[Bro] Warning: "Bro node ... possibly still running"

[Fernandez, Mark I]

Issue #1: My node.cfg file specifies "type=standalone", but I get a BroCtl warning that "Bro node 'worker-1' possibly still running on host...".

Operating on Bro 2.4.1 and BroControl 1.4.

Background:
I configured a local cluster with one manager, one proxy, and two workers.  Worker-1 is monitoring eth1, and worker-2 is monitoring eth2.  The host was suffering too much packet loss, as indicated in the notice.log with the messages "PacketFilter::Dropped_Packets" and "CaptureLoss::Too_Much_Loss".  Therefore, I backed down from a local cluster, to just a standalone configuration in node.cfg.  First, monitored only eth1 for a few days to observe packet loss, and then changed to monitor only eth2 for a few days.  When I edit node.cfg and then run broctl, I get the following warnings:

Warning: broctl node config has changed (run the broctl "deploy" command)
Warning: Bro node "worker-1" possibly still running on host "localhost" (PID www)
Warning: Bro node "worker-2" possibly still running on host "localhost" (PID xxx)
Warning: Bro node "proxy" possibly still running on host "localhost" (PID yyy)
Warning: Bro node "manager" possibly still running on host "localhost" (PID zzz)

This is very curious that broctl "remembers" the previous node.cfg settings.  Of course, none of the PIDs are valid anymore, because those processes were terminated when I changed from a cluster to standalone.  But for some reason, broctl believes these processes might still be running.  Where does BroCtl store this information?

Issue #2: Originally, when I changed node.cfg back to standalone, and then ran BroCtl "deploy" to implement the new configuration, the original manager, proxy, and worker processes were not terminated.  BroCtl left these processes running, and then started a new set of processes for the new config.  I discovered this a few days later because the notice.logs had entries from "bro" (standalone), and still was getting entries from "worker-1" and "worker-2" even though the cluster configuration was removed two days prior.  I would run BroCtl "nodes" and it would correctly show that Bro is standalone monitoring eth1 only.  I was confused.  Finally, I ran process list on the host, and it revealed the original manager, proxy, and workers were all still running.  To clear the situation, I ran BroCtl "stop", then ran "kill -9" on every Bro-related PID, and then ran BroCtl "deploy".  This cleared away the issue of "worker-1" and "worker-2" from writing to the notice.logs; however, I still observe Issue #1, where BroCtl gives the warning messages that "Warning: Bro node ... possibly still running".

I have a crontab to run BroCtl "cron" every five minutes.  Does BroCtl "cron" affect how various configs are "remembered"?  Should I disable that crontab item before making any changes to node.cfg and/or before running BroCtl "deploy"?


Thanks!
Mark I. Fernandez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161118/c25a9c36/attachment-0001.html