[Bro] Warning: "Bro node ... possibly still running"

[Daniel Thayer]

In order to prevent this problem, you should run "broctl stop"
before removing (or renaming) any nodes in your node.cfg.


On 11/18/16 6:44 AM, Fernandez, Mark I wrote:
> *_Issue #1_*: My node.cfg file specifies “type=standalone”, but I get a
> BroCtl warning that “Bro node ‘worker-1’ possibly still running on host…”.
>
>
>
> Operating on Bro 2.4.1 and BroControl 1.4.
>
>
>
> *_Background_*:
>
> I configured a local cluster with one manager, one proxy, and two
> workers.  Worker-1 is monitoring eth1, and worker-2 is monitoring eth2.
> The host was suffering too much packet loss, as indicated in the
> notice.log with the messages “PacketFilter::Dropped_Packets” and
> “CaptureLoss::Too_Much_Loss”.  Therefore, I backed down from a local
> cluster, to just a standalone configuration in node.cfg.  First,
> monitored only eth1 for a few days to observe packet loss, and then
> changed to monitor only eth2 for a few days.  When I edit node.cfg and
> then run broctl, I get the following warnings:
>
>
>
> Warning: broctl node config has changed (run the broctl “deploy” command)
>
> Warning: Bro node “worker-1” possibly still running on host “localhost”
> (PID www)
>
> Warning: Bro node “worker-2” possibly still running on host “localhost”
> (PID xxx)
>
> Warning: Bro node “proxy” possibly still running on host “localhost”
> (PID yyy)
>
> Warning: Bro node “manager” possibly still running on host “localhost”
> (PID zzz)
>
>
>
> This is very curious that broctl “remembers” the previous node.cfg
> settings.  Of course, none of the PIDs are valid anymore, because those
> processes were terminated when I changed from a cluster to standalone.
>  But for some reason, broctl believes these processes might still be
> running.  Where does BroCtl store this information?
>
>
>
> *_Issue #2_*: Originally, when I changed node.cfg back to standalone,
> and then ran BroCtl “deploy” to implement the new configuration, the
> original manager, proxy, and worker processes were not terminated.
> BroCtl left these processes running, and then started a new set of
> processes for the new config.  I discovered this a few days later
> because the notice.logs had entries from “bro” (standalone), and still
> was getting entries from “worker-1” and “worker-2” even though the
> cluster configuration was removed two days prior.  I would run BroCtl
> “nodes” and it would correctly show that Bro is standalone monitoring
> eth1 only.  I was confused.  Finally, I ran process list on the host,
> and it revealed the original manager, proxy, and workers were all still
> running.  To clear the situation, I ran BroCtl “stop”, then ran “kill
> -9” on every Bro-related PID, and then ran BroCtl “deploy”.  This
> cleared away the issue of “worker-1” and “worker-2” from writing to the
> notice.logs; however, I still observe *_Issue #1_*, where BroCtl gives
> the warning messages that “Warning: Bro node ... possibly still running".
>
>
>
> I have a crontab to run BroCtl “cron” every five minutes.  Does BroCtl
> “cron” affect how various configs are “remembered”?  Should I disable
> that crontab item before making any changes to node.cfg and/or before
> running BroCtl “deploy”?
>
>
>
>
>
> Thanks!
>
> *Mark I. Fernandez*
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>