[Bro] Warning: "Bro node ... possibly still running"

Fernandez, Mark I mfernandez at mitre.org
Fri Nov 18 06:44:31 PST 2016


Daniel,

Thank you.  To clarify, I should run broctl stop before I even edit the node.cfg file?  I did not do so the first time.  Bro was still running, I edited node.cfg, then ran broctl deploy.  Indeed, while I was troubleshooting this issue, I tried every variation.  I would run broctl stop, then edit node.cfg, then broctl deploy.  This had no effect on the original manager, proxy and worker processes; and the only way to terminate these processes was to run "kill -9".  Even earlier this morning, I ran broctl stop, edited node.cfg, and when I ran broctl, it gave the warnings.  

Now that the damage is done, how do I undo this condition?  I believe the system is monitoring and logging as intended, but for trust and confidence in the system state, I would like to clear away these warnings.  Any advice on how to do so?

Mark


-----Original Message-----
From: Daniel Thayer [mailto:dnthayer at illinois.edu] 
Sent: Friday, November 18, 2016 9:02 AM
To: Fernandez, Mark I <mfernandez at mitre.org>; bro at bro.org
Subject: Re: [Bro] Warning: "Bro node ... possibly still running"

In order to prevent this problem, you should run "broctl stop"
before removing (or renaming) any nodes in your node.cfg.


On 11/18/16 6:44 AM, Fernandez, Mark I wrote:
> *_Issue #1_*: My node.cfg file specifies "type=standalone", but I get a
> BroCtl warning that "Bro node 'worker-1' possibly still running on host...".
>
>
>
> Operating on Bro 2.4.1 and BroControl 1.4.
>
>
>
> *_Background_*:
>
> I configured a local cluster with one manager, one proxy, and two
> workers.  Worker-1 is monitoring eth1, and worker-2 is monitoring eth2.
> The host was suffering too much packet loss, as indicated in the
> notice.log with the messages "PacketFilter::Dropped_Packets" and
> "CaptureLoss::Too_Much_Loss".  Therefore, I backed down from a local
> cluster, to just a standalone configuration in node.cfg.  First,
> monitored only eth1 for a few days to observe packet loss, and then
> changed to monitor only eth2 for a few days.  When I edit node.cfg and
> then run broctl, I get the following warnings:
>
>
>
> Warning: broctl node config has changed (run the broctl "deploy" command)
>
> Warning: Bro node "worker-1" possibly still running on host "localhost"
> (PID www)
>
> Warning: Bro node "worker-2" possibly still running on host "localhost"
> (PID xxx)
>
> Warning: Bro node "proxy" possibly still running on host "localhost"
> (PID yyy)
>
> Warning: Bro node "manager" possibly still running on host "localhost"
> (PID zzz)
>
>
>
> This is very curious that broctl "remembers" the previous node.cfg
> settings.  Of course, none of the PIDs are valid anymore, because those
> processes were terminated when I changed from a cluster to standalone.
>  But for some reason, broctl believes these processes might still be
> running.  Where does BroCtl store this information?
>
>
>
> *_Issue #2_*: Originally, when I changed node.cfg back to standalone,
> and then ran BroCtl "deploy" to implement the new configuration, the
> original manager, proxy, and worker processes were not terminated.
> BroCtl left these processes running, and then started a new set of
> processes for the new config.  I discovered this a few days later
> because the notice.logs had entries from "bro" (standalone), and still
> was getting entries from "worker-1" and "worker-2" even though the
> cluster configuration was removed two days prior.  I would run BroCtl
> "nodes" and it would correctly show that Bro is standalone monitoring
> eth1 only.  I was confused.  Finally, I ran process list on the host,
> and it revealed the original manager, proxy, and workers were all still
> running.  To clear the situation, I ran BroCtl "stop", then ran "kill
> -9" on every Bro-related PID, and then ran BroCtl "deploy".  This
> cleared away the issue of "worker-1" and "worker-2" from writing to the
> notice.logs; however, I still observe *_Issue #1_*, where BroCtl gives
> the warning messages that "Warning: Bro node ... possibly still running".
>
>
>
> I have a crontab to run BroCtl "cron" every five minutes.  Does BroCtl
> "cron" affect how various configs are "remembered"?  Should I disable
> that crontab item before making any changes to node.cfg and/or before
> running BroCtl "deploy"?
>
>
>
>
>
> Thanks!
>
> *Mark I. Fernandez*
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



More information about the Bro mailing list