[Bro] BROKER + CLUSTER - stuck (Mike Dopheide)

[Daniel Thayer]

When "broctl deploy" hangs, were there any bro processes
running (before you ran "deploy")?

Also, when you ran "stop+check+install+start", which
command hangs?  Also, in this case, did you verify that
all bro processes were stopped by the "stop" command?



On 3/8/17 3:52 PM, fatema bannatwala wrote:
> Cool! Thanks Justin for the explanation :)
>
> Tried using stop first+deploy but hangs and never completes.
> Also tried doing stop+check+install+start, but hangs again.
>
> Hence, ended up doing install+restart, to bring back the cluster up. :(
> The manager doesn't look overloaded though.
> I think next will try out the check.bro change you suggested at the
> beginning, and see if that helps..
>
> Thanks!
>
> On Wed, Mar 8, 2017 at 3:56 PM, Azoff, Justin S <jazoff at illinois.edu
> <mailto:jazoff at illinois.edu>> wrote:
>
>
>     > On Mar 8, 2017, at 3:39 PM, fatema bannatwala <fatema.bannatwala at gmail.com <mailto:fatema.bannatwala at gmail.com>> wrote:
>     >
>     > Hmm, looks like the manager is also running with low memory:
>     > $ free -g
>     >               total        used        free      shared  buff/cache   available
>     > Mem:             70           9          46           0          14          60
>     > Swap:             7           4           3
>     >
>
>     That's fine.. you have 46G of ram free.
>
>     > $ top
>     > top - 15:30:04 up 47 days, 22:33,  8 users,  load average: 1.26, 1.25, 1.37
>     > Tasks: 495 total,   2 running, 491 sleeping,   2 stopped,   0 zombie
>     > %Cpu(s):  2.9 us,  1.7 sy,  0.4 ni, 94.9 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
>     > KiB Mem : 73949688 total, 48927592 free,  9836872 used, 15185224 buff/cache
>     > KiB Swap:  8388600 total,  4192868 free,  4195732 used. 63369176 avail Mem
>     >
>     >
>     > Anyways, not going into that rabbit hole :)
>     > So the correct sequence to deploy any config changes in a cluster would be:
>     > stop -> check -> install -> start
>     > I was looking at the cmds available and looks like "restart --clean" would do the trick?
>     > or I can just script the above sequence in my restart-bro script :)
>
>     Yeah.. deploy basically automates check+install+restart.
>
>     stop+check+install+start is basically the same as stop+deploy.
>
>     The reason why deploy does check first is because if you
>     accidentally broke your configuration, you need to start it again
>     with the old configuration, fix the configuration, and then retry -
>     effectively wasting a restart.
>
>     Doing check first means that you don't stop bro unless you know the
>     new configuration will work.
>
>     The worst thing you can do is stop+install+check .  If you do that,
>     you can end up with a broken bro installation that needs to be fixed
>     before you can start it up again.  A lot of people were doing this
>     which is why added deploy.
>
>
>
>
>     --
>     - Justin Azoff
>
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>