[Bro] BROKER + CLUSTER - stuck (Mike Dopheide)

[fatema bannatwala]

There were bro processes running before I used deploy cmd ( manager acts as
Proxies-4, logger as well as manager, so around 5-6 bro processes run on
manager machine).

When I ran stop+check+install+start, all the bro processes stopped properly
(atleast that's what console output said), and it hung on check.

After restoring the working cluster, I ran 'broctl check' on manager just
to see what it does when run standalone, while the cluster is running:
it just never completed, and turned out, after couple of minutes, that the
manager started swapping and hung, so I had to run 'sudo kill -9 bro'
to get back the hold of machine, then restarted bro normally by doing a
'install and restart'.. :/

Thanks,
Fatema.

On Wed, Mar 8, 2017 at 5:03 PM, Daniel Thayer <dnthayer at illinois.edu> wrote:

> When "broctl deploy" hangs, were there any bro processes
> running (before you ran "deploy")?
>
> Also, when you ran "stop+check+install+start", which
> command hangs?  Also, in this case, did you verify that
> all bro processes were stopped by the "stop" command?
>
>
>
> On 3/8/17 3:52 PM, fatema bannatwala wrote:
>
>> Cool! Thanks Justin for the explanation :)
>>
>> Tried using stop first+deploy but hangs and never completes.
>> Also tried doing stop+check+install+start, but hangs again.
>>
>> Hence, ended up doing install+restart, to bring back the cluster up. :(
>> The manager doesn't look overloaded though.
>> I think next will try out the check.bro change you suggested at the
>> beginning, and see if that helps..
>>
>> Thanks!
>>
>> On Wed, Mar 8, 2017 at 3:56 PM, Azoff, Justin S <jazoff at illinois.edu
>> <mailto:jazoff at illinois.edu>> wrote:
>>
>>
>>
>>     > On Mar 8, 2017, at 3:39 PM, fatema bannatwala <
>> fatema.bannatwala at gmail.com <mailto:fatema.bannatwala at gmail.com>> wrote:
>>     >
>>     > Hmm, looks like the manager is also running with low memory:
>>     > $ free -g
>>     >               total        used        free      shared
>> buff/cache   available
>>     > Mem:             70           9          46           0
>> 14          60
>>     > Swap:             7           4           3
>>     >
>>
>>     That's fine.. you have 46G of ram free.
>>
>>     > $ top
>>     > top - 15:30:04 up 47 days, 22:33,  8 users,  load average: 1.26,
>> 1.25, 1.37
>>     > Tasks: 495 total,   2 running, 491 sleeping,   2 stopped,   0 zombie
>>     > %Cpu(s):  2.9 us,  1.7 sy,  0.4 ni, 94.9 id,  0.1 wa,  0.0 hi,  0.0
>> si,  0.0 st
>>     > KiB Mem : 73949688 total, 48927592 free,  9836872 used, 15185224
>> buff/cache
>>     > KiB Swap:  8388600 total,  4192868 free,  4195732 used. 63369176
>> avail Mem
>>     >
>>     >
>>     > Anyways, not going into that rabbit hole :)
>>     > So the correct sequence to deploy any config changes in a cluster
>> would be:
>>     > stop -> check -> install -> start
>>     > I was looking at the cmds available and looks like "restart
>> --clean" would do the trick?
>>     > or I can just script the above sequence in my restart-bro script :)
>>
>>     Yeah.. deploy basically automates check+install+restart.
>>
>>     stop+check+install+start is basically the same as stop+deploy.
>>
>>     The reason why deploy does check first is because if you
>>     accidentally broke your configuration, you need to start it again
>>     with the old configuration, fix the configuration, and then retry -
>>     effectively wasting a restart.
>>
>>     Doing check first means that you don't stop bro unless you know the
>>     new configuration will work.
>>
>>     The worst thing you can do is stop+install+check .  If you do that,
>>     you can end up with a broken bro installation that needs to be fixed
>>     before you can start it up again.  A lot of people were doing this
>>     which is why added deploy.
>>
>>
>>
>>
>>     --
>>     - Justin Azoff
>>
>>
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/af602150/attachment.html