[Bro] BROKER + CLUSTER - stuck (Mike Dopheide)

[fatema bannatwala]

Thanks Justin for the input!
Yeah, you are right, tested the deploy cmd on a standalone node, and it
does not hang there.
I will test out the check.bro suggestions on the prod cluster.

The cluster nodes use an average of ~30-35Gigs of memory (having ~125G in
total)
And the capture loss also doesn't report any loss i.e 0.025% etc
Hence thought that the nodes were doing Ok, not sure if they are getting
loads of traffic and hence getting overloaded.

Also, I have noticed that when doing a restart on the cluster, it takes
longer now (in 2.5) than it used to take when running the old version
(2.4.1),
maybe the custom scripts can be the culprit, but had same scripts in the
old version as well.


On Wed, Mar 8, 2017 at 2:42 PM, Azoff, Justin S <jazoff at illinois.edu> wrote:

>
> > On Mar 8, 2017, at 2:32 PM, fatema bannatwala <
> fatema.bannatwala at gmail.com> wrote:
> >
> > Thanks Mike for clarification. Does it means I can use it and redef it
> in my local.bro ?
> > I was thinking is there anyone else encountering same issues with deploy
> cmd, and
> > if it can be added to Bro documentation on how to solve the hanging
> issue.
> > Because thinking of it as a normal user perspective: I upgrade the Bro
> cluster to 2.5, and when use broctl deploy, it just hangs there, while user
> having no clue why it would happen.
> >
> > Thanks,
> > Fatema.
>
> I used to see this issue on heavily overloaded clusters, but haven't been
> able to reproduce it in a while.  All check does is run a sc
>
> One thing you could try is changing the check.bro script that comes with
> broctl to do
>
> terminate();
>
> instead of
>
> terminate_communication();
>
>
> I think Daniel also had a branch of broctl that just used bro -a instead
> of the check.bro script.
>
> --
> - Justin Azoff
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/b704d84c/attachment.html