[Bro] Manager and logger threads crash immediately on deploy

[Mike Dopheide]

Is that (8 proxies and 32 workers) on ALL 8 hosts?   For 64 total proxies?

That seems like a lot to me.

On Wed, Jul 12, 2017 at 1:44 PM Chris Herdt <cherdt at umn.edu> wrote:

> I've successfully run smaller Bro clusters, but now that I'm scaling out
> I'm seeing the manager and logger threads crash immediately when I deploy
> the configuration.
>
> What I'm trying to run:
> - 1 manager, 1 logger on 1 host
> - 8 proxies and 32 workers on 8 hosts
>
> I'm using Bro 2.5.1. Each worker host has 2 Myricom 10G NICs w/2 ports
> each, using the 3.0.10 Myricom SNF driver. I'm attempting to run 9
> processes (lb_procs) per worker node, each pinned to its own CPU core.
>
> What I'm finding is that any time the number of worker processes exceeds
> ~160 (not a magic number--not consistent, but around that value based on
> observation), the manager and logger threads crash. If I keep the number of
> worker processes at or below ~160 (either by reducing processes per node,
> reducing nodes per host, or reducing hosts in the cluster) it runs
> successfully. Ideally, the cluster would have 288 worker processes.
>
> This does not seem to be related packet volume, as the manager and logger
> threads crash even if I am not sending any traffic to the worker nodes.
>
> Any troubleshooting or optimization suggestions are appreciated.
>
> --
> Chris Herdt
> Systems Administrator
> University of Minnesota
> cherdt at umn.edu
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170712/cbfcec3b/attachment-0001.html