[Bro] Manager and logger threads crash immediately on deploy

Mike Dopheide dopheide at gmail.com
Wed Jul 12 12:32:19 PDT 2017


Okay, I think I'm following now, but I want to restate it so that other's
with more large cluster experience can chime in.

1 physical host = manager and logger
8 x physical hosts = proxy + 4*(worker w/ 9lb_procs)

I'm not sure if having multiple worker nodes per physical host is all that
common.  I assume you're doing that so each 'worker' node only monitors one
of your four 10G links per physical host.  Ignoring the actual traffic
capture aspect, have you tried running only one worker node w/ 36 lb_procs
per physical host?

-Dop




On Wed, Jul 12, 2017 at 2:01 PM, Chris Herdt <cherdt at umn.edu> wrote:

> On Wed, Jul 12, 2017 at 1:56 PM, Mike Dopheide <dopheide at gmail.com> wrote:
>
>> Is that (8 proxies and 32 workers) on ALL 8 hosts?   For 64 total proxies?
>>
>> That seems like a lot to me.
>>
>> On Wed, Jul 12, 2017 at 1:44 PM Chris Herdt <cherdt at umn.edu> wrote:
>>
>>> I've successfully run smaller Bro clusters, but now that I'm scaling out
>>> I'm seeing the manager and logger threads crash immediately when I deploy
>>> the configuration.
>>>
>>> What I'm trying to run:
>>> - 1 manager, 1 logger on 1 host
>>> - 8 proxies and 32 workers on 8 hosts
>>>
>>> I'm using Bro 2.5.1. Each worker host has 2 Myricom 10G NICs w/2 ports
>>> each, using the 3.0.10 Myricom SNF driver. I'm attempting to run 9
>>> processes (lb_procs) per worker node, each pinned to its own CPU core.
>>>
>>> What I'm finding is that any time the number of worker processes exceeds
>>> ~160 (not a magic number--not consistent, but around that value based on
>>> observation), the manager and logger threads crash. If I keep the number of
>>> worker processes at or below ~160 (either by reducing processes per node,
>>> reducing nodes per host, or reducing hosts in the cluster) it runs
>>> successfully. Ideally, the cluster would have 288 worker processes.
>>>
>>> This does not seem to be related packet volume, as the manager and
>>> logger threads crash even if I am not sending any traffic to the worker
>>> nodes.
>>>
>>> Any troubleshooting or optimization suggestions are appreciated.
>>>
>>>
>
> 8 total proxies, 32 total workers.
> (1 proxy node + 4 worker nodes) * 8 hosts
>
>
> --
> Chris Herdt
> Systems Administrator
> University of Minnesota
> cherdt at umn.edu
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170712/28b57ab0/attachment.html 


More information about the Bro mailing list