[Bro] cluster manager crash

[Hovsep Levi]

Bowen,

This is what I call an architecture limitation of Bro; it's well known but
not really formally acknowledged, you can read the archives and perceive.
if you use faster CPUs it will mask the problem by using less workers (in
theory).  I'm not sure where the ideal worker threshold is and it's
relation to events per second.

Some people avoid this issue by segmenting the cluster per server; you lose
some functionality but at least your cluster runs without incident.
(mostly).  Example:  a four server bro cluster becomes four bro clusters,
each running it's own manager and writing to local disk.  If you're just
using Bro as a network recorder this is a fairly even trade off.

I've never had a day where Bro didn't crash due to memory exhaustion and
have to perform full restarts once per hour to prevent manager crashes.
The only way to fix it is to become a Bro developer. :>


Regards,

Hovsep



On Wed, Sep 7, 2016 at 1:44 PM, Bowen Li <newfire.bw at gmail.com> wrote:

> Hi all,
>     I have an issue about cluster manager crash when lots of log event
> send to it.
>     I set up a bro cluster  on my server, the cluster have 32 workers and
> 1 proxy and handle about 5Gb/s. After run about one and a half hour, the
> cluster no longer produces logs, but workers still extracts files. So it
> seems that the manager was crashed.
>     Is there any possibility that the manager doesn't work anymore when
> workers send lots of log event? If so, what`s the limit of the log event?
> Or maybe the issue won`t happen if I run a real cluster on several servers?
>     By the way, if I want to handle 10Gb/s, how much memory should I leave
> for each worker ? If I do memory usage restrictions, will it affect
> the performance of the cluster?
>     Any insight would be helpful.
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160928/d5441039/attachment.html