[Bro] Version: 2.0-907 -- Bro manager memory exhaustion

[Tritium Cat]

On Wed, Aug 1, 2012 at 1:23 AM, Vlad Grigorescu <vladg at cmu.edu> wrote:

> > I based the design on the very rough suggestion each core will handle
> >60-120 Mbps of traffic.  So 20 workers on a 48 core would be 1.2 Gbps to
> >2.4 Gbps.  I want to say I even recall reading about that threshold in a
> >paper somewhere.
>
> As you mentioned, that's a very rough estimate. There are several factors
> in play here, including:
>
> 1) What you're loading into Bro,
> 2) What your traffic looks like,
> 3) Resource contention,
> 4) Disk I/O,
> 5) Tuning some script-level optimizations.
>
> I think that what you're running into is resource contention - I believe
> that the 80-150 Mbps/core guideline is based on empirical evidence with
> 4-8 workers/system. AFAIK, running more than 12 or so workers/system is
> venturing into not-well-knonwn territory (I know of one setup that start
> experiencing stability and performance issues moving past 12
> workers/system). I also don't think that that guideline is really linear -
> I've analyzed ~1 Gbps with PF_RING and 8 workers (albeit without any
> scripts that do MD5 hashing).
>
> Additionally, once you start talking about so many workers, I think that
> your disk I/O on the manager will be a real issue - that's simply a lot of
> data to write out to disk.
>
>
You might be right.  I had considered disk I/O and ran the manager on a
decent raid-10 array (the elsa server), disk i/o did not appear to be the
problem so much as CPU and memory.  That observation carried over to
development builds with a threaded manager, but at an accelerated rate;
right now I'm watching the disk I/O under threshold and the manager is
consuming 100M of memory per second until memory exhaustion.

My configuration is default with Conn::Log turned off.  I'm still trying to
obtain an idea of the performance baseline before adding additional scripts.

For my current setup it seems to work best if I segment the cluster so I'll
continue with that approach.

--TC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120801/57435107/attachment.html