[Bro] Version: 2.0-907 -- Bro manager memory exhaustion
tritium.cat at gmail.com
Wed Aug 1 13:00:59 PDT 2012
On Wed, Aug 1, 2012 at 1:23 AM, Vlad Grigorescu <vladg at cmu.edu> wrote:
> > I based the design on the very rough suggestion each core will handle
> >60-120 Mbps of traffic. So 20 workers on a 48 core would be 1.2 Gbps to
> >2.4 Gbps. I want to say I even recall reading about that threshold in a
> >paper somewhere.
> As you mentioned, that's a very rough estimate. There are several factors
> in play here, including:
> 1) What you're loading into Bro,
> 2) What your traffic looks like,
> 3) Resource contention,
> 4) Disk I/O,
> 5) Tuning some script-level optimizations.
> I think that what you're running into is resource contention - I believe
> that the 80-150 Mbps/core guideline is based on empirical evidence with
> 4-8 workers/system. AFAIK, running more than 12 or so workers/system is
> venturing into not-well-knonwn territory (I know of one setup that start
> experiencing stability and performance issues moving past 12
> workers/system). I also don't think that that guideline is really linear -
> I've analyzed ~1 Gbps with PF_RING and 8 workers (albeit without any
> scripts that do MD5 hashing).
> Additionally, once you start talking about so many workers, I think that
> your disk I/O on the manager will be a real issue - that's simply a lot of
> data to write out to disk.
You might be right. I had considered disk I/O and ran the manager on a
decent raid-10 array (the elsa server), disk i/o did not appear to be the
problem so much as CPU and memory. That observation carried over to
development builds with a threaded manager, but at an accelerated rate;
right now I'm watching the disk I/O under threshold and the manager is
consuming 100M of memory per second until memory exhaustion.
My configuration is default with Conn::Log turned off. I'm still trying to
obtain an idea of the performance baseline before adding additional scripts.
For my current setup it seems to work best if I segment the cluster so I'll
continue with that approach.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Bro