[Bro] Version: 2.0-907 -- Bro manager memory exhaustion

Scott Campbell scampbell at lbl.gov
Wed Aug 1 13:59:09 PDT 2012


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I have noticed that there seems to be a large volume of "soft"
information for high performance bro installations.  This is
particularly true with PF_RING/DNA/libzero configuration and use.
Finding nice simple examples are also a bit scarce.

My proposal to address this is as follows: For anybody willing to
document this info - something as simple as a cut and paste of the
PF_RING/ixgbe startup configs and the node.cfg - I will purchase a
beer*.  We will put this info up on the Bro web site for the
edification of those staring into the abyss of 10 (or 100!) G.

That is it.  This project is personal and not funded or supported by
ICSI or anybody else for that matter.  Can you imagine the IRB?

cheers!
scott

* Till I run out of money dedicated to the Bro documentation liquidity
fund.  :-)

On 8/1/12 3:00 PM, Tritium Cat wrote:
> On Wed, Aug 1, 2012 at 1:23 AM, Vlad Grigorescu <vladg at cmu.edu 
> <mailto:vladg at cmu.edu>> wrote:
> 
>> I based the design on the very rough suggestion each core will
>> handle 60-120 Mbps of traffic.  So 20 workers on a 48 core would
>> be 1.2
> Gbps to
>> 2.4 Gbps.  I want to say I even recall reading about that
>> threshold
> in a
>> paper somewhere.
> 
> As you mentioned, that's a very rough estimate. There are several 
> factors in play here, including:
> 
> 1) What you're loading into Bro, 2) What your traffic looks like, 
> 3) Resource contention, 4) Disk I/O, 5) Tuning some script-level
> optimizations.
> 
> I think that what you're running into is resource contention - I
> believe that the 80-150 Mbps/core guideline is based on empirical
> evidence with 4-8 workers/system. AFAIK, running more than 12 or so
> workers/system is venturing into not-well-knonwn territory (I know
> of one setup that start experiencing stability and performance
> issues moving past 12 workers/system). I also don't think that that
> guideline is really linear - I've analyzed ~1 Gbps with PF_RING and
> 8 workers (albeit without any scripts that do MD5 hashing).
> 
> Additionally, once you start talking about so many workers, I think
> that your disk I/O on the manager will be a real issue - that's
> simply a lot of data to write out to disk.
> 
> 
> You might be right.  I had considered disk I/O and ran the manager
> on a decent raid-10 array (the elsa server), disk i/o did not
> appear to be the problem so much as CPU and memory.  That
> observation carried over to development builds with a threaded
> manager, but at an accelerated rate; right now I'm watching the
> disk I/O under threshold and the manager is consuming 100M of
> memory per second until memory exhaustion.
> 
> My configuration is default with Conn::Log turned off.  I'm still
> trying to obtain an idea of the performance baseline before adding
> additional scripts.
> 
> For my current setup it seems to work best if I segment the cluster
> so I'll continue with that approach.
> 
> --TC
> 
> 
> 
> _______________________________________________ Bro mailing list 
> bro at bro-ids.org 
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFQGZidK2Plq8B7ZBwRAm3fAKCeVvu8jsSW3B5qmbcExWOWqA55UQCfaMhJ
dMvW/6xUyqnS2TxWzvjDGoU=
=hRud
-----END PGP SIGNATURE-----



More information about the Bro mailing list