[Bro] Bro with 10Gb NIC's or higher

Michał Purzyński michalpurzynski1 at gmail.com
Fri Jan 9 12:01:33 PST 2015


Do you really see and can handle 1Gbit/sec of traffic per core? I'm curious.

I would say, with a 2.6Ghz CPU my educated guess would be somewhere
about 250Mbit/sec / core with Bro. Of course configuration is
everything here, I'm just looking into "given you do it right, that's
what's possible".

On Fri, Jan 9, 2015 at 8:00 PM, Aashish Sharma <asharma at lbl.gov> wrote:
> While, we at LBNL continue to work towards a formal documentation, I think I'd reply then causing further delays:
>
> Here is the 100G cluster setup we've done:
>
> - 5 nodes running 10 workers + 1 proxy each on them
> - 100G split by arista to 5x10G
> - 10G on each node is further split my myricom to 10x1G/worker with shunting enabled !!
>
> Note: Scott Campbell did some very early work on the concept of shunting
>             (http://dl.acm.org/citation.cfm?id=2195223.2195788)
>
> We are using react-framework to talk to arista written by Justin Azoff.
>
> With Shunting enabled cluster isn't even truly seeing 10G anymore.
>
> oh btw, Capture_loss is a good policy to run for sure. With above setup we get ~ 0.xx % packet drops.
>
> (Depending on kind of traffic you are monitoring you may need a slightly different shunting logic)
>
>
> Here is hardware specs / node:
>
> - Motherboard-SM, X9DRi-F
> - Intel E5-2643V2 3.5GHz Ivy Bridge (2x6-=12 Cores)
> - 128GB DDRIII 1600MHz ECC/REG - (8x16GB Modules Installed)
> - 10G-PCIE2-8C2-2S+; Myricom 10G "Gen2" (5 GT/s) PCI Express NIC with two SFP+
> -  Myricom 10G-SR Modules
>
> On tapping side we have
> - Arista 7504  (gets fed 100G TX/RX + backup and other 10Gb links)
> - Arista 7150 (Symetric hashing via DANZ - splitting tcp sessions 1/link - 5 links to nodes
>
> on Bro side:
> 5 nodes accepting 5 links from 7150
> Each node running 10 workers + 1 proxy
> Myricom spliting/load balancing to each worker on the node.
>
>
> Hope this helps,
>
> let us know if you have any further questions.
>
> Thanks,
> Aashish
>
> On Fri, Jan 09, 2015 at 06:20:17PM +0000, Mike Patterson wrote:
>> You're right, it's 32 on mine.
>>
>> I posted some specs for my system a couple of years ago now, I think.
>>
>> 6-8GB per worker should give some headroom (my workers usually use about 5 apiece I think).
>>
>> Mike
>>
>> --
>> Simple, clear purpose and principles give rise to complex and
>> intelligent behavior. Complex rules and regulations give rise
>> to simple and stupid behavior. - Dee Hock
>>
>> > On Jan 9, 2015, at 1:03 PM, Donaldson, John <donaldson8 at llnl.gov> wrote:
>> >
>> > I'd agree with all of this. We're monitoring a few 10Gbps network segments with DAG 9.2X2s, too. I'll add in that, when processing that much traffic on a single device, you'll definitely not want to skimp on memory.
>> >
>> > I'm not sure which configurations you're using that might be limiting you to 16 streams -- we're  run with at least 24 streams, and (at least with the 9.2X2s) you should be able to work with up to 32 receive streams.
>> >
>> > v/r
>> >
>> > John Donaldson
>> >
>> >> -----Original Message-----
>> >> From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of
>> >> Mike Patterson
>> >> Sent: Thursday, January 08, 2015 7:29 AM
>> >> To: coen bakkers
>> >> Cc: bro at bro.org
>> >> Subject: Re: [Bro] Bro with 10Gb NIC's or higher
>> >>
>> >> Succinctly, yes, although that provision is a big one.
>> >>
>> >> I'm running Bro on two 10 gig interfaces, an Intel X520 and an Endace DAG
>> >> 9.2X2. Both perform reasonably well. Although my hardware is somewhat
>> >> underspecced (Dell R710s of differing vintages), I still get tons of useful data.
>> >>
>> >> If your next question would be "how should I spec my hardware", that's
>> >> quite difficult to answer because it depends on a lot. Get the hottest CPUs
>> >> you can afford, with as many cores. If you're actually sustaining 10+Gb you'll
>> >> probably want at least 20-30 cores. I'm sustaining 4.5Gb or so on 8 3.7Ghz
>> >> cores, but Bro reports 10% or so loss. Note that some hardware
>> >> configurations will limit the number of streams you can feed to Bro, eg my
>> >> DAG can only produce 16 streams so even if I had it in a 24 core box, I'd only
>> >> be making use of 2/3 of my CPU.
>> >>
>> >> Mike
>> >>
>> >>> On Jan 7, 2015, at 5:04 AM, coen bakkers <cbakkers at yahoo.de> wrote:
>> >>>
>> >>> Does anyone have experience with higher speed NIC's and Bro? Will it
>> >> sustain 10Gb speeds or more provide the hardware is spec'd appropriately?
>> >>>
>> >>> regards,
>> >>>
>> >>> Coen
>> >>> _______________________________________________
>> >>> Bro mailing list
>> >>> bro at bro-ids.org
>> >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> >>
>> >>
>> >> _______________________________________________
>> >> Bro mailing list
>> >> bro at bro-ids.org
>> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> --
> Aashish Sharma  (asharma at lbl.gov)
> Cyber Security,
> Lawrence Berkeley National Laboratory
> http://go.lbl.gov/pgp-aashish
> Office: (510)-495-2680  Cell: (510)-612-7971
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


More information about the Bro mailing list