[Bro] Bro with 10Gb NIC's or higher

[Aashish Sharma]

While, we at LBNL continue to work towards a formal documentation, I think I'd reply then causing further delays:

Here is the 100G cluster setup we've done:

- 5 nodes running 10 workers + 1 proxy each on them 
- 100G split by arista to 5x10G 
- 10G on each node is further split my myricom to 10x1G/worker with shunting enabled !!

Note: Scott Campbell did some very early work on the concept of shunting
	    (http://dl.acm.org/citation.cfm?id=2195223.2195788) 

We are using react-framework to talk to arista written by Justin Azoff. 

With Shunting enabled cluster isn't even truly seeing 10G anymore.

oh btw, Capture_loss is a good policy to run for sure. With above setup we get ~ 0.xx % packet drops. 

(Depending on kind of traffic you are monitoring you may need a slightly different shunting logic) 


Here is hardware specs / node: 

- Motherboard-SM, X9DRi-F
- Intel E5-2643V2 3.5GHz Ivy Bridge (2x6-=12 Cores) 
- 128GB DDRIII 1600MHz ECC/REG - (8x16GB Modules Installed)
- 10G-PCIE2-8C2-2S+; Myricom 10G "Gen2" (5 GT/s) PCI Express NIC with two SFP+
-  Myricom 10G-SR Modules 

On tapping side we have 
- Arista 7504  (gets fed 100G TX/RX + backup and other 10Gb links)
- Arista 7150 (Symetric hashing via DANZ - splitting tcp sessions 1/link - 5 links to nodes

on Bro side:
5 nodes accepting 5 links from 7150 
Each node running 10 workers + 1 proxy 
Myricom spliting/load balancing to each worker on the node. 


Hope this helps, 

let us know if you have any further questions. 

Thanks, 
Aashish 

On Fri, Jan 09, 2015 at 06:20:17PM +0000, Mike Patterson wrote:
> You're right, it's 32 on mine.
> 
> I posted some specs for my system a couple of years ago now, I think.
> 
> 6-8GB per worker should give some headroom (my workers usually use about 5 apiece I think).
> 
> Mike
> 
> -- 
> Simple, clear purpose and principles give rise to complex and
> intelligent behavior. Complex rules and regulations give rise
> to simple and stupid behavior. - Dee Hock
> 
> > On Jan 9, 2015, at 1:03 PM, Donaldson, John <donaldson8 at llnl.gov> wrote:
> > 
> > I'd agree with all of this. We're monitoring a few 10Gbps network segments with DAG 9.2X2s, too. I'll add in that, when processing that much traffic on a single device, you'll definitely not want to skimp on memory.
> > 
> > I'm not sure which configurations you're using that might be limiting you to 16 streams -- we're  run with at least 24 streams, and (at least with the 9.2X2s) you should be able to work with up to 32 receive streams.
> > 
> > v/r 
> > 
> > John Donaldson
> > 
> >> -----Original Message-----
> >> From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of
> >> Mike Patterson
> >> Sent: Thursday, January 08, 2015 7:29 AM
> >> To: coen bakkers
> >> Cc: bro at bro.org
> >> Subject: Re: [Bro] Bro with 10Gb NIC's or higher
> >> 
> >> Succinctly, yes, although that provision is a big one.
> >> 
> >> I'm running Bro on two 10 gig interfaces, an Intel X520 and an Endace DAG
> >> 9.2X2. Both perform reasonably well. Although my hardware is somewhat
> >> underspecced (Dell R710s of differing vintages), I still get tons of useful data.
> >> 
> >> If your next question would be "how should I spec my hardware", that's
> >> quite difficult to answer because it depends on a lot. Get the hottest CPUs
> >> you can afford, with as many cores. If you're actually sustaining 10+Gb you'll
> >> probably want at least 20-30 cores. I'm sustaining 4.5Gb or so on 8 3.7Ghz
> >> cores, but Bro reports 10% or so loss. Note that some hardware
> >> configurations will limit the number of streams you can feed to Bro, eg my
> >> DAG can only produce 16 streams so even if I had it in a 24 core box, I'd only
> >> be making use of 2/3 of my CPU.
> >> 
> >> Mike
> >> 
> >>> On Jan 7, 2015, at 5:04 AM, coen bakkers <cbakkers at yahoo.de> wrote:
> >>> 
> >>> Does anyone have experience with higher speed NIC's and Bro? Will it
> >> sustain 10Gb speeds or more provide the hardware is spec'd appropriately?
> >>> 
> >>> regards,
> >>> 
> >>> Coen
> >>> _______________________________________________
> >>> Bro mailing list
> >>> bro at bro-ids.org
> >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >> 
> >> 
> >> _______________________________________________
> >> Bro mailing list
> >> bro at bro-ids.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-- 
Aashish Sharma	(asharma at lbl.gov) 				 
Cyber Security, 
Lawrence Berkeley National Laboratory  
http://go.lbl.gov/pgp-aashish 
Office: (510)-495-2680  Cell: (510)-612-7971
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150109/8ece2184/attachment.bin