[Bro] Bro with 10Gb NIC's or higher

[Kelly Kerby]

Hello,

We at UT Austin are fairly new to Bro and new  to the list (been following, but never posted), but I thought I'd share my experience.

We have had good luck monitoring our traffic which sustains ~17-20 Gbps during peak hours with 2 devices made by a
company called Netronome. The traffic is distributed between the 2 clustered devices using an integrated load balancer which
evenly spreads the traffic across all the processors which have been pinned to corresponding bro workers.

We see very little traffic loss - random ~2-3% drops per Bro instance with the occasional larger ~10% drop.

Our configuration:

- 2 clustered devices 40 cores each with 32 workers and 4 proxies
- Primary device with 2 10 gig cards

Hope this is helpful.

-Kelly
UT Austin

On 1/9/15 1:11 PM, Mike Reeves wrote:
> In all of the 10G deployments I have done I always do multiple boxes behind a flow based load balancer. That way I can use commodity boxes without special NICs and keep them at a reasonable price point. The bang for the buck goes down when you talk 4 x 12 core HT processors etc. vs a dual 10 core HT. You also get the ability to have some fault tolerance where if you have hardware issues you are not blind. I have a few deployments that are going from 10G to 100G and the only thing we have to change is the inbound interfaces on the LB gear. The other positive is as usage goes up I can add additional capacity incrementally instead of having to re-solution.
>
> Thanks
>
> Mike
>
>
>
>
>> On Jan 9, 2015, at 1:20 PM, Mike Patterson <mike.patterson at uwaterloo.ca> wrote:
>>
>> You're right, it's 32 on mine.
>>
>> I posted some specs for my system a couple of years ago now, I think.
>>
>> 6-8GB per worker should give some headroom (my workers usually use about 5 apiece I think).
>>
>> Mike
>>
>> --
>> Simple, clear purpose and principles give rise to complex and
>> intelligent behavior. Complex rules and regulations give rise
>> to simple and stupid behavior. - Dee Hock
>>
>>> On Jan 9, 2015, at 1:03 PM, Donaldson, John <donaldson8 at llnl.gov> wrote:
>>>
>>> I'd agree with all of this. We're monitoring a few 10Gbps network segments with DAG 9.2X2s, too. I'll add in that, when processing that much traffic on a single device, you'll definitely not want to skimp on memory.
>>>
>>> I'm not sure which configurations you're using that might be limiting you to 16 streams -- we're  run with at least 24 streams, and (at least with the 9.2X2s) you should be able to work with up to 32 receive streams.
>>>
>>> v/r
>>>
>>> John Donaldson
>>>
>>>> -----Original Message-----
>>>> From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of
>>>> Mike Patterson
>>>> Sent: Thursday, January 08, 2015 7:29 AM
>>>> To: coen bakkers
>>>> Cc: bro at bro.org
>>>> Subject: Re: [Bro] Bro with 10Gb NIC's or higher
>>>>
>>>> Succinctly, yes, although that provision is a big one.
>>>>
>>>> I'm running Bro on two 10 gig interfaces, an Intel X520 and an Endace DAG
>>>> 9.2X2. Both perform reasonably well. Although my hardware is somewhat
>>>> underspecced (Dell R710s of differing vintages), I still get tons of useful data.
>>>>
>>>> If your next question would be "how should I spec my hardware", that's
>>>> quite difficult to answer because it depends on a lot. Get the hottest CPUs
>>>> you can afford, with as many cores. If you're actually sustaining 10+Gb you'll
>>>> probably want at least 20-30 cores. I'm sustaining 4.5Gb or so on 8 3.7Ghz
>>>> cores, but Bro reports 10% or so loss. Note that some hardware
>>>> configurations will limit the number of streams you can feed to Bro, eg my
>>>> DAG can only produce 16 streams so even if I had it in a 24 core box, I'd only
>>>> be making use of 2/3 of my CPU.
>>>>
>>>> Mike
>>>>
>>>>> On Jan 7, 2015, at 5:04 AM, coen bakkers <cbakkers at yahoo.de> wrote:
>>>>>
>>>>> Does anyone have experience with higher speed NIC's and Bro? Will it
>>>> sustain 10Gb speeds or more provide the hardware is spec'd appropriately?
>>>>>
>>>>> regards,
>>>>>
>>>>> Coen
>>>>> _______________________________________________
>>>>> Bro mailing list
>>>>> bro at bro-ids.org
>>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>>
>>>>
>>>> _______________________________________________
>>>> Bro mailing list
>>>> bro at bro-ids.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3858 bytes
Desc: S/MIME Cryptographic Signature
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150112/23b87765/attachment.bin