[Bro] cluster question

[Michał Purzyński]

The correct formula, valid for every nic is between 0-1Gbit per worker. Depends on the traffic type, CPU, bios settings, kernel settings and sometimes version, OS type version and settings, NIC and running scripts.

The old 250 comes from a dark past and was for unknown traffic on unknown hardware.

Sometimes I process 500 Mbit/sec per worker on Myricom sometimes I have quite a packet drop. Oh well ;)

Also afpacket >> pfring. And there's afpacket, netmap, Myricom, napatech and dozen others ;)


> On 7 Oct 2016, at 19:29, Neslog <neslog at gmail.com> wrote:
> 
> Is that formula based on Myricom NIC or using PF_Ring?  What's the best way to calculate the expected increase when switching to a custom nic?
> 
> 
>> On Oct 7, 2016 12:31 PM, "Hovsep Levi" <hovsep.sanjay.levi at gmail.com> wrote:
>> You sound a little confused, multi-node scaling is a feature of Bro and really the only way to monitor high volume locations.  See the LBNL paper on Bro at 100G for an example.  When using a front-end load-balancer you are distributing the traffic directly to the worker nodes which in turn produce metadata to be sent to the manager node.
>> 
>> The decision to use more than one box is relative to the processing requirements, the basic formula is something like one 3.0 Ghz core per 250Mbps of traffic.
>> 
>> If you use multiple managers you break global visibility in the scripting context, proxies share state among the entire cluster which operates as a sort of giant shared memory space.  Multiple managers is essentially independent Bro clusters.  I think a basic example would be a scanning script or SQL injection script... if the threshold is 25 and 10.1.1.1 attacks your entire network each cluster only sees 1/n of that activity and may not fire an event because of the limited context.
>> 
>> As for the bandwidth concerns you mention I'm not sure what you mean exactly.  The metadata produced by the workers and sent to the manager (logs) are a fraction of the monitored raw traffic.
>> 
>> HTH,
>> 
>> -Hovsep
>> 
>> 
>> 
>>> On Fri, Oct 7, 2016 at 12:02 PM, erik clark <philosnef at gmail.com> wrote:
>>> I noticed the previous gentleman running 160 workers (I assume 16 boxes with 10 workers each??) in a cluster, and had a general question about this.
>>> 
>>> If I am pumping out well above 5Gb/s, doesn't that mean running in a cluser that I am pushing 5 right back out the other side? If so, this doesn't seem to scale well beyond 5ish Gb/s. 
>>> 
>>> At what point, and how many pps, should we move away from a single manager host talking to cluster hosts? Even if there is no processing by bro on the manager, you still have bandwidth issues, unless you are loading up your bro manager with multiple 10 gig nics, and are loadbalancing upstream, in which case, why aren't you just load balancing to stand alone boxes each with their own manager, logger, and set of workers?
>>> 
>>> It seems to me that running multiple physical bro hosts tied to a single manager is the poor mans solution to running proper load balancing hardware upstream. Am I mistaken?
>>> 
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> 
>> 
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161007/5be5d35c/attachment.html