[Zeek] Number of CPU cores for 100Gbps

[Woot4moo]

Thanks for the details. I am aware of MarkII and am reading through it.

How as a community can we update that clustering documentation? If it’s not
accurate it could very easily turn people away

On Sun, Apr 28, 2019 at 6:29 PM Michał Purzyński <michalpurzynski1 at gmail.com>
wrote:

> These rules aren't current anymore and frankly, have never been accurate.
>
> Your Zeek speed depends on the traffic you have, if you have some elephant
> flows (and how you deal with them), scripts you run, etc. I remember
> pushing between 5-10Gbit/sec through a server with 24 cores (not threads),
> with room to spare.
>
> You will also need memory, and depending on scripts you intend to write,
> that might be quite a lot. We run with 192GB / server.
>
> Do you have 100Gbit of traffic or 100Gbit interfaces?
>
> Either way, you're gonna build yourself a cluster with a packet broker in
> front of it. Arista works well, other people use different brands,
> depending on your needs and your budget.
>
> Give those tuning guides I wrote with Suricata developers a read, while on
> it, they apply to Zeek as well. Of course Suricata can process way more
> traffic per core, than Zeek, because the processing it does is way simpler.
>
> https://github.com/pevma/SEPTun
> https://github.com/pevma/SEPTun-Mark-II
>
>
> On Sun, Apr 28, 2019 at 11:35 AM Woot4moo <tscheponik at gmail.com> wrote:
>
>> My understanding is that 4,000+ CPU cores would be necessary to support
>> this throughput. In the recent meeting from CERN I recall seeing someone
>> describe 200Gbps, which would imply 8,000+ CPU cores. Is this accurate, or
>> am I doing a conversion incorrectly?
>>
>> I am basing this purely on this quote, from
>>
>> https://docs.zeek.org/en/stable/cluster/
>>
>> “The rule of thumb we have followed recently is to allocate
>> approximately 1 core for every 250Mbps of traffic that is being analyzed.
>> However, this estimate could be extremely traffic mix-specific. It has
>> generally worked for mixed traffic with many users and servers. For
>> example, if your traffic peaks around 2Gbps (combined) and you want to
>> handle traffic at peak load, you may want to have 8 cores available (2048 /
>> 250 == 8.2). ”
>>
> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190428/7cc25178/attachment.html