[Zeek] log delays and logger CPUs
michalpurzynski1 at gmail.com
Wed Oct 16 10:53:59 PDT 2019
So there's a couple of things you could do.
1. Indeed using pin_cpus for the manager or the logging will result in all
threads being pinned to the same CPU, definitely not ideal and worth
filling a bug
2. You could (per Justin's idea from Zeek Week) use the path func and write
a script that will split biggest log files into two or more files - and
each will get a separate thread
3. And maybe implement some filtering for traffic you don't care for, if
there is any, examples here
On Wed, Oct 16, 2019 at 10:17 AM Justin Azoff <justin at corelight.com> wrote:
> On Wed, Oct 16, 2019 at 12:23 PM Chris Herdt <cherdt at umn.edu> wrote:
>> I have a cluster running Bro 2.6.4. One host runs a manager and logger, 8
>> other hosts run proxy and worker nodes.
>> Lately the logger node has not been able to keep up with the logs, and
>> I've noticed that the most recent entries in the current/conn.log are
>> significantly delayed (I've seen delays as high as 90 minutes).
>> The logger process has maxed out CPU usage on core 1. The node.cfg file
>> specifies 8 CPU cores (all on the same NUMA node as the NVMe drive where
>> the logs are written):
>> `broctl nodes` shows that only 1 CPU core is pinned:
>> Can pin_cpus be used with a logger node? Any other suggestions for
>> improving logger performance?
> Ah.. pin_cpus is more intended to work with multiple worker processes.
> It's definitely doing the wrong thing in your case. It's pinning the first
> logger (out of one total) to core 1, and then it never uses 3,5,7....
> You're better off removing that setting. It should run fine across all
> cores, even with the numa hit.. the volume of logs that would go across the
> numa bus would only be a small fraction of the total bandwidth.
> You could still pin it to those cores, but you'd have to do it manually
> using taskset for now. This is probably something that could be fixed in
> broctl(zeekctl) to better handle pin_cpus option when only a single process
> is being started.
> Zeek mailing list
> zeek at zeek.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Zeek