[Zeek] log delays and logger CPUs

[Justin Azoff]

On Wed, Oct 16, 2019 at 12:23 PM Chris Herdt <cherdt at umn.edu> wrote:

> I have a cluster running Bro 2.6.4. One host runs a manager and logger, 8
> other hosts run proxy and worker nodes.
>
> Lately the logger node has not been able to keep up with the logs, and
> I've noticed that the most recent entries in the current/conn.log are
> significantly delayed (I've seen delays as high as 90 minutes).
>
> The logger process has maxed out CPU usage on core 1. The node.cfg file
> specifies 8 CPU cores (all on the same NUMA node as the NVMe drive where
> the logs are written):
>
> [logger]
> type=logger
> host=bromanager-01.umn.edu
> pin_cpus=1,3,5,7,9,11,13,15
>
> `broctl nodes` shows that only 1 CPU core is pinned:
>


> Can pin_cpus be used with a logger node? Any other suggestions for
> improving logger performance?
>

Ah.. pin_cpus is more intended to work with multiple worker processes.
It's definitely doing the wrong thing in your case.  It's pinning the first
logger (out of one total) to core 1, and then it never uses 3,5,7....
You're better off removing that setting.  It should run fine across all
cores, even with the numa hit.. the volume of logs that would go across the
numa bus would only be a small fraction of the total bandwidth.

You could still pin it to those cores, but you'd have to do it manually
using taskset for now.  This is probably something that could be fixed in
broctl(zeekctl) to better handle pin_cpus option when only a single process
is being started.

-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191016/8ce2ecd0/attachment.html