[Bro] Manager swapping..

[Azoff, Justin S]

> On Mar 22, 2017, at 11:05 AM, fatema bannatwala <fatema.bannatwala at gmail.com> wrote:
> 
> The bro processes are really big on the manager:
> 
> PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
> 104772 bro       20   0 24.926g 0.017t   1300 S  45.7 25.0   4542:04 bro
> 125346 bro       20   0  0.221t 0.027t   3444 S  40.4 39.4 187:28.80 bro
> 125366 bro       25   5 1510856 275516    728 R  40.1  0.4 222:22.58 bro
> 104776 bro       25   5  540736 228920    360 S   8.9  0.3 893:42.05 bro

Which process is which in this output?  Can you use broctl top manager logger instead?  that will include the output about which process is which along with the cpu/mem usage.  How to troubleshoot the issue depends a lot on if it is the manage process or the logger process causing the problems.

> Also, the free -g output looks like this:
> $ free -g
>               total        used        free      shared  buff/cache   available
> Mem:             70          47           0           0          22          21
> Swap:             7           7           0

Looks like you have some headroom there, but not much.

> Next thing I am going to try is to disable some of the protocols from logging (don't know how much help it would be) and restart Bro.

Well, if it's the logger node, reducing the log volume can help.  Depending on how many cpu cores you have one thing that can help is using logging filters to split logs out into multiple files.  That lets the logger node dedicate more threads to writing logs.

> Any other suggestions/Best practices to follow, to avoid this situation in future (really not looking forward to the quick and dirty fix of restarting Bro whenever this happens :) )?
> 
> Also, I have proper ethtool settings (tso off gso off gro off rx off tx off sg off) on the manager as well (as suggested in some of the posts for better performance).

That shouldn't really matter on the manager, but it can't hurt.

> Was just brainstorming, and thinking if multi-threading can be used for logger as well, just like worker threads? 
> As a single Bro logger process is becoming big, why not to distribute the work load across multiple logger processes.
> Is it possible to do? and if it impacts manager on the same node?
> Anybody tried that?

The logger does distribute the work across multiple threads, but it has a central component that has to receive all the messages.  

Someone else on the mailing list was having issues with logger scaling and I pointed them to the parts of broctl that needed to be tweaked to let you run multiple logger nodes.  If you're currently using something like the kafka log writer or something like logstash to ship bro logs off to another system it will kind of work.  The 'issue' is that you end up with 2 log directories that each contain the logs from half the workers.  As long as you have something else that can merge them back together and correlate everything that's not a problem.

Hopefully multiple logger nodes can be supported officially at some point.


-- 
- Justin Azoff