[Bro] Manager swapping..

[fatema bannatwala]

Was just brainstorming, and thinking if multi-threading can be used for
logger as well, just like worker threads?
As a single Bro logger process is becoming big, why not to distribute the
work load across multiple logger processes.
Is it possible to do? and if it impacts manager on the same node?
Anybody tried that?

On Wed, Mar 22, 2017 at 11:05 AM, fatema bannatwala <
fatema.bannatwala at gmail.com> wrote:

> Hey all,
>
> We have logger and manager running on the same node, and it started to use
> complete swap and bro logs in current dir stopped rotating.
>
> We have run in this type of issue before when running Bro2.4, and it
> turned out that moving proxies to the worker nodes solved the high load
> issue on manager, and things started working normally.
>
> Now, we have all the proxies on the worker nodes (4 in total) and logger
> is running on the same node as manager, so my guess would be, that might be
> causing the high load on manager.
>
> The bro processes are really big on the manager:
>
> PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
> 104772 bro       20   0 24.926g 0.017t   1300 S  45.7 25.0   4542:04 bro
> 125346 bro       20   0  0.221t 0.027t   3444 S  40.4 39.4 187:28.80 bro
> 125366 bro       25   5 1510856 275516    728 R  40.1  0.4 222:22.58 bro
> 104776 bro       25   5  540736 228920    360 S   8.9  0.3 893:42.05 bro
>
> Also, the free -g output looks like this:
> $ free -g
>               total        used        free      shared  buff/cache
> available
> Mem:             70          47           0           0          22
>    21
> Swap:             7           7           0
>
> Next thing I am going to try is to disable some of the protocols from
> logging (don't know how much help it would be) and restart Bro.
>
> Any other suggestions/Best practices to follow, to avoid this situation in
> future (really not looking forward to the quick and dirty fix of restarting
> Bro whenever this happens :) )?
>
> Also, I have proper ethtool settings (tso off gso off gro off rx off tx
> off sg off) on the manager as well (as suggested in some of the posts for
> better performance).
>
> Thanks,
> Fatema.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170322/19f43239/attachment.html