[Bro] Bro cluster requirements and manager logging backlog bug

Azoff, Justin S jazoff at illinois.edu
Fri Jan 6 14:16:58 PST 2017


Looks like you worked out the broctl changes the right way.  That code is a bit crufty, but what you have will work.  There's a much easier way to do the distributing of workers/proxies:

>>> import itertools
>>> loggers = ['logger-1', 'logger-2']
>>> logger_cycler = itertools.cycle(loggers)
>>> next(logger_cycler)
'logger-1'
>>> next(logger_cycler)
'logger-2'
>>> next(logger_cycler)
'logger-1'
>>> next(logger_cycler)
'logger-2'

> On Jan 6, 2017, at 4:58 PM, Hovsep Levi <hovsep.sanjay.levi at gmail.com> wrote:
> 
> I'm using four loggers and the memory usage remains stable.  When I re-enable writing logs to disk there's a difference since logs/current is a symlink to the first logger, spool/logger-1; the other loggers write into their own spool directories (ex: "spool/logger-3").  I think you mentioned this before.

Yep.  It's an issue for purely local logging, and I'm not sure if rotation would work (but maybe it does? you tell me :-))

For people that use splunk/logstash/kafka it's mostly a non-issue since it will get re-aggregated anyway.

> For some reason logger-1 and logger-3 are doing all of the work, there are no logs in logger-2 and logger-4 and the communication.log files for each doesn't show any worker communications.  At startup there was "peer sent worker-1-1" but nothing afterwards.  I'm not sure yet if this happens when Kafka only logging is enabled.  The cluster-layout.bro looks correct and shows the 4 loggers are distributed among the workers correctly, so it's not that.
> 


> When I reduced the number of loggers to 2 it's the same phenomenon, logger-1 is working OK but logger-2 seems to be stalled.  Only one worker has sent data and it's very low volume.
> 
> Overall the multiple logger setup shows promise for fixing the issue but there's a few more things to discover and tune.  It seems the reason the cluster is stable is because only half of the logs are being received when using multiple loggers.


It's very promising that you were seeing traffic to logger-1 and logger-3, so it is at least proving that multiple loggers will work.  If you ran 4 workers but only one was doing anything I'd be worried.  I'd be interested in knowing what happens if you ran 6 or 8 loggers.

Can you post what the resulting cluster-layout looked like for 2 and 4 workers?  Maybe it's a simple problem and it's just not evenly distributing things.


-- 
- Justin Azoff




More information about the Bro mailing list