[Bro] Bro cluster requirements and manager logging backlog bug

[Azoff, Justin S]

> On Dec 22, 2016, at 12:29 PM, Hovsep Levi <hovsep.sanjay.levi at gmail.com> wrote:
> 
> Thanks for the help, I'm going to give your suggestions a try.
> 
> 
> Unfortunately I wasn't able to stabilize the cluster.  I tried splitting the conn log into six different types, inbound(dns,http,other} and outbound{dns,http,other} in addition to the http inbound/outbound split but the logger process continues to buffer about 1G of memory per minute.
> 
> Short of a re-write of the logging process the only option is to upgrade CPUs ?  I tried running more than one logger but that doesn't seem to work.

There may be some inefficiencies in the thread queuing code the logger uses, but the only people that seem to have these major issues have the slow AMD cpus.

Multiple loggers is something we hope to add once broker is integrated.  There's a few places I hope to be able to do some sort of consistent ring hashing to scale out different tasks.  Many tasks in bro are easily partitioned, like logging and sumstats.

> Maybe streaming logs via Kafka and disabling writing to disk has a chance.

Ah! if that is your end goal, you could try looking into having your workers write directly to kafka and bypass the manager entirely.



-- 
- Justin Azoff