[Bro] Bro cluster requirements and manager logging backlog bug
Azoff, Justin S
jazoff at illinois.edu
Thu Dec 22 10:23:14 PST 2016
> On Dec 22, 2016, at 1:07 PM, Hovsep Levi <hovsep.sanjay.levi at gmail.com> wrote:
> There may be some inefficiencies in the thread queuing code the logger uses, but the only people that seem to have these major issues have the slow AMD cpus.
> Multiple loggers is something we hope to add once broker is integrated. There's a few places I hope to be able to do some sort of consistent ring hashing to scale out different tasks. Many tasks in bro are easily partitioned, like logging and sumstats.
> I wasn't implying poor code just code not optimized for our deployment. Maybe the multiple logger approach would do it but in the meanwhile I'm looking for a quick fix.
Yeah.. mostly it's just not designed or optimized for systems that have 48 slow cores.. It's possible that broker may end up performing a bit better (since it uses threads for more things) but at some point we would still need to scale out to multiple machines anyway.
> > Maybe streaming logs via Kafka and disabling writing to disk has a chance.
> Ah! if that is your end goal, you could try looking into having your workers write directly to kafka and bypass the manager entirely.
> I thought there was some degree of normalization that occurred at the manager node ? Would having workers write directly to Kafka limit any features of Bro ?
Not really, the main thing it does is log aggregation and rotation, which if you are sending to kafka you don't really need.
> What you are saying sounds like using Kafka on the manager isn't going to fix anything as it will encounter the same resource bottleneck.
Correct, it would still go through the same single receiving process.
It's probably not that much work (for kafka use cases at least) to be able to run two(or four) logger processes.
It's really just a matter of running more of them on different ports and updating the cluster layout so half the workers have one port and half the workers have the other port (broctl already does this kind of thing when you configure multiple proxy nodes).
It's a little more complicated for non kafka uses cases because you would end up logger-1/conn.log logger-2/conn.log and log rotation wouldn't work right.
I think getting it to work for the kafka use cases may only require a few lines of code to be changed (basically do for logger nodes what broctl already does for proxy nodes)
- Justin Azoff
More information about the Bro