[Bro] Bro cluster requirements and manager logging backlog bug
hovsep.sanjay.levi at gmail.com
Thu Dec 22 10:07:59 PST 2016
> There may be some inefficiencies in the thread queuing code the logger
> uses, but the only people that seem to have these major issues have the
> slow AMD cpus.
> Multiple loggers is something we hope to add once broker is integrated.
> There's a few places I hope to be able to do some sort of consistent ring
> hashing to scale out different tasks. Many tasks in bro are easily
> partitioned, like logging and sumstats.
I wasn't implying poor code just code not optimized for our deployment.
Maybe the multiple logger approach would do it but in the meanwhile I'm
looking for a quick fix.
> > Maybe streaming logs via Kafka and disabling writing to disk has a
> Ah! if that is your end goal, you could try looking into having your
> workers write directly to kafka and bypass the manager entirely.
I thought there was some degree of normalization that occurred at the
manager node ? Would having workers write directly to Kafka limit any
features of Bro ?
What you are saying sounds like using Kafka on the manager isn't going to
fix anything as it will encounter the same resource bottleneck.
Here's the config I was going to use:
# Kafka output
#redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG etc...);
#redef Kafka::kafka_conf = table(
# ["metadata.broker.list"] = "10.1.1.1:9092"
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Bro