[Bro] Bro cluster requirements and manager logging backlog bug

[Hovsep Levi]

>
> There may be some inefficiencies in the thread queuing code the logger
> uses, but the only people that seem to have these major issues have the
> slow AMD cpus.
>


> Multiple loggers is something we hope to add once broker is integrated.
> There's a few places I hope to be able to do some sort of consistent ring
> hashing to scale out different tasks.  Many tasks in bro are easily
> partitioned, like logging and sumstats.
>
>
I wasn't implying poor code just code not optimized for our deployment.
Maybe the multiple logger approach would do it but in the meanwhile I'm
looking for a quick fix.



> > Maybe streaming logs via Kafka and disabling writing to disk has a
> chance.
>
> Ah! if that is your end goal, you could try looking into having your
> workers write directly to kafka and bypass the manager entirely.
>
>

I thought there was some degree of normalization that occurred at the
manager node ?  Would having workers write directly to Kafka limit any
features of Bro ?

What you are saying sounds like using Kafka on the manager isn't going to
fix anything as it will encounter the same resource bottleneck.

Here's the config I was going to use:

# Kafka output
#@load logs-to-kafka.bro
#redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG etc...);
#redef Kafka::kafka_conf = table(
#    ["metadata.broker.list"] = "10.1.1.1:9092"
#);
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161222/ac995842/attachment.html