[Bro] Logging and memory leak

[Hovsep Levi]

Not really, I was going to reply to the old thread regarding this.

I'm in the process of switching back to a single logger and considering
trying Kafka export from each worker directly.  Right now the logs are
backlogged by about 20 minutes which I suspect is the bottleneck issue.
Apparently when using only Kafka export it has taken 24+ hours to reach
this state as opposed to previously with file based logging where the logs
would be delayed by 20 minutes within an hour's time.

I'm sure the weird issues still exist with multiple loggers I just can't
see them as easily right now, my Logstash parser doesn't handle them yet.
The priority has been to get the cluster stable, after that I'll have time
to work on optimization.  It seems with the current configuration a cluster
restart once per day is going to be required.

I'm also about to add another 44 workers to resolve the 9-17% packet loss
per worker during peak.  I'm expecting the individual worker export to have
its own set of challenges so my time may be better spent re-writing the
logger node for high volume.  Right now I don't know how to reconfigure Bro
to Kafka export from the workers directly, have to read more source.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170201/f823aacb/attachment.html