[Bro] elastic search / bro questions

Michal Purzynski michal at rsbac.org
Sat Nov 8 03:39:38 PST 2014

How about using Heka to read and parse the logs, and MozDef to collect 
them? That's what we do here with I believ 7k eps, soon to be more. Or 
just Heka. I'd go for both, we're working on a plug and play configuration.

One of the good things about Heka is - it's insane fast. Tests were 
showing 10Gbit/sec pipe saturated with logs.





> On Thu, Nov 6, 2014 at 7:54 PM, Joe Blow <blackhole.em at gmail.com 
> <mailto:blackhole.em at gmail.com>> wrote:
>     Hey all,
>     Just going to throw this out there and hope some people are
>     willing to potentially share some learning experiences if they
>     have any.
>     We have a system which generates around 15k-30k BRO events/sec and
>     are trying to ingest these logs into a fairly beefy elasticsearch
>     cluster.  Total cluster memory ~300GB, storage ~300TB.
>     Long story short, we're having some problems keeping up with this
>     feed.  Does anyone have any performance tuning with this module? 
>     I've played a lot with rsyslog batch sizes with elasticsearch and
>     was hoping there would be some simple directive i could try and
>     apply to BRO.
>     Does anyone have this experience here?  Does this module batch
>     anything?
>     Thanks in advance.
>     Cheers,
>     JB
>     _______________________________________________
>     Bro mailing list
>     bro at bro-ids.org <mailto:bro at bro-ids.org>
>     http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141108/ba3aaa2d/attachment.html 

More information about the Bro mailing list