[Bro] elastic search / bro questions

[Michal Purzynski]

How about using Heka to read and parse the logs, and MozDef to collect 
them? That's what we do here with I believ 7k eps, soon to be more. Or 
just Heka. I'd go for both, we're working on a plug and play configuration.

One of the good things about Heka is - it's insane fast. Tests were 
showing 10Gbit/sec pipe saturated with logs.

Heka

http://blog.mozilla.org/services/2013/04/30/introducing-heka/
https://github.com/mozilla-services/heka
https://hekad.readthedocs.org/en/v0.8.0/

MozDef

https://github.com/jeffbryner/MozDef
http://mozdef.readthedocs.org/en/latest/

> On Thu, Nov 6, 2014 at 7:54 PM, Joe Blow <blackhole.em at gmail.com 
> <mailto:blackhole.em at gmail.com>> wrote:
>
>     Hey all,
>
>     Just going to throw this out there and hope some people are
>     willing to potentially share some learning experiences if they
>     have any.
>
>     We have a system which generates around 15k-30k BRO events/sec and
>     are trying to ingest these logs into a fairly beefy elasticsearch
>     cluster.  Total cluster memory ~300GB, storage ~300TB.
>
>     Long story short, we're having some problems keeping up with this
>     feed.  Does anyone have any performance tuning with this module? 
>     I've played a lot with rsyslog batch sizes with elasticsearch and
>     was hoping there would be some simple directive i could try and
>     apply to BRO.
>
>     Does anyone have this experience here?  Does this module batch
>     anything?
>
>     Thanks in advance.
>
>     Cheers,
>
>     JB
>
>     _______________________________________________
>     Bro mailing list
>     bro at bro-ids.org <mailto:bro at bro-ids.org>
>     http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141108/ba3aaa2d/attachment.html