[Bro] elastic search / bro questions

Joe Blow blackhole.em at gmail.com
Mon Nov 10 06:46:37 PST 2014


Hey all,

We're still fighting with certain troubles but have completed isolating all
data nodes from our master nodes.  That seemed to help with general
availability of the cluster, and data throughput (we think some of the data
nodes couldn't talk to the masters, creating a bunch of stability issues).

Now all that is kind of an ES thing, though i thought it might be valuable
i've added it.

I now have a question for the BRO folks regarding indexing to the 'bro'
index (as opposed to the 'bro-201410242100').  We have our 'bro' index up
over 10B records.  When the index needs to be brought back up after
(routine) catastrophic failures, we find ourselves waiting for a really,
really long time while the massive 'bro' primary shards initialize.

My question is this.  Many of these ES issues appear that they can be
alleviated if we were shoving all of the bro logs into 'bro-YYYYmmddHHMM',
instead of some there, and some in the giant 'bro' index.  Is there any
reason why we can't force all of the ES logging into the time based
indicies instead of the one giant bro index?  Would anyone know where to
start hacking the BRO code to try and make this possible?

By the way, thanks tons for the help everyone, i'll definitely be posting a
full lessons learned once we get everything up the way we're expecting.

Cheers,

JB



On Sat, Nov 8, 2014 at 6:39 AM, Michal Purzynski <michal at rsbac.org> wrote:

>  How about using Heka to read and parse the logs, and MozDef to collect
> them? That's what we do here with I believ 7k eps, soon to be more. Or just
> Heka. I'd go for both, we're working on a plug and play configuration.
>
> One of the good things about Heka is - it's insane fast. Tests were
> showing 10Gbit/sec pipe saturated with logs.
>
> Heka
>
> http://blog.mozilla.org/services/2013/04/30/introducing-heka/
> https://github.com/mozilla-services/heka
> https://hekad.readthedocs.org/en/v0.8.0/
>
> MozDef
>
> https://github.com/jeffbryner/MozDef
> http://mozdef.readthedocs.org/en/latest/
>
>  On Thu, Nov 6, 2014 at 7:54 PM, Joe Blow <blackhole.em at gmail.com> wrote:
>
>>  Hey all,
>>
>>  Just going to throw this out there and hope some people are willing to
>> potentially share some learning experiences if they have any.
>>
>>  We have a system which generates around 15k-30k BRO events/sec and are
>> trying to ingest these logs into a fairly beefy elasticsearch cluster.
>> Total cluster memory ~300GB, storage ~300TB.
>>
>>  Long story short, we're having some problems keeping up with this
>> feed.  Does anyone have any performance tuning with this module?  I've
>> played a lot with rsyslog batch sizes with elasticsearch and was hoping
>> there would be some simple directive i could try and apply to BRO.
>>
>>  Does anyone have this experience here?  Does this module batch anything?
>>
>>  Thanks in advance.
>>
>> Cheers,
>>
>>  JB
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>
>
> _______________________________________________
> Bro mailing listbro at bro-ids.orghttp://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141110/d7b194ef/attachment.html 


More information about the Bro mailing list