[Bro] Get the license usage down in Splunk when indexing Bro

Mike Eriksson mike at swedishmike.org
Tue Jul 11 22:15:33 PDT 2017


Troy,

Many thanks for your reply.

For this one host we're looking at around 170 - 190GB of data per day
according to Splunk. Which feels like a lot, but could also be the 'true'
value for a host with our traffic and all logs turned on if you see what I
mean?

So far we've not looked at duplicates in the logs - and I'll have to check
the size of the weird.log when I get in to work - so that's definitely
worth looking at.

If you don't mind me asking, if we find that we have a similar issue like
yours with dropped packets etc - what steps did you go through in order to
work on this issue? Spread the load out on more cores? Lessen the amount of
scripts run? All of the above, none of the above or something completely
different?

Cheers, Mike

On Tue, Jul 11, 2017 at 8:52 PM Troy Ward <pyrodie18 at gmail.com> wrote:

> Mike,
>
> How much data are we talking about?  Have you done the analysis to see
> what logs are actually causing you problems?  I am currently ingesting
> somewhere in the neighborhood of 50GB of bro logs a day but at one point it
> was a lot more.  After doing some digging we found out that our sensor was
> saturated and dropping a ton of packets which had the bro wierd log and
> conn log going through the roof because it some connections where appearing
> 3 or 4 times due to it thinking they were different connections.
>
> Troy
>
>
>>
>> Today's Topics:
>>
>>    1. Re: Get the license usage down in Splunk when indexing Bro
>>       logs (Joshua Buysse)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Tue, 11 Jul 2017 13:55:13 -0500
>> From: Joshua Buysse <buysse at umn.edu>
>> Subject: Re: [Bro] Get the license usage down in Splunk when indexing
>>         Bro     logs
>> To: Dave Crawford <bro at pingtrip.com>
>> Cc: bro at bro.org
>> Message-ID:
>>         <
>> CAGp0rD1qGn8jyPR5eZn1kq0RbEetRuwBABFdPLkOO6zrTek4uw at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> I think I know where Mike's misunderstanding comes from - the JSON logs
>> are
>> larger original size (for license volume), but will use less space on
>> indexer disk than the default TSV because the extractions are search-time
>> instead of index-time.
>>
>> On Tue, Jul 11, 2017 at 9:57 AM, Dave Crawford <bro at pingtrip.com> wrote:
>>
>> >
>> >
>> > The JSON logs will always be larger than the default tab delimited. With
>> > JSON every log event includes the "column" names versus a set of
>> headers in
>> > he delimited format.
>> >
>> > > On Jul 11, 2017, at 6:48 AM, Mike Eriksson <mike at swedishmike.org>
>> wrote:
>> > >
>> > > Hi all,
>> > >
>> > > We're currently working on deploying Bro sensors to various offices
>> and
>> > I've come to realise that the Bro logs are quite 'expensive' when it
>> comes
>> > to Splunk licenses. To say the least.
>> > >
>> > > We have discussed various solutions but most of them fall down on us
>> > losing the ability to correlate events unless we shift all the logs in
>> to
>> > Splunk.
>> > >
>> > > At the moment we're running it pretty much 'out of the box' so we can
>> > save some GB's per day to turn of certain scripts, but it will probably
>> not
>> > be enough.
>> > >
>> > > Someone mentioned that turning on JSON logging instead of the standard
>> > logging on Bro could save considerable amounts of space on your SIEM.
>> Have
>> > any of you guys tested this and can you back that  statement up?
>> > >
>> > > I was hoping that someone else had encountered this before and had
>> come
>> > up with some solution(s) to this issue?
>> > >
>> > > Thanks in advance, Mike
>> > >
>> > >
>> > > _______________________________________________
>> > > Bro mailing list
>> > > bro at bro-ids.org
>> > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> >
>> >
>> > _______________________________________________
>> > Bro mailing list
>> > bro at bro-ids.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> >
>>
>>
>>
>> --
>> Joshua Buysse
>> University of Minnesota - University Information Security
>>
>> "On two occasions I have been asked, 'Pray, Mr. Babbage, if you
>> put into the machine wrong figures, will the right answers come
>> out?' I am not able rightly to apprehend the kind of confusion of
>> ideas that could provoke such a question."
>>   - Charles Babbage
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170711/b813770f/attachment-0001.html
>>
>> ------------------------------
>>
>> _______________________________________________
>> Bro mailing list
>> Bro at bro.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>> End of Bro Digest, Vol 135, Issue 9
>> ***********************************
>>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170712/5f9fc882/attachment-0001.html 


More information about the Bro mailing list