[Bro] Get the license usage down in Splunk when indexing Bro

[Troy Ward]

Mike,

How much data are we talking about?  Have you done the analysis to see what
logs are actually causing you problems?  I am currently ingesting somewhere
in the neighborhood of 50GB of bro logs a day but at one point it was a lot
more.  After doing some digging we found out that our sensor was saturated
and dropping a ton of packets which had the bro wierd log and conn log
going through the roof because it some connections where appearing 3 or 4
times due to it thinking they were different connections.

Troy


>
> Today's Topics:
>
>    1. Re: Get the license usage down in Splunk when indexing Bro
>       logs (Joshua Buysse)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 11 Jul 2017 13:55:13 -0500
> From: Joshua Buysse <buysse at umn.edu>
> Subject: Re: [Bro] Get the license usage down in Splunk when indexing
>         Bro     logs
> To: Dave Crawford <bro at pingtrip.com>
> Cc: bro at bro.org
> Message-ID:
>         <CAGp0rD1qGn8jyPR5eZn1kq0RbEetRuwBABFdPLkOO6zrTek4uw at mail.
> gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> I think I know where Mike's misunderstanding comes from - the JSON logs are
> larger original size (for license volume), but will use less space on
> indexer disk than the default TSV because the extractions are search-time
> instead of index-time.
>
> On Tue, Jul 11, 2017 at 9:57 AM, Dave Crawford <bro at pingtrip.com> wrote:
>
> >
> >
> > The JSON logs will always be larger than the default tab delimited. With
> > JSON every log event includes the "column" names versus a set of headers
> in
> > he delimited format.
> >
> > > On Jul 11, 2017, at 6:48 AM, Mike Eriksson <mike at swedishmike.org>
> wrote:
> > >
> > > Hi all,
> > >
> > > We're currently working on deploying Bro sensors to various offices and
> > I've come to realise that the Bro logs are quite 'expensive' when it
> comes
> > to Splunk licenses. To say the least.
> > >
> > > We have discussed various solutions but most of them fall down on us
> > losing the ability to correlate events unless we shift all the logs in to
> > Splunk.
> > >
> > > At the moment we're running it pretty much 'out of the box' so we can
> > save some GB's per day to turn of certain scripts, but it will probably
> not
> > be enough.
> > >
> > > Someone mentioned that turning on JSON logging instead of the standard
> > logging on Bro could save considerable amounts of space on your SIEM.
> Have
> > any of you guys tested this and can you back that  statement up?
> > >
> > > I was hoping that someone else had encountered this before and had come
> > up with some solution(s) to this issue?
> > >
> > > Thanks in advance, Mike
> > >
> > >
> > > _______________________________________________
> > > Bro mailing list
> > > bro at bro-ids.org
> > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >
>
>
>
> --
> Joshua Buysse
> University of Minnesota - University Information Security
>
> "On two occasions I have been asked, 'Pray, Mr. Babbage, if you
> put into the machine wrong figures, will the right answers come
> out?' I am not able rightly to apprehend the kind of confusion of
> ideas that could provoke such a question."
>   - Charles Babbage
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/
> 20170711/b813770f/attachment-0001.html
>
> ------------------------------
>
> _______________________________________________
> Bro mailing list
> Bro at bro.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> End of Bro Digest, Vol 135, Issue 9
> ***********************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170711/d38193af/attachment.html