[Bro] problem ingesting bro json logs into splunk

[Brandon Lattin]

I do wonder if it's even faster having the pre-search-time extractions in
the tsidx files. I suppose if you're going for a specific IP, the bloom
filters may help?

I've been really hesitant to move to JSON, simply because of the added raw
volume impact on licensing. Bro is already over 250GB/day for us using TSV
files.

On Thu, Jul 14, 2016 at 11:44 AM, philosnef <philosnef at yahoo.com> wrote:

> It is the TS IDX files in Splunk that grow out of control when using the
> Bro TSV app. Hope this helps for anyone interested.
>
>
> On Thursday, July 14, 2016 12:30 PM, "Azoff, Justin S" <
> jazoff at illinois.edu> wrote:
>
>
>
> > On Jul 14, 2016, at 12:16 PM, philosnef <philosnef at yahoo.com> wrote:
> >
> > The problem with the Spunk app is that indexing is occuring at time of
> ingest. This causes the indices of the Bro data to grow extremely fast.
> Using json and not the Bro app means that the data is indexed by Splunk,
> resulting in far smaller indices on the splunk indexing servers. This is
> specifically why we moved away from TSV and to JSON, since it was nuking
> disk storage for those indices...
>
>
> Odd, I'd expect it to be about the same.  The indexed data should be the
> same, and even though every json record includes the field names, they
> compress well.
>
> It's possible that the bro app indexing the fields individually is what
> makes the indexes larger... if you do something like
>
>     id_resp_p=6379
>
> (or whatever the field shows up as for you)
>
> does that find the records immediately, or does it have to scan through
> all the data?
>
> without individual field indexes you would have to do something like
>
>     6379 id_resp_p=6379
>
> and hope that speeds it up, if you're trying to do something like
>
>     id_orig_p=80
>
> Then this will be pretty slow:
>
>     80 id_orig_p=80
>
>
> --
> - Justin Azoff
>
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 
Brandon Lattin
Security Analyst
University of Minnesota - University Information Security
Office: 612-626-6672
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/e841a123/attachment.html