[Bro] problem ingesting bro json logs into splunk

[philosnef]

Heh. We have a multi-TB license for splunk... Bro is one of the largest consumers of that license.... 

    On Thursday, July 14, 2016 12:54 PM, Brandon Lattin <lattin at umn.edu> wrote:
 

 I do wonder if it's even faster having the pre-search-time extractions in the tsidx files. I suppose if you're going for a specific IP, the bloom filters may help?
I've been really hesitant to move to JSON, simply because of the added raw volume impact on licensing. Bro is already over 250GB/day for us using TSV files.
On Thu, Jul 14, 2016 at 11:44 AM, philosnef <philosnef at yahoo.com> wrote:

It is the TS IDX files in Splunk that grow out of control when using the Bro TSV app. Hope this helps for anyone interested. 

    On Thursday, July 14, 2016 12:30 PM, "Azoff, Justin S" <jazoff at illinois.edu> wrote:
 

 
> On Jul 14, 2016, at 12:16 PM, philosnef <philosnef at yahoo.com> wrote:
> 
> The problem with the Spunk app is that indexing is occuring at time of ingest. This causes the indices of the Bro data to grow extremely fast. Using json and not the Bro app means that the data is indexed by Splunk, resulting in far smaller indices on the splunk indexing servers. This is specifically why we moved away from TSV and to JSON, since it was nuking disk storage for those indices...

Odd, I'd expect it to be about the same.  The indexed data should be the same, and even though every json record includes the field names, they compress well.

It's possible that the bro app indexing the fields individually is what makes the indexes larger... if you do something like

    id_resp_p=6379

(or whatever the field shows up as for you)

does that find the records immediately, or does it have to scan through all the data?

without individual field indexes you would have to do something like

    6379 id_resp_p=6379

and hope that speeds it up, if you're trying to do something like

    id_orig_p=80

Then this will be pretty slow:

    80 id_orig_p=80


-- 
- Justin Azoff


   
_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




-- 
Brandon LattinSecurity Analyst
University of Minnesota - University Information Security
Office: 612-626-6672

  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/dcaeefd9/attachment-0001.html