[Bro] problem ingesting bro json logs into splunk
philosnef at yahoo.com
Thu Jul 14 09:44:01 PDT 2016
It is the TS IDX files in Splunk that grow out of control when using the Bro TSV app. Hope this helps for anyone interested.
On Thursday, July 14, 2016 12:30 PM, "Azoff, Justin S" <jazoff at illinois.edu> wrote:
> On Jul 14, 2016, at 12:16 PM, philosnef <philosnef at yahoo.com> wrote:
> The problem with the Spunk app is that indexing is occuring at time of ingest. This causes the indices of the Bro data to grow extremely fast. Using json and not the Bro app means that the data is indexed by Splunk, resulting in far smaller indices on the splunk indexing servers. This is specifically why we moved away from TSV and to JSON, since it was nuking disk storage for those indices...
Odd, I'd expect it to be about the same. The indexed data should be the same, and even though every json record includes the field names, they compress well.
It's possible that the bro app indexing the fields individually is what makes the indexes larger... if you do something like
(or whatever the field shows up as for you)
does that find the records immediately, or does it have to scan through all the data?
without individual field indexes you would have to do something like
and hope that speeds it up, if you're trying to do something like
Then this will be pretty slow:
- Justin Azoff
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Bro