[Bro] problem ingesting bro json logs into splunk

Azoff, Justin S jazoff at illinois.edu
Thu Jul 14 09:30:34 PDT 2016


> On Jul 14, 2016, at 12:16 PM, philosnef <philosnef at yahoo.com> wrote:
> 
> The problem with the Spunk app is that indexing is occuring at time of ingest. This causes the indices of the Bro data to grow extremely fast. Using json and not the Bro app means that the data is indexed by Splunk, resulting in far smaller indices on the splunk indexing servers. This is specifically why we moved away from TSV and to JSON, since it was nuking disk storage for those indices...

Odd, I'd expect it to be about the same.  The indexed data should be the same, and even though every json record includes the field names, they compress well.

It's possible that the bro app indexing the fields individually is what makes the indexes larger... if you do something like

    id_resp_p=6379

(or whatever the field shows up as for you)

does that find the records immediately, or does it have to scan through all the data?

without individual field indexes you would have to do something like

    6379 id_resp_p=6379

and hope that speeds it up, if you're trying to do something like

    id_orig_p=80

Then this will be pretty slow:

    80 id_orig_p=80


-- 
- Justin Azoff




More information about the Bro mailing list