[Bro] Bro logs from JSON to TSV

ElBadry Shaker, Moustafa moustafa.elbadry at oregonstate.edu
Fri Jan 5 08:34:58 PST 2018

Great. Thanks Justin for sharing this. Definitely helps us a lot.


On 1/4/18, 12:36 PM, "Azoff, Justin S" <jazoff at illinois.edu> wrote:

    > On Jan 3, 2018, at 4:38 PM, ElBadry Shaker, Moustafa <moustafa.elbadry at oregonstate.edu> wrote:
    > I want the exact TSV format. 
    > We currently have our Bro cluster writing logs in JSON. There are couple of network traffic analytics tools like RITA (Real Intelligence Threat Analytics) and some AWK scripts that we want to use. The problem is that the tools we want to use work only with Bro’s default TSV format. 
    > Moustafa   
    Ah, I see now.  You have a few of options here.
    You could just tell bro to write out the logs in both formats at the same time.  For older logs there is only a script for bro that can re-log to json, but not the other way, most people have the opposite problem.
    There is an open issue for RITA to support json: https://github.com/ocmdev/rita/issues/146
    A tool to convert the json logs back into the TSV format could be written, but ultimately that would be a waste of time.  Better to update RITA to support json instead of writing more tools to work with the tsv format that only bro uses.
    For awk stuff you can swap out bro-cut for jq or https://github.com/JustinAzoff/json-cut
    json-cut it doesn't support all the options that bro-cut supports and may be a bit buggy, but it's easier to extract a few fields from a json log as TSV and 2x faster than jq.  If I can find a nice, small json library for C we can probably update bro-cut to natively support the json logs.
    For now, to extract note and msg from a stream of notice logs with bro-cut and json-cut you just do
        zcat notice.* | bro-cut note msg | awk ...
        zcat notice.* | json-cut note msg | awk ...
    For jq you use something like
        zcat notice.* | jq -r '[.note, .msg]|@tsv' | awk ...
    If the awk scripts are hardcoding top level field numbers like $3 and $5 instead of using bro-cut... they should not do that :-)
    Justin Azoff

More information about the Bro mailing list