[Bro] - recommended DB for Bro logs

Zeolla@GMail.com zeolla at gmail.com
Sun Dec 9 08:53:35 PST 2018


I've put bro data in Solr,.ElasticSearch, HDFS, Splunk, and Mongodb with
success but for different use cases.  What are you looking to do with the
data?

The Apache Metron project supports bro logs natively and can index in hdfs,
solr, or elasticsearch.  If you don't want to buy into the entire project
(a bit of a heavy lift if you don't already run Ambari and Hadoop or aren't
interested in security data analytics) there may be reusable components
that are helpful.  Let me know if you're interested in digging in and I can
help.  A part of this project is the kafka writer plugin, used as a buffer
between bro and an indexed store.
https://packages.bro.org/packages/view/7388aa77-4fb7-11e8-88be-0a645a3f3086

This isn't meant to be a commercial, I've heard great things about bro data
going into Postgres and redis as well.

See also:
https://packages.bro.org/tags/view/737d1f7c-4fb7-11e8-88be-0a645a3f3086
https://packages.bro.org/tags/view/738aaeb0-4fb7-11e8-88be-0a645a3f3086

Jon

On Sun, Dec 9, 2018, 10:56 AM Clark Gaylord <cgaylord at vt.edu> wrote:

> I have done some proof of concept work with PostgreSQL (mostly in AWS RDS)
> and have been very happy with the results so far. Of course the rub is you
> need to set up the schema, but it is pretty straightforward to ingest after
> that from the JSON.
>
> What I've done is load JSON into a text field of a temp table, then cast
> that as JSON on insert (there was a little trick to getting this right that
> I don't recall off the top of my head). My load process is currently out of
> service but I can try to look up my code for this if you need it.
>
> Anyway, works like a champ since PG has not only JSON but inet and cidr
> data types!
> https://www.postgresql.org/docs/11.1/datatype-net-types.html
>
> You could do a document database that would handle the JSON gracefully,
> but then you're constantly paying the parse tax. Works great if you don't
> actually want to use your data, though. :-)
>
> If you use standard bro text files you've got more parsing to do but it's
> certainly doable. I like having JSON bro output to avoid that heavy lifting.
>
> Cheers
> Clark
>
> --
>
> --
> Clark Gaylord
> cgaylord at vt.edu
> ... Autocorrect may have improved this message ...
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-- 

Jon Zeolla
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181209/5386f17f/attachment-0001.html 


More information about the Bro mailing list