[Bro] - recommended DB for Bro logs

[Clark Gaylord]

Looks like Metron doesn't support IPv6. Hence useless in 2018. Too bad.

--
Clark Gaylord
cgaylord at vt.edu
... autocorrect may have improved this message ...

On Sun, Dec 9, 2018, 11:53 Zeolla at GMail.com <zeolla at gmail.com wrote:

> I've put bro data in Solr,.ElasticSearch, HDFS, Splunk, and Mongodb with
> success but for different use cases.  What are you looking to do with the
> data?
>
> The Apache Metron project supports bro logs natively and can index in
> hdfs, solr, or elasticsearch.  If you don't want to buy into the entire
> project (a bit of a heavy lift if you don't already run Ambari and Hadoop
> or aren't interested in security data analytics) there may be reusable
> components that are helpful.  Let me know if you're interested in digging
> in and I can help.  A part of this project is the kafka writer plugin, used
> as a buffer between bro and an indexed store.
> https://packages.bro.org/packages/view/7388aa77-4fb7-11e8-88be-0a645a3f3086
>
> This isn't meant to be a commercial, I've heard great things about bro
> data going into Postgres and redis as well.
>
> See also:
> https://packages.bro.org/tags/view/737d1f7c-4fb7-11e8-88be-0a645a3f3086
> https://packages.bro.org/tags/view/738aaeb0-4fb7-11e8-88be-0a645a3f3086
>
> Jon
>
> On Sun, Dec 9, 2018, 10:56 AM Clark Gaylord <cgaylord at vt.edu> wrote:
>
>> I have done some proof of concept work with PostgreSQL (mostly in AWS
>> RDS) and have been very happy with the results so far. Of course the rub is
>> you need to set up the schema, but it is pretty straightforward to ingest
>> after that from the JSON.
>>
>> What I've done is load JSON into a text field of a temp table, then cast
>> that as JSON on insert (there was a little trick to getting this right that
>> I don't recall off the top of my head). My load process is currently out of
>> service but I can try to look up my code for this if you need it.
>>
>> Anyway, works like a champ since PG has not only JSON but inet and cidr
>> data types!
>> https://www.postgresql.org/docs/11.1/datatype-net-types.html
>>
>> You could do a document database that would handle the JSON gracefully,
>> but then you're constantly paying the parse tax. Works great if you don't
>> actually want to use your data, though. :-)
>>
>> If you use standard bro text files you've got more parsing to do but it's
>> certainly doable. I like having JSON bro output to avoid that heavy lifting.
>>
>> Cheers
>> Clark
>>
>> --
>>
>> --
>> Clark Gaylord
>> cgaylord at vt.edu
>> ... Autocorrect may have improved this message ...
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> --
>
> Jon Zeolla
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181209/19543bdd/attachment.html