[Bro] Elasticsearch Writer vs logstash

Michał Purzyński michalpurzynski1 at gmail.com
Fri Jan 30 08:55:42 PST 2015


I keep Bro logging to files just to keep some local cache that's easy
for quick browsing/query and to offload Bro from writing to ES.

Heka reads Bro logs, transforms with Lua scripts inside a sandbox and
can output to whatever you want. I think there's ES output from Heka
already.

On Fri, Jan 30, 2015 at 1:51 PM, Hosom, Stephen M <hosom at battelle.org> wrote:
> Some things to think about:
>
>
>
> 1.       Logstash is easy, but all the easiness that comes with it comes at
> a performance hit.
>
> a.       If you go this way, you could probably make this ‘easier’ by
> logging Bro’s logs to JSON for Logstash to send to Elasticsearch.
>
>                                                                i.      This
> will put you in an odd spot compared to other Bro deployments. Not many
> people log JSON logs. If you do this, you’ll want to use jq as a replacement
> for bro-cut.
>
> b.      Make sure you look at Heka as an alternative.
>
> 2.       Some people have had success with the NSQ writer and using NSQ, but
> that is also not what most people would consider a “production” deployment.
>
>
>
> If you do nothing else, please use a recent version of Elasticsearch. Older
> versions of Elasticsearch were MUCH worse on performance and lacked features
> that are very nice to have. You’ll want to look into tuning Elasticsearch as
> well. There are MANY articles out there on how to tune Elasticsearch for
> indexing large data volumes.
>
>
>
> Finally, keep in mind that a lot of how you keep Bro’s logs can vary
> depending on the size of your environment and your tolerance level for risk.
> If you can’t risk losing indexed logs when Elasticsearch is down, then
> you’ll want to look into a queuing system like Redis, NSQ, or RabbitMQ.
> Seems like everyone has their pet implementation of AMQP, so I’ll let you
> sort that one out. This conversation could really go on forever… feel free
> to hop on #bro on freenode if you want to chat.
>
>
>
> From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of anthony
> kasza
> Sent: Friday, January 30, 2015 2:29 AM
> To: Luis Miguel Silva
> Cc: bro
> Subject: Re: [Bro] Elasticsearch Writer vs logstash
>
>
>
> I thought the ES writer had some issues it needed worked out around indexes
> or something. Seth?
>
> -AK
>
> On Jan 29, 2015 11:17 PM, "Luis Miguel Silva"
> <luismiguelferreirasilva at gmail.com> wrote:
>
> ...I just found a website that has a tutorial on how to parse bro logs with
> logstash AND points to the config used in the distro Security Onion.
>
>
>
> So I'd just like to know what your thoughts are on using the elasticsearch
> writer vs logstash?
>
>
>
> Thank you,
>
> Luis
>
>
>
> On Thu, Jan 29, 2015 at 11:48 PM, Luis Miguel Silva
> <luismiguelferreirasilva at gmail.com> wrote:
>
> Dear all,
>
>
>
> I'm interested in dumping my bro logs into an elastic search instance and,
> based on what I was able to learn thus far, it seems I have two different
> options:
>
> - use the elasticsearch writer (which the documentation says should not be
> used in production as it doesn't have any error checking)
>
> - or use logstash to read info directly from the bro logs and externally
> dump it into elasticsearch
>
>
>
> It seems to me the logstash route is better, given that I should be able to
> massage the data into more "user friendly" fields that can be easily queried
> with elasticsearch.
>
>
>
> So my question is, based on your experience, what is the best option? And,
> if you do use logstash, can you share your logstash config?
>
>
>
> Thanks in advance,
>
> Luis
>
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



More information about the Bro mailing list