[Bro] Elasticsearch Writer vs logstash
Hosom, Stephen M
hosom at battelle.org
Fri Jan 30 04:51:06 PST 2015
Some things to think about:
1. Logstash is easy, but all the easiness that comes with it comes at a performance hit.
a. If you go this way, you could probably make this ‘easier’ by logging Bro’s logs to JSON for Logstash to send to Elasticsearch.
i. This will put you in an odd spot compared to other Bro deployments. Not many people log JSON logs. If you do this, you’ll want to use jq as a replacement for bro-cut.
b. Make sure you look at Heka as an alternative.
2. Some people have had success with the NSQ writer and using NSQ, but that is also not what most people would consider a “production” deployment.
If you do nothing else, please use a recent version of Elasticsearch. Older versions of Elasticsearch were MUCH worse on performance and lacked features that are very nice to have. You’ll want to look into tuning Elasticsearch as well. There are MANY articles out there on how to tune Elasticsearch for indexing large data volumes.
Finally, keep in mind that a lot of how you keep Bro’s logs can vary depending on the size of your environment and your tolerance level for risk. If you can’t risk losing indexed logs when Elasticsearch is down, then you’ll want to look into a queuing system like Redis, NSQ, or RabbitMQ. Seems like everyone has their pet implementation of AMQP, so I’ll let you sort that one out. This conversation could really go on forever… feel free to hop on #bro on freenode if you want to chat.
From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of anthony kasza
Sent: Friday, January 30, 2015 2:29 AM
To: Luis Miguel Silva
Subject: Re: [Bro] Elasticsearch Writer vs logstash
I thought the ES writer had some issues it needed worked out around indexes or something. Seth?
On Jan 29, 2015 11:17 PM, "Luis Miguel Silva" <luismiguelferreirasilva at gmail.com<mailto:luismiguelferreirasilva at gmail.com>> wrote:
...I just found a website that has a tutorial on how to parse bro logs with logstash<http://www.appliednsm.com/parsing-bro-logs-with-logstash/> AND points to the config used in the distro Security Onion<http://www.appliednsm.com/wp-content/uploads/logstash-SObro22-parse.conf_.txt>.
So I'd just like to know what your thoughts are on using the elasticsearch writer vs logstash?
On Thu, Jan 29, 2015 at 11:48 PM, Luis Miguel Silva <luismiguelferreirasilva at gmail.com<mailto:luismiguelferreirasilva at gmail.com>> wrote:
I'm interested in dumping my bro logs into an elastic search instance and, based on what I was able to learn thus far, it seems I have two different options:
- use the elasticsearch writer (which the documentation says should not be used in production as it doesn't have any error checking)
- or use logstash to read info directly from the bro logs and externally dump it into elasticsearch
It seems to me the logstash route is better, given that I should be able to massage the data into more "user friendly" fields that can be easily queried with elasticsearch.
So my question is, based on your experience, what is the best option? And, if you do use logstash, can you share your logstash config?
Thanks in advance,
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Bro