[Bro] Elasticsearch Writer vs logstash
jswan at sugf.com
Fri Jan 30 09:08:45 PST 2015
Yet another option: use nxlog on the Bro node. Have it forward the logs to Logstash as raw JSON and use the json_lines codec in Logstash to feed to Elasticsearch.
The reason I like this option is that it allows you to do complex processing locally rather than use a lot of complex grok filters in Logstash, which can be really slow. Nxlog can do type conversions, regex filters, and a lot more. It also keeps your Logstash config simple. I configure nxlog to use a separate TCP or UDP output using JSON formatting, then my Logstash config just looks like:
port => 5000
type => bro_dns
codec => json_lines
I would love to have the native Elasticsearch writer fixed up and blessed for production use, though!
From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of James Lay
Sent: Friday, January 30, 2015 6:42 AM
To: bro at bro.org
Subject: Re: [Bro] Elasticsearch Writer vs logstash
On Thu, 2015-01-29 at 23:48 -0700, Luis Miguel Silva wrote:
I'm interested in dumping my bro logs into an elastic search instance and, based on what I was able to learn thus far, it seems I have two different options:
- use the elasticsearch writer (which the documentation says should not be used in production as it doesn't have any error checking)
- or use logstash to read info directly from the bro logs and externally dump it into elasticsearch
It seems to me the logstash route is better, given that I should be able to massage the data into more "user friendly" fields that can be easily queried with elasticsearch.
So my question is, based on your experience, what is the best option? And, if you do use logstash, can you share your logstash config?
Thanks in advance,
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
I've used bro and logstash with good success...one setup is everything is on one machine, the other is remote using rsyslog to get the data to logstash. I tried going direct bro->elasticsearch, but logstash creates logstash-* shards, and bro creates bro-* shards, and kibana had a hard time seeing both. I'm currently just piping conn.log, but here's my logstash entry:
An interesting gotcha is the fact that the above doesn't see sizes as values but strings, so I had to add a mutate to get that to work:
convert => [ "resp_bytes", "integer" ]
convert => [ "resp_ip_bytes", "integer" ]
convert => [ "orig_bytes", "integer" ]
convert => [ "orig_ip_bytes", "integer" ]
Hope that helps...feel free to ping me off list if you need any help.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Bro