[Bro] Elasticsearch Writer vs logstash

[Swan, Jay]

Yet another option: use nxlog on the Bro node. Have it forward the logs to Logstash as raw JSON and use the json_lines codec in Logstash to feed to Elasticsearch.

The reason I like this option is that it allows you to do complex processing locally rather than use a lot of complex grok filters in Logstash, which can be really slow. Nxlog can do type conversions, regex filters, and a lot more. It also keeps your Logstash config simple. I configure nxlog to use a separate TCP or UDP output using JSON formatting, then my Logstash config just looks like:

input {
               tcp {
                              port => 5000
                              type => bro_dns
                              codec => json_lines
               }
               Etc….
}

I would love to have the native Elasticsearch writer fixed up and blessed for production use, though!

Jay

From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of James Lay
Sent: Friday, January 30, 2015 6:42 AM
To: bro at bro.org
Subject: Re: [Bro] Elasticsearch Writer vs logstash

On Thu, 2015-01-29 at 23:48 -0700, Luis Miguel Silva wrote:
Dear all,

I'm interested in dumping my bro logs into an elastic search instance and, based on what I was able to learn thus far, it seems I have two different options:
- use the elasticsearch writer (which the documentation says should not be used in production as it doesn't have any error checking)
- or use logstash to read info directly from the bro logs and externally dump it into elasticsearch

It seems to me the logstash route is better, given that I should be able to massage the data into more "user friendly" fields that can be easily queried with elasticsearch.

So my question is, based on your experience, what is the best option? And, if you do use logstash, can you share your logstash config?

Thanks in advance,
Luis



_______________________________________________

Bro mailing list

bro at bro-ids.org<mailto:bro at bro-ids.org>

http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

I've used bro and logstash with good success...one setup is everything is on one machine, the other is remote using rsyslog to get the data to logstash.  I tried going direct bro->elasticsearch, but logstash creates logstash-* shards, and bro creates bro-* shards, and kibana had a hard time seeing both.  I'm currently just piping conn.log, but here's my logstash entry:

"(?<unixtime>(.*?))\t(?<uid>(.*?))\t(?<src_ip>(.*?))\t(?<src_port>(.*?))\t(?<dst_ip>(.*?))\t(?<dst_port>(.*?))\t(?<proto>(.*?))\t(?<service>(.*?))\t(?<duration>(.*?))\t(?<orig_bytes>(.*?))\t(?<resp_bytes>(.*?))\t(?<conn_state>(.*?))\t(?<local_orig>(.*?))\t(?<missed_bytes>(.*?))\t(?<history>(.*?))\t(?<orig_packts>(.*?))\t(?<orig_ip_bytes>(.*?))\t(?<resp_packts>(.*?))\t(?<resp_ip_bytes>(.*?))\t(?<tun_parent>(.*))"

An interesting gotcha is the fact that the above doesn't see sizes as values but strings, so I had to add a mutate to get that to work:

mutate {
convert => [ "resp_bytes", "integer" ]
convert => [ "resp_ip_bytes", "integer" ]
convert => [ "orig_bytes", "integer" ]
convert => [ "orig_ip_bytes", "integer" ]
}

Hope that helps...feel free to ping me off list if you need any help.

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150130/4ca17375/attachment.html