[Bro] Elasticsearch Writer vs logstash

James Lay jlay at slave-tothe-box.net
Fri Jan 30 05:41:47 PST 2015


On Thu, 2015-01-29 at 23:48 -0700, Luis Miguel Silva wrote:
> Dear all,
> 
> 
> 
> I'm interested in dumping my bro logs into an elastic search instance
> and, based on what I was able to learn thus far, it seems I have two
> different options:
> - use the elasticsearch writer (which the documentation says should
> not be used in production as it doesn't have any error checking)
> - or use logstash to read info directly from the bro logs and
> externally dump it into elasticsearch
> 
> 
> It seems to me the logstash route is better, given that I should be
> able to massage the data into more "user friendly" fields that can be
> easily queried with elasticsearch.
> 
> 
> So my question is, based on your experience, what is the best option?
> And, if you do use logstash, can you share your logstash config?
> 
> 
> Thanks in advance,
> Luis
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


I've used bro and logstash with good success...one setup is everything
is on one machine, the other is remote using rsyslog to get the data to
logstash.  I tried going direct bro->elasticsearch, but logstash creates
logstash-* shards, and bro creates bro-* shards, and kibana had a hard
time seeing both.  I'm currently just piping conn.log, but here's my
logstash entry:

"(?<unixtime>(.*?))\t(?<uid>(.*?))\t(?<src_ip>(.*?))\t(?<src_port>(.*?))\t(?<dst_ip>(.*?))\t(?<dst_port>(.*?))\t(?<proto>(.*?))\t(?<service>(.*?))\t(?<duration>(.*?))\t(?<orig_bytes>(.*?))\t(?<resp_bytes>(.*?))\t(?<conn_state>(.*?))\t(?<local_orig>(.*?))\t(?<missed_bytes>(.*?))\t(?<history>(.*?))\t(?<orig_packts>(.*?))\t(?<orig_ip_bytes>(.*?))\t(?<resp_packts>(.*?))\t(?<resp_ip_bytes>(.*?))\t(?<tun_parent>(.*))" 

An interesting gotcha is the fact that the above doesn't see sizes as
values but strings, so I had to add a mutate to get that to work:

mutate {
			convert => [ "resp_bytes", "integer" ]
			convert => [ "resp_ip_bytes", "integer" ]
			convert => [ "orig_bytes", "integer" ]
			convert => [ "orig_ip_bytes", "integer" ]
		}

Hope that helps...feel free to ping me off list if you need any help.

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150130/06c9d7ba/attachment.html 


More information about the Bro mailing list