[Bro] Elasticsearch Writer vs logstash

James Lay jlay at slave-tothe-box.net
Fri Jan 30 05:41:47 PST 2015

On Thu, 2015-01-29 at 23:48 -0700, Luis Miguel Silva wrote:
> Dear all,
> I'm interested in dumping my bro logs into an elastic search instance
> and, based on what I was able to learn thus far, it seems I have two
> different options:
> - use the elasticsearch writer (which the documentation says should
> not be used in production as it doesn't have any error checking)
> - or use logstash to read info directly from the bro logs and
> externally dump it into elasticsearch
> It seems to me the logstash route is better, given that I should be
> able to massage the data into more "user friendly" fields that can be
> easily queried with elasticsearch.
> So my question is, based on your experience, what is the best option?
> And, if you do use logstash, can you share your logstash config?
> Thanks in advance,
> Luis
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

I've used bro and logstash with good success...one setup is everything
is on one machine, the other is remote using rsyslog to get the data to
logstash.  I tried going direct bro->elasticsearch, but logstash creates
logstash-* shards, and bro creates bro-* shards, and kibana had a hard
time seeing both.  I'm currently just piping conn.log, but here's my
logstash entry:


An interesting gotcha is the fact that the above doesn't see sizes as
values but strings, so I had to add a mutate to get that to work:

mutate {
			convert => [ "resp_bytes", "integer" ]
			convert => [ "resp_ip_bytes", "integer" ]
			convert => [ "orig_bytes", "integer" ]
			convert => [ "orig_ip_bytes", "integer" ]

Hope that helps...feel free to ping me off list if you need any help.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150130/06c9d7ba/attachment.html 

More information about the Bro mailing list