[Bro] filebeat +elk

Perry, David perry29 at llnl.gov
Wed Mar 28 13:42:01 PDT 2018


I had logstash reading Bro http log and then turned on DNS lookup in logstash.  It quickly got overloaded.  I turned off DNS in logstash and had no more issues of that sort.  Logstash geoip-lite is able to keep up.

I am not using JSON files, btw.

David


On Mar 28, 2018, at 11:59 AM, Blake Moss <blake_moss at byu.edu<mailto:blake_moss at byu.edu>> wrote:

On this subject, We’ve had issues with both filebeats and logstash reading logs (written to files) once events per second reaches upwards of 3k. We are currently looking into using the bro kafka plugin. Has anyone else had issues with logstash or filebeats bottlenecking?

From: craig bowser<mailto:reswob10 at gmail.com>
Sent: Wednesday, March 28, 2018 12:44 PM
To: Daniel Guerra<mailto:daniel.guerra69 at gmail.com>
Cc: bro at bro.org<mailto:bro at bro.org>
Subject: Re: [Bro] filebeat +elk

So at job I was using logstash on bro and reading each file, parsing and enhancing the data then sending to elasticsearch. But then that was talking too many resources from bro, do not I'm using filebeat to send each log to a logstash server which parses, enhances and sends to elasticsearch.

At home I'm using syslog-ng to send bro logs to logstash

The suggestion to use rabbitmq is good as well.

On Wed, Mar 28, 2018, 2:23 PM Daniel Guerra <daniel.guerra69 at gmail.com<mailto:daniel.guerra69 at gmail.com>> wrote:

I would use json to stdout with a python script to

insert it in elasticsearch. I think its the most efficient

and stable method. The latest elasticsearch needs

separate index for the different log types.

There is a bro-pkg for json to stdout.



Op 28/03/2018 om 18:52 schreef erik clark:
I am trying to ingest bro 2.5 json logs into an elk stack, using filebeat to push the logs. Is that even the best way to do this? I have found MUCH outdated material on ingesting bro logs into an elk stack, but very little that is up to date, and some of which is up to date but is using older versions of software from elastic.co<http://elastic.co/>. If anyone has a modern bro/elk integration document they use(d) to set their environment up, it would be greatly appreciated if you could share. Thanks!

Erik



_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro<http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>

_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro<http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro<http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180328/97020655/attachment-0001.html 


More information about the Bro mailing list