[Bro] filebeat +elk

[Patrick Kelley]

I've had some issues as you described with Logstash. About the same EPS.  I
moved away from Filebeat some time ago. Unrelated issues.

Kafka has worked quite well. I recommend the Apache Metron.

https://metron.apache.org/current-book/metron-sensors/bro-plugin-kafka/index.html

bro -N should output the following:

Apache::Kafka (dynamic, version 0.2)

On Wed, Mar 28, 2018 at 2:59 PM, Blake Moss <blake_moss at byu.edu> wrote:

> On this subject, We’ve had issues with both filebeats and logstash reading
> logs (written to files) once events per second reaches upwards of 3k. We
> are currently looking into using the bro kafka plugin. Has anyone else had
> issues with logstash or filebeats bottlenecking?
>
>
>
> *From: *craig bowser <reswob10 at gmail.com>
> *Sent: *Wednesday, March 28, 2018 12:44 PM
> *To: *Daniel Guerra <daniel.guerra69 at gmail.com>
> *Cc: *bro at bro.org
> *Subject: *Re: [Bro] filebeat +elk
>
>
> So at job I was using logstash on bro and reading each file, parsing and
> enhancing the data then sending to elasticsearch. But then that was talking
> too many resources from bro, do not I'm using filebeat to send each log to
> a logstash server which parses, enhances and sends to elasticsearch.
>
> At home I'm using syslog-ng to send bro logs to logstash
>
> The suggestion to use rabbitmq is good as well.
>
> On Wed, Mar 28, 2018, 2:23 PM Daniel Guerra <daniel.guerra69 at gmail.com>
> wrote:
>
>> I would use json to stdout with a python script to
>>
>> insert it in elasticsearch. I think its the most efficient
>>
>> and stable method. The latest elasticsearch needs
>>
>> separate index for the different log types.
>>
>> There is a bro-pkg for json to stdout.
>>
>>
>>
>>
>> Op 28/03/2018 om 18:52 schreef erik clark:
>>
>> I am trying to ingest bro 2.5 json logs into an elk stack, using filebeat
>> to push the logs. Is that even the best way to do this? I have found MUCH
>> outdated material on ingesting bro logs into an elk stack, but very little
>> that is up to date, and some of which is up to date but is using older
>> versions of software from elastic.co. If anyone has a modern bro/elk
>> integration document they use(d) to set their environment up, it would be
>> greatly appreciated if you could share. Thanks!
>>
>> Erik
>>
>>
>> _______________________________________________
>> Bro mailing listbro at bro-ids.orghttp://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 

*Patrick Kelley, CISSP, C|EH, ITIL*
*CTO*
patrick.kelley at criticalpathsecurity.com
(o) 770-224-6482
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180328/6e90ad3a/attachment.html