[Zeek] Bro Logs Ingestion

[Bible, Landy]

David,

Think of Kafka as a message queue between Zeek and ELK.

I think it depends partly on the scale of your setup. On a small system, it's probably not needed, but as you scale up it becomes much more useful. I ran a Zeek system with ELK for a few years before moving from security to networking. My ELK system ingested about 3 billion messages a week. I used Redis between Zeek and ELK. Redis was just a message queue. It provided a buffer between Zeek and ELK. That helped smooth out bursts of log messages from Zeek since it could generate messages a lot faster than my ELK cluster could process them. It also meant I could take the ELK cluster down for maintenance without stopping Zeek. Messages would just queue up in Redis until I brought ELK back up.

Once I found Kafka I had planned to replace Redis with it, but I was offered the networking gig before I made that happen. Kafka had the benefits of being properly clustered itself and wrote to disk instead of RAM which was the main limitation of using Redis. Kafka also had the feature of being able to support multiple independent consumers so that I could have data feeding into multiple systems if I wanted to.


--

Landy Bible

​Senior Network Engineer

The University of Tulsa

________________________________
From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> on behalf of David Decker <x.faith at gmail.com>
Sent: Sunday, May 5, 2019 3:58 PM
To: Zeek at zeek.org
Subject: [Zeek] Bro Logs Ingestion

Sorry beginner question here:

But I know you can ingest logs into Splunk, and Elastic Search.

So I know SecurityOnion has an ELK stack and it looks like they get sent right to Logstash - ES - Kibana

RockNSM looks almost the same but it has a stop off at Kafka before forwarding to Logstash.

Trying to figure out is there a benefit for Kafka.

Also looking at using Splunk instead of ES.
I know I can use the TA and monitor the logs from splunk, but would it be better to monitor from Kafka?

I guess I need to understand more of how Kafka fits.

Thanks
Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190505/15658dc5/attachment.html