[Bro] Bro Elastic/ELK Experiences

[Azoff, Justin S]

> On Jul 24, 2017, at 12:25 PM, Landy Bible <landy-bible at utulsa.edu> wrote:
> 
> Craig,
> 
> I'm currently Redis, but I'm migrating to Kafka.
> 
> I initially chose Redis because it was super easy to set up. It's worked well for me so far, but it's limited by the amount of ram I have in my servers. I currently run 3 servers, but they aren't actually clustered. Logstash runs on my Bro server to forward the logs to Redis, and will fail over in the event that one of my Redis servers goes down. I had to write some scripts to block incoming writes if the node got too full while still allowing the Logstash agents on the other side to read from the queue. When I'm doing ELK maintenance I have to keep a close eye on my Redis boxes, I've only got a few hours worth of headroom on them, and if they fill up they'll crash and I lose data. Also, Redis operates as a single queue. I can't have multiple processes consuming the same data for different uses.
> 
> Migrating to Kafka will allow me to do real clustering at the queuing layer. Since it writes to disk I have days or weeks worth of storage rather than hours, so my ELK maintenance won't be so stressful. And since it has the concept of consumer offsets I can have the ELK system consuming the data, as well as my other custom processes consume it for other uses without impacting ELK. The downside is that it's a bit more complex to get a Kafka cluster up and running than it is to get a single Redis node up. I'm only testing it right now but so far it seems like it's worth the effort.
> 
> If you're ever in the Tulsa area, hit me up. I'd be happy to show you some details about how we're running things at TU.

You should also consider http://nsq.io/ as a replacement for redis instead of kafka.

Log stash or the bro elasticsearch plugin can write to it.  It's a lot simpler to run compared to kafka.

-- 
- Justin Azoff