<div dir="ltr"><div dir="ltr">There are some good patterns here. We observed that it helps a lot to just ship logs from each NSM sensor as soon as data is collected, with minimal, if any, processing. That's why we ship logs (with syslog-ng) to a RabbitMQ instance, via AMQPS. No extra processing is done there, it's just buffering. We then have a set of python workers fetching messages from Rabbit and doing all of the processing.</div><div dir="ltr"><br></div><div>No Kafka here but just simple solutions and avoiding any processing on endpoints.</div><div><br></div><div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, May 5, 2019 at 7:12 PM Mustafa Qasim <<a href="mailto:alajal@gmail.com">alajal@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>the biggest reason is absorbing back pressure from logstash or other ingesting tools. In past the back pressure from logstash would cause CPU spikes on the originating endpoint.</div><div><br></div><div>second, we can write programs to clean, modify and enrich data before throwing at the ingesting tools making our log processing pipelines indipendedent. Giving us flexibility of migrating from logstash to Humio or Splunk and not worry about wasting all the efforts you put into logstash pipelines.</div><br clear="all"><div><div dir="ltr" class="gmail-m_-870303575112016214gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>------<br><b>Mustafa Qasim</b><br></div><div>PGP: <a href="http://pgp.mit.edu/pks/lookup?op=get&search=0x0A9C8A5EC57E0A7C" target="_blank">C57E0A7C</a></div></div></div></div></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 6, 2019 at 7:08 AM David Decker <<a href="mailto:x.faith@gmail.com" target="_blank">x.faith@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Sorry beginner question here:<div><br></div><div>But I know you can ingest logs into Splunk, and Elastic Search. </div><div><br></div><div>So I know SecurityOnion has an ELK stack and it looks like they get sent right to Logstash - ES - Kibana</div><div><br></div><div>RockNSM looks almost the same but it has a stop off at Kafka before forwarding to Logstash. </div><div><br></div><div>Trying to figure out is there a benefit for Kafka. </div><div><br></div><div>Also looking at using Splunk instead of ES.</div><div>I know I can use the TA and monitor the logs from splunk, but would it be better to monitor from Kafka? <br><br></div><div>I guess I need to understand more of how Kafka fits. </div><div><br></div><div>Thanks</div><div>Dave</div></div>
_______________________________________________<br>
Zeek mailing list<br>
<a href="mailto:zeek@zeek.org" target="_blank">zeek@zeek.org</a><br>
<a href="http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek" rel="noreferrer" target="_blank">http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek</a></blockquote></div>
_______________________________________________<br>
Zeek mailing list<br>
<a href="mailto:zeek@zeek.org" target="_blank">zeek@zeek.org</a><br>
<a href="http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek" rel="noreferrer" target="_blank">http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek</a></blockquote></div></div>