[Zeek] Bro Logs Ingestion

Michał Purzyński michalpurzynski1 at gmail.com
Sun May 5 19:16:51 PDT 2019


There are some good patterns here. We observed that it helps a lot to just
ship logs from each NSM sensor as soon as data is collected, with minimal,
if any, processing. That's why we ship logs (with syslog-ng) to a RabbitMQ
instance, via AMQPS. No extra processing is done there, it's just
buffering. We then have a set of python workers fetching messages from
Rabbit and doing all of the processing.

No Kafka here but just simple solutions and avoiding any processing on
endpoints.



On Sun, May 5, 2019 at 7:12 PM Mustafa Qasim <alajal at gmail.com> wrote:

> the biggest reason is absorbing back pressure from logstash or other
> ingesting tools. In past the back pressure from logstash would cause CPU
> spikes on the originating endpoint.
>
> second, we can write programs to clean, modify and enrich data before
> throwing at the ingesting tools making our log processing pipelines
> indipendedent. Giving us flexibility of migrating from logstash to Humio or
> Splunk and not worry about wasting all the efforts you put into logstash
> pipelines.
>
> ------
> *Mustafa Qasim*
> PGP: C57E0A7C
> <http://pgp.mit.edu/pks/lookup?op=get&search=0x0A9C8A5EC57E0A7C>
>
>
> On Mon, May 6, 2019 at 7:08 AM David Decker <x.faith at gmail.com> wrote:
>
>> Sorry beginner question here:
>>
>> But I know you can ingest logs into Splunk, and Elastic Search.
>>
>> So I know SecurityOnion has an ELK stack and it looks like they get sent
>> right to Logstash - ES - Kibana
>>
>> RockNSM looks almost the same but it has a stop off at Kafka before
>> forwarding to Logstash.
>>
>> Trying to figure out is there a benefit for Kafka.
>>
>> Also looking at using Splunk instead of ES.
>> I know I can use the TA and monitor the logs from splunk, but would it be
>> better to monitor from Kafka?
>>
>> I guess I need to understand more of how Kafka fits.
>>
>> Thanks
>> Dave
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190505/fd63691a/attachment-0001.html 


More information about the Zeek mailing list