[Bro] ElasticSearch plugin

Joe Blow blackhole.em at gmail.com
Mon Jun 13 12:21:38 PDT 2016

What if you *hate* using logstash because liblognormalize is much, much
faster than a regex engine, like logstash?  I'd much prefer to simply get
the data into rsyslog (which should be trivial), then use rsyslog queueing
and batching which is much more flexible, IMO.  Use RELP to reliably
forward, send to kafka, do whatever, but once you've got it into rsyslog
you've got pretty solid queueing which you can use, along with a whole host
of output modules.

I'd just love to see a better pure syslog integration, without having to
write to disks, then read from the disks.  It starts potentially cause
problems when you're capturing 10Gb/s+.



On Mon, Jun 13, 2016 at 2:45 PM, Gary Faulkner <gfaulkner.nsm at gmail.com>

> I believe supporting dots in field names again is on the fix list for
> Elastic Stack v5 which is currently in development, so that part might
> at least get fixed on the Elastic end. Technically I believe that fix is
> a plus even for folks using another plugin such as Kafka as those folks
> still potentially had to do something to rewrite the field names. I
> can't speak for anything else that might be broken in the plugin.
> Here is the reference to that bit in the Elastic Blog:
> https://www.elastic.co/blog/elasticsearch-5-0-0-alpha3-released
> I could still see cases where someone could have a low volume Bro +
> local ES where it might not be desirable or necessary to run a bunch of
> other stuff in between Bro and ES, but maybe it just isn't too big a
> deal then to just let Logstash do the work of reading local log files
> assuming that one is OK with still writing normal Bro log output in
> addition to ES. An example might be if I wanted to do something akin to
> a stand-alone Security Onion node, but with Bro and Elastic instead of
> Bro and ELSA.
> I'm not using that functionality currently and will probably look at
> something like Kafka, but I already have a log volume where the overhead
> of running something like Kafka probably makes sense.
> ~Gary
> On 6/13/16 10:44 AM, Seth Hall wrote:
> > Is there anyone here relying on the elasticsearch writer plugin in the
> bro-plugins repository?  It doesn't appear to work with current versions of
> elasticsearch anymore and it has always had trouble at sites with high
> rates of logging.
> >
> > If we don't get much of a response on this we will be deprecating and/or
> removing the elasticsearch writer.  There should be more reliable
> mechanisms available soon anyway by either writing to a Kafka server and
> then forwarding to ElasticSearch or writing files as JSON and the
> forwarding to ElasticSearch.
> >
> > Thanks,
> >    .Seth
> >
> > --
> > Seth Hall
> > International Computer Science Institute
> > (Bro) because everyone has a network
> > http://www.bro.org/
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160613/e0948677/attachment.html 

More information about the Bro mailing list