[Bro] ElasticSearch plugin

[Joe Blow]

So it's settled then!!  When will the RELP writer be done?!? :)

Cheers,

JB

On Tue, Jun 14, 2016 at 10:04 AM, Vlad Grigorescu <vladg at illinois.edu>
wrote:

> Seth Hall <seth at icir.org> writes:
>
> > Is there anyone here relying on the elasticsearch writer plugin in the
> > bro-plugins repository?  It doesn't appear to work with current
> > versions of elasticsearch anymore and it has always had trouble at
> > sites with high rates of logging.
>
> I think we should be a bit cautious here. Let's not forget that this is
> really an ElasticSearch and NSQ writer. I've had very good success with
> NSQ at high rate, so I don't really see much value to the second
> argument.
>
> > If we don't get much of a response on this we will be deprecating
> > and/or removing the elasticsearch writer. There should be more
> > reliable mechanisms available soon anyway by either writing to a Kafka
> > server and then forwarding to ElasticSearch or writing files as JSON
> > and the forwarding to ElasticSearch.
>
> Do we know what the specific problems are with new versions of
> ElasticSearch? Since the writer is just writing out JSON, either it's
> doing something that's not compatible (which I'd think would be an easy
> fix), or there's an issue with the JSON writer, which would affect
> people regardless of how they get their logs to ElasticSearch.
>
> The only concrete issue I've heard of is 'no periods in field names',
> which I believe there are fixes for here:
>
>
> https://github.com/danielguerra69/bro-debian-elasticsearch/tree/master/bro-patch
>
> I think the better solution would simply be to make the record separator
> redef-able in the formatter. I can *maybe* see the argument for using
> '.' instead of '$' in the ASCII logs, but since the other separators are
> user-definable, I think this one should be as well.
>
> As far as which is more reliable, I think that should be up to the users
> to decide. Personally, I'd rather use NSQ for a number of reasons
> (easier to setup and manage, latency is over an order of magnitude less
> compared with Kafka, etc.), and there are issues with JSON output to the
> disk as well (unnecessary IOPs as someone mentioned).
>
> This is already out of the Bro source code; I see more benefits than
> downsides to leaving it in the bro-plugins repo.
>
> I do agree that a RELP writer would be a great addition, and then we
> could just use their great collection of output modules:
>
> http://www.rsyslog.com/doc/v8-stable/configuration/modules/idx_output.html
>
>   --Vlad
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160614/edc689bb/attachment-0001.html