[Bro] ElasticSearch plugin

[Vlad Grigorescu]

Seth Hall <seth at icir.org> writes:

> Is there anyone here relying on the elasticsearch writer plugin in the
> bro-plugins repository?  It doesn't appear to work with current
> versions of elasticsearch anymore and it has always had trouble at
> sites with high rates of logging.

I think we should be a bit cautious here. Let's not forget that this is
really an ElasticSearch and NSQ writer. I've had very good success with
NSQ at high rate, so I don't really see much value to the second
argument.

> If we don't get much of a response on this we will be deprecating
> and/or removing the elasticsearch writer. There should be more
> reliable mechanisms available soon anyway by either writing to a Kafka
> server and then forwarding to ElasticSearch or writing files as JSON
> and the forwarding to ElasticSearch.

Do we know what the specific problems are with new versions of
ElasticSearch? Since the writer is just writing out JSON, either it's
doing something that's not compatible (which I'd think would be an easy
fix), or there's an issue with the JSON writer, which would affect
people regardless of how they get their logs to ElasticSearch.

The only concrete issue I've heard of is 'no periods in field names',
which I believe there are fixes for here:

https://github.com/danielguerra69/bro-debian-elasticsearch/tree/master/bro-patch

I think the better solution would simply be to make the record separator
redef-able in the formatter. I can *maybe* see the argument for using
'.' instead of '$' in the ASCII logs, but since the other separators are
user-definable, I think this one should be as well.

As far as which is more reliable, I think that should be up to the users
to decide. Personally, I'd rather use NSQ for a number of reasons
(easier to setup and manage, latency is over an order of magnitude less
compared with Kafka, etc.), and there are issues with JSON output to the
disk as well (unnecessary IOPs as someone mentioned).

This is already out of the Bro source code; I see more benefits than
downsides to leaving it in the bro-plugins repo.

I do agree that a RELP writer would be a great addition, and then we
could just use their great collection of output modules:

http://www.rsyslog.com/doc/v8-stable/configuration/modules/idx_output.html

  --Vlad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 800 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160614/de5d50e0/attachment.bin