[Bro] How use logs-to-elasticsearch.bro

[Seth Hall]

> On Mar 1, 2016, at 11:47 AM, Derek Ditch <derek.ditch at gmail.com> wrote:
> 
> I actually ran into this again this morning. I patched the Elasticsearch writer and I’m still testing it [1]. It uses some code adapted from “g-clef” that built it into a Kafka output plugin.

I'm not sure about that mechanism, but I think it should be integrated deeper into Bro, like probably into the json formatter and then exposed in the writers as a configuration option.  Alternately, I could see having a configuration option for the writers (that flows through into the json formatter) which provides the structured output instead of flattened output as is done now.

> Seth, I know some people prefer different timestamp formats, it might be best to parameterize that so that it can be modified in script land using the existing Bro formatting libraries. I’ve found that TS_8601 works extremely well with the Elasticsearch Joda parsing library.

Instead of making the change as you've specified, can you add ISO8601 output as a config option as the ascii logger does?  You can even set the default to be ISO8601, but I think there is some value in having that be configurable in the same way that it is in other parts of Bro.

> I also changed the ts field name to @timestamp, since it’s almost universal now for the standard field for use in Elasticsearch data (used by Fluentd, Logstash, and even Spark).

That's actually part of a larger change that I would like to address soon .  The timestamps in each log right now are protocol specific.  You definitely don't always want to use the ts field as the @timestamp field (to keep things concrete).  We need to add a new field that is displayed whenever json output is enabled and I could even see a justification for adding it to the ascii logs that represents when the log was written.  It's basically metadata about the log line.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/