[Bro] Bro with elasticsearch 2.0

[Jay Swan]

This isn't directly relevant to the original question, but it bit me:
Elasticsearch field names that contain a "." require extra work if you're
querying them from one of the language APIs that uses the dot as a
dereference character. I wrote a blog post about this a while back:

https://jayswan.github.io/2015/03/01/searching-raw-fields-with-python/

If you plan to do a lot of work with Bro logs in Elasticsearch from the
language clients, it's a good thing to be aware of.

Jay

On Fri, Nov 6, 2015 at 4:22 PM, Tim Desrochers <tgdesrochers at gmail.com>
wrote:

> Yes mine are there too. I should clarify more. It seems to be an issue
> with fields that have a dot as the last special character. So "seen.where"
> from Intel or "basic_constraints.ca" from x509. Those fields don't seem
> to be indexed in the new version.
> On Nov 6, 2015 5:59 PM, Daniel Guerra <daniel.guerra69 at gmail.com> wrote:
>
> Check the mapping script i use. All the id.orig etc are all there.
>
>
> https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/elasticsearchMapping.sh
>
> On 06 Nov 2015, at 23:56, Tim Desrochers <tgdesrochers at gmail.com> wrote:
>
> Both links reference the no dot issue.  And state they are working on a
> plugin (I hadn't seen a plugin mentioned before now) so that will most
> likely fix the issue. But I still do not get any logs with dots in the
> label.
> I see errors in the /var/log/elasticsearch directory stating cannot index
> field error no dot in label allowed. So I know the info is not making it
> into the cluster it get stopped at my indexers.
>
> https://github.com/logstash-plugins/logstash-filter-mutate/issues/54
>
>
> https://discuss.elastic.co/t/please-read-upgrading-logstash-and-elasticsearch-to-2-0/33791
>
> Any ideas on a temp fix before a plugin can be issued?
> On Nov 6, 2015 5:28 PM, Daniel Guerra <daniel.guerra69 at gmail.com> wrote:
>
> Hi,
> I have no problems with elastic 2.0. I have made a docker container for
> this
> check. It runs the git version and i patched some things (check the
> dockerfile).
> https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/
> I just added a elasticsearch data volume that has all indexes for bro and
> kibana.
> https://hub.docker.com/r/danielguerra/bro-elasticsearch-kibana-volume/
>
>
> On 06 Nov 2015, at 22:13, Tim Desrochers <tgdesrochers at gmail.com> wrote:
>
> This may not be a question for this forum but I have raised it in the
> elasticsearch forum with no answers.
>
> I just upgraded my ES cluster to  elasticsearch 2.0 and it seems that
> elasticsearch no longer allows for dot (.) In field names and will not send
> that data into the cluster. This means that any info from the Intel log,
> x509 log, and other fields will no longer be indexed.
>
> Is there a work around for this. Is there a way to have bro print fields
> with underscores instead of periods or, and this may be easier, is there a
> way to have logstash look for any field name with dot and replace them with
> an underscore.
>
> As with may things upgrades in one product drives changes in others. I'm
> not sure the reason ES 2.0 decided that field names cannot include dots but
> I'd love to find a way to make this work with bro once again.
>
> Thanks
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151107/2f06ccec/attachment-0001.html