[Bro] Bro with elasticsearch 2.0

[Daniel Guerra]

It has some interesting changes
https://www.elastic.co/downloads/past-releases/elasticsearch-2-0-0 <https://www.elastic.co/downloads/past-releases/elasticsearch-2-0-0>

I’m having trouble with 2.0 too but it works great with 1.7.

> On 07 Nov 2015, at 16:27, Jay Swan <sanjuanswan at gmail.com> wrote:
> 
> This isn't directly relevant to the original question, but it bit me: Elasticsearch field names that contain a "." require extra work if you're querying them from one of the language APIs that uses the dot as a dereference character. I wrote a blog post about this a while back:
> 
> https://jayswan.github.io/2015/03/01/searching-raw-fields-with-python/ <https://jayswan.github.io/2015/03/01/searching-raw-fields-with-python/>
> 
> If you plan to do a lot of work with Bro logs in Elasticsearch from the language clients, it's a good thing to be aware of.
> 
> Jay
> 
> On Fri, Nov 6, 2015 at 4:22 PM, Tim Desrochers <tgdesrochers at gmail.com <mailto:tgdesrochers at gmail.com>> wrote:
> Yes mine are there too. I should clarify more. It seems to be an issue with fields that have a dot as the last special character. So "seen.where" from Intel or "basic_constraints.ca <http://basic_constraints.ca/>" from x509. Those fields don't seem to be indexed in the new version.
> 
> On Nov 6, 2015 5:59 PM, Daniel Guerra <daniel.guerra69 at gmail.com <mailto:daniel.guerra69 at gmail.com>> wrote:
> Check the mapping script i use. All the id.orig etc are all there.
> 
> https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/elasticsearchMapping.sh <https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/elasticsearchMapping.sh>
> 
>> On 06 Nov 2015, at 23:56, Tim Desrochers <tgdesrochers at gmail.com <mailto:tgdesrochers at gmail.com>> wrote:
>> 
>> Both links reference the no dot issue.  And state they are working on a plugin (I hadn't seen a plugin mentioned before now) so that will most likely fix the issue. But I still do not get any logs with dots in the label. 
>> I see errors in the /var/log/elasticsearch directory stating cannot index field error no dot in label allowed. So I know the info is not making it into the cluster it get stopped at my indexers.
>> 
>> https://github.com/logstash-plugins/logstash-filter-mutate/issues/54 <https://github.com/logstash-plugins/logstash-filter-mutate/issues/54>
>> https://discuss.elastic.co/t/please-read-upgrading-logstash-and-elasticsearch-to-2-0/33791 <https://discuss.elastic.co/t/please-read-upgrading-logstash-and-elasticsearch-to-2-0/33791>
>> Any ideas on a temp fix before a plugin can be issued? 
>> 
>> On Nov 6, 2015 5:28 PM, Daniel Guerra <daniel.guerra69 at gmail.com <mailto:daniel.guerra69 at gmail.com>> wrote:
>> Hi,
>> I have no problems with elastic 2.0. I have made a docker container for this
>> check. It runs the git version and i patched some things (check the dockerfile).
>> https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/ <https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/>
>> I just added a elasticsearch data volume that has all indexes for bro and kibana.
>> https://hub.docker.com/r/danielguerra/bro-elasticsearch-kibana-volume/ <https://hub.docker.com/r/danielguerra/bro-elasticsearch-kibana-volume/>
>> 
>> 
>>> On 06 Nov 2015, at 22:13, Tim Desrochers <tgdesrochers at gmail.com <mailto:tgdesrochers at gmail.com>> wrote:
>>> 
>>> This may not be a question for this forum but I have raised it in the elasticsearch forum with no answers.
>>> 
>>> I just upgraded my ES cluster to  elasticsearch 2.0 and it seems that elasticsearch no longer allows for dot (.) In field names and will not send that data into the cluster. This means that any info from the Intel log, x509 log, and other fields will no longer be indexed.
>>> 
>>> Is there a work around for this. Is there a way to have bro print fields with underscores instead of periods or, and this may be easier, is there a way to have logstash look for any field name with dot and replace them with an underscore.
>>> 
>>> As with may things upgrades in one product drives changes in others. I'm not sure the reason ES 2.0 decided that field names cannot include dots but I'd love to find a way to make this work with bro once again.
>>> 
>>> Thanks
>>> 
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org <mailto:bro at bro-ids.org>
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org <mailto:bro at bro-ids.org>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151109/81458619/attachment.html