[Bro] Bro -> Elastic Search -> Kibana Issues | Default Analyzed Fields

[Arash Fallah]

Does anyone have any suggestions on working around this? I was thinking of
using LogStash but I'm worried about scalability with that solution.

I understand Bro wasn't explicitly made to interface with ElasticSearch and
that this might be a process.

On Mon, Oct 26, 2015 at 8:47 PM, Arash Fallah <af7 at umbc.edu> wrote:

> Exactly, which is why I suggest disabling string analyzation in Elastic
> Search for this field (and all IP fields). Otherwise, the IPv6 entries are
> ruining the results.
>
> On Oct 26, 2015, at 8:00 PM, Mark Parsons <mark.parsons at gmail.com> wrote:
>
> Arash,
> I imagine that is because elasticsearch currently does not support IPv6
> fields.
>
> See this elasticsearch github issue for more background
>
> https://github.com/elastic/elasticsearch/issues/3714
>
> Thanks,
> Mark
>
> Sent from my iPhone
>
> On Oct 26, 2015, at 3:16 PM, Arash Fallah <af7 at umbc.edu> wrote:
>
> Real Quick Update:
>
> My initial analysis was incorrect. IPv4 fields are processed correctly.
> The issue is with IPv6 fields. Same concept, different trigger. Here is a
> picture illustrating the problem:
>
> http://i.imgur.com/pdxRbmX.png
>
> On Mon, Oct 26, 2015 at 2:50 PM, Arash Fallah <af7 at umbc.edu> wrote:
>
>> I am having an issue when trying to process Bro data through Elastic
>> Search and Kibana.
>>
>> Specifically, I am doing basic quantitative statistics such as pulling
>> the Top 5 Originating IP's addresses by id_orig_h. However, Elastic Search
>> and Kibana breaks apart this field as shown below in the screenshot linked
>> below:
>>
>> http://i.imgur.com/m3BH6LP.png
>>
>> Basically, for some strings, the default Elastic Search analyzer will
>> segment them into different pieces based on the "." character. For example,
>> 130.85.12.20 will be broken apart into 130, 85, 12, and 80. This is because
>> in the Core Type mappings, the not index attribute is not explicitly set to
>> no (it defaults to yes). There is no way to adjust this for existing fields.
>>
>> Here is the current mapping created by Bro:
>>
>> "id.resp_h" : {
>>        "type" : "string"
>> }
>>
>> It should be:
>>
>> "id.resp_h" : {
>>        "type" : "string"
>>         "index": "not_analyzed"
>> }
>>
>> Suggestions?
>>
>>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151028/7610bb56/attachment.html