[Bro] Bro -> Elastic Search -> Kibana Issues | Default Analyzed Fields

[Arash Fallah]

Thanks Jay, it worked.

I used index templates as suggested by Jay,
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html,
to override the mapping for all string fields in Bro. I do not know the
complete ramifications of this but it works as a bandaid fix for now.

I got the entire mapping by doing the following: curl -XGET '
http://localhost:9200/bro-*/_mapping/?pretty=true'

Then I modified it via the index template linked here:
http://pastebin.com/qRFCzaMT

Warning, Bro might change its ElasticSearch mapping / field names at any
time and this index template should only be used as a reference. I fully
expect the current ElasticSearch writer to change so to take full advantage
of ElasticSearch and Kibana, it is very much a work in progress.

On Thu, Oct 29, 2015 at 9:36 AM, Jay Swan <sanjuanswan at gmail.com> wrote:

> I think you could use dynamic templates in Elasticsearch to have it set
> desired string fields to "not_analyzed" at index creation time, but I've
> never tried it myself:
>
>
> https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html
>
> Jay
>
> On Wed, Oct 28, 2015 at 10:45 AM, Arash Fallah <af7 at umbc.edu> wrote:
>
>> Does anyone have any suggestions on working around this? I was thinking
>> of using LogStash but I'm worried about scalability with that solution.
>>
>> I understand Bro wasn't explicitly made to interface with ElasticSearch
>> and that this might be a process.
>>
>> On Mon, Oct 26, 2015 at 8:47 PM, Arash Fallah <af7 at umbc.edu> wrote:
>>
>>> Exactly, which is why I suggest disabling string analyzation in Elastic
>>> Search for this field (and all IP fields). Otherwise, the IPv6 entries are
>>> ruining the results.
>>>
>>> On Oct 26, 2015, at 8:00 PM, Mark Parsons <mark.parsons at gmail.com>
>>> wrote:
>>>
>>> Arash,
>>> I imagine that is because elasticsearch currently does not support IPv6
>>> fields.
>>>
>>> See this elasticsearch github issue for more background
>>>
>>> https://github.com/elastic/elasticsearch/issues/3714
>>>
>>> Thanks,
>>> Mark
>>>
>>> Sent from my iPhone
>>>
>>> On Oct 26, 2015, at 3:16 PM, Arash Fallah <af7 at umbc.edu> wrote:
>>>
>>> Real Quick Update:
>>>
>>> My initial analysis was incorrect. IPv4 fields are processed correctly.
>>> The issue is with IPv6 fields. Same concept, different trigger. Here is a
>>> picture illustrating the problem:
>>>
>>> http://i.imgur.com/pdxRbmX.png
>>>
>>> On Mon, Oct 26, 2015 at 2:50 PM, Arash Fallah <af7 at umbc.edu> wrote:
>>>
>>>> I am having an issue when trying to process Bro data through Elastic
>>>> Search and Kibana.
>>>>
>>>> Specifically, I am doing basic quantitative statistics such as pulling
>>>> the Top 5 Originating IP's addresses by id_orig_h. However, Elastic Search
>>>> and Kibana breaks apart this field as shown below in the screenshot linked
>>>> below:
>>>>
>>>> http://i.imgur.com/m3BH6LP.png
>>>>
>>>> Basically, for some strings, the default Elastic Search analyzer will
>>>> segment them into different pieces based on the "." character. For example,
>>>> 130.85.12.20 will be broken apart into 130, 85, 12, and 80. This is because
>>>> in the Core Type mappings, the not index attribute is not explicitly set to
>>>> no (it defaults to yes). There is no way to adjust this for existing fields.
>>>>
>>>> Here is the current mapping created by Bro:
>>>>
>>>> "id.resp_h" : {
>>>>        "type" : "string"
>>>> }
>>>>
>>>> It should be:
>>>>
>>>> "id.resp_h" : {
>>>>        "type" : "string"
>>>>         "index": "not_analyzed"
>>>> }
>>>>
>>>> Suggestions?
>>>>
>>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>> <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>>>
>>>
>>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151030/0c230339/attachment-0001.html