[Bro] script to extract elastic search mapping from header of bro-logs

[Adam Pumphrey]

You might be able to accomplish the desired end result with a dynamic template in elasticsearch.  They can be useful for this sort of thing.  Instead of doing a type -> type mapping, you’d be applying data type handling rules in ES based on the names of the fields you’re interested in.  You can do this with the ‘path_match’ option and patterns like “*.orig_h” – if you’re using/allowing dots in the field names..  Attached an example.   You can also override the default behavior for built-in data types, create sub fields or configure type to type mappings.

https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html

Adam

On Apr 26, 2017, at 2:14 AM, Frank Meier <franky.meier.1 at gmx.de> wrote:

Hello,

many of us use Elastic Search as a sink for bro-logs. I am thinking
about written a script to extract the correct mapping from the bro
header.

This would mean:
* mapping data types:
                string, addr, enum -> string
                int, count, port -> long
                interval, double -> double
                time -> epoch_millis
* setting 'not_analyzed' for types like addr where this makes no sense
* handle container types (table, set, vector)

Any ideas? Has anyone done this before?

Franky
_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro<http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170504/5abb1162/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: es_index_mapping_template.json
Type: application/json
Size: 1976 bytes
Desc: es_index_mapping_template.json
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170504/5abb1162/attachment.bin