[Bro] Bro -> Elasticsearch -> Kibana4beta -> GeoLocation

[Daniel Guerra]

Hi All,

The problem was solved like this.
The geoip script adds the geo_location
With the proper mapping kibana shows
geoip data ;).
Check 
https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/

##! Add geo_location for the originator and responder of a connection
##! to the connection logs.

module Conn;

export
{
        redef record Conn::Info +=
        {
                geo_location: string &optional &log;
        };
}

event connection_state_remove(c: connection)
{
        local resp_loc = lookup_location(c$id$resp_h);
        if (resp_loc?$longitude && resp_loc?$latitude)
	#geo location is just a cat lat,long
          c$conn$geo_location= cat(resp_loc$latitude,",",resp_loc$longitude);
}

add the mapping before reading data with 
curl -XPUT elasticsearch:9200/_template/fixstrings_bro -d '{
  "template": "bro-*",
    "mappings" : {
	"conn" : {
		"geo_location" : { "type" : "geo_point” }
	}
   }}'

> On 30 Oct 2015, at 19:25, Daniel Guerra <daniel.guerra69 at gmail.com> wrote:
> 
> The funny thing is that elasticsearch stores the data internal
> like the bro output is.
> 
> quote from the object document
> Internally, this document is indexed as a simple, flat list of key-value pairs, something like this:
> 
> {
>   "region":             "US",
>   "manager.age":        30,
>   "manager.name.first": "John",
>   "manager.name.last":  "Smith"
> }
> Maybe this is an elasticsearch problem …
> To make it all work ElasticSearch.cc <http://elasticsearch.cc/> has to change to do
> the geopoint mapping. And maybe stop analyse strings like
> user_agent to avoid chopping of the result in the first word.
> This could be solved by using url formatted strings you want
> to show in graphs etc (no spaces).
> The last thing is some naming collisions elasticsearch is 
> confused about, like version in ssh & socks, but thats easy
> to change in their main scripts.
> 
> Daniel
>> On 30 Oct 2015, at 14:46, Seth Hall <seth at icir.org <mailto:seth at icir.org>> wrote:
>> 
>> 
>>> On Oct 29, 2015, at 9:33 PM, Daniel Guerra <daniel.guerra69 at gmail.com <mailto:daniel.guerra69 at gmail.com>> wrote:
>>> 
>>> I use the elasticsearch plugin in bro. I know logstash works fine but its
>>> very cpu intensive. Thanx anyway. 
>> 
>> Technically it can be done, but it would require changes to the JSON formatter (in the core).  This is actually a pretty reasonable request (and I like the idea a lot!).  It might not be too much work to implement it, it just needs to be done.
>> 
>>  .Seth
>> 
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro.org/ <http://www.bro.org/>
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151104/fd7f7ade/attachment.html