[Bro] Bro -> Elasticsearch -> Kibana4beta -> GeoLocation

[Daniel Guerra]

Hi,

The docker image is currently not working (should have checked first). It works fine on my laptop
but docker has some troubles with it. You can build the dockerfile yourself
https://github.com/danielguerra69/bro-debian-elasticsearch <https://github.com/danielguerra69/bro-debian-elasticsearch>

> On 04 Nov 2015, at 01:59, Daniel Guerra <daniel.guerra69 at gmail.com> wrote:
> 
> Hi All,
> 
> The problem was solved like this.
> The geoip script adds the geo_location
> With the proper mapping kibana shows
> geoip data ;).
> Check 
> https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/ <https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/>
> 
> ##! Add geo_location for the originator and responder of a connection
> ##! to the connection logs.
> 
> module Conn;
> 
> export
> {
>         redef record Conn::Info +=
>         {
>                 geo_location: string &optional &log;
>         };
> }
> 
> event connection_state_remove(c: connection)
> {
>         local resp_loc = lookup_location(c$id$resp_h);
>         if (resp_loc?$longitude && resp_loc?$latitude)
> 	#geo location is just a cat lat,long
>           c$conn$geo_location= cat(resp_loc$latitude,",",resp_loc$longitude);
> }
> 
> add the mapping before reading data with 
> curl -XPUT elasticsearch:9200/_template/fixstrings_bro -d '{
>   "template": "bro-*",
>     "mappings" : {
> 	"conn" : {
> 		"geo_location" : { "type" : "geo_point” }
> 	}
>    }}'
> 
>> On 30 Oct 2015, at 19:25, Daniel Guerra <daniel.guerra69 at gmail.com <mailto:daniel.guerra69 at gmail.com>> wrote:
>> 
>> The funny thing is that elasticsearch stores the data internal
>> like the bro output is.
>> 
>> quote from the object document
>> Internally, this document is indexed as a simple, flat list of key-value pairs, something like this:
>> 
>> {
>>   "region":             "US",
>>   "manager.age":        30,
>>   "manager.name.first": "John",
>>   "manager.name.last":  "Smith"
>> }
>> Maybe this is an elasticsearch problem …
>> To make it all work ElasticSearch.cc <http://elasticsearch.cc/> has to change to do
>> the geopoint mapping. And maybe stop analyse strings like
>> user_agent to avoid chopping of the result in the first word.
>> This could be solved by using url formatted strings you want
>> to show in graphs etc (no spaces).
>> The last thing is some naming collisions elasticsearch is 
>> confused about, like version in ssh & socks, but thats easy
>> to change in their main scripts.
>> 
>> Daniel
>>> On 30 Oct 2015, at 14:46, Seth Hall <seth at icir.org <mailto:seth at icir.org>> wrote:
>>> 
>>> 
>>>> On Oct 29, 2015, at 9:33 PM, Daniel Guerra <daniel.guerra69 at gmail.com <mailto:daniel.guerra69 at gmail.com>> wrote:
>>>> 
>>>> I use the elasticsearch plugin in bro. I know logstash works fine but its
>>>> very cpu intensive. Thanx anyway. 
>>> 
>>> Technically it can be done, but it would require changes to the JSON formatter (in the core).  This is actually a pretty reasonable request (and I like the idea a lot!).  It might not be too much work to implement it, it just needs to be done.
>>> 
>>>  .Seth
>>> 
>>> --
>>> Seth Hall
>>> International Computer Science Institute
>>> (Bro) because everyone has a network
>>> http://www.bro.org/ <http://www.bro.org/>
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151104/4c7ef25d/attachment.html