[Bro] ElasticSearch plugin

[Daniel Guerra]

Hi All,

I have been playing with elastic for a while. It works well and
besides the dot there are a few script changes needed to
avoid name/type confusion. A few have been solved but
I use these changes in my docker image on this subject.
Mapping is also very important to make things work. After
this you are ready to dump. For the kibana config I used
 elasticsearchdump (a alpine elasticdump). I preconfigured
kibana with searches, visualisations and dashboards.
In the ideal world, I would write to kafka combined with
an elastic-river for kafka. Graylog is implented like this.
But compiling the kafka plugin ends in complains, it needs
more time and reading installing etc. TODO ... 
Currently I’m quite happy with my elastic combination,
it is way faster when there are no errors, and elastic does
a lot with the current git. Elastic is memory hungry and prefers
to run on 3 nodes.

Regards,

Daniel

For the details on docker check this (I had to split them because
of dockerhub compile time) .
#docker-compose
https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/docker-compose.yml

#docker image (check develop for your source experiments)
https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/

#preperations
https://github.com/danielguerra69/debian-bro-develop

#compiling bro
https://github.com/danielguerra69/bro-debian-elasticsearch


>>>>>>>>bro script changes<<<<<

RUN sed -i "s/version:     count           \&log/socks_version:     count           \&log/g" /usr/local/bro/share/bro/base/protocols/socks/main.bro
RUN sed -i "s/\$version=/\$socks_version=/g" /usr/local/bro/share/bro/base/protocols/socks/main.bro
RUN sed -i "s/version:          string \&log/ssl_version:     string \&log/g" /usr/local/bro/share/bro/base/protocols/ssl/main.bro
RUN sed -i "s/\$version=/\$ssl_version=/g" /usr/local/bro/share/bro/base/protocols/ssl/main.bro
RUN sed -i "s/version:         count        \&log/ssh_version:         count        \&log/g" /usr/local/bro/share/bro/base/protocols/ssh/main.bro
RUN sed -i "s/\$version =/\$ssh_version =/g" /usr/local/bro/share/bro/base/protocols/ssh/main.bro
RUN sed -i "s/version: string \&log/snmp_version: string \&log/g" /usr/local/bro/share/bro/base/protocols/snmp/main.bro
RUN sed -i "s/\$version=/\$snmp_version=/g" /usr/local/bro/share/bro/base/protocols/snmp/main.bro


>>>>>>> mapping script <<<<<<< 

#!/bin/bash
until curl -XGET elasticsearch:9200/; do
  >&2 echo "Elasticsearch is unavailable - sleeping"
  sleep 5
done

>&2 echo "Elasticsearch is up - executing command"
curl -XPUT elasticsearch:9200/_template/fixstrings_bro -d '{
  "template": "bro-*",
    "index": {
      "number_of_shards": 7,
      "number_of_replicas": 1
    },
    "mappings" : {
      "http" : {
        "properties" : {
          "status_msg" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "user_agent" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "uri" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      },
        "conn" : {
          "properties" : {
            "orig_location" : {
              "type" : "geo_point"
            },
            "resp_location" : {
              "type" : "geo_point"
            }
          }
      },
      "files" : {
        "properties" : {
          "mime_type" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      },
      "location": {
        "properties" : {
          "ext_location" : {
            "type" : "geo_point"
          }
        }
      },
      "notice" : {
        "properties" : {
          "note" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      },
      "ssl" : {
        "properties" : {
          "validation_status" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "server_name" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      },
      "dns" : {
        "properties" : {
          "answers" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "query" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      },
      "intel" : {
        "properties" : {
          "sources" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "seen_indicator_type" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "seen_where" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      },
      "weird" : {
        "properties" : {
          "name" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "query" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      }
    }
  }'



> On 15 Jun 2016, at 04:14, Seth Hall <seth at icir.org> wrote:
> 
> 
>> On Jun 14, 2016, at 12:03 PM, Landy Bible <landy-bible at utulsa.edu> wrote:
>> 
>> For what it's worth, using the de_dot filter in logstash
> 
> Hah!  Interesting.
> 
> I wanted to briefly thank everyone that has participated in this thread so far.  It's really worthwhile to hear where people are struggling and see how everyone has addressed things for their own situation.  We are still working on making it easier to do the sort of integration that everyone is working toward and should hopefully be addressing some of the pain points in the 2.5 release.
> 
>  .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro