[Bro] Bro 2.2 beta and logstash

[James Lay]

Hey thanks a bunch for this Aaron...this totally helps me out.

James

On 2013-10-16 13:24, Aaron Gee-Clough wrote:
> I had it working for the dns.log for bro 2.1 (I'm not running 2.2
> yet), but I ended up bailing on logstash for other reasons (logstash
> would stop sending logs to ElasticSearch, for reasons I could never
> figure out...it never logged anything during the pauses). I was also
> parsing the bro logs after they were sent over to another server via
> syslog, which added another timestamp, so I wasn't trying to parse 
> the
> bro timestamp in grok. That timestamp shouldn't be hard to parse,
> though, the bro timestamp is the first  %{NUMBER} in the grok pattern
> below. (and, obviously, you'll need to change the date parsing also 
> if
> you use the bro timestamp.)
>
> In any case, here's the logstash config I used to get it going:
>
>
> input {
>     file {
>         type    => "brodns"
>         path => ["<insert path to dns.log file here>"]
>         sincedb_path => ["<insert path that the logstash user can
> write to here>"]
>     }
> }
>
> filter {
>     grok {
>         type => "brodns"
>         match => ["message",
> 
> "^%{SYSLOGTIMESTAMP:syslog_timestamp}%{SPACE}%{NUMBER}%{SPACE}%{WORD:conn_id}%{SPACE}%{IP:source_ip}%{SPACE}%{NUMBER:source_port}%{SPACE}%{IP:dest_ip}%{SPACE}%{NUMBER:dest_port}%{SPACE}%{WORD:proto}%{SPACE}%{NUMBER}%{SPACE}%{NOTSPACE:query}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE:query_class}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE:query_type}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE:response_code}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE:answers}%{SPACE}%{NOTSPACE:ttls}"]
>     }
>     date {
>         match => [ "syslog_timestamp","MMM  d HH:mm:ss", "MMM dd 
> HH:mm:ss"  ]
>         remove_field => ['syslog_timestamp']
>     }
> }
>
> output {
>     elasticsearch_http{
>         host => "<insert ES Server here>"
>         flush_size => 5000
>     }
> }
>
> aaron
>
> On 10/15/2013 05:48 PM, James Lay wrote:
>> Hey all,
>>
>> Just starting into this...wondering if anyone has done any legwork 
>> to
>> get this into logstash at all?  I've got it inputing in now, but 
>> none of
>> the fields are set yet.  I see where with 2.2.beta I can enable 
>> elastic
>> search:
>>
>> @load tuning/logs-to-elasticsearch
>>
>> but that doesn't seem to work with the embedded option of logstash.
>> Any hints getting this to be useful would help...thank you.
>>
>> James
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro