[Bro] How use logs-to-elasticsearch.bro

Aaron Gee-Clough lists at g-clef.net
Wed Mar 2 13:42:12 PST 2016


Hi, all. (sorry for missing this conversation yesterday)

I'm the author of that Bro Kafka logging plugin ( 
https://github.com/g-clef/KafkaLogger ). If folks have any questions 
about it or issues with it, please let me know. (Happy to fix bugs if 
people hit them.)

The way I'm using it right now is that bro logs to a Kafka topic, then 
Logstash pulls the events off Kafka for insertion into Elasticsearch. 
That's working quite well for me at the moment (several thousand events 
per second going to Kafka with no noticeable impact to bro or kafka).

Aaron

On 03/01/2016 05:53 PM, Derek Ditch wrote:
> I would also add, that I use ELK almost exclusively for Bro logs, but 
> I go through a Kafka output plugin. There’s an easy setup using Chef 
> to automate for a simple test environment over at http://rocknsm.io/.
>
> Disclaimer, I’m one of the authors of that open source project.
>> Derek Ditch
> dcode at rocknsm.io <mailto:dcode at rocknsm.io>
> GPG: 0x2543A3B5
>
>> On 01Mar 2016, at 13:38, Tim Desrochers <tgdesrochers at gmail.com 
>> <mailto:tgdesrochers at gmail.com>> wrote:
>>
>> I use bro with ELK in production and it works great. I use bro to 
>> json and all my logs are in json. Then use logstash to pick up the 
>> logs and the good folks at elastic have created a plugin for de_dot. 
>> It's not perfect but with some mutates it works fine for the time 
>> being. Kibana is a fine interface to build dashboards and query the 
>> data.
>>
>> Bro and ELK integration works great with a little tweaking. I'm happy 
>> to share come configs if you're interested.
>>
>> On Mar 1, 2016 11:31, "Michael Shirk" <shirkdog.bsd at gmail.com 
>> <mailto:shirkdog.bsd at gmail.com>> wrote:
>>
>>     I am happy this came up, as I have been going through the same
>>     issues for testing Brownian vs. ELK with Bro filters
>>
>>     If it is not supported in Bro's JSON output, it would be nice to
>>     be able to configure it, as there may already be some parsing of
>>     the default JSON output of Bro with tools like Splunk.
>>
>>     --
>>     Michael Shirk
>>     Daemon Security, Inc.
>>     http://www.daemon-security.com <http://www.daemon-security.com/>
>>
>>     On Mar 1, 2016 11:06, "Seth Hall" <seth at icir.org
>>     <mailto:seth at icir.org>> wrote:
>>
>>
>>         > On Mar 1, 2016, at 3:18 AM, Daniel Guerra
>>         <daniel.guerra69 at gmail.com
>>         <mailto:daniel.guerra69 at gmail.com>> wrote:
>>         >
>>         > There is a problem with elasticsearch 2.0 and higher.
>>         > It doesn’t accept dots in field names and there are
>>         > some timestamp issues.
>>
>>         I know this discussion has been going on for a while and
>>         unfortunately I've been a bit behind the curve on keeping up
>>         with it closely.  As someone who seems to have been coping
>>         with this problem for a while, what do you recommend? Would
>>         it be best if we could do nested json documents in the json
>>         output? i.e....
>>
>>         {"ts":1223421341234.1234, "id": {"orig_h": "1.2.3.4",
>>         "orig_p":1234.......etc }}
>>
>>           .Seth
>>
>>         --
>>         Seth Hall
>>         International Computer Science Institute
>>         (Bro) because everyone has a network
>>         http://www.bro.org/
>>
>>
>>         _______________________________________________
>>         Bro mailing list
>>         bro at bro-ids.org <mailto:bro at bro-ids.org>
>>         http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>         <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>>
>>
>>     _______________________________________________
>>     Bro mailing list
>>     bro at bro-ids.org <mailto:bro at bro-ids.org>
>>     http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>     <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org <mailto:bro at bro-ids.org>
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



More information about the Bro mailing list