[Bro-Dev] [JIRA] (BIT-1543) Kafka Logger - Writes Bro Logs to Kafka

Nick Allen (JIRA) jira at bro-tracker.atlassian.net
Wed Mar 2 06:56:00 PST 2016


Nick Allen created BIT-1543:
-------------------------------

             Summary: Kafka Logger - Writes Bro Logs to Kafka
                 Key: BIT-1543
                 URL: https://bro-tracker.atlassian.net/browse/BIT-1543
             Project: Bro Issue Tracker
          Issue Type: Problem
          Components: Bro
            Reporter: Nick Allen


As part of the Apache Metron project, we needed a way to send Bro logs to Kafka. From my research it seems like this is a common request. I'd rather give this code back to the Bro community than maintain it as part of Apache Metron.

This Bro plugin logs all Bro output to Kafka. Configuring this plugin is as simple as adding the following Bro script.

{{  
  @load Bro/Kafka/logs-to-kafka.bro
  redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
  redef Kafka::topic_name = "bro";
  redef Kafka::kafka_conf = table(
      ["metadata.broker.list"] = "localhost:9092"
  );
}}

This plugin has the following features.

* The user can specify a subset of all logs that should be sent to kafka. For example, to only send conn, http, and dns logs, specify the following.

{{redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
}}

* Full configurability of Kafka connectivity. Any configuration setting accepted by the librdkafka library can be passed to the plugin to tune how the logs are sent to Kafka.

{{redef Kafka::kafka_conf = table(
   ["metadata.broker.list"] = "localhost:9092",
   ["client.id"] = "bro"
);
}}

* The plugin will wait a configurable period of time (for example, 3 seconds) after shutdown to attempt to send any queued messages to Kafka.

{{redef Kafka::max_wait_on_shutdown = 3000;
}}

* There are two message formats to choose from. By default, the standard Bro JSON format is used. There is an alternative 'tagged JSON' format that is provided by the plugin. Currently, all messages are sent to a single Bro topic. This 'tagged JSON' format helps a Kafka consumer distinguish which log stream the message originated from. This format prepends the log stream identifier to the JSON message.

{{{'conn': { ... }}
{'http': { ... }}
{'dns': { ... }}}}

To enable this alternative format, simply specify the following.

{{redef Kafka::tag_json = T;}}



--
This message was sent by Atlassian JIRA
(v7.2.0-OD-03-010#72000)


More information about the bro-dev mailing list