[Bro-Dev] [JIRA] (BIT-1543) Kafka Logger - Writes Bro Logs to Kafka

[Robin Sommer (JIRA)]

     [ https://bro-tracker.atlassian.net/browse/BIT-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robin Sommer reassigned BIT-1543:
---------------------------------

    Assignee: Robin Sommer  (was: Seth Hall)

> Kafka Logger - Writes Bro Logs to Kafka
> ---------------------------------------
>
>                 Key: BIT-1543
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1543
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>            Reporter: Nick Allen
>            Assignee: Robin Sommer
>
> As part of the Apache Metron project, we needed a way to send Bro logs to Kafka. From my research it seems like this is a common request. I'd rather give this code back to the Bro community than maintain it as part of Apache Metron.
> This Bro plugin logs all Bro output to Kafka. Configuring this plugin is as simple as adding the following Bro script.
> {{  
>   @load Bro/Kafka/logs-to-kafka.bro
>   redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
>   redef Kafka::topic_name = "bro";
>   redef Kafka::kafka_conf = table(
>       ["metadata.broker.list"] = "localhost:9092"
>   );
> }}
> This plugin has the following features.
> * The user can specify a subset of all logs that should be sent to kafka. For example, to only send conn, http, and dns logs, specify the following.
> {{redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
> }}
> * Full configurability of Kafka connectivity. Any configuration setting accepted by the librdkafka library can be passed to the plugin to tune how the logs are sent to Kafka.
> {{redef Kafka::kafka_conf = table(
>    ["metadata.broker.list"] = "localhost:9092",
>    ["client.id"] = "bro"
> );
> }}
> * The plugin will wait a configurable period of time (for example, 3 seconds) after shutdown to attempt to send any queued messages to Kafka.
> {{redef Kafka::max_wait_on_shutdown = 3000;
> }}
> * There are two message formats to choose from. By default, the standard Bro JSON format is used. There is an alternative 'tagged JSON' format that is provided by the plugin. Currently, all messages are sent to a single Bro topic. This 'tagged JSON' format helps a Kafka consumer distinguish which log stream the message originated from. This format prepends the log stream identifier to the JSON message.
> {{{'conn': { ... }}
> {'http': { ... }}
> {'dns': { ... }}}}
> To enable this alternative format, simply specify the following.
> {{redef Kafka::tag_json = T;}}



--
This message was sent by Atlassian JIRA
(v7.2.0-OD-03-012#72000)