[Zeek] Cannot send logs to their individual Kafka topics

Wed Apr 3 05:38:43 PDT 2019

Are you using master?  The easiest way to fix this is likely to add a key
of "topic_name" and a value of "dns" to your $config table, similar to as
shown here
<https://github.com/apache/metron-bro-plugin-kafka#example-6---sending-a-log-to-multiple-topics>.
Please let me know if that works for you.

There is a known issue in master where the plugin is not falling back to
use $path as the destination topic name, and I have a PR open
<https://github.com/apache/metron-bro-plugin-kafka/pull/26> for it but
unfortunately haven't had a lot of time to finish (it is just pending some
btests - functionally it is done) and get that merged.

- Jon Zeolla
Zeolla at GMail.Com

On Tue, Apr 2, 2019 at 11:37 AM Arda Savran <asavran at layerxtech.com> wrote:

> Hello folks:
>
> I have successfully been able to send everything to a remote single Kafka
> Topic from a local Bro machine and following is my local.bro file to make
> that happen:
>
> *##! Local site policy. Customize as appropriate.*
> *##!*
> *##! This file will not be overwritten when upgrading or reinstalling!*
>
> *#@load packages*
>
> *@load
> /usr/local/bro/lib/bro/plugins/packages/metron-bro-plugin-kafka/scripts/Apache/Kafka*
> *redef Kafka::send_all_active_logs = T;*
> *redef Kafka::tag_json = T;*
> *redef Kafka::kafka_conf = table(["metadata.broker.list"] =
> "XX.XX.XX.XX:9092");*
>
> However, when I change that to write logs to their individual Kafka topics
> I get an error message under stderr.log. Following is my updated local.bro:
>
> *##! Local site policy. Customize as appropriate.*
> *##!*
> *##! This file will not be overwritten when upgrading or reinstalling!*
>
> *#@load packages*
>
> *#@load
> /usr/local/bro/lib/bro/plugins/packages/metron-bro-plugin-kafka/scripts/Apache/Kafka*
> *#redef Kafka::send_all_active_logs = T;*
> *#redef Kafka::tag_json = T;*
> *#redef Kafka::kafka_conf = table(["metadata.broker.list"] =
> "XX.XX.XX.XX:9092");*
>
> *###########*
> *###########*
>
> *@load
> /usr/local/bro/lib/bro/plugins/packages/metron-bro-plugin-kafka/scripts/Apache/Kafka*
> *redef Kafka::topic_name = "";*
> *redef Kafka::tag_json = T;*
> *redef Kafka::debug = "all";*
>
> *event bro_init() &priority=-10*
> *{*
> *# handles DNS*
> *local dns_filter: Log::Filter = [*
> *$name = "kafka-dns",*
> *$writer = Log::WRITER_KAFKAWRITER,*
> *$config = table(["metadata.broker.list"] = "XX.XX.XX.XX:9092"),*
> *$path = "dns"*
> *];*
> *Log::add_filter(DNS::LOG, dns_filter);*
> *}*
>
> *###########*
> *###########*
>
> I enter "broctl check" and "broctl deploy" after that; but get the
> following:
>
> [root at localhost current]# tail -f stderr.log
> %7|1554218121.957|STATE|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed
> state DOWN -> CONNECT
> %7|1554218121.957|BROADCAST|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: Broadcasting state change
> %7|1554218121.957|BROKERFAIL|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: failed: err:
> Local: Broker transport failure: (errno: Connection refused)
> %7|1554218121.957|FAIL|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#
> 127.0.0.1:9092 failed: Connection refused
> %7|1554218121.957|STATE|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed
> state CONNECT -> DOWN
> %7|1554218121.957|BROADCAST|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: Broadcasting state change
> %7|1554218121.957|BUFQ|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Purging bufq
> with 0 buffers
> %7|1554218121.957|BUFQ|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Updating 0
> buffers on connection reset
> %7|1554218122.309|NOINFO|rdkafka#producer-1| [thrd:main]: Topic  partition
> count is zero: should refresh metadata
> %7|1554218122.309|METADATA|rdkafka#producer-1| [thrd:main]: Skipping
> metadata refresh of 1 topic(s): no usable brokers
> %7|1554218122.957|CONNECT|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: broker in state
> DOWN connecting
> %7|1554218122.958|CONNECT|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connecting to
> ipv4#127.0.0.1:9092 (plaintext) with socket 29
> %7|1554218122.958|STATE|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed
> state DOWN -> CONNECT
> %7|1554218122.958|BROADCAST|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: Broadcasting state change
> %7|1554218122.958|BROKERFAIL|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: failed: err:
> Local: Broker transport failure: (errno: Connection refused)
> %7|1554218122.958|FAIL|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#
> 127.0.0.1:9092 failed: Connection refused
> %7|1554218122.958|STATE|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed
> state CONNECT -> DOWN
> %7|1554218122.958|BROADCAST|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: Broadcasting state change
> %7|1554218122.958|BUFQ|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Purging bufq
> with 0 buffers
> %7|1554218122.958|BUFQ|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Updating 0
> buffers on connection reset
> %7|1554218122.958|RECONNECT|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Delaying next
> reconnect by 301ms
> %7|1554218123.259|RECONNECT|rdkafka#producer-1|
> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Delaying next
> reconnect by 53ms
> %7|1554218123.309|NOINFO|rdkafka#producer-1| [thrd:main]: Topic  partition
> count is zero: should refresh metadata
>
> Yes, I have iptables enabled on the local bro machine but it works with
> the first configuration option file. How come bro thinks that the kafka
> broker is local. It is supposed to send the messages to XX.XX.XX.XX.
>
> Thanks in advance.
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190403/189db9c9/attachment.html 