[Zeek] Cannot send logs to their individual Kafka topics
Arda Savran
asavran at layerxtech.com
Wed Apr 3 06:52:36 PDT 2019
I used the master.
I changed the beginning of my local.bro as follows and did a "broctl check"
and "broctl deploy":
#@load packages
#@load
/usr/local/bro/lib/bro/plugins/packages/metron-bro-plugin-kafka/scripts/Apache/Kafka
#redef Kafka::send_all_active_logs = T;
#redef Kafka::tag_json = T;
#redef Kafka::kafka_conf = table(["metadata.broker.list"] =
"XX.XX.XX.XX:9092");
###########
###########
@load
/usr/local/bro/lib/bro/plugins/packages/metron-bro-plugin-kafka/scripts/Apache/Kafka
redef Kafka::topic_name = "";
redef Kafka::tag_json = T;
redef Kafka::debug = "all";
event bro_init() &priority=-10
{
# handles DNS
local dns_filter: Log::Filter = [
$name = "kafka-dns",
$writer = Log::WRITER_KAFKAWRITER,
$config = table(["metadata.broker.list"] = " XX.XX.XX.XX:9092"),
*$config = table(["topic_name"] = "bro_dns"),*
$path = "dns"
];
Log::add_filter(DNS::LOG, dns_filter);
}
Still having no luck:
[root at localhost current]# tail -f stderr.log
%7|1554299460.116|CONNECT|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connecting to
ipv4#127.0.0.1:9092 (plaintext) with socket 34
%7|1554299460.116|STATE|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed
state DOWN -> CONNECT
%7|1554299460.116|BROADCAST|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: Broadcasting state change
%7|1554299460.116|BROKERFAIL|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: failed: err:
Local: Broker transport failure: (errno: Connection refused)
%7|1554299460.116|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]:
localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection
refused
%7|1554299460.116|STATE|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed
state CONNECT -> DOWN
%7|1554299460.116|BROADCAST|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: Broadcasting state change
%7|1554299460.116|BUFQ|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]:
localhost:9092/bootstrap: Purging bufq with 0 buffers
%7|1554299460.116|BUFQ|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]:
localhost:9092/bootstrap: Updating 0 buffers on connection reset
%7|1554299460.116|RECONNECT|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Delaying next
reconnect by 435ms
%7|1554299460.394|NOINFO|rdkafka#producer-1| [thrd:main]: Topic bro_dns
partition count is zero: should refresh metadata
%7|1554299460.394|METADATA|rdkafka#producer-1| [thrd:main]: Skipping
metadata refresh of 1 topic(s): no usable brokers
%7|1554299460.552|RECONNECT|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Delaying next
reconnect by 276ms
%7|1554299460.827|CONNECT|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: broker in state
DOWN connecting
%7|1554299460.827|CONNECT|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connecting to
ipv4#127.0.0.1:9092 (plaintext) with socket 34
%7|1554299460.827|STATE|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed
state DOWN -> CONNECT
%7|1554299460.827|BROADCAST|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: Broadcasting state change
%7|1554299460.827|BROKERFAIL|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: failed: err:
Local: Broker transport failure: (errno: Connection refused)
%7|1554299460.827|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]:
localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection
refused
%7|1554299460.827|STATE|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed
state CONNECT -> DOWN
%7|1554299460.827|BROADCAST|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: Broadcasting state change
%7|1554299460.827|BUFQ|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]:
localhost:9092/bootstrap: Purging bufq with 0 buffers
%7|1554299460.827|BUFQ|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]:
localhost:9092/bootstrap: Updating 0 buffers on connection reset
%7|1554299461.394|NOINFO|rdkafka#producer-1| [thrd:main]: Topic bro_dns
partition count is zero: should refresh metadata
%7|1554299461.394|METADATA|rdkafka#producer-1| [thrd:main]: Skipping
metadata refresh of 1 topic(s): no usable brokers
%7|1554299461.827|CONNECT|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: broker in state
DOWN connecting
%7|1554299461.828|CONNECT|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connecting to
ipv4#127.0.0.1:9092 (plaintext) with socket 34
%7|1554299461.828|STATE|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed
state DOWN -> CONNECT
%7|1554299461.828|BROADCAST|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: Broadcasting state change
%7|1554299461.828|BROKERFAIL|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: failed: err:
Local: Broker transport failure: (errno: Connection refused)
%7|1554299461.828|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]:
localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection
refused
%7|1554299461.828|STATE|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed
state CONNECT -> DOWN
%7|1554299461.828|BROADCAST|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: Broadcasting state change
%7|1554299461.828|BUFQ|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]:
localhost:9092/bootstrap: Purging bufq with 0 buffers
%7|1554299461.829|BUFQ|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]:
localhost:9092/bootstrap: Updating 0 buffers on connection reset
%7|1554299461.829|RECONNECT|rdkafka#producer-1|
[thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Delaying next
reconnect by 715ms
Do you have any other suggestions for me?
Thanks
On Wed, Apr 3, 2019 at 8:38 AM Zeolla at GMail.com <zeolla at gmail.com> wrote:
> Are you using master? The easiest way to fix this is likely to add a key
> of "topic_name" and a value of "dns" to your $config table, similar to as
> shown here
> <https://github.com/apache/metron-bro-plugin-kafka#example-6---sending-a-log-to-multiple-topics>.
> Please let me know if that works for you.
>
> There is a known issue in master where the plugin is not falling back to
> use $path as the destination topic name, and I have a PR open
> <https://github.com/apache/metron-bro-plugin-kafka/pull/26> for it but
> unfortunately haven't had a lot of time to finish (it is just pending some
> btests - functionally it is done) and get that merged.
>
> - Jon Zeolla
> Zeolla at GMail.Com
>
>
> On Tue, Apr 2, 2019 at 11:37 AM Arda Savran <asavran at layerxtech.com>
> wrote:
>
>> Hello folks:
>>
>> I have successfully been able to send everything to a remote single Kafka
>> Topic from a local Bro machine and following is my local.bro file to make
>> that happen:
>>
>> *##! Local site policy. Customize as appropriate.*
>> *##!*
>> *##! This file will not be overwritten when upgrading or reinstalling!*
>>
>> *#@load packages*
>>
>> *@load
>> /usr/local/bro/lib/bro/plugins/packages/metron-bro-plugin-kafka/scripts/Apache/Kafka*
>> *redef Kafka::send_all_active_logs = T;*
>> *redef Kafka::tag_json = T;*
>> *redef Kafka::kafka_conf = table(["metadata.broker.list"] =
>> "XX.XX.XX.XX:9092");*
>>
>> However, when I change that to write logs to their individual Kafka
>> topics I get an error message under stderr.log. Following is my updated
>> local.bro:
>>
>> *##! Local site policy. Customize as appropriate.*
>> *##!*
>> *##! This file will not be overwritten when upgrading or reinstalling!*
>>
>> *#@load packages*
>>
>> *#@load
>> /usr/local/bro/lib/bro/plugins/packages/metron-bro-plugin-kafka/scripts/Apache/Kafka*
>> *#redef Kafka::send_all_active_logs = T;*
>> *#redef Kafka::tag_json = T;*
>> *#redef Kafka::kafka_conf = table(["metadata.broker.list"] =
>> "XX.XX.XX.XX:9092");*
>>
>> *###########*
>> *###########*
>>
>> *@load
>> /usr/local/bro/lib/bro/plugins/packages/metron-bro-plugin-kafka/scripts/Apache/Kafka*
>> *redef Kafka::topic_name = "";*
>> *redef Kafka::tag_json = T;*
>> *redef Kafka::debug = "all";*
>>
>> *event bro_init() &priority=-10*
>> *{*
>> *# handles DNS*
>> *local dns_filter: Log::Filter = [*
>> *$name = "kafka-dns",*
>> *$writer = Log::WRITER_KAFKAWRITER,*
>> *$config = table(["metadata.broker.list"] = "XX.XX.XX.XX:9092"),*
>> *$path = "dns"*
>> *];*
>> *Log::add_filter(DNS::LOG, dns_filter);*
>> *}*
>>
>> *###########*
>> *###########*
>>
>> I enter "broctl check" and "broctl deploy" after that; but get the
>> following:
>>
>> [root at localhost current]# tail -f stderr.log
>> %7|1554218121.957|STATE|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed
>> state DOWN -> CONNECT
>> %7|1554218121.957|BROADCAST|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: Broadcasting state change
>> %7|1554218121.957|BROKERFAIL|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: failed: err:
>> Local: Broker transport failure: (errno: Connection refused)
>> %7|1554218121.957|FAIL|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#
>> 127.0.0.1:9092 failed: Connection refused
>> %7|1554218121.957|STATE|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed
>> state CONNECT -> DOWN
>> %7|1554218121.957|BROADCAST|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: Broadcasting state change
>> %7|1554218121.957|BUFQ|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Purging bufq
>> with 0 buffers
>> %7|1554218121.957|BUFQ|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Updating 0
>> buffers on connection reset
>> %7|1554218122.309|NOINFO|rdkafka#producer-1| [thrd:main]: Topic
>> partition count is zero: should refresh metadata
>> %7|1554218122.309|METADATA|rdkafka#producer-1| [thrd:main]: Skipping
>> metadata refresh of 1 topic(s): no usable brokers
>> %7|1554218122.957|CONNECT|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: broker in state
>> DOWN connecting
>> %7|1554218122.958|CONNECT|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connecting to
>> ipv4#127.0.0.1:9092 (plaintext) with socket 29
>> %7|1554218122.958|STATE|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed
>> state DOWN -> CONNECT
>> %7|1554218122.958|BROADCAST|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: Broadcasting state change
>> %7|1554218122.958|BROKERFAIL|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: failed: err:
>> Local: Broker transport failure: (errno: Connection refused)
>> %7|1554218122.958|FAIL|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#
>> 127.0.0.1:9092 failed: Connection refused
>> %7|1554218122.958|STATE|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed
>> state CONNECT -> DOWN
>> %7|1554218122.958|BROADCAST|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: Broadcasting state change
>> %7|1554218122.958|BUFQ|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Purging bufq
>> with 0 buffers
>> %7|1554218122.958|BUFQ|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Updating 0
>> buffers on connection reset
>> %7|1554218122.958|RECONNECT|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Delaying next
>> reconnect by 301ms
>> %7|1554218123.259|RECONNECT|rdkafka#producer-1|
>> [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Delaying next
>> reconnect by 53ms
>> %7|1554218123.309|NOINFO|rdkafka#producer-1| [thrd:main]: Topic
>> partition count is zero: should refresh metadata
>>
>> Yes, I have iptables enabled on the local bro machine but it works with
>> the first configuration option file. How come bro thinks that the kafka
>> broker is local. It is supposed to send the messages to XX.XX.XX.XX.
>>
>> Thanks in advance.
>>
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190403/b145311a/attachment-0001.html
More information about the Zeek
mailing list