[Zeek] [Non-DoD Source] Re: Kafka plugin causes logger to segfault
Zeolla@GMail.com
zeolla at gmail.com
Tue Apr 23 14:46:30 PDT 2019
Are you able to turn debug on[1] and share the details? If you need to
bring this off list for sensitivity reasons feel free to contact me
directly.
1:
https://github.com/apache/metron-bro-plugin-kafka/blob/master/README.md#debug
Jon Zeolla
On Tue, Apr 23, 2019, 4:18 PM Weasel, Gary W CIV DISA RE (US) <
gary.w.weasel2.civ at mail.mil> wrote:
> That was a typo when copying over into the email. It's a colon in the
> actual config.
>
> I'm running bro 2.6.1.
>
> It turns out there was something wrong with the Kafka pipeline, and after
> we resolved those issues, the logger stopped crashing with the confluent
> version of librdkafka, but still crashes immediately with the regular
> version (the version prescribed by zeek packages).
>
> v/r
> Gary
>
> -----Original Message-----
> From: Zeolla at GMail.com <zeolla at gmail.com>
> Sent: Tuesday, April 23, 2019 3:28 PM
> To: Patrick Cain <pcain at coopercain.com>
> Cc: Weasel, Gary W CIV DISA RE (US) <gary.w.weasel2.civ at mail.mil>;
> zeek at zeek.org
> Subject: [Non-DoD Source] Re: [Zeek] Kafka plugin causes logger to segfault
>
> All active links contained in this email were disabled. Please verify the
> identity of the sender, and confirm the authenticity of all links contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
> ________________________________
>
>
>
> 172.16.0.40.9092 doesn't appear to be an IP address to me. Did you mean
> 172.16.0.40:9092 < Caution-http://172.16.0.40:9092 > ?
>
>
> - Jon Zeolla
> Zeolla at GMail.Com
>
>
> On Tue, Apr 23, 2019 at 3:16 PM Patrick Cain <pcain at coopercain.com <
> Caution-mailto:pcain at coopercain.com > > wrote:
>
>
> Hi,
>
> You don't say what version you're running, but with 2.5 and 2.6 I
> use these
> lines along with the kafka config:
>
> ### JSON LOGGING
> @load tuning/json-logs
> # Set the log separator
> redef Log::default_scope_sep = "_";
> # Set the time in iso format
> redef LogAscii::json_timestamps = JSON::TS_ISO8601;
>
> Your kafka config looks close to mine (I leave the topic_name
> field blank.)
> My kafka emitter has been running on Centos 6, Centos 7 and RHEL7
> systems
> for about a year.
> Can you manually connect to your broker from the zeek box? I have
> had
> issues in the past when the logger was happy but other things in
> the pipe to
> zookeeper and kafka were unhappy.
>
> Pat
> -----Original Message-----
> From: zeek-bounces at zeek.org < Caution-mailto:zeek-bounces at zeek.org
> > <zeek-bounces at zeek.org < Caution-mailto:zeek-bounces at zeek.org > > On
> Behalf Of Weasel,
> Gary W CIV DISA RE (US)
> Sent: Monday, April 22, 2019 11:10 AM
> To: 'zeek at zeek.org < Caution-mailto:zeek at zeek.org > ' <
> zeek at zeek.org < Caution-mailto:zeek at zeek.org > >
> Subject: [Zeek] Kafka plugin causes logger to segfault
>
> All,
>
> I'm currently at my wits end dealing with the Kafka plugin, I'm
> having great
> difficulty stopping it from crashing.
>
> When I use the library of librdkafka as prescribed from
> Caution-
> https://packages.zeek.org/packages/view/7388aa77-4fb7-11e8-88be-0a645a3f3086
> (librdkafka-0.11.5 < Caution-
> https://packages.zeek.org/packages/view/7388aa77-4fb7-11e8-88be-0a645a3f3086(librdkafka-0.11.5
> > ), my logger crashes immediately after startup. When
> using an alternative version of librdkafka
> (librdkakfa1-0.11.4_confluent4.1.3) the logger doesn't immediately
> crash but
> within a minute of starting it usually does.
>
> The stderr.log says the same every time, /run-bro: line 110: <pid>
> Segmentation fault nohup "$mybro" "$@"
>
> I have downloaded the most recent version of
> Caution-https://github.com/apache/metron-bro-plugin-kafka <
> Caution-https://github.com/apache/metron-bro-plugin-kafka > and still
> experience this.
>
> I am building an RPM (running CentOS) for the Kafka plugin and
> installing
> that way, since the box is offline and unable to reach
> bro-packages. When I
> tried to use librdkafka-0.11.5 I've also built an RPM for that.
>
> The following is my only added configuration
>
> @load Apache/Kafka/logs-to-kafka.bro
> redef Kafka::logs_to_send = set(Conn::LOG); redef
> Kafka::kafka_conf = table(
> ["metadata.broker.list"] = "172.16.0.40.9092"
> );
> redef Kafka::topic_name = "bro";
> redef Kafka::tag_json = T;
>
> The interesting thing to note: the logger does not crash if no
> logs are
> being sent (i.e. I comment out the logs_to_send line).
>
> The only other plugins I'm running are Bro::AF_Packet and
> Corelight::CommunityID.
>
> Anyone have any insight or doing something different?
>
> v/r
> Gary
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org < Caution-mailto:zeek at zeek.org >
> Caution-http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek <
> Caution-http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek >
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org < Caution-mailto:zeek at zeek.org >
> Caution-http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek <
> Caution-http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190423/45891ad3/attachment.html
More information about the Zeek
mailing list