[Zeek] Zeek 3.1.2 and Kafka - No data flow

[Zeolla@GMail.com]

I have the plugin working with 3.1.2 here
<https://github.com/apache/metron-bro-plugin-kafka/pull/44> - feedback is
welcome.

- Jon Zeolla
Zeolla at GMail.Com


On Wed, Apr 29, 2020 at 3:35 PM Zeolla at GMail.com <zeolla at gmail.com> wrote:

> Were you able to get this working?  I'm planning to work on the bro to
> zeek cutover for the plugin soon.
>
> - Jon Zeolla
> Zeolla at GMail.Com
>
>
> On Mon, Apr 27, 2020 at 6:39 AM Zeolla at GMail.com <zeolla at gmail.com> wrote:
>
>> I have not run it on 3.1.2 yet but I recommend making your changes to the
>> plugin and running the end to end testing script at
>> https://github.com/apache/metron-bro-plugin-kafka/blob/master/docker/run_end_to_end.sh
>>
>> It was meant to help isolate issues when making changes to the plugin.
>> Also, we welcome PRs against the project so please feel free to
>> contribute.  Thanks,
>>
>> Jon Zeolla
>>
>> On Sun, Apr 26, 2020, 9:12 PM Hovsep Levi <hovsep.sanjay.levi at gmail.com>
>> wrote:
>>
>>> Hello Zeeks
>>>
>>>
>>> Has anyone succeeded to enable Kafka plugin with Zeek 3.1.2 ?  I am
>>> trying to modernize the metron-kafka plugin and have partial success.  My
>>> problem seems to be with script-land referencing.
>>>
>>> The logger node is loading the plugin OK and connects to the Kafka
>>> broker.  The broker IP is redef information found from site/local.zeek.
>>>
>>> $ bin/zeekctl diag logger-1
>>> [logger-1]
>>>
>>> No core file found.
>>>
>>> Zeek 3.1.2-debug
>>>
>>> Zeek plugins:
>>> Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)
>>>
>>> ==== No reporter.log
>>>
>>> ==== stderr.log
>>> %7|1587948661.341|RECV|rdkafka#producer-3| [..kafka messages..] ..
>>>
>>>
>>>
>>>
>>> But the worker node has a problem referencing existing variable
>>> declaration.  The logs-to-kafka.bro script expects it.  There is also
>>> suspicion with the Zeek plugins info that is different from the logger node
>>> and maybe the problem.
>>>
>>> $ bin/zeekctl diag worker-1-1
>>> [worker-1-1]
>>>
>>> No core file found.
>>>
>>> Zeek 3.1.2-debug
>>>
>>> Zeek plugins: (none found)   <<< ??? Normal for worker node ???
>>>
>>> ==== No reporter.log
>>>
>>> ==== stderr.log
>>>
>>> error in
>>> /opt/zeek/spool/installed-scripts-do-not-touch/site/custom_plugins/APACHE_KAFKA/scripts/Apache/Kafka/./logs-to-kafka.bro,
>>> line 24: unknown identifier logs_to_send, at or near "logs_to_send"
>>>
>>>
>>>
>>> The configuration is not default and explained below:
>>>
>>>
>>> The Kafka logger was installed to site/custom_plugins/APACHE_KAFKA
>>>
>>>
>>> share/zeek/site/local.zeek uses:
>>>
>>> @load custom_plugins/APACHE_KAFKA/scripts/Apache/Kafka
>>>
>>>
>>>
>>> lib/zeek/plugins/custom_plugins is a symlink to
>>> share/zeek/site/custom_plugins
>>>
>>>
>>> Using the lib symlink seems to be the only way to load the plugin, then
>>> the @load statement brings redef customizations and scripts.  This works ok
>>> for the logger node but not the worker who cannot interface with the plugin
>>> ?
>>>
>>> Another idea is have non-logger nodes bypass loading logs-to-kafka.bro
>>> but this isn't fully understood.
>>>
>>>
>>> TIA
>>>
>>> /hovsep
>>>
>>>
>>>
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200504/f519feed/attachment.html