[Zeek] Zeek 3.1.2 and Kafka - No data flow

[Hovsep Levi]

It is not working yet for me and was set aside to fix another time.

Very glad to hear about pull 44, I will test !

/Hovsep


On Mon, May 4, 2020 at 5:01 PM Zeolla at GMail.com <zeolla at gmail.com> wrote:

> I have the plugin working with 3.1.2 here
> <https://github.com/apache/metron-bro-plugin-kafka/pull/44> - feedback is
> welcome.
>
> - Jon Zeolla
> Zeolla at GMail.Com
>
>
> On Wed, Apr 29, 2020 at 3:35 PM Zeolla at GMail.com <zeolla at gmail.com> wrote:
>
>> Were you able to get this working?  I'm planning to work on the bro to
>> zeek cutover for the plugin soon.
>>
>> - Jon Zeolla
>> Zeolla at GMail.Com
>>
>>
>> On Mon, Apr 27, 2020 at 6:39 AM Zeolla at GMail.com <zeolla at gmail.com>
>> wrote:
>>
>>> I have not run it on 3.1.2 yet but I recommend making your changes to
>>> the plugin and running the end to end testing script at
>>> https://github.com/apache/metron-bro-plugin-kafka/blob/master/docker/run_end_to_end.sh
>>>
>>> It was meant to help isolate issues when making changes to the plugin.
>>> Also, we welcome PRs against the project so please feel free to
>>> contribute.  Thanks,
>>>
>>> Jon Zeolla
>>>
>>> On Sun, Apr 26, 2020, 9:12 PM Hovsep Levi <hovsep.sanjay.levi at gmail.com>
>>> wrote:
>>>
>>>> Hello Zeeks
>>>>
>>>>
>>>> Has anyone succeeded to enable Kafka plugin with Zeek 3.1.2 ?  I am
>>>> trying to modernize the metron-kafka plugin and have partial success.  My
>>>> problem seems to be with script-land referencing.
>>>>
>>>> The logger node is loading the plugin OK and connects to the Kafka
>>>> broker.  The broker IP is redef information found from site/local.zeek.
>>>>
>>>> $ bin/zeekctl diag logger-1
>>>> [logger-1]
>>>>
>>>> No core file found.
>>>>
>>>> Zeek 3.1.2-debug
>>>>
>>>> Zeek plugins:
>>>> Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)
>>>>
>>>> ==== No reporter.log
>>>>
>>>> ==== stderr.log
>>>> %7|1587948661.341|RECV|rdkafka#producer-3| [..kafka messages..] ..
>>>>
>>>>
>>>>
>>>>
>>>> But the worker node has a problem referencing existing variable
>>>> declaration.  The logs-to-kafka.bro script expects it.  There is also
>>>> suspicion with the Zeek plugins info that is different from the logger node
>>>> and maybe the problem.
>>>>
>>>> $ bin/zeekctl diag worker-1-1
>>>> [worker-1-1]
>>>>
>>>> No core file found.
>>>>
>>>> Zeek 3.1.2-debug
>>>>
>>>> Zeek plugins: (none found)   <<< ??? Normal for worker node ???
>>>>
>>>> ==== No reporter.log
>>>>
>>>> ==== stderr.log
>>>>
>>>> error in
>>>> /opt/zeek/spool/installed-scripts-do-not-touch/site/custom_plugins/APACHE_KAFKA/scripts/Apache/Kafka/./logs-to-kafka.bro,
>>>> line 24: unknown identifier logs_to_send, at or near "logs_to_send"
>>>>
>>>>
>>>>
>>>> The configuration is not default and explained below:
>>>>
>>>>
>>>> The Kafka logger was installed to site/custom_plugins/APACHE_KAFKA
>>>>
>>>>
>>>> share/zeek/site/local.zeek uses:
>>>>
>>>> @load custom_plugins/APACHE_KAFKA/scripts/Apache/Kafka
>>>>
>>>>
>>>>
>>>> lib/zeek/plugins/custom_plugins is a symlink to
>>>> share/zeek/site/custom_plugins
>>>>
>>>>
>>>> Using the lib symlink seems to be the only way to load the plugin, then
>>>> the @load statement brings redef customizations and scripts.  This works ok
>>>> for the logger node but not the worker who cannot interface with the plugin
>>>> ?
>>>>
>>>> Another idea is have non-logger nodes bypass loading logs-to-kafka.bro
>>>> but this isn't fully understood.
>>>>
>>>>
>>>> TIA
>>>>
>>>> /hovsep
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Zeek mailing list
>>>> zeek at zeek.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200507/5454da5a/attachment-0001.html