[Zeek] Zeek 3.1.2 and Kafka - No data flow

[Zeolla@GMail.com]

Master officially supports Zeek 3.1 now so please let me know if you
run into any issues.

- Jon Zeolla
Zeolla at GMail.Com


On Thu, May 7, 2020 at 1:15 PM Hovsep Levi <hovsep.sanjay.levi at gmail.com>
wrote:

> It is not working yet for me and was set aside to fix another time.
>
> Very glad to hear about pull 44, I will test !
>
> /Hovsep
>
>
> On Mon, May 4, 2020 at 5:01 PM Zeolla at GMail.com <zeolla at gmail.com> wrote:
>
>> I have the plugin working with 3.1.2 here
>> <https://github.com/apache/metron-bro-plugin-kafka/pull/44> - feedback
>> is welcome.
>>
>> - Jon Zeolla
>> Zeolla at GMail.Com
>>
>>
>> On Wed, Apr 29, 2020 at 3:35 PM Zeolla at GMail.com <zeolla at gmail.com>
>> wrote:
>>
>>> Were you able to get this working?  I'm planning to work on the bro to
>>> zeek cutover for the plugin soon.
>>>
>>> - Jon Zeolla
>>> Zeolla at GMail.Com
>>>
>>>
>>> On Mon, Apr 27, 2020 at 6:39 AM Zeolla at GMail.com <zeolla at gmail.com>
>>> wrote:
>>>
>>>> I have not run it on 3.1.2 yet but I recommend making your changes to
>>>> the plugin and running the end to end testing script at
>>>> https://github.com/apache/metron-bro-plugin-kafka/blob/master/docker/run_end_to_end.sh
>>>>
>>>> It was meant to help isolate issues when making changes to the plugin.
>>>> Also, we welcome PRs against the project so please feel free to
>>>> contribute.  Thanks,
>>>>
>>>> Jon Zeolla
>>>>
>>>> On Sun, Apr 26, 2020, 9:12 PM Hovsep Levi <hovsep.sanjay.levi at gmail.com>
>>>> wrote:
>>>>
>>>>> Hello Zeeks
>>>>>
>>>>>
>>>>> Has anyone succeeded to enable Kafka plugin with Zeek 3.1.2 ?  I am
>>>>> trying to modernize the metron-kafka plugin and have partial success.  My
>>>>> problem seems to be with script-land referencing.
>>>>>
>>>>> The logger node is loading the plugin OK and connects to the Kafka
>>>>> broker.  The broker IP is redef information found from site/local.zeek.
>>>>>
>>>>> $ bin/zeekctl diag logger-1
>>>>> [logger-1]
>>>>>
>>>>> No core file found.
>>>>>
>>>>> Zeek 3.1.2-debug
>>>>>
>>>>> Zeek plugins:
>>>>> Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)
>>>>>
>>>>> ==== No reporter.log
>>>>>
>>>>> ==== stderr.log
>>>>> %7|1587948661.341|RECV|rdkafka#producer-3| [..kafka messages..] ..
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> But the worker node has a problem referencing existing variable
>>>>> declaration.  The logs-to-kafka.bro script expects it.  There is also
>>>>> suspicion with the Zeek plugins info that is different from the logger node
>>>>> and maybe the problem.
>>>>>
>>>>> $ bin/zeekctl diag worker-1-1
>>>>> [worker-1-1]
>>>>>
>>>>> No core file found.
>>>>>
>>>>> Zeek 3.1.2-debug
>>>>>
>>>>> Zeek plugins: (none found)   <<< ??? Normal for worker node ???
>>>>>
>>>>> ==== No reporter.log
>>>>>
>>>>> ==== stderr.log
>>>>>
>>>>> error in
>>>>> /opt/zeek/spool/installed-scripts-do-not-touch/site/custom_plugins/APACHE_KAFKA/scripts/Apache/Kafka/./logs-to-kafka.bro,
>>>>> line 24: unknown identifier logs_to_send, at or near "logs_to_send"
>>>>>
>>>>>
>>>>>
>>>>> The configuration is not default and explained below:
>>>>>
>>>>>
>>>>> The Kafka logger was installed to site/custom_plugins/APACHE_KAFKA
>>>>>
>>>>>
>>>>> share/zeek/site/local.zeek uses:
>>>>>
>>>>> @load custom_plugins/APACHE_KAFKA/scripts/Apache/Kafka
>>>>>
>>>>>
>>>>>
>>>>> lib/zeek/plugins/custom_plugins is a symlink to
>>>>> share/zeek/site/custom_plugins
>>>>>
>>>>>
>>>>> Using the lib symlink seems to be the only way to load the plugin,
>>>>> then the @load statement brings redef customizations and scripts.  This
>>>>> works ok for the logger node but not the worker who cannot interface with
>>>>> the plugin ?
>>>>>
>>>>> Another idea is have non-logger nodes bypass loading logs-to-kafka.bro
>>>>> but this isn't fully understood.
>>>>>
>>>>>
>>>>> TIA
>>>>>
>>>>> /hovsep
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Zeek mailing list
>>>>> zeek at zeek.org
>>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>>>
>>>> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200507/88d2c84e/attachment.html