[Bro] which kafka plugin to use?

[Zeolla@GMail.com]

For what it's worth, I'm currently using the plugin available under
https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka in
my production bro environment, which is an 8 node cluster with > 25,000
events per second and it's working just fine for me, but I would love to
get others to test it.  I'm not making any changes to the core kafka plugin
itself for the move, just packaging it and incrementing some version
numbers - the real heavy lift is within Metron itself, not the bro plugin.

Jon

On Tue, Aug 15, 2017 at 12:25 PM Zeolla at GMail.com <zeolla at gmail.com> wrote:

> To clarify, the Metron project developed the kafka plugin for its own uses
> and then contributed it into bro-plugins.  Recently I worked with the
> initial creator of the plugin to unify all of the updates that have
> happened to it over the years (in a way that complies with its LICENSE)
> here
> <https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka>
> .
>
> I'm in the process of porting it to be a bro package and moving it to
> https://github.com/apache/metron-bro-plugin-kafka which will be its final
> resting point.  I'm currently battling through some CentOS 6 -> 7 upgrades
> in Metron, and then upgrading bro to 2.5.1 (from 2.4) in Metron (and all of
> the associated automation/testing), and then finally I will be publishing
> the kafka plugin module and submitting a PR to
> https://github.com/bro/packages.  Some very, *very* early movement
> towards packaging the kafka plugin can be found here
> <https://github.com/JonZeolla/metron-bro-plugin-kafka> (caution, it
> almost definitely does not work - I'm trying to figure out how to handle
> the librdkafka dependancy in the package, any feedback would be helpful).
>
> I would /love/ to have this ready to go for brocon (which is my goal).
>
> Jon
>
> On Tue, Aug 15, 2017 at 12:00 PM Erich M Nahum <nahum at us.ibm.com> wrote:
>
>> > The original kafka plugin, hosted at https://github.com/bro/bro-plugins
>> > , is now gone.
>>
>> D'oh, I now see it is also available in aux/plugins/kafka
>>
>>
>>
>> > When trying to build from the git tree at https://github.com/g-clef/
>> > KafkaLogger,
>> > I get the following build error:
>> >
>> > [ 33%] Building CXX object CMakeFiles/Kafka-KafkaWriter.linux-
>> > x86_64.dir/src/AddingJson.cc.o
>> > /usr/src/KafkaLogger/src/AddingJson.cc:3:20: fatal error: config.h:
>> > No such file or directory
>> > compilation terminated.
>> > CMakeFiles/Kafka-KafkaWriter.linux-x86_64.dir/build.make:80: recipe
>> > for target 'CMakeFiles/Kafka-KafkaWriter.linux-x86_64.dir/src/
>> > AddingJson.cc.o' failed
>>
>> Perhaps this is useful to Aaron Gee-Clough.  I forgot to mention that
>> I'm using Ubuntu 16.04 running apt-get upgrade periodically.
>>
>>
>>
>> > I see there's now a Metro fork of the kafka plugin at
>> >
>> >
>> https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka
>> >
>> > but I am reluctant to try it based on email comments that it is beta.
>> >
>> > Any comments/suggestions?
>>
>> While I can use the version in the bro source, I guess my question still
>> stands:
>> what's the long-term outlook for kafka support?
>>
>>
>>
>> -Erich
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> --
>
> Jon
>
-- 

Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170815/b1dfbef5/attachment.html