[Bro] which kafka plugin to use?
zeolla at gmail.com
Tue Aug 15 09:52:29 PDT 2017
For what it's worth, I'm currently using the plugin available under
my production bro environment, which is an 8 node cluster with > 25,000
events per second and it's working just fine for me, but I would love to
get others to test it. I'm not making any changes to the core kafka plugin
itself for the move, just packaging it and incrementing some version
numbers - the real heavy lift is within Metron itself, not the bro plugin.
On Tue, Aug 15, 2017 at 12:25 PM Zeolla at GMail.com <zeolla at gmail.com> wrote:
> To clarify, the Metron project developed the kafka plugin for its own uses
> and then contributed it into bro-plugins. Recently I worked with the
> initial creator of the plugin to unify all of the updates that have
> happened to it over the years (in a way that complies with its LICENSE)
> I'm in the process of porting it to be a bro package and moving it to
> https://github.com/apache/metron-bro-plugin-kafka which will be its final
> resting point. I'm currently battling through some CentOS 6 -> 7 upgrades
> in Metron, and then upgrading bro to 2.5.1 (from 2.4) in Metron (and all of
> the associated automation/testing), and then finally I will be publishing
> the kafka plugin module and submitting a PR to
> https://github.com/bro/packages. Some very, *very* early movement
> towards packaging the kafka plugin can be found here
> <https://github.com/JonZeolla/metron-bro-plugin-kafka> (caution, it
> almost definitely does not work - I'm trying to figure out how to handle
> the librdkafka dependancy in the package, any feedback would be helpful).
> I would /love/ to have this ready to go for brocon (which is my goal).
> On Tue, Aug 15, 2017 at 12:00 PM Erich M Nahum <nahum at us.ibm.com> wrote:
>> > The original kafka plugin, hosted at https://github.com/bro/bro-plugins
>> > , is now gone.
>> D'oh, I now see it is also available in aux/plugins/kafka
>> > When trying to build from the git tree at https://github.com/g-clef/
>> > KafkaLogger,
>> > I get the following build error:
>> > [ 33%] Building CXX object CMakeFiles/Kafka-KafkaWriter.linux-
>> > x86_64.dir/src/AddingJson.cc.o
>> > /usr/src/KafkaLogger/src/AddingJson.cc:3:20: fatal error: config.h:
>> > No such file or directory
>> > compilation terminated.
>> > CMakeFiles/Kafka-KafkaWriter.linux-x86_64.dir/build.make:80: recipe
>> > for target 'CMakeFiles/Kafka-KafkaWriter.linux-x86_64.dir/src/
>> > AddingJson.cc.o' failed
>> Perhaps this is useful to Aaron Gee-Clough. I forgot to mention that
>> I'm using Ubuntu 16.04 running apt-get upgrade periodically.
>> > I see there's now a Metro fork of the kafka plugin at
>> > but I am reluctant to try it based on email comments that it is beta.
>> > Any comments/suggestions?
>> While I can use the version in the bro source, I guess my question still
>> what's the long-term outlook for kafka support?
>> Bro mailing list
>> bro at bro-ids.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Bro