[Bro] Question about Bro manager write data to kafka

[Zeolla@GMail.com]

I am working on a newer version of the Kafka writer plugin (as a part of
the Apache Metron project, which is where the plugin was initially created)
which has support for sending to kerberized Kafka, some bug fixes, better
debug logging, etc.  It currently exists here
<https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka>,
but I'm going to be turning it into a bro package and moving it here
<https://github.com/apache/metron-bro-plugin-kafka> eventually (once it has
more testing).  If you're willing to beta test a bit, perhaps it's worth
giving a shot, in addition to Justin's comments?

Jon

On Tue, Jun 13, 2017 at 9:27 AM Azoff, Justin S <jazoff at illinois.edu> wrote:

>
> > On Jun 13, 2017, at 9:09 AM, Alkene Pan <alkenepan at gmail.com> wrote:
> >
> > Hi Bro, i'm encountered a performance issue about Bro manager write data
> to kafka. Can anyone help me please?
> ...
> >
> > Bro Cluster Config details:
> > [manager]
> > type=manager
> > host=localhost
> >
> > [proxy-1]
> > type=proxy
> > host=localhost
> >
> > [worker-1]
> > type=worker
> > host=localhost
> > interface=eno1
> > aux_scripts= -C
> > lb_method=pf_ring
> > lb_procs=15
> > pin_cpus=3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
> >
> ...
>
> > The mechanism is correct? Or Bro Manager exist performance issue about
> write a huge data into Kafka? Or incorrect configuration? Please kindly let
> me know if you have any recommendation. Thank you so much.
>
> You're not running a logger process which will easily double the
> performance of your cluster.  Add
>
> [logger]
> type=logger
> host=localhost
>
> to your node.cfg
>
> If you install the bro 2.5.1 beta you can have two or more loggers defined:
>
>
> [logger-1]
> type=logger
> host=localhost
>
> [logger-2]
> type=logger
> host=localhost
>
> (This is specifically intended for things like kafka)
>
> --
> - Justin Azoff
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-- 

Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170613/8c3019fb/attachment-0001.html