[Bro] NSQ plugin getting deprecated in 2.5

Vlad Grigorescu vladg at illinois.edu
Mon Sep 19 08:59:53 PDT 2016


I can try to summarize the current status of the plugin, to give this
discussion some additional context:

 * A large stream of log output to NSQ is working. I was pushing about 1
   billion log lines/day for months with no issues.

 * ElasticSearch output stopped working with ElasticSearch version 2.0,
   since they changed the delimiter rules. However, this should be
   fixable with the change Seth introduced for 2.5 (and perhaps we
   should update that to be the default?)

 * A medium to large stream of log output to ElasticSearch requires a
   lot of tuning and I think is still problematic. I think memory slowly
   creeps up in most cases (ElasticSearch starts garbage-collecting, and
   stops responding for a while). I haven't done work with ElasticSearch
   2.0 to see how that affects this. Perhaps splitting out the logger
   node will help with this? I'm not sure.

 * I think that the main issue that Seth was referencing is that the log
   writer doesn't check the response code from NSQ or ElasticSearch. If
   the server responds with a 500 or other error code, it might make
   sense to retry sending the messages a couple of times? Right now,
   they just get dropped, so this can be a lossy log writer.

So, I'm a bit hesitant to deprecate this, since I think it still works
for NSQ, and it still works (in some cases) for ElasticSearch.
Ironically, it works better for ElasticSearch in 2.5 than it would for
2.4.1, since the delimiter configuration option was introduced.

That being said, I'm also hesitant to take this on myself, simply
because we don't have an ElasticSearch cluster at NCSA.

I think it makes sense to generalize this as an HTTP/JSON log writer,
but we still need to tackle the question of what we do with messages
that fail to be delivered.

Generalizing it might be a bit tricky. For example, ElasticSearch needs
to post to http://1.2.3.4:9000/$log_name, while NSQ needs to add a
line containing the log_name before each log line.

  --Vlad

"Azoff, Justin S" <jazoff at illinois.edu> writes:

>> On Sep 16, 2016, at 5:50 PM, Robin Sommer <robin at icir.org> wrote:
>> 
>> 
>> I'm wondering if there's anybody who'd be interested in taking over
>> ownership of the plugin. We are planing to move bro-plugins/* into
>> separately distributed Bro packages anyways, using the new Bro package
>> manager. If somebody wanted to take ownership of the plugin that way,
>> they could just starting maintaining a package for it. An option could
>> also be turning it into a NSQ-only plugin?
>> 
>> Robin
>
> I was thinking the same thing.
>
> I think the issues with the elasticsearch plugin was that bro -> remote ES never worked well in practice and users were better off using something else to get logs into ES.
>
> But bro -> local NSQ was rock solid for the people that used it.
>
> Also, another thing to keep in mind is that there are only a few lines in the entire plugin that are actually specific to NSQ, with a few strings moved into options it could possibly be turned into a generic http/json log writer.
>
>
> -- 
> - Justin Azoff
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 800 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160919/d29c0491/attachment.bin 


More information about the Bro mailing list