[Bro-Dev] Bro 2.6-beta plans

[Jon Siwek]

On Wed, Sep 12, 2018 at 9:18 AM Azoff, Justin S <jazoff at illinois.edu> wrote:
>
> Just finished the migration to master across the board, and it's looking REALLY good.

Great, thanks for helping test and provide performance data.

> The manager box in this cluster only runs the manager and logger processes, no proxies.  It also has something like 20 idle cores,
> so this isn't a problem at all, but could affect people who run a cluster-in-a-box.

An idea in this type of situation could be to tune Broker::max_threads
per node type.  E.g. leave at 1 for workers and bump to ~4 for
manager/logger since there's idle cores on their host and they're
inherently in a less-scalable/centralized location.  That may not
lower overall cpu usage, but may help prevent some bottlenecks in the
processing of remote messages.  Particularly the work of processing
data store communication should distribute among threads, potentially
each data store could be processing messages independently on separate
threads.  (The default scripts have 3 stores, one for each known-*
script).

> I do seem to be seeing a bunch of reporter errors like
>
> Reporter::ERROR string with embedded NUL: "\\x00\\x00\\x00\\x00OPTIONS"

Ok, not necessarily that bad, but would be nice to find where that's
coming from to handle it more properly

- Jon