[Bro-Dev] Performance Issues after the fe7e1ee commit?

[Jon Siwek]

On Wed, Jun 6, 2018 at 2:10 PM Azoff, Justin S <jazoff at illinois.edu> wrote:

> I haven't noticed a huge difference on a real multi process cluster, at least not a 30x change, so this is odd that pcap processing is so much slower.
> Especially since broker should be completely disabled when pcap files are being read and caf shouldn't even be doing anything.

It's not that simple.

You could think of broker being enabled in this case of simply reading
a pcap because it was querying the library for whether there were any
peers (and that was actually a at least part of the perf. problem).

Beyond that, you can still think of broker being enabled when you're
reading a pcap and you are also using data stores (which the Known*
scripts in Bro now do by default).  Communication with a local master
data store is still asynchronous communication (with local
threads/actors) that ends up going through CAF messaging.  There's
also essentially a per-packet synchronization being done between Bro
and Broker/CAF now to ensure that this type of asynchronous workload
ends up producing the same results between any given Bro run.  So
there's now also that bit of extra overhead in pcap processing.

- Jon