[Bro-Dev] Performance Issues after the fe7e1ee commit?

Jon Siwek jsiwek at corelight.com
Thu Jun 7 15:05:33 PDT 2018

On Thu, Jun 7, 2018 at 12:21 PM Robin Sommer <robin at icir.org> wrote:
> Hmm, I'm still seeing much larger runtimes on that M57 trace using our
> test configuration, even with yesterday's change:
> ; Master, pre-Broker
> # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro
> 73.72user 7.90system 1:06.53elapsed 122%CPU (0avgtext+0avgdata 199092maxresident)
> ; Current master
> # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro
> 2191.59user 1606.69system 12:39.92elapsed 499%CPU (0avgtext+0avgdata 228400maxresident)
> Bro must still be blocking somewhere when reading from trace files.

Thanks, if you pull master changes [1] again there's likely some improvement.

You can also try comparing the timing of:

# zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro
Known::use_host_store=F Known::use_service_store=F

With that, I get the same timings as from before pre-Broker.  At least
a good chunk of the difference when using data stores is that, for
every query, Bro will immediately block until getting a response back
from the data store thread/actor.  Not doing this type of synchronous
data store access when reading pcaps leads to the possibility of
results differing between runs (and I recall such differences being
likely from experience in running the unit test suite).

- Jon

[1] https://github.com/bro/broker/commit/9b56fea4999d4e11a5cd2caaafd934759015fab5

More information about the bro-dev mailing list