[Bro-Dev] Performance Issues after the fe7e1ee commit?

[Jon Siwek]

On Fri, Jun 8, 2018 at 10:23 AM Robin Sommer <robin at icir.org> wrote:

>
> > # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro
> > Known::use_host_store=F Known::use_service_store=F
> > Known::use_cert_store=F
>
> That indeed gets it way down, though still not back to the same level
> on my box:
>
> 170.49user 58.14system 1:55.94elapsed 197%CPU
>
> (pre-master: 73.72user 7.90system 1:06.53elapsed 122%CPU)

Just double-checking: same configure/build flags were used between the two?

Here's the more precise results I got from running on a macOS and Linux system:

# Linux master (b51e6f3) --build-type=debug
$ export BROPATH=$BROPATH:../testing/external/scripts; time bro -r
2009-M57-day11-18.trace test-all-policy testing-setup
real    0m32.572s
user    0m40.926s
sys     0m7.873s

# Linux master (b51e6f3) --build-type=debug
$ export BROPATH=$BROPATH:../testing/external/scripts; time bro -r
2009-M57-day11-18.trace test-all-policy testing-setup
Known::use_host_store=F Known::use_cert_store=F
Known::use_service_store=F
real    0m25.603s
user    0m23.311s
sys     0m7.952s

# Linux pre-broker (7a6f502) --build-type=debug
$ export BROPATH=$BROPATH:../testing/external/scripts; time bro -r
2009-M57-day11-18.trace test-all-policy testing-setup
real    0m24.785s
user    0m21.230s
sys     0m8.341s

# macOS master (b51e6f3) --build-type=debug
$ export BROPATH=$BROPATH:../testing/external/scripts; time bro -r
2009-M57-day11-18.trace test-all-policy testing-setup.bro
real    0m38.775s
user    0m42.365s
sys     0m8.950s

# macOS master (b51e6f3) --build-type=debug
$ export BROPATH=$BROPATH:../testing/external/scripts; time bro -r
2009-M57-day11-18.trace test-all-policy testing-setup.bro
Known::use_host_store=F Known::use_cert_store=F
Known::use_service_store=F
real    0m32.975s
user    0m33.774s
sys     0m4.409s

# macOS pre-broker (7a6f502) --build-type=debug
$ export BROPATH=$BROPATH:../testing/external/scripts; time bro -r
2009-M57-day11-18.trace test-all-policy testing-setup.bro
real    0m30.785s
user    0m31.840s
sys     0m3.788s

> Are there more places where Bro's waiting for Broker in pcap mode?

Not that I can think of.

> Yeah, I remember that discussion. It's the trade-off between
> performance and consistency. Where's the code that's doing this
> blocking?

I just know that Manager::AdvanceTime() and also
Manager::TrackStoreQuery() -> FlushPendingQueries() can block/spin
waiting on actor/threading in pcap mode.

> Would it be possible to not block right away, but sync up
> with Broker when events are flushed the next time? (Like we had
> discussed before as a general mechanism for making async operations
> consistent)

I think the way to make an async operation most consistent is to model
it as a synchronous operation?  That's what I'm doing now at least,
and if I try moving FlushPendingQueries() to later (before the
AdvanceTime() call) in an attempt to possibly have more queries in the
queue/buffer, I get the external test suites failing.  At least on my
own test systems, I don't have much performance to recover by going
this route anyway, so maybe we need to dig more into why your results
are different.  Only thing I'm thinking is that your test system maybe
does a poorer job of scheduling the right thread to run in order for
the FlushPendingQueries() spin-loop to actually finish.

- Jon