[Bro-Dev] Performance Issues after the fe7e1ee commit?

McMullan, Tim Tim.McMullan at sig.com
Fri Jun 8 12:18:47 PDT 2018

Hi Jon,

Thanks for looking into this!  I tried this again after pulling from master and at least for our case it is looking good!

### old
# /hostname/bro/bin/bro --version
/hostname/bro/bin/bro version 2.5-519
# time /hostname/bro/bin/bro -r 20180606-1242-prodfilers.pcap master.bro

real    0m32.772s
user    0m28.821s
sys     0m3.873s

### new
# /hostname/bro-devel/bin/bro --version
/hostname/bro-devel/bin/bro version 2.5-658
# time /hostname/bro-devel/bin/bro -r 20180606-1242-prodfilers.pcap master.bro

real    0m34.684s
user    0m31.502s
sys     0m4.266s

Thanks again!

-----Original Message-----
From: Jon Siwek [mailto:jsiwek at corelight.com]
Sent: Thursday, June 07, 2018 6:06 PM
To: Robin Sommer <robin at icir.org>
Cc: McMullan, Tim <Tim.McMullan at msx.bala.susq.com>; <bro-dev at bro.org> <bro-dev at bro.org>; Wallior, Julien <Julien.Wallior at msx.bala.susq.com>; Trejo, Devin <Devin.Trejo at msx.bala.susq.com>
Subject: Re: [Bro-Dev] Performance Issues after the fe7e1ee commit?

On Thu, Jun 7, 2018 at 12:21 PM Robin Sommer <robin at icir.org> wrote:
> Hmm, I'm still seeing much larger runtimes on that M57 trace using our
> test configuration, even with yesterday's change:
> ; Master, pre-Broker
> # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro
> 73.72user 7.90system 1:06.53elapsed 122%CPU (0avgtext+0avgdata 199092maxresident)
> ; Current master
> # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro
> 2191.59user 1606.69system 12:39.92elapsed 499%CPU (0avgtext+0avgdata 228400maxresident)
> Bro must still be blocking somewhere when reading from trace files.

Thanks, if you pull master changes [1] again there's likely some improvement.

You can also try comparing the timing of:

# zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro
Known::use_host_store=F Known::use_service_store=F

With that, I get the same timings as from before pre-Broker.  At least
a good chunk of the difference when using data stores is that, for
every query, Bro will immediately block until getting a response back
from the data store thread/actor.  Not doing this type of synchronous
data store access when reading pcaps leads to the possibility of
results differing between runs (and I recall such differences being
likely from experience in running the unit test suite).

- Jon

[1] https://github.com/bro/broker/commit/9b56fea4999d4e11a5cd2caaafd934759015fab5


IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.

More information about the bro-dev mailing list