[Bro-Dev] Test timing measurements (Re: [Bro-Commits] [git/btest] topic/robin/timing: Adding a timing mode that records test execution times per host. (808fd76))
robin at icir.org
Fri Jan 31 11:24:28 PST 2014
(Moving from bro-commits to bro-dev).
On Fri, Jan 31, 2014 at 12:51 -0500, you wrote:
> Instruction counts are probably going to have a strong dependency on
> the compiler version / options used to generate the code. I believe
> these counts could additionally be influenced by e.g. library
> upgrades, even when restricted to a single host and using a specific
> compiler / options.
True, but I'm not sure that's necessarily a bad thing. If the count
changes signficantly, it's worth understanding where it's coming from
I would say. btest won't complain as long as deviations are within a
reasonable range (1% by default, don't know if that's the right
I'm also not sure if instruction count is the right feature; there are
plenty others one could measure, like cycles etc. I was just thinking
this might be the most stable.
> One alternative approach to tracking IDs for timing baselines might be
> to use system tools to gather a list of all libraries bro is linked
A problem with this is that btest doesn't know about Bro. :-) The way
I'm doing it currently is that instruction count is measured for all
BTEST-EXEC commands that are part of a test, which are then summed up
for a single number. I'd like to keep it the way that btest can
measure arbitrary command lines (which is part of the challenge of
finding a stable way of doing so ...).
> Additionally, formatting the temporary file in a human-readable way
> and keeping it as part of / in addition to the baseline can yield
> potentially useful information when looking into timing differences.
It's, more or less, human-readable:
> cat Baseline/_Timing/2a6b457d90e93b6688f312f87f677c5c
What I'm mostly wondering about is if it's worth commiting data that's
very specific to a single user/machine to the repos?
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin
More information about the bro-dev