[Bro-Dev] Test timing measurements (Re: [Bro-Commits] [git/btest] topic/robin/timing: Adding a timing mode that records test execution times per host. (808fd76))

[Robin Sommer]

(Moving from bro-commits to bro-dev).

On Fri, Jan 31, 2014 at 12:51 -0500, you wrote:

> Instruction counts are probably going to have a strong dependency on
> the compiler version / options used to generate the code. I believe
> these counts could additionally be influenced by e.g. library
> upgrades, even when restricted to a single host and using a specific
> compiler / options.

True, but I'm not sure that's necessarily a bad thing. If the count
changes signficantly, it's worth understanding where it's coming from
I would say. btest won't complain as long as deviations are within a
reasonable range (1% by default, don't know if that's the right
value).

I'm also not sure if instruction count is the right feature; there are
plenty others one could measure, like cycles etc. I was just thinking
this might be the most stable.

> One alternative approach to tracking IDs for timing baselines might be
> to use system tools to gather a list of all libraries bro is linked
> against.

A problem with this is that btest doesn't know about Bro. :-) The way
I'm doing it currently is that instruction count is measured for all
BTEST-EXEC commands that are part of a test, which are then summed up
for a single number. I'd like to keep it the way that btest can
measure arbitrary command lines (which is part of the challenge of
finding a stable way of doing so ...).

> Additionally, formatting the temporary file in a human-readable way
> and keeping it as part of / in addition to the baseline can yield
> potentially useful information when looking into timing differences.

It's, more or less, human-readable:

    > cat Baseline/_Timing/2a6b457d90e93b6688f312f87f677c5c
    tests.m57-long 705347795206
    tests.ipv6 104508274160
    tests.m57-short 68458131160

What I'm mostly wondering about is if it's worth commiting data that's
very specific to a single user/machine to the repos?

Robin


-- 
Robin Sommer * Phone +1 (510) 722-6541 *     robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 * www.icir.org/robin