[Bro-Dev] Test timing measurements (Re: [Bro-Commits] [git/btest] topic/robin/timing: Adding a timing mode that records test execution times per host. (808fd76))
gc355804 at ohio.edu
Tue Feb 4 10:28:50 PST 2014
On 1/31/14 2:24 PM, Robin Sommer wrote:
> (Moving from bro-commits to bro-dev).
> On Fri, Jan 31, 2014 at 12:51 -0500, you wrote:
>> Instruction counts are probably going to have a strong dependency on
>> the compiler version / options used to generate the code. I believe
>> these counts could additionally be influenced by e.g. library
>> upgrades, even when restricted to a single host and using a specific
>> compiler / options.
> True, but I'm not sure that's necessarily a bad thing.
I don't think it's a bad thing either, exactly. When I wrote this, I
was thinking that it'd make it a little easier to interpret the results
if code changes could be tested independently of system changes ... but
you're right that the system *is* going to have an effect on how bro
runs and should therefore be included in the benchmark.
I do think it's important to be able to isolate system-level vs.
code-level changes, but just storing the commit should be enough to do
that. I think this is already happening, so ... looks good to me.
> What I'm mostly wondering about is if it's worth commiting data that's
> very specific to a single user/machine to the repos?
From an academic standpoint, think it could be useful to review these
counts just to see how much they actually vary from platform to
platform. Also, if the instruction counts *are* pretty consistent
cross-platform, then it might e.g. be an indication that we / the
compiler aren't taking advantage of some more complex instructions that
exist on a particular platform.
Additionally, given the number of people working on the project, this
kind of data probably wouldn't take much space. It also seems like it'd
be pretty easy to purge this data in the event it didn't end up being
More information about the bro-dev