[Bro-Dev] Test timing measurements (Re: [Bro-Commits] [git/btest] topic/robin/timing: Adding a timing mode that records test execution times per host. (808fd76))

[Gilbert Clark]

On 1/31/14 2:24 PM, Robin Sommer wrote:
> (Moving from bro-commits to bro-dev).
>
> On Fri, Jan 31, 2014 at 12:51 -0500, you wrote:
>
>> Instruction counts are probably going to have a strong dependency on
>> the compiler version / options used to generate the code. I believe
>> these counts could additionally be influenced by e.g. library
>> upgrades, even when restricted to a single host and using a specific
>> compiler / options.
> True, but I'm not sure that's necessarily a bad thing.
I don't think it's a bad thing either, exactly.  When I wrote this, I 
was thinking that it'd make it a little easier to interpret the results 
if code changes could be tested independently of system changes ... but 
you're right that the system *is* going to have an effect on how bro 
runs and should therefore be included in the benchmark.

I do think it's important to be able to isolate system-level vs. 
code-level changes, but just storing the commit should be enough to do 
that.  I think this is already happening, so ... looks good to me.
> What I'm mostly wondering about is if it's worth commiting data that's 
> very specific to a single user/machine to the repos? 
 From an academic standpoint, think it could be useful to review these 
counts just to see how much they actually vary from platform to 
platform.  Also, if the instruction counts *are* pretty consistent 
cross-platform, then it might e.g. be an indication that we / the 
compiler aren't taking advantage of some more complex instructions that 
exist on a particular platform.

Additionally, given the number of people working on the project, this 
kind of data probably wouldn't take much space.  It also seems like it'd 
be pretty easy to purge this data in the event it didn't end up being 
useful.

--Gilbert