[Tmrg] Round table: level of realism of tests?

[Lachlan Andrew]

Greetings Sally,

On 05/11/2007, Sally Floyd <sallyfloyd at mac.com> wrote:
> My apologies for responding a month late on this!
> My unanswered email folder has been building up on me.

No worries.  I know you're really busy.

> > Motivated by past debates over different labs' tests, I was also more
> > interested in repeatability than realism.  If we get different results
> > using simulation from dummynet or different results using dummynet
> > from real WAN testbeds, it would be ideal if the results are "clean"
> > enough to find out what   causes   the difference.  Than means many of
> > the tests may lack important attributes like "web" cross traffic --
> > although of course there must also be enough tests with cross traffic
> > to see how the algorithm will perform in practice.
>
> Hmmm.

*grin*

> My own view would be that any clearly unrealistic test scenarios
> should be explicitly labeled as such.

Good idea.  We could classify tests as those "trying to understand"
behaviour vs those "evaluating" behaviour.  The first group could
usefully have tests which would be misleading if they were in the
second group.

> One of my other views (as
> expressed in the 2002 Hotnets paper on "Internet Research Needs
> Better Models") is that a reliance on unrealistic scenarios in
> evaluating transport protocols (e.g., scenarios with only one-way
> traffic, only long-lived flows, or only flows all with the same
> RTT) could do a serious dis-service in the design and evaluation
> of transport protocols.

True, that is a danger we must try to avoid.  It has to be balanced
against the need to hold some variables fixed while others are varied.
 For example, if we're looking at the impact of the number of hops on
fairness, there is a case to keep RTTs equal to eliminate the effect
of RTT-unfairness for that experiment.

We can also do things to maximize repeatability without reducing
realism, like agreeing on some deterministic (pseudo-random?) cross
traffic rather than every experiment having unique cross traffic.

I've always thought of this round-table as only a start, which is why
I felt comfortable suggesting having too many "understanding"
experiments, at the expense of "evaluating" experiments.  But the
table is round, so my suggestion won't determine what happens.

Cheers,
Lachlan

-- 
Lachlan Andrew  Dept of Computer Science, Caltech
1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA
Ph: +1 (626) 395-8820    Fax: +1 (626) 568-3603
http://netlab.caltech.edu/~lachlan