[Tmrg] Round table: level of realism of tests?

[Sally Floyd]

Lachlan -

My apologies for responding a month late on this!
My unanswered email folder has been building up on me.

...
> Motivated by past debates over different labs' tests, I was also more
> interested in repeatability than realism.  If we get different results
> using simulation from dummynet or different results using dummynet
> from real WAN testbeds, it would be ideal if the results are "clean"
> enough to find out what   causes   the difference.  Than means many of
> the tests may lack important attributes like "web" cross traffic --
> although of course there must also be enough tests with cross traffic
> to see how the algorithm will perform in practice.

Hmmm.

My own view would be that any clearly unrealistic test scenarios
should be explicitly labeled as such.   One of my other views (as
expressed in the 2002 Hotnets paper on "Internet Research Needs
Better Models") is that a reliance on unrealistic scenarios in
evaluating transport protocols (e.g., scenarios with only one-way
traffic, only long-lived flows, or only flows all with the same
RTT) could do a serious dis-service in the design and evaluation
of transport protocols.

Thus, my own view would be that test scenarios that were clearly
repeatable but clearly unrealistic could do more harm than good,
if people were going to rely on that for evaluating transport
protocols.

But maybe others have a different view, who knows?

- Sally
http://www.icir.org/floyd/


The 2002 Hotnets paper:
http://www.icir.org/models/hotnetsFinal.pdf