[Tmrg] Round table: level of realism of tests?

[Sally Floyd]

>> One of my other views (as
>> expressed in the 2002 Hotnets paper on "Internet Research Needs
>> Better Models") is that a reliance on unrealistic scenarios in
>> evaluating transport protocols (e.g., scenarios with only one-way
>> traffic, only long-lived flows, or only flows all with the same
>> RTT) could do a serious dis-service in the design and evaluation
>> of transport protocols.
>
> True, that is a danger we must try to avoid.  It has to be balanced
> against the need to hold some variables fixed while others are varied.
>  For example, if we're looking at the impact of the number of hops on
> fairness, there is a case to keep RTTs equal to eliminate the effect
> of RTT-unfairness for that experiment.

My assumption would be that if we are looking at a basic test of
tradeoffs between bandwidth, delay, and packet drop rates,
and we were varying bandwidth, *all* of these basic tests would
have realistic scenarios, with a realistic range of RTTs for traffic
on the congested link, a realistic range of packet sizes (including
40-byte packets from TCP ACK packets from reverse-path traffic),
a realistic distribution of connection sizes, a realistic mix of
TCP and UDP traffic, and the like.  And that if we were looking
at the effect of different levels of reverse-path traffic, all of the
other variable would be held fixed at realistic values.

That is, I am assuming that we are not creating scenarios for
people to use to debug their own congestion control mechanisms.
(People seem to be able to do that for themselves.)
I am assuming that the *first* priority is to create scenarios to
*evaluate* congestion control mechanisms.  Our own, and other people's.
And I think that requires realistic scenarios, for the most part.

> We can also do things to maximize repeatability without reducing
> realism, like agreeing on some deterministic (pseudo-random?) cross
> traffic rather than every experiment having unique cross traffic.

That sounds fine to me.

It would certainly be a good thing if different simulators and different
testbeds all gave the same *general* results for the same scenario.
Certainly, within a single simulator, results should be repeatable.
(E.g., in ns-2 version x, with simulation script y, and seed z for the
pseudo-random number generator, all users should get the same
results.  And in a particular testbed, with a particular set of code,
and a particular set-up, repeated experiments should get the same
results.)

However, I assume that there is a limit to the repeatability across
different simulators or different testbeds, or between simulators
and testbeds.  Certainly it would be good for experiments to be
done in both simulators and testbeds, and for the overall quantitative
results to be the same.

As discussed in the 2001 paper on "Difficulties in Simulating
the Internet" ("http://www.icir.org/floyd/papers/simulate_2001.pdf"),
simulators and testbeds in some cases have different roles to play.
E.g., testbeds will probably be more useful for exploring interactions
with the various features of commercial routers, firewalls,
middleboxes, and the like.  And simulators will probably be more
useful to the individual research (particularly the one who does
not have a testbed at their disposal) to play with a wide range of
scenarios, to develop intuition, and to easily explore functionality
that is not yet implemented in testbeds or in the real world.  So
it would be fine with me, for example, if there were some scenarios
that would run on testbeds but not in simulators.  Or vice versa.

> I've always thought of this round-table as only a start, which is why
> I felt comfortable suggesting having too many "understanding"
> experiments, at the expense of "evaluating" experiments.  But the
> table is round, so my suggestion won't determine what happens.

Great.  I will be there pushing for both the "understanding" and
the "evaluating" experiments to have as many realistic scenarios
as possible.  To avoid encouraging researchers to develop an
"understanding" that doesn't have much to do with the real world...

- Sally
http://www.icir.org/floyd/