[Tmrg] Towards a Common TCP Evaluation Suite

[Lawrence D. Dunn]

Lachlan,
   Here are my "raw notes" taken during your talk...

Larry
--
<start notes>

  >  4:01p Lachlan A.
      >  Towards a Common TCP Evauation Suite
         Lachlan Andrew1, Cesar Marcondes2, Sally Floyd3, Lawrence
         Dunn4, Romaric Guillier5, Wang Gang6, Lars Eggert7,Sangtae Ha8,
         Injong Rhee8,
         1 Caltech, 2 UCLA, 3 ICSI, 4 Cisco Systems, 5 INRIA, 6 NEC
         China, 7 Nokia, 8 NCSU
      >  He mentioned this was in part, in response to Dunn's "Injong and Doug
         agree on results" goal...
      >  and reduce arguments, need to repeat experiments which may/may
         not match someone else's
      >  want them to be as realistic as possible; not a "benchmark", as
         sally points out people will tune to meet the benchmark
      >  balance reductionism (doug's 1-variable at a time) with realism
      >  design space
          >  start w/ simulator, or testbed
          >  choose topology (delay, bitrate)
          >  flows under test
          >  cross traffic
          >  metrics to track
          >  comments
              >  richard- convergence is the wrong name; please be
                 precise in terminology- like suppressing slowstart is
                 not the same as "a new flow starts..."
              >  is 80% for transients misleading? maybe a spread of
                 points, to help characterize shape a bit?
              >  richard- on impact on reno test- endstations are
                 uncontrollable, and he's afraid things will sync or end
                 up in lockstep due to endstation effects; lachlan- we
                 watch experimental loading; light host links/cpu,
                 tighter bottleneck
              >


<end notes>
At 10:21 AM -0700 3/10/08, Lachlan Andrew wrote:
>Greetings all,
>
>The TCP test suite presentation at PFLDnet received a lot of useful
>feedback.  I took some notes afterwards, but missed many of the questions.
>Could those who raised issues during the talk please repeat them to the list?
>(The slides are at <...>.)
>
>The following are the comments I remember, and my responses.
>
>- For the RTT distribution, why limit it to the nine discrete values,
>   when a modified dummynet can give per-flow delays?
>
>   I said that it was to make it more platform-independent.  However,
>   the point remains:  Why 6 nodes, not 4 or 8?  We'll have to revisit
>   this when we look at whether these link delays give the right RTT
>   distribution when long-RTT flows complete.
>
>- What is "real"?  How do we know that the "realistic" cases we study
>   are better than simpler ones?
>
>   We should probably quote measurement studies to back things up.
>   I would add that we must justify that the benefits of capturing any
>   given real effect is worth the cost, for example in terms of making
>   results potentially misleading due to statistical effects, or
>   difficult to interpret for other reasons.
>
>- It is not "realistic" to assume that the newly arriving flow misses
>   slow start.
>
>   I still think it is a useful case to consider, but we should justify it.
>   If we say it is the "worst case", then we'd have to justify not applying
>   the same logic to losses due to cross traffic.  The worst case is if
>   there are none during the transient, which I think would happen more
>   often than a flow exiting slow start on the first RTT.
>
>   If we have time, it would be interesting to see the sensitivity of
>   convergence time to cross traffic; instead of one test with 10%,
>   we could have one with 0% and one with 30%.
>
>- The "convergence time" section needs a new name
>
>- When we have multiple flows being generated by a single host, we need
>   to avoid them getting into lock-step because of host issues.
>
>   I think that other processing issues will keep us to fairly low rates
>   (<= 622M?)  in testbeds, at least Linux ones, and so this shouldn't
>   be a problem.
>
>- For the 3-hop topology in which each flow gets 60ms delay, we currently
>   specify having seven delay elements.  It is possible to achieve this
>   with only two delay elements (one of the "access" links, and one of
>   the bottleneck links).
>
>   Are these equivalent?  They both result in the same RTTs; most TCP
>   throughput models only care what the RTTs, not which links it occurs on.
>   We can't do the same with shifting buffers around, but I think we can
>   shift propagation delays around freely, can't we?
>
>   If they're not equivalent, could we standardize on the one with
>   two delays instead of seven?  It would mean we only need 4 dummynets
>   instead of 7.
>
>Also, Lars suggested that all authors of the PFLDnet paper become
>acknowledgements, and that we build up a new set of authors based on
>who keeps the suite moving, independent of who was at the round table.
>
>Cheers,
>Lachlan
>
>
>
>--
>Lachlan Andrew  Dept of Computer Science, Caltech
>1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA
>Ph: +1 (626) 395-8820    Fax: +1 (626) 568-3603
>http://netlab.caltech.edu/lachlan