From jheffner at psc.edu  Mon Oct  1 08:53:59 2007
From: jheffner at psc.edu (John Heffner)
Date: Mon, 01 Oct 2007 11:53:59 -0400
Subject: [Tmrg] Round table: Buffer sizes
In-Reply-To: <aa7d2c6d0709291751y23d681d7xf03ceb8ab863203b@mail.gmail.com>
References: <aa7d2c6d0709251307wf1971a6s26a5f71e82f367bd@mail.gmail.com>	<46F98369.8030605@psc.edu>
	<aa7d2c6d0709291751y23d681d7xf03ceb8ab863203b@mail.gmail.com>
Message-ID: <47011817.6050504@psc.edu>

Lachlan Andrew wrote:
> Greetings John,
> 
> On 25/09/2007, John Heffner <jheffner at psc.edu> wrote:
>> Lachlan Andrew wrote:
>>> suitable buffer sizes for this core set of tests?
>>
>> Given the relatively wide range of buffer sizes in hardware out there at
>> any given speed, and the significant effect buffer size can have on
>> congestion control, it seems like this should be an extra dimension
>> rather than fixed per link type.
> 
> Absolutely.  As Sally suggests, it would be good to have some tests
> specifically along this dimension.  That leaves open what value(s?) to
> use for this parameter for tests along other dimensions.
> 
>> Maybe select a couple different sizes based on a set of drain times?
>> Say, {1 ms, 10 ms, 100 ms, 1 sec}?  Convert this to bytes or packets by
>> dividing by the link byte or MTU packet rate.
> 
> Sounds good.  As I pointed out, converting delay to packets isn't as
> simple as dividing by MTU packet rate, because we have reverse ACKs
> which aren't full sized.  If we have an equal number of forward and
> backward flows every second ACK is delayed, we need a buffer of
> roughly  (3/2)(drain_time/MTU_rate)  packets.  Without delayed ACKs,
> it becomes  rougly  2(drain_time/MTU_rate).

What I was talking about was for a byte buffer, use 
(drain_time/byte_rate); for a packet buffer, (drain_time/packet_rate).

Some devices use packet buffers, and others use byte buffers.  I'm not 
sure there's a significant majority either way.  As you say, their 
behavior is definitely different when not all packets are the same size 
(mixed-MTU or bi-directional traffic).  I'm not sure if this effect is 
strong enough that you need to do both, but it couldn't hurt.  I'll 
admit I haven't been following this group closely enough to know exactly 
what the objective is for the "core" set of tests.


> In my limited experience, buffer sizes are typically powers of 2.  I
> propose that all "core" tests use buffers with a limit (in packets)
> equal to a power of 2.  Do others agree?

Hm, that's not really my experience.  Linux, for instance, uses a 
default of 100 or 1000 packets with most ethernet drivers.  Especially 
when talking about packets rather than bytes, a power of two seems kind 
of arbitrary.  I think a lot of older Fast/Gig switches used 64k 
buffers, or 42-43 packets at 1500 bytes.  OTOH, I don't think there's 
anything bad about using a power of two if it's convenient.

   -John

From sallyfloyd at mac.com  Mon Oct  1 13:30:52 2007
From: sallyfloyd at mac.com (Sally Floyd)
Date: Mon, 1 Oct 2007 13:30:52 -0700
Subject: [Tmrg] Round table: Buffer sizes
In-Reply-To: <aa7d2c6d0709291751y23d681d7xf03ceb8ab863203b@mail.gmail.com>
References: <aa7d2c6d0709251307wf1971a6s26a5f71e82f367bd@mail.gmail.com>
	<46F98369.8030605@psc.edu>
	<aa7d2c6d0709291751y23d681d7xf03ceb8ab863203b@mail.gmail.com>
Message-ID: <65b35fd1d6c8959209fe942f1c5ddece@mac.com>

> In my limited experience, buffer sizes are typically powers of 2.  I
> propose that all "core" tests use buffers with a limit (in packets)
> equal to a power of 2.  Do others agree?

It makes sense to me to limit buffer sizes to powers of two.

I also think that the purpose of a "core" set of tests is to
explore how congestion control mechanisms perform under a
range of conditions (including boundary conditions of various
kinds).  That is, in my view, the point of a core set of tests is
not "whoever gets the highest score on these tests wins", but
in contrast, a set of tests that hopefully will shed some light
on the strong and weak points (or the tradeoffs in design)
for the congestion control mechanism under test.  With
some added guide of "we believe that these tests represent
fairly realistic scenarios", and "these other tests represent fairly
unrealistic scenarios, but are included to test boundary
conditions of various kinds, or to test conjectured conditions
of the future."

With that view in mine, I assume that a core set of tests will
include tests with buffers in packets and with buffers in bytes,
and with both Drop-Tail and AQM.

And I assume that a core set of tests will include (but not
necessarily be limited to) a realistic mix of packet sizes for
data packets on the congested link.  My memory is that for
some links that have been measured, a realistic mix means
90% of data packets with 1500 bytes, with a mix for the
remaining data packets of 500 bytes, 4000 bytes, 200 bytes,
and the like.

I also assume that a core set of tests is a set that an average
researcher can run on their computer over the weekend
(or in only a few days).  So of course, many of the tests will
have to be short samples of the possible space.

- Sally
http://www.icir.org/floyd/


From jhealy at swin.edu.au  Tue Oct  2 00:22:10 2007
From: jhealy at swin.edu.au (James Healy)
Date: Tue, 02 Oct 2007 17:22:10 +1000
Subject: [Tmrg] Logging active TCP details in FreeBSD 5, 6 and 7
In-Reply-To: <aa7d2c6d0709070822s544dc68cgca0a262986395f8@mail.gmail.com>
References: <46E0F2BE.8090405@swin.edu.au>
	<aa7d2c6d0709070822s544dc68cgca0a262986395f8@mail.gmail.com>
Message-ID: <4701F1A2.50807@swin.edu.au>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Lachlan,

Lachlan Andrew wrote:
> Would it be possible to give it an interface like Web100?  We're
> currently building our benchmarking suite to use (a slightly
> optimized) Web100 to monitor system internals, and it would be great
> to be able to use SIFTR without too much modification of our suite.

Unfortunately, we don't currently have the resources to expand SIFTR to
include a web100 compatible interface. We can certainly see the merit in
the idea though, and would be happy to liase with anybody that would
like to look into it.

regards,

James Healy
Research Assistant
http://caia.swin.edu.au
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (FreeBSD)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHAfGi4oawkrbYo/kRAphxAJ4usqbK3vdLNU593Z03CPD41hqKxgCgtrLo
I2tJ3waiVZixie/shJKsxFk=
=fD0N
-----END PGP SIGNATURE-----

Swinburne University of Technology
CRICOS Provider Code: 00111D

NOTICE
This e-mail and any attachments are confidential and intended only for the use of the addressee. They may contain information that is privileged or protected by copyright. If you are not the intended recipient, any dissemination, distribution, printing, copying or use is strictly prohibited. The University does not warrant that this e-mail and any attachments are secure and there is also a risk that it may be corrupted in transmission. It is your responsibility to check any attachments for viruses or defects before opening them. If you have received this transmission in error, please contact us on +61 3 9214 8000 and delete it immediately from your system. We do not accept liability in connection with computer virus, data corruption, delay, interruption, unauthorised access or unauthorised amendment.

Please consider the environment before printing this email.

From jheffner at psc.edu  Tue Oct  2 11:18:51 2007
From: jheffner at psc.edu (John Heffner)
Date: Tue, 02 Oct 2007 14:18:51 -0400
Subject: [Tmrg] Logging active TCP details in FreeBSD 5, 6 and 7
In-Reply-To: <4701F1A2.50807@swin.edu.au>
References: <46E0F2BE.8090405@swin.edu.au>	<aa7d2c6d0709070822s544dc68cgca0a262986395f8@mail.gmail.com>
	<4701F1A2.50807@swin.edu.au>
Message-ID: <47028B8B.4080907@psc.edu>

James Healy wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi Lachlan,
> 
> Lachlan Andrew wrote:
>> Would it be possible to give it an interface like Web100?  We're
>> currently building our benchmarking suite to use (a slightly
>> optimized) Web100 to monitor system internals, and it would be great
>> to be able to use SIFTR without too much modification of our suite.
> 
> Unfortunately, we don't currently have the resources to expand SIFTR to
> include a web100 compatible interface. We can certainly see the merit in
> the idea though, and would be happy to liase with anybody that would
> like to look into it.

SIFTR and Web100 (the TCP ESTATS MIB) are very different ways of 
instrumenting TCP.  I'm not sure it would even make sense to try to use 
the same interface.

   -John

From lachlan.andrew at gmail.com  Tue Oct  2 14:02:05 2007
From: lachlan.andrew at gmail.com (Lachlan Andrew)
Date: Tue, 2 Oct 2007 14:02:05 -0700
Subject: [Tmrg] Round table: Buffer sizes
In-Reply-To: <65b35fd1d6c8959209fe942f1c5ddece@mac.com>
References: <aa7d2c6d0709251307wf1971a6s26a5f71e82f367bd@mail.gmail.com>
	<46F98369.8030605@psc.edu>
	<aa7d2c6d0709291751y23d681d7xf03ceb8ab863203b@mail.gmail.com>
	<65b35fd1d6c8959209fe942f1c5ddece@mac.com>
Message-ID: <aa7d2c6d0710021402s19af222fg76fd940ea3f60488@mail.gmail.com>

Greetings Sally,

Thanks for clearly stating your vision of the core tests.  My
tentative goals are below.

On 01/10/2007, Sally Floyd <sallyfloyd at mac.com> wrote:
>
> I also think that the purpose of a "core" set of tests is to
> explore how congestion control mechanisms perform under a
> range of conditions (including boundary conditions of various
> kinds).  That is, in my view, the point of a core set of tests is
> not "whoever gets the highest score on these tests wins", but
> in contrast, a set of tests that hopefully will shed some light
> on the strong and weak points (or the tradeoffs in design)
> for the congestion control mechanism under test.

Agreed.

> With
> some added guide of "we believe that these tests represent
> fairly realistic scenarios", and "these other tests represent fairly
> unrealistic scenarios, but are included to test boundary
> conditions of various kinds, or to test conjectured conditions
> of the future."
>
> With that view in mind, I assume that a core set of tests will
> include tests with buffers in packets and with buffers in bytes,
> and with both Drop-Tail and AQM.

It would be good to have such tests, but that is more
ambitious/comprehensive than I had in mind for the initial November
round table.

My aim was more "white-box" than "black-box": trying to find some
simple   tests which can be performed on a range of physical and
simulated testbeds.  If (big if!) physical routers predominantly have
buffers in packets, then I'd prefer to   start   with a subset of the
core tests which only use buffers in packets.

Motivated by past debates over different labs' tests, I was also more
interested in repeatability than realism.  If we get different results
using simulation from dummynet or different results using dummynet
from real WAN testbeds, it would be ideal if the results are "clean"
enough to find out what   causes   the difference.  Than means many of
the tests may lack important attributes like "web" cross traffic --
although of course there must also be enough tests with cross traffic
to see how the algorithm will perform in practice.

Once one or two tests have been defined precisely enough to be
repeatable by different labs, it would of course be good to extend to
a "wider core", like the one you describe.

> And I assume that a core set of tests will include (but not
> necessarily be limited to) a realistic mix of packet sizes for
> data packets on the congested link.  My memory is that for
> some links that have been measured, a realistic mix means
> 90% of data packets with 1500 bytes, with a mix for the
> remaining data packets of 500 bytes, 4000 bytes, 200 bytes,
> and the like.

Yes, some tests like that would be good.  However, since the MTU is a
property of an interface, it increases the number of senders required
to run the experiments.  Also, any experiments with over 1500-byte
packets can't be run on GbE hardware.

Since the TCP algorithms themselves determine the percentages of
traffic, we should specify the traffic in terms of the number of
flows   with each MTU, rather than the amount of traffic.  How about
specifying 90% of flows use 1500-byte, and 10% of flows use 536-byte?

Limiting to only two distinct MTUs also reduces the number of
computers needed.  It can be done with only 8 NICs (2 senders, 2
receivers and 4 on a Dummynet).

> I also assume that a core set of tests is a set that an average
> researcher can run on their computer over the weekend
> (or in only a few days).  So of course, many of the tests will
> have to be short samples of the possible space.

Yes, short samples will be important.  Again, at this stage I'd like
to design the tests for hardware implementation, and then check that
they're also OK for simulation, rather than the other way around.  We
can look into exploiting the extra flexibility of simulation in "Round
Table II".

Of course, the actual focus will depend on what the attendees are
interested in.  Cesar has put together a tentative agenda at
<http://wil.cs.caltech.edu/mwiki/index.php?title=Round_table_agenda>.
Feedback is welcome.

Cheers,
Lachlan

-- 
Lachlan Andrew  Dept of Computer Science, Caltech
1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA
Phone: +1 (626) 395-8820    Fax: +1 (626) 568-3603

From lachlan.andrew at gmail.com  Tue Oct  2 14:27:30 2007
From: lachlan.andrew at gmail.com (Lachlan Andrew)
Date: Tue, 2 Oct 2007 14:27:30 -0700
Subject: [Tmrg] Round table: Buffer sizes
In-Reply-To: <aa7d2c6d0710021402s19af222fg76fd940ea3f60488@mail.gmail.com>
References: <aa7d2c6d0709251307wf1971a6s26a5f71e82f367bd@mail.gmail.com>
	<46F98369.8030605@psc.edu>
	<aa7d2c6d0709291751y23d681d7xf03ceb8ab863203b@mail.gmail.com>
	<65b35fd1d6c8959209fe942f1c5ddece@mac.com>
	<aa7d2c6d0710021402s19af222fg76fd940ea3f60488@mail.gmail.com>
Message-ID: <aa7d2c6d0710021427k370c3deesaf937037ca861cc4@mail.gmail.com>

Greetings again,

On 02/10/2007, Lachlan Andrew <lachlan.andrew at gmail.com> wrote:
> On 01/10/2007, Sally Floyd <sallyfloyd at mac.com> wrote:
> > some links that have been measured, a realistic mix means
> > 90% of data packets with 1500 bytes, with a mix for the
> > remaining data packets of 500 bytes, 4000 bytes, 200 bytes,
> > and the like.
>
> Since the TCP algorithms themselves determine the percentages of
> traffic, we should specify the traffic in terms of the number of
> flows   with each MTU, rather than the amount of traffic.  How about
> specifying 90% of flows use 1500-byte, and 10% of flows use 536-byte?

On second thoughts,  TSO and iperf's blocking themselves produce a
significant number of packets below the MTU.  If we have 100% of flows
with an MTU of 1500 and use TSO, we may automatically get 90% 1500
byte packets, and 10% smaller ones.

If we rely on this artifact, we should control for it (specify iperf
parameters?), and specify how to get comparable results in
simulations.

Having one flow with 10% of its packets small is very different from
having 10% of flows with all of their packets small.  Sally, I assume
the study you referred to was pre-TSO, but many of the smaller packets
could still have been runts from connections with larger MTUs.

Thoughts?

Cheers,
Lachlan

-- 
Lachlan Andrew  Dept of Computer Science, Caltech
1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA
Phone: +1 (626) 395-8820    Fax: +1 (626) 568-3603

From jheffner at psc.edu  Wed Oct  3 10:54:12 2007
From: jheffner at psc.edu (John Heffner)
Date: Wed, 03 Oct 2007 13:54:12 -0400
Subject: [Tmrg] Round table: Buffer sizes
In-Reply-To: <aa7d2c6d0710021427k370c3deesaf937037ca861cc4@mail.gmail.com>
References: <aa7d2c6d0709251307wf1971a6s26a5f71e82f367bd@mail.gmail.com>	<46F98369.8030605@psc.edu>	<aa7d2c6d0709291751y23d681d7xf03ceb8ab863203b@mail.gmail.com>	<65b35fd1d6c8959209fe942f1c5ddece@mac.com>	<aa7d2c6d0710021402s19af222fg76fd940ea3f60488@mail.gmail.com>
	<aa7d2c6d0710021427k370c3deesaf937037ca861cc4@mail.gmail.com>
Message-ID: <4703D744.1010505@psc.edu>

Lachlan Andrew wrote:
> Greetings again,
> 
> On 02/10/2007, Lachlan Andrew <lachlan.andrew at gmail.com> wrote:
>> On 01/10/2007, Sally Floyd <sallyfloyd at mac.com> wrote:
>>> some links that have been measured, a realistic mix means
>>> 90% of data packets with 1500 bytes, with a mix for the
>>> remaining data packets of 500 bytes, 4000 bytes, 200 bytes,
>>> and the like.
>> Since the TCP algorithms themselves determine the percentages of
>> traffic, we should specify the traffic in terms of the number of
>> flows   with each MTU, rather than the amount of traffic.  How about
>> specifying 90% of flows use 1500-byte, and 10% of flows use 536-byte?
> 
> On second thoughts,  TSO and iperf's blocking themselves produce a
> significant number of packets below the MTU.  If we have 100% of flows
> with an MTU of 1500 and use TSO, we may automatically get 90% 1500
> byte packets, and 10% smaller ones.
> 
> If we rely on this artifact, we should control for it (specify iperf
> parameters?), and specify how to get comparable results in
> simulations.
> 
> Having one flow with 10% of its packets small is very different from
> having 10% of flows with all of their packets small.  Sally, I assume
> the study you referred to was pre-TSO, but many of the smaller packets
> could still have been runts from connections with larger MTUs.
> 
> Thoughts?

TSO traffic will often give you only MSS-sized segments until the window 
gets large enough that you start sending full 64k packets down to the 
driver.  (If I remember correctly how it works now -- there have been so 
many changes..)

One thing to try might be using setsockopt(TCP_MAXSEG) on some flows. 
That's probably easier than changing interface MTUs.

   -John

From lachlan.andrew at gmail.com  Wed Oct  3 11:04:30 2007
From: lachlan.andrew at gmail.com (Lachlan Andrew)
Date: Wed, 3 Oct 2007 11:04:30 -0700
Subject: [Tmrg] Round table: Buffer sizes
In-Reply-To: <4703D744.1010505@psc.edu>
References: <aa7d2c6d0709251307wf1971a6s26a5f71e82f367bd@mail.gmail.com>
	<46F98369.8030605@psc.edu>
	<aa7d2c6d0709291751y23d681d7xf03ceb8ab863203b@mail.gmail.com>
	<65b35fd1d6c8959209fe942f1c5ddece@mac.com>
	<aa7d2c6d0710021402s19af222fg76fd940ea3f60488@mail.gmail.com>
	<aa7d2c6d0710021427k370c3deesaf937037ca861cc4@mail.gmail.com>
	<4703D744.1010505@psc.edu>
Message-ID: <aa7d2c6d0710031104j6f99c65am9455f1f38e3ba112@mail.gmail.com>

On 03/10/2007, John Heffner <jheffner at psc.edu> wrote:
>
> TSO traffic will often give you only MSS-sized segments until the window
> gets large enough that you start sending full 64k packets down to the
> driver.  (If I remember correctly how it works now -- there have been so
> many changes..)

OK.  I remember seeing fragments coming out, but as you say, that
might have been a transient state of the code.

> One thing to try might be using setsockopt(TCP_MAXSEG) on some flows.
> That's probably easier than changing interface MTUs.

Thanks!  That will certainly make things more flexible.

Cheers,
Lachlan

-- 
Lachlan Andrew  Dept of Computer Science, Caltech
1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA
Phone: +1 (626) 395-8820    Fax: +1 (626) 568-3603

From lstewart at room52.net  Tue Oct  9 18:51:53 2007
From: lstewart at room52.net (Lawrence Stewart)
Date: Wed, 10 Oct 2007 11:51:53 +1000
Subject: [Tmrg] Software for FreeBSD TCP R&D: SIFTR v1.1.4 and DPD v1.0
	released
Message-ID: <470C3039.9030302@room52.net>

Hi All,

Further to Grenville's recent email regarding SIFTR, we just wanted to 
give you a quick heads up regarding the availability of a new SIFTR
(Statistical Information for TCP Research) version and the debut release
of DPD (Deterministic Packet Discard).

SIFTR v1.1.4 addresses a couple of issues, one of which is applicable to
users of SIFTR in FreeBSD 7-CURRENT. Read the changelog and readme for
more information.

DPD is a new FreeBSD kernel module we developed to further aid us in our
ongoing TCP research. It allows for the deterministic dropping of TCP
packets from within the FreeBSD kernel via a simple sysctl interface.
This is particularly useful for anyone that is interested in observing
TCP reacting to packet loss events (e.g. congestion control
researchers). Being able to drop the same packet(s) across multiple
tests allows for simpler comparisons of TCP behaviour. We've found it
particularly useful in evaluating and observing the behaviour of
different congestion control mechanisms, and hope it may be of use to
others out there. Please refer to the DPD readme for more in-depth
information.

The software and documentation is freely available under a BSD licence from:

http://caia.swin.edu.au/urp/newtcp/tools.html

We would be very happy to hear from anyone regarding bugs and
suggestions as well.

Cheers,
Lawrence

http://caia.swin.edu.au



From lachlan.andrew at gmail.com  Sat Oct 27 13:00:01 2007
From: lachlan.andrew at gmail.com (Lachlan Andrew)
Date: Sat, 27 Oct 2007 13:00:01 -0700
Subject: [Tmrg] convergence time
Message-ID: <aa7d2c6d0710271300y66ede9aei7a559b22f31c6502@mail.gmail.com>

Greetings all,

With the TCP evaluation round table just under two weeks away, let's
keep the discussions moving.  It would be great for people to start
threads on whatever issues they think we should agree on. Even if
you're not coming in person or through VRVS, feel free to start a
thread.


How should we measure the responsiveness of a TCP algorithm?  One way
is to measure "convergence time" as the time after a step change in
traffic until rate is within x% of its final value.

If we agree on that, we need to decide:

- What should x be?  I think it is not very critical, as long as we
agree on it.  If it is too low (like within 10%), it becomes too
sensitive to how we measure rates.  If it is too high (like within
50%), it doesn't capture whole convergence process.  Is 30% OK?

- How do we determine the "final value" of rate?  If everything is
symmetric, we could just take is at the "fair" (equal) rates, but I
think we should define it independently for fairness, for those cases
in which flows never reach equal rates.  For experiments with just one
step change followed by a long period of "steady state" (possibly with
cross traffic coming and going), we can just average over a period "a
long time" after the event.  How long should that be?  It could be
something like "when the rate of change has dropped to 5% of the
original rate of change" or some such.

- What timescale should we average the "current" rate over?  Rates
vary due to AIMD and cross traffic, as well as the convergence
process.  For loss-based protocols, including hybrid loss+delay, we
could base it on the rate (or window) just before or just after a loss
event.  For non-loss-based protocols, we could simply average over one
RTT.  Thoughts?

- The convergence time depends on setting, such as the number of
competing flows, and the RTT.  Should we specify a few settings
specifically for determining "the convergence speed of the algorithm",
or should we just say how to measure convergence time for each
experiment?

- The rise time of a single flow to an empty systems is not very
interesting, because it many measures the impact of slow start.  Is it
interesting to consider the response of one existing flow to one new
flow?  An alternative is to consider time to settle when a flow
*departs*, although that mainly measure the aggressiveness of the
protocol, rather than its responsiveness.

- We need to make these repeatable.  That is particularly hard with
cross traffic.  Should we specify a minimum number of runs to average
over?  If so, there is a tradeoff between accuracy and time to
complete the tests.  Would averaging over 5 tests be enough?

Cheers,
Lachlan

-- 
Lachlan Andrew  Dept of Computer Science, Caltech
1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA
Phone: +1 (626) 395-8820    Fax: +1 (626) 568-3603

From weixl at caltech.edu  Sun Oct 28 23:41:24 2007
From: weixl at caltech.edu (Xiaoliang (David) Wei)
Date: Sun, 28 Oct 2007 23:41:24 -0700 (PDT)
Subject: [Tmrg] convergence time
In-Reply-To: <aa7d2c6d0710271300y66ede9aei7a559b22f31c6502@mail.gmail.com>
References: <aa7d2c6d0710271300y66ede9aei7a559b22f31c6502@mail.gmail.com>
Message-ID: <Pine.GSO.4.58.0710282328230.23481@inky>

Hi Lachlan,

    Thanks for the good summary. It seems to me that part of the problems
occur because the concept of convergence really depends on degree of
stability of the protocols and timescale of the measurement. These are
inherent if we measure "current rate".

    Another option, to eliminate the dependency to stability and
timescale, is that we don't study the convergence of current rate.
Instead, we study the convergence of the aggregate average rate. That is,
if the instanenous rate of a flow at time t is x(t), we define the
aggregate average rate of the flow at time t to be
X(t) = 1/t * sum u=0->t x(u).
("sum" can be "integrate" if the time is continous).

Then we study the convergence of the curve X(t) to the "final value".
This process might be easier as:
1. X(t) is easier to measure because we can just look at the amount we
have transfered from time 0 to time t;
2. X(t) converges even x(t) has a limit-cycle oscillation, so it is less
sensitive to stability
3. If x(t) converges fast, X(t) converges fast too. We can still compare
the convergence with X(t)
4. X(t) does have meaning in user-experience. It measures how long the
users have to participate in the network to get to the desired rate.


-David


On Sat, 27 Oct 2007, Lachlan Andrew wrote:

> Greetings all,
>
> With the TCP evaluation round table just under two weeks away, let's
> keep the discussions moving.  It would be great for people to start
> threads on whatever issues they think we should agree on. Even if
> you're not coming in person or through VRVS, feel free to start a
> thread.
>
>
> How should we measure the responsiveness of a TCP algorithm?  One way
> is to measure "convergence time" as the time after a step change in
> traffic until rate is within x% of its final value.
>
> If we agree on that, we need to decide:
>
> - What should x be?  I think it is not very critical, as long as we
> agree on it.  If it is too low (like within 10%), it becomes too
> sensitive to how we measure rates.  If it is too high (like within
> 50%), it doesn't capture whole convergence process.  Is 30% OK?
>
> - How do we determine the "final value" of rate?  If everything is
> symmetric, we could just take is at the "fair" (equal) rates, but I
> think we should define it independently for fairness, for those cases
> in which flows never reach equal rates.  For experiments with just one
> step change followed by a long period of "steady state" (possibly with
> cross traffic coming and going), we can just average over a period "a
> long time" after the event.  How long should that be?  It could be
> something like "when the rate of change has dropped to 5% of the
> original rate of change" or some such.
>
> - What timescale should we average the "current" rate over?  Rates
> vary due to AIMD and cross traffic, as well as the convergence
> process.  For loss-based protocols, including hybrid loss+delay, we
> could base it on the rate (or window) just before or just after a loss
> event.  For non-loss-based protocols, we could simply average over one
> RTT.  Thoughts?
>
> - The convergence time depends on setting, such as the number of
> competing flows, and the RTT.  Should we specify a few settings
> specifically for determining "the convergence speed of the algorithm",
> or should we just say how to measure convergence time for each
> experiment?
>
> - The rise time of a single flow to an empty systems is not very
> interesting, because it many measures the impact of slow start.  Is it
> interesting to consider the response of one existing flow to one new
> flow?  An alternative is to consider time to settle when a flow
> *departs*, although that mainly measure the aggressiveness of the
> protocol, rather than its responsiveness.
>
> - We need to make these repeatable.  That is particularly hard with
> cross traffic.  Should we specify a minimum number of runs to average
> over?  If so, there is a tradeoff between accuracy and time to
> complete the tests.  Would averaging over 5 tests be enough?
>
> Cheers,
> Lachlan
>
> --
> Lachlan Andrew  Dept of Computer Science, Caltech
> 1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA
> Phone: +1 (626) 395-8820    Fax: +1 (626) 568-3603
> _______________________________________________
> Tmrg-interest mailing list
> Tmrg-interest at ICSI.Berkeley.EDU
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest
>

From andrea.baiocchi at uniroma1.it  Tue Oct 30 02:39:30 2007
From: andrea.baiocchi at uniroma1.it (Andrea Baiocchi)
Date: Tue, 30 Oct 2007 10:39:30 +0100
Subject: [Tmrg] convergence time
In-Reply-To: <aa7d2c6d0710271300y66ede9aei7a559b22f31c6502@mail.gmail.com>
References: <aa7d2c6d0710271300y66ede9aei7a559b22f31c6502@mail.gmail.com>
Message-ID: <a06230905c34ca8ab29b4@[192.168.205.5]>

Dear Lachlan, dear all,
   I am following this mailing list with great interest, especially 
about raound tabel and TCP measurements/evaluation issues. Many 
thanks to Lachlan for his most appreciable initiative. I try to 
contribute.


At 13:00 -0700 27-10-2007, Lachlan Andrew wrote:
>How should we measure the responsiveness of a TCP algorithm?

can it be sensible to look at responsiveness from a point of view 
closer to the user, by measuring the time required to deliver a given 
amount of data Bklg as a function of the value of Bklg? This is a 
curve not a single value, but useful indication could be extracted 
from there (i.e. asymptotic growth rate).

>- The rise time of a single flow to an empty systems is not very
>interesting, because it many measures the impact of slow start.  Is it
>interesting to consider the response of one existing flow to one new
>flow?  An alternative is to consider time to settle when a flow
>*departs*, although that mainly measure the aggressiveness of the
>protocol, rather than its responsiveness.

My proposal above suffers from slow start dependence. It could be 
"generalized" (albeit also complicated) by considering the amount of 
time required to deliver one INCREMENT of Bklg, say DeltaB, after an 
amount Bklg0 has already been delivered. As an example, given Bklg0, 
time required to deliver further data DeltaB=alpha*Bklg0, with 
alpha=0.1. Parameters to choose for this measurement are in general 
Bklg0 and alpha. This last measurement could also be normalized to 
the overall time required to deliver Bklg0 amount of data.

for both measurements it would clerayl be key to precisely define the 
scenario (corss traffic, link/topology changes, whatever stresses 
"responsiveness" of TCP).


Thank you for you attention.
Best regards,

Andrea Baiocchi

-- 
*********************************************************
Andrea Baiocchi, PhD
INFOCOM Dept. - University of Roma "La Sapienza"
Via Eudossiana 18 - 00184 Roma (Italy)
E-mail: andrea.baiocchi at uniroma1.it
Phone  +39 0644585654     
Fax:     +39 064744481
**********************************************************

From lachlan.andrew at gmail.com  Wed Oct 31 10:18:13 2007
From: lachlan.andrew at gmail.com (Lachlan Andrew)
Date: Wed, 31 Oct 2007 09:18:13 -0800
Subject: [Tmrg] convergence time
In-Reply-To: <Pine.GSO.4.58.0710282328230.23481@inky>
References: <aa7d2c6d0710271300y66ede9aei7a559b22f31c6502@mail.gmail.com>
	<Pine.GSO.4.58.0710282328230.23481@inky>
Message-ID: <aa7d2c6d0710311018m3751dd7etc24d6bc2fb75da47@mail.gmail.com>

Greetings David,

On 28/10/2007, Xiaoliang (David) Wei <weixl at caltech.edu> wrote:
>     Another option, to eliminate the dependency to stability and
> timescale, is that we don't study the convergence of current rate.
> Instead, we study the convergence of the aggregate average rate. That is,
> if the instanenous rate of a flow at time t is x(t), we define the
> aggregate average rate of the flow at time t to be
> X(t) = 1/t * sum u=0->t x(u).
> ("sum" can be "integrate" if the time is continous).

Good idea.

> Then we study the convergence of the curve X(t) to the "final value".
> This process might be easier as:
> 1. X(t) is easier to measure because we can just look at the amount we
> have transfered from time 0 to time t;
> 2. X(t) converges even x(t) has a limit-cycle oscillation, so it is less
> sensitive to stability
> 3. If x(t) converges fast, X(t) converges fast too. We can still compare
> the convergence with X(t)
> 4. X(t) does have meaning in user-experience. It measures how long the
> users have to participate in the network to get to the desired rate.

They're all good points.

The main drawback is that  X(t)  converges (much) more slowly, since
it always gives some weight to the early rates.  If we want to observe
the impact of each of several newly arriving flows, we need to space
them out further if we use  X(t)  than we do if we use  x(t), or else
the transients will interact.

The time required to find the "final" value could already be quite
long, especially in the case of Reno, which takes hours to reach
equilibrium on large BDP paths.

What do others think?

Cheers,
Lachlan

-- 
Lachlan Andrew  Dept of Computer Science, Caltech
1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA
Ph: +1 (626) 395-8820    Fax: +1 (626) 568-3603
http://netlab.caltech.edu/~lachlan

From lachlan.andrew at gmail.com  Wed Oct 31 10:31:24 2007
From: lachlan.andrew at gmail.com (Lachlan Andrew)
Date: Wed, 31 Oct 2007 09:31:24 -0800
Subject: [Tmrg] convergence time
In-Reply-To: <a06230905c34ca8ab29b4@192.168.205.5>
References: <aa7d2c6d0710271300y66ede9aei7a559b22f31c6502@mail.gmail.com>
	<a06230905c34ca8ab29b4@192.168.205.5>
Message-ID: <aa7d2c6d0710311031na932b90u491409fcd26dfa40@mail.gmail.com>

Greetings Andrea,

On 30/10/2007, Andrea Baiocchi <andrea.baiocchi at uniroma1.it> wrote:
>
> At 13:00 -0700 27-10-2007, Lachlan Andrew wrote:
> >How should we measure the responsiveness of a TCP algorithm?
>
> can it be sensible to look at responsiveness from a point of view
> closer to the user, by measuring the time required to deliver a given
> amount of data Bklg as a function of the value of Bklg? This is a
> curve not a single value, but useful indication could be extracted
> from there (i.e. asymptotic growth rate).

That is a very useful metric for the "efficiency" or "aggressiveness"
of the algorithm.  It is essentially the "flow completion time" metric
that Nandita and Nick have been promoting.

When I spoke of "responsiveness", I meant "response to changes in
network conditions", which is something not captured by the metric you
mention.  Whatever we call them, we need to measure both effects.

> My proposal above suffers from slow start dependence. It could be
> "generalized" (albeit also complicated) by considering the amount of
> time required to deliver one INCREMENT of Bklg, say DeltaB, after an
> amount Bklg0 has already been delivered. As an example, given Bklg0,
> time required to deliver further data DeltaB=alpha*Bklg0, with
> alpha=0.1. Parameters to choose for this measurement are in general
> Bklg0 and alpha. This last measurement could also be normalized to
> the overall time required to deliver Bklg0 amount of data.

That metric is equivalent to the average rate over some interval.  Can
you think of a way to average out the effects of AIMD in that
measurement?  It would either need  alpha  to be quite large, or to be
specially tuned to the time between loss events.  Otherwise, for given
parameters  alpha  and  Bklg0,  an algorithm could alternate between
doing well and doing poorly as the RTT increases, as the interval
moves from (just before a loss) to (just after a loss).

> Thank you for you attention.

Thank you for your input!

Cheers,
Lachlan

-- 
Lachlan Andrew  Dept of Computer Science, Caltech
1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA
Ph: +1 (626) 395-8820    Fax: +1 (626) 568-3603
http://netlab.caltech.edu/~lachlan

From lachlan.andrew at gmail.com  Wed Oct 31 12:16:45 2007
From: lachlan.andrew at gmail.com (Lachlan Andrew)
Date: Wed, 31 Oct 2007 11:16:45 -0800
Subject: [Tmrg] [Iccrg] Re:  convergence time
In-Reply-To: <238973.33571.qm@web51808.mail.re2.yahoo.com>
References: <238973.33571.qm@web51808.mail.re2.yahoo.com>
Message-ID: <aa7d2c6d0710311216y4e04a357o57e5a2008f0dcb61@mail.gmail.com>

Greetings Dirceu,

On 31/10/2007, Dirceu Cavendish <dirceu_cavendish at yahoo.com> wrote:
> I find transient interaction effects VERY interesting to study...

True.  But they are also complex.  My aim with this roundtable was to
agree on some simple "single numbers" to make comparison between
different people's experiments easier.  If we measure convergence time
as the time for  X(t)  to reach within 20% of its final value, but in
the experiment,  X(t)  never reaches its final value, then we are left
with no numeric measure of the convergence time.

What information will  X(t)  tell us about the interactions that isn't
more apparent from  x(t)?  The benefits David mentioned apply mainly
to the case without interactions.

> The bottom line is: agreeing on X(t) performance metrics instead of x(t)
> does not LIMIT in any way the expressiveness of experimental results, since
> we can always reduce X(t) to x(t) by using a single flow...

I agree that we can get  x(t)  from  X(t)  by differentiating it
(regardless of how many flows).
In terms of "expressiveness", they both carry exactly the same amount
of information, if we want to compare the whole functions, rather than
thresholding thiem.  My question is how we can make that comparison.
Does anyone have any suggestions?

To me, we need to agree on some sort of data reduction, and
thresholding is a simple example.  Thresholding  X(t)  seems more
problematic.  Of course, that problem may go away if there is a better
alternative than thresholding.

Thanks,
Lachlan

-- 
Lachlan Andrew  Dept of Computer Science, Caltech
1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA
Ph: +1 (626) 395-8820    Fax: +1 (626) 568-3603
http://netlab.caltech.edu/~lachlan