From wanggang at research.nec.com.cn Tue Aug 7 19:21:40 2007 From: wanggang at research.nec.com.cn (Wang gang) Date: Wed, 8 Aug 2007 10:21:40 +0800 Subject: [Tmrg] A reminder about An NS2 TCP Evaluation Tool Suite Message-ID: <015901c7d962$dacd2b90$c44c1cac@ad.research.nec.com.cn> Dear colleagues, We have released the tool 'An NS2 TCP Evaluation Tool Suite' for some time. Since then, we have received some feed back from users. We expect to receive wider comments, and seek collaborations or contributions to make the tool towards a useful one. The download page is, http://labs.nec.com.cn/tcpeval.htm Here is a brief introduction, This tool is motivated by the observation that there is significant overlap among (but lack of an agreed set of) the topologies, traffic, and metrics used by many researchers in the evaluation of TCP alternatives: effort could be saved by starting research from an existing framework. As such, our tool includes several typical topologies and traffic models; it measures some of the most important metrics commonly used in TCP evaluation; and it can automatically generate simulation statistics and graphs ready for inclusion in latex and html documents. The tool also contains an extendable open-source framework. With community effort, we hope the tool evolves into a widely accepted, well-defined set of TCP performance evaluation benchmarks. Best Regards. Gang Wang. ---------------------------------------- Gang Wang NEC Labs, China 010-62705962/63 (ext.511) wanggang at research.nec.com.cn -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/tmrg-interest/attachments/20070808/ec1f89e5/attachment.html From lachlan.andrew at gmail.com Wed Aug 15 19:44:39 2007 From: lachlan.andrew at gmail.com (Lachlan Andrew) Date: Wed, 15 Aug 2007 19:44:39 -0700 Subject: [Tmrg] A reminder about An NS2 TCP Evaluation Tool Suite In-Reply-To: <015901c7d962$dacd2b90$c44c1cac@ad.research.nec.com.cn> References: <015901c7d962$dacd2b90$c44c1cac@ad.research.nec.com.cn> Message-ID: Greetings Gang Wang, Your tool looks nice. Here are some suggestions: 1. Each topology seems to specify a single bottleneck capacity and RTT. Is there a way to make it test for a a range of capacities, like 10, 100, 155 and 622 Mbps? 2. It specifies a diff_RTT variable so that flows can have different RTTs, but it seems that RTTs are equally spaced within the allowed range. This equal spacing may cause artefacts. More importantly, real RTTs aren't uniformly distributed. It would be good to have a more realistic distribution of RTTs. (If they're generated randomly, it will be important to make it repeatable still.) 3. Jain's measure of fairness does not reflect the user's experience. - A fairness measure should give more weight to flows receiving very low throughput. If 10 flows get equal throughput, and one flow gets nothing, that is very unfair, but scores highly in Jain's index. - This can partly be overcome by applying Jain's index to the *download times* instead of the rates. As an approximation of the download time, you could use the reciprocal of the rate. - Jain's measure also doesn't consider the impact of multiple bottlenecks. In a parking-lot topology with links of unequal capacity, the "fairest" solution IMO is for the flow which only uses the high-capacity link not to be restricted by the fact that there is another low-capacity link which it doesn't use. Jain's index only gives a high score if the high-capacity link is under-utilized. 4. The parking lot topology is very symmetric. It would be interesting to look at parking-lot topologies with different bandwidths on the different bottlenecks. Cheers, Lachlan On 07/08/07, Wang gang wrote: > > Dear colleagues, > > We have released the tool 'An NS2 TCP Evaluation Tool Suite' for some time. > Since > then, we have received some feed back from users. We expect to receive wider > comments, > and seek collaborations or contributions to make the tool towards a useful > one. > > The download page is, > http://labs.nec.com.cn/tcpeval.htm > > > Here is a brief introduction, > This tool is motivated by the observation that there is significant overlap > among (but lack > of an agreed set of) the topologies, traffic, and metrics used by many > researchers in the > evaluation of TCP alternatives: effort could be saved by starting research > from an existing > framework. As such, our tool includes several typical topologies and > traffic models; it measures > some of the most important metrics commonly used in TCP evaluation; and it > can automatically > generate simulation statistics and graphs ready for inclusion in latex and > html documents. The > tool also contains an extendable open-source framework. With community > effort, we hope the > tool evolves into a widely accepted, well-defined set of TCP performance > evaluation benchmarks. > > Best Regards. > > Gang Wang. > > ---------------------------------------- > Gang Wang > NEC Labs, China > 010-62705962/63 (ext.511) > > wanggang at research.nec.com.cn > > > > _______________________________________________ > Tmrg-interest mailing list > Tmrg-interest at ICSI.Berkeley.EDU > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest > > -- Lachlan Andrew Dept of Computer Science, Caltech 1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA Phone: +1 (626) 395-8820 Fax: +1 (626) 568-3603 From lachlan.andrew at gmail.com Mon Aug 20 07:52:35 2007 From: lachlan.andrew at gmail.com (Lachlan Andrew) Date: Mon, 20 Aug 2007 07:52:35 -0700 Subject: [Tmrg] A reminder about An NS2 TCP Evaluation Tool Suite In-Reply-To: <5.1.1.8.2.20070820232343.07eb4e30@mail.jp.nec.com> References: <015901c7d962$dacd2b90$c44c1cac@ad.research.nec.com.cn> <5.1.1.8.2.20070820232343.07eb4e30@mail.jp.nec.com> Message-ID: Greetings Hide, On 20/08/07, Hideyuki Shimonishi wrote: > > Nice to talk to you again. Yes, good to hear from you. I hope you don't mind, but I'm Cc'ing this to tmrg. > It may be useful to consider distribution of per-flow throughput, rather > than some statistical values. > Also, in multiple-bottleneck topology, we may have to consider > alpha-proportional fairness, i.e. resource fairness v.s. throughput fairness. Good point. I was also thinking that it would be good both to evaluate the total "utility" based on some sort of alpha-fairness, and also try to evaluate what "alpha" is the best approximation in the case of multiple links. > Some results are shown in my PFLDnet 2007 presentation. > Some results about throughput distribution are shown in pp17-18. > Some results about fairness are shown in left figure of page 21, which > shows AReno, compound-TCP, and Hamilton-TCP are rather throughput fair, and > others are rather resource fair. I do not think this figure is the best, we > may need to use another statistics to show this tradeoff. OK, I'll check out those figures. > >4. The parking lot topology is very symmetric. It would be > >interesting to look at parking-lot topologies with different > >bandwidths on the different bottlenecks. > > As you may know since Cesar has presented our tool at ICCRG, our NEC-UCLA > tool should be one other option to do simulations in complex topologies. Yes, I saw that presentation. One feature I really like about that tool is the way it compares very systematically against Reno using exactly the same traffic. Cheers, Lachlan -- Lachlan Andrew Dept of Computer Science, Caltech 1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA Phone: +1 (626) 395-8820 Fax: +1 (626) 568-3603 From h-shimonishi at cd.jp.nec.com Mon Aug 20 10:21:40 2007 From: h-shimonishi at cd.jp.nec.com (Hideyuki Shimonishi) Date: Tue, 21 Aug 2007 02:21:40 +0900 Subject: [Tmrg] A reminder about An NS2 TCP Evaluation Tool Suite In-Reply-To: References: <5.1.1.8.2.20070820232343.07eb4e30@mail.jp.nec.com> <015901c7d962$dacd2b90$c44c1cac@ad.research.nec.com.cn> <5.1.1.8.2.20070820232343.07eb4e30@mail.jp.nec.com> Message-ID: <5.1.1.8.2.20070821021410.04f3eb40@mail.jp.nec.com> Hi Lachlan, At 07/08/20 07:52 -0700, Lachlan Andrew wrote: >Greetings Hide, > >On 20/08/07, Hideyuki Shimonishi wrote: > > > > Nice to talk to you again. > >Yes, good to hear from you. I hope you don't mind, but I'm Cc'ing >this to tmrg. > > > It may be useful to consider distribution of per-flow throughput, rather > > than some statistical values. > > Also, in multiple-bottleneck topology, we may have to consider > > alpha-proportional fairness, i.e. resource fairness v.s. throughput fairness. > >Good point. I was also thinking that it would be good both to >evaluate the total "utility" based on some sort of alpha-fairness, >and also try to evaluate what "alpha" is the best approximation in the >case of multiple links. I have no idea what alpha is better or not, but I think it would be valuable to study what protocol is more resource-fair than Reno and what protocol is more throughput-fair than Reno. > > Some results are shown in my PFLDnet 2007 presentation. > > Some results about throughput distribution are shown in pp17-18. > > Some results about fairness are shown in left figure of page 21, which > > shows AReno, compound-TCP, and Hamilton-TCP are rather throughput fair, and > > others are rather resource fair. I do not think this figure is the best, we > > may need to use another statistics to show this tradeoff. > >OK, I'll check out those figures. Also, please check slide 14. It looks like that AReno and Hamilton have similar alpha with Reno, and others are more resource fair. > > >4. The parking lot topology is very symmetric. It would be > > >interesting to look at parking-lot topologies with different > > >bandwidths on the different bottlenecks. > > > > As you may know since Cesar has presented our tool at ICCRG, our NEC-UCLA > > tool should be one other option to do simulations in complex topologies. > >Yes, I saw that presentation. One feature I really like about that >tool is the way it compares very systematically against Reno using >exactly the same traffic. Thanks. Your comments two years ago about the tool was really helpful to me to develop the method ! Thanks, HIDE >Cheers, >Lachlan > >-- >Lachlan Andrew Dept of Computer Science, Caltech >1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA >Phone: +1 (626) 395-8820 Fax: +1 (626) 568-3603 From lachlan.andrew at gmail.com Mon Aug 20 21:19:35 2007 From: lachlan.andrew at gmail.com (Lachlan Andrew) Date: Mon, 20 Aug 2007 21:19:35 -0700 Subject: [Tmrg] TCP evaluation suite round-table In-Reply-To: References: Message-ID: Greetings all, On 8-9 November, a few of us will be getting together at Caltech for a "round table" to try to agree on some basic parameters and metrics for TCP evaluation. We won't try to answer things like "what fairness metric is best", but we can agree on some basic parameters. The situation we're trying to avoid is: Group A finds that at 500Mbps, flow 1 reaches 10% of its final throughput after 30s Group B finds that at 622Mbps, flow 1 reaches 20% of its final throughput after 20s I'll try to have live video-conferencing via VRVS so thta those who can't come in person can still participate. Unfortunately, our videoconferencing room is small, and so physical attendence will probably be limited to a dozen or so people. Please let me know if you're interested. As basic goals, I'd like to come away from the roundtable with: - a set of bandwidths that are of interest, say 10, 155, 622, 2500 Mbps - a set of buffer sizes that are of interest, like BDP or 16384 packets - a set of distributions of RTT that are of interest - an agreed notion of "convergence time" -- e.g., "the average over period x is within y% of the final average - an agreed notion of "time to converge to fairness" -- e.g., "the ratio of averages over period x is within y% of the final ratio" -- should this metric depend on the final ratio achieved? - an agreed notion of "intraflow variability" -- e.g., what timescales are of interest? - an agreed set of traffic models for background traffic Injong has added to that list: - a measure of total link utilization - fluctuation in utilization due to fluctuation in background traffic. - What is the per-flow fair bandwidth share? I think some of those will be "easy", and we can sort them out on the list before an interactive meeting, to save time for more debatable ones. It would be good if everyone can throw in some ideas for the next couple of months so that we can see issues are the hard ones. It would be good to have a common set of scenarios which could be tested by simulation, emulation and real networks. Obviously, the simulation is the most flexible, and so it may have a larger set of tests, but we can at least simulate the emulated cases. Ulterior motive: I'd like people also to simulate/emulate the scenarios that can also be tested on WAN-in-Lab :) If this works, we can get more ambitious in a second roundtable elsewhere. Cheers, Lachlan -- Lachlan Andrew Dept of Computer Science, Caltech 1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA Phone: +1 (626) 395-8820 Fax: +1 (626) 568-3603 From lachlan.andrew at gmail.com Mon Aug 20 21:36:41 2007 From: lachlan.andrew at gmail.com (Lachlan Andrew) Date: Mon, 20 Aug 2007 21:36:41 -0700 Subject: [Tmrg] TCP evaluation suite round-table In-Reply-To: References: Message-ID: Greetings again, On 20/08/07, Lachlan Andrew wrote: > As basic goals, I'd like to come away from the roundtable with: > - a set of bandwidths that are of interest, say 10, 155, 622, 2500 Mbps > > I think some of those will be "easy", and we can sort them out on the > list before an interactive meeting, to save time for more debatable > ones. To try my hypothesis, I think that bandwidths should be easy to agree on on-list. Obvious candidates are: 10 Mbit/s -- old Ethernet, the right ball-part for current ADSL/cable 54Mbit/s -- 802.11a/g 100Mbit/s -- Fast Ethernet 155Mbit/s -- OC3/STM-1 400Mbit/s -- used by Doug and Injong's Dummynet studies (IIRC) 622Mbit/s -- OC12/STM-4 1000Mbit/s -- GbE 2488Mbit/s -- OC48/STM-16 9952Mbit/s -- OC192/STM-64 10Gbit/s -- 10GbE Can we assume that Moore's law now allows Dummynet to run at 622Mbit/s? If so, I'd strike 400Mbit/s off in favour of OC12. With my WAN-in-Lab hat on, I'd vote for GbE, OC48 and 10GbE, since we have those. Can we agree on using 10Mbit/s 100Mbit/s 622Mbit/s 1000Mbit/s 2488Mbit/s 10Gbit/s in all simulations/experiments, unless there is a reason to deviate? Cheers, Lachlan -- Lachlan Andrew Dept of Computer Science, Caltech 1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA Phone: +1 (626) 395-8820 Fax: +1 (626) 568-3603 From lachlan.andrew at gmail.com Mon Aug 20 22:06:38 2007 From: lachlan.andrew at gmail.com (Lachlan Andrew) Date: Mon, 20 Aug 2007 22:06:38 -0700 Subject: [Tmrg] TCP evaluation suite round-table In-Reply-To: References: Message-ID: Greetings again, On 20/08/07, Lachlan Andrew wrote: > - a measure of total link utilization Again, I think this should be easy to agree on. Rather than simply measuring total bit/s over the link (which can be achieved by inducing congestion collapse), I would advocate using the sum of the *receive* rates of the flows using each link. For multi-link topologies, this would count the rate of multi-hop flows multiple times, and so is different from the "total network throughput" (sum of all flow rates). It is also more "useful" than measuring the total network throughput, since the latter is maximized by giving all capacity to any single-hop flows using a link. Thoughts? Lachlan -- Lachlan Andrew Dept of Computer Science, Caltech 1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA Phone: +1 (626) 395-8820 Fax: +1 (626) 568-3603 From lars.eggert at nokia.com Tue Aug 21 02:12:07 2007 From: lars.eggert at nokia.com (Lars Eggert) Date: Tue, 21 Aug 2007 12:12:07 +0300 Subject: [Tmrg] TCP evaluation suite round-table In-Reply-To: References: Message-ID: Hi, On 2007-8-21, at 7:36, ext Lachlan Andrew wrote: > Can we assume that Moore's law now allows Dummynet to run at > 622Mbit/s? If so, I'd strike 400Mbit/s off in favour of OC12. during my thesis, I found that dummynet doesn't simulate high- datarate paths very accurately anymore once the CPU becomes loaded: However, simulating wide-area Gigabit links with Dummynet is problematic [ZEC2003]. Dummynet uses the kernel firewall to identify packets for processing, and depends heavily on the kernel timers to control when packets leave the transmission buffer. Both mechanisms incur significant overheads at high data rates. Furthermore, high data rates cause high interrupt loads, which can decrease system responsiveness and eventually lead to livelock [MOGUL1997]. Because Dummynet processing occurs at the IP layer, device interrupts cause delays that reduce the accuracy of the simulation. These delays can also interfere with user-space processing, and as a result affect the benchmark processes themselves. [ZEC2003] Marko Zec and Miljenko Mikuc. Real-Time IP Network Simulation at Gigabit Data Rates. Proc. International Conference on Telecommunications (ConTEL), Zagreb, Croatia, June 11-13, 2003. As you said, Moore's law may have pushed up the region of bandwidth that can be accurately simulated, but it'd be good to have some verification. Lars -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2446 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/tmrg-interest/attachments/20070821/c6095385/attachment.bin From doug.leith at nuim.ie Wed Aug 22 00:26:57 2007 From: doug.leith at nuim.ie (Douglas Leith) Date: Wed, 22 Aug 2007 08:26:57 +0100 Subject: [Tmrg] Tmrg-interest Digest, Vol 11, Issue 4 In-Reply-To: References: Message-ID: Re dummynet, my experience these days is that it can run up to 1Gb using modern hardware. In my experience more of an issue is that end hosts can still have difficulty at high bandwidth-delay products due to sack processing overhead etc. and it is this that has placed the upper limit on test speeds rather than anything else. I'm not sure where things break these days but it would be easy enough to check. Doug On 21 Aug 2007, at 20:00, tmrg-interest-request at ICSI.Berkeley.EDU wrote: > Send Tmrg-interest mailing list submissions to > tmrg-interest at ICSI.Berkeley.EDU > > To subscribe or unsubscribe via the World Wide Web, visit > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest > or, via email, send a message with subject or body 'help' to > tmrg-interest-request at ICSI.Berkeley.EDU > > You can reach the person managing the list at > tmrg-interest-owner at ICSI.Berkeley.EDU > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Tmrg-interest digest..." > > > Today's Topics: > > 1. TCP evaluation suite round-table (Lachlan Andrew) > 2. Re: TCP evaluation suite round-table (Lachlan Andrew) > 3. Re: TCP evaluation suite round-table (Lachlan Andrew) > 4. Re: TCP evaluation suite round-table (Lars Eggert) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 20 Aug 2007 21:19:35 -0700 > From: "Lachlan Andrew" > Subject: [Tmrg] TCP evaluation suite round-table > To: tmrg-interest at ICSI.Berkeley.EDU > Message-ID: > > Content-Type: text/plain; charset=ISO-8859-1 > > Greetings all, > > On 8-9 November, a few of us will be getting together at Caltech for a > "round table" to try to agree on some basic parameters and metrics for > TCP evaluation. > > We won't try to answer things like "what fairness metric is best", but > we can agree on some basic parameters. The situation we're trying to > avoid is: > > Group A finds that at 500Mbps, flow 1 reaches 10% of its final > throughput after 30s > Group B finds that at 622Mbps, flow 1 reaches 20% of its final > throughput after 20s > > I'll try to have live video-conferencing via VRVS > so thta those who can't come in > person can still participate. Unfortunately, our videoconferencing > room is small, and so physical attendence will probably be limited to > a dozen or so people. Please let me know if you're interested. > > As basic goals, I'd like to come away from the roundtable with: > - a set of bandwidths that are of interest, say 10, 155, 622, 2500 > Mbps > - a set of buffer sizes that are of interest, like BDP or 16384 > packets > - a set of distributions of RTT that are of interest > - an agreed notion of "convergence time" > -- e.g., "the average over period x is within y% of the > final average > - an agreed notion of "time to converge to fairness" > -- e.g., "the ratio of averages over period x is within y% > of the final ratio" > -- should this metric depend on the final ratio achieved? > - an agreed notion of "intraflow variability" > -- e.g., what timescales are of interest? > - an agreed set of traffic models for background traffic > > Injong has added to that list: > - a measure of total link utilization > - fluctuation in utilization due to fluctuation in background traffic. > - What is the per-flow fair bandwidth share? > > I think some of those will be "easy", and we can sort them out on the > list before an interactive meeting, to save time for more debatable > ones. It would be good if everyone can throw in some ideas for the > next couple of months so that we can see issues are the hard ones. > > It would be good to have a common set of scenarios which could be > tested by simulation, emulation and real networks. Obviously, the > simulation is the most flexible, and so it may have a larger set of > tests, but we can at least simulate the emulated cases. > > Ulterior motive: I'd like people also to simulate/emulate the > scenarios that can also be tested on WAN-in-Lab :) > > If this works, we can get more ambitious in a second roundtable > elsewhere. > > Cheers, > Lachlan > > -- > Lachlan Andrew Dept of Computer Science, Caltech > 1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA > Phone: +1 (626) 395-8820 Fax: +1 (626) 568-3603 > > > ------------------------------ > > Message: 2 > Date: Mon, 20 Aug 2007 21:36:41 -0700 > From: "Lachlan Andrew" > Subject: Re: [Tmrg] TCP evaluation suite round-table > To: tmrg-interest at ICSI.Berkeley.EDU > Message-ID: > > Content-Type: text/plain; charset=ISO-8859-1 > > Greetings again, > > On 20/08/07, Lachlan Andrew wrote: >> As basic goals, I'd like to come away from the roundtable with: >> - a set of bandwidths that are of interest, say 10, 155, 622, 2500 >> Mbps >> >> I think some of those will be "easy", and we can sort them out on the >> list before an interactive meeting, to save time for more debatable >> ones. > > To try my hypothesis, I think that bandwidths should be easy to agree > on on-list. > > Obvious candidates are: > 10 Mbit/s -- old Ethernet, the right ball-part for current ADSL/cable > 54Mbit/s -- 802.11a/g > 100Mbit/s -- Fast Ethernet > 155Mbit/s -- OC3/STM-1 > 400Mbit/s -- used by Doug and Injong's Dummynet studies (IIRC) > 622Mbit/s -- OC12/STM-4 > 1000Mbit/s -- GbE > 2488Mbit/s -- OC48/STM-16 > 9952Mbit/s -- OC192/STM-64 > 10Gbit/s -- 10GbE > > Can we assume that Moore's law now allows Dummynet to run at > 622Mbit/s? If so, I'd strike 400Mbit/s off in favour of OC12. > > With my WAN-in-Lab hat on, I'd vote for GbE, OC48 and 10GbE, since we > have those. > > Can we agree on using > > 10Mbit/s > 100Mbit/s > 622Mbit/s > 1000Mbit/s > 2488Mbit/s > 10Gbit/s > > in all simulations/experiments, unless there is a reason to deviate? > > Cheers, > Lachlan > > > -- > Lachlan Andrew Dept of Computer Science, Caltech > 1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA > Phone: +1 (626) 395-8820 Fax: +1 (626) 568-3603 > > > ------------------------------ > > Message: 3 > Date: Mon, 20 Aug 2007 22:06:38 -0700 > From: "Lachlan Andrew" > Subject: Re: [Tmrg] TCP evaluation suite round-table > To: tmrg-interest at ICSI.Berkeley.EDU > Message-ID: > > Content-Type: text/plain; charset=ISO-8859-1 > > Greetings again, > > On 20/08/07, Lachlan Andrew wrote: > >> - a measure of total link utilization > > Again, I think this should be easy to agree on. > > Rather than simply measuring total bit/s over the link (which can be > achieved by inducing congestion collapse), I would advocate using the > sum of the *receive* rates of the flows using each link. > > For multi-link topologies, this would count the rate of multi-hop > flows multiple times, and so is different from the "total network > throughput" (sum of all flow rates). It is also more "useful" than > measuring the total network throughput, since the latter is maximized > by giving all capacity to any single-hop flows using a link. > > Thoughts? > > Lachlan > > -- > Lachlan Andrew Dept of Computer Science, Caltech > 1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA > Phone: +1 (626) 395-8820 Fax: +1 (626) 568-3603 > > > ------------------------------ > > Message: 4 > Date: Tue, 21 Aug 2007 12:12:07 +0300 > From: Lars Eggert > Subject: Re: [Tmrg] TCP evaluation suite round-table > To: l.andrew at ieee.org > Cc: tmrg-interest at ICSI.Berkeley.EDU > Message-ID: > Content-Type: text/plain; charset="us-ascii" > > Hi, > > On 2007-8-21, at 7:36, ext Lachlan Andrew wrote: >> Can we assume that Moore's law now allows Dummynet to run at >> 622Mbit/s? If so, I'd strike 400Mbit/s off in favour of OC12. > > during my thesis, I found that dummynet doesn't simulate high- > datarate paths very accurately anymore once the CPU becomes loaded: > > However, simulating wide-area Gigabit links with Dummynet > is problematic [ZEC2003]. Dummynet uses the kernel firewall > to identify packets for processing, and depends heavily on > the kernel timers to control when packets leave the > transmission buffer. Both mechanisms incur significant > overheads at high data rates. Furthermore, high data rates > cause high interrupt loads, which can decrease system > responsiveness and eventually lead to livelock [MOGUL1997]. > Because Dummynet processing occurs at the IP layer, device > interrupts cause delays that reduce the accuracy of the > simulation. These delays can also interfere with user-space > processing, and as a result affect the benchmark processes > themselves. > > [ZEC2003] Marko Zec and Miljenko Mikuc. Real-Time IP Network > Simulation at Gigabit Data Rates. Proc. International > Conference on Telecommunications (ConTEL), Zagreb, > Croatia, June 11-13, 2003. > > As you said, Moore's law may have pushed up the region of bandwidth > that can be accurately simulated, but it'd be good to have some > verification. > > Lars > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: smime.p7s > Type: application/pkcs7-signature > Size: 2446 bytes > Desc: not available > Url : http://mailman.ICSI.Berkeley.EDU/pipermail/tmrg-interest/ > attachments/20070821/c6095385/attachment-0001.bin > > ------------------------------ > > _______________________________________________ > Tmrg-interest mailing list > Tmrg-interest at ICSI.Berkeley.EDU > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest > > > End of Tmrg-interest Digest, Vol 11, Issue 4 > ******************************************** From lars.eggert at nokia.com Wed Aug 22 01:31:19 2007 From: lars.eggert at nokia.com (Lars Eggert) Date: Wed, 22 Aug 2007 11:31:19 +0300 Subject: [Tmrg] Tmrg-interest Digest, Vol 11, Issue 4 In-Reply-To: References:

Message-ID: <29750F5F-0B96-4C03-B3AE-0CDD8DA21C6E@nokia.com> On 2007-8-22, at 10:26, ext Douglas Leith wrote: > Re dummynet, my experience these days is that it can run up to 1Gb > using modern hardware. In my experience more of an issue is that end > hosts can still have difficulty at high bandwidth-delay products due > to sack processing overhead etc. and it is this that has placed the > upper limit on test speeds rather than anything else. I'm not sure > where things break these days but it would be easy enough to check. It may be interesting to compare the results in a dummynet setup with one that uses real hardwarde (Lachlan's setup, for example.) What I saw a few years back was that dummynet bunched together packets in bursts, due to the way device interrupts were handled by BSD at the time. Essentially, device driver processing interrupts IP- layer processing, and so dummynet only got cycles intermittently. The result was that although you'd get a simulated path that had the desired bandwidth/delay properties over longer timescales (>> CPU quantum), if you looked at the packet-level trace, there were some oddities there. For my stuff (queueing), that caused some issues, but I'm not sure if it'd matter much for TCP evaluation. Lars -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2446 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/tmrg-interest/attachments/20070822/3c3b302d/attachment.bin From doug.leith at nuim.ie Wed Aug 22 05:45:29 2007 From: doug.leith at nuim.ie (Douglas Leith) Date: Wed, 22 Aug 2007 13:45:29 +0100 Subject: [Tmrg] Tmrg-interest Digest, Vol 11, Issue 4 In-Reply-To: <29750F5F-0B96-4C03-B3AE-0CDD8DA21C6E@nokia.com> References:

<29750F5F-0B96-4C03-B3AE-0CDD8DA21C6E@nokia.com> Message-ID: On 22 Aug 2007, at 09:31, Lars Eggert wrote: > On 2007-8-22, at 10:26, ext Douglas Leith wrote: >> Re dummynet, my experience these days is that it can run up to 1Gb >> using modern hardware. In my experience more of an issue is that end >> hosts can still have difficulty at high bandwidth-delay products due >> to sack processing overhead etc. and it is this that has placed the >> upper limit on test speeds rather than anything else. I'm not sure >> where things break these days but it would be easy enough to check. > > It may be interesting to compare the results in a dummynet setup > with one that uses real hardwarde (Lachlan's setup, for example.) Sounds like a good idea. > What I saw a few years back was that dummynet bunched together > packets in bursts, due to the way device interrupts were handled by > BSD at the time. Essentially, device driver processing interrupts > IP-layer processing, and so dummynet only got cycles > intermittently. The result was that although you'd get a simulated > path that had the desired bandwidth/delay properties over longer > timescales (>> CPU quantum), if you looked at the packet-level > trace, there were some oddities there. For my stuff (queueing), > that caused some issues, but I'm not sure if it'd matter much for > TCP evaluation. Makes sense. Doug From ldunn at cisco.com Thu Aug 23 10:22:33 2007 From: ldunn at cisco.com (Lawrence D. Dunn) Date: Thu, 23 Aug 2007 12:22:33 -0500 Subject: [Tmrg] Fwd: Re: TCP evaluation suite round-table Message-ID: tmrg-interest folks, In response to Lachlan's post on an evaluation suite round-table, I sent him a couple thoughts unicast. Lachlan felt they might be relevant/interesting to the list members (I wasn't totally sure) ;-) so I'm forwarding my note, below. I'll also send his quite-thoughtful reply in a second... Larry -- >Date: Tue, 21 Aug 2007 09:08:33 -0500 >To: l.andrew at ieee.org, ld >From: "Lawrence D. Dunn" >Subject: Re: [Tmrg] TCP evaluation suite round-table >Cc: >Bcc: >X-Attachments: > >Lachlan, (unicast), > ( I almost sent this to the list, but figured, since parts sound > like a bit of a tangential ramble, I'd start with a unicast, > and if I feel the same way in a day or so, maybe send it to the list... ;-) > > I think this one *might* generate some discussion/debate. > > For example, your last sentence (..."useful"...) seems to imply > that giving all capacity to single-hop flows is somehow bad, or wrong. > Though I probably agree, it seems that maybe this connects > "utilization" to some notion of "fairness" (i.e. on what basis > have we concluded that behavior-X is bad/wrong/misleading?) > Are utilization and fairness meant to be coupled, or orthogonal, > or varies-by-scheme-and-that's-OK? > > Also, it might be worth considering, for link utilization, > whether a single-valued metric is the right choice. > For example, is a "square wave" w/ average utilization > of 50% somehow "better" or "worse" that a somewhat-noisy > utilization that seems to hover near 50%? Do we care? > Maybe we should stay away from better/worse judgements at this > early stage, but a single-valued metric (average of the sums > or sum of averages in your example below? ) > probably means that we can't tell the difference. > On one hand, maybe it's best not to complicate things. > On another hand, maybe adding standard-deviation, > or some other metric(s)-of-your-choice, might at least help > capture differentiation between two behaviors > that is useful. > > Or not. Maybe I'm over-thinking it. ;-) > > Is it correct to assume that we're excluding exotica like some > TCP proxy that "fans-out" a single flow, multicast-fashion, > to multiple downstream receivers? I suspect that if such > a proxy were placed after the bottleneck, it might wreak havoc > with the sum-of-receive-rates approach. > (East counter-point: "well, that's not really TCP" - perhaps so, > but it makes me think that the tighter we can bound what > is in/out of the measurement bounds, the fewer disputes > we might have later on. OTOH, tighter-bounds might stifle > some cool/creative approaches, so maybe it's best to not > worry about it). > > Having hinted at conflicting sides of most/all points, I'll > try to be quiet for a bit... ;-) > >Best regards, >Larry >-- > >At 10:06 PM -0700 8/20/07, Lachlan Andrew wrote: >>Greetings again, >> >>On 20/08/07, Lachlan Andrew wrote: >> >>> - a measure of total link utilization >> >>Again, I think this should be easy to agree on. >> >>Rather than simply measuring total bit/s over the link (which can be >>achieved by inducing congestion collapse), I would advocate using the >>sum of the *receive* rates of the flows using each link. >> >>For multi-link topologies, this would count the rate of multi-hop >>flows multiple times, and so is different from the "total network >>throughput" (sum of all flow rates). It is also more "useful" than >>measuring the total network throughput, since the latter is maximized >>by giving all capacity to any single-hop flows using a link. >> >>Thoughts? >> >>Lachlan >> >>-- >>Lachlan Andrew Dept of Computer Science, Caltech >>1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA >>Phone: +1 (626) 395-8820 Fax: +1 (626) 568-3603 >>_______________________________________________ >>Tmrg-interest mailing list >>Tmrg-interest at ICSI.Berkeley.EDU >>http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest From ldunn at cisco.com Thu Aug 23 10:23:20 2007 From: ldunn at cisco.com (Lawrence D. Dunn) Date: Thu, 23 Aug 2007 12:23:20 -0500 Subject: [Tmrg] Fwd: Re: TCP evaluation suite round-table Message-ID: tmrg-interest folks, Here's Lachlan's reply (with his permission). Larry -- >X-from-outside-Cisco: 64.233.184.228 >X-IronPort-Anti-Spam-Filtered: true >X-IronPort-Anti-Spam-Result: Ao8CANusykZA6bjknmdsb2JhbAASjXsBAQIHBAYPGIkzAiQ >X-IronPort-AV: i="4.19,290,1183359600"; > d="scan'208"; a="37870124:sNHT18213240" >DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; > d=gmail.com; s=beta; > >h=domainkey-signature:received:received:message-id:date:from:reply-to:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; > >b=heDStVgnVKrH8deXPyWpHuWTsBnuo6gOKIYBO3uCwI5yGkQtEIgE6HtmwyMj0SyqCFKSzK+0wBZUbD3QxPxfL8Nofw+ZOdP/2wqic2nvA+oIqqzgPfFv9+iFjUIetiMPl1iTQUliMK7P34DRpZNIN3ZoXUwoBgsq4PbSc19QCWI= >Date: Tue, 21 Aug 2007 09:15:04 -0700 >From: "Lachlan Andrew" >Reply-To: l.andrew at ieee.org >To: "Lawrence D. Dunn" , "David Hayes" >Subject: Re: [Tmrg] TCP evaluation suite round-table >Authentication-Results: sj-dkim-3; >header.From=lachlan.andrew at gmail.com; dkim=pass ( > sig from gmail.com/beta verified; ); > >Greetings Larry, > >On 21/08/07, Lawrence D. Dunn wrote: >> Lachlan, (unicast), >> ( I almost sent this to the list, but figured, since parts sound >> like a bit of a tangential ramble, I'd start with a unicast, >> and if I feel the same way in a day or so, maybe send it to the >>list... ;-) > >As always, you raise some really good points. The list would benefit >from all of them. > >> For example, your last sentence (..."useful"...) seems to imply >> that giving all capacity to single-hop flows is somehow bad, or wrong. >> Though I probably agree, it seems that maybe this connects >> "utilization" to some notion of "fairness" (i.e. on what basis >> have we concluded that behavior-X is bad/wrong/misleading?) >> Are utilization and fairness meant to be coupled, or orthogonal, >> or varies-by-scheme-and-that's-OK? > >My personal opinion is that Frank Kelly was spot on. Fairness and >utilization are intimately coupled, and it is easy to evaluate them >together if we can agree on what we're trying to achieve. > >I agree that we should steer away from value judgements at the moment. > However, if we're trying to decide on a small set of measurements >from which we *can* make value judgements, then I think we should >avoid introducing misleading coupling of quantities like fairness >and throughput. > >A metric which is clearly positively correlated with one quantity >(increases with throughput) and negatively correlated with another >(decreases with fairness) is IMO going to be less useful in eventually >making those value judgements, than one which is positively correlated >with one quantity and *independent* of the other. > >If we're going to introduce a measure which is dependent on both >throughput and fairness, I'd advocate "aggregate utility", which >increases with both. > >> Also, it might be worth considering, for link utilization, >> whether a single-valued metric is the right choice. >> For example, is a "square wave" w/ average utilization >> of 50% somehow "better" or "worse" that a somewhat-noisy >> utilization that seems to hover near 50%? Do we care? >> Maybe we should stay away from better/worse judgements at this >> early stage, but a single-valued metric (average of the sums >> or sum of averages in your example below? ) >> probably means that we can't tell the difference. >> On one hand, maybe it's best not to complicate things. >> On another hand, maybe adding standard-deviation, >> or some other metric(s)-of-your-choice, might at least help >> capture differentiation between two behaviors >> that is useful. >> >> Or not. Maybe I'm over-thinking it. ;-) > >Again, a very good point. On question to ask would be why variation >in utilization is bad. To me, it is only bad if it has a measurable >effect on some quantity that applications care about, like loss, >jitter or expected file transfer time. That quantity may be >experienced by the "big" TCP flow, or by CBR cross traffic, etc. >Rather than measuring variation for its own sake, I'd prefer to find >metrics which measure its harm. > >> Is it correct to assume that we're excluding exotica like some >> TCP proxy that "fans-out" a single flow, multicast-fashion, >> to multiple downstream receivers? I suspect that if such >> a proxy were placed after the bottleneck, it might wreak havoc >> with the sum-of-receive-rates approach. > >I agree that it would mean that the sum-of-receive-rates could be much >greater than the link capacity, but I see that as appropriate and in >fact a big benefit of this approach. > >This is very related to my discussions with Michael Welzl on iccrg >about rate-given-corruption (did you see them?). If a flow is giving >benefit to two users, to me it is twice as useful as one giving >benefit only to one, and "should" be given a greater share of the >bottleneck resource. Again, how much more depends on what "utility" >we're trying to maximize, and how much it "costs" the network to >transport the data. > >> (East counter-point: "well, that's not really TCP" - perhaps so, >> but it makes me think that the tighter we can bound what >> is in/out of the measurement bounds, the fewer disputes >> we might have later on. OTOH, tighter-bounds might stifle >> some cool/creative approaches, so maybe it's best to not >> worry about it). > >Yes. My concern with what is in/out is routing. Maximizing the sum >of (link_count*throughput) encourages longer paths. If we're dealing >with fixed routing, then I think we should try to come up with metrics >that are meaningful for all protocols which decide when and how much >to transmit, whether TCP or not. > >Feel free to forward any of this to the list. > >Cheers, >Lachlan > >-- >Lachlan Andrew Dept of Computer Science, Caltech >1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA >Phone: +1 (626) 395-8820 Fax: +1 (626) 568-3603 From sallyfloyd at mac.com Wed Aug 29 20:55:34 2007 From: sallyfloyd at mac.com (Sally Floyd) Date: Wed, 29 Aug 2007 20:55:34 -0700 Subject: [Tmrg] TCP evaluation suite round-table In-Reply-To: References: Message-ID: <7dd379fa8c871c6bfe026828dd76c716@mac.com> > Can we agree on using > > 10Mbit/s > 100Mbit/s > 622Mbit/s > 1000Mbit/s > 2488Mbit/s > 10Gbit/s > > in all simulations/experiments, unless there is a reason to deviate? That sounds reasonable to me, for basic scenarios. It is probably also worthwhile to look at a scenario with a congested low-bandwidth access link (dial-up, since it still exists), and with a low-bandwidth congested wireless link (1-2 Mbps), assuming that those are still around. Because it is good to researchers to also explore what would happen if someone uses a particular congestion control mechanism over a very-low-bandwidth path. - Sally http://www.icir.org/floyd/ From sallyfloyd at mac.com Wed Aug 29 21:14:18 2007 From: sallyfloyd at mac.com (Sally Floyd) Date: Wed, 29 Aug 2007 21:14:18 -0700 Subject: [Tmrg] TCP evaluation suite round-table In-Reply-To: References: Message-ID: Lachlan - > Rather than simply measuring total bit/s over the link (which can be > achieved by inducing congestion collapse), I would advocate using the > sum of the *receive* rates of the flows using each link. I think that is a fine idea. But it would be easy for each simulation or experiment to also report the total bps over the link, in each direction. E.g., to compare with the aggregate receive rates. Or to allow "local" metrics about throughput vs. delay vs. drop rates for understanding the queue management. Or some such. - Sally http://www.icir.org/floyd/