Internet Engineering Task Force Names Internet Draft Gang Wang Draft-irtf-tmrg-ns2-tool-00.txt Yong Xia NEC Labs, China Expires: October 2007 April 18, 2007 An NS2 TCP Evaluation Tool Suite Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This Internet-Draft will expire on October 18, 2007. Abstract This document discusses the details that have been considered and implemented in an NS2 TCP evaluation tool suite. It works on NS2, a discrete event simulator targeted at networking research and aims at saving the time and efforts spent by TCP researchers on labor intensive simulations. Its rationale of design is centered on several criteria: it includes several typical topologies; it measures some of the most important metrics commonly used in TCP performance evaluation; it is easy to use - a set of statistics is automatically generated at the end of each simulation. Additionally, this tool Expires October 18, 2007 [Page 1] Internet-Draft An NS2 TCP Evaluation Tool Suite April 2007 represents the first step towards a useful open-access TCP variant analysis mechanism. Modifications and improvements are encouraged. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119. Table of Contents 1. Introduction.................................................3 2. TOOL SUITE COMPONENT.........................................4 2.1. Network Topologies......................................5 2.1.1. Single-Bottleneck Dumb-Bell........................5 2.1.2. Multiple-Bottleneck Parking-Lot....................6 2.1.3. Simple Network.....................................6 2.2. Traffic Models..........................................7 2.2.1. Long-lived FTP.....................................7 2.2.2. Short-lived Web....................................7 2.2.3. Streaming Video....................................7 2.2.4. Interactive Voice..................................7 2.3. Performance Metrics.....................................8 2.3.1. Throughput, Delay, jitter and Loss Rates...........8 2.3.1.1. Throughput....................................8 2.3.1.2. Delay.........................................9 2.3.1.3. Jitter........................................9 2.3.1.4. Loss Rates....................................9 2.3.2. Response Times and Oscillations....................9 2.3.3. Fairness and Convergence..........................10 2.3.4. Robustness in Challenging Environments............10 2.4. Output.................................................10 3. Usage Details...............................................10 4. Security Considerations.....................................11 5. IANA Considerations.........................................11 6. Acknowledgments.............................................11 7. References..................................................12 Author's Addresses.............................................13 Intellectual Property Statement................................14 Disclaimer of Validity.........................................14 Copyright Statement............................................14 Acknowledgment.................................................14 Name Expires October 18, 2007 [Page 2] Internet-Draft An NS2 TCP Evaluation Tool Suite April 2007 1. Introduction The Additive-Increase-Multiplicative-Decrease (AIMD) congestion control algorithm employed by TCP [1] is designed to share network resources among competing users fairly and efficiently. However, research results show that TCP does not perform very well with fast long-distance networks [2]. To solve this problem, many TCP variants using high speed transport protocols have been proposed. They include HighSpeed TCP (HS-TCP) [3], Scalable TCP (S-TCP) [4], Binary increase control TCP (BIC-TCP) [5], CUBIC TCP [6], H-TCP [7], XCP [8], VCP [9] and FAST [10]. The proliferation of these many options logically brings the following questions to mind: what is the most effective high speed transport protocol for general usage? In order to answer such questions, a number of high speed TCP variant performance evaluations have been conducted. They include experiments [11] - [12] using a discrete event simulator targeted at networking research, NS2 [13], as well as test-bed analysis [14] and real world experiments [15]. Although test-beds and real world experiments can produce much more realistic results than simulations, implementing a large and complex experimental scenario is really a challenging task. So a great many researchers employ NS2 simulations to test their ideas. However, even NS2 includes a large number of examples and tutorials, carrying out extensive simulations is still exceedingly labor intensive. Suppose one is designing a new congestion control protocol (i.e., a new TCP variant) with all the basic ideas already laid out. What they often do is to implement an algorithm in NS2. The next step is to write a script which defines a network topology and a traffic model. After a simulation run is completed, a trace file is obtained. Another script language (e.g., awk) is then utilized to post-process the file and generate some performance evaluation metrics, as a basis to refine the protocol. One usually needs to change simulation parameters, the network topology, or the traffic model in order to gradually reach a better and better understanding of the protocol in a set of targeted scenarios. This design process continues for many rounds until the researcher is satisfied with the performance of the protocol. The above scenario has been replicated by a great number of TCP researchers, given long-standing interest in TCP-related research. Consequently, a huge amount of time is spent in duplicating similar network topologies, traffic models and performance metrics. One might ask: Why don't we combine these repeated efforts to avoid reinventing the wheel again and again. We could then better use our time to concentrate on more important things. Name Expires October 18, 2007 [Page 3] Internet-Draft An NS2 TCP Evaluation Tool Suite April 2007 In this draft, we try to address the above question with a suite of NS2 simulation scripts that: 1. Define a set of commonly used network topologies, traffic models and performance evaluation metrics. 2. Automate simulation and post-processing procedures (to the greatest extent possible). When carrying out TCP evaluations on NS2 using this tool suite, researchers need only select the typical scenario they wish to use. Simulation results are presented in a series of figures automatically. They do not need to write complicated scenario code and strip the huge amount of trace to obtain useful results. What's more, this simulation suite does not attempt to be a final TCP performance benchmark. Instead, it intends to serve as a starting point. Other researchers in the community are encouraged to continue to develop it towards a comprehensive NS2 TCP evaluation tool suite. Ultimately, the test suites are needed not only for high-speed transport protocols, but for other proposed changes to congestion control mechanisms as well: E.g., ECN added to SYN/ACK packets, to make small transfers more robust; changes in RTO estimation; proposals for ways to distinguish between loss due to congestion and to corruption, etc. 2. TOOL SUITE COMPONENT The tool works as an NS2 patch. Its architecture is shown in Fig.1, which is primarily composed of the following components: network topologies, traffic models, performance evaluation metrics, and a set of output graphs. Name Expires October 18, 2007 [Page 4] Internet-Draft An NS2 TCP Evaluation Tool Suite April 2007 Topology Traffic Model Metrics +----------------+ +----------------+ +-----------------+ | Dumb-bell | |Long-lived FTP | |Network Level | | Parking-lot | |Short-Lived HTTP| | | | Simple-network | |Voice | |Application Level| +----------------+ |Video Streaming | +-----------------+ +----------------+ | | | | | | | | | +-------------------->---<--------------------+ | | V +------------------+ | | | Result in graphs | +------------------+ Figure 1 Architecture of the tool suite. 2.1. Network Topologies The tool suite incorporates 3 commonly used topologies in TCP Performance Evaluations. They are Single-Bottleneck Dumb-Bell, Multiple-Bottleneck Parking-Lot, and Simple Network. More realistic and complex topologies can be introduced into the tool easily. 2.1.1. Single-Bottleneck Dumb-Bell Its topology is shown in Figure 2, in which senders and receivers connect to router 1 or router 2. The bandwidth between routers is much lower than that between other links, which causes the link between routers to be a bottleneck. Name Expires October 18, 2007 [Page 5] Internet-Draft An NS2 TCP Evaluation Tool Suite April 2007 PC1 PC4 \ / \ Bottleneck / PC2--- Router1 --------------Router2--- PC5 / \ / \ PC3 PC6 PC : Personal Computer Figure 2 Dumb-bell topology. 2.1.2. Multiple-Bottleneck Parking-Lot The Parking-Lot topology shown in Figure 3 is similar to the Dumb- Bell topology except that it includes multiple senders and receivers to connect intermediate routers. PC1 PC7 PC4 \ | / \ | / PC2--- Router1 ---Router2...----RouterN--- PC5 / | \ / | \ PC3 PC8 PC6 Figure 3 Parking-lot topology. 2.1.3. Simple Network Here a simple network topology is proposed in Figure 4. In this configuration, the core routers represent the backbone of the network with access routers responsible for sender or receiver nodes to access the network. It is similar to the transit and stub domains in GT-ITM [16]. We have employed static routing as the default routing protocol. Name Expires October 18, 2007 [Page 6] Internet-Draft An NS2 TCP Evaluation Tool Suite April 2007 CR1 PC1 / \ PC5 \ / \ / _--AR2-----CR2 CR4----AR4-- / \ / \ PC2 \ / PC6 CR3 | /\ PC3 PC4 PC: Personal Computer AR: Access Router CR: Core Router Figure 4 Simple Network topology. All of the links in the above topologies have settable parameters. 2.2. Traffic Models The tool attempts to apply the most recent research contributions for traffic settings. The applications involved include four common traffic patterns. 2.2.1. Long-lived FTP FTP traffic uses non-stop file transmission, which begins at a random time and runs on the top of TCP. Implementation details and choice of TCP variants are decided by users, not the scope of this tool suite. 2.2.2. Short-lived Web The web traffic module employs the PackMime HTTP traffic generation model [17], which is now available in the recent NS2 release, NS-2.30 and 2.31. 2.2.3. Streaming Video Streaming traffic is generated using CBR traffic over UDP. Sending rate and packet size are settable. 2.2.4. Interactive Voice There are currently two synthetic voice traffic generation methods available in this tool suite. One is based on CBR-like streaming Name Expires October 18, 2007 [Page 7] Internet-Draft An NS2 TCP Evaluation Tool Suite April 2007 traffic. The other is generated according to a two-state ON/OFF model, in which ON and OFF states are exponentially distributed. The mean ON period is 1.0 s, and the mean OFF duration is 1.35 s. These values are set in accordance with ITU-T recommendations [18]. The voice packet size is 200 bytes, including the 160 bytes data packet (codec G711, 64 kbps rate and 20 ms duration), 20 byte IP header, 8 byte UDP header, and 12 byte RTP header. These parameters can be changed with other audio codec. 2.3. Performance Metrics A comprehensive description on the metrics of TCP performance evaluation is shown in [19]. In the first step, the tool suite tries to implement some commonly used and important metrics in the draft. Here we classify them into network metrics and application metrics. They are listed as follows. 2.3.1. Throughput, Delay, jitter and Loss Rates 2.3.1.1. Throughput For network metrics, we collect bottleneck link utilization as the aggregate link throughput. Throughput is sometimes different from goodput, because goodput consists solely of useful transmitted traffic, where throughput may also include retransmitted traffic [19]. But users care more about the useful bits the network can provide. So the tool collects application level end-to-end goodput no matter what the transport protocol is employed. For FTP traffic, it measures the transmitted traffic during some intervals in bits per second. For HTTP traffic, the PackMime HTTP model collects request /response goodput and response time to measure HTTP traffic performance. Voice and video traffic respond differently from above. Their performances are affected by packet delay, delay jitter and packet loss rate as well as goodput. So their goodput is measured in transmitted packet rate excluding lost packets and delayed packets in excess of a predefined threshold. Name Expires October 18, 2007 [Page 8] Internet-Draft An NS2 TCP Evaluation Tool Suite April 2007 2.3.1.2. Delay We use bottleneck queue size as an indication of queuing delay in bottlenecks. Besides mean and min/max queue size statistics, we also utilize percentile queue size in this tool suite to display what queue states are like during most of the time. FTP traffic is not affected much by the packet transmission delay it experiences. For HTTP traffic, we report on the response time, defined as the duration between the client's sending out requests and receiving the response from the server. For voice and streaming traffic, packet delay is a one way measurement, as defined by the duration between sending and receiving at the end nodes. 2.3.1.3. Jitter Delay jitter is quite important for delay sensitive traffic, such as voice and video. Large jitter requires much more buffer size at the receiver side and may cause high loss rates in strict delay requirements. We employ standard packet delay deviation to show jitter for interactive and streaming traffic. 2.3.1.4. Loss Rates To obtain network statistics, we measure the bottleneck queue loss rate. We do not collect loss rates for FTP and HTTP traffic because they are less affected by this metric. For interactive and streaming traffic, high packet loss rates result in the failure of the receiver to decode the packet. In this tool suite, they are measured during specified intervals. The received packet is considered lost if its delay is beyond a predefined threshold. 2.3.2. Response Times and Oscillations One of the key concerns in the design of congestion control mechanisms has been the response time to sudden network changes. On the one hand, the mechanism should respond rapidly to changes in the network environment. On the other hand, it has to make sure changes are not too severe to ensure the stability of the network [19]. This Name Expires October 18, 2007 [Page 9] Internet-Draft An NS2 TCP Evaluation Tool Suite April 2007 tool suite is designed so the response time and fluctuations can be easily observed using a series of figures it generates, if we set kinds of simulation scenarios, including variable bandwidth, round trip delay, various traffic start times and other parameters. 2.3.3. Fairness and Convergence In this tool suite, the fairness measurement employs the Jain fairness index [20] to survey the fair bandwidth share of end-to-end FTP flows that traverse the same route. Convergence times are the time elapsed between multiple flows from an unfair share of link bandwidth to a fair state. They are quite important for environments with high-bandwidth long-delay flows. We have implemented such scenario to show this metric. The total simulation time is 1000 seconds. Five reverse FTP flows start at the beginning of the simulation, and five forward FTP flows start at every 200 seconds. So after the simulation, the forward FTP throughput figure shows the convergence performance. 2.3.4. Robustness in Challenging Environments A static link packet error model has been introduced in the tool suite to investigate TCP performance in challenging environments. Link failure, routing changes and other diagnostic markers can easily be tested by changing the tool's parameters. 2.4. Output The tool suite employs the RPI NS2 Graphing and Statistics Package [21] to plot the above metrics. At the end of the simulation, it generates a series of figures about the metrics which the researchers are interested in. It also creates html files in order to make the results presented more intuitively, and a backup of the original data files are stored in the tmp directory for users. 3. Usage Details Here we present a brief introduction on how to use this software. Users are recommended to read the software's manual incorporated in its release for more details. The main body of this tool includes three files; those are create_traffic.tcl, create_topology.tcl and create_graph.tcl. As their file names indicate, create_traffic.tcl defines the traffic parameters in the simulation. Create_topology.tcl implements the three common network topologies. Crate_graph.tcl creates graph statistics and displays them at the end of simulation. Name Expires October 18, 2007 [Page 10] Internet-Draft An NS2 TCP Evaluation Tool Suite April 2007 Three example scripts are given in the package. Let's take Dumb-Bell topology as an example. The simulation script is test_dumb_bell.tcl, and its default parameters are set in def_dumb_bell. Those parameters can be changed as the scenario needs. Totally, the parameter setting includes three parts, scenario setting, topology setting and the traffic setting. The scenario setting defines the overall parameters, such as how long the simulation runs, what kinds of figures will be displayed, and using AQM or not. Topology setting sets the specific parameters for this topology. For Dumb-bell, it sets the bottleneck bandwidth, round trip time, propagation delay and Packet Error Rate in the bottleneck link. The traffic setting entry defines the traffic parameters used in the simulation, such as the number of FTP traffic, what TCP variant employed by FTP and other kinds of traffic settings. After the running command "ns test_dumb_bell.tcl", results in text format and graphs in encapsulated postscript format are stored in results/tmp/expX, where X stands for the number of simulations you have already carried out. At the same time, results can be viewed through the file /tmp/expX/index100.html in a more intuitive way. Parking-Lot and Simple Network simulations are the similar process as Dumb-Bell topology. 4. Security Considerations There are no security considerations in this document. 5. IANA Considerations There are no security considerations in this document. 6. Acknowledgments The authors would like to thank Sally Floyd for her kindly reviews and a lot of valuable advice. Name Expires October 18, 2007 [Page 11] Internet-Draft An NS2 TCP Evaluation Tool Suite April 2007 7. References [1] V. Jacobson. "Congestion Avoidance and Control". SIGCOMM'88, August 1998. [2] S. Floyd, S. Ratnasamy, and S. Shenker, "Modifying TCP's Congestion Control for High Speeds, May 2002", available from http://www.icir.org/floyd/notes.html. [3] S. Floyd, "HighSpeed TCP for Large Congestion Windows", RFC 3649, Dec.2003. [4] T. Kelly, "Scalable TCP: Improving Performance in HighSpeed Wide Area Networks." ACM Computer Commun. Rev., vol 33, no 2, pp.83-91, April 2003. [5] L. Xu, K. Harfoush, and I. Rhee, "Binary Increase Congestion Control (BIC) for Fast Long-Distance Networks", IEEE INFOCOM 2004, pp.2514-2524, March 2004. [6] I. Rhee and L. Xu. Cubic: "A New TCP-Friendly High-Speed TCP Variant". In Proceedings of Third International Workshop on Protocols for Fast Long-Distance Networks, 2005 [7] D. Leigh and R. Shorten, "H-TCP Protocol for High-Speed Long Distance Networks", PFLDnet2004. [8] D. Kataabi, M. Handley, and C. Rohrs. "Congestion Control for High Bandwidth-Delay Product Networks". SIGCOMM'02, August 2002 [9] Y. Xia, L. Subramanian, Ion Stoica, and S. Kalyanaraman, "One more Bit Is Enough", Proc. Of Sigcomm 2005, Philadelphia, Pennsylvania, USA. [10] C. Jin, D. Wei, and S. Low. "FAST TCP: Motivation, Architecture, Algorithms, Performance". INFOCOM'04, March 2004. [11] M. Nabeshima and K. YATA, "Performance Evaluation and Comparison of Transport Protocols for Fast Long-Distance Networks." IEICE Trans. Commun., vol. E89-B, NO4 APRIL 2006. [12] S. Mascolo and F. Vacirca, "The Effect of Reverse Traffic on the Performance of New TCP Congestion Control Algorithms". PFLDnet 2006. [13] NS2. http://www.isi.edu/nsnam/ns Name Expires October 18, 2007 [Page 12] Internet-Draft An NS2 TCP Evaluation Tool Suite April 2007 [14] S. Ha, Y. Kim, L. Le, I. Rhee and L. Xu, "A Step toward Realistic Performance Evaluation of High-Speed TCP Variants", PFLDnet2006. [15] H. Bullot, R.Les Cottrell and R. Hughes-Jones, "Evaluation of advanced TCP stacks on fast long distance production networks", Journal of Grid Computing. Vol 1, pp.345-359,2003. [16] GT-ITM. http://www- static.cc.gatech.edu/fac/Ellen.Zegura/graphs.html. [17] J. Cao, W.S. Cleveland, Y. Gao, K. Jeffay, F.D. Smith, and M.C, Weigle, "Stochastic Models for Generating Synthetic HTTP Source Traffic", Proceedings of IEEE INFOCOM Hong Kong, March 2004. [18] ITU-T Recommendation P.59, Artificial conversational speech, ITU-T, Mar. 1993. [19] S. Floyd, "Metrics for the Evaluation of Congestion Control Mechanisms", [Online]. Available: http://www.icir.org/tmrg/. [20] R. Jain, "The Art of Computer Systems Performance Analysis", 1991. [21] RPI ns-2 Graphing and Statistics Package. http://networks.ecse.rpi.edu/~harrisod/graph.html Author's Addresses Gang Wang NEC Laboratories China 14/F, Bldg.A, Innovation Plaza, Tsinghua Science Park No.1, Zhong Guan Chun East Road, Beijing China Phone: +81 10 6270 5962 ext# 511 Email: wanggang@research.nec.com.cn Yong Xia NEC Laboratories China 14/F, Bldg.A, Innovation Plaza, Tsinghua Science Park No.1, Zhong Guan Chun East Road, Beijing China Phone: +81 10 6270 5962 ext# 320 Email: xiayong@research.nec.com.cn Name Expires October 18, 2007 [Page 13] Internet-Draft An NS2 TCP Evaluation Tool Suite April 2007 Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. Disclaimer of Validity This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright Statement Copyright (C) The Internet Society (2007). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society. Name Expires October 18, 2007 [Page 14]