From vsemenishin66 at yahoo.com Mon Dec 27 09:32:03 2010 From: vsemenishin66 at yahoo.com (Vladimir Semenishin) Date: Mon, 27 Dec 2010 09:32:03 -0800 (PST) Subject: [Netalyzr] Network Buffering Message-ID: <604896.59467.qm@web45501.mail.sp1.yahoo.com> Is their a solution to uplink network buffering? I have tried to use qos, but it seems to only work for a few minutes. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/netalyzr/attachments/20101227/f8c170b2/attachment.html From nweaver at ICSI.Berkeley.EDU Wed Dec 29 07:48:06 2010 From: nweaver at ICSI.Berkeley.EDU (Nicholas Weaver) Date: Wed, 29 Dec 2010 07:48:06 -0800 Subject: [Netalyzr] Network Buffering In-Reply-To: <604896.59467.qm@web45501.mail.sp1.yahoo.com> References: <604896.59467.qm@web45501.mail.sp1.yahoo.com> Message-ID: <059E3A34-880F-49B2-869F-C0517332FEFD@icsi.berkeley.edu> Unfortunately there is no real solution currently available. This is an endemic problem in many many network devices, e.g, Jim Gettys (one of the original X11 developers and OLPC developers) has been working on this problem for a while now. WARNING, the following is highly technical: http://gettys.wordpress.com/2010/12/13/mitigations-and-solutions-of-bufferbloat-in-home-routers-and-operating-systems/ The best solution for now is just to know about it: Don't try to use voice applications (like Skype) or realtime applications (like online games, or even much websurfing) while also doing file transfers (like BitTorrent). Either alone will work fine, but due to the buffer behavior, the file transfers will disrupt other applications. On Dec 27, 2010, at 9:32 AM, Vladimir Semenishin wrote: > Is their a solution to uplink network buffering? I have tried to use qos, but it seems to only work for a few minutes. > > _______________________________________________ > Netalyzr mailing list > Netalyzr at mailman.ICSI.Berkeley.EDU > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/netalyzr From jg at freedesktop.org Wed Dec 29 09:59:42 2010 From: jg at freedesktop.org (Jim Gettys) Date: Wed, 29 Dec 2010 12:59:42 -0500 Subject: [Netalyzr] Network Buffering In-Reply-To: <059E3A34-880F-49B2-869F-C0517332FEFD@icsi.berkeley.edu> References: <604896.59467.qm@web45501.mail.sp1.yahoo.com> <059E3A34-880F-49B2-869F-C0517332FEFD@icsi.berkeley.edu> Message-ID: <4D1B770E.1030607@freedesktop.org> On 12/29/2010 10:48 AM, Nicholas Weaver wrote: > Unfortunately there is no real solution currently available. > > This is an endemic problem in many many network devices, e.g, Jim Gettys (one of the original X11 developers and OLPC developers) has been working on this problem for a while now. WARNING, the following is highly technical: http://gettys.wordpress.com/2010/12/13/mitigations-and-solutions-of-bufferbloat-in-home-routers-and-operating-systems/ > > The best solution for now is just to know about it: Don't try to use voice applications (like Skype) or realtime applications (like online games, or even much websurfing) while also doing file transfers (like BitTorrent). > Actually, you can mitigate your broadband link quite well with a mid or high end ($100) home router; you set the uplink/downlink bandwidth to somewhat below the ISP provisioned bandwidth. Works quite well, at least most of the time (if your ISP does not run RED on the head end of the broadband, you may have problems due to congestion in your local area; there is a paper which shows this problem is also quite common). See: http://gettys.wordpress.com/2010/12/13/mitigations-and-solutions-of-bufferbloat-in-home-routers-and-operating-systems/ for details. Note that if your wireless link is lower bandwidth than your uplink speed, you may have bufferbloat in your home router (and in your operating system). This happens commonly on FIOS, where even a 802.11g link can easily be below the bandwidth of the ISP. I see this on my home 50/10 Comcast service, in some parts of my house. Again, there are some things you can do, particularly if you are willing to run open source router code. The real solutions (as opposed to mitigation) involve AQM. Note that after mitigation, my home broadband service is generally below 20ms latency. See: http://www.dslreports.com/r3/smokeping.cgi?target=network.2ea89843611d2ac85ee91c449b367f39.NY to see how my home service now behaves. Best regards, Jim > Either alone will work fine, but due to the buffer behavior, the file transfers will disrupt other applications. > > > On Dec 27, 2010, at 9:32 AM, Vladimir Semenishin wrote: > >> Is their a solution to uplink network buffering? I have tried to use qos, but it seems to only work for a few minutes. >> >> _______________________________________________ >> Netalyzr mailing list >> Netalyzr at mailman.ICSI.Berkeley.EDU >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/netalyzr > > > _______________________________________________ > Netalyzr mailing list > Netalyzr at mailman.ICSI.Berkeley.EDU > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/netalyzr > From vern at icir.org Wed Dec 29 19:01:10 2010 From: vern at icir.org (Vern Paxson) Date: Wed, 29 Dec 2010 19:01:10 -0800 Subject: [Netalyzr] Network Buffering In-Reply-To: <4D1B770E.1030607@freedesktop.org> (Wed, 29 Dec 2010 12:59:42 EST). Message-ID: <20101230030110.B8284369FC7@taffy.ICSI.Berkeley.EDU> > Actually, you can mitigate your broadband link quite well with a mid or > high end ($100) home router; you set the uplink/downlink bandwidth to > somewhat below the ISP provisioned bandwidth. What's the idea here? Won't this just shift the buffering to the home router, which might itself have buffer bloat? Vern From jg at freedesktop.org Thu Dec 30 07:13:49 2010 From: jg at freedesktop.org (Jim Gettys) Date: Thu, 30 Dec 2010 10:13:49 -0500 Subject: [Netalyzr] Network Buffering In-Reply-To: <20101230030110.B8284369FC7@taffy.ICSI.Berkeley.EDU> References: <20101230030110.B8284369FC7@taffy.ICSI.Berkeley.EDU> Message-ID: <4D1CA1AD.6060605@freedesktop.org> On 12/29/2010 10:01 PM, Vern Paxson wrote: >> Actually, you can mitigate your broadband link quite well with a mid or >> high end ($100) home router; you set the uplink/downlink bandwidth to >> somewhat below the ISP provisioned bandwidth. > > What's the idea here? Won't this just shift the buffering to the home > router, which might itself have buffer bloat? > No, not if the upstream/downstream bandwidth in the broadband connection is the bottleneck, rather than the 802.11 or ethernet hop inside the house. Thankfully, 10Mbps ethernet is pretty much a thing of the past. That leaves Wifi as a common offender, as I discovered (having several chimney's in my house, wireless coverage is "interesting). - Jim From jg at freedesktop.org Thu Dec 30 07:26:18 2010 From: jg at freedesktop.org (Jim Gettys) Date: Thu, 30 Dec 2010 10:26:18 -0500 Subject: [Netalyzr] Network Buffering In-Reply-To: <20101230030110.B8284369FC7@taffy.ICSI.Berkeley.EDU> References: <20101230030110.B8284369FC7@taffy.ICSI.Berkeley.EDU> Message-ID: <4D1CA49A.3090604@freedesktop.org> On 12/29/2010 10:01 PM, Vern Paxson wrote: >> Actually, you can mitigate your broadband link quite well with a mid or >> high end ($100) home router; you set the uplink/downlink bandwidth to >> somewhat below the ISP provisioned bandwidth. > > What's the idea here? Won't this just shift the buffering to the home > router, which might itself have buffer bloat? > Also remember that the user's computers are also often a problem in the 802.11 upstream case; modern wireless (and ethernet) devices have very large ring buffers. This is not just a home router problem, but an end-node problem. On my laptop, the driver apparently buffers ~256 packets. (I guess I should be thankful, the hardware supports ring buffer sizes up to 4096). - Jim From vern at icir.org Thu Dec 30 09:02:33 2010 From: vern at icir.org (Vern Paxson) Date: Thu, 30 Dec 2010 09:02:33 -0800 Subject: [Netalyzr] Network Buffering In-Reply-To: <4D1CA1AD.6060605@freedesktop.org> (Thu, 30 Dec 2010 10:13:49 EST). Message-ID: <20101230170233.9F82C36A5A3@taffy.ICSI.Berkeley.EDU> [Jim dropped from subthread] > No, not if the upstream/downstream bandwidth in the broadband connection > is the bottleneck, rather than the 802.11 or ethernet hop inside the house. Do you understand his logic here? Seems all he'll do is shift the bottleneck to the local router. If it's overbuffered, then nothing has changed. Or is he presuming it's not overbuffered? Vern From jg at freedesktop.org Thu Dec 30 11:24:34 2010 From: jg at freedesktop.org (Jim Gettys) Date: Thu, 30 Dec 2010 14:24:34 -0500 Subject: [Netalyzr] Network Buffering In-Reply-To: <20101230170233.9F82C36A5A3@taffy.ICSI.Berkeley.EDU> References: <20101230170233.9F82C36A5A3@taffy.ICSI.Berkeley.EDU> Message-ID: <4D1CDC72.5050209@freedesktop.org> On 12/30/2010 12:02 PM, Vern Paxson wrote: > [Jim dropped from subthread] > >> No, not if the upstream/downstream bandwidth in the broadband connection >> is the bottleneck, rather than the 802.11 or ethernet hop inside the house. > > Do you understand his logic here? Seems all he'll do is shift the bottleneck > to the local router. If it's overbuffered, then nothing has changed. Or > is he presuming it's not overbuffered? > I'll give a concrete example here, as it's easy to be confused (I know, I was confused for several months, until I realised there were (at least) two separate places for bufferbloat in Linux. I now have Comcast 50/10 service. At least with my current E3000 router, I have to shape the bandwidth to somewhat less than that nominal bandwidth to get my latencies under control. Let's say, for sake of this argument, that it's 40/7 I end up with (I seem to have done better with other routers, but I've not done systematic experiments). This takes the bad broadband hop out of the running; I'll never fill buffers in either direction over the broadband connection (presuming the ISP is running RED on the head end, which it appears Comcast does, at least on my CMTS). The smokeping to my home system over ethernet shows constant good latency, even under load. Let's see what happens over wireless: Upstream: In some parts of my house, I can get more upstream bandwidth than my router can dispose of; in other parts, less bandwidth than I have available upstream. So in one case, the bottleneck is inside the home router (routing from 802.11 to ethernet); in the other, it's in my laptop. By twisting the ifqueuelen parameter in either my laptop, or inside the router (when I'm running an open source router), I can remove the largest single offender of buffering. But this still leaves the excessive ring buffer buffering in the wireless interfaces (something around 256 packets, which is much better than the 1000 default of the transmit queue). Since those drivers don't currently implement the right feature for ethtool, you'd have to patch the driver to get good buffering out of the wireless driver. In one reply to my posting, one person posted a one line patch to set the wireless driver buffering to something reasonable and reported he gets the expected good behaviour. Downstream: In the downstream case, I am almost always (unless I'm running at 802.11n near the router) going to have bufferbloat in my home router, routing between the incoming ethernet from the cable modem to 802.11. Again, messing with the parameters in the router (when I've been using OpenWRT) has the expected effect. Conclusions: 1) Whether you have problems with home router or OS bufferbloat is critically dependent on the bandwidth of the 802.11 hop versus that of the broadband connection. Before I upped my Comcast service tier, I was less likely to suffer from 802.11 bufferbloat; so we can expect home router bufferbloat to become more of a problem in the future, and it is often much worse when it occurs than the broadband hop. 2) you can mitigate by tuning the transmit queue and/or ring buffer sizes to something reasonable. But most people can't log into their home router and do the mitigations there. 3) since the bandwidth's are all over the place (a factor of 100 is possible, given 802.11), we're really going to want AQM, and current RED isn't likely to hack it. Van thinks his nRED algorithm probably will. This post over what to do in detail: http://gettys.wordpress.com/2010/12/13/mitigations-and-solutions-of-bufferbloat-in-home-routers-and-operating-systems/