[Netalyzr] Network Buffering

Jim Gettys jg at freedesktop.org
Thu Dec 30 11:24:34 PST 2010


On 12/30/2010 12:02 PM, Vern Paxson wrote:
> [Jim dropped from subthread]
>
>> No, not if the upstream/downstream bandwidth in the broadband connection
>> is the bottleneck, rather than the 802.11 or ethernet hop inside the house.
>
> Do you understand his logic here?  Seems all he'll do is shift the bottleneck
> to the local router.  If it's overbuffered, then nothing has changed.  Or
> is he presuming it's not overbuffered?
>

I'll give a concrete example here, as it's easy to be confused (I know, 
I was confused for several months, until I realised there were (at 
least) two separate places for bufferbloat in Linux.

I now have Comcast 50/10 service.  At least with my current E3000 
router, I have to shape the bandwidth to somewhat less than that nominal 
bandwidth to get my latencies under control.  Let's say, for sake of 
this argument, that it's 40/7 I end up with (I seem to have done better 
with other routers, but I've not done systematic experiments).  This 
takes the bad broadband hop out of the running; I'll never fill buffers 
in either direction over the broadband connection (presuming the ISP is 
running RED on the head end, which it appears Comcast does, at least on 
my CMTS).

The smokeping to my home system over ethernet shows constant good 
latency, even under load.

Let's see what happens over wireless:

Upstream:

In some parts of my house, I can get more upstream bandwidth than my 
router can dispose of; in other parts, less bandwidth than I have 
available upstream.

So in one case, the bottleneck is inside the home router (routing from 
802.11 to ethernet); in the other, it's in my laptop.

By twisting the ifqueuelen parameter in either my laptop, or inside the 
router (when I'm running an open source router), I can remove the 
largest single offender of buffering.

But this still leaves the excessive ring buffer buffering in the 
wireless interfaces (something around 256 packets, which is much better 
than the 1000 default of the transmit queue).  Since those drivers don't 
currently implement the right feature for ethtool, you'd have to patch 
the driver to get good buffering out of the wireless driver.  In one 
reply to my posting, one person posted a one line patch to set the 
wireless driver buffering to something reasonable and reported he gets 
the expected good behaviour.


Downstream:

In the downstream case, I am almost always (unless I'm running at 
802.11n near the router) going to have bufferbloat in my home router, 
routing between the incoming ethernet from the cable modem to 802.11.

Again, messing with the parameters in the router (when I've been using 
OpenWRT) has the expected effect.

Conclusions:

1) Whether you have problems with home router or OS bufferbloat is 
critically dependent on the bandwidth of the 802.11 hop versus that of 
the broadband connection.  Before I upped my Comcast service tier, I was 
less likely to suffer from 802.11 bufferbloat; so we can expect home 
router bufferbloat to become more of a problem in the future, and it is 
often much worse when it occurs than the broadband hop.

2) you can mitigate by tuning the transmit queue and/or ring buffer 
sizes to something reasonable.  But most people can't log into their 
home router and do the mitigations there.

3) since the bandwidth's are all over the place (a factor of 100 is 
possible, given 802.11), we're really going to want AQM, and current RED 
isn't likely to hack it. Van thinks his nRED algorithm probably will.

This post over what to do in detail:

http://gettys.wordpress.com/2010/12/13/mitigations-and-solutions-of-bufferbloat-in-home-routers-and-operating-systems/


More information about the Netalyzr mailing list