[Netalyzr] Network Buffering
jg at freedesktop.org
Thu Dec 30 11:24:34 PST 2010
On 12/30/2010 12:02 PM, Vern Paxson wrote:
> [Jim dropped from subthread]
>> No, not if the upstream/downstream bandwidth in the broadband connection
>> is the bottleneck, rather than the 802.11 or ethernet hop inside the house.
> Do you understand his logic here? Seems all he'll do is shift the bottleneck
> to the local router. If it's overbuffered, then nothing has changed. Or
> is he presuming it's not overbuffered?
I'll give a concrete example here, as it's easy to be confused (I know,
I was confused for several months, until I realised there were (at
least) two separate places for bufferbloat in Linux.
I now have Comcast 50/10 service. At least with my current E3000
router, I have to shape the bandwidth to somewhat less than that nominal
bandwidth to get my latencies under control. Let's say, for sake of
this argument, that it's 40/7 I end up with (I seem to have done better
with other routers, but I've not done systematic experiments). This
takes the bad broadband hop out of the running; I'll never fill buffers
in either direction over the broadband connection (presuming the ISP is
running RED on the head end, which it appears Comcast does, at least on
The smokeping to my home system over ethernet shows constant good
latency, even under load.
Let's see what happens over wireless:
In some parts of my house, I can get more upstream bandwidth than my
router can dispose of; in other parts, less bandwidth than I have
So in one case, the bottleneck is inside the home router (routing from
802.11 to ethernet); in the other, it's in my laptop.
By twisting the ifqueuelen parameter in either my laptop, or inside the
router (when I'm running an open source router), I can remove the
largest single offender of buffering.
But this still leaves the excessive ring buffer buffering in the
wireless interfaces (something around 256 packets, which is much better
than the 1000 default of the transmit queue). Since those drivers don't
currently implement the right feature for ethtool, you'd have to patch
the driver to get good buffering out of the wireless driver. In one
reply to my posting, one person posted a one line patch to set the
wireless driver buffering to something reasonable and reported he gets
the expected good behaviour.
In the downstream case, I am almost always (unless I'm running at
802.11n near the router) going to have bufferbloat in my home router,
routing between the incoming ethernet from the cable modem to 802.11.
Again, messing with the parameters in the router (when I've been using
OpenWRT) has the expected effect.
1) Whether you have problems with home router or OS bufferbloat is
critically dependent on the bandwidth of the 802.11 hop versus that of
the broadband connection. Before I upped my Comcast service tier, I was
less likely to suffer from 802.11 bufferbloat; so we can expect home
router bufferbloat to become more of a problem in the future, and it is
often much worse when it occurs than the broadband hop.
2) you can mitigate by tuning the transmit queue and/or ring buffer
sizes to something reasonable. But most people can't log into their
home router and do the mitigations there.
3) since the bandwidth's are all over the place (a factor of 100 is
possible, given 802.11), we're really going to want AQM, and current RED
isn't likely to hack it. Van thinks his nRED algorithm probably will.
This post over what to do in detail:
More information about the Netalyzr