This fascinating article about HTTP performance caught my eye on Slashdot today. It parallels the W3C paper about HTTP, keep-alives and pipelining that I used as a reference while working on zsync. zsync uses HTTP 1.1 with keep-alive and pipelining, and certainly has terrible performance without it.

The most interesting thing is how the whole article is really about problems caused by not having pipelining enabled by default in web browsers. The recommendation ‒ run 4 different hostnames, as a way of raising the browser based restriction (actually mandated by the HTTP RFC) on the number of connections in parallel to the same site from 2 to 8 ‒ is effectively saying that the system as-is is broken. The RFC limits to 2 connections at a time because there is a tragedy of the commons here: every individual user gains by opening as many parellel connections as they can, but this hurts overall network performance (HTTP keep-alives are designed specifically to avoid having one HTTP connection overhead per HTTP request). Having lots of connections reduces latency, but it creates a lot of TCP connection overhead on the network and for the server.

The article is up-front about the fact that the performance problems are all solved by enabling pipelining. The discussion of upstream bandwidth is interesting, and I hadn't considered the effect of asymmetric Internet connections in this light before. But the upstream bandwidth wouldn't affect the latency (except for the first request) if pipelining was used; zsync works fine on ADSL, because it uses pipelining to transmit the next request up while the current one is being received down. Upstream bandwidth could still affect latency, but far less than the effect the article shows for servers without pipelining; and making multiple connections would certainly do no better.

Since the owners of broken servers are free to disable HTTP 1.1 keep-alives, or just put a proxy in front, I second the idea of just turning on pipelining in the mainstream browsers. HTTP 1.1 already solved this problem. The only reason we aren't benefitting from the solution is that it is disabled due to broken servers; but they will never be fixed if the option is never turned on.