AOL Rewrites the HTTP Specs

Shortcut to: What's the Big Deal?

After battling with AOL's decision to dictate their interpretation of the HTTP/1.1 spec, we have these findings to share with the public.

This past Sunday (15/12/1996), AOL appeared to have "upgraded" their proxy software. As the upgrade progressed, it became apparent that the new AOL proxy software refused to communicate with webservers sending HTTP/1.1 response headers, and answered their customer's request with the following message:

  <H1>UNSUPPORTED WEB VERSION</H1>

  <H2>The Web address you requested is not available in a version supported
  by AOL.  This is an issue with the Web site, and not with AOL. The owner
  of this site is using an unsupported HTTP language. If you receive this
  message frequently, you may want to set your web graphics preferences to
  COMPRESSED at Keyword: PREFERENCES</H2>

As near as we can tell, the proxy address is hardcoded into the AOL browsers, and at first this changed seemed to only affect Win95 versions of the AOL browser. However, after a couple days of work, AOL seems to have progressed further with their upgrade and the results now seem to affect every platform and version of the AOL browsers. Netscape and MSIE used over AOL connections appear to work fine, probably since they are not configured (by default) to use the AOL proxy servers.

Since many Apache group members provide services to clients whose content must be servable to AOL users, several of us had to take some action with our Apache 1.2 servers and implement a BrowserMatch change that is at this time handling the AOL requests by sending an HTTP/1.0 response header. AOL appeared yesterday to still be holding in cache the failure message above for failed attempts made since the Sunday deployment of the proxy upgrade. It appears that this morning most of these messages have expired on previously failed connections.

We can confirm that the problem is the HTTP/1.1 response header and nothing else. Other Apache 1.2 servers that we are running, which don't care if AOL ever sees them, are still responding with HTTP/1.1 responses and are unreachable from AOL browsers. One of these is the Apache web site itself.

AOL's response has been as follows:

> ---------------- Begin Forwarded Message ----------------
> This message comes from sites that do not issue explicit HTTP/1.0
> responses to HTTP/1.0 requests.  In the past we've had problems with
> sites returning HTTP/0.9 responses and only the "compressed" side was
> able to filter them out (issuing "cannot retreive..." messages.  Now
> those sites can be caught on both compressed and uncompressed sides of
> the house.
>
> However, a new problem was emerging on the horizon.  New HTTP/1.1 web
> servers are starting to generate HTTP/1.1 responses to HTTP/1.0 requests
> when they should be generating only HTTP/1.0 responses.  We wanted to
> stem the tide of those faults proliferating and becoming a de facto
> standard by blocking them now.  Hopefully the authors of those web
> servers will change their software to only generate HTTP/1.1 responses
> when an HTTP/1.1 request is submitted.
> ----------------- End Forwarded Message -----------------

This is completely incorrect: the HTTP spec states that minor-version-number revisions are backwards compatible, that giving a 1.1 response to a 1.0 client is completely acceptable, since the changes made are additive in nature. Section 3.1 of the HTTP specification states:

The <minor> number is incremented when the changes made to the protocol add features which do not change the general message parsing algorithm, but which may add to the message semantics and imply additional capabilities of the sender. The <major> number is incremented when the format of a message within the protocol is changed.

Even if the developers of the AOL proxy were unaware of HTTP/1.1, the HTTP/1.0 specification states exactly the same thing, section 3.1. In order for their proxy to be minimally compliant with HTTP/1.0, they must allow the behavior Apache is showing.

What's the big deal? The big deal is that if HTTP/1.1 servers had to pretend to be broken for HTTP/1.0 clients, the rate of graceful technological evolution of the web would slow, and the much-much-needed improvements in HTTP/1.1 would be delayed. But an even bigger deal is that we have a large company which is publicly snubbing consensus-developed Internet standards - a process they have been a party to, so they have no excuse for ignorance. If a large company can single-handedly prevent interoperability between their software (used by 8 million people on a daily basis) and the most widely used web server in the world, what hope do public Internet standards have?

Some more thoughts on the subject:

First, we should emphasize that responding with an HTTP/1.1 version identifier does not mean that the message header will use HTTP/1.1 features. This is in the specification; the version identifier is designed to identify the capabilities of the server. When Apache sends a response to an HTTP/1.0 request, it responds with a valid HTTP/1.0 response -- the only thing which might be surprising to an HTTP/1.0 client is the version identifier.

Both HTTP/1.0 and HTTP/1.1 specifications describe the version identifier in the following way: "...The protocol versioning policy is intended to allow the sender to indicate the format of a message and its capacity for understanding further HTTP communication, rather than the features obtained via that communication..."

Further, the specification for both requires that an HTTP/1.1 message (either the client's or the server's) must be comprehensible to an HTTP/1.0 client or server. So what we're looking at isn't whether or not we should send comprehensible responses, but whether or not we should tell the remote server what version of HTTP we're capable of speaking.

Sending a version number which indicates the capabilities of the sender provides another opportunity to negotiate a transition from HTTP/1.0 to HTTP/1.1, for HTTP/1.1-capable clients. It also correctly identifies the version of HTTP which the server uses.

We could also partition the logic so that only clients ever initiate a transition from HTTP/1.0 to HTTP/1.1, but that creates an equally large possibility for failure, namely that HTTP/1.0 servers will reject such requests. Along those lines, it is worth noting that this 'problem' started several days after releasing Apache 1.2b2, when AOL decided to start dropping HTTP/1.1 responses -- not because there they caused any problems, but rather because someone (or some people) decided it was appropriate.

It would be preferable to have either client or server be able to offer a transition to another protocol.