Header name lowercase requirement is excessive

[andrewgodwin]

It was pointed out in https://code.djangoproject.com/ticket/32586 that the initial ASGI requirement that header names in responses MUST be lowercased - inherited from ASGI 1, I believe - doesn't actually match the Django codebase, and upon inspection, this is because, unfortunately, a very small number of HTTP clients are case-sensitive to headers like Content-Type.

I believe the appropriate action is to relax the MUST requirement, and say that servers can/should try to preserve case when passed to them, but nothing requires it; this saves us from entirely flipping the expectation here. Thoughts?

[tomchristie]

Probably a good direction, yup. It's possible there's some places we need to be careful with here.

Eg from a cursory look, there might be some bits of Starlette that would have minor breakages if an ASGI server implementation switched to not lowercasing the incoming headers.

Eg... https://github.com/encode/starlette/blob/master/starlette/datastructures.py#L28

# Determine the hostname to use when reconstructing the URL from the ASGI scope.
host_header = None
    for key, value in scope["headers"]:
        if key == b"host":
            host_header = value.decode("latin-1")
            break

Not really a big deal to review and fix up that project, but there might be other bits of the ecosystem that'd need some review.

The other thing is I don't think I quite understand the wording in "Header names should be lowercased, but it is not required; servers should preserve header case on a best-effort basis.". It's not clear to me if it's advising lowercasing or advising not lowercasing?

Is the difference in wording intended to mean that servers SHOULD lowercase incoming request headers, but SHOULD preserve case on outgoing response headers?

Perhaps something like "Header names may optionally be lowercase, but this is not required" is more clear. Not sure.

[andrewgodwin]

Yes, this is not for incoming request headers, but for outgoing request headers. I agree that all incoming ones should be lowercased; it's the response headers that it seems we occasionally need to keep case sensitivity.

How about this wording instead:

Response header names are case-insensitive according to the HTTP RFCs, and we encourage ASGI applications to lowercase their response headers. However, some HTTP clients are case sensitive to headers like Content-Type, and so for this reason ASGI servers should make a best-effort attempt to preserve the case of response headers.

[adriangb]

Response header names are case-insensitive according to the HTTP RFCs, and we encourage ASGI applications to lowercase their response headers. However, some HTTP clients are case sensitive to headers like Content-Type, and so for this reason ASGI servers should make a best-effort attempt to preserve the case of response headers.

➕ this seems quite reasonable.

I have to assume that there are no clients that break with non-lowercase headers (i.e. they only accept lowercase headers and so they were okay so far because the ASGI server lowercases them) so that if an ASGI server updated itself to preserve case it might break their system?

[pgjones]

I've a request not to lowercase the incoming headers for a use case whereby Hypercorn is used as a proxy and the casing is sensitive downstream. Not lower-casing, does however cause some bugs e.g. pgjones/hypercorn#77. I'm not sure what the least unpleasant solution is.

[synodriver]

I think an asgi middleware could do that. The middleware can be configured whether to lowercase the incoming header or not.

[andrewgodwin]

There's also a point where we can't accommodate literally every single HTTP feature - ASGI has always been about handling 98% of cases, not 100%. I might argue that proxying stuff to specialist servers is a case where it's reasonable to ask someone to write a bit more HTTP handling code than merely running a web app.

[thrau]

From my current understanding, if neither WSGI nor ASGI is capable of preserving header casing, it effectively makes any compliant python web server unusable as application-layer proxy for any backend that expects headers to arrive as they were sent by the client. I wouldn't consider an L7 proxy to be in the 2% bucket of use-cases, but I suppose that's debatable.

More generally, I would be interested in understanding the argument to force lower-casing of incoming request headers in the first place. Are there any benefits for server-side implementations that are so fundamental that it warrants a MUST in the spec?

[andrewgodwin]

At the time the spec was designed, we did an extensive review of a lot of different backends, applications and servers in the Python ecosystem and beyond, and header case being important almost never came up. Combine that with the fact that ASGI (like WSGI) is meant to be an application abstraction, it made sense to remove one potential point of confusion away from frameworks having to implement it and just making it part of the spec - which was especially important given we wanted to make middleware more powerful and easier to write, and having a simpler spec helps there tremendously.

My view is that writing a perfect L7 proxy that has to back a header-case-sensitive server is pretty far on the long tail of use cases, especially as there are several other HTTP features that ASGI deliberately does not implement as it's designed to be an application web abstraction, not a perfect HTTP connection layer. We can move from "must" to "best effort" as I suggested further upthread, but in my experience, if you want to write an L7 proxy, you're probably going to need to go low-level at some point.