How to handle non-chunked transfer-encoding headers

[shinrich]

As part of PR #7473, the Transfer-Encoding header was marked hop-by-hop. This means all Transfer-encoding headers were are removed before the proxy request is created. Transfer-encoding headers are only added back as appropriate for the sending from ATS to the origin.

This change was made to address issue #6907.

The mozilla specification https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding indicates that the Transfer-Encoding is a hop-by-hop header, so assuming we interpret the incoming transfer-encoding header the change in PR #7473 is correct and lines up with the spec.

However @bryancall pointed out that ATS really only interprets the chunked value of the Transfer-Encoding header. According to the Mozilla document there are several other values that deal with compressing the data as it is transferred.

Transfer-Encoding: compress
Transfer-Encoding: deflate
Transfer-Encoding: gzip
Transfer-Encoding: identity

In addition in our logic to translate HTTP/1.1 headers to HTTP/2, the transfer-encoding header is stripped. Which again is fine for chunked, but ignores any compression-oriented values of the header.

It seems these days that the vast majority of uses of transfer-encoding headers are just chunked. In fact it is very hard to find articles discussing the other options. So we don't need a particularly efficient solution here. In fact it may be sufficient to just log warnings when we drop these headers.

[masaori335]

RFC 7230 says below in section 3.3.1 for a request but doesn't say anything for a response for intermediate.

A server that receives a request message with a transfer coding it
does not understand SHOULD respond with 501 (Not Implemented).
https://tools.ietf.org/html/rfc7230#section-3.3.1

[shinrich]

I think ATS should follow the server guidance and return 501. I think it will affect a very small number of transactions. If it is between returning 501 and properly uncompressing the data stream, I would much rather return the error code.

[shinrich]

From the PR @bryancall identifies

We should be responding with 415

https://tools.ietf.org/html/rfc7231#section-3.1.2.2
An origin server MAY respond with a status code of 415 (Unsupported
Media Type) if a representation in the request message has a content
coding that is not acceptable.

[shinrich]

https://tools.ietf.org/html/rfc7230#section-3.3.1 discusses transfer-encoding header and gives the guidance that a server that does not understand a transfer-encoding header value should return 501..

https://tools.ietf.org/html/rfc7231#section-3.1.2.2 discusses the content-encoding header which is not the same as the transfer-encoding header. It says that. a server that does not understand an encoding value may return 415.

I think that the guidance is https://tools.ietf.org/html/rfc7230#section-3.3.1 is more appropriate for our situation here with the transfer-encoding header and so we should be returning 501 not 415.

[bryancall]

I reread the RFCs - 415 is for Content-Encoding and 501 would be for Transfer-Encoding. I am good with 501s.

[bryancall]

A 502 would be more appropriate for an origin server that send a non-supported Transfer-Encoding:

6.6.2.  501 Not Implemented

   The 501 (Not Implemented) status code indicates that the server does
   not support the functionality required to fulfill the request.  This
   is the appropriate response when the server does not recognize the
   request method and is not capable of supporting it for any resource.

   A 501 response is cacheable by default; i.e., unless otherwise
   indicated by the method definition or explicit cache controls (see
   Section 4.2.2 of [RFC7234]).

6.6.3.  502 Bad Gateway

   The 502 (Bad Gateway) status code indicates that the server, while
   acting as a gateway or proxy, received an invalid response from an
   inbound server it accessed while attempting to fulfill the request.