Consider ignoring Hop-by-hop Headers such as Transfer-Encoding

[trevorturk]

Following on from a conversation with @olleolleolle in socketry/protocol-http1#11 (comment), this is an exploratory issue which at least should serve as a good search result for those hitting the same problem I did.

To summarize the issue in brief: I'm attempting to migrate an app from https://github.com/typhoeus/typhoeus to https://github.com/socketry/async-http and I have a collection of old VCR cassettes where some hit an Protocol::HTTP1::BadRequest exception from https://github.com/socketry/protocol-http1/blob/e8291e87fd7263f5b36e083cb6ff0fcd9c387fb3/lib/protocol/http1/connection.rb#L484 when reused. This error occurs when a VCR cassette contains both a Transfer-Encoding and a Content-Length.

Thinking that this may be an issue with async-http or related gems, I created a short example demonstrating the issue here: https://github.com/trevorturk/async-http-vcr and the commit showing me removing the problematic Transfer-Encoding header is here: trevorturk/async-http-vcr@d6827d2#diff-c87944a807f1ce4f3f8634203a7cca16f126e07820a98a68aea2b7c147f34a1a

After @ioquatix investigated, he commented (socketry/protocol-http1#11 (comment)) that be believes async-http and related gems are operating as they must and suggested removing all "Hop-by-hop Headers" from VCR cassette recordings. More information about these headers is provided here: https://tools.ietf.org/html/rfc2616#page-92.

[krainboltgreene]

This would be quite a change for many users. We're talking millions of lines of yaml removed.

Further, let's say you wrote a test to make sure your server doesn't server those headers (as per spec), your test now can never be failing, but not because you did the right thing.

[trevorturk]

Yes, I'm not sure what's best to do, if anything at all. FWIW, removing the Transfer-Encoding header from my existing cassette yaml files did solve my specific problem, so at least I'm hoping this issue will be helpful for future potential Googlers encountering the same thing, even if we decide to close this issue!

[krainboltgreene]

I think having an opt-out option would be fine. ignore_hop_by_hop: true being the default.

[ioquatix]

WebMock assumes you can send the response including all the included headers.

At least for some adaptors (like async-http) that won't work. For example, if HTTP/2 was being used, transfer-encoding doesn't make any sense at all.

With HTTP/1, some headers are specific to the connection state rather than metadata pertaining to the response itself. Another example would be connection: close.

You also need to consider how to support trailers as that doesn't seem to be supported in the current design.