Ensure empty body response for 1xx/204/304 per RFC 9112 sec 6.3 #7756

bdraco · 2023-10-27T16:25:37Z

What do these changes do?

Transfer-Encoding and Content-Length will be removed from the server response when sending a 1xx (Informational), 204 (No Content), or 304 (Not Modified) status since no body is expected for these status codes

Content-Length forbidden on 1xx/204 per https://datatracker.ietf.org/doc/html/rfc7230#section-3.3.2 and discouraged on 304 per https://datatracker.ietf.org/doc/html/rfc7232#section-4.1

We need to ensure Content-Length is kept for HEAD to ensure keep-alive works as expected and is allowed per https://datatracker.ietf.org/doc/html/rfc2616#section-14.13

Transfer-Encoding removed per https://datatracker.ietf.org/doc/html/rfc9112#section-6.1

any recipient on the response chain (including the origin server) can remove transfer codings when they are not needed.

aiohttp would incorrectly send an body of 0\r\n\r\n when chunked was set for Transfer-Encoding with the above status code.

This change ensures aiohttp does not send these invalid bodies.

Are there changes in behavior for the user?

The Transfer-Encoding and Content-Length headers will be removed when sending a 1xx (Informational), 204 (No Content), or 304 (Not Modified) except for HEAD responses since these status since no body is expected for these status codes.

An invalid body of 0\r\n\r\n will no longer be sent with these status codes.

Related issue number

Checklist

I think the code is well written
Unit tests for the changes exist
Documentation reflects the changes
If you provide code modification, please add yourself to CONTRIBUTORS.txt
- The format is <Name> <Surname>.
- Please keep alphabetical order, the file is sorted by names.
Add a new news fragment into the CHANGES folder
- name it <issue_id>.<type> for example (588.bugfix)
- if you don't have an issue_id change it to the pr id after creating the pr
- ensure type is one of the following:
  - .feature: Signifying a new feature.
  - .bugfix: Signifying a bug fix.
  - .doc: Signifying a documentation improvement.
  - .removal: Signifying a deprecation or removal of public API.
  - .misc: A ticket has been closed, but it is not of interest to users.
- Make sure to use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files."

TODO:

figure out why client closes connection on HEAD requests with content length
figure out why client functional tests fail with AIOHTTP_NO_EXTENSIONS=1 test_keepalive_after_empty_body_status and test_keepalive_after_empty_body_status_stream_response -- They do not fail anymore after the fixed in Fix py http parser not treating 204/304/1xx as an empty body #7755
timeline
more functional testing

`Transfer-Encoding` and `Content-Length` will be removed from the server response when sending a 1xx (Informational), 204 (No Content), or 304 (Not Modified) status since no body is expected for these status codes Older aiohttp would incorrectly send an body of "0\r\n\r\n" when chunked was set for Transfer-Encoding. This changes ensures aiohttp can still reforward requests from these versions without http parser failing since the parser would previous try to read these incorrect responses, but no longer does

codecov · 2023-10-27T16:43:20Z

Codecov Report

Merging #7756 (8eebfe1) into master (af2bb1e) will increase coverage by 0.01%.
The diff coverage is 98.52%.

@@            Coverage Diff             @@
##           master    #7756      +/-   ##
==========================================
+ Coverage   97.36%   97.38%   +0.01%     
==========================================
  Files         106      106              
  Lines       31813    32000     +187     
  Branches     3639     3652      +13     
==========================================
+ Hits        30975    31162     +187     
  Misses        633      633              
  Partials      205      205

Flag	Coverage Δ
CI-GHA	`97.30% <98.52%> (+0.01%)`	⬆️
OS-Linux	`96.97% <98.52%> (+0.01%)`	⬆️
OS-Windows	`95.46% <98.52%> (+0.02%)`	⬆️
OS-macOS	`96.63% <98.52%> (+0.01%)`	⬆️
Py-3.10.11	`95.39% <98.52%> (+0.02%)`	⬆️
Py-3.10.13	`96.84% <98.52%> (+0.01%)`	⬆️
Py-3.11.6	`96.51% <98.52%> (+0.02%)`	⬆️
Py-3.12.0	`?`
Py-3.8.10	`95.35% <98.52%> (+0.02%)`	⬆️
Py-3.8.18	`96.76% <98.52%> (+0.01%)`	⬆️
Py-3.9.13	`95.35% <98.52%> (+0.02%)`	⬆️
Py-3.9.18	`96.79% <98.52%> (+0.01%)`	⬆️
Py-pypy7.3.11	`96.22% <98.52%> (+0.03%)`	⬆️
VM-macos	`96.63% <98.52%> (+0.01%)`	⬆️
VM-ubuntu	`96.97% <98.52%> (+0.01%)`	⬆️
VM-windows	`95.46% <98.52%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
aiohttp/helpers.py	`95.69% <100.00%> (+0.03%)`	⬆️
aiohttp/web_fileresponse.py	`99.28% <100.00%> (ø)`
tests/test_client_functional.py	`98.50% <100.00%> (+0.06%)`	⬆️
tests/test_helpers.py	`99.07% <100.00%> (+0.03%)`	⬆️
tests/test_web_functional.py	`98.06% <100.00%> (+0.01%)`	⬆️
tests/test_web_response.py	`99.50% <100.00%> (+0.02%)`	⬆️
aiohttp/web_response.py	`97.72% <86.36%> (-0.19%)`	⬇️

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

aiohttp/web_response.py

tests/test_web_response.py

Dreamsorcerer · 2023-10-27T17:39:21Z

Thanks for digging into this, I didn't realise from the original issue that it was a response from aiohttp that was causing the problem.

bdraco · 2023-10-27T19:39:45Z

Probably could use a test to check the raw body from a 304 with chunked pass through as well for the proxy case so it doesn't accidentally add the bad body

bdraco · 2023-10-27T21:46:45Z

Need to figure out how I regressed

tests/test_client_functional.py::test_bad_payload_chunked_encoding[pyloop]

bdraco · 2023-10-27T23:00:15Z

So it looks like if writer.enable_chunking() isn't called which is now the case for HEAD it won't keep the connection alive...

need to figure that out later though as out of time

bdraco · 2023-10-27T23:00:43Z

Which also means we need keep-alive tests for 304/204/1xx

aiohttp/web_response.py

bdraco · 2023-10-28T17:27:21Z

There is still another issue here because in Response we also force content length "0" when content length is missing and the body is empty. This is probably for clients that don't handle implicit empty body well. We might need to do that in StreamResponse as well. Also the rfc for http 1.0 says content length 0 is invalid ... but in practice it might be better to keep it as is as I expect many clients actually rely on that from experience

Also there doesn't appear to be adequate coverage for validating how things behave with StreamResponse vs Response as well so might need to add that first and than back track and fix any discrepancies

Will need to reconcile that before addressing review comments

Dreamsorcerer · 2023-10-28T17:35:45Z

There is still another issue here because in Response we also force content length "0" when content length is missing and the body is empty. This is probably for clients that don't handle implicit empty body well.

I think that is needed for the client to know where the message ends. The only exception where it would be acceptable to have no content-length header is when the body must be empty (or chunked is being used):
https://www.rfc-editor.org/rfc/rfc9110#section-8.6-5

Also the rfc for http 1.0 says content length 0 is invalid

It says "Any Content-Length greater than or equal to zero is a valid value."

bdraco · 2023-10-28T18:03:38Z

It says "Any Content-Length greater than or equal to zero is a valid value."

Right on, I mentally dropped the equal when I read that

aiohttp/web_response.py

tests/test_client_functional.py

Co-authored-by: Sam Bull <git@sambull.org>

aiohttp/helpers.py

patchback · 2023-11-02T19:58:20Z

Backport to 3.9: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply f63cf18 on top of patchback/backports/3.9/f63cf18fec3e8e6608c7fe7bff9f46a0976f6c25/pr-7756

Backporting merged PR #7756 into master

Ensure you have a local repo clone of your fork. Unless you cloned it
from the upstream, this would be your origin remote.
Make sure you have an upstream repo added as a remote too. In these
instructions you'll refer to it by the name upstream. If you don't
have it, here's how you can add it:
```
$ git remote add upstream https://github.com/aio-libs/aiohttp.git
```

Ensure you have the latest copy of upstream and prepare a branch
that will hold the backported code:

$ git fetch upstream
$ git checkout -b patchback/backports/3.9/f63cf18fec3e8e6608c7fe7bff9f46a0976f6c25/pr-7756 upstream/3.9

Now, cherry-pick PR Ensure empty body response for 1xx/204/304 per RFC 9112 sec 6.3 #7756 contents into that branch:
```
$ git cherry-pick -x f63cf18fec3e8e6608c7fe7bff9f46a0976f6c25
```
If it'll yell at you with something like fatal: Commit f63cf18fec3e8e6608c7fe7bff9f46a0976f6c25 is a merge but no -m option was given., add -m 1 as follows instead:
```
$ git cherry-pick -m1 -x f63cf18fec3e8e6608c7fe7bff9f46a0976f6c25
```
At this point, you'll probably encounter some merge conflicts. You must
resolve them in to preserve the patch from PR Ensure empty body response for 1xx/204/304 per RFC 9112 sec 6.3 #7756 as close to the
original as possible.

Push this branch to your fork on GitHub:

$ git push origin patchback/backports/3.9/f63cf18fec3e8e6608c7fe7bff9f46a0976f6c25/pr-7756

Create a PR, ensure that the CI is green. If it's not — update it so that
the tests and any other checks pass. This is it!
Now relax and wait for the maintainers to process your pull request
when they have some cycles to do reviews. Don't worry — they'll tell you if
any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

bdraco · 2023-11-02T20:11:01Z

I'll get the backport done shortly

…libs#7756)  `Transfer-Encoding` and `Content-Length` will be removed from the server response when sending a 1xx (Informational), 204 (No Content), or 304 (Not Modified) status since no body is expected for these status codes - `Content-Length` forbidden on 1xx/204 per https://datatracker.ietf.org/doc/html/rfc7230#section-3.3.2 and discouraged on 304 per https://datatracker.ietf.org/doc/html/rfc7232#section-4.1 We need to ensure `Content-Length` is kept for `HEAD` to ensure keep-alive works as expected and is allowed per https://datatracker.ietf.org/doc/html/rfc2616#section-14.13 - `Transfer-Encoding` removed per https://datatracker.ietf.org/doc/html/rfc9112#section-6.1 > any recipient on the response chain (including the origin server) can remove transfer codings when they are not needed. `aiohttp` would incorrectly send an body of `0\r\n\r\n` when `chunked` was set for `Transfer-Encoding` with the above status code. This change ensures `aiohttp` does not send these invalid bodies. The `Transfer-Encoding` and `Content-Length` headers will be removed when sending a 1xx (Informational), 204 (No Content), or 304 (Not Modified) except for `HEAD` responses since these status since no body is expected for these status codes. An invalid body of `0\r\n\r\n` will no longer be sent with these status codes.  - [x] I think the code is well written - [x] Unit tests for the changes exist - [ ] Documentation reflects the changes - [ ] If you provide code modification, please add yourself to `CONTRIBUTORS.txt` * The format is <Name> <Surname>. * Please keep alphabetical order, the file is sorted by names. - [x] Add a new news fragment into the `CHANGES` folder * name it `<issue_id>.<type>` for example (588.bugfix) * if you don't have an `issue_id` change it to the pr id after creating the pr * ensure type is one of the following: * `.feature`: Signifying a new feature. * `.bugfix`: Signifying a bug fix. * `.doc`: Signifying a documentation improvement. * `.removal`: Signifying a deprecation or removal of public API. * `.misc`: A ticket has been closed, but it is not of interest to users. * Make sure to use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files." TODO: - [x] figure out why client closes connection on HEAD requests with content length - [x] figure out why client functional tests fail with AIOHTTP_NO_EXTENSIONS=1 `test_keepalive_after_empty_body_status` and `test_keepalive_after_empty_body_status_stream_response` -- They do not fail anymore after the fixed in aio-libs#7755 - [x] timeline - [x] more functional testing --------- Co-authored-by: Sam Bull <git@sambull.org> (cherry picked from commit f63cf18)

bdraco · 2023-11-02T20:34:39Z

Thanks @Dreamsorcerer

… (#7778)  `Transfer-Encoding` and `Content-Length` will be removed from the server response when sending a 1xx (Informational), 204 (No Content), or 304 (Not Modified) status since no body is expected for these status codes - `Content-Length` forbidden on 1xx/204 per https://datatracker.ietf.org/doc/html/rfc7230#section-3.3.2 and discouraged on 304 per https://datatracker.ietf.org/doc/html/rfc7232#section-4.1 We need to ensure `Content-Length` is kept for `HEAD` to ensure keep-alive works as expected and is allowed per https://datatracker.ietf.org/doc/html/rfc2616#section-14.13 - `Transfer-Encoding` removed per https://datatracker.ietf.org/doc/html/rfc9112#section-6.1 > any recipient on the response chain (including the origin server) can remove transfer codings when they are not needed. `aiohttp` would incorrectly send an body of `0\r\n\r\n` when `chunked` was set for `Transfer-Encoding` with the above status code. This change ensures `aiohttp` does not send these invalid bodies. The `Transfer-Encoding` and `Content-Length` headers will be removed when sending a 1xx (Informational), 204 (No Content), or 304 (Not Modified) except for `HEAD` responses since these status since no body is expected for these status codes. An invalid body of `0\r\n\r\n` will no longer be sent with these status codes.  - [x] I think the code is well written - [x] Unit tests for the changes exist - [ ] Documentation reflects the changes - [ ] If you provide code modification, please add yourself to `CONTRIBUTORS.txt` * The format is <Name> <Surname>. * Please keep alphabetical order, the file is sorted by names. - [x] Add a new news fragment into the `CHANGES` folder * name it `<issue_id>.<type>` for example (588.bugfix) * if you don't have an `issue_id` change it to the pr id after creating the pr * ensure type is one of the following: * `.feature`: Signifying a new feature. * `.bugfix`: Signifying a bug fix. * `.doc`: Signifying a documentation improvement. * `.removal`: Signifying a deprecation or removal of public API. * `.misc`: A ticket has been closed, but it is not of interest to users. * Make sure to use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files." TODO: - [x] figure out why client closes connection on HEAD requests with content length - [x] figure out why client functional tests fail with AIOHTTP_NO_EXTENSIONS=1 `test_keepalive_after_empty_body_status` and `test_keepalive_after_empty_body_status_stream_response` -- They do not fail anymore after the fixed in #7755 - [x] timeline - [x] more functional testing --------- Co-authored-by: Sam Bull <git@sambull.org> (cherry picked from commit f63cf18)  ## What do these changes do?  ## Are there changes in behavior for the user?  ## Related issue number  ## Checklist - [ ] I think the code is well written - [ ] Unit tests for the changes exist - [ ] Documentation reflects the changes - [ ] If you provide code modification, please add yourself to `CONTRIBUTORS.txt` * The format is <Name> <Surname>. * Please keep alphabetical order, the file is sorted by names. - [ ] Add a new news fragment into the `CHANGES` folder * name it `<issue_id>.<type>` for example (588.bugfix) * if you don't have an `issue_id` change it to the pr id after creating the pr * ensure type is one of the following: * `.feature`: Signifying a new feature. * `.bugfix`: Signifying a bug fix. * `.doc`: Signifying a documentation improvement. * `.removal`: Signifying a deprecation or removal of public API. * `.misc`: A ticket has been closed, but it is not of interest to users. * Make sure to use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files."

[![Mend Renovate logo banner](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [aiohttp](https://togithub.com/aio-libs/aiohttp) | `==3.8.6` -> `==3.9.0` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/aiohttp/3.9.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/aiohttp/3.9.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/aiohttp/3.8.6/3.9.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/aiohttp/3.8.6/3.9.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>aio-libs/aiohttp (aiohttp)</summary> ### [`v3.9.0`](https://togithub.com/aio-libs/aiohttp/blob/HEAD/CHANGES.rst#390-2023-11-18) [Compare Source](https://togithub.com/aio-libs/aiohttp/compare/v3.8.6...v3.9.0) \================== ## Features - Introduced `AppKey` for static typing support of `Application` storage. See https://docs.aiohttp.org/en/stable/web_advanced.html#application-s-config `#5864 <https://github.com/aio-libs/aiohttp/issues/5864>`\_ - Added a graceful shutdown period which allows pending tasks to complete before the application's cleanup is called. The period can be adjusted with the `shutdown_timeout` parameter. -- by :user:`Dreamsorcerer`. See https://docs.aiohttp.org/en/latest/web_advanced.html#graceful-shutdown `#7188 <https://github.com/aio-libs/aiohttp/issues/7188>`\_ - Added `handler_cancellation <https://docs.aiohttp.org/en/stable/web_advanced.html#web-handler-cancellation>`\_ parameter to cancel web handler on client disconnection. -- by :user:`mosquito` This (optionally) reintroduces a feature removed in a previous release. Recommended for those looking for an extra level of protection against denial-of-service attacks. `#7056 <https://github.com/aio-libs/aiohttp/issues/7056>`\_ - Added support for setting response header parameters `max_line_size` and `max_field_size`. `#2304 <https://github.com/aio-libs/aiohttp/issues/2304>`\_ - Added `auto_decompress` parameter to `ClientSession.request` to override `ClientSession._auto_decompress`. -- by :user:`Daste745` `#3751 <https://github.com/aio-libs/aiohttp/issues/3751>`\_ - Changed `raise_for_status` to allow a coroutine. `#3892 <https://github.com/aio-libs/aiohttp/issues/3892>`\_ - Added client brotli compression support (optional with runtime check). `#5219 <https://github.com/aio-libs/aiohttp/issues/5219>`\_ - Added `client_max_size` to `BaseRequest.clone()` to allow overriding the request body size. -- :user:`anesabml`. `#5704 <https://github.com/aio-libs/aiohttp/issues/5704>`\_ - Added a middleware type alias `aiohttp.typedefs.Middleware`. `#5898 <https://github.com/aio-libs/aiohttp/issues/5898>`\_ - Exported `HTTPMove` which can be used to catch any redirection request that has a location -- :user:`dreamsorcerer`. `#6594 <https://github.com/aio-libs/aiohttp/issues/6594>`\_ - Changed the `path` parameter in `web.run_app()` to accept a `pathlib.Path` object. `#6839 <https://github.com/aio-libs/aiohttp/issues/6839>`\_ - Performance: Skipped filtering `CookieJar` when the jar is empty or all cookies have expired. `#7819 <https://github.com/aio-libs/aiohttp/issues/7819>`\_ - Performance: Only check origin if insecure scheme and there are origins to treat as secure, in `CookieJar.filter_cookies()`. `#7821 <https://github.com/aio-libs/aiohttp/issues/7821>`\_ - Performance: Used timestamp instead of `datetime` to achieve faster cookie expiration in `CookieJar`. `#7824 <https://github.com/aio-libs/aiohttp/issues/7824>`\_ - Added support for passing a custom server name parameter to HTTPS connection. `#7114 <https://github.com/aio-libs/aiohttp/issues/7114>`\_ - Added support for using Basic Auth credentials from :file:`.netrc` file when making HTTP requests with the :py:class:`~aiohttp.ClientSession` `trust_env` argument is set to `True`. -- by :user:`yuvipanda`. `#7131 <https://github.com/aio-libs/aiohttp/issues/7131>`\_ - Turned access log into no-op when the logger is disabled. `#7240 <https://github.com/aio-libs/aiohttp/issues/7240>`\_ - Added typing information to `RawResponseMessage`. -- by :user:`Gobot1234` `#7365 <https://github.com/aio-libs/aiohttp/issues/7365>`\_ - Removed `async-timeout` for Python 3.11+ (replaced with `asyncio.timeout()` on newer releases). `#7502 <https://github.com/aio-libs/aiohttp/issues/7502>`\_ - Added support for `brotlicffi` as an alternative to `brotli` (fixing Brotli support on PyPy). `#7611 <https://github.com/aio-libs/aiohttp/issues/7611>`\_ - Added `WebSocketResponse.get_extra_info()` to access a protocol transport's extra info. `#7078 <https://github.com/aio-libs/aiohttp/issues/7078>`\_ - Allow `link` argument to be set to None/empty in HTTP 451 exception. `#7689 <https://github.com/aio-libs/aiohttp/issues/7689>`\_ ## Bugfixes - Implemented stripping the trailing dots from fully-qualified domain names in `Host` headers and TLS context when acting as an HTTP client. This allows the client to connect to URLs with FQDN host name like `https://example.com./`. \-- by :user:`martin-sucha`. `#3636 <https://github.com/aio-libs/aiohttp/issues/3636>`\_ - Fixed client timeout not working when incoming data is always available without waiting. -- by :user:`Dreamsorcerer`. `#5854 <https://github.com/aio-libs/aiohttp/issues/5854>`\_ - Fixed `readuntil` to work with a delimiter of more than one character. `#6701 <https://github.com/aio-libs/aiohttp/issues/6701>`\_ - Added `__repr__` to `EmptyStreamReader` to avoid `AttributeError`. `#6916 <https://github.com/aio-libs/aiohttp/issues/6916>`\_ - Fixed bug when using `TCPConnector` with `ttl_dns_cache=0`. `#7014 <https://github.com/aio-libs/aiohttp/issues/7014>`\_ - Fixed response returned from expect handler being thrown away. -- by :user:`Dreamsorcerer` `#7025 <https://github.com/aio-libs/aiohttp/issues/7025>`\_ - Avoided raising `UnicodeDecodeError` in multipart and in HTTP headers parsing. `#7044 <https://github.com/aio-libs/aiohttp/issues/7044>`\_ - Changed `sock_read` timeout to start after writing has finished, avoiding read timeouts caused by an unfinished write. -- by :user:`dtrifiro` `#7149 <https://github.com/aio-libs/aiohttp/issues/7149>`\_ - Fixed missing query in tracing method URLs when using `yarl` 1.9+. `#7259 <https://github.com/aio-libs/aiohttp/issues/7259>`\_ - Changed max 32-bit timestamp to an aware datetime object, for consistency with the non-32-bit one, and to avoid a `DeprecationWarning` on Python 3.12. `#7302 <https://github.com/aio-libs/aiohttp/issues/7302>`\_ - Fixed `EmptyStreamReader.iter_chunks()` never ending. -- by :user:`mind1m` `#7616 <https://github.com/aio-libs/aiohttp/issues/7616>`\_ - Fixed a rare `RuntimeError: await wasn't used with future` exception. -- by :user:`stalkerg` `#7785 <https://github.com/aio-libs/aiohttp/issues/7785>`\_ - Fixed issue with insufficient HTTP method and version validation. `#7700 <https://github.com/aio-libs/aiohttp/issues/7700>`\_ - Added check to validate that absolute URIs have schemes. `#7712 <https://github.com/aio-libs/aiohttp/issues/7712>`\_ - Fixed unhandled exception when Python HTTP parser encounters unpaired Unicode surrogates. `#7715 <https://github.com/aio-libs/aiohttp/issues/7715>`\_ - Updated parser to disallow invalid characters in header field names and stop accepting LF as a request line separator. `#7719 <https://github.com/aio-libs/aiohttp/issues/7719>`\_ - Fixed Python HTTP parser not treating 204/304/1xx as an empty body. `#7755 <https://github.com/aio-libs/aiohttp/issues/7755>`\_ - Ensure empty body response for 1xx/204/304 per RFC 9112 sec 6.3. `#7756 <https://github.com/aio-libs/aiohttp/issues/7756>`\_ - Fixed an issue when a client request is closed before completing a chunked payload. -- by :user:`Dreamsorcerer` `#7764 <https://github.com/aio-libs/aiohttp/issues/7764>`\_ - Edge Case Handling for ResponseParser for missing reason value. `#7776 <https://github.com/aio-libs/aiohttp/issues/7776>`\_ - Fixed `ClientWebSocketResponse.close_code` being erroneously set to `None` when there are concurrent async tasks receiving data and closing the connection. `#7306 <https://github.com/aio-libs/aiohttp/issues/7306>`\_ - Added HTTP method validation. `#6533 <https://github.com/aio-libs/aiohttp/issues/6533>`\_ - Fixed arbitrary sequence types being allowed to inject values via version parameter. -- by :user:`Dreamsorcerer` `#7835 <https://github.com/aio-libs/aiohttp/issues/7835>`\_ - Performance: Fixed increase in latency with small messages from websocket compression changes. `#7797 <https://github.com/aio-libs/aiohttp/issues/7797>`\_ ## Improved Documentation - Fixed the `ClientResponse.release`'s type in the doc. Changed from `comethod` to `method`. `#5836 <https://github.com/aio-libs/aiohttp/issues/5836>`\_ - Added information on behavior of base_url parameter in `ClientSession`. `#6647 <https://github.com/aio-libs/aiohttp/issues/6647>`\_ - Fixed `ClientResponseError` docs. `#6700 <https://github.com/aio-libs/aiohttp/issues/6700>`\_ - Updated Redis code examples to follow the latest API. `#6907 <https://github.com/aio-libs/aiohttp/issues/6907>`\_ - Added a note about possibly needing to update headers when using `on_response_prepare`. -- by :user:`Dreamsorcerer` `#7283 <https://github.com/aio-libs/aiohttp/issues/7283>`\_ - Completed `trust_env` parameter description to honor `wss_proxy`, `ws_proxy` or `no_proxy` env. `#7325 <https://github.com/aio-libs/aiohttp/issues/7325>`\_ - Expanded SSL documentation with more examples (e.g. how to use certifi). -- by :user:`Dreamsorcerer` `#7334 <https://github.com/aio-libs/aiohttp/issues/7334>`\_ - Fix, update, and improve client exceptions documentation. `#7733 <https://github.com/aio-libs/aiohttp/issues/7733>`\_ ## Deprecations and Removals - Added `shutdown_timeout` parameter to `BaseRunner`, while deprecating `shutdown_timeout` parameter from `BaseSite`. -- by :user:`Dreamsorcerer` `#7718 <https://github.com/aio-libs/aiohttp/issues/7718>`\_ - Dropped Python 3.6 support. `#6378 <https://github.com/aio-libs/aiohttp/issues/6378>`\_ - Dropped Python 3.7 support. -- by :user:`Dreamsorcerer` `#7336 <https://github.com/aio-libs/aiohttp/issues/7336>`\_ - Removed support for abandoned `tokio` event loop. -- by :user:`Dreamsorcerer` `#7281 <https://github.com/aio-libs/aiohttp/issues/7281>`\_ ## Misc - Made `print` argument in `run_app()` optional. `#3690 <https://github.com/aio-libs/aiohttp/issues/3690>`\_ - Improved performance of `ceil_timeout` in some cases. `#6316 <https://github.com/aio-libs/aiohttp/issues/6316>`\_ - Changed importing Gunicorn to happen on-demand, decreasing import time by ~53%. -- :user:`Dreamsorcerer` `#6591 <https://github.com/aio-libs/aiohttp/issues/6591>`\_ - Improved import time by replacing `http.server` with `http.HTTPStatus`. `#6903 <https://github.com/aio-libs/aiohttp/issues/6903>`\_ - Fixed annotation of `ssl` parameter to disallow `True`. -- by :user:`Dreamsorcerer`. `#7335 <https://github.com/aio-libs/aiohttp/issues/7335>`\_ *** </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/allenporter/pyrainbird).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

…libs#7756) (aio-libs#7778)  `Transfer-Encoding` and `Content-Length` will be removed from the server response when sending a 1xx (Informational), 204 (No Content), or 304 (Not Modified) status since no body is expected for these status codes - `Content-Length` forbidden on 1xx/204 per https://datatracker.ietf.org/doc/html/rfc7230#section-3.3.2 and discouraged on 304 per https://datatracker.ietf.org/doc/html/rfc7232#section-4.1 We need to ensure `Content-Length` is kept for `HEAD` to ensure keep-alive works as expected and is allowed per https://datatracker.ietf.org/doc/html/rfc2616#section-14.13 - `Transfer-Encoding` removed per https://datatracker.ietf.org/doc/html/rfc9112#section-6.1 > any recipient on the response chain (including the origin server) can remove transfer codings when they are not needed. `aiohttp` would incorrectly send an body of `0\r\n\r\n` when `chunked` was set for `Transfer-Encoding` with the above status code. This change ensures `aiohttp` does not send these invalid bodies. The `Transfer-Encoding` and `Content-Length` headers will be removed when sending a 1xx (Informational), 204 (No Content), or 304 (Not Modified) except for `HEAD` responses since these status since no body is expected for these status codes. An invalid body of `0\r\n\r\n` will no longer be sent with these status codes.  - [x] I think the code is well written - [x] Unit tests for the changes exist - [ ] Documentation reflects the changes - [ ] If you provide code modification, please add yourself to `CONTRIBUTORS.txt` * The format is <Name> <Surname>. * Please keep alphabetical order, the file is sorted by names. - [x] Add a new news fragment into the `CHANGES` folder * name it `<issue_id>.<type>` for example (588.bugfix) * if you don't have an `issue_id` change it to the pr id after creating the pr * ensure type is one of the following: * `.feature`: Signifying a new feature. * `.bugfix`: Signifying a bug fix. * `.doc`: Signifying a documentation improvement. * `.removal`: Signifying a deprecation or removal of public API. * `.misc`: A ticket has been closed, but it is not of interest to users. * Make sure to use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files." TODO: - [x] figure out why client closes connection on HEAD requests with content length - [x] figure out why client functional tests fail with AIOHTTP_NO_EXTENSIONS=1 `test_keepalive_after_empty_body_status` and `test_keepalive_after_empty_body_status_stream_response` -- They do not fail anymore after the fixed in aio-libs#7755 - [x] timeline - [x] more functional testing --------- Co-authored-by: Sam Bull <git@sambull.org> (cherry picked from commit f63cf18)     - [ ] I think the code is well written - [ ] Unit tests for the changes exist - [ ] Documentation reflects the changes - [ ] If you provide code modification, please add yourself to `CONTRIBUTORS.txt` * The format is <Name> <Surname>. * Please keep alphabetical order, the file is sorted by names. - [ ] Add a new news fragment into the `CHANGES` folder * name it `<issue_id>.<type>` for example (588.bugfix) * if you don't have an `issue_id` change it to the pr id after creating the pr * ensure type is one of the following: * `.feature`: Signifying a new feature. * `.bugfix`: Signifying a bug fix. * `.doc`: Signifying a documentation improvement. * `.removal`: Signifying a deprecation or removal of public API. * `.misc`: A ticket has been closed, but it is not of interest to users. * Make sure to use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files."

bdraco mentioned this pull request Oct 27, 2023

** 3.9 Beta testing ** #7675

Closed

8 tasks

bdraco changed the title ~~Remove Transfer-Encoding for empty body response per RFC 9112 sec 6.3~~ Ensure empty body response for 1xx/204/304 per RFC 9112 sec 6.3 Oct 27, 2023

Dreamsorcerer requested changes Oct 27, 2023

View reviewed changes

aiohttp/web_response.py Outdated Show resolved Hide resolved

Dreamsorcerer reviewed Oct 27, 2023

View reviewed changes

tests/test_web_response.py Outdated Show resolved Hide resolved

bdraco added 4 commits October 27, 2023 15:25

review

dce3e25

make enable_chunking conditional on status_code_indicates_empty_body

e3db2b8

Handle HEAD response as well

3fb3f79

move asserts out of side effect (keep mock since it needs to be async)

8f9f752

bdraco added 6 commits October 27, 2023 17:20

fix regression

65d711f

remove unreachable

efcbcaf

add functional tests

9c8c8fa

add a head test as well

8ed3c5e

reduce comment

028e8a2

reduce comment

9d54c05

add test for empty body status

d60e402

Dreamsorcerer reviewed Oct 28, 2023

View reviewed changes

aiohttp/web_response.py Outdated Show resolved Hide resolved

Dreamsorcerer reviewed Oct 28, 2023

View reviewed changes

aiohttp/web_response.py Outdated Show resolved Hide resolved

bdraco added 2 commits October 28, 2023 12:57

fixes for empty body with client

ce8e12e

must_be_empty_body

591875d

move must_be_empty_body block out one level

5b557b9

Dreamsorcerer reviewed Oct 31, 2023

View reviewed changes

aiohttp/web_response.py Outdated Show resolved Hide resolved

Dreamsorcerer reviewed Oct 31, 2023

View reviewed changes

aiohttp/web_response.py Show resolved Hide resolved

Dreamsorcerer reviewed Oct 31, 2023

View reviewed changes

aiohttp/web_response.py Outdated Show resolved Hide resolved

Dreamsorcerer reviewed Oct 31, 2023

View reviewed changes

tests/test_client_functional.py Outdated Show resolved Hide resolved

bdraco and others added 4 commits October 31, 2023 10:31

Update aiohttp/web_response.py

bea21de

Co-authored-by: Sam Bull <git@sambull.org>

simplify check

8f7c0e2

remove superfluous close

a7dc41c

adjust logic for empty body on 200 with connect

5630ca5

Dreamsorcerer reviewed Oct 31, 2023

View reviewed changes

aiohttp/helpers.py Outdated Show resolved Hide resolved

bdraco added 8 commits October 31, 2023 11:04

sync and adjust test

79d55e2

adjust CONNECT check for 2xx

286a108

handle head with body unset

e851e0a

adjust test_head_returns_empty_body

02f22fa

fix 304 test

15c9814

Merge branch 'master' into ignore_transfer_encoding_304_rfc_9112_sq

ae1c29a

ruff

3e5a6ee

make it just a bit safer

cccbdb3

Dreamsorcerer reviewed Oct 31, 2023

View reviewed changes

aiohttp/helpers.py Show resolved Hide resolved

Update links

8eebfe1

Dreamsorcerer added the backport-3.9 label Nov 2, 2023

Dreamsorcerer merged commit f63cf18 into aio-libs:master Nov 2, 2023
29 of 34 checks passed

bdraco mentioned this pull request Nov 14, 2023

Ensure empty body responses never generate an invalid chunked response home-assistant/supervisor#4710

Merged

12 tasks

Dreamsorcerer mentioned this pull request Feb 2, 2024

Failure introduced from 3.8.5 --> 3.8.6; Passing on 3.9+ #8122

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensure empty body response for 1xx/204/304 per RFC 9112 sec 6.3 #7756

Ensure empty body response for 1xx/204/304 per RFC 9112 sec 6.3 #7756

bdraco commented Oct 27, 2023 •

edited

Loading

codecov bot commented Oct 27, 2023 •

edited

Loading

Dreamsorcerer commented Oct 27, 2023

bdraco commented Oct 27, 2023

bdraco commented Oct 27, 2023

bdraco commented Oct 27, 2023

bdraco commented Oct 27, 2023

bdraco commented Oct 28, 2023 •

edited

Loading

Dreamsorcerer commented Oct 28, 2023 •

edited

Loading

bdraco commented Oct 28, 2023

patchback bot commented Nov 2, 2023 •

edited

Loading

bdraco commented Nov 2, 2023

bdraco commented Nov 2, 2023

Ensure empty body response for 1xx/204/304 per RFC 9112 sec 6.3 #7756

Ensure empty body response for 1xx/204/304 per RFC 9112 sec 6.3 #7756

Conversation

bdraco commented Oct 27, 2023 • edited Loading

What do these changes do?

Are there changes in behavior for the user?

Related issue number

Checklist

codecov bot commented Oct 27, 2023 • edited Loading

Codecov Report

Dreamsorcerer commented Oct 27, 2023

bdraco commented Oct 27, 2023

bdraco commented Oct 27, 2023

bdraco commented Oct 27, 2023

bdraco commented Oct 27, 2023

bdraco commented Oct 28, 2023 • edited Loading

Dreamsorcerer commented Oct 28, 2023 • edited Loading

bdraco commented Oct 28, 2023

patchback bot commented Nov 2, 2023 • edited Loading

Backport to 3.9: 💔 cherry-picking failed — conflicts found

Backporting merged PR #7756 into master

bdraco commented Nov 2, 2023

bdraco commented Nov 2, 2023

bdraco commented Oct 27, 2023 •

edited

Loading

codecov bot commented Oct 27, 2023 •

edited

Loading

bdraco commented Oct 28, 2023 •

edited

Loading

Dreamsorcerer commented Oct 28, 2023 •

edited

Loading

patchback bot commented Nov 2, 2023 •

edited

Loading