Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-130806: emit ResourceWarning if GzipFile unclosed #130905

Merged
merged 2 commits into from
Mar 13, 2025

Conversation

cmaloney
Copy link
Contributor

@cmaloney cmaloney commented Mar 6, 2025

This may indicate accidental data loss, the crc and size fields of the GzipFile are only written on close.

Ways to make sure all data is written:

  1. Use the file-like object as a “With Statement Context Manager”.
  2. Ensure .close() is always called which flushes data before closing.
  3. If the underlying stream needs to be kept open, use .detach()

Since at least 3.12 flushing has been necessary in GzipFile (see gh-105808 which was a release blocker), this makes that more visible. Users have been encountering as they upgrade to 3.12 (ex. gh-129726).

This may indicate accidental data loss.

Ways to make sure all data is written:
1. Use the [file-like object](https://docs.python.org/3/glossary.html#term-file-object) as a [“With Statement Context Manager”](https://docs.python.org/3/reference/datamodel.html#context-managers).
   - All objects which [inherit](https://docs.python.org/3/tutorial/classes.html#inheritance) from [IOBase](https://docs.python.org/3/library/io.html#io.IOBase) support this.
   - [`BufferedIOBase`](https://docs.python.org/3/library/io.html#io.BufferedIOBase), [`BufferedWriter`](https://docs.python.org/3/library/io.html#io.BufferedWriter), and [`GzipFile`](https://docs.python.org/3/library/gzip.html#gzip.GzipFile) all support this.
   - Ensures `.close()` is called in both exception and regular cases.
1. Ensure [`.close()`](https://docs.python.org/3/library/io.html#io.IOBase.close) is always called which flushes data before closing.
1. If the underlying stream need to be kept open, use [`.detach()`](https://docs.python.org/3/library/io.html#io.BufferedIOBase.detach)

Since 3.12 flushing has been necessary in GzipFile (see pythongh-105808 which was a release blocker), this makes that more visible. Users have been encountering as they upgrade to 3.12 (ex. pythongh-129726).

There are a number of cases of unclosed file-like objects being deleted in CPython libraries and the test suite. This issue includes resolving those cases where the new ResourceWarning is emitted.
Co-authored-by: Victor Stinner <vstinner@python.org>
Copy link
Member

@vstinner vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vstinner
Copy link
Member

vstinner commented Mar 7, 2025

cc @serhiy-storchaka @pitrou

@vstinner vstinner merged commit 93089c0 into python:main Mar 13, 2025
43 checks passed
@vstinner
Copy link
Member

Merged, thank you.

@cmaloney cmaloney deleted the gzipwarning branch March 14, 2025 07:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants