Fix up support for Python 3.12 #1290

rmartin16 · 2023-05-25T12:47:36Z

Changes

Enable new tarfile extraction protections in Python 3.12
- https://docs.python.org/3.12/whatsnew/3.12.html#other-language-changes
- This applies the new "data" filter to unpacking tar files
- This prevents the unpacked files from using absolute filepaths or symlinks from referencing absolute filepaths or filepaths outside of the target directory
- Additionally, the permissions are sanitized to basically ensure they are sane; e.g. world execute permissions are removed if the owner doesn't have execute permissions.

Notes

DeprecationWarning from pytest for ast constants
- [Python 3.12] Assertion rewriting triggers DeprecationWarning pytest-dev/pytest#10977
DeprecationWarning from dateutil for datetime.utcfromtimestamp() usage
- DeprecationWarning: datetime.utcfromtimestamp() is deprecated dateutil/dateutil#1284

PR Checklist:

All new features have been tested
All new features have been documented
I have read the CONTRIBUTING.md file
I will abide by the code of conduct

rmartin16 · 2023-05-25T14:01:44Z

interesting....setuptools is no longer installed in to virtual envs by default...fyi

rmartin16 · 2023-05-25T15:44:03Z

Test coverage still fails for Python 3.12. My guess is coverage hasn't been updated to account for AST changes.

freakboy3742 · 2023-05-25T23:13:57Z

pyproject.toml

@@ -51,6 +52,10 @@ multi_line_output = 3
 testpaths = ["tests"]
 filterwarnings = [
    "error",
+    # Work around https://github.com/pytest-dev/pytest/issues/10977 for Python 3.12
+    'ignore:(ast\.Str|ast\.NameConstant|ast\.Num|Attribute s) is deprecated and will be removed.*:DeprecationWarning:',


I'd rather not do this sort of masking for a pre-release; if we get closer to 3.12 final and it hasn't been fixed, then sure - but our 3.12 support is mostly for "canary" testing at this point (to catch issues like the tarball filter argument). I don't think we need to introduce workarounds for problems that should be fixed before 3.12 final.

That's makes sense; I was kinda leery of it since there isn't a good way to track when it can come back out.

For at least the pytest error, I'm expecting them to fix it pretty quickly and have a new release available...not sure about the dateutil one.

I mostly just want GitHub to add proper support for allow-failure so CI doesn't look like it failed because testing failed on a non-critical platform...:(

Furthermore, though, these changes won't get full coverage until the pytest DeprecationWarning is fixed.

rmartin16 · 2023-05-29T16:57:51Z

Pytest support for 3.12 is blocked on the 3.12.0b2 release.

Python 3.12 support pytest-dev/pytest#10894

rmartin16 · 2023-06-10T20:03:49Z

Waiting on new release for python-dateutil
The missing coverage for Python 3.12 seems to be covered...
- Python 3.12.0b1/2 Missing Coverage for Exiting a Conditional in a Finally Block nedbat/coveragepy#1640
- 3.12 tracing regression: a conditional in a finally block will revisit the condition before exiting the block. python/cpython#105658
  - Unresolved with 3.12.0rc1

rmartin16 · 2023-09-01T00:39:24Z

Sooo....traction with PRs in python-dateutil have been lethargic and Irit hasn't moved on python/cpython#105658.

Given that and the looming 3.12 release, I'm inclined to push a commit with configuration to suppress these errors.

Let me know if you'd rather go a different direction. Alternatively, we could merge this now and just include such mitigations with the updates to CI for 3.12.

freakboy3742 · 2023-09-01T02:38:00Z

Sooo....traction with PRs in python-dateutil have been lethargic and Irit hasn't moved on python/cpython#105658.

Given that and the looming 3.12 release, I'm inclined to push a commit with configuration to suppress these errors.

Let me know if you'd rather go a different direction. Alternatively, we could merge this now and just include such mitigations with the updates to CI for 3.12.

The coverage issue doesn't bother me that much - I'd be happy merging something to ignore that one on the basis that the line is covered (and the overall coverage report supports that), with an open ticket to revert the ignore once the underlying issue is resolved.

The dateutil issue is more of a concern though. Is that actually an error, or just a very loud warning that is being raised as an error by pytest for some reason? It's not clear to me why a DeprecationWarning is presenting as an error. If it is a hard error, I don't think we can ignore it; but if it's an ignorable warning and cookiecutter still works, then I'd be happy with an explicit ignore (again with a ticket to revisit once datetutil is fixed).

rmartin16 · 2023-09-01T02:52:14Z

The dateutil issue is more of a concern though. Is that actually an error, or just a very loud warning that is being raised as an error by pytest for some reason? It's not clear to me why a DeprecationWarning is presenting as an error. If it is a hard error, I don't think we can ignore it; but if it's an ignorable warning and cookiecutter still works, then I'd be happy with an explicit ignore (again with a ticket to revisit once datetutil is fixed).

We updated the pytest configuration a while back to report all warnings as errors.

[tool.pytest.ini_options]
testpaths = ["tests"]
filterwarnings = [
    "error",
]

Nonetheless, the dateutil error is just about datetime.utcfromtimestamp() being deprecating; it won't be an actual problem in 3.12.

Additionally, the transitive dependency is through arrow which currently includes the dependency python-dateutil >= 2.7.0....so, we'll automatically get the update once released.

briefcase==0.3.16.dev133+g77cf5df2.d20230829
├── cookiecutter [required: >=2.2,<3.0, installed: 2.3.0]
│   ├── arrow [required: Any, installed: 1.2.3]
│   │   └── python-dateutil [required: >=2.7.0, installed: 2.8.2]

freakboy3742 · 2023-09-01T02:55:07Z

The dateutil issue is more of a concern though. Is that actually an error, or just a very loud warning that is being raised as an error by pytest for some reason? It's not clear to me why a DeprecationWarning is presenting as an error. If it is a hard error, I don't think we can ignore it; but if it's an ignorable warning and cookiecutter still works, then I'd be happy with an explicit ignore (again with a ticket to revisit once datetutil is fixed).

We updated the pytest configuration a while back to report all warnings as errors.
[tool.pytest.ini_options]
testpaths = ["tests"]
filterwarnings = [
    "error",
]
Nonetheless, the dateutil error is just about datetime.utcfromtimestamp() being deprecating; it won't be an actual problem in 3.12.

Additionally, the transitive dependency is through arrow which currently includes the dependency python-dateutil >= 2.7.0....so, we'll automatically get the update once released.
briefcase==0.3.16.dev133+g77cf5df2.d20230829
├── cookiecutter [required: >=2.2,<3.0, installed: 2.3.0]
│   ├── arrow [required: Any, installed: 1.2.3]
│   │   └── python-dateutil [required: >=2.7.0, installed: 2.8.2]

Ah - that makes sense. In which case, I'm happy with smothering the warnings for now, with some open housekeeping tickets to clean up the code once the problems have been resolved upstream.

- python-dateutil uses the deprecated datetime.utcfromtimestamp(); its next release after 2.8.2 will no longer use this function. - Missing coverage is being falsely reported for an exit from a finally block; the coverage exclusion can be removed with python/cpython#105658.

rmartin16 · 2023-09-01T03:56:48Z

freakboy3742

/me sheds at tear at the beauty... :-)

rmartin16 force-pushed the py312-support branch from 31276cf to bd6c055 Compare May 25, 2023 12:53

rmartin16 force-pushed the py312-support branch from 928e391 to 27760db Compare May 25, 2023 15:32

freakboy3742 reviewed May 25, 2023

View reviewed changes

rmartin16 force-pushed the py312-support branch from 27760db to 30cd541 Compare May 25, 2023 23:24

rmartin16 mentioned this pull request Jun 2, 2023

CI Tests Failure with Release of Python 3.12.0-alpha.7 #1165

Closed

rmartin16 force-pushed the py312-support branch from 30cd541 to b62e9c9 Compare June 10, 2023 19:34

rmartin16 force-pushed the py312-support branch from b62e9c9 to 0615d74 Compare June 12, 2023 14:49

rmartin16 force-pushed the py312-support branch from 0615d74 to 946ea95 Compare July 4, 2023 14:20

rmartin16 force-pushed the py312-support branch from 946ea95 to e13c53b Compare July 11, 2023 14:42

rmartin16 force-pushed the py312-support branch from e13c53b to 2f1460c Compare August 7, 2023 21:01

rmartin16 and others added 2 commits August 31, 2023 23:12

Enable new tarfile unpacking protections in Python 3.12

71f576f

rmartin16 force-pushed the py312-support branch from 2f1460c to fc7f648 Compare September 1, 2023 03:29

This was referenced Sep 1, 2023

Remove python-dateutil error suppression for pytest once >2.8.2 is released #1428

Closed

Remove finally block exit coverage exclusion when python/cpython#105658 is resolved #1429

Closed

rmartin16 marked this pull request as ready for review September 1, 2023 03:56

freakboy3742 approved these changes Sep 1, 2023

View reviewed changes

freakboy3742 merged commit cc2dae1 into beeware:main Sep 1, 2023
35 checks passed

rmartin16 deleted the py312-support branch September 1, 2023 04:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix up support for Python 3.12 #1290

Fix up support for Python 3.12 #1290

rmartin16 commented May 25, 2023 •

edited

Loading

rmartin16 commented May 25, 2023 •

edited

Loading

rmartin16 commented May 25, 2023 •

edited

Loading

freakboy3742 May 25, 2023

rmartin16 May 25, 2023

rmartin16 May 25, 2023

rmartin16 commented May 29, 2023

rmartin16 commented Jun 10, 2023 •

edited

Loading

rmartin16 commented Sep 1, 2023

freakboy3742 commented Sep 1, 2023

rmartin16 commented Sep 1, 2023

freakboy3742 commented Sep 1, 2023

rmartin16 commented Sep 1, 2023

freakboy3742 left a comment

Fix up support for Python 3.12 #1290

Fix up support for Python 3.12 #1290

Conversation

rmartin16 commented May 25, 2023 • edited Loading

Changes

Notes

PR Checklist:

rmartin16 commented May 25, 2023 • edited Loading

rmartin16 commented May 25, 2023 • edited Loading

freakboy3742 May 25, 2023

Choose a reason for hiding this comment

rmartin16 May 25, 2023

Choose a reason for hiding this comment

rmartin16 May 25, 2023

Choose a reason for hiding this comment

rmartin16 commented May 29, 2023

rmartin16 commented Jun 10, 2023 • edited Loading

rmartin16 commented Sep 1, 2023

freakboy3742 commented Sep 1, 2023

rmartin16 commented Sep 1, 2023

freakboy3742 commented Sep 1, 2023

rmartin16 commented Sep 1, 2023

freakboy3742 left a comment

Choose a reason for hiding this comment

rmartin16 commented May 25, 2023 •

edited

Loading

rmartin16 commented May 25, 2023 •

edited

Loading

rmartin16 commented May 25, 2023 •

edited

Loading

rmartin16 commented Jun 10, 2023 •

edited

Loading