Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix collections.abc imports on Python 3.13.0 and 3.13.1 #2657

Merged
merged 3 commits into from
Dec 19, 2024

Conversation

DanielNoord
Copy link
Collaborator

@DanielNoord DanielNoord commented Dec 18, 2024

Supersedes #2656
References pylint-dev/pylint#10112
Reverts and improves on #2647
Is likely related to python/cpython#125415

The issue was partially identified in #2656 but not fixed correctly there I think.

The issue 3.13.1 introduces is that submodules of stdlib modules can now also be frozen and even change their name. This is somewhat annoying, but fixable.
However, while debugging I noticed that find_spec is actually pretty dangerous to call. It imports the full module and can therefore raise warnings etc.

In this PR I take a different approach. Only if we know we're dealing with a stdlib module or stdlib submodule we try and retrieve the frozen module. Otherwise, don't import the module as that is just costly for no additional gain.
The issue with that approach is that it only works on Python 3.10+. To work around this I just copied to old code to an else branch that is only executed on 3.9 and below. This is fine as these versions don't deal with submodules being frozen.

As can be seen from the test runs the tests does serve as a regression. It fails on main, but passes with the fixes in this PR.
The PyPy failure is unrelated. The checks job won't pass until pylint itself is fixed.

There was also already a Changelog entry for 3.3.7, but I just reused that. I think a rebase went wrong there.

I tested this locally against 3.13.0 and 3.13.1. I did see some failures on pylint but my local setup is very broken currently so I can't really determine whether it is related to this PR. Anyhow, I think this fix by itself is worth merging as it clearly fixes a test that should pass. We can worry about pylint later.

Lastly, @Pierre-Sassoulas @jacobtylerwalls I know I won't have time for a release of astroid and pylint in the coming days/weeks. If you approve the PR would you mind doing the release? I can answer questions, but my time off ends today so I need to get back to $work.

@DanielNoord DanielNoord added Brain 🧠 Needs a brain tip High priority Issue with more than 10 reactions backport maintenance/3.3.x python 3.13 labels Dec 18, 2024
@Pierre-Sassoulas
Copy link
Member

Nice 😌, I was running in circle with this bug ! (I don't have time for a full review atm, but should we temporarily test in CI for python 3.13.0 and 3.13.1 ?) I can do the releases, sure.

@DanielNoord
Copy link
Collaborator Author

Nice 😌, I was running in circle with this bug ! (I don't have time for a full review atm, but should we temporarily test in CI for python 3.13.0 and 3.13.1 ?) I can do the releases, sure.

I'm 100% sure this fixes the issue for astroid on both .0 and .1. I tested this extensively locally.

As for pylint I'm not sure, but we could.

Copy link
Member

@Pierre-Sassoulas Pierre-Sassoulas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great !

astroid/interpreter/_import/spec.py Outdated Show resolved Hide resolved
Co-authored-by: Pierre Sassoulas <pierre.sassoulas@gmail.com>
@DanielNoord
Copy link
Collaborator Author

@Pierre-Sassoulas could you reapprove and merge? :)

Copy link

codecov bot commented Dec 19, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.21%. Comparing base (8305199) to head (37a5ca3).
Report is 3 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2657      +/-   ##
==========================================
+ Coverage   93.19%   93.21%   +0.02%     
==========================================
  Files          93       93              
  Lines       11078    11083       +5     
==========================================
+ Hits        10324    10331       +7     
+ Misses        754      752       -2     
Flag Coverage Δ
linux 93.09% <100.00%> (+0.01%) ⬆️
pypy 93.21% <100.00%> (+0.02%) ⬆️
windows 93.21% <100.00%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
astroid/brain/brain_collections.py 100.00% <100.00%> (+5.55%) ⬆️
astroid/const.py 100.00% <ø> (ø)
astroid/interpreter/_import/spec.py 97.65% <100.00%> (+0.06%) ⬆️

... and 3 files with indirect coverage changes

@Pierre-Sassoulas Pierre-Sassoulas merged commit 631a76c into pylint-dev:main Dec 19, 2024
19 of 20 checks passed
Copy link
Contributor

The backport to maintenance/3.3.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-maintenance/3.3.x maintenance/3.3.x
# Navigate to the new working tree
cd .worktrees/backport-maintenance/3.3.x
# Create a new branch
git switch --create backport-2657-to-maintenance/3.3.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 631a76cbdec372481397c000b9883d219c6f8ae1
# Push it to GitHub
git push --set-upstream origin backport-2657-to-maintenance/3.3.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-maintenance/3.3.x

Then, create a pull request where the base branch is maintenance/3.3.x and the compare/head branch is backport-2657-to-maintenance/3.3.x.

@Pierre-Sassoulas
Copy link
Member

Working on the manual cherry-pick atm.

@DanielNoord DanielNoord deleted the collections-fix branch December 19, 2024 12:32
Pierre-Sassoulas pushed a commit that referenced this pull request Dec 19, 2024
* Add test for importing `collections.abc`

* Fix issue with importing of frozen submodules
@Pierre-Sassoulas
Copy link
Member

The cherry-pick is hard, maybe we can release 3.4.0 directly ?

@jacobtylerwalls
Copy link
Member

I can help with the cherry-pick over the weekend if need be.

DanielNoord added a commit that referenced this pull request Dec 20, 2024
)

* Fix `collections.abc` imports on Python 3.13.0 and 3.13.1 (#2657)

* Add test for importing `collections.abc`

* Fix issue with importing of frozen submodules

* Fix `search_paths`

* Do not reassign submodule_path parameters in method bodies

This makes it easier to use less generic annotations with mypy.

---------

Co-authored-by: Daniël van Noord <13665637+DanielNoord@users.noreply.github.com>
Co-authored-by: correctmost <134317971+correctmost@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport maintenance/3.3.x Brain 🧠 Needs a brain tip High priority Issue with more than 10 reactions python 3.13
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants