Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Meta issue for cleaning up import system cruft #97850

Open
25 of 31 tasks
warsaw opened this issue Oct 4, 2022 · 13 comments
Open
25 of 31 tasks

Meta issue for cleaning up import system cruft #97850

warsaw opened this issue Oct 4, 2022 · 13 comments
Assignees
@brettcannon @warsaw
Labels
topic-importlib

Comments

@warsaw
Copy link
Member

warsaw commented Oct 4, 2022 •
edited by bedevere-app bot
Loading

This is a meta issue tracking all the things that need to be cleaned up in the import system. From removing long deprecated APIs to migrating internal access to __spec__, we'll have several task lists for each related set of clean ups.

@brettcannon @ericsnowcurrently for visibility.

Removing deprecated APIs

Over in What's New for Python 3.11 there's a long list of things that have been deprecated long enough that they can be removed in Python 3.12. Here's that task list for tracking purposes. Watch for linked PRs (maybe one big one or many small ones -- we'll see!).

@brettcannon @ericsnowcurrently for visibility.

Update docs

  • Make sure examples and details from PEP 302 are covered in the importlib docs (/cc @Yhg1s )
  • Remove https://docs.python.org/3/reference/import.html#open-issues (PR)
  • It would be really nice to have a diagram.
  • (import_machinery.rst) how about a section devoted just to the attributes of modules and packages, perhaps expanding upon or supplanting the related entries in the data model reference page?
  • runpy, pkgutil, et al in the library manual should all get “See Also” links at the top pointing to the new import system section.
  • Add more explanation regarding the different ways in which __main__ is initialized?
  • Add more info on __main__ quirks/pitfalls (i.e. copy from PEP 395).

Missing deprecations

Here's a list of related APIs that have not yet been deprecated, but should be. We'll deprecate them in 3.12 and remove them in 3.14.

  • pkgutil.find_loader()
  • pkgutil.get_loader() (or it needs to be reimplemented to not use find_loader())

Other:
@warsaw warsaw changed the title Task list for cleaning up import system cruft for Python 3.12 Meta issue for cleaning up import system cruft Oct 4, 2022
@bedevere-bot bedevere-bot mentioned this issue Oct 5, 2022
Merged
@warsaw warsaw self-assigned this Oct 5, 2022
sobolevn added a commit to sobolevn/cpython that referenced this issue Oct 5, 2022
@sobolevn
pythongh-97850: Remove deprecated functions from importlib.utils
cb4e97c
@bedevere-bot bedevere-bot mentioned this issue Oct 5, 2022
Merged
@sobolevn
Copy link
Member

sobolevn commented Oct 5, 2022

I've removed importlib.utils.set_loader, importlib.utils.set_package, and
importlib.utils.module_for_loader from importlib.utils in #97898 🎉

warsaw added a commit that referenced this issue Oct 5, 2022
@warsaw
gh-97850: Remove all known instances of module_repr() (#97876)
5dc3599 
Remove all known instances of module_repr()
@brettcannon brettcannon self-assigned this Oct 5, 2022
brettcannon added a commit to brettcannon/cpython that referenced this issue Oct 5, 2022
@brettcannon
Remove the open issues section from the import reference
0591f9f 
Tracking in python#97850 instead.
@brettcannon brettcannon mentioned this issue Oct 5, 2022
Merged
sobolevn added a commit to sobolevn/cpython that referenced this issue Oct 5, 2022
@sobolevn
pythongh-97850: Remove deprecated functions from importlib.utils
b3a8ee6
carljm added a commit to carljm/cpython that referenced this issue Oct 6, 2022
@carljm
Merge branch 'main' into dictwatch
4bb5d9d 
* main: (66 commits)
  pythongh-65961: Raise `DeprecationWarning` when `__package__` differs from `__spec__.parent` (python#97879)
  docs(typing): add "see PEP 675" to LiteralString (python#97926)
  pythongh-97850: Remove all known instances of module_repr() (python#97876)
  I changed my surname early this year (python#96671)
  pythongh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (python#97768)
  pythongh-91539: improve performance of get_proxies_environment  (python#91566)
  build(deps): bump actions/stale from 5 to 6 (python#97701)
  pythonGH-95172 Make the same version `versionadded` oneline (python#95172)
  pythongh-88050: Fix asyncio subprocess to kill process cleanly when process is blocked (python#32073)
  pythongh-93738: Documentation C syntax (Function glob patterns -> literal markup) (python#97774)
  pythongh-93357: Port test cases to IsolatedAsyncioTestCase, part 2 (python#97896)
  pythongh-95196: Disable incorrect pickling of the C implemented classmethod descriptors (pythonGH-96383)
  pythongh-97758: Fix a crash in getpath_joinpath() called without arguments (pythonGH-97759)
  pythongh-74696: Pass root_dir to custom archivers which support it (pythonGH-94251)
  pythongh-97661: Improve accuracy of sqlite3.Cursor.fetchone docs (python#97662)
  pythongh-87092: bring compiler code closer to a preprocessing-opt-assembler organisation (pythonGH-97644)
  pythonGH-96704: Add {Task,Handle}.get_context(), use it in call_exception_handler() (python#96756)
  pythongh-93738: Documentation C syntax (:c:type:`PyTypeObject*` -> :c:expr:`PyTypeObject*`) (python#97778)
  pythongh-97825: fix AttributeError when calling subprocess.check_output(input=None) with encoding or errors args (python#97826)
  Add re.VERBOSE flag documentation example (python#97678)
  ...
@miss-islington miss-islington mentioned this issue Oct 6, 2022
Merged
brettcannon added a commit that referenced this issue Oct 6, 2022
@brettcannon
gh-97850: Remove the open issues section from the import reference (#…
f8edc6f 
…97935)

Remove the open issues section from the import reference

Tracking in #97850 instead.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Oct 6, 2022
@brettcannon @miss-islington
pythongh-97850: Remove the open issues section from the import refere…
2ce98a8 
…nce (pythonGH-97935)

Remove the open issues section from the import reference

Tracking in python#97850 instead.
(cherry picked from commit f8edc6f)

Co-authored-by: Brett Cannon <brett@python.org>
@miss-islington miss-islington mentioned this issue Oct 6, 2022
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Oct 6, 2022
@brettcannon @miss-islington
pythongh-97850: Remove the open issues section from the import refere…
1fc2f7e 
…nce (pythonGH-97935)

Remove the open issues section from the import reference

Tracking in python#97850 instead.
(cherry picked from commit f8edc6f)

Co-authored-by: Brett Cannon <brett@python.org>
miss-islington added a commit that referenced this issue Oct 6, 2022
@miss-islington @brettcannon
gh-97850: Remove the open issues section from the import reference (G…
bc03bf4 
…H-97935)

Remove the open issues section from the import reference

Tracking in #97850 instead.
(cherry picked from commit f8edc6f)

Co-authored-by: Brett Cannon <brett@python.org>
miss-islington added a commit that referenced this issue Oct 6, 2022
@miss-islington
[3.11] gh-97850: Remove the open issues section from the import refer…
ae2ab47 
…ence (GH-97935) (GH-97994)

Remove the open issues section from the import reference

Tracking in #97850 instead.
(cherry picked from commit f8edc6f)


Co-authored-by: Brett Cannon <brett@python.org>

Automerge-Triggered-By: GH:brettcannon
carljm added a commit to carljm/cpython that referenced this issue Oct 6, 2022
@carljm
Merge branch 'main' into dictwatch
e5d2169 
* main:
  fixes pythongh-96078: os.sched_yield release the GIL while calling sched_yield(2). (pythongh-97965)
  pythongh-65961: Do not rely solely on `__cached__` (pythonGH-97990)
  pythongh-97850: Remove the open issues section from the import reference (python#97935)
  Docs: pin sphinx-lint (pythonGH-97992)
  pythongh-94590: add signatures to operator itemgetter, attrgetter, methodcaller (python#94591)
  Add Pynche's move to the What's new in 3.11 (python#97974)
  pythongh-97781: Apply changes from importlib_metadata 5. (pythonGH-97785)
  pythongh-86482: Document assignment expression need for ()s (python#23291)
  pythongh-97943: PyFunction_GetAnnotations should return a borrowed reference. (python#97949)
  pythongh-94808: Coverage: Test that maximum indentation level is handled (python#95926)
warsaw pushed a commit that referenced this issue Oct 7, 2022
@sobolevn
gh-97850: Remove deprecated functions from importlib.utils (#97898)
27025e1 
* gh-97850: Remove deprecated functions from `importlib.utils`

* Rebase and remove `set_package` from diff
@warsaw warsaw mentioned this issue Oct 7, 2022
Closed
6 tasks
carljm added a commit to carljm/cpython that referenced this issue Oct 8, 2022
@carljm
Merge branch 'main' into typewatch
1af2a98 
* main: (53 commits)
  pythongh-94808: Coverage: Test that maximum indentation level is handled (python#95926)
  pythonGH-88050: fix race in closing subprocess pipe in asyncio  (python#97951)
  pythongh-93738: Disallow pre-v3 syntax in the C domain (python#97962)
  pythongh-95986: Fix the example using match keyword (python#95989)
  pythongh-97897: Prevent os.mkfifo and os.mknod segfaults with macOS 13 SDK (pythonGH-97944)
  pythongh-94808: Cover `PyUnicode_Count` in CAPI (python#96929)
  pythongh-94808: Cover `PyObject_PyBytes` case with custom `__bytes__` method (python#96610)
  pythongh-95691: Doc BufferedWriter and BufferedReader (python#95703)
  pythonGH-88968: Add notes about socket ownership transfers (python#97936)
  pythongh-96865: [Enum] fix Flag to use CONFORM boundary (pythonGH-97528)
  pythongh-65961: Raise `DeprecationWarning` when `__package__` differs from `__spec__.parent` (python#97879)
  docs(typing): add "see PEP 675" to LiteralString (python#97926)
  pythongh-97850: Remove all known instances of module_repr() (python#97876)
  I changed my surname early this year (python#96671)
  pythongh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (python#97768)
  pythongh-91539: improve performance of get_proxies_environment  (python#91566)
  build(deps): bump actions/stale from 5 to 6 (python#97701)
  pythonGH-95172 Make the same version `versionadded` oneline (python#95172)
  pythongh-88050: Fix asyncio subprocess to kill process cleanly when process is blocked (python#32073)
  pythongh-93738: Documentation C syntax (Function glob patterns -> literal markup) (python#97774)
  ...
carljm added a commit to carljm/cpython that referenced this issue Oct 8, 2022
@carljm
Merge branch 'main' into typewatch
415ed49 
* main:
  pythonGH-97002: Prevent `_PyInterpreterFrame`s from backing more than one `PyFrameObject` (pythonGH-97996)
  pythongh-97973: Return all necessary information from the tokenizer (pythonGH-97984)
  fixes pythongh-96078: os.sched_yield release the GIL while calling sched_yield(2). (pythongh-97965)
  pythongh-65961: Do not rely solely on `__cached__` (pythonGH-97990)
  pythongh-97850: Remove the open issues section from the import reference (python#97935)
  Docs: pin sphinx-lint (pythonGH-97992)
  pythongh-94590: add signatures to operator itemgetter, attrgetter, methodcaller (python#94591)
  Add Pynche's move to the What's new in 3.11 (python#97974)
  pythongh-97781: Apply changes from importlib_metadata 5. (pythonGH-97785)
  pythongh-86482: Document assignment expression need for ()s (python#23291)
  pythongh-97943: PyFunction_GetAnnotations should return a borrowed reference. (python#97949)
carljm added a commit to carljm/cpython that referenced this issue Oct 8, 2022
@carljm
Merge branch 'main' into typewatch
0de5aff 
* main:
  pythongh-86298: Ensure that __loader__ and __spec__.loader agree in warnings.warn_explicit() (pythonGH-97803)
  pythongh-82874: Convert remaining importlib format uses to f-str. (python#98005)
  Docs: Fix backtick errors found by sphinx-lint (python#97998)
  pythongh-97850: Remove deprecated functions from `importlib.utils` (python#97898)
  Remove extra spaces in custom openSSL documentation. (python#93568)
  pythonGH-90985: Revert  "Deprecate passing a message into cancel()" (python#97999)
mpage pushed a commit to mpage/cpython that referenced this issue Oct 11, 2022
@warsaw @mpage
pythongh-97850: Remove all known instances of module_repr() (python#9…
0aca8ca 
…7876)

Remove all known instances of module_repr()
mpage pushed a commit to mpage/cpython that referenced this issue Oct 11, 2022
@brettcannon @mpage
pythongh-97850: Remove the open issues section from the import refere…
9ffbad4 
…nce (python#97935)

Remove the open issues section from the import reference

Tracking in python#97850 instead.
mpage pushed a commit to mpage/cpython that referenced this issue Oct 11, 2022
@sobolevn @mpage
pythongh-97850: Remove deprecated functions from importlib.utils (p…
04a52de 
…ython#97898)

* pythongh-97850: Remove deprecated functions from `importlib.utils`

* Rebase and remove `set_package` from diff
sobolevn added a commit to sobolevn/cpython that referenced this issue Oct 21, 2022
@sobolevn
pythongh-97850: Deprecate find_module and get_module in pkgutil
b7de6df
@warsaw
Copy link
Member Author

warsaw commented Apr 29, 2023

From the meta checklist:

Deprecate pkgutil.find_loader()

I kind of think we're okay here, at least for 3.12. This uses importlib.util.find_spec() so I feel like it's not so urgent to deprecate and actually probably has some utility.

Deprecate pkgutil.get_loader()

Similarly, I'm not sure this even needs to be deprecated. It is implemented in terms of the above, and thus find_spec(), and it has some potentially useful wrapper semantics.

I'm going to mark both issues as complete and not deprecate it in 3.12.

@warsaw
Copy link
Member Author

warsaw commented Apr 29, 2023

@brettcannon @ericsnowcurrently - I think this PR is ready for review. I feel good that we've implemented all the removals we need for 3.12. Like the other issues, we can address the doc updates after beta 1.

@warsaw
Copy link
Member Author

warsaw commented May 3, 2023

The removal branch has been merged, but I'm leaving this meta issue open for now so we can address the documentation checklist during the beta cycle.

This was referenced May 3, 2023
Merged
Merged
AlexWaygood pushed a commit that referenced this issue May 3, 2023
@AA-Turner
GH-97850: Suppress cross-references to the removed module_repr me…
8b03e5f 
…thod (#104133)

Suppress cross-references to ``module_repr``
AlexWaygood added a commit that referenced this issue May 3, 2023
@AA-Turner @AlexWaygood
GH-97850: Suppress cross-references to removed importlib.util fun…
423d7fa 
…ctions (#104134)

`importlib.utils` -> `importlib.util` in a few places

---------

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
@ericsnowcurrently
Copy link
Member

Thanks again for tackling this, @warsaw!

@hugovk
Copy link
Member

hugovk commented May 3, 2023 •
edited
Loading

[For the imp module, I created issues for most of the 71 projects in the PyPI top 5k (listed at python/cpython#98573 (comment)), where I could find their tracker. And some already had issues.

I've checked off projects which had already fixed but not yet released, or have fixed since I created the issue, or where the project is unmaintained or otherwise abandoned and a fix is unlikely in any case.

Currently 24/71 (34%) are checked off. In the issue I pointed to the 3.11 imp docs which have suggestions of how to upgrade each functions/constants, and most seemed fine with that so far, except for one noting imp.load_source has no suggestions.

Edit 2024-10-17: 57/71 (80%)

brettcannon added a commit that referenced this issue May 3, 2023
@sobolevn @CAM-Gerlach
@brettcannon @arhadthedev
gh-97850: Deprecate find_loader and get_loader in pkgutil (GH-9…
d6e83fb 
…8520)

Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
Co-authored-by: Brett Cannon <brett@python.org>
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
@erlend-aasland erlend-aasland mentioned this issue May 3, 2023
Merged
@mdmintz
Copy link

mdmintz commented May 4, 2023

Due to nose being unmaintained (number 41 in your list), that's why there's pynose now (number 55 in your list).
It's a complete drop-in replacement for all occurrences. (Let me know if that isn't the case.)

@simaoafonso-pwt simaoafonso-pwt mentioned this issue May 30, 2023
Closed
@Irtiza90 Irtiza90 mentioned this issue Jun 8, 2023
Closed
@vstinner vstinner mentioned this issue Jun 20, 2023
Closed
@bedevere-bot bedevere-bot mentioned this issue Aug 31, 2023
Merged
AlexWaygood added a commit that referenced this issue Aug 31, 2023
@AlexWaygood
gh-97850: Note in py312 whatsnew that importlib.util.set_loader and…
013a99a 
… `importlib.util.module_for_loader` have been removed (#108719)

Note in py312 whatsnew that `importlib.util.set_loader` and `importlib.util.module_for_loader` have been removed
@bedevere-bot bedevere-bot mentioned this issue Aug 31, 2023
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Aug 31, 2023
@AlexWaygood @miss-islington
pythongh-97850: Note in py312 whatsnew that `importlib.util.set_loade…
e9da6ca 
…r` and `importlib.util.module_for_loader` have been removed (pythonGH-108719)

Note in py312 whatsnew that `importlib.util.set_loader` and `importlib.util.module_for_loader` have been removed
(cherry picked from commit 013a99a)

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Yhg1s pushed a commit that referenced this issue Aug 31, 2023
@miss-islington @AlexWaygood
[3.12] gh-97850: Note in py312 whatsnew that `importlib.util.set_load…
a92c60c 
…er` and `importlib.util.module_for_loader` have been removed (GH-108719) (#108723)

gh-97850: Note in py312 whatsnew that `importlib.util.set_loader` and `importlib.util.module_for_loader` have been removed (GH-108719)

Note in py312 whatsnew that `importlib.util.set_loader` and `importlib.util.module_for_loader` have been removed
(cherry picked from commit 013a99a)

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
@hugovk
Copy link
Member

hugovk commented Nov 5, 2023

Edit 2023-11-05: 36/71 (52%)

We're now over 50% of those in the checklist.

@wallrite wallrite mentioned this issue Apr 18, 2024
Closed
@picnixz picnixz mentioned this issue May 28, 2024
Merged
@hugovk
Copy link
Member

hugovk commented Jun 7, 2024

Edit 2024-06-07: 55/71 (78%)

We're now at 78% (fixed or unmaintained/abandoned).

@ferdnyc ferdnyc mentioned this issue Jun 16, 2024
Open
@cHiv0rz cHiv0rz mentioned this issue Oct 18, 2024
Closed
hugovk added a commit that referenced this issue Nov 1, 2024
@picnixz @hugovk @brettcannon
gh-97850: remove find_loader and get_loader from pkgutil (#…
464a7a9 
…119656)

Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Brett Cannon <brett@python.org>
picnixz added a commit to picnixz/cpython that referenced this issue Dec 8, 2024
@picnixz @hugovk @brettcannon
pythongh-97850: remove find_loader and get_loader from ``pkgu…
21d2e70 
…til`` (python#119656)

Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Brett Cannon <brett@python.org>
@tomasr8
Copy link
Member

tomasr8 commented Dec 18, 2024

Related issue: #121604

Here's a couple more things that might need attention:

In general, lots of methods/attributes are marked as deprecated in the docs for a long time (for example: DEBUG_BYTECODE_SUFFIXES is deprecated since 3.5). It's probably not a good idea to remove them now (especially those that don't yet raise a DeprecationWarning), but we could mark them as deprecated-removed instead, planning to remove them in 3.16?

Would like to know your thoughts on what to do about these :)

@brettcannon
Copy link
Member

👍

That's a @jaraco question.

  • Loader.load_module is deprecated with the deprecation stating it will be removed in 3.12. Should we try to remove this method in 3.14? If not, we should update the deprecation message.

If it has been documented as deprecated for 5 releases and raising an exception for as long, then it can come out.

but we could mark them as deprecated-removed instead, planning to remove them in 3.16?

We should keep it all around for 5 releases of being documented as deprecated as well as raising an exception. There's no rush here as we are just trying to nudge people to better APIs long-term more than avoiding buggy code that's blocking some other change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brettcannon brettcannon

@warsaw warsaw

Labels
topic-importlib
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@brettcannon @warsaw @ericsnowcurrently @hugovk @sobolevn @mdmintz @tomasr8