Add Python 3.12 to the CI matrix #2148

webknjaz · 2024-12-16T04:58:14Z

Contributor checklist

Included tests for the changes.
PR title is short, clear, and ready to be included in the user-facing changelog.

Maintainer checklist

Verified one of these labels is present: backwards incompatible, feature, enhancement, deprecation, bug, dependency, docs or skip-changelog as they determine changelog listing.
Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

hugovk · 2024-12-16T06:51:03Z

3.12 is failing on the CI: https://github.com/jazzband/pip-tools/actions/runs/12346208386/job/34451449901?pr=2148

WhyNotHugo · 2024-12-16T12:14:29Z

Failure for piplowest is due to it referencing pkgutil.ImpImporter, which is gone is Python 3.12.

I think that we'll need to bump the minimal support version of pip.

WhyNotHugo · 2024-12-16T12:14:32Z

I think that failure with pipsupported might be due to to this change in pip:

freeze no longer excludes the setuptools, distribute, and wheel from the output when running on Python 3.12 or later, where they are not included in a virtual environment by default. Use --exclude if you wish to exclude any of these packages. (#4256)

WhyNotHugo · 2024-12-16T12:18:22Z

References to pkgutil.ImpImporter were made conditional in pypa/pip@f1a7a6f, which was first included in 23.1.2, tagged 2023-04-26.

I wonder if we need to update the minimal supported pip, or if we can claim "For Python 3.12, the minimal pip is 23.1.2", while remaining compatible with older versions in Python < 3.12.

WhyNotHugo · 2024-12-16T12:28:06Z

See also: #2104

macro1 · 2024-12-16T21:50:00Z

pip itself only supports the most recent version. is there a reason to be careful to maximize backward compatibility?

webknjaz · 2024-12-16T22:31:36Z

pip itself only supports the most recent version. is there a reason to be careful to maximize backward compatibility?

I think historically, pip-tools would support the previous pip version and the current one. But now with so low activity we're not catching up with the changes on time. The README even used to have a compatibility table.

#926 / #1950

I didn't really want this PR to become a discussion of the support drops. Only wanted to add modern Python to the matrix. Obviously, it needs some tweaks, likely incorporating the ideas from #2105.

macro1 · 2024-12-16T22:43:18Z

I didn't really want this PR to become a discussion of the support drops.

that's fine. we can discuss it elsewhere as well. really i meant to voice my support for dropping support for older versions over adding complexity to the pipeline

webknjaz · 2024-12-16T23:26:04Z

I think that for as long as a good range of Pythons is supported, it's okay to depend on newer Pip versions. But I don't have stats on why people would want to use older versions of it, honestly. This would require somebody to lead a discussion and make sure different user groups are reached, aware and heard.

Ref jazzband#2148 (comment)

webknjaz · 2024-12-17T00:08:11Z

References to pkgutil.ImpImporter were made conditional in pypa/pip@f1a7a6f, which was first included in 23.1.2, tagged 2023-04-26.

I wonder if we need to update the minimal supported pip, or if we can claim "For Python 3.12, the minimal pip is 23.1.2", while remaining compatible with older versions in Python < 3.12.

Post tox invocation, this is reproducible via

$ .tox/py312-piplowest/bin/python -I -m pip install               
ERROR: Exception:
Traceback (most recent call last):
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/commands/install.py", line 315, in run
    session = self.get_default_session(options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 98, in get_default_session
    self._session = self.enter_context(self._build_session(options))
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 125, in _build_session
    session = PipSession(
              ^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/network/session.py", line 343, in __init__
    self.headers["User-Agent"] = user_agent()
                                 ^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/network/session.py", line 175, in user_agent
    setuptools_dist = get_default_environment().get_distribution("setuptools")
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 180, in get_distribution
    return next(matches, None)
           ^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 177, in <genexpr>
    for distribution in self.iter_all_distributions()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/base.py", line 594, in iter_all_distributions
    for dist in self._iter_distributions():
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 168, in _iter_distributions
    for dist in finder.find_eggs(location):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 136, in find_eggs
    yield from self._find_eggs_in_dir(location)
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 103, in _find_eggs_in_dir
    from pip._vendor.pkg_resources import find_distributions
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/__main__.py", line 31, in <module>
    sys.exit(_main())
             ^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/cli/main.py", line 70, in main
    return command.main(cmd_args)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 101, in main
    return self._main(args)
           ^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
    self.handle_pip_version_check(options)
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 179, in handle_pip_version_check
    session = self._build_session(
              ^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 125, in _build_session
    session = PipSession(
              ^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/network/session.py", line 343, in __init__
    self.headers["User-Agent"] = user_agent()
                                 ^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/network/session.py", line 175, in user_agent
    setuptools_dist = get_default_environment().get_distribution("setuptools")
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 180, in get_distribution
    return next(matches, None)
           ^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 177, in <genexpr>
    for distribution in self.iter_all_distributions()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/base.py", line 594, in iter_all_distributions
    for dist in self._iter_distributions():
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 168, in _iter_distributions
    for dist in finder.find_eggs(location):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 136, in find_eggs
    yield from self._find_eggs_in_dir(location)
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 103, in _find_eggs_in_dir
    from pip._vendor.pkg_resources import find_distributions
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?

Ref jazzband#2148 (comment)

webknjaz · 2024-12-17T00:15:09Z

@WhyNotHugo I applied the suggested update.

Ref jazzband#2148 (comment)

WhyNotHugo

LGTM. Just one questions for my own curiosity.

WhyNotHugo · 2024-12-17T04:44:58Z

pyproject.toml

@@ -38,7 +38,8 @@ dependencies = [
  # direct dependencies
  "build >= 1.0.0",
  "click >= 8",
-  "pip >= 22.2",
+  "pip >= 23.1; python_version >= '3.12'",
+  "pip >= 22.2; python_version < '3.12'",


Is the conditional required here? Without it, would the rule supersede the previous, or both apply?

I think that not all pip versions would merge these properly otherwise. But I haven't checked in a while. FWIW, this is the typical idiom people tend to follow.

webknjaz added maintenance Related to maintenance processes feature Request for a new feature ci Related to continuous integration tasks labels Dec 16, 2024

webknjaz requested review from WhyNotHugo, hugovk and chrysle December 16, 2024 04:58

webknjaz enabled auto-merge December 16, 2024 04:58

webknjaz added a commit to webknjaz/pip-tools that referenced this pull request Dec 16, 2024

Handle pip 23.1.2 @ Python 3.12 freeze exclusion

737a65c

Ref jazzband#2148 (comment)

webknjaz added a commit to webknjaz/pip-tools that referenced this pull request Dec 17, 2024

Handle pip 23.1.2 @ Python 3.12 freeze exclusion

f0d2f50

Ref jazzband#2148 (comment)

webknjaz force-pushed the maintenance/cpython-3.12 branch from 737a65c to adc094a Compare December 17, 2024 00:14

webknjaz and others added 3 commits December 17, 2024 01:15

Add Python 3.12 to the CI matrix

ed0f13f

Handle pip 23.1.2 @ Python 3.12 freeze exclusion

6a4e173

Ref jazzband#2148 (comment)

Bump lowest tested pip to v23.1

088299b

webknjaz force-pushed the maintenance/cpython-3.12 branch from adc094a to 088299b Compare December 17, 2024 00:15

WhyNotHugo approved these changes Dec 17, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Python 3.12 to the CI matrix #2148

Add Python 3.12 to the CI matrix #2148

webknjaz commented Dec 16, 2024

hugovk commented Dec 16, 2024

WhyNotHugo commented Dec 16, 2024

WhyNotHugo commented Dec 16, 2024

WhyNotHugo commented Dec 16, 2024 •

edited

Loading

WhyNotHugo commented Dec 16, 2024

macro1 commented Dec 16, 2024

webknjaz commented Dec 16, 2024

macro1 commented Dec 16, 2024

webknjaz commented Dec 16, 2024

webknjaz commented Dec 17, 2024

webknjaz commented Dec 17, 2024

WhyNotHugo left a comment

WhyNotHugo Dec 17, 2024

webknjaz Dec 17, 2024

Add Python 3.12 to the CI matrix #2148

Are you sure you want to change the base?

Add Python 3.12 to the CI matrix #2148

Conversation

webknjaz commented Dec 16, 2024

Contributor checklist

Maintainer checklist

hugovk commented Dec 16, 2024

WhyNotHugo commented Dec 16, 2024

WhyNotHugo commented Dec 16, 2024

WhyNotHugo commented Dec 16, 2024 • edited Loading

WhyNotHugo commented Dec 16, 2024

macro1 commented Dec 16, 2024

webknjaz commented Dec 16, 2024

macro1 commented Dec 16, 2024

webknjaz commented Dec 16, 2024

webknjaz commented Dec 17, 2024

webknjaz commented Dec 17, 2024

WhyNotHugo left a comment

Choose a reason for hiding this comment

WhyNotHugo Dec 17, 2024

Choose a reason for hiding this comment

webknjaz Dec 17, 2024

Choose a reason for hiding this comment

WhyNotHugo commented Dec 16, 2024 •

edited

Loading