Add hidden support for Python 3.13. #2318

jsirois · 2024-01-07T21:46:32Z

This uses a patched version of Pip as well as a new build system to
affect spinning up CI for the Python 3.13 release in October ahead of
time without leaking details of this testing-only setup to the
production distribution.

This uses a patched version of Pip as well as a new build system to affect spinning up CI for the Python 3.13 release in October ahead of time without leaking details of this testing-only setup to the production distribution.

jsirois · 2024-01-08T09:04:09Z

pex/bin/pex.py

@@ -958,15 +958,21 @@ def _compatible_with_current_platform(interpreter, platforms):
 def main(args=None):
    args = args[:] if args else sys.argv[1:]
    args = [transform_legacy_arg(arg) for arg in args]
-    parser = configure_clp()
+
+    parser = catch(configure_clp)


Testing out the hiddenness of pep24 from the production CLI --pip-version support revealed these two spots where raised exceptions led to noisy backtraces instead of clean bad value messages.

jsirois · 2024-01-08T09:04:13Z

pex/build_system/pep_517.py

@@ -183,8 +180,8 @@ def _invoke_build_hook(
            args=args,
            env=build_system.env,
            cwd=project_directory,
-            stdout=stdout if stdout is not None else sys.stderr.fileno(),
-            stderr=stderr if stderr is not None else sys.stderr.fileno(),
+            stdout=subprocess.PIPE,


Working through an issue with non-reproducible local project locks when fiddling with the proper setup.{cfg,py} configuration (see various comments in those files) I became keenly aware of all the sdist build output leaked to stdout / stderr here. No callers passed stdout / stderr and always piping does the right thing since Job knows to slurp the streams up and only report output when the Job fails, which is exactly what you want for a non-noisy CLI that is only noisy when it needs to be to aid debug of a build failure.

jsirois · 2024-01-08T09:04:17Z

pex/pip/version.py

                )
+                if not applicable_versions:
+                    raise ValueError(


When working through --pip-version pip24 hiddenness, I was getting a semi-cryptic error message from max getting an empty iterable. Erroring was correct, but now the message is more clear in this case.

jsirois · 2024-01-08T09:04:29Z

tests/bin/test_sh_boot.py

-            "pypy2.7",
-        ]
+
+    supported_versions = sorted(


The switch to the dynamic list is necessitated by the new support for mutating Requires-Python metadata for the Pex dist under CI for Python 3.13. In that case the list needs to include python3.13 and pypy3.13 whereas in the nominal production cases it does not.

jsirois · 2024-01-08T09:11:23Z

setup.py

+        },
+        # This supports expanding the supported Python range via the _PEX_REQUIRES_PYTHON env var
+        # for testing unreleased Pythons.
+        python_requires=os.environ.get("_PEX_REQUIRES_PYTHON"),


This is what necessitated the build system change from flit to setuptools. AFAICT setuptools provides the only pep-517 build backend where Requires-Dist metadata can be easily altered which is what I needed to be able to do to keep Python 3.13 support officially turned off while testing against Python 3.13 alpha, beta and rc releases from now through the end of September as the 3.13 release works towards final.

And, although this was thoroughly vetted by tests - which caught issues with the new packaging relentlessly, the eyeball looks like so:

$ tox -epackage -- --additional-format sdist --additional-format wheel ROOT: will run in automatically provisioned tox, host /home/jsirois/bin/tox.venv/bin/python3.11 is missing [requires (has)]: tox<4 (4.11.4), virtualenv<20.16 (20.25.0) ROOT: install_deps> python -I -m pip install 'tox<4' 'tox>=3.25.1' 'virtualenv<20.16' ROOT: provision> .tox/.tox/bin/python -m tox -epackage -- --additional-format sdist --additional-format wheel package create: /home/jsirois/dev/pantsbuild/jsirois-pex/.tox/package package installdeps: build package installed: build==1.0.3,packaging==23.2,pyproject_hooks==1.0.0 package run-test-pre: PYTHONHASHSEED='2295051996' package run-test: commands[0] | python scripts/package.py --additional-format sdist --additional-format wheel Building Pex PEX to `dist/pex` ... Built Pex PEX @ v2.1.156-1-g6a6e403a: sha256: 8eeaef638aa9db10766f7805c231687b36a27dff84031a0721c336cdb451b872 size: 3668094 Building additional distribution formats to `dist`: 1.) sdist, 2.) wheel ... Built: dist/pex-2.1.156-py2.py3-none-any.whl dist/pex-2.1.156.tar.gz ____________________________________________________________________________________________________ summary _____________________________________________________________________________________________________ package: commands succeeded congratulations :)

sdist:
$ diff -u1 <(tar -tzf ~/downloads/pex-2.1.156.tar.gz | sort) <(tar -tzf dist/pex-2.1.156.tar.gz | grep -v -E "/$" | sort) --- /dev/fd/63 2024-01-08 01:41:03.688560680 -0800 +++ /dev/fd/62 2024-01-08 01:41:03.688560680 -0800 @@ -1,11 +1,6 @@ -pex-2.1.156/.github/FUNDING.yml -pex-2.1.156/.github/workflows/ci.yml -pex-2.1.156/.github/workflows/release.yml -pex-2.1.156/.gitignore -pex-2.1.156/.readthedocs.yaml pex-2.1.156/CHANGES.md pex-2.1.156/LICENSE +pex-2.1.156/MANIFEST.in pex-2.1.156/PKG-INFO pex-2.1.156/README.rst -pex-2.1.156/RELEASE.rst pex-2.1.156/docker/base/Dockerfile @@ -278,2 +273,3 @@ pex-2.1.156/pex/vendor/_vendored/pip/.prefix/bin/pip3.8 +pex-2.1.156/pex/vendor/_vendored/pip/__init__.py pex-2.1.156/pex/vendor/_vendored/pip/pip-20.3.4.dist-info/INSTALLER @@ -801,2 +797,4 @@ pex-2.1.156/scripts/typecheck.py +pex-2.1.156/setup.cfg +pex-2.1.156/setup.py pex-2.1.156/testing/__init__.py

wheel:
$ diff -u1 <(zipinfo -1 ~/downloads/pex-2.1.156-py2.py3-none-any.whl | sort) <(zipinfo -1 dist/pex-2.1.156-py2.py3-none-any.whl | grep -v -E "/$" | sort) --- /dev/fd/63 2024-01-08 01:42:12.688549984 -0800 +++ /dev/fd/62 2024-01-08 01:42:12.688549984 -0800 @@ -5,2 +5,3 @@ pex-2.1.156.dist-info/entry_points.txt +pex-2.1.156.dist-info/top_level.txt pex/__init__.py @@ -256,2 +257,3 @@ pex/vendor/_vendored/pip/.prefix/bin/pip3.8 +pex/vendor/_vendored/pip/__init__.py pex/vendor/_vendored/pip/pip-20.3.4.dist-info/INSTALLER

It turns out Virtualenv.create_atomic was used in a few other spots.

Sanitize PyPy testing / interrogation along the way by centralizing in PythonInterpreter / PythonIdentity.

Also add a note about why we always add this file, even when Pex itself does not need it.

jsirois · 2024-01-08T13:00:02Z

Ok, this is finally good to go for review.

benjyw

Sounds like this was a little painful, with the flit->setuptools switch and the pypy symlinks issue... All LGTM though!

jsirois · 2024-01-08T22:41:16Z

Sounds like this was a little painful, with the flit->setuptools switch and the pypy symlinks issue... All LGTM though!

A bit, but flit->setuptools was a 1 time thing to allow this style of anticipating Python releases going forward and that seems worth it. I'd much rather not be caught back on my heels any longer.

Thanks for taking a look.

Add hidden support for Python 3.13.

6a6e403

This uses a patched version of Pip as well as a new build system to affect spinning up CI for the Python 3.13 release in October ahead of time without leaking details of this testing-only setup to the production distribution.

jsirois force-pushed the 3.13/prep branch from 1d0849b to 6a6e403 Compare January 8, 2024 08:33

jsirois commented Jan 8, 2024

View reviewed changes

jsirois requested review from benjyw, huonw and hrfuller January 8, 2024 09:05

jsirois commented Jan 8, 2024

View reviewed changes

jsirois added 4 commits January 8, 2024 02:30

Centralize PyPy 7.3.14 venv workaround.

7e36f42

It turns out Virtualenv.create_atomic was used in a few other spots.

Skip the relocatable_venv test for PyPy 7.3.14.

e9d0aa4

Sanitize PyPy testing / interrogation along the way by centralizing in PythonInterpreter / PythonIdentity.

Centralize ~un-needed __init__.py hack better.

b3899bd

Also add a note about why we always add this file, even when Pex itself does not need it.

Re-run tox -evendor.

3b8d7f7

benjyw approved these changes Jan 8, 2024

View reviewed changes

jsirois merged commit 039d9f6 into pex-tool:main Jan 8, 2024
26 checks passed

jsirois deleted the 3.13/prep branch January 8, 2024 22:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add hidden support for Python 3.13. #2318

Add hidden support for Python 3.13. #2318

jsirois commented Jan 7, 2024

jsirois Jan 8, 2024 •

edited

Loading

jsirois Jan 8, 2024 •

edited

Loading

jsirois Jan 8, 2024

jsirois Jan 8, 2024

jsirois Jan 8, 2024

jsirois Jan 8, 2024

jsirois commented Jan 8, 2024

benjyw left a comment

jsirois commented Jan 8, 2024

Add hidden support for Python 3.13. #2318

Add hidden support for Python 3.13. #2318

Conversation

jsirois commented Jan 7, 2024

jsirois Jan 8, 2024 • edited Loading

Choose a reason for hiding this comment

jsirois Jan 8, 2024 • edited Loading

Choose a reason for hiding this comment

jsirois Jan 8, 2024

Choose a reason for hiding this comment

jsirois Jan 8, 2024

Choose a reason for hiding this comment

jsirois Jan 8, 2024

Choose a reason for hiding this comment

jsirois Jan 8, 2024

Choose a reason for hiding this comment

jsirois commented Jan 8, 2024

benjyw left a comment

Choose a reason for hiding this comment

jsirois commented Jan 8, 2024

jsirois Jan 8, 2024 •

edited

Loading

jsirois Jan 8, 2024 •

edited

Loading