Add the ability for Pants to provide Python via a union (with a pyenv impl) #18352

thejcannon · 2023-02-24T12:53:19Z

Adding the ability for an optional union (PythonProvider) implementer to return a PythonExecutable (newly outfitted with append_only_caches) so that Python for user code can be supplied completely by rule-code. Additionally added a pyenv union implementation.

The dirty details are:

Python is essentially NOT relocatable (hats off to IndyGreg). Python itself, as well as ecosystem tools, are baked with assumptions about absolute paths.
Python installation is not atomic. That means we do our work behind a lock. We run the bash script on every session, but it should be fast as it only does the work if the target python doesnt exist.

thejcannon · 2023-02-24T12:54:13Z

src/python/pants/init/plugin_resolver.py

@@ -80,7 +80,6 @@ async def resolve_plugins(
            internal_only=True,
            python=python,
            requirements=requirements,
-            interpreter_constraints=request.interpreter_constraints,


This was None and thats a no-no. However there's no type hint for this param so mypy isn't complaining.

thejcannon · 2023-02-24T12:57:40Z

We still require some kind of python for executing non-user code. Namely:

pex to ask about ICs
gunzipping, and a few other places

I can tackle that in another PR.

kaos

This is huge. Amazing stuff!

src/python/pants/backend/python/providers/pyenv/register.py

src/python/pants/backend/python/providers/pyenv/rules.py

src/python/pants/backend/python/providers/pyenv/rules_integration_test.py

kaos · 2023-02-24T13:47:16Z

That means we do our work in a tempdir and symlink the result, on every session.

how much needs to be redone though? would be nice if we could avoid reinstalling Python on every restart of pantsd (that's what session means, right? or is it every pants run? cause that sounds like it would be a little bit of a buzzkill)

Co-authored-by: Andreas Stenius <git@astekk.se>

thejcannon · 2023-02-24T14:13:13Z

Oh sorry bad phrasing. We run the process on every run, but bail if the python already exists. So we should only install it once. I'll fix the phrasing.

src/python/pants/backend/python/providers/pyenv/rules.py

…o pyenvpython

kaos

🚀

thejcannon · 2023-02-24T16:27:12Z

Another open question: Should we start using this in this repo? There's two places it could be useful:

For running the repo itself ./pants run ...
For our tests that are multi-interpreter. Right now we skip, but with enabling the pyenv plugin in the test, we wouldn't have to (although there'd be timeout issues 😬 )

benjyw · 2023-02-24T16:33:09Z

Awesome! Users will love it. Reviewing now.

benjyw

Potentially massive!!

src/python/pants/backend/python/providers/pyenv/rules.py

benjyw · 2023-02-24T16:25:11Z

src/python/pants/backend/python/providers/pyenv/rules.py

+            input_digest=pyenv.digest,
+            description=f"Choose specific version for Python {python_to_use}",
+            env={"PATH": env_vars.get("PATH", "")},
+            # Caching the result is OK, since if the user really needs a different patch,


I concur. And this will only matter when new Python versions are released, which is... not often.

src/python/pants/backend/python/providers/pyenv/rules.py

thejcannon · 2023-02-24T17:08:41Z

Oh one alternative to the current bash script is a Python script which uses locks and subprocesses. If we fix the issue highlighted above regarding non-user-code usage of Python (ideally with pyoxy or Python Build Standalone) a Python script to install Python is possible.

src/python/pants/backend/python/providers/pyenv/rules.py

thejcannon · 2023-03-02T21:20:33Z

Okie dokie folks, hot changes coming in:

Switched the locking and installation to Python to be cross-platform(-ish). The sentinel file-checking is still in bash because it's the hot-path and bash go vroom.
Added a bunch of plumbing to make it so users can out-of-band install versions the same way we do in-band
- We now have a new (internal) target type which is runnable representing running pyenv install.
- And a synthetic target is constructed at the root named pyenv-install
- This should be used for people to do more complicated things, like installing an optimized version of Python
Added more help docs

All-in-all I'm pretty pleased with it. Although I wish we had a convention/namespace surrounding synthetic targets location/names.

kaos

Awesome 🚀

kaos · 2023-03-03T03:06:37Z

src/python/pants/backend/python/providers/pyenv/target_types.py

+class PyenvInstall(Target):
+    alias = "_pyenv_install"
+    help = "<internal target>"
+    core_fields = (PyenvInstallSentinelField,)


this is a bit boiler-platey. not suggesting to do anything about it here and now, but if we find this pattern useful, it could make sense to streamline it a bit :)

src/python/pants/backend/python/providers/pyenv/rules.py

stuhood

Nice! Thanks.

As to Docker: this should work fine, since named caches have been used properly AFAICT.

stuhood · 2023-03-03T17:55:23Z

src/python/pants/backend/python/providers/pyenv/rules.py

+        A subsystem for pyenv-provided-Python (https://github.com/pyenv/pyenv).
+
+        Enabling this subsystem will switch Pants from trying to find an appropriate Python on your
+        system to using pyenv to install the correct Python(s).


This intro doesn't really make it clear that this is a Pants-managed pyenv, rather than a pyenv install on your system (which we do have support for via interpreter_search_path=["<PYENV>"]).

...Which also highlights that some doc changes might be good for this, so that those two strategies reference one another.

Ahh yeah, docs always elude me. I'll see what I can do

I'm gonna punt on docs changes in this PR and open another one for that.

stuhood · 2023-03-03T17:57:21Z

src/python/pants/backend/python/providers/pyenv/rules.py

+        If you wish to customize the pyenv installation of a Python, a synthetic target is exposed
+        at the root of your repo which is runnable. This target can be run with the relevant
+        environment variables set to enable optimizations. You will need to wipe the specific
+        version directory if Python was already installed. Example:
+
+            sudo rm -rf <named_caches_dir>/pyenv/versions/<specific_version>
+            # env vars from https://github.com/pyenv/pyenv/blob/master/plugins/python-build/README.md#building-for-maximum-performance
+            PYTHON_CONFIGURE_OPTS='--enable-optimizations --with-lto' {bin_name()} run :pants-pyenv-install -- 3.10


Hm... this is very magical. It feels like something that should be provided via subsystem options instead, otherwise the build could be very non-hermetic with folks using interpreters which were customized in different ways.

If we expose this through options it's the same end result. Customized python builds...

And this way is the only "middle ground" I could come up with to make having some (uncacheable) build step take 30 minutes. Make the user "own" running that.

That might be the end result (if people end up putting things in a .pantsrc file), but the happy path would be to put the option in pants.toml instead, which would cause the option to be applied by default.

Well maybe I'm just viewing this from my perspective, but I know my engineers wouldn't wait for a 30 minute process. And they'd bug me constantly about it. And kill it. Meaning they'd have to restart.

So I can't put it in pants.toml.

And putting it there feels weird because if you change the option we don't invalidate existing pythons either. (Same for .pants.rc)

All in all, the end goal is providing Python. That's the 80% case.

For the 20%, let's give them a back door. But it doesn't have to be of the same elk as the front door 😉

So I can't put it in pants.toml.

This is true for optimization flags, but not true for the other flags you might want to pass to a pyenv install: particularly CONFIGURE_OPTS.

And putting it there feels weird because if you change the option we don't invalidate existing pythons either. (Same for .pants.rc)

We could? Fingerprinting the options could be done by embedding them in the installation script, and taking its digest, for example.

Sure anything is possible. Getting back to it though, the build is already non-hermetic and we still need a way to allow users to install Python out-of-band or suffer a 30-minute build step.

OK I think I've decided to have this functionality be in its own sub-plugin of the pyenv provider plugin. So it's "off" by default.

src/python/pants/backend/python/providers/pyenv/rules.py

src/python/pants/core/goals/run.py

thejcannon · 2023-03-05T18:47:15Z

Marked as experimental so we can merge with a few conversations ongoing.

This PR decouples the Python Pants uses for its own nefarious purposes (like running PEX or `gunzip`) from the user search paths by either using `sys.executable` locally or downloading and using Python Build Standalone in a Docker environment. Additionally when making this change, the Python/pex code was refactored so that we always use this Python to run pex, with Python either being chosen by pex, or by using `PEX_PYTHON` env var at runtime. I think this is a nice cleanup of the handshake between `CompletePexEnvironment.create_argv` and `CompletePexEnvironment.environment_dict` used to have. On a side note, this (with `scie-pants` and #18352) this marks a complete divorce from any Python on the user system (or lack thereof) and running Python code.

This PR decouples the Python Pants uses for its own nefarious purposes (like running PEX or `gunzip`) from the user search paths by either using `sys.executable` locally or downloading and using Python Build Standalone in a Docker environment. Additionally when making this change, the Python/pex code was refactored so that we always use this Python to run pex, with Python either being chosen by pex, or by using `PEX_PYTHON` env var at runtime. I think this is a nice cleanup of the handshake between `CompletePexEnvironment.create_argv` and `CompletePexEnvironment.environment_dict` used to have. On a side note, this (with `scie-pants` and pantsbuild#18352) this marks a complete divorce from any Python on the user system (or lack thereof) and running Python code.

This PR decouples the Python Pants uses for its own nefarious purposes (like running PEX or `gunzip`) from the user search paths by either using `sys.executable` locally or downloading and using Python Build Standalone in a Docker environment. Additionally when making this change, the Python/pex code was refactored so that we always use this Python to run pex, with Python either being chosen by pex, or by using `PEX_PYTHON` env var at runtime. I think this is a nice cleanup of the handshake between `CompletePexEnvironment.create_argv` and `CompletePexEnvironment.environment_dict` used to have. On a side note, this (with `scie-pants` and #18352) this marks a complete divorce from any Python on the user system (or lack thereof) and running Python code.

oh hey it works

b6f394d

thejcannon added the category:new feature label Feb 24, 2023

thejcannon requested review from stuhood, kaos, jsirois, benjyw and Eric-Arellano February 24, 2023 12:53

tweak

e5a3429

thejcannon commented Feb 24, 2023

View reviewed changes

kaos reviewed Feb 24, 2023

View reviewed changes

Update src/python/pants/backend/python/providers/pyenv/register.py

08d1cad

Co-authored-by: Andreas Stenius <git@astekk.se>

jsirois reviewed Feb 24, 2023

View reviewed changes

src/python/pants/backend/python/providers/pyenv/rules.py Outdated Show resolved Hide resolved

thejcannon added 3 commits February 24, 2023 09:07

review comments

5e53821

Merge branch 'pyenvpython' of https://github.com/thejcannon/pants int…

ee1b586

…o pyenvpython

version

4a94869

kaos approved these changes Feb 24, 2023

View reviewed changes

fix test

39b00a0

benjyw approved these changes Feb 24, 2023

View reviewed changes

review

85f9dcc

benjyw reviewed Feb 24, 2023

View reviewed changes

src/python/pants/backend/python/providers/pyenv/rules.py Outdated Show resolved Hide resolved

one mroe ascii

b53c79e

thejcannon commented Feb 24, 2023

View reviewed changes

src/python/pants/backend/python/providers/pyenv/rules.py Outdated Show resolved Hide resolved

thejcannon added 2 commits February 24, 2023 13:59

Merge branch 'main' into pyenvpython

c332a67

probably fix test

26a85b0

ok here we go

4f9f7ad

jsirois reviewed Mar 2, 2023

View reviewed changes

src/python/pants/backend/python/providers/pyenv/rules.py Outdated Show resolved Hide resolved

booyah

ca50f77

thejcannon added 2 commits March 2, 2023 15:40

always an oopsie

6a795cd

gotta register it

04c0186

kaos approved these changes Mar 3, 2023

View reviewed changes

thejcannon added 4 commits March 3, 2023 08:02

isort

9e3ee5d

Merge branch 'main' into pyenvpython

8d4cbba

pants-

17c6aeb

1 fixes

2355236

stuhood approved these changes Mar 3, 2023

View reviewed changes

all good now

4404347

thejcannon commented Mar 3, 2023

View reviewed changes

src/python/pants/core/goals/run.py Outdated Show resolved Hide resolved

thejcannon added 4 commits March 4, 2023 19:35

or {}

2306193

fmt

09a8247

link

49a1f21

Mark experimental

dca39f7

thejcannon added 2 commits March 5, 2023 13:07

BUILD file

08f4737

dumb spacing

a1be452

thejcannon merged commit 7398971 into pantsbuild:main Mar 5, 2023

This was referenced Mar 6, 2023

Document the new Python provider union, and the pyenv implementation #18426

Open

Always provide Python-for-Pants-scripts #18433

Merged

thejcannon mentioned this pull request Mar 14, 2023

Always provide Python-for-Pants-scripts (Cherry-pick of #18433) #18495

Merged

stuhood mentioned this pull request Mar 15, 2023

Move the pants-pyenv-install target to a sub-plugin #18499

Merged

benjyw mentioned this pull request Sep 22, 2023

Read Interpreter Constraints from .python-version pyenv file #18730

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the ability for Pants to provide Python via a union (with a pyenv impl) #18352

Add the ability for Pants to provide Python via a union (with a pyenv impl) #18352

thejcannon commented Feb 24, 2023 •

edited

Loading

thejcannon Feb 24, 2023

thejcannon commented Feb 24, 2023

kaos left a comment

kaos commented Feb 24, 2023

thejcannon commented Feb 24, 2023

kaos left a comment

thejcannon commented Feb 24, 2023

benjyw commented Feb 24, 2023

benjyw left a comment

benjyw Feb 24, 2023

thejcannon commented Feb 24, 2023

thejcannon commented Mar 2, 2023 •

edited

Loading

kaos left a comment

kaos Mar 3, 2023

stuhood left a comment

stuhood Mar 3, 2023 •

edited

Loading

stuhood Mar 3, 2023

thejcannon Mar 3, 2023

thejcannon Mar 5, 2023

stuhood Mar 3, 2023

thejcannon Mar 3, 2023

stuhood Mar 3, 2023

thejcannon Mar 3, 2023

thejcannon Mar 3, 2023

stuhood Mar 3, 2023

thejcannon Mar 3, 2023

thejcannon Mar 6, 2023

thejcannon commented Mar 5, 2023

Add the ability for Pants to provide Python via a union (with a pyenv impl) #18352

Add the ability for Pants to provide Python via a union (with a pyenv impl) #18352

Conversation

thejcannon commented Feb 24, 2023 • edited Loading

Choose a reason for hiding this comment

thejcannon commented Feb 24, 2023

kaos left a comment

Choose a reason for hiding this comment

kaos commented Feb 24, 2023

thejcannon commented Feb 24, 2023

kaos left a comment

Choose a reason for hiding this comment

thejcannon commented Feb 24, 2023

benjyw commented Feb 24, 2023

benjyw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thejcannon commented Feb 24, 2023

thejcannon commented Mar 2, 2023 • edited Loading

kaos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood left a comment

Choose a reason for hiding this comment

stuhood Mar 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thejcannon commented Mar 5, 2023

thejcannon commented Feb 24, 2023 •

edited

Loading

thejcannon commented Mar 2, 2023 •

edited

Loading

stuhood Mar 3, 2023 •

edited

Loading