Add ability to `run` any `PythonSourceField` #15849

thejcannon · 2022-06-16T18:10:26Z

(Minor user API change 😄 )

This PR encompasses a few changes which combined achieve the goal of the proposal doc.

It:

Adds support for run on a PythonSourceField. The implementation is 100% shifted from running a pex_binary
- A new field was added to the PythonSourceField targets: run_goal_use_sandbox
- Unfortunately, for consistency, this (including the new field) applies to _test_utils and _test targets as well 🤷‍♂️
Add a global flag --use-deprecated-pex-binary-run-semantics which toggles between the "new" and "old" behavior and must be set (will have the default changed in 2.14 and be removed in 2.15).
Special-cases the handling of an ambiguity between 2 targets if it is between a pex_binary and python_source. This allows us a path forward without breaking clients who (very very very often) use ./pants run path/to/file.py to refer to the pex_binary. Which target gets selected is based on --use-deprecated-pex-binary-run-semantics.
Removes run_in_sandbox field from pex_binary (this never made it to a stable release, so non-breaking)

[ci skip-rust]
[ci skip-build-wheels]

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

thejcannon

This has a little bit of time-sensitivity due to pulling pex_binary's run_in_sandbox field 😉

src/python/pants/backend/python/goals/run_pex_binary_integration_test.py

src/python/pants/backend/python/goals/run_python_source.py

thejcannon · 2022-06-16T18:14:41Z

src/python/pants/backend/python/goals/run_python_source.py

+        # @TODO: How should we make this customizable?
+        # `False` is backwards-compatible behavior right now
+        run_in_sandbox=False,
+        # Setting --venv is kosher because the PEX we create is just for the thirdparty deps.


Not just kosher, but desirable due to pex-tool/pex#1802. Happy to hear better comment wordsmithing.

No longer necessary if we upgrade PEX to a version which fixes pex-tool/pex#1802

thejcannon · 2022-06-16T18:15:11Z

src/python/pants/backend/python/goals/run_python_source_integration_test.py

+            python_sources(name="main_lib",
+                sources=["main.py"],
+                # Force-exclude any dep on bar.py, so the only way to consume it is via the dist.
+                dependencies=[":dist", "!:lib"],


This is slightly tweaked from the pex_binary case because we can't use transitive excludes.

src/python/pants/backend/python/target_types.py

src/python/pants/backend/python/util_rules/pex_environment.py

thejcannon · 2022-06-16T18:24:51Z

Can I/should I pull the plug on the in-repo pex_binarys that are just scripts? 😄

EDIT: I'll do this in follow-up PR. I want to be able to demonstrate the ambiguity easily.

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

thejcannon · 2022-06-17T01:35:54Z

Example of deprecations (while there is ambiguity and the pex_binary still exists):

joshuacannon@CEPHANDRIUS:~/work/pants$ ./pants run build-support/bin/generate_docs.py
20:28:54.63 [INFO] Initializing scheduler...
20:28:55.00 [INFO] Scheduler initialized.
20:28:55.27 [WARN] DEPRECATED: referring to a `pex_binary` by using the filename specified in `entry_point` is scheduled to be removed in version 2.14.0dev0.

In Pants 2.14, a `pex_binary` can no longer be referred to by the filename that the `entry_point` field uses.

This is due to a change in Pants 2.13, which allows you to use the `run` goal directly on a `python_source` target without requiring a `pex_binary`. As a consequence Pants removed the ability to refer to the `pex_binary` via its `entry_point`, as otherwise it would be ambiguous which target to use.

Note that because of this change you are able to remove any `pex_binary` targets you have declared just to support the `run` goal (usually these are developer scripts), as using `run` with the filename will have the equivalent behavior.

To fix this deprecation, use the `pex_binary`'s address.
20:28:55.30 [WARN] DEPRECATED: the option --pex-run-packaged-firstparty-code defaulting to false is scheduled to be removed in version 2.14.0dev0.

In Pants 2.14, running a `pex_binary` will actually package the PEX and run it, as if you ran `package` followed by executing the built PEX.

To fix this deprecation, explictly set `run_packaged_firstparty_code` in the `[pex]` section of `pants.toml`. Set it to `false` to use the "old" behavior. Set it to `true` to use the "new" behavior.

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano

Yay!!

Docs need to be updated for the Python package and run pages before merging. I'm gonna be OOO till Thursday, but lmk if you want an assist when I get back.

Our deprecation messages might want to suggest for people that they get rid of pex_binary targets they use to have. No need. And you can delete all of build-support/bin I think :)

src/python/pants/backend/python/util_rules/pex_environment.py

src/python/pants/backend/python/goals/run_python_source.py

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

thejcannon · 2022-06-18T00:22:39Z

Yay!!

Docs need to be updated for the Python package and run pages before merging. I'm gonna be OOO till Thursday, but lmk if you want an assist when I get back.

Our deprecation messages might want to suggest for people that they get rid of pex_binary targets they use to have. No need. And you can delete all of build-support/bin I think :)

I'll take a stab at the docs.

The deprecation message does make the suggestion:

Note that because of this change you are able to remove any pex_binary targets you have declared just to support the run goal (usually these are developer scripts), as using run on the python_source will have the equivalent behavior.

I'll dedicate a separate PR for deletion of our binary targets 😄

benjyw · 2022-06-21T15:48:36Z

| My vote is well-documented flag on run (and eventually repl)

An issue with that is that it's meaningless for compiled languages.

thejcannon · 2022-06-21T15:56:06Z

| My vote is well-documented flag on run (and eventually repl)

An issue with that is that it's meaningless for compiled languages.

That's the well documented part 😉 This is relevant for any language which can run on the loose files, so mostly scripting languages.

Maybe I just make it a field, and hope either the default covers most cases or its rare people will want to toggle between the two modes on the same target easily?

benjyw · 2022-06-21T21:03:28Z

Naming nit: --run-packaged-firstparty-code is ungainly. How about --package (So it's --run-as-package in global position or ./pants run --as-package in goal position.

benjyw · 2022-06-21T21:10:54Z

I'm still trying to figure out if I think this is intuitive to Winston. I think on balance yes? "Run a source file" seems to intuitively mean "run it and its deps directly from source" and "Run a package" seems to intuitively mean "build an executable and run it". I think the risks re --changed and --loop not doing the right thing are acceptable.

thejcannon · 2022-06-22T00:48:50Z

Naming nit: --run-packaged-firstparty-code is ungainly. How about --package (So it's --run-as-package in global position or ./pants run --as-package in goal position.

I like the idea, but the option is on the pex subsystem 😢

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

thejcannon · 2022-06-22T00:53:45Z

I think the risks re --changed and --loop not doing the right thing are acceptable.

They do the right thing, just on the metadata they have (which might not be right). It's a slight difference, but an important one IMO.

Because the use-case we're explicitly not considering isn't that it "doesnt work", but that run shouldn't be used to validate your metadata.

thejcannon · 2022-06-22T01:02:49Z

src/python/pants/backend/python/target_types.py

+        were packaged in a `pex_binary`. Additionally, it may be necessary if your sources depend
+        transitively on "generated" files which will be materialized in the sandbox in a source
+        root, but are not in-repo.


Ya know, we can technically inspect the dep tree for this and issue a warning if the field isn't explictly set or something similar 🤔

Not saying we should block this PR on it, but worth an issue filed once this is in.

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

thejcannon · 2022-06-22T01:10:29Z

Docs are added
Field has been added as well
No longer draft PR

src/python/pants/backend/python/target_types.py

…to runpython # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

thejcannon · 2022-06-22T13:50:51Z

Naming nit: --run-packaged-firstparty-code is ungainly. How about --package (So it's --run-as-package in global position or ./pants run --as-package in goal position.

I like the idea, but the option is on the pex subsystem cry

Also FWIW this will likely be set in pants.toml and will be deprecated/removed after 2 releases. So kinda like the use_old_style_macros (or whatever it was called)

thejcannon · 2022-06-22T14:09:55Z

Hmmm this just occurred to me, how should we handle restartable? Why is that a field and not a run flag?

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

run's counterpart to pantsbuild#15799. Support is added for pex_binary (which will be spiritually lifted to the python_source with pantsbuild#15849) [ci skip-rust] [ci skip-build-wheels]

`run`'s counterpart to #15799. Support is added for `pex_binary` (which will be spiritually lifted to the `python_source` with #15849) [ci skip-rust] [ci skip-build-wheels]

Eric-Arellano

Huzzah!

docs/markdown/Python/python-goals/python-run-goal.md

src/python/pants/backend/python/target_types.py

src/python/pants/engine/internals/specs_rules.py

src/python/pants/option/global_options.py

Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

…to runpython # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano

🎉

Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>

With #15849 the default for the `--tailor-pex-binary-targets` flag should naturally be `False`, so this begins the path to that. [ci skip-rust] [ci skip-build-wheels]

Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com> [ci skip-build-wheels]

…ld#15962) With pantsbuild#15849 the default for the `--tailor-pex-binary-targets` flag should naturally be `False`, so this begins the path to that. [ci skip-rust] [ci skip-build-wheels]

…16022) (Minor user API change 😄 ) This PR encompasses a few changes which combined achieve the goal of [the proposal doc](https://docs.google.com/document/d/1R9m-DtM8Y5UKx5iam5ipeK531tZR40_P7El4ucYq4Po). It: - Adds support for `run` on a `PythonSourceField`. The implementation is 100% shifted from `run`ning a `pex_binary` - A new field was added to the `PythonSourceField` targets: `run_goal_use_sandbox` - Unfortunately, for consistency, this (including the new field) applies to `_test_utils` and `_test` targets as well 🤷‍♂️ - Add a global flag `--use-deprecated-pex-binary-run-semantics` which toggles between the "new" and "old" behavior and must be set (will have the default changed in 2.14 and be removed in 2.15). - Special-cases the handling of an ambiguity between 2 targets if it is between a `pex_binary` and `python_source`. This allows us a path forward without breaking clients who (very very very often) use `./pants run path/to/file.py` to refer to the `pex_binary`. Which target gets selected is based on `--use-deprecated-pex-binary-run-semantics`. - Removes `run_in_sandbox` field from `pex_binary` (this never made it to a stable release, so non-breaking) [ci skip-rust] [ci skip-build-wheels]

…ick of #15962) (#16023) With #15849 the default for the `--tailor-pex-binary-targets` flag should naturally be `False`, so this begins the path to that. [ci skip-rust] [ci skip-build-wheels]

Namely #15849 and #15962 [ci skip-rust]

Run Python as first-class

4668af8

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

thejcannon added the category:new feature label Jun 16, 2022

thejcannon marked this pull request as draft June 16, 2022 18:10

thejcannon commented Jun 16, 2022

View reviewed changes

thejcannon mentioned this pull request Jun 16, 2022

Support running pex_binary targets directly on the in-repo sources. #15689

Merged

thejcannon added 2 commits June 16, 2022 20:32

blast of code

37d389d

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

add TODO

f519a50

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

use helper

10f7992

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano reviewed Jun 17, 2022

View reviewed changes

suggestions

2adea33

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

field

445e2eb

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

thejcannon marked this pull request as ready for review June 22, 2022 00:54

thejcannon commented Jun 22, 2022

View reviewed changes

thejcannon added 2 commits June 21, 2022 20:06

docs too!

4872e9b

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Merge branch 'main' into runpython

9a18162

thejcannon commented Jun 22, 2022

View reviewed changes

src/python/pants/backend/python/target_types.py Outdated Show resolved Hide resolved

thejcannon added 2 commits June 21, 2022 20:23

oops

77da92e

Merge branch 'runpython' of github.com-thejcannon:thejcannon/pants in…

4e7642d

…to runpython # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

docs

22637a8

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

thejcannon mentioned this pull request Jun 28, 2022

Deprecate not setting tailor_pex_binary_targets explictly #15962

Merged

thejcannon added 5 commits June 28, 2022 11:32

no timeout

701a8eb

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

fix tests

d23c9f6

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

bump to 400

f9037a6

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

fix test

041ff3c

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

actually fix test

f40a5b5

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

thejcannon mentioned this pull request Jun 28, 2022

Add --debug-adapter flag to run (Cherry-pick of #15829) #15988

Merged

Eric-Arellano approved these changes Jun 29, 2022

View reviewed changes

thejcannon and others added 5 commits June 29, 2022 19:08

Update docs/markdown/Python/python-goals/python-run-goal.md

a517edb

Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>

Update docs/markdown/Python/python-goals/python-run-goal.md

95c04db

Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>

Update docs/markdown/Python/python-goals/python-run-goal.md

79a5056

Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>

eric's

80b30f7

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Merge branch 'runpython' of github.com-thejcannon:thejcannon/pants in…

b6cb80c

…to runpython # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano approved these changes Jun 30, 2022

View reviewed changes

thejcannon merged commit 27628c3 into pantsbuild:main Jun 30, 2022

thejcannon deleted the runpython branch June 30, 2022 01:23

Eric-Arellano added a commit to Eric-Arellano/pants that referenced this pull request Jun 30, 2022

Add ability to run any PythonSourceField (pantsbuild#15849)

5320f5b

Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>

thejcannon removed the needs-cherrypick label Jun 30, 2022

thejcannon added a commit to thejcannon/pants that referenced this pull request Jul 1, 2022

Add ability to run any PythonSourceField (pantsbuild#15849)

da76a13

Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com> [ci skip-build-wheels]

thejcannon mentioned this pull request Jul 1, 2022

Deprecate not setting tailor_pex_binary_targets explictly (Cherry-pick of #15962) #16023

Merged

thejcannon mentioned this pull request Jul 1, 2022

Remove unnecessary pex_binary targets #16025

Merged

thejcannon added a commit that referenced this pull request Jul 5, 2022

Remove unnecessary pex_binary targets (#16025)

d59c230

Namely #15849 and #15962 [ci skip-rust]

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability to `run` any `PythonSourceField` #15849

Add ability to `run` any `PythonSourceField` #15849

thejcannon commented Jun 16, 2022 •

edited

Loading

thejcannon left a comment

thejcannon Jun 16, 2022

thejcannon Jun 21, 2022

thejcannon Jun 16, 2022

thejcannon commented Jun 16, 2022 •

edited

Loading

thejcannon commented Jun 17, 2022 •

edited

Loading

Eric-Arellano left a comment

thejcannon commented Jun 18, 2022

benjyw commented Jun 21, 2022

thejcannon commented Jun 21, 2022 •

edited

Loading

benjyw commented Jun 21, 2022

benjyw commented Jun 21, 2022

thejcannon commented Jun 22, 2022

thejcannon commented Jun 22, 2022

thejcannon Jun 22, 2022

thejcannon commented Jun 22, 2022

thejcannon commented Jun 22, 2022

thejcannon commented Jun 22, 2022

Eric-Arellano left a comment

Eric-Arellano left a comment

Add ability to run any PythonSourceField #15849

Add ability to run any PythonSourceField #15849

Conversation

thejcannon commented Jun 16, 2022 • edited Loading

thejcannon left a comment

Choose a reason for hiding this comment

thejcannon Jun 16, 2022

Choose a reason for hiding this comment

thejcannon Jun 21, 2022

Choose a reason for hiding this comment

thejcannon Jun 16, 2022

Choose a reason for hiding this comment

thejcannon commented Jun 16, 2022 • edited Loading

thejcannon commented Jun 17, 2022 • edited Loading

Eric-Arellano left a comment

Choose a reason for hiding this comment

thejcannon commented Jun 18, 2022

benjyw commented Jun 21, 2022

thejcannon commented Jun 21, 2022 • edited Loading

benjyw commented Jun 21, 2022

benjyw commented Jun 21, 2022

thejcannon commented Jun 22, 2022

thejcannon commented Jun 22, 2022

thejcannon Jun 22, 2022

Choose a reason for hiding this comment

thejcannon commented Jun 22, 2022

thejcannon commented Jun 22, 2022

thejcannon commented Jun 22, 2022

Eric-Arellano left a comment

Choose a reason for hiding this comment

Eric-Arellano left a comment

Choose a reason for hiding this comment

Add ability to `run` any `PythonSourceField` #15849

Add ability to `run` any `PythonSourceField` #15849

thejcannon commented Jun 16, 2022 •

edited

Loading

thejcannon commented Jun 16, 2022 •

edited

Loading

thejcannon commented Jun 17, 2022 •

edited

Loading

thejcannon commented Jun 21, 2022 •

edited

Loading