[internal] Add `experimental_resolve` field to `pex_binary` #12734

Eric-Arellano · 2021-09-01T23:00:20Z

Part of #11165 and builds off of #12703.

Rather than having a single option [python-setup].experimental_lockfile, users set [python-setup].experimental_resolves_to_lockfiles to define 0-n "named resolves" that associate a lockfile with a name:

[python-setup]
experimental_resolves_to_lockfiles = { lock1 = "lock1.txt", lock2 = "lock2.txt" }

Then, individual pex_binary targets can specify which resolve to use:

pex_binary(name="reversion", entry_point="reversion.py", experimental_resolve="lock1")

In a followup, we'll add a mechanism to set the default resolve.

Users can generate that lockfile with ./pants generate-lockfiles (all resolves) or ./pants generate-lockfiles --resolve=<name>:

❯ ./pants generate-lockfiles --resolve=lock1 --resolve=lock2
15:55:56.60 [INFO] Completed: Generate lockfile for lock1
15:55:56.61 [INFO] Completed: Generate lockfile for lock2
15:55:57.02 [INFO] Wrote lockfile for the resolve `lock1` to lock1.txt
15:55:57.02 [INFO] Wrote lockfile for the resolve `lock2` to lock2.txt

Then, it will be consumed with ./pants package and ./pants run. Pants will extract the proper subset from that lockfile, meaning that the lockfile can safely be a superset of what is used for the particular build.

❯ ./pants package build-support/bin:
...
15:56:33.87 [INFO] Completed: Installing lock1.txt for the resolve `lock1`
15:56:34.39 [INFO] Completed: Installing lock2.txt for the resolve `lock2`
15:56:34.48 [INFO] Completed: Extracting 1 requirement to build build-support.bin/generate_user_list.pex from lock1_lockfile.pex: pystache==0.5.4
...

If the lockfile is incompatible, we will (soon) warn or error with instructions to either use a new resolve or regenerate the lockfile.

In followups, this field will be hooked up to other targets like python_awslambda and python_tests.

We will likely also add a new field compatible_resolves to python_library, per #12714, which is a list of resolves. "Root targets" like python_tests and pex_binary will validate that all their dependencies are compatible. When you operate directly on a python_library target, like running MyPy on it, we will choose any of the possible resolves. You will be able to set your own default for this field.

[ci skip-rust]
[ci skip-build-wheels]

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano · 2021-09-01T23:03:59Z

src/python/pants/backend/python/target_types.py

+            name_description = "names"
+        super().__init__(
+            f"Unrecognized resolve {name_description} from {description_of_origin}: "
+            f"{unrecognized_str}\n\nAll valid resolve names: {sorted(all_valid_names)}"


NB: the all_valid_names will be different when you're using the resolve field vs generate-lockfiles --resolve. In the latter case, it includes tool lockfiles. This is accurate: it's an error to use a tool resolve for user code.

But it could be confusing. I wonder if this error message should better explain this nuance? Something like:

All valid resolve names (from [python-setup].experimental_resolves_to_lockfiles):

vs

All valid resolve names (from [python-setup].experimental_resolves_to_lockfiles and all activated Python tools):

Are tool resolves and user resolves conceptually in the same namespace?
Is the black tool resolve always named black so that all you have to do to replace the built-in lockfile is add an entry for black in [python-setup].experimental_resolves_to_lockfiles? Or would I add a user resolve (of any random name like my_black_lockfile) and set [black].resolve = my_black_lockfile?

In other words, how ambiguous might this be when a tool resolve is included in this list?

A tool resolve is always its option scope, so black, pytest, mypy_protobuf. The lockfile path is set via [black].lockfile and so on, rather than [python-setup].experimental_resolves_to_lockfiles which is only for your own code.

When you run ./pants generate-lockfiles, we eagerly validate that your own resolve names do not conflict with any tool resolves.

So the only time the two types of resolves should really interact is when you run ./pants generate-lockfiles --resolve=black --resolve=my-custom-name, which needs to be unambiguous. It also is currently an error if you set pex_binary(resolve="black") as that resolve needs to not be from a tool.

Does that make sense?

Eric-Arellano · 2021-09-01T23:06:53Z

src/python/pants/backend/python/target_types.py

+                description_of_origin=f"the field `resolve` in the target {self.address}",
+            )
+
+    def resolve_and_lockfile(self, python_setup: PythonSetup) -> tuple[str, str] | None:


This modeling took a lot of iterations. Originally I only had call sites pass the resolve name to PexFromTargetsRequest, and it would do the lookup for the corresponding lockfile from [python-setup]. This was great for boilerplate, but I think it's crucial the error message for unrecognized resolve names explains the origin of the error. We need to preserve the Address of the offending target.

It was less awkward imo to handle this validation here and having the callers pass the resolve_and_lockfile, rather than passing resolve_and_description_of_origin

Eric-Arellano · 2021-09-01T23:07:41Z

src/python/pants/backend/python/util_rules/pex_from_targets.py

@@ -246,11 +253,33 @@ async def pex_from_targets(request: PexFromTargetsRequest, python_setup: PythonS
                "`[python-setup].resolve_all_constraints` is enabled, so "
                "`[python-setup].requirement_constraints` must also be set."
            )
+        elif request.resolve_and_lockfile:


Note that, for now, if the field is not set, we fall back to [python-setup].experimental_lockfile. That option will be going away soon.

Although we are pausing the lockfile work in Pants until after PEX's lockfile support is ready to consume, I do think that landing this and following up with one more to actually delete experimental_lockfile would be a good place to rest on this.

I don't know how feasible that will be: pantsbuild/pants must ignore some targets from testprojects/ when resolving. We would need an opt-out mechanism for those targets (a fake lockfile won't work because they're invalid requirements).

I'd love to finish this part of the project - I really don't think it's much work. But trying to keep it scoped.

…ckfile

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

benjyw · 2021-09-02T04:55:32Z

Quick naming bikeshed: I think experimental_resolves is fine as a name, we don't need to be quite so literal about the structure of the type. People are going to have to read the docs here anyway.

Eric-Arellano · 2021-09-02T05:05:06Z

@stuhood had the same suggestion in #12703. My response (updated a bit):

Are you okay with waiting a little longer for this rename? #12742 makes me think we want >1 resolve related option, e.g. default_resolve to be a dedicated option. I'm open to renaming but would like to keep this for now while iterating. It helps my mental model.

Is that okay?

benjyw · 2021-09-02T05:18:25Z

As long as we rename before this is released, sure.

stuhood

Thanks! Awesome to see it come together.

stuhood · 2021-09-07T22:12:24Z

src/python/pants/backend/python/goals/lockfile.py

+@rule
+async def setup_user_lockfile_requests(
+    requested: _SpecifiedUserResolves, python_setup: PythonSetup
+) -> _UserLockfileRequests:
+    # First, associate all resolves with their consumers.
+    all_build_targets = await Get(UnexpandedTargets, AddressSpecs([DescendantAddresses("")]))
+    resolves_to_roots = defaultdict(list)
+    for tgt in all_build_targets:
+        if not tgt.has_field(PythonResolveField):
+            continue
+        tgt[PythonResolveField].validate(python_setup)
+        resolve = tgt[PythonResolveField].value
+        if resolve is None:
+            continue
+        resolves_to_roots[resolve].append(tgt.address)
+
+    # Expand the resolves for all specified.
+    transitive_targets_per_resolve = await MultiGet(
+        Get(TransitiveTargets, TransitiveTargetsRequest(resolves_to_roots[resolve]))
+        for resolve in requested
+    )


It seems like it would probably be simpler for this rule to execute per-resolve, rather than on a batch of resolves. The only thing that is saved by batching them is making a single pass to filter targets by resolve, but that should be cheap/linear time.

Eh we generally always encourage people to use MultiGet, code like this is common. We could add a helper rule for an individual resolve if necessary, but that adds new boilerplate.

I think this is fine for now and we can refactor need be.

stuhood · 2021-09-07T22:20:23Z

src/python/pants/backend/python/util_rules/pex_from_targets.py

        elif python_setup.lockfile:
            resolved_dists = await Get(
                ResolvedDistributions,
                PexRequest(
-                    description=f"Resolving {python_setup.lockfile}",
+                    description=f"Installing {python_setup.lockfile}",


"Installing" sounds like a sideeffect to me, but I guess that it is overloaded between "building a wheel" and "installing something somewhere".

@benjyw thoughts? I'm not fully happy with any of the terms.

stuhood · 2021-09-07T22:21:42Z

src/python/pants/backend/python/util_rules/pex_from_targets.py

@@ -246,11 +253,33 @@ async def pex_from_targets(request: PexFromTargetsRequest, python_setup: PythonS
                "`[python-setup].resolve_all_constraints` is enabled, so "
                "`[python-setup].requirement_constraints` must also be set."
            )
+        elif request.resolve_and_lockfile:


Although we are pausing the lockfile work in Pants until after PEX's lockfile support is ready to consume, I do think that landing this and following up with one more to actually delete experimental_lockfile would be a good place to rest on this.

As explained in #12734, this allows you to set the resolve when running a specific test target. The resolve will not yet be consumed when running Pylint and MyPy on the `python_tests` code, which is blocked by implementing #12714. [ci skip-rust] [ci skip-build-wheels]

[internal] Add experimental_resolve field to pex_binary

ad5a905

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano requested review from stuhood, jsirois, benjyw and chrisjrn September 1, 2021 23:00

Fix bad field name reference

88cc938

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano commented Sep 1, 2021

View reviewed changes

Eric-Arellano mentioned this pull request Sep 2, 2021

Redesign Python 3rdparty dependency management (lockfiles) #12314

Closed

Eric-Arellano added 2 commits September 1, 2021 20:24

Merge branch 'main' of github.com:pantsbuild/pants into pex-binary-lo…

9ff6af7

…ckfile

Use a generator expression

51a158e

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

stuhood approved these changes Sep 7, 2021

View reviewed changes

Eric-Arellano merged commit 824b8d7 into pantsbuild:main Sep 7, 2021

Eric-Arellano deleted the pex-binary-lockfile branch September 7, 2021 23:38

Eric-Arellano mentioned this pull request Sep 8, 2021

[internal] Add experimental_resolve to python_tests #12773

Merged

stuhood mentioned this pull request Sep 9, 2021

Upgrade to Pex 2.1.48 and leverage packed layout. (cherrypick of #12715, #12808) #12833

Closed

jsirois mentioned this pull request Sep 10, 2021

Prepare the 2.8.0.dev1 release. #12845

Merged

Eric-Arellano mentioned this pull request Sep 13, 2021

jvm: inject coursier_lockfile dependency for java targets #12784

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[internal] Add `experimental_resolve` field to `pex_binary` #12734

[internal] Add `experimental_resolve` field to `pex_binary` #12734

Eric-Arellano commented Sep 1, 2021 •

edited

Loading

Eric-Arellano Sep 1, 2021

cognifloyd Sep 2, 2021

Eric-Arellano Sep 2, 2021

Eric-Arellano Sep 1, 2021

Eric-Arellano Sep 1, 2021

stuhood Sep 7, 2021

Eric-Arellano Sep 7, 2021

benjyw commented Sep 2, 2021

Eric-Arellano commented Sep 2, 2021 •

edited

Loading

benjyw commented Sep 2, 2021

stuhood left a comment •

edited

Loading

stuhood Sep 7, 2021

Eric-Arellano Sep 7, 2021

stuhood Sep 7, 2021

Eric-Arellano Sep 7, 2021 •

edited

Loading

stuhood Sep 7, 2021

[internal] Add experimental_resolve field to pex_binary #12734

[internal] Add experimental_resolve field to pex_binary #12734

Conversation

Eric-Arellano commented Sep 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjyw commented Sep 2, 2021

Eric-Arellano commented Sep 2, 2021 • edited Loading

benjyw commented Sep 2, 2021

stuhood left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eric-Arellano Sep 7, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

[internal] Add `experimental_resolve` field to `pex_binary` #12734

[internal] Add `experimental_resolve` field to `pex_binary` #12734

Eric-Arellano commented Sep 1, 2021 •

edited

Loading

Eric-Arellano commented Sep 2, 2021 •

edited

Loading

stuhood left a comment •

edited

Loading

Eric-Arellano Sep 7, 2021 •

edited

Loading