[internal] Add `compatible_resolves` to `jvm_artifact` target #13893

Eric-Arellano · 2021-12-16T00:45:39Z

Part of #13621.

As explained in #13621 (comment), jvm_artifact targets now declare explicitly which resolves they should be used with. Rather than where before we inferred this based on analyzing which first-party targets consumed the artifacts and worked backward by seeing the resolves those targets themselves used.

Having jvm_artifact targets declare which resolves they use is necessary for us to update dependency inference so that you can only infer deps on artifacts that belong to your resolve. Whereas right now you can infer deps on the entire universe of artifacts. We need to know what's in a resolve before any dependency inference has happened.

This change also brings clearer modeling for users, less magic. While there's more up-front work when you add a new jvm_artifact, it makes it much clearer what your intent is for where the resolve should be used.

[ci skip-rust]
[ci skip-build-wheels]

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano · 2021-12-16T00:47:55Z

src/python/pants/jvm/goals/coursier.py

+        if not tgt.has_field(JvmArtifactCompatibleResolvesField):
+            continue
+        # TODO: add a `default_compatible_resolves` field.
+        # TODO: error if no resolves for the target? Which would mean it's practically invisible.


This feels like a bug waiting to happen. I think we should error.

FYI the fix will be to either make sure [jvm].default_compatible_resolves is set or to manually set it for the target.

pants/src/python/pants/jvm/subsystems.py

Lines 23 to 29 in 18350c5

register(

"--default-resolve",

type=str,

# TODO: Default this to something like `jvm-default`.

default=None,

help="The name of the resolve to use by default.",

)

is relevant here, and could probably be used now (even though it isn't a list option yet).

And as previously mentioned, I think that we should set an actual default value there.

But yes: no resolves should be an error.

This is already using --default-resolve if you look at the next line.

I think that we should set an actual default value there.

Agreed, waiting to do in a followup.

Eric-Arellano · 2021-12-16T00:48:13Z

src/python/pants/jvm/goals/coursier.py

-        CoursierResolvedLockfile, ArtifactRequirements, artifact_requirements
+        CoursierResolvedLockfile,
+        ArtifactRequirements,
+        # TODO: error if no artifacts associated with a resolve?


I think we probably should allow it? You'd have an empty lockfile generated, which is a little weird but also legal.

At the very least, it is common in tests.

Yeah we should allow it. We have in pantsbuild/pants a jvm_testprojects resolve. Currently, it's unused, but it's convenient we have it defined already in pants.toml so that we can add to it.

Yeah we should allow it. We have in pantsbuild/pants a jvm_testprojects resolve. Currently, it's unused, but it's convenient we have it defined already in pants.toml so that we can add to it.

It's the default resolve, so it's not unused... it's used for the code under testprojects/src/jvm, right?

They don't use any jvm_artifacts. Note that the lockfile has always been empty, and remains empty with this change

Ah, sure. Not unused, just empty.

Eric-Arellano · 2021-12-16T00:48:44Z

src/python/pants/jvm/goals/coursier_integration_test.py

    )


 @maybe_skip_jdk_test
-def test_coursier_resolve_noop_does_not_touch_lockfile(rule_runner: RuleRunner) -> None:


No changes, just DRY

Eric-Arellano · 2021-12-16T00:48:51Z

src/python/pants/jvm/goals/coursier_integration_test.py

    return rule_runner


 @maybe_skip_jdk_test
-def test_coursier_resolve_creates_missing_lockfile(rule_runner: RuleRunner) -> None:


No changes, just DRY

Eric-Arellano · 2021-12-16T00:49:25Z

src/python/pants/jvm/goals/coursier_integration_test.py

+
+
+@maybe_skip_jdk_test
+def test_updates_lockfile(rule_runner: RuleRunner) -> None:


This was the test right below. No changes, just DRY

Eric-Arellano · 2021-12-16T00:49:42Z

src/python/pants/jvm/goals/coursier_integration_test.py


-@maybe_skip_jdk_test
-def test_coursier_resolve_updates_bogus_lockfile(rule_runner: RuleRunner) -> None:


Deleted because this test doesn't seem like it's pulling its weight

Eric-Arellano · 2021-12-16T00:50:08Z

src/python/pants/jvm/target_types.py

@@ -154,6 +154,21 @@ class JvmProvidesTypesField(StringSequenceField):
    )


+class JvmArtifactCompatibleResolvesField(JvmCompatibleResolvesField):
+    help = (


Writing is hard...feedback welcomed for both these new help messages

So, it sortof feels like rather than customizing the help for use with JvmArtifact, you could expand the help for JvmCompatibleResolvesField to indicate that it applies across all targets, and defines a universe of code that uses a particular thirdparty resolve... i.e., move some of the help you've added here onto the existing field instead.

I still need to update JvmCompatibleResolvesField in a followup, but I do think it's valuable for these to diverge. In particular, the behavior is different for how we validate that the entire transitive closure is compatible with each other. For jvm_artifact, it by definition is the leaf(?) of the closure as it does not have the dependencies field. So it doesn't make sense to talk about validating its dependencies.

Hm. Maybe. But I think that similar to interpreter_constraints, you need to understand the big picture before you can successfully set the field anywhere. Splitting the big picture up across multiple fields makes that a bit harder. But there might be a few details that are only relevant here.

Yeah I think we should do a comprehensive audit of help messages for JVM at the same time we write the docs for it. (I like writing docs, so I can do that when the time is right if you'd like)

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano · 2021-12-16T01:03:20Z

The next followup will add [jvm].default_compatible_resolves. Then I'm going to proceed with updating dependency inference as planned in #13621 (comment).

Reminder that we are still missing validation that the resolve chosen for consumption is valid for the entire transitive closure, per #13876. That will be a followup to this all.

stuhood

Thanks!

Before landing this, there is a thing to fix: the pantsbuild/pants repo uses three different resolves. Because we don't have JVM lockfile staleness checks, these changes didn't cause check to fail: but if you regenerate the lockfiles for the repo right now, it will likely cause some code to fail to build. You'll need to actually use the new field in a few places under https://github.com/pantsbuild/pants/tree/main/3rdparty/jvm to make the resolves accurate again.

stuhood · 2021-12-16T17:22:42Z

src/python/pants/jvm/goals/coursier.py

-        CoursierResolvedLockfile, ArtifactRequirements, artifact_requirements
+        CoursierResolvedLockfile,
+        ArtifactRequirements,
+        # TODO: error if no artifacts associated with a resolve?


At the very least, it is common in tests.

stuhood · 2021-12-16T17:26:07Z

src/python/pants/jvm/target_types.py

@@ -154,6 +154,21 @@ class JvmProvidesTypesField(StringSequenceField):
    )


+class JvmArtifactCompatibleResolvesField(JvmCompatibleResolvesField):
+    help = (


So, it sortof feels like rather than customizing the help for use with JvmArtifact, you could expand the help for JvmCompatibleResolvesField to indicate that it applies across all targets, and defines a universe of code that uses a particular thirdparty resolve... i.e., move some of the help you've added here onto the existing field instead.

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

…target-api [ci skip-rust] [ci skip-build-wheels]

stuhood

Thanks!

[internal] Add compatible_resolves to jvm_artifact target

f1088f2

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano requested review from stuhood, chrisjrn and tdyas December 16, 2021 00:45

Eric-Arellano commented Dec 16, 2021

View reviewed changes

Eric-Arellano added 2 commits December 15, 2021 17:54

Fix MyPy

9186735

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Fix bad imports, oops

60fd440

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

stuhood approved these changes Dec 16, 2021

View reviewed changes

Eric-Arellano added 3 commits December 16, 2021 11:13

Update our own lockfiles

6c8d128

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Review feedback and fix a bug

954cd41

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Merge branch 'main' of github.com:pantsbuild/pants into jvm-artifact-…

4ac5f24

…target-api [ci skip-rust] [ci skip-build-wheels]

Eric-Arellano enabled auto-merge (squash) December 16, 2021 19:00

stuhood approved these changes Dec 16, 2021

View reviewed changes

Eric-Arellano merged commit 5f4f8c1 into pantsbuild:main Dec 16, 2021

Eric-Arellano deleted the jvm-artifact-target-api branch December 16, 2021 19:40

jsirois mentioned this pull request Dec 17, 2021

Prepare the 2.9.0rc0 release. #13916

Merged

stuhood mentioned this pull request Jan 26, 2022

Adds CoursierWrapperProcess and JvmProcess, makes JVM executions less tenuous #14270

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[internal] Add `compatible_resolves` to `jvm_artifact` target #13893

[internal] Add `compatible_resolves` to `jvm_artifact` target #13893

Eric-Arellano commented Dec 16, 2021

Eric-Arellano Dec 16, 2021

stuhood Dec 16, 2021 •

edited

Loading

Eric-Arellano Dec 16, 2021

Eric-Arellano Dec 16, 2021

stuhood Dec 16, 2021

Eric-Arellano Dec 16, 2021

stuhood Dec 16, 2021

Eric-Arellano Dec 16, 2021

stuhood Dec 16, 2021

Eric-Arellano Dec 16, 2021

Eric-Arellano Dec 16, 2021

Eric-Arellano Dec 16, 2021

Eric-Arellano Dec 16, 2021

Eric-Arellano Dec 16, 2021

stuhood Dec 16, 2021

Eric-Arellano Dec 16, 2021

stuhood Dec 16, 2021

Eric-Arellano Dec 16, 2021

Eric-Arellano commented Dec 16, 2021

stuhood left a comment

stuhood Dec 16, 2021

stuhood Dec 16, 2021

stuhood left a comment

	register(
	"--default-resolve",
	type=str,
	# TODO: Default this to something like `jvm-default`.
	default=None,
	help="The name of the resolve to use by default.",
	)



		@maybe_skip_jdk_test
		def test_updates_lockfile(rule_runner: RuleRunner) -> None:


		@maybe_skip_jdk_test
		def test_coursier_resolve_updates_bogus_lockfile(rule_runner: RuleRunner) -> None:

[internal] Add compatible_resolves to jvm_artifact target #13893

[internal] Add compatible_resolves to jvm_artifact target #13893

Conversation

Eric-Arellano commented Dec 16, 2021

Choose a reason for hiding this comment

stuhood Dec 16, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eric-Arellano commented Dec 16, 2021

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood left a comment

Choose a reason for hiding this comment

[internal] Add `compatible_resolves` to `jvm_artifact` target #13893

[internal] Add `compatible_resolves` to `jvm_artifact` target #13893

stuhood Dec 16, 2021 •

edited

Loading