More declarative target generators for plugins #14377

stuhood · 2022-02-06T01:00:53Z

#13882 will require adjusting how many instances of a target are created based on how many times the parametrize keyword appears in its fields (including in the overrides field, in the case of target generators). The parametrize keyword is not itself a valid field value (and would also be challenging to render as such in peek), and so target generation must be updated to consume something other than the instantiated target generator to decide which fields to generate with.

To prepare for these changes, the base ("template") and overridden fields to use in target generation are added to the GenerateTargetsRequest class, and made required in generate_file_level_targets. To reduce the total amount of churn in callers of generate_file_level_targets, this change introduces the TargetGenerator and TargetFilesGenerator base classes for Target. The latter allows file-target generation to be requested declaratively, and both allow specifying which fields should be "moved" and "copied" from the target generator to the generated targets. These moved/copied declarations are used to construct the template for generated targets.

[ci skip-rust]
[ci skip-build-wheels]

src/python/pants/backend/go/target_type_rules.py

src/python/pants/backend/project_info/peek_test.py

stuhood · 2022-02-09T22:55:55Z

src/python/pants/engine/internals/graph.py

+        # Split out the `propagated_fields` before construction.
+        generator_fields = dict(target_adaptor.kwargs)
+        template_fields = {}
+        # TODO: Require for all instances before landing.


The impact of not having done this is just that TargetGenerator subclasses (but not TargetFilesGenerator subclasses) are not able to consume the moved/copied fields, and are still consuming those fields from the generator class rather than the template. It's inconsistent, but I'd like to defer it to #14408 given the size of this PR.

src/python/pants/engine/target_test.py

src/python/pants/help/help_info_extracter.py

src/python/pants/init/engine_initializer.py

stuhood · 2022-02-09T23:14:01Z

The commits are useful to review independently. The first is unfortunately a mishmash, because it was refactored so many times. But the second one is entirely updating file-generating targets to use the new API.

stuhood · 2022-02-09T23:20:40Z

src/python/pants/backend/python/target_types.py

-    copied_fields = (PythonTestsDependenciesField,)
+    copied_fields = (
+        PythonTestsDependenciesField,
+        InterpreterConstraintsField,


We will fail (with a useful error message) to parametrize any core_field of a target generator, and so in order to parametrize interpreter_constraints, we'd need to make InterpreterConstraintsField a moved_field here. But there were tests (and maybe implementations?) relying on the current behavior of "all" targets in the graph having the field, so it felt best to tackle it in a followup.

Yeah imo this should be a moved_field. We should not be using the python_sources in the interpreter constraints calculation, that is a bug.

But agreed that it's best fixed in a followup.

Eric-Arellano

Leaving feedback on first commit. Will keep reviewing.

src/python/pants/backend/go/util_rules/go_mod.py

src/python/pants/engine/internals/build_files_test.py

src/python/pants/engine/internals/graph.py

src/python/pants/build_graph/address.py

src/python/pants/engine/target.py

src/python/pants/engine/internals/graph.py

[ci skip-rust] [ci skip-build-wheels]

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

[ci skip-rust] [ci skip-build-wheels]

stuhood

Have applied the review feedback, and split out some #14408-specific affordances to apply over there.

src/python/pants/engine/internals/graph.py

Eric-Arellano

Thanks for removing parametrization!

Eric-Arellano · 2022-02-10T17:23:37Z

src/python/pants/backend/codegen/avro/target_types.py

-        # Note: Avro files cannot import from other Avro files, so do not add dependencies.
-        add_dependencies_on_all_siblings=False,


It's a bummer we lose this comment. I think it's good to force plugin authors to reason about this, even though it's more boilerplate. What about requiring that you implement the Settings? Getting this option wrong is a big deal, it makes file-level targets irrelevant. And we messed it up in the past with files and resources.

(Fine if followup).

I think that "dependencies on all siblings" should be rare to set, and unlike the inverse it's challenging to detect that you've set it accidentally (because dependencies are automatically more coarse-grained than they should be, and so things will succeed). The inverse (not setting it when you wanted to) is much, much easier to detect.

For example: scala has accidentally been setting this via add_dependencies_on_all_siblings, without us realizing (or remembering, at least): see #14382. In the interests of making dependency inference a stronger default, and removing the boilerplate required to do the right thing, I don't think that we should require it.

Eric-Arellano · 2022-02-10T17:51:42Z

src/python/pants/backend/java/target_types.py

+    copied_fields = (
+        *COMMON_TARGET_FIELDS,
+        Dependencies,
+        JvmCompatibleResolvesField,


This should probably be moved? You move the singular resolve field, which makes sense.

Mm, yes. Let's see if it encounters the same issue as InterpreterConstraintsField, if so, will defer. I'll be getting a PR out to replace this field later today I think.

Ah: yea, there are some consumers/tests expecting all targets in a cycle to have either a resolve or JvmCompatibleResolvesField. I'm going to back this out, and get another PR out for replacing this field.

Eric-Arellano · 2022-02-10T17:54:32Z

src/python/pants/backend/python/target_types.py

-    copied_fields = (PythonTestsDependenciesField,)
+    copied_fields = (
+        PythonTestsDependenciesField,
+        InterpreterConstraintsField,


Yeah imo this should be a moved_field. We should not be using the python_sources in the interpreter constraints calculation, that is a bug.

But agreed that it's best fixed in a followup.

src/python/pants/backend/scala/compile/scalac_test.py

src/python/pants/backend/scala/target_types.py

Eric-Arellano · 2022-02-10T20:05:17Z

src/python/pants/engine/target.py

-                continue
-            elif field.value != field.default:
-                generated_target_fields[field.alias] = field.value
+        generated_target_fields = dict(template)


Why call this template rather than directly generated_target_fields?

My thinking on this was that the generator might add additional fields... I didn't want to give the impression that it needed to be a complete set.

Eric-Arellano · 2022-02-10T20:05:58Z

src/python/pants/engine/target.py

+    # The base Address to generate for. Note that due to parametrization, this may not
+    # always be the Address of the underlying target.


What would it be if not the underlying target?

In the case of parametrization, the generator will have been constructed with its core_fields, and the moved_fields will have been parametrized in the template. The template and the template_address will then be aligned in terms of which parameters have been used, but the generator target cannot be: no choices were made about its fields, since any parametrized fields have been moved off.

Eric-Arellano · 2022-02-10T20:17:50Z

Also this is a Plugin API change, not internal. At least one user had been using target gen for their plugin, although I think not anymore.

kaos · 2022-02-10T20:30:09Z

At least one user had been using target gen for their plugin, although I think not anymore.

I'm using target gen in a plugin internally at work :)

kaos

I added the comments I could remember from #14408 so there are comments in there that may apply to this PR
#14408 (review)

src/python/pants/backend/shell/target_types.py

src/python/pants/engine/target.py

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

…y some code which expects it to exist on all targets. # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

stuhood force-pushed the stuhood/more-declarative-generators branch 11 times, most recently from 45a187c to 575e0af Compare February 7, 2022 22:39

stuhood mentioned this pull request Feb 8, 2022

[internal] BSP: add union to gather BuildTarget instances #14392

Merged

stuhood force-pushed the stuhood/more-declarative-generators branch 3 times, most recently from 6b0f46e to ab0b8aa Compare February 9, 2022 22:52

stuhood commented Feb 9, 2022

View reviewed changes

stuhood requested review from Eric-Arellano, benjyw and kaos February 9, 2022 22:58

stuhood force-pushed the stuhood/more-declarative-generators branch from ab0b8aa to 4001102 Compare February 9, 2022 22:59

stuhood marked this pull request as ready for review February 9, 2022 22:59

stuhood commented Feb 9, 2022

View reviewed changes

Eric-Arellano reviewed Feb 10, 2022

View reviewed changes

src/python/pants/engine/internals/graph.py Show resolved Hide resolved

stuhood added 5 commits February 9, 2022 20:48

Add Target(Files)Generator to make file generation declarative.

bdce990

[ci skip-rust] [ci skip-build-wheels]

Update all generate_file_level_targets consumers.

ca8cf52

[ci skip-rust] [ci skip-build-wheels]

Make generate_file_level_targets private.

28002db

[ci skip-rust] [ci skip-build-wheels]

Expose TargetGenerator.moved_fields in the help.

9140a3e

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Review feedback and a test fix for pantsbuild#14419.

89bcdf1

[ci skip-rust] [ci skip-build-wheels]

stuhood force-pushed the stuhood/more-declarative-generators branch from e0ccb1c to 89bcdf1 Compare February 10, 2022 04:59

stuhood commented Feb 10, 2022

View reviewed changes

src/python/pants/engine/internals/graph.py Show resolved Hide resolved

Eric-Arellano approved these changes Feb 10, 2022

View reviewed changes

kaos approved these changes Feb 10, 2022

View reviewed changes

src/python/pants/backend/shell/target_types.py Show resolved Hide resolved

src/python/pants/engine/target.py Outdated Show resolved Hide resolved

src/python/pants/engine/target.py Show resolved Hide resolved

stuhood changed the title ~~[internal] More declarative target generators~~ More declarative target generators for plugins Feb 10, 2022

Review feedback.

25d878f

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

stuhood enabled auto-merge (squash) February 10, 2022 22:42

stuhood mentioned this pull request Feb 10, 2022

Add initial support for a parametrize builtin to generate multiple copies of a target #14408

Merged

Eric-Arellano approved these changes Feb 10, 2022

View reviewed changes

stuhood merged commit 6bbd9d8 into pantsbuild:main Feb 10, 2022

stuhood deleted the stuhood/more-declarative-generators branch February 11, 2022 00:03

Eric-Arellano mentioned this pull request Feb 16, 2022

Fix target parametrization by making it more eager #14496

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More declarative target generators for plugins #14377

More declarative target generators for plugins #14377

stuhood commented Feb 6, 2022 •

edited

Loading

stuhood Feb 9, 2022 •

edited

Loading

stuhood commented Feb 9, 2022 •

edited

Loading

stuhood Feb 9, 2022 •

edited

Loading

Eric-Arellano Feb 10, 2022

Eric-Arellano left a comment

stuhood left a comment

Eric-Arellano left a comment

Eric-Arellano Feb 10, 2022

stuhood Feb 10, 2022 •

edited

Loading

Eric-Arellano Feb 10, 2022

stuhood Feb 10, 2022

stuhood Feb 10, 2022

Eric-Arellano Feb 10, 2022

Eric-Arellano Feb 10, 2022

stuhood Feb 10, 2022 •

edited

Loading

Eric-Arellano Feb 10, 2022

stuhood Feb 10, 2022

Eric-Arellano commented Feb 10, 2022

kaos commented Feb 10, 2022

kaos left a comment

		# Note: Avro files cannot import from other Avro files, so do not add dependencies.
		add_dependencies_on_all_siblings=False,

		# The base Address to generate for. Note that due to parametrization, this may not
		# always be the Address of the underlying target.

More declarative target generators for plugins #14377

More declarative target generators for plugins #14377

Conversation

stuhood commented Feb 6, 2022 • edited Loading

stuhood Feb 9, 2022 • edited Loading

Choose a reason for hiding this comment

stuhood commented Feb 9, 2022 • edited Loading

stuhood Feb 9, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eric-Arellano left a comment

Choose a reason for hiding this comment

stuhood left a comment

Choose a reason for hiding this comment

Eric-Arellano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood Feb 10, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood Feb 10, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eric-Arellano commented Feb 10, 2022

kaos commented Feb 10, 2022

kaos left a comment

Choose a reason for hiding this comment

stuhood commented Feb 6, 2022 •

edited

Loading

stuhood Feb 9, 2022 •

edited

Loading

stuhood commented Feb 9, 2022 •

edited

Loading

stuhood Feb 9, 2022 •

edited

Loading

stuhood Feb 10, 2022 •

edited

Loading

stuhood Feb 10, 2022 •

edited

Loading