Add initial support for a `parametrize` builtin to generate multiple copies of a target #14408

stuhood · 2022-02-09T05:40:38Z

As introduced in #13882 and designed in https://docs.google.com/document/d/1mzZWnXiE6OkgMH_Dm7WeA3ck9veusQmr_c6GWPb_tsQ/edit#, this change adds the parametrize builtin.

Examples:

# Create two copies of each generated target, with addresses qualified with `@resolve=a` and `@resolve=b`.
python_sources(
  ..,
  resolve=parametrize(“a”, “b”),
)

# ...alternatively, define an alias for some or all of the values:
python_sources(
  ..,
  resolve=parametrize(“a”, b="bread and butter"),
)

# Create four copies of each generated target, with addresses qualified with both parametrized fields.
scala_sources(
  ..,
  resolve=parametrize(“a”, “b”),
  jdk=parametrize("c", "d"),
)

# Apply parametrization to only some of the generated targets:
scala_sources(
  ..,
  overrides={
    "JdkDependent.scala": {"jdk": parametrize("c", "d")},
  },
)

parametrize may be used in any field (other than name) of non-generator targets, but is limited to moved fields of generator targets (i.e., those which will be moved from the generator to the generated target).

Using parametrize in the overrides of a target generator requires adjustments to the target generator's implementation (see #14430): this change only adjusts the implementation for target file generators (TargetFileGenerator).

kaos

Cool stuff :)

I've raised the question regarding the use of dict rather than FrozenDict and Mapping for immutability. Maybe I've overlooked something.. ?

src/python/pants/build_graph/address.py

src/python/pants/engine/internals/parametrize.py

kaos

This. Is. So. Awesome! :D

lgtm, but I haven't reviewed the changes to all backends.

src/python/pants/backend/shell/target_types.py

src/python/pants/engine/target.py

kaos · 2022-02-10T15:23:04Z

src/python/pants/engine/internals/graph.py

-            f"{bullet_list(addr.spec for addr in generated)}\n\n"
+            f"The address `{address}` was not generated by the target `{base_address}`, which "
+            "only generated these addresses:\n\n"
+            f"{bullet_list(addr.spec for addr in parametrizations.parametrizations)}\n\n"


Perhaps a moot point, but I tend to prefer relying on Address.__str__ for presentational messages. (which in this case is the same... currently.)
So the moot point is, that if that were to change for whatever reason in the future, it'll be much less impact if we didn't sprinkle with addr.spec... ;)

Yea, reasonable. Thanks!

kaos · 2022-02-10T15:32:20Z

src/python/pants/engine/internals/graph.py

+
+    target: Target | None
+    generate_request = target_types_to_generate_requests.request_for(target_type)
+    if generate_request:


This is a huge if block... possible/worth to split out (or perhaps await Joshuas rule parsing that supports following function calls)

Sorry, I rushed this one in without doing this. I'm going to open a ticket for followup (since I'm out of town for the next few days), but will clean this up before 2.11.x.

Eric-Arellano · 2022-02-10T23:11:44Z

src/python/pants/build_graph/address.py

+            relative_file_path=self._relative_file_path,
+            parameters=merged_parameters,
+        )
+
    def maybe_convert_to_target_generator(self) -> Address:


Is target_generator a misnomer? Maybe base_target?

Yea, it probably is. It used to be build_target, although I like base.

Probably best as a followup though?

Fine with me.

Maybe open an issue tracking all the little fixes like this? I know you have some dedicated tickets for bigger changes.

src/python/pants/build_graph/address_test.py

Eric-Arellano · 2022-02-10T23:12:35Z

src/python/pants/core/register.py

+
+def build_file_aliases():
+    return BuildFileAliases(
+        objects={"parametrize": Parametrize},


V happy this is not a CAOF, I'm eagerly awaiting dropping that infrastructure.

src/python/pants/engine/internals/parametrize.py

…d Target templates. [ci skip-rust] [ci skip-build-wheels]

[ci skip-rust] [ci skip-build-wheels]

Eric-Arellano

It looks like this implementation lets you parametrize any field. As discussed on the design doc, I still strongly suspect that that is not optimal: there are some fields where it does not make semantic sense to parametrize, like skip_flake8.

A major downside of parametrization if you're not careful is dependency inference will cause ambiguity. We have some blessed fields like resolve where will update things to handle that, but it seems very wrong if we were to try to write something generically that has skip_flake8 in the key for module_mapping - you should be able to depend on something regardless of skip_flake8.

Further, it's much easier to loosen the restriction than to add it.

I know you're trying to land this before heading out for a couple days, so this doesn't need to block landing this PR. But what do you think about this, and if we could in a followup add ParametrizableFieldMarker?

Eric-Arellano · 2022-02-11T00:29:42Z

src/python/pants/engine/internals/graph.py


-    for field_type in target.field_types:
+    # TODO: Move to Target constructor.


Nack, we need to use the engine for singleton so that we don't warn hundreds of times.

Yea... but we could use the @memoized infrastructure for that, and it would cause it to be triggered in all cases, including for generated targets. Right now it's only triggered for generator targets, and only for non-moved fields.

src/python/pants/engine/internals/graph_test.py

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano

Thanks for making those changes!

(Approval is assuming that you're willing to entertain changing parametrization to not work with every field. I feel pretty strongly about this. But trying to not block.)

stuhood · 2022-02-11T01:53:47Z

Approval is assuming that you're willing to entertain changing parametrization to not work with every field. I feel pretty strongly about this. But trying to not block.

Yea, willing to entertain it. "dynamic" parametrization (which the design doc suggests will need a marker class) is still to come, so we'll be making more changes to resolution.

My suspicion on needing to mark fields as supporting "static" parametrization though is that it is likely to add boilerplate to field definition without giving adequate benefit. If someone parametrizes the skip_black field, then they'll already get a useful warning about ambiguity. But if we don't anticipate a usecase that someone would like to parametrize, then we've blocked them from doing something unnecessarily until they get an upstream change approved.

Eric-Arellano · 2022-02-11T02:01:35Z

s that it is likely to add boilerplate to field definition without giving adequate benefit.

Boilerplate isn't compelling here imo. It's a single import from pants.engine.target. Even if it were more painful, I'd rather optimize for user semantics than inconvenience writing the plugin.

then they'll already get a useful warning about ambiguity.

True, but they might not make the connection why it's not a great idea to parameterize skip_flake8. Vs a very eager error message saying this isn't legal, but you can do it with X Y and Z from this target.

But if we don't anticipate a usecase that someone would like to parametrize, then we've blocked them from doing something unnecessarily until they get an upstream change approved.

Yeah, that is more compelling. This is where I think it's easier to loosen than restrict. If we find this is happening enough, we can open the floodgates.

stuhood · 2022-02-11T02:07:05Z

s that it is likely to add boilerplate to field definition without giving adequate benefit.

Boilerplate isn't compelling here imo. It's a single import from pants.engine.target. Even if it were more painful, I'd rather optimize for user semantics than inconvenience writing the plugin.

That depends whether the restriction is per-Field or per-Target. The latter case is the more likely one to be safe (how can I know that all usages of a Field with any target are unsafe to parametrize?)

then they'll already get a useful warning about ambiguity.

True, but they might not make the connection why it's not a great idea to parameterize skip_flake8. Vs a very eager error message saying this isn't legal, but you can do it with X Y and Z from this target.

The warning would include @skip_flake8=true vs @skip_flake8=false, which is... pretty clear, IMO.

stuhood changed the title ~~Add initial support for the parametrize builtin to generate multiple copies of a target~~ Add initial support for a parametrize builtin to generate multiple copies of a target Feb 9, 2022

kaos reviewed Feb 9, 2022

View reviewed changes

src/python/pants/build_graph/address.py Outdated Show resolved Hide resolved

src/python/pants/engine/internals/parametrize.py Show resolved Hide resolved

src/python/pants/engine/internals/parametrize.py Show resolved Hide resolved

stuhood mentioned this pull request Feb 9, 2022

More declarative target generators for plugins #14377

Merged

stuhood force-pushed the stuhood/parameterization branch from 9ce6c3f to fc1a34f Compare February 9, 2022 23:00

stuhood mentioned this pull request Feb 10, 2022

Complete support for parametrize in overrides #14430

Open

stuhood force-pushed the stuhood/parameterization branch from fc1a34f to de0aee1 Compare February 10, 2022 07:06

stuhood requested review from Eric-Arellano and benjyw February 10, 2022 07:24

stuhood force-pushed the stuhood/parameterization branch from de0aee1 to 9cdd67e Compare February 10, 2022 07:25

stuhood marked this pull request as ready for review February 10, 2022 07:25

kaos reviewed Feb 10, 2022

View reviewed changes

Eric-Arellano reviewed Feb 10, 2022

View reviewed changes

stuhood added 2 commits February 10, 2022 16:04

Add the Parametrize builtin, and use it to create multiple generate…

fa0c64f

…d Target templates. [ci skip-rust] [ci skip-build-wheels]

Add support for parametrization of overrides and non-generated targets.

7280f3d

[ci skip-rust] [ci skip-build-wheels]

stuhood force-pushed the stuhood/parameterization branch from 9cdd67e to 7280f3d Compare February 11, 2022 00:04

Eric-Arellano reviewed Feb 11, 2022

View reviewed changes

Review feedback.

194ea5a

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano approved these changes Feb 11, 2022

View reviewed changes

stuhood enabled auto-merge (squash) February 11, 2022 02:04

stuhood merged commit 2d3bb35 into pantsbuild:main Feb 11, 2022

stuhood deleted the stuhood/parameterization branch February 11, 2022 02:08

stuhood mentioned this pull request Feb 11, 2022

Add generic support for parametrizing metadata like interpreter constraints and resolves #13882

Closed

Eric-Arellano mentioned this pull request Feb 16, 2022

Fix target parametrization by making it more eager #14496

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add initial support for a `parametrize` builtin to generate multiple copies of a target #14408

Add initial support for a `parametrize` builtin to generate multiple copies of a target #14408

stuhood commented Feb 9, 2022 •

edited

Loading

kaos left a comment

kaos left a comment

kaos Feb 10, 2022

stuhood Feb 11, 2022

kaos Feb 10, 2022

stuhood Feb 11, 2022

Eric-Arellano Feb 10, 2022

stuhood Feb 11, 2022

stuhood Feb 11, 2022

Eric-Arellano Feb 11, 2022

stuhood Feb 11, 2022

Eric-Arellano Feb 10, 2022

Eric-Arellano left a comment •

edited

Loading

Eric-Arellano Feb 11, 2022

stuhood Feb 11, 2022

Eric-Arellano left a comment •

edited

Loading

stuhood commented Feb 11, 2022 •

edited

Loading

Eric-Arellano commented Feb 11, 2022

stuhood commented Feb 11, 2022 •

edited

Loading


		for field_type in target.field_types:
		# TODO: Move to Target constructor.

Add initial support for a parametrize builtin to generate multiple copies of a target #14408

Add initial support for a parametrize builtin to generate multiple copies of a target #14408

Conversation

stuhood commented Feb 9, 2022 • edited Loading

kaos left a comment

Choose a reason for hiding this comment

kaos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eric-Arellano left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eric-Arellano left a comment • edited Loading

Choose a reason for hiding this comment

stuhood commented Feb 11, 2022 • edited Loading

Eric-Arellano commented Feb 11, 2022

stuhood commented Feb 11, 2022 • edited Loading

Add initial support for a `parametrize` builtin to generate multiple copies of a target #14408

Add initial support for a `parametrize` builtin to generate multiple copies of a target #14408

stuhood commented Feb 9, 2022 •

edited

Loading

Eric-Arellano left a comment •

edited

Loading

Eric-Arellano left a comment •

edited

Loading

stuhood commented Feb 11, 2022 •

edited

Loading

stuhood commented Feb 11, 2022 •

edited

Loading