Allow `@union`s to declare additional types which are provided to implementers #16717

stuhood · 2022-08-31T17:33:09Z

As described in #12934, each @union Get callsite currently implicitly represents an API where exactly the parameters provided to the Get are available, and no others.

In order for other types to be made available to plugins (most immediately, a new EnvironmentName type to differentiate subgraphs for different environments in #13682), we need a new syntax to indicate that @unions provide additional types as part of their API.

To that end, this change allows a @union declaration to specify additional types that will be in scope for callees (and consequently must be available to callers):

# Declares that callsites using a `Vehicle` must have `Fuel` in scope, and that `Fuel` will be
# in scope for callees/implementers of the `@union`.
@union(in_scope_types=[Fuel])
class Vehicle(ABC):
    ...

Fixes #12934.

[ci skip-build-wheels]

#16717 increases the cost of rule graph solving by about 50%: profiling showed that hashing and allocation contribute a significant amount of time there. The default `SipHasher` used in Rust is secure and randomized, but more expensive than a simpler hasher such as `FnvHasher` (see https://nnethercote.github.io/perf-book/hashing.html). This change switches to `FnvHasher` in the `rule_graph` crate, and fills out its usage in the `graph` crate. Additionally, `combinations_of_one` generates many tiny `Vec`s of a known fixed size, but was not pre-allocating them at that size. [ci skip-build-wheels]

…ld#16726) pantsbuild#16717 increases the cost of rule graph solving by about 50%: profiling showed that hashing and allocation contribute a significant amount of time there. The default `SipHasher` used in Rust is secure and randomized, but more expensive than a simpler hasher such as `FnvHasher` (see https://nnethercote.github.io/perf-book/hashing.html). This change switches to `FnvHasher` in the `rule_graph` crate, and fills out its usage in the `graph` crate. Additionally, `combinations_of_one` generates many tiny `Vec`s of a known fixed size, but was not pre-allocating them at that size. [ci skip-build-wheels]

…lize "re-entering" the rule graph via a `Query`. [ci skip-build-wheels]

[ci skip-build-wheels]

stuhood · 2022-09-02T04:08:46Z

Sorry for the big change... I'm really looking forward to replacing this rule graph implementation.

Commits are useful to review independently, particularly the top one which demonstrates usage. See also #16721 for additional usages.

thejcannon · 2022-09-02T11:39:45Z

src/python/pants/engine/internals/scheduler_test.py

@@ -144,7 +149,7 @@ def num_wheels(self) -> int:


 @rule
-def car_num_wheels(car: Car) -> int:
+def car_num_wheels(car: Car, _: Fuel) -> int:


What's the difference between this new syntax, and having a Fuel object be passed into the construction of the union member(s)?

That would be equivalent from a "number of @rule instances created" perspective.

But from a practical perspective, adjusting every type everywhere to pass a type (and then actually passing it) is precisely the kind of thing the engine was designed to avoid.

So instead of adjusting types and callsites we adjust rules declarations? I can dig it.

src/python/pants/engine/unions.py

src/rust/engine/rule_graph/src/builder.rs

Eric-Arellano · 2022-09-02T16:03:57Z

src/rust/engine/src/python.rs

+    let gil = Python::acquire_gil();
+    let py = gil.python();


FYI this is deprecated in the newest PyO3. Better is to use with_gil

thejcannon · 2022-09-02T16:20:09Z

src/python/pants/engine/internals/scheduler_test.py

+    pass
+
+
+@union(in_scope_types=[Fuel])


in_scope sounds weird to me. It doesn't quite give that "you have to declare additional arguments" vibe.

It doesn't quite give that "you have to declare additional arguments" vibe.

To be clear: you don't have to declare any additional arguments. These are just types that are guaranteed to be available to the callee. So in this example, you can see that motorcycle_num_wheels doesn't use Fuel. It could, but it doesn't need to.

I'm open to changing the name as part of #16721 (and beyond), so I'm going to roll with this for now.

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

In order to differentiate subgraphs of the `@rule` graph for different platforms and environments for #7735 (to support #13682), the environment that is in use needs to become a very widespread parameter in the `@rule` graph. To do that, this change: 1. Adjusts all production `QueryRule`s to declare that an `EnvironmentName` must be provided. 2. Adjusts most `@union`s to use #16717 to declare that an `EnvironmentName` will be available. 3. Computes the `Platform` from the `EnvironmentName` (currently as a noop, since we don't actually have multiple instances of the `EnvironmentName`). A caveat for point 1 is that to avoid churn, we only update _production_ `QueryRule`s in this change. For continuity in tests, `RuleRunner` will install a singleton `EnvironmentName` by default, and uses a (probably temporary) facility to filter the `EnvironmentName` back out of all `QueryRule`s. Tests that actually want to test with multiple environments should disable that facility.

stuhood added the category:plugin api change label Aug 31, 2022

stuhood mentioned this pull request Aug 31, 2022

Minor performance optimizations in rule_graph and graph #16726

Merged

stuhood mentioned this pull request Sep 1, 2022

Allow a @union's API to be satisfied by a non-singleton @rule #16751

Open

stuhood force-pushed the stuhood/union-scope-api branch 3 times, most recently from 3468ccb to a281656 Compare September 2, 2022 01:39

stuhood added 3 commits September 1, 2022 20:55

Record in_scope_params at Get registration time, and use to forma…

232abe2

…lize "re-entering" the rule graph via a `Query`. [ci skip-build-wheels]

Remove redundant is_union implementation.

6950d10

[ci skip-build-wheels]

Allow the @union decorator to include in_scope_types.

738cc2b

[ci skip-build-wheels]

stuhood force-pushed the stuhood/union-scope-api branch from a281656 to 738cc2b Compare September 2, 2022 03:56

stuhood requested review from Eric-Arellano, kaos and thejcannon September 2, 2022 03:56

stuhood marked this pull request as ready for review September 2, 2022 04:07

thejcannon reviewed Sep 2, 2022

View reviewed changes

Eric-Arellano approved these changes Sep 2, 2022

View reviewed changes

stuhood mentioned this pull request Sep 2, 2022

The environment to use becomes a context-specific parameter #16721

Merged

thejcannon reviewed Sep 2, 2022

View reviewed changes

stuhood merged commit 7b2cb44 into pantsbuild:main Sep 2, 2022

stuhood deleted the stuhood/union-scope-api branch September 2, 2022 19:13

stuhood added a commit to stuhood/pants that referenced this pull request Sep 2, 2022

Review feedback from pantsbuild#16717.

da600f0

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

stuhood added a commit to stuhood/pants that referenced this pull request Sep 2, 2022

Review feedback from pantsbuild#16717.

09ce8f3

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

stuhood added a commit to stuhood/pants that referenced this pull request Sep 2, 2022

Review feedback from pantsbuild#16717.

1db903c

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

stuhood added a commit to stuhood/pants that referenced this pull request Sep 2, 2022

Review feedback from pantsbuild#16717.

081c571

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

stuhood mentioned this pull request Sep 8, 2022

Add support for cross-building Python applications to test, package inside-of/for a Docker image #13682

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow `@union`s to declare additional types which are provided to implementers #16717

Allow `@union`s to declare additional types which are provided to implementers #16717

stuhood commented Aug 31, 2022 •

edited

Loading

stuhood commented Sep 2, 2022

thejcannon Sep 2, 2022

stuhood Sep 2, 2022 •

edited

Loading

thejcannon Sep 2, 2022

Eric-Arellano Sep 2, 2022

thejcannon Sep 2, 2022

stuhood Sep 2, 2022

stuhood Sep 2, 2022

Allow @unions to declare additional types which are provided to implementers #16717

Allow @unions to declare additional types which are provided to implementers #16717

Conversation

stuhood commented Aug 31, 2022 • edited Loading

stuhood commented Sep 2, 2022

thejcannon Sep 2, 2022

Choose a reason for hiding this comment

stuhood Sep 2, 2022 • edited Loading

Choose a reason for hiding this comment

thejcannon Sep 2, 2022

Choose a reason for hiding this comment

Eric-Arellano Sep 2, 2022

Choose a reason for hiding this comment

thejcannon Sep 2, 2022

Choose a reason for hiding this comment

stuhood Sep 2, 2022

Choose a reason for hiding this comment

stuhood Sep 2, 2022

Choose a reason for hiding this comment

Allow `@union`s to declare additional types which are provided to implementers #16717

Allow `@union`s to declare additional types which are provided to implementers #16717

stuhood commented Aug 31, 2022 •

edited

Loading

stuhood Sep 2, 2022 •

edited

Loading