[Plugin API] Add type `CurrentGoals` to allow rules to find out the current running goals. #17674

kaos · 2022-11-29T14:35:24Z

Example:

from pants.engine.internals.scheduler import CurrentExecutingGoal

@rule
def my_rule(..., current_goal: CurrentExecutingGoal) -> MyResult:
  if "my-goal" == current_goal.name:
    # conditional stuff for my-goal

  return MyResult()

This is for #17671 (comment)

…ut the current goal.

src/python/pants/engine/goal.py

stuhood

I'm concerned that this will break in the presence of #10542, which doesn't have many other blockers left.

Could #17671 (comment) use a flag on the request type instead?

kaos · 2022-12-01T14:14:42Z

I'm concerned that this will break in the presence of #10542, which doesn't have many other blockers left.

Good point. I'll have another look with that in mind.

Could #17671 (comment) use a flag on the request type instead?

Ah, I'd like that, but as the linting goal would also trigger fetching all dependencies, it would still reach the visibility backend the regular way.

One thing I tried (and failed, but maybe that could be fixed) was to introduce a "context value" in the rule graph, akin to how React does. Conceptual example:

@rule
def context_value() -> MyContextValue:
  return MyContextValue(defaultValue)

@rule
def validate_visibility_dependency_rules(ctx_value: MyContextValue, request: ...) -> ValidatedDependencies:
  ...

# ---

# Here I may not understand masked_types properly, my thoughts where to "replace" any previous value with a new one
@goal_rule(_masked_types=[MyContextValue])
async def visibility_linter_goal(...):
  # The provided MyContextValue here is the key - I want it to replace any previous source for it with this new one.
  await Get(ValidatedDependencies, {request: RequestType, ctx_value: MyContextValue})
  ...

kaos · 2022-12-14T13:31:45Z

@stuhood I've adjusted to implement most of the goal rule identification on the rust side of the engine, hopefully that'll work also when running goals in parallel.

kaos · 2022-12-15T15:17:57Z

This does not block #10542 but needs an alternate solution to unblock #7654, as do RunId and SessionValues so this may be considered good enough until such an alternative solution is developed.

stuhood · 2022-12-15T21:41:14Z

src/python/pants/engine/goal.py

+    @final
+    @classmethod
+    def is_goal(cls) -> bool:
+        return True


This should probably use is_subclass instead rather than a new property.

Agree, I can take another look at that. I tried that first and failed, but I've learned so much each day, so I may be able to pull it off this time. Perhaps :P

stuhood · 2022-12-15T21:42:32Z

src/python/pants/engine/internals/scheduler.py

@@ -75,6 +75,13 @@
 Workunit = Dict[str, Any]


+@dataclass(frozen=True)
+class CurrentGoals:
+    """The currently running goals."""


Suggested change

"""The currently running goals."""

"""The currently running goals if any."""

It's possible for no goals to be running: probably the intrinsic @rule should fail in that case.

Why fail? The .rule_names could be empty in that case, is that not a valid scenario..?

stuhood · 2022-12-15T21:43:00Z

src/python/pants/engine/internals/scheduler.py

+class CurrentGoals:
+    """The currently running goals."""
+
+    rule_names: tuple[str, ...]


Should this be rule names, or the Goal.subsystem.name?

I'd prefer the Goal.subsystem.name but failed to figure out how to get that :P
Then again, I realized this may be more robust, or too rigid I suppose, depending on the use case. Maybe it would be nice to capture both?

stuhood · 2022-12-15T21:49:01Z

src/rust/engine/src/nodes.rs

+  async fn run_node(self, context: Context, workunit: &mut RunningWorkunit) -> NodeResult<Value> {
+    let gil = Python::acquire_gil();
+    let py = gil.python();
+    let names = workunit.get_current_goals();


Rather than doing this via workunits, you should probably make it a mutable property of the Session, similar to the SessionValues:

pants/src/rust/engine/src/session.rs

Lines 86 to 87 in 7f364f8

// Per-Session values that have been set for this session.

session_values: Mutex<PyObject>,

Then, in run_goal_rules, you would try-finally setting and clearing the CurrentGoals value.

Ahh.. ok. Interesting. Yea, going via workunits has been a source of much headache and fiddling to tie the pieces together.

But wait.. if we have that in try/finally in the python world, that's pretty close to what I had originally only applied at a different spot, right?

aaa50d6

Where I stored it in SessionValues directly rather than as a property of the session, though.

Hmm. Yea, it is.

I think that I was confused by the mutation in that case... I think that you could store this directly in the SessionValues, which we already support setting? Then you don't need to mutate the CurrentGoals type, and can get it in a normal rule from the SessionValues via session_values[CurrentGoals]:

pants/src/python/pants/engine/internals/options_parsing.py

Line 48 in d209d15

options_bootstrapper = session_values[OptionsBootstrapper]

Uhm, just coming back to this again--and in my initial version I stored the CurrentExecutingGoal type in the session values as you suggest. But I made it mutatable as my understanding is that I can't change the session values once the session has been created..?

pants/src/python/pants/bin/local_pants_runner.py

Lines 100 to 106 in aaa50d6

session_values=SessionValues(

{

OptionsBootstrapper: options_bootstrapper,

CompleteEnvironmentVars: env,

CurrentExecutingGoal: CurrentExecutingGoal(),

}

),

kaos · 2023-04-11T17:48:56Z

src/python/pants/init/engine_initializer.py

+    goal_param_types: ClassVar[tuple[type, ...]] = (
+        Specs,
+        Console,
+        Workspace,
+        EnvironmentName,
+        CurrentExecutingGoal,
+    )


@stuhood I found this spot for it. Feels so much more appropriate than in session values! ;)

Unfortunately, this will cause anything that consumes the CurrentExecutingGoal to get a new "identity" for each goal. So if you consumed this field below a graph calculation (for example), you would end up with two copies of the graph in memory.

Happy to have a video call to talk through this if need be.

Ugh. OK. Let's do that, can sync on slack.

kaos · 2023-04-11T17:52:42Z

src/python/pants/init/engine_initializer.py

+            params = Params(
+                specs, self.console, workspace, env_name, CurrentExecutingGoal(goal, goal_product)
+            )


regarding #10542 I think that we must still be able to provide goal specific Params even when running multiple goals in parallel and as such this "should" be OK, no? (as it is flagged as uncachable)

Yes, this is safe from a #10542 perspective. But it has the other problem I mentioned above: #17674 (comment) ... =(

stuhood

As discussed offline, I'm fine with this as long as it is well labeled with the caveat that any @rule consuming the CurrentGoals will need to re-run for different input goals.

kaos · 2023-04-21T18:23:29Z

As discussed offline, I'm fine with this as long as it is well labeled with the caveat that any @rule consuming the CurrentGoals will need to re-run for different input goals.

Turns out this is a bigger caveat than I want us to swallow. Looking for alternative venues.

kaos · 2023-04-22T14:16:15Z

Superseded by #18788

[Plugin API] Add type CurrentExecutingGoal to allow rules to find o…

aaa50d6

…ut the current goal.

kaos added the category:new feature label Nov 29, 2022

kaos requested a review from stuhood November 29, 2022 14:35

kaos mentioned this pull request Nov 29, 2022

Add visibility linter to report all violations. #17671

Merged

kaos commented Nov 29, 2022

View reviewed changes

src/python/pants/engine/goal.py Show resolved Hide resolved

stuhood reviewed Dec 1, 2022

View reviewed changes

kaos force-pushed the current-goal-rule branch from aaa50d6 to f5274fc Compare December 14, 2022 13:22

kaos requested a review from stuhood December 14, 2022 13:30

kaos changed the title ~~[Plugin API] Add type CurrentExecutingGoal to allow rules to find out the current goal.~~ [Plugin API] Add type CurrentGoals to allow rules to find out the current running goals. Dec 15, 2022

stuhood reviewed Dec 15, 2022

View reviewed changes

Merge branch 'main' into current-goal-rule

66bed4c

kaos mentioned this pull request Dec 20, 2022

Add error boundary support for rules #17745

Closed

kaos added 2 commits April 11, 2023 12:02

Merge branch 'main' into current-goal-rule

bcffc1a

Move CurrentExecutingGoal to the goal param types.

de8391b

kaos force-pushed the current-goal-rule branch from 62d62b7 to de8391b Compare April 11, 2023 17:46

kaos commented Apr 11, 2023

View reviewed changes

kaos requested a review from stuhood April 11, 2023 17:50

kaos commented Apr 11, 2023

View reviewed changes

stuhood approved these changes Apr 17, 2023

View reviewed changes

kaos mentioned this pull request Apr 22, 2023

New API type: CurrentExecutingGoals. #18788

Merged

kaos closed this Apr 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Plugin API] Add type `CurrentGoals` to allow rules to find out the current running goals. #17674

[Plugin API] Add type `CurrentGoals` to allow rules to find out the current running goals. #17674

kaos commented Nov 29, 2022 •

edited

Loading

stuhood left a comment

kaos commented Dec 1, 2022

kaos commented Dec 14, 2022

kaos commented Dec 15, 2022

stuhood Dec 15, 2022

kaos Dec 15, 2022

stuhood Dec 15, 2022

kaos Dec 15, 2022

stuhood Dec 15, 2022

kaos Dec 15, 2022

stuhood Dec 15, 2022

kaos Dec 15, 2022

kaos Dec 15, 2022

stuhood Dec 17, 2022 •

edited

Loading

kaos Jan 6, 2023

kaos Apr 11, 2023

stuhood Apr 11, 2023

kaos Apr 11, 2023

kaos Apr 11, 2023 •

edited

Loading

stuhood Apr 11, 2023

stuhood left a comment

kaos commented Apr 21, 2023

kaos commented Apr 22, 2023

	"""The currently running goals."""
	"""The currently running goals if any."""

	// Per-Session values that have been set for this session.
	session_values: Mutex<PyObject>,

	session_values=SessionValues(
	{
	OptionsBootstrapper: options_bootstrapper,
	CompleteEnvironmentVars: env,
	CurrentExecutingGoal: CurrentExecutingGoal(),
	}
	),

[Plugin API] Add type CurrentGoals to allow rules to find out the current running goals. #17674

[Plugin API] Add type CurrentGoals to allow rules to find out the current running goals. #17674

Conversation

kaos commented Nov 29, 2022 • edited Loading

stuhood left a comment

Choose a reason for hiding this comment

kaos commented Dec 1, 2022

kaos commented Dec 14, 2022

kaos commented Dec 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood Dec 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaos Apr 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood left a comment

Choose a reason for hiding this comment

kaos commented Apr 21, 2023

kaos commented Apr 22, 2023

[Plugin API] Add type `CurrentGoals` to allow rules to find out the current running goals. #17674

[Plugin API] Add type `CurrentGoals` to allow rules to find out the current running goals. #17674

kaos commented Nov 29, 2022 •

edited

Loading

stuhood Dec 17, 2022 •

edited

Loading

kaos Apr 11, 2023 •

edited

Loading