allow specifying multiple Params to a Get()

[cosmicexplorer]

Note: #12946 is prework for this change.

---

Currently, we can depend on any number of params in the rule graph in an @rule declaration, which may be specified either via Params when invoking scheduler.product_request(), via "singleton" rules provided at scheduler construction, or as outputs of other rules (see src/python/pants/engine/README.md). However, when we want to invoke a ruleset inside the body of another rule, we can only use rulesets requiring exactly one param due to the current limitations of the await Get(...) syntax. We would like to do:

@rule
def _d() -> D:
  return D()

@rule
def _e(c: C, d: D) -> E:
  return E(c, d)

@rule
def _c(b: B) -> C:
  return C(b)

@rule
def _f(c: C, d: D) -> F:
  return F(c, d)

@rule
async def _a(b: B) -> A:
  c = await Get(C, B, b) # we can do this now
  d = await Get(D) # (if D requires no params) -- this won't work right now
  e = await Get(Em Params((C, c), (D, d))) # this is a proposed syntax
  f = await Get(F, Params(C(...), D(...))) # proposed syntax for inline construction
  # ...

This would likely require editing pants.engine.selectors.Get to allow for a variable number of subjects, which would also involve updating the _RuleVisitor in rules.py to understand the new Params() syntax in arguments for Get nodes. There would then likely need to be some small modifications on the rust side where we parse the output of a PyGeneratorResponse (or whatever that struct is called), and also a change where we then generate Gets in order to correctly hook these up into the rule graph. Since we know that this multiple dispatch works already in @rule parameter selectors, it's not likely to require major changes to support this in await Get(...).

[stuhood]

d = yield Get(D) # (if D requires no params) -- this won't work right now

Should probably not work (EDIT: See below)... that should be accomplished via a parameter selector to the rule instead, as that would allow the work to compute D to proceed concurrently before the rule starts running. But having an error message to that effect would be helpful.

EDIT: We've since (Dec 2021) seen that it is useful to be able to make some operations lazy by moving them under conditionals in your @rule body, so this is worth supporting too.

[cosmicexplorer]

Agreed on that, we'd probably want to raise an error message to that effect if we see anything like:

• Get[D]()
• Get[D](Params())

In general though, we won't be able to catch all of these potential concurrency losses with local heuristics. A very silly form of inter-procedural optimization added to the _RuleVisitor would potentially be able to recognize not just zero-param Gets, but also something like:

@rule
def f(b: B, c: C) -> A:
  d = yield Get[D](B, b) # this could just be another selector
  # ...

and suggest instead:

@rule
def f(b: B, c: C, d: D) -> A:
  # ...

Calling it "inter-procedural optimization" is a bit much -- however, we definitely could (in _RuleVisitor):

1. collect the @rule parameter selectors before we descend into the body of f
2. when analyzing a Get() node, check if all of its params are already provided as parameter selectors to f
3. raise an error describing how to modify the rule to move the input into a parameter selector

This would disallow some currently-valid constructions, but we can either add syntax to Get() to get around that, or we could not (I think this is probably better) and the user has to use an intermediate datatype to avoid the check from triggering (which we could describe in a lengthy, descriptive error message).

[stuhood]

In general though, we won't be able to catch all of these potential concurrency losses with local heuristics. A very silly form of inter-procedural optimization added to the _RuleVisitor would potentially be able to recognize not just zero-param Gets, but also something like:

Mm, I don't think that that is worth doing, because it would catch only the very most basic cases... catching anything more advanced would require flow analysis, and I don't think that that is worth it.

EDIT: Oh... or maybe not. But still low priority I think.

[cosmicexplorer]

Low priority but perhaps a high benefit / effort ratio since we'd just be adding an if statement when we process Get() nodes! And it would also happen to cover the case of yield Get(D, Params()) mentioned above automatically, which is cool.

[Eric-Arellano]

I updated the PR description and comments to use the new syntax for rules and typed Get statements.

How feasible is implementing this?

@rule
async def _a(b: B) -> A:
  c = await Get[C](B, b) # we can do this now
  e = await Get[E](Params((C, c), (D, d))) # this is a proposed syntax
  f = await Get[F](Params(C(...), D(...))) # proposed syntax for inline construction
  # ...

The conversation in this thread became focused on how to handle yield Get[A](), which is a separate conversation better saved for a followup, imo.

[Eric-Arellano]

I hit a need tonight for being able to await Get for a singleton.

After a refactor I've done to Pytest coverage, getting the CoverageConfig file is now a singleton rule. But that setup is non-trivial, as we map the transitive closure of every source file -> source root. I don't want to add this as a param to the rule because it's a waste of performance if coverage is not being used.

[stuhood]

#12946 now describes some useful prework for this one, so we should probably consider this one to be blocked on that one.

[stuhood]

An alternative syntax that I quite like:

output = await Get(Output, {input1: Input1, input2: Input2})

This works because Get inputs already must always have eq/hash. Putting the declared type of the value in the dict value also has the benefit of looking like a type-ascription (and is growable, if we want to add more information in the dict value: such as whether a param is optionally consumed).

A downside though is the extra cost of creating a dict (and actually hashing and potentially comparing the values) when using multiple parameters. I think that using multiple parameters will be rare enough (for example in #13682: only at the boundaries between platforms) that this will not be an issue.