Design v2 Target API

[stuhood]

The current state of the "target" API that is consumed by @rules is very scattered, mostly due to a need for backward compatibility, and the incremental introduction of v2 to the rest of the codebase.

Now that @console_rules have begun to be used more widely, and this API (that was originally only designed to act as adaptors to allow pantsd to consume @rules for BUILD file parsing) is beginning to be used by end users writing @rules, it's time for another round of design.

A previous bit of design was done in https://docs.google.com/document/d/102EFbwk6cpM9-_4ZSYhMQYA0zL1UKbXzW-DCd8KsFeg/edit?usp=sharing, but some of the considerations there are stale: see comments on this ticket, and example usages of these APIs in the v2 python test runner and in v2 list for more information.

---

A quick explanation of the status quo as of March 2018 April 2019 August 2019 ~~February 2020:

None of these APIs are public yet, and they're all going to need to change... they came out of the engine experiment, and they support more features than we need. So we need to figure out which bits to prune.

Currently the hierarchy of construction (bottom to top) is:

• HydratedStruct - A Struct that has been deserialized from a BUILD file. Currently this type supports inlining other Structs via address references: see this example ...where because the configurations field is expecting to receive Structs, the inline address reference there is expanded and inlined into
• TargetAdaptor - A Struct subclass representing a target. So in the case of expanding a legacy build graph, the above step operates on concrete subclasses of TargetAdaptor. A TargetAdaptor still has sources represented as a PathGlobs object in a SourcesField wrapper: ie, the source globs haven't been expanded.
• HydratedTarget - An object containing enough information to actually construct the legacy BuildGraph interface (with an EagerFilesetWithSpec object representing fully expanded and fingerprinted globs). As described on Declare v2 products on v1 Tasks #4769 and in the document linked above, this is not the interface we want to expose to anyone, as it is too granular to avoid expanding sources if the usecase doesn't require them.
• LegacyHydratedTarget - a HydratedTarget but with a BuildFileAddress rather than an Address, which is needed for V1 to work properly.

[stuhood]

Relates to #4641: people are running into the partial coverage of the previous target API in 1.3.0.

[stuhood]

Relates to #3991 ... it's possible that with v1 build file parsing "gone" in master, that approach might be more feasible than before.

[stuhood]

We should reevaluate this post #5580: it's possible that @jsirois's idea to make Structs inheritable/mergeable/etc with typed fields is totally excellent once we have easier @rule APIs.

[stuhood]

@jsirois : I had another thought here around how to do extensibility without subclassing (which remained an open question in the doc linked in the description).

I think that a somewhat natural way to constrain the types that are legal in a particular field while still allowing for extension, would be to have "named TypeConstraints unions". As an example: for determining which Configuration/Field types are legal on Structs, rule registration could mutate or aggregate a TypeConstraint called "LegalFieldTypes" (or "legal_field_types", depending). @rules which needed to consume "all legal field types" would use that constraint by name... meanwhile, the @rule graph would still need to have @rules installed that could satisfy each of the unioned members of the named constraint... so if you had a yield Get(HydratedField, LegalFieldTypes, x) call, each member of the LegalFieldTypes union would need to have an @rule that could provide a HydratedField.

This "named type unions" concept could also apply to other plugin extension points: @rule authors would add some possible type to a union, and then add an @rule to satisfy it.

[stuhood]

This "named type unions" concept could also apply to other plugin extension points: @rule authors would add some possible type to a union, and then add an @rule to satisfy it.

It occurs to me that in order for consuming rules to consume the output of this type of union, they'd need to be oblivious to the union members... and in order for that to happen and still be useful, the union members would probably all need a shared parent class. So... this might actually look a bit like subtyping with a closed universe of subclasses?

[stuhood]

This is related to #6449, because the implementation of dep inference relies on implementers being able to inject new Field types and rules to provide them.

I think that all of the sketches described above have begun to solidify in my mind around "subtyping with a closed universe of subclasses". But that will require a bit of refactoring of how unions work now, and I don't think we should start that until #5788 is completed.

So while it is a bit of a yakshave, I'm going to consider this blocked by #5788.

[cosmicexplorer]

I'm probably missing something, but:

This "named type unions" concept could also apply to other plugin extension points: @rule authors would add some possible type to a union, and then add an @rule to satisfy it.

I don't understand what the named union approach provides -- the requirement of a closed universe of possible types of input is already achieved by requiring a unique path from subject to product in the rule graph for every subj/prod pair requested.

It occurs to me that in order for consuming rules to consume the output of this type of union, they'd need to be oblivious to the union members... and in order for that to happen and still be useful, the union members would probably all need a shared parent class. So... this might actually look a bit like subtyping with a closed universe of subclasses?

Rules consuming some product A are already oblivious to the path from the subject to A? if we're thinking about requiring some common parent class anyway, it seems like instead we could make A just a normal datatype containing all of the info that rules consuming A would need, and rely on the fact that there must be a single unique path from subject to A for polymorphism? In the case of A = Field, I would think a rule author can make whatever kind of type F they want to represent their fields, and then define an @rule from F to A.

each member of the LegalFieldTypes union would need to have an @rule that could provide a HydratedField

This seems like exactly the guarantee we already have by requiring a unique path from product to subject. I'm looking at the doc linked up top and I cannot understand what the explicit, closed union concept provides over the implicit closed union that the rule graph already provides.

I have no problem with the idea, I just think requiring a unique path in the rule graph is a very simple model to use and debug and I would prefer to use the polymorphism from that already than require anyone to start subclassing anything. The property "this type subclasses this other type" means "this type provides some known set of methods/attrs" -- it seems like that can be achieved by using a datatype A which provides the methods/attrs you require, and then requiring that there be a unique path from whatever type the rule author can think of to A. I don't see what utility further narrowing that has as of now, it seems to make the model more complex. I could very easily be missing something obvious.

[stuhood]

Re-reading my own comments, I think that I flip-flopped a few times between discussing "output" types (like HydratedField) and "input" types (like Field / SourcesField / BundleField). It's not the case that the input types need to be subclasses of any particular type, because the @rule needn't actually consume them, and can just pass them from place to place.

Given that confusion, I think I'll probably refresh the design doc for another round of review.

[stuhood]

I don't understand what the named union approach provides -- the requirement of a closed universe of possible types of input is already achieved by requiring a unique path from subject to product in the rule graph for every subj/prod pair requested.

@cosmicexplorer : It provides extensibility into existing @rules that might not be aware of all of the types of a union. Concretely, it allows replacing:

if isinstance(x, SourcesField):
  hf = yield Get(HydratedField, SourcesField, x)
elif isinstance(x, BundleField):
  hf = yield Get(HydratedField, BundleField, x)
else:
  ..

with:

hf = yield Get(HydratedField, FieldsUnionType, x)

The former has a hardcoded list of legal fields, and so you cannot add additional Field types (or @rules to satisfy them) without editing that rule.

[stuhood]

I rebased the AggregationRule concept out of #6170 before landing it, because it doesn't quite fit this usecase, but was in the same area. Pushed it over here for reference: 25322b1

[stuhood]

EDIT: This question is resolved for now: see #8368!

---

From a discussion with @benjyw in https://pantsbuild.slack.com/archives/C0D7TNJHL/p1569606120265000, my feeling is that there might be a fundamental decision around how to use @union; do we (either, I think?):

1. allow for "runtime" typechecks against a @union? IE, give @rules the ability to check whether an object is a member of a @union?
2. add some sort of "default"ing facility to @union, so that if something isn't a member of the type, a different @rule runs.

---

The fundamental difference between these choices in a case like

pants/src/python/pants/rules/core/test.py

Lines 71 to 73 in 44d4ab5

	# NB: This has the effect of "casting" a TargetAdaptor to a member of the TestTarget union. If the
	# TargetAdaptor is not a member of the union, it will fail at runtime with a useful error message.
	result = yield Get(TestResult, TestTarget, target.adaptor)

is that the former would mean "don't request a TestResult if we can't test something", and the latter would mean: "you can always get a TestResult, but sometimes it will be a noop provided by a default implementation".

I'm leaning toward the former, because it's less magical, but it all relates to how strongly typed targets are in general.

[benjyw]

Agreed re the former probably being nicer.

[illicitonion]

I think we definitely want runtime checks; @console_rules should each have explicit boilerplate to exclude non-matching targets, because they should all probably handle that differently. We can probably provide some helper functions to make it less boilerplatey as the need arises.

[cosmicexplorer]

We can probably provide some helper functions to make it less boilerplatey as the need arises.

I think this is the approach we should take for all v2 functionality! It seems to have allowed us to make more progress on this sort of API question recently.

While I would really like to eventually figure out a way to express UnionRule-y relations in a way which somehow doesn't allow for the case where the union relation doesn't exist, since as of right now it seems like there's no clear way to do that, I think choosing specifically to allow the explicit check for now, with an eye to some way to clean that up later, sounds like a great approach.

[stuhood]

Closing the loop: #4535 (comment) was resolved in #8368!

[stuhood]

This is also related to #7022: it's possible that target definitions (and macros) should themselves be implemented as @rules, which would allow them to declare their dependencies.

[stuhood]

Attaching notes from a design discussion today.

The summary of these is that we think that there are roughly two levels to be concerned with:

1. Parsing: ie, how do I define the universe of legal things that can be parsed from BUILD files into typed Target objects. The notes suggest that:
   • we should have two base classes: Field and Target.
     • A Target has a classmember that is a union of its legal Field types (we could maybe generate this synthetically like we do for Goal.Options), and would have a getter target.get(Field) for the instance of a field (or a default? or fields themselves have defaults, so it is infallible?)
     • Field might have a required class method Field.default(Target) that would create a default instance of the Field for that Target type (aka, "default Sources (glob) for PythonLibrary").
2. Codegen: ie, how do I go from a typed Target object of one type (ie ThriftTarget) to the type that a caller/dependee wants (ie, ScalaTarget or Classpath).

Both of these can be worked on roughly independently: the only issue with starting Codegen before Parsing is that the types that you would consume as input to codegenny-conversions would be our existing awkward placeholder types (adaptors/etc).

Additionally, we thought that dependency inference is likely to "just work" almost regardless of design. And we thought that it was not crystal clear "how many" unions we need to have for the lint/fmt case, and that more experimentation was necessary there.

[cosmicexplorer]

we should have two base classes: Field and Target.

I think that conceptually, we could absolutely use a @union for each of these and just create UnionRules. We could even go further, and make each of these into the @dataclasses Field and Target, where "adding a new Field type A" would then mean "defining a type A and writing a @ruleset mapping A -> Field".

In practice, I think a base class just seems like a better API, so I am definitely still behind using that approach. I think all of the above are actually reducible to each other anyway, so I don't think we would be losing anything or introducing any inconsistency by taking this approach now.

Super hype about the result of this discussion today!

[Eric-Arellano]

Closing now that the Target API has been in usage for about a month, including for codegen.

There's a bit of remaining work:

• Finish documentation at https://pants.readme.io/docs/target-api.
• Deprecate TargetAdaptor in favor of the Target API.

But the actual API is implemented and used in the wild.