Expand engine readme #5759

cosmicexplorer · 2018-04-27T06:43:41Z

Problem

It's not clear how to make rule definitions available to the scheduler, and there's no discussion of the type checking added to datatype() fields in #5723.

Solution

Add a section on datatype fields, typing, and how they interact with the engine.
Add a section on making rules available to the scheduler in and out of unit tests.
Add a section describing types of rules after discussion on this PR.

cosmicexplorer · 2018-04-27T06:49:02Z

The bit about being able to expose rules from backends needs #5747 to be merged before it is true -- I'm working on passing CI there, not a problem, just making a note.

illicitonion

The README changes look great to me, thanks for putting them together!

I'm slightly skeptical of the TypeCheckError change... Can you explain the motivation?

illicitonion · 2018-04-27T10:41:15Z

src/python/pants/util/objects.py

@@ -47,8 +47,8 @@ def datatype(field_decls, superclass_name=None, **kwargs):

  class DataType(namedtuple_cls):
    @classmethod
-    def make_type_error(cls, msg):
-      return TypeCheckError(cls.__name__, msg)
+    def make_type_error(cls, msg, *args, **kwargs):


This seems like a weird change. Why are we now accepting arguments that we'll ignore?

The idea here is that you can pass in e.g. another exception instance as the next positional argument, so that its stacktrace can be preserved. *args and **kwargs are eventually forwarded to the Exception class constructor, unless I'm mistaken, so I don't see how we're ignoring these? I may be missing something quite obvious.

Aah, sorry, I was totally misreading the TypeCheckError constructor!

Part of that might be because I don't know how normal humans format python code, part of that might be because this chaining of exceptions is currently a little ad-hoc/awkward. See the TODO in one of the other exception classes in that file -- there could be a method to make this pattern more concise.

cosmicexplorer · 2018-05-04T01:09:01Z

This needs to be updated a bit after some recent changes to datatype(). I'll rework this and comment again when it is updated.

stuhood

Thanks!

stuhood · 2018-05-04T22:58:22Z

src/python/pants/engine/README.md

+
+1. an `@rule`, which has a single product type and selects its inputs as described above.
+2. a `SingletonRule`, which matches a product type with a value so the type can then be `Select`ed in an `@rule`.
+3. a `RootRule`, which declares a type that can be used as a *subject*, which means it can be provided as an input to a `product_request()` or a `Get` statement.


So, the reason RootRule exists is only for the former case: Scheduler.product_request, et al. Those represent the root of a request. In the absence of RootRules, any subject type that was only involved in a request "at runtime" (ie via product_request), would show up as an an unused or impossible path in the rulegraph, because no @rule could provide that type. RootRule allows for plugging that hole in rulegraph verification by saying: "yea, no other @rule provides this type, but trust me: someone might request one at runtime".

Another potential name for RootRule might be "ParamRule", or something similar... because it's essentially saying that the type represents a sort of "public API entrypoint" via a product_request. I'm fairly confident that RootRule is a necessary API, but the name is not at all final... Suggestions welcome!

You do not need to use RootRule if any Get requests exist in the graph, because the Get requests are statically verified in the graph. We know before runtime that they might be requested.

I pilfered most of that comment into a few sentences in 5032fe9.

ParamRule would be a great name, and thinking it over, the only problem is that RootRule is also a good name, for different reasons.

stuhood · 2018-05-04T23:03:44Z

src/python/pants/engine/README.md

@@ -181,3 +225,108 @@ in the context of scala and mixed scala & java builds.  Twitter spiked on a proj
 a target-level scheduling system scoped to just the jvm compilation tasks.  This bore fruit and
 served as further impetus to get a "tuple-engine" designed and constructed to bring the benefits
 seen in the jvm compilers to the wider pants world of tasks.
+
+## Datatypes in Depth


This section... feels like too much. But maybe it's not obvious to me yet that people will need to be convinced to add types to things... in the presence of @rules? The idea behind @rules is really to force people to use types, whether or not we motivate them to in some other way.

Maybe if you trimmed it down a bit?

That makes sense to me. Would it be better just removed?

I think that incorporating a small amount of "types are good, use them!" into the section that already describes datatype would be good? But probably not much more than a paragraph.

Sorry... it's great writing, but it doesn't feel like the right place.

Made that edit in af466ad.

The published version is currently broken because of a transitive dep bump.

…in depth"

cosmicexplorer · 2018-05-12T09:04:16Z

Un-WIPing this after the most recent commits in response to PR comments.

stuhood

Thanks!

stuhood · 2018-05-13T18:25:43Z

src/python/pants/engine/README.md

+print(x[0])         # 'a string'
+
+# datatype objects can be easily inspected:
+print(x)            # 'FormattedInt(content=a string)'


As a separate change, it looks like we should be using the repr of fields, rather than str. That would result in quotes around string fields in this case.

This is the output of the __str__() method on datatypes (which also displays any type constraints on fields) -- the __repr__() method uses the repr of each field. Does that sound appropriate, or is there a further change to be made?

(if print() uses repr, then i may have taken a shortcut in producing that example code/output -- i really hope not because i hate seeing that)

Looks like that's exactly what happened. I can make a quick followup to fix that.

See #5818 for small followup.

illicitonion approved these changes Apr 27, 2018

View reviewed changes

cosmicexplorer changed the title ~~Expand engine readme~~ [WIP] Expand engine readme May 4, 2018

stuhood reviewed May 4, 2018

View reviewed changes

Use custom fork of grpcio-compiler (pantsbuild#5810)

65b2a07

The published version is currently broken because of a transitive dep bump.

stuhood force-pushed the master branch from b6bb42d to 9e2fdb5 Compare May 11, 2018 23:54

cosmicexplorer force-pushed the expand-engine-readme branch from 33d0edb to b291cac Compare May 12, 2018 08:13

cosmicexplorer added 7 commits May 12, 2018 01:15

pass on args and kwargs to the TypeCheckError ctor

91c4afd

make some small changes to engine/README.md and add a section

826f54b

expand description of datatype fields

970662d

describe how to register rules and what rules you can register

3316140

add "datatypes in depth" section

b291cac

update engine readme for new datatype features and remove "datatypes …

af466ad

…in depth"

describe RootRules from pr discussion

5032fe9

cosmicexplorer changed the title ~~[WIP] Expand engine readme~~ Expand engine readme May 12, 2018

stuhood approved these changes May 13, 2018

View reviewed changes

stuhood merged commit 97db480 into pantsbuild:master May 13, 2018

cosmicexplorer mentioned this pull request May 13, 2018

Add examples of __repr__ for datatypes and add example of type check error #5818

Closed

cosmicexplorer mentioned this pull request Jun 4, 2018

Initial @rules for Options computation via the engine. #5889

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expand engine readme #5759

Expand engine readme #5759

cosmicexplorer commented Apr 27, 2018 •

edited

Loading

cosmicexplorer commented Apr 27, 2018

illicitonion left a comment

illicitonion Apr 27, 2018

cosmicexplorer Apr 27, 2018

illicitonion Apr 27, 2018

cosmicexplorer Apr 27, 2018

cosmicexplorer commented May 4, 2018

stuhood left a comment

stuhood May 4, 2018

cosmicexplorer May 12, 2018

cosmicexplorer May 12, 2018

stuhood May 4, 2018

cosmicexplorer May 7, 2018

stuhood May 7, 2018

cosmicexplorer May 12, 2018

cosmicexplorer commented May 12, 2018

stuhood left a comment

stuhood May 13, 2018

cosmicexplorer May 13, 2018

cosmicexplorer May 13, 2018

cosmicexplorer May 13, 2018

cosmicexplorer May 13, 2018

Expand engine readme #5759

Expand engine readme #5759

Conversation

cosmicexplorer commented Apr 27, 2018 • edited Loading

Problem

Solution

cosmicexplorer commented Apr 27, 2018

illicitonion left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cosmicexplorer commented May 4, 2018

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cosmicexplorer commented May 12, 2018

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cosmicexplorer commented Apr 27, 2018 •

edited

Loading