Improve Get errors for unhashable or mismatched types #7502

stuhood · 2019-04-04T22:31:10Z

Problem

The error for unhashable types used as Get params is unfortunate, because the call to identify fails and returns a null pointer to rust, which triggers an error... somewhere else. See #7499.

Solution

As suggested in #7499, have extern_generator_send directly identify Get params, meaning that failures to hash take advantage of the fact that extern_generator_send is already fallible. This avoids a significant number of calls back to python to identify Values (an optimization discussed in #7318 and #7114).

Result

Improved usability. Fixes #7499.

stuhood · 2019-04-04T22:31:23Z

Commits are useful to review independently.

src/rust/engine/src/externs.rs

src/rust/engine/src/rule_graph.rs

illicitonion · 2019-04-05T08:59:41Z

tests/python/pants_test/engine/test_scheduler.py

@@ -195,7 +195,7 @@ def test_union_rules(self):
    self.assertTrue(isinstance(a, A))
    # Fails due to no union relationship from A -> UnionBase.
    expected_msg = """\
-Exception: WithDeps(Inner(InnerEntry { params: {UnionWrapper}, rule: Task(Task { product: A, clause: [Select { product: UnionWrapper }], gets: [Get { product: A, subject: UnionA }, Get { product: A, subject: UnionB }], func: a_union_test(), cacheable: true }) })) did not declare a dependency on JustGet(Get { product: A, subject: A })
+Exception: (A, [UnionWrapper], [Get(A, UnionA), Get(A, UnionB)], a_union_test()) for UnionWrapper did not declare a dependency on Get(A, A)


I find this less informative than the previous error. What are all of these comma-separated parts of a string? As a rule-author, I don't want to have to dig into the rust code in Pants to work out what these things all are... (The old message wasn't great either, but at least it had keys associated with each value...)

So, this is currently the exact output for rule-graph visualization and tracing, but they should clearly evolve. This change mostly just got us into alignment between an error case and those other cases.

If we think it is in scope for this change, I can look at it here... it is mostly a python signature, and we could roughly shape it that way. The only issue is the Gets and other used Params. A strawperson refactoring of this example that doesn't lose any information might be:

def a_union_test(UnionWrapper) -> A { for UnionWrapper; Get(A, UnionA), Get(A, UnionB) }

or

def a_union_test -> A { for UnionWrapper; Get(A, UnionA), Get(A, UnionB) }

... I'm going to open a ticket to discuss this.

But, in the meantime, I'd like to merge this output as-is in order to make it obvious which cases exist, and where (based on these tests now actually using Display).

I really like the stack trace in particular with rust type names as it lets me reconstruct the rust part in my head much more easily, which is useful for specifically an error trace. This seems like a large UI change wrapped in another functional change. I would really prefer that we solve the output problem in a followup PR addressing #7509 and considering the difference between what we want to display in stack traces vs what we want to display in pretty graphs, while scoping this one to the identify() and related changes.

I've dropped these changes from the PR and will wait to see what happens with #7509.

src/rust/engine/src/externs.rs

benjyw

Looks good as far as my limited understanding of this code goes.

cosmicexplorer

Some thoughts on comments to add in places mostly. Being able to use rust enums more naturally is very exciting!

cosmicexplorer · 2019-04-07T00:00:14Z

src/python/pants/engine/native.py

+            TypeId(c.to_id(res.product)),
+            c.to_value(res.subject),
+            c.identify(res.subject),
+          )


This is a fantastic change. Is this relying on mostly the rust C representation of enums (with tag and (in this case) get fields), or is there some cffi magic going on to allow addressing unions in this deeply ergonomic way? Is there a link we could add documenting why this works to access our rust enums?

In this case, I actually think we don't want to do this, as we probably want to colocate parsing the python response and producing the FFI object that represents it (so the below code is needless complexity for now). But I wrote this comment out anyway so I can link to it in #7367:

Could this part be moved into a helper method? Something like:

@memoized_property def _generator_response_tags_enum(self): class _Tags(enum([self._lib.Get, self._lib.GetMulti, self._lib.Broke, self._lib.Throw])): pass return _Tags def _set_generator_response(self, response, c, input_res, tag): attr, value = self._generator_response_tags_enum(tag).resolve_for_enum_variant({ self._lib.Get: lambda: ('get', ( TypeId(c.to_id(input_res.product)), c.to_value(input_res.subject), c.identify(input_res.subject), ), # ... response.tag = tag setattr(response, attr, value) }) def extern_generator_send(...): # ... if isinstance(res, Get): self._set_generator_response(response, c, res, self._lib.Get) # ... return response[0]

This is a fantastic change. Is this relying on mostly the rust C representation of enums (with tag and (in this case) get fields), or is there some cffi magic going on to allow addressing unions in this deeply ergonomic way? Is there a link we could add documenting why this works to access our rust enums?

This is mostly relying on the C API that is generated by cbindgen. Kudos on integrating that.

cosmicexplorer · 2019-04-07T00:27:23Z

src/python/pants/engine/native.py

-    ffi = getattr(self._ffi_module, 'ffi')
-    _FFISpecification(ffi).register_cffi_externs()
-    return ffi
+    return getattr(self._ffi_module, 'ffi')


Considering that register_cffi_externs() modifies the ffi object and not the lib object, I would think we would want to keep that call in this method to avoid spooky mutation of the ffi object happening whenever we first happen to access the lib object.

Unfortunately, constructing the _FFISpecification inside of self.ffi triggered infinite recursion when I attempted to feed it self.lib (which required self.ffi, etc, etc).

cosmicexplorer · 2019-04-07T00:28:00Z

src/rust/engine/src/externs.rs

+  GetMulti(TypeIdBuffer, HandleBuffer, IdentBuffer),
+  // NB: Broke not Break because C keyword.
+  Broke(Handle),
+  Throw(Handle),


Much better!! Thank you for taking the time to figure out how to do this with cffi, it makes the code leagues more readable. It also obviates the need for that TODO without additional complexity in building the engine!

cosmicexplorer · 2019-04-07T00:32:04Z

src/rust/engine/src/externs.rs

+    with_vec(self.ptr(), self.len() as usize, |vec| Self::lift(&vec[0]))
+  }
+}
+


This is all really great! I really like that we can manipulate python collections in this structured, efficient, and safe way.

cosmicexplorer · 2019-04-07T00:36:31Z

src/rust/engine/src/externs.rs

+      "Expected exactly 1 item in Buffer, but had: {}",
+      self.len()
+    );
+    with_vec(self.ptr(), self.len() as usize, |vec| Self::lift(&vec[0]))


Any reason this doesn't delegate to .to_vec()? Like:

fn unwrap_one(&self) -> Output { assert_equal!(self.len(), 1, "Expected exactly 1 item in Buffer, but had: {}", self.len()); self.to_vec()[0] }

with_vec doesn't actually "create" a vec: instead, it "views" some memory as a Vec via:

unsafe { Vec::from_raw_parts(c_ptr, c_len, c_len) }

That means that by avoiding calling to_vec(), we avoid actually copying the buffer containing the one item.

cosmicexplorer · 2019-04-07T00:39:46Z

src/rust/engine/src/externs.rs

+  idents_ptr: *mut Ident,
+  idents_len: u64,
+  // A Handle to hold the underlying array alive.
+  handle_: Handle,


I know this part isn't new, but could we add a comment on one of these fields explaining why handle_ needs to exist? Is it so python doesn't gc the pointer?

That's correct. Since CI is green here, I'm going to go ahead and merge this one, but can revisit.

cosmicexplorer · 2019-04-07T00:42:21Z

src/rust/engine/src/externs.rs

+///
+/// The response from a call to extern_generator_send. Gets include Idents for their Handles
+/// in order to avoid roundtripping to intern them, and to eagerly trigger errors for unhashable
+/// types on the python side where possible.


Good explanation.

cosmicexplorer · 2019-04-07T00:48:36Z

tests/python/pants_test/engine/test_scheduler.py

@@ -213,6 +222,12 @@ def test_get_type_match_failure(self):
      # `a_typecheck_fail_test` above expects `wrapper.inner` to be a `B`.
      self.scheduler.product_request(A, [Params(TypeCheckFailWrapper(A()))])

+  def test_unhashable_failure(self):
+    """Test that unhashable Get(...) params result in a structured error."""
+    expected_msg = """TypeError: unhashable type: 'list'"""


I would like for this message to check more specifically which object has the unhashable type, but I can't see how to do that easily, so this is definitely specific enough for now.

illicitonion

Looks great, thanks!

stuhood added 3 commits April 4, 2019 13:58

Extract a RawBuffer trait for reuse in an upcoming new buffer.

e9c53be

Add a test that unhashable Get params do not trigger panics.

701d799

Identify all Get params before returning them in extern_generator_send.

b5e083d

stuhood requested review from benjyw, illicitonion and cosmicexplorer April 4, 2019 22:31

illicitonion reviewed Apr 5, 2019

View reviewed changes

benjyw approved these changes Apr 5, 2019

View reviewed changes

stuhood force-pushed the stuhood/improve-get-errors branch 2 times, most recently from 8cd695e to c5a9244 Compare April 6, 2019 15:15

stuhood added 2 commits April 6, 2019 08:28

Take advantage of cbindgen's ability to generate tagged unions!

9520bec

Improve type parameter names.

b44073c

stuhood force-pushed the stuhood/improve-get-errors branch from c5a9244 to b44073c Compare April 6, 2019 15:28

cosmicexplorer approved these changes Apr 7, 2019

View reviewed changes

cosmicexplorer mentioned this pull request Apr 7, 2019

share enum definitions across rust and python #7367

Closed

3 tasks

illicitonion approved these changes Apr 8, 2019

View reviewed changes

stuhood merged commit d0c670f into pantsbuild:master Apr 10, 2019

stuhood deleted the stuhood/improve-get-errors branch April 10, 2019 06:04

stuhood mentioned this pull request May 16, 2019

Use @union to make the v2 test runner generic #7661

Merged

stuhood mentioned this pull request May 29, 2019

Render the @union description when a target type is not testable with v2 #7819

Closed

stuhood mentioned this pull request Sep 27, 2019

Type mismatches in await Gets give unclear error message #8349

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve Get errors for unhashable or mismatched types #7502

Improve Get errors for unhashable or mismatched types #7502

stuhood commented Apr 4, 2019 •

edited

Loading

stuhood commented Apr 4, 2019

illicitonion Apr 5, 2019

stuhood Apr 5, 2019 •

edited

Loading

stuhood Apr 5, 2019

stuhood Apr 5, 2019

stuhood Apr 6, 2019

cosmicexplorer Apr 6, 2019

stuhood Apr 6, 2019

benjyw left a comment

cosmicexplorer left a comment

cosmicexplorer Apr 7, 2019

cosmicexplorer Apr 7, 2019 •

edited

Loading

stuhood Apr 10, 2019

cosmicexplorer Apr 7, 2019

stuhood Apr 10, 2019

cosmicexplorer Apr 7, 2019

cosmicexplorer Apr 7, 2019

cosmicexplorer Apr 7, 2019

stuhood Apr 10, 2019

cosmicexplorer Apr 7, 2019

stuhood Apr 10, 2019

cosmicexplorer Apr 7, 2019

cosmicexplorer Apr 7, 2019

illicitonion left a comment

Improve Get errors for unhashable or mismatched types #7502

Improve Get errors for unhashable or mismatched types #7502

Conversation

stuhood commented Apr 4, 2019 • edited Loading

Problem

Solution

Result

stuhood commented Apr 4, 2019

Choose a reason for hiding this comment

stuhood Apr 5, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjyw left a comment

Choose a reason for hiding this comment

cosmicexplorer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cosmicexplorer Apr 7, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

illicitonion left a comment

Choose a reason for hiding this comment

stuhood commented Apr 4, 2019 •

edited

Loading

stuhood Apr 5, 2019 •

edited

Loading

cosmicexplorer Apr 7, 2019 •

edited

Loading