use tag builder to avoid materializing result #3666
Conversation
To be honest, I am not sure what the specific problem there is, hence I am not yet confident to decided whether this additional complexity is worth it or not. |
| #[allow(unreachable_code)] | ||
| let _type_hinter = || #cls::#fn_ident(#(#unreachable_args),*); | ||
| let tag = (&_pyo3::callback::TagBuilder::new(&_type_hinter)).#tag(); | ||
| tag.convert(py, #call) |
There was a problem hiding this comment.
Wouldn't it be simpler if we tried to wrap call directly, e.g.
let call = move || #call;
let tag = (&_pyo3::callback::TagBuilder::new(&call)).#tag();
tag.convert(py, call());or does that break the lifetime extension as well?
There was a problem hiding this comment.
This is actually exactly what I tried first! 😂
The problem is that #call might contain diverging branches to return errors (e.g. by ?), and the return type of the function call is generally incompatible with this (e.g. because we OkWrap non-results).
Given that we know the full function signature, another option is to just carry the return type through the macro as tokens rather than rely on inference. The only complication here is that any lifetimes in the return type need to be stripped (we used to have code to do that; it wasn't much fun and quite complex, though this hack with _type_hinter is on a similar level of evil).
There was a problem hiding this comment.
This complexity is why I would like to take one step back to actually look at a breaking test case. I mean, __next__ and __anext__ will not take any arguments besides self and the GIL token, right? At least the existing code from our repo seems to compile without the single-expression approach to the conversion.
There was a problem hiding this comment.
Absolutely, sorry, I should have started there - see #3661 (comment), which hopefully you also find gets broken without a single expression.
There was a problem hiding this comment.
Thinking back when we introduced the requirement for single expression form, I would even say that using the alternative from back then (the holder bindings) might actually be preferable as it would avoid complicating the already complicated specialization and would more generally allow us to inspect the return value using arbitrary code, not just specialization.
There was a problem hiding this comment.
Yes, I think you might be right. At the time I was hopeful that was a good simplification, but it's caused us pain a few times now.
There was a problem hiding this comment.
Ok, I will try to resurrect that PR if I can find it and try to get that into mergeable form before continuing in #3661.
|
I think that this isn't needed after #3667, so will close. |
@adamreichold possible solution for https://github.com/PyO3/pyo3/pull/3661/files#r1431111113
I'm unsure what I think of the horrors here, but it solves the problem 😂