Runtime type information for objects crossing the Rust–Python boundary #2490

CLOVIS-AI · 2022-07-01T13:32:09Z

Thank you for contributing to pyo3!

Please consider adding the following to your pull request:

an entry in CHANGELOG.md
~~docs to all new functions and / or detail in the guide~~ Documentation will be added when the whole inspect module will be done
tests for all new or changed functions

First step towards #2454.
The goal of this PR is to provide runtime information about Python types for any object that is transferred to/from Python.

Example usage:

println!("a: {}", <(bool, usize, Vec<HashMap<String, u32>>)>::type_input());

a: Tuple[bool, int, Sequence[Mapping[str, int]]]

This PR introduces two new methods:

FromPyObject::type_input: the Python type when the object appears as a parameter to a Rust function,
IntoPy::type_output: the Python type when the object appears as a return value of a Rust function.

Both methods are implemented for the most important Rust types (primitives like usize, collections like HashMap).
The default implementation is to return Any (which is the recommended type when the exact type is missing, and avoids this PR being a breaking change).

In the future, this will allow the inspection module to generate code calling type_input and type_output to delay the Python type generation until runtime, which is required because macros are executed before the Rust compiler resolves the types.

Future work:

Implementing these methods for all other objects (PyDict, hashbrown tables,...); this is fairly easy and may be done on a case-by-case basis when more precise information is needed
Automatically generating the precise type information as part of #[pyclass]

mejrs · 2022-07-02T15:38:47Z

You'll need to rebase on main to make CI work (because of #2489 ) :)

CLOVIS-AI · 2022-07-04T06:49:09Z

@mejrs it's rebased :)

CLOVIS-AI · 2022-07-04T11:49:36Z

Fixed cargo fmt and cargo clippy

davidhewitt · 2022-07-07T07:45:03Z

👍 looks like you were thinking along the same lines for a first step as me with this PR! Nice work!

So I guess the only question here is whether type_input should be used for downcast errors as suggested in #2454? I think I'd like to confirm that use case works before we introduce new public APIs.

Similarly I wonder if users can already use these to include function signatures in docstrings? Might need some macro support. One to think about, can probably implement this in a separate PR if we want it.

Just for completeness, do other use cases for these beyond the introspection API come to mind? I'm sure users will always surprise us 😊

davidhewitt · 2022-07-07T07:49:50Z

Also probably obvious from my other comments, though just wanted to add I'd prefer we merge this after 0.17 is released so that we allow ourselves time to iterate on this further if needed after merge.

CLOVIS-AI · 2022-07-10T09:44:09Z

Just for completeness, do other use cases for these beyond the introspection API come to mind? I'm sure users will always surprise us blush

Well, this PR itself is just the building block of "what is the actual Python type of this type as seen from Python". I expect that this can be used for hundreds of different things 😓. Indeed, debugging/error information seems like a good use case.

So I guess the only question here is whether type_input should be used for downcast errors as suggested in #2454? I think I'd like to confirm that use case works before we introduce new public APIs.

'should' is your decision, but it definitely sounds like it 'could'. I don't think it should be a part of this PR though.

I'd prefer we merge this after 0.17 is released so that we allow ourselves time to iterate on this further if needed after merge.

Do you have an estimate on when that will be? I'm a bit tight on schedule and there's a lot of things to do until .pyi generation is a thing :)

davidhewitt · 2022-07-12T07:23:47Z

Tracking preparation for 0.17 at #2493 - TBH at the moment I'm working on getting those pieces complete whenever I can. I'd like it to be at most a week or two, though it depends how much time I can manage to free up to make progress.

CLOVIS-AI · 2022-08-01T09:15:07Z

Fixed rustdoc & conflicts, rebased on current main, better commit messages.

CLOVIS-AI · 2022-08-01T09:27:09Z

I'm not sure I understand why test fails. What should I do to fix it?

messense · 2022-08-01T09:33:33Z

I'm not sure I understand why test fails. What should I do to fix it?

Run TRYBUILD=overwrite cargo test to update ui tests output, see https://github.com/dtolnay/trybuild#workflow

davidhewitt · 2022-09-06T06:53:23Z

@CLOVIS-AI sorry for the long delay with this caused by the 0.17 release. I'm now able to widen bandwidth again, I propose let's merge this and then we can continue iterating on the stub generation? I think this PR is good as-is, just needs a rebase :)

CLOVIS-AI · 2022-09-06T08:14:50Z

Updated the expected test output & rebased.

mejrs

This looks great, I have just a couple of nits :)

mejrs · 2022-09-06T12:17:20Z

src/inspect/types.rs

+    Any,
+    /// The type `typing.None`.
+    None,
+    /// The type `typing.NoReturn`, which represents functions that never end (similar to `never` in Rust).


Minor nit - such functions can still panic (in rust) or raise an exception (in python)

Suggested change

/// The type `typing.NoReturn`, which represents functions that never end (similar to `never` in Rust).

/// The type `typing.NoReturn`, which represents functions that never return (similar to `never` in Rust).

Done, added clarification in parenthesis

mejrs · 2022-09-06T12:20:04Z

src/inspect/types.rs

+        /// The module this class comes from.
+        module: ModuleName,
+        /// The name of this class, as it appears in a type hint.
+        name: &'static str,


Could someone want to create names at runtime? In that case Cow<'static, str> is more flexible 🤔

Oh, I didn't know it was possible to create a Cow in a const context. That does sound more flexible.

I don't really see a use case for creating names at runtime, but I don't really see a reason to forbid it either.

mejrs · 2022-09-06T12:20:20Z

src/inspect/types.rs

+    /// The type is in the current module: it doesn't need to be imported in this module, but needs to be imported in others.
+    CurrentModule,
+    /// The type is in the specified module.
+    Module(&'static str),


Same as above.

davidhewitt · 2022-09-06T18:35:17Z

Thanks! With apologies will need one final rebase as #2587 has conflicted on the CHANGELOG. (Really need to find some time to work on #2337 soon.)

Also the coverage job is reporting that a lot of new implementations are not covered by tests, it would be much appreciated if we could add a few more. I find it's much better to add coverage at the same time as new APIs wherever possible rather than hope it gets covered later 😄

The TypeInfo structure represents Python types used in hints. Its Display implementation converts it to the exact syntax for it to appear in a type hint.

CLOVIS-AI · 2022-09-06T19:37:56Z

The rebase is done. I do not have the enough time on my hands at the moment to increase the code coverage.

I'm not familiar with GitHub's UI for code coverage so I have some trouble reading it, but it seems to me like it is complaining about all the one-liner functions that simply return a TypeInfo instance? It's not possible for a bug to hide there (they are just Rust structure instanciations).

davidhewitt · 2022-09-06T21:26:09Z

Understood. Given that I'm the source of the delay that caused you to lose your availability, seems reasonable to me to proceed to merge in that case.

CLOVIS-AI mentioned this pull request Jul 1, 2022

Python Interface (.pyi) generation and runtime inspection #2454

Open

7 tasks

CLOVIS-AI force-pushed the type-info branch from bf755eb to 3a874fc Compare July 1, 2022 13:44

CLOVIS-AI force-pushed the type-info branch from 3a874fc to 3d6ca07 Compare July 4, 2022 06:48

CLOVIS-AI force-pushed the type-info branch from 3d6ca07 to b14a3b1 Compare July 4, 2022 11:49

CLOVIS-AI force-pushed the type-info branch from b14a3b1 to a824aa8 Compare July 4, 2022 12:24

CLOVIS-AI force-pushed the type-info branch from a824aa8 to e02305b Compare August 1, 2022 09:14

CLOVIS-AI force-pushed the type-info branch from e02305b to dc4ad08 Compare August 1, 2022 09:17

CLOVIS-AI force-pushed the type-info branch from dc4ad08 to cd90279 Compare September 6, 2022 08:13

mejrs approved these changes Sep 6, 2022

View reviewed changes

CLOVIS-AI force-pushed the type-info branch 3 times, most recently from 0d34f72 to 28cc154 Compare September 6, 2022 17:16

CLOVIS-AI added 5 commits September 6, 2022 21:30

Added the missing 'Unreleased' section in the changelog

13e8efa

Represent Python types

8898bc9

The TypeInfo structure represents Python types used in hints. Its Display implementation converts it to the exact syntax for it to appear in a type hint.

IntoPy and FromPyObject allow the retrieval of the type information

d7c1a29

Declare the Python type of Rust primitives

2ffc4bb

Declare the Python type of Rust collections

0e6ae64

Python types: use Cow instead of bare references

453e355

CLOVIS-AI force-pushed the type-info branch from 28cc154 to 453e355 Compare September 6, 2022 19:30

davidhewitt merged commit ec58ad2 into PyO3:main Sep 6, 2022

CLOVIS-AI deleted the type-info branch February 1, 2024 11:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Runtime type information for objects crossing the Rust–Python boundary #2490

Runtime type information for objects crossing the Rust–Python boundary #2490

CLOVIS-AI commented Jul 1, 2022 •

edited

Loading

mejrs commented Jul 2, 2022

CLOVIS-AI commented Jul 4, 2022

CLOVIS-AI commented Jul 4, 2022

davidhewitt commented Jul 7, 2022

davidhewitt commented Jul 7, 2022 •

edited

Loading

CLOVIS-AI commented Jul 10, 2022

davidhewitt commented Jul 12, 2022

CLOVIS-AI commented Aug 1, 2022 •

edited

Loading

CLOVIS-AI commented Aug 1, 2022

messense commented Aug 1, 2022

davidhewitt commented Sep 6, 2022

CLOVIS-AI commented Sep 6, 2022

mejrs left a comment

mejrs Sep 6, 2022

CLOVIS-AI Sep 6, 2022

mejrs Sep 6, 2022

CLOVIS-AI Sep 6, 2022

CLOVIS-AI Sep 6, 2022

mejrs Sep 6, 2022

CLOVIS-AI Sep 6, 2022

davidhewitt commented Sep 6, 2022

CLOVIS-AI commented Sep 6, 2022

davidhewitt commented Sep 6, 2022

	/// The type `typing.NoReturn`, which represents functions that never end (similar to `never` in Rust).
	/// The type `typing.NoReturn`, which represents functions that never return (similar to `never` in Rust).

Runtime type information for objects crossing the Rust–Python boundary #2490

Runtime type information for objects crossing the Rust–Python boundary #2490

Conversation

CLOVIS-AI commented Jul 1, 2022 • edited Loading

mejrs commented Jul 2, 2022

CLOVIS-AI commented Jul 4, 2022

CLOVIS-AI commented Jul 4, 2022

davidhewitt commented Jul 7, 2022

davidhewitt commented Jul 7, 2022 • edited Loading

CLOVIS-AI commented Jul 10, 2022

davidhewitt commented Jul 12, 2022

CLOVIS-AI commented Aug 1, 2022 • edited Loading

CLOVIS-AI commented Aug 1, 2022

messense commented Aug 1, 2022

davidhewitt commented Sep 6, 2022

CLOVIS-AI commented Sep 6, 2022

mejrs left a comment

Choose a reason for hiding this comment

mejrs Sep 6, 2022

Choose a reason for hiding this comment

CLOVIS-AI Sep 6, 2022

Choose a reason for hiding this comment

mejrs Sep 6, 2022

Choose a reason for hiding this comment

CLOVIS-AI Sep 6, 2022

Choose a reason for hiding this comment

CLOVIS-AI Sep 6, 2022

Choose a reason for hiding this comment

mejrs Sep 6, 2022

Choose a reason for hiding this comment

CLOVIS-AI Sep 6, 2022

Choose a reason for hiding this comment

davidhewitt commented Sep 6, 2022

CLOVIS-AI commented Sep 6, 2022

davidhewitt commented Sep 6, 2022

CLOVIS-AI commented Jul 1, 2022 •

edited

Loading

davidhewitt commented Jul 7, 2022 •

edited

Loading

CLOVIS-AI commented Aug 1, 2022 •

edited

Loading