A new example that shows how to integrate Python plugins that use pyclasses into a Rust app #2873

alexpyattaev · 2023-01-10T22:03:21Z

Example showing integration of a Python plugin into a Rust app while having option to test pyclass based API without the main app. This also illustrates some aspects related to import of Python modules into a Rust app while also having an API module available for the Python code to be able to produce Rust objects.

CI seems to fail on my local machine for reasons unrelated to the example just added:

error: unused macro definition: `check_struct`
  --> pyo3-ffi-check/src/main.rs:13:18
   |
13 |     macro_rules! check_struct {
   |                  ^^^^^^^^^^^^
   |

mejrs

Thanks for the PR! I have some thoughts:

This PR uses the word "plugin" a lot. What does it mean? If you mean "extension module", you should call it that, not "plugin". We don't call it that anywhere, so this is very likely to confuse readers, and it confused me too until I realized you meant extension module (I think).

I'm also not sure this is the correct approach. Personally I'd update the existing examples - just put this at the bottom of their lib.rs:

#[cfg(test)]
mod tests{
    #[test] 
    fn test_module(){
         // make module
         // call module etc
    }
}

This could be added to maturin new too.

There are two advantages to this:

When users create their first python module, they won't have to figure out how to test it. It will just be there
Putting it in lib.rs (or anywhere else in the crate) allows users to access crate private things. main.rs does not let you do this.

What do you think?

examples/plugin/README.md

mejrs · 2023-01-10T23:03:59Z

examples/plugin/src/main.rs

+    pyo3::append_to_inittab!(pylib_module);
+    //spawn runtime
+    pyo3::prepare_freethreaded_python();
+    //import path for python
+    let path = Path::new("./");
+    //do useful work
+    Python::with_gil(|py| {
+        //add the current directory to import path of Python (do not use this in production!)
+        let syspath: &PyList = py.import("sys")?.getattr("path")?.extract()?;
+        syspath.insert(0, &path)?;
+        println!("Import path is: {:?}", syspath);
+
+        // Now we can load our plugin.py file
+        let plugin = PyModule::import(py, "plugin")?;


You can just call the initialization function, and avoid messing with sys.path entirely:

let module = PyModule::new(py, "plugin")?; plugin_api::plugin_api(py, &module)?;

No, this will not work as expected, since plugin.py will not land on the import path. And then, if plugin.py has any dependencies of its own, they will fail to import which will confuse the user until they figure out they need to mess with import path for it to work as you'd expect Python to work. I will actually add some deps to plugin.py to illustrate this.

examples/plugin/src/main.rs

examples/README.md

examples/plugin/src/main.rs

alexpyattaev · 2023-01-11T08:37:56Z

Thanks for the PR!
You are welcome, I figured if it took me 2 days to figure this out, it may be worth sharing. Also PyO3 is awesome =)

This PR uses the word "plugin" a lot. What does it mean? If you mean "extension module", you should call it that, not "plugin". We don't call it that anywhere, so this is very likely to confuse readers, and it confused me too until I realized you meant extension module (I think).
By plugin I mean a literal plugin you'd put in a bigger application. Specifically, if you are building something like blender video editor, you would want to have the "main" application in rust, but then also have some scripting API as well. For that to work, you would normally want to define a bunch of your application-specific interface classes, such that Python plugins can construct complex objects (in example it is the Gadget pyclass) and then pass them into the Rust code. It is much more efficient than parsing complex data structures from Python classes, and also allows you to test the interface properly before integration into the main app.

I'm also not sure this is the correct approach. Personally I'd update the existing examples - just put this at the bottom of their lib.rs:
#[cfg(test)]
mod tests{
    #[test] 
    fn test_module(){
         // make module
         // call module etc
    }
}
This could be added to maturin new too.

There are two advantages to this:
* When users create their first python module, they won't have to figure out how to test it. It will just be there

* Putting it in lib.rs (or anywhere else in the crate) allows users to access _crate_ private things. `main.rs` does not let you do this.
What do you think?

This will only work with custom build configuration (as in my build_extmod.sh) to make the actual plugin. With the way linking is set up with Python you can not have both tests and extension modules working at the same time. Though it is a "larger issue" in some sense.

adamreichold · 2023-01-11T08:46:40Z

Depending on what you guys decide on w.r.t. the plug-ins-versus-better-testing angle, I think it should be stressed that while putting the plug-in API in a separate crate and testing plug-ins this way is possible, it is not possible to build the plug-in API/host separately from the main application, i.e. it currently needs to be linked statically into the main application to keep their ABI compatible.

alexpyattaev · 2023-01-11T10:14:31Z

That is very much true, and is a fundamental limitation of Rust toolchain at the moment. Not sure if such remarks belong in the example though, but feel free to suggest where they would go.

mejrs

Sorry for not getting back to you sooner 😅. I have some comments remaining.

Anyway, I wonder what the others think E.g. @davidhewitt ?

examples/plugin/README.md

examples/README.md

alexpyattaev · 2023-01-25T14:02:03Z

No rush here=) Thanks for the suggestions, I think your description is quite well fitting, and indeed way shorter.

davidhewitt

Yep, this looks excellent to me, thanks very much to both of you for putting this example together!

I'll add the template files and merge when I get a moment 👍

examples/plugin/README.md

…to a Rust app while having option to test API without the main app

davidhewitt · 2023-01-27T08:35:14Z

bors r+

2873: A new example that shows how to integrate Python plugins that use pyclasses into a Rust app r=davidhewitt a=alexpyattaev Example showing integration of a Python plugin into a Rust app while having option to test pyclass based API without the main app. This also illustrates some aspects related to import of Python modules into a Rust app while also having an API module available for the Python code to be able to produce Rust objects. CI seems to fail on my local machine for reasons unrelated to the example just added: ``` error: unused macro definition: `check_struct` --> pyo3-ffi-check/src/main.rs:13:18 | 13 | macro_rules! check_struct { | ^^^^^^^^^^^^ | ``` 2889: Added support for PyErr_WriteUnraisable r=davidhewitt a=mitsuhiko Fixes #2884 2923: hygiene: fix `#[pymethods(crate = "...")]` r=davidhewitt a=davidhewitt Got to the bottom of the hygiene issue in test of #2914 Turns out that `#[pymethods] #[pyo3(crate = "...")]` works, but `#[pymethods(crate = "...")]` was ignoring the argument. Added a tweak to fix this and a snippet in the hygiene test (which fails on `main`). 2924: remove unneeded into_iter calls r=davidhewitt a=davidhewitt Clippy complaining about these to me this morning locally. Co-authored-by: Alex Pyattaev <alex.pyattaev@gmail.com> Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com> Co-authored-by: Armin Ronacher <armin.ronacher@active-4.com>

bors · 2023-01-27T09:31:06Z

Build failed (retrying...):

conclusion

2873: A new example that shows how to integrate Python plugins that use pyclasses into a Rust app r=davidhewitt a=alexpyattaev Example showing integration of a Python plugin into a Rust app while having option to test pyclass based API without the main app. This also illustrates some aspects related to import of Python modules into a Rust app while also having an API module available for the Python code to be able to produce Rust objects. CI seems to fail on my local machine for reasons unrelated to the example just added: ``` error: unused macro definition: `check_struct` --> pyo3-ffi-check/src/main.rs:13:18 | 13 | macro_rules! check_struct { | ^^^^^^^^^^^^ | ``` 2889: Added support for PyErr_WriteUnraisable r=davidhewitt a=mitsuhiko Fixes #2884 Co-authored-by: Alex Pyattaev <alex.pyattaev@gmail.com> Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com> Co-authored-by: Armin Ronacher <armin.ronacher@active-4.com>

bors · 2023-01-27T10:46:06Z

Build failed (retrying...):

conclusion

2873: A new example that shows how to integrate Python plugins that use pyclasses into a Rust app r=davidhewitt a=alexpyattaev Example showing integration of a Python plugin into a Rust app while having option to test pyclass based API without the main app. This also illustrates some aspects related to import of Python modules into a Rust app while also having an API module available for the Python code to be able to produce Rust objects. CI seems to fail on my local machine for reasons unrelated to the example just added: ``` error: unused macro definition: `check_struct` --> pyo3-ffi-check/src/main.rs:13:18 | 13 | macro_rules! check_struct { | ^^^^^^^^^^^^ | ``` Co-authored-by: Alex Pyattaev <alex.pyattaev@gmail.com> Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>

bors · 2023-01-27T12:00:43Z

Build failed:

conclusion

alexpyattaev · 2023-01-27T17:02:14Z

Not sure what bors is unhappy with... Any ideas?

There seems to be some issue fetching python 3.11 on Darwin from "https://github.com/actions/python-versions/releases/download/3.11.0-3730290910/python-3.11.0-darwin-x64.tar.gz", is that it?

davidhewitt · 2023-01-27T21:12:43Z

bors retry

davidhewitt · 2023-01-27T21:14:08Z

There's a compatibility issue with our test suite on 3.11.1 (#2817) so we're using 3.11.0 for testing at the moment (3.11.2 should fix this). I think we just hit a rate limit with all the bors tests running!

bors · 2023-01-27T22:01:42Z

Build succeeded:

conclusion

mejrs reviewed Jan 10, 2023

View reviewed changes

mejrs reviewed Jan 24, 2023

View reviewed changes

examples/plugin/README.md Outdated Show resolved Hide resolved

examples/README.md Outdated Show resolved Hide resolved

davidhewitt reviewed Jan 26, 2023

View reviewed changes

examples/plugin/README.md Show resolved Hide resolved

davidhewitt added the CI-skip-changelog Skip checking changelog entry label Jan 27, 2023

alexpyattaev and others added 2 commits January 27, 2023 08:34

added a plugin example that shows how to integrate a Python plugin in…

2228f58

…to a Rust app while having option to test API without the main app

add cargo-generate template for plugin example

d7601a4

davidhewitt force-pushed the main branch from 5c8613b to d7601a4 Compare January 27, 2023 08:34

bors bot merged commit efb8a12 into PyO3:main Jan 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

A new example that shows how to integrate Python plugins that use pyclasses into a Rust app #2873

A new example that shows how to integrate Python plugins that use pyclasses into a Rust app #2873

alexpyattaev commented Jan 10, 2023

mejrs left a comment •

edited

Loading

mejrs Jan 10, 2023

alexpyattaev Jan 11, 2023

alexpyattaev commented Jan 11, 2023

adamreichold commented Jan 11, 2023

alexpyattaev commented Jan 11, 2023

mejrs left a comment

alexpyattaev commented Jan 25, 2023

davidhewitt left a comment

davidhewitt commented Jan 27, 2023

bors bot commented Jan 27, 2023

bors bot commented Jan 27, 2023

bors bot commented Jan 27, 2023

alexpyattaev commented Jan 27, 2023 •

edited

Loading

davidhewitt commented Jan 27, 2023

davidhewitt commented Jan 27, 2023

bors bot commented Jan 27, 2023

A new example that shows how to integrate Python plugins that use pyclasses into a Rust app #2873

A new example that shows how to integrate Python plugins that use pyclasses into a Rust app #2873

Conversation

alexpyattaev commented Jan 10, 2023

mejrs left a comment • edited Loading

Choose a reason for hiding this comment

mejrs Jan 10, 2023

Choose a reason for hiding this comment

alexpyattaev Jan 11, 2023

Choose a reason for hiding this comment

alexpyattaev commented Jan 11, 2023

adamreichold commented Jan 11, 2023

alexpyattaev commented Jan 11, 2023

mejrs left a comment

Choose a reason for hiding this comment

alexpyattaev commented Jan 25, 2023

davidhewitt left a comment

Choose a reason for hiding this comment

davidhewitt commented Jan 27, 2023

bors bot commented Jan 27, 2023

bors bot commented Jan 27, 2023

bors bot commented Jan 27, 2023

alexpyattaev commented Jan 27, 2023 • edited Loading

davidhewitt commented Jan 27, 2023

davidhewitt commented Jan 27, 2023

bors bot commented Jan 27, 2023

mejrs left a comment •

edited

Loading

alexpyattaev commented Jan 27, 2023 •

edited

Loading