pyximport alike tool for easier edit/compile/test cycle

[messense]

I have had similar successes with PyO3. You can rapidly prototype in Python, leveraging spaghetti code that Rust is rightly concerned with and also Python's stronger ecosystem in the space of ML, and then dip into Rust as the bottlenecks become clear.

Has anyone seen attempts for lower overhead PyO3 integration? Creating a new Rust module each time and the maturin develop loops can be frustrating when massive gains might come from a few dozen lines of Rust.

Perhaps I just haven't seen the best workflow for small Rust additions to a Python program using PyO3? I always appreciated Cython's low overhead auto-compiling (i.e. if you import primes it'll search for prime.pyx and auto-compile + cache before import) and would love the same for Rust + PyO3.

Originally posted in: https://www.reddit.com/r/rust/comments/qjxwni/comment/hitpi1z/?utm_source=share&utm_medium=web2x&context=3

[Smerity]

I'm not certain how I might be able to help but I'd be excited to try to. Python and Rust are a perfect combination for me and this would help oxidize far larger parts of my Python programs!

[messense]

Some design notes:

• We could start with a import hook implementation which can do compile, install, load on the fly when import xyz is executed where xyz is a maturin based project in cwd. pyximport is a good material for reference.

• Then for the xyz.rsx single file idea, we could generate a maturin based project for it assuming it's using PyO3 or detect what bindings it uses and do the similar compile, install, load processes. I don't quite like the .rsx extension name though, maybe .rs is enough.

[messense]

• We could start with a import hook implementation which can do compile, install, load on the fly when import xyz is executed where xyz is a maturin based project in cwd. pyximport is a good material for reference.

I have a preliminary prototype in #729

[messense]

https://maturin.rs/develop.html#import-hook

[ofirnk]

maturin.rs/develop.html#import-hook

@messense I think it's worth adding a recomendation to run

cargo watch -s maturin develop

without that it's not clear (to me?) how to keep the .so object in latest version

[davidhewitt]

Does the import-hook automatically rebuild if you launch a new Python interpreter? Need to be careful about replacing the .so while the Python interpreter is running, you'll probably get weird crashes like segfaults and bus errors if function addresses change after recompilation and the interpreter then loads a new segment of the .so.

[ofirnk]

Does the import-hook automatically rebuild if you launch a new Python interpreter? Need to be careful about replacing the .so while the Python interpreter is running, you'll probably get weird crashes like segfaults and bus errors if function addresses change after recompilation and the interpreter then loads a new segment of the .so.

it does not get re-built when re-running import_hook.install() after a code change

from what I saw in the implementation - as long as the import (of pyo3_pure from the example) is successful - it doesn't get rebuilt.

but - if a new method is added - let's call it multi_as_string - it won't be available.

if an explicit import exists ex. from pyo3_pure import multi_as_string so find_spec is invoked - but doesn't trigger a new build since multi_as_string is not a cargo project (or single .rs ).

Possible workaround is to iterate on

for module_name in reverse(fullname.split(".")):
    # continue if not a cargo module
    break;

we can continue the discussion here on in a new issue - don't wanna spam if it's doesn't seem relevant