Please warn if no rust is exported in a mixed rust/python module

[gilescope]

When I create a mixed rust and python project, it seems one needs to re-export the rust into the python.

It would be great if maturin warned if nothing was imported from the rust module - i.e. the user thought it happened automagically.

[madhavajay]

@gilescope I am having a similar problem. This took me some messing around.

In the end the key was inside the init.py in the "myproj" python src dir:

# re-export rust myproj module at this level
from .myproj import *

# export vanilla_python.py functions as vanilla_python module
from . import vanilla_python

Now I can import myproj.rust_ffi and also myproj.vanilla_python on the same package level.

[madhavajay]

So while this is letting me execute the full path to functions, I have an issue with a nested module.

#[pymodule]
fn myproj(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_wrapped(wrap_pymodule!(message))?;
    Ok(())
}

#[pymodule]
fn submod(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_wrapped(wrap_pyfunction!(run_class_method_message))?;
    Ok(())
}

With this I can do:

import myproj
myproj.submod.run_class_method_message()

And I can do:

from myproj import submod

But I can't do:

from myproj.submod import run_class_method_message

I get the error:

ModuleNotFoundError: No module named 'myproj.submod'

Am I doing something wrong?

[programmerjake]

But I can't do:

from myproj.submod import run_class_method_message

I get the error:

ModuleNotFoundError: No module named 'myproj.submod'

Am I doing something wrong?

no, that's a limitation in how CPython loads extension modules.

[madhavajay]

Okay, so I have actually found a way to fix this using some python duct tape.

The fix below allows you to import normal vanilla python libs like:

from myproj.vanilla import abc

As well as rust modules at depth like:

from myproj.suba.subc import xyz

Your directory structure:

./
├── Cargo.toml
├── myproj. <-- this is your mixed python src
│   ├── init.py
│   ├── import_fixer.py
│   ├── suba
│   │   ├── init.py
│   │   └── subc
│   │   └── init.py
│   ├── subb
│   │   └── init.py
│   └── vanilla.py
└── src

Your rust modules:

#[pymodule]
fn myproj(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_wrapped(wrap_pymodule!(suba))?;
    m.add_wrapped(wrap_pymodule!(subb))?;
    Ok(())
}

#[pymodule]
fn suba(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_wrapped(wrap_pyfunction!(geh))?;
    m.add_wrapped(wrap_pymodule!(subc))?;
    Ok(())
}

#[pymodule]
fn subb(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_wrapped(wrap_pyfunction!(def))?;
    Ok(())
}

#[pymodule]
fn subc(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_wrapped(wrap_pyfunction!(xyz))?;
    Ok(())
}

Inside myproj/init.py:

# here you can import the normal kind of vanilla python __init__ that you want
from . import vanilla.py

For each submodule from rust you want, add a directory and a init.py

In each init.py put:

import importlib

# find and run the import fixer
package_name = __name__.split(".")[0]
import_fixer = importlib.import_module(".import_fixer", package=package_name)
import_fixer.fix_imports(locals(), __file__)

Finally inside import_fixer.py put:

# -*- coding: utf-8 -*-
# Author: github.com/madhavajay
"""Fixes pyo3 mixed modules for import in python"""

import importlib
import os
from typing import Dict, Any, List

# gets the name of the top level module / package
package_name = __name__.split(".")[0] # myproj

# convert the subdirs from "package_name" into a list of sub module names
def get_module_name_from_init_path(path: str) -> List[str]:
    _, path_and_file = os.path.splitdrive(os.path.dirname(path))
    module_path = path_and_file.split(package_name)[-1]
    parts = module_path.split(os.path.sep)[1:]
    return parts


# step through the main base module from rust at myproj.myproj and unpack each level
def unpack_module_from_parts(module: Any, module_parts: List[str]) -> Any:
    for part in module_parts:
        module = getattr(module, part)
    return module


# take the local scope of the caller and populate it with the correct properties
def fix_imports(lcl: Dict[str, Any], init_file_path: str, debug: bool = False) -> None:
    # rust library is available as package_name.package_name
    import_string = f".{package_name}"
    base_module = importlib.import_module(import_string, package=package_name)
    module_parts = get_module_name_from_init_path(init_file_path)
    submodule = unpack_module_from_parts(base_module, module_parts)
    if debug:
        module_path = ".".join(module_parts)
        print(f"Parsed module_name: {module_path} from: {init_file_path}")

    # re-export functions
    keys = ["builtin_function_or_method", "module"]
    for k in dir(submodule):
        if type(getattr(submodule, k)).__name__ in keys:
            if debug:
                print(f"Loading: {submodule}.{k}")
            lcl[k] = getattr(submodule, k)

Which begs the question, could this be autogenerated and included in pyo3?

[konstin]

When I create a mixed rust and python project, it seems one needs to re-export the rust into the python.

That is not necessary. In the pyo3_mixed example you can e.g. do from pyo3_mixed import pyo3_mixed; pyo3_mixed.get_21(). However as @programmerjake said, cpython can't import from a submodule of a native module directly.

[madhavajay]

@konstin, what are your thoughts on my solution above?
It provides the ability to mix both vanilla python and native modules / functions on the same import syntax and path with minimal effort. PyO3 could have a build flag which prepends this to any "/module/subdir/init.py" files inside the mixed python source providing these re-exports in a way that the user thinks logically and allowing any existing custom code in init to overwrite the locals() keys if desired.