Crates with multiple modules / submodules

[davidhewitt]

This is a continuation of PyO3/pyo3#759

If a shared library exposes multiple modules, then it is possible to make all of them importable from Python by (ab)using symlinks, one for each module exposed, all pointing to the built .so.

Would it make sense for maturin to handle this case (presumably with some extra configuration)?

[konstin]

How do other projects such as numpy, pandas, etc. handle this?

[davidhewitt]

That's a great question. I suspect they set up a series of Python files to wrap the c extensions, but I'd need to do further research...

[messense]

I think this isn't hard to support for pure Rust layout since we're already emitting __init__.py, and we can get a list of PyInit__XYZ symbols from the .so file and treat the ones that are not the same as the main module name as submodules and add them to sys.modules.

The limitation is that this will only support submodules like x.y not x.y.z.....

[konstin]

I think the best solution to this is to export one _native module that links all other modules and then create the real module structure that you want in python (e.g. create a foo.py with from _native.foo import *)

[WillAyd]

How do other projects such as numpy, pandas, etc. handle this?

In pandas we build almost all of our extensions into a _libs folder in the source, and every module maps back to an individual shared lib

[Czaki]

Multiple projects (numpy, pandas etc) are collections of multiple .so files.
Sometimes these files are accessed by wrapping python modules, sometimes directly.

It is useful sometimes (for example when freezing aplication)