Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ModuleNotFoundError: No module named 'supermodule.submodule'; 'supermodule' is not a package #759

Open
tcztzy opened this issue Feb 8, 2020 · 18 comments

Comments

@tcztzy
Copy link

tcztzy commented Feb 8, 2020

🐛 Bug Reports

I am working for writing Python extension, but facing import submodule error.

ModuleNotFoundError: No module named 'supermodule.submodule'; 'supermodule' is not a package

What do I expected

Import submodule like usual Python package.

🌍 Environment

$ uname -a
Linux xxxxxx 4.19.84-microsoft-standard #1 SMP Wed Nov 13 11:44:37 UTC 2019 x86_64 GNU/Linux
  • Your python version:
$ python -V
Python 3.8.1
  • How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?:
$ pyenv virtualenv system xxxxxx
  • Your rust version (rustc --version):
rustc 1.43.0-nightly (c9290dcee 2020-02-04)
  • Are you using the latest pyo3 version? Have you tried using latest master (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")?
    Yes.

💥 Reproducing

Please provide a minimal working example. This means both the rust code and the python.

Please also write what exact flags are required to reproduce your results.
src/lib.rs

use pyo3::prelude::*;
use pyo3::{wrap_pyfunction, wrap_pymodule};
use pyo3::types::IntoPyDict;

#[pyfunction]
fn subfunction() -> String {
    "Subfunction".to_string()
}

#[pymodule]
fn submodule(_py: Python, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pyfunction!(subfunction))?;
    Ok(())
}

#[pymodule]
fn supermodule(_py: Python, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pymodule!(submodule))?;
    Ok(())
}

Cargo.toml

[package]
name = "supermodule"
version = "0.1.0"
authors = ["Tang Ziya <tcztzy@gmail.com>"]
edition = "2018"

[package.metadata.maturin]
classifier = [
    "Development Status :: 3 - Alpha",
    "Programming Language :: Python",
    "Programming Language :: Rust",
]

[lib]
name = "supermodule"
crate-type = ["cdylib", "rlib"]

[features]
default = []

[dependencies]
pyo3 = { git = "https://github.com/PyO3/pyo3", features = ["extension-module"] }

[dev-dependencies]
$ maturin develop
$ python -c "from supermodule.submodule import subfunction"
ModuleNotFoundError: No module named 'supermodule.submodule'; 'supermodule' is not a package
@davidhewitt
Copy link
Member

This is unfortunately a flaw in the way Python imports modules from native extensions. I think if you run import supermodule first, then from supermodule.submodule import subfunction will succeed.

@waymost
Copy link

waymost commented Feb 10, 2020

I've run into this same issue with version 0.8.5 on Mac and no combination of imports that I've tried has worked, including the above suggestion.

@davidhewitt
Copy link
Member

Thanks for the note. I must have mis-remembered that.

Based on stack overflow the solution looks like you might be able to create a symlink to the main library which has the name of the submodule: https://stackoverflow.com/questions/48706842/python-3-x-c-extension-module-and-submodule

@sunjay
Copy link

sunjay commented Jul 6, 2020

You can fix this in a somewhat hacky way by adding the module to sys.modules manually. From your Rust code, this can look like:

#[pymodule]
fn supermodule(py: Python, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pymodule!(submodule))?;

    py.run("\
import sys
sys.modules['supermodule.submodule'] = submodule
    ", None, Some(module.dict()))?;

    Ok(())
}

The documentation claims that sys.modules is writable, so this approach should not break anytime soon. The only caveat is that while this works when executed by a Python interpreter, it does not work with static analysis tools like pylint.

After a lot of trial and error, I am fairly convinced that the only "fully robust" solution that will also make static analysis tools happy is to have your entire module hierarchy represented in the file system. Whether you do this with Python files or with separate extension modules is up to you. I agree with the comments in PyO3/maturin#266 that we should look into how numpy, pandas, etc. handle this and see if there is a better way.

It would be great if in the meantime PyO3 could emit code to modify sys.modules automatically so from supermodule.submodule import ... would work out of the box.

@davidhewitt
Copy link
Member

A solution for this came up in #1517 (comment)

We could try to support this pattern internally in PyO3 (or at least add documentation).

@chris-laplante
Copy link
Contributor

Somewhat unrelated but maybe this will help someone. In my particular case I had a module on-disk with a number of submodules. I only wanted to import a subset of the submodules. Tried many solutions (e.g. PyModule::from_code, py_run! workarounds, etc) but this is what finally worked for me:

let sys: &PyModule = py.import("sys").unwrap();
let syspath: &PyList = sys.getattr("path").unwrap().try_into().unwrap();

syspath
    .insert(0, "/path/to/python/library/")
    .unwrap();

let my_module = PyModule::new(py, "my_module")?;
let my_mod_submodule_1 = py.import("my_module.submodule1").unwrap();
let my_mod_submodule_2 = py.import("my_module.submodule2").unwrap();
my_module.setattr("submodule1", my_mod_submodule_1)?;
my_module.setattr("submodule2", my_mod_submodule_2)?;

@vxgmichel
Copy link
Contributor

For the record, an alternative to the py.run! solution is:

#[pymodule]
fn supermodule(py: Python, m: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pymodule!(submodule))?;

    py.import("sys")?
        .getattr("modules")?
        .set_item("supermodule.submodule", submodule)?;

    Ok(())
}

@davidhewitt
Copy link
Member

NB it looks like submodules may also have complexity with things like pickling: #2469 (comment)

@hameer-spire
Copy link

For the record, an alternative to the py.run! solution is:

#[pymodule]
fn supermodule(py: Python, m: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pymodule!(submodule))?;

    py.import("sys")?
        .getattr("modules")?
        .set_item("supermodule.submodule", submodule)?;

    Ok(())
}

The modern incantation to do this is something like:

#[pymodule]
fn supermodule(py: Python, m: &PyModule) -> PyResult<()> {
    let submodule = pyo3::wrap_pymodule!(submodule);
    m.add_wrapped(submodule)?;
    py.import("sys")?
        .getattr("modules")?
        .set_item("supermodule.submodule", submodule(py))?;
    Ok(())
}

@jdiggans-twist
Copy link

jdiggans-twist commented Sep 26, 2022

I'm encountering this same issue and when using the solution suggested in #759 (comment) (via copy/paste).

After maturin develop and attempting import supermodule.submodule in a REPL I'm now getting:

thread '<unnamed>' panicked at 'failed to wrap pymodule: PyErr { type: <class 'ImportError'>, value: ImportError('PyO3 modules may only be initialized once per interpreter process'), traceback: None }', supermodule/src/lib.rs:432:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "python3.9/site-packages/supermodule/__init__.py", line 1, in <module>
    from .supermodule import *
pyo3_runtime.PanicException: failed to wrap pymodule: PyErr { type: <class 'ImportError'>, value: ImportError('PyO3 modules may only be initialized once per interpreter process'), traceback: None }

I don't see any issues in this repo on this particular error (multiple initialization) - any idea where I've gone wrong?

(I see from https://github.com/PyO3/pyo3/blob/7bdc504252a2f972ba3490c44249b202a4ce6180/guide/src/migration.md#each-pymodule-can-now-only-be-initialized-once-per-process that this is new behavior as of the 28 August release of 0.17 but it's not clear from that writeup what 'the ability to initialize a #[pymodule] more than once in the same process' means - i.e. I am just starting a REPL and importing the package one time to trigger the error).

@jdiggans-twist
Copy link

If I back down to pyo3 v0.16.6, the import works without error so it's something new to 0.17.x.

@davidhewitt
Copy link
Member

@jdiggans-twist I've created a separate issue #2644 to discuss.

@ariebovenberg
Copy link

One addition: the __name__ of the submodules isn't set correctly either.

Here is the complete solution I use for this:

#[pymodule]
fn supermodule(py: Python, m: &PyModule) -> PyResult<()> {
    let submodule = create_my_submodule()?;  // function that creates the &PyModule
    m.add_submodule(submodule)?;
    py.import("sys")?
        .getattr("modules")?
        .set_item("supermodule.submodule", submodule)?;
    // needs to be set *after* `add_submodule()`
    submodule.setattr("__name__", "supermodule.submodule")?;
    Ok(())
}

@Anexen
Copy link

Anexen commented Nov 15, 2023

Here is my solution that correctly sets __name__ for both submodule and the function:

#[pymodule]
fn supermodule(py: Python, m: &PyModule) -> PyResult<()> {
    let child_module = PyModule::new(py, "supermodule.submodule")?;
    submodule(py, child_module)?;
    m.add("submodule", child_module)?;
    py.import("sys")?.getattr("modules")?.set_item("supermodule.submodule", child_module)?;
    Ok(())
}

@nleroy917
Copy link

Hi guys,

I was able to use all the above to get submodules going for my package. Thank you! Something I couldn't figure out was adding type stubs to the package after this. I followed the tutorial, but it seems highly specific to modules that don't have submodules. Is there a way to add type's to projects that include submodules without creating the python source folder?

Thanks!

@davidhewitt
Copy link
Member

@nleroy917 maybe #3591 (reply in thread) helps you?

@nleroy917
Copy link

@davidhewitt Thank you for pointing me there, that seems to be the exact problem I was having. In the meantime, I got something working after reading this in the documentation:

As we now specify our own package content, we have to provide the init.py file, so the folder is treated as a package and we can import things from it. We can always use the same content that Maturin creates for us if we do not specify a Python source folder. For PyO3 bindings it would be:

from .my_project import *

It's not perfect, and I am repeating myself a lot, but it works and is sufficient for now. I sincerely appreciate your work on this project! It's become an indispensable tool in my workflow

@squidfunk
Copy link

FYI: The workaround posted in #759 (comment) also works for the experimental declarative API (which is awesome, btw!) ☺️

#[pymodule]
mod my_module {
    use super::*;

    #[pymodule]
    mod my_submodule {
        use super::*;

        /// Hack: workaround for https://github.com/PyO3/pyo3/issues/759
        #[pymodule_init]
        fn init(m: &Bound<'_, PyModule>) -> PyResult<()> {
            Python::with_gil(|py| {
                py.import_bound("sys")?
                    .getattr("modules")?
                    .set_item("my_module.my_submodule", m)
            })
        }
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests