Importing mixed Rust/Python project fails when structured as recommended

[ArniDagur]

The README of this project says the following:

Mixed rust/python projects

To create a mixed rust/python project, create a folder with your module name (i.e. lib.name in Cargo.toml) next to your Cargo.toml and add your python sources there:

my-project
├── Cargo.toml
├── my_project
│   ├── __init__.py
│   └── bar.py
├── Readme.md
└── src
    └── lib.rs

maturin will add the native extension as a module in your python folder. When using develop, maturin will copy the native library and for cffi also the glue code to your python folder. You should add those files to your gitignore.

With cffi you can do from .my_project import lib and then use lib.my_native_function, with pyo3/rust-cpython you can directly from .my_project import my_native_function.

Example layout with pyo3 after maturin develop:

my-project
├── Cargo.toml
├── my_project
│   ├── __init__.py
│   ├── bar.py
│   └── my_project.cpython-36m-x86_64-linux-gnu.so
├── Readme.md
└── src
    └── lib.rs

This is the structure I'm using for python-adblock. The project is written entirely in Rust, but I'm using a mixed Rust/Python project in order to include Python type stubs. I have an __init__.py which imports the neccesary symbols from the native module.

However, some of my users have reported that the imports inside __init__.py fail as follows:

>>> import adblock
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/app/adblock/__init__.py", line 1, in <module>
    from .adblock import __version__, Engine, FilterSet, BlockerResult, UrlSpecificResources
ModuleNotFoundError: No module named 'adblock.adblock'

or

>>> import adblock
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/app/adblock/__init__.py", line 1, in <module>
    from adblock.adblock import __version__, Engine, FilterSet, BlockerResult, UrlSpecificResources
ModuleNotFoundError: No module named 'adblock.adblock'

See ArniDagur/python-adblock#17. As you can see, the result is the same whether I use from .adblock import ... or from adblock.adblock import .... I have only managed to reproduce this when the package is installed system wide (pip3 install .) and the current working directory is the root of my project (git repository).

Does anyone know what is going on? Am I doing something different from what is described in the README.md, or is the advice in the README.md wrong in some way?

[messense]

pip isn't used, but I've added python3 -m sysconfig for these archs.

https://gitlab.alpinelinux.org/omni/aports/-/jobs/391397

https://gitlab.alpinelinux.org/omni/aports/-/jobs/391398

armv7: Platform: "linux-armv8l"
armhf: Platform: "linux-armv8l"

Interesting. I think we are expecting Platform: "linux-armv7l"

I wonder what's is Rust target triple on these platforms.

See also randombit/botan#1543 (comment)

[stusmall]

I'm running into this same issue, but I might have a shorter path to reproducing it. I'm including as much detail as possible because I might be doing something weird in here. I'm not an experienced python dev and I'm kinda winging it as I go.

1. Check out maturin @ revision d531e49 (head of main when I pulled)
2. Install with cargo install -f --path .
3. Goto the cffi-mixed test crate with cd test-crates/cffi-mixed
4. Create and activate a venv. python3 -m venv venv; source venv/bin/activate
5. Install tox pip install tox
6. Run tox tox --skip-missing-interpreters (I'm just using the system interpreter to simply things):

(venv) ➜  cffi-mixed git:(main) tox --skip-missing-interpreters
.package create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.package
.package installdeps: maturin>=0.12,<0.13
py36 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py36
SKIPPED: InterpreterNotFound: python3.6
py37 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py37
SKIPPED: InterpreterNotFound: python3.7
py38 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py38
py38 installdeps: pytest
py38 inst: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.tmp/package/1/cffi_mixed-0.1.0.tar.gz
py38 installed: attrs==21.4.0,cffi==1.15.0,cffi-mixed @ file:///home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.tmp/package/1/cffi_mixed-0.1.0.tar.gz,iniconfig==1.1.1,packaging==21.3,pluggy==1.0.0,py==1.11.0,pycparser==2.21,pyparsing==3.0.7,pytest==7.0.1,tomli==2.0.1
py38 run-test-pre: PYTHONHASHSEED='1058416854'
py38 run-test: commands[0] | pytest
================================= test session starts ==================================
platform linux -- Python 3.8.12, pytest-7.0.1, pluggy-1.0.0
cachedir: test-crates/cffi-mixed/.tox/py38/.pytest_cache
rootdir: /home/stusmall/Workspace/maturin
collected 0 items / 1 error

======================================== ERRORS ========================================
______________ ERROR collecting test-crates/cffi-mixed/test_cffi_mixed.py ______________
ImportError while importing test module '/home/stusmall/Workspace/maturin/test-crates/cffi-mixed/test_cffi_mixed.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../../.pyenv/versions/3.8.12/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test_cffi_mixed.py:3: in <module>
    import cffi_mixed
cffi_mixed/__init__.py:1: in <module>
    from .cffi_mixed import ffi, lib
E   ModuleNotFoundError: No module named 'cffi_mixed.cffi_mixed'
=============================== short test summary info ================================
ERROR test_cffi_mixed.py
!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!
=================================== 1 error in 0.08s ===================================
ERROR: InvocationError for command /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py38/bin/pytest (exited with code 2)
_______________________________________ summary ________________________________________
SKIPPED:  py36: InterpreterNotFound: python3.6
SKIPPED:  py37: InterpreterNotFound: python3.7
ERROR:   py38: commands failed

For some extra version info just in case it helps for reproducing the issue:

(venv) ➜  cffi-mixed git:(main) python3 --version
Python 3.8.12
(venv) ➜  cffi-mixed git:(main) pip --version
pip 21.1.1 from /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/venv/lib/python3.8/site-packages/pip (python 3.8)
(venv) ➜  cffi-mixed git:(main) tox --version
3.24.5 imported from /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/venv/lib/python3.8/site-packages/tox/__init__.py

What is interesting is that if I run maturin develop before running tox --skip-missing-interpreters then the test works.

env) ➜  cffi-mixed git:(main) maturin develop
🍹 Building a mixed python/rust project
🐍 Using CPython 3.8 at /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/venv/bin/python to generate the cffi bindings
📡 Using build options bindings from pyproject.toml
Collecting cffi
  Using cached cffi-1.15.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (446 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Installing collected packages: pycparser, cffi
Successfully installed cffi-1.15.0 pycparser-2.21
WARNING: You are using pip version 21.1.1; however, version 22.0.3 is available.
You should consider upgrading via the '/home/stusmall/Workspace/maturin/test-crates/cffi-mixed/venv/bin/python -m pip install --upgrade pip' command.
   Compiling cffi-mixed v0.1.0 (/home/stusmall/Workspace/maturin/test-crates/cffi-mixed)
    Finished dev [unoptimized + debuginfo] target(s) in 0.55s
📦 Built wheel to /tmp/.tmpsizYQW/cffi_mixed-0.1.0-py3-none-linux_x86_64.whl
🛠  Installed cffi-mixed-0.1.0
(venv) ➜  cffi-mixed git:(main) tox --skip-missing-interpreters
py36 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py36
SKIPPED: InterpreterNotFound: python3.6
py37 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py37
SKIPPED: InterpreterNotFound: python3.7
py38 inst-nodeps: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.tmp/package/1/cffi_mixed-0.1.0.tar.gz
py38 installed: attrs==21.4.0,cffi==1.15.0,cffi-mixed @ file:///home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.tmp/package/1/cffi_mixed-0.1.0.tar.gz,iniconfig==1.1.1,packaging==21.3,pluggy==1.0.0,py==1.11.0,pycparser==2.21,pyparsing==3.0.7,pytest==7.0.1,tomli==2.0.1
py38 run-test-pre: PYTHONHASHSEED='2429669310'
py38 run-test: commands[0] | pytest
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.8.12, pytest-7.0.1, pluggy-1.0.0
cachedir: test-crates/cffi-mixed/.tox/py38/.pytest_cache
rootdir: /home/stusmall/Workspace/maturin
collected 3 items

test_cffi_mixed.py ...                                                                                                                                                                                        [100%]

================================================================================================= 3 passed in 0.02s =================================================================================================
______________________________________________________________________________________________________ summary ______________________________________________________________________________________________________
SKIPPED:  py36: InterpreterNotFound: python3.6
SKIPPED:  py37: InterpreterNotFound: python3.7
  py38: commands succeeded
  congratulations :)

So before I go further, I want to repeat, I'm not a python dev. So this might be missing the point of the initial bug or misunderstanding how modules work. But it looks to me that when from .cffi_mixed import ffi, lib is run instead of importing from cffi_mixed the wheel it is opening the directory on disk cff_mixed. When this happens before maturin develop is run, it doesn't have native.so in place, but it does afterwards and so the tests pass

[messense]

@stusmall Sorry for the late reply, I think your hypothesis is correct. We've changed the default project template to include a python folder in #855 to avoid such issue.