--pyargs cannot locate namespace packages installed in editable mode

[taschini]

The --pyargs command-line switch cannot locate libraries within a namespace package that are installed in editable mode.

How to reproduce

1. In a fresh virtual environment install the attached pinned requirements:

   $ virtualenv env
   $ source env/bin/activate
   $ pip install -r pinned_requirements.txt
   

   This will install three libraries (more.basicauth, more.chameleon, more.forwarded) in editable mode, together with their dependencies.

   These libraries are all defined within the more namespace package.

2. Run

   $ py.test --pyargs more.chamaleon

This is the output I get:

$ py.test --pyargs more.chamaleon
==================================================================== test session starts =====================================================================
platform darwin -- Python 2.7.11, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
rootdir:[…]/pytest_bug, inifile: 
plugins: cov-2.2.1

================================================================ no tests ran in 0.00 seconds ================================================================
ERROR: file or package not found: more.chamaleon

Expected output

I expect that py.test invocation to produce similar output as

$ py.test env/src/more.chameleon/more
==================================================================== test session starts =====================================================================
platform darwin -- Python 2.7.11, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
rootdir: […]/pytest_bug/env/src/more.chameleon, inifile: 
plugins: cov-2.2.1
collected 5 items 

env/src/more.chameleon/more/chameleon/tests/test_chameleon.py .....

================================================================== 5 passed in 0.51 seconds ==================================================================

[RonnyPfannschmidt]

i think this might be duplicate of #478

do you agree?

[taschini]

Yes, I actually just noticed that too.

[RonnyPfannschmidt]

also note, that this is a bug in setuptools namespaces vs setup.py develop

[taschini]

I cannot judge whether the difference in behavior between setuptools and setup.py develop is a bug. Do you happen to have a link to the related issue on the tracker of setuptools?

[RonnyPfannschmidt]

pypa/pip#3
pypa/setuptools#250

[taschini]

Thank you, quite an interesting read. Would you mind if I give a look at _tryconvertpyarg to see if there is a way to work around some of these issues?

[taschini]

also note, that this is a bug in setuptools namespaces vs setup.py develop

Actually, the --pyargs command-line switch cannot locate libraries within a namespace package that have been installed with setup.py develop either:

# Create fresh environment:
virtualenv --no-site-packages env
source env/bin/activate

# Get the sources:
mkdir src
git clone https://github.com/morepath/more.chameleon src/more.chameleon
git clone https://github.com/morepath/more.basicauth src/more.basicauth
git clone https://github.com/morepath/more.forwarded src/more.forwarded

# Install the libraries in "develop" mode:
for k in src/*; do (cd $k; python setup.py develop); done

# Install the test requirements:
pip install pytest WebTest pytest-cov

# verify that the libraries are installed
python -c 'import more.chameleon, more.basicauth, more.forwarded'

# Try to run py.test
py.test --pyargs more.chameleon

That still fails with

==================================================================== test session starts =====================================================================
platform darwin -- Python 2.7.11, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
rootdir: […]/pytest_bug2, inifile: 
plugins: cov-2.2.1

================================================================ no tests ran in 0.00 seconds ================================================================
ERROR: file or package not found: more.chameleon

[RonnyPfannschmidt]

i think the problem is finding namespace package path properly - py.test currently does not support that kind of discovery

it needs active support and logic i think

[taschini]

As far as I'm concerned, #1597 solves this issue, which I think it can now be closed.

[nicoddemus]

Oh yeah, thanks!

[dimpase]

I am still seeing this issue on Python 3.9 and pytest 7.1.3, probably #1597 has not addressed the full extent of the problem. We have a mix of namespace and "normal" Python modules in https://github.com/sagemath/sage
and are trying to switch to pytest, cf e.g. https://trac.sagemath.org/ticket/33826
Still seeing, with --pyargs sage.misc

ERROR: module or package not found: sage.misc (missing __init__.py?)

sort of errors, while

Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from sage.misc import banner
>>> 

is really there. Is it because it tries to import the parent? (which is also a namespace package) ?

[RonnyPfannschmidt]

Pytest is unaware of pep420, nobody fixed that so far

[dimpase]

Pytest is unaware of pep420, nobody fixed that so far

If this was documented, it would have saved us unnecessary banging of heads on keyboard.

[nicoddemus]

@dimpase indeed we should have this documented somewhere, to avoid users wasting time on this.

Perhaps we can improve the error message:

ERROR: pytest cannot import module or package 'sage.misc'
Possible reasons:
- Missing an __init__.py file
- Namespace package (not supported)
- Not reachable via PYTHONPATH
See <link to docs> for more information.

[dimpase]

should there be an issue on making pytest PEP 420-aware?

[The-Compiler]

@dimpase see the very first comment above: #478

[nicoddemus]

Created #10569.