Attempt to clarify the confusion regarding __init__ files and unique test names

[nicoddemus]

Fix #529

[nicoddemus]

Took this out because it seemed like out of place in the tox section.

[coveralls]

Coverage remained the same at 92.71% when pulling 581857a on nicoddemus:init-files-docs into 906b40f on pytest-dev:master.

[coveralls]

Coverage remained the same at 92.71% when pulling 581857a on nicoddemus:init-files-docs into 906b40f on pytest-dev:master.

[philtay]

I'd add another "test layout". The src/ one, which solves a lot of problems. You can also add __init__.py files in your tests/ directory and avoid the "unique names" limit.

https://blog.ionelmc.ro/2014/05/25/python-packaging/
https://hynek.me/articles/testing-packaging/

[nicoddemus]

While I completely agree with the points in @ionelmc's post, I'm not sure it is a good fit for this particular section of the docs. The section is titled Choosing a test layout and it is all about choosing either embedding the tests as part of the package or putting them in a separate folder which is not part of your package; having your root package at the root of the repository or at a src sub-folder is another issue which is not related to where you put your tests.

But again, I'm not sure, so I would like to hear more opinions (@ionelmc please do chime in if you want).

[ionelmc]

The src-layout is more concerned with where the code is, rather than where the tests are. It's true that my blog post touches the subject of test location but that's rather a problem of me tackling too many problems in one article :)

[philtay]

The "Tests outside application code" section says:

Note that using this scheme your test files must have unique names,
because pytest will import them as top-level modules since there
are no packages to derive a full package name from.

This statement is not (entirely) true. You can easily have non-unique names if you add __init__.py files (unique basenames are a severe limitation if not a bug). But if you do so then you'll have the problem of testing against local vs installed package. A solution is the src layout. In other words you can solve the problem either making the tests non-importable or making the package(s) you want to test non-importable. This should be in the docs. IMO it's the problem root.

In the "Tox" section I'd mention the changedir option that, together with the Pytest import-mode=append option can solve this problem too.

[nicoddemus]

This statement is not (entirely) true. You can easily have non-unique names if you add init.py files (unique basenames are a severe limitation if not a bug).

You are right, it should be mentioned that adding __init__.py files here will solve the issue of duplicate test module names.

But if you do so then you'll have the problem of testing against local vs installed package. In other words you can solve the problem either making the tests non-importable or making the package(s) you want to test non-importable.

Why is that, could you clarify this bit?

In the "Tox" section I'd mention the changedir option that, together with the Pytest import-mode=append option can solve this problem too.

I'm not aware of this solution as well, could you explain this in more details? Thanks!

[philtay]

Why is that, could you clarify this bit?

Suppose that you need to use duplicate test names and you create the following layout:

setup.py
tox.ini
mypkg/
    __init__.py
tests/
    __init__.py

When you run pytest the root directory (the one which contains setup.py) is prepended to sys.path and, as a consequence, the local mypkg takes precedence over the installed one. A solution is to remove the __init__.py file from the tests directory. In this way the root directory won't be prepended anymore, but you lose the possibility of having non-unique test names.

A better approach is:

setup.py
tox.ini
src/
    mypkg/
        __init__.py
tests/
    __init__.py

Now it doesn't matter anymore if you have an __init__.py file in tests/. The root directory will still be prepended to sys.path but you don't care because you can only import the installed mypkg. Nesting mypkg/ in src/ makes it "non-importable" from the root directory.

I'm not aware of this solution as well, could you explain this in more details? Thanks!

Tox put the current directory (i.e. the root directory) in sys.path by default. If you configure it like

[testenv]
changedir = {envtmpdir}

then the directory in sys.path will be a non-relevant one (in this case the environment temporary directory). But you still have the problem that an __init__.py file in tests/ will make pytest prepending the root directory. If you invoke it with the option --import-mode=append then the root directory will be appended (instead of prepended). In this case the installed package will have the precedence and will be tested. Basically you can safely have __init__.py files in tests/ even without adopting a src/ layout.

[nicoddemus]

Many thanks for the detailed explanation, I have clear picture now of all the relevant issues here. I will work on adding more info to the docs tomorrow.

Thanks again!

[nicoddemus]

This section became more complicated than I hoped for. I'm considering just recommending straight away to add __init__.py files except to the tests directory, which seems to solve all issues discussed here. This would greatly simplify this section.

@ionelmc @philtay thoughts?

[philtay]

It doesn't work. pytest uses the full directory path as import name. You can choose between using __init__.py files or not, but if you do then you have to put them everywhere or it'll break.

[philtay]

IMO this section is complex (not complicated), because this is a complex problem with different solutions. I suggest to document all of them as it's hard to figure out these things (probably impossible for a newbie).

[nicoddemus]

but if you do then you have to put them everywhere or it'll break.

Are you certain? I just tested this and it works:

.tmp/
    tests/
        foo/
            __init__.py
            test_foo.py
        bar/
            __init__.py
            test_foo.py

Executing pytest from .tmp works as expected, with only .tmp/tests being added to sys.path.

[philtay]

Try this:

.tmp/
    tests/
        foo/
            __init__.py
            test_foo.py
        email/
            __init__.py
            test_email.py

pytest will import email.test_email, but ooops email is a stdlib package...
The same goes with an installed package. It will try to import mypkg.test_xxx but there isn't a test_xxx.py in your installed mypkg.

[nicoddemus]

Good point, thanks!

[nicoddemus]

Applied your suggestions, let me know what you think.

[philtay]

LGTM

[coveralls]

Coverage remained the same at 92.71% when pulling 4a93483 on nicoddemus:init-files-docs into 906b40f on pytest-dev:master.

[coveralls]

Coverage remained the same at 92.71% when pulling 272aba9 on nicoddemus:init-files-docs into 906b40f on pytest-dev:master.