"source" config combined with pytest-xdist leads to incorrect coverage

[DanCardin]

Describe the bug
Either source behaves weirdly in combination with xdist, or the documentation around
how source/include work could be improved.

When supplying certain source/include values in particular ways with xdist, you can
either get 0% or 100%.

In my real-life usecase, I actually see 16% coverage reported,
where the last xdist process reports some amount of coverage while everything
else reports 0. But perhaps due to the simpler test-setup i lay out below, it only
seems to result in 0s.

It seems, by description, similar to #389

To Reproduce
How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:

1. What version of Python are you using? 3.9.6
2. What version of coverage.py shows the problem? 6.3.2
3. What versions of what packages do you have installed? pytest-xdist==2.5.0

❯ tree
.
├── poetry.lock
├── pyproject.toml
├── src
│  └── foo
│     ├── __init__.py
│     └── meow.py
└── test.py

# pyproject.toml
[tool.poetry]
name = "foo"
version = "0.0.0"
description = ""
authors = []
packages = [
    { include = "foo", from = "src" },
]

[tool.coverage.report]
show_missing = true
skip_covered = true

[tool.coverage.run]
source = ["src"]
parallel = true
branch = true

# meow.py
def meow():
    print("meow")


def meow2():
    print("meow")


def meow3():
    print("meow")


meow()

# test.py
from foo.meow import meow, meow2, meow3


def test_meow():
    meow()


def test_meow2():
    meow2()


def test_meow3():
    meow3

If you run the following, you get the correct result (86% in this case)

coverage run -m py.test test.py && coverage combine && coverage report

If you run the following, you get 0%

coverage run -m py.test -n 3 test.py && coverage combine && coverage report

Perhaps notable other options I tried:

source = ["src/"]  # 0%
source = ["foo"]  # 0%
source = ["src/foo"]  # 0%
source = ["src/foo"]  # 0%
include = ["src"]  # 0%
include = ["src/*"]  # 86% hurray!

So tl;dr the behavior of source seems to be different with/without xdist, perhaps
in combination given the fact that the package is nested within a src/ directory
instead of foo/ being implicitly on the path.

*Expected behavior
I expect the behavior of the source option to react the same regardless of
use of the -n flag from pytest-xdist.

Alternatively, if this is difficult to work around, just that perhaps the docs
be made more clear that one needs to use include with a pattern.

[ssbarnea]

I can confirm that there is a clear bug in this area and when I commented source and used include the reported coverage went from 12% to 23%, even if xdist run on 10 cores (m1).

This is still far from the real coverage on the project, which is around 84% as measured with pytest-cov.

I should mention that I reached that place while trying to avoid using pytest-cov because it was reporting some warnings about modules being imported too soon. Funny bit is that even with these warnings, the pytest-cov reported far more than coverage.

# pyproject.toml
[tool.coverage.run]
parallel = true
concurrency = ["multiprocessing", "thread"]

I am really curious to find a project using just coverage.py + pytest + xdist, what does properly record the coverage, maybe I could stop what is causing the incomplete coverage.

Another thing that I found quite weird was that I seen the coverage files being exact 102400 in file size.

[ssbarnea]

After spending a good number of hours today trying to make it work w/o pytest-cov, I think I found the trick. In fact it was only one project that managed to get it working and that was via https://github.com/python-attrs/attrs/pull/1011/files change. change the the magic sauce that nobody documented was the installation of a 6 years package named coverage-enable-subprocess -- magically once that package is installed, my coverage magically switched from 23% to 89%.

[nedbat]

I want to understand what's going wrong here more, but:

Alternatively, if this is difficult to work around, just that perhaps the docs be made more clear that one needs to use include with a pattern.

The help for coverage run says:

  --include=PAT1,PAT2,...
                        Include only files whose paths match one of these
                        patterns. Accepts shell-style wildcards, which must be
                        quoted.

The docs for specifying source files say:

The include and omit file name patterns follow typical shell syntax: * matches any number of characters and ? matches a single character. Patterns that start with a wildcard character are used as-is, other patterns are interpreted relative to the current directory...

If you have suggestions for how to make it clearer that include needs a pattern, I'm open to them.

[DanCardin]

I dont remember writing that 😅, the content in the docs seems clear enough. Instead it might be better to warn in the event that includes are supplied which ultimately result in no matches? But honestly that's more of an unrelated feature request.

More relevant to the bit where i said "Alternatively, if this is difficult to work around", i guess i'd now instead suggest warning if xdist is detected to be used in combination with source; if source cannot be made to work.

[nedbat]

Definitely this is because you have to configure coverage.py to measure subprocesses. You can do that a number of ways:

1. Use the steps in https://coverage.readthedocs.io/en/6.5.0/subprocess.html
2. Use the coverage-enable-subprocess package
3. Use pytest-cov

It might be time for coverage.py to have a better way to do this itself...

[nedbat]

This is related to #367 and #378.

[adam-grant-hendry]

Not sure if this helps, but on Windows (with coverage, pytest, and pytest-xdist), if I run

coverage run --debug=trace --parallel-mode -m pytest -n auto

I consistently get an OSError: [WinError 6] The handle is invalid. Perhaps there is a race condition when trying to write out the .coverage.* files?

Afterwards, if I run

coverage combine && coverage report -m

I get 0% coverage like everyone else is saying. If I don't use xdist and just run

coverage run --debug=trace -m pytest && coverage combine && coverage report -m

Everything works fine...¯(º_o)/¯

traceback

Traceback (most recent call last): File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\pytest\__main__.py", line 5, in raise SystemExit(pytest.console_main()) File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\_pytest\config\__init__.py", line 192, in console_main code = main() File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\_pytest\config\__init__.py", line 150, in main config = _prepareconfig(args, plugins) File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\_pytest\config\__init__.py", line 331, in _prepareconfig config = pluginmanager.hook.pytest_cmdline_parse( File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\pluggy\_hooks.py", line 493, in __call__ return self._hookexec(self.name, self._hookimpls, kwargs, firstresult) File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\pluggy\_manager.py", line 115, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\pluggy\_callers.py", line 130, in _multicall teardown[0].send(outcome) File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\_pytest\helpconfig.py", line 104, in pytest_cmdline_parse config: Config = outcome.get_result() File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\pluggy\_result.py", line 114, in get_result raise exc.with_traceback(exc.__traceback__) File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\pluggy\_callers.py", line 77, in _multicall res = hook_impl.function(*args) File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\_pytest\config\__init__.py", line 1075, in pytest_cmdline_parse self.parse(args) File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\_pytest\config\__init__.py", line 1425, in parse self._preparse(args, addopts=addopts) File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\_pytest\config\__init__.py", line 1327, in _preparse self.hook.pytest_load_initial_conftests( File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\pluggy\_hooks.py", line 493, in __call__ return self._hookexec(self.name, self._hookimpls, kwargs, firstresult) File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\pluggy\_manager.py", line 115, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\pluggy\_callers.py", line 152, in _multicall return outcome.get_result() File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\pluggy\_result.py", line 114, in get_result raise exc.with_traceback(exc.__traceback__) File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\pluggy\_callers.py", line 77, in _multicall res = hook_impl.function(*args) File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\_pytest\config\__init__.py", line 1145, in pytest_load_initial_conftests args, args_source = early_config._decide_args( File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\_pytest\config\__init__.py", line 1264, in _decide_args result.extend(sorted(glob.iglob(path, recursive=True))) File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\coverage\control.py", line 393, in _should_trace self._debug.write(disposition_debug_msg(disp)) File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\coverage\debug.py", line 91, in write self.output.write(msg+"\n") File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\coverage\debug.py", line 403, in write self.outfile.write(filter_text(text, self.filters)) OSError: [WinError 6] The handle is invalid

[adam-grant-hendry]

Also unfortunately, adding coverage-enable-subprocess to my project doesn't seem to work on my end :/