False positives using when using pytest-xdist -f (--looponfail)

[y2kbugger]

When doing TDD, after the first mypy error, the plugin start emitting errors on collection.

It will continue normally until you hit an actual mypy fail, then in the follow round (after mypy error is fixed) this warning is emitted forever until pytest is killed and restarted.

Steps to Reproduce (assumes python 3.8 and pipenv available):

1. mkdir ~/tmp20200412
2. cd ~/tmp20200412
3. git clone git@bitbucket.org:y2kbugger/test_driven_development_by_example_tdd_follow_along.git
4. cd ./test*
5. git checkout b0a2f56
6. replicate versions exactly from Pipfile.lock pipenv sync
7. put venv on PATH pipenv shell
8. run pytest pytest --mypy -f money.py test_money.py

In another terminal:

1. cd ~/tmp20200412/test*
2. cause mypy error echo 'lol: str = 2' >> money.py
3. undo mypy error git checkout b0a2f56 -- .

###################################### waiting for changes #######################################
### Watching:   /home/y2k/devel/test_driven_development_by_example_tdd_follow_along
# MODIFIED /home/y2k/devel/test_driven_development_by_example_tdd_follow_along/money.py
============================= test session starts ==============================
platform linux -- Python 3.8.2, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /home/y2k/devel/test_driven_development_by_example_tdd_follow_along
plugins: forked-1.1.3, mypy-0.6.1, xdist-1.31.0, flake8-1.0.4
collected 0 items / 1 error
collected 0 items / 1 error

==================================== ERRORS ====================================
________________________ ERROR collecting test session _________________________
../../.local/share/virtualenvs/test_driven_development_by_example_tdd_fol-TPMT2dI5/lib/python3.8/site-packages/_pytest/runner.py:244: in from_call
    result = func()
../../.local/share/virtualenvs/test_driven_development_by_example_tdd_fol-TPMT2dI5/lib/python3.8/site-packages/_pytest/runner.py:264: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
../../.local/share/virtualenvs/test_driven_development_by_example_tdd_fol-TPMT2dI5/lib/python3.8/site-packages/_pytest/main.py:498: in collect
    yield from self._collect(fspath, parts)
../../.local/share/virtualenvs/test_driven_development_by_example_tdd_fol-TPMT2dI5/lib/python3.8/site-packages/_pytest/main.py:537: in _collect
    assert not names, "invalid arg {!r}".format((argpath, names))
E   AssertionError: invalid arg (local('/home/y2k/devel/test_driven_development_by_example_tdd_follow_along'), ['mypy'])
=========================== short test summary info ============================
ERROR  - AssertionError: invalid arg (local('/home/y2k/devel/test_driven_deve...
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.22s ===============================
######################################### LOOPONFAILING ##########################################
money.py::mypy
test_money.py::test_multiplication
test_money.py::test_franc_multiplication
test_money.py::test_equality
test_money.py::test_currency
::mypy
###################################### waiting for changes #######################################
### Watching:   /home/y2k/devel/test_driven_development_by_example_tdd_follow_along

Note I kill with keyboard interrupt here:

^C

Then everything goes fine again:

(test_driven_development_by_example_tdd_follow_along) [y2k@15r test_driven_development_by_example_tdd_follow_along]
$ pytest --mypy -f money.py test_money.py 
============================= test session starts ==============================
platform linux -- Python 3.8.2, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /home/y2k/devel/test_driven_development_by_example_tdd_follow_along
plugins: forked-1.1.3, mypy-0.6.1, xdist-1.31.0, flake8-1.0.4
collected 7 items
collected 7 items

money.py .                                                               [ 12%]
test_money.py ......                                                     [ 87%]
. .                                                                      [100%]. [100%]
===================================== mypy =====================================

Success: no issues found in 2 source files
============================== 9 passed in 0.34s ===============================
###################################### waiting for changes #######################################
### Watching:   /home/y2k/devel/test_driven_development_by_example_tdd_follow_along

[dmtucker]

Thanks for the excellent repro steps! 💯

The problem seems to have been introduced in 0.6.0 (#79) 🤔

[y2kbugger]

I did notice that I was passing both the source file and the test file to pytest but I think I was able to reproduce when only passing the test file. I will get back to you on that.

…

On Sun, Apr 12, 2020, 17:17 David Tucker ***@***.***> wrote: Thanks for the excellent repro steps! 💯 The problem seems to have been introduced in 0.6.0 (#79 <#79>) 🤔 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#85 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABORVLM2V5ZGUMKNUNLAR63RMIVVFANCNFSM4MGQ6OJA> .

[y2kbugger]

Can confirm error occurs even if ran properly:

pytest -f --mypy ./test_money.py 

[dmtucker]

When a test fails, pytest-xdist holds onto the node ID of the failing test. The MypyStatusItem that is injected in pytest_collection_modifyitems has parent=session which produces a node ID of ::mypy (where mypy is nodeid_name).
Running that node ID explicitly (like xdist does) reproduces this:

pytest --verbose --mypy ::mypy

================================================ test session starts =================================================
platform linux -- Python 3.8.2, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /home/dtux/Projects/pytest-mypy/venv/bin/python
cachedir: .pytest_cache
Using --randomly-seed=1597611273
rootdir: /home/dtux/Projects/pytest-mypy
plugins: xdist-2.0.0, forked-1.3.0, randomly-2.1.1, cov-2.5.1, mypy-0.6.3.dev21+gd052c4c.d20200816
collected 0 items / 1 error                                                                                          

======================================================= ERRORS =======================================================
___________________________________________ ERROR collecting test session ____________________________________________
venv/lib/python3.8/site-packages/_pytest/runner.py:294: in from_call
    result = func()  # type: Optional[_T]
venv/lib/python3.8/site-packages/_pytest/runner.py:324: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
venv/lib/python3.8/site-packages/_pytest/main.py:576: in collect
    yield from self._collect(fspath, parts)
venv/lib/python3.8/site-packages/_pytest/main.py:617: in _collect
    assert not names, "invalid arg {!r}".format((argpath, names))
E   AssertionError: invalid arg (local('/home/dtux/Projects/pytest-mypy'), ['mypy'])
============================================== short test summary info ===============================================
ERROR  - AssertionError: invalid arg (local('/home/dtux/Projects/pytest-mypy'), ['mypy'])
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================== 1 error in 0.08s ==================================================

[offbyone]

Would you mind cutting a release with this? I would love to have this working.

[dmtucker]

Fixed in https://pypi.org/project/pytest-mypy/0.8.0/