6.0.0rc1 pytest.mark.xfail mypy error: No overload variant

[simonjayhawkins]

• a detailed description of the bug or suggestion
• output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

using example from https://docs.pytest.org/en/documentation-restructure/how-to/skipping.html#raises-parameter

import pytest

@pytest.mark.xfail(raises=RuntimeError)
def test_function():
    ...

running mypy

$ mypy ~/pytest-mypy.py
C:\Users\simon\pytest-mypy.py:3: error: No overload variant of "__call__" of "_XfailMarkDecorator" matches argument type "Type[RuntimeError]"
C:\Users\simon\pytest-mypy.py:3: note: Possible overload variant:
C:\Users\simon\pytest-mypy.py:3: note:     def __call__(self, condition: Union[str, bool] = ..., *conditions: Union[str, bool], reason: str = ..., run: bool = ..., raises: Union[BaseException, Tuple[BaseException, ...]] = ..., strict: bool = ...) -> MarkDecorator
C:\Users\simon\pytest-mypy.py:3: note:     <1 more non-matching overload not shown>
Found 1 error in 1 file (checked 1 source file)

should the type annotation for raise in _XfailMarkDecorator be raises: Union[Type[BaseException], Tuple[Type[BaseException], ...]] = ...,

xref pandas-dev/pandas#35261 (comment)

[asottile]

yep, this annotation is indeed incorrect, would you like to send a patch to correct it?

also pytest seems to raise a confusing internal error if non-types are passed into raises:

$ pytest t.py
============================= test session starts ==============================
platform linux -- Python 3.8.2, pytest-5.4.3, py-1.9.0, pluggy-0.13.1
rootdir: /tmp/x
collected 1 item                                                               

t.py 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/_pytest/main.py", line 191, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/_pytest/main.py", line 247, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/_pytest/main.py", line 272, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/_pytest/runner.py", line 85, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/_pytest/runner.py", line 100, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/_pytest/runner.py", line 188, in call_and_report
INTERNALERROR>     report = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/pluggy/callers.py", line 203, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/_pytest/skipping.py", line 147, in pytest_runtest_makereport
INTERNALERROR>     if evalxfail.invalidraise(call.excinfo.value):
INTERNALERROR>   File "/tmp/x/venv/lib/python3.8/site-packages/_pytest/mark/evaluate.py", line 53, in invalidraise
INTERNALERROR>     return not isinstance(exc, raises)
INTERNALERROR> TypeError: isinstance() arg 2 must be a type or tuple of types

============================ no tests ran in 0.02s =============================

[bluetech]

Thanks for testing the release candidate @simonjayhawkins. You identified the problem correctly. Since I want to make sure this is fixed for 6.0.0 I submitted a fix for this myself in #7492.

[simonjayhawkins]

Thanks for testing the release candidate @simonjayhawkins. You identified the problem correctly. Since I want to make sure this is fixed for 6.0.0 I submitted a fix for this myself in #7492.

Thanks @bluetech . It took me a while to get the environment setup with conda on Windows yesterday to get the tests running before submitting a PR. In pandas, we have some other mypy failures that i'll work through today, so may have some more changes.