Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add --mypy-xfail #180

Merged
merged 1 commit into from
Sep 17, 2024
Merged

Add --mypy-xfail #180

merged 1 commit into from
Sep 17, 2024

Conversation

dmtucker
Copy link
Collaborator

@dmtucker dmtucker commented Sep 5, 2024

Resolve #177

I could see a number of ways to approach this need:

  • Have errors emitted as actual warnings.
  • Disable failures, but provide a separate report (similar to pytest-cov).
  • Treat the failures as xfail.

I spent some time with each of these and ruled out emitting warnings for the following reasons.

  • Warnings are not pretty. Here's what my testing looked like:
    demo/bad/__init__.py::mypy1.11.1
      /home/dtux/Projects/pytest-mypy/src/pytest_mypy/__init__.py:237: MypyWarning: 1: error: Incompatible types in assignment (expression has type "str", variable has type "int")  [assignment]
        warnings.warn(MypyWarning(file_error_formatter(self, results, errors)))
    • /home/dtux/Projects/pytest-mypy/src/pytest_mypy/__init__.py:237 ...don't care
    • warnings.warn(MypyWarning(file_error_formatter(self, results, errors))) ...don't care
  • Moreover, IIUC, warnings not really appropriate for end-users because they are aimed at developers (hence the info we don't care about above).

This implements the latter 2 bullets via a new CLI option: --mypy-xfail.

  • If a mypy check (including the status check) passes, Pytest reports a pass:
    $ venv/bin/pytest --mypy-xfail demo/good.py 
    ====================== test session starts =======================
    platform linux -- Python 3.10.12, pytest-8.3.2, pluggy-1.5.0
    Using --randomly-seed=3285518892
    rootdir: /home/dtux/Projects/pytest-mypy
    configfile: tox.ini
    plugins: cov-4.1.0, mypy-0.10.4.dev48+gb3a6274.d20240813, randomly-3.15.0, xdist-1.34.0, forked-1.6.0
    collected 2 items                                                
    
    demo/good.py ..                                            [100%]
    ============================== mypy ==============================
    Success: no issues found in 1 source file
    
    ======================= 2 passed in 0.09s ========================
  • If a check fails, Pytest reports an xfail (with reasonable reasons, shown with -ra):
    $ venv/bin/pytest --mypy-xfail -ra demo/bad.py
    ====================== test session starts =======================
    platform linux -- Python 3.10.12, pytest-8.3.2, pluggy-1.5.0
    Using --randomly-seed=2239210653
    rootdir: /home/dtux/Projects/pytest-mypy
    configfile: tox.ini
    plugins: cov-4.1.0, mypy-0.10.4.dev48+gb3a6274.d20240813, randomly-3.15.0, xdist-1.34.0, forked-1.6.0
    collected 2 items                                                
    
    demo/bad.py xx                                             [100%]
    ============================== mypy ==============================
    demo/bad.py:1: error: Incompatible types in assignment (expression has type "str", variable has type "int")  [assignment]
    Found 1 error in 1 file (checked 1 source file)
    
    ==================== short test summary info =====================
    XFAIL demo/bad.py::mypy1.11.1 - The mypy errors in this file were xfailed by --mypy-xfail.
    XFAIL demo/bad.py::mypy1.11.1-status - mypy's non-zero exit status (1) was xfailed by --mypy-xfail.
    ======================= 2 xfailed in 0.28s =======================

Copy link

@jaraco jaraco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the investigation and explanation. This approach makes sense and is not too complicated.

I do worry that with this feature, I or others will want finer-grained control (only specific classes of failure or specific files). Still, for the application I was seeking, this would have worked great. I could have added the xfail directive to all of the projects, rolled out the stricter changes, and then project-by-project addressed the expected failures until they subsided to then remove the directive.

src/pytest_mypy/__init__.py Outdated Show resolved Hide resolved
src/pytest_mypy/__init__.py Outdated Show resolved Hide resolved
@dmtucker dmtucker merged commit 34a215c into realpython:main Sep 17, 2024
6 checks passed
@dmtucker dmtucker deleted the issue177 branch September 17, 2024 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Option to treat mypy failures as warnings
2 participants