pre-commit CI is failing

[nicoa]

Edit: see #1401 for the failure as well.

TL;DR: with the current setup, it seems to still stick to old click, which then breaks the CI in #1398 / #1400 when using a click 8 feature. If manually upgraded to newest click (8.0.1) it drops the above errors but introduces a whole lot of new ones.

The code I'm referring to here (which breaks the "old" hook invocation) is the one in the mentioned PR, not the one on master branch.

hook with old click

the current hook (cached) leads to

❯ pre-commit run mypy --all-files
mypy.....................................................................Failed
- hook id: mypy
- exit code: 1

piptools/scripts/compile.py:48: error: Unexpected keyword argument "package_name" for "version_option"
/Users/me/.cache/pre-commit/repo3nny31gn/py_env-python3.9/lib/python3.9/site-packages/mypy/typeshed/third_party/2and3/click/decorators.pyi:241: note: "version_option" defined here
piptools/scripts/sync.py:24: error: Unexpected keyword argument "package_name" for "version_option"
/Users/me/.cache/pre-commit/repo3nny31gn/py_env-python3.9/lib/python3.9/site-packages/mypy/typeshed/third_party/2and3/click/decorators.pyi:241: note: "version_option" defined here
Found 2 errors in 2 files (checked 37 source files)

upgraded hook

upgrading the hook as

  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v0.812
    hooks:
      - id: mypy
        # Avoid error: Duplicate module named 'setup'
        # https://github.com/python/mypy/issues/4008
        exclude: ^tests/test_data/
        additional_dependencies: [click>=8.0.0]

leads to

mypy.....................................................................Failed
- hook id: mypy
- exit code: 1

piptools/utils.py:22: error: Module 'pip._internal.vcs' does not explicitly export attribute 'is_url'; implicit reexport disabled
piptools/utils.py:50: error: Argument 1 to "key_from_req" has incompatible type "Optional[Requirement]"; expected "InstallRequirement"
piptools/utils.py:57: error: "InstallRequirement" has no attribute "key"
piptools/utils.py:111: error: Item "None" of "Optional[Link]" has no attribute "url"
piptools/utils.py:113: error: Item "None" of "Optional[Link]" has no attribute "url"
piptools/repositories/pypi.py:31: error: Module 'pip._vendor.requests' does not explicitly export attribute 'RequestException'; implicit reexport disabled
piptools/repositories/pypi.py:31: error: Module 'pip._vendor.requests' does not explicitly export attribute 'Session'; implicit reexport disabled
piptools/repositories/pypi.py:63: error: Incompatible types in assignment (expression has type "Command", variable has type "InstallCommand")
piptools/repositories/pypi.py:126: error: Argument 1 to "find_all_candidates" of "PyPIRepository" has incompatible type "Optional[str]"; expected "str"
piptools/repositories/pypi.py:134: error: Invalid index type "Union[Union[Version, LegacyVersion], str]" for "Dict[_BaseVersion, Set[InstallationCandidate]]"; expected type "_BaseVersion"
piptools/repositories/pypi.py:140: error: Argument 1 to "make_candidate_evaluator" of "PackageFinder" has incompatible type "Optional[str]"; expected "str"
piptools/repositories/pypi.py:147: error: Argument 2 to "make_install_requirement" has incompatible type "Union[LegacyVersion, Version]"; expected "Union[str, Version]"
piptools/repositories/pypi.py:169: error: Argument 1 to "make_requirement_preparer" of "RequirementCommand" has incompatible type "**Dict[str, object]"; expected "TempDirectory"
piptools/repositories/pypi.py:169: error: Argument 1 to "make_requirement_preparer" of "RequirementCommand" has incompatible type "**Dict[str, object]"; expected "Values"
piptools/repositories/pypi.py:169: error: Argument 1 to "make_requirement_preparer" of "RequirementCommand" has incompatible type "**Dict[str, object]"; expected "RequirementTracker"
piptools/repositories/pypi.py:169: error: Argument 1 to "make_requirement_preparer" of "RequirementCommand" has incompatible type "**Dict[str, object]"; expected "PipSession"
piptools/repositories/pypi.py:169: error: Argument 1 to "make_requirement_preparer" of "RequirementCommand" has incompatible type "**Dict[str, object]"; expected "PackageFinder"
piptools/repositories/pypi.py:169: error: Argument 1 to "make_requirement_preparer" of "RequirementCommand" has incompatible type "**Dict[str, object]"; expected "bool"
piptools/repositories/pypi.py:169: error: Argument 1 to "make_requirement_preparer" of "RequirementCommand" has incompatible type "**Dict[str, object]"; expected "str"
piptools/repositories/pypi.py:186: error: "BaseResolver" has no attribute "_resolve_one"
piptools/repositories/pypi.py:190: error: "BaseResolver" has no attribute "_get_dist_for"
piptools/repositories/pypi.py:223: error: Incompatible types in assignment (expression has type "Set[InstallationCandidate]", target has type "Set[InstallRequirement]")
piptools/repositories/pypi.py:316: error: Argument 1 to "debug" of "LogContext" has incompatible type "Optional[str]"; expected "str"
piptools/repositories/pypi.py:362: error: Argument 1 to "find_all_candidates" of "PyPIRepository" has incompatible type "Optional[str]"; expected "str"
piptools/repositories/pypi.py:367: error: Invalid index type "Union[Version, LegacyVersion, str]" for "Dict[_BaseVersion, Set[InstallationCandidate]]"; expected type "_BaseVersion"
piptools/repositories/pypi.py:424: error: Cannot assign to a method
piptools/repositories/pypi.py:424: error: Incompatible types in assignment (expression has type "Callable[[Wheel, List[Tag]], bool]", variable has type "Callable[[Wheel, Iterable[Tag]], bool]")
piptools/repositories/pypi.py:425: error: Cannot assign to a method
piptools/repositories/pypi.py:431: error: Cannot assign to a method
piptools/repositories/pypi.py:432: error: Cannot assign to a method
piptools/cache.py:170: error: Argument 1 to "key_from_req" has incompatible type "Requirement"; expected "InstallRequirement"
piptools/writer.py:50: error: Argument 1 to "key_from_ireq" has incompatible type "Optional[InstallRequirement]"; expected "InstallRequirement"
piptools/writer.py:242: error: "InstallRequirement" has no attribute "_source_ireqs"; maybe "source_dir"?
piptools/resolver.py:71: error: Returning Any from function declared to return "InstallRequirement"
piptools/resolver.py:79: error: Item "None" of "Optional[Requirement]" has no attribute "specifier"
piptools/resolver.py:107: error: Returning Any from function declared to return "InstallRequirement"
piptools/resolver.py:202: error: Need type annotation for 'required_by'
piptools/resolver.py:202: error: Item "None" of "Optional[str]" has no attribute "lower"
piptools/resolver.py:295: error: Argument "key" to "sorted" has incompatible type "Callable[[InstallRequirement], str]"; expected "Callable[[RequirementSummary], SupportsLessThan]"
piptools/resolver.py:299: error: Argument "key" to "sorted" has incompatible type "Callable[[InstallRequirement], str]"; expected "Callable[[RequirementSummary], SupportsLessThan]"
piptools/resolver.py:346: error: "InstallRequirement" has no attribute "_source_ireqs"; maybe "source_dir"?
piptools/repositories/local.py:9: error: Module 'pip._vendor.requests' does not explicitly export attribute 'Session'; implicit reexport disabled
piptools/repositories/local.py:24: error: "InstallationCandidate" has no attribute "req"
piptools/repositories/local.py:25: error: Item "None" of "Optional[Requirement]" has no attribute "specifier"
piptools/repositories/local.py:26: error: "InstallationCandidate" has no attribute "req"
piptools/repositories/local.py:67: error: Signature of "find_best_match" incompatible with supertype "BaseRepository"
piptools/repositories/local.py:73: error: Argument 1 to "as_tuple" has incompatible type "InstallationCandidate"; expected "InstallRequirement"
piptools/repositories/local.py:74: error: Incompatible return value type (got "InstallRequirement", expected "InstallationCandidate")
piptools/repositories/local.py:76: error: Incompatible return value type (got "InstallRequirement", expected "InstallationCandidate")
piptools/repositories/local.py:86: error: "InstallationCandidate" has no attribute "hash_options"
piptools/scripts/compile.py:48: error: Unexpected keyword argument "package_name" for "version_option"
piptools/scripts/compile.py:346: error: Argument 1 to "LocalRequirementsRepository" has incompatible type "Dict[str, InstallRequirement]"; expected "Mapping[str, InstallationCandidate]"
tests/test_repository_pypi.py:7: error: Module 'pip._vendor.requests' does not explicitly export attribute 'HTTPError'; implicit reexport disabled
tests/test_repository_pypi.py:7: error: Module 'pip._vendor.requests' does not explicitly export attribute 'Session'; implicit reexport disabled
piptools/sync.py:66: error: Argument 1 to "key_from_req" has incompatible type "Requirement"; expected "InstallRequirement"
piptools/sync.py:72: error: "Requirement" has no attribute "requires"
piptools/sync.py:77: error: "Requirement" has no attribute "version"
piptools/sync.py:92: error: Argument 1 to "key_from_req" has incompatible type "Requirement"; expected "InstallRequirement"
piptools/sync.py:159: error: Argument 1 to "key_from_req" has incompatible type "Requirement"; expected "InstallRequirement"
piptools/sync.py:162: error: "Requirement" has no attribute "version"
piptools/sync.py:201: error: Value of type variable "SupportsLessThanT" of "sorted" cannot be "InstallRequirement"
piptools/sync.py:225: error: Value of type variable "SupportsLessThanT" of "sorted" cannot be "InstallRequirement"
piptools/sync.py:225: error: List item 6 has incompatible type "List[InstallRequirement]"; expected "Union[str, bytes, _PathLike[str], _PathLike[bytes]]"
piptools/scripts/sync.py:24: error: Unexpected keyword argument "package_name" for "version_option"
piptools/scripts/sync.py:129: error: Argument 2 to "diff" has incompatible type "List[Distribution]"; expected "Iterable[Requirement]"
piptools/scripts/sync.py:148: error: Argument 2 to "sync" has incompatible type "Set[str]"; expected "Iterable[InstallRequirement]"
Found 66 errors in 10 files (checked 37 source files)

Those are all correct and needs to be fixed I guess, right?

UPDATE: Actually I'm not sure the pre-commit is correctly invoking everything, compare this gitter thread. To me it seemed that this is related to click (removing typeshed), pip and/or setuptools upgrades, but I'm not deep enough into neither typing nor those packages to assess whats happening here.

[webknjaz]

It seems so

[ssbarnea]

Running mypy on pip-tools is messed big time and pre-commit ci is correct to comaplain about it. I did run mypy locally myself using mypy . and got >60 errors, quite genuine I think.

I also observed the running pre-commit locally does pass mypy but only because the mypy hook definition inside .pre-commit-config.yaml is incorrect.

How did we endup with an issue like this? Wasn't CI supposed to prevent something like thing from happening. I am more worried about this aspect than the reported issues. Sadly my knowledge of pip-tools internals is very limited and I am afraid that help from someone that worked on adding type hints is needed, especially to explain the differences between InstallRequirement and Requirement.

[nicoa]

it seems to be running again. If there is nobody withspoking, I'd close this issue soon.

[ssbarnea]

Some of these errors can be fixed by upgrading hooks, mypy in particular is a source of surprises as it can give different results with different python versions. Still closing it makes sense as we were not able to reproduce.