Update mypy to a custom version

[cdce8p]

Proposed change

Unfortunately, the mypy releases don't happen quite as regular as I would like. Usually that isn't a big issue, however this time the next version will include some features and bug fixes which are required for some major typing improvements.

The release itself was originally planned for the end of January, the release branch has even been cut already, unfortunately though there haven't been any updates over the last two weeks. It might happen early next week, or next month.

That's why I decided to create a project to build custom mypy version and upload them to PyPI. These can be used interchangeably which mypy itself, basically as a stop gap, until the next official release. The wheels are build using almost the same workflow, i.e. also compiled with mypyc. Only difference being that I chose not to compile / upload musllinux wheels. (If necessary, I can add that and push a new release.)

https://pypi.org/project/mypy-dev/1.9.0b1/
https://github.com/cdce8p/mypy-dev/tree/1.9.0b1
https://github.com/cdce8p/mypy-dev/releases/tag/1.9.0b1

Release 1.9.0b1 tracks the current HEAD of the release-1.9.0 branch python/mypy@155909a.

Changes for Mypy 1.9.0 (incomplete)

• Support for basic TypeVar defaults. Basically current use cases. E.g. for DataUpdateCoordinator.
  As long as the default isn't a TypeVarTuple or references another TypeVar.
  => This will also enable us to add generic typing to the Event class to get rid of some type: ignores around async_listen.
• Bugfix for TypeVarTuple inference with default arguments. Mypy will now emit warnings for wrong typing - used for
  Improve dispatcher typing #106872
  Consider TypeVarTuple to be invariant python/mypy#16759
• Bugfix for dict.__setitem__ with custom key types. Required for aiohttp.web.AppKey and
  Add HassDict implementation #103844
  Fix missing type store for overloads python/mypy#16803
  https://docs.aiohttp.org/en/stable/web_advanced.html#application-s-config
• Better typing stubs. Notably for asyncio callback types - anything with a callback and only *args, e.g. call_soon. We could use something similar for async_add_job, async_add_hass_job, ... .

Plan for the future

As soon as there is an official release, I'll create a PR to use that one instead. Beyond that, if there is a similar situation in the future, it might make sense to temporarily switch again. So far, I don't expect that. This release is somewhat special as a lot of typing improvements are currently blocked.

Good to know

mypy --version will print the development version as if mypy would have been installed from git. I.e. not 1.9.0b1 but this instead

mypy 1.9.0+dev.155909ad1bde747d89fcd091621d7cd9b1e15818 (compiled: yes)

Just in case someone finds an issue and wants to open a bug report on the official mypy repo.

Refs

• 1.9.0 Release Tracking Issue python/mypy#16843

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Deprecation (breaking change to happen in the future)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Additional information

• This PR fixes or closes issue: fixes #
• This PR is related to issue:
• Link to documentation pull request:

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• I have followed the perfect PR recommendations
• The code has been formatted using Ruff (ruff format homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
• Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[MartinHjelmare]

Have we considered switching to pyright? What would be the consequences of that?

[cdce8p]

Have we considered switching to pyright? What would be the consequences of that?

Technically it would be possible. However, likely quite a bit of work to get everything working as it is now. There is a lot of custom configuration especially around strict typing integrations. Personally, I use VSCode with Pylance / pright every day. I've even set "typeCheckingMode": "strict". It works, I do test all new typing changes I make against both mypy and pyright and report issues as necessary, but there are differences.

Most notably in how pyright infers variables without annotations, the vast majority. Whereas mypy assumes the type to be the one of the first assignment, pyright is happy to add any additional ones as needed and combine them into a union. Maybe that's a preference thing but I prefer the way mypy handles that. I don't want to go around the whole codebase and just add type annotations to limit x to int in the example below. It's more convenient to add int | str if it should be that.

def f(some_val: bool) -> None:
    x = 1
    reveal_type(x)
    if some_val:
        x = ""
    reveal_type(x)

# mypy
test.py:3: note: Revealed type is "builtins.int"
test.py:5: error: Incompatible types in assignment (expression has type "str", variable has type "int")  [assignment]
test.py:6: note: Revealed type is "builtins.int"

# pyright
test.py:3:17 - information: Type of "x" is "Literal[1]"
test.py:6:17 - information: Type of "x" is "Literal['', 1]"

--
The main advantage I would see in pyright is the higher development velocity. It usually takes quite a while to get new features in mypy and even releases do take time (and are irregular). That doesn't matter so much if, like in this case, I add the features I want / need for Home Assistant myself 😄 Back to pyright, Pylance recently reduced their release frequency to monthly, so we might end up waiting there too.

--
Overall I would recommend to stay with mypy. That might change in the future, but I don't see a need for that yet. This PR is just a helpful stop gap measure to work around the unpredictable release schedule. So we can start to update the typing now instead of waiting for the release, which will most likely be based one the same commit (just with a version change).

[cdce8p]

Just to provide some more context regarding pyright. Did a run over the homeassistant directory without any additional configuration.

44246 errors, 36 warnings, 0 informations

Quite possible that a lot of these would go away fairly quickly but it isn't just plug and play.

Update
Got it down to ~5500 errors, but that is where the real work would start.

[MartinHjelmare]

We've discussed this in the core team and think it's ok. It's not great to not use upstream packages, but we think this is worth it in this case. 👍

Thanks!