Pin mypy version?

[srittau]

I suggest that we pin the mypy requirement to the latest released version and update the requirement after each new release. Historically, mypy and typeshed development were closely linked, but by now typeshed is used by many different projects. This would allow repeatable CI builds and means that local dev environments match the CI environment better. It also means that wrong commits to mypy won't break typeshed's unrelated builds as it has happened in the past. The last point becomes more important as more people contribute to typeshed. On the other hand, it can make work for the mypy maintainers a bit harder, as it means that some potential issues only crop up when synching typeshed in the mypy codebase.

[srittau]

@JelleZijlstra @ilevkivskyi @JukkaL

[JelleZijlstra]

I can definitely see the argument for doing this. On the other hand, the mypy test is currently the only test in our CI that uses typeshed to typecheck real-world code, and that finds issues pretty regularly.

What would be nice to have for typeshed is a tool like black-primer (https://black.readthedocs.io/en/latest/black_primer.html#) that runs on a variety of projects that rely on typeshed.

A good first step could be to stop running the full mypy test suite in typeshed CI and run only mypy's own self-check, where it typechecks its own code. That part isn't as vulnerable to the minor changes in error messages that sometimes require the complicated merge dance, and it would still let us see typeshed in real-world use in CI.

[JukkaL]

I don't think that running the full mypy test suite is very useful, since most tests don't use the full stubs anyway. Mypy self check should be pretty predictable, and covers a good deal of stdlib. Pinning mypy version wouldn't we as important if we didn't run all the tests. However, pinning mypy would have other benefits, such as being able to use a precompiled binary (or at least doing this more reliably).

... that runs on a variety of projects that rely on typeshed

I like this idea.

Additionally, I strongly believe that once we've migrated to modular typeshed, we should add support for per-distribution tests, at least simple ones, to catch more regressions. We can then require non-trivial stub changes to include tests.

[hauntsaninja]

Things have changed quite a bit since this issue was posted. We now:
a) Only run a subset of mypy's test suite, see https://github.com/python/mypy/blob/fe8fac16178447bb0abab3e759221294d4a70f3c/runtests.py#L77
b) Run mypy_primer in CI
c) Have another mypy + typeshed maintainer

The close coupling has advantages, for instance, last week it prevented shipping broken stubs in #4815
I'm going to close for now, but we can always reconsider if we think this is still a painpoint for typeshed.