type hints need to be distributed with cpython/projects, instead of being in a separate repo

[ngie-eign]

While testing out type hints with mypy dealing with some of the customizations my group made to cpython 3.8.2, I was surprised and shocked that the type hints for stdlib live in a separate repo, instead of being distributed with the base interpreter.

Distributing the .pyi files with the base interpreter and individual projects should be done to reduce friction for end-users that customize cpython and other projects to meet their needs without having to jump through hoops to effectively maintain a duplicate type hints file for underlying implementations that implement the customizations. What will happen over time, I fear, is that the .pyi files will become crufty/incorrect or out of sync with the implementations as functions/constants are added/removed by other developers.

Tying the interface files to the project that they describe makes sense, intuitively and pragmatically and will result in less of a maintenance nightmare.

Please see this mypy issue for more details: python/mypy#8745 .

[Michael0x2a]

What will happen over time, I fear, is that the .pyi files will become crufty/incorrect or out of sync with the implementations as functions/constants are added/removed by other developers.

Your concern that the .pyi files might be incorrect in some places isn't a hypothetical concern -- that's been the state of affairs for the past few years. (This is a volunteer project, after all.) Some of the more obscure corners of the standard library are fairly incomplete, and some of the more magical corners (such as the map function) can't yet be typed with full precision using typing features that are available today.

Given this state of affairs, the solution we seem to have converged on is to (a) make it as easy as possible for people to contribute small improvements and (b) make it as easy as possible to distribute these changes. Keeping typeshed separate from the Python release cycle is one way of doing this.

We also tend to backport any improvements we make to the typing ecosystem. For example, Protocols were only added to Python proper as of Python 3.8, but are available for use in Python 2.7+ using the typing-extensions ecosystem. So since new typing features are generally made available to all users of type hints, it makes sense to ensure that any improvements we make to the stdlib using new type features are also made available to the same set of users.

(From this point of view, it's also not great that typeshed tends to come bundled with type checkers -- see #2491 for more discussion.)

I think the "what if existing type hints fall out of sync" problem is less of a concern, at least for our primary use case of supporting users of CPython. The standard library is pretty stable/the descriptions of any changes are fairly stable. I don't think anybody has really thought about how to support forks of CPython yet: it seems to be a pretty uncommon use case so far, for better or for worse.

---

Regarding your other issue about mypy, you shouldn't need to fork mypy. Instead, use the --custom-typeshed-dir flag and point it at your custom fork of typeshed.

This may become somewhat easier once #2491 (comment) is finished. Then, you could maybe pip-install your custom version of typeshed instead of the default one and not have to mess around with mypy flags.

[srittau]

As @Michael0x2a pointed out, shipping the stubs with the Python interpreter at this point would make no sense for quality reasons and because it is a fast moving target. This could change in the future, but is a decision of the CPython project anyway, not of the typeshed project.

[hauntsaninja]

On the subject of "what if existing type hints fall out of sync", typeshed CI can detect a lot of issues automatically by comparing stubs to what it finds via runtime introspection of the stdlib. Although this approach isn't perfect, it's able to catch a lot of common issues and generally help prevent divergence. See #754 for more details.