[red-knot] Support stub packages #17204

MichaReiser · 2025-04-04T13:26:55Z

Summary

This PR adds support for stub packages, except for partial stub packages (a stub package is always considered non-partial).

I read the specification at typing.python.org/en/latest/spec/distributing.html#stub-only-packages but I found it lacking some details, especially on how to handle namespace packages or when the regular and stub packages disagree on whether they're namespace packages. I tried to document my decisions in the mdtests where the specification isn't clear and compared the behavior to Pyright.

Mypy seems to only support stub packages in the venv folder. At least, it never picked up my stub packages otherwise. I decided not to spend too much time fighting mypyp, which is why I focused the comparison around Pyright

Closes #16612

Test plan

Added mdtests

github-actions · 2025-04-04T13:28:41Z

`mypy_primer` results

Changes were detected when running on open source projects

scrapy (https://github.com/scrapy/scrapy)
- error[lint:unresolved-import] /tmp/mypy_primer/projects/scrapy/tests/test_webclient.py:13:8: Cannot resolve import `OpenSSL.SSL`
- error[lint:unresolved-import] /tmp/mypy_primer/projects/scrapy/scrapy/core/downloader/contextfactory.py:6:6: Cannot resolve import `OpenSSL`
- error[lint:unresolved-import] /tmp/mypy_primer/projects/scrapy/tests/test_core_downloader.py:9:8: Cannot resolve import `OpenSSL.SSL`
+ error[lint:unresolved-attribute] /tmp/mypy_primer/projects/scrapy/scrapy/utils/ssl.py:23:9: Type `X509Name` has no attribute `_name`
- error[lint:unresolved-import] /tmp/mypy_primer/projects/scrapy/scrapy/core/downloader/tls.py:4:6: Cannot resolve import `OpenSSL`
- error[lint:unresolved-import] /tmp/mypy_primer/projects/scrapy/tests/mockserver.py:15:6: Cannot resolve import `OpenSSL`
- error[lint:unresolved-import] /tmp/mypy_primer/projects/scrapy/scrapy/utils/ssl.py:6:8: Cannot resolve import `OpenSSL.SSL`
- error[lint:unresolved-import] /tmp/mypy_primer/projects/scrapy/scrapy/utils/ssl.py:7:8: Cannot resolve import `OpenSSL.version`
- error[lint:unresolved-import] /tmp/mypy_primer/projects/scrapy/scrapy/utils/ssl.py:12:10: Cannot resolve import `OpenSSL.crypto`
- Found 1509 diagnostics
+ Found 1502 diagnostics

MichaReiser · 2025-04-04T16:25:48Z

crates/red_knot_python_semantic/src/module_resolver/resolver.rs

-        let mut components = name.components();
-        let module_name = components.next_back()?;
-
-        match resolve_package(search_path, components, &resolver_state) {
-            Ok(resolved_package) => {


I moved most of this block into resolve_module_in_search_path except the early-returning if package is a regular package (not a namespace package)

crates/red_knot_python_semantic/src/module_resolver/resolver.rs

carljm

LGTM!

crates/red_knot_python_semantic/src/module_resolver/resolver.rs

crates/red_knot_python_semantic/resources/mdtest/import/stub_packages.md

AlexWaygood

This looks basically good! Two edge cases (which I'm not sure if you handle correctly or not here -- but would be good to test explicitly in any event) relate to the case of a single-file module in site-packages. E.g. if foo.py is a single-file module at the top-level in site-packages, I don't think it's valid to have a foo-stubs.pyi file in site-packages; foo-stubs.pyi should not be treated as the "stubs package" for the runtime foo module. (I'd be curious to see what mypy/pyright do there.) But it is definitely valid to have a foo-stubs/__init__.pyi file in site-packages, and that should be treated as the stub for the runtime foo package.

AlexWaygood · 2025-04-04T18:29:16Z

crates/red_knot_python_semantic/resources/mdtest/import/stub_packages.md

+The runtime package is a regular package but the stubs are namespace packages. Pyright skips the
+stub package if the "regular" package isn't a namespace package. I'm not aware that the behavior
+here is specificed, and using the stubs without probing the runtime package first requires slightly
+fewer lookups.


this behaviour makes sense to me. I think in general, stubs packages should work exactly the same regardless of whether the runtime package is installed or not, and this behaviour seems consistent with that principle.

AlexWaygood · 2025-04-04T18:30:17Z

crates/red_knot_python_semantic/resources/mdtest/import/stub_packages.md

+It's recommended that stub packages use `__init__.pyi` files over `__init__.py` but it doesn't seem
+to be an enforced convention. At least, Pyright is fine with the following.


lol, I have no opinion at all on this 😆 I'd be fine if we didn't support it, but I'm also fine if we do! It seems very unlikely to come up

Yeah, not supporting it is harder, that's why I went with supporting it :)

MichaReiser · 2025-04-07T12:25:57Z

I added a test for foo-stubs.pyi. Pyright ignores those modules too. I didn't test mypy.

There's already a test covering foo-stubs/__init__.py

AlexWaygood · 2025-04-07T13:11:02Z

I didn't test mypy.

I confirmed that mypy also ignores a foo-stubs.pyi file in site-packages if you're importing a foo module at runtime!

AlexWaygood · 2025-04-07T13:11:53Z

Thanks @MichaReiser! :-D

* main: (42 commits) [playground] New default program (#17277) [red-knot] Add `--python-platform` CLI option (#17284) [red-knot] Allow ellipsis default params in stub functions (#17243) [red-knot] Fix stale syntax errors in playground (#17280) Update Rust crate clap to v4.5.35 (#17273) Fix RUF100 to detect unused file-level noqa directives with specific codes (#17042) (#17061) [ci] Fix pattern for code changes (#17275) [`airflow`] Update oudated `AIR301`, `AIR302` rules (#17123) [docs] fix formatting of "See Style Guide" link (#17272) [red-knot] Support stub packages (#17204) ruff_annotate_snippets: address unused code warnings [red-knot] Add a couple more tests for `*` imports (#17270) [red-knot] Add 'Format document' to playground (#17217) Update actions/setup-node action to v4.3.0 (#17259) Update actions/upload-artifact action to v4.6.2 (#17261) Update actions/download-artifact action to v4.2.1 (#17258) Update actions/setup-python action to v5.5.0 (#17260) Update actions/cache action to v4.2.3 (#17256) Update Swatinem/rust-cache action to v2.7.8 (#17255) Update actions/checkout action to v4.2.2 (#17257) ...

* main: (222 commits) [playground] New default program (#17277) [red-knot] Add `--python-platform` CLI option (#17284) [red-knot] Allow ellipsis default params in stub functions (#17243) [red-knot] Fix stale syntax errors in playground (#17280) Update Rust crate clap to v4.5.35 (#17273) Fix RUF100 to detect unused file-level noqa directives with specific codes (#17042) (#17061) [ci] Fix pattern for code changes (#17275) [`airflow`] Update oudated `AIR301`, `AIR302` rules (#17123) [docs] fix formatting of "See Style Guide" link (#17272) [red-knot] Support stub packages (#17204) ruff_annotate_snippets: address unused code warnings [red-knot] Add a couple more tests for `*` imports (#17270) [red-knot] Add 'Format document' to playground (#17217) Update actions/setup-node action to v4.3.0 (#17259) Update actions/upload-artifact action to v4.6.2 (#17261) Update actions/download-artifact action to v4.2.1 (#17258) Update actions/setup-python action to v5.5.0 (#17260) Update actions/cache action to v4.2.3 (#17256) Update Swatinem/rust-cache action to v2.7.8 (#17255) Update actions/checkout action to v4.2.2 (#17257) ...

MichaReiser added the ty Multi-file analysis & type inference label Apr 4, 2025

MichaReiser force-pushed the micha/stub-packages branch 4 times, most recently from 76bf9f8 to 3e40e95 Compare April 4, 2025 16:25

MichaReiser commented Apr 4, 2025

View reviewed changes

MichaReiser marked this pull request as ready for review April 4, 2025 16:26

MichaReiser requested review from AlexWaygood, carljm, dcreager and sharkdp as code owners April 4, 2025 16:26

MichaReiser commented Apr 4, 2025

View reviewed changes

crates/red_knot_python_semantic/src/module_resolver/resolver.rs Show resolved Hide resolved

carljm approved these changes Apr 4, 2025

View reviewed changes

crates/red_knot_python_semantic/src/module_resolver/resolver.rs Outdated Show resolved Hide resolved

crates/red_knot_python_semantic/src/module_resolver/resolver.rs Outdated Show resolved Hide resolved

AlexWaygood reviewed Apr 4, 2025

View reviewed changes

crates/red_knot_python_semantic/resources/mdtest/import/stub_packages.md Outdated Show resolved Hide resolved

AlexWaygood reviewed Apr 4, 2025

View reviewed changes

crates/red_knot_python_semantic/resources/mdtest/import/stub_packages.md Outdated Show resolved Hide resolved

AlexWaygood reviewed Apr 4, 2025

View reviewed changes

crates/red_knot_python_semantic/resources/mdtest/import/stub_packages.md Outdated Show resolved Hide resolved

AlexWaygood reviewed Apr 4, 2025

View reviewed changes

MichaReiser added 3 commits April 7, 2025 13:52

[red-knot] Support stub packages

90302e5

Improve logging

f945b61

Address 'easy' review comments

9f62e3c

MichaReiser force-pushed the micha/stub-packages branch from df4817d to 9f62e3c Compare April 7, 2025 11:58

Handle foo-stubs.pyi modules

62eac06

MichaReiser merged commit 6cc2d02 into main Apr 7, 2025
23 checks passed

MichaReiser deleted the micha/stub-packages branch April 7, 2025 12:40

		It's recommended that stub packages use `__init__.pyi` files over `__init__.py` but it doesn't seem
		to be an enforced convention. At least, Pyright is fine with the following.

[red-knot] Support stub packages #17204

[red-knot] Support stub packages #17204

Uh oh!

Conversation

MichaReiser commented Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

github-actions bot commented Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mypy_primer results

Uh oh!

MichaReiser Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

carljm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlexWaygood left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlexWaygood Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

AlexWaygood Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

MichaReiser Apr 7, 2025

Choose a reason for hiding this comment

Uh oh!

MichaReiser commented Apr 7, 2025

Uh oh!

Uh oh!

AlexWaygood commented Apr 7, 2025

Uh oh!

AlexWaygood commented Apr 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MichaReiser commented Apr 4, 2025 •

edited

Loading

github-actions bot commented Apr 4, 2025 •

edited

Loading

`mypy_primer` results

MichaReiser Apr 4, 2025 •

edited

Loading

AlexWaygood left a comment •

edited

Loading