[ty] Substitute for `typing.Self` when checking protocol members #21569

dcreager · 2025-11-21T19:04:02Z

This patch updates our protocol assignability checks to substitute for any occurrences of typing.Self in method signatures, replacing it with the class being checked for assignability against the protocol.

This requires a new helper method on signatures, apply_self, which substitutes occurrences of typing.Self without binding the self parameter.

We also update the try_upcast_to_callable method. Before, it would return a Type, since certain types upcast to a union of callables, not to a single callable. However, even in that case, we know that every element of the union is a callable. We now return a vector of CallableType. (Actually a smallvec to handle the most common case of a single callable; and wrapped in a new type so that we can provide helper methods.) If there is more than one element in the result, it represents a union of callables. This lets callers get at the CallableType instances in a more type-safe way. (This makes it easier for our protocol checking code to call the new apply_self helper.) We also provide an into_type method so that callers that really do want a Type can get the original result easily.

dcreager · 2025-11-21T19:04:36Z

crates/ty_python_semantic/resources/mdtest/protocols.md

 static_assert(is_assignable_to(NominalWithSelf, UsesSelf))
-# TODO: should pass
-static_assert(is_subtype_of(NominalWithSelf, UsesSelf))  # error: [static-assert-error]
+static_assert(is_subtype_of(NominalWithSelf, UsesSelf))


There are other Self-related tests in this file that are still TODO, since they also require being able to check against generic protocol members correctly.

astral-sh-bot · 2025-11-21T19:06:07Z

Diagnostic diff on typing conformance tests

Changes were detected when running ty on typing conformance tests

--- old-output.txt	2025-11-24 18:49:42.174021411 +0000
+++ new-output.txt	2025-11-24 18:49:45.948033270 +0000
@@ -504,6 +504,7 @@
 generics_self_basic.py:64:38: error[invalid-return-type] Function always implicitly returns `None`, which is not assignable to return type `Self@set_value`
 generics_self_basic.py:67:26: error[invalid-type-form] Special form `typing.Self` expected no type parameter
 generics_self_protocols.py:61:19: error[invalid-argument-type] Argument to function `accepts_shape` is incorrect: Expected `ShapeProtocol`, found `BadReturnType`
+generics_self_protocols.py:64:19: error[invalid-argument-type] Argument to function `accepts_shape` is incorrect: Expected `ShapeProtocol`, found `ReturnDifferentClass`
 generics_self_usage.py:50:34: error[invalid-assignment] Object of type `def foo(self) -> int` is not assignable to `(typing.Self, /) -> int`
 generics_self_usage.py:73:14: error[invalid-type-form] Variable of type `typing.Self` is not allowed in a type expression
 generics_self_usage.py:73:23: error[invalid-type-form] Variable of type `typing.Self` is not allowed in a type expression
@@ -1002,5 +1003,5 @@
 typeddicts_usage.py:28:17: error[missing-typed-dict-key] Missing required key 'name' in TypedDict `Movie` constructor
 typeddicts_usage.py:28:18: error[invalid-key] Unknown key "title" for TypedDict `Movie`: Unknown key "title"
 typeddicts_usage.py:40:24: error[invalid-type-form] The special form `typing.TypedDict` is not allowed in type expressions. Did you mean to use a concrete TypedDict or `collections.abc.Mapping[str, object]` instead?
-Found 1004 diagnostics
+Found 1005 diagnostics
 WARN A fatal error occurred while checking some files. Not all project files were analyzed. See the diagnostics list above for details.

astral-sh-bot · 2025-11-21T21:06:36Z

`mypy_primer` results

Changes were detected when running on open source projects

scikit-build-core (https://github.com/scikit-build/scikit-build-core)
- src/scikit_build_core/build/wheel.py:98:20: error[no-matching-overload] No overload of bound method `__init__` matches arguments
- Found 45 diagnostics
+ Found 44 diagnostics

sympy (https://github.com/sympy/sympy)
- sympy/polys/domains/domain.py:586:51: warning[unused-ignore-comment] Unused blanket `type: ignore` directive
- Found 14879 diagnostics
+ Found 14878 diagnostics

core (https://github.com/home-assistant/core)
- homeassistant/helpers/storage.py:395:13: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[Any] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[dict[str, Any] | list[Any] | str | ... omitted 3 union elements]]` cannot be called with key of type `Literal["version"]` on object of type `list[JsonValueType]`
+ homeassistant/helpers/storage.py:395:13: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[JsonValueType] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[JsonValueType]]` cannot be called with key of type `Literal["version"]` on object of type `list[JsonValueType]`
- homeassistant/helpers/storage.py:396:17: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[Any] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[dict[str, Any] | list[Any] | str | ... omitted 3 union elements]]` cannot be called with key of type `Literal["minor_version"]` on object of type `list[JsonValueType]`
+ homeassistant/helpers/storage.py:396:17: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[JsonValueType] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[JsonValueType]]` cannot be called with key of type `Literal["minor_version"]` on object of type `list[JsonValueType]`
- homeassistant/helpers/storage.py:398:22: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[Any] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[dict[str, Any] | list[Any] | str | ... omitted 3 union elements]]` cannot be called with key of type `Literal["data"]` on object of type `list[JsonValueType]`
+ homeassistant/helpers/storage.py:398:22: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[JsonValueType] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[JsonValueType]]` cannot be called with key of type `Literal["data"]` on object of type `list[JsonValueType]`
- homeassistant/helpers/storage.py:403:17: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[Any] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[dict[str, Any] | list[Any] | str | ... omitted 3 union elements]]` cannot be called with key of type `Literal["version"]` on object of type `list[JsonValueType]`
+ homeassistant/helpers/storage.py:403:17: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[JsonValueType] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[JsonValueType]]` cannot be called with key of type `Literal["version"]` on object of type `list[JsonValueType]`
- homeassistant/helpers/storage.py:404:17: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[Any] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[dict[str, Any] | list[Any] | str | ... omitted 3 union elements]]` cannot be called with key of type `Literal["minor_version"]` on object of type `list[JsonValueType]`
+ homeassistant/helpers/storage.py:404:17: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[JsonValueType] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[JsonValueType]]` cannot be called with key of type `Literal["minor_version"]` on object of type `list[JsonValueType]`
- homeassistant/helpers/storage.py:409:57: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[Any] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[dict[str, Any] | list[Any] | str | ... omitted 3 union elements]]` cannot be called with key of type `Literal["version"]` on object of type `list[JsonValueType]`
+ homeassistant/helpers/storage.py:409:57: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[JsonValueType] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[JsonValueType]]` cannot be called with key of type `Literal["version"]` on object of type `list[JsonValueType]`
- homeassistant/helpers/storage.py:409:74: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[Any] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[dict[str, Any] | list[Any] | str | ... omitted 3 union elements]]` cannot be called with key of type `Literal["data"]` on object of type `list[JsonValueType]`
+ homeassistant/helpers/storage.py:409:74: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[JsonValueType] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[JsonValueType]]` cannot be called with key of type `Literal["data"]` on object of type `list[JsonValueType]`
- homeassistant/helpers/storage.py:413:25: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[Any] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[dict[str, Any] | list[Any] | str | ... omitted 3 union elements]]` cannot be called with key of type `Literal["version"]` on object of type `list[JsonValueType]`
+ homeassistant/helpers/storage.py:413:25: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[JsonValueType] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[JsonValueType]]` cannot be called with key of type `Literal["version"]` on object of type `list[JsonValueType]`
- homeassistant/helpers/storage.py:413:42: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[Any] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[dict[str, Any] | list[Any] | str | ... omitted 3 union elements]]` cannot be called with key of type `Literal["minor_version"]` on object of type `list[JsonValueType]`
+ homeassistant/helpers/storage.py:413:42: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[JsonValueType] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[JsonValueType]]` cannot be called with key of type `Literal["minor_version"]` on object of type `list[JsonValueType]`
- homeassistant/helpers/storage.py:413:65: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[Any] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[dict[str, Any] | list[Any] | str | ... omitted 3 union elements]]` cannot be called with key of type `Literal["data"]` on object of type `list[JsonValueType]`
+ homeassistant/helpers/storage.py:413:65: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[JsonValueType] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[JsonValueType]]` cannot be called with key of type `Literal["data"]` on object of type `list[JsonValueType]`
- homeassistant/helpers/storage.py:416:24: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[Any] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[dict[str, Any] | list[Any] | str | ... omitted 3 union elements]]` cannot be called with key of type `Literal["version"]` on object of type `list[JsonValueType]`
+ homeassistant/helpers/storage.py:416:24: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[JsonValueType] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[JsonValueType]]` cannot be called with key of type `Literal["version"]` on object of type `list[JsonValueType]`
- homeassistant/helpers/storage.py:418:30: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[Any] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[dict[str, Any] | list[Any] | str | ... omitted 3 union elements]]` cannot be called with key of type `Literal["data"]` on object of type `list[JsonValueType]`
+ homeassistant/helpers/storage.py:418:30: error[invalid-argument-type] Method `__getitem__` of type `Overload[(i: SupportsIndex, /) -> dict[str, Any] | list[JsonValueType] | str | ... omitted 3 union elements, (s: slice[Any, Any, Any], /) -> list[JsonValueType]]` cannot be called with key of type `Literal["data"]` on object of type `list[JsonValueType]`

Memory usage changes were detected when running on open source projects

flake8 (https://github.com/pycqa/flake8)
-     memo metadata = ~7MB
+     memo metadata = ~8MB

astral-sh-bot · 2025-11-21T21:20:45Z

`ecosystem-analyzer` results

Lint rule	Removed	Changed
`invalid-argument-type`	0	12
`unused-ignore-comment`	1	0
Total	1	12

Full report with detailed diff (timing results)

dcreager · 2025-11-21T21:40:09Z

Typing conformance result is a new true positive

dcreager · 2025-11-21T21:51:42Z

Ecosystem results look good now, too — showing a more precise type in the diagnostics

AlexWaygood · 2025-11-22T10:50:33Z

(I'd love to take a look at this one before it goes in, if that's okay, just to make sure I understand everything that's going on!)

dcreager · 2025-11-22T16:30:55Z

(I'd love to take a look at this one before it goes in, if that's okay, just to make sure I understand everything that's going on!)

👍 no problem, I'll wait to merge on an okay from you

AlexWaygood

Thank you!!

Typing conformance result is a new true positive

Would you be able to add a regression test for this to protocols.md?

crates/ty_python_semantic/src/types.rs

crates/ty_python_semantic/src/types/protocol_class.rs

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

* origin/main: (27 commits) [ty] Add hint about resolved Python version when a user attempts to import a member added on a newer version (#21615) Use release commit for actions/checkout (#21610) [ty] Add failing mdtest for known `Protocol` panic (#21594) [`parser`] Fix panic when parsing IPython escape command expressions (#21480) Fix cargo shear in CI (#21609) Update actions/checkout digest to c2d88d3 (#21601) Update dependency ruff to v0.14.6 (#21603) Update astral-sh/setup-uv action to v7.1.4 (#21602) Update Rust crate clap to v4.5.53 (#21604) Update taiki-e/install-action action to v2.62.56 (#21608) Update Rust crate hashbrown to v0.16.1 (#21605) Update Rust crate indexmap to v2.12.1 (#21606) Update Rust crate syn to v2.0.111 (#21607) [ty] Check method definitions on subclasses for Liskov violations (#21436) [ty] Fix panic for unclosed string literal in type annotation position (#21592) [ty] Fix rendering of unused suppression diagnostic (#21580) [ty] Improve lsp handling of hover/goto on imports (#21572) [ty] Improve diagnostics when a submodule is not available as an attribute on a module-literal type (#21561) [ty] Improve concise diagnostics for invalid exceptions when a user catches a tuple of objects (#21578) [ty] upgrade salsa (#21575) ...

dcreager · 2025-11-24T18:24:04Z

Would you be able to add a regression test for this to protocols.md?

Done

crates/ty_python_semantic/src/types/class.rs

crates/ty_python_semantic/src/types.rs

…d-typevar * origin/main: (30 commits) [ty] Double click to insert inlay hint (#21600) [ty] Switch the error code from `unresolved-attribute` to `possibly-missing-attribute` for submodules that may not be available (#21618) [ty] Substitute for `typing.Self` when checking protocol members (#21569) [ty] Don't suggest things that aren't subclasses of `BaseException` after `raise` [ty] Add hint about resolved Python version when a user attempts to import a member added on a newer version (#21615) Use release commit for actions/checkout (#21610) [ty] Add failing mdtest for known `Protocol` panic (#21594) [`parser`] Fix panic when parsing IPython escape command expressions (#21480) Fix cargo shear in CI (#21609) Update actions/checkout digest to c2d88d3 (#21601) Update dependency ruff to v0.14.6 (#21603) Update astral-sh/setup-uv action to v7.1.4 (#21602) Update Rust crate clap to v4.5.53 (#21604) Update taiki-e/install-action action to v2.62.56 (#21608) Update Rust crate hashbrown to v0.16.1 (#21605) Update Rust crate indexmap to v2.12.1 (#21606) Update Rust crate syn to v2.0.111 (#21607) [ty] Check method definitions on subclasses for Liskov violations (#21436) [ty] Fix panic for unclosed string literal in type annotation position (#21592) [ty] Fix rendering of unused suppression diagnostic (#21580) ...

* main: [ty] Extend Liskov checks to also cover classmethods and staticmethods (astral-sh#21598) Dogfood ty on the `scripts` directory (astral-sh#21617) [ty] support generic aliases in `type[...]`, like `type[C[int]]` (astral-sh#21552) [ty] Retain the function-like-ness of `Callable` types when binding `self` (astral-sh#21614) [ty] Distinguish "unconstrained" from "constrained to any type" (astral-sh#21539) Disable ty workspace diagnostics for VSCode users (astral-sh#21620) [ty] Double click to insert inlay hint (astral-sh#21600) [ty] Switch the error code from `unresolved-attribute` to `possibly-missing-attribute` for submodules that may not be available (astral-sh#21618) [ty] Substitute for `typing.Self` when checking protocol members (astral-sh#21569) [ty] Don't suggest things that aren't subclasses of `BaseException` after `raise` [ty] Add hint about resolved Python version when a user attempts to import a member added on a newer version (astral-sh#21615)

dcreager added 2 commits November 21, 2025 13:37

upcast into CallableType not Type

296d3e6

apply Self types when checking protocols

062f148

dcreager requested review from AlexWaygood, carljm and sharkdp as code owners November 21, 2025 19:04

dcreager added the ty Multi-file analysis & type inference label Nov 21, 2025

dcreager commented Nov 21, 2025

View reviewed changes

dcreager added 4 commits November 21, 2025 14:22

fix that test

c065a73

fix some calls

58ab6b7

early return

1ea65e8

fix stack overflow

1d428cb

dcreager added the ecosystem-analyzer label Nov 21, 2025

use literal fallback

9cd5254

dcreager added ecosystem-analyzer and removed ecosystem-analyzer labels Nov 21, 2025

carljm approved these changes Nov 22, 2025

View reviewed changes

AlexWaygood approved these changes Nov 23, 2025

View reviewed changes

dcreager and others added 6 commits November 24, 2025 12:34

Update crates/ty_python_semantic/src/types.rs

1575381

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

option for no callable

08145db

simpler callable construction

b7ef551

clippy

e4f47ee

add mdtest for new typing conformance pass

9b288dc

AlexWaygood approved these changes Nov 24, 2025

View reviewed changes

crates/ty_python_semantic/src/types/class.rs Outdated Show resolved Hide resolved

crates/ty_python_semantic/src/types.rs Show resolved Hide resolved

remove unneeded option

a5c87aa

AlexWaygood approved these changes Nov 24, 2025

View reviewed changes

dcreager merged commit 03fe560 into main Nov 24, 2025
41 checks passed

dcreager deleted the dcreager/protocol-self branch November 24, 2025 19:05

[ty] Substitute for typing.Self when checking protocol members #21569

[ty] Substitute for typing.Self when checking protocol members #21569

Uh oh!

Conversation

dcreager commented Nov 21, 2025

Uh oh!

dcreager Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

astral-sh-bot bot commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Diagnostic diff on typing conformance tests

Uh oh!

astral-sh-bot bot commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mypy_primer results

Uh oh!

astral-sh-bot bot commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ecosystem-analyzer results

Uh oh!

dcreager commented Nov 21, 2025

Uh oh!

dcreager commented Nov 21, 2025

Uh oh!

AlexWaygood commented Nov 22, 2025

Uh oh!

dcreager commented Nov 22, 2025

Uh oh!

AlexWaygood left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dcreager commented Nov 24, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[ty] Substitute for `typing.Self` when checking protocol members #21569

[ty] Substitute for `typing.Self` when checking protocol members #21569

astral-sh-bot bot commented Nov 21, 2025 •

edited

Loading

astral-sh-bot bot commented Nov 21, 2025 •

edited

Loading

`mypy_primer` results

astral-sh-bot bot commented Nov 21, 2025 •

edited

Loading

`ecosystem-analyzer` results