Enable recursive type aliases behind a flag

[ilevkivskyi]

This PR exposes recursive type aliases that were secretly there for last ~3 years. For now they will still be behind an opt-in flag, because they are not production ready.

As we discussed with @JukkaL during PyCon, I use couple hacks to make them minimally usable, as proper solutions will take time. I may clean up some of them in near future (or may not).

You can see few added test cases to get an idea of what is supported, example:

Nested = Sequence[Union[T, Nested[T]]]

def flatten(seq: Nested[T]) -> List[T]:
    flat: List[T] = []
    for item in seq:
        if isinstance(item, Sequence):
            res.extend(flatten(item))
        else:
            res.append(item)
    return flat

reveal_type(flatten([1, [2, [3]]]))  # N: Revealed type is "builtins.list[builtins.int]"

There are several known issues, missing features, etc. Some comments:

• People use get_proper_type() too much (I also used to do this). This can cause giant error messages, infinite recursion, and even performance issues. We should try to avoid expanding unless necessary, and always pass on original type when possible.
• We should fail gracefully on invalid aliases A = Union[A, ...] and B = Type[B] (first is theoretically meaningless, second is technically valid, but hard to support)
• We need a better formatting for recursive types. Currently it is just two verbosity levels: deep unroll with full names or just Name[Arg, ...].
• Local (function level) recursive aliases may need to be prohibited (or we can store them in global namespace, like we do with local named tuples etc).
• Recursive classes (most notably class str(Sequence[str]): ...) can interfere with recursive aliases causing infinite recursion. I will likely fix this soon (should be easy unless I am missing something).
• Recursive named tuples and typed dicts are still not supported (unless you use dirty NT = Union[_NT, _NT] trick). Btw supporting recursive and generic named tuples/typed dicts should be quite easy if we agree to store them internally as type aliases (or maybe we can introduce couple other non-proper types).
• We should get rid of is_same_type() altogether, replacing with either equality or (proper) equivalence call by call. Also I am probably going to unify (proper) subtype visitors.

[AlexWaygood]

• Recursive named tuples and typed dicts are still not supported

I think there's already an open PR to support recursive namedtuples btw:

• Support recursive NamedTuple classes #13029

[ilevkivskyi]

I think there's already an open PR to support recursive namedtuples btw:

Yeah, but it will not work, our internal representation is not suitable for them being recursive.

[ilevkivskyi]

@hauntsaninja @JelleZijlstra could one of you (or both) please take a look at this?

[JelleZijlstra]

This is exciting progress. I have an optional suggestion.

[JelleZijlstra]

Shouldn't we always return True if is_pep_613(s) returns True?

It may also be reasonable to require explicit TypeAlias for recursive aliases.

[ilevkivskyi]

Shouldn't we always return True if is_pep_613(s) returns True?

We are interested only in aliases that can be recursive, so as a micro-optimization we only do this for r.h.s that can be recursive. I will add a comment on this.

It may also be reasonable to require explicit TypeAlias for recursive aliases.

Hm, yeah this will simplify logic a bit. OTOH it may be hard to reliably detect cases where it was intended and missing, to give a good error message. I will keep the current logic as is, it is easy to change later, as this part is decoupled from everything else.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

dragonchain (https://github.com/dragonchain/dragonchain)
- dragonchain/lib/database/redis.py:399:36: error: Argument 2 to "zadd" of "SortedSetCommands" has incompatible type "Dict[str, int]"; expected "Mapping[Union[str, bytes], Union[bytes, float, str]]"
+ dragonchain/lib/database/redis.py:399:36: error: Argument 2 to "zadd" of "SortedSetCommands" has incompatible type "Dict[str, int]"; expected "Mapping[Union[str, bytes], Union[bytes, float, int, str]]"
- dragonchain/lib/database/redis_utest.py:218:9: error: "Callable[[Union[str, bytes], Mapping[Union[str, bytes], Union[bytes, float, str]], bool, bool, bool, bool, Optional[Any], Optional[Any]], int]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:218:9: error: "Callable[[Union[str, bytes], Mapping[Union[str, bytes], Union[bytes, float, int, str]], bool, bool, bool, bool, Optional[Any], Optional[Any]], int]" has no attribute "assert_called_once_with"

[ilevkivskyi]

I implemented all of the minor follow-up items from the list above. We are only left with more major items: make TypedDicts and NamedTuples recursive and generic.

TBH I was thinking again about this, and it seems to me it would be so much simpler to do this by storing them in symbol tables as TypeAliases, instead of as TypeInfos that are used for fallbacks. One could say storing them as type aliases is a hack, but IMO current design is also a hack (docstring says TypeInfos should be in one-to-one correspondence with classes). So we would be replacing one hack with another (IMO less ugly) hack.

As I mentioned elsewhere it also kind of makes sense, as these two are conceptually similar (and this analogy would be even stronger if we would allow inline NamedTuples/TypedDicts):

Point = Tuple[int, int]
Point = NamedTuple('Point', [('x', int), ('y', int)])

Opts = Dict[str, int]
Opts = TypedDict('Opts', {'start': int, 'limit': int})

The other option of adding new dedicated nodes and/or non-proper types also seems dangerous, as we already have various special-casing for TypeAliasType to prevent infinite recursion etc.

[ilevkivskyi]

Hm, it may be easy to make the above proposed logic conditional on the --enable-recursive-aliases flag. I will try this tomorrow.

Btw in the meantime I discovered another case were can't get rid of infinite recursion, see #13352. I even switched to full nerd mode and read some recent research papers, it looks like it may be a still open problem (funny thing is that the term I invented for myself to call them "diverging" is actually the official name for these things).

[ilevkivskyi]

OK, so I tried to do the above and it worked quite well, couple comments:

• After all I think switching from TypeInfo to TypeAlias under a flag is a bad idea, this is a significant change, and it is better if it will be constantly tested (I was anyway running all tests with this flag while working on this).
• There are still around a dozen or so tests failing if I enable the switch. First, it was assumed in several cases that named tuple will be fully analyzed only once, which can't be the case anymore. Also various internal machinery (like fixup.py, fine grained deps etc) expect named tuples to be TypeInfos. It is all fixable in one-two days, but I wanted to try another experiment: allow TypeAliasType to point to either TypeAlias or TypeInfo, this may be simpler to implement.

[ilevkivskyi]

After all, the most promising seem to be a compromise approach, where we still store named tuples as TypeInfos in symbol tables, and TypeAliasType still always points to a TypeAlias. But, whenever a type analyzer sees a TypeInfo with a tuple type, it creates a type alias, see #13371.

[ilevkivskyi]

The last major item of my "grand plan" is to support generic NamedTuples and TypedDicts. The latter should be easy unless I am missing something. But for tuple types we have a problem with fallbacks. In some sense this is not new, we have special delayed treatment for tuple fallbacks. But now we will probably need to delay it "even more".

To illustrate, what Instance would we put in bases in class C(Tuple[T, S]): ...? In case of actual types we can just schedule a patch and fill in the fallback after semantic analysis. But for type variables there is nothing we can put there before we apply type arguments. This is problematic because bases are stored in TypeInfo, i.e. it is just one type for all instances of C. My current idea is to just say we don't care, and put tuple[Any, ...] in the base, and compute the fallback wherever needed on the fly. We already do this in several places, so it sounds totally doable. Most notably we would need to special case map_instance_to_supertype(). You may say, well it is a bit dangerous to introduce a potential cycle map_instance_to_supertype -> join_types -> is_subtype -> map_instance_to_supertype, but in fact we already have a loop map_instance_to_supertype -> expand_type -> make_simplified_union -> is_subtype -> map_instance_to_supertype and it doesn't case much trouble (I have found only one bug on the tracker caused by this #6730).

I will think a bit more, and if I will not come up with a better idea, I will move on with the "on the fly fallbacks" tomorrow.

[sg495]

I'll plainly admit my ignorance here: I still don't quite understand how this test actually runs fine in 3.9.

For me, running the first example from the test in 3.9.13 results in NameError:

from typing import Dict, List, Union
JSON = Union[str, List[JSON], Dict[str, JSON]]
# NameError: name 'JSON' is not defined

On the other hand, the same line written with forward references runs without issues (and mypy 0.991 correctly catches the nested list item error):

from typing import Dict, List, Union
JSON = Union[str, List["JSON"], Dict[str, "JSON"]]
class Bad: ...
x: JSON = ["foo", {"bar": [Bad()]}]
# mypy error list-item - List item 0 has incompatible type "Bad"; expected "Union[str, List[JSON], Dict[str, JSON]]"

@ilevkivskyi: when you have a second, could you please write a brief comment clarifying why that worked? I know the example works fine with forward refs, but I'm intrigued by the possibility of it running without (it'd make for much neater code 😄).

[JelleZijlstra]

This test won't work at runtime. Mypy's test suite does not actually run its tests at runtime.

PEP-695 may provide a mechanism for writing recursive type aliases without stringification, but that's still in the future.