Simplify and tighten type aliases #3524

ilevkivskyi · 2017-06-11T00:09:13Z

Fixes #2887
Fixes #3191
In addition this prohibits reassigning aliases, before something like this was allowed:

if random():
    Alias = Sequence[int]
else:
    Alias = Sequence[float]

def fun(arg: Alias) -> None:
    ...

Now this will say: Cannot assign multiple types to name "Alias" without an explicit "Type[...]" annotation. See #3494 for background.

Finally, this simplifies the logic in semanal.py, so that most processing of type aliases happens in one function.

ilevkivskyi · 2017-06-15T07:39:15Z

Marked this as WIP, since I have found another place (in checkexpr.py) where type variables resolution should be updated.

…a test

ilevkivskyi · 2017-06-22T21:41:46Z

OK, I fixed remaining points (also this PR now may help with #3355)
This is ready for review now.

gvanrossum · 2017-06-23T17:45:13Z

I fear that @JukkaL will have to review this, but I ran this against our Dropbox codebase, and found some occurrences of the following pattern:

class A: ...
class B: ...
def f(a: bool) -> None:
    cls = A
    if a:
        cls = B

We get several errors on the last line:

__tmp__.py:6: error: Cannot assign to a type
__tmp__.py:6: error: Cannot assign multiple types to name "c" without an explicit "Type[...]" annotation
__tmp__.py:6: error: Incompatible types in assignment (expression has type Type[B], variable has type Type[A])

Previously this code passed.

ilevkivskyi · 2017-06-23T19:20:54Z

@gvanrossum
I tried your example on master, and it gives an error:

Incompatible types in assignment (expression has type Type[B], variable has type Type[A])

probably B needs to inherit from A in order to be safe. The point is that previously type aliases were never created at function scope (only variables with types like Type[...]). I changed this to be in line with how aliases behave at module scope (i.e. depending on the presence of an explicit annotation).

But after you mention it is actually used like this, I think maybe it makes sense. If Jukka also agrees, then I will change this back (it is only one line). To me both behaviours make sense, on one hand explicit is better than explicit plus there will be less exceptions to explain, on another hand, it would be boring and probably un-Pythonic to put explicit annotations everywhere.

gvanrossum · 2017-06-23T20:34:14Z

You're right, in the actual code this has class B(A). And that passes in master but still gives two errors with your PR:

__tmp__.py:6: error: Cannot assign to a type
__tmp__.py:6: error: Cannot assign multiple types to name "cls" without an explicit "Type[...]" annotation

The solution would be to start with cls: Type[A] = A which is both verbose and a bit unintuitive.

ilevkivskyi · 2017-06-23T20:41:52Z

@gvanrossum

The solution would be to start with cls: Type[A] = A which is both verbose and a bit unintuitive.

OK, I will revert this soon. (But we need to document that type aliases should be defined at the module level, at a class and function scope they are always automatically variables with type Type[...], there is #3494 for the documentation part, I will make a separate PR).

While we are waiting for review by @JukkaL, could you please test whether this PR helps with #3355?
(There is still no definitive repro, but you said that the crash happens in your internal codebase).

gvanrossum · 2017-06-23T20:59:21Z

I wish I had a repro for #3355, but it's worse -- it occasionally happens to other users when they use --quick mode, but I haven't found a way to trigger it reliably myself yet. :-(

…mypy into rework-type-aliases

ilevkivskyi · 2017-06-29T11:24:02Z

@gvanrossum As promised, I removed creation of aliases at function scope.

ilevkivskyi · 2017-07-04T16:16:36Z

@JukkaL I fixed the merge problem, this should be ready for review again.

JukkaL

Mostly looks good, just a bunch of minor things.

JukkaL · 2017-07-05T15:21:30Z

test-data/unit/check-generics.test

+from typing import Tuple
+from lib import Transform
+
+def int_tf(m: int) -> Transform[int, str]:


Also use reveal_type to check that the declared type resulting from Transform[x, y] is as expected.

JukkaL · 2017-07-06T14:23:59Z

mypy/semanal.py

+                    # when this type alias gets "inlined", the Any is not explicit anymore,
+                    # so we need to replace it with non-explicit Anys
+                    res = make_any_non_explicit(res)
+                    if isinstance(res, Instance) and not res.args and isinstance(rvalue, RefExpr):


Add comment that describes the purpose of this special case.

JukkaL · 2017-07-06T14:24:37Z

mypy/semanal.py

@@ -1643,26 +1623,59 @@ def alias_fallback(self, tp: Type) -> Instance:
        fb_info.mro = [fb_info, self.object_type().type]
        return Instance(fb_info, [])

+    def analyze_alias(self, rvalue: Expression,
+                      allow_unnormalized: bool) -> Tuple[Optional[Type], List[str]]:


Add docstring. Explain arguments and the return value.

JukkaL · 2017-07-06T14:25:41Z

mypy/nodes.py

@@ -2267,6 +2268,7 @@ class SymbolTableNode:
    mod_id = ''  # type: Optional[str]
    # If this not None, override the type of the 'node' attribute.
    type_override = None  # type: Optional[mypy.types.Type]
+    alias_tvars = None  # type: Optional[List[str]]


Add comment.

Are these short or full names? If short ones, it would be more correct to have full names. For example, a.T and b.T could refer to different type variables -- this is very unlikely, though.

It looks like these are qualified names. I added a test for this, and a comment here.

JukkaL · 2017-07-06T14:28:24Z

mypy/fixup.py

@@ -88,6 +88,9 @@ def visit_symbol_table(self, symtab: SymbolTable) -> None:
                    if stnode is not None:
                        value.node = stnode.node
                        value.type_override = stnode.type_override
+                        if not value.type_override and self.quick_and_dirty:


I don't understand this -- add a comment. Is there a test case that tests this change?

Now I am also not sure it is needed. It was an attempt to fix #3355, but now I have a better idea. I removed this for now, and will make a separate PR later.

JukkaL · 2017-07-06T14:30:12Z

mypy/nodes.py

@@ -2342,6 +2346,7 @@ def serialize(self, prefix: str, name: str) -> JsonDict:
                data['node'] = self.node.serialize()
            if self.type_override is not None:
                data['type_override'] = self.type_override.serialize()
+                data['alias_tvars'] = self.alias_tvars


Is there a test case that triggers this (and the corresponding deserialization bit)?

Good catch! Added an incremental test.

JukkaL · 2017-07-06T14:31:30Z

mypy/semanal.py

+                    node.kind = TYPE_ALIAS
+                    node.type_override = res
+                    node.alias_tvars = alias_tvars
+                    if isinstance(s.rvalue, IndexExpr):


Add comment that explains the motivation of this if statement.

JukkaL · 2017-07-06T14:33:31Z

test-data/unit/check-type-aliases.test

+from typing import TypeVar, Sequence, Type
+T = TypeVar('T')
+
+A: Type[float] = int


Also test assignment after initialization (it should be fine since there is an explicit annotation)?

Yes, added it below.

JukkaL · 2017-07-06T14:45:30Z

test-data/unit/check-generics.test

+
+[case testGenericTypeAliasesImportingWithoutTypeVarError]
+from a import Alias
+x: Alias[int, str]  # E: Bad number of arguments for type alias, expected: 1, given: 2


Add reveal_type(x) to see what type gets assigned in this case.

JukkaL · 2017-07-06T16:05:35Z

mypy/semanal.py

    def check_and_set_up_type_alias(self, s: AssignmentStmt) -> None:
        """Check if assignment creates a type alias and set it up as needed."""
-        # For now, type aliases only work at the top level of a module.
+        # Type aliases are created only at module scope, at class and function scopes
+        # assignments create class valued members and local variables with type object types.


This caused an error in an internal Dropbox codebase. We have code that looks like this:

from typing import Union class A: B = Union[int, str] def f(self) -> B: # this PR makes this an invalid type pass

Maybe type aliases in the class body should still be supported? Alternatively, maybe anything where the rvalue is a valid type that is an IndexExpr should still be considered as a type alias, i.e. only restrict simple cases like X = Y.

OK, now the rules a following: explicit annotation -- always a variable, otherwise:

module scope -- always an alias

class scope -- an alias for subscripted, a variable for simple class objects

function scope -- always a variable

ilevkivskyi · 2017-07-07T08:53:19Z

@JukkaL Thanks for review! All comments should be addressed now.

JukkaL

Thanks, now looks good!

ilevkivskyi added 4 commits June 10, 2017 21:15

Start reworking type aliases

d07259a

Fix another alias issue

34de9ae

Add some tests

6765a08

Prohibit reassigning aliases

8dc40d6

ilevkivskyi changed the title ~~Simplify and tighten type aliases~~ [WIP] Simplify and tighten type aliases Jun 15, 2017

ilevkivskyi added 4 commits June 22, 2017 21:36

Merge upstream

b76dafc

Merge remote-tracking branch 'upstream/master' into rework-type-aliases

aa3831e

Factor out common code, fix runtime aliases and deserialization; add …

50056c4

…a test

Fix typo

b2212dd

ilevkivskyi changed the title ~~[WIP] Simplify and tighten type aliases~~ Simplify and tighten type aliases Jun 22, 2017

gvanrossum mentioned this pull request Jun 23, 2017

Problem with alias for Optional[str] #3599

Closed

ilevkivskyi requested a review from JukkaL June 23, 2017 19:21

ilevkivskyi assigned JukkaL and unassigned JukkaL Jun 23, 2017

Ivan Levkivskyi added 3 commits June 29, 2017 12:46

Don't create aliases at function scope, create variables instead

598a511

Merge branch 'rework-type-aliases' of https://github.com/ilevkivskyi/…

4d8cef1

…mypy into rework-type-aliases

Fix broken merge

f90dabf

ilevkivskyi assigned JukkaL Jul 4, 2017

ilevkivskyi and others added 2 commits July 4, 2017 17:43

Merge branch 'master' into rework-type-aliases

7970238

Fix strict-optional self-test :-)

8c35be0

ilevkivskyi mentioned this pull request Jul 4, 2017

Release mypy 0.520 #3655

Closed

Merge branch 'master' into rework-type-aliases

91e2be4

JukkaL requested changes Jul 6, 2017

View reviewed changes

ilevkivskyi added 4 commits July 7, 2017 00:28

First part of review comments

5cc1953

Second part of review comments

7e7c1b7

Minor fix; reduce nesting in check_and_set_up_type_alias

efe6342

Better type on error; shorten docstring; minor refactoring

91bda84

JukkaL approved these changes Jul 7, 2017

View reviewed changes

JukkaL merged commit 70d5f45 into python:master Jul 7, 2017

ilevkivskyi mentioned this pull request Jul 19, 2017

New --quick crash: "assert override is not None" #3355

Closed

ilevkivskyi mentioned this pull request Aug 7, 2017

Qualified alias names are mishandled #3145

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simplify and tighten type aliases #3524

Simplify and tighten type aliases #3524

ilevkivskyi commented Jun 11, 2017 •

edited

Loading

ilevkivskyi commented Jun 15, 2017

ilevkivskyi commented Jun 22, 2017

gvanrossum commented Jun 23, 2017

ilevkivskyi commented Jun 23, 2017

gvanrossum commented Jun 23, 2017

ilevkivskyi commented Jun 23, 2017

gvanrossum commented Jun 23, 2017

ilevkivskyi commented Jun 29, 2017

ilevkivskyi commented Jul 4, 2017

JukkaL left a comment

JukkaL Jul 5, 2017

JukkaL Jul 6, 2017

JukkaL Jul 6, 2017

JukkaL Jul 6, 2017

ilevkivskyi Jul 7, 2017

JukkaL Jul 6, 2017

ilevkivskyi Jul 7, 2017

JukkaL Jul 6, 2017

ilevkivskyi Jul 7, 2017

JukkaL Jul 6, 2017

JukkaL Jul 6, 2017

ilevkivskyi Jul 7, 2017

JukkaL Jul 6, 2017

JukkaL Jul 6, 2017

ilevkivskyi Jul 7, 2017

ilevkivskyi commented Jul 7, 2017

JukkaL left a comment

Simplify and tighten type aliases #3524

Simplify and tighten type aliases #3524

Conversation

ilevkivskyi commented Jun 11, 2017 • edited Loading

ilevkivskyi commented Jun 15, 2017

ilevkivskyi commented Jun 22, 2017

gvanrossum commented Jun 23, 2017

ilevkivskyi commented Jun 23, 2017

gvanrossum commented Jun 23, 2017

ilevkivskyi commented Jun 23, 2017

gvanrossum commented Jun 23, 2017

ilevkivskyi commented Jun 29, 2017

ilevkivskyi commented Jul 4, 2017

JukkaL left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilevkivskyi commented Jul 7, 2017

JukkaL left a comment

Choose a reason for hiding this comment

ilevkivskyi commented Jun 11, 2017 •

edited

Loading