[2.7] urlparse.urlparse should use AnyStr, but can't due to NamedTuple

[mr-c]

Fair chance I'm doing something wrong here, any pointers?

(env3) michael@dan:~/schema_salad$ make mypy
MYPYPATH=typeshed/2.7 mypy --py2 --disallow-untyped-calls schema_salad
Traceback (most recent call last):
  File "/home/michael/src/mypy/env3/bin/mypy", line 6, in <module>
    main(__file__)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/main.py", line 54, in main
    res = type_check_only(sources, bin_dir, options)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/main.py", line 102, in type_check_only
    python_path=options.python_path)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/build.py", line 209, in build
    dispatch(sources, manager)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/build.py", line 1325, in dispatch
    process_graph(graph, manager)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/build.py", line 1456, in process_graph
    process_stale_scc(graph, scc)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/build.py", line 1482, in process_stale_scc
    graph[id].semantic_analysis()
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/build.py", line 1292, in semantic_analysis
    self.manager.semantic_analyzer.visit_file(self.tree, self.xpath)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/semanal.py", line 246, in visit_file
    self.accept(d)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/semanal.py", line 2251, in accept
    node.accept(self)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/nodes.py", line 650, in accept
    return visitor.visit_class_def(self)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/semanal.py", line 561, in visit_class_def
    defn.defs.accept(self)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/nodes.py", line 715, in accept
    return visitor.visit_block(self)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/semanal.py", line 1012, in visit_block
    self.accept(s)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/semanal.py", line 2251, in accept
    node.accept(self)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/nodes.py", line 462, in accept
    return visitor.visit_func_def(self)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/semanal.py", line 316, in visit_func_def
    self.analyze_function(defn)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/semanal.py", line 468, in analyze_function
    defn.type = self.anal_type(defn.type)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/semanal.py", line 1037, in anal_type
    return t.accept(a)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/types.py", line 496, in accept
    return visitor.visit_callable_type(self)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/typeanal.py", line 180, in visit_callable_type
    return t.copy_modified(arg_types=self.anal_array(t.arg_types),
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/typeanal.py", line 247, in anal_array
    res.append(t.accept(self))
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/types.py", line 637, in accept
    return visitor.visit_tuple_type(self)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/typeanal.py", line 194, in visit_tuple_type
    return TupleType(self.anal_array(t.items), fallback, t.line)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/typeanal.py", line 247, in anal_array
    res.append(t.accept(self))
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/types.py", line 340, in accept
    return visitor.visit_type_var(self)
  File "/home/michael/src/mypy/env3/lib/python3.5/site-packages/mypy/typeanal.py", line 177, in visit_type_var
    raise RuntimeError('TypeVarType is already analyzed')
RuntimeError: TypeVarType is already analyzed

diff --git a/typeshed/2.7/urlparse.pyi b/typeshed/2.7/urlparse.pyi
index ee2e4c2..b7862c7 100644
--- a/typeshed/2.7/urlparse.pyi
+++ b/typeshed/2.7/urlparse.pyi
@@ -1,6 +1,6 @@
 # Stubs for urlparse (Python 2)

-from typing import Dict, List, NamedTuple, Tuple, Sequence, Union, overload
+from typing import AnyStr, Dict, Generic, List, NamedTuple, Tuple, Sequence, Union, overload

 uses_relative = []  # type: List[str]
 uses_netloc = []  # type: List[str]
@@ -24,9 +24,9 @@ class ResultMixin(object):
     def port(self) -> int: ...

 class SplitResult(NamedTuple('SplitResult', [
-        ('scheme', str), ('netloc', str), ('path', str), ('query', str), ('fragment', str)
-    ]), ResultMixin):
-    def geturl(self) -> str: ...
+        ('scheme', AnyStr), ('netloc', AnyStr), ('path', AnyStr), ('query', AnyStr), ('fragment', AnyStr)
+    ]), ResultMixin, Generic[AnyStr]):
+    def geturl(self) -> AnyStr: ...

 class ParseResult(NamedTuple('ParseResult', [
         ('scheme', str), ('netloc', str), ('path', str), ('params', str), ('query', str),
@@ -35,7 +35,7 @@ class ParseResult(NamedTuple('ParseResult', [
     def geturl(self) -> str: ...

 def urlparse(url: Union[str, unicode], scheme: str = ..., allow_fragments: bool = ...) -> ParseResult: ...
-def urlsplit(url: Union[str, unicode], scheme: str = ..., allow_fragments: bool = ...) -> SplitResult: ...
+def urlsplit(url: AnyStr, scheme: str = ..., allow_fragments: bool = ...) -> SplitResult[AnyStr]: ...
 @overload
 def urlunparse(data: Tuple[str, str, str, str, str, str]) -> str: ...
 @overload

[gvanrossum]

Not sure exactly what you are doing or what you are trying. Are you trying to fix something in the urlparse stubs? What exactly is the typeshed/2.7 directory? (I don't have one like that.)

[mr-c]

Hey @gvanrossum, thanks for looking at this.

Yes, I am trying to improve the urlparse stubs.

the typeshed/2.7 directory is from the schema_salad project: https://github.com/common-workflow-language/schema_salad/blob/master/typeshed/2.7
It is where our local type overrides live until they get mainstreamed into mypy.

I'm trying to model urlparse.urlparse() with a restricted generic type: if given a str it returns a ParseResult whose scheme, netloc, path, query, and fragment members are of type str. Likewise if I call urlparse.urlparser() with a unicode object the same ParseResult members are of type unicode.

I'm having difficulty combining a restricted generic type with a NamedTuple (which is how ParseResult is implemented).

Any assistance is appreciated :-)

[gvanrossum]

Oh, that's a tough puzzle. I wonder if this would be useful?

from typing import *

class B(NamedTuple('B', [('a', Any)])):
    pass

class C(NamedTuple('C', [('a', str)])):
    pass

class D(NamedTuple('C', [('a', bytes)])):
    pass

@overload
def foo(a: str) -> C:
    pass
@overload
def foo(a: bytes) -> D:
    pass

a = foo('')
reveal_type(a)
b = foo(b'')
reveal_type(b)

But I don't know if that satisfies all requirements.

[gnprice]

@mr-c By the way, regardless of what you may or may not be doing wrong in a given program, mypy should never crash! :-) I just went and filed this one on the mypy tracker, as python/mypy#1682 , so we can fix it -- if you ever run into another crash, we'd always love to see it reported there.

[mr-c]

@gvanrossum Thanks, that worked!

@gnprice Right-o, thank you for the reminder and for filing the follow up issue for me!