Failing to recognize generic constraints in tuples

[mikeknep]

• Are you reporting a bug, or opening a feature request? Bug

The basic context here is I need to map data into different shapes using various Mappings, and then write those shapes to different targets using various Writers. I want to ensure that when I define a job configuration, I get a Mapping and Writer that are compatible with one another (i.e. they map to and write the same shape).

Initially I tried returning a Tuple[Mapping[E], Writer[E]], but found that mypy does not report an error when I deliberately mismatch a Mapping and Writer. Fortunately, however, an error is raised when I define a (data)class to hold these two objects instead of just a tuple. I think using a dataclass here is probably a good idea regardless (for reader comprehension purposes), but I would nevertheless expect the tuple implementation to behave the same way. It seems like someone could try the tuple implementation, not see any errors, and incorrectly assume they have a typechecking safety net that isn't actually there.

from dataclasses import dataclass
from typing import Generic, Tuple, TypeVar
from typing_extensions import Protocol


M = TypeVar("M", covariant=True)
class Mapping(Protocol[M]):
    def map(self) -> M:
        ...


W = TypeVar("W", contravariant=True)
class Writer(Protocol[W]):
    def write(self, shape: W):
        ...


class Circle:
    pass


class CircleMapping:
    def map(self) -> Circle:
        return Circle()


class CircleWriter:
    def write(self, shape: Circle):
        print("in CircleWriter.write")


class Square:
    pass


class SquareMapping:
    def map(self) -> Square:
        return Square()


class SquareWriter:
    def write(self, shape: Square):
        print("in SquareWriter.write")


E = TypeVar("E")


## Tuple strategy
Tools = Tuple[Mapping[E], Writer[E]]

def get_tools(shape_name: str) -> Tools:
    if shape_name == "square":
        return (SquareMapping(), CircleWriter())   # mypy does not flag an error here!
    else:
        return (CircleMapping(), CircleWriter())

def run_1(tools: Tools):
    (mapping, writer) = tools
    shape = mapping.map()
    writer.write(shape)



## Dataclass strategy
@dataclass
class EtlTools(Generic[E]):
    mapping: Mapping[E]
    writer: Writer[E]

def get_etl_tools(shape_name: str) -> EtlTools:
    if shape_name == "square":
        return EtlTools(mapping=SquareMapping(), writer=CircleWriter()) # mypy error, cannot infer type argument 1 of EtlTools
    else:
        return EtlTools(mapping=CircleMapping(), writer=CircleWriter())

def run_2(etl_tools: EtlTools):
    shape = etl_tools.mapping.map()
    etl_tools.writer.write(shape)

• What is the actual behavior/output?
  Mypy only errors on the dataclass implementation, not the tuple implementation

• What is the behavior/output you expect?
  The tuple implementation should report an error as well

• What are the versions of mypy and Python you are using?
  mypy = "==0.720"
  python version 3.7.4

• What are the mypy flags you are using? (For example --strict-optional)
  None

[JelleZijlstra]

This is probably because you're missing a type argument to Tools, which is a generic alias. Try returning Tools[E].

The --disallow-any-generics option (which ideally should be on by default) would have told you this.

[mikeknep]

@JelleZijlstra Thanks for the flag tip. I'm trying your suggestion, but here with correct pairings:

def get_tools(shape_name: str) -> Tools[E]:
    if shape_name == "square":
        return (SquareMapping(), SquareWriter())
    else:
        return (CircleMapping(), CircleWriter())

Error message from mypy:

Incompatible return value type (got "Tuple[SquareMapping, SquareWriter]", expected "Tuple[Mapping[E], Writer[E]]")
Incompatible return value type (got "Tuple[CircleMapping, CircleWriter]", expected "Tuple[Mapping[E], Writer[E]]")

[JelleZijlstra]

Interesting, that seems like a bug to me.

[JukkaL]

Mypy treats a type variable in the return type (but not in an argument type) as meaning that the value of that variable is specified by the calling context. A typical example is set(), which returns Set[T] for arbitrary T. In the example get_tools doesn't return Tools[E] for an arbitrary E, and this is what mypy is complaining about. This is probably poorly documented, though.

[ilevkivskyi]

@JukkaL I think this is then a duplicate of #2885 (added the docs label there).