Infer generic types for default implementations

[TV4Fun]

It would be useful in some cases to have an abstract class where a method's return type is overridable, but a default implementation is still provided for the most common case. Consider an example like this:

from abc import ABC, abstractmethod
from typing import Generic, TypeVar

_InputType = TypeVar('_InputType', contravariant=True)
_IntermediateType = TypeVar('_IntermediateType')
_OutputType = TypeVar('_OutputType', covariant=True)


class GenericBase(Generic[_InputType, _IntermediateType, _OutputType], ABC):
    @abstractmethod
    def first_step(self, pipeline_input: _InputType) -> _IntermediateType: ...

    def second_step(self, state: _IntermediateType) -> _OutputType:
        # By default, pass through state unmodified
        return state

    def execute(self, pipeline_input: _InputType) -> _OutputType:
        state = self.first_step(pipeline_input)
        return self.second_step(state)


class GenericImpl(GenericBase[str, int, float]):
    def first_step(self, pipeline_input: str) -> int:
        return len(pipeline_input)

    def second_step(self, state: int) -> float:
        return state * 1.5


a = GenericImpl()
print(a.execute("aaa"))

In this case, I would like GenericBase to provide a default implementation of second_step for the most common cases, but allow implementations to change its return type if they would like. Mypy does not allow for this though, and produces an error:

generic_abstract.py:16: error: Incompatible return value type (got "_IntermediateType", expected "_OutputType")

Without explicit casting, there doesn't seem to be a way to provide a default implementation for second_step that avoids this error. One possible workaround is to define an interface class and make GenericBase a subclass of that:

from abc import ABC, abstractmethod
from typing import Generic, TypeVar

_InputType = TypeVar('_InputType', contravariant=True)
_IntermediateType = TypeVar('_IntermediateType')
_OutputType = TypeVar('_OutputType', covariant=True)


class GenericInterface(Generic[_InputType, _IntermediateType, _OutputType], ABC):
    @abstractmethod
    def first_step(self, pipeline_input: _InputType) -> _IntermediateType: ...

    @abstractmethod
    def second_step(self, state: _IntermediateType) -> _OutputType: ...

    def execute(self, pipeline_input: _InputType) -> _OutputType:
        state = self.first_step(pipeline_input)
        return self.second_step(state)


class GenericBase(GenericInterface[_InputType, _IntermediateType, _IntermediateType]):
    def second_step(self, state: _IntermediateType) -> _IntermediateType:
        # By default, pass through state unmodified
        return state


class GenericImpl(GenericInterface[str, int, float]):
    def first_step(self, pipeline_input: str) -> int:
        return len(pipeline_input)

    def second_step(self, state: int) -> float:
        return state * 1.5


class OtherGenericImpl(GenericBase[str, int]):
    def first_step(self, pipeline_input: str) -> int:
        return int(pipeline_input)


a = GenericImpl()
print(a.execute("aaa"))
b = OtherGenericImpl()
print(b.execute("123"))

This works, but seems a little un-pythonic. It complicates the class hierarchy and requires us to define an extra class without any real purpose. A better approach would be if, when reading the first definition of GenericBase, Mypy inferred _IntermediateType and _OutputType to be the same type. This would mean that an implementation would be valid iff either _IntermediateType and _OutputType were compatible types, or if it overrode second_step to match the type arguments given to GenericBase. To make this work, type arguments should be evaluated in the implementation, not the base class definition. So the examples I gave above would work, but something like this:

class WrongGenericImpl(GenericBase[str, int, str]):
    def first_step(self, pipeline_input: str) -> int:
        return len(pipeline_input)

would produce an error. There are probably complications I haven't considered in implementing this, but I think the basic idea is worth considering. Otherwise, does anyone have a suggestion for how else I might implement the design pattern in my first example?

[emmatyping]

but allow implementations to change its return type if they would like

I think this is the source of your issues. Changing the return type would violate the Liskov substitution principle. I would say you probably want to put some bound on the return type or otherwise weaken the typing a small amount by making the default Any.

Regardless I don't think there is anything to be done about this issue in mypy, so closing.