Key/value types with isinstance() and collections

[Terrance]

I'm annotating a function that recursively processes items in a mapping/sequence structure, but I'm warned that the key and value types aren't known. With generics, and without the actual processing logic, I have the following:

from typing import Any, AnyStr, Dict, List, Mapping, overload, Sequence, TypeVar

TAny = TypeVar("TAny")
TKey = TypeVar("TKey")
TValue = TypeVar("TValue")

@overload
def unwrap(obj: AnyStr) -> AnyStr: ...
@overload
def unwrap(obj: Mapping[TKey, TValue]) -> Dict[TKey, TValue]: ...
@overload
def unwrap(obj: Sequence[TValue]) -> List[TValue]: ...
@overload
def unwrap(obj: TAny) -> TAny: ...

def unwrap(obj: Any) -> Any:
    if isinstance(obj, Mapping):
        # (parameter) obj: Mapping[Unknown, Unknown]
        return {key: unwrap(value) for key, value in obj.items()}  # Type of "key" is unknown
                                                                   # Type of "value" is unknown
    elif isinstance(obj, Sequence) and not isinstance(obj, (str, bytes)):
        # (parameter) obj: Sequence[Unknown]
        return [unwrap(item) for item in obj]  # Type of "item" is unknown
    else:
        return obj

Ideally the isinstance() checks would be sufficient to match the types from the corresponding overloads, but I'm guessing the presence of Any makes it ambiguous? What's the best way to match up the key/value types here?

This could probably be fixed with the new TypeGuard, but that feels a bit unnecessary here.

[erictraut]

The problem here is that the type checker does not know the type of the type arguments for Mapping and Sequence based on the isinstance call, so it's assuming Mapping[Unknown, Unknown] and Sequence[Unknown]. When you access the items stored within the mapping or sequence, the resulting types are therefore Unknown.

Presumably you have the reportUnknownVariableType diagnostic rule enabled (or are using strict type checking mode), so the type checker is emitting a diagnostic when you use these variables. In this particular case, there's really no harm in these variables having an unknown or Any type.

It would be convenient if we could use a subscripted type in an isinstance call:

if isinstance(obj, Mapping[Any, Any]):

but this isn't legal at runtime.

This is a rare case where I would probably resort to using a cast operator. It's a bit ugly, but it gets the job done.

def unwrap(obj: Any) -> Any:
    if isinstance(obj, Mapping):
        return {
            key: unwrap(value) for key, value in cast(Mapping[Any, Any], obj).items()
        }
    elif isinstance(obj, Sequence) and not isinstance(obj, (str, bytes)):
        return [unwrap(item) for item in cast(Sequence[Any], obj)]
    else:
        return obj

[Terrance]

Indeed, I'm using strict type checking -- aiming for full type coverage in a few of my projects. I'm not too fond of cast() either, but I suspect that's the easiest way to go. 👍

---

I've also noticed that AnyStr warns of overlapping with Sequence (though the resulting external type hints are accurate):

@overload
def unwrap(obj: AnyStr) -> AnyStr: ...  # Overload 1 for "unwrap" overlaps overload 2 and returns an incompatible type
@overload
def unwrap(obj: Sequence[TValue]) -> List[TValue]: ...

unwrap("str")  # (type: str)
unwrap((1, 2))  # (type: List[int])

If I'm reading python/typing#253 correctly, the overlap isn't allowed here because of the differing return types -- setting the sequence overload to (obj: Sequence[TValue]) -> Sequence[TValue] rather than -> List[TValue] makes it valid.

Whilst it'd be nice to expose the narrower return type, it seems I can't do that without a non-str sequence type to avoid the overlap (MutableSequence is close but not quite right here).