feat(index): append

[cmp0xff]

• Tests added: Please use assert_type() to assert the type of any return value

Index.append used not to be typed. In this PR, typings for Index.append are added and tested.

[twoertwein]

it might be nicer here to have a second S1 here as the resulting Index can contain a mix of different types.

def append(self, other: Index[S2] | Sequence[Index[S2]]) -> Index[S1 | S2]: ...

[cmp0xff]

45e0ad3, but this one works less well.

• mypy is not happy with Index[int].append(Index[int | str]) and gives Index[Any]
• pyright is not happy with Index[int | str].append([Index[int], Index[str]]) and gives Index[int | Any]. In particular, the typing for [Index[int], Index[str]] seems to be list[Index[int] | Index[str]], instead of list[Index[int | str]].

[twoertwein]

• mypy is not happy with Index[int].append(Index[int | str]) and gives Index[Any]
• pyright is not happy with Index[int | str].append([Index[int], Index[str]]) and gives Index[int | Any]. In particular, the typing for [Index[int], Index[str]] seems to be list[Index[int] | Index[str]], instead of list[Index[int | str]].

While that is annoying for testing on the CI, I think that is the safer choice for user: rather expect a wider type that includes Any than suggesting it is a narrower type. This needs input from @Dr-Irv.

If S1 and S2 were covariant, it seems to work for at least pyright in a simple toy example (but they are invariant)

from __future__ import annotations
from typing import TypeVar, reveal_type, Generic, Sequence

S1 = TypeVar("S1", bound=int | str, covariant=True)
S2 = TypeVar("S2", bound=int | str, covariant=True)

class Index(Generic[S1]):
    def __init__(self, data: list[S1]) -> None: ...

    def append(self: Index[S1], other: Sequence[Index[S2]]) -> Index[S1 | S2]: ...

strings = Index(["a"])
reveal_type(strings)
ints = Index([1])
reveal_type(ints)

reveal_type(strings.append([ints]))
reveal_type(ints.append([strings]))

string_ints = Index(["a", 1])
reveal_type(string_ints)
reveal_type(string_ints.append([ints]))
reveal_type(strings.append([string_ints]))

reveal_type(strings.append([ints]))
reveal_type(strings.append([strings, ints]))

reveal_type(strings.append([ints, strings, string_ints]))

[cmp0xff]

If S1 and S2 were covariant, it seems to work for at least pyright in a simple toy example (but they are invariant)

from __future__ import annotations
from typing import TypeVar, reveal_type, Generic, Sequence

S1 = TypeVar("S1", bound=int | str, covariant=True)
S2 = TypeVar("S2", bound=int | str, covariant=True)

class Index(Generic[S1]):
    def __init__(self, data: list[S1]) -> None: ...

    def append(self: Index[S1], other: Sequence[Index[S2]]) -> Index[S1 | S2]: ...

Hi, I am new to covariance / contravariance, but I read PEP484 (covariance-and-contravariance) and it says covariant is for classes, not for functions, where the latter case is prohibited. In your example, S1 is find, but not S2. Could you help me and explain a bit? Thanks.

B_co = TypeVar('B_co', covariant=True)

def bad_func(x: B_co) -> B_co:  # Flagged as error by a type checker
    ...

[twoertwein]

We can't change S1 to be covariant. While the following is not exactly what we like to have, it is probably the closest we can get (but it doesn't work with mypy, unless the caller casts).

from __future__ import annotations
from typing import TypeVar, reveal_type, Generic, Sequence

S1 = TypeVar("S1", bound=int | str)
IndexT = TypeVar("IndexT", bound="Index")


class Index(Generic[S1]):
    def __init__(self, data: list[S1]) -> None: ...

    def append(self: Index[S1], other: Sequence[IndexT]) -> Index[S1] | IndexT: ...


strings = Index(["a"])
reveal_type(strings)
ints = Index([1])
reveal_type(ints)

reveal_type(strings.append([ints]))
reveal_type(ints.append([strings]))

string_ints = Index(["a", 1])
reveal_type(string_ints)
reveal_type(string_ints.append([ints]))
reveal_type(strings.append([string_ints]))

reveal_type(strings.append([ints]))
reveal_type(strings.append([strings, ints]))

reveal_type(strings.append([ints, strings, string_ints]))

[cmp0xff]

Hi, I ran the script myself. In the most complicated case, I see Index[int | str] | Index[int] | Index[str]. To be honest, as a user I would rather see Index[Unknown], because it's simpler, and in both cases I would probably still need a manual cast. Nevertheless, 3844062

[Dr-Irv]

I don't think that's right. I think the result would be Index[S1] | Index[S2] .

I'm not sure how the downstream Index stuff would work with the union type inside the generic.

[Dr-Irv]

Why can't we use Sequence[Index] here without the TypeVar ?

[Dr-Irv]

these tests need to work without having ignore in them. So you'll need to fix the types in the append() declarations to make that happen.

[Dr-Irv]

I don't think we should be supporting Index[list[str]] because list[str] is not hashable and labels in an Index should be hashable.

But this is a bug in pandas, I think. See pandas-dev/pandas#61937

So can you remove this test.

Then we will have to separate out list[str] from S1 and have an I1 that includes everything in S1 except list[str], while S1 includes everything.

So can you make that change as well?