feat(index): append #1282

cmp0xff · 2025-07-18T14:20:43Z

Tests added: Please use assert_type() to assert the type of any return value

Index.append used not to be typed. In this PR, typings for Index.append are added and tested.

pandas-stubs/core/indexes/base.pyi

twoertwein · 2025-07-18T14:55:36Z

pandas-stubs/core/indexes/base.pyi

+    @overload
+    def append(self, other: Index[Never]) -> Index: ...
+    @overload
+    def append(self, other: Index[S1] | Sequence[Index[S1]]) -> Index[S1]: ...


it might be nicer here to have a second S1 here as the resulting Index can contain a mix of different types.

def append(self, other: Index[S2] | Sequence[Index[S2]]) -> Index[S1 | S2]: ...

45e0ad3, but this one works less well.

mypy is not happy with Index[int].append(Index[int | str]) and gives Index[Any]

pyright is not happy with Index[int | str].append([Index[int], Index[str]]) and gives Index[int | Any]. In particular, the typing for [Index[int], Index[str]] seems to be list[Index[int] | Index[str]], instead of list[Index[int | str]].

mypy is not happy with Index[int].append(Index[int | str]) and gives Index[Any]

pyright is not happy with Index[int | str].append([Index[int], Index[str]]) and gives Index[int | Any]. In particular, the typing for [Index[int], Index[str]] seems to be list[Index[int] | Index[str]], instead of list[Index[int | str]].

While that is annoying for testing on the CI, I think that is the safer choice for user: rather expect a wider type that includes Any than suggesting it is a narrower type. This needs input from @Dr-Irv.

If S1 and S2 were covariant, it seems to work for at least pyright in a simple toy example (but they are invariant)

from __future__ import annotations from typing import TypeVar, reveal_type, Generic, Sequence S1 = TypeVar("S1", bound=int | str, covariant=True) S2 = TypeVar("S2", bound=int | str, covariant=True) class Index(Generic[S1]): def __init__(self, data: list[S1]) -> None: ... def append(self: Index[S1], other: Sequence[Index[S2]]) -> Index[S1 | S2]: ... strings = Index(["a"]) reveal_type(strings) ints = Index([1]) reveal_type(ints) reveal_type(strings.append([ints])) reveal_type(ints.append([strings])) string_ints = Index(["a", 1]) reveal_type(string_ints) reveal_type(string_ints.append([ints])) reveal_type(strings.append([string_ints])) reveal_type(strings.append([ints])) reveal_type(strings.append([strings, ints])) reveal_type(strings.append([ints, strings, string_ints]))

If S1 and S2 were covariant, it seems to work for at least pyright in a simple toy example (but they are invariant)

from __future__ import annotations from typing import TypeVar, reveal_type, Generic, Sequence S1 = TypeVar("S1", bound=int | str, covariant=True) S2 = TypeVar("S2", bound=int | str, covariant=True) class Index(Generic[S1]): def __init__(self, data: list[S1]) -> None: ... def append(self: Index[S1], other: Sequence[Index[S2]]) -> Index[S1 | S2]: ...

Hi, I am new to covariance / contravariance, but I read PEP484 (covariance-and-contravariance) and it says covariant is for classes, not for functions, where the latter case is prohibited. In your example, S1 is find, but not S2. Could you help me and explain a bit? Thanks.

B_co = TypeVar('B_co', covariant=True) def bad_func(x: B_co) -> B_co: # Flagged as error by a type checker ...

We can't change S1 to be covariant. While the following is not exactly what we like to have, it is probably the closest we can get (but it doesn't work with mypy, unless the caller casts).

from __future__ import annotations from typing import TypeVar, reveal_type, Generic, Sequence S1 = TypeVar("S1", bound=int | str) IndexT = TypeVar("IndexT", bound="Index") class Index(Generic[S1]): def __init__(self, data: list[S1]) -> None: ... def append(self: Index[S1], other: Sequence[IndexT]) -> Index[S1] | IndexT: ... strings = Index(["a"]) reveal_type(strings) ints = Index([1]) reveal_type(ints) reveal_type(strings.append([ints])) reveal_type(ints.append([strings])) string_ints = Index(["a", 1]) reveal_type(string_ints) reveal_type(string_ints.append([ints])) reveal_type(strings.append([string_ints])) reveal_type(strings.append([ints])) reveal_type(strings.append([strings, ints])) reveal_type(strings.append([ints, strings, string_ints]))

Hi, I ran the script myself. In the most complicated case, I see Index[int | str] | Index[int] | Index[str]. To be honest, as a user I would rather see Index[Unknown], because it's simpler, and in both cases I would probably still need a manual cast. Nevertheless, 3844062

…les#r2216273709

Dr-Irv · 2025-07-24T20:29:13Z

pandas-stubs/core/indexes/base.pyi

+    @overload
+    def append(self, other: Index[S1] | Sequence[Index[S1]]) -> Self: ...
+    @overload
+    def append(self, other: Index[S2] | Sequence[Index[S2]]) -> Index[S1 | S2]: ...


I don't think that's right. I think the result would be Index[S1] | Index[S2] .

I'm not sure how the downstream Index stuff would work with the union type inside the generic.

When we append Index([1]) to Index(["a"]), we actually get Index(["a", 1]), which has the type Index[str | int].

If we can't narrow it to a specific type that is one of the types in S1, then I think it will need to be just Index without specifying the subtype, or Index[Any], or even UnknownIndex .

Right now, if you did Index(["a", 1]), the revealed type by pyright is Index[str | int], but with mypy it is Index[Any], so I think I want to avoid placing unions inside the generic type because I don't know what else will break and there could be inconsistencies in the type checkers in handling this.

67e6bde, except for that mypy is unable to handle pd.Index([1, "a"]).append(pd.Index([1, "a"])) and gives Index[Any].

Dr-Irv · 2025-07-24T20:29:39Z

pandas-stubs/core/indexes/base.pyi

+    @overload
+    def append(self, other: Index[S2] | Sequence[Index[S2]]) -> Index[S1 | S2]: ...
+    @overload
+    def append(self, other: Sequence[_T_INDEX]) -> Self | _T_INDEX: ...


Why can't we use Sequence[Index] here without the TypeVar ?

The idea came from #1282 (comment). Removed in 67e6bde.

Dr-Irv · 2025-07-24T20:31:39Z

tests/test_indexes.py

+    check(assert_type(first.append(third), "pd.Index[int | str]"), pd.Index)  # type: ignore[assert-type]
+    check(assert_type(first.append([third]), "pd.Index[int | str]"), pd.Index)  # type: ignore[assert-type]
+    check(
+        assert_type(  # type: ignore[assert-type]
+            first.append([second, third]),  # pyright: ignore[reportAssertTypeFailure]
+            "pd.Index[int | str]",
+        ),
+        pd.Index,
+    )
+
+    check(assert_type(third.append([]), "pd.Index[int | str]"), pd.Index)  # type: ignore[assert-type]
+    check(
+        assert_type(third.append(cast("list[Index[Any]]", [])), "pd.Index[int | str]"),  # type: ignore[assert-type]
+        pd.Index,
+    )
+    check(assert_type(third.append([first]), "pd.Index[int | str]"), pd.Index)  # type: ignore[assert-type]
+    check(
+        assert_type(  # type: ignore[assert-type]
+            third.append([first, second]),  # pyright: ignore[reportAssertTypeFailure]
+            "pd.Index[int | str]",
+        ),
+        pd.Index,


these tests need to work without having ignore in them. So you'll need to fix the types in the append() declarations to make that happen.

Three type: ignore remains, which I believe are mypy bugs. I tried to add

@overload def append(self, other: Sequence[Never]) -> Self: ...

as the first overload, which did not help.

Dr-Irv · 2025-07-24T20:46:26Z

tests/test_indexes.py

+    """Test pd.Index[list[str]].append"""
+    first = pd.Index([["str", "rts"]])
+    second = pd.Index([["srt", "trs"]])
+    check(assert_type(first.append([]), "pd.Index[list[str]]"), pd.Index, list)
+    check(assert_type(first.append(second), "pd.Index[list[str]]"), pd.Index, list)
+    check(assert_type(first.append([second]), "pd.Index[list[str]]"), pd.Index, list)


I don't think we should be supporting Index[list[str]] because list[str] is not hashable and labels in an Index should be hashable.

But this is a bug in pandas, I think. See pandas-dev/pandas#61937

So can you remove this test.

Then we will have to separate out list[str] from S1 and have an I1 that includes everything in S1 except list[str], while S1 includes everything.

So can you make that change as well?

Removing Index[list[str]] is hard. The following code runs:

import pandas as pd pd.Index(["a_b"]).str.split("_")

It produces Index([['a', 'b']], dtype='object'). We have Index[S1] all over pandas-stubs.

Yes, but it really isn't supported to have list[str] in an Index:

ind = pd.Index(["a_b", "c_d"]) spl = ind.str.split("_") spl.duplicated()

This will create an exception because the list is unhashable.

Might be best in this PR to just leave out the test with append and Index[list[str]] and not worry for now about splitting it out until the issue in pandas is straightened out.

Dr-Irv

Let's not return things like Index[str | int] and just make that Index

Dr-Irv · 2025-07-29T14:17:39Z

pandas-stubs/core/indexes/base.pyi

-    def append(self, other: Index[S2] | Sequence[Index[S2]]) -> Index[S1 | S2]: ...
-    @overload
-    def append(self, other: Sequence[_T_INDEX]) -> Self | _T_INDEX: ...
+    def append(self, other: Index[S2]) -> Index[S1 | S2]: ...


Suggested change

def append(self, other: Index[S2]) -> Index[S1 | S2]: ...

def append(self, other: Index[S2]) -> Index: ...

I really want to avoid having the union types here.

Dr-Irv · 2025-07-29T14:18:24Z

tests/test_indexes.py

    check(
        assert_type(  # type: ignore[assert-type]
-            first.append([second, third]),  # pyright: ignore[reportAssertTypeFailure]
-            "pd.Index[int | str]",
+            first.append(third), "pd.Index[int | str]"


Should be Index[Any] (or just pd.Index)

d7a538d. Now just two # type: ignore[assert-type]s remain.

Dr-Irv · 2025-07-29T17:38:55Z

tests/test_indexes.py

+            third.append([]), "pd.Index[int | str]"
+        ),
+        pd.Index,
+    )
+    check(
+        assert_type(  # type: ignore[assert-type]
+            third.append(cast("list[Index[Any]]", [])), "pd.Index[int | str]"


Change these to pd.Index in the assert_type statements, and then I don't think you need the # type: ignore statements., i.e., remove the union part of the generic paramter.

feat(index): append

390e5d9

twoertwein reviewed Jul 18, 2025

View reviewed changes

cmp0xff added 3 commits July 18, 2025 17:18

fix(comment): pandas-dev#1282 (comment)

3a34057

fix(comment): https://github.com/pandas-dev/pandas-stubs/pull/1282/fi…

45e0ad3

…les#r2216273709

fix(pyrefly): ignore

6c8ba76

cmp0xff requested a review from twoertwein July 18, 2025 16:43

fix(comment): pandas-dev#1282 (comment)

3844062

cmp0xff mentioned this pull request Jul 23, 2025

refactor: #718 only drop TimestampSeries #1274

Open

2 tasks

Dr-Irv requested changes Jul 24, 2025

View reviewed changes

cmp0xff added 3 commits July 29, 2025 12:46

fix(comment): pandas-dev#1282 (comment)

ac0857b

fix(comment): pandas-dev#1282 (comment)

67e6bde

fix: Sequence[Index] instead of Sequence

fa3c401

cmp0xff requested a review from Dr-Irv July 29, 2025 11:13

Dr-Irv requested changes Jul 29, 2025

View reviewed changes

fix(comment): avoid union types pandas-dev#1282 (comment)

d7a538d

cmp0xff requested a review from Dr-Irv July 29, 2025 15:36

Dr-Irv requested changes Jul 29, 2025

View reviewed changes

	def append(self, other: Index[S2]) -> Index[S1 \| S2]: ...
	def append(self, other: Index[S2]) -> Index: ...

Uh oh!

feat(index): append #1282

Are you sure you want to change the base?

feat(index): append #1282

Conversation

cmp0xff commented Jul 18, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Dr-Irv left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!