BUG: Fix TypeError in assert_index_equal when comparing CategoricalIndex with check_categorical=True and exact=False #61941

Aniketsy · 2025-07-24T22:40:10Z

Fixes #61935

Fixes a bug where assert_index_equal raises a TypeError instead of AssertionError when comparing two CategoricalIndex objects with check_categorical=True and exact=False.
Ensures consistency with expected testing behavior by properly raising an AssertionError in these cases.

Please let me know if my approach or fix needs any improvements . I’m open to feedback and happy to make changes based on suggestions.

Aniketsy · 2025-07-25T15:59:08Z

Hi @mroeschke
I've opened a pull request addressing
BUG: Fix TypeError in assert_index_equal when comparing CategoricalIndex with check_categorical=True and exact=False ([#61941])
The changes are ready for review.

I'd really appreciate it if you could take a look and provide feedback .
Please let me know if anything needs to be improved or clarified.

Thanks!

pandas/_testing/asserters.py

mroeschke

This also needs a unit test

Aniketsy · 2025-07-25T18:16:23Z

Hi @mroeschke,

Thank you for your review. I’ve updated the PR based on your feedback ,please have a look when convenient.

Additionally, I noticed one check failure (pre-commit.ci-pr) and wanted to ask if you could help clarify the reason behind it. Apologies if this isn't the appropriate way to raise this, please do let me know the correct approach if needed.

Thanks again!

Aniketsy · 2025-07-26T12:08:05Z

Hi @jorisvandenbossche, I ran pre-commit locally and all hooks passed. However, the GitHub checks are still showing a failure. Could you please advise if I’ve missed something?

Aniketsy · 2025-07-28T13:35:12Z

Hi @mroeschke
When you have a moment, could you please review this PR? I've been working on resolving the check failure, but haven't been able to pinpoint the issue yet. Any insights or suggestions you could provide would be greatly appreciated.

Thank you!

mroeschke · 2025-07-28T17:13:32Z

pandas/tests/util/test_assert_index_equal.py

+    ci1 = CategoricalIndex(["a", "b", "c"], categories=["a", "b", "c"], ordered=False)
+    ci2 = CategoricalIndex(["a", "x", "c"], categories=["a", "b", "c"], ordered=False)


Can you test when 1 index is CategoricalIndex and the other is just an Index?

Sorry I had meant an additional test in addition to the one you had

Please , let me know if , I need to add an additional test .

@Aniketsy yes, please add it as an additional test, and keep the original one you added (because we want to cover the case of the bug report #61935, where both left and right are CategoricalIndex)

@jorisvandenbossche Should I keep these as two separate tests for clarity, or merge them into one parametrized test?

mroeschke · 2025-07-28T18:36:05Z

pandas/_testing/asserters.py

+                if hasattr(left, "_internal_get_values") and hasattr(
+                    right, "_internal_get_values"


Can you use isinstance checks to call _internal_get_values if the object is a CategoricalIndex?

Thankyou ! for correcting me . I have updated that

There is one check fail due to:
CategoricalIndex does not have an _internal_get_values() method.
So should i use .value or something else . Please suggest

Aniketsy · 2025-08-14T19:33:58Z

Hi @mroeschke
I just wanted to check in on this PR to see if there’s anything further you’d like me to update or improve.
Thankyou !

Aniketsy · 2025-08-22T06:25:38Z

Hi @jorisvandenbossche, I’ve added the separate test as suggested. Please let me know if you’d prefer me to merge these into a single parametrized test instead.

Aniketsy · 2025-09-12T11:50:51Z

Hi @jorisvandenbossche , just a gentle reminder to review the changes whenever you get a chance. Thanks!

jbrockmendel · 2025-09-22T17:11:47Z

pandas/_testing/asserters.py

-            mismatch = left._values != right._values
+            try:
+                mismatch = left._values != right._values
+            except TypeError:


comment pointing back to the original issue, otherwise ill be confused as to why this can happen

Sure , I will add comment

jbrockmendel · 2025-09-22T17:12:01Z

pandas/_testing/asserters.py

+            try:
+                mismatch = left._values != right._values
+            except TypeError:
+                if isinstance(left, CategoricalIndex) and isinstance(


are there cases where this doesn't hold?

I’m not sure, but I don’t think so.

jbrockmendel · 2025-09-22T17:12:33Z

pandas/_testing/asserters.py

+                if isinstance(left, CategoricalIndex) and isinstance(
+                    right, CategoricalIndex
+                ):
+                    mismatch = left.codes != right.codes


if we raised TypeError it'll be because the dtypes dont match and arent comparable. in that case, comparing codes doesn't make much sense

got it that makes sense.

Aniketsy · 2025-09-23T16:39:19Z

@jbrockmendel just to confirm before proceeding to apply changes as per suggestion.

except TypeError:
                if isinstance(left, CategoricalIndex) and isinstance(
                    right, CategoricalIndex
                ):
                    mismatch = left.codes != right.codes
                else:
                    mismatch = left.values != right.values

Should I simply replace it with this .

except TypeError:
                mismatch = np.ones(len(left), dtype=bool)

Also do i need to update tests as i have added for two cases.
Please correct me if I'm wrong .

jbrockmendel · 2025-09-26T18:04:50Z

In that context I don't think it makes sense so show the mismatch. Just say that the types aren't comparable.

…dex with check_categorical=True and exact=False

…Index

…s AssertionError

Aniketsy · 2025-09-27T05:50:01Z

@jbrockmendel I’ve updated the changes. Please review them when you have time. Thanks!

…s AssertionError

Aniketsy · 2025-10-01T04:13:29Z

It looks like the errors in the CI are from another file and not caused by the changes in this PR. As far as I can tell, this shouldn’t be related, but happy to be corrected if I’m wrong.

jbrockmendel · 2025-10-01T16:42:31Z

the code checks failures should be fixed by merging main

…oricalindex-typeerror

Aniketsy · 2025-10-02T08:49:16Z

merged main, but still its showing check failures .

jbrockmendel · 2025-10-02T16:02:11Z

Looks like those are caused by #62545

Aniketsy · 2025-10-03T17:01:24Z

@jbrockmendel The CI check failure is resolved.

mroeschke · 2025-10-03T17:17:43Z

Thanks @Aniketsy

mroeschke requested changes Jul 25, 2025

View reviewed changes

pandas/_testing/asserters.py Outdated Show resolved Hide resolved

mroeschke added the Testing pandas testing functions or related to the test suite label Jul 25, 2025

mroeschke requested changes Jul 25, 2025

View reviewed changes

Aniketsy force-pushed the bugfix-assert-categoricalindex-typeerror branch from 333eb89 to 5edf8ce Compare July 25, 2025 18:06

Aniketsy force-pushed the bugfix-assert-categoricalindex-typeerror branch from de5b16b to 08739f2 Compare July 26, 2025 10:38

mroeschke reviewed Jul 28, 2025

View reviewed changes

jbrockmendel reviewed Sep 22, 2025

View reviewed changes

Aniketsy added 11 commits September 27, 2025 10:59

BUG: Fix TypeError in assert_index_equal when comparing CategoricalIn…

6871bd5

…dex with check_categorical=True and exact=False

STYLE: Fix E501 line too long in assert_index_equal error message

d637203

TST: Add test case for categorical index assertion fix

7a4aae6

BUG: Fix (GH#61941) and unit test

1a7c48c

Add unit test and (GH#61941)

d600c06

Add unit test and (GH#61941)

eaeb094

Add unit test and (GH#61941)

7037a72

TST: Fix formatting in test_assert_index_equal (via pre-commit)

80aaf2c

style: fix formatting with pre-commit (Ruff)

1a1ee91

Fix: Check fail

5419cf0

TST: Add test for CategoricalIndex vs Index in assert_index_equal

db8006d

Aniketsy added 5 commits September 27, 2025 11:00

TST: Use isinstance for CategoricalIndex in assert_index_equal

2ba4cfb

Use the public .codes property for internal comparison of Categorical…

b82f867

…Index

TST: Add tests for CategoricalIndex mismatch scenarios

9012972

BUG: assert_index_equal CategoricalIndex incomparable categories give…

b99e6d2

…s AssertionError

BUG: assert_index_equal CategoricalIndex incomparable categories give…

5d4af24

…s AssertionError

Aniketsy force-pushed the bugfix-assert-categoricalindex-typeerror branch from 0ef7755 to 5d4af24 Compare September 27, 2025 05:36

BUG: assert_index_equal CategoricalIndex incomparable categories give…

bc10d92

…s AssertionError

Merge remote-tracking branch 'upstream/main' into bugfix-assert-categ…

ac62aa5

…oricalindex-typeerror

Merge branch 'main' into bugfix-assert-categoricalindex-typeerror

6ef5a73

mroeschke added this to the 3.0 milestone Oct 3, 2025

mroeschke approved these changes Oct 3, 2025

View reviewed changes

mroeschke merged commit 1028791 into pandas-dev:main Oct 3, 2025
42 checks passed

		ci1 = CategoricalIndex(["a", "b", "c"], categories=["a", "b", "c"], ordered=False)
		ci2 = CategoricalIndex(["a", "x", "c"], categories=["a", "b", "c"], ordered=False)

		if hasattr(left, "_internal_get_values") and hasattr(
		right, "_internal_get_values"

Uh oh!

BUG: Fix TypeError in assert_index_equal when comparing CategoricalIndex with check_categorical=True and exact=False #61941

BUG: Fix TypeError in assert_index_equal when comparing CategoricalIndex with check_categorical=True and exact=False #61941

Uh oh!

Conversation

Aniketsy commented Jul 24, 2025 • edited by jorisvandenbossche Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Aniketsy commented Jul 25, 2025

Uh oh!

Uh oh!

mroeschke left a comment

Choose a reason for hiding this comment

Uh oh!

Aniketsy commented Jul 25, 2025

Uh oh!

Aniketsy commented Jul 26, 2025

Uh oh!

Aniketsy commented Jul 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aniketsy commented Aug 14, 2025

Uh oh!

Aniketsy commented Aug 22, 2025

Uh oh!

Aniketsy commented Sep 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aniketsy commented Sep 23, 2025

Uh oh!

jbrockmendel commented Sep 26, 2025

Uh oh!

Aniketsy commented Sep 27, 2025

Uh oh!

Aniketsy commented Oct 1, 2025

Uh oh!

jbrockmendel commented Oct 1, 2025

Uh oh!

Aniketsy commented Oct 2, 2025

Uh oh!

jbrockmendel commented Oct 2, 2025

Uh oh!

Aniketsy commented Oct 3, 2025

Uh oh!

Uh oh!

mroeschke commented Oct 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

Aniketsy commented Jul 24, 2025 •

edited by jorisvandenbossche

Loading