Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

REGR: Performance decrease in factorize #48620

Merged
merged 8 commits into from
Sep 22, 2022
Merged

REGR: Performance decrease in factorize #48620

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
b072b82
PERF: Only check isna in certain cases in factorize
rhshadrach Sep 17, 2022
65f4c4b
Remove type-ignore
rhshadrach Sep 17, 2022
3a60993
Reformat
rhshadrach Sep 17, 2022
a9962b7
Merge branch 'main' of https://github.com/pandas-dev/pandas into grou…
rhshadrach Sep 18, 2022
135b1d3
Check for object dtype
rhshadrach Sep 18, 2022
79f399d
Merge branch 'main' of https://github.com/pandas-dev/pandas into grou…
rhshadrach Sep 20, 2022
58c6377
whatsnew note
rhshadrach Sep 20, 2022
131d36e
Merge branch 'main' into groupby_na_regr_v2
rhshadrach Sep 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.5.1.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ including other versions of pandas.
Fixed regressions
~~~~~~~~~~~~~~~~~
- Regression in :func:`.read_csv` causing an ``EmptyDataError`` when using an UTF-8 file handle that was already read from (:issue:`48646`)
- Fixed performance regression in :func:`factorize` when ``na_sentinel`` is not ``None`` and ``sort=False`` (:issue:`48620`)
-

.. ---------------------------------------------------------------------------
Expand Down
23 changes: 12 additions & 11 deletions pandas/core/algorithms.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -569,17 +569,6 @@ def factorize_array(

hash_klass, values = _get_hashtable_algo(values)

# factorize can now handle differentiating various types of null values.
# However, for backwards compatibility we only use the null for the
# provided dtype. This may be revisited in the future, see GH#48476.
null_mask = isna(values)
if null_mask.any():
na_value = na_value_for_dtype(values.dtype, compat=False)
# Don't modify (potentially user-provided) array
# error: No overload variant of "where" matches argument types "Any", "object",
# "ndarray[Any, Any]"
values = np.where(null_mask, na_value, values) # type: ignore[call-overload]

table = hash_klass(size_hint or len(values))
uniques, codes = table.factorize(
values,
Expand Down Expand Up @@ -813,6 +802,18 @@ def factorize(
na_sentinel_arg = None
else:
na_sentinel_arg = na_sentinel

if not dropna and not sort and is_object_dtype(values):
# factorize can now handle differentiating various types of null values.
# These can only occur when the array has object dtype.
# However, for backwards compatibility we only use the null for the
# provided dtype. This may be revisited in the future, see GH#48476.
null_mask = isna(values)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to check for integer dtype? Not sure if this offers any performance gains

Copy link
Member Author

@rhshadrach rhshadrach Sep 18, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to determine which dtypes can hold nan? I didn't see one in core.dtypes, but might have missed it.

For integer dtypes, this op takes 13.1 µs in the ASVs I commented on in the linked PR - compared to 2.3ms for object dtype. There could be some gains here if we can eliminate certain dtypes.

Copy link
Member

@phofl phofl Sep 18, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, this is probably checked under the hood then. I don't think that this offers much gains then.

The more interesting question is probably, if there is a dtype that can hold different nans apart from object. I don't think so. So we could only check for object dtype?

Edit: I think what you are looking or is something like np_can_hold_na, but you would have to check for every na value, so probably not of much use here

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed - we can check only object dtype here. ASVs for this are currently running, I plan update this PR later this evening (GMT-4).

if null_mask.any():
na_value = na_value_for_dtype(values.dtype, compat=False)
# Don't modify (potentially user-provided) array
values = np.where(null_mask, na_value, values)

codes, uniques = factorize_array(
values,
na_sentinel=na_sentinel_arg,
Expand Down