Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: groupby any/all raising with pd.NA object data #42085

Merged
merged 4 commits into from
Jun 21, 2021
Merged

BUG: groupby any/all raising with pd.NA object data #42085

merged 4 commits into from
Jun 21, 2021

Conversation

mzeitlin11
Copy link
Member

@mzeitlin11 mzeitlin11 added Bug Groupby Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate labels Jun 17, 2021
jreback
jreback requested changes Jun 18, 2021
View reviewed changes
Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @rhshadrach if you can look

doc/source/whatsnew/v1.4.0.rst Outdated
@@ -187,7 +187,7 @@ Plotting

Groupby/resample/rolling
^^^^^^^^^^^^^^^^^^^^^^^^
-
- :meth:`.GroupBy.any` and :meth:`.GroupBy.all` raising with ``object`` data containing ``pd.NA`` even when ``skipna=True`` (:issue:`37501`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok to move to 1.3

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have moved

@jreback jreback added this to the 1.3 milestone Jun 18, 2021
rhshadrach
rhshadrach requested changes Jun 18, 2021
View reviewed changes
Copy link
Member

@rhshadrach rhshadrach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good - minor request below.

pandas/core/groupby/groupby.py Outdated
# GH#37501: don't raise on pd.NA when skipna=True
vals = np.array(
[bool(x) if not (skipna and isna(x)) else True for x in vals]
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While slightly more verbose, it seems better to me to avoid evaluating skipna at each iteration:

if skipna:
    vals = np.array(
        [bool(x) if not isna(x) else True for x in vals]
    )
else:
    vals = np.array([bool(x) for x in vals])

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, have changed

@jreback jreback merged commit 81b26c4 into pandas-dev:master Jun 21, 2021
@jreback
Copy link
Contributor

jreback commented Jun 21, 2021

thanks @mzeitlin11

@jreback
Copy link
Contributor

jreback commented Jun 21, 2021

@meeseeksdev backport 1.3.x

meeseeksmachine pushed a commit to meeseeksmachine/pandas that referenced this pull request Jun 21, 2021
@mzeitlin11 @meeseeksmachine
Backport PR pandas-dev#42085: BUG: groupby any/all raising with pd.NA…
42daec8 
… object data
@meeseeksmachine meeseeksmachine mentioned this pull request Jun 21, 2021
Merged
@lumberbot-app
Copy link

lumberbot-app bot commented Jun 21, 2021

Something went wrong ... Please have a look at my logs.

@mzeitlin11 mzeitlin11 deleted the any_all_object branch June 21, 2021 17:27
simonjayhawkins pushed a commit that referenced this pull request Jun 21, 2021
@meeseeksmachine @mzeitlin11
Backport PR #42085: BUG: groupby any/all raising with pd.NA object da…
fa3ba91 
…ta (#42164)

Co-authored-by: Matthew Zeitlin <37011898+mzeitlin11@users.noreply.github.com>
neinkeinkaffee pushed a commit to neinkeinkaffee/pandas that referenced this pull request Jun 21, 2021
@mzeitlin11 @neinkeinkaffee
BUG: groupby any/all raising with pd.NA object data (pandas-dev#42085)
0f57e84
@mzeitlin11 mzeitlin11 mentioned this pull request Jun 21, 2021
Closed
JulianWgs pushed a commit to JulianWgs/pandas that referenced this pull request Jul 3, 2021
@mzeitlin11 @JulianWgs
BUG: groupby any/all raising with pd.NA object data (pandas-dev#42085)
df95975
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rhshadrach rhshadrach rhshadrach requested changes

@jreback jreback jreback approved these changes

Assignees
No one assigned
Labels
Bug Groupby Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
Projects
None yet
Milestone
1.3
Development

Successfully merging this pull request may close these issues.

BUG: Groupby any/all raises TypeError for pd.NA
3 participants
@mzeitlin11 @jreback @rhshadrach