REF: Compute complete result_index upfront in groupby

[rhshadrach]

• closes BUG: Group keys contain NA values despite dropping them in groupby method #55919
• closes BUG: DataFrameGroupBy.nunique fails with categorical, multiple groupings, and as_index=False #52848
• closes Categorical in GroupBy with aggregations raise error under specific conditions #36698
• closes BUG: Groupby with categorical multiIndex and timedelta returns incorrect type. #43891
• closes DOC: Grouper.ngroups vs Grouper.group_info[2] #49980
• closes BUG: groupby().groups with multiple group keys always treats sort as True #56966
• closes BUG: groupby.groups and groupby.indices give incorrect sort order when dropna=False #56851
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

Just a PoC at this point, need to try to move behavior changes out of here as much as possible.

The main change this makes is it moves the computation of unobserved groups upfront. Currently, we only include unobserved groups upfront if there is a single grouping (e.g. df.groupby('a')) but not two or more (e.g. df.groupby(['a', 'b'])). When there are two or more, we go through the groupby computations with only observed groups, and then tack on the unobserved groups at the end. By always including unobserved groups upfront, we can simplify the logic in the groupby code. Having unobserved groups sometimes included and sometimes not included upfront is also a footgun.

In order to make this change, I found I needed to rework a bit of how NA values are handled in Grouping._codes_and_uniques. This in turn fixed some NA bugs.

This PR fixes a number of issues that stem from this (shown in the test changes)

• Unobserved groups are not included in groupby.apply if there is more than one grouping
• Unobserved groups do not appear in groubpy.groups
• When there are multiple groupings, len(df.groupby(..., dropna=True)) counts groups that have NA values
• When there are multiple groupings, df.groupby(..., dropna=True).groups has groups that have NA values
• The value of .all for empty groups was np.nan but should be True
• The value of .any for empty groups was np.nan but should be False
• The value of .prod for empty groups was np.nan but should be 1

In addition, it allows us to remove various similar objects throughout groupby:

• BaseGrouper.group_keys_seq
• BaseGrouper.reconstructed_codes
• Grouping.group_index
• Grouping.result_index
• Grouping.group_arraylike
• BinGrouper.reconstructed_codes

and a few methods

• GroupBy._reindex_output
• BaseGrouper._sort_idx
• BaseGrouper._get_compressed_codes

But it does add BaseGrouper.result_index_and_codes. This computes the (aggregated) result index that takes into account dropna and observed, along with the codes for the groups themselves.

ASVs; no performance regressions (with the standard 10% cutoff) other than groupby transform operations with multiple categorical groupings.

| Change   | Before [2b67593b] <test>   | After [e285742a] <gb_observed_pre>   |   Ratio | Benchmark (Parameter)                                                                                              |
|----------|----------------------------|--------------------------------------|---------|--------------------------------------------------------------------------------------------------------------------|
| +        | 308±4μs                    | 1.25±0ms                             |    4.08 | groupby.MultipleCategories.time_groupby_transform                                                                  |
| -        | 5.10±0.04ms                | 4.61±0.06ms                          |    0.91 | groupby.Apply.time_scalar_function_multi_col(5)                                                                    |
| -        | 239±4μs                    | 216±0.2μs                            |    0.9  | groupby.Categories.time_groupby_sort(False)                                                                        |
| -        | 60.1±2μs                   | 53.8±0.4μs                           |    0.9  | groupby.GroupByMethods.time_dtype_as_field('float', 'size', 'direct', 1, 'cython')                                 |
| -        | 241±5μs                    | 215±1μs                              |    0.89 | groupby.Categories.time_groupby_ordered_sort(False)                                                                |
| -        | 556±7μs                    | 488±60μs                             |    0.88 | arithmetic.IntFrameWithScalar.time_frame_op_with_scalar(<class 'numpy.int64'>, 5.0, <built-in function mul>)       |
| -        | 235±5μs                    | 205±2μs                              |    0.87 | groupby.Categories.time_groupby_extra_cat_sort(False)                                                              |
| -        | 419±8μs                    | 356±10μs                             |    0.85 | arithmetic.IntFrameWithScalar.time_frame_op_with_scalar(<class 'numpy.float64'>, 2, <built-in function le>)        |
| -        | 549±10μs                   | 461±50μs                             |    0.84 | arithmetic.IntFrameWithScalar.time_frame_op_with_scalar(<class 'numpy.float64'>, 3.0, <built-in function truediv>) |
| -        | 21.7±0.7μs                 | 18.1±0.06μs                          |    0.83 | groupby.GroupByMethods.time_dtype_as_field('uint', 'count', 'direct', 1, 'cython')                                 |
| -        | 22.5±0.3μs                 | 18.6±0.2μs                           |    0.83 | groupby.GroupByMethods.time_dtype_as_group('float', 'count', 'direct', 1, 'cython')                                |
| -        | 22.2±0.2μs                 | 18.4±0.8μs                           |    0.83 | groupby.GroupByMethods.time_dtype_as_group('uint', 'count', 'direct', 1, 'cython')                                 |
| -        | 22.3±0.8μs                 | 18.3±0.1μs                           |    0.82 | groupby.GroupByMethods.time_dtype_as_field('float', 'count', 'direct', 1, 'cython')                                |
| -        | 21.6±0.9μs                 | 17.8±0.3μs                           |    0.82 | groupby.GroupByMethods.time_dtype_as_field('int16', 'count', 'direct', 1, 'cython')                                |
| -        | 22.0±0.6μs                 | 18.1±0.3μs                           |    0.82 | groupby.GroupByMethods.time_dtype_as_group('int', 'count', 'direct', 1, 'cython')                                  |
| -        | 21.9±0.7μs                 | 17.8±0.3μs                           |    0.81 | groupby.GroupByMethods.time_dtype_as_field('int', 'count', 'direct', 1, 'cython')                                  |
| -        | 22.0±0.2μs                 | 17.9±0.07μs                          |    0.81 | groupby.GroupByMethods.time_dtype_as_group('int16', 'count', 'direct', 1, 'cython')                                |
| -        | 21.6±0.1μs                 | 17.0±0.06μs                          |    0.79 | groupby.GroupByMethods.time_dtype_as_group('object', 'count', 'direct', 1, 'cython')                               |
| -        | 520±50μs                   | 370±30μs                             |    0.71 | arithmetic.IntFrameWithScalar.time_frame_op_with_scalar(<class 'numpy.int64'>, 2, <built-in function sub>)         |
| -        | 13.7±1ms                   | 1.80±0ms                             |    0.13 | groupby.MultipleCategories.time_groupby_nosort                                                                     |
| -        | 15.8±2ms                   | 1.73±0ms                             |    0.11 | groupby.MultipleCategories.time_groupby_extra_cat_nosort                                                           |
| -        | 13.7±0.2ms                 | 1.57±0.01ms                          |    0.11 | groupby.MultipleCategories.time_groupby_ordered_nosort                                                             |
| -        | 33.6±0.4ms                 | 1.40±0ms                             |    0.04 | groupby.MultipleCategories.time_groupby_sort                                                                       |
| -        | 35.5±1ms                   | 1.23±0ms                             |    0.03 | groupby.MultipleCategories.time_groupby_extra_cat_sort                                                             |
| -        | 33.3±0.8ms                 | 1.15±0ms                             |    0.03 | groupby.MultipleCategories.time_groupby_ordered_sort                                                               |

I haven't verified this, but I believe this would also make #55261 trivial to implement just by changing a few lines of result_index_and_ids

[rhshadrach]

Ref: #56016 (comment)

[rhshadrach]

@jbrockmendel @mroeschke - this is now ready

[mroeschke]

Does this happen to fix #35202

[rhshadrach]

Does this happen to fix #35202

No, see the bottom half of #55738 (comment).

[mroeschke]

LGTM

[mroeschke]

Thanks @rhshadrach

[jorisvandenbossche]

@rhshadrach is the following change in behaviour intentional?

df = pd.DataFrame(
    {
        "key_cat": pd.Categorical(["a", "a", "b", "b"]),
        "key_noncat": [1, 1, 1, 2],
        "values": [1.0, 2.0, 3.0, 4.0],
    }
)
df.groupby(["key_cat", "key_noncat"], observed=False)["values"].agg(lambda x: x.sum())

Using released pandas, this injected NaNs for the unobserved values:

# using pandas 2.2.0
>>> df.groupby(["key_cat", "key_noncat"], observed=False)["values"].agg(lambda x: x.sum())
key_cat  key_noncat
a        1             3.0
         2             NaN
b        1             3.0
         2             4.0
Name: values, dtype: float64

versus after (I assume) this PR on pandas main:

# using pandas main
>>> df.groupby(["key_cat", "key_noncat"], observed=False)["values"].agg(lambda x: x.sum())
key_cat  key_noncat
a        1             3.0
         2             0.0
b        1             3.0
         2             4.0
Name: values, dtype: float64

The reason this now gives a 0 is because this actually calls the UDF on the empty group, and the sum of an empty Series is 0 (while before, the UDF was never called for the unobserved group).

I don't think I have a strong opinion on what the behaviour should be, but if it is intentional, this probably warrants a whatsnew note (it was confusing to debug my failing tests).

Apart of the different result, this also means that your UDF needs to be able to handle empty input (before, the UDF essentially never got empty input for those cases, I think?)

[jorisvandenbossche]

I also see that your top post has a long list of changes / fixes:

This PR fixes a number of issues that stem from this (shown in the test changes)

• ...

I think those are mostly added to the whatsnew, but it might be clearer to have a separate section explaining the general change (unobserved groups are now properly included through all APIs), and then list some of the consequence of it (instead of those bullet points that will become part of a large list of bullet points in the groupby section)

[rhshadrach]

Thanks @jorisvandenbossche - indeed this is intentional; xref #36698. I'll add a whatsnew note try to make this a notable bug fix section in a followup.