API/DEPR: numeric_only kwarg for apply/reductions

[jbrockmendel]

A lot of complexity in DataFrame._reduce/apply/groupby ops is driven by the numeric_only=None. In these cases we try to apply the reduction to non-numeric dtypes and if it fails, exclude them. This hugely complicated our exception handling.

We should consider removing that option and require users to subset the appropriate columns before doing their operations.

[WillAyd]

This is in need of some clean up and could certainly be refactored; might be a tough sell to do away with the implicit dropping of "nuisance" columns though

I know the OP mentions DataFrame ops, but worth mention this is very prevalent in GroupBy as well

[TomAugspurger]

might be a tough sell to do away with the implicit dropping of "nuisance" columns though.

Agreed with this.

Ignoring object-dtype for the moment, can we determine the valid dtypes to include / drop before doing the operation?

[jbrockmendel]

@TomAugspurger can you provide some context for the block quote? It looks like a dask thing that might be unrelated (or could be related in a really interesting way).

Ignoring object-dtype for the moment, can we determine the valid dtypes to include / drop before doing the operation?

I think so. Will become more certain as the exception-catching PRs in groupby progress.

[TomAugspurger]

Copy-paste fail. Fixed.

[jbrockmendel]

Related: what would you expect the behavior to be if you did df.groupby("group").mean(numeric_only=False) on a DataFrame that includes non-numeric columns?

Motivated by a case I'm troubleshooting in tests.groupby.test_function.test_arg_passthru where it looks like it is trying the op on non-numeric (in this case categorical[str]) columns and then dropping those.

[jbrockmendel]

@jorisvandenbossche you might be interested in weighing in here. The current behavior of suppressing exceptions and excluding columns is a big part of the reason why groupby is so hard to reason about

[jorisvandenbossche]

I agree that it seems a big breakage to go away from the automatic dropping of "nuisance" columns.

what would you expect the behavior to be if you did df.groupby("group").mean(numeric_only=False) on a DataFrame that includes non-numeric columns?

I would expect it to raise, like it does for df.mean(numeric_only=False) ...

[jbrockmendel]

@jreback on the call today you seemed positive on this idea, asked how many tests rely on current behavior. Looks like 50 total, roughly a third in DataFrame reductions (mostly trying to do math on strings), a third in groupby.apply/agg (some strings, some Categorical), and the rest window (mostly dt64)

I've spent a big part of the afternoon trying to get the deprecation for #36076 right and its a PITA, would be a lot easier following this.