REF/API: String methods refactor

[TomAugspurger]

edit: probably closing this: See #29640 for a simpler alternative.

In preparation for #29597, this changes the string methods implemetation to

• Use masks to apply the function just to valid values.
• Disables all unnecessary inference on the result.

This works quite well for the new StringDtype, since we know the values are always string. The object-dtype behavior is a bit more complex. I'll note that inline below.

Additionally, it fixes #29624

edit: Oh, forgot another change. String methods returning numeric values will return a nullable integer type. So Series[string].str.count('a') returns an Int64Dtype, rather than a maybe int or maybe float (depending on the presence of NAs). Will document that.

[TomAugspurger]

cc @jorisvandenbossche

[TomAugspurger]

I think this optimization would be something like

result = np.zeros(dtype="int64")
result[notna] = lib.map_infer(np.asarray(arr[notna]), f)
return IntegerArray(result, mask)

which skips the object-dtype ndarray allocation.

[TomAugspurger]

Previously, this was handled inside _map by

1. calling the f on each element inside a big try/except
2. If the callable failed, we'd define a new callable that wrapped f inside a try / except, and mapped the new callable over the values.

That means something like Series(['a'] * 1000 + [None]).str.get(0) would call this f 2002 times! The first 1,000 were good, then we failed on 1,001. So we define a new wrapped callable and call that 1,001 more times.

Better to do the wrapping here.

[jorisvandenbossche]

Cool!

Did only a very quick skim, some questions:

• did you check performance? More or less the same as before?
• long term, we might want something like lib.map_infer but where we can pass a mask? (so we don't need to filter first, and assign to a subset)

[jorisvandenbossche]

Do we need to do this check? (just trying to think if users could have had objects that would work with get but would not be following this check)

[jorisvandenbossche]

I have been thinking about this as well recently in context of BooleanArray, but we might want to add a mask keyword to the pd.array(..) function?
I think that would be useful in general, and would also here give a way to not go the object way.

[jreback]

long term, we might want something like lib.map_infer but where we can pass a mask? (so we don't need to filter first, and assign to a subset)

there already is a lib.map_infer_mask FYI

[jorisvandenbossche]

there already is a lib.map_infer_mask FYI

Which is what was being used before actually. Tom, what's the reason for no longer using it? Being able to pass the 'na' value might be useful (now it takes the original value from the array, which is not always what you want I suppose)

[TomAugspurger]

did you check performance? More or less the same as before?

checking now. Slowdown in things like str.get and .str.decode in a few cases.

Will look into re-using map_infer_mask. If I can pass through an na_result rather than the original, that may be enough.

[TomAugspurger]

I wonder if it makes sense to just apply this change to just StringDtype? That'd keep the diff a lot smaller at least, and is probably less like to introduce new bugs.

[TomAugspurger]

Closing in favor of #29640