ENH: add expand kw to str.get_dummies

[sinhrks]

Ref: #10008.

Though this still under work (needs #10089 to simplify get_dummies flow), would like to discuss followings.

#### .str.extract note: overlaps with #11386

Currently it returns Series for a single group and DataFrame for multiples. To support expand kw, we have to choose :

1. Add expand option keeping existing behavior with warning for future change to extract=True (current impl).
2. Add expand option keeping existing behavior. Standardize extract=None (or other option) to select returning dimensionality automatically.
3. Add expand option with default True (or False). This breaks the API.
4. Make Index.str.extract return MultiIndex in multiple group case without adding expand option.

.str.get_dummies

1. Add expand kw with default True. Currently this always returns DataFrame (and raises TypeError in Index). This doesn't break an API (current impl).
2. Make Index.str.get_dummies return MultiIndex without adding expand option.

CC @mortada

[mortada]

I think it'd be good to change this example to showcase expand=False when it actually has multiple groups, i.e.,

>>> Series(['a1', 'b2', 'c3']).str.extract('([ab])(\d)', expand=False)
0        [a, 1]
1        [b, 2]
2    [nan, nan]
Name: [0, 1], dtype: object

I'd also move this one example down, so we'd have:

"A pattern with more than one group will return a DataFrame."

"But you can specify expand=False to return Series."

[jorisvandenbossche]

Question: it there actually a need to have the option of expand here for those two functions?

If we add them, my opinion about the discussion points:

• extract:
  • I would keep the current behaviour (seems also as a good behaviour (series for single group, dataframe for multiple groups), I see no reason to change?)
  • Have expand=None to let you able to override the default behaviour
• get_dummies:
  • OK. How would this look for expand=False? A series/index with lists as elements? (question is if we want to encourage this? -> coming back to my initial question)

[sinhrks]

@jorisvandenbossche Correct. There is an option to make Index.str work as the same as Series.str without adding expand kw. Added as alternatives. Though I prefer to make them to have unified kw/behavior.

get_dummies: How would this look for expand=False? A series/index with lists as elements?

Result will be a Series/Index of tuples, as the same as str.split(expand=False).

[jorisvandenbossche]

Not that important for this discussion, but str.split gives a list, not a tuple for me?

[jorisvandenbossche]

Though I prefer to make them to have unified kw/behavior.

Well, I would also like that very much, but the default values of the keyword would in any case not be unified. So therefore, as it is not really possible to unify it that way, I was considering the option of not adding the keyword at all (which is also no unified behaviour)

[jreback]

status?

[sinhrks]

I hope to work on this, but it requires Index.fillna to make flow simple (#10089).

[jreback]

@sinhrks what are we doing with this one?

[sinhrks]

There are 2 points, and I think 1st point (add expand=False to return Series) is less useful (only for consistency). How about adding Index.str.get_dummies and close the issue?

1. Add expand kw with default True. Currently this always returns DataFrame (and raises TypeError in Index). This doesn't break an API (current impl).
2. Make Index.str.get_dummies return MultiIndex without adding expand option.

[jreback]

@sinhrks I suppose you could add Index.str.get_dummies not really how useful this is, but it makes things consistent.