Add DataFrame.unnest/2

[costaraphael]

Building on top of #756, this adds the DataFrame.unnest/2 function.

[philss]

[josevalim]

Do we need both unnest and explode? Maybe it should be a single function that figures it out based if the column type is a list or a struct? 🤔

[cigrainger]

Do we need both unnest and explode? Maybe it should be a single function that figures it out based if the column type is a list or a struct? 🤔

Unnest is column-wise and explode is row-wise, so that's the rub. Where it gets interesting is in lists of structs and, similarly, the kind of nesting that you can do with nested dataframes in R. So in R, extreme 'tidy-ness' leads to interesting patterns, like so: https://r4ds.had.co.nz/many-models.html#nested-data

I think your question logically leads to the question of whether we want to support that sort of thing and, if so, how. I think it's really powerful and would love to support it. I think having a good, intuitive API around unnesting is essential to making it work.

So, for example, would explode on a list of structs also unnest or do you have to do those steps one after the other?Could we support doing both through an option or a third function? Would unifying to unnest feel unintuitive if you're just exploding a list column?

Edit: if we have a path to represent trained Scholar models as struct dtypes now, that basically completes the circle.

[josevalim]

I got the differences now, thanks @cigrainger and @costaraphael! Let's ship this.