[WIP] complete and expand df

[versipellis]

First PR I'm ever making to a big project, so bear with my noviceness :)

This PR spawns from a conversation on the Slack #data channel. It takes a dataframe as an input, and some vector of columns from that dataframe, and turns implicit into explicit missing values.

expanddf(df, indexcols) returns a dataframe of every possible combination of index cols, the length of which would be the product of the length of every column's unique values.

completedf(df, indexcols, fill=missing) wraps expanddf, but joins the original dataframe back, and provides the utility to fill the newly-missing values.

I haven't written the documentation in anticipation that feedback might cause a number of changes to the implementation. See below for examples in use.

The equivalent in...
R Tidyverse: https://tidyr.tidyverse.org/reference/complete.html
Python Pandas: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.reindex.html
Stata: https://www.stata.com/manuals13/dfillin.pdf

Example:

julia> df = DataFrame(A=["05","05","05","06","06","07","07",missing],B=["a",missing,"c","a",missing,"c","d","a"],C=[1,1,10,43,5,10,3,10])

julia> indexcols = [:A,:B]

julia> df
8×3 DataFrame
│ Row │ A       │ B       │ C     │
│     │ String⍰ │ String⍰ │ Int64 │
├─────┼─────────┼─────────┼───────┤
│ 1   │ 05      │ a       │ 1     │
│ 2   │ 05      │ missing │ 1     │
│ 3   │ 05      │ c       │ 10    │
│ 4   │ 06      │ a       │ 43    │
│ 5   │ 06      │ missing │ 5     │
│ 6   │ 07      │ c       │ 10    │
│ 7   │ 07      │ d       │ 3     │
│ 8   │ missing │ a       │ 10    │

julia> expanddf(df, indexcols)
16×2 DataFrame
│ Row │ A       │ B       │
│     │ String⍰ │ String⍰ │
├─────┼─────────┼─────────┤
│ 1   │ 05      │ a       │
│ 2   │ 06      │ a       │
│ 3   │ 07      │ a       │
│ 4   │ missing │ a       │
│ 5   │ 05      │ missing │
⋮
│ 11  │ 07      │ c       │
│ 12  │ missing │ c       │
│ 13  │ 05      │ d       │
│ 14  │ 06      │ d       │
│ 15  │ 07      │ d       │
│ 16  │ missing │ d       │

julia> completedf(df, indexcols)
16×3 DataFrame
│ Row │ A       │ B       │ C       │
│     │ String⍰ │ String⍰ │ Int64⍰  │
├─────┼─────────┼─────────┼─────────┤
│ 1   │ 05      │ a       │ 1       │
│ 2   │ 06      │ a       │ 43      │
│ 3   │ 07      │ a       │ missing │
│ 4   │ missing │ a       │ 10      │
│ 5   │ 05      │ missing │ 1       │
⋮
│ 11  │ 07      │ c       │ 10      │
│ 12  │ missing │ c       │ missing │
│ 13  │ 05      │ d       │ missing │
│ 14  │ 06      │ d       │ missing │
│ 15  │ 07      │ d       │ 3       │
│ 16  │ missing │ d       │ missing │

julia> completedf(df, indexcols, fill=10000)
16×3 DataFrame
│ Row │ A       │ B       │ C     │
│     │ String⍰ │ String⍰ │ Int64 │
├─────┼─────────┼─────────┼───────┤
│ 1   │ 05      │ a       │ 1     │
│ 2   │ 06      │ a       │ 43    │
│ 3   │ 07      │ a       │ 10000 │
│ 4   │ missing │ a       │ 10    │
│ 5   │ 05      │ missing │ 1     │
⋮
│ 11  │ 07      │ c       │ 10    │
│ 12  │ missing │ c       │ 10000 │
│ 13  │ 05      │ d       │ 10000 │
│ 14  │ 06      │ d       │ 10000 │
│ 15  │ 07      │ d       │ 3     │
│ 16  │ missing │ d       │ 10000 │

[bkamins]

Thank you for an excellent idea and contribution. I have left some suggestions in comments.

[versipellis]

Thank you for the review! I'll go through them in the next few days and make the changes/see if there's a sane approach for some of them.

[bkamins]

Thank you. Also one more thing. If a column is categorical you probably should not use unique but levels - is this what you have indented? Probably @nalimilan can comment what is the best strategy to check if a column is categorcal given the planned changes in CategoricalArrays.jl (@nalimilan - note that df in this PR can be SubDataFrame in particular).

[nalimilan]

Thanks. Regarding the API, I wonder whether we need both expand and complete (I would drop the "df" suffix anyway): wouldn't it make sense to have a single function, with an argument to determine whether the other columns should be kept (complete) or not (expand)?

[versipellis]

Thanks. Regarding the API, I wonder whether we need both expand and complete (I would drop the "df" suffix anyway): wouldn't it make sense to have a single function, with an argument to determine whether the other columns should be kept (complete) or not (expand)?

That would certainly be more elegant. I kept the different functions mostly to keep in mind users transitioning from R.

Thank you. Also one more thing. If a column is categorical you probably should not use unique but levels - is this what you have indented? Probably @nalimilan can comment what is the best strategy to check if a column is categorcal given the planned changes in CategoricalArrays.jl (@nalimilan - note that df in this PR can be SubDataFrame in particular).

Quite honestly, I forgot about categorical as a column type, and haven't used it much myself. I can go take a look at how to implement this though :)

[versipellis]

@bkamins @nalimilan I think I've addressed all the comments that were raised, except for the one about nesting. I'm not sure if that discussion is completely within the scope of this PR however - would you both be OK if I resolve that comment for the time being?

[bkamins]

@nalimilan - I am OK to leave out nesting. The only question is if we can later introduce it without breaking things (I do not know this functionality of dplyr in detail).

[versipellis]

Hi @bkamins, my apologies, real life has been absolutely a nightmare. I'll be working on this PR in the coming week or two.

I'm not very good with writing tests, however, which is where I've mostly been stuck with this up till now. I had been planning to write tests to make sure each of the args works, but that's all I got to. Do you have anything else you'd like to see covered in the testing?

[bkamins]

now it should be leftjoin

[bkamins]

Why do you think we need replaceallmissing kwarg? I think that fill is enough. If later someone wants to replace the remaining missing values that were originally in the data frame in non-indexcols then it is easily done.

[bkamins]

Excellent - thank you. This time frame is OK (as you probably know we are pushing to finalize DataFrames.jl to 1.0 release and that is why I was asking about this feature).

So the steps I would recommend you to take are:

1. do not do anything until you have time to work on PR (we are actively adding new things now, so this way you will avoid rebasing the PR several times)
2. rebase the PR (Resolve conflicts button above)
3. address all the unresolved comments
4. add docstring
5. Write tests that will cover a Cartesian product of the following things:
   • data frame type: DataFrame, SubDataFrame
   • number of rows: 0, 1, many
   • idxcols: many columns, one column, zero columns (also testing for the case when what idxcols slects is not present in the data frame)
   • all combinations of kwarg values (for fill use two values: the default and some other value to check it is probably used to fill the columns)

[bkamins]

rebase the PR

Hi @versipellis - I recommended to rebase the PR, you have merged branches. Now it is impossible to make a review (you have added 8,500 lines and removed 5,000). I think that in the current state of this PR it is best if you just take the changes you have made, rebase it to master, add only one commit with changes and force push.

[versipellis]

rebase the PR

Hi @versipellis - I recommended to rebase the PR, you have merged branches. Now it is impossible to make a review (you have added 8,500 lines and removed 5,000). I think that in the current state of this PR it is best if you just take the changes you have made, rebase it to master, add only one commit with changes and force push.

Sorry about that - I had switched laptops in the middle of the covid-19 stuff and working on this, and my own local repos were a giant mess; rebase was doing something funky. I'll do that.

[bkamins]

That is what I assumed. That is why I recommend to use "force" in git - to prune old stuff and just make one commit on top of current master. Ideally - with the changes recommended in the comments (but they can be added later if that would be simpler for you.

Fortunately this should be simple as you just add a function (and not change code in many places).

[bkamins]

I will implement this functionality in a separate PR for 1.4 release.

[bkamins]

Closing in favor of #3012