Add imputation for missing values

[bashtage]

Has missing value handling been implemented? If so, is there any way to configure what to do? I would like to examine moving linearmodels to formulaic since I would really like an extensible formula parser, which I couldn't do with patsy.

[matthewwardrop]

Hi @bashtage! Can you clarify what you mean by "missing values"? Do you mean imputation in the original data set? Or missing values in a second dataset that you are massaging to look like the first? Or just any case where the are nulls in the data?

[bashtage]

patsy has an input called NA_action that lets you tell it what to do with NAs. I was trying to use formulas in linearmodels but noticed that I am strict with dropping and prefer to raise. I didn't see any obvious way to achieve the same result in formulaic.

[matthewwardrop]

Ah! After thinking about this a little, here is how I'd like to implement this.

I will add a new stateful transform called impute which allow you to do things like: a + impute(b, mean). It will keep track of its state, and passing future dataframes into the formula will remember which value to impute when data is missing. If after the columns are all evaluated (but before encoding) there are any null values, the materialiser will either: ignore the nulls, drop the nulls or raise, depending on the argument passed to the materialiser during its construction.

In full, then, you would something like:

from formulaic import model_matrix
df = ...
model_matrix('a + impute(b, mean)', df, na_action='drop'|'raise'|'ignore')

In this example, the na_action could only be triggered based on a (since b would have null values imputed).

Does that sound good?

[bashtage]

That sounds like an excellent solution.

As a wish list item, it would be great if impute could have an interface so that it would be possible for end users to supply their own imputers. For example, statsmodels as MICE which could be wrapped if there was an interface to do sophisticated imputing, or someone might want to use a PCA-based computer, or one that uses some non-model data such as a regression projection.

[matthewwardrop]

I was imagining that the mean passed above is a python function that exists in the namespace (it could have been np.mean, etc). So that should cover all local imputations (imputations local to a column), or non-local imputations where the function already has the relevant context (via a closure or some such). Is that sufficient?

If not, what would a non-local imputation look like (as I imagine would be the case for PCA) expressed in a formula?