Showing Low Variance columns

[reza1615]

Sometimes a dataset may have a categorical feature with multiple levels, where distribution of such levels are skewed and one level may dominate over other levels. This means there is not much variation in the information provided by such feature. For a ML model, such feature may not add a lot of information and thus can be ignored for modeling.
source

• Count of unique values in a feature / sample size < 10%
• Count of most common value / Count of second most common value > 20 times.

I suggest to add a part in column description to show is this column is Low Variance or not.
(e.g. Low_Variance = True/False)
Also it can be like * or ! as a new icon before column's name to show which columns are Low variance.
⑉ or 𝄇 or 🔰 or 🚩

To make it more visual
Also we can have a section on top of the column description with different flags to show

• is the categorical column, sparse
• does the column have Gaussian distribution
• is the categorical column, Low variance
• is it with high ration missing values
• is it with many outliers
  ...

this flags can also shown in the top of drop-down menu

[aschonfeld]

@reza, I really like this feature. Couple questions:

1. Can I just display the count of a value in parenthesis next to each value in the "Unique Values" section of the "Describe" popup?
2. What is the actual formula to determine if low variance? I see the following:

• (count of unique values / # rows in dataframe)
• (count of most common value) / (count of second most common)

3. Looks like I'm cutting off the count of outliers at 100 (i'll fix this)
4. How to determine sparseness?
5. How to determine Gaussian distribution? (skew = 0 & kurtosis = 3)?
6. What would be considered a "high" ratio of missing values?

[FedererKK]

Not referring specifically to categorical features, but my 2 cents

5. Jarque-Brera test for normality, or maybe even better a flag to show moments of different orders.
6. It is very much data dependant, so an input box where the user specifies a threshold maybe?

[aschonfeld]

Thanks for feedback @FedererKK i was going to start on this soon

[aschonfeld]

@FedererKK this is what I have so far: https://youtu.be/quco79Val4w

Can you give me some more information on what you meant by "a flag to show moments of different orders"?

Please let me know what else you think might be helpful to be determine variance (charts, calculations, etc...)

Thanks

[aschonfeld]

@FedererKK I also found this on stackoverflow

[aschonfeld]

@FedererKK @reza1615 I think this will be the final version

It will only be available for columns of numeric data (ints, floats). Please let me know if you think there's anything else I should add.

[aschonfeld]

added in v1.10.0