Encode categorical columns

[reza1615]

It would be helpful to have column encoding as new column or for some cases inplace. Most common algorithms are

• OneHotEncoder https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html#sklearn.preprocessing.OneHotEncoder
• OrdinalEncoder https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OrdinalEncoder.html#sklearn.preprocessing.OrdinalEncoder
• LabelEncoder https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html#sklearn.preprocessing.LabelEncoder
• FeatureHasher https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html#sklearn.feature_extraction.FeatureHasher

[aschonfeld]

@reza1615 So I just started looking at the OneHotEncoder documentation and I'm wondering how this will work. Based on what I see in this article OneHotEncoder will end up returning an entirely new dataframe and not a column (series). Can you send over an example of how you would expect to use each of these? thanks

[reza1615]

@aschonfeld please take a look the attached file for the encoders
encoders.zip

[aschonfeld]

So based on your OneHotEncoder example in encoders.ipynb you're not using OneHotEncoder from sklearn but pd.get_dummies?

[reza1615]

Yes dummies is equal to onehot + drop the first onehot column

[aschonfeld]

@reza1615 quick question on OneHot encoder, why are we setting drop_first to True?

[reza1615]

@aschonfeld : to avoid dependency among the variables.
If you don't drop the first column then your dummy variables will be correlated. This may affect some models adversely and the effect is stronger when the cardinality is smaller. For example iterative models may have trouble converging and lists of variable importances may be distorted.

[aschonfeld]

Maybe I'm doing something wrong then. I'm running this piece of code:

df = pd.DataFrame({"car": ["Honda", "Benze", "Ford", "Honda", "Benze", "Ford"]})
pd.get_dummies(data, columns=['car'], drop_first=True)

This only returns a dataframe with car_Ford & car_Honda. If this is expected then I'll move forward with the code. Maybe this scenario I'm testing isn't valid 🤦

[reza1615]

It is correct. we don't need car_benze. (in real life we need benze 😁)
because in this data when the car isn't ford or hunda it is benze. if we had car_benze we increased co-relation of data.

[aschonfeld]

Added in v1.19.0