Make Binary Classification metrics more robust (ValueError: Found unknown categories...)

[npatki]

Problem Description

The Binary Classification metrics, are designed to:

1. Train a binary classifier on the synthetic data, and then
2. Test the classifier on the real data

The classifier only works if all the possible category values are available during the training phase. In practice, it's possible that the synthetic data may be missing some categories.

For example, consider that there may be exceedingly rare category in the real data: credit_fraud occurs <1% of the time. This case may never be covered by the synthetic data due to sheer luck. If you had data like this, the classifier would fail with a ValueError because there status='credit_fraud' is an unknown category at the time of testing.

Expected behavior

We expect the metric to be more robust, meaning that it should not crash if it encounters this case. At a bare minimum, it may just skip over any rows with unknown values. So these rows would never even factor into the final F1 score that is returned.

[josalhor]

I think I have traced back the problem.

The problem is in the OneHotEncoder used in utils.py, this one:

SDMetrics/sdmetrics/utils.py

Line 161 in 87790f2

enc = OneHotEncoder()

By default OneHotEncoders will throw up that exception if transforming a category not seen in training. Reproduction of the error:

import pandas as pd
from sklearn.preprocessing import OneHotEncoder

# Create synthetic data (training data)
synthetic_data = pd.DataFrame({
    'feature1': [1, 2, 3, 4, 5],
    'feature2': [5, 4, 3, 2, 1],
    'status': ['normal', 'normal', 'normal', 'normal', 'normal']  # No 'credit_fraud' category
})

# Create real data (testing data)
real_data = pd.DataFrame({
    'feature1': [1, 2, 3, 6, 7],
    'feature2': [5, 4, 3, 2, 1],
    'status': ['normal', 'normal', 'credit_fraud', 'normal', 'credit_fraud']  # Contains 'credit_fraud'
})

one_hot_encoder = OneHotEncoder()

X_train = synthetic_data.drop('status', axis=1)
y_train = one_hot_encoder.fit_transform(synthetic_data[['status']]).toarray()

X_test = real_data.drop('status', axis=1)
y_test = one_hot_encoder.transform(real_data[['status']]).toarray()  # This line will cause an error

A way to fix it would be to simply add the parameter:

one_hot_encoder = OneHotEncoder(handle_unknown='ignore')

Would you accept @npatki a PR with this change?