Strange behaviour with Xgboost when dealing with categorical data type, Possibly a bug

[abhishek0093]

Hi Community !
I'm experiencing a strange behavior with Xgboost module , when I'm using it with categorical data. I have attached a sample file here with comments and results to reproduce the issue (link).

The issue I'm facing is that during training I turned on the categorical feature and followed the guidelines of documentation. I encoded the categories to lie in range [0, num_categories). The training phase follows the expected transformation of data in dmatrix .
But during test phase, when I'm doing the same transformations and enabling categorical, the transformation in the dmatrix for big(>2) dataframe doesn't follow the same encoding as happened during the training phase. Also, if I make predictions row-by-row each of the categorical_feature, each of them is getting encoded as 0.
Strangely, if I treat those categorical feature as of type integer, I'm getting the correct expected transformation for dmatrix. But, its result is different from treating the features as categorical and I'm not sure which one (or any) is correct result.

I think there is some problem with how categories are being transformed during the testing phase on a completely new dataset. Xgboost tries to encode each of the provided category to lie in the range of [0, num_categories), regardless of how they were treated during the training. For example if during training phase I had 100 unique categories each within the range [0, 100). Now if during test phase I provide same categorical column having original cat_id of 89 and 98, xgboost transform it to cat_id 0, and 1 which I think shouldn't be.

I would like to hear from the community that If there is something that I'm missing or this is some unexpected behavior.

Edit (14Oct, 2023) : Also not sure, if there should be a parameter to validate that the data type of dmatrix passed to model for prediction has same data_type of features as used during training. I'm trying to say something similar "validate_features" of predict method , but to validate type of data supplied.

[trivialfis]

Let me look into it.

[trivialfis]

Hi, in the notebook, the test data was created from fresh and doesn't maintain the category mapping from the training dataset. XGBoost by itself doesn't save the information on how data is mapped (which is part of the user's data engineering pipeline).

You can optionally use the OrdinalEncoder from sklearn (or other libraries providing similar features), or use your own mapping function with pandas (for instance, pd.Categorical.from_codes).

[trivialfis]

Added a brief document for this #9678 .

[abhishek0093]

Hi @trivialfis ,
Thankyou for responding to the issue.
However, I'm not sure if I understood your comment fully because I'm still facing the same issue even after using the same OrdinalEncoder() object both at training and testing time (link).
Could you please help with a small example to state your point as it seems I'm missing something here. I have a dataset (similar to in attached notebook) and I want to train on that dataset containing categorical data(present as integer, not necessarily continuous). But later I want to deploy the built model into production and for that I want to ensure the dmatrix constructed during test phase on an entirely new dataset follows the same transformation rules as during the training phase which currently doesn't seems to happen.

Edit 17Oct 2023 : I had thought to put every step inside a pipeline and use .fit_transform() during training and later during test time to use .transfom() to remember all transforms. But I'm not sure how to put dmatrix creation inside pipeline(I can do it for sklearn wrapper , but not sure of the native xgboost) or it is possible.

[trivialfis]

@abhishek0093 I think one doesn't need to put DMatrix construction into a pipeline, if you can get sklearn interface working then you can get DMatrix working. Looking into the fit method, you would find that the first thing XGBoost does is create a DMatrix.

[trivialfis]

I can try to write an example for using ord encoder tomorrow if you haven't got it right yet.

[abhishek0093]

@trivialfis I have already built some of the code to use with native xgboost, so it would be advantageous to use native module. Also I think sklearn wrapper currently doesn't support some methods when dealing with categorical data which was the reason for me to switch to native module.

I have already tried using various options but none seems to work. Currently the problem is that Dmatrix created during testing phase doesn’t know that data is already encoded and that it just need to parse it as it is without doing internal processing to make it to lie in range [0,n) as is required during training. One way was to put it inside a pipeline and only use transform() during testing phase, but not sure how we could do this. I think there is something which I’m missing or unexpected here.

Can you please give a working example for the issue to make sure we get correct dmatrix encoding during test phase on a new dataset while dealing with categorical data.

[trivialfis]

@abhishek0093 I put together a quick script based on your notebook, see if it addresses the issue:

import numpy as np
import pandas as pd
import xgboost as xgb
from sklearn.compose import make_column_selector, make_column_transformer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OrdinalEncoder

np.random.seed(3)


def make_df():
    """Generate data for demo/testing."""
    n_samples = 512
    df = pd.DataFrame(
        np.random.randint(32, 96, size=(n_samples, 3)),
        columns=["brand_id", "retailer_id", "category_id"],
    )

    df["price"] = np.random.randint(100, 200, size=(n_samples,))
    df["stock_status"] = np.random.choice([True, False], n_samples)
    df["on_sale"] = np.random.choice([True, False], n_samples)
    df["label"] = np.random.normal(loc=0.0, scale=1.0, size=n_samples)

    features_list = [
        "brand_id",
        "retailer_id",
        "category_id",
        "stock_status",
        "on_sale",
        "price",
    ]
    categorical_features = ["brand_id", "retailer_id", "category_id"]

    X = df[features_list]
    y = df["label"]

    return X, y, categorical_features


X, y, cat_feats = make_df()


def native():
    """Using the native XGBoost interface."""
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, random_state=3, test_size=0.2
    )

    enc = OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=np.nan)
    enc.set_output(transform="pandas")

    enc = enc.fit(X_train[cat_feats])

    def enc_transform(X: pd.DataFrame) -> pd.DataFrame:
        # don't make change inplace so that we can have demonstrations for encoding
        X = X.copy()
        cat_cols = enc.transform(X[cat_feats])
        for i, name in enumerate(cat_feats):
            cat_cols[name] = pd.Categorical.from_codes(
                codes=cat_cols[name].astype(np.int32), categories=enc.categories_[i]
            )
        X[cat_feats] = cat_cols
        return X

    X_train_enc = enc_transform(X_train)
    X_test_enc = enc_transform(X_test)

    Xy_train = xgb.QuantileDMatrix(X_train_enc, y_train, enable_categorical=True)
    Xy_test = xgb.QuantileDMatrix(
        X_test_enc, y_test, enable_categorical=True, ref=Xy_train
    )
    booster = xgb.train({}, Xy_train)
    booster.predict(Xy_test)

    # This addresses the question in the issue, to show that the encoding is done
    # consistently.
    # We first obtain result from newly encoded data
    predt0 = booster.inplace_predict(enc_transform(X_train.head(16)))
    # then we obtain result from already encoded data from training.
    predt1 = booster.inplace_predict(X_train_enc.head(16))
    np.testing.assert_allclose(predt0, predt1)


def pipeline():
    """Using sklearn pipeline."""
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, random_state=3, test_size=0.2
    )

    enc = make_column_transformer(
        (
            OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=np.nan),
            make_column_selector(pattern=".*_id"),
        ),
        remainder="passthrough",
        verbose_feature_names_out=False,
    )
    # No need to set pandas output, we use `feature_types` to indicate the type of
    # features.

    # enc.set_output(transform="pandas")

    feature_types = ["c" if fn in cat_feats else "q" for fn in X_train.columns]
    reg = xgb.XGBRegressor(
        feature_types=feature_types, enable_categorical=True, n_estimators=10
    )
    p = make_pipeline(enc, reg)
    p.fit(X_train, y_train)
    for a, b in zip(reg.get_booster().feature_types, feature_types):
        assert a == b

    predt0 = p.predict(X_train.iloc[:16, :])
    predt1 = p.predict(X_train)[:16]

    # This addresses the question in the issue, to show that the encoding is done
    # consistently.
    np.testing.assert_allclose(predt0, predt1)


if __name__ == "__main__":
    pipeline()
    native()

[abhishek0093]

Hi @trivialfis ,
Thanks a lot for the help, the issue is resolved now. I appreciate you for taking out time and resolving the issue.

I think the mistake done by me might be common and the above mentioned way doesn't seems to be present on the documentation page for the categorical data. Can you please put this example on documentation page or link this issue there as it might be helpful for others facing the same issue. If you need my help in this,I would be very happy to help.

[trivialfis]

@abhishek0093 I pushed the example into #9678 with small cleanups, feel free take a look when you get a chance.