Class imbalance / weights?

[otsaw]

Hi, I'm trying out Tangram for a binary classification problem. I have data with about 600 observations in one class and 6000 in another. Can I do a weighted model with Tangram that would balance the classes? Or does Tangram do some balancing implicitly under the hood?

[isabella]

Hi @otsaw! Tangram currently does not automatically handle class imbalance. What we suggest doing is upsampling your minority class or downsampling your majority class. To upsample the minority class, duplicate the rows that correspond to the minority class until the number of minority class and majority class rows are equal in your csv. You will have to create two CSV's and pass the one with the upsampled data as --file-train and the one that does not have upsampling as --file-test. To downsample the majority class, select only 600 of the 6000 majority class rows and the full 600 of the minority class rows and pass this csv with 1200 rows to Tangram. This is obviously not ideal! We want to add native support for handling class imbalance.

[otsaw]

Thanks. Downsampling seems easier , but I don't like throwing away 80% of my data. Upsampling seems to require at least a custom train-test split that avoids the replicated rows ending up in both, and a custom config tweaking at least min_examples_per_node to avoid overfitting. I'll give it try, see where I get.

[isabella]

You are correct, it does involve a custom train/test split. I'm creating an issue for adding support for class imbalanced datasets. tangramdotdev/tangram#39 (comment)

[otsaw]

I ended up with downsampling as it seemed difficult to set the parameters to work with upsampling. The downsampled models seem fine in terms of overall metrics, so that didn't turn out to be that much of an issue, but there is some amount of excess variation when looking at a single prediction due to the stochastics of the downsampling. If I sample and train again from the same data, a single prediction can change a lot. So, downsampling looks good enough to me for now, but support for weights or upsampling would of course be better.