Model Fitting Bias

[Ankitamani]

Hello TabPFN developers,

I have been using the TabPFN2 prior Labs Web App (https://ux.priorlabs.ai/predict) for Model Fitting and Testing for the data set with 18 Features and a target to obtain target predictions (regression task). The files for the model fitting (training data) and model testing (testing data) were created using an 80-20 split for training and testing data. The workflow includes 1) generating a model building data with Features (1-18) and target column (File: BuildingDataSet1), which is uploaded to TabPFN; the target is selected and task as regression. 2) Estimate Model performance (R2 8.56%) for TabPFN2 (File: PriorWebAppWebpage) 3) model fitting data (File: FittingDataSet1) used in the Test Set Prediction section of the web app to obtain the Model fitting predictions for the Target column. The Model predictions for the Fitting Target Set are evaluated against the actual Target values (File: FittingTargetPredictionsSet1). The Model Fitting does not have a good performance (fails) and holds a bias; I have attached the plot file for the Scatter plot of Target vs Predicted Target values obtained from model fitting (File: FittingTargetPredictionsSet1) reflecting the bias.

Why does the Model have a bias and fail to fit the data? Please suggest.

BuildingDataSet1.csv

PriorWebAppWebpagepdf.pdf

FittingDataSet1.csv

FittingTargetPredictionsSet1.csv

ModelFittingTargetvsPredicted.pdf

[noahho]

Hi @Ankitamani,

Thank you for the detailed description and additional context. Since both TabPFN2 and baseline models like Random Forest and XGBoost are underperforming, this suggests the issue might be dataset-related rather than specific to TabPFN2.

Also, could you clarify what exactly is shown in the ModelFittingTargetvsPredicted.pdf?

Best,
Noah

[LeoGrin]

Thanks @Ankitamani! My guess would be that the poor prediction comes from the very small dataset size (35 samples, and less so for the train set).

[Ankitamani]

Hi Noah,

Thanks you for the quick response, to clarify the plot, I used Scatter plot to check the model fitting for the actual values of the target vs the predicted values obtained from TabPFN2 predictions. I use the linear model (slope and intercept) to see if I can further apply corrections to the predicted values obtained from TabPFN2.

What is your view on dataset-related rather than specific to TabPFN2? Basically asking for suggestion for how/ if dataset can be improved.

Thanks,
Ankita

[Ankitamani]

Hi Leo,

Thanks for your response. I agree that the data set is small; so I also tried using the built-in mtcar data in R and performed Model Fitting; and the predictions from TabPFN2 show great model fitting (File: ModelFitting8020ActualPredictedmpg.pdf). Although sample size is a limitation, TabPFN2 performs a great task with the mtcar data (target mpg) with training data of only 26 samples but fails for my dataset with 35 samples in the training set.

ModelFitting8020ActualPredictedmpg.pdf

[noahho]

It seems like the tasks are both small, but one of them might be easier so you will get better results. We are working on finetuning - where you can learn over all your related tasks- then hopefully get a model that works well even on your second dataset.