Long refit time of AutoSklearn models

[AmirAlavi]

A note to investigate further: We noticed extremely long refit time for AutoSklearn models. For example, if we set the time_left_for_this_task to be just 1hr, and so the per_run_time_limit is just 6minutes, we noticed that the refit could somehow take hours in some cases.

We didn't investigate it thoroughly, but I think I recall discovering that the HistGradientBoosting models would take a very long time to fit¹. I know ensembling adds another layer to this, but I think we had observed this even for ensemble_size=1. @eddiebergman in #1677 illuminated that ensembling also includes all the models from different folds in cross validation.

I'm wondering if you had also noticed any performance issues with that, and if the new updates address it? (perhaps the upgrade to newer sklearn will take care of it)

Footnotes

1. And maybe even also their prediction time is slow too. We attempted to do permutation feature importance analysis on an AutoSklearnClassifier and it was extremely slow if the classifier was the HistogramGradientBoostingTree, compared to a raw sklearn pipeline. ↩

[eliwoods]

I am noticing this as well. Even for small models where the run time is 2 minutes and the task time is 20 minutes, it takes over an hour (sometimes multiple hours) to refit. Here are the following relevant versions:

• Ubuntu 20.04
• Python 3.8.10
• auto-sklearn 0.15.0