A Curious Question about Prediction Quality

[reshalfahsi]

Hi there,

Is it possible to obtain prediction quality? Is there any computation under the hood within this package to find the conformal prediction? If there weren't any could you please add it as the feature request?

Thank you.

[sonichi]

I'm making it a feature request. What package do you use today to get prediction quality for models like lightgbm or random forest?

[reshalfahsi]

Have you heard about nonconformist package? I currently use that one.

[sonichi]

It seems that you can first run flaml to do hyperparameter tuning and model selection, and then pass flaml.model to the package. Give that a try?

[reshalfahsi]

I haven't but maybe I will give it a try.

[reshalfahsi]

Hi @sonichi,

It seems the nonconformist package requires the user to train alongside the wrapper from the package. You have to wrap the model and do the magic. I recommend that you should implement the algorithm by yourself because when I apply XGBoost model I have to tweak the package a little bit.

[sonichi]

Yes it uses a wrapper. I was thinking that you could take the tuned model from flaml, wrap it with the package and retrain. Is there any issue with this approach? What tweak is required for XGBoost?

[sonichi]

@reshalfahsi I'm closing this issue as it is inactive for a month. If you still request this feature, feel free to reopen and follow up.