-
Notifications
You must be signed in to change notification settings - Fork 906
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Train splitting support in historical_forecasts #2064
Comments
Hi @adahan, Thanks you for this feature request, I think it's a partial duplicate of #2004. I really like the approach you are describing and would like to know your opinion on the following aspects:
Once merged with the master branch, #2050 will allow user to provide a "static" validation series to the |
You are right, its close to #2004. Thanks for your feedbacks and here are my opinions :
#2050 in combination with historical iterations callbacks might provide the best way to offer a flexible way to handle this as it could update those static val timeseries at each retrain and serve so many other use cases. |
Is your feature request related to a current problem? Please describe.
historical_forecasts doesn't not support Early stopping on val loss because historical_forecasts doesn't support train timeseries splitting.
Describe proposed solution
It would be great to add a split_before param to historical_forecasts so when retrain is not False we can split the training data in train/validation set before each retrain. Therefore enabling Early stopping on val loss for historical forecast which would speedup the process. This would also requires to add a load_from_checkpoint option to reload the model at best validation error.
Describe potential alternatives
Instead of adding parameters to historical_forecasts, we add both split_before and load_from_checkpoint to the properties of the model itself same as the callbacks property for instance and historical_forecasts could simply load split_before and load_from_checkpoint and use it both fit and historical_forecasts and would automatically split and/or load from checkpoint without external call.
Additional context
Passing early_stop on val_loss to callbacks will throw: Early stopping conditioned on metric
val_loss
which is not available. Pass in or modify yourEarlyStopping
callback to use any of the following:train_loss
when using historical_forecasts.The text was updated successfully, but these errors were encountered: