diff --git a/README.md b/README.md
index c87c8c3e35..ee5260bfc0 100644
--- a/README.md
+++ b/README.md
@@ -174,6 +174,9 @@ series.plot()
flavours of probabilistic forecasting (such as estimating parametric distributions or quantiles).
Some anomaly detection scorers are also able to exploit these predictive distributions.
+* **Conformal Prediction Support:** Our conformal prediction models allow to generate probabilistic forecasts with
+ calibrated quantile intervals for any pre-trained global forecasting model.
+
* **Past and Future Covariates support:** Many models in Darts support past-observed and/or future-known
covariate (external data) time series as inputs for producing forecasts.
@@ -221,51 +224,54 @@ on bringing more models and features.
| Model | Sources | Target Series Support:
Sampled/ Distribution Parameters | Training & Forecasting on Multiple Series |
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------|--------------------------------------------------------------------------|--------------------------------------------------------------------------|-------------------------------------------|
| **Baseline Models** ([LocalForecastingModel](https://unit8co.github.io/darts/userguide/covariates.html#local-forecasting-models-lfms)) | | | | | |
-| [NaiveMean](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.baselines.html#darts.models.forecasting.baselines.NaiveMean) | | ✅ ✅ | 🔴 🔴 🔴 | 🔴 🔴 | 🔴 |
-| [NaiveSeasonal](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.baselines.html#darts.models.forecasting.baselines.NaiveSeasonal) | | ✅ ✅ | 🔴 🔴 🔴 | 🔴 🔴 | 🔴 |
-| [NaiveDrift](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.baselines.html#darts.models.forecasting.baselines.NaiveDrift) | | ✅ ✅ | 🔴 🔴 🔴 | 🔴 🔴 | 🔴 |
-| [NaiveMovingAverage](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.baselines.html#darts.models.forecasting.baselines.NaiveMovingAverage) | | ✅ ✅ | 🔴 🔴 🔴 | 🔴 🔴 | 🔴 |
+| [NaiveMean](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.baselines.html#darts.models.forecasting.baselines.NaiveMean) | | ✅ ✅ | 🔴 🔴 🔴 | 🔴 🔴 | 🔴 |
+| [NaiveSeasonal](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.baselines.html#darts.models.forecasting.baselines.NaiveSeasonal) | | ✅ ✅ | 🔴 🔴 🔴 | 🔴 🔴 | 🔴 |
+| [NaiveDrift](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.baselines.html#darts.models.forecasting.baselines.NaiveDrift) | | ✅ ✅ | 🔴 🔴 🔴 | 🔴 🔴 | 🔴 |
+| [NaiveMovingAverage](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.baselines.html#darts.models.forecasting.baselines.NaiveMovingAverage) | | ✅ ✅ | 🔴 🔴 🔴 | 🔴 🔴 | 🔴 |
| **Statistical / Classic Models** ([LocalForecastingModel](https://unit8co.github.io/darts/userguide/covariates.html#local-forecasting-models-lfms)) | | | | | |
-| [ARIMA](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.arima.html#darts.models.forecasting.arima.ARIMA) | | ✅ 🔴 | 🔴 ✅ 🔴 | ✅ 🔴 | 🔴 |
-| [VARIMA](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.varima.html#darts.models.forecasting.varima.VARIMA) | | 🔴 ✅ | 🔴 ✅ 🔴 | ✅ 🔴 | 🔴 |
-| [AutoARIMA](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.auto_arima.html#darts.models.forecasting.auto_arima.AutoARIMA) | | ✅ 🔴 | 🔴 ✅ 🔴 | 🔴 🔴 | 🔴 |
-| [StatsForecastAutoArima](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.sf_auto_arima.html#darts.models.forecasting.sf_auto_arima.StatsForecastAutoARIMA) (faster AutoARIMA) | [Nixtla's statsforecast](https://github.com/Nixtla/statsforecast) | ✅ 🔴 | 🔴 ✅ 🔴 | ✅ 🔴 | 🔴 |
-| [ExponentialSmoothing](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.exponential_smoothing.html#darts.models.forecasting.exponential_smoothing.ExponentialSmoothing) | | ✅ 🔴 | 🔴 🔴 🔴 | ✅ 🔴 | 🔴 |
-| [StatsforecastAutoETS](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.sf_auto_ets.html#darts.models.forecasting.sf_auto_ets.StatsForecastAutoETS) | [Nixtla's statsforecast](https://github.com/Nixtla/statsforecast) | ✅ 🔴 | 🔴 ✅ 🔴 | ✅ 🔴 | 🔴 |
-| [StatsforecastAutoCES](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.sf_auto_ces.html#darts.models.forecasting.sf_auto_ces.StatsForecastAutoCES) | [Nixtla's statsforecast](https://github.com/Nixtla/statsforecast) | ✅ 🔴 | 🔴 🔴 🔴 | 🔴 🔴 | 🔴 |
-| [BATS](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.tbats_model.html#darts.models.forecasting.tbats_model.BATS) and [TBATS](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.tbats_model.html#darts.models.forecasting.tbats_model.TBATS) | [TBATS paper](https://robjhyndman.com/papers/ComplexSeasonality.pdf) | ✅ 🔴 | 🔴 🔴 🔴 | ✅ 🔴 | 🔴 |
-| [Theta](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.theta.html#darts.models.forecasting.theta.Theta) and [FourTheta](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.theta.html#darts.models.forecasting.theta.FourTheta) | [Theta](https://robjhyndman.com/papers/Theta.pdf) & [4 Theta](https://github.com/Mcompetitions/M4-methods/blob/master/4Theta%20method.R) | ✅ 🔴 | 🔴 🔴 🔴 | 🔴 🔴 | 🔴 |
-| [StatsForecastAutoTheta](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.sf_auto_theta.html#darts.models.forecasting.sf_auto_theta.StatsForecastAutoTheta) | [Nixtla's statsforecast](https://github.com/Nixtla/statsforecast) | ✅ 🔴 | 🔴 🔴 🔴 | ✅ 🔴 | 🔴 |
-| [Prophet](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.prophet_model.html#darts.models.forecasting.prophet_model.Prophet) | [Prophet repo](https://github.com/facebook/prophet) | ✅ 🔴 | 🔴 ✅ 🔴 | ✅ 🔴 | 🔴 |
-| [FFT](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.fft.html#darts.models.forecasting.fft.FFT) (Fast Fourier Transform) | | ✅ 🔴 | 🔴 🔴 🔴 | 🔴 🔴 | 🔴 |
-| [KalmanForecaster](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.kalman_forecaster.html#darts.models.forecasting.kalman_forecaster.KalmanForecaster) using the Kalman filter and N4SID for system identification | [N4SID paper](https://people.duke.edu/~hpgavin/SystemID/References/VanOverschee-Automatica-1994.pdf) | ✅ ✅ | 🔴 ✅ 🔴 | ✅ 🔴 | 🔴 |
-| [Croston](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.croston.html#darts.models.forecasting.croston.Croston) method | | ✅ 🔴 | 🔴 🔴 🔴 | 🔴 🔴 | 🔴 |
+| [ARIMA](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.arima.html#darts.models.forecasting.arima.ARIMA) | | ✅ 🔴 | 🔴 ✅ 🔴 | ✅ 🔴 | 🔴 |
+| [VARIMA](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.varima.html#darts.models.forecasting.varima.VARIMA) | | 🔴 ✅ | 🔴 ✅ 🔴 | ✅ 🔴 | 🔴 |
+| [AutoARIMA](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.auto_arima.html#darts.models.forecasting.auto_arima.AutoARIMA) | | ✅ 🔴 | 🔴 ✅ 🔴 | 🔴 🔴 | 🔴 |
+| [StatsForecastAutoArima](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.sf_auto_arima.html#darts.models.forecasting.sf_auto_arima.StatsForecastAutoARIMA) (faster AutoARIMA) | [Nixtla's statsforecast](https://github.com/Nixtla/statsforecast) | ✅ 🔴 | 🔴 ✅ 🔴 | ✅ 🔴 | 🔴 |
+| [ExponentialSmoothing](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.exponential_smoothing.html#darts.models.forecasting.exponential_smoothing.ExponentialSmoothing) | | ✅ 🔴 | 🔴 🔴 🔴 | ✅ 🔴 | 🔴 |
+| [StatsforecastAutoETS](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.sf_auto_ets.html#darts.models.forecasting.sf_auto_ets.StatsForecastAutoETS) | [Nixtla's statsforecast](https://github.com/Nixtla/statsforecast) | ✅ 🔴 | 🔴 ✅ 🔴 | ✅ 🔴 | 🔴 |
+| [StatsforecastAutoCES](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.sf_auto_ces.html#darts.models.forecasting.sf_auto_ces.StatsForecastAutoCES) | [Nixtla's statsforecast](https://github.com/Nixtla/statsforecast) | ✅ 🔴 | 🔴 🔴 🔴 | 🔴 🔴 | 🔴 |
+| [BATS](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.tbats_model.html#darts.models.forecasting.tbats_model.BATS) and [TBATS](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.tbats_model.html#darts.models.forecasting.tbats_model.TBATS) | [TBATS paper](https://robjhyndman.com/papers/ComplexSeasonality.pdf) | ✅ 🔴 | 🔴 🔴 🔴 | ✅ 🔴 | 🔴 |
+| [Theta](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.theta.html#darts.models.forecasting.theta.Theta) and [FourTheta](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.theta.html#darts.models.forecasting.theta.FourTheta) | [Theta](https://robjhyndman.com/papers/Theta.pdf) & [4 Theta](https://github.com/Mcompetitions/M4-methods/blob/master/4Theta%20method.R) | ✅ 🔴 | 🔴 🔴 🔴 | 🔴 🔴 | 🔴 |
+| [StatsForecastAutoTheta](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.sf_auto_theta.html#darts.models.forecasting.sf_auto_theta.StatsForecastAutoTheta) | [Nixtla's statsforecast](https://github.com/Nixtla/statsforecast) | ✅ 🔴 | 🔴 🔴 🔴 | ✅ 🔴 | 🔴 |
+| [Prophet](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.prophet_model.html#darts.models.forecasting.prophet_model.Prophet) | [Prophet repo](https://github.com/facebook/prophet) | ✅ 🔴 | 🔴 ✅ 🔴 | ✅ 🔴 | 🔴 |
+| [FFT](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.fft.html#darts.models.forecasting.fft.FFT) (Fast Fourier Transform) | | ✅ 🔴 | 🔴 🔴 🔴 | 🔴 🔴 | 🔴 |
+| [KalmanForecaster](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.kalman_forecaster.html#darts.models.forecasting.kalman_forecaster.KalmanForecaster) using the Kalman filter and N4SID for system identification | [N4SID paper](https://people.duke.edu/~hpgavin/SystemID/References/VanOverschee-Automatica-1994.pdf) | ✅ ✅ | 🔴 ✅ 🔴 | ✅ 🔴 | 🔴 |
+| [Croston](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.croston.html#darts.models.forecasting.croston.Croston) method | | ✅ 🔴 | 🔴 🔴 🔴 | 🔴 🔴 | 🔴 |
| **Global Baseline Models** ([GlobalForecastingModel](https://unit8co.github.io/darts/userguide/covariates.html#global-forecasting-models-gfms)) | | | | | |
-| [GlobalNaiveAggregate](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.global_baseline_models.html#darts.models.forecasting.global_baseline_models.GlobalNaiveAggregate) | | ✅ ✅ | 🔴 🔴 🔴 | 🔴 🔴 | ✅ |
-| [GlobalNaiveDrift](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.global_baseline_models.html#darts.models.forecasting.global_baseline_models.GlobalNaiveDrift) | | ✅ ✅ | 🔴 🔴 🔴 | 🔴 🔴 | ✅ |
-| [GlobalNaiveSeasonal](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.global_baseline_models.html#darts.models.forecasting.global_baseline_models.GlobalNaiveSeasonal) | | ✅ ✅ | 🔴 🔴 🔴 | 🔴 🔴 | ✅ |
+| [GlobalNaiveAggregate](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.global_baseline_models.html#darts.models.forecasting.global_baseline_models.GlobalNaiveAggregate) | | ✅ ✅ | 🔴 🔴 🔴 | 🔴 🔴 | ✅ |
+| [GlobalNaiveDrift](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.global_baseline_models.html#darts.models.forecasting.global_baseline_models.GlobalNaiveDrift) | | ✅ ✅ | 🔴 🔴 🔴 | 🔴 🔴 | ✅ |
+| [GlobalNaiveSeasonal](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.global_baseline_models.html#darts.models.forecasting.global_baseline_models.GlobalNaiveSeasonal) | | ✅ ✅ | 🔴 🔴 🔴 | 🔴 🔴 | ✅ |
| **Regression Models** ([GlobalForecastingModel](https://unit8co.github.io/darts/userguide/covariates.html#global-forecasting-models-gfms)) | | | | | |
-| [RegressionModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.regression_model.html#darts.models.forecasting.regression_model.RegressionModel): generic wrapper around any sklearn regression model | | ✅ ✅ | ✅ ✅ ✅ | 🔴 🔴 | ✅ |
-| [LinearRegressionModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.linear_regression_model.html#darts.models.forecasting.linear_regression_model.LinearRegressionModel) | | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
-| [RandomForest](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.random_forest.html#darts.models.forecasting.random_forest.RandomForest) | | ✅ ✅ | ✅ ✅ ✅ | 🔴 🔴 | ✅ |
-| [LightGBMModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.lgbm.html#darts.models.forecasting.lgbm.LightGBMModel) | | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
-| [XGBModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.xgboost.html#darts.models.forecasting.xgboost.XGBModel) | | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
-| [CatBoostModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.catboost_model.html#darts.models.forecasting.catboost_model.CatBoostModel) | | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
+| [RegressionModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.regression_model.html#darts.models.forecasting.regression_model.RegressionModel): generic wrapper around any sklearn regression model | | ✅ ✅ | ✅ ✅ ✅ | 🔴 🔴 | ✅ |
+| [LinearRegressionModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.linear_regression_model.html#darts.models.forecasting.linear_regression_model.LinearRegressionModel) | | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
+| [RandomForest](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.random_forest.html#darts.models.forecasting.random_forest.RandomForest) | | ✅ ✅ | ✅ ✅ ✅ | 🔴 🔴 | ✅ |
+| [LightGBMModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.lgbm.html#darts.models.forecasting.lgbm.LightGBMModel) | | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
+| [XGBModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.xgboost.html#darts.models.forecasting.xgboost.XGBModel) | | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
+| [CatBoostModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.catboost_model.html#darts.models.forecasting.catboost_model.CatBoostModel) | | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
| **PyTorch (Lightning)-based Models** ([GlobalForecastingModel](https://unit8co.github.io/darts/userguide/covariates.html#global-forecasting-models-gfms)) | | | | | |
-| [RNNModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.rnn_model.html#darts.models.forecasting.rnn_model.RNNModel) (incl. LSTM and GRU); equivalent to DeepAR in its probabilistic version | [DeepAR paper](https://arxiv.org/abs/1704.04110) | ✅ ✅ | 🔴 ✅ 🔴 | ✅ ✅ | ✅ |
-| [BlockRNNModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.block_rnn_model.html#darts.models.forecasting.block_rnn_model.BlockRNNModel) (incl. LSTM and GRU) | | ✅ ✅ | ✅ 🔴 🔴 | ✅ ✅ | ✅ |
-| [NBEATSModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.nbeats.html#darts.models.forecasting.nbeats.NBEATSModel) | [N-BEATS paper](https://arxiv.org/abs/1905.10437) | ✅ ✅ | ✅ 🔴 🔴 | ✅ ✅ | ✅ |
-| [NHiTSModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.nhits.html#darts.models.forecasting.nhits.NHiTSModel) | [N-HiTS paper](https://arxiv.org/abs/2201.12886) | ✅ ✅ | ✅ 🔴 🔴 | ✅ ✅ | ✅ |
-| [TCNModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.tcn_model.html#darts.models.forecasting.tcn_model.TCNModel) | [TCN paper](https://arxiv.org/abs/1803.01271), [DeepTCN paper](https://arxiv.org/abs/1906.04397), [blog post](https://medium.com/unit8-machine-learning-publication/temporal-convolutional-networks-and-forecasting-5ce1b6e97ce4) | ✅ ✅ | ✅ 🔴 🔴 | ✅ ✅ | ✅ |
-| [TransformerModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.transformer_model.html#darts.models.forecasting.transformer_model.TransformerModel) | | ✅ ✅ | ✅ 🔴 🔴 | ✅ ✅ | ✅ |
-| [TFTModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.tft_model.html#darts.models.forecasting.tft_model.TFTModel) (Temporal Fusion Transformer) | [TFT paper](https://arxiv.org/pdf/1912.09363.pdf), [PyTorch Forecasting](https://pytorch-forecasting.readthedocs.io/en/latest/models.html) | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
-| [DLinearModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.dlinear.html#darts.models.forecasting.dlinear.DLinearModel) | [DLinear paper](https://arxiv.org/pdf/2205.13504.pdf) | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
-| [NLinearModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.nlinear.html#darts.models.forecasting.nlinear.NLinearModel) | [NLinear paper](https://arxiv.org/pdf/2205.13504.pdf) | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
-| [TiDEModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.tide_model.html#darts.models.forecasting.tide_model.TiDEModel) | [TiDE paper](https://arxiv.org/pdf/2304.08424.pdf) | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
-| [TSMixerModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.tsmixer_model.html#darts.models.forecasting.tsmixer_model.TSMixerModel) | [TSMixer paper](https://arxiv.org/pdf/2303.06053.pdf), [PyTorch Implementation](https://github.com/ditschuk/pytorch-tsmixer) | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
+| [RNNModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.rnn_model.html#darts.models.forecasting.rnn_model.RNNModel) (incl. LSTM and GRU); equivalent to DeepAR in its probabilistic version | [DeepAR paper](https://arxiv.org/abs/1704.04110) | ✅ ✅ | 🔴 ✅ 🔴 | ✅ ✅ | ✅ |
+| [BlockRNNModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.block_rnn_model.html#darts.models.forecasting.block_rnn_model.BlockRNNModel) (incl. LSTM and GRU) | | ✅ ✅ | ✅ 🔴 🔴 | ✅ ✅ | ✅ |
+| [NBEATSModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.nbeats.html#darts.models.forecasting.nbeats.NBEATSModel) | [N-BEATS paper](https://arxiv.org/abs/1905.10437) | ✅ ✅ | ✅ 🔴 🔴 | ✅ ✅ | ✅ |
+| [NHiTSModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.nhits.html#darts.models.forecasting.nhits.NHiTSModel) | [N-HiTS paper](https://arxiv.org/abs/2201.12886) | ✅ ✅ | ✅ 🔴 🔴 | ✅ ✅ | ✅ |
+| [TCNModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.tcn_model.html#darts.models.forecasting.tcn_model.TCNModel) | [TCN paper](https://arxiv.org/abs/1803.01271), [DeepTCN paper](https://arxiv.org/abs/1906.04397), [blog post](https://medium.com/unit8-machine-learning-publication/temporal-convolutional-networks-and-forecasting-5ce1b6e97ce4) | ✅ ✅ | ✅ 🔴 🔴 | ✅ ✅ | ✅ |
+| [TransformerModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.transformer_model.html#darts.models.forecasting.transformer_model.TransformerModel) | | ✅ ✅ | ✅ 🔴 🔴 | ✅ ✅ | ✅ |
+| [TFTModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.tft_model.html#darts.models.forecasting.tft_model.TFTModel) (Temporal Fusion Transformer) | [TFT paper](https://arxiv.org/pdf/1912.09363.pdf), [PyTorch Forecasting](https://pytorch-forecasting.readthedocs.io/en/latest/models.html) | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
+| [DLinearModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.dlinear.html#darts.models.forecasting.dlinear.DLinearModel) | [DLinear paper](https://arxiv.org/pdf/2205.13504.pdf) | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
+| [NLinearModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.nlinear.html#darts.models.forecasting.nlinear.NLinearModel) | [NLinear paper](https://arxiv.org/pdf/2205.13504.pdf) | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
+| [TiDEModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.tide_model.html#darts.models.forecasting.tide_model.TiDEModel) | [TiDE paper](https://arxiv.org/pdf/2304.08424.pdf) | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
+| [TSMixerModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.tsmixer_model.html#darts.models.forecasting.tsmixer_model.TSMixerModel) | [TSMixer paper](https://arxiv.org/pdf/2303.06053.pdf), [PyTorch Implementation](https://github.com/ditschuk/pytorch-tsmixer) | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
| **Ensemble Models** ([GlobalForecastingModel](https://unit8co.github.io/darts/userguide/covariates.html#global-forecasting-models-gfms)): Model support is dependent on ensembled forecasting models and the ensemble model itself | | | | | |
-| [NaiveEnsembleModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.baselines.html#darts.models.forecasting.baselines.NaiveEnsembleModel) | | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
-| [RegressionEnsembleModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.regression_ensemble_model.html#darts.models.forecasting.regression_ensemble_model.RegressionEnsembleModel) | | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
+| [NaiveEnsembleModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.baselines.html#darts.models.forecasting.baselines.NaiveEnsembleModel) | | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
+| [RegressionEnsembleModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.regression_ensemble_model.html#darts.models.forecasting.regression_ensemble_model.RegressionEnsembleModel) | | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
+| **Conformal Models** ([GlobalForecastingModel](https://unit8co.github.io/darts/userguide/covariates.html#global-forecasting-models-gfms)): Model support is dependent on the forecasting model used | | | | | |
+| [ConformalNaiveModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.conformal_models.html#darts.models.forecasting.conformal_models.ConformalNaiveModel) | [Conformalized Prediction](https://arxiv.org/pdf/1905.03222) | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
+| [ConformalQRModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.conformal_models.html#darts.models.forecasting.conformal_models.ConformalQRModel) | [Conformalized Quantile Regression](https://arxiv.org/pdf/1905.03222) | ✅ ✅ | ✅ ✅ ✅ | ✅ ✅ | ✅ |
## Community & Contact
Anyone is welcome to join our [Gitter room](https://gitter.im/u8darts/darts) to ask questions, make proposals,
diff --git a/darts/ad/anomaly_model/forecasting_am.py b/darts/ad/anomaly_model/forecasting_am.py
index e90d088c7f..88ce67b9ce 100644
--- a/darts/ad/anomaly_model/forecasting_am.py
+++ b/darts/ad/anomaly_model/forecasting_am.py
@@ -120,10 +120,9 @@ def fit(
If set to 'value', `start` corresponds to the index value/label of the first predicted point. Will raise
an error if the value is not in `series`' index. Default: `'value'`
num_samples
- Number of times a prediction is sampled from a probabilistic model. Should be left set to 1 for
- deterministic models.
+ Number of times a prediction is sampled from a probabilistic model. Must be `1` for deterministic models.
verbose
- Whether to print progress.
+ Whether to print the progress.
show_warnings
Whether to show warnings related to historical forecasts optimization, or parameters `start` and
`train_length`.
@@ -201,10 +200,9 @@ def score(
If set to 'value', `start` corresponds to the index value/label of the first predicted point. Will raise
an error if the value is not in `series`' index. Default: `'value'`
num_samples
- Number of times a prediction is sampled from a probabilistic model. Should be left set to 1 for
- deterministic models.
+ Number of times a prediction is sampled from a probabilistic model. Must be `1` for deterministic models.
verbose
- Whether to print progress.
+ Whether to print the progress.
show_warnings
Whether to show warnings related to historical forecasts optimization, or parameters `start` and
`train_length`.
@@ -289,10 +287,9 @@ def predict_series(
If set to 'value', `start` corresponds to the index value/label of the first predicted point. Will raise
an error if the value is not in `series`' index. Default: `'value'`
num_samples
- Number of times a prediction is sampled from a probabilistic model. Should be left set to 1 for
- deterministic models.
+ Number of times a prediction is sampled from a probabilistic model. Must be `1` for deterministic models.
verbose
- Whether to print progress.
+ Whether to print the progress.
show_warnings
Whether to show warnings related to historical forecasts optimization, or parameters `start` and
`train_length`.
@@ -385,10 +382,9 @@ def eval_metric(
If set to 'value', `start` corresponds to the index value/label of the first predicted point. Will raise
an error if the value is not in `series`' index. Default: `'value'`
num_samples
- Number of times a prediction is sampled from a probabilistic model. Should be left set to 1 for
- deterministic models.
+ Number of times a prediction is sampled from a probabilistic model. Must be `1` for deterministic models.
verbose
- Whether to print progress.
+ Whether to print the progress.
show_warnings
Whether to show warnings related to historical forecasts optimization, or parameters `start` and
`train_length`.
@@ -491,10 +487,9 @@ def show_anomalies(
If set to 'value', `start` corresponds to the index value/label of the first predicted point. Will raise
an error if the value is not in `series`' index. Default: `'value'`
num_samples
- Number of times a prediction is sampled from a probabilistic model. Should be left set to 1 for
- deterministic models.
+ Number of times a prediction is sampled from a probabilistic model. Must be `1` for deterministic models.
verbose
- Whether to print progress.
+ Whether to print the progress.
show_warnings
Whether to show warnings related to historical forecasts optimization, or parameters `start` and
`train_length`.
diff --git a/darts/metrics/__init__.py b/darts/metrics/__init__.py
index 72bc38c89a..d8b15c3c2f 100644
--- a/darts/metrics/__init__.py
+++ b/darts/metrics/__init__.py
@@ -6,52 +6,67 @@
and quantile forecasts. For probabilistic and quantile forecasts, use parameter `q` to define the quantile(s) to
compute the deterministic metrics on:
- - Aggregated over time:
- Absolute metrics:
- - :func:`MERR `: Mean Error
- - :func:`MAE `: Mean Absolute Error
- - :func:`MSE `: Mean Squared Error
- - :func:`RMSE `: Root Mean Squared Error
- - :func:`RMSLE `: Root Mean Squared Log Error
-
- Relative metrics:
- - :func:`MASE `: Mean Absolute Scaled Error
- - :func:`MSSE `: Mean Squared Scaled Error
- - :func:`RMSSE `: Root Mean Squared Scaled Error
- - :func:`MAPE `: Mean Absolute Percentage Error
- - :func:`sMAPE `: symmetric Mean Absolute Percentage Error
- - :func:`OPE `: Overall Percentage Error
- - :func:`MARRE `: Mean Absolute Ranged Relative Error
-
- Other metrics:
- - :func:`R2 `: Coefficient of Determination
- - :func:`CV `: Coefficient of Variation
-
- - Per time step:
- Absolute metrics:
- - :func:`ERR `: Error
- - :func:`AE `: Absolute Error
- - :func:`SE `: Squared Error
- - :func:`SLE `: Squared Log Error
-
- Relative metrics:
- - :func:`ASE `: Absolute Scaled Error
- - :func:`SSE `: Squared Scaled Error
- - :func:`APE `: Absolute Percentage Error
- - :func:`sAPE `: symmetric Absolute Percentage Error
- - :func:`ARRE `: Absolute Ranged Relative Error
-
-For probabilistic forecasts (storchastic predictions with `num_samples >> 1`):
- - Aggregated over time:
+- Aggregated over time:
+ Absolute metrics:
+ - :func:`MERR `: Mean Error
+ - :func:`MAE `: Mean Absolute Error
+ - :func:`MSE `: Mean Squared Error
+ - :func:`RMSE `: Root Mean Squared Error
+ - :func:`RMSLE `: Root Mean Squared Log Error
+
+ Relative metrics:
+ - :func:`MASE `: Mean Absolute Scaled Error
+ - :func:`MSSE `: Mean Squared Scaled Error
+ - :func:`RMSSE `: Root Mean Squared Scaled Error
+ - :func:`MAPE `: Mean Absolute Percentage Error
+ - :func:`sMAPE `: symmetric Mean Absolute Percentage Error
+ - :func:`OPE `: Overall Percentage Error
+ - :func:`MARRE `: Mean Absolute Ranged Relative Error
+
+ Other metrics:
+ - :func:`R2 `: Coefficient of Determination
+ - :func:`CV `: Coefficient of Variation
+
+- Per time step:
+ Absolute metrics:
+ - :func:`ERR `: Error
+ - :func:`AE `: Absolute Error
+ - :func:`SE `: Squared Error
+ - :func:`SLE `: Squared Log Error
+
+ Relative metrics:
+ - :func:`ASE `: Absolute Scaled Error
+ - :func:`SSE `: Squared Scaled Error
+ - :func:`APE `: Absolute Percentage Error
+ - :func:`sAPE `: symmetric Absolute Percentage Error
+ - :func:`ARRE `: Absolute Ranged Relative Error
+
+For probabilistic forecasts (storchastic predictions with `num_samples >> 1`) and quantile forecasts:
+
+- Aggregated over time:
+ Quantile metrics:
- :func:`MQL `: Mean Quantile Loss
- :func:`QR `: Quantile Risk
+
+ Quantile interval metrics:
- :func:`MIW `: Mean Interval Width
- - Per time step:
+ - :func:`MWS `: Mean Interval Winkler Score
+ - :func:`MIC `: Mean Interval Coverage
+ - :func:`MINCS_QR `: Mean Interval Non-Conformity Score for Quantile Regression
+
+- Per time step:
+ Quantile metrics:
- :func:`QL `: Quantile Loss
+
+ Quantile interval metrics:
- :func:`IW `: Interval Width
+ - :func:`WS `: Interval Winkler Score
+ - :func:`IC `: Interval Coverage
+ - :func:`INCS_QR `: Interval Non-Conformity Score for Quantile Regression
For Dynamic Time Warping (DTW) (aggregated over time):
- - :func:`DTW `: Dynamic Time Warping Metric
+
+- :func:`DTW `: Dynamic Time Warping Metric
"""
from darts.metrics.metrics import (
@@ -62,13 +77,19 @@
coefficient_of_variation,
dtw_metric,
err,
+ ic,
+ incs_qr,
iw,
+ iws,
mae,
mape,
marre,
mase,
merr,
+ mic,
+ mincs_qr,
miw,
+ miws,
mql,
mse,
msse,
@@ -86,6 +107,44 @@
sse,
)
+ALL_METRICS = {
+ ae,
+ ape,
+ arre,
+ ase,
+ coefficient_of_variation,
+ dtw_metric,
+ err,
+ iw,
+ iws,
+ mae,
+ mape,
+ marre,
+ mase,
+ merr,
+ miw,
+ miws,
+ mql,
+ mse,
+ msse,
+ ope,
+ ql,
+ qr,
+ r2_score,
+ rmse,
+ rmsle,
+ rmsse,
+ sape,
+ se,
+ sle,
+ smape,
+ sse,
+ ic,
+ mic,
+ incs_qr,
+ mincs_qr,
+}
+
TIME_DEPENDENT_METRICS = {
ae,
ape,
@@ -98,8 +157,23 @@
sle,
sse,
iw,
+ iws,
+ ic,
+ incs_qr,
}
+Q_INTERVAL_METRICS = {
+ iw,
+ iws,
+ miw,
+ miws,
+ ic,
+ mic,
+ incs_qr,
+}
+
+NON_Q_METRICS = {dtw_metric}
+
__all__ = [
"ae",
"ape",
@@ -130,4 +204,10 @@
"sse",
"iw",
"miw",
+ "iws",
+ "miws",
+ "ic",
+ "mic",
+ "incs_qr",
+ "mincs_qr",
]
diff --git a/darts/metrics/metrics.py b/darts/metrics/metrics.py
index eb99ef6ab0..63c7b4ac2a 100644
--- a/darts/metrics/metrics.py
+++ b/darts/metrics/metrics.py
@@ -216,6 +216,7 @@ def wrapper_multi_ts_support(*args, **kwargs):
iterable=zip(*input_series),
verbose=verbose,
total=len(actual_series),
+ desc=f"metric `{func.__name__}()`",
)
# `vals` is a list of series metrics of length `len(actual_series)`. Each metric has shape
@@ -657,7 +658,7 @@ def err(
"""Error (ERR).
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed per
- component/column and time step :math:`t` as:
+ component/column, (optional) quantile, and time step :math:`t` as:
.. math:: y_t - \\hat{y}_t
@@ -702,23 +703,25 @@ def err(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- a sequence (list) of uni/multivariate series with `series_reduction`, `component_reduction` and
`time_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n time steps, n components) without time
- and component reductions. For:
+ A numpy array of metric scores. The array has shape (n time steps, n components * n quantiles) without time
+ and component reductions, and shape (n time steps, n quantiles) without time but component reduction and
+ `len(q) > 1`. For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- single uni/multivariate series and at least `time_reduction=None`.
- a sequence of uni/multivariate series including `series_reduction` and at least one of
`component_reduction=None` or `time_reduction=None`.
- List[float]
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
@@ -748,7 +751,7 @@ def merr(
"""Mean Error (MERR).
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed per
- component/column as:
+ component/column and (optional) quantile as:
.. math:: \\frac{1}{T}\\sum_{t=1}^T{(y_t - \\hat{y}_t)}
@@ -788,19 +791,22 @@ def merr(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- - sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n components,) without component reduction. For:
+ A numpy array of metric scores. The array has shape (n components * n quantiles,) without component reduction,
+ and shape (n quantiles,) with component reduction and `len(q) > 1`.
+ For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- - sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
- List[float]
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
return np.nanmean(
@@ -831,7 +837,7 @@ def ae(
"""Absolute Error (AE).
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed per
- component/column and time step :math:`t` as:
+ component/column, (optional) quantile, and time step :math:`t` as:
.. math:: |y_t - \\hat{y}_t|
@@ -876,23 +882,25 @@ def ae(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- a sequence (list) of uni/multivariate series with `series_reduction`, `component_reduction` and
`time_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n time steps, n components) without time
- and component reductions. For:
+ A numpy array of metric scores. The array has shape (n time steps, n components * n quantiles) without time
+ and component reductions, and shape (n time steps, n quantiles) without time but component reduction and
+ `len(q) > 1`. For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- single uni/multivariate series and at least `time_reduction=None`.
- a sequence of uni/multivariate series including `series_reduction` and at least one of
`component_reduction=None` or `time_reduction=None`.
- List[float]
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
@@ -922,7 +930,7 @@ def mae(
"""Mean Absolute Error (MAE).
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed per
- component/column as:
+ component/column and (optional) quantile as:
.. math:: \\frac{1}{T}\\sum_{t=1}^T{|y_t - \\hat{y}_t|}
@@ -962,19 +970,22 @@ def mae(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- - sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n components,) without component reduction. For:
+ A numpy array of metric scores. The array has shape (n components * n quantiles,) without component reduction,
+ and shape (n quantiles,) with component reduction and `len(q) > 1`.
+ For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- - sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
- List[float]
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
return np.nanmean(
@@ -1009,7 +1020,7 @@ def ase(
It is the Absolute Error (AE) scaled by the Mean AE (MAE) of the naive m-seasonal forecast.
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed per
- component/column and time step :math:`t` as:
+ component/column, (optional) quantile, and time step :math:`t` as:
.. math:: \\frac{AE(y_{t_p+1:t_p+T}, \\hat{y}_{t_p+1:t_p+T})}{E_m},
@@ -1073,23 +1084,25 @@ def ase(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- a sequence (list) of uni/multivariate series with `series_reduction`, `component_reduction` and
`time_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n time steps, n components) without time
- and component reductions. For:
+ A numpy array of metric scores. The array has shape (n time steps, n components * n quantiles) without time
+ and component reductions, and shape (n time steps, n quantiles) without time but component reduction and
+ `len(q) > 1`. For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- single uni/multivariate series and at least `time_reduction=None`.
- a sequence of uni/multivariate series including `series_reduction` and at least one of
`component_reduction=None` or `time_reduction=None`.
- List[float]
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
References
@@ -1126,7 +1139,7 @@ def mase(
It is the Mean Absolute Error (MAE) scaled by the MAE of the naive m-seasonal forecast.
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed per
- component/column as:
+ component/column and (optional) quantile as:
.. math:: \\frac{MAE(y_{t_p+1:t_p+T}, \\hat{y}_{t_p+1:t_p+T})}{E_m},
@@ -1185,19 +1198,22 @@ def mase(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- - sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n components,) without component reduction. For:
+ A numpy array of metric scores. The array has shape (n components * n quantiles,) without component reduction,
+ and shape (n quantiles,) with component reduction and `len(q) > 1`.
+ For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- - sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
- List[float]
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
References
@@ -1234,7 +1250,7 @@ def se(
"""Squared Error (SE).
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed per
- component/column and time step :math:`t` as:
+ component/column, (optional) quantile, and time step :math:`t` as:
.. math:: (y_t - \\hat{y}_t)^2.
@@ -1279,23 +1295,25 @@ def se(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- a sequence (list) of uni/multivariate series with `series_reduction`, `component_reduction` and
`time_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n time steps, n components) without time
- and component reductions. For:
+ A numpy array of metric scores. The array has shape (n time steps, n components * n quantiles) without time
+ and component reductions, and shape (n time steps, n quantiles) without time but component reduction and
+ `len(q) > 1`. For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- single uni/multivariate series and at least `time_reduction=None`.
- a sequence of uni/multivariate series including `series_reduction` and at least one of
`component_reduction=None` or `time_reduction=None`.
- List[float]
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
@@ -1325,7 +1343,7 @@ def mse(
"""Mean Squared Error (MSE).
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed per
- component/column as:
+ component/column and (optional) quantile as:
.. math:: \\frac{1}{T}\\sum_{t=1}^T{(y_t - \\hat{y}_t)^2}.
@@ -1365,19 +1383,22 @@ def mse(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- - sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n components,) without component reduction. For:
+ A numpy array of metric scores. The array has shape (n components * n quantiles,) without component reduction,
+ and shape (n quantiles,) with component reduction and `len(q) > 1`.
+ For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- - sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
- List[float]
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
return np.nanmean(
@@ -1412,7 +1433,7 @@ def sse(
It is the Squared Error (SE) scaled by the Mean SE (MSE) of the naive m-seasonal forecast.
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed per
- component/column and time step :math:`t` as:
+ component/column, (optional) quantile, and time step :math:`t` as:
.. math:: \\frac{SE(y_{t_p+1:t_p+T}, \\hat{y}_{t_p+1:t_p+T})}{E_m},
@@ -1476,23 +1497,25 @@ def sse(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- a sequence (list) of uni/multivariate series with `series_reduction`, `component_reduction` and
`time_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n time steps, n components) without time
- and component reductions. For:
+ A numpy array of metric scores. The array has shape (n time steps, n components * n quantiles) without time
+ and component reductions, and shape (n time steps, n quantiles) without time but component reduction and
+ `len(q) > 1`. For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- single uni/multivariate series and at least `time_reduction=None`.
- a sequence of uni/multivariate series including `series_reduction` and at least one of
`component_reduction=None` or `time_reduction=None`.
- List[float]
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
References
@@ -1529,7 +1552,7 @@ def msse(
It is the Mean Squared Error (MSE) scaled by the MSE of the naive m-seasonal forecast.
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed per
- component/column as:
+ component/column and (optional) quantile as:
.. math:: \\frac{MSE(y_{t_p+1:t_p+T}, \\hat{y}_{t_p+1:t_p+T})}{E_m},
@@ -1588,19 +1611,22 @@ def msse(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- - sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n components,) without component reduction. For:
+ A numpy array of metric scores. The array has shape (n components * n quantiles,) without component reduction,
+ and shape (n quantiles,) with component reduction and `len(q) > 1`.
+ For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- - sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
- List[float]
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
References
@@ -1636,7 +1662,7 @@ def rmse(
"""Root Mean Squared Error (RMSE).
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed per
- component/column as:
+ component/column and (optional) quantile as:
.. math:: \\sqrt{\\frac{1}{T}\\sum_{t=1}^T{(y_t - \\hat{y}_t)^2}}
@@ -1676,19 +1702,22 @@ def rmse(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- - sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n components,) without component reduction. For:
+ A numpy array of metric scores. The array has shape (n components * n quantiles,) without component reduction,
+ and shape (n quantiles,) with component reduction and `len(q) > 1`.
+ For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- - sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
- List[float]
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
return np.sqrt(
@@ -1721,7 +1750,7 @@ def rmsse(
It is the Root Mean Squared Error (RMSE) scaled by the RMSE of the naive m-seasonal forecast.
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed per
- component/column as:
+ component/column and (optional) quantile as:
.. math:: \\frac{RMSE(y_{t_p+1:t_p+T}, \\hat{y}_{t_p+1:t_p+T})}{E_m},
@@ -1780,19 +1809,22 @@ def rmsse(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- - sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n components,) without component reduction. For:
+ A numpy array of metric scores. The array has shape (n components * n quantiles,) without component reduction,
+ and shape (n quantiles,) with component reduction and `len(q) > 1`.
+ For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- - sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
- List[float]
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
References
@@ -1826,7 +1858,7 @@ def sle(
"""Squared Log Error (SLE).
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed per
- component/column and time step :math:`t` as:
+ component/column, (optional) quantile, and time step :math:`t` as:
.. math:: \\left(\\log{(y_t + 1)} - \\log{(\\hat{y} + 1)}\\right)^2
@@ -1873,23 +1905,25 @@ def sle(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- a sequence (list) of uni/multivariate series with `series_reduction`, `component_reduction` and
`time_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n time steps, n components) without time
- and component reductions. For:
+ A numpy array of metric scores. The array has shape (n time steps, n components * n quantiles) without time
+ and component reductions, and shape (n time steps, n quantiles) without time but component reduction and
+ `len(q) > 1`. For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- single uni/multivariate series and at least `time_reduction=None`.
- a sequence of uni/multivariate series including `series_reduction` and at least one of
`component_reduction=None` or `time_reduction=None`.
- List[float]
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
@@ -1920,7 +1954,7 @@ def rmsle(
"""Root Mean Squared Log Error (RMSLE).
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed per
- component/column as:
+ component/column and (optional) quantile as:
.. math:: \\sqrt{\\frac{1}{T}\\sum_{t=1}^T{\\left(\\log{(y_t + 1)} - \\log{(\\hat{y}_t + 1)}\\right)^2}}
@@ -1962,19 +1996,22 @@ def rmsle(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- - sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n components,) without component reduction. For:
+ A numpy array of metric scores. The array has shape (n components * n quantiles,) without component reduction,
+ and shape (n quantiles,) with component reduction and `len(q) > 1`.
+ For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- - sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
- List[float]
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
return np.sqrt(
@@ -2060,23 +2097,25 @@ def ape(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- a sequence (list) of uni/multivariate series with `series_reduction`, `component_reduction` and
`time_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n time steps, n components) without time
- and component reductions. For:
+ A numpy array of metric scores. The array has shape (n time steps, n components * n quantiles) without time
+ and component reductions, and shape (n time steps, n quantiles) without time but component reduction and
+ `len(q) > 1`. For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- single uni/multivariate series and at least `time_reduction=None`.
- a sequence of uni/multivariate series including `series_reduction` and at least one of
`component_reduction=None` or `time_reduction=None`.
- List[float]
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
@@ -2113,7 +2152,7 @@ def mape(
"""Mean Absolute Percentage Error (MAPE).
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed as a
- percentage value per component/column with:
+ percentage value per component/column and (optional) quantile with:
.. math:: 100 \\cdot \\frac{1}{T} \\sum_{t=1}^{T}{\\left| \\frac{y_t - \\hat{y}_t}{y_t} \\right|}
@@ -2161,19 +2200,22 @@ def mape(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- - sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n components,) without component reduction. For:
+ A numpy array of metric scores. The array has shape (n components * n quantiles,) without component reduction,
+ and shape (n quantiles,) with component reduction and `len(q) > 1`.
+ For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- - sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
- List[float]
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
@@ -2205,7 +2247,7 @@ def sape(
"""symmetric Absolute Percentage Error (sAPE).
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed as a
- percentage value per component/column and time step :math:`t` with:
+ percentage value per component/column, (optional) quantile and time step :math:`t` with:
.. math::
200 \\cdot \\frac{\\left| y_t - \\hat{y}_t \\right|}{\\left| y_t \\right| + \\left| \\hat{y}_t \\right|}
@@ -2259,23 +2301,25 @@ def sape(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- a sequence (list) of uni/multivariate series with `series_reduction`, `component_reduction` and
`time_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n time steps, n components) without time
- and component reductions. For:
+ A numpy array of metric scores. The array has shape (n time steps, n components * n quantiles) without time
+ and component reductions, and shape (n time steps, n quantiles) without time but component reduction and
+ `len(q) > 1`. For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- single uni/multivariate series and at least `time_reduction=None`.
- a sequence of uni/multivariate series including `series_reduction` and at least one of
`component_reduction=None` or `time_reduction=None`.
- List[float]
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
@@ -2312,7 +2356,7 @@ def smape(
"""symmetric Mean Absolute Percentage Error (sMAPE).
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed as a
- percentage value per component/column with:
+ percentage value per component/column and (optional) quantile with:
.. math::
200 \\cdot \\frac{1}{T}
@@ -2363,19 +2407,22 @@ def smape(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- - sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n components,) without component reduction. For:
+ A numpy array of metric scores. The array has shape (n components * n quantiles,) without component reduction,
+ and shape (n quantiles,) with component reduction and `len(q) > 1`.
+ For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- - sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
- List[float]
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
@@ -2406,7 +2453,7 @@ def ope(
"""Overall Percentage Error (OPE).
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed as a
- percentage value per component/column with:
+ percentage value per component/column and (optional) quantile with:
.. math:: 100 \\cdot \\left| \\frac{\\sum_{t=1}^{T}{y_t}
- \\sum_{t=1}^{T}{\\hat{y}_t}}{\\sum_{t=1}^{T}{y_t}} \\right|.
@@ -2452,19 +2499,22 @@ def ope(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- - sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n components,) without component reduction. For:
+ A numpy array of metric scores. The array has shape (n components * n quantiles,) without component reduction,
+ and shape (n quantiles,) with component reduction and `len(q) > 1`.
+ For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- - sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
- List[float]
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
@@ -2506,7 +2556,7 @@ def arre(
"""Absolute Ranged Relative Error (ARRE).
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed as a
- percentage value per component/column and time step :math:`t` with:
+ percentage value per component/column, (optional) quantile and time step :math:`t` with:
.. math:: 100 \\cdot \\left| \\frac{y_t - \\hat{y}_t} {\\max_t{y_t} - \\min_t{y_t}} \\right|
@@ -2556,23 +2606,25 @@ def arre(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- a sequence (list) of uni/multivariate series with `series_reduction`, `component_reduction` and
`time_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n time steps, n components) without time
- and component reductions. For:
+ A numpy array of metric scores. The array has shape (n time steps, n components * n quantiles) without time
+ and component reductions, and shape (n time steps, n quantiles) without time but component reduction and
+ `len(q) > 1`. For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- single uni/multivariate series and at least `time_reduction=None`.
- a sequence of uni/multivariate series including `series_reduction` and at least one of
`component_reduction=None` or `time_reduction=None`.
- List[float]
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
@@ -2612,7 +2664,7 @@ def marre(
"""Mean Absolute Ranged Relative Error (MARRE).
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed as a
- percentage value per component/column with:
+ percentage value per component/column and (optional) quantile with:
.. math:: 100 \\cdot \\frac{1}{T} \\sum_{t=1}^{T} {\\left| \\frac{y_t - \\hat{y}_t} {\\max_t{y_t} -
\\min_t{y_t}} \\right|}
@@ -2666,9 +2718,9 @@ def marre(
- single multivariate series and at least `component_reduction=None`.
- sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
- List[float]
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
return np.nanmean(
@@ -2698,7 +2750,7 @@ def r2_score(
"""Coefficient of Determination :math:`R^2` (see [1]_ for more details).
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed per
- component/column as:
+ component/column and (optional) quantile as:
.. math:: 1 - \\frac{\\sum_{t=1}^T{(y_t - \\hat{y}_t)^2}}{\\sum_{t=1}^T{(y_t - \\bar{y})^2}},
@@ -2742,19 +2794,22 @@ def r2_score(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- - sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n components,) without component reduction. For:
+ A numpy array of metric scores. The array has shape (n components * n quantiles,) without component reduction,
+ and shape (n quantiles,) with component reduction and `len(q) > 1`.
+ For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- - sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
- List[float]
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
References
@@ -2790,7 +2845,7 @@ def coefficient_of_variation(
"""Coefficient of Variation (percentage).
For the true series :math:`y` and predicted series :math:`\\hat{y}` of length :math:`T`, it is computed per
- component/column as a percentage value with:
+ component/column and (optional) quantile as a percentage value with:
.. math:: 100 \\cdot \\text{RMSE}(y_t, \\hat{y}_t) / \\bar{y},
@@ -2833,19 +2888,22 @@ def coefficient_of_variation(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- - sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n components,) without component reduction. For:
+ A numpy array of metric scores. The array has shape (n components * n quantiles,) without component reduction,
+ and shape (n quantiles,) with component reduction and `len(q) > 1`.
+ For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- - sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
- List[float]
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
@@ -2921,19 +2979,22 @@ def dtw_metric(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- - sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n components,) without component reduction. For:
+ A numpy array of metric scores. The array has shape (n components * n quantiles,) without component reduction,
+ and shape (n quantiles,) with component reduction and `len(q) > 1`.
+ For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- - sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
- List[float]
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
@@ -2967,7 +3028,7 @@ def qr(
sample values summed up along the time axis (QL computes the quantile and loss per time step).
For the true series :math:`y` and predicted stochastic/probabilistic series (containing N samples) :math:`\\hat{y}`
- of of shape :math:`T \\times N`, it is computed per column/component as:
+ of of shape :math:`T \\times N`, it is computed per column/component and quantile as:
.. math:: 2 \\frac{QL(Z, \\hat{Z}_q)}{Z},
@@ -3007,19 +3068,22 @@ def qr(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- - sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n components,) without component reduction. For:
+ A numpy array of metric scores. The array has shape (n components * n quantiles,) without component reduction,
+ and shape (n quantiles,) with component reduction and `len(q) > 1`.
+ For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- - sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
- List[float]
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
if not pred_series.is_stochastic:
@@ -3075,7 +3139,7 @@ def ql(
QL computes the quantile of all sample values and the loss per time step.
For the true series :math:`y` and predicted stochastic/probabilistic series (containing N samples) :math:`\\hat{y}`
- of of shape :math:`T \\times N`, it is computed per column/component and time step :math:`t` as:
+ of of shape :math:`T \\times N`, it is computed per column/component, quantile and time step :math:`t` as:
.. math:: 2 \\max((q - 1) (y_t - \\hat{y}_{t,q}), q (y_t - \\hat{y}_{t,q})),
@@ -3120,23 +3184,25 @@ def ql(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- a sequence (list) of uni/multivariate series with `series_reduction`, `component_reduction` and
`time_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n time steps, n components) without time
- and component reductions. For:
+ A numpy array of metric scores. The array has shape (n time steps, n components * n quantiles) without time
+ and component reductions, and shape (n time steps, n quantiles) without time but component reduction and
+ `len(q) > 1`. For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- single uni/multivariate series and at least `time_reduction=None`.
- a sequence of uni/multivariate series including `series_reduction` and at least one of
`component_reduction=None` or `time_reduction=None`.
- List[float]
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
y_true, y_pred = _get_values_or_raise(
@@ -3175,7 +3241,7 @@ def mql(
time axis.
For the true series :math:`y` and predicted stochastic/probabilistic series (containing N samples) :math:`\\hat{y}`
- of of shape :math:`T \\times N`, it is computed per column/component as:
+ of of shape :math:`T \\times N`, it is computed per column/component and quantile as:
.. math:: 2 \\frac{1}{T}\\sum_{t=1}^T{\\max((q - 1) (y_t - \\hat{y}_{t,q}), q (y_t - \\hat{y}_{t,q}))},
@@ -3215,19 +3281,22 @@ def mql(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- - sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n components,) without component reduction. For:
+ A numpy array of metric scores. The array has shape (n components * n quantiles,) without component reduction,
+ and shape (n quantiles,) with component reduction and `len(q) > 1`.
+ For:
+ - the input from the `float` return case above but with `len(q) > 1`.
- single multivariate series and at least `component_reduction=None`.
- - sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
- List[float]
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
return np.nanmean(
@@ -3259,16 +3328,17 @@ def iw(
) -> METRIC_OUTPUT_TYPE:
"""Interval Width (IL).
- IL gives the width of predicted quantile intervals.
+ IL gives the length / width of predicted quantile intervals.
For the true series :math:`y` and predicted stochastic or quantile series :math:`\\hat{y}` of length :math:`T`,
- it is computed per component/column, quantile interval, and time step
+ it is computed per component/column, quantile interval :math:`(q_l,q_h)`, and time step
:math:`t` as:
- .. math:: \\hat{y}_{t,qh} - \\hat{y}_{t,ql}
+ .. math:: U_t - L_t,
- where :math:`\\hat{y}_{t,qh}` are the upper bound quantile values (of all predicted quantiles or samples) at time
- :math:`t`, and :math:`\\hat{y}_{t,ql}` are the lower bound quantile values.
+ where :math:`U_t` are the predicted upper bound quantile values :math:`\\hat{y}_{q_h,t}` (of all predicted
+ quantiles or samples) at time :math:`t`, and :math:`L_t` are the predicted lower bound quantile values
+ :math:`\\hat{y}_{q_l,t}`.
Parameters
----------
@@ -3310,23 +3380,25 @@ def iw(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q_interval) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- a sequence (list) of uni/multivariate series with `series_reduction`, `component_reduction` and
`time_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n time steps, n components) without time
- and component reductions. For:
+ A numpy array of metric scores. The array has shape (n time steps, n components * n q intervals) without time
+ and component reductions, and shape (n time steps, n q intervals) without time but component reduction and
+ `len(q_interval) > 1`. For:
+ - the input from the `float` return case above but with `len(q_interval) > 1`.
- single multivariate series and at least `component_reduction=None`.
- single uni/multivariate series and at least `time_reduction=None`.
- a sequence of uni/multivariate series including `series_reduction` and at least one of
`component_reduction=None` or `time_reduction=None`.
- List[float]
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
y_true, y_pred = _get_values_or_raise(
@@ -3355,18 +3427,19 @@ def miw(
n_jobs: int = 1,
verbose: bool = False,
) -> METRIC_OUTPUT_TYPE:
- """Mean Interval Width (IL).
+ """Mean Interval Width (MIL).
- IL gives the width of predicted quantile intervals aggregated over time.
+ MIL gives the time-aggregated length / width of predicted quantile intervals.
For the true series :math:`y` and predicted stochastic or quantile series :math:`\\hat{y}` of length :math:`T`,
- it is computed per component/column, quantile interval, and time step
+ it is computed per component/column, quantile interval :math:`(q_l,q_h)`, and time step
:math:`t` as:
- .. math:: \\frac{1}{T}\\sum_{t=1}^T{\\hat{y}_{t,qh} - \\hat{y}_{t,ql}}
+ .. math:: \\frac{1}{T}\\sum_{t=1}^T{U_t - L_t},
- where :math:`\\hat{y}_{t,qh}` are the upper bound quantile values (of all predicted quantiles or samples) at time
- :math:`t`, and :math:`\\hat{y}_{t,ql}` are the lower bound quantile values.
+ where :math:`U_t` are the predicted upper bound quantile values :math:`\\hat{y}_{q_h,t}` (of all predicted
+ quantiles or samples) at time :math:`t`, and :math:`L_t` are the predicted lower bound quantile values
+ :math:`\\hat{y}_{q_l,t}`.
Parameters
----------
@@ -3403,19 +3476,22 @@ def miw(
Returns
-------
float
- A single metric score for:
+ A single metric score for (with `len(q_interval) <= 1`):
- single univariate series.
- single multivariate series with `component_reduction`.
- - sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
np.ndarray
- A numpy array of metric scores. The array has shape (n components,) without component reduction. For:
+ A numpy array of metric scores. The array has shape (n components * n q intervals,) without component reduction,
+ and shape (n q intervals,) with component reduction and `len(q_interval) > 1`.
+ For:
+ - the input from the `float` return case above but with `len(q_interval) > 1`.
- single multivariate series and at least `component_reduction=None`.
- - sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
- List[float]
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
Same as for type `float` but for a sequence of series.
- List[np.ndarray]
+ list[np.ndarray]
Same as for type `np.ndarray` but for a sequence of series.
"""
return np.nanmean(
@@ -3428,3 +3504,633 @@ def miw(
),
axis=TIME_AX,
)
+
+
+@interval_support
+@multi_ts_support
+@multivariate_support
+def iws(
+ actual_series: Union[TimeSeries, Sequence[TimeSeries]],
+ pred_series: Union[TimeSeries, Sequence[TimeSeries]],
+ intersect: bool = True,
+ *,
+ q_interval: Union[tuple[float, float], Sequence[tuple[float, float]]] = None,
+ q: Optional[Union[float, list[float], tuple[np.ndarray, pd.Index]]] = None,
+ time_reduction: Optional[Callable[..., np.ndarray]] = None,
+ component_reduction: Optional[Callable[[np.ndarray], float]] = np.nanmean,
+ series_reduction: Optional[Callable[[np.ndarray], Union[float, np.ndarray]]] = None,
+ n_jobs: int = 1,
+ verbose: bool = False,
+) -> METRIC_OUTPUT_TYPE:
+ """Interval Winkler Score (IWS) [1]_.
+
+ IWS gives the length / width of the quantile intervals plus a penalty if the observation is outside the interval.
+
+ For the true series :math:`y` and predicted stochastic or quantile series :math:`\\hat{y}` of length :math:`T`,
+ it is computed per component/column, quantile interval :math:`(q_l,q_h)`, and time step :math:`t` as:
+
+ .. math::
+ \\begin{equation}
+ \\begin{cases}
+ (U_t - L_t) + \\frac{1}{q_l} (L_t - y_t) & \\text{if } y_t < L_t \\\\
+ (U_t - L_t) & \\text{if } L_t \\leq y_t \\leq U_t \\\\
+ (U_t - L_t) + \\frac{1}{1 - q_h} (y_t - U_t) & \\text{if } y_t > U_t
+ \\end{cases}
+ \\end{equation}
+
+ where :math:`U_t` are the predicted upper bound quantile values :math:`\\hat{y}_{q_h,t}` (of all predicted
+ quantiles or samples) at time :math:`t`, and :math:`L_t` are the predicted lower bound quantile values
+ :math:`\\hat{y}_{q_l,t}`.
+
+ Parameters
+ ----------
+ actual_series
+ The (sequence of) actual series.
+ pred_series
+ The (sequence of) predicted series.
+ intersect
+ For time series that are overlapping in time without having the same time index, setting `True`
+ will consider the values only over their common time interval (intersection in time).
+ q_interval
+ The quantile interval(s) to compute the metric on. Must be a tuple (single interval) or sequence tuples
+ (multiple intervals) with elements (low quantile, high quantile).
+ q
+ Quantiles `q` not supported by this metric; use `q_interval` instead.
+ component_reduction
+ Optionally, a function to aggregate the metrics over the component/column axis. It must reduce a `np.ndarray`
+ of shape `(t, c)` to a `np.ndarray` of shape `(t,)`. The function takes as input a ``np.ndarray`` and a
+ parameter named `axis`, and returns the reduced array. The `axis` receives value `1` corresponding to the
+ component axis. If `None`, will return a metric per component.
+ time_reduction
+ Optionally, a function to aggregate the metrics over the time axis. It must reduce a `np.ndarray`
+ of shape `(t, c)` to a `np.ndarray` of shape `(c,)`. The function takes as input a ``np.ndarray`` and a
+ parameter named `axis`, and returns the reduced array. The `axis` receives value `0` corresponding to the
+ time axis. If `None`, will return a metric per time step.
+ series_reduction
+ Optionally, a function to aggregate the metrics over multiple series. It must reduce a `np.ndarray`
+ of shape `(s, t, c)` to a `np.ndarray` of shape `(t, c)` The function takes as input a ``np.ndarray`` and a
+ parameter named `axis`, and returns the reduced array. The `axis` receives value `0` corresponding to the
+ series axis. For example with `np.nanmean`, will return the average over all series metrics. If `None`, will
+ return a metric per component.
+ n_jobs
+ The number of jobs to run in parallel. Parallel jobs are created only when a ``Sequence[TimeSeries]`` is
+ passed as input, parallelising operations regarding different ``TimeSeries``. Defaults to `1`
+ (sequential). Setting the parameter to `-1` means using all the available processors.
+ verbose
+ Optionally, whether to print operations progress
+
+ Returns
+ -------
+ float
+ A single metric score for (with `len(q_interval) <= 1`):
+
+ - single univariate series.
+ - single multivariate series with `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction`, `component_reduction` and
+ `time_reduction`.
+ np.ndarray
+ A numpy array of metric scores. The array has shape (n time steps, n components * n q intervals) without time
+ and component reductions, and shape (n time steps, n q intervals) without time but component reduction and
+ `len(q_interval) > 1`. For:
+
+ - the input from the `float` return case above but with `len(q_interval) > 1`.
+ - single multivariate series and at least `component_reduction=None`.
+ - single uni/multivariate series and at least `time_reduction=None`.
+ - a sequence of uni/multivariate series including `series_reduction` and at least one of
+ `component_reduction=None` or `time_reduction=None`.
+ list[float]
+ Same as for type `float` but for a sequence of series.
+ list[np.ndarray]
+ Same as for type `np.ndarray` but for a sequence of series.
+
+ References
+ ----------
+ .. [1] https://otexts.com/fpp3/distaccuracy.html
+ """
+ y_true, y_pred = _get_values_or_raise(
+ actual_series,
+ pred_series,
+ intersect,
+ remove_nan_union=True,
+ q=q,
+ )
+ y_pred_lo, y_pred_hi = _get_quantile_intervals(y_pred, q=q, q_interval=q_interval)
+ interval_width = y_pred_hi - y_pred_lo
+
+ # `c_alpha = 2 / alpha` corresponds to:
+ # - `1 / (1 - q_hi)` for the high quantile
+ # - `1 / q_lo` for the low quantile
+ c_alpha_hi = 1 / (1 - q_interval[:, 1])
+ c_alpha_lo = 1 / q_interval[:, 0]
+
+ score = np.where(
+ y_true < y_pred_lo,
+ interval_width + c_alpha_lo * (y_pred_lo - y_true),
+ np.where(
+ y_true > y_pred_hi,
+ interval_width + c_alpha_hi * (y_true - y_pred_hi),
+ interval_width,
+ ),
+ )
+ return score
+
+
+@interval_support
+@multi_ts_support
+@multivariate_support
+def miws(
+ actual_series: Union[TimeSeries, Sequence[TimeSeries]],
+ pred_series: Union[TimeSeries, Sequence[TimeSeries]],
+ intersect: bool = True,
+ *,
+ q_interval: Union[tuple[float, float], Sequence[tuple[float, float]]] = None,
+ q: Optional[Union[float, list[float], tuple[np.ndarray, pd.Index]]] = None,
+ component_reduction: Optional[Callable[[np.ndarray], float]] = np.nanmean,
+ series_reduction: Optional[Callable[[np.ndarray], Union[float, np.ndarray]]] = None,
+ n_jobs: int = 1,
+ verbose: bool = False,
+) -> METRIC_OUTPUT_TYPE:
+ """Mean Interval Winkler Score (IWS) [1]_.
+
+ MIWS gives the time-aggregated length / width of the quantile intervals plus a penalty if the observation is
+ outside the interval.
+
+ For the true series :math:`y` and predicted stochastic or quantile series :math:`\\hat{y}` of length :math:`T`,
+ it is computed per component/column, quantile interval :math:`(q_l,q_h)`, and time step :math:`t` as:
+
+ .. math:: \\frac{1}{T}\\sum_{t=1}^T{W_t(y_t, \\hat{y}_{t}, q_h, q_l)},
+
+ where :math:`W` is the Winkler Score :func:`~darts.metrics.metrics.iws`.
+
+ Parameters
+ ----------
+ actual_series
+ The (sequence of) actual series.
+ pred_series
+ The (sequence of) predicted series.
+ intersect
+ For time series that are overlapping in time without having the same time index, setting `True`
+ will consider the values only over their common time interval (intersection in time).
+ q_interval
+ The quantile interval(s) to compute the metric on. Must be a tuple (single interval) or sequence tuples
+ (multiple intervals) with elements (low quantile, high quantile).
+ q
+ Quantiles `q` not supported by this metric; use `q_interval` instead.
+ component_reduction
+ Optionally, a function to aggregate the metrics over the component/column axis. It must reduce a `np.ndarray`
+ of shape `(t, c)` to a `np.ndarray` of shape `(t,)`. The function takes as input a ``np.ndarray`` and a
+ parameter named `axis`, and returns the reduced array. The `axis` receives value `1` corresponding to the
+ component axis. If `None`, will return a metric per component.
+ series_reduction
+ Optionally, a function to aggregate the metrics over multiple series. It must reduce a `np.ndarray`
+ of shape `(s, t, c)` to a `np.ndarray` of shape `(t, c)` The function takes as input a ``np.ndarray`` and a
+ parameter named `axis`, and returns the reduced array. The `axis` receives value `0` corresponding to the
+ series axis. For example with `np.nanmean`, will return the average over all series metrics. If `None`, will
+ return a metric per component.
+ n_jobs
+ The number of jobs to run in parallel. Parallel jobs are created only when a ``Sequence[TimeSeries]`` is
+ passed as input, parallelising operations regarding different ``TimeSeries``. Defaults to `1`
+ (sequential). Setting the parameter to `-1` means using all the available processors.
+ verbose
+ Optionally, whether to print operations progress
+
+ Returns
+ -------
+ float
+ A single metric score for (with `len(q_interval) <= 1`):
+
+ - single univariate series.
+ - single multivariate series with `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ np.ndarray
+ A numpy array of metric scores. The array has shape (n components * n q intervals,) without component reduction,
+ and shape (n q intervals,) with component reduction and `len(q_interval) > 1`.
+ For:
+
+ - the input from the `float` return case above but with `len(q_interval) > 1`.
+ - single multivariate series and at least `component_reduction=None`.
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
+ Same as for type `float` but for a sequence of series.
+ list[np.ndarray]
+ Same as for type `np.ndarray` but for a sequence of series.
+
+ References
+ ----------
+ .. [1] https://otexts.com/fpp3/distaccuracy.html
+ """
+ return np.nanmean(
+ _get_wrapped_metric(iws, n_wrappers=3)(
+ actual_series,
+ pred_series,
+ intersect,
+ q=q,
+ q_interval=q_interval,
+ ),
+ axis=TIME_AX,
+ )
+
+
+@interval_support
+@multi_ts_support
+@multivariate_support
+def ic(
+ actual_series: Union[TimeSeries, Sequence[TimeSeries]],
+ pred_series: Union[TimeSeries, Sequence[TimeSeries]],
+ intersect: bool = True,
+ *,
+ q_interval: Union[tuple[float, float], Sequence[tuple[float, float]]] = None,
+ q: Optional[Union[float, list[float], tuple[np.ndarray, pd.Index]]] = None,
+ time_reduction: Optional[Callable[..., np.ndarray]] = None,
+ component_reduction: Optional[Callable[[np.ndarray], float]] = np.nanmean,
+ series_reduction: Optional[Callable[[np.ndarray], Union[float, np.ndarray]]] = None,
+ n_jobs: int = 1,
+ verbose: bool = False,
+) -> METRIC_OUTPUT_TYPE:
+ """Interval Coverage (IC).
+
+ IC gives a binary outcome with `1` if the observation is within the interval, and `0` otherwise.
+
+ For the true series :math:`y` and predicted stochastic or quantile series :math:`\\hat{y}` of length :math:`T`,
+ it is computed per component/column, quantile interval :math:`(q_l,q_h)`, and time step :math:`t` as:
+
+ .. math::
+ \\begin{equation}
+ \\begin{cases}
+ 1 & \\text{if } L_t < y_t < U_t \\\\
+ 0 & \\text{otherwise}
+ \\end{cases}
+ \\end{equation}
+
+ where :math:`U_t` are the predicted upper bound quantile values :math:`\\hat{y}_{q_h,t}` (of all predicted
+ quantiles or samples) at time :math:`t`, and :math:`L_t` are the predicted lower bound quantile values
+ :math:`\\hat{y}_{q_l,t}`.
+
+ Parameters
+ ----------
+ actual_series
+ The (sequence of) actual series.
+ pred_series
+ The (sequence of) predicted series.
+ intersect
+ For time series that are overlapping in time without having the same time index, setting `True`
+ will consider the values only over their common time interval (intersection in time).
+ q_interval
+ The quantile interval(s) to compute the metric on. Must be a tuple (single interval) or sequence tuples
+ (multiple intervals) with elements (low quantile, high quantile).
+ q
+ Quantiles `q` not supported by this metric; use `q_interval` instead.
+ component_reduction
+ Optionally, a function to aggregate the metrics over the component/column axis. It must reduce a `np.ndarray`
+ of shape `(t, c)` to a `np.ndarray` of shape `(t,)`. The function takes as input a ``np.ndarray`` and a
+ parameter named `axis`, and returns the reduced array. The `axis` receives value `1` corresponding to the
+ component axis. If `None`, will return a metric per component.
+ time_reduction
+ Optionally, a function to aggregate the metrics over the time axis. It must reduce a `np.ndarray`
+ of shape `(t, c)` to a `np.ndarray` of shape `(c,)`. The function takes as input a ``np.ndarray`` and a
+ parameter named `axis`, and returns the reduced array. The `axis` receives value `0` corresponding to the
+ time axis. If `None`, will return a metric per time step.
+ series_reduction
+ Optionally, a function to aggregate the metrics over multiple series. It must reduce a `np.ndarray`
+ of shape `(s, t, c)` to a `np.ndarray` of shape `(t, c)` The function takes as input a ``np.ndarray`` and a
+ parameter named `axis`, and returns the reduced array. The `axis` receives value `0` corresponding to the
+ series axis. For example with `np.nanmean`, will return the average over all series metrics. If `None`, will
+ return a metric per component.
+ n_jobs
+ The number of jobs to run in parallel. Parallel jobs are created only when a ``Sequence[TimeSeries]`` is
+ passed as input, parallelising operations regarding different ``TimeSeries``. Defaults to `1`
+ (sequential). Setting the parameter to `-1` means using all the available processors.
+ verbose
+ Optionally, whether to print operations progress
+
+ Returns
+ -------
+ float
+ A single metric score for (with `len(q_interval) <= 1`):
+
+ - single univariate series.
+ - single multivariate series with `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction`, `component_reduction` and
+ `time_reduction`.
+ np.ndarray
+ A numpy array of metric scores. The array has shape (n time steps, n components * n q intervals) without time
+ and component reductions, and shape (n time steps, n q intervals) without time but component reduction and
+ `len(q_interval) > 1`. For:
+
+ - the input from the `float` return case above but with `len(q_interval) > 1`.
+ - single multivariate series and at least `component_reduction=None`.
+ - single uni/multivariate series and at least `time_reduction=None`.
+ - a sequence of uni/multivariate series including `series_reduction` and at least one of
+ `component_reduction=None` or `time_reduction=None`.
+ list[float]
+ Same as for type `float` but for a sequence of series.
+ list[np.ndarray]
+ Same as for type `np.ndarray` but for a sequence of series.
+ """
+ y_true, y_pred = _get_values_or_raise(
+ actual_series,
+ pred_series,
+ intersect,
+ remove_nan_union=True,
+ q=q,
+ )
+ y_pred_lo, y_pred_hi = _get_quantile_intervals(y_pred, q=q, q_interval=q_interval)
+ return np.where((y_pred_lo <= y_true) & (y_true <= y_pred_hi), 1.0, 0.0)
+
+
+@interval_support
+@multi_ts_support
+@multivariate_support
+def mic(
+ actual_series: Union[TimeSeries, Sequence[TimeSeries]],
+ pred_series: Union[TimeSeries, Sequence[TimeSeries]],
+ intersect: bool = True,
+ *,
+ q_interval: Union[tuple[float, float], Sequence[tuple[float, float]]] = None,
+ q: Optional[Union[float, list[float], tuple[np.ndarray, pd.Index]]] = None,
+ component_reduction: Optional[Callable[[np.ndarray], float]] = np.nanmean,
+ series_reduction: Optional[Callable[[np.ndarray], Union[float, np.ndarray]]] = None,
+ n_jobs: int = 1,
+ verbose: bool = False,
+) -> METRIC_OUTPUT_TYPE:
+ """Mean Interval Coverage (MIC).
+
+ MIC gives the time-aggregated Interval Coverage :func:`~darts.metrics.metrics.ic` - the ratio of observations
+ being within the interval.
+
+ For the true series :math:`y` and predicted stochastic or quantile series :math:`\\hat{y}` of length :math:`T`,
+ it is computed per component/column, quantile interval :math:`(q_l,q_h)`, and time step :math:`t` as:
+
+ .. math:: \\frac{1}{T}\\sum_{t=1}^T{C(y_t, \\hat{y}_{t}, q_h, q_l)},
+
+ where :math:`C` is the Interval Coverage :func:`~darts.metrics.metrics.ic`.
+
+ Parameters
+ ----------
+ actual_series
+ The (sequence of) actual series.
+ pred_series
+ The (sequence of) predicted series.
+ intersect
+ For time series that are overlapping in time without having the same time index, setting `True`
+ will consider the values only over their common time interval (intersection in time).
+ q_interval
+ The quantile interval(s) to compute the metric on. Must be a tuple (single interval) or sequence tuples
+ (multiple intervals) with elements (low quantile, high quantile).
+ q
+ Quantiles `q` not supported by this metric; use `q_interval` instead.
+ component_reduction
+ Optionally, a function to aggregate the metrics over the component/column axis. It must reduce a `np.ndarray`
+ of shape `(t, c)` to a `np.ndarray` of shape `(t,)`. The function takes as input a ``np.ndarray`` and a
+ parameter named `axis`, and returns the reduced array. The `axis` receives value `1` corresponding to the
+ component axis. If `None`, will return a metric per component.
+ series_reduction
+ Optionally, a function to aggregate the metrics over multiple series. It must reduce a `np.ndarray`
+ of shape `(s, t, c)` to a `np.ndarray` of shape `(t, c)` The function takes as input a ``np.ndarray`` and a
+ parameter named `axis`, and returns the reduced array. The `axis` receives value `0` corresponding to the
+ series axis. For example with `np.nanmean`, will return the average over all series metrics. If `None`, will
+ return a metric per component.
+ n_jobs
+ The number of jobs to run in parallel. Parallel jobs are created only when a ``Sequence[TimeSeries]`` is
+ passed as input, parallelising operations regarding different ``TimeSeries``. Defaults to `1`
+ (sequential). Setting the parameter to `-1` means using all the available processors.
+ verbose
+ Optionally, whether to print operations progress
+
+ Returns
+ -------
+ float
+ A single metric score for (with `len(q_interval) <= 1`):
+
+ - single univariate series.
+ - single multivariate series with `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ np.ndarray
+ A numpy array of metric scores. The array has shape (n components * n q intervals,) without component reduction,
+ and shape (n q intervals,) with component reduction and `len(q_interval) > 1`.
+ For:
+
+ - the input from the `float` return case above but with `len(q_interval) > 1`.
+ - single multivariate series and at least `component_reduction=None`.
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
+ Same as for type `float` but for a sequence of series.
+ list[np.ndarray]
+ Same as for type `np.ndarray` but for a sequence of series.
+ """
+ return np.nanmean(
+ _get_wrapped_metric(ic, n_wrappers=3)(
+ actual_series,
+ pred_series,
+ intersect,
+ q=q,
+ q_interval=q_interval,
+ ),
+ axis=TIME_AX,
+ )
+
+
+@interval_support
+@multi_ts_support
+@multivariate_support
+def incs_qr(
+ actual_series: Union[TimeSeries, Sequence[TimeSeries]],
+ pred_series: Union[TimeSeries, Sequence[TimeSeries]],
+ intersect: bool = True,
+ *,
+ q_interval: Union[tuple[float, float], Sequence[tuple[float, float]]] = None,
+ symmetric: bool = True,
+ q: Optional[Union[float, list[float], tuple[np.ndarray, pd.Index]]] = None,
+ time_reduction: Optional[Callable[..., np.ndarray]] = None,
+ component_reduction: Optional[Callable[[np.ndarray], float]] = np.nanmean,
+ series_reduction: Optional[Callable[[np.ndarray], Union[float, np.ndarray]]] = None,
+ n_jobs: int = 1,
+ verbose: bool = False,
+) -> METRIC_OUTPUT_TYPE:
+ """Interval Non-Conformity Score for Quantile Regression (INCS_QR).
+
+ INCS_QR gives the absolute error to the closest predicted quantile interval bound when the observation is outside
+ the interval. Otherwise, it gives the negative absolute error to the closer bound.
+
+ For the true series :math:`y` and predicted stochastic or quantile series :math:`\\hat{y}` of length :math:`T`,
+ it is computed per component/column, quantile interval :math:`(q_l,q_h)`, and time step :math:`t` as:
+
+ .. math:: \\max(L_t - y_t, y_t - U_t)
+
+ where :math:`U_t` are the predicted upper bound quantile values :math:`\\hat{y}_{q_h,t}` (of all predicted
+ quantiles or samples) at time :math:`t`, and :math:`L_t` are the predicted lower bound quantile values
+ :math:`\\hat{y}_{q_l,t}`.
+
+ Parameters
+ ----------
+ actual_series
+ The (sequence of) actual series.
+ pred_series
+ The (sequence of) predicted series.
+ intersect
+ For time series that are overlapping in time without having the same time index, setting `True`
+ will consider the values only over their common time interval (intersection in time).
+ q_interval
+ The quantile interval(s) to compute the metric on. Must be a tuple (single interval) or sequence tuples
+ (multiple intervals) with elements (low quantile, high quantile).
+ symmetric
+ Whether to return symmetric non-conformity scores. If `False`, returns asymmetric scores (individual scores
+ for lower- and upper quantile interval bounds; returned in the component axis).
+ q
+ Quantiles `q` not supported by this metric; use `q_interval` instead.
+ component_reduction
+ Optionally, a function to aggregate the metrics over the component/column axis. It must reduce a `np.ndarray`
+ of shape `(t, c)` to a `np.ndarray` of shape `(t,)`. The function takes as input a ``np.ndarray`` and a
+ parameter named `axis`, and returns the reduced array. The `axis` receives value `1` corresponding to the
+ component axis. If `None`, will return a metric per component.
+ time_reduction
+ Optionally, a function to aggregate the metrics over the time axis. It must reduce a `np.ndarray`
+ of shape `(t, c)` to a `np.ndarray` of shape `(c,)`. The function takes as input a ``np.ndarray`` and a
+ parameter named `axis`, and returns the reduced array. The `axis` receives value `0` corresponding to the
+ time axis. If `None`, will return a metric per time step.
+ series_reduction
+ Optionally, a function to aggregate the metrics over multiple series. It must reduce a `np.ndarray`
+ of shape `(s, t, c)` to a `np.ndarray` of shape `(t, c)` The function takes as input a ``np.ndarray`` and a
+ parameter named `axis`, and returns the reduced array. The `axis` receives value `0` corresponding to the
+ series axis. For example with `np.nanmean`, will return the average over all series metrics. If `None`, will
+ return a metric per component.
+ n_jobs
+ The number of jobs to run in parallel. Parallel jobs are created only when a ``Sequence[TimeSeries]`` is
+ passed as input, parallelising operations regarding different ``TimeSeries``. Defaults to `1`
+ (sequential). Setting the parameter to `-1` means using all the available processors.
+ verbose
+ Optionally, whether to print operations progress
+
+ Returns
+ -------
+ float
+ A single metric score for (with `len(q_interval) <= 1`):
+
+ - single univariate series.
+ - single multivariate series with `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction`, `component_reduction` and
+ `time_reduction`.
+ np.ndarray
+ A numpy array of metric scores. The array has shape (n time steps, n components * n q intervals) without time
+ and component reductions, and shape (n time steps, n q intervals) without time but component reduction and
+ `len(q_interval) > 1`. For:
+
+ - the input from the `float` return case above but with `len(q_interval) > 1`.
+ - single multivariate series and at least `component_reduction=None`.
+ - single uni/multivariate series and at least `time_reduction=None`.
+ - a sequence of uni/multivariate series including `series_reduction` and at least one of
+ `component_reduction=None` or `time_reduction=None`.
+ list[float]
+ Same as for type `float` but for a sequence of series.
+ list[np.ndarray]
+ Same as for type `np.ndarray` but for a sequence of series.
+ """
+ y_true, y_pred = _get_values_or_raise(
+ actual_series,
+ pred_series,
+ intersect,
+ remove_nan_union=True,
+ q=q,
+ )
+ y_pred_lo, y_pred_hi = _get_quantile_intervals(y_pred, q=q, q_interval=q_interval)
+ if symmetric:
+ return np.maximum(y_pred_lo - y_true, y_true - y_pred_hi)
+ else:
+ return np.concatenate([y_pred_lo - y_true, y_true - y_pred_hi], axis=SMPL_AX)
+
+
+@interval_support
+@multi_ts_support
+@multivariate_support
+def mincs_qr(
+ actual_series: Union[TimeSeries, Sequence[TimeSeries]],
+ pred_series: Union[TimeSeries, Sequence[TimeSeries]],
+ intersect: bool = True,
+ *,
+ q_interval: Union[tuple[float, float], Sequence[tuple[float, float]]] = None,
+ symmetric: bool = True,
+ q: Optional[Union[float, list[float], tuple[np.ndarray, pd.Index]]] = None,
+ component_reduction: Optional[Callable[[np.ndarray], float]] = np.nanmean,
+ series_reduction: Optional[Callable[[np.ndarray], Union[float, np.ndarray]]] = None,
+ n_jobs: int = 1,
+ verbose: bool = False,
+) -> METRIC_OUTPUT_TYPE:
+ """Mean Interval Non-Conformity Score for Quantile Regression (MINCS_QR).
+
+ MINCS_QR gives the time-aggregated INCS_QR :func:`~darts.metrics.metrics.incs_qr`.
+
+ For the true series :math:`y` and predicted stochastic or quantile series :math:`\\hat{y}` of length :math:`T`,
+ it is computed per component/column, quantile interval :math:`(q_l,q_h)`, and time step :math:`t` as:
+
+ .. math:: \\frac{1}{T}\\sum_{t=1}^T{INCS_QR(y_t, \\hat{y}_{t}, q_h, q_l)},
+
+ where :math:`INCS_QR` is the Interval Non-Conformity Score for Quantile Regression
+ :func:`~darts.metrics.metrics.incs_qr`.
+
+ Parameters
+ ----------
+ actual_series
+ The (sequence of) actual series.
+ pred_series
+ The (sequence of) predicted series.
+ intersect
+ For time series that are overlapping in time without having the same time index, setting `True`
+ will consider the values only over their common time interval (intersection in time).
+ q_interval
+ The quantile interval(s) to compute the metric on. Must be a tuple (single interval) or sequence tuples
+ (multiple intervals) with elements (low quantile, high quantile).
+ symmetric
+ Whether to return symmetric non-conformity scores. If `False`, returns asymmetric scores (individual scores
+ for lower- and upper quantile interval bounds; returned in the component axis).
+ q
+ Quantiles `q` not supported by this metric; use `q_interval` instead.
+ component_reduction
+ Optionally, a function to aggregate the metrics over the component/column axis. It must reduce a `np.ndarray`
+ of shape `(t, c)` to a `np.ndarray` of shape `(t,)`. The function takes as input a ``np.ndarray`` and a
+ parameter named `axis`, and returns the reduced array. The `axis` receives value `1` corresponding to the
+ component axis. If `None`, will return a metric per component.
+ series_reduction
+ Optionally, a function to aggregate the metrics over multiple series. It must reduce a `np.ndarray`
+ of shape `(s, t, c)` to a `np.ndarray` of shape `(t, c)` The function takes as input a ``np.ndarray`` and a
+ parameter named `axis`, and returns the reduced array. The `axis` receives value `0` corresponding to the
+ series axis. For example with `np.nanmean`, will return the average over all series metrics. If `None`, will
+ return a metric per component.
+ n_jobs
+ The number of jobs to run in parallel. Parallel jobs are created only when a ``Sequence[TimeSeries]`` is
+ passed as input, parallelising operations regarding different ``TimeSeries``. Defaults to `1`
+ (sequential). Setting the parameter to `-1` means using all the available processors.
+ verbose
+ Optionally, whether to print operations progress
+
+ Returns
+ -------
+ float
+ A single metric score for (with `len(q_interval) <= 1`):
+
+ - single univariate series.
+ - single multivariate series with `component_reduction`.
+ - a sequence (list) of uni/multivariate series with `series_reduction` and `component_reduction`.
+ np.ndarray
+ A numpy array of metric scores. The array has shape (n components * n q intervals,) without component reduction,
+ and shape (n q intervals,) with component reduction and `len(q_interval) > 1`.
+ For:
+
+ - the input from the `float` return case above but with `len(q_interval) > 1`.
+ - single multivariate series and at least `component_reduction=None`.
+ - a sequence of uni/multivariate series including `series_reduction` and `component_reduction=None`.
+ list[float]
+ Same as for type `float` but for a sequence of series.
+ list[np.ndarray]
+ Same as for type `np.ndarray` but for a sequence of series.
+ """
+ return np.nanmean(
+ _get_wrapped_metric(incs_qr, n_wrappers=3)(
+ actual_series,
+ pred_series,
+ intersect,
+ q=q,
+ q_interval=q_interval,
+ symmetric=symmetric,
+ ),
+ axis=TIME_AX,
+ )
diff --git a/darts/models/__init__.py b/darts/models/__init__.py
index 17640b195d..f4218c4ea8 100644
--- a/darts/models/__init__.py
+++ b/darts/models/__init__.py
@@ -108,14 +108,18 @@
except ImportError:
XGBModel = NotImportedModule(module_name="XGBoost")
+# Conformal Prediction
+# Filtering
from darts.models.filtering.gaussian_process_filter import GaussianProcessFilter
from darts.models.filtering.kalman_filter import KalmanFilter
-
-# Filtering
from darts.models.filtering.moving_average_filter import MovingAverageFilter
-from darts.models.forecasting.baselines import NaiveEnsembleModel
# Ensembling
+from darts.models.forecasting.baselines import NaiveEnsembleModel
+from darts.models.forecasting.conformal_models import (
+ ConformalNaiveModel,
+ ConformalQRModel,
+)
from darts.models.forecasting.ensemble_model import EnsembleModel
__all__ = [
@@ -140,7 +144,7 @@
"VARIMA",
"BlockRNNModel",
"DLinearModel",
- "GlobalNaiveDrift",
+ "GlobalNaiveAggregate",
"GlobalNaiveDrift",
"GlobalNaiveSeasonal",
"NBEATSModel",
@@ -165,4 +169,6 @@
"MovingAverageFilter",
"NaiveEnsembleModel",
"EnsembleModel",
+ "ConformalNaiveModel",
+ "ConformalQRModel",
]
diff --git a/darts/models/forecasting/__init__.py b/darts/models/forecasting/__init__.py
index 37a50aa4bc..b3559f9b62 100644
--- a/darts/models/forecasting/__init__.py
+++ b/darts/models/forecasting/__init__.py
@@ -50,4 +50,7 @@
Ensemble Models (`GlobalForecastingModel `_)
- :class:`~darts.models.forecasting.baselines.NaiveEnsembleModel`
- :class:`~darts.models.forecasting.regression_ensemble_model.RegressionEnsembleModel`
+Conformal Models (`GlobalForecastingModel `_)
+ - :class:`~darts.models.forecasting.conformal_models.ConformalNaiveModel`
+ - :class:`~darts.models.forecasting.conformal_models.ConformalQRModel`
"""
diff --git a/darts/models/forecasting/conformal_models.py b/darts/models/forecasting/conformal_models.py
new file mode 100644
index 0000000000..33ffb766d2
--- /dev/null
+++ b/darts/models/forecasting/conformal_models.py
@@ -0,0 +1,1880 @@
+"""
+Conformal Models
+---------------
+
+A collection of conformal prediction models for pre-trained global forecasting models.
+"""
+
+import copy
+import os
+from abc import ABC, abstractmethod
+from collections.abc import Sequence
+from typing import Any, BinaryIO, Callable, Optional, Union
+
+try:
+ from typing import Literal
+except ImportError:
+ from typing_extensions import Literal
+
+import numpy as np
+import pandas as pd
+
+from darts import TimeSeries, metrics
+from darts.logging import get_logger, raise_log
+from darts.metrics.metrics import METRIC_TYPE
+from darts.models.forecasting.forecasting_model import GlobalForecastingModel
+from darts.models.utils import TORCH_AVAILABLE
+from darts.utils import _build_tqdm_iterator, _with_sanity_checks
+from darts.utils.historical_forecasts.utils import (
+ _adjust_historical_forecasts_time_index,
+)
+from darts.utils.timeseries_generation import _build_forecast_series
+from darts.utils.ts_utils import (
+ SeriesType,
+ get_series_seq_type,
+ series2seq,
+)
+from darts.utils.utils import (
+ _check_quantiles,
+ generate_index,
+ likelihood_component_names,
+ n_steps_between,
+ quantile_names,
+ random_method,
+ sample_from_quantiles,
+)
+
+if TORCH_AVAILABLE:
+ from darts.models.forecasting.torch_forecasting_model import TorchForecastingModel
+else:
+ TorchForecastingModel = None
+
+logger = get_logger(__name__)
+
+
+class ConformalModel(GlobalForecastingModel, ABC):
+ @random_method
+ def __init__(
+ self,
+ model: GlobalForecastingModel,
+ quantiles: list[float],
+ symmetric: bool = True,
+ cal_length: Optional[int] = None,
+ num_samples: int = 500,
+ random_state: Optional[int] = None,
+ stride_cal: bool = False,
+ ):
+ """Base Conformal Prediction Model.
+
+ Base class for any probabilistic conformal model. A conformal model calibrates the predictions from any
+ pre-trained global forecasting model. It does not have to be trained, and can generated calibrated forecasts
+ directly using the underlying trained forecasting model. Since it is a probabilistic model, you can generate
+ forecasts in two ways (when calling `predict()`, `historical_forecasts()`, ...):
+
+ - Predict the calibrated quantile intervals directly: Pass parameters `predict_likelihood_parameters=True`, and
+ `num_samples=1` to the forecast method.
+ - Predict stochastic samples from the calibrated quantile intervals: Pass parameters
+ `predict_likelihood_parameters=False`, and `num_samples>>1` to the forecast method.
+
+ Conformal models can be applied to any of Darts' global forecasting model, as long as the model has been
+ fitted before. In general the workflow of the models to produce one calibrated forecast/prediction is as
+ follows:
+
+ - Extract a calibration set: The number of calibration examples from the most recent past to use for one
+ conformal prediction can be defined at model creation with parameter `cal_length`. If `stride_cal` is `True`,
+ then the same `stride` from the forecasting methods is applied to the calibration set, and more calibration
+ examples are required (`cal_length * stride` historical forecasts that were generated with `stride=1`).
+ To make your life simpler, we support two modes:
+ - Automatic extraction of the calibration set from the past of your input series (`series`,
+ `past_covariates`, ...). This is the default mode and our predict/forecasting/backtest/.... API is
+ identical to any other forecasting model
+ - Supply a fixed calibration set with parameters `cal_series`, `cal_past_covariates`, ... .
+ - Generate historical forecasts on the calibration set (using the forecasting model)
+ - Compute the errors/non-conformity scores (specific to each conformal model) on these historical forecasts
+ - Compute the quantile values from the errors / non-conformity scores (using our desired quantiles set at model
+ creation with parameter `quantiles`).
+ - Compute the conformal prediction: Add the calibrated intervals to (or adjust the existing intervals of) the
+ forecasting model's predictions.
+
+ Some notes:
+
+ - When computing historical_forecasts(), backtest(), residuals(), ... the above is applied for each forecast
+ (the forecasting model's historical forecasts are only generated once for efficiency).
+ - For multi-horizon forecasts, the above is applied for each step in the horizon separately
+
+ Parameters
+ ----------
+ model
+ A pre-trained global forecasting model.
+ quantiles
+ A list of quantiles centered around the median `q=0.5` to use. For example quantiles
+ [0.1, 0.2, 0.5, 0.8 0.9] correspond to two intervals with (0.9 - 0.1) = 80%, and (0.8 - 0.2) 60% coverage
+ around the median (model forecast).
+ symmetric
+ Whether to use symmetric non-conformity scores. If `False`, uses asymmetric scores (individual scores
+ for lower- and upper quantile interval bounds).
+ cal_length
+ The number of past forecast residuals/errors to consider as calibration input for each conformal forecast.
+ If `None`, considers all past residuals.
+ num_samples
+ Number of times a prediction is sampled from the underlying `model` if it is probabilistic. Uses `1` for
+ deterministic models. This is different to the `num_samples` produced by the conformal model which can be
+ set in downstream forecasting tasks.
+ random_state
+ Control the randomness of probabilistic conformal forecasts (sample generation) across different runs.
+ stride_cal
+ Whether to apply the same historical forecast `stride` to the non-conformity scores of the calibration set.
+ """
+ if not isinstance(model, GlobalForecastingModel) or not model._fit_called:
+ raise_log(
+ ValueError("`model` must be a pre-trained `GlobalForecastingModel`."),
+ logger=logger,
+ )
+ _check_quantiles(quantiles)
+ super().__init__(add_encoders=None)
+
+ # quantiles and interval setup
+ self.quantiles = np.array(quantiles)
+ self.idx_median = quantiles.index(0.5)
+ self.q_interval = [
+ (q_l, q_h)
+ for q_l, q_h in zip(
+ quantiles[: self.idx_median], quantiles[self.idx_median + 1 :][::-1]
+ )
+ ]
+ self.interval_range = np.array([
+ q_high - q_low
+ for q_high, q_low in zip(
+ self.quantiles[self.idx_median + 1 :][::-1],
+ self.quantiles[: self.idx_median],
+ )
+ ])
+ if symmetric:
+ # symmetric considers both tails together
+ self.interval_range_sym = copy.deepcopy(self.interval_range)
+ else:
+ # asymmetric considers tails separately
+ self.interval_range_sym = 1 - (1 - self.interval_range) / 2
+ self.symmetric = symmetric
+
+ # model setup
+ self.model = model
+ self.cal_length = cal_length
+ self.stride_cal = stride_cal
+ self.num_samples = num_samples if model.supports_probabilistic_prediction else 1
+ self._likelihood = "quantile"
+ self._fit_called = True
+
+ def fit(
+ self,
+ series: Union[TimeSeries, Sequence[TimeSeries]],
+ past_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ future_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ **kwargs,
+ ) -> "ConformalModel":
+ """Fit/train the underlying forecasting model on (potentially multiple) series.
+
+ Optionally, one or multiple past and/or future covariates series can be provided as well, depending on the
+ forecasting model used. The number of covariates series must match the number of target series.
+
+ Notes
+ -----
+ Conformal Models do not required calling `fit()`, since they use pre-trained global forecasting models.
+ You can call `predict()` directly. Also, make sure that the input series used in `predict()` corresponds to
+ a calibration set, and not the same as used during training with `fit()`.
+
+ Parameters
+ ----------
+ series
+ One or several target time series. The model will be trained to forecast these time series.
+ The series may or may not be multivariate, but if multiple series are provided
+ they must have the same number of components.
+ past_covariates
+ One or several past-observed covariate time series. These time series will not be forecast, but can
+ be used by some models as an input. The covariate(s) may or may not be multivariate, but if multiple
+ covariates are provided they must have the same number of components. If `past_covariates` is provided,
+ it must contain the same number of series as `series`.
+ future_covariates
+ One or several future-known covariate time series. These time series will not be forecast, but can
+ be used by some models as an input. The covariate(s) may or may not be multivariate, but if multiple
+ covariates are provided they must have the same number of components. If `future_covariates` is provided,
+ it must contain the same number of series as `series`.
+
+ Returns
+ -------
+ self
+ Fitted model.
+ """
+ # does not have to be trained, but we allow it for unified API
+ self.model.fit(
+ series=series,
+ past_covariates=past_covariates,
+ future_covariates=future_covariates,
+ **kwargs,
+ )
+ return self
+
+ def predict(
+ self,
+ n: int,
+ series: Union[TimeSeries, Sequence[TimeSeries]] = None,
+ past_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ future_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ cal_series: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ cal_past_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ cal_future_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ stride: int = 1,
+ num_samples: int = 1,
+ verbose: bool = False,
+ predict_likelihood_parameters: bool = False,
+ show_warnings: bool = True,
+ ) -> Union[TimeSeries, Sequence[TimeSeries]]:
+ """Forecasts calibrated quantile intervals (or samples from calibrated intervals) for `n` time steps after the
+ end of the `series`.
+
+ It is important that the input series for prediction correspond to a calibration set - a set different to the
+ series that the underlying forecasting `model` was trained one.
+
+ Since it is a probabilistic model, you can generate forecasts in two ways:
+
+ - Predict the calibrated quantile intervals directly: Pass parameters `predict_likelihood_parameters=True`, and
+ `num_samples=1` to the forecast method.
+ - Predict stochastic samples from the calibrated quantile intervals: Pass parameters
+ `predict_likelihood_parameters=False`, and `num_samples>>1` to the forecast method.
+
+ Under the hood, the simplified workflow to produce one calibrated forecast/prediction for every step in the
+ horizon `n` is as follows:
+
+ - Extract a calibration set: The number of calibration examples from the most recent past to use for one
+ conformal prediction can be defined at model creation with parameter `cal_length`. To make your life simpler,
+ we support two modes:
+ - Automatic extraction of the calibration set from the past of your input series (`series`,
+ `past_covariates`, ...). This is the default mode.
+ - Supply a fixed calibration set with parameters `cal_series`, `cal_past_covariates`, ... .
+ - Generate historical forecasts on the calibration set (using the forecasting model)
+ - Compute the errors/non-conformity scores (specific to each conformal model) on these historical forecasts
+ - Compute the quantile values from the errors / non-conformity scores (using our desired quantiles set at model
+ creation with parameter `quantiles`).
+ - Compute the conformal prediction: Add the calibrated intervals to (or adjust the existing intervals of) the
+ forecasting model's predictions.
+
+ Parameters
+ ----------
+ n
+ Forecast horizon - the number of time steps after the end of the series for which to produce predictions.
+ series
+ A series or sequence of series, representing the history of the target series whose future is to be
+ predicted. If `cal_series` is `None`, will use the past of this series for calibration.
+ past_covariates
+ Optionally, a (sequence of) past-observed covariate time series for every input time series in `series`.
+ Their dimension must match that of the past covariates used for training. If `cal_series` is `None`, will
+ use this series for calibration.
+ future_covariates
+ Optionally, a (sequence of) future-known covariate time series for every input time series in `series`.
+ Their dimension must match that of the past covariates used for training. If `cal_series` is `None`, will
+ use this series for calibration.
+ cal_series
+ Optionally, a (sequence of) target series for every input time series in `series` to use for calibration
+ instead of `series`.
+ cal_past_covariates
+ Optionally, a (sequence of) past covariates series for every input time series in `series` to use for
+ calibration instead of `past_covariates`.
+ cal_future_covariates
+ Optionally, a future covariates series for every input time series in `series` to use for calibration
+ instead of `future_covariates`.
+ stride
+ The number of time steps between two consecutive predictions (and non-conformity scores) of the
+ calibration set. Right-bound by the first time step of the generated forecast.
+ num_samples
+ Number of times a prediction is sampled from the calibrated quantile predictions using linear
+ interpolation in-between the quantiles. For larger values, the sample distribution approximates the
+ calibrated quantile predictions.
+ verbose
+ Whether to print the progress.
+ predict_likelihood_parameters
+ If set to `True`, generates the quantile predictions directly. Only supported with `num_samples = 1`.
+ show_warnings
+ Whether to show warnings related auto-regression and past covariates usage.
+
+ Returns
+ -------
+ Union[TimeSeries, Sequence[TimeSeries]]
+ If `series` is not specified, this function returns a single time series containing the `n`
+ next points after then end of the training series.
+ If `series` is given and is a simple ``TimeSeries``, this function returns the `n` next points
+ after the end of `series`.
+ If `series` is given and is a sequence of several time series, this function returns
+ a sequence where each element contains the corresponding `n` points forecasts.
+ """
+ if series is None:
+ # then there must be a single TS, and that was saved in super().fit as self.training_series
+ if self.model.training_series is None:
+ raise_log(
+ ValueError(
+ "Input `series` must be provided. This is the result either from fitting on multiple series, "
+ "or from not having fit the model yet."
+ ),
+ logger,
+ )
+ series = self.model.training_series
+
+ called_with_single_series = get_series_seq_type(series) == SeriesType.SINGLE
+
+ # guarantee that all inputs are either list of TimeSeries or None
+ series = series2seq(series)
+ if past_covariates is None and self.model.past_covariate_series is not None:
+ past_covariates = [self.model.past_covariate_series] * len(series)
+ if future_covariates is None and self.model.future_covariate_series is not None:
+ future_covariates = [self.model.future_covariate_series] * len(series)
+ past_covariates = series2seq(past_covariates)
+ future_covariates = series2seq(future_covariates)
+
+ super().predict(
+ n,
+ series,
+ past_covariates,
+ future_covariates,
+ num_samples,
+ verbose,
+ predict_likelihood_parameters,
+ show_warnings,
+ )
+
+ # if a calibration set is given, use it. Otherwise, use past of input as calibration
+ if cal_series is None:
+ cal_series = series
+ cal_past_covariates = past_covariates
+ cal_future_covariates = future_covariates
+
+ cal_series = series2seq(cal_series)
+ if len(cal_series) != len(series):
+ raise_log(
+ ValueError(
+ f"Mismatch between number of `cal_series` ({len(cal_series)}) "
+ f"and number of `series` ({len(series)})."
+ ),
+ logger=logger,
+ )
+ cal_past_covariates = series2seq(cal_past_covariates)
+ cal_future_covariates = series2seq(cal_future_covariates)
+
+ # generate model forecast to calibrate
+ preds = self.model.predict(
+ n=n,
+ series=series,
+ past_covariates=past_covariates,
+ future_covariates=future_covariates,
+ num_samples=self.num_samples,
+ verbose=verbose,
+ predict_likelihood_parameters=False,
+ show_warnings=show_warnings,
+ )
+ # convert to multi series case with `last_points_only=False`
+ preds = [[pred] for pred in preds]
+
+ # generate all possible forecasts for calibration
+ cal_hfcs = self.model.historical_forecasts(
+ series=cal_series,
+ past_covariates=cal_past_covariates,
+ future_covariates=cal_future_covariates,
+ num_samples=self.num_samples,
+ forecast_horizon=n,
+ retrain=False,
+ overlap_end=True,
+ last_points_only=False,
+ verbose=verbose,
+ show_warnings=show_warnings,
+ predict_likelihood_parameters=False,
+ )
+ cal_preds = self._calibrate_forecasts(
+ series=series,
+ forecasts=preds,
+ cal_series=cal_series,
+ cal_forecasts=cal_hfcs,
+ num_samples=num_samples,
+ forecast_horizon=n,
+ stride=stride,
+ overlap_end=True,
+ last_points_only=False,
+ verbose=verbose,
+ show_warnings=show_warnings,
+ predict_likelihood_parameters=predict_likelihood_parameters,
+ )
+ # convert historical forecasts output to simple forecast / prediction
+ if called_with_single_series:
+ return cal_preds[0][0]
+ else:
+ return [cp[0] for cp in cal_preds]
+
+ @_with_sanity_checks("_historical_forecasts_sanity_checks")
+ def historical_forecasts(
+ self,
+ series: Union[TimeSeries, Sequence[TimeSeries]],
+ past_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ future_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ cal_series: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ cal_past_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ cal_future_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ forecast_horizon: int = 1,
+ num_samples: int = 1,
+ train_length: Optional[int] = None,
+ start: Optional[Union[pd.Timestamp, float, int]] = None,
+ start_format: Literal["position", "value"] = "value",
+ stride: int = 1,
+ retrain: Union[bool, int, Callable[..., bool]] = True,
+ overlap_end: bool = False,
+ last_points_only: bool = True,
+ verbose: bool = False,
+ show_warnings: bool = True,
+ predict_likelihood_parameters: bool = False,
+ enable_optimization: bool = True,
+ fit_kwargs: Optional[dict[str, Any]] = None,
+ predict_kwargs: Optional[dict[str, Any]] = None,
+ sample_weight: Optional[Union[TimeSeries, Sequence[TimeSeries], str]] = None,
+ ) -> Union[TimeSeries, list[TimeSeries], list[list[TimeSeries]]]:
+ """Generates calibrated historical forecasts by simulating predictions at various points in time throughout the
+ history of the provided (potentially multiple) `series`. This process involves retrospectively applying the
+ model to different time steps, as if the forecasts were made in real-time at those specific moments. This
+ allows for an evaluation of the model's performance over the entire duration of the series, providing insights
+ into its predictive accuracy and robustness across different historical periods.
+
+ Currently, conformal models only support the pre-trained historical forecasts mode (`retrain=False`).
+ Parameters `retrain` and `train_length` are ignored.
+
+ **Pre-trained Mode:** First, all historical forecasts are generated using the underlying pre-trained global
+ forecasting model (see :meth:`ForecastingModel.historical_forecasts()
+ ` for more info). Then it
+ repeatedly builds a calibration set by either expanding from the beginning of the historical forecasts or by
+ using a fixed-length `cal_length` (the start point can also be configured with `start` and `start_format`).
+ The next forecast of length `forecast_horizon` is then calibrated on this calibration set. Subsequently, the
+ end of the calibration set is moved forward by `stride` time steps, and the process is repeated.
+ You can also use a fixed calibration set to calibrate all forecasts equally by passing `cal_series`, and
+ optional `cal_past_covariates` and `cal_future_covariates`.
+
+ By default, with `last_points_only=True`, this method returns a single time series (or a sequence of time
+ series) composed of the last point from each calibrated historical forecast. This time series will thus have a
+ frequency of `series.freq * stride`.
+ If `last_points_only=False`, it will instead return a list (or a sequence of lists) of the full calibrate
+ historical forecast series each with frequency `series.freq`.
+
+ Parameters
+ ----------
+ series
+ A (sequence of) target time series used to successively compute the historical forecasts. If `cal_series`
+ is `None`, will use the past of this series for calibration.
+ past_covariates
+ Optionally, a (sequence of) past-observed covariate time series for every input time series in `series`.
+ Their dimension must match that of the past covariates used for training. If `cal_series` is `None`, will
+ use this series for calibration.
+ future_covariates
+ Optionally, a (sequence of) future-known covariate time series for every input time series in `series`.
+ Their dimension must match that of the past covariates used for training. If `cal_series` is `None`, will
+ use this series for calibration.
+ cal_series
+ Optionally, a (sequence of) target series for every input time series in `series` to use as a fixed
+ calibration set instead of `series`.
+ cal_past_covariates
+ Optionally, a (sequence of) past covariates series for every input time series in `series` to use as a fixed
+ calibration set instead of `past_covariates`.
+ cal_future_covariates
+ Optionally, a future covariates series for every input time series in `series` to use as a fixed
+ calibration set instead of `future_covariates`.
+ forecast_horizon
+ The forecast horizon for the predictions.
+ num_samples
+ Number of times a prediction is sampled from the calibrated quantile predictions using linear
+ interpolation in-between the quantiles. For larger values, the sample distribution approximates the
+ calibrated quantile predictions.
+ train_length
+ Currently ignored by conformal models.
+ start
+ Optionally, the first point in time at which a prediction is computed. This parameter supports:
+ ``float``, ``int``, ``pandas.Timestamp``, and ``None``.
+ If a ``float``, it is the proportion of the time series that should lie before the first prediction point.
+ If an ``int``, it is either the index position of the first prediction point for `series` with a
+ `pd.DatetimeIndex`, or the index value for `series` with a `pd.RangeIndex`. The latter can be changed to
+ the index position with `start_format="position"`.
+ If a ``pandas.Timestamp``, it is the time stamp of the first prediction point.
+ If ``None``, the first prediction point will automatically be set to:
+
+ - the first predictable point if `retrain` is ``False``, or `retrain` is a Callable and the first
+ predictable point is earlier than the first trainable point.
+ - the first trainable point if `retrain` is ``True`` or ``int`` (given `train_length`),
+ or `retrain` is a ``Callable`` and the first trainable point is earlier than the first predictable point.
+ - the first trainable point (given `train_length`) otherwise
+
+ Note: If the model uses a shifted output (`output_chunk_shift > 0`), then the first predicted point is also
+ shifted by `output_chunk_shift` points into the future.
+ Note: Raises a ValueError if `start` yields a time outside the time index of `series`.
+ Note: If `start` is outside the possible historical forecasting times, will ignore the parameter
+ (default behavior with ``None``) and start at the first trainable/predictable point.
+ start_format
+ Defines the `start` format.
+ If set to ``'position'``, `start` corresponds to the index position of the first predicted point and can
+ range from `(-len(series), len(series) - 1)`.
+ If set to ``'value'``, `start` corresponds to the index value/label of the first predicted point. Will raise
+ an error if the value is not in `series`' index. Default: ``'value'``.
+ stride
+ The number of time steps between two consecutive predictions.
+ retrain
+ Currently ignored by conformal models.
+ overlap_end
+ Whether the returned forecasts can go beyond the series' end or not.
+ last_points_only
+ Whether to return only the last point of each historical forecast. If set to ``True``, the method returns a
+ single ``TimeSeries`` (for each time series in `series`) containing the successive point forecasts.
+ Otherwise, returns a list of historical ``TimeSeries`` forecasts.
+ verbose
+ Whether to print the progress.
+ show_warnings
+ Whether to show warnings related to historical forecasts optimization, or parameters `start` and
+ `train_length`.
+ predict_likelihood_parameters
+ If set to `True`, generates the quantile predictions directly. Only supported with `num_samples = 1`.
+ enable_optimization
+ Whether to use the optimized version of `historical_forecasts` when supported and available.
+ Default: ``True``.
+ fit_kwargs
+ Currently ignored by conformal models.
+ predict_kwargs
+ Optionally, some additional arguments passed to the model `predict()` method.
+ sample_weight
+ Currently ignored by conformal models.
+
+ Returns
+ -------
+ TimeSeries
+ A single historical forecast for a single `series` and `last_points_only=True`: it contains only the
+ predictions at step `forecast_horizon` from all historical forecasts.
+ list[TimeSeries]
+ A list of historical forecasts for:
+
+ - a sequence (list) of `series` and `last_points_only=True`: for each series, it contains only the
+ predictions at step `forecast_horizon` from all historical forecasts.
+ - a single `series` and `last_points_only=False`: for each historical forecast, it contains the entire
+ horizon `forecast_horizon`.
+ list[list[TimeSeries]]
+ A list of lists of historical forecasts for a sequence of `series` and `last_points_only=False`. For each
+ series, and historical forecast, it contains the entire horizon `forecast_horizon`. The outer list
+ is over the series provided in the input sequence, and the inner lists contain the historical forecasts for
+ each series.
+ """
+ called_with_single_series = get_series_seq_type(series) == SeriesType.SINGLE
+ series = series2seq(series)
+ past_covariates = series2seq(past_covariates)
+ future_covariates = series2seq(future_covariates)
+
+ if cal_series is not None:
+ cal_series = series2seq(cal_series)
+ if len(cal_series) != len(series):
+ raise_log(
+ ValueError(
+ f"Mismatch between number of `cal_series` ({len(cal_series)}) "
+ f"and number of `series` ({len(series)})."
+ ),
+ logger=logger,
+ )
+ cal_past_covariates = series2seq(cal_past_covariates)
+ cal_future_covariates = series2seq(cal_future_covariates)
+
+ # generate all possible forecasts (overlap_end=True) to have enough residuals
+ hfcs = self.model.historical_forecasts(
+ series=series,
+ past_covariates=past_covariates,
+ future_covariates=future_covariates,
+ num_samples=self.num_samples,
+ forecast_horizon=forecast_horizon,
+ retrain=False,
+ overlap_end=overlap_end,
+ last_points_only=last_points_only,
+ verbose=verbose,
+ show_warnings=show_warnings,
+ predict_likelihood_parameters=False,
+ enable_optimization=enable_optimization,
+ fit_kwargs=fit_kwargs,
+ predict_kwargs=predict_kwargs,
+ )
+ # optionally, generate calibration forecasts
+ if cal_series is None:
+ cal_hfcs = None
+ else:
+ cal_hfcs = self.model.historical_forecasts(
+ series=cal_series,
+ past_covariates=cal_past_covariates,
+ future_covariates=cal_future_covariates,
+ num_samples=self.num_samples,
+ forecast_horizon=forecast_horizon,
+ retrain=False,
+ overlap_end=True,
+ last_points_only=last_points_only,
+ verbose=verbose,
+ show_warnings=show_warnings,
+ predict_likelihood_parameters=False,
+ enable_optimization=enable_optimization,
+ fit_kwargs=fit_kwargs,
+ predict_kwargs=predict_kwargs,
+ )
+ calibrated_forecasts = self._calibrate_forecasts(
+ series=series,
+ forecasts=hfcs,
+ cal_series=cal_series,
+ cal_forecasts=cal_hfcs,
+ num_samples=num_samples,
+ start=start,
+ start_format=start_format,
+ forecast_horizon=forecast_horizon,
+ stride=stride,
+ overlap_end=overlap_end,
+ last_points_only=last_points_only,
+ verbose=verbose,
+ show_warnings=show_warnings,
+ predict_likelihood_parameters=predict_likelihood_parameters,
+ )
+ return (
+ calibrated_forecasts[0]
+ if called_with_single_series
+ else calibrated_forecasts
+ )
+
+ def backtest(
+ self,
+ series: Union[TimeSeries, Sequence[TimeSeries]],
+ past_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ future_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ cal_series: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ cal_past_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ cal_future_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ historical_forecasts: Optional[
+ Union[TimeSeries, Sequence[TimeSeries], Sequence[Sequence[TimeSeries]]]
+ ] = None,
+ forecast_horizon: int = 1,
+ num_samples: int = 1,
+ train_length: Optional[int] = None,
+ start: Optional[Union[pd.Timestamp, float, int]] = None,
+ start_format: Literal["position", "value"] = "value",
+ stride: int = 1,
+ retrain: Union[bool, int, Callable[..., bool]] = True,
+ overlap_end: bool = False,
+ last_points_only: bool = False,
+ metric: Union[METRIC_TYPE, list[METRIC_TYPE]] = metrics.mape,
+ reduction: Union[Callable[..., float], None] = np.mean,
+ verbose: bool = False,
+ show_warnings: bool = True,
+ predict_likelihood_parameters: bool = False,
+ enable_optimization: bool = True,
+ metric_kwargs: Optional[Union[dict[str, Any], list[dict[str, Any]]]] = None,
+ fit_kwargs: Optional[dict[str, Any]] = None,
+ predict_kwargs: Optional[dict[str, Any]] = None,
+ sample_weight: Optional[Union[TimeSeries, Sequence[TimeSeries], str]] = None,
+ ) -> Union[float, np.ndarray, list[float], list[np.ndarray]]:
+ """Compute error values that the model produced for historical forecasts on (potentially multiple) `series`.
+
+ If `historical_forecasts` are provided, the metric(s) (given by the `metric` function) is evaluated directly on
+ all forecasts and actual values. The same `series` and `last_points_only` value must be passed that were used
+ to generate the historical forecasts. Finally, the method returns an optional `reduction` (the mean by default)
+ of all these metric scores.
+
+ If `historical_forecasts` is ``None``, it first generates the historical forecasts with the parameters given
+ below (see :meth:`ConformalModel.historical_forecasts()
+ ` for more info) and then
+ evaluates as described above.
+
+ The metric(s) can be further customized `metric_kwargs` (e.g. control the aggregation over components, time
+ steps, multiple series, other required arguments such as `q` for quantile metrics, ...).
+
+ Notes
+ -----
+ Darts has several metrics to evaluate probabilistic forecasts. For conformal models, we recommend using
+ quantile interval metrics (see `here `_).
+ You can specify which intervals to evaluate by setting `metric_kwargs={'q_interval': my_intervals}`. To check
+ all intervals used by your conformal model `my_model`, you can set ``{'q_interval': my_model.q_interval}``.
+
+ Parameters
+ ----------
+ series
+ A (sequence of) target time series used to successively compute the historical forecasts. If `cal_series`
+ is `None`, will use the past of this series for calibration.
+ past_covariates
+ Optionally, a (sequence of) past-observed covariate time series for every input time series in `series`.
+ Their dimension must match that of the past covariates used for training. If `cal_series` is `None`, will
+ use this series for calibration.
+ future_covariates
+ Optionally, a (sequence of) future-known covariate time series for every input time series in `series`.
+ Their dimension must match that of the past covariates used for training. If `cal_series` is `None`, will
+ use this series for calibration.
+ cal_series
+ Optionally, a (sequence of) target series for every input time series in `series` to use as a fixed
+ calibration set instead of `series`.
+ cal_past_covariates
+ Optionally, a (sequence of) past covariates series for every input time series in `series` to use as a fixed
+ calibration set instead of `past_covariates`.
+ cal_future_covariates
+ Optionally, a future covariates series for every input time series in `series` to use as a fixed
+ calibration set instead of `future_covariates`.
+ historical_forecasts
+ Optionally, the (or a sequence of / a sequence of sequences of) historical forecasts time series to be
+ evaluated. Corresponds to the output of :meth:`historical_forecasts()
+ `. The same `series` and
+ `last_points_only` values must be passed that were used to generate the historical forecasts. If provided,
+ will skip historical forecasting and ignore all parameters except `series`, `last_points_only`, `metric`,
+ and `reduction`.
+ forecast_horizon
+ The forecast horizon for the predictions.
+ num_samples
+ Number of times a prediction is sampled from the calibrated quantile predictions using linear
+ interpolation in-between the quantiles. For larger values, the sample distribution approximates the
+ calibrated quantile predictions.
+ train_length
+ Currently ignored by conformal models.
+ start
+ Optionally, the first point in time at which a prediction is computed. This parameter supports:
+ ``float``, ``int``, ``pandas.Timestamp``, and ``None``.
+ If a ``float``, it is the proportion of the time series that should lie before the first prediction point.
+ If an ``int``, it is either the index position of the first prediction point for `series` with a
+ `pd.DatetimeIndex`, or the index value for `series` with a `pd.RangeIndex`. The latter can be changed to
+ the index position with `start_format="position"`.
+ If a ``pandas.Timestamp``, it is the time stamp of the first prediction point.
+ If ``None``, the first prediction point will automatically be set to:
+
+ - the first predictable point if `retrain` is ``False``, or `retrain` is a Callable and the first
+ predictable point is earlier than the first trainable point.
+ - the first trainable point if `retrain` is ``True`` or ``int`` (given `train_length`),
+ or `retrain` is a ``Callable`` and the first trainable point is earlier than the first predictable point.
+ - the first trainable point (given `train_length`) otherwise
+
+ Note: If the model uses a shifted output (`output_chunk_shift > 0`), then the first predicted point is also
+ shifted by `output_chunk_shift` points into the future.
+ Note: Raises a ValueError if `start` yields a time outside the time index of `series`.
+ Note: If `start` is outside the possible historical forecasting times, will ignore the parameter
+ (default behavior with ``None``) and start at the first trainable/predictable point.
+ start_format
+ Defines the `start` format.
+ If set to ``'position'``, `start` corresponds to the index position of the first predicted point and can
+ range from `(-len(series), len(series) - 1)`.
+ If set to ``'value'``, `start` corresponds to the index value/label of the first predicted point. Will raise
+ an error if the value is not in `series`' index. Default: ``'value'``.
+ stride
+ The number of time steps between two consecutive predictions.
+ retrain
+ Currently ignored by conformal models.
+ overlap_end
+ Whether the returned forecasts can go beyond the series' end or not.
+ last_points_only
+ Whether to return only the last point of each historical forecast. If set to ``True``, the method returns a
+ single ``TimeSeries`` (for each time series in `series`) containing the successive point forecasts.
+ Otherwise, returns a list of historical ``TimeSeries`` forecasts.
+ metric
+ A metric function or a list of metric functions. Each metric must either be a Darts metric (see `here
+ `_), or a custom metric that has an
+ identical signature as Darts' metrics, uses decorators :func:`~darts.metrics.metrics.multi_ts_support` and
+ :func:`~darts.metrics.metrics.multi_ts_support`, and returns the metric score.
+ reduction
+ A function used to combine the individual error scores obtained when `last_points_only` is set to `False`.
+ When providing several metric functions, the function will receive the argument `axis = 1` to obtain single
+ value for each metric function.
+ If explicitly set to `None`, the method will return a list of the individual error scores instead.
+ Set to ``np.mean`` by default.
+ verbose
+ Whether to print the progress.
+ show_warnings
+ Whether to show warnings related to historical forecasts optimization, or parameters `start` and
+ `train_length`.
+ predict_likelihood_parameters
+ If set to `True`, generates the quantile predictions directly. Only supported with `num_samples = 1`.
+ enable_optimization
+ Whether to use the optimized version of `historical_forecasts` when supported and available.
+ Default: ``True``.
+ metric_kwargs
+ Additional arguments passed to `metric()`, such as `'n_jobs'` for parallelization, `'component_reduction'`
+ for reducing the component wise metrics, seasonality `'m'` for scaled metrics, etc. Will pass arguments to
+ each metric separately and only if they are present in the corresponding metric signature. Parameter
+ `'insample'` for scaled metrics (e.g. mase`, `rmsse`, ...) is ignored, as it is handled internally.
+ fit_kwargs
+ Currently ignored by conformal models.
+ predict_kwargs
+ Optionally, some additional arguments passed to the model `predict()` method.
+ sample_weight
+ Currently ignored by conformal models.
+
+ Returns
+ -------
+ float
+ A single backtest score for single uni/multivariate series, a single `metric` function and:
+
+ - `historical_forecasts` generated with `last_points_only=True`
+ - `historical_forecasts` generated with `last_points_only=False` and using a backtest `reduction`
+ np.ndarray
+ An numpy array of backtest scores. For single series and one of:
+
+ - a single `metric` function, `historical_forecasts` generated with `last_points_only=False`
+ and backtest `reduction=None`. The output has shape (n forecasts, *).
+ - multiple `metric` functions and `historical_forecasts` generated with `last_points_only=False`.
+ The output has shape (*, n metrics) when using a backtest `reduction`, and (n forecasts, *, n metrics)
+ when `reduction=None`
+ - multiple uni/multivariate series including `series_reduction` and at least one of
+ `component_reduction=None` or `time_reduction=None` for "per time step metrics"
+ list[float]
+ Same as for type `float` but for a sequence of series. The returned metric list has length
+ `len(series)` with the `float` metric for each input `series`.
+ list[np.ndarray]
+ Same as for type `np.ndarray` but for a sequence of series. The returned metric list has length
+ `len(series)` with the `np.ndarray` metrics for each input `series`.
+ """
+ historical_forecasts = historical_forecasts or self.historical_forecasts(
+ series=series,
+ past_covariates=past_covariates,
+ future_covariates=future_covariates,
+ cal_series=cal_series,
+ cal_past_covariates=cal_past_covariates,
+ cal_future_covariates=cal_future_covariates,
+ num_samples=num_samples,
+ train_length=train_length,
+ start=start,
+ start_format=start_format,
+ forecast_horizon=forecast_horizon,
+ stride=stride,
+ retrain=retrain,
+ last_points_only=last_points_only,
+ verbose=verbose,
+ show_warnings=show_warnings,
+ predict_likelihood_parameters=predict_likelihood_parameters,
+ enable_optimization=enable_optimization,
+ fit_kwargs=fit_kwargs,
+ predict_kwargs=predict_kwargs,
+ overlap_end=overlap_end,
+ sample_weight=sample_weight,
+ )
+ return super().backtest(
+ series=series,
+ historical_forecasts=historical_forecasts,
+ forecast_horizon=forecast_horizon,
+ num_samples=num_samples,
+ train_length=train_length,
+ start=start,
+ start_format=start_format,
+ stride=stride,
+ retrain=retrain,
+ overlap_end=overlap_end,
+ last_points_only=last_points_only,
+ metric=metric,
+ reduction=reduction,
+ verbose=verbose,
+ show_warnings=show_warnings,
+ predict_likelihood_parameters=predict_likelihood_parameters,
+ enable_optimization=enable_optimization,
+ metric_kwargs=metric_kwargs,
+ fit_kwargs=fit_kwargs,
+ predict_kwargs=predict_kwargs,
+ sample_weight=sample_weight,
+ )
+
+ def residuals(
+ self,
+ series: Union[TimeSeries, Sequence[TimeSeries]],
+ past_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ future_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ cal_series: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ cal_past_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ cal_future_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ historical_forecasts: Optional[
+ Union[TimeSeries, Sequence[TimeSeries], Sequence[Sequence[TimeSeries]]]
+ ] = None,
+ forecast_horizon: int = 1,
+ num_samples: int = 1,
+ train_length: Optional[int] = None,
+ start: Optional[Union[pd.Timestamp, float, int]] = None,
+ start_format: Literal["position", "value"] = "value",
+ stride: int = 1,
+ retrain: Union[bool, int, Callable[..., bool]] = True,
+ overlap_end: bool = False,
+ last_points_only: bool = True,
+ metric: METRIC_TYPE = metrics.err,
+ verbose: bool = False,
+ show_warnings: bool = True,
+ predict_likelihood_parameters: bool = False,
+ enable_optimization: bool = True,
+ metric_kwargs: Optional[dict[str, Any]] = None,
+ fit_kwargs: Optional[dict[str, Any]] = None,
+ predict_kwargs: Optional[dict[str, Any]] = None,
+ sample_weight: Optional[Union[TimeSeries, Sequence[TimeSeries], str]] = None,
+ values_only: bool = False,
+ ) -> Union[TimeSeries, list[TimeSeries], list[list[TimeSeries]]]:
+ """Compute the residuals that the model produced for historical forecasts on (potentially multiple) `series`.
+
+ This function computes the difference (or one of Darts' "per time step" metrics) between the actual
+ observations from `series` and the fitted values obtained by training the model on `series` (or using a
+ pre-trained model with `retrain=False`). Not all models support fitted values, so we use historical forecasts
+ as an approximation for them.
+
+ In sequence this method performs:
+
+ - use pre-computed `historical_forecasts` or compute historical forecasts for each series (see
+ :meth:`~darts.models.forecasting.conformal_models.ConformalModel.historical_forecasts` for more details).
+ How the historical forecasts are generated can be configured with parameters `num_samples`, `train_length`,
+ `start`, `start_format`, `forecast_horizon`, `stride`, `retrain`, `last_points_only`, `fit_kwargs`, and
+ `predict_kwargs`.
+ - compute a backtest using a "per time step" `metric` between the historical forecasts and `series` per
+ component/column and time step (see
+ :meth:`~darts.models.forecasting.conformal_models.ConformalModel.backtest` for more details). By default,
+ uses the residuals :func:`~darts.metrics.metrics.err` (error) as a `metric`.
+ - create and return `TimeSeries` (or simply a np.ndarray with `values_only=True`) with the time index from
+ historical forecasts, and values from the metrics per component and time step.
+
+ This method works for single or multiple univariate or multivariate series.
+ It uses the median prediction (when dealing with stochastic forecasts).
+
+ Notes
+ -----
+ Darts has several metrics to evaluate probabilistic forecasts. For conformal models, we recommend using
+ "per time step" quantile interval metrics (see `here
+ `_). You can specify which intervals to
+ evaluate by setting `metric_kwargs={'q_interval': my_intervals}`. To check all intervals used by your conformal
+ model `my_model`, you can set ``{'q_interval': my_model.q_interval}``.
+
+ Parameters
+ ----------
+ series
+ A (sequence of) target time series used to successively compute the historical forecasts. If `cal_series`
+ is `None`, will use the past of this series for calibration.
+ past_covariates
+ Optionally, a (sequence of) past-observed covariate time series for every input time series in `series`.
+ Their dimension must match that of the past covariates used for training. If `cal_series` is `None`, will
+ use this series for calibration.
+ future_covariates
+ Optionally, a (sequence of) future-known covariate time series for every input time series in `series`.
+ Their dimension must match that of the past covariates used for training. If `cal_series` is `None`, will
+ use this series for calibration.
+ cal_series
+ Optionally, a (sequence of) target series for every input time series in `series` to use as a fixed
+ calibration set instead of `series`.
+ cal_past_covariates
+ Optionally, a (sequence of) past covariates series for every input time series in `series` to use as a fixed
+ calibration set instead of `past_covariates`.
+ cal_future_covariates
+ Optionally, a future covariates series for every input time series in `series` to use as a fixed
+ calibration set instead of `future_covariates`.
+ historical_forecasts
+ Optionally, the (or a sequence of / a sequence of sequences of) historical forecasts time series to be
+ evaluated. Corresponds to the output of :meth:`historical_forecasts()
+ `. The same `series` and
+ `last_points_only` values must be passed that were used to generate the historical forecasts. If provided,
+ will skip historical forecasting and ignore all parameters except `series`, `last_points_only`, `metric`,
+ and `reduction`.
+ forecast_horizon
+ The forecast horizon for the predictions.
+ num_samples
+ Number of times a prediction is sampled from the calibrated quantile predictions using linear
+ interpolation in-between the quantiles. For larger values, the sample distribution approximates the
+ calibrated quantile predictions.
+ train_length
+ Currently ignored by conformal models.
+ start
+ Optionally, the first point in time at which a prediction is computed. This parameter supports:
+ ``float``, ``int``, ``pandas.Timestamp``, and ``None``.
+ If a ``float``, it is the proportion of the time series that should lie before the first prediction point.
+ If an ``int``, it is either the index position of the first prediction point for `series` with a
+ `pd.DatetimeIndex`, or the index value for `series` with a `pd.RangeIndex`. The latter can be changed to
+ the index position with `start_format="position"`.
+ If a ``pandas.Timestamp``, it is the time stamp of the first prediction point.
+ If ``None``, the first prediction point will automatically be set to:
+
+ - the first predictable point if `retrain` is ``False``, or `retrain` is a Callable and the first
+ predictable point is earlier than the first trainable point.
+ - the first trainable point if `retrain` is ``True`` or ``int`` (given `train_length`),
+ or `retrain` is a ``Callable`` and the first trainable point is earlier than the first predictable point.
+ - the first trainable point (given `train_length`) otherwise
+
+ Note: If the model uses a shifted output (`output_chunk_shift > 0`), then the first predicted point is also
+ shifted by `output_chunk_shift` points into the future.
+ Note: Raises a ValueError if `start` yields a time outside the time index of `series`.
+ Note: If `start` is outside the possible historical forecasting times, will ignore the parameter
+ (default behavior with ``None``) and start at the first trainable/predictable point.
+ start_format
+ Defines the `start` format.
+ If set to ``'position'``, `start` corresponds to the index position of the first predicted point and can
+ range from `(-len(series), len(series) - 1)`.
+ If set to ``'value'``, `start` corresponds to the index value/label of the first predicted point. Will raise
+ an error if the value is not in `series`' index. Default: ``'value'``.
+ stride
+ The number of time steps between two consecutive predictions.
+ retrain
+ Currently ignored by conformal models.
+ overlap_end
+ Whether the returned forecasts can go beyond the series' end or not.
+ last_points_only
+ Whether to return only the last point of each historical forecast. If set to ``True``, the method returns a
+ single ``TimeSeries`` (for each time series in `series`) containing the successive point forecasts.
+ Otherwise, returns a list of historical ``TimeSeries`` forecasts.
+ metric
+ Either one of Darts' "per time step" metrics (see `here
+ `_), or a custom metric that has an
+ identical signature as Darts' "per time step" metrics, uses decorators
+ :func:`~darts.metrics.metrics.multi_ts_support` and :func:`~darts.metrics.metrics.multi_ts_support`,
+ and returns one value per time step.
+ verbose
+ Whether to print the progress.
+ show_warnings
+ Whether to show warnings related to historical forecasts optimization, or parameters `start` and
+ `train_length`.
+ predict_likelihood_parameters
+ If set to `True`, generates the quantile predictions directly. Only supported with `num_samples = 1`.
+ enable_optimization
+ Whether to use the optimized version of `historical_forecasts` when supported and available.
+ Default: ``True``.
+ metric_kwargs
+ Additional arguments passed to `metric()`, such as `'n_jobs'` for parallelization, `'m'` for scaled
+ metrics, etc. Will pass arguments only if they are present in the corresponding metric signature. Ignores
+ reduction arguments `"series_reduction", "component_reduction", "time_reduction"`, and parameter
+ `'insample'` for scaled metrics (e.g. mase`, `rmsse`, ...), as they are handled internally.
+ fit_kwargs
+ Currently ignored by conformal models.
+ predict_kwargs
+ Optionally, some additional arguments passed to the model `predict()` method.
+ sample_weight
+ Currently ignored by conformal models.
+ values_only
+ Whether to return the residuals as `np.ndarray`. If `False`, returns residuals as `TimeSeries`.
+
+ Returns
+ -------
+ TimeSeries
+ Residual `TimeSeries` for a single `series` and `historical_forecasts` generated with
+ `last_points_only=True`.
+ list[TimeSeries]
+ A list of residual `TimeSeries` for a sequence (list) of `series` with `last_points_only=True`.
+ The residual list has length `len(series)`.
+ list[list[TimeSeries]]
+ A list of lists of residual `TimeSeries` for a sequence of `series` with `last_points_only=False`.
+ The outer residual list has length `len(series)`. The inner lists consist of the residuals from
+ all possible series-specific historical forecasts.
+ """
+ historical_forecasts = historical_forecasts or self.historical_forecasts(
+ series=series,
+ past_covariates=past_covariates,
+ future_covariates=future_covariates,
+ cal_series=cal_series,
+ cal_past_covariates=cal_past_covariates,
+ cal_future_covariates=cal_future_covariates,
+ num_samples=num_samples,
+ train_length=train_length,
+ start=start,
+ start_format=start_format,
+ forecast_horizon=forecast_horizon,
+ stride=stride,
+ retrain=retrain,
+ last_points_only=last_points_only,
+ verbose=verbose,
+ show_warnings=show_warnings,
+ predict_likelihood_parameters=predict_likelihood_parameters,
+ enable_optimization=enable_optimization,
+ fit_kwargs=fit_kwargs,
+ predict_kwargs=predict_kwargs,
+ overlap_end=overlap_end,
+ sample_weight=sample_weight,
+ )
+ return super().residuals(
+ series=series,
+ historical_forecasts=historical_forecasts,
+ forecast_horizon=forecast_horizon,
+ num_samples=num_samples,
+ train_length=train_length,
+ start=start,
+ start_format=start_format,
+ stride=stride,
+ retrain=retrain,
+ overlap_end=overlap_end,
+ last_points_only=last_points_only,
+ metric=metric,
+ verbose=verbose,
+ show_warnings=show_warnings,
+ predict_likelihood_parameters=predict_likelihood_parameters,
+ enable_optimization=enable_optimization,
+ metric_kwargs=metric_kwargs,
+ fit_kwargs=fit_kwargs,
+ predict_kwargs=predict_kwargs,
+ sample_weight=sample_weight,
+ values_only=values_only,
+ )
+
+ @random_method
+ def _calibrate_forecasts(
+ self,
+ series: Sequence[TimeSeries],
+ forecasts: Union[Sequence[Sequence[TimeSeries]], Sequence[TimeSeries]],
+ cal_series: Optional[Sequence[TimeSeries]] = None,
+ cal_forecasts: Optional[
+ Union[Sequence[Sequence[TimeSeries]], Sequence[TimeSeries]]
+ ] = None,
+ num_samples: int = 1,
+ start: Optional[Union[pd.Timestamp, float, int]] = None,
+ start_format: Literal["position", "value"] = "value",
+ forecast_horizon: int = 1,
+ stride: int = 1,
+ overlap_end: bool = False,
+ last_points_only: bool = True,
+ verbose: bool = False,
+ show_warnings: bool = True,
+ predict_likelihood_parameters: bool = False,
+ ) -> Union[TimeSeries, list[TimeSeries], list[list[TimeSeries]]]:
+ """Generate calibrated historical forecasts.
+
+ In general the workflow of the models to produce one calibrated forecast/prediction per step in the horizon
+ is as follows:
+
+ - Generate historical forecasts for `series` and optional calibration set (`cal_series`) (using the forecasting
+ model)
+ - Extract a calibration set: The forecasts from the most recent past to use as calibration
+ for one conformal prediction. The number of examples to use can be defined at model creation with parameter
+ `cal_length`. We support two modes:
+ - Automatic extraction of the calibration set from the past of your input series (`series`,
+ `past_covariates`, ...). This is the default mode and our predict/forecasting/backtest/.... API is
+ identical to any other forecasting model
+ - Supply a fixed calibration set with parameters `cal_series`, `cal_past_covariates`, ... .
+ - Compute the errors/non-conformity scores (specific to each conformal model) on these historical forecasts
+ - Compute the quantile values from the errors / non-conformity scores (using our desired quantiles set at model
+ creation with parameter `quantiles`).
+ - Compute the conformal prediction: Add the calibrated intervals to (or adjust the existing intervals of) the
+ forecasting model's predictions.
+ """
+ # TODO: add proper handling of `cal_stride` > 1
+ # cal_stride = stride if self.stride_cal else 1
+ cal_length = self.cal_length
+ metric, metric_kwargs = self._residuals_metric
+ residuals = self.model.residuals(
+ series=series if cal_series is None else cal_series,
+ historical_forecasts=forecasts if cal_series is None else cal_forecasts,
+ overlap_end=overlap_end if cal_series is None else True,
+ last_points_only=last_points_only,
+ verbose=verbose,
+ show_warnings=show_warnings,
+ values_only=True,
+ metric=metric,
+ metric_kwargs=metric_kwargs,
+ )
+
+ outer_iterator = enumerate(zip(series, forecasts, residuals))
+ if len(series) > 1:
+ # Use tqdm on the outer loop only if there's more than one series to iterate over
+ # (otherwise use tqdm on the inner loop).
+ outer_iterator = _build_tqdm_iterator(
+ outer_iterator,
+ verbose,
+ total=len(series),
+ desc="conformal forecasts",
+ )
+
+ cp_hfcs = []
+ for series_idx, (series_, s_hfcs, res) in outer_iterator:
+ cp_preds = []
+
+ # no historical forecasts were generated
+ if not s_hfcs:
+ cp_hfcs.append(cp_preds)
+ continue
+
+ last_hfc = s_hfcs if last_points_only else s_hfcs[-1]
+
+ # compute the minimum required number of useful calibration residuals
+ # at least one or `cal_length` examples
+ min_n_cal = cal_length or 1
+ # `last_points_only=False` requires additional examples to use most recent information
+ # from all steps in the horizon
+ if not last_points_only:
+ min_n_cal += forecast_horizon - 1
+
+ # determine first forecast index for conformal prediction
+ if cal_series is None:
+ # we need at least one residual per point in the horizon prior to the first conformal forecast
+ first_idx_train = forecast_horizon + self.output_chunk_shift
+ # plus some additional examples based on `cal_length`
+ if cal_length is not None:
+ first_idx_train += cal_length - 1
+ # check if later we need to drop some residuals without useful information (unknown residuals)
+ if overlap_end:
+ delta_end = n_steps_between(
+ end=last_hfc.end_time(),
+ start=series_.end_time(),
+ freq=series_.freq,
+ )
+ else:
+ delta_end = 0
+ else:
+ # calibration set is decoupled from `series` forecasts; we can start with the first forecast
+ first_idx_train = 0
+ # check if we need to drop some residuals without useful information
+ cal_series_ = cal_series[series_idx]
+ cal_last_hfc = cal_forecasts[series_idx][-1]
+ delta_end = n_steps_between(
+ end=cal_last_hfc.end_time(),
+ start=cal_series_.end_time(),
+ freq=cal_series_.freq,
+ )
+
+ # drop residuals without useful information
+ last_res_idx = None
+ if last_points_only and delta_end > 0:
+ # useful residual information only up until the forecast
+ # ending at the last time step in `series`
+ last_res_idx = -delta_end
+ elif not last_points_only and delta_end >= forecast_horizon:
+ # useful residual information only up until the forecast
+ # starting at the last time step in `series`
+ last_res_idx = -(delta_end - forecast_horizon + 1)
+ if last_res_idx is None and cal_series is None:
+ # drop at least the one residuals/forecast from the end, since we can only use prior residuals
+ last_res_idx = -(self.output_chunk_shift + 1)
+ # with last points only, ignore the last `horizon` residuals to avoid look-ahead bias
+ if last_points_only:
+ last_res_idx -= forecast_horizon - 1
+
+ if last_res_idx is not None:
+ res = res[:last_res_idx]
+
+ if first_idx_train >= len(s_hfcs) or len(res) < min_n_cal:
+ set_name = "" if cal_series is None else "cal_"
+ raise_log(
+ ValueError(
+ "Could not build the minimum required calibration input with the provided "
+ f"`{set_name}series` and `{set_name}*_covariates` at series index: {series_idx}. "
+ f"Expected to generate at least `{min_n_cal}` calibration forecasts with known residuals "
+ f"before the first conformal forecast, but could only generate `{len(res)}`."
+ ),
+ logger=logger,
+ )
+ # adjust first index based on `start`
+ first_idx_start = 0
+ if start is not None:
+ # adjust forecastable index in case of output shift or `last_points_only=True`
+ adjust_idx = (
+ self.output_chunk_shift
+ + int(last_points_only) * (forecast_horizon - 1)
+ ) * series_.freq
+ historical_forecastable_index = (
+ s_hfcs[first_idx_train].start_time() - adjust_idx,
+ s_hfcs[-1].start_time() - adjust_idx,
+ )
+ # TODO: add proper start handling with `cal_stride>1`
+ # adjust forecastable index based on start, assuming hfcs were generated with `stride=1`
+ first_idx_start, _ = _adjust_historical_forecasts_time_index(
+ series=series_,
+ series_idx=series_idx,
+ start=start,
+ start_format=start_format,
+ stride=stride,
+ historical_forecasts_time_index=historical_forecastable_index,
+ show_warnings=show_warnings,
+ )
+ # find position relative to start
+ first_idx_start = n_steps_between(
+ first_idx_start + adjust_idx,
+ s_hfcs[0].start_time(),
+ freq=series_.freq,
+ )
+
+ # get final first index
+ first_fc_idx = max([first_idx_train, first_idx_start])
+ # bring into shape (forecasting steps, n components, n samples * n examples)
+ if last_points_only:
+ # -> (1, n components, n samples * n examples)
+ res = res.T
+ else:
+ res = np.array(res)
+ # -> (forecast horizon, n components, n samples * n examples)
+ # rearrange the residuals to avoid look-ahead bias and to have the same number of examples per
+ # point in the horizon. We want the most recent residuals in the past for each step in the horizon.
+ # Meaning that to conformalize any forecast at some time `t` with `horizon=n`:
+ # - for `horizon=1` of that forecast calibrate with residuals from all 1-step forecasts up until
+ # forecast time `t-1`
+ # - for `horizon=n` of that forecast calibrate with residuals from all n-step forecasts up until
+ # forecast time `t-n`
+ # The rearranged residuals will look as follows, where `res_ti_cj_hk` is the
+ # residuals at time `ti` for component `cj` at forecasted step/horizon `hk`.
+ # ```
+ # [ # forecast horizon
+ # [ # components
+ # [res_t0_c0_h1, ...] # residuals at different times
+ # [..., res_tn_cn_h1],
+ # ],
+ # ...,
+ # [
+ # [res_t0_c0_hn, ...],
+ # [..., res_tn_cn_hn],
+ # ],
+ # ]
+ # ```
+ res_ = []
+ for irr in range(forecast_horizon - 1, -1, -1):
+ res_end_idx = -(forecast_horizon - (irr + 1))
+ res_.append(res[irr : res_end_idx or None, abs(res_end_idx)])
+ res = np.concatenate(res_, axis=2).T
+
+ # get the last forecast index based on the residual examples
+ if cal_series is None:
+ last_fc_idx = res.shape[2] + (
+ forecast_horizon + self.output_chunk_shift
+ )
+ else:
+ last_fc_idx = len(s_hfcs)
+
+ q_hat = None
+ # with a calibration set, the calibrated interval is constant across all forecasts
+ if cal_series is not None:
+ if cal_length is not None:
+ res = res[:, :, -cal_length:]
+ q_hat = self._calibrate_interval(res)
+
+ def conformal_predict(idx_, pred_vals_):
+ if cal_series is None:
+ # get the last residual index for calibration, `cal_end` is exclusive
+ # to avoid look-ahead bias, use only residuals from before the historical forecast start point;
+ # for `last_points_only=True`, the last residual historically available at the forecasting
+ # point is `forecast_horizon + self.output_chunk_shift - 1` steps before. The same applies to
+ # `last_points_only=False` thanks to the residual rearrangement
+ cal_end = (
+ first_fc_idx
+ + idx_ * stride
+ - (forecast_horizon + self.output_chunk_shift - 1)
+ )
+ # first residual index is shifted back by the horizon to get `cal_length` points for
+ # the last point in the horizon
+ cal_start = cal_end - cal_length if cal_length is not None else None
+
+ cal_res = res[:, :, cal_start:cal_end]
+ q_hat_ = self._calibrate_interval(cal_res)
+ else:
+ # with a calibration set, use a constant q_hat
+ q_hat_ = q_hat
+ vals = self._apply_interval(pred_vals_, q_hat_)
+ if not predict_likelihood_parameters:
+ vals = sample_from_quantiles(
+ vals, self.quantiles, num_samples=num_samples
+ )
+ return vals
+
+ # historical conformal prediction
+ # for each forecast, compute calibrated quantile intervals based on past residuals
+ if last_points_only:
+ inner_iterator = enumerate(
+ s_hfcs.all_values(copy=False)[first_fc_idx:last_fc_idx:stride]
+ )
+ else:
+ inner_iterator = enumerate(s_hfcs[first_fc_idx:last_fc_idx:stride])
+ comp_names_out = (
+ self._cp_component_names(series_)
+ if predict_likelihood_parameters
+ else None
+ )
+ if len(series) == 1:
+ # Only use progress bar if there's no outer loop
+ inner_iterator = _build_tqdm_iterator(
+ inner_iterator,
+ verbose,
+ total=(last_fc_idx - 1 - first_fc_idx) // stride + 1,
+ desc="conformal forecasts",
+ )
+
+ if last_points_only:
+ for idx, pred_vals in inner_iterator:
+ pred_vals = np.expand_dims(pred_vals, 0)
+ cp_pred = conformal_predict(idx, pred_vals)
+ cp_preds.append(cp_pred)
+ cp_preds = _build_forecast_series(
+ points_preds=np.concatenate(cp_preds, axis=0),
+ input_series=series_,
+ custom_columns=comp_names_out,
+ time_index=generate_index(
+ start=s_hfcs._time_index[first_fc_idx],
+ length=len(cp_preds),
+ freq=series_.freq * stride,
+ name=series_._time_index.name,
+ ),
+ with_static_covs=False,
+ with_hierarchy=False,
+ )
+ else:
+ for idx, pred in inner_iterator:
+ pred_vals = pred.all_values(copy=False)
+ cp_pred = conformal_predict(idx, pred_vals)
+ cp_pred = _build_forecast_series(
+ points_preds=cp_pred,
+ input_series=series_,
+ custom_columns=comp_names_out,
+ time_index=pred._time_index,
+ with_static_covs=False,
+ with_hierarchy=False,
+ )
+ cp_preds.append(cp_pred)
+ cp_hfcs.append(cp_preds)
+ return cp_hfcs
+
+ def save(
+ self, path: Optional[Union[str, os.PathLike, BinaryIO]] = None, **pkl_kwargs
+ ) -> None:
+ """
+ Saves the conformal model under a given path or file handle.
+
+ Additionally, two files are stored if `self.model` is a `TorchForecastingModel`.
+
+ Example for saving and loading a :class:`ConformalNaiveModel`:
+
+ .. highlight:: python
+ .. code-block:: python
+
+ from darts.datasets import AirPassengersDataset
+ from darts.models import ConformalNaiveModel, LinearRegressionModel
+
+ series = AirPassengersDataset().load()
+ forecasting_model = LinearRegressionModel(lags=4).fit(series)
+
+ model = ConformalNaiveModel(
+ model=forecasting_model,
+ quantiles=[0.1, 0.5, 0.9],
+ )
+
+ model.save("my_model.pkl")
+ model_loaded = ConformalNaiveModel.load("my_model.pkl")
+ ..
+
+ Parameters
+ ----------
+ path
+ Path or file handle under which to save the ensemble model at its current state. If no path is specified,
+ the ensemble model is automatically saved under ``"{ConformalNaiveModel}_{YYYY-mm-dd_HH_MM_SS}.pkl"``.
+ If the forecasting model is a `TorchForecastingModel`, two files (model object and checkpoint) are saved
+ under ``"{path}.{ModelClass}.pt"`` and ``"{path}.{ModelClass}.ckpt"``.
+ pkl_kwargs
+ Keyword arguments passed to `pickle.dump()`
+ """
+
+ if path is None:
+ # default path
+ path = self._default_save_path() + ".pkl"
+
+ super().save(path, **pkl_kwargs)
+
+ if TORCH_AVAILABLE and issubclass(type(self.model), TorchForecastingModel):
+ path_tfm = f"{path}.{type(self.model).__name__}.pt"
+ self.model.save(path=path_tfm)
+
+ @staticmethod
+ def load(path: Union[str, os.PathLike, BinaryIO]) -> "ConformalModel":
+ model: ConformalModel = GlobalForecastingModel.load(path)
+
+ if TORCH_AVAILABLE and issubclass(type(model.model), TorchForecastingModel):
+ path_tfm = f"{path}.{type(model.model).__name__}.pt"
+ model.model = TorchForecastingModel.load(path_tfm)
+ return model
+
+ @abstractmethod
+ def _calibrate_interval(
+ self, residuals: np.ndarray
+ ) -> tuple[np.ndarray, np.ndarray]:
+ """Computes the lower and upper calibrated forecast intervals based on residuals.
+
+ Parameters
+ ----------
+ residuals
+ The residuals are expected to have shape (horizon, n components, n historical forecasts * n samples)
+ """
+ pass
+
+ @abstractmethod
+ def _apply_interval(self, pred: np.ndarray, q_hat: tuple[np.ndarray, np.ndarray]):
+ """Applies the calibrated interval to the predicted quantiles. Returns an array with `len(quantiles)`
+ conformalized quantile predictions (lower quantiles, model forecast, upper quantiles) per component.
+
+ E.g. output is `(target1_q1, target1_pred, target1_q2, target2_q1, ...)`
+ """
+ pass
+
+ @property
+ @abstractmethod
+ def _residuals_metric(self) -> tuple[METRIC_TYPE, Optional[dict]]:
+ """Gives the "per time step" metric and optional metric kwargs used to compute residuals /
+ non-conformity scores."""
+ pass
+
+ def _cp_component_names(self, input_series) -> list[str]:
+ """Gives the component names for generated forecasts."""
+ return likelihood_component_names(
+ input_series.components, quantile_names(self.quantiles)
+ )
+
+ @property
+ def output_chunk_length(self) -> Optional[int]:
+ # conformal models can predict any horizon if the calibration set is large enough
+ return None
+
+ @property
+ def output_chunk_shift(self) -> int:
+ return self.model.output_chunk_shift
+
+ @property
+ def _model_encoder_settings(self):
+ raise NotImplementedError(f"not supported by `{self.__class__.__name__}`.")
+
+ @property
+ def extreme_lags(
+ self,
+ ) -> tuple[
+ Optional[int],
+ Optional[int],
+ Optional[int],
+ Optional[int],
+ Optional[int],
+ Optional[int],
+ int,
+ Optional[int],
+ ]:
+ raise NotImplementedError(f"not supported by `{self.__class__.__name__}`.")
+
+ @property
+ def min_train_series_length(self) -> int:
+ raise NotImplementedError(f"not supported by `{self.__class__.__name__}`.")
+
+ @property
+ def min_train_samples(self) -> int:
+ raise NotImplementedError(f"not supported by `{self.__class__.__name__}`.")
+
+ @property
+ def supports_multivariate(self) -> bool:
+ return self.model.supports_multivariate
+
+ @property
+ def supports_past_covariates(self) -> bool:
+ return self.model.supports_past_covariates
+
+ @property
+ def supports_future_covariates(self) -> bool:
+ return self.model.supports_future_covariates
+
+ @property
+ def supports_static_covariates(self) -> bool:
+ return self.model.supports_static_covariates
+
+ @property
+ def supports_sample_weight(self) -> bool:
+ return self.model.supports_sample_weight
+
+ @property
+ def supports_likelihood_parameter_prediction(self) -> bool:
+ return True
+
+ @property
+ def supports_probabilistic_prediction(self) -> bool:
+ return True
+
+ @property
+ def uses_past_covariates(self) -> bool:
+ return self.model.uses_past_covariates
+
+ @property
+ def uses_future_covariates(self) -> bool:
+ return self.model.uses_future_covariates
+
+ @property
+ def uses_static_covariates(self) -> bool:
+ return self.model.uses_static_covariates
+
+ @property
+ def considers_static_covariates(self) -> bool:
+ return self.model.considers_static_covariates
+
+ @property
+ def likelihood(self) -> str:
+ return self._likelihood
+
+
+class ConformalNaiveModel(ConformalModel):
+ def __init__(
+ self,
+ model: GlobalForecastingModel,
+ quantiles: list[float],
+ symmetric: bool = True,
+ cal_length: Optional[int] = None,
+ num_samples: int = 500,
+ random_state: Optional[int] = None,
+ stride_cal: bool = False,
+ ):
+ """Naive Conformal Prediction Model.
+
+ A probabilistic model that adds calibrated intervals around the median forecast from a pre-trained
+ global forecasting model. It does not have to be trained and can generated calibrated forecasts
+ directly using the underlying trained forecasting model. It supports two symmetry modes:
+
+ - `symmetric=True`:
+ - The lower and upper interval bounds are calibrated with the same magnitude.
+ - Non-conformity scores: uses metric `ae()` (see absolute error :func:`~darts.metrics.metrics.ae`) to
+ compute the non-conformity scores on the calibration set.
+ - `symmetric=False`
+ - The lower and upper interval bounds are calibrated separately.
+ - Non-conformity scores: uses metric `err()` (see error :func:`~darts.metrics.metrics.err`) to compute the
+ non-conformity scores on the calibration set for the upper bounds, an `-err()` for the lower bounds.
+
+ Since it is a probabilistic model, you can generate forecasts in two ways (when calling `predict()`,
+ `historical_forecasts()`, ...):
+
+ - Predict the calibrated quantile intervals directly: Pass parameters `predict_likelihood_parameters=True`, and
+ `num_samples=1` to the forecast method.
+ - Predict stochastic samples from the calibrated quantile intervals: Pass parameters
+ `predict_likelihood_parameters=False`, and `num_samples>>1` to the forecast method.
+
+ Conformal models can be applied to any of Darts' global forecasting model, as long as the model has been
+ fitted before. In general the workflow of the models to produce one calibrated forecast/prediction is as
+ follows:
+
+ - Extract a calibration set: The number of calibration examples from the most recent past to use for one
+ conformal prediction can be defined at model creation with parameter `cal_length`. If `stride_cal` is `True`,
+ then the same `stride` from the forecasting methods is applied to the calibration set, and more calibration
+ examples are required (`cal_length * stride` historical forecasts that were generated with `stride=1`).
+ To make your life simpler, we support two modes:
+ - Automatic extraction of the calibration set from the past of your input series (`series`,
+ `past_covariates`, ...). This is the default mode and our predict/forecasting/backtest/.... API is
+ identical to any other forecasting model
+ - Supply a fixed calibration set with parameters `cal_series`, `cal_past_covariates`, ... .
+ - Generate historical forecasts on the calibration set (using the forecasting model)
+ - Compute the errors/non-conformity scores (as defined above) on these historical forecasts
+ - Compute the quantile values from the errors / non-conformity scores (using our desired quantiles set at model
+ creation with parameter `quantiles`).
+ - Compute the conformal prediction: Add the calibrated intervals to the forecasting model's predictions.
+
+ Some notes:
+
+ - When computing historical_forecasts(), backtest(), residuals(), ... the above is applied for each forecast
+ (the forecasting model's historical forecasts are only generated once for efficiency).
+ - For multi-horizon forecasts, the above is applied for each step in the horizon separately
+
+ Parameters
+ ----------
+ model
+ A pre-trained global forecasting model.
+ quantiles
+ A list of quantiles centered around the median `q=0.5` to use. For example quantiles
+ [0.1, 0.2, 0.5, 0.8 0.9] correspond to two intervals with (0.9 - 0.1) = 80%, and (0.8 - 0.2) 60% coverage
+ around the median (model forecast).
+ symmetric
+ Whether to use symmetric non-conformity scores. If `True`, uses metric `ae()` (see
+ :func:`~darts.metrics.metrics.ae`) to compute the non-conformity scores. If `False`, uses metric `-err()`
+ (see :func:`~darts.metrics.metrics.err`) for the lower, and `err()` for the upper quantile interval bound.
+ cal_length
+ The number of past forecast residuals/errors to consider as calibration input for each conformal forecast.
+ If `None`, considers all past residuals.
+ num_samples
+ Number of times a prediction is sampled from the underlying `model` if it is probabilistic. Uses `1` for
+ deterministic models. This is different to the `num_samples` produced by the conformal model which can be
+ set in downstream forecasting tasks.
+ random_state
+ Control the randomness of probabilistic conformal forecasts (sample generation) across different runs.
+ stride_cal
+ Whether to apply the same historical forecast `stride` to the non-conformity scores of the calibration set.
+ """
+ super().__init__(
+ model=model,
+ quantiles=quantiles,
+ symmetric=symmetric,
+ cal_length=cal_length,
+ num_samples=num_samples,
+ random_state=random_state,
+ stride_cal=stride_cal,
+ )
+
+ def _calibrate_interval(
+ self, residuals: np.ndarray
+ ) -> tuple[np.ndarray, np.ndarray]:
+ def q_hat_from_residuals(residuals_):
+ # compute quantiles of shape (forecast horizon, n components, n quantile intervals)
+ return np.quantile(
+ residuals_,
+ q=self.interval_range_sym,
+ method="higher",
+ axis=2,
+ ).transpose((1, 2, 0))
+
+ # residuals shape (horizon, n components, n past forecasts)
+ if self.symmetric:
+ # symmetric (from metric `ae()`)
+ q_hat = q_hat_from_residuals(residuals)
+ return -q_hat, q_hat[:, :, ::-1]
+ else:
+ # asymmetric (from metric `err()`)
+ q_hat = q_hat_from_residuals(
+ np.concatenate([-residuals, residuals], axis=1)
+ )
+ n_comps = residuals.shape[1]
+ return -q_hat[:, :n_comps, :], q_hat[:, n_comps:, ::-1]
+
+ def _apply_interval(self, pred: np.ndarray, q_hat: tuple[np.ndarray, np.ndarray]):
+ # convert stochastic predictions to median
+ if pred.shape[2] != 1:
+ pred = np.expand_dims(np.quantile(pred, 0.5, axis=2), -1)
+ # shape (forecast horizon, n components, n quantiles)
+ pred = np.concatenate([pred + q_hat[0], pred, pred + q_hat[1]], axis=2)
+ # -> (forecast horizon, n components * n quantiles)
+ return pred.reshape(len(pred), -1)
+
+ @property
+ def _residuals_metric(self) -> tuple[METRIC_TYPE, Optional[dict]]:
+ return (metrics.ae if self.symmetric else metrics.err), None
+
+
+class ConformalQRModel(ConformalModel):
+ def __init__(
+ self,
+ model: GlobalForecastingModel,
+ quantiles: list[float],
+ symmetric: bool = True,
+ cal_length: Optional[int] = None,
+ num_samples: int = 500,
+ random_state: Optional[int] = None,
+ stride_cal: bool = False,
+ ):
+ """Conformalized Quantile Regression Model.
+
+ A probabilistic model that calibrates the quantile predictions from a pre-trained probabilistic global
+ forecasting model. It does not have to be trained and can generated calibrated forecasts
+ directly using the underlying trained forecasting model. It supports two symmetry modes:
+
+ - `symmetric=True`:
+ - The lower and upper quantile predictions are calibrated with the same magnitude.
+ - Non-conformity scores: uses metric `incs_qr(symmetric=True)` (see Non-Conformity Score for Quantile
+ Regression :func:`~darts.metrics.metrics.incs_qr`) to compute the non-conformity scores on the calibration
+ set.
+ - `symmetric=False`
+ - The lower and upper quantile predictions are calibrated separately.
+ - Non-conformity scores: uses metric `incs_qr(symmetric=False)` (see Non-Conformity Score for Quantile
+ Regression :func:`~darts.metrics.metrics.incs_qr`) to compute the non-conformity scores for the upper and
+ lower bound separately.
+
+ Since it is a probabilistic model, you can generate forecasts in two ways (when calling `predict()`,
+ `historical_forecasts()`, ...):
+
+ - Predict the calibrated quantile intervals directly: Pass parameters `predict_likelihood_parameters=True`, and
+ `num_samples=1` to the forecast method.
+ - Predict stochastic samples from the calibrated quantile intervals: Pass parameters
+ `predict_likelihood_parameters=False`, and `num_samples>>1` to the forecast method.
+
+ Conformal models can be applied to any of Darts' global forecasting model, as long as the model has been
+ fitted before. In general the workflow of the models to produce one calibrated forecast/prediction is as
+ follows:
+
+ - Extract a calibration set: The number of calibration examples from the most recent past to use for one
+ conformal prediction can be defined at model creation with parameter `cal_length`. If `stride_cal` is `True`,
+ then the same `stride` from the forecasting methods is applied to the calibration set, and more calibration
+ examples are required (`cal_length * stride` historical forecasts that were generated with `stride=1`).
+ To make your life simpler, we support two modes:
+ - Automatic extraction of the calibration set from the past of your input series (`series`,
+ `past_covariates`, ...). This is the default mode and our predict/forecasting/backtest/.... API is
+ identical to any other forecasting model
+ - Supply a fixed calibration set with parameters `cal_series`, `cal_past_covariates`, ... .
+ - Generate historical forecasts (quantile predictions) on the calibration set (using the forecasting model)
+ - Compute the errors/non-conformity scores (as defined above) on these historical quantile predictions
+ - Compute the quantile values from the errors / non-conformity scores (using our desired quantiles set at model
+ creation with parameter `quantiles`).
+ - Compute the conformal prediction: Calibrate the predicted quantiles from the forecasting model's predictions.
+
+ Some notes:
+
+ - When computing historical_forecasts(), backtest(), residuals(), ... the above is applied for each forecast
+ (the forecasting model's historical forecasts are only generated once for efficiency).
+ - For multi-horizon forecasts, the above is applied for each step in the horizon separately
+
+ Parameters
+ ----------
+ model
+ A pre-trained probabilistic global forecasting model using a `likelihood`.
+ quantiles
+ A list of quantiles centered around the median `q=0.5` to use. For example quantiles
+ [0.1, 0.2, 0.5, 0.8 0.9] correspond to two intervals with (0.9 - 0.1) = 80%, and (0.8 - 0.2) 60% coverage
+ around the median (model forecast).
+ symmetric
+ Whether to use symmetric non-conformity scores. If `True`, uses symmetric metric
+ `incs_qr(..., symmetric=True)` (see :func:`~darts.metrics.metrics.incs_qr`) to compute the non-conformity
+ scores. If `False`, uses asymmetric metric `incs_qr(..., symmetric=False)` with individual scores for the
+ lower- and upper quantile interval bounds.
+ cal_length
+ The number of past forecast residuals/errors to consider as calibration input for each conformal forecast.
+ If `None`, considers all past residuals.
+ num_samples
+ Number of times a prediction is sampled from the underlying `model` if it is probabilistic. Uses `1` for
+ deterministic models. This is different to the `num_samples` produced by the conformal model which can be
+ set in downstream forecasting tasks.
+ random_state
+ Control the randomness of probabilistic conformal forecasts (sample generation) across different runs.
+ stride_cal
+ Whether to apply the same historical forecast `stride` to the non-conformity scores of the calibration set.
+ """
+ if not model.supports_probabilistic_prediction:
+ raise_log(
+ ValueError(
+ "`model` must must support probabilistic forecasting. Consider using a `likelihood` at "
+ "forecasting model creation, or use another conformal model."
+ ),
+ logger=logger,
+ )
+ super().__init__(
+ model=model,
+ quantiles=quantiles,
+ symmetric=symmetric,
+ cal_length=cal_length,
+ num_samples=num_samples,
+ random_state=random_state,
+ stride_cal=stride_cal,
+ )
+
+ def _calibrate_interval(
+ self, residuals: np.ndarray
+ ) -> tuple[np.ndarray, np.ndarray]:
+ n_comps = residuals.shape[1] // (
+ len(self.interval_range) * (1 + int(not self.symmetric))
+ )
+ n_intervals = len(self.interval_range)
+
+ def q_hat_from_residuals(residuals_):
+ # TODO: is there a more efficient way?
+ # compute quantiles with shape (horizon, n components, n quantile intervals)
+ # over all past residuals
+ q_hat_tmp = np.quantile(
+ residuals_, q=self.interval_range_sym, method="higher", axis=2
+ ).transpose((1, 2, 0))
+ q_hat_ = np.empty((len(residuals_), n_comps, n_intervals))
+ for i in range(n_intervals):
+ for c in range(n_comps):
+ q_hat_[:, c, i] = q_hat_tmp[:, i + c * n_intervals, i]
+ return q_hat_
+
+ if self.symmetric:
+ # symmetric has one nc-score per interval (from metric `incs_qr(symmetric=True)`)
+ # residuals shape (horizon, n components * n intervals, n past forecasts)
+ q_hat = q_hat_from_residuals(residuals)
+ return -q_hat, q_hat[:, :, ::-1]
+ else:
+ # asymmetric has two nc-score per interval (for lower and upper quantiles, from metric
+ # `incs_qe(symmetric=False)`)
+ # lower and upper residuals are concatenated along axis=1;
+ # residuals shape (horizon, n components * n intervals * 2, n past forecasts)
+ half_idx = residuals.shape[1] // 2
+ q_hat_lo = q_hat_from_residuals(residuals[:, :half_idx])
+ q_hat_hi = q_hat_from_residuals(residuals[:, half_idx:])
+ return -q_hat_lo, q_hat_hi[:, :, ::-1]
+
+ def _apply_interval(self, pred: np.ndarray, q_hat: tuple[np.ndarray, np.ndarray]):
+ # get quantile predictions with shape (n times, n components, n quantiles)
+ pred = np.quantile(pred, self.quantiles, axis=2).transpose((1, 2, 0))
+ # shape (forecast horizon, n components, n quantiles)
+ pred = np.concatenate(
+ [
+ pred[:, :, : self.idx_median] + q_hat[0], # lower quantiles
+ pred[:, :, self.idx_median : self.idx_median + 1], # model forecast
+ pred[:, :, self.idx_median + 1 :] + q_hat[1], # upper quantiles
+ ],
+ axis=2,
+ )
+ # -> (forecast horizon, n components * n quantiles)
+ return pred.reshape(len(pred), -1)
+
+ @property
+ def _residuals_metric(self) -> tuple[METRIC_TYPE, Optional[dict]]:
+ return metrics.incs_qr, {
+ "q_interval": self.q_interval,
+ "symmetric": self.symmetric,
+ }
diff --git a/darts/models/forecasting/ensemble_model.py b/darts/models/forecasting/ensemble_model.py
index c585efd6c3..72187f8334 100644
--- a/darts/models/forecasting/ensemble_model.py
+++ b/darts/models/forecasting/ensemble_model.py
@@ -239,9 +239,10 @@ def _stack_ts_multiseq(self, predictions_list):
# stacks multiple sequences of timeseries elementwise
return [self._stack_ts_seq(ts_list) for ts_list in zip(*predictions_list)]
+ @property
def _model_encoder_settings(self):
raise NotImplementedError(
- "Encoders are not supported by EnsembleModels. Instead add encoder to the underlying `forecasting_models`."
+ "Encoders are not supported by EnsembleModels. Instead add encoders to the underlying `forecasting_models`."
)
def _make_multiple_predictions(
@@ -436,15 +437,6 @@ def save(
@staticmethod
def load(path: Union[str, os.PathLike, BinaryIO]) -> "EnsembleModel":
- """
- Loads the ensemble model from a given path or file handle.
-
- Parameters
- ----------
- path
- Path or file handle from which to load the ensemble model.
- """
-
model: EnsembleModel = GlobalForecastingModel.load(path)
for i, m in enumerate(model.forecasting_models):
diff --git a/darts/models/forecasting/forecasting_model.py b/darts/models/forecasting/forecasting_model.py
index c191fd1e3d..315be3c9e8 100644
--- a/darts/models/forecasting/forecasting_model.py
+++ b/darts/models/forecasting/forecasting_model.py
@@ -37,10 +37,12 @@
from darts.utils import _build_tqdm_iterator, _parallel_apply, _with_sanity_checks
from darts.utils.historical_forecasts.utils import (
_adjust_historical_forecasts_time_index,
+ _extend_series_for_overlap_end,
_get_historical_forecast_predict_index,
_get_historical_forecast_train_index,
_historical_forecasts_general_checks,
_historical_forecasts_sanitize_kwargs,
+ _process_historical_forecast_for_backtest,
_reconciliate_historical_time_indices,
)
from darts.utils.timeseries_generation import (
@@ -327,8 +329,7 @@ def predict(
n
Forecast horizon - the number of time steps after the end of the series for which to produce predictions.
num_samples
- Number of times a prediction is sampled from a probabilistic model. Should be left set to 1
- for deterministic models.
+ Number of times a prediction is sampled from a probabilistic model. Must be `1` for deterministic models.
verbose
Optionally, set the prediction verbosity. Not effective for all models.
show_warnings
@@ -348,8 +349,12 @@ def predict(
),
logger,
)
-
- if self.output_chunk_shift and n > self.output_chunk_length:
+ is_autoregression = (
+ False
+ if self.output_chunk_length is None
+ else (n > self.output_chunk_length)
+ )
+ if self.output_chunk_shift and is_autoregression:
raise_log(
ValueError(
"Cannot perform auto-regression `(n > output_chunk_length)` with a model that uses a "
@@ -633,11 +638,11 @@ def historical_forecasts(
series: Union[TimeSeries, Sequence[TimeSeries]],
past_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
future_covariates: Optional[Union[TimeSeries, Sequence[TimeSeries]]] = None,
+ forecast_horizon: int = 1,
num_samples: int = 1,
train_length: Optional[int] = None,
start: Optional[Union[pd.Timestamp, float, int]] = None,
start_format: Literal["position", "value"] = "value",
- forecast_horizon: int = 1,
stride: int = 1,
retrain: Union[bool, int, Callable[..., bool]] = True,
overlap_end: bool = False,
@@ -650,42 +655,60 @@ def historical_forecasts(
predict_kwargs: Optional[dict[str, Any]] = None,
sample_weight: Optional[Union[TimeSeries, Sequence[TimeSeries], str]] = None,
) -> Union[TimeSeries, list[TimeSeries], list[list[TimeSeries]]]:
- """Compute the historical forecasts that would have been obtained by this model on
- (potentially multiple) `series`.
-
- This method repeatedly builds a training set: either expanding from the beginning of `series` or moving with
- a fixed length `train_length`. It trains the model on the training set, emits a forecast of length equal to
- forecast_horizon, and then moves the end of the training set forward by `stride` time steps.
-
- By default, this method will return one (or a sequence of) single time series made up of
- the last point of each historical forecast.
- This time series will thus have a frequency of ``series.freq * stride``.
- If `last_points_only` is set to `False`, it will instead return one (or a sequence of) list of the
- historical forecasts series.
-
- By default, this method always re-trains the models on the entire available history, corresponding to an
- expanding window strategy. If `retrain` is set to `False`, the model must have been fit before. This is not
- supported by all models.
+ """Generates historical forecasts by simulating predictions at various points in time throughout the history of
+ the provided (potentially multiple) `series`. This process involves retrospectively applying the model to
+ different time steps, as if the forecasts were made in real-time at those specific moments. This allows for an
+ evaluation of the model's performance over the entire duration of the series, providing insights into its
+ predictive accuracy and robustness across different historical periods.
+
+ There are two main modes for this method:
+
+ - Re-training Mode (Default, `retrain=True`): The model is re-trained at each step of the simulation, and
+ generates a forecast using the updated model.
+ - Pre-trained Mode (`retrain=False`): The forecasts are generated at each step of the simulation without
+ re-training. It is only supported for pre-trained global forecasting models. This mode is significantly
+ faster as it skips the re-training step.
+
+ By choosing the appropriate mode, you can balance between computational efficiency and the need for up-to-date
+ model training.
+
+ **Re-training Mode:** This mode repeatedly builds a training set by either expanding from the beginning of
+ the `series` or by using a fixed-length `train_length` (the start point can also be configured with `start`
+ and `start_format`). The model is then trained on this training set, and a forecast of length `forecast_horizon`
+ is generated. Subsequently, the end of the training set is moved forward by `stride` time steps, and the process
+ is repeated.
+
+ **Pre-trained Mode:** This mode is only supported for pre-trained global forecasting models. It uses the same
+ simulation steps as in the *Re-training Mode* (ignoring `train_length`), but generates the forecasts directly
+ without re-training.
+
+ By default, with `last_points_only=True`, this method returns a single time series (or a sequence of time
+ series) composed of the last point from each historical forecast. This time series will thus have a frequency of
+ `series.freq * stride`.
+ If `last_points_only=False`, it will instead return a list (or a sequence of lists) of the full historical
+ forecast series each with frequency `series.freq`.
Parameters
----------
series
- The (or a sequence of) target time series used to successively train and compute the historical forecasts.
+ A (sequence of) target time series used to successively train (if `retrain` is not ``False``) and compute
+ the historical forecasts.
past_covariates
- Optionally, one (or a sequence of) past-observed covariate series. This applies only if the model
- supports past covariates.
+ Optionally, a (sequence of) past-observed covariate time series for every input time series in `series`.
+ This applies only if the model supports past covariates.
future_covariates
- Optionally, one (or a sequence of) of future-known covariate series. This applies only if the model
- supports future covariates.
+ Optionally, a (sequence of) future-known covariate time series for every input time series in `series`.
+ This applies only if the model supports future covariates.
+ forecast_horizon
+ The forecast horizon for the predictions.
num_samples
- Number of times a prediction is sampled from a probabilistic model. Use values `>1` only for probabilistic
+ Number of times a prediction is sampled from a probabilistic model. Use values ``>1`` only for probabilistic
models.
train_length
- Number of time steps in our training set (size of backtesting window to train on). Only effective when
- `retrain` is not ``False``. Default is set to `train_length=None` where it takes all available time steps
- up until prediction time, otherwise the moving window strategy is used. If larger than the number of time
- steps available, all steps up until prediction time are used, as in default case. Needs to be at least
- `min_train_series_length`.
+ Optionally, use a fixed length / number of time steps for every constructed training set (rolling window
+ mode). Only effective when `retrain` is not ``False``. The default is ``None``, where it uses all time
+ steps up until the prediction time (expanding window mode). If larger than the number of available time
+ steps, uses the expanding mode. Needs to be at least `min_train_series_length`.
start
Optionally, the first point in time at which a prediction is computed. This parameter supports:
``float``, ``int``, ``pandas.Timestamp``, and ``None``.
@@ -699,7 +722,7 @@ def historical_forecasts(
- the first predictable point if `retrain` is ``False``, or `retrain` is a Callable and the first
predictable point is earlier than the first trainable point.
- the first trainable point if `retrain` is ``True`` or ``int`` (given `train_length`),
- or `retrain` is a Callable and the first trainable point is earlier than the first predictable point.
+ or `retrain` is a ``Callable`` and the first trainable point is earlier than the first predictable point.
- the first trainable point (given `train_length`) otherwise
Note: If `start` is not within the trainable / forecastable points, uses the closest valid start point that
@@ -709,21 +732,18 @@ def historical_forecasts(
Note: If `start` is outside the possible historical forecasting times, will ignore the parameter
(default behavior with ``None``) and start at the first trainable/predictable point.
start_format
- Defines the `start` format. Only effective when `start` is an integer and `series` is indexed with a
- `pd.RangeIndex`.
- If set to 'position', `start` corresponds to the index position of the first predicted point and can range
- from `(-len(series), len(series) - 1)`.
- If set to 'value', `start` corresponds to the index value/label of the first predicted point. Will raise
- an error if the value is not in `series`' index. Default: ``'value'``
- forecast_horizon
- The forecast horizon for the predictions.
+ Defines the `start` format.
+ If set to ``'position'``, `start` corresponds to the index position of the first predicted point and can
+ range from `(-len(series), len(series) - 1)`.
+ If set to ``'value'``, `start` corresponds to the index value/label of the first predicted point. Will raise
+ an error if the value is not in `series`' index. Default: ``'value'``.
stride
The number of time steps between two consecutive predictions.
retrain
Whether and/or on which condition to retrain the model before predicting.
- This parameter supports 3 different datatypes: ``bool``, (positive) ``int``, and
- ``Callable`` (returning a ``bool``).
- In the case of ``bool``: retrain the model at each step (`True`), or never retrains the model (`False`).
+ This parameter supports 3 different types: ``bool``, (positive) ``int``, and ``Callable`` (returning a
+ ``bool``).
+ In the case of ``bool``: retrain the model at each step (`True`), or never retrain the model (`False`).
In the case of ``int``: the model is retrained every `retrain` iterations.
In the case of ``Callable``: the model is retrained whenever callable returns `True`.
The callable must have the following positional arguments:
@@ -732,35 +752,35 @@ def historical_forecasts(
- `pred_time` (pd.Timestamp or int): timestamp of forecast time (end of the training series)
- `train_series` (TimeSeries): train series up to `pred_time`
- `past_covariates` (TimeSeries): past_covariates series up to `pred_time`
- - `future_covariates` (TimeSeries): future_covariates series up
- to `min(pred_time + series.freq * forecast_horizon, series.end_time())`
+ - `future_covariates` (TimeSeries): future_covariates series up to `min(pred_time + series.freq *
+ forecast_horizon, series.end_time())`
Note: if any optional `*_covariates` are not passed to `historical_forecast`, ``None`` will be passed
to the corresponding retrain function argument.
- Note: some models do require being retrained every time and do not support anything other
- than `retrain=True`.
+ Note: some models require being retrained every time and do not support anything other than
+ `retrain=True`.
overlap_end
Whether the returned forecasts can go beyond the series' end or not.
last_points_only
- Whether to retain only the last point of each historical forecast.
- If set to `True`, the method returns a single ``TimeSeries`` containing the successive point forecasts.
+ Whether to return only the last point of each historical forecast. If set to ``True``, the method returns a
+ single ``TimeSeries`` (for each time series in `series`) containing the successive point forecasts.
Otherwise, returns a list of historical ``TimeSeries`` forecasts.
verbose
- Whether to print progress.
+ Whether to print the progress.
show_warnings
Whether to show warnings related to historical forecasts optimization, or parameters `start` and
`train_length`.
predict_likelihood_parameters
- If set to `True`, the model predict the parameters of its Likelihood parameters instead of the target. Only
+ If set to `True`, the model predicts the parameters of its `likelihood` instead of the target. Only
supported for probabilistic models with a likelihood, `num_samples = 1` and `n<=output_chunk_length`.
- Default: ``False``
+ Default: ``False``.
enable_optimization
- Whether to use the optimized version of historical_forecasts when supported and available.
+ Whether to use the optimized version of `historical_forecasts` when supported and available.
Default: ``True``.
fit_kwargs
- Additional arguments passed to the model `fit()` method.
+ Optionally, some additional arguments passed to the model `fit()` method.
predict_kwargs
- Additional arguments passed to the model `predict()` method.
+ Optionally, some additional arguments passed to the model `predict()` method.
sample_weight
Optionally, some sample weights to apply to the target `series` labels for training. Only effective when
`retrain` is not ``False``. They are applied per observation, per label (each step in
@@ -938,7 +958,9 @@ def retrain_func(
# (otherwise use tqdm on the inner loop).
outer_iterator = series
else:
- outer_iterator = _build_tqdm_iterator(series, verbose)
+ outer_iterator = _build_tqdm_iterator(
+ series, verbose, total=len(series), desc="historical forecasts"
+ )
# deactivate the warning after displaying it once if show_warnings is True
show_predict_warnings = show_warnings
@@ -1034,7 +1056,10 @@ def retrain_func(
if len(series) == 1:
# Only use tqdm if there's no outer loop
iterator = _build_tqdm_iterator(
- historical_forecasts_time_index[::stride], verbose
+ historical_forecasts_time_index[::stride],
+ verbose,
+ total=(len(historical_forecasts_time_index) - 1) // stride + 1,
+ desc="historical forecasts",
)
else:
iterator = historical_forecasts_time_index[::stride]
@@ -1184,11 +1209,11 @@ def backtest(
historical_forecasts: Optional[
Union[TimeSeries, Sequence[TimeSeries], Sequence[Sequence[TimeSeries]]]
] = None,
+ forecast_horizon: int = 1,
num_samples: int = 1,
train_length: Optional[int] = None,
start: Optional[Union[pd.Timestamp, float, int]] = None,
start_format: Literal["position", "value"] = "value",
- forecast_horizon: int = 1,
stride: int = 1,
retrain: Union[bool, int, Callable[..., bool]] = True,
overlap_end: bool = False,
@@ -1204,51 +1229,49 @@ def backtest(
predict_kwargs: Optional[dict[str, Any]] = None,
sample_weight: Optional[Union[TimeSeries, Sequence[TimeSeries], str]] = None,
) -> Union[float, np.ndarray, list[float], list[np.ndarray]]:
- """Compute error values that the model would have produced when
- used on (potentially multiple) `series`.
+ """Compute error values that the model produced for historical forecasts on (potentially multiple) `series`.
- If `historical_forecasts` are provided, the metric (given by the `metric` function) is evaluated directly on
- the forecast and the actual values. The same `series` must be passed that was used to generate the historical
- forecasts. Otherwise, it repeatedly builds a training set: either expanding from the
- beginning of `series` or moving with a fixed length `train_length`. It trains the current model on the
- training set, emits a forecast of length equal to `forecast_horizon`, and then moves the end of the training
- set forward by `stride` time steps. The metric is then evaluated on the forecast and the actual values.
- Finally, the method returns a `reduction` (the mean by default) of all these metric scores.
+ If `historical_forecasts` are provided, the metric(s) (given by the `metric` function) is evaluated directly on
+ all forecasts and actual values. The same `series` and `last_points_only` value must be passed that were used
+ to generate the historical forecasts. Finally, the method returns an optional `reduction` (the mean by default)
+ of all these metric scores.
- By default, this method uses each historical forecast (whole) to compute error scores.
- If `last_points_only` is set to `True`, it will use only the last point of each historical
- forecast. In this case, no reduction is used.
+ If `historical_forecasts` is ``None``, it first generates the historical forecasts with the parameters given
+ below (see :meth:`ForecastingModel.historical_forecasts()
+ ` for more info) and then
+ evaluates as described above.
- By default, this method always re-trains the models on the entire available history, corresponding to an
- expanding window strategy. If `retrain` is set to `False` (useful for models for which training might be
- time-consuming, such as deep learning models), the trained model will be used directly to emit the forecasts.
+ The metric(s) can be further customized `metric_kwargs` (e.g. control the aggregation over components, time
+ steps, multiple series, other required arguments such as `q` for quantile metrics, ...).
Parameters
----------
series
- The (or a sequence of) target time series used to successively train and evaluate the historical forecasts.
+ A (sequence of) target time series used to successively train (if `retrain` is not ``False``) and compute
+ the historical forecasts.
past_covariates
- Optionally, one (or a sequence of) past-observed covariate series. This applies only if the model
- supports past covariates.
+ Optionally, a (sequence of) past-observed covariate time series for every input time series in `series`.
+ This applies only if the model supports past covariates.
future_covariates
- Optionally, one (or a sequence of) future-known covariate series. This applies only if the model
- supports future covariates.
+ Optionally, a (sequence of) future-known covariate time series for every input time series in `series`.
+ This applies only if the model supports future covariates.
historical_forecasts
Optionally, the (or a sequence of / a sequence of sequences of) historical forecasts time series to be
evaluated. Corresponds to the output of :meth:`historical_forecasts()
`. The same `series` and
- `last_points_only` values must be passed that were used to generate the historical forecasts.
- If provided, will skip historical forecasting and ignore all parameters except `series`,
- `last_points_only`, `metric`, and `reduction`.
+ `last_points_only` values must be passed that were used to generate the historical forecasts. If provided,
+ will skip historical forecasting and ignore all parameters except `series`, `last_points_only`, `metric`,
+ and `reduction`.
+ forecast_horizon
+ The forecast horizon for the predictions.
num_samples
- Number of times a prediction is sampled from a probabilistic model. Use values `>1` only for probabilistic
+ Number of times a prediction is sampled from a probabilistic model. Use values ``>1`` only for probabilistic
models.
train_length
- Number of time steps in our training set (size of backtesting window to train on). Only effective when
- `retrain` is not ``False``. Default is set to `train_length=None` where it takes all available time steps
- up until prediction time, otherwise the moving window strategy is used. If larger than the number of time
- steps available, all steps up until prediction time are used, as in default case. Needs to be at least
- `min_train_series_length`.
+ Optionally, use a fixed length / number of time steps for every constructed training set (rolling window
+ mode). Only effective when `retrain` is not ``False``. The default is ``None``, where it uses all time
+ steps up until the prediction time (expanding window mode). If larger than the number of available time
+ steps, uses the expanding mode. Needs to be at least `min_train_series_length`.
start
Optionally, the first point in time at which a prediction is computed. This parameter supports:
``float``, ``int``, ``pandas.Timestamp``, and ``None``.
@@ -1262,7 +1285,7 @@ def backtest(
- the first predictable point if `retrain` is ``False``, or `retrain` is a Callable and the first
predictable point is earlier than the first trainable point.
- the first trainable point if `retrain` is ``True`` or ``int`` (given `train_length`),
- or `retrain` is a Callable and the first trainable point is earlier than the first predictable point.
+ or `retrain` is a ``Callable`` and the first trainable point is earlier than the first predictable point.
- the first trainable point (given `train_length`) otherwise
Note: If `start` is not within the trainable / forecastable points, uses the closest valid start point that
@@ -1272,40 +1295,39 @@ def backtest(
Note: If `start` is outside the possible historical forecasting times, will ignore the parameter
(default behavior with ``None``) and start at the first trainable/predictable point.
start_format
- Defines the `start` format. Only effective when `start` is an integer and `series` is indexed with a
- `pd.RangeIndex`.
- If set to 'position', `start` corresponds to the index position of the first predicted point and can range
- from `(-len(series), len(series) - 1)`.
- If set to 'value', `start` corresponds to the index value/label of the first predicted point. Will raise
- an error if the value is not in `series`' index. Default: ``'value'``
- forecast_horizon
- The forecast horizon for the point predictions.
+ Defines the `start` format.
+ If set to ``'position'``, `start` corresponds to the index position of the first predicted point and can
+ range from `(-len(series), len(series) - 1)`.
+ If set to ``'value'``, `start` corresponds to the index value/label of the first predicted point. Will raise
+ an error if the value is not in `series`' index. Default: ``'value'``.
stride
The number of time steps between two consecutive predictions.
retrain
Whether and/or on which condition to retrain the model before predicting.
- This parameter supports 3 different datatypes: ``bool``, (positive) ``int``, and
- ``Callable`` (returning a ``bool``).
- In the case of ``bool``: retrain the model at each step (`True`), or never retrains the model (`False`).
+ This parameter supports 3 different types: ``bool``, (positive) ``int``, and ``Callable`` (returning a
+ ``bool``).
+ In the case of ``bool``: retrain the model at each step (`True`), or never retrain the model (`False`).
In the case of ``int``: the model is retrained every `retrain` iterations.
In the case of ``Callable``: the model is retrained whenever callable returns `True`.
The callable must have the following positional arguments:
- - `counter` (int): current `retrain` iteration
- - `pred_time` (pd.Timestamp or int): timestamp of forecast time (end of the training series)
- - `train_series` (TimeSeries): train series up to `pred_time`
- - `past_covariates` (TimeSeries): past_covariates series up to `pred_time`
- - `future_covariates` (TimeSeries): future_covariates series up
- to `min(pred_time + series.freq * forecast_horizon, series.end_time())`
+ - `counter` (int): current `retrain` iteration
+ - `pred_time` (pd.Timestamp or int): timestamp of forecast time (end of the training series)
+ - `train_series` (TimeSeries): train series up to `pred_time`
+ - `past_covariates` (TimeSeries): past_covariates series up to `pred_time`
+ - `future_covariates` (TimeSeries): future_covariates series up to `min(pred_time + series.freq *
+ forecast_horizon, series.end_time())`
Note: if any optional `*_covariates` are not passed to `historical_forecast`, ``None`` will be passed
to the corresponding retrain function argument.
- Note: some models do require being retrained every time and do not support anything other
- than `retrain=True`.
+ Note: some models require being retrained every time and do not support anything other than
+ `retrain=True`.
overlap_end
Whether the returned forecasts can go beyond the series' end or not.
last_points_only
- Whether to use the whole historical forecasts or only the last point of each forecast to compute the error.
+ Whether to return only the last point of each historical forecast. If set to ``True``, the method returns a
+ single ``TimeSeries`` (for each time series in `series`) containing the successive point forecasts.
+ Otherwise, returns a list of historical ``TimeSeries`` forecasts.
metric
A metric function or a list of metric functions. Each metric must either be a Darts metric (see `here
`_), or a custom metric that has an
@@ -1318,15 +1340,16 @@ def backtest(
If explicitly set to `None`, the method will return a list of the individual error scores instead.
Set to ``np.mean`` by default.
verbose
- Whether to print progress.
+ Whether to print the progress.
show_warnings
- Whether to show warnings related to parameters `start`, and `train_length`.
+ Whether to show warnings related to historical forecasts optimization, or parameters `start` and
+ `train_length`.
predict_likelihood_parameters
- If set to `True`, the model predict the parameters of its Likelihood parameters instead of the target. Only
- supported for probabilistic models with `likelihood="quantile"`, `num_samples = 1` and
- `n<=output_chunk_length`. Default: ``False``.
+ If set to `True`, the model predicts the parameters of its `likelihood` instead of the target. Only
+ supported for probabilistic models with a likelihood, `num_samples = 1` and `n<=output_chunk_length`.
+ Default: ``False``.
enable_optimization
- Whether to use the optimized version of historical_forecasts when supported and available.
+ Whether to use the optimized version of `historical_forecasts` when supported and available.
Default: ``True``.
metric_kwargs
Additional arguments passed to `metric()`, such as `'n_jobs'` for parallelization, `'component_reduction'`
@@ -1334,9 +1357,9 @@ def backtest(
each metric separately and only if they are present in the corresponding metric signature. Parameter
`'insample'` for scaled metrics (e.g. mase`, `rmsse`, ...) is ignored, as it is handled internally.
fit_kwargs
- Additional arguments passed to the model `fit()` method.
+ Optionally, some additional arguments passed to the model `fit()` method.
predict_kwargs
- Additional arguments passed to the model `predict()` method.
+ Optionally, some additional arguments passed to the model `predict()` method.
sample_weight
Optionally, some sample weights to apply to the target `series` labels for training. Only effective when
`retrain` is not ``False``. They are applied per observation, per label (each step in
@@ -1416,58 +1439,13 @@ def backtest(
# remember input series type
series_seq_type = get_series_seq_type(series)
- series = series2seq(series)
-
- # check that `historical_forecasts` have correct type
- expected_seq_type = None
- forecast_seq_type = get_series_seq_type(historical_forecasts)
- if last_points_only and not series_seq_type == forecast_seq_type:
- # lpo=True -> fc sequence type must be the same
- expected_seq_type = series_seq_type
- elif not last_points_only and forecast_seq_type != series_seq_type + 1:
- # lpo=False -> fc sequence type must be one order higher
- expected_seq_type = series_seq_type + 1
-
- if expected_seq_type is not None:
- raise_log(
- ValueError(
- f"Expected `historical_forecasts` of type {expected_seq_type} "
- f"with `last_points_only={last_points_only}` and `series` of type "
- f"{series_seq_type}. However, received `historical_forecasts` of type "
- f"{forecast_seq_type}. Make sure to pass the same `last_points_only` "
- f"value that was used to generate the historical forecasts."
- ),
- logger=logger,
- )
-
- # we must wrap each fc in a list if `last_points_only=True`
- nested = last_points_only and forecast_seq_type == SeriesType.SEQ
- historical_forecasts = series2seq(
- historical_forecasts, seq_type_out=SeriesType.SEQ_SEQ, nested=nested
+ # validate historical forecasts and covert to multiple series with multiple forecasts case
+ series, historical_forecasts = _process_historical_forecast_for_backtest(
+ series=series,
+ historical_forecasts=historical_forecasts,
+ last_points_only=last_points_only,
)
- # check that the number of series-specific forecasts corresponds to the
- # number of series in `series`
- if len(series) != len(historical_forecasts):
- error_msg = (
- f"Mismatch between the number of series-specific `historical_forecasts` "
- f"(n={len(historical_forecasts)}) and the number of `TimeSeries` in `series` "
- f"(n={len(series)}). For `last_points_only={last_points_only}`, expected "
- )
- expected_seq_type = (
- series_seq_type if last_points_only else series_seq_type + 1
- )
- if expected_seq_type == SeriesType.SINGLE:
- error_msg += (
- f"a single `historical_forecasts` of type {expected_seq_type}."
- )
- else:
- error_msg += f"`historical_forecasts` of type {expected_seq_type} with length n={len(series)}."
- raise_log(
- ValueError(error_msg),
- logger=logger,
- )
-
# we have multiple forecasts per series: rearrange forecasts to call each metric only once;
# flatten historical forecasts, get matching target series index, remember cumulative target lengths
# for later reshaping back to original
@@ -1662,7 +1640,7 @@ def gridsearch(
A reduction function (mapping array to float) describing how to aggregate the errors obtained
on the different validation series when backtesting. By default it'll compute the mean of errors.
verbose
- Whether to print progress.
+ Whether to print the progress.
n_jobs
The number of jobs to run in parallel. Parallel jobs are created only when there are two or more parameters
combinations to evaluate. Each job will instantiate, train, and evaluate a different instance of the model.
@@ -1755,7 +1733,10 @@ def gridsearch(
# iterate through all combinations of the provided parameters and choose the best one
iterator = _build_tqdm_iterator(
- zip(params_cross_product), verbose, total=len(params_cross_product)
+ zip(params_cross_product),
+ verbose,
+ total=len(params_cross_product),
+ desc="gridsearch",
)
def _evaluate_combination(param_combination) -> float:
@@ -1843,13 +1824,14 @@ def residuals(
historical_forecasts: Optional[
Union[TimeSeries, Sequence[TimeSeries], Sequence[Sequence[TimeSeries]]]
] = None,
+ forecast_horizon: int = 1,
num_samples: int = 1,
train_length: Optional[int] = None,
start: Optional[Union[pd.Timestamp, float, int]] = None,
start_format: Literal["position", "value"] = "value",
- forecast_horizon: int = 1,
stride: int = 1,
retrain: Union[bool, int, Callable[..., bool]] = True,
+ overlap_end: bool = False,
last_points_only: bool = True,
metric: METRIC_TYPE = metrics.err,
verbose: bool = False,
@@ -1859,10 +1841,10 @@ def residuals(
metric_kwargs: Optional[dict[str, Any]] = None,
fit_kwargs: Optional[dict[str, Any]] = None,
predict_kwargs: Optional[dict[str, Any]] = None,
- values_only: bool = False,
sample_weight: Optional[Union[TimeSeries, Sequence[TimeSeries], str]] = None,
+ values_only: bool = False,
) -> Union[TimeSeries, list[TimeSeries], list[list[TimeSeries]]]:
- """Compute the residuals produced by this model on a (or sequence of) `TimeSeries`.
+ """Compute the residuals that the model produced for historical forecasts on (potentially multiple) `series`.
This function computes the difference (or one of Darts' "per time step" metrics) between the actual
observations from `series` and the fitted values obtained by training the model on `series` (or using a
@@ -1871,7 +1853,7 @@ def residuals(
In sequence this method performs:
- - compute historical forecasts for each series or use pre-computed `historical_forecasts` (see
+ - use pre-computed `historical_forecasts` or compute historical forecasts for each series (see
:meth:`~darts.models.forecasting.forecasting_model.ForecastingModel.historical_forecasts` for more details).
How the historical forecasts are generated can be configured with parameters `num_samples`, `train_length`,
`start`, `start_format`, `forecast_horizon`, `stride`, `retrain`, `last_points_only`, `fit_kwargs`, and
@@ -1879,7 +1861,7 @@ def residuals(
- compute a backtest using a "per time step" `metric` between the historical forecasts and `series` per
component/column and time step (see
:meth:`~darts.models.forecasting.forecasting_model.ForecastingModel.backtest` for more details). By default,
- uses the residuals :func:`~darts.metrics.metrics.err` as a `metric`.
+ uses the residuals :func:`~darts.metrics.metrics.err` (error) as a `metric`.
- create and return `TimeSeries` (or simply a np.ndarray with `values_only=True`) with the time index from
historical forecasts, and values from the metrics per component and time step.
@@ -1889,13 +1871,14 @@ def residuals(
Parameters
----------
series
- The univariate TimeSeries instance which the residuals will be computed for.
+ A (sequence of) target time series used to successively train (if `retrain` is not ``False``) and compute
+ the historical forecasts.
past_covariates
- One or several past-observed covariate time series.
+ Optionally, a (sequence of) past-observed covariate time series for every input time series in `series`.
+ This applies only if the model supports past covariates.
future_covariates
- One or several future-known covariate time series.
- forecast_horizon
- The forecasting horizon used to predict each fitted value.
+ Optionally, a (sequence of) future-known covariate time series for every input time series in `series`.
+ This applies only if the model supports future covariates.
historical_forecasts
Optionally, the (or a sequence of / a sequence of sequences of) historical forecasts time series to be
evaluated. Corresponds to the output of :meth:`historical_forecasts()
@@ -1903,15 +1886,16 @@ def residuals(
`last_points_only` values must be passed that were used to generate the historical forecasts. If provided,
will skip historical forecasting and ignore all parameters except `series`, `last_points_only`, `metric`,
and `reduction`.
+ forecast_horizon
+ The forecast horizon for the predictions.
num_samples
- Number of times a prediction is sampled from a probabilistic model. Use values `>1` only for probabilistic
+ Number of times a prediction is sampled from a probabilistic model. Use values ``>1`` only for probabilistic
models.
train_length
- Number of time steps in our training set (size of backtesting window to train on). Only effective when
- `retrain` is not ``False``. Default is set to `train_length=None` where it takes all available time steps
- up until prediction time, otherwise the moving window strategy is used. If larger than the number of time
- steps available, all steps up until prediction time are used, as in default case. Needs to be at least
- `min_train_series_length`.
+ Optionally, use a fixed length / number of time steps for every constructed training set (rolling window
+ mode). Only effective when `retrain` is not ``False``. The default is ``None``, where it uses all time
+ steps up until the prediction time (expanding window mode). If larger than the number of available time
+ steps, uses the expanding mode. Needs to be at least `min_train_series_length`.
start
Optionally, the first point in time at which a prediction is computed. This parameter supports:
``float``, ``int``, ``pandas.Timestamp``, and ``None``.
@@ -1925,7 +1909,7 @@ def residuals(
- the first predictable point if `retrain` is ``False``, or `retrain` is a Callable and the first
predictable point is earlier than the first trainable point.
- the first trainable point if `retrain` is ``True`` or ``int`` (given `train_length`),
- or `retrain` is a Callable and the first trainable point is earlier than the first predictable point.
+ or `retrain` is a ``Callable`` and the first trainable point is earlier than the first predictable point.
- the first trainable point (given `train_length`) otherwise
Note: If `start` is not within the trainable / forecastable points, uses the closest valid start point that
@@ -1935,38 +1919,39 @@ def residuals(
Note: If `start` is outside the possible historical forecasting times, will ignore the parameter
(default behavior with ``None``) and start at the first trainable/predictable point.
start_format
- Defines the `start` format. Only effective when `start` is an integer and `series` is indexed with a
- `pd.RangeIndex`.
- If set to 'position', `start` corresponds to the index position of the first predicted point and can range
- from `(-len(series), len(series) - 1)`.
- If set to 'value', `start` corresponds to the index value/label of the first predicted point. Will raise
- an error if the value is not in `series`' index. Default: ``'value'``
- forecast_horizon
- The forecast horizon for the point predictions.
+ Defines the `start` format.
+ If set to ``'position'``, `start` corresponds to the index position of the first predicted point and can
+ range from `(-len(series), len(series) - 1)`.
+ If set to ``'value'``, `start` corresponds to the index value/label of the first predicted point. Will raise
+ an error if the value is not in `series`' index. Default: ``'value'``.
stride
The number of time steps between two consecutive predictions.
retrain
Whether and/or on which condition to retrain the model before predicting.
- This parameter supports 3 different datatypes: ``bool``, (positive) ``int``, and
- ``Callable`` (returning a ``bool``).
- In the case of ``bool``: retrain the model at each step (`True`), or never retrains the model (`False`).
+ This parameter supports 3 different types: ``bool``, (positive) ``int``, and ``Callable`` (returning a
+ ``bool``).
+ In the case of ``bool``: retrain the model at each step (`True`), or never retrain the model (`False`).
In the case of ``int``: the model is retrained every `retrain` iterations.
In the case of ``Callable``: the model is retrained whenever callable returns `True`.
The callable must have the following positional arguments:
- - `counter` (int): current `retrain` iteration
- - `pred_time` (pd.Timestamp or int): timestamp of forecast time (end of the training series)
- - `train_series` (TimeSeries): train series up to `pred_time`
- - `past_covariates` (TimeSeries): past_covariates series up to `pred_time`
- - `future_covariates` (TimeSeries): future_covariates series up
- to `min(pred_time + series.freq * forecast_horizon, series.end_time())`
+ - `counter` (int): current `retrain` iteration
+ - `pred_time` (pd.Timestamp or int): timestamp of forecast time (end of the training series)
+ - `train_series` (TimeSeries): train series up to `pred_time`
+ - `past_covariates` (TimeSeries): past_covariates series up to `pred_time`
+ - `future_covariates` (TimeSeries): future_covariates series up to `min(pred_time + series.freq *
+ forecast_horizon, series.end_time())`
Note: if any optional `*_covariates` are not passed to `historical_forecast`, ``None`` will be passed
to the corresponding retrain function argument.
- Note: some models do require being retrained every time and do not support anything other
- than `retrain=True`.
+ Note: some models require being retrained every time and do not support anything other than
+ `retrain=True`.
+ overlap_end
+ Whether the returned forecasts can go beyond the series' end or not.
last_points_only
- Whether to use the whole historical forecasts or only the last point of each forecast to compute the error.
+ Whether to return only the last point of each historical forecast. If set to ``True``, the method returns a
+ single ``TimeSeries`` (for each time series in `series`) containing the successive point forecasts.
+ Otherwise, returns a list of historical ``TimeSeries`` forecasts.
metric
Either one of Darts' "per time step" metrics (see `here
`_), or a custom metric that has an
@@ -1974,15 +1959,16 @@ def residuals(
:func:`~darts.metrics.metrics.multi_ts_support` and :func:`~darts.metrics.metrics.multi_ts_support`,
and returns one value per time step.
verbose
- Whether to print progress.
+ Whether to print the progress.
show_warnings
- Whether to show warnings related to parameters `start`, and `train_length`.
+ Whether to show warnings related to historical forecasts optimization, or parameters `start` and
+ `train_length`.
predict_likelihood_parameters
- If set to `True`, the model predict the parameters of its Likelihood parameters instead of the target. Only
- supported for probabilistic models with `likelihood="quantile"`, `num_samples = 1` and
- `n<=output_chunk_length`. Default: ``False``.
+ If set to `True`, the model predicts the parameters of its `likelihood` instead of the target. Only
+ supported for probabilistic models with a likelihood, `num_samples = 1` and `n<=output_chunk_length`.
+ Default: ``False``.
enable_optimization
- Whether to use the optimized version of historical_forecasts when supported and available.
+ Whether to use the optimized version of `historical_forecasts` when supported and available.
Default: ``True``.
metric_kwargs
Additional arguments passed to `metric()`, such as `'n_jobs'` for parallelization, `'m'` for scaled
@@ -1990,11 +1976,9 @@ def residuals(
reduction arguments `"series_reduction", "component_reduction", "time_reduction"`, and parameter
`'insample'` for scaled metrics (e.g. mase`, `rmsse`, ...), as they are handled internally.
fit_kwargs
- Additional arguments passed to the model `fit()` method.
+ Optionally, some additional arguments passed to the model `fit()` method.
predict_kwargs
- Additional arguments passed to the model `predict()` method.
- values_only
- Whether to return the residuals as `np.ndarray`. If `False`, returns residuals as `TimeSeries`.
+ Optionally, some additional arguments passed to the model `predict()` method.
sample_weight
Optionally, some sample weights to apply to the target `series` labels for training. Only effective when
`retrain` is not ``False``. They are applied per observation, per label (each step in
@@ -2005,6 +1989,8 @@ def residuals(
If a string, then the weights are generated using built-in weighting functions. The available options are
`"linear"` or `"exponential"` decay - the further in the past, the lower the weight. The weights are
computed per time `series`.
+ values_only
+ Whether to return the residuals as `np.ndarray`. If `False`, returns residuals as `TimeSeries`.
Returns
-------
@@ -2043,33 +2029,33 @@ def residuals(
enable_optimization=enable_optimization,
fit_kwargs=fit_kwargs,
predict_kwargs=predict_kwargs,
- overlap_end=False,
+ overlap_end=overlap_end,
sample_weight=sample_weight,
)
- residuals = self.backtest(
+ # remember input series type
+ series_seq_type = get_series_seq_type(series)
+ # validate historical forecasts and covert to multiple series with multiple forecasts case
+ series, historical_forecasts = _process_historical_forecast_for_backtest(
series=series,
historical_forecasts=historical_forecasts,
last_points_only=last_points_only,
- metric=metric,
- reduction=None,
- metric_kwargs=metric_kwargs,
)
- # remember input series type
- series_seq_type = get_series_seq_type(series)
+ # optionally, add nans to end of series to get residuals of same shape for each forecast
+ if overlap_end:
+ series = _extend_series_for_overlap_end(
+ series=series, historical_forecasts=historical_forecasts
+ )
- # convert forecasts and residuals to list of lists of series/arrays
- forecast_seq_type = get_series_seq_type(historical_forecasts)
- historical_forecasts = series2seq(
- historical_forecasts,
- seq_type_out=SeriesType.SEQ_SEQ,
- nested=last_points_only and forecast_seq_type == SeriesType.SEQ,
+ residuals = self.backtest(
+ series=series,
+ historical_forecasts=historical_forecasts,
+ last_points_only=False,
+ metric=metric,
+ reduction=None,
+ metric_kwargs=metric_kwargs,
)
- if series_seq_type == SeriesType.SINGLE:
- residuals = [residuals]
- if last_points_only:
- residuals = [[res] for res in residuals]
# sanity check residual output
q, q_interval = metric_kwargs.get("q"), metric_kwargs.get("q_interval")
@@ -2632,6 +2618,7 @@ def _optimized_historical_forecasts(
verbose: bool = False,
show_warnings: bool = True,
predict_likelihood_parameters: bool = False,
+ **kwargs,
) -> Union[TimeSeries, Sequence[TimeSeries], Sequence[Sequence[TimeSeries]]]:
logger.warning(
"`optimized historical forecasts is not available for this model, use `historical_forecasts` instead."
@@ -2836,12 +2823,11 @@ def predict(
One future-known covariate time series for every input time series in `series`. They must match the
past covariates that have been used with the :func:`fit()` function for training in terms of dimension.
num_samples
- Number of times a prediction is sampled from a probabilistic model. Should be left set to 1
- for deterministic models.
+ Number of times a prediction is sampled from a probabilistic model. Must be `1` for deterministic models.
verbose
- Optionally, whether to print progress.
+ Whether to print the progress.
predict_likelihood_parameters
- If set to `True`, the model predict the parameters of its Likelihood parameters instead of the target. Only
+ If set to `True`, the model predicts the parameters of its `likelihood` instead of the target. Only
supported for probabilistic models with a likelihood, `num_samples = 1` and `n<=output_chunk_length`.
Default: ``False``
show_warnings
@@ -3048,8 +3034,7 @@ def predict(
the covariate time series that has been used with the :func:`fit()` method for training, and it must
contain at least the next `n` time steps/indices after the end of the training target series.
num_samples
- Number of times a prediction is sampled from a probabilistic model. Should be left set to 1
- for deterministic models.
+ Number of times a prediction is sampled from a probabilistic model. Must be `1` for deterministic models.
verbose
Optionally, set the prediction verbosity. Not effective for all models.
show_warnings
@@ -3223,8 +3208,7 @@ def predict(
training target series. If `series` is set, it must contain at least the time steps/indices corresponding
to the new target series (historic future covariates), plus the next `n` time steps/indices after the end.
num_samples
- Number of times a prediction is sampled from a probabilistic model. Should be left set to 1
- for deterministic models.
+ Number of times a prediction is sampled from a probabilistic model. Must be `1` for deterministic models.
verbose
Optionally, set the prediction verbosity. Not effective for all models.
show_warnings
diff --git a/darts/models/forecasting/regression_model.py b/darts/models/forecasting/regression_model.py
index 5302dc0ab1..b5c76a0f0e 100644
--- a/darts/models/forecasting/regression_model.py
+++ b/darts/models/forecasting/regression_model.py
@@ -988,9 +988,9 @@ def predict(
Number of times a prediction is sampled from a probabilistic model. Should be set to 1
for deterministic models.
verbose
- Optionally, whether to print progress.
+ Whether to print the progress.
predict_likelihood_parameters
- If set to `True`, the model predict the parameters of its Likelihood parameters instead of the target. Only
+ If set to `True`, the model predicts the parameters of its `likelihood` instead of the target. Only
supported for probabilistic models with a likelihood, `num_samples = 1` and `n<=output_chunk_length`.
Default: ``False``
**kwargs : dict, optional
diff --git a/darts/models/forecasting/torch_forecasting_model.py b/darts/models/forecasting/torch_forecasting_model.py
index f73052dc5c..89ca19401f 100644
--- a/darts/models/forecasting/torch_forecasting_model.py
+++ b/darts/models/forecasting/torch_forecasting_model.py
@@ -14,9 +14,6 @@
as well as past and future values of some future covariates.
* SplitCovariatesTorchModel(TorchForecastingModel) for torch models consuming past-observed as well as future
values of some future covariates.
-
- * TorchParametricProbabilisticForecastingModel(TorchForecastingModel) is the super-class of all probabilistic torch
- forecasting models.
"""
import copy
@@ -706,7 +703,7 @@ def fit(
Optionally, a custom PyTorch-Lightning Trainer object to perform training. Using a custom ``trainer`` will
override Darts' default trainer.
verbose
- Optionally, whether to print the progress. Ignored if there is a `ProgressBar` callback in
+ Whether to print the progress. Ignored if there is a `ProgressBar` callback in
`pl_trainer_kwargs`.
epochs
If specified, will train the model for ``epochs`` (additional) epochs, irrespective of what ``n_epochs``
@@ -932,7 +929,7 @@ def fit_from_dataset(
Optionally, a custom PyTorch-Lightning Trainer object to perform prediction. Using a custom `trainer` will
override Darts' default trainer.
verbose
- Optionally, whether to print the progress. Ignored if there is a `ProgressBar` callback in
+ Whether to print the progress. Ignored if there is a `ProgressBar` callback in
`pl_trainer_kwargs`.
epochs
If specified, will train the model for ``epochs`` (additional) epochs, irrespective of what ``n_epochs``
@@ -1237,7 +1234,7 @@ def lr_find(
Optionally, a custom PyTorch-Lightning Trainer object to perform training. Using a custom ``trainer`` will
override Darts' default trainer.
verbose
- Optionally, whether to print the progress. Ignored if there is a `ProgressBar` callback in
+ Whether to print the progress. Ignored if there is a `ProgressBar` callback in
`pl_trainer_kwargs`.
epochs
If specified, will train the model for ``epochs`` (additional) epochs, irrespective of what ``n_epochs``
@@ -1366,7 +1363,7 @@ def predict(
batch_size
Size of batches during prediction. Defaults to the models' training ``batch_size`` value.
verbose
- Optionally, whether to print the progress. Ignored if there is a `ProgressBar` callback in
+ Whether to print the progress. Ignored if there is a `ProgressBar` callback in
`pl_trainer_kwargs`.
n_jobs
The number of jobs to run in parallel. ``-1`` means using all processors. Defaults to ``1``.
@@ -1376,8 +1373,7 @@ def predict(
(and optionally future covariates) back into the model. If this parameter is not provided,
it will be set ``output_chunk_length`` by default.
num_samples
- Number of times a prediction is sampled from a probabilistic model. Should be left set to 1
- for deterministic models.
+ Number of times a prediction is sampled from a probabilistic model. Must be `1` for deterministic models.
dataloader_kwargs
Optionally, a dictionary of keyword arguments used to create the PyTorch `DataLoader` instance for the
inference/prediction dataset. For more information on `DataLoader`, check out `this link
@@ -1388,7 +1384,7 @@ def predict(
Optionally, enable monte carlo dropout for predictions using neural network based models.
This allows bayesian approximation by specifying an implicit prior over learned models.
predict_likelihood_parameters
- If set to `True`, the model predict the parameters of its Likelihood parameters instead of the target. Only
+ If set to `True`, the model predicts the parameters of its `likelihood` instead of the target. Only
supported for probabilistic models with a likelihood, `num_samples = 1` and `n<=output_chunk_length`.
Default: ``False``.
show_warnings
@@ -1514,7 +1510,7 @@ def predict_from_dataset(
batch_size
Size of batches during prediction. Defaults to the models ``batch_size`` value.
verbose
- Optionally, whether to print the progress. Ignored if there is a `ProgressBar` callback in
+ Whether to print the progress. Ignored if there is a `ProgressBar` callback in
`pl_trainer_kwargs`.
n_jobs
The number of jobs to run in parallel. ``-1`` means using all processors. Defaults to ``1``.
@@ -1524,8 +1520,7 @@ def predict_from_dataset(
(and optionally future covariates) back into the model. If this parameter is not provided,
it will be set ``output_chunk_length`` by default.
num_samples
- Number of times a prediction is sampled from a probabilistic model. Should be left set to 1
- for deterministic models.
+ Number of times a prediction is sampled from a probabilistic model. Must be `1` for deterministic models.
dataloader_kwargs
Optionally, a dictionary of keyword arguments used to create the PyTorch `DataLoader` instance for the
inference/prediction dataset. For more information on `DataLoader`, check out `this link
@@ -1536,7 +1531,7 @@ def predict_from_dataset(
Optionally, enable monte carlo dropout for predictions using neural network based models.
This allows bayesian approximation by specifying an implicit prior over learned models.
predict_likelihood_parameters
- If set to `True`, the model predict the parameters of its Likelihood parameters instead of the target. Only
+ If set to `True`, the model predicts the parameters of its `likelihood` instead of the target. Only
supported for probabilistic models with a likelihood, `num_samples = 1` and `n<=output_chunk_length`.
Default: ``False``
diff --git a/darts/tests/conftest.py b/darts/tests/conftest.py
index b0b97a0131..90bf29e20b 100644
--- a/darts/tests/conftest.py
+++ b/darts/tests/conftest.py
@@ -1,4 +1,5 @@
import logging
+import os
import shutil
import tempfile
@@ -40,15 +41,31 @@ def tear_down_tests():
@pytest.fixture(scope="module")
def tmpdir_module():
- """Sets up a temporary directory that will be deleted after the test module (script) finished."""
+ """Sets up and moves into a temporary directory that will be deleted after the test module (script) finished."""
temp_work_dir = tempfile.mkdtemp(prefix="darts")
+ # remember origin
+ cwd = os.getcwd()
+ # move to temp dir
+ os.chdir(temp_work_dir)
+ # go into test with temp dir as input
yield temp_work_dir
+ # move back to origin
shutil.rmtree(temp_work_dir)
+ # remove temp dir
+ os.chdir(cwd)
@pytest.fixture(scope="function")
def tmpdir_fn():
- """Sets up a temporary directory that will be deleted after the test function finished."""
+ """Sets up and moves into a temporary directory that will be deleted after the test function finished."""
temp_work_dir = tempfile.mkdtemp(prefix="darts")
+ # remember origin
+ cwd = os.getcwd()
+ # move to temp dir
+ os.chdir(temp_work_dir)
+ # go into test with temp dir as input
yield temp_work_dir
+ # move back to origin
+ os.chdir(cwd)
+ # remove temp dir
shutil.rmtree(temp_work_dir)
diff --git a/darts/tests/metrics/test_metrics.py b/darts/tests/metrics/test_metrics.py
index f3e2b88229..ba58b6fe3f 100644
--- a/darts/tests/metrics/test_metrics.py
+++ b/darts/tests/metrics/test_metrics.py
@@ -79,6 +79,53 @@ def metric_iw(y_true, y_pred, q_interval=None, **kwargs):
return res.reshape(len(y_pred), -1)
+def metric_iws(y_true, y_pred, q_interval=None, **kwargs):
+ # this tests assumes `y_pred` are stochastic values
+ if isinstance(q_interval, tuple):
+ q_interval = [q_interval]
+ q_interval = np.array(q_interval)
+ q_lo = q_interval[:, 0]
+ q_hi = q_interval[:, 1]
+ y_pred_lo = np.quantile(y_pred, q_lo, axis=2).transpose(1, 2, 0)
+ y_pred_hi = np.quantile(y_pred, q_hi, axis=2).transpose(1, 2, 0)
+ interval_width = y_pred_hi - y_pred_lo
+ res = np.where(
+ y_true < y_pred_lo,
+ interval_width + 1 / q_lo * (y_pred_lo - y_true),
+ interval_width,
+ )
+ res = np.where(
+ y_true > y_pred_hi, interval_width + 1 / (1 - q_hi) * (y_true - y_pred_hi), res
+ )
+ return res.reshape(len(y_pred), -1)
+
+
+def metric_ic(y_true, y_pred, q_interval=None, **kwargs):
+ # this tests assumes `y_pred` are stochastic values
+ if isinstance(q_interval, tuple):
+ q_interval = [q_interval]
+ q_interval = np.array(q_interval)
+ q_lo = q_interval[:, 0]
+ q_hi = q_interval[:, 1]
+ y_pred_lo = np.quantile(y_pred, q_lo, axis=2).transpose(1, 2, 0)
+ y_pred_hi = np.quantile(y_pred, q_hi, axis=2).transpose(1, 2, 0)
+ res = np.where((y_pred_lo <= y_true) & (y_true <= y_pred_hi), 1, 0)
+ return res.reshape(len(y_pred), -1)
+
+
+def metric_incs_qr(y_true, y_pred, q_interval=None, **kwargs):
+ # this tests assumes `y_pred` are stochastic values
+ if isinstance(q_interval, tuple):
+ q_interval = [q_interval]
+ q_interval = np.array(q_interval)
+ q_lo = q_interval[:, 0]
+ q_hi = q_interval[:, 1]
+ y_pred_lo = np.quantile(y_pred, q_lo, axis=2).transpose(1, 2, 0)
+ y_pred_hi = np.quantile(y_pred, q_hi, axis=2).transpose(1, 2, 0)
+ res = np.maximum(y_pred_lo - y_true, y_true - y_pred_hi)
+ return res.reshape(len(y_pred), -1)
+
+
class TestMetrics:
np.random.seed(42)
pd_train = pd.Series(
@@ -1853,6 +1900,9 @@ def test_wrong_error_scale(self):
[
# only time dependent quantile interval metrics
(metrics.iw, metric_iw),
+ (metrics.iws, metric_iws),
+ (metrics.ic, metric_ic),
+ (metrics.incs_qr, metric_incs_qr),
],
)
def test_metric_quantile_interval_accuracy(self, config):
@@ -1899,6 +1949,12 @@ def check_ref(**test_kwargs):
# time dependent but with time reduction
metrics.iw,
metrics.miw,
+ metrics.iws,
+ metrics.miws,
+ metrics.ic,
+ metrics.mic,
+ metrics.incs_qr,
+ metrics.mincs_qr,
],
[True, False], # univariate series
[True, False], # single series
diff --git a/darts/tests/models/forecasting/test_conformal_model.py b/darts/tests/models/forecasting/test_conformal_model.py
new file mode 100644
index 0000000000..10acaa8b1e
--- /dev/null
+++ b/darts/tests/models/forecasting/test_conformal_model.py
@@ -0,0 +1,1614 @@
+import copy
+import itertools
+import os
+
+import numpy as np
+import pandas as pd
+import pytest
+
+from darts import TimeSeries, concatenate
+from darts.datasets import AirPassengersDataset
+from darts.metrics import ae, err, ic, incs_qr, mic
+from darts.models import (
+ ConformalNaiveModel,
+ ConformalQRModel,
+ LinearRegressionModel,
+ NaiveSeasonal,
+ NLinearModel,
+)
+from darts.models.forecasting.forecasting_model import ForecastingModel
+from darts.tests.conftest import TORCH_AVAILABLE, tfm_kwargs
+from darts.utils import timeseries_generation as tg
+from darts.utils.utils import (
+ likelihood_component_names,
+ quantile_interval_names,
+ quantile_names,
+)
+
+IN_LEN = 3
+OUT_LEN = 3
+regr_kwargs = {"lags": IN_LEN, "output_chunk_length": OUT_LEN}
+tfm_kwargs = copy.deepcopy(tfm_kwargs)
+tfm_kwargs["pl_trainer_kwargs"]["fast_dev_run"] = True
+torch_kwargs = dict(
+ {"input_chunk_length": IN_LEN, "output_chunk_length": OUT_LEN, "random_state": 0},
+ **tfm_kwargs,
+)
+pred_lklp = {"num_samples": 1, "predict_likelihood_parameters": True}
+q = [0.1, 0.5, 0.9]
+
+
+def train_model(
+ *args, model_type="regression", model_params=None, quantiles=None, **kwargs
+):
+ model_params = model_params or {}
+ if model_type == "regression":
+ return LinearRegressionModel(
+ **regr_kwargs,
+ **model_params,
+ random_state=42,
+ ).fit(*args, **kwargs)
+ elif model_type in ["regression_prob", "regression_qr"]:
+ return LinearRegressionModel(
+ likelihood="quantile",
+ quantiles=quantiles,
+ **regr_kwargs,
+ **model_params,
+ random_state=42,
+ ).fit(*args, **kwargs)
+ else:
+ return NLinearModel(**torch_kwargs, **model_params).fit(*args, **kwargs)
+
+
+# pre-trained global model for conformal models
+models_cls_kwargs_errs = [
+ (
+ ConformalNaiveModel,
+ {"quantiles": q},
+ "regression",
+ ),
+]
+
+if TORCH_AVAILABLE:
+ models_cls_kwargs_errs.append((
+ ConformalNaiveModel,
+ {"quantiles": q},
+ "torch",
+ ))
+
+
+class TestConformalModel:
+ """
+ Tests all general model behavior for Naive Conformal Model with symmetric non-conformity score.
+ Additionally, checks correctness of predictions for:
+ - ConformalNaiveModel with symmetric & asymmetric non-conformity scores
+ - ConformaQRlModel with symmetric & asymmetric non-conformity scores
+ """
+
+ np.random.seed(42)
+
+ # forecasting horizon used in runnability tests
+ horizon = OUT_LEN + 1
+
+ # some arbitrary static covariates
+ static_covariates = pd.DataFrame([[0.0, 1.0]], columns=["st1", "st2"])
+
+ # real timeseries for functionality tests
+ ts_length = 13 + horizon
+ ts_passengers = (
+ AirPassengersDataset()
+ .load()[:ts_length]
+ .with_static_covariates(static_covariates)
+ )
+ ts_pass_train, ts_pass_val = (
+ ts_passengers[:-horizon],
+ ts_passengers[-horizon:],
+ )
+
+ # an additional noisy series
+ ts_pass_train_1 = ts_pass_train + 0.01 * tg.gaussian_timeseries(
+ length=len(ts_pass_train),
+ freq=ts_pass_train.freq_str,
+ start=ts_pass_train.start_time(),
+ )
+
+ # an additional time series serving as covariates
+ year_series = tg.datetime_attribute_timeseries(ts_passengers, attribute="year")
+ month_series = tg.datetime_attribute_timeseries(ts_passengers, attribute="month")
+ time_covariates = year_series.stack(month_series)
+ time_covariates_train = time_covariates[:-horizon]
+
+ # various ts with different static covariates representations
+ ts_w_static_cov = tg.linear_timeseries(length=ts_length).with_static_covariates(
+ pd.Series([1, 2])
+ )
+ ts_shared_static_cov = ts_w_static_cov.stack(tg.sine_timeseries(length=ts_length))
+ ts_comps_static_cov = ts_shared_static_cov.with_static_covariates(
+ pd.DataFrame([[0, 1], [2, 3]], columns=["st1", "st2"])
+ )
+
+ def test_model_construction_naive(self):
+ local_model = NaiveSeasonal(K=5)
+ global_model = LinearRegressionModel(**regr_kwargs)
+ series = self.ts_pass_train
+
+ model_err_msg = "`model` must be a pre-trained `GlobalForecastingModel`."
+ # un-trained local model
+ with pytest.raises(ValueError) as exc:
+ ConformalNaiveModel(model=local_model, quantiles=q)
+ assert str(exc.value) == model_err_msg
+
+ # pre-trained local model
+ local_model.fit(series)
+ with pytest.raises(ValueError) as exc:
+ ConformalNaiveModel(model=local_model, quantiles=q)
+ assert str(exc.value) == model_err_msg
+
+ # un-trained global model
+ with pytest.raises(ValueError) as exc:
+ ConformalNaiveModel(model=global_model, quantiles=q)
+ assert str(exc.value) == model_err_msg
+
+ # pre-trained local model should work
+ global_model.fit(series)
+ _ = ConformalNaiveModel(model=global_model, quantiles=q)
+
+ # non-centered quantiles
+ with pytest.raises(ValueError) as exc:
+ ConformalNaiveModel(model=global_model, quantiles=[0.2, 0.5, 0.6])
+ assert str(exc.value) == (
+ "quantiles lower than `q=0.5` need to share same difference to `0.5` as quantiles higher than `q=0.5`"
+ )
+
+ # quantiles missing median
+ with pytest.raises(ValueError) as exc:
+ ConformalNaiveModel(model=global_model, quantiles=[0.1, 0.9])
+ assert str(exc.value) == "median quantile `q=0.5` must be in `quantiles`"
+
+ # too low and high quantiles
+ with pytest.raises(ValueError) as exc:
+ ConformalNaiveModel(model=global_model, quantiles=[-0.1, 0.5, 1.1])
+ assert str(exc.value) == "All provided quantiles must be between 0 and 1."
+
+ def test_model_construction_cqr(self):
+ model_det = train_model(self.ts_pass_train, model_type="regression")
+ model_prob_q = train_model(
+ self.ts_pass_train, model_type="regression_prob", quantiles=q
+ )
+ model_prob_poisson = train_model(
+ self.ts_pass_train,
+ model_type="regression",
+ model_params={"likelihood": "poisson"},
+ )
+
+ # deterministic global model
+ with pytest.raises(ValueError) as exc:
+ ConformalQRModel(model=model_det, quantiles=q)
+ assert str(exc.value).startswith(
+ "`model` must must support probabilistic forecasting."
+ )
+ # probabilistic model works
+ _ = ConformalQRModel(model=model_prob_q, quantiles=q)
+ # works also with different likelihood
+ _ = ConformalQRModel(model=model_prob_poisson, quantiles=q)
+
+ @pytest.mark.parametrize("config", models_cls_kwargs_errs)
+ def test_save_model_parameters(self, config):
+ # model creation parameters were saved before. check if re-created model has same params as original
+ model_cls, kwargs, model_type = config
+ model = model_cls(
+ model=train_model(
+ self.ts_pass_train, model_type=model_type, quantiles=kwargs["quantiles"]
+ ),
+ **kwargs,
+ )
+ model_fresh = model.untrained_model()
+ assert model._model_params.keys() == model_fresh._model_params.keys()
+ for param, val in model._model_params.items():
+ if isinstance(val, ForecastingModel):
+ # Conformal Models require a forecasting model as input, which has no equality
+ continue
+ assert val == model_fresh._model_params[param]
+
+ @pytest.mark.parametrize(
+ "config", itertools.product(models_cls_kwargs_errs, [{}, pred_lklp])
+ )
+ def test_save_load_model(self, tmpdir_fn, config):
+ # check if save and load methods work and if loaded model creates same forecasts as original model
+ (model_cls, kwargs, model_type), pred_kwargs = config
+ model = model_cls(
+ train_model(
+ self.ts_pass_train, model_type=model_type, quantiles=kwargs["quantiles"]
+ ),
+ **kwargs,
+ )
+
+ # check if save and load methods work and
+ # if loaded conformal model creates same forecasts as original ensemble models
+ expected_suffixes = [
+ ".pkl",
+ ".pkl.NLinearModel.pt",
+ ".pkl.NLinearModel.pt.ckpt",
+ ]
+
+ # test save
+ model.save()
+ model.save(os.path.join(tmpdir_fn, f"{model_cls.__name__}.pkl"))
+
+ model_prediction = model.predict(5, **pred_kwargs)
+
+ assert os.path.exists(tmpdir_fn)
+ files = os.listdir(tmpdir_fn)
+ if model_type == "torch":
+ # 1 from conformal model, 2 from torch, * 2 as `save()` was called twice
+ assert len(files) == 6
+ for f in files:
+ assert f.startswith(model_cls.__name__)
+ suffix_counts = {
+ suffix: sum(1 for p in os.listdir(tmpdir_fn) if p.endswith(suffix))
+ for suffix in expected_suffixes
+ }
+ assert all(count == 2 for count in suffix_counts.values())
+ else:
+ assert len(files) == 2
+ for f in files:
+ assert f.startswith(model_cls.__name__) and f.endswith(".pkl")
+
+ # test load
+ pkl_files = []
+ for filename in os.listdir(tmpdir_fn):
+ if filename.endswith(".pkl"):
+ pkl_files.append(os.path.join(tmpdir_fn, filename))
+ for p in pkl_files:
+ loaded_model = model_cls.load(p)
+ assert model_prediction == loaded_model.predict(5, **pred_kwargs)
+
+ @pytest.mark.parametrize("config", models_cls_kwargs_errs)
+ def test_single_ts(self, config):
+ model_cls, kwargs, model_type = config
+ model = model_cls(
+ train_model(
+ self.ts_pass_train, model_type=model_type, quantiles=kwargs["quantiles"]
+ ),
+ **kwargs,
+ )
+ pred = model.predict(n=self.horizon, **pred_lklp)
+ assert pred.n_components == self.ts_pass_train.n_components * 3
+ assert not np.isnan(pred.all_values()).any().any()
+
+ pred_fc = model.model.predict(n=self.horizon)
+ assert pred_fc.time_index.equals(pred.time_index)
+ # the center forecasts must be equal to the forecasting model forecast
+ fc_columns = likelihood_component_names(
+ self.ts_pass_val.columns, quantile_names([0.5])
+ )
+ np.testing.assert_array_almost_equal(
+ pred[fc_columns].all_values(), pred_fc.all_values()
+ )
+ assert pred.static_covariates is None
+
+ # using a different `n`, gives different results, since we can generate more residuals for the horizon
+ pred1 = model.predict(n=1, **pred_lklp)
+ assert not pred1 == pred
+
+ # giving the same series as calibration set must give the same results
+ pred_cal = model.predict(
+ n=self.horizon, cal_series=self.ts_pass_train, **pred_lklp
+ )
+ np.testing.assert_array_almost_equal(pred.all_values(), pred_cal.all_values())
+
+ # wrong dimension
+ with pytest.raises(ValueError):
+ model.predict(
+ n=self.horizon,
+ series=self.ts_pass_train.stack(self.ts_pass_train),
+ **pred_lklp,
+ )
+
+ @pytest.mark.parametrize("config", models_cls_kwargs_errs)
+ def test_multi_ts(self, config):
+ model_cls, kwargs, model_type = config
+ model = model_cls(
+ train_model(
+ [self.ts_pass_train, self.ts_pass_train_1],
+ model_type=model_type,
+ quantiles=kwargs["quantiles"],
+ ),
+ **kwargs,
+ )
+ with pytest.raises(ValueError):
+ # when model is fit from >1 series, one must provide a series in argument
+ model.predict(n=1)
+
+ pred = model.predict(n=self.horizon, series=self.ts_pass_train, **pred_lklp)
+ assert pred.n_components == self.ts_pass_train.n_components * 3
+ assert not np.isnan(pred.all_values()).any().any()
+
+ # the center forecasts must be equal to the forecasting model forecast
+ fc_columns = likelihood_component_names(
+ self.ts_pass_val.columns, quantile_names([0.5])
+ )
+ pred_fc = model.model.predict(n=self.horizon, series=self.ts_pass_train)
+ assert pred_fc.time_index.equals(pred.time_index)
+ np.testing.assert_array_almost_equal(
+ pred[fc_columns].all_values(), pred_fc.all_values()
+ )
+
+ # using a calibration series also requires an input series
+ with pytest.raises(ValueError):
+ # when model is fit from >1 series, one must provide a series in argument
+ model.predict(n=1, cal_series=self.ts_pass_train, **pred_lklp)
+ # giving the same series as calibration set must give the same results
+ pred_cal = model.predict(
+ n=self.horizon,
+ series=self.ts_pass_train,
+ cal_series=self.ts_pass_train,
+ **pred_lklp,
+ )
+ np.testing.assert_array_almost_equal(pred.all_values(), pred_cal.all_values())
+
+ # check prediction for several time series
+ pred_list = model.predict(
+ n=self.horizon,
+ series=[self.ts_pass_train, self.ts_pass_train_1],
+ **pred_lklp,
+ )
+ pred_fc_list = model.model.predict(
+ n=self.horizon,
+ series=[self.ts_pass_train, self.ts_pass_train_1],
+ )
+ assert (
+ len(pred_list) == 2
+ ), f"Model {model_cls} did not return a list of prediction"
+ for pred, pred_fc in zip(pred_list, pred_fc_list):
+ assert pred.n_components == self.ts_pass_train.n_components * 3
+ assert pred_fc.time_index.equals(pred.time_index)
+ assert not np.isnan(pred.all_values()).any().any()
+ np.testing.assert_array_almost_equal(
+ pred_fc.all_values(),
+ pred[fc_columns].all_values(),
+ )
+
+ # using a calibration series requires to have same number of series as target
+ with pytest.raises(ValueError) as exc:
+ # when model is fit from >1 series, one must provide a series in argument
+ model.predict(
+ n=1,
+ series=[self.ts_pass_train, self.ts_pass_val],
+ cal_series=self.ts_pass_train,
+ **pred_lklp,
+ )
+ assert (
+ str(exc.value)
+ == "Mismatch between number of `cal_series` (1) and number of `series` (2)."
+ )
+ # using a calibration series requires to have same number of series as target
+ with pytest.raises(ValueError) as exc:
+ # when model is fit from >1 series, one must provide a series in argument
+ model.predict(
+ n=1,
+ series=[self.ts_pass_train, self.ts_pass_val],
+ cal_series=[self.ts_pass_train] * 3,
+ **pred_lklp,
+ )
+ assert (
+ str(exc.value)
+ == "Mismatch between number of `cal_series` (3) and number of `series` (2)."
+ )
+
+ # giving the same series as calibration set must give the same results
+ pred_cal_list = model.predict(
+ n=self.horizon,
+ series=[self.ts_pass_train, self.ts_pass_train_1],
+ cal_series=[self.ts_pass_train, self.ts_pass_train_1],
+ **pred_lklp,
+ )
+ for pred, pred_cal in zip(pred_list, pred_cal_list):
+ np.testing.assert_array_almost_equal(
+ pred.all_values(), pred_cal.all_values()
+ )
+
+ # using copies of the same series as calibration set must give the same interval widths for
+ # each target series
+ pred_cal_list = model.predict(
+ n=self.horizon,
+ series=[self.ts_pass_train, self.ts_pass_train_1],
+ cal_series=[self.ts_pass_train, self.ts_pass_train],
+ **pred_lklp,
+ )
+
+ pred_0_vals = pred_cal_list[0].all_values()
+ pred_1_vals = pred_cal_list[1].all_values()
+
+ # lower range
+ np.testing.assert_array_almost_equal(
+ pred_0_vals[:, 1] - pred_0_vals[:, 0], pred_1_vals[:, 1] - pred_1_vals[:, 0]
+ )
+ # upper range
+ np.testing.assert_array_almost_equal(
+ pred_0_vals[:, 2] - pred_0_vals[:, 1], pred_1_vals[:, 2] - pred_1_vals[:, 1]
+ )
+
+ # wrong dimension
+ with pytest.raises(ValueError):
+ model.predict(
+ n=self.horizon,
+ series=[
+ self.ts_pass_train,
+ self.ts_pass_train.stack(self.ts_pass_train),
+ ],
+ **pred_lklp,
+ )
+
+ @pytest.mark.parametrize(
+ "config",
+ itertools.product(
+ [(ConformalNaiveModel, {"quantiles": [0.1, 0.5, 0.9]}, "regression")],
+ [
+ {"lags_past_covariates": IN_LEN},
+ {"lags_future_covariates": (IN_LEN, OUT_LEN)},
+ {},
+ ],
+ ),
+ )
+ def test_covariates(self, config):
+ (model_cls, kwargs, model_type), covs_kwargs = config
+ model_fc = LinearRegressionModel(**regr_kwargs, **covs_kwargs)
+ # Here we rely on the fact that all non-Dual models currently are Past models
+ if model_fc.supports_future_covariates:
+ cov_name = "future_covariates"
+ is_past = False
+ elif model_fc.supports_past_covariates:
+ cov_name = "past_covariates"
+ is_past = True
+ else:
+ cov_name = None
+ is_past = None
+
+ covariates = [self.time_covariates_train, self.time_covariates_train]
+ if cov_name is not None:
+ cov_kwargs = {cov_name: covariates}
+ cov_kwargs_train = {cov_name: self.time_covariates_train}
+ cov_kwargs_notrain = {cov_name: self.time_covariates}
+ else:
+ cov_kwargs = {}
+ cov_kwargs_train = {}
+ cov_kwargs_notrain = {}
+
+ model_fc.fit(series=[self.ts_pass_train, self.ts_pass_train_1], **cov_kwargs)
+
+ model = model_cls(model=model_fc, **kwargs)
+ if cov_name == "future_covariates":
+ assert model.supports_future_covariates
+ assert not model.supports_past_covariates
+ assert model.uses_future_covariates
+ assert not model.uses_past_covariates
+ elif cov_name == "past_covariates":
+ assert not model.supports_future_covariates
+ assert model.supports_past_covariates
+ assert not model.uses_future_covariates
+ assert model.uses_past_covariates
+ else:
+ assert not model.supports_future_covariates
+ assert not model.supports_past_covariates
+ assert not model.uses_future_covariates
+ assert not model.uses_past_covariates
+
+ with pytest.raises(ValueError):
+ # when model is fit from >1 series, one must provide a series in argument
+ model.predict(n=1)
+
+ if cov_name is not None:
+ with pytest.raises(ValueError):
+ # when model is fit using multiple covariates, covariates are required at prediction time
+ model.predict(n=1, series=self.ts_pass_train)
+
+ with pytest.raises(ValueError):
+ # when model is fit using covariates, n cannot be greater than output_chunk_length...
+ # (for short covariates)
+ # past covariates model can predict up until output_chunk_length
+ # with train future covariates we cannot predict at all after end of series
+ model.predict(
+ n=OUT_LEN + 1 if is_past else 1,
+ series=self.ts_pass_train,
+ **cov_kwargs_train,
+ )
+ else:
+ # model does not support covariates
+ with pytest.raises(ValueError):
+ model.predict(
+ n=1,
+ series=self.ts_pass_train,
+ past_covariates=self.time_covariates,
+ )
+ with pytest.raises(ValueError):
+ model.predict(
+ n=1,
+ series=self.ts_pass_train,
+ future_covariates=self.time_covariates,
+ )
+
+ # ... unless future covariates are provided
+ _ = model.predict(
+ n=self.horizon, series=self.ts_pass_train, **cov_kwargs_notrain
+ )
+
+ pred = model.predict(
+ n=self.horizon, series=self.ts_pass_train, **cov_kwargs_notrain, **pred_lklp
+ )
+ pred_fc = model_fc.predict(
+ n=self.horizon,
+ series=self.ts_pass_train,
+ **cov_kwargs_notrain,
+ )
+ fc_columns = likelihood_component_names(
+ self.ts_pass_val.columns, quantile_names([0.5])
+ )
+ np.testing.assert_array_almost_equal(
+ pred[fc_columns].all_values(),
+ pred_fc.all_values(),
+ )
+
+ if cov_name is None:
+ return
+
+ # when model is fit using 1 training and 1 covariate series, time series args are optional
+ model_fc = LinearRegressionModel(**regr_kwargs, **covs_kwargs)
+ model_fc.fit(series=self.ts_pass_train, **cov_kwargs_train)
+ model = model_cls(model_fc, **kwargs)
+
+ if is_past:
+ # can only predict up until ocl
+ with pytest.raises(ValueError):
+ _ = model.predict(n=OUT_LEN + 1, **pred_lklp)
+ # wrong covariates dimension
+ with pytest.raises(ValueError):
+ covs = cov_kwargs_train[cov_name]
+ covs = {cov_name: covs.stack(covs)}
+ _ = model.predict(n=OUT_LEN + 1, **covs, **pred_lklp)
+ # with past covariates from train we can predict up until output_chunk_length
+ pred1 = model.predict(n=OUT_LEN, **pred_lklp)
+ pred2 = model.predict(n=OUT_LEN, series=self.ts_pass_train, **pred_lklp)
+ pred3 = model.predict(n=OUT_LEN, **cov_kwargs_train, **pred_lklp)
+ pred4 = model.predict(
+ n=OUT_LEN, **cov_kwargs_train, series=self.ts_pass_train, **pred_lklp
+ )
+ else:
+ # with future covariates we need additional time steps to predict
+ with pytest.raises(ValueError):
+ _ = model.predict(n=1, **pred_lklp)
+ with pytest.raises(ValueError):
+ _ = model.predict(n=1, series=self.ts_pass_train, **pred_lklp)
+ with pytest.raises(ValueError):
+ _ = model.predict(n=1, **cov_kwargs_train, **pred_lklp)
+ with pytest.raises(ValueError):
+ _ = model.predict(
+ n=1, **cov_kwargs_train, series=self.ts_pass_train, **pred_lklp
+ )
+ # wrong covariates dimension
+ with pytest.raises(ValueError):
+ covs = cov_kwargs_notrain[cov_name]
+ covs = {cov_name: covs.stack(covs)}
+ _ = model.predict(n=OUT_LEN + 1, **covs, **pred_lklp)
+ pred1 = model.predict(n=OUT_LEN, **cov_kwargs_notrain, **pred_lklp)
+ pred2 = model.predict(
+ n=OUT_LEN, series=self.ts_pass_train, **cov_kwargs_notrain, **pred_lklp
+ )
+ pred3 = model.predict(n=OUT_LEN, **cov_kwargs_notrain, **pred_lklp)
+ pred4 = model.predict(
+ n=OUT_LEN, **cov_kwargs_notrain, series=self.ts_pass_train, **pred_lklp
+ )
+
+ assert pred1 == pred2
+ assert pred1 == pred3
+ assert pred1 == pred4
+
+ @pytest.mark.parametrize(
+ "config,ts",
+ itertools.product(
+ models_cls_kwargs_errs,
+ [ts_w_static_cov, ts_shared_static_cov, ts_comps_static_cov],
+ ),
+ )
+ def test_use_static_covariates(self, config, ts):
+ """
+ Check that both static covariates representations are supported (component-specific and shared)
+ for both uni- and multivariate series when fitting the model.
+ Also check that the static covariates are present in the forecasted series
+ """
+ model_cls, kwargs, model_type = config
+ model = model_cls(
+ train_model(ts, model_type=model_type, quantiles=kwargs["quantiles"]),
+ **kwargs,
+ )
+ assert model.uses_static_covariates
+ pred = model.predict(OUT_LEN)
+ assert pred.static_covariates is None
+
+ @pytest.mark.parametrize(
+ "config",
+ itertools.product(
+ [True, False], # univariate series
+ [True, False], # single series
+ [True, False], # use covariates
+ [True, False], # datetime index
+ [1, 3, 5], # different horizons
+ ),
+ )
+ def test_predict(self, config):
+ (is_univar, is_single, use_covs, is_datetime, horizon) = config
+ series = self.ts_pass_train
+ if not is_univar:
+ series = series.stack(series)
+ if not is_datetime:
+ series = TimeSeries.from_values(series.all_values(), columns=series.columns)
+ if use_covs:
+ pc, fc = series, series
+ fc = fc.append_values(fc.values()[: max(horizon, OUT_LEN)])
+ if horizon > OUT_LEN:
+ pc = pc.append_values(pc.values()[: horizon - OUT_LEN])
+ model_kwargs = {
+ "lags_past_covariates": IN_LEN,
+ "lags_future_covariates": (IN_LEN, OUT_LEN),
+ }
+ else:
+ pc, fc = None, None
+ model_kwargs = {}
+ if not is_single:
+ series = [
+ series,
+ series.with_columns_renamed(
+ col_names=series.columns.tolist(),
+ col_names_new=(series.columns + "_s2").tolist(),
+ ),
+ ]
+ if use_covs:
+ pc = [pc] * 2
+ fc = [fc] * 2
+
+ # testing lags_past_covariates None but past_covariates during prediction
+ model_instance = LinearRegressionModel(
+ lags=IN_LEN, output_chunk_length=OUT_LEN, **model_kwargs
+ )
+ model_instance.fit(series=series, past_covariates=pc, future_covariates=fc)
+ model = ConformalNaiveModel(model_instance, quantiles=q)
+
+ preds = model.predict(
+ n=horizon,
+ series=series,
+ past_covariates=pc,
+ future_covariates=fc,
+ **pred_lklp,
+ )
+
+ if is_single:
+ series = [series]
+ preds = [preds]
+
+ for s_, preds_ in zip(series, preds):
+ cols_expected = likelihood_component_names(s_.columns, quantile_names(q))
+ assert preds_.columns.tolist() == cols_expected
+ assert len(preds_) == horizon
+ assert preds_.start_time() == s_.end_time() + s_.freq
+ assert preds_.freq == s_.freq
+
+ def test_output_chunk_shift(self):
+ model_params = {"output_chunk_shift": 1}
+ model = ConformalNaiveModel(
+ train_model(self.ts_pass_train, model_params=model_params, quantiles=q),
+ quantiles=q,
+ )
+ pred = model.predict(n=1, **pred_lklp)
+ pred_fc = model.model.predict(n=1)
+
+ assert pred_fc.time_index.equals(pred.time_index)
+ # the center forecasts must be equal to the forecasting model forecast
+ fc_columns = likelihood_component_names(
+ self.ts_pass_train.columns, quantile_names([0.5])
+ )
+
+ np.testing.assert_array_almost_equal(
+ pred[fc_columns].all_values(), pred_fc.all_values()
+ )
+
+ pred_cal = model.predict(n=1, cal_series=self.ts_pass_train, **pred_lklp)
+ assert pred_fc.time_index.equals(pred_cal.time_index)
+ # the center forecasts must be equal to the forecasting model forecast
+ np.testing.assert_array_almost_equal(pred_cal.all_values(), pred.all_values())
+
+ @pytest.mark.parametrize(
+ "config",
+ list(
+ itertools.product(
+ [1, 3, 5], # horizon
+ [True, False], # univariate series
+ [True, False], # single series
+ [q, [0.2, 0.3, 0.5, 0.7, 0.8]],
+ [
+ (ConformalNaiveModel, "regression"),
+ (ConformalNaiveModel, "regression_prob"),
+ (ConformalQRModel, "regression_qr"),
+ ], # model type
+ [True, False], # symmetric non-conformity score
+ [None, 1], # train length
+ )
+ ),
+ )
+ def test_conformal_model_predict_accuracy(self, config):
+ """Verifies that naive conformal model computes the correct intervals for:
+ - different horizons (smaller, equal, larger than ocl)
+ - uni/multivariate series
+ - single/multi series
+ - single/multi quantile intervals
+ - deterministic/probabilistic forecasting model
+ - naive conformal and conformalized quantile regression
+ - symmetric/asymmetric non-conformity scores
+
+ The naive approach computes it as follows:
+
+ - pred_upper = pred + q_interval(absolute error, past)
+ - pred_middle = pred
+ - pred_lower = pred - q_interval(absolute error, past)
+
+ Where q_interval(absolute error) is the `q_hi - q_hi` quantile value of all historic absolute errors
+ between `pred`, and the target series.
+ """
+ (
+ n,
+ is_univar,
+ is_single,
+ quantiles,
+ (model_cls, model_type),
+ symmetric,
+ cal_length,
+ ) = config
+ idx_med = quantiles.index(0.5)
+ q_intervals = [
+ (q_hi, q_lo)
+ for q_hi, q_lo in zip(quantiles[:idx_med], quantiles[idx_med + 1 :][::-1])
+ ]
+ series = self.helper_prepare_series(is_univar, is_single)
+ pred_kwargs = (
+ {"num_samples": 1000}
+ if model_type in ["regression_prob", "regression_qr"]
+ else {}
+ )
+
+ model_fc = train_model(series, model_type=model_type, quantiles=q)
+ model = model_cls(
+ model=model_fc,
+ quantiles=quantiles,
+ symmetric=symmetric,
+ cal_length=cal_length,
+ )
+ pred_fc_list = model.model.predict(n, series=series, **pred_kwargs)
+ pred_cal_list = model.predict(n, series=series, **pred_lklp)
+ pred_cal_list_with_cal = model.predict(
+ n, series=series, cal_series=series, **pred_lklp
+ )
+
+ if issubclass(model_cls, ConformalNaiveModel):
+ metric = ae if symmetric else err
+ metric_kwargs = {}
+ else:
+ metric = incs_qr
+ metric_kwargs = {"q_interval": q_intervals, "symmetric": symmetric}
+ # compute the expected intervals
+ residuals_list = model.model.residuals(
+ series,
+ retrain=False,
+ forecast_horizon=n,
+ overlap_end=True,
+ last_points_only=False,
+ stride=1,
+ values_only=True,
+ metric=metric,
+ metric_kwargs=metric_kwargs,
+ **pred_kwargs,
+ )
+ if is_single:
+ pred_fc_list = [pred_fc_list]
+ pred_cal_list = [pred_cal_list]
+ residuals_list = [residuals_list]
+ pred_cal_list_with_cal = [pred_cal_list_with_cal]
+
+ for pred_fc, pred_cal, pred_cal_with_cal, residuals in zip(
+ pred_fc_list, pred_cal_list, pred_cal_list_with_cal, residuals_list
+ ):
+ residuals = np.concatenate(residuals[:-1], axis=2)
+
+ pred_vals = pred_fc.all_values()
+ pred_vals_expected = self.helper_compute_pred_cal(
+ residuals,
+ pred_vals,
+ n,
+ quantiles,
+ model_type,
+ symmetric,
+ cal_length=cal_length,
+ )
+ self.helper_compare_preds(pred_cal, pred_vals_expected, model_type)
+ self.helper_compare_preds(pred_cal_with_cal, pred_vals_expected, model_type)
+
+ @pytest.mark.parametrize(
+ "config",
+ itertools.product(
+ [1, 3, 5], # horizon
+ [True, False], # univariate series
+ [True, False], # single series,
+ [0, 1], # output chunk shift
+ [None, 1], # train length
+ [False, True], # use covariates
+ [q, [0.2, 0.3, 0.5, 0.7, 0.8]], # quantiles
+ ),
+ )
+ def test_naive_conformal_model_historical_forecasts(self, config):
+ """Checks correctness of naive conformal model historical forecasts for:
+ - different horizons (smaller, equal and larger the OCL)
+ - uni and multivariate series
+ - single and multiple series
+ - with and without output shift
+ - with and without training length
+ - with and without covariates in the forecast and calibration sets.
+ """
+ n, is_univar, is_single, ocs, cal_length, use_covs, quantiles = config
+ n_q = len(quantiles)
+ half_idx = n_q // 2
+ if ocs and n > OUT_LEN:
+ # auto-regression not allowed with ocs
+ return
+
+ series = self.helper_prepare_series(is_univar, is_single)
+ model_params = {"output_chunk_shift": ocs}
+
+ # for covariates, we check that shorter & longer covariates in the calibration set give expected results
+ covs_kwargs = {}
+ cal_covs_kwargs_overlap = {}
+ cal_covs_kwargs_short = {}
+ cal_covs_kwargs_exact = {}
+ if use_covs:
+ model_params["lags_past_covariates"] = regr_kwargs["lags"]
+ past_covs = series
+ if n > OUT_LEN:
+ append_vals = [[[1.0]] * (1 if is_univar else 2)] * (n - OUT_LEN)
+ if is_single:
+ past_covs = past_covs.append_values(append_vals)
+ else:
+ past_covs = [pc.append_values(append_vals) for pc in past_covs]
+ covs_kwargs["past_covariates"] = past_covs
+ # produces examples with all points in `overlap_end=True` (last example has no useful information)
+ cal_covs_kwargs_overlap["cal_past_covariates"] = past_covs
+ # produces one example less (drops the one with unuseful information)
+ cal_covs_kwargs_exact["cal_past_covariates"] = (
+ past_covs[: -(1 + ocs)]
+ if is_single
+ else [pc[: -(1 + ocs)] for pc in past_covs]
+ )
+ # produces another example less (drops the last one which contains useful information)
+ cal_covs_kwargs_short["cal_past_covariates"] = (
+ past_covs[: -(2 + ocs)]
+ if is_single
+ else [pc[: -(2 + ocs)] for pc in past_covs]
+ )
+
+ # forecasts from forecasting model
+ model_fc = train_model(series, model_params=model_params, **covs_kwargs)
+ hfc_fc_list = model_fc.historical_forecasts(
+ series,
+ retrain=False,
+ forecast_horizon=n,
+ overlap_end=True,
+ last_points_only=False,
+ stride=1,
+ **covs_kwargs,
+ )
+ # residuals to compute the conformal intervals
+ residuals_list = model_fc.residuals(
+ series,
+ historical_forecasts=hfc_fc_list,
+ overlap_end=True,
+ last_points_only=False,
+ values_only=True,
+ metric=ae, # absolute error
+ **covs_kwargs,
+ )
+
+ # conformal forecasts
+ model = ConformalNaiveModel(
+ model=model_fc, quantiles=quantiles, cal_length=cal_length
+ )
+ # without calibration set
+ hfc_conf_list = model.historical_forecasts(
+ series=series,
+ forecast_horizon=n,
+ overlap_end=True,
+ last_points_only=False,
+ stride=1,
+ **covs_kwargs,
+ **pred_lklp,
+ )
+ # with calibration set and covariates that can generate all calibration forecasts in the overlap
+ hfc_conf_list_with_cal = model.historical_forecasts(
+ series=series,
+ forecast_horizon=n,
+ overlap_end=True,
+ last_points_only=False,
+ stride=1,
+ cal_series=series,
+ **covs_kwargs,
+ **cal_covs_kwargs_overlap,
+ **pred_lklp,
+ )
+
+ if is_single:
+ hfc_conf_list = [hfc_conf_list]
+ residuals_list = [residuals_list]
+ hfc_conf_list_with_cal = [hfc_conf_list_with_cal]
+ hfc_fc_list = [hfc_fc_list]
+
+ # validate computed conformal intervals that did not use a calibration set
+ # conformal models start later since they need past residuals as input
+ first_fc_idx = len(hfc_fc_list[0]) - len(hfc_conf_list[0])
+ for hfc_fc, hfc_conf, hfc_residuals in zip(
+ hfc_fc_list, hfc_conf_list, residuals_list
+ ):
+ for idx, (pred_fc, pred_cal) in enumerate(
+ zip(hfc_fc[first_fc_idx:], hfc_conf)
+ ):
+ # need to ignore additional `ocs` (output shift) residuals
+ residuals = np.concatenate(
+ hfc_residuals[: first_fc_idx - ocs + idx], axis=2
+ )
+
+ pred_vals = pred_fc.all_values()
+ pred_vals_expected = self.helper_compute_pred_cal(
+ residuals,
+ pred_vals,
+ n,
+ quantiles,
+ cal_length=cal_length,
+ model_type="regression",
+ symmetric=True,
+ )
+ np.testing.assert_array_almost_equal(
+ pred_cal.all_values(), pred_vals_expected
+ )
+
+ # validate computed conformal intervals that used a calibration set
+ for hfc_conf_with_cal, hfc_conf in zip(hfc_conf_list_with_cal, hfc_conf_list):
+ # last forecast with calibration set must be equal to the last without calibration set
+ # (since calibration set is the same series)
+ assert hfc_conf_with_cal[-1] == hfc_conf[-1]
+ hfc_0_vals = hfc_conf_with_cal[0].all_values()
+ for hfc_i in hfc_conf_with_cal[1:]:
+ hfc_i_vals = hfc_i.all_values()
+ for q_idx in range(n_q):
+ np.testing.assert_array_almost_equal(
+ hfc_0_vals[:, half_idx::n_q] - hfc_0_vals[:, q_idx::n_q],
+ hfc_i_vals[:, half_idx::n_q] - hfc_i_vals[:, q_idx::n_q],
+ )
+
+ if use_covs:
+ # `cal_covs_kwargs_exact` will not compute the last example in overlap_end (this one has anyways no
+ # useful information). Result is expected to be identical to the case when using `cal_covs_kwargs_overlap`
+ hfc_conf_list_with_cal_exact = model.historical_forecasts(
+ series=series,
+ forecast_horizon=n,
+ overlap_end=True,
+ last_points_only=False,
+ stride=1,
+ cal_series=series,
+ **covs_kwargs,
+ **cal_covs_kwargs_exact,
+ **pred_lklp,
+ )
+
+ # `cal_covs_kwargs_short` will compute example less that contains useful information
+ hfc_conf_list_with_cal_short = model.historical_forecasts(
+ series=series,
+ forecast_horizon=n,
+ overlap_end=True,
+ last_points_only=False,
+ stride=1,
+ cal_series=series,
+ **covs_kwargs,
+ **cal_covs_kwargs_short,
+ **pred_lklp,
+ )
+ if is_single:
+ hfc_conf_list_with_cal_exact = [hfc_conf_list_with_cal_exact]
+ hfc_conf_list_with_cal_short = [hfc_conf_list_with_cal_short]
+
+ # must match
+ assert len(hfc_conf_list_with_cal_exact) == len(
+ hfc_conf_list_with_cal_short
+ )
+ for hfc_cal_exact, hfc_cal in zip(
+ hfc_conf_list_with_cal_exact, hfc_conf_list_with_cal
+ ):
+ assert len(hfc_cal_exact) == len(hfc_cal)
+ for hfc_cal_exact_, hfc_cal_ in zip(hfc_cal_exact, hfc_cal):
+ assert hfc_cal_exact_.time_index.equals(hfc_cal_.time_index)
+ assert hfc_cal_exact_.columns.equals(hfc_cal_.columns)
+ np.testing.assert_array_almost_equal(
+ hfc_cal_exact_.all_values(), hfc_cal_.all_values()
+ )
+
+ # second last forecast with shorter calibration set (that has one example less) must be equal to the
+ # second last without calibration set
+ for hfc_conf_with_cal, hfc_conf in zip(
+ hfc_conf_list_with_cal_short, hfc_conf_list
+ ):
+ assert hfc_conf_with_cal[-2] == hfc_conf[-2]
+
+ # checking that last points only is equal to the last forecasted point
+ hfc_lpo_list = model.historical_forecasts(
+ series=series,
+ forecast_horizon=n,
+ overlap_end=True,
+ last_points_only=True,
+ stride=1,
+ **covs_kwargs,
+ **pred_lklp,
+ )
+ hfc_lpo_list_with_cal = model.historical_forecasts(
+ series=series,
+ forecast_horizon=n,
+ overlap_end=True,
+ last_points_only=True,
+ stride=1,
+ cal_series=series,
+ **covs_kwargs,
+ **cal_covs_kwargs_overlap,
+ **pred_lklp,
+ )
+ if is_single:
+ hfc_lpo_list = [hfc_lpo_list]
+ hfc_lpo_list_with_cal = [hfc_lpo_list_with_cal]
+
+ for hfc_lpo, hfc_conf in zip(hfc_lpo_list, hfc_conf_list):
+ hfc_conf_lpo = concatenate([hfc[-1:] for hfc in hfc_conf], axis=0)
+ assert hfc_lpo == hfc_conf_lpo
+
+ for hfc_lpo, hfc_conf in zip(hfc_lpo_list_with_cal, hfc_conf_list_with_cal):
+ hfc_conf_lpo = concatenate([hfc[-1:] for hfc in hfc_conf], axis=0)
+ assert hfc_lpo == hfc_conf_lpo
+
+ def test_probabilistic_historical_forecast(self):
+ """Checks correctness of naive conformal historical forecast from probabilistic fc model compared to
+ deterministic one,
+ """
+ series = self.helper_prepare_series(False, False)
+ # forecasts from forecasting model
+ model_det = ConformalNaiveModel(
+ train_model(series, model_type="regression", quantiles=q),
+ quantiles=q,
+ )
+ model_prob = ConformalNaiveModel(
+ train_model(series, model_type="regression_prob", quantiles=q),
+ quantiles=q,
+ )
+ hfcs_det = model_det.historical_forecasts(
+ series,
+ forecast_horizon=2,
+ last_points_only=True,
+ stride=1,
+ **pred_lklp,
+ )
+ hfcs_prob = model_prob.historical_forecasts(
+ series,
+ forecast_horizon=2,
+ last_points_only=True,
+ stride=1,
+ **pred_lklp,
+ )
+ assert isinstance(hfcs_det, list) and len(hfcs_det) == 2
+ assert isinstance(hfcs_prob, list) and len(hfcs_prob) == 2
+ for hfc_det, hfc_prob in zip(hfcs_det, hfcs_prob):
+ assert hfc_det.columns.equals(hfc_prob.columns)
+ assert hfc_det.time_index.equals(hfc_prob.time_index)
+ self.helper_compare_preds(
+ hfc_prob, hfc_det.all_values(), model_type="regression_prob"
+ )
+
+ def helper_prepare_series(self, is_univar, is_single):
+ series = self.ts_pass_train
+ if not is_univar:
+ series = series.stack(series + 3.0)
+ if not is_single:
+ series = [series, series + 5]
+ return series
+
+ def helper_compare_preds(self, cp_pred, pred_expected, model_type, tol_rel=0.1):
+ if isinstance(cp_pred, TimeSeries):
+ cp_pred = cp_pred.all_values(copy=False)
+ if model_type == "regression":
+ # deterministic fc model should give almost identical results
+ np.testing.assert_array_almost_equal(cp_pred, pred_expected)
+ else:
+ # probabilistic fc models have some randomness
+ diffs_rel = np.abs((cp_pred - pred_expected) / pred_expected)
+ assert (diffs_rel < tol_rel).all().all()
+
+ @staticmethod
+ def helper_compute_pred_cal(
+ residuals, pred_vals, n, quantiles, model_type, symmetric, cal_length=None
+ ):
+ """Generates expected prediction results for naive conformal model from:
+
+ - residuals and predictions from deterministic/probabilistic model
+ - any forecast horizon
+ - any quantile intervals
+ - symmetric/ asymmetric non-conformity scores
+ - any train length
+ """
+ cal_length = cal_length or 0
+ n_comps = pred_vals.shape[1]
+ half_idx = len(quantiles) // 2
+
+ # get alphas from quantiles (alpha = q_hi - q_lo) per interval
+ alphas = np.array(quantiles[half_idx + 1 :][::-1]) - np.array(
+ quantiles[:half_idx]
+ )
+ if not symmetric:
+ # asymmetric non-conformity scores look only on one tail -> alpha/2
+ alphas = 1 - (1 - alphas) / 2
+ if model_type == "regression_prob":
+ # naive conformal model converts probabilistic forecasts to median (deterministic)
+ pred_vals = np.expand_dims(np.quantile(pred_vals, 0.5, axis=2), -1)
+ elif model_type == "regression_qr":
+ # conformalized quantile regression consumes quantile forecasts
+ pred_vals = np.quantile(pred_vals, quantiles, axis=2).transpose(1, 2, 0)
+
+ is_naive = model_type in ["regression", "regression_prob"]
+ pred_expected = []
+ for alpha_idx, alpha in enumerate(alphas):
+ q_hats = []
+ # compute the quantile `alpha` of all past residuals (absolute "per time step" errors between historical
+ # forecasts and the target series)
+ for idx in range(n):
+ res_end = residuals.shape[2] - idx
+ if cal_length:
+ res_start = res_end - cal_length
+ else:
+ res_start = n - (idx + 1)
+ res_n = residuals[idx][:, res_start:res_end]
+ if is_naive and symmetric:
+ # identical correction for upper and lower bounds
+ # metric is `ae()`
+ q_hat_n = np.quantile(res_n, q=alpha, method="higher", axis=1)
+ q_hats.append((-q_hat_n, q_hat_n))
+ elif is_naive:
+ # correction separately for upper and lower bounds
+ # metric is `err()`
+ q_hat_hi = np.quantile(res_n, q=alpha, method="higher", axis=1)
+ q_hat_lo = np.quantile(-res_n, q=alpha, method="higher", axis=1)
+ q_hats.append((-q_hat_lo, q_hat_hi))
+ elif symmetric: # CQR symmetric
+ # identical correction for upper and lower bounds
+ # metric is `incs_qr(symmetric=True)`
+ q_hat_n = np.quantile(res_n, q=alpha, method="higher", axis=1)
+ q_hats.append((-q_hat_n, q_hat_n))
+ else: # CQR asymmetric
+ # correction separately for upper and lower bounds
+ # metric is `incs_qr(symmetric=False)`
+ half_idx = len(res_n) // 2
+
+ # residuals have shape (n components * n intervals * 2)
+ # the factor 2 comes from the metric being computed for lower, and upper bounds separately
+ # (comp_1_qlow_1, comp_1_qlow_2, ... comp_n_qlow_m, comp_1_qhigh_1, ...)
+ q_hat_lo = np.quantile(
+ res_n[:half_idx], q=alpha, method="higher", axis=1
+ )
+ q_hat_hi = np.quantile(
+ res_n[half_idx:], q=alpha, method="higher", axis=1
+ )
+ q_hats.append((
+ -q_hat_lo[alpha_idx :: len(alphas)],
+ q_hat_hi[alpha_idx :: len(alphas)],
+ ))
+ # bring to shape (horizon, n components, 2)
+ q_hats = np.array(q_hats).transpose((0, 2, 1))
+ # the prediction interval is given by pred +/- q_hat
+ pred_vals_expected = []
+ for col_idx in range(n_comps):
+ q_col = q_hats[:, col_idx]
+ pred_col = pred_vals[:, col_idx]
+ if is_naive:
+ # conformal model corrects deterministic predictions
+ idx_q_lo = slice(0, None)
+ idx_q_med = slice(0, None)
+ idx_q_hi = slice(0, None)
+ else:
+ # conformal model corrects quantile predictions
+ idx_q_lo = slice(alpha_idx, alpha_idx + 1)
+ idx_q_med = slice(len(alphas), len(alphas) + 1)
+ idx_q_hi = slice(
+ pred_col.shape[1] - (alpha_idx + 1),
+ pred_col.shape[1] - alpha_idx,
+ )
+ # correct lower and upper bounds
+ pred_col_expected = np.concatenate(
+ [
+ pred_col[:, idx_q_lo] + q_col[:, :1], # lower quantile
+ pred_col[:, idx_q_med], # median forecast
+ pred_col[:, idx_q_hi] + q_col[:, 1:],
+ ], # upper quantile
+ axis=1,
+ )
+ pred_col_expected = np.expand_dims(pred_col_expected, 1)
+ pred_vals_expected.append(pred_col_expected)
+ pred_vals_expected = np.concatenate(pred_vals_expected, axis=1)
+ pred_expected.append(pred_vals_expected)
+
+ # reorder to have columns going from lowest quantiles to highest per component
+ pred_expected_reshaped = []
+ for comp_idx in range(n_comps):
+ for q_idx in [0, 1, 2]:
+ for pred_idx in range(len(pred_expected)):
+ # upper quantiles will have reversed order
+ if q_idx == 2:
+ pred_idx = len(pred_expected) - 1 - pred_idx
+ pred_ = pred_expected[pred_idx][:, comp_idx, q_idx]
+ pred_ = pred_.reshape(-1, 1, 1)
+
+ # q_hat_idx = q_idx + comp_idx * 3 + alpha_idx * 3 * n_comps
+ pred_expected_reshaped.append(pred_)
+ # only add median quantile once
+ if q_idx == 1:
+ break
+ return np.concatenate(pred_expected_reshaped, axis=1)
+
+ @pytest.mark.parametrize(
+ "config",
+ itertools.product(
+ [1, 3, 5], # horizon
+ [0, 1], # output chunk shift
+ [False, True], # use covariates
+ ),
+ )
+ def test_too_short_input_predict(self, config):
+ """Checks conformal model predict with minimum required input and too short input."""
+ n, ocs, use_covs = config
+ if ocs and n > OUT_LEN:
+ return
+ icl = IN_LEN
+ min_len = icl + ocs + n
+ series = tg.linear_timeseries(length=min_len)
+ series_train = [tg.linear_timeseries(length=IN_LEN + OUT_LEN + ocs)] * 2
+
+ model_params = {"output_chunk_shift": ocs}
+ covs_kwargs = {}
+ cal_covs_kwargs = {}
+ covs_kwargs_train = {}
+ covs_kwargs_too_short = {}
+ cal_covs_kwargs_short = {}
+ if use_covs:
+ model_params["lags_past_covariates"] = regr_kwargs["lags"]
+ covs_kwargs_train["past_covariates"] = series_train
+ # use shorter covariates, to test whether residuals are still properly extracted
+ past_covs = series
+ # for auto-regression, we require longer past covariates
+ if n > OUT_LEN:
+ past_covs = past_covs.append_values([1.0] * (n - OUT_LEN))
+ covs_kwargs["past_covariates"] = past_covs
+ covs_kwargs_too_short["past_covariates"] = past_covs[:-1]
+ # giving covs in calibration set requires one calibration example less
+ cal_covs_kwargs["cal_past_covariates"] = past_covs[: -(1 + ocs)]
+ cal_covs_kwargs_short["cal_past_covariates"] = past_covs[: -(2 + ocs)]
+
+ model = ConformalNaiveModel(
+ train_model(
+ series=series_train,
+ model_params=model_params,
+ **covs_kwargs_train,
+ ),
+ quantiles=q,
+ )
+
+ # prediction works with long enough input
+ preds1 = model.predict(n=n, series=series, **covs_kwargs)
+ assert not np.isnan(preds1.all_values()).any().any()
+ preds2 = model.predict(
+ n=n, series=series, **covs_kwargs, cal_series=series, **cal_covs_kwargs
+ )
+ assert not np.isnan(preds2.all_values()).any().any()
+ # series too short: without covariates, make `series` shorter. Otherwise, use the shorter covariates
+ series_ = series[:-1] if not use_covs else series
+
+ with pytest.raises(ValueError) as exc:
+ _ = model.predict(n=n, series=series_, **covs_kwargs_too_short)
+ if not use_covs:
+ assert str(exc.value).startswith(
+ "Could not build the minimum required calibration input with the provided `cal_series`"
+ )
+ else:
+ # if `past_covariates` are too short, then it raises error from the forecasting_model.predict()
+ assert str(exc.value).startswith(
+ "The `past_covariates` at list/sequence index 0 are not long enough."
+ )
+
+ with pytest.raises(ValueError) as exc:
+ _ = model.predict(
+ n=n,
+ series=series,
+ cal_series=series_,
+ **covs_kwargs,
+ **cal_covs_kwargs_short,
+ )
+ if not use_covs or n > 1:
+ assert str(exc.value).startswith(
+ "Could not build the minimum required calibration input with the provided `cal_series`"
+ )
+ else:
+ # if `cal_past_covariates` are too short and `horizon=1`, then it raises error from the forecasting model
+ assert str(exc.value).startswith(
+ "Cannot build a single input for prediction with the provided model"
+ )
+
+ @pytest.mark.parametrize(
+ "config",
+ itertools.product(
+ [False, True], # last points only
+ [False, True], # overlap end
+ [None, 2], # train length
+ [0, 1], # output chunk shift
+ [1, 3, 5], # horizon
+ [True, False], # use covs
+ ),
+ )
+ def test_too_short_input_hfc(self, config):
+ """Checks conformal model historical forecasts with minimum required input and too short input."""
+ (
+ last_points_only,
+ overlap_end,
+ cal_length,
+ ocs,
+ n,
+ use_covs,
+ ) = config
+ if ocs and n > OUT_LEN:
+ return
+
+ icl = IN_LEN
+ ocl = OUT_LEN
+ horizon_ocs = n + ocs
+ add_cal_length = cal_length - 1 if cal_length is not None else 0
+ # min length to generate 1 conformal forecast
+ min_len_val_series = (
+ icl + horizon_ocs * (1 + int(not overlap_end)) + add_cal_length
+ )
+
+ series_train = [tg.linear_timeseries(length=icl + ocl + ocs)] * 2
+ series = tg.linear_timeseries(length=min_len_val_series)
+
+ # define cal series to get the minimum required cal set
+ if overlap_end:
+ # with overlap_end `series` has the exact length to generate one forecast after the end of the input series
+ # Therefore, `series` has already the minimum length for one calibrated forecast
+ cal_series = series
+ else:
+ # without overlap_end, we use a shorter input, since the last forecast is within the input series
+ # (it generates more residuals with useful information than the minimum requirements)
+ cal_series = series[:-horizon_ocs]
+
+ series_with_cal = series[: -(horizon_ocs + add_cal_length)]
+
+ model_params = {"output_chunk_shift": ocs}
+ covs_kwargs_train = {}
+ covs_kwargs = {}
+ covs_with_cal_kwargs = {}
+ cal_covs_kwargs = {}
+ covs_kwargs_short = {}
+ cal_covs_kwargs_short = {}
+ if use_covs:
+ model_params["lags_past_covariates"] = regr_kwargs["lags"]
+ covs_kwargs_train["past_covariates"] = series_train
+
+ # `- horizon_ocs` to generate forecasts extending up until end of target series
+ if not overlap_end:
+ past_covs = series[:-horizon_ocs]
+ else:
+ past_covs = series
+
+ # calibration set is always generated internally with `overlap_end=True`
+ # make shorter to not compute residuals without useful information
+ cal_past_covs = cal_series[: -(1 + ocs)]
+
+ # last_points_only requires `horizon` residuals less
+ if last_points_only:
+ cal_past_covs = cal_past_covs[: (-(n - 1) or None)]
+
+ # for auto-regression, we require longer past covariates
+ if n > OUT_LEN:
+ past_covs = past_covs.append_values([1.0] * (n - OUT_LEN))
+ cal_past_covs = cal_past_covs.append_values([1.0] * (n - OUT_LEN))
+
+ # covariates lengths to generate exactly one forecast
+ covs_kwargs["past_covariates"] = past_covs
+ # giving a calibration set requires fewer forecasts
+ covs_with_cal_kwargs["past_covariates"] = past_covs[:-horizon_ocs]
+ cal_covs_kwargs["cal_past_covariates"] = cal_past_covs
+
+ # use too short covariates to check that errors are raised
+ covs_kwargs_short["past_covariates"] = covs_kwargs["past_covariates"][:-1]
+ cal_covs_kwargs_short["cal_past_covariates"] = cal_covs_kwargs[
+ "cal_past_covariates"
+ ][:-1]
+
+ model = ConformalNaiveModel(
+ train_model(
+ series=series_train,
+ model_params=model_params,
+ **covs_kwargs_train,
+ ),
+ quantiles=q,
+ cal_length=cal_length,
+ )
+
+ hfc_kwargs = {
+ "last_points_only": last_points_only,
+ "overlap_end": overlap_end,
+ "forecast_horizon": n,
+ }
+ # prediction works with long enough input
+ hfcs = model.historical_forecasts(
+ series=series,
+ **covs_kwargs,
+ **hfc_kwargs,
+ )
+ hfcs_cal = model.historical_forecasts(
+ series=series_with_cal,
+ cal_series=cal_series,
+ **covs_with_cal_kwargs,
+ **cal_covs_kwargs,
+ **hfc_kwargs,
+ )
+ if last_points_only:
+ hfcs = [hfcs]
+ hfcs_cal = [hfcs_cal]
+
+ assert len(hfcs) == len(hfcs_cal) == 1
+ for hfc, hfc_cal in zip(hfcs, hfcs_cal):
+ assert not np.isnan(hfc.all_values()).any().any()
+ assert not np.isnan(hfc_cal.all_values()).any().any()
+
+ # input too short: without covariates, make `series` shorter. Otherwise, use the shorter covariates
+ series_ = series[:-1] if not use_covs else series
+ cal_series_ = cal_series[:-1] if not use_covs else cal_series
+
+ with pytest.raises(ValueError) as exc:
+ _ = model.historical_forecasts(
+ series=series_,
+ **covs_kwargs_short,
+ **hfc_kwargs,
+ )
+ assert str(exc.value).startswith(
+ "Could not build the minimum required calibration input with the provided `series` and `*_covariates`"
+ )
+
+ with pytest.raises(ValueError) as exc:
+ _ = model.historical_forecasts(
+ series=series_with_cal,
+ cal_series=cal_series_,
+ **covs_with_cal_kwargs,
+ **cal_covs_kwargs_short,
+ **hfc_kwargs,
+ )
+ if (not use_covs or n > 1 or (cal_length or 1) > 1) and not (
+ last_points_only and use_covs and cal_length is None
+ ):
+ assert str(exc.value).startswith(
+ "Could not build the minimum required calibration input with the provided `cal_series`"
+ )
+ else:
+ assert str(exc.value).startswith(
+ "Cannot build a single input for prediction with the provided model"
+ )
+
+ @pytest.mark.parametrize("quantiles", [[0.1, 0.5, 0.9], [0.1, 0.3, 0.5, 0.7, 0.9]])
+ def test_backtest_and_residuals(self, quantiles):
+ """Residuals and backtest are already tested for quantile, and interval metrics based on stochastic or quantile
+ forecasts. So, a simple check that they give expected results should be enough.
+ """
+ n_q = len(quantiles)
+ half_idx = n_q // 2
+ q_interval = [
+ (q_lo, q_hi)
+ for q_lo, q_hi in zip(quantiles[:half_idx], quantiles[half_idx + 1 :][::-1])
+ ]
+ lpo = False
+
+ # series long enough for 2 hfcs
+ series = self.helper_prepare_series(True, True).append_values([0.1])
+ # conformal model
+ model = ConformalNaiveModel(model=train_model(series), quantiles=quantiles)
+
+ hfc = model.historical_forecasts(
+ series=series, forecast_horizon=5, last_points_only=lpo, **pred_lklp
+ )
+ bt = model.backtest(
+ series=series,
+ historical_forecasts=hfc,
+ last_points_only=lpo,
+ metric=mic,
+ metric_kwargs={"q_interval": model.q_interval},
+ )
+ # default backtest is equal to backtest with metric kwargs
+ np.testing.assert_array_almost_equal(
+ bt,
+ model.backtest(
+ series=series,
+ historical_forecasts=hfc,
+ last_points_only=lpo,
+ metric=mic,
+ metric_kwargs={"q_interval": q_interval},
+ ),
+ )
+ np.testing.assert_array_almost_equal(
+ mic(
+ [series] * len(hfc),
+ hfc,
+ q_interval=q_interval,
+ series_reduction=np.mean,
+ ),
+ bt,
+ )
+
+ residuals = model.residuals(
+ series=series,
+ historical_forecasts=hfc,
+ last_points_only=lpo,
+ metric=ic,
+ metric_kwargs={"q_interval": q_interval},
+ )
+ # default residuals is equal to residuals with metric kwargs
+ assert residuals == model.residuals(
+ series=series,
+ historical_forecasts=hfc,
+ last_points_only=lpo,
+ metric=ic,
+ metric_kwargs={"q_interval": q_interval},
+ )
+ expected_vals = ic([series] * len(hfc), hfc, q_interval=q_interval)
+ expected_residuals = []
+ for vals, hfc_ in zip(expected_vals, hfc):
+ expected_residuals.append(
+ TimeSeries.from_times_and_values(
+ times=hfc_.time_index,
+ values=vals,
+ columns=likelihood_component_names(
+ series.components, quantile_interval_names(q_interval)
+ ),
+ )
+ )
+ assert residuals == expected_residuals
+
+ def test_predict_probabilistic_equals_quantile(self):
+ """Tests that sampled quantiles predictions have approx. the same quantiles as direct quantile predictions."""
+ quantiles = [0.1, 0.3, 0.5, 0.7, 0.9]
+
+ # multiple multivariate series
+ series = self.helper_prepare_series(False, False)
+
+ # conformal model
+ model = ConformalNaiveModel(model=train_model(series), quantiles=quantiles)
+ # direct quantile predictions
+ pred_quantiles = model.predict(n=3, series=series, **pred_lklp)
+ # smapled predictions
+ pred_samples = model.predict(n=3, series=series, num_samples=500)
+ for pred_q, pred_s in zip(pred_quantiles, pred_samples):
+ assert pred_q.n_samples == 1
+ assert pred_q.n_components == series[0].n_components * len(quantiles)
+ assert pred_s.n_samples == 500
+ assert pred_s.n_components == series[0].n_components
+
+ vals_q = pred_q.all_values()
+ vals_s = pred_s.all_values()
+ vals_s_q = np.quantile(vals_s, quantiles, axis=2).transpose((1, 2, 0))
+ vals_s_q = vals_s_q.reshape(vals_q.shape)
+ self.helper_compare_preds(
+ vals_s_q,
+ vals_q,
+ model_type="regression_prob",
+ )
diff --git a/darts/tests/models/forecasting/test_ensemble_models.py b/darts/tests/models/forecasting/test_ensemble_models.py
index 7fa663ddce..92fb932a5e 100644
--- a/darts/tests/models/forecasting/test_ensemble_models.py
+++ b/darts/tests/models/forecasting/test_ensemble_models.py
@@ -766,14 +766,10 @@ def get_global_ensemble_model(output_chunk_length=5):
)
@pytest.mark.parametrize("model_cls", [NaiveEnsembleModel, RegressionEnsembleModel])
- def test_save_load_ensemble_models(self, tmpdir_module, model_cls):
+ def test_save_load_ensemble_models(self, tmpdir_fn, model_cls):
# check if save and load methods work and
# if loaded ensemble model creates same forecasts as original ensemble models
- cwd = os.getcwd()
- os.chdir(tmpdir_module)
- os.mkdir(model_cls.__name__)
- full_model_path_str = os.path.join(tmpdir_module, model_cls.__name__)
- os.chdir(full_model_path_str)
+ full_model_path_str = os.getcwd()
kwargs = {}
expected_suffixes = [".pkl", ".pkl.RNNModel_2.pt", ".pkl.RNNModel_2.pt.ckpt"]
@@ -827,5 +823,3 @@ def test_save_load_ensemble_models(self, tmpdir_module, model_cls):
for p in pkl_files:
loaded_model = model_cls.load(p)
assert model_prediction == loaded_model.predict(5)
-
- os.chdir(cwd)
diff --git a/darts/tests/models/forecasting/test_global_forecasting_models.py b/darts/tests/models/forecasting/test_global_forecasting_models.py
index f8eea72615..22343b3cf7 100644
--- a/darts/tests/models/forecasting/test_global_forecasting_models.py
+++ b/darts/tests/models/forecasting/test_global_forecasting_models.py
@@ -276,12 +276,10 @@ def test_save_model_parameters(self, config):
),
],
)
- def test_save_load_model(self, tmpdir_module, model):
+ def test_save_load_model(self, tmpdir_fn, model):
# check if save and load methods work and if loaded model creates same forecasts as original model
- cwd = os.getcwd()
- os.chdir(tmpdir_module)
model_path_str = type(model).__name__
- full_model_path_str = os.path.join(tmpdir_module, model_path_str)
+ full_model_path_str = os.path.join(tmpdir_fn, model_path_str)
model.fit(self.ts_pass_train)
model_prediction = model.predict(self.forecasting_horizon)
@@ -293,9 +291,7 @@ def test_save_load_model(self, tmpdir_module, model):
assert os.path.exists(full_model_path_str)
assert (
len([
- p
- for p in os.listdir(tmpdir_module)
- if p.startswith(type(model).__name__)
+ p for p in os.listdir(tmpdir_fn) if p.startswith(type(model).__name__)
])
== 4
)
@@ -305,8 +301,6 @@ def test_save_load_model(self, tmpdir_module, model):
assert model_prediction == loaded_model.predict(self.forecasting_horizon)
- os.chdir(cwd)
-
@pytest.mark.parametrize("config", models_cls_kwargs_errs)
def test_single_ts(self, config):
model_cls, kwargs, err = config
diff --git a/darts/tests/models/forecasting/test_historical_forecasts.py b/darts/tests/models/forecasting/test_historical_forecasts.py
index 967e5e5e7c..42c22e5319 100644
--- a/darts/tests/models/forecasting/test_historical_forecasts.py
+++ b/darts/tests/models/forecasting/test_historical_forecasts.py
@@ -15,6 +15,7 @@
ARIMA,
AutoARIMA,
CatBoostModel,
+ ConformalNaiveModel,
LightGBMModel,
LinearRegressionModel,
NaiveDrift,
@@ -24,6 +25,7 @@
from darts.tests.conftest import TORCH_AVAILABLE, tfm_kwargs
from darts.utils import n_steps_between
from darts.utils import timeseries_generation as tg
+from darts.utils.utils import likelihood_component_names, quantile_names
if TORCH_AVAILABLE:
import torch
@@ -1589,13 +1591,13 @@ def f_encoder(idx):
assert ohfc[0].start_time() == first_ts_expected
# check hist fc end
assert ohfc[-1].end_time() == last_ts_expected
- for hfc, ohfc in zip(hfc, ohfc):
- assert hfc.columns.equals(series.columns)
- assert ohfc.columns.equals(series.columns)
- assert len(ohfc) == n_pred_points_expected
- assert (hfc.time_index == ohfc.time_index).all()
+ for hfc_, ohfc_ in zip(hfc, ohfc):
+ assert hfc_.columns.equals(series.columns)
+ assert ohfc_.columns.equals(series.columns)
+ assert len(ohfc_) == n_pred_points_expected
+ assert (hfc_.time_index == ohfc_.time_index).all()
np.testing.assert_array_almost_equal(
- hfc.all_values(), ohfc.all_values()
+ hfc_.all_values(), ohfc_.all_values()
)
def test_hist_fc_end_exact_with_covs(self):
@@ -2853,3 +2855,543 @@ def test_historical_forecast_additional_sanity_checks(self):
assert str(err.value).startswith(
"Since `start_format='position'`, `start` must be an integer, received"
)
+
+ @pytest.mark.parametrize(
+ "config",
+ itertools.product(
+ [False, True], # use covariates
+ [True, False], # last points only
+ [True, False], # overlap end
+ [1, 3], # stride
+ [
+ 3, # horizon < ocl
+ 5, # horizon == ocl
+ 7, # horizon > ocl -> autoregression
+ ],
+ [False, True], # use integer indexed series
+ [False, True], # use multi-series
+ [0, 1], # output chunk shift
+ ),
+ )
+ def test_conformal_historical_forecasts(self, config):
+ """Tests historical forecasts output naive conformal model with last points only, covariates, stride,
+ different horizons and overlap end.
+ Tests that the returned dimensions, lengths and start / end times are correct.
+ """
+ (
+ use_covs,
+ last_points_only,
+ overlap_end,
+ stride,
+ horizon,
+ use_int_idx,
+ use_multi_series,
+ ocs,
+ ) = config
+ q = [0.1, 0.5, 0.9]
+ pred_lklp = {"num_samples": 1, "predict_likelihood_parameters": True}
+ # compute minimum series length to generate n forecasts
+ icl = 3
+ ocl = 5
+ horizon_ocs = horizon + ocs
+ min_len_val_series = icl + horizon_ocs + int(not overlap_end) * horizon_ocs
+ n_forecasts = 3
+ # get train and val series of that length
+ series_train, series_val = (
+ self.ts_pass_train[:10],
+ self.ts_pass_val[: min_len_val_series + n_forecasts - 1],
+ )
+ if use_int_idx:
+ series_train = TimeSeries.from_values(
+ series_train.all_values(), columns=series_train.columns
+ )
+ series_val = TimeSeries.from_times_and_values(
+ values=series_val.all_values(),
+ times=pd.RangeIndex(
+ start=series_train.end_time() + series_train.freq,
+ stop=series_train.end_time()
+ + (len(series_val) + 1) * series_train.freq,
+ step=series_train.freq,
+ ),
+ columns=series_train.columns,
+ )
+ # check that too short input raises error
+ series_val_too_short = series_val[:-n_forecasts]
+
+ # optionally, generate covariates
+ if use_covs:
+ pc = tg.gaussian_timeseries(
+ start=series_train.start_time(),
+ end=series_val.end_time() + max(0, horizon - ocl) * series_train.freq,
+ freq=series_train.freq,
+ )
+ fc = tg.gaussian_timeseries(
+ start=series_train.start_time(),
+ end=series_val.end_time()
+ + (max(ocl, horizon) + ocs) * series_train.freq,
+ freq=series_train.freq,
+ )
+ else:
+ pc, fc = None, None
+
+ # first train the ForecastingModel
+ model_kwargs = (
+ {}
+ if not use_covs
+ else {"lags_past_covariates": icl, "lags_future_covariates": (icl, ocl)}
+ )
+ forecasting_model = LinearRegressionModel(
+ lags=icl, output_chunk_length=ocl, output_chunk_shift=ocs, **model_kwargs
+ )
+ forecasting_model.fit(series_train, past_covariates=pc, future_covariates=fc)
+
+ # add an offset and rename columns in second series to make sure that conformal hist fc works as expected
+ if use_multi_series:
+ series_val = [
+ series_val,
+ (series_val + 10)
+ .shift(1)
+ .with_columns_renamed(series_val.columns, "test_col"),
+ ]
+ pc = [pc, pc.shift(1)] if pc is not None else None
+ fc = [fc, fc.shift(1)] if fc is not None else None
+
+ # conformal model
+ model = ConformalNaiveModel(forecasting_model, quantiles=q)
+
+ # cannot perform auto regression with output chunk shift
+ if ocs and horizon > ocl:
+ with pytest.raises(ValueError) as exc:
+ _ = model.historical_forecasts(
+ series=series_val_too_short,
+ past_covariates=pc,
+ future_covariates=fc,
+ retrain=False,
+ last_points_only=last_points_only,
+ overlap_end=overlap_end,
+ stride=stride,
+ forecast_horizon=horizon,
+ **pred_lklp,
+ )
+ assert str(exc.value).startswith("Cannot perform auto-regression")
+ return
+
+ # compute conformal historical forecasts
+ hist_fct = model.historical_forecasts(
+ series=series_val,
+ past_covariates=pc,
+ future_covariates=fc,
+ retrain=False,
+ last_points_only=last_points_only,
+ overlap_end=overlap_end,
+ stride=stride,
+ forecast_horizon=horizon,
+ **pred_lklp,
+ )
+ # raises error with too short target series
+ with pytest.raises(ValueError) as exc:
+ _ = model.historical_forecasts(
+ series=series_val_too_short,
+ past_covariates=pc,
+ future_covariates=fc,
+ retrain=False,
+ last_points_only=last_points_only,
+ overlap_end=overlap_end,
+ stride=stride,
+ forecast_horizon=horizon,
+ **pred_lklp,
+ )
+ assert str(exc.value).startswith(
+ "Could not build the minimum required calibration input with the provided `series`"
+ )
+
+ if not isinstance(series_val, list):
+ series_val = [series_val]
+ hist_fct = [hist_fct]
+
+ for (
+ series,
+ hfc,
+ ) in zip(series_val, hist_fct):
+ if not isinstance(hfc, list):
+ hfc = [hfc]
+
+ n_preds_with_overlap = (
+ len(series)
+ - icl # input for first prediction
+ - horizon_ocs # skip first forecasts to avoid look-ahead bias
+ + 1 # minimum one forecast
+ )
+ if not last_points_only:
+ # last points only = False gives a list of forecasts per input series
+ # where each forecast contains the predictions over the entire horizon
+ n_pred_series_expected = n_preds_with_overlap
+ n_pred_points_expected = horizon
+ first_ts_expected = series.time_index[icl] + series.freq * (
+ horizon_ocs + ocs
+ )
+ last_ts_expected = series.end_time() + series.freq * horizon_ocs
+ # no overlapping means less predictions
+ if not overlap_end:
+ n_pred_series_expected -= horizon_ocs
+ last_ts_expected -= series.freq * horizon_ocs
+ else:
+ # last points only = True gives one contiguous time series per input series
+ # with only predictions from the last point in the horizon
+ n_pred_series_expected = 1
+ n_pred_points_expected = n_preds_with_overlap
+ first_ts_expected = series.time_index[icl] + series.freq * (
+ horizon_ocs + ocs + horizon - 1
+ )
+ last_ts_expected = series.end_time() + series.freq * horizon_ocs
+ # no overlapping means less predictions
+ if not overlap_end:
+ n_pred_points_expected -= horizon_ocs
+ last_ts_expected -= series.freq * horizon_ocs
+
+ # adapt based on stride
+ if stride > 1:
+ if not last_points_only:
+ n_pred_series_expected = n_pred_series_expected // stride + int(
+ n_pred_series_expected % stride
+ )
+ else:
+ n_pred_points_expected = n_pred_points_expected // stride + int(
+ n_pred_points_expected % stride
+ )
+ first_ts_expected = hfc[0].start_time()
+ last_ts_expected = hfc[-1].end_time()
+
+ cols_excpected = likelihood_component_names(
+ series.columns, quantile_names(q)
+ )
+ # check length match between optimized and default hist fc
+ assert len(hfc) == n_pred_series_expected
+ # check hist fc start
+ assert hfc[0].start_time() == first_ts_expected
+ # check hist fc end
+ assert hfc[-1].end_time() == last_ts_expected
+ for hfc_ in hfc:
+ assert hfc_.columns.tolist() == cols_excpected
+ assert len(hfc_) == n_pred_points_expected
+
+ @pytest.mark.parametrize(
+ "config",
+ itertools.product(
+ [False, True], # last points only
+ [None, 1, 2], # cal length
+ [False, True], # use start
+ ["value", "position"], # start format
+ [False, True], # use integer indexed series
+ [False, True], # use multi-series
+ [0, 1], # output chunk shift
+ ),
+ )
+ def test_conformal_historical_start_cal_length(self, config):
+ """Tests naive conformal model with start, train length, calibration set, and center forecasts against
+ the forecasting model's forecast."""
+ (
+ last_points_only,
+ cal_length,
+ use_start,
+ start_format,
+ use_int_idx,
+ use_multi_series,
+ ocs,
+ ) = config
+ q = [0.1, 0.5, 0.9]
+ pred_lklp = {"num_samples": 1, "predict_likelihood_parameters": True}
+ # compute minimum series length to generate n forecasts
+ icl = 3
+ ocl = 5
+ horizon = 5
+ horizon_ocs = horizon + ocs
+ add_cal_length = cal_length - 1 if cal_length is not None else 0
+ add_start = 2 * int(use_start)
+ min_len_val_series = icl + 2 * horizon_ocs + add_cal_length + add_start
+ n_forecasts = 3
+ # get train and val series of that length
+ series_train, series_val = (
+ self.ts_pass_train[:10],
+ self.ts_pass_val[: min_len_val_series + n_forecasts - 1],
+ )
+
+ if use_int_idx:
+ series_train = TimeSeries.from_values(
+ series_train.all_values(), columns=series_train.columns
+ )
+ series_val = TimeSeries.from_times_and_values(
+ values=series_val.all_values(),
+ times=pd.RangeIndex(
+ start=series_train.end_time() + series_train.freq,
+ stop=series_train.end_time()
+ + (len(series_val) + 1) * series_train.freq,
+ step=series_train.freq,
+ ),
+ columns=series_train.columns,
+ )
+
+ # first train the ForecastingModel
+ forecasting_model = LinearRegressionModel(
+ lags=icl,
+ output_chunk_length=ocl,
+ output_chunk_shift=ocs,
+ )
+ forecasting_model.fit(series_train)
+
+ # optionally compute the start as a positional index
+ start_position = icl + horizon_ocs + add_cal_length + add_start
+ start = None
+ if use_start:
+ if start_format == "value":
+ start = series_val.time_index[start_position]
+ else:
+ start = start_position
+
+ # add an offset and rename columns in second series to make sure that conformal hist fc works as expected
+ if use_multi_series:
+ series_val = [
+ series_val,
+ (series_val + 10)
+ .shift(1)
+ .with_columns_renamed(series_val.columns, "test_col"),
+ ]
+
+ # compute regular historical forecasts
+ hist_fct_all = forecasting_model.historical_forecasts(
+ series=series_val,
+ retrain=False,
+ start=start,
+ start_format=start_format,
+ last_points_only=last_points_only,
+ forecast_horizon=horizon,
+ )
+ # compute conformal historical forecasts (skips some of the first forecasts to get minimum required cal set)
+ model = ConformalNaiveModel(
+ forecasting_model, quantiles=q, cal_length=cal_length
+ )
+ hist_fct = model.historical_forecasts(
+ series=series_val,
+ retrain=False,
+ start=start,
+ start_format=start_format,
+ last_points_only=last_points_only,
+ forecast_horizon=horizon,
+ **pred_lklp,
+ )
+ # using a calibration series should not skip any forecasts
+ hist_fct_cal = model.historical_forecasts(
+ series=series_val,
+ cal_series=series_val,
+ retrain=False,
+ start=start,
+ start_format=start_format,
+ last_points_only=last_points_only,
+ forecast_horizon=horizon,
+ **pred_lklp,
+ )
+
+ if not isinstance(series_val, list):
+ series_val = [series_val]
+ hist_fct = [hist_fct]
+ hist_fct_all = [hist_fct_all]
+ hist_fct_cal = [hist_fct_cal]
+
+ for idx, (
+ series,
+ hfc,
+ hfc_all,
+ hfc_cal,
+ ) in enumerate(zip(series_val, hist_fct, hist_fct_all, hist_fct_cal)):
+ if not isinstance(hfc, list):
+ hfc = [hfc]
+ hfc_all = [hfc_all]
+ hfc_cal = [hfc_cal]
+
+ # multi series: second series is shifted by one time step (+/- idx);
+ # start_format = "value" requires a shift
+ add_start_series_2 = idx * int(use_start) * int(start_format == "value")
+
+ n_preds_without_overlap = (
+ len(series)
+ - icl # input for first prediction
+ - horizon_ocs # skip first forecasts to avoid look-ahead bias
+ - horizon_ocs # cannot compute with `overlap_end=False`
+ + 1 # minimum one forecast
+ - add_cal_length # skip based on train length
+ - add_start # skip based on start
+ + add_start_series_2 # skip based on start if second series
+ )
+ if not last_points_only:
+ n_pred_series_expected = n_preds_without_overlap
+ n_pred_points_expected = horizon
+ # seconds series is shifted by one time step (- idx)
+ first_ts_expected = series.time_index[
+ start_position - add_start_series_2 + ocs
+ ]
+ last_ts_expected = series.end_time()
+ else:
+ n_pred_series_expected = 1
+ n_pred_points_expected = n_preds_without_overlap
+ # seconds series is shifted by one time step (- idx)
+ first_ts_expected = (
+ series.time_index[start_position - add_start_series_2]
+ + (horizon_ocs - 1) * series.freq
+ )
+ last_ts_expected = series.end_time()
+
+ cols_excpected = likelihood_component_names(
+ series.columns, quantile_names(q)
+ )
+ # check historical forecasts dimensions
+ assert len(hfc) == n_pred_series_expected
+ # check hist fc start
+ assert hfc[0].start_time() == first_ts_expected
+ # check hist fc end
+ assert hfc[-1].end_time() == last_ts_expected
+ for hfc_ in hfc:
+ assert hfc_.columns.tolist() == cols_excpected
+ assert len(hfc_) == n_pred_points_expected
+
+ # with a calibration set, we can calibrate all possible historical forecasts from base forecasting model
+ assert len(hfc_cal) == len(hfc_all)
+ for hfc_all_, hfc_cal_ in zip(hfc_all, hfc_cal):
+ assert hfc_all_.start_time() == hfc_cal_.start_time()
+ assert len(hfc_all_) == len(hfc_cal_)
+ assert hfc_all_.freq == hfc_cal_.freq
+
+ # the center forecast must be equal to the forecasting model's forecast
+ np.testing.assert_array_almost_equal(
+ hfc_all_.all_values(), hfc_cal_.all_values()[:, 1:2]
+ )
+
+ # check that with a calibration set, all prediction intervals have the same width
+ vals_cal_0 = hfc_cal[0].values()
+ vals_cal_i = hfc_cal_.values()
+ np.testing.assert_array_almost_equal(
+ vals_cal_0[:, 0] - vals_cal_0[:, 1],
+ vals_cal_i[:, 0] - vals_cal_i[:, 1],
+ )
+ np.testing.assert_array_almost_equal(
+ vals_cal_0[:, 1] - vals_cal_0[:, 2],
+ vals_cal_i[:, 1] - vals_cal_i[:, 2],
+ )
+
+ @pytest.mark.parametrize(
+ "config",
+ list(
+ itertools.product(
+ [False, True], # last points only
+ [None, 2], # cal length
+ ["value", "position"], # start format
+ [1, 2], # stride
+ [0, 1], # output chunk shift
+ )
+ ),
+ )
+ def test_conformal_historical_forecast_start(self, caplog, config):
+ """Tests naive conformal model with `start` being the first forecastable index is identical to a start
+ before forecastable index (including stride).
+ """
+ (
+ last_points_only,
+ cal_length,
+ start_format,
+ stride,
+ ocs,
+ ) = config
+ # TODO: adjust this test (the input length of `series_val`), once `stride_cal` has been properly implemented
+ q = [0.1, 0.5, 0.9]
+ pred_lklp = {"num_samples": 1, "predict_likelihood_parameters": True}
+ # compute minimum series length to generate n forecasts
+ icl = 3
+ ocl = 5
+ horizon = 2
+ horizon_ocs = horizon + ocs
+ add_cal_length = cal_length - 1 if cal_length is not None else 0
+ min_len_val_series = icl + 2 * horizon_ocs + add_cal_length
+ n_forecasts = 3
+ # to get `n_forecasts` with `stride`, we need more points
+ n_forecasts_stride = stride * n_forecasts - int(1 % stride > 0)
+ # get train and val series of that length
+ series_train, series_val = (
+ self.ts_pass_train[:10],
+ self.ts_pass_val[: min_len_val_series + n_forecasts_stride - 1],
+ )
+
+ # first train the ForecastingModel
+ forecasting_model = LinearRegressionModel(
+ lags=icl,
+ output_chunk_length=ocl,
+ output_chunk_shift=ocs,
+ )
+ forecasting_model.fit(series_train)
+
+ # optionally compute the start as a positional index
+ start_position = icl + horizon_ocs + add_cal_length
+ if start_format == "value":
+ start = series_val.time_index[start_position]
+ start_too_early = series_val.time_index[start_position - stride]
+ else:
+ start = start_position
+ start_too_early = start_position - stride
+ start_first_fc = series_val.time_index[start_position] + series_val.freq * (
+ horizon_ocs - 1 if last_points_only else ocs
+ )
+ too_early_warn_exp = "is before the first predictable/trainable historical"
+
+ hfc_params = {
+ "series": series_val,
+ "retrain": False,
+ "start_format": start_format,
+ "stride": stride,
+ "last_points_only": last_points_only,
+ "forecast_horizon": horizon,
+ }
+ # compute regular historical forecasts
+ hist_fct_all = forecasting_model.historical_forecasts(start=start, **hfc_params)
+ assert len(hist_fct_all) == n_forecasts
+ assert hist_fct_all[0].start_time() == start_first_fc
+ assert (
+ hist_fct_all[1].start_time() - stride * series_val.freq
+ == hist_fct_all[0].start_time()
+ )
+
+ # compute conformal historical forecasts (starting at first possible conformal forecast)
+ model = ConformalNaiveModel(
+ forecasting_model, quantiles=q, cal_length=cal_length, stride_cal=stride > 1
+ )
+ with caplog.at_level(logging.WARNING):
+ hist_fct = model.historical_forecasts(
+ start=start, **hfc_params, **pred_lklp
+ )
+ assert too_early_warn_exp not in caplog.text
+ caplog.clear()
+ assert len(hist_fct) == len(hist_fct_all)
+ assert hist_fct_all[0].start_time() == hist_fct[0].start_time()
+ assert (
+ hist_fct[1].start_time() - stride * series_val.freq
+ == hist_fct[0].start_time()
+ )
+
+ # start one earlier raises warning but still starts at same time
+ with caplog.at_level(logging.WARNING):
+ hist_fct_too_early = model.historical_forecasts(
+ start=start_too_early, **hfc_params, **pred_lklp
+ )
+ assert too_early_warn_exp in caplog.text
+ caplog.clear()
+ assert hist_fct_too_early == hist_fct
+
+ # using a calibration series should not skip any forecasts
+ hist_fct_cal = model.historical_forecasts(
+ start=start,
+ cal_series=series_val[:-horizon_ocs],
+ **hfc_params,
+ **pred_lklp,
+ )
+ assert len(hist_fct_all) == len(hist_fct_cal)
+ assert hist_fct_all[0].start_time() == hist_fct_cal[0].start_time()
+
+ # cal_series yields same calibration set on the last hist fc
+ assert hist_fct[-1] == hist_fct_cal[-1]
diff --git a/darts/tests/models/forecasting/test_local_forecasting_models.py b/darts/tests/models/forecasting/test_local_forecasting_models.py
index b9d0bf5084..e1e7361a60 100644
--- a/darts/tests/models/forecasting/test_local_forecasting_models.py
+++ b/darts/tests/models/forecasting/test_local_forecasting_models.py
@@ -142,8 +142,6 @@ def test_save_model_parameters(self):
@pytest.mark.parametrize("model", [ARIMA(1, 1, 1), LinearRegressionModel(lags=12)])
def test_save_load_model(self, tmpdir_module, model):
# check if save and load methods work and if loaded model creates same forecasts as original model
- cwd = os.getcwd()
- os.chdir(tmpdir_module)
model_path_str = type(model).__name__
model_path_pathlike = pathlib.Path(model_path_str + "_pathlike")
model_path_binary = model_path_str + "_binary"
@@ -186,8 +184,6 @@ def test_save_load_model(self, tmpdir_module, model):
for loaded_model in loaded_models:
assert model_prediction == loaded_model.predict(self.forecasting_horizon)
- os.chdir(cwd)
-
def test_save_load_model_invalid_path(self):
# check if save and load methods raise an error when given an invalid path
model = ARIMA(1, 1, 1)
diff --git a/darts/tests/models/forecasting/test_probabilistic_models.py b/darts/tests/models/forecasting/test_probabilistic_models.py
index 141fd43dcd..fd63793463 100644
--- a/darts/tests/models/forecasting/test_probabilistic_models.py
+++ b/darts/tests/models/forecasting/test_probabilistic_models.py
@@ -12,6 +12,7 @@
BATS,
TBATS,
CatBoostModel,
+ ConformalNaiveModel,
ExponentialSmoothing,
LightGBMModel,
LinearRegressionModel,
@@ -61,13 +62,16 @@
lgbm_available = not isinstance(LightGBMModel, NotImportedModule)
cb_available = not isinstance(CatBoostModel, NotImportedModule)
+# conformal models require a fitted base model
+# in tests below, the model is re-trained for new input series.
+# using a fake trained model should allow the same API with conformal models
+conformal_forecaster = LinearRegressionModel(lags=10, output_chunk_length=5)
+conformal_forecaster._fit_called = True
+
# model_cls, model_kwargs, err_univariate, err_multivariate
models_cls_kwargs_errs = [
(ExponentialSmoothing, {}, 0.3, None),
(ARIMA, {"p": 1, "d": 0, "q": 1, "random_state": 42}, 0.03, None),
-]
-
-models_cls_kwargs_errs += [
(
BATS,
{
@@ -92,6 +96,17 @@
0.04,
0.04,
),
+ (
+ ConformalNaiveModel,
+ {
+ "model": conformal_forecaster,
+ "cal_length": 1,
+ "random_state": 42,
+ "quantiles": [0.1, 0.5, 0.9],
+ },
+ 0.04,
+ 0.04,
+ ),
]
xgb_test_params = {
@@ -137,7 +152,7 @@
**tfm_kwargs,
},
0.06,
- 0.05,
+ 0.06,
),
(
BlockRNNModel,
@@ -285,7 +300,7 @@ def test_probabilistic_forecast_accuracy_multivariate(self, config):
def helper_test_probabilistic_forecast_accuracy(self, model, err, ts, noisy_ts):
model.fit(noisy_ts[:100])
- pred = model.predict(n=100, num_samples=100)
+ pred = model.predict(n=50, num_samples=100)
# test accuracy of the median prediction compared to the noiseless ts
mae_err_median = mae(ts[100:], pred)
diff --git a/darts/tests/models/forecasting/test_regression_models.py b/darts/tests/models/forecasting/test_regression_models.py
index d6d1b6db11..1424a1f865 100644
--- a/darts/tests/models/forecasting/test_regression_models.py
+++ b/darts/tests/models/forecasting/test_regression_models.py
@@ -1005,33 +1005,31 @@ def test_models_runnability(self, config):
model, mode = config
train_y, test_y = self.sine_univariate1.split_before(0.7)
# testing past covariates
+ model_instance = model(lags=4, lags_past_covariates=None, multi_models=mode)
with pytest.raises(ValueError):
# testing lags_past_covariates None but past_covariates during training
- model_instance = model(lags=4, lags_past_covariates=None, multi_models=mode)
model_instance.fit(
series=self.sine_univariate1,
past_covariates=self.sine_multivariate1,
)
+ model_instance = model(lags=4, lags_past_covariates=3, multi_models=mode)
with pytest.raises(ValueError):
# testing lags_past_covariates but no past_covariates during fit
- model_instance = model(lags=4, lags_past_covariates=3, multi_models=mode)
model_instance.fit(series=self.sine_univariate1)
# testing future_covariates
+ model_instance = model(lags=4, lags_future_covariates=None, multi_models=mode)
with pytest.raises(ValueError):
# testing lags_future_covariates None but future_covariates during training
- model_instance = model(
- lags=4, lags_future_covariates=None, multi_models=mode
- )
model_instance.fit(
series=self.sine_univariate1,
future_covariates=self.sine_multivariate1,
)
+ model_instance = model(lags=4, lags_future_covariates=(0, 3), multi_models=mode)
with pytest.raises(ValueError):
# testing lags_covariate but no covariate during fit
- model_instance = model(lags=4, lags_future_covariates=3, multi_models=mode)
model_instance.fit(series=self.sine_univariate1)
# testing input_dim
diff --git a/darts/tests/utils/test_utils.py b/darts/tests/utils/test_utils.py
index d629851cea..003d2253aa 100644
--- a/darts/tests/utils/test_utils.py
+++ b/darts/tests/utils/test_utils.py
@@ -1,3 +1,5 @@
+import itertools
+
import numpy as np
import pandas as pd
import pytest
@@ -15,6 +17,7 @@
n_steps_between,
quantile_interval_names,
quantile_names,
+ sample_from_quantiles,
)
@@ -631,3 +634,94 @@ def test_quantile_interval_names(self, config):
q, names_expected = config
names = quantile_interval_names(q, "a")
assert names == names_expected
+
+ @pytest.mark.parametrize("ndim", [2, 3])
+ def test_generate_samples_shape(self, ndim):
+ """Checks that the output shape of generated samples from quantiles and quantile predictions
+ is as expected."""
+ n_time_steps = 10
+ n_columns = 5
+ n_quantiles = 20
+ num_samples = 50
+
+ q = np.linspace(0, 1, n_quantiles)
+ q_pred = np.random.rand(n_time_steps, n_columns, n_quantiles)
+ if ndim == 2:
+ q_pred = q_pred.reshape((n_time_steps, n_columns * n_quantiles))
+ y_pred = sample_from_quantiles(q_pred, q, num_samples)
+ assert y_pred.shape == (n_time_steps, n_columns, num_samples)
+
+ @pytest.mark.parametrize(
+ "config",
+ itertools.product(
+ [1, 2], # n times
+ [2, 3], # ndim
+ [1, 2], # n components
+ ),
+ )
+ def test_generate_samples_output(self, config):
+ """Tests sample generation from quantiles and quantile predictions for:
+
+ - single/multiple time steps
+ - from 2 or 3 dimensions
+ - uni/multivariate
+ """
+ np.random.seed(42)
+ n_times, ndim, n_comps = config
+ num_samples = 100000
+
+ q = np.array([0.2, 0.5, 0.75])
+ q_pred = np.array([[[1.0, 2.0, 3.0]]])
+ if n_times == 2:
+ q_pred = np.concatenate([q_pred, np.array([[[5.0, 7.0, 9.0]]])], axis=0)
+ if n_comps == 2:
+ q_pred = np.concatenate([q_pred, q_pred + 1.0], axis=1)
+ if ndim == 2:
+ q_pred = q_pred.reshape((len(q_pred), -1))
+ y_pred = sample_from_quantiles(q_pred, q, num_samples)
+
+ q_pred = q_pred.reshape((q_pred.shape[0], n_comps, len(q)))
+ for i in range(n_comps):
+ # edges must be identical to min/max predicted quantiles
+ assert y_pred[:, i].min() == q_pred[:, i].min()
+ assert y_pred[:, i].max() == q_pred[:, i].max()
+
+ # check that sampled quantiles values equal to the predicted quantiles
+ assert np.quantile(y_pred[:, i], q[0], axis=1) == pytest.approx(
+ q_pred[:, i, 0], abs=0.02
+ )
+ assert np.quantile(y_pred[:, i], q[1], axis=1) == pytest.approx(
+ q_pred[:, i, 1], abs=0.02
+ )
+ assert np.quantile(y_pred[:, i], q[2], axis=1) == pytest.approx(
+ q_pred[:, i, 2], abs=0.02
+ )
+
+ # for each component and quantile, check that the expected ratio of sampled values is approximately
+ # equal to the quantile
+ assert (y_pred[:, i] == q_pred[:, i, 0:1]).mean(axis=1) == pytest.approx(
+ 0.2, abs=0.02
+ )
+ assert (
+ (q_pred[:, i, 0:1] < y_pred[:, i]) & (y_pred[:, i] <= q_pred[:, i, 1:2])
+ ).mean(axis=1) == pytest.approx(0.3, abs=0.02)
+ assert (
+ (q_pred[:, i, 1:2] < y_pred[:, i]) & (y_pred[:, i] < q_pred[:, i, 2:3])
+ ).mean(axis=1) == pytest.approx(0.25, abs=0.02)
+ assert (y_pred[:, i] == q_pred[:, i, 2:3]).mean(axis=1) == pytest.approx(
+ 0.25, abs=0.02
+ )
+
+ # between the quantiles, the values must be linearly interpolated
+ # check that number of unique values is approximately equal to the difference between two adjacent quantiles
+ mask1 = (q_pred[:, i, 0:1] < y_pred[:, i]) & (
+ y_pred[:, i] < q_pred[:, i, 1:2]
+ )
+ share_unique1 = len(np.unique(y_pred[:, i][mask1])) / num_samples
+ assert share_unique1 == pytest.approx(n_times * (q[1] - q[0]), abs=0.05)
+
+ mask2 = (q_pred[:, i, 1:2] < y_pred[:, i]) & (
+ y_pred[:, i] < q_pred[:, i, 2:3]
+ )
+ share_unique2 = len(np.unique(y_pred[:, i][mask2])) / num_samples
+ assert share_unique2 == pytest.approx(n_times * (q[2] - q[1]), abs=0.05)
diff --git a/darts/utils/historical_forecasts/optimized_historical_forecasts_regression.py b/darts/utils/historical_forecasts/optimized_historical_forecasts_regression.py
index 7230a6d06e..30f293c3b9 100644
--- a/darts/utils/historical_forecasts/optimized_historical_forecasts_regression.py
+++ b/darts/utils/historical_forecasts/optimized_historical_forecasts_regression.py
@@ -37,7 +37,9 @@ def _optimized_historical_forecasts_last_points_only(
Rely on _check_optimizable_historical_forecasts() to check that the assumptions are verified.
"""
forecasts_list = []
- iterator = _build_tqdm_iterator(series, verbose)
+ iterator = _build_tqdm_iterator(
+ series, verbose, total=len(series), desc="historical forecasts"
+ )
for idx, series_ in enumerate(iterator):
past_covariates_ = past_covariates[idx] if past_covariates is not None else None
future_covariates_ = (
@@ -200,7 +202,9 @@ def _optimized_historical_forecasts_all_points(
Rely on _check_optimizable_historical_forecasts() to check that the assumptions are verified.
"""
forecasts_list = []
- iterator = _build_tqdm_iterator(series, verbose)
+ iterator = _build_tqdm_iterator(
+ series, verbose, total=len(series), desc="historical forecasts"
+ )
for idx, series_ in enumerate(iterator):
past_covariates_ = past_covariates[idx] if past_covariates is not None else None
future_covariates_ = (
diff --git a/darts/utils/historical_forecasts/utils.py b/darts/utils/historical_forecasts/utils.py
index cca6af103e..89465a3289 100644
--- a/darts/utils/historical_forecasts/utils.py
+++ b/darts/utils/historical_forecasts/utils.py
@@ -9,9 +9,8 @@
from darts.logging import get_logger, raise_log
from darts.timeseries import TimeSeries
-from darts.utils import n_steps_between
-from darts.utils.ts_utils import get_series_seq_type, series2seq
-from darts.utils.utils import generate_index
+from darts.utils.ts_utils import SeriesType, get_series_seq_type, series2seq
+from darts.utils.utils import generate_index, n_steps_between
logger = get_logger(__name__)
@@ -477,7 +476,7 @@ def _get_historical_forecastable_time_index(
Returns
-------
- Union[pd.DatetimeIndex, pd.RangeIndex, Tuple[int, int], Tuple[pd.Timestamp, pd.Timestamp], None]
+ Union[pd.DatetimeIndex, pd.RangeIndex, tuple[int, int], tuple[pd.Timestamp, pd.Timestamp], None]
The longest time_index that can be used for historical forecasting, either as a range or a tuple.
Examples
@@ -1042,3 +1041,93 @@ def _process_predict_start_points_bounds(
bounds[:, 1] -= steps_too_long
cum_lengths = np.cumsum(np.diff(bounds) // stride + 1)
return bounds, cum_lengths
+
+
+def _process_historical_forecast_for_backtest(
+ series: Union[TimeSeries, Sequence[TimeSeries]],
+ historical_forecasts: Union[
+ TimeSeries, Sequence[TimeSeries], Sequence[Sequence[TimeSeries]]
+ ],
+ last_points_only: bool,
+):
+ """Checks that the `historical_forecasts` have the correct format based on the input `series` and
+ `last_points_only`. If all checks have passed, it converts `series` and `historical_forecasts` format into a
+ multiple series case with `last_points_only=False`.
+ """
+ # remember input series type
+ series_seq_type = get_series_seq_type(series)
+ series = series2seq(series)
+
+ # check that `historical_forecasts` have correct type
+ expected_seq_type = None
+ forecast_seq_type = get_series_seq_type(historical_forecasts)
+ if last_points_only and not series_seq_type == forecast_seq_type:
+ # lpo=True -> fc sequence type must be the same
+ expected_seq_type = series_seq_type
+ elif not last_points_only and forecast_seq_type != series_seq_type + 1:
+ # lpo=False -> fc sequence type must be one order higher
+ expected_seq_type = series_seq_type + 1
+
+ if expected_seq_type is not None:
+ raise_log(
+ ValueError(
+ f"Expected `historical_forecasts` of type {expected_seq_type} "
+ f"with `last_points_only={last_points_only}` and `series` of type "
+ f"{series_seq_type}. However, received `historical_forecasts` of type "
+ f"{forecast_seq_type}. Make sure to pass the same `last_points_only` "
+ f"value that was used to generate the historical forecasts."
+ ),
+ logger=logger,
+ )
+
+ # we must wrap each fc in a list if `last_points_only=True`
+ nested = last_points_only and forecast_seq_type == SeriesType.SEQ
+ historical_forecasts = series2seq(
+ historical_forecasts, seq_type_out=SeriesType.SEQ_SEQ, nested=nested
+ )
+
+ # check that the number of series-specific forecasts corresponds to the
+ # number of series in `series`
+ if len(series) != len(historical_forecasts):
+ error_msg = (
+ f"Mismatch between the number of series-specific `historical_forecasts` "
+ f"(n={len(historical_forecasts)}) and the number of `TimeSeries` in `series` "
+ f"(n={len(series)}). For `last_points_only={last_points_only}`, expected "
+ )
+ expected_seq_type = series_seq_type if last_points_only else series_seq_type + 1
+ if expected_seq_type == SeriesType.SINGLE:
+ error_msg += f"a single `historical_forecasts` of type {expected_seq_type}."
+ else:
+ error_msg += f"`historical_forecasts` of type {expected_seq_type} with length n={len(series)}."
+ raise_log(
+ ValueError(error_msg),
+ logger=logger,
+ )
+ return series, historical_forecasts
+
+
+def _extend_series_for_overlap_end(
+ series: Sequence[TimeSeries],
+ historical_forecasts: Sequence[Sequence[TimeSeries]],
+):
+ """Extends each target `series` to the end of the last historical forecast for that series.
+ Fills the values all missing dates with `np.nan`.
+
+ Assumes the input meets the multiple `series` case with `last_points_only=False` (e.g. the output of
+ `darts.utils.historical_forecasts.utils_process_historical_forecast_for_backtest()`).
+ """
+ series_extended = []
+ append_vals = [np.nan] * series[0].n_components
+ for series_, hfcs_ in zip(series, historical_forecasts):
+ # find number of missing target time steps based on the last forecast
+ missing_steps = n_steps_between(
+ hfcs_[-1].end_time(), series[0].end_time(), freq=series[0].freq
+ )
+ # extend the target if it is too short
+ if missing_steps > 0:
+ series_extended.append(
+ series_.append_values(np.array([append_vals] * missing_steps))
+ )
+ else:
+ series_extended.append(series_)
+ return series_extended
diff --git a/darts/utils/timeseries_generation.py b/darts/utils/timeseries_generation.py
index bb9a6d8a1e..1094303736 100644
--- a/darts/utils/timeseries_generation.py
+++ b/darts/utils/timeseries_generation.py
@@ -746,6 +746,7 @@ def _build_forecast_series(
with_static_covs: bool = True,
with_hierarchy: bool = True,
pred_start: Optional[Union[pd.Timestamp, int]] = None,
+ time_index: Union[pd.DatetimeIndex, pd.RangeIndex] = None,
) -> TimeSeries:
"""
Builds a forecast time series starting after the end of an input time series, with the
@@ -764,24 +765,26 @@ def _build_forecast_series(
with_hierarchy
If set to `False`, do not copy the input_series `hierarchy` attribute
pred_start
- Optionally, give a custom prediction start point.
+ Optionally, give a custom prediction start point. Only effective if `time_index` is `None`.
+ time_index
+ Optionally, the index to use for the forecast time series.
Returns
-------
TimeSeries
New TimeSeries instance starting after the input series
"""
- time_index_length = (
- len(points_preds)
- if isinstance(points_preds, np.ndarray)
- else len(points_preds[0])
- )
-
- time_index = _generate_new_dates(
- time_index_length,
- input_series=input_series,
- start=pred_start,
- )
+ if time_index is None:
+ time_index_length = (
+ len(points_preds)
+ if isinstance(points_preds, np.ndarray)
+ else len(points_preds[0])
+ )
+ time_index = _generate_new_dates(
+ time_index_length,
+ input_series=input_series,
+ start=pred_start,
+ )
values = (
points_preds
if isinstance(points_preds, np.ndarray)
diff --git a/darts/utils/torch.py b/darts/utils/torch.py
index 710e0809b8..81edf78d01 100644
--- a/darts/utils/torch.py
+++ b/darts/utils/torch.py
@@ -4,24 +4,21 @@
"""
from functools import wraps
-from inspect import signature
-from typing import Any, Callable, TypeVar
+from typing import Callable, TypeVar
+import numpy as np
import torch.nn as nn
import torch.nn.functional as F
-from numpy.random import randint
from sklearn.utils import check_random_state
from torch import Tensor
from torch.random import fork_rng, manual_seed
-from darts.logging import get_logger, raise_if_not
+from darts.logging import get_logger, raise_log
+from darts.utils.utils import MAX_NUMPY_SEED_VALUE, MAX_TORCH_SEED_VALUE, _is_method
T = TypeVar("T")
logger = get_logger(__name__)
-MAX_TORCH_SEED_VALUE = (1 << 31) - 1 # to accommodate 32-bit architectures
-MAX_NUMPY_SEED_VALUE = (1 << 31) - 1
-
class MonteCarloDropout(nn.Dropout):
"""
@@ -53,26 +50,6 @@ def mc_dropout_enabled(self) -> bool:
return self._mc_dropout_enabled or self.training
-def _is_method(func: Callable[..., Any]) -> bool:
- """Check if the specified function is a method.
-
- Parameters
- ----------
- func
- the function to inspect.
-
- Returns
- -------
- bool
- true if `func` is a method, false otherwise.
- """
- spec = signature(func)
- if len(spec.parameters) > 0:
- if list(spec.parameters.keys())[0] == "self":
- return True
- return False
-
-
def random_method(decorated: Callable[..., T]) -> Callable[..., T]:
"""Decorator usable on any method within a class that will provide an isolated torch random context.
@@ -82,22 +59,22 @@ def random_method(decorated: Callable[..., T]) -> Callable[..., T]:
----------
decorated
A method to be run in an isolated torch random context.
-
"""
# check that @random_method has been applied to a method.
- raise_if_not(
- _is_method(decorated), "@random_method can only be used on methods.", logger
- )
+ if not _is_method(decorated):
+ raise_log(ValueError("@random_method can only be used on methods."), logger)
@wraps(decorated)
def decorator(self, *args, **kwargs) -> T:
if "random_state" in kwargs.keys():
+ # get random state for first time from model constructor
self._random_instance = check_random_state(kwargs["random_state"])
elif not hasattr(self, "_random_instance"):
+ # get random state for first time from other method
self._random_instance = check_random_state(
- randint(0, high=MAX_NUMPY_SEED_VALUE)
+ np.random.randint(0, high=MAX_NUMPY_SEED_VALUE)
)
-
+ # handle the randomness
with fork_rng():
manual_seed(self._random_instance.randint(0, high=MAX_TORCH_SEED_VALUE))
return decorated(self, *args, **kwargs)
diff --git a/darts/utils/utils.py b/darts/utils/utils.py
index f05699d44c..450f7e00e9 100644
--- a/darts/utils/utils.py
+++ b/darts/utils/utils.py
@@ -7,12 +7,13 @@
from enum import Enum
from functools import wraps
from inspect import Parameter, getcallargs, signature
-from typing import Callable, Optional, TypeVar, Union
+from typing import Any, Callable, Optional, TypeVar, Union
import numpy as np
import pandas as pd
from joblib import Parallel, delayed
from pandas._libs.tslibs.offsets import BusinessMixin
+from sklearn.utils import check_random_state
from tqdm import tqdm
from tqdm.notebook import tqdm as tqdm_notebook
@@ -25,6 +26,9 @@
logger = get_logger(__name__)
+MAX_TORCH_SEED_VALUE = (1 << 31) - 1 # to accommodate 32-bit architectures
+MAX_NUMPY_SEED_VALUE = (1 << 31) - 1
+
# Enums
class SeasonalityMode(Enum):
@@ -265,6 +269,26 @@ def _parallel_apply(
return returned_data
+def _is_method(func: Callable[..., Any]) -> bool:
+ """Check if the specified function is a method.
+
+ Parameters
+ ----------
+ func
+ the function to inspect.
+
+ Returns
+ -------
+ bool
+ true if `func` is a method, false otherwise.
+ """
+ spec = signature(func)
+ if len(spec.parameters) > 0:
+ if list(spec.parameters.keys())[0] == "self":
+ return True
+ return False
+
+
def _check_quantiles(quantiles):
raise_if_not(
all([0 < q < 1 for q in quantiles]),
@@ -587,3 +611,114 @@ def expand_arr(arr: np.ndarray, ndim: int):
if len(shape) != ndim:
arr = arr.reshape(shape + tuple(1 for _ in range(ndim - len(shape))))
return arr
+
+
+def sample_from_quantiles(
+ vals: np.ndarray,
+ quantiles: np.ndarray,
+ num_samples: int,
+):
+ """Generates `num_samples` samples from quantile predictions using linear interpolation. The generated samples
+ should have quantile values close to the quantile predictions. For the lowest and highest quantiles, the lowest
+ and highest quantile predictions are repeated.
+
+ Parameters
+ ----------
+ vals
+ A numpy array of quantile predictions/values. Either an array with two dimensions
+ (n times, n components * n quantiles), or with three dimensions (n times, n components, n quantiles).
+ In the two-dimensional case, the order is first by ascending column, then by ascending quantile value
+ `(comp_0_q_0, comp_0_q_1, ... comp_n_q_m)`
+ quantiles
+ A numpy array of quantiles.
+ num_samples
+ The number of samples to generate.
+ """
+ if not 2 <= vals.ndim <= 3:
+ raise_log(
+ ValueError(
+ "`vals` must have either two dimensions with `(n times, n components * n quantiles)` or three "
+ "dimensions with shape `(n times, n components, n quantiles)`"
+ )
+ )
+ n_time_steps = len(vals)
+ n_quantiles = len(quantiles)
+ if vals.ndim == 2:
+ if vals.shape[1] % n_quantiles > 0:
+ raise_log(
+ ValueError(
+ "`vals` with two dimension must have shape `(n times, n components * n quantiles)`."
+ )
+ )
+ vals = vals.reshape((n_time_steps, -1, n_quantiles))
+ elif vals.ndim == 3 and vals.shape[2] != n_quantiles:
+ raise_log(
+ ValueError(
+ "`vals` with three dimension must have shape `(n times, n components, n quantiles)`."
+ )
+ )
+ n_columns = vals.shape[1]
+
+ # Generate uniform random samples
+ random_samples = np.random.uniform(0, 1, (n_time_steps, n_columns, num_samples))
+ # Find the indices of the quantiles just below and above the random samples
+ lower_indices = np.searchsorted(quantiles, random_samples, side="right") - 1
+ upper_indices = lower_indices + 1
+
+ # Handle edge cases
+ lower_indices = np.clip(lower_indices, 0, n_quantiles - 1)
+ upper_indices = np.clip(upper_indices, 0, n_quantiles - 1)
+
+ # Gather the corresponding quantile values and vals values
+ q_lower = quantiles[lower_indices]
+ q_upper = quantiles[upper_indices]
+ z_lower = np.take_along_axis(vals, lower_indices, axis=2)
+ z_upper = np.take_along_axis(vals, upper_indices, axis=2)
+
+ y = z_lower
+ # Linear interpolation
+ mask = q_lower != q_upper
+ y[mask] = z_lower[mask] + (z_upper[mask] - z_lower[mask]) * (
+ random_samples[mask] - q_lower[mask]
+ ) / (q_upper[mask] - q_lower[mask])
+ return y
+
+
+def random_method(decorated: Callable[..., T]) -> Callable[..., T]:
+ """Decorator usable on any method within a class that will provide a random context.
+
+ The decorator will store a `_random_instance` property on the object in order to persist successive calls to the
+ RNG.
+
+ This is the equivalent to `darts.utils.torch.random_method` but for non-torch models.
+
+ Parameters
+ ----------
+ decorated
+ A method to be run in an isolated torch random context.
+ """
+ # check that @random_method has been applied to a method.
+ if not _is_method(decorated):
+ raise_log(ValueError("@random_method can only be used on methods."), logger)
+
+ @wraps(decorated)
+ def decorator(self, *args, **kwargs):
+ if "random_state" in kwargs.keys():
+ # get random state for first time from model constructor
+ self._random_instance = check_random_state(
+ kwargs["random_state"]
+ ).get_state()
+ elif not hasattr(self, "_random_instance"):
+ # get random state for first time from other method
+ self._random_instance = check_random_state(
+ np.random.randint(0, high=MAX_NUMPY_SEED_VALUE)
+ ).get_state()
+
+ # handle the randomness
+ np.random.set_state(self._random_instance)
+ result = decorated(self, *args, **kwargs)
+ # update the random state after the function call
+ self._random_instance = np.random.get_state()
+ return result
+
+ return decorator
diff --git a/docs/source/conf.py b/docs/source/conf.py
index 21a00c2efe..eb798536ba 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -49,10 +49,10 @@
"inherited-members": None,
"show-inheritance": None,
"ignore-module-all": True,
- "exclude-members": "ForecastingModel,LocalForecastingModel,FutureCovariatesLocalForecastingModel,"
+ "exclude-members": "LocalForecastingModel,FutureCovariatesLocalForecastingModel,"
+ "TransferableFutureCovariatesLocalForecastingModel,GlobalForecastingModel,TorchForecastingModel,"
+ "PastCovariatesTorchModel,FutureCovariatesTorchModel,DualCovariatesTorchModel,MixedCovariatesTorchModel,"
- + "SplitCovariatesTorchModel,TorchParametricProbabilisticForecastingModel,"
+ + "SplitCovariatesTorchModel,"
+ "min_train_series_length,"
+ "untrained_model,first_prediction_index,future_covariate_series,past_covariate_series,"
+ "initialize_encoders,register_datapipe_as_function,register_function,functions,"
diff --git a/docs/userguide/covariates.md b/docs/userguide/covariates.md
index 97f82c6d92..8df7dc94eb 100644
--- a/docs/userguide/covariates.md
+++ b/docs/userguide/covariates.md
@@ -154,6 +154,7 @@ GFMs are models that can be trained on multiple target (and covariate) time seri
| [TiDEModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.tide_model.html#darts.models.forecasting.tide_model.TiDEModel) | ✅ | ✅ | ✅ |
| [TSMixerModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.tsmixer_model.html#darts.models.forecasting.tsmixer_model.TSMixerModel) | ✅ | ✅ | ✅ |
| Ensemble Models (f) | ✅ | ✅ | ✅ |
+| Conformal Prediction Models (g) | ✅ | ✅ | ✅ |
**Table 1: Darts' forecasting models and their covariate support**
@@ -170,6 +171,8 @@ GFMs are models that can be trained on multiple target (and covariate) time seri
(f) Ensemble Model including [RegressionEnsembleModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.regression_ensemble_model.html#darts.models.forecasting.regression_ensemble_model.RegressionEnsembleModel), and [NaiveEnsembleModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.baselines.html#darts.models.forecasting.baselines.NaiveEnsembleModel). The covariate support is given by the covariate support of the ensembled forecasting models.
+(g) Conformal Prediction Model including [ConformalNaiveModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.conformal_models.html#darts.models.forecasting.conformal_models.ConformalNaiveModel), and [ConformalQRModel](https://unit8co.github.io/darts/generated_api/darts.models.forecasting.conformal_models.html#darts.models.forecasting.conformal_models.ConformalQRModel). The covariate support is given by the covariate support of the underlying forecasting model.
+
----
## Quick guide on how to use past and/or future covariates with Darts' forecasting models
diff --git a/examples/23-Conformal-Prediction-examples.ipynb b/examples/23-Conformal-Prediction-examples.ipynb
new file mode 100644
index 0000000000..10573d8b43
--- /dev/null
+++ b/examples/23-Conformal-Prediction-examples.ipynb
@@ -0,0 +1,885 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "45bd6e88-1be9-4de1-9933-143eda71d501",
+ "metadata": {},
+ "source": [
+ "# Conformal Prediction Models\n",
+ "\n",
+ "The following is a in depth demonstration of the regression models in Darts - from basic to advanced features, including:\n",
+ "\n",
+ "- Darts' regression models\n",
+ "- lags and lagged data extraction\n",
+ "- covariates usage\n",
+ "- parameters output_chunk_length in relation with multi_models\n",
+ "- one-shot and auto-regressive predictions\n",
+ "- multi output support\n",
+ "- probablistic forecasting\n",
+ "- explainability\n",
+ "- and more"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "3ef9bc25-7b86-4de5-80e9-6eff27025b44",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ ""
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "# fix python path if working locally\n",
+ "from utils import fix_pythonpath_if_working_locally\n",
+ "\n",
+ "fix_pythonpath_if_working_locally()\n",
+ "\n",
+ "# activate javascript\n",
+ "from shap import initjs\n",
+ "\n",
+ "initjs()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "9d9d76e9-5753-4762-a1cb-c8c61d0313d2",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "The autoreload extension is already loaded. To reload it, use:\n",
+ " %reload_ext autoreload\n"
+ ]
+ }
+ ],
+ "source": [
+ "%load_ext autoreload\n",
+ "%autoreload 2\n",
+ "%matplotlib inline\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd\n",
+ "\n",
+ "from darts import concatenate, metrics\n",
+ "from darts.datasets import ElectricityConsumptionZurichDataset\n",
+ "from darts.models import ConformalNaiveModel, LinearRegressionModel"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "eacf6328-6b51-43e9-8b44-214f5df15684",
+ "metadata": {},
+ "source": [
+ "### Input Dataset\n",
+ "For this notebook, we use the Electricity Consumption Dataset from households in Zurich, Switzerland.\n",
+ "\n",
+ "The dataset has a quarter-hourly frequency (15 Min time intervals), but we resample it to hourly \n",
+ "frequency to keep things simple.\n",
+ "\n",
+ "**Target series** (the series we want to forecast):\n",
+ "- **Value_NE5**: Electricity consumption by households on grid level 5 (in kWh).\n",
+ "\n",
+ "**Covariates** (external data to help improve forecasts):\n",
+ "The dataset also comes with weather measurements that we can use as covariates. For simplicity, we use:\n",
+ "- **T [°C]**: Measured temperature\n",
+ "- **StrGlo [W/m2]**: Measured solar irradation\n",
+ "- **RainDur [min]**: Measured raining duration"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "ea0d05f6-03cc-4422-afed-36acb2b94fa7",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/dennisbader/miniconda3/envs/darts310/lib/python3.10/site-packages/xarray/groupers.py:403: FutureWarning: 'H' is deprecated and will be removed in a future version, please use 'h' instead.\n",
+ " self.index_grouper = pd.Grouper(\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAGvCAYAAABB3D9ZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABn60lEQVR4nO3ddXgU59oG8HuTIBE0WAhS0iBNoRQvGgKkQHB3ioTQQ6GlWClSpDgt2iJFkh6gSPEUSClSpBR3D0WKS9C4zfcHX/bsbiarMzs7m/t3XXvBZuyZ3ZFn33lFIwiCACIiIiIVclE6ACIiIiJrMZEhIiIi1WIiQ0RERKrFRIaIiIhUi4kMERERqRYTGSIiIlItJjJERESkWkxkiIiISLWYyFgpPT0dt27dQnp6utKhSIr7pS7Oul+Ac+6bM+4T4Lz7BTjvvjnTfjGRISIiItViIkNERESqxUSGiIiIVIuJDBEREakWExkiIiJSLSYyREREpFpMZIiIiEi1mMgQERGRajGRISIiItViIkNERESqxUSGiIiIVIuJDBEREakWExkiIiJSLSYyRKRqDRs2xNChQ5UOg4gUwkSGnM7Jkyexbds2pKWlKR0KmdCqVSs0adJEdNrff/8NPz8/nD592s5RGTdx4kRoNBp8+umnen8/e/YsNBoNbt++DQC4ffs2NBqN3svV1RV+fn44evQoAODPP//MNI9Go8HVq1ftvVtEqsVERiHJyclKh+CU7ty5gxo1aqBt27b4+eeflQ6HTOjfvz/27duHO3fuZJoWHh6OgIAAVK1aVYHIjMudOzdWrFiB69evm5x3z549ePjwIR4+fIj79+/j2LFjqFatmt48165d087z8OFDlC1bVq7QiZwOExkFdOvWDfnz58fGjRuVDsXpLF26VPv//v37KxgJmaNly5YoUqQIIiIi9P4eHx+PDRs2IDg4GN27d0eJEiXg4eGBSpUqYe3atUbXqdFosHXrVr2/5c+fX28b9+/fR5cuXVCgQAF4e3ujTZs22pIUc5QvXx5BQUEYN26cyXm9vb1RrFgx7atw4cLIkSOH3jxFihTRm8fV1dXsWIiyOzelA8huHjx4gHXr1gEAOnXqBEEQFI7IufDz1Fe9enU8evTIrtssVqwYTp48ada8bm5u6N27NyIiIvDNN99Ao9EAAH799VckJyejS5cuOHz4MEaPHo28efNix44d6NWrF/z8/FCrVi2r4ouPj0dQUBDq16+PgwcPws3NDVOmTEGzZs1w/vx55MyZ06z1zJgxAzVq1MCJEydQo0YNq2LJUKVKFSQmJiIgIADjxo1DUFCQTesjyk6YyNhZXFyc0iE4NSYy+h49eoT79+8rHYZR/fr1w+zZs/Hnn39qb+ArV65Eu3btUKxYMQwfPhwuLm8Lj4cMGYKoqCj8+uuvVicy69atg4uLC5YvX65NnMLDw5E/f378+eef+Pjjj81aT9WqVdG5c2eMHj0ae/fuzXK+OnXqaOMH3h6jL1++hIuLC3x8fPDTTz+hWrVqSEpKwqpVq9C4cWP8+eefaNCggVX7R5TdMJGxs/3798u+jcTEROTOnVv27TiihIQEpUNwKMWKFXP4bVaoUAF16tTBypUrERQUhH/++QeHDh1CVFQU0tLSMG3aNGzYsAH3799HUlISkpKS4OnpaXV8p06dwo0bN5AnTx69vycmJuKff/6xaF1TpkzBe++9h927d6NIkSKi86xfvx7vvfceACA9PR3379/XPjoqX748ypcvr523du3auHv3Lr777jsmMkRmYiJjZ3InMsOGDcPChQsxe/bsbNkkdcGCBUqH4FDMfcSjtP79+2Pw4MH48ccfER4ejtKlS6Nx48YYO3Ysli9fjnnz5qFSpUrw9PTE0KFDjVaW12g0mUrmUlJStP9PT09HtWrVsGbNmkzLFi5c2KK43333XQwYMACjR4/GihUrROcpWbIk/P39tds2rB9j6KOPPsLq1astioMoO2NlXzvLqB8jl7lz5yI1NRVffvmlrNtxRIcOHVI6BLJS586d4erqil9++QU///wz+vbtC41GgxMnTqB169bo2bMnKleuDD8/P0RHRxtdV+HChfHw4UPt++joaMTHx2vfV61aFdHR0ShSpAj8/f31Xvny5bM49m+++QbXr1+X7Nw+c+YMfHx8JFkXUXbARIacBovi1cvLywtdunTBmDFj8ODBA/Tp0wcAULp0aezZswdHjhzBlStXMHDgQJOVlxs1aoQffvgBp0+fxsmTJ/Hpp5/qlYL06NEDhQoVQps2bXDo0CHcunULBw4cwBdffIF79+5ZHHvRokUxbNiwLEsDY2Ji8OjRI+3r6dOnSExMBADMmzcPW7duRXR0NC5duoSvv/4amzZtwuDBgy2Ogyi7YiJjR69evVI6BCKH1b9/f7x48QJNmjRBqVKlALyt3FulShU0bdoUDRs2RLFixdC2bVuj6/n+++9RsmRJNGjQAN27d8eIESPg4eGhne7h4YGDBw+iVKlSaN++Pd577z3069cPCQkJyJs3r1Wxjxw5El5eXqLTmjRpAh8fH/j4+MDX1xe1atXSNg9PTk7GiBEj8MEHH6B+/fo4fPgwduzYgfbt21sVB1F2pBHYzMMq6enpuHPnDkqXLq3XIsGYYcOGYe7cuXp/k/rjz2iFYe26rdkvR6G77xkyPgM175cxzrpfgHPumzPuE+C8+wU47745036xsq/M0tLS0L9/fzx58gQvXrxQOhwiIiKnwkRGZhEREewqn0hFsnpEBAC7du1C/fr17RgNEZnCREZmamn+SkRvnT17Nstpvr6+9guEiMzCREZhgiCI1u0gImVk9PlCROqg7ho+Di4pKQlLliwxOo89evolIlKrs2fPomfPnti5c6fSoZCDYomMjH788UeT8+h23EVERPqqVauG9PR0rFmzhmOpkSiWyMho+PDhJueR88Tcvn27Xtfs2U12HW+KyJmkp6cbnZ6YmIiIiAicOnXKThGRo2Ei48TatGmDmTNnKh2GXbx8+TLT39zcWOBI5OwmT56Mvn37onr16oiJiVE6HFIAExmFyV1UOn78eFnX7yieP3+udAhGxcfHm/xlSUSWmz59uvb/+/btUzASUgoTGXIKYkmCozxPP3HiBDw9PREQEICkpCSlwyFyWo5yzpN9MZEhpyB2AXOEZu2CIKBmzZoAgGvXrmHp0qUKR0RE5FyYyCiMvyCkYcnnmJKSgujoaBmj+R/DDhH5DJ9IPryeZk9MZBT2zTffoEaNGjhx4oTSoaiaJRewoKAglCtXDnPmzJExorcSExNl3wYRvcVEJntiIiOxR48eIS0tzez579y5g5MnT6JOnToyRuX8xIaCELuoXblyBX/99RcA85rH20rto8oS6YqIiED16tURGRmpdChEWrzKSigiIgLFixdHgwYNLP5lkJqaKlNU2UNCQoJZ87Vu3VrmSPQ5Qj0dss2LFy/QokULdOnSBcnJyUqHo6i+ffvi1KlTdj+PzMUS0OyJiYyE+vbtC0EQcOTIEZw/f17pcLKF06dPY9GiRShXrlymaWJJxM2bN42uTxAE3LlzR7L4DD158kS2dZM8Ro8ejZ07d2LDhg2YO3eu0uFkK5cuXbJo/tWrV8sUCTkyJjIyefTokdIhqFpKSgo2b95s9EIWGxuLatWq4bPPPsPQoUMl2W6fPn3wzjvvYNSoUZKszzCZMjX2FjmeP/74Q/v/I0eOKBiJNARBQGhoKGrUqIFr164pHU6Wli1bhooVK1q0DEtksicmMjJp1qyZ0iGo2ty5c9GhQwdUrFgRL168EJ3n8uXL2v+fOXMm03RrKv7997//BQDMnj3b4mXF8NGS+unWc3KGTg3/+OMPrFixAidPnkSbNm2UDidLYWFhSodAKsFEhhzSV199pf3/jh07ROcxlSTExcXh1q1bksYlhVGjRmHVqlVKh0FmcrZk9J9//tH+35FLZMxhWPLduXNnhSIhJXEwGnJ4WZWsmNMiqFy5crhz5w6KFSsmdVhmEbsJZpT2VK9eHe+99569QyIbsHmvY/n777/13nt5eSkUCSmJJTIO6MiRI+jYsSN27dqldCgOwZZEJjU1FVOmTJE6JLMZ+zWf0QycHM+TJ0+0LZScrUTGmcTFxem9Z6KZPbFExgHVrVsXALBp0yaemMi6XoK5NxhH/Qwd8bEXva1DEhISgpIlS+Lq1atKh0NGDBkyRO+9o57rJC+WyJDDy+riZG7fMUoyVmo0bdo0O0ZC5vr444+RmpqKW7du4ZdfflE6HDLi5cuXeu+ZyGRPTGTI4b1580b072poasnHEuoWGxur9543Ssdx+vTpTH+zpFd1ch5MZLKBN2/eYPbs2YiKilI6FKt88cUXmf52584d/Pzzz2Ytn3HzMawYSGSKIAhMRh3UpEmTMv2NiUz2xDoyDuTFixcoUKCA5OsdPXo0Fi1aBAC4ffs2SpcuLfk27C04ONjiEayVqO/Am6C6sQTGMrt378aTJ0/QrVs35MiRQ/L16yaW/G4oA0tkHIhYiYGlN2sxGUkMAPz55582r88RSPG5kLyuXbuGr776SrSzQrXizTNrV69eRfPmzfHJJ59g6dKlsmyjf//+GDdunOTfw507d7B48WIOIaJSTGQciNjJ2a1bNwUicS4Zn6u9S0eSkpKydYlMrVq1MGvWLFStWhWpqalo3749ateujXv37ikdmkWc7Tu0dpgFw4q1hn799Vft/w1bE2VFEAScOnXK7ME4w8PDMXXqVGzdulX0eikIAlq2bAmNRoNly5aZtU4AqF27NgYNGoR27dqZvQw5DiYyDiQ9PR03btzQ+5tuhTYpukfPjr8o7X0jEgQBzZo1Q6FChfTG6cluXr16pf3/ihUrsGXLFhw9ehT9+/dXMCrLOOP58vjxY6uWGzFihMSRvO3lunr16siVK1emitXG7N+/X/Tvb9680fYEbskQBw8fPgTgWGNpOeOxJxcmMg5EEAT06NEj099q1aoFb29vFClSRLUVdrOT48eP4/fff0dsbKzeUAvZ2cWLF7X/37dvn4KRGGd489i8ebNCkdiPbkmKMStWrDA63ZofDN999532/2PGjDF7ubS0NNHtWZIMObJff/0VPj4+mDBhgtKhqAITGQeSnp6O48ePZ/r78ePH8fz5c8TExKB58+YKRKZu9v5l8/r1a7tuT20c+Zfmnj179N4fPHhQ770jx24uwwSgc+fOOHDggOTrtdTChQvNnjerRMaS7yerec+dO2f2OuTSuXNnPH78GJMnT8b169eVDsfhMZGxUVxcHM6ePWt1ca0uWzpIS09Pz3TRFaOGC7HUicBPP/0EwPSF1txWTWfOnMG0adO0xdGGnK1OhRTU8pmIfadqid0WS5Ys0Xu/a9cuNGnSBNu3bzd7HVJ+Tln1HZUhq8fs5o611LNnT/j4+IgmcH379jVrHfbSsmVLpUPQM2bMGLRt29ah6rqx+bUN0tPTUbNmTVy9ehVFihSxeX0nTpywetmpU6fim2++sTkGRyBWKmUP3bt3NzmPIAioWrUqAGD79u04evRopnmyw43PFo7c14erq6vR6Wr4IWANQRBw4sQJrFixAv3790dISAgAYO/evWbvs1Q3tsuXL6NmzZpG57HlGDp79izWrFkDAGjYsGGm/XO0HsOjo6MRGxtrlwEx79+/j/79+6NMmTJYtGiR3rXs+fPn8Pb21r5//fq1wzwmZomMDa5du6b9Fa9Us72UlBR89913ZicxarjJ5sqVS/J1XrhwweQ8165dMzmP7gX02LFjovNcuXLF/MCyKUtalNiT2Pmh+zdnSGSyeiRTs2ZNLF261GQSkRUp6u/NmDED77//fqbBIA3Z8mgpJibG6HRHvEbOmDHD5DwvXrzATz/9ZNOjqL59++L333/HkiVLMtUPGz58uN77rCpcK4GJjA2kaEVkqzlz5mDkyJFmz6+GC7HYhUS3sqg1QkNDbVo+gzkXOXObnmZnlrQoMUYQBFy/fl3W41p33bt375ZtO45M7BHMq1evsG3bNpNJhyW+/vprs+b7+eefsW3bNsm26+ju3btn8hgfMGAABg4ciIoVK1p9b9JtZXn+/Hm9aYcOHbJqnfbAREblRo8erXQIkhM7YW1tspuYmGgyCXHEX2KOxtqEQeyzHT16tM0Xx9DQUJQvXx6dO3e2eh2XL19G7969sWXLFh4DWWjYsGGmv7Vr1w5t27ZFnz59AKjn/DGM07DLC0f0888/o1SpUplateratGkTgLel9FIkl4bnuiN/v0xkSBVsfW5tzg1Yil/19+/ft3kdjigxMRG1a9dG+fLlcefOHaPzvnnzxqwOzmbOnIkGDRogNTXV6rhWrlwJANi4caPVQ1DUr18fq1atQvv27UXjVmJoC3uzps5JxqOFjRs3AgA++eQTs5aLi4vDmjVrcPv2bYu3aYo153mnTp0kj8PQypUrMW7cOLMaMmR1rbt37x5++eUX0UfahtcdU+eoNVxcHDddcNzISFFpaWl49uyZItsWy/xtPYmsSVLELjqm1vPxxx9bvB01mDNnDo4ePYro6Gi88847Wc43ffp05MuXL1M9pwULFmS5THx8vCQxmtvaJCoqCm3bttXeiJ8/fy5pLKdOnUJERITDVRrNIHZ+ZfyaFxMZGWnWevPnz29yHkEQ4OXlhZ49e6JChQpmrVdqhufw2bNn9d5LXfJw4sQJ9O/fH1OnTjXr0dn06dONThe7Ls+ZM0fvvRw9wrNExkk58hebFXNu6BmtsXx8fCxqfikneyQyht9njRo1LH7WfPnyZYvmV4t//vlH731WzWPHjBljcdIoVf0W3YQkK/fu3UPz5s2xbds2NGrUSPJYrl27hurVq6Nv376YMmWKTeuSi6X72Lp1a8m2vW7dOu3/k5KSJFtvhi1btpicZ/bs2ZJv15jff/9d+3/dce+yYipxFPv+DK9dGXUKbSntNLUNR8JERoXOnz9v9tgk1jh8+DDOnj2L1NRUtGnTRrbtWMLWk8iaG9T169fZGVUWpCxtkKrSvDnr+fTTT41OtzWR0S1lsKVfKGe1detWWdevO6RLVkxV2Jb6hm3pjzBrfrRlVYpdrFgx0S4inA0TGRtI0QmeNSpXrmz1IwxzTlJ7FYnfuXMH3bt3N/rYIYM5FyhjHKGFmZoZHjdiN/wffvjBqnXLncisXr0aISEhOHbsmHYcHrljcWSFChWSZb2O3KrFElK3gLM0MTF1jRaLz7DENGO+mJgY1K5d26LtG9uOo2IiY4MvvvhCsW0fOHDArGJKQ450cHbu3Blr167FF198YVaFSluKSQVBsOqXljk3cFvFxMQolhRby7ClhyAIDtnsPD09Hb169cKuXbvw0UcfmZzfmrF6Vq9ejTlz5sjyqEQOcgxzcvjwYZw6dcrkfI78eEIulu6zNZ+ROY/UbOXI3x179rWB0gOUffbZZ7Ks114HrG4PvleuXJG18l9WCci2bdvwxx9/YNSoUaLTdUdwlsPdu3dRrlw5pKam4ty5cwgICJB1e1Jp0aIFXr58Kcm6fH19ERcXZ7JXXUO3bt3Sey/2HVvaGseSgQuBt6UQvXr1AgBZH/dKSY7ze/LkyaJ/T0tL0/teHflmKBex4/LNmzfw9PQULa1x1M/IUeMCWCKTbaWkpDhU6YzcJ0lW+9q2bVv8+OOPqF27tmgMcj9qGDZsGBITE5Gamop+/frJui1rJCYmiv5dygQvKSlJ22W8JYz1qZFBju9PNzn65ZdftP+fNGmS5NuS0sOHD9GmTRsMGzZM8nWLnV9BQUEoXry4XknNmTNnJN+2Ka9evcL69evx7NkzxVpi6tq9ezeKFCmCWrVqiR6f5vYSf+3aNUyYMMGs0uy7d+9aHKc5pPoxYyuLE5mpU6eiadOmCAwMRJcuXbTPRSMjI1GrVi3Ur19f+3r06JF2uUuXLqFbt26oW7cuwsLC9AZnS0xMxPjx49GgQQO0aNEiU1fXkZGRCAkJQWBgICZNmoSUlBRr9zfbEwQBFy9ehK+vLz788EOHKQ7/66+/UKNGDcyaNSvLeWxJvEwt++DBA9G/GzYjzugzQyq6Tbyz6mNi4cKF6Natmyz9bmTl5s2bqFatGtzd3aHRaEyWPto6PpY1/e/8/fffJueRI1nP6vqTVdLnKPr374/t27fL8hhT7HP+888/8eTJEzRt2lT7N3OGAZFajx490LVrV7Ro0cJk02Y5GP5Aatq0KRITE3Hy5Em9Fk0ZTPUBk/FZV69eHZMnT8Z7771nMgZrho8w59yxte6iVCxOZHr06IHIyEgcOHAA33zzDcaPH6+9ANesWROHDh3SvooVKwbgbZHrqFGj0LVrV+zbtw8VK1bUGxto6dKlePXqFXbu3Ilp06ZhxowZ2i/zxo0bmDt3Lr777jvs2LEDDx48wIoVK6TY92yrXbt2ePr0Kc6fP29VPRspvHjxQu/9d999h5MnT+Krr76S5XGOtXVkdAdJA8wbWNJaYmM0RUdH4/PPP8e6devQoUMH2batKykpCR988IHeRWr9+vVGlwkKCrJpm45UOmgJRy5uN7Rr1y7Z1m3s+zM1tpHcMip4Hz9+3Kxk057Hoi3VE+xdtcHasa3sweI6MrqdYWk0GiQnJ5ssrjt16hTc3d21TXkHDBiAJk2a4OHDh/Dx8cHOnTvx/fffw8vLC5UrV0aDBg2we/duDBgwAFFRUQgODtbWHQgNDcWUKVOybEaZnJyc6Vm1m5sbcubMaemuGqXW1g3p6el6FTUfPnyoty/p6emZDlg59nXu3LlZTsvqBDV83m4JQRCs+vWQnp5udP9t/WwMlzd8r9svzenTp7OcX8rv6Ny5c2Z1ca67TSl6XrZ134ytIyvWfG66x4SpC7k9vi9HYKqbf6X213C75t54Da+JYuuSgu6xFBMTk+mHk6llzJWWlmbyWDQsaRQ7n8xZr9TMafVlVWXfGTNmIDIyEklJSQgMDISfnx8uXbqEc+fOoXHjxihYsCC6dOmCjh07AnhbTO3v769d3t3dHSVKlMDNmzfh6emJmJgYvenlypXDpUuXtMvqNh8rW7Ys7t+/j8TEROTOnTtTbOHh4ZlG1u3UqZNNY7E4E8NfR69evTJZlClHd9dPnz7NclpWifHt27etHhk7JSXFZB8WYhc5U4O12frZGCYAhusz/JzEthcZGYkff/wRoaGh2nPOFsa+G1OxWOvFixdZrs/w+X5aWprojTM1NTXTOkz9ArdmHy5evIijR4+ibt26WXYMaGr9ctVZUMq9e/eMTr9z545V9aBsZfj5mzNEQEpKiuj3Zul39vr1a+TNmzdT6bOup0+f4s6dOxg/fjzWrFljVu/IGctY4vnz5yaPRcPHwz/99BN69eqFGzduICAgQPSR6qNHj2S5P+gqU6aMyXmsSmRGjx6NkSNH4uTJk9oLStWqVbFu3ToUK1YMly9fxogRI+Dt7Y2goCAkJCTA09NTbx2enp5ISEhAfHw8XF1d9ZIST09PbVfhhst6eXlp/y6WyPTt2zdTJUCWyPxPjhw59N7nyZMHpUuX1r4X2y/d6VnZvn07vvzyS/Tt2xfjxo0zOf+SJUuynJbVr5LSpUtbnciYU5IjVnRaokQJlCpVKstlzPlsjDG8yBmur0iRIkanp6ena7sBGDVqFIYPH25TPEDWvfYaKlWqlGSPVgoUKCC6b3fv3kXJkiX1fpX16NFDr4fYDK6urpnWIUefSCNGjMChQ4dQo0YNVKtWzei8GfFkPNrU3afHjx+jYMGCVh/TauLm5obx48fbfbuGx0OePHlMLpMjR45M10Sx49CYmTNnYuzYsfj000+NXiMKFSqE0qVLa5M8cyrPrlmzxuJBdAsWLGjy/Pr333/1pj979gwDBgzA4cOHMXbsWNHOQQVBsOhzkYvVW3d1dUWtWrVw4sQJ/P333/D19UXx4sXh4uKCihUromvXrtqxTNzd3TMVVcfFxcHd3R0eHh5IS0vT++UUFxcHDw8P0WUzHju4u7uLxpUzZ054eXnpvXLnzg0XFxfJX2pk2I27RqMxuV/R0dEmP4t27drh9u3bmDBhAv7991+bPrusbo66sVrKnMRT7JGWqbh1p1nzzNpwSANTn1XG32JiYjBy5EhERESYXN7Sl2Gym5VKlSrh2bNnkpwLYsdhxnoN/yaWxBjbf2OsaSmW0cDhxIkTJtf/6NEjDBs2DKVKlcLvv/+unf/3339HqVKl8N577yE1NVWW65MjXa9MldjIRez8MXc5U8ehsVfGUB2LFy82muxnfPeWOHDggMXLGDsmMv4v9mPv8OHDAN428hHTp08f5MiRAz179lT0uLX56E5PTxc9SHW/PD8/P72i4ISEBNy7dw9+fn7ImzcvvL299aZfv34dfn5+ostGR0fD19dXtDSGTLOmcpalQ8K///77NlVEs2YkXlPkqpSm+6jO0j5IxBjGmdVFsHXr1pgzZw4GDBhg8zYNmVvKcuXKFXz++eeSb9+ezOnEzRbLly/H/Pnz8eDBA4SEhGj/3rJlS6Snp+POnTtGEzNnYezxSnY2ePBgq5aztLWp3JVy165diwsXLsi6DWMsSmTi4+Oxa9cuxMfHIzU1FXv37sWpU6dQpUoVHDlyRHuwXr16FevXr0f9+vUBANWqVUNCQgIiIyORnJyMFStWICAgAD4+PgCAkJAQLF++HHFxcbhw4QIOHjyI4OBgAECzZs2wZ88eXL16FbGxsVi5cqUsPVNmF+aUTBjeyK5cuYJx48Zh7dq1Zm0jPj7epotzVs1qbXmcZ+2JbGo53cdoplr2SEnO8VMseVy0efNmSbapVOsHW7dr6rMy5zFdVo+/zp49i1atWjlFK83o6GilQwCgzHFm7BixtoWmscYS1rL1MbGSfcpYVEdGo9Fg27ZtmDlzpvbZ2JQpU+Dv74/IyEhMmDABiYmJKFy4MHr37q1NRnLmzIlZs2bh22+/xYwZMxAQEADdniAHDhyIKVOmoFmzZsibNy9Gjx6tbR3l7++PoUOH4ssvv0RcXBwaNWrkkB2HqYXhiWzOid2zZ0/t/ytXrmxW77OGJ8X58+dRsmRJFChQwOSyWZXIyNmPjLV0m0xLUV/E2mbiUrJk+47Up5MgCNrvOWMfTH3vth4Xf/75p9HptvxKrV27NhITE/Hbb7+hc+fOZtXvyJCSkoJhw4YhLS0Nc+bMUezRjqORo7RXCXIMZmtrIqJkU2yLEhl3d/csK2l++eWX+PLLL7Nc9v3338/yV3ru3LmNDnnfqlUrtGrVypJQKQu2dmB04MABsxIZ3YN69erV6NWrF4oUKYJ///3XZOXGrE4IJRIZUzd1qZMOcx4t2TLmlDksbYWwYcMGmSKxzPPnz1GlShVoNBrs27fPrKTZVqa6njDs8MySUkXdeoOvX7+2KJFZuHChdhDPYsWKYcaMGWYvKwdH6W/EHo00dDt7lYscn+fAgQMlX6e9qLPGKlltz549Ni2fcQLFxMRg/fr1ZhWNZoxF8+TJE2zbts3k/FklB0q0FLPkF729SlKy6kNJKm3btrVo/i5dusgTiIViY2Nx7tw5nD17Fl999ZVZy9h6Q+jdu7dF869evdqs+QxLUJ49e4a5c+dqu6UQo1svLTIyUvv/LVu22G1E+6woXcqYwR4jdMs1Bp4uc5qRW8qa3rUdBROZbE7sQm7ORSckJARdu3ZF165dLdqeOUW7WSUsttx0dIfLkItUj5ZMrdNUnYm//voLy5Yts/rmldH1gZqZ+0jH1n2dPXu2RfMfOHDArPkMmyq3b98ew4YNQ8WKFUXPj06dOqFAgQKZWrEBjlEa4iiJjDlDJNj6eRkmm3IkCJbWTXOEY0BOTGQoE3PG9MnoPMmaMTxMKV++vOTrzC4ePnyIevXqISwsDFmNSJwVJS925m7b3PkyEma1XsANk9CbN29q/2/YyV9MTAw2btyI1NRU9O3bV7R3blLO/PnzjU5X6zFqSMn9YCKTzYkdfFn1GWDres2l1g7C7HHzMPWLXrdOhrn1ItLS0tCsWTP4+fnh4sWLNsUnN1O9M2dw1JuDNV3km2JYZ0p3RG7AMSq4Our3IcbW89jS5U1VGJfC+fPnZd8GExlSjL0PvrFjx5rdBb4hR78YSnHDMPVoyVTRuDWf0fr16/H777/j9u3bilWqN1bZX1f79u3Nmu/kyZOYNm2aLSHJQopExnAdhp2GGQ4O6QjJqZpKiK5cuYInT57o/S0lJUW2ekYTJ06UZb26Fi9eLPs2lMREJpubN28eHj16hI0bN2LLli0mL7S6RdzWuHXrVqZu952FtQmaMYY3AFMjCR85csTibeiOIWPOY0U5WNrBlznGjh2Lq1evSr5eW0j9CA0wv7daJWUMo2Fv1pwPwNteqzOOydevX6N8+fLw8vLCf/7zHynDA6CuJM9ROf4ZQLLz8fFBp06d0L59e5OtmuxREdTYRfzZs2d2/aXtaKVApnoPXr58ucXrVMON0Fq2Jt5qYPj9sRfd/2ndurVVyz158gS7d+8GoP+ofcmSJapu3eOsnPcKRlaZOXOm0eliN3axJti2JACG40HprrNfv34YO3as1et2dIafm2ERtxyc+Reh3H3uyMVYnQZzHi3Z47hRA1MlmMa0bt0aFSpUwL59+/T+bqqelqXnk7kt2Rwd68iQw8gY6NMSEyZMkDSGb775Jstpuv1j2IOpi5LcJ6+l/ZRYg4mM/WzZssWs+XTHlzNkqh5Vo0aNjPY3Q+a7du1apk5EK1SogIiICIwePVqyITrINhb17EskduMW6wnWWW6OSv3KeP78uez1O+7du4fRo0fbpZMwKejW5TGX7hASjsCWwVQzGFYENjxGS5cubfM2nIWbm5vkyezQoUP1KlAfO3YMNWvW1L53lmufpVQzRAGRuV69eoX8+fNbPSiaGEerryIHQRCQkpKCihUryt7Veffu3VWTxABv47XU9OnTZYhEWYbngWGClx3OE3PJUSJn2Aps1apVeokM2R8TGbKI2EVSbODAkSNHSr5tR+9x1tvb26Zn8gAQFxcHb29viSIyTk1JDAAcPnxY6RBkYWniYTi/4RARTGTsi5+38lhHhmy2Y8cOu2xn0aJFdtmOJRITE3HmzBnJRq2eNWuWBFEZl5CQgPDwcNm3Q+bRaDR4/fo1Ro0ahUWLFpmsqGt447x8+XKm9dFbztwiz1KnTp3KctrBgwftGIn0WCJDFlHy14etw8zL4fjx46hatSq+/fZbSW4g9mja6eHhIfs2yDJff/21NlE3t4M0QRBEkx6WEPyPj4+P3ZtLO2oiOWbMmEyjsWdYsGCBzetnqyVSDSUPVkd+FDJ+/HiHvYABbx/LJScn8ybnoHRLG83tWLF9+/YoVqyYXCGRkzF27m/atMmOkUiPiQxZRMkboRJdrdt7f+UYpfvGjRvw9fVFqVKlbK7DI6fk5GSlQ1AFQRAQFxdn9rhTJK+Ma0R0dDRGjhyZ6VGfozB2LWvcuLEdI5EeExmymCOXjChJihIZUz0rW6NPnz54+fIlHj9+jPHjx0u+fqk4+3gwWbGm92G5xv0h69WvXx/fffed0mFkydgPhUqVKtkxEukxkSGLHDt2DM2bN1c6DIfkqI+WdPv5kWM8KKlYM7yCMzDW+Z2Ys2fPyhOIE7JnKd/jx4/tti1ryF2hl3VkSDUuXryIuLg4pcMgC+h2oObIrTiya/0dS2+2LVq0MJo0O1pvxkpy5MSdpOO4VzUispkgCHqJjFgvzI4iuyQyhomLpT04mxopfM2aNRbHRNZzxG4hLOWopcnmYiJD5MS2bduml8gcP35cwWis50xjB61du1bvvaWPlohIHxMZIiPUXkpw9+7dTGPzqNGMGTOUDkEycrRMIzKlbNmyWU6TokSGdWSISDZqSWSMXQjVXvSty5n2hdQjT548SocgGyYyREZYctNx1F/az58/VzoEmznTzV+KCtfO9Hk4g9GjRysdguJYIkPkoNT+aElNSQxLZMzniMN1ZGczZ85UOgSbqP38YiJD5MTmzp2rdAg2e/XqleoTSl1S3DQ+//xzCSKh7MSZziFDHDSSyIm9evVK6RBssmfPHgwePNiphi+QIpGx14jzlD2ovbIvExkiI5z5V4yjEfusw8LCFIhEXmovxid1cuZrGR8tERHZERMZImkxkSEisiMmMuRo1H5MMpEhMsKZi2MdTXb5rNV+0yDno/Y6MkxkiIyIiYnBkiVL2I08ScaRB+4k5+XMPxRY2ZfIiI8++ggA4O7ujvj4eIWjcW7OfKHVdfPmTaVDIJIcS2SIHFxCQgIuXLigdBjkBLZs2aJ0CER61P64k4kMkZlq1qypdAhERGSAiQyRmRITE5UOwalll0dLREpw5vOLiQwROQRnvtASKU3u84t1ZIiIiIiswESGiIgom7p8+TJWrVqldBg2YfNrIiKibCghIQGVKlVSOgybsUSGiBwC68gQyUfs/Lpy5Yqs67cXJjJE5BCYyBDZl9r7j8nARIaIiMjJif1QYCJDREREqsVEhohIQoa/GJ88eaJQJETZg5SJDOvIEBEZ2LRpk9IhEJEKMJEhIiJycnLXkWGJDBFle4YXQhcXXp6I5BQbG6t0CJLglYKIHBITGSLpiJWYTJgwQYFIpMcrBRE5BJbIENnXjRs3JFsXHy0RERlwlqahRCQvJjJE5JBYIkOkHiyRIaJsj4+WiOTjzEOA8EpBRA7B8ELLR0tEZA4mMkRERE6OJTJERESkWnKXcLKODBGRAWf+BUlE0mEiQ0QOgYkLkXyc+fxiIkNEDkEQBAiCgGvXriE9PV3pcIhIJZjIEJHD+Prrr1GhQgV06tSJrZaIVIR1ZIiIAMycORMAsHnzZoUjISK1YCJDRA7BmZ/hE5F8mMgQkUNKSUlROgQip+HMPxSYyBAREZFNWEeGiLI9Z/7FSETysTiRmTp1Kpo2bYrAwEB06dIFhw4d0k6LiIhAkyZN0KhRI8yfP1/vwnTp0iV069YNdevWRVhYGB4+fKidlpiYiPHjx6NBgwZo0aIFoqKi9LYZGRmJkJAQBAYGYtKkSSxyJsoG2ASbSD0U/SEiWOjWrVtCUlKSIAiCcPHiRSEwMFB49eqVcOjQIaFFixbC3bt3hadPnwodO3YUtm7dKgiCICQlJQkhISHC1q1bhcTERGH+/PlCaGiodp3z5s0ThgwZIrx580Y4e/asEBgYKNy+fVsQBEGIjo4WgoKChEuXLglv3rwRwsLChMWLF1satuTS0tIEAHzxxZdEr6JFi+q9X7x4seIx8cWXs7zKlSsn6z1s48aNCt2NBcHiEpl33nkHOXPmBPB27Ibk5GQ8e/YMO3fuRMeOHVGiRAkUKlQIPXv2xK5duwAAp06dgru7O9q0aYNcuXJhwIABuHz5srZUZufOnQgLC4OXlxcqV66MBg0aYPfu3QCAqKgoBAcHIyAgAF5eXggNDdWuV0xycjJiY2P1XomJiUhPT5f8RUTy4TlGJC0572Fy3GPNjdHNmoBnzJiByMhIJCUlITAwEH5+frh16xZCQkK085QrVw4//vgjAODmzZvw9/fXTnN3d0eJEiVw8+ZNeHp6IiYmRm96uXLlcOnSJe2ytWvX1k4rW7Ys7t+/j8TEROTOnTtTbOHh4Vi2bJne3zp16oTOnTtbs6tEZCdpaWl6758/f65QJETOJzk5GXfu3JFt/U+ePJFl/WXKlDE5j1WJzOjRozFy5EicPHkSN27cAADEx8fDy8tLO4+npyfi4+MBAAkJCfD09NRbh6enJxISEhAfHw9XV1e9pMTYshnbSEhIEE1k+vbtix49eujvpJubthRJKvy1SCQtFxf9AuJ8+fIpFAmR88mRIwdKly6tfS/1Paxw4cJ667cnqxIZAHB1dUWtWrWwdu1a+Pn5wcPDA7GxsdrpcXFx8PDwAPC2BCYuLk5v+bi4OLi7u8PDwwNpaWl6JSzGls3Yhru7u2hcOXPmlDxpISL7S0pKUjoEIqdi+GNB6nXLuX6j27Z1Benp6bh37x7KlCmjLZ0BgOvXr8PPzw8A4OfnpzctISEB9+7dg5+fH/LmzQtvb2+zl42Ojoavr69oaQwROY85c+YoHQIRqYBFiUx8fDx27dqF+Ph4pKamYu/evTh16hSqVKmCkJAQbNq0Cffv38ezZ8+wZs0aNG/eHABQrVo1JCQkIDIyEsnJyVixYgUCAgLg4+MDAAgJCcHy5csRFxeHCxcu4ODBgwgODgYANGvWDHv27MHVq1cRGxuLlStXatdLRM5DMGi+qdtFAxE5NsPz154serSk0Wiwbds2zJw5E4IgoGTJkpgyZQr8/f3h7++P6Oho9O7dG+np6Wjbti1at24N4O3jnlmzZuHbb7/FjBkzEBAQgMmTJ2vXO3DgQEyZMgXNmjVD3rx5MXr0aLzzzjsAAH9/fwwdOhRffvkl4uLi0KhRI/Tr10+6T4CIiMjJKZloyE0jOPPeySg9PR2urq5Kh0FERGRSRmFDBqnvYevXr1esdTCHKCAiIiLVYiJDREREqsVEhoiIiFSLiQwRERHZRMnqtkxkiIiInJwzt+thIkNERJQNpKWloW3btqhatSpu3bqldDiSsXqIAiIiIlIHjUaDrl27Ytu2bQCAnj17KhyRdFgiQ0RE5OQEQcDGjRu1748ePSr5+pXCRIaIiIhUi4kMERGRk2NlXyIiIiIHxESGiIjIyWk0GlnXzzoyREREJBs+WiIiIiLKAktkiIiIiKzARIaIiIhUi4kMERER2YSPloiIiIiswESGiIjIybHVEhEREZEDYiJDREREqsVEhoiIiFSLiQwRERGpFhMZIiIiJ8fKvkREREQOiIkMERGRk3v48KHSIciGiQwREZGTS0pKknX97NmXiIiIyApMZIiIiEi1mMgQERGRajGRISIiItViIkNERESqxUSGiIiIVIuJDBEREakWExkiIiJSLSYyREREZBN2iEdERERkBSYyREREpFpMZIiIiEi1mMgQERGRajGRISIiItViIkNEREQ2YaslIiIiIiswkSEiIiKbaDQaxbbNRIaIiIhswkdLRERERFZgIkNERESqxUSGiIiIVIuJDBEREakWExkiIiJSLSYyREREZBO2WiIiIiKyAhMZIiIisgk7xCMiIiLV4qMlIiIiIiswkSEiIiLVYiJDREREqsVExkrnzp1TOgQiIqJsj4mMlT799FOlQyAiIsr2mMhYKS4uTukQiIiIHAJbLanQpUuXlA6BiIgo22MiQ0RERDZhh3hERESkWny0RERERGQFJjJERESkWhYlMsnJyZg0aRJCQkIQGBiIsLAw3LhxAwAQGRmJWrVqoX79+trXo0ePtMteunQJ3bp1Q926dREWFoaHDx9qpyUmJmL8+PFo0KABWrRogaioKL3tRkZGarc5adIkpKSk2LLPREREJCElHy1BsEB8fLywbNky4dGjR0JqaqqwatUqoXXr1oIgCML27duFwYMHiy6XlJQkhISECFu3bhUSExOF+fPnC6Ghodrp8+bNE4YMGSK8efNGOHv2rBAYGCjcvn1bEARBiI6OFoKCgoRLly4Jb968EcLCwoTFixdbErYsAPDFF1988cUXX4CwcuVKxe7HFpXIuLu7IzQ0FEWLFoWrqyu6dOmCBw8e4OXLl0aXO3XqFNzd3dGmTRvkypULAwYMwOXLl7WlMjt37kRYWBi8vLxQuXJlNGjQALt37wYAREVFITg4GAEBAfDy8kJoaCh27dqV5baSk5MRGxur90pMTER6erqkLyIiInpL6nusJfdaN1sCP3/+PAoWLIj8+fMDeNttf+PGjVGwYEF06dIFHTt2BADcvHkT/v7+2uXc3d1RokQJ3Lx5E56enoiJidGbXq5cOW0/LTdv3kTt2rW108qWLYv79+8jMTERuXPnzhRTeHg4li1bpve3Tp06oXPnzrbsaibNmjXL9AiMiIgoO3r27Bnu3Lkj+XrLlCljch6rE5nY2FhMmzYNgwYNAgBUrVoV69atQ7FixXD58mWMGDEC3t7eCAoKQkJCAjw9PfWW9/T0REJCAuLj4+Hq6qqXlHh6eiI+Ph4AMi3r5eWl/btYItO3b1/06NFDfyfd3JAzZ05rd1VUvnz5JF0fERGRWnl7e6N06dKKbNuqRCYpKQnDhw9HvXr10KZNGwCAr6+vdnrFihXRtWtX7N+/H0FBQXB3d8/UpX9cXBzc3d3h4eGBtLQ0vRKWuLg4eHh4AECmZWNjY7V/F5MzZ07JkxYxSnb+Q0RE5EhcXFzg4qJMQ2iLt5qamooxY8agcOHCGDp0aJbz6d7o/fz8tK2bgLelKffu3YOfnx/y5s0Lb29vvenXr1+Hn5+f6LLR0dHw9fUVLY0hIiIi+xPU1CHe1KlTkZSUhIkTJ+olK0eOHMGLFy8AAFevXsX69etRv359AEC1atWQkJCAyMhIJCcnY8WKFQgICICPjw8AICQkBMuXL0dcXBwuXLiAgwcPIjg4GMDbuih79uzB1atXERsbi5UrV6J58+Y277itWCJDRESkPIseLT18+BCRkZHIlSsXgoKCtH9fsGABjh07hgkTJiAxMRGFCxdG7969tclIzpw5MWvWLHz77beYMWMGAgICMHnyZO3yAwcOxJQpU9CsWTPkzZsXo0ePxjvvvAMA8Pf3x9ChQ/Hll18iLi4OjRo1Qr9+/STYdSIiIlI7jaBkeZCKde/eHWvXrlU6DCIiIsUtW7YMoaGhimybQxQQERGRajGRsRLryBARESmPiQwRERHZRFWtloiIiIgcBRMZIiIiUi0mMkRERGQTPlpSIVb2JSIiUh4TGSIiIlItJjJERERkEz5aIiIiIrICExkiIiJSLSYyREREpFpMZIiIiMgmrCNDREREZAUmMkRERKRaTGSIiIjIJny0RERERGQFJjJERESkWkxkrMSxloiIiN7ioyUiIiIiKzCRISIiItViIkNEREQ24aMlIiIiIiswkSEiIiLVYiJDRERENuGjJRVi82siIiLlMZEhIiIi1WIiQ0RERDbhoyUiIiIiKzCRISIiItViIkNEREQ24aMlIiIiIiswkSEiIiLVYiJDRERENuGjJRV68OCB0iEQERFle0xkrLR3716lQyAiIsr2mMgQERGRTfhoiYiIiMgKTGSIiIhItZjIEBERkU3q1q2r2LY1gpIPtlRMo9EoHQIREZFDYB0ZIiIiIiswkSEiIiLVYiJDREREqsVEhoiIiFSLiYwV9uzZo3QIREREBCYyVnFzc1M6BCIiIgITGavkzJlT6RCIiIgITGSskp6ernQIREREBCYyVmEi49jy5cundAhERGQnTGSswETGsQUHBysdAhER2QkTGStwVAfHxu+HiCj7YCJjBZbIODYmMkRE2QcTGSswkXFsTGSIiLIPJjJWYCLj2JjIEBFlH0xkrMBExrExkSEiyj6YyFiBiQwREZFjYCJjBf7id2z8foiIsg8mMlbgjdKx8fshIso+mMiQ02EiQ0SUfTCRIafDRIaIKPtgIkNOh4kMEVH2wUSGnE5sbKzSIRARkZ0wkSGnc/DgQaVDICIiO2EiQ0RERKplUSKTnJyMSZMmISQkBIGBgQgLC8ONGze00yMiItCkSRM0atQI8+fP16urcOnSJXTr1g1169ZFWFgYHj58qJ2WmJiI8ePHo0GDBmjRogWioqL0thsZGand5qRJk5CSkmLt/hIREZEzESwQHx8vLFu2THj06JGQmpoqrFq1SmjdurUgCIJw6NAhoUWLFsLdu3eFp0+fCh07dhS2bt0qCIIgJCUlCSEhIcLWrVuFxMREYf78+UJoaKh2vfPmzROGDBkivHnzRjh79qwQGBgo3L59WxAEQYiOjhaCgoKES5cuCW/evBHCwsKExYsXWxK25LZv3y4A4IsvvvjiS8bXDz/8oHgMfJl+de7cWdF7skUlMu7u7ggNDUXRokXh6uqKLl264MGDB3j58iV27tyJjh07okSJEihUqBB69uyJXbt2AQBOnToFd3d3tGnTBrly5cKAAQNw+fJlbanMzp07ERYWBi8vL1SuXBkNGjTA7t27AQBRUVEIDg5GQEAAvLy8EBoaql2vmOTkZMTGxuq9EhMTkZ6eLumLiIjk9Z///EfpEMgMGo1G8nusJfdaN1uCP3/+PAoWLIj8+fPj1q1bCAkJ0U4rV64cfvzxRwDAzZs34e/vr53m7u6OEiVK4ObNm/D09ERMTIze9HLlyuHSpUvaZWvXrq2dVrZsWdy/fx+JiYnInTt3ppjCw8OxbNkyvb916tQJnTt3tmVX9Tx9+tSi+fPmzYvXr19Ltn0yrmLFirh48aLSYRCRje7cuaN0CGSGuLg42b6rMmXKmJzH6kQmNjYW06ZNw6BBgwAA8fHx8PLy0k739PREfHw8ACAhIQGenp56y3t6eiIhIQHx8fFwdXXVS0qMLZuxjYSEBNFEpm/fvujRo4f+Trq5IWfOnNbuaiaFCxe2aP5cuXLpvV+7di26desmWTykr0CBAkqHQEQSKF26tNIhkBk8PDwU/a6sSmSSkpIwfPhw1KtXD23atAHwdkd0+++Ii4uDh4cHgLclMHFxcXrriIuLg7u7Ozw8PJCWlqZXwmJs2YxtuLu7i8aWM2dOSZMWMS4utjX2Cg4OligSEiOwQzwip2DrtZbsw8XFRdHvyuItp6amYsyYMShcuDCGDh2q/XuZMmX0WjBdv34dfn5+AAA/Pz+9aQkJCbh37x78/PyQN29eeHt7m71sdHQ0fH19RUtjHBUTF/s6fPiw0iEQkUQiIyOVDoFMMHwKYm8WJzJTp05FUlISJk6cCI1Go/17SEgINm3ahPv37+PZs2dYs2YNmjdvDgCoVq0aEhISEBkZieTkZKxYsQIBAQHw8fHRLrt8+XLExcXhwoULOHjwoPbm36xZM+zZswdXr15FbGwsVq5cqV2vWsyfP1/pEIiIVOWHH34AALRs2VLv7/nz50eJEiWUCImy0KJFC2UDsKSJ04MHD4Rq1aoJderUEerVq6d9nT59WhAEQVi5cqXQqFEjoWHDhsK8efOE9PR07bIXL14UunTpItSpU0cIDQ0VHjx4oJ2WkJAgjB07VqhXr54QEhIi7Nq1S2+727dvF5o1aybUr19fmDBhgpCUlGRDQy3bWdr8WhAEvffPnj1TvLmcM78+/PBDo9PfffddxWPkiy++xF+ff/65sHr1aiE5OVl7zdWdni9fPmHlypWKx8nX/15KUz4CFWIi4ziv4cOH672vXbu20LJlS6PLLF++XPG4+eKLL/FXRv9junSn58uXTwgPD1c8Tr7eviZMmGDnO3BmrEnlALp06aJ0CKpVsGBBvffVq1dXKBIiEpMjRw6lQyAZVatWTekQONaSHPLnz2/R/O+//36mv/Xs2dPoMqamO4uKFStKvk7dul1EJK8xY8ZIvk6ew47DEb4LJjIyKFSokM3ryGi1lZUPPvjA5m04A7GTyNSJJThA8+y8efMqHQKRXWR0pWEuc26MbJZNung0WOHjjz82On39+vUWrc+colfDnokdIQu2B0uTDo1Gg6+++kr7fu3atVKHJAlLO1UkUiupr1UajSbbXP8M/fvvv0qHkIkjfBdMZKxg2FOvoapVq9q0viZNmmSaZ/jw4XrvHeHgsYfx48cbnW6Y6Gg0GtStWxc7d+7Exo0bRYemcIQSGWs0bNhQ6RDIyZUqVUrpEMySXa5/hkqWLKl0CA6JiYwD0Gg0ovVkCOjYsaNVyzVv3hwdOnQQLYJWayKji8cLyaFp06aSr9Mw6TBVOdScJCW7JDKVKlVSOgSTHOG7YCKjUo5w8NiDq6ur0emGn4M5n4szjF7u5mbTeK9EotSQ5AuCkG2uf2FhYUqHkKllqCFHOGaYyEhAd2ynXr16KRKD7gjh2ZlaEhlrTn7dfXOEiwc5HzmOKzmSDrkTGX9/f1nXn8FU1/5KVGr+5JNP9N6XK1fO7jFYiomMBJYuXYpLly5h2bJl2m61jTF1cFpTtLpq1SqTy2QHUiQyplqMWatPnz5wcXHBggULLF42ODhYlcnL9u3blQ6BLCBHkm/qnFyxYoXJ+YsXL673Xu4bvFzXAEOmzmlbEzZrxvnLGDrIXI5QOsZERgKenp4ICAhAaGhols1qv//+ewBAUFAQChQokGm64QFt6r2hd99915KQFdG+fXulQwBg+mItx4n5+eefIzw8HC9evMCQIUMs3saHH36o994RLh7mUHwMFrKIPfptMnxvToegug0gsnOrJUPu7u5GpwcGBlq8TrEGFLpq1apl8TrlxkTGToYNG4abN2/ijz/+sGp5W3+NO8KjJ1P1Xaxh6qQTY6p+iRy/9jISEbX3H2OYUJniLDccS1siAsBHH30kQyTyGjx4sOzbMGx95yzHiBxMJSqG1z/DVk3WfLaGP/QM1/HTTz/ZvA2pMZGRgLlJRpkyZbQ382LFimn/blhsKsbw4Kpfv74FEQJt2rSxaH6pmLNvtsiTJ4/ee3NOqgoVKmiHMli6dGmm6easQ+lmkEo9Zpo+fbpF8xt+lobdCCjF0oTs1KlTFs3frl07h7jAm1KmTBm997r1/aRi+DlMnDjR6HRzZJcO8Sy9bm/YsMHmbZq6tgQEBNi8Dallj6PBAe3atQvFixdHkyZNzGpinJaWpvfe1JhC//3vf/Xeq+Giao0qVarovTe3V9AjR47g1q1boq0CzLlIGjvZzWlRZE1Hf47A1nFzHKFkELCu7oAlgoKCVFmnyR48PT313ltzbMt9PjjK+WbqWmIYpxQ/sNR4bWIio5APP/wQd+/exR9//AEXFxe9m6fYgWTpY5kOHTrYHKOz0mg0yJEjB955550sp5ti7GQXmyb1Tc0RLh5ixDpz1OXr62unSPR5e3vrvTf1fdg6kKujfj+2EusDpmXLlkaXMfVZWNOFQnBwsHa+adOmmZzfUZh6tDxjxgy995YeR9Z8loZMPa53xGObiYyCdJMX3YNDrDLqRx99hPfeew8AEBERYdG6AfELtz1GpbX3QS/F9swpkbG0dYetiYyjXDxM7ceyZcsy/U33cUXRokUlj8kclj4ysfTRrSGNRqOKEhlTMRo2Q+7YsWOmHwCmKpSaGtLF0laagiAgT548uHLlCjZv3owRI0Zg/vz5Jtdhifj4eEnXZ67SpUtbNL8cx1izZs303jvKtccYJjISkOJgat26tfb/zZo1y7ROFxcXnDlzBjdu3MjUzl+MOQdfnTp1jE43NTyAI5KiV1Apnr/379/f5nUYUvrGaM5nK1bK9ddff6F69eoYN25cpjoZ9mJpK0ClP2tH0bNnz0x/M0xGDT+rKVOm6L3P+AGWFWtLQMuXL4927dohR44cokOR2OLQoUOSrg/IXB9JjOFnkZqaatE2pCg9MfXY1RETGyYyDmLs2LHo06cPBgwYgC+++EJ0nly5cpndzNrwYDPV5Fusy3ulHgNYQoqTyrCvE3P6eTF1o3PGZseCIFh1g69evTpOnDiBb7/9VoaoMt9YTdUfM4etPSerpUTGUob7JfaoxNZj39JHUeYsY6nKlStLuj4A+O677yxexlRJouGxrsbOB6XARMZBuLu7Izw8HD/99JMkLQcMDz5TrYfkaK1gyB4XdmtKZFq1aqX33pq+F3TJUUfGUS4mjnBzNiztMqzPYk79FlP7YapO2qJFi0xuQw1srYPx/vvvO8SxLcU6fv31V+3/lyxZYvP6DHl6ehr9rIoUKZLpbx4eHliwYAHq1auHI0eOZJpu2KjDkBSfi2Fpl0aj0StdUqqUVRcTGZl06tQJwNvBC5UgdgDnzp1bgUjkJUeLB1sr+4pNl+Jxle46lWp+6giJjGEpwOTJk/Xem+r2vWzZspn2w/CXrWHlYEMDBgwwOl0tJTKrV6+2aH7D/XKUJMQUUx39jR8/Hu3bt8fGjRuxa9cuRfoAyup4GTJkCA4dOiTa4s9UEiHFZ/vpp59m+tvOnTvRtGlTTJ06FeXLl7d5G7ZiIiMBsQMwIiICv/32G9avX69ARJlvdF5eXpnmkfqCJIW5c+fatLw9Rs41HCHYnKJ0KcZuMUxk+vTpY/M6TTFVOvXbb7/JHoMhw/MtX758eu9z5cpldJmqVatmWodh/ziGpXSGnGXQTlP15KRKxubOnQsfH59MwxEAmc9Ha5J0w3X8+OOPeu83bdpkdPm8efPCxcUFHTp0yFTZ1Z7M+bwXLFiAvHnzWtynk7UMj3WNRoMKFSogKioKY8aMsUsMpjCRkYmHhwdatGiRqcM2c+n+QjTVvFGMRqPBzp07kTNnTlSvXh0NGjQwOb+txo4da/F6DW/GAwcOtCkGORIywxtjr169TNYTMLwgmfr8zaHb31DXrl1leY5vSLfjOLFShpCQEJvWb02FcktvrtYMemdrL9TO0o2+VMN5DB06FA8ePEC/fv1MrkMsEdVNVgsXLmxyHYYlZobHQMaQMVktbw1Tj3kMGT5KMjeGIUOG4MWLFxg9enSmac5wzFmDiYyDat26NYYOHYq2bduKNmkF/ncTESv602g0aN68OZ4+fYpjx47Z5VHEZ599ZvEyhjd4Wx9/iZ3Is2fPNjmPMX/99ZdNy0vlP//5D0aNGoXhw4fj888/t3l9vXv3tnkdGo1GWww/bNgwm9dnDlNJhuH38/XXX2cqzZJ7sD7A9tIMsVJUezPsiNPwc5GixEaj0WiTgPLly4sm/RMmTED+/PmRM2dOs3qvNdW1RLt27awL1gixFl66DH8ImCr1M8bc67kzVicQw0RGAnI8C9doNJg7dy62bNmiN5yBrs2bN+Pvv/82OuJ2RpGpGKnjFitu1x1XpWrVqtq6Q1Ixp7mhrT2JfvDBBxbNb23rHmMyOvGbOXMmvvvuO9HK2aZaWxk+yxZrqWaK2H7t2bMHBw4cwKxZsyxenzUs7e0U0C9ZMDwfHKWlx1dffaX33l4jMBtjTvNfS5u2i+nVqxf++ecfnDt3TjRR9fb2xt27d3H//n3R4SVs/Q7tVddHd0iUoUOH2rxNQ4IgYNu2bQCAggULon///pK04nN0TGRULFeuXPjoo48kGYxRrlKGefPmoVGjRmjVqhWGDRuW6eKsRK+R5mzj2LFjaNq0KSIiIkR/3dkaZ82aNW1aHsh8w6hXr55Fy4tVaDRMdsy5SXl6eqJBgwZWHYdS1IcwZ7o9SmQyWjJpNBp06tTJ4hu6YQ+1UiTDU6dOtWl5U8epRqPR6zywW7duVm/Lz89P9LFSBi8vLxQqVCjLOKSWP39+ydfZtWtXbN68GQcOHJBlpHHgbWn+lStXcOPGjUw/4pwVExnKkhQjNXt7e2Pv3r3Yvn270YuUVKSq7FuzZk1ERUWZ1fmgGFM3IVOlJ4ZjQInF/OTJE8sD01GpUiWj08UqPVrao7Ep1tyAxLrJN0U3bjla3gBvv7MdO3bgypUrKFiwoMXLy/H419ZBWw1LP8TqSdWpUwdz585Fz549Je9hV0mGj0oNjztT9fl0OznN4OLignbt2mkfn+l+5y4uLpKVLFWoUEG07zBnxUTGSrojgJYtW1bBSP4no0dNcwahBEzfbG3tpt1edDtEM9VsVgpS3PjEKizqGjRokMl1/Pzzz0an6z7WA0w/UjH8hSi2n7olNlKUKlnzWZo6vsVK+YyVyEiVyLi6uiIkJET7GUn9eNGagS6t2betW7fC09MTnTt3ztQBZ1YlqJ9//jkmT55sl/NPjBwlMobf34QJE/TeV6hQwejyP/30k8ltTJ06VXterl+/PlslH1JiImOlzZs3o169ehgzZozDPIMcO3Ys7t69a/ZQ7o7Y/Noav/32G7y8vODv729WqycfHx+bticIgl7yZNj8V4o6MlL0ZWNqBHTD9+ZUIC5btix++OEH9OjRQ68DMWvZqx8g3QqkhiVR9jj2xTo7M8Uwrqwq/RtjTSlomzZt8OLFiyy7jpAiQctoGFC3bl2b1wXYJ5ExrJfWtm1bo8ubM65YqVKlcO3aNZw5cwYNGjRAcHAwGjZsiLx582L//v02x2ytjJK4jBazGb3NO+qPWyYyVipbtiz++9//ytbturVKlChh1kndvXt3s/4mN3NiNRy7RZeLiwuqV6+OR48e4erVq6K19A1PbimGul+9ejVy586N/Pnz2+UYMFXvQ+y94X5KVcn1s88+w+rVq1GqVCnR6RmP48xpmm2vjv0WL16MWrVqISQkBIMHD840XY4K+7rrlKLTMGsG3LS2/lxWrX6kqtM2d+5c7Nu3D1FRUVYtb0iOyr6mjgmxccUs3Qbwto+pjMRBo9Fg3759ePLkSaYSVbm5u7trh2vZvXs31q5dq+0w8fvvv8fhw4fx+++/2zUmczGRyaa8vLwQFBSkfd+oUSOL+zrJ6kZmCXOaBxq7SGVM8/T0lKTSszk0Gg0qVaqE+/fv4969e6IVEO3Rq6ulTSvt1bLjp59+wv79+7Fx40bZYxIjNvJ78eLFcfToUezYsQM5c+aU/PsRK73QHVtHis7L1FJqas5nmyNHDgQFBTlEE3NzWXPM6HaeaW7lXo1GY3V9QmuPkUGDBiEmJkbbJLxw4cLo2rWrtrTZ1dUVdevWhbu7u1XrlxsTmWxszJgx6NixIzp06IBvvvnG5vVpNBqLO2kzVVfEFHN+0VtycltSObJgwYJZtgrYu3ev2esBMlfuNYdhvz2m9tMezY6Bt0XwDRs2zPKip3uDt7aVS0axvtjy1owfZM5NKqOTNbFK0GL15Bo2bIg9e/Zg3759oo9QTPUga+v3Y/jIMysZj8ZN1fnI4IgjhztqkhceHo4vv/wSmzZtsvmRtpzc3NwcNkkxBxOZbMzDwwO//vorNm7cCA8Pj0zTDS9IpoqVBUEw2RW4qW0AwKhRo+Dl5YXw8HDRZXT71TEn8TDnwnrhwgVMmDABBw8eFJ2u2xV3kyZNTK7v2bNnJufRZdilujkxG7YqM7WMOYmMPW5CQ4YMwapVq3Do0CGUKVPGqh6K165di/379yMiIiLTNCnqF4lZtGgRDh06hC1btmSaJlbJVaPRoHHjxtqST0vrWIitz9b5y5cvjwULFqBmzZr4888/AQA7duzATz/9ZFbyrdFojPbJAzhuIqM7VpHhYzq5zgUfHx/MmTMH7du3t3ldcnLURNBcTGTIaoYHv5ubG9599118/PHHAICJEyeaXIfYxWLmzJl4+fKldvgCw+3s3bsXLVu2xIIFCyRLZCpWrIiJEydmaqWRYdy4cVi5ciWOHTtmcymSGGvG7rG0WN5w1GilLl45c+ZEz549Le73Rlfu3LnRsGFD0Y4BLU1kspo/Y3iRjMTVzc0N9erV0z7SO3r0KD788EOMGTPGrMes48aN03tvasA/w3o1UiQy3377LYYMGYJjx45px9EqUqQIBgwYYHZppKnBS+Xof0UK69evx4gRI/DHH3+I/nAzZE0i8/XXXwNQpr5hVqTuMsERMZHJRqS+cWk0Gm2y4e3trS3KzuhLw5bHVcbquwQEBCAyMhJDhgwxa126RbpZ9ZJsiru7O/r27Wt2k2Mp6g/pEvvuOnfuDH9/f+TIkcOsSpO2jmMlF6l/wVta8pTVeXHmzBksWrQIa9euFZ1eq1YtnDlzxuxO5wz3M6vO3TIY9jVk6vw17HZf7JGZNQlzRsmom5sbevfujZEjR2qnibV08/f3R1hYGHx9fbFv3z6Lt2cNc65tJUuWxOzZs0VLVKUqkZk2bRpu375t8ajittCtFCzWfNuc/VDDSO3GOMcQriSLlJQUo9NdXFywcOFCNG7cGPXr19feQNzc3CR73i6Ftm3bIjg4GBcvXrTLaM0dOnRAy5YtJe0cTOxCmyNHDly8eBEXL15ElSpVcPbsWaPrEBvF1hHIMZyDqfXrtuiqWLGi6Dzvvvsu/vOf/0gamyXMefyhy8/PD7179zY6eKE133nv3r1RokQJlC5dGgULFkS3bt0QHx+PHDlyoE2bNqLLLF26FIIg2O0Ys/UYMqdyrbnbKF26tE2xWGrdunUIDw9H48aNRUsov//+ezRq1MiuMdkbS2RIy/BENazIalgnQ6PRwMvLCz179jT75DW8sAUGBmr72DAckVYqLi4uiIqKwuHDh0XHaZHCihUrtP9ftGgRGjdujH79+qFKlSq4dOmSVevUfRTUuHFj0Xly5Mih7UXW0ou5OTcZ3XochmMBOSpz9uvLL79E5cqVUbx4cfzyyy+oWrWqdpot3ewbI3XfQrolI1JtQ4yLiwuaNGmirdDs4uKCAQMGoE+fPma1KHRUGf1tFSxYEL169co03ZzPMqNzRrGRqO2laNGiGD16NGrUqCE6vWHDhia7QnD078oUlsiQluGJW7RoUYwdOxZbt27F4sWLkZ6ebnPfBmKdTJ0+fRqXLl3K8mYtFTlP1n79+qFfv356f9NNbqwxe/ZsuLu7w8/PT1ufQUrmdCwXFBSEJUuW4OHDhxg1apTkMcjBnO/Z3d0dZ86cQVpamrakas+ePTh+/LjoaPJS+Pjjj7X1xqx5zGe4X3Xq1Mk0j+EjMyXGMlOCNQlcp06dcO7cOfj6+oq2PjRcp1grsPXr1+PmzZtZ1q1zBBqNBk2aNMHOnTuVDkU2TGRIS+xiMGXKlCw7pJOqMzNfX1/4+vpmOT2j0iWQdUddzqhAgQJYuHChbOs396bmqHVrsmLufmk0Gr3HbY0bN5Y1ma5duzYWLVqEs2fPYvr06bh165ZFy4vt16effoolS5YAeFspWbduhlgdmeyUyPTp0wcRERFGuzYwNrK94Tpr1aqF5s2b4/Dhw9rWmS4uLvD397cyapIKHy2RlhyPJqQQGhqKkiVLIkeOHNizZ49dtmkPO3fuRPXq1bF8+XKlQ1GUFI9DMh5LduzYMVOCba/eg80xcOBAjB49Gvny5ZPk/JkxYwaGDBmCmTNnZuqXRmz9zprI5MmTByVKlADwv/5+Vq5ciStXrmgTPVtpNBrs3LkTz549s2rcKyUZfu+GlbBv3rxpz3AkxxIZ0urZsydGjRqF+Ph4sx6LWNMCyJoLae7cuREdHY03b96YbOmhJs2bN0fz5s0lW5+lN2xHualJkcgMGzYMPXv2FG0a70iJjNTy5ctndCR1tbdGMZeLiwuOHj2KI0eOoEWLFgDeHt/mNjqwhFiFWjWqWbMmjh8/DgDo0aOHwtHYxnnPcLJYvnz5cP78eaxevRq9e/c2Ob89SxJy5crlVEmMHCpXrqwt5p4zZ45V68gYHA6AtrtytShSpIg2OVu6dCmAt8e0YbNkR2FOIrlw4UK4uLhkqn+VFVPNyh0leZWDr68vOnXqZFYfMeZwpiRQrCXf/v37UblyZYSGhqJLly4KRSYNlshkU1mdpGXKlIGLi4tZv2KteTbsTBcHqbVr107ba6w1lQddXFxw+vRpXL161awR2cX6W+nTpw+uX78OLy8vq4ZNcBShoaF49913UbZs2SyHkVCDwYMH45NPPtGrJ5bBnP5xDM83S4bgyO6c6VrVqVMnDB06VO9vuXPnxpYtW+zeXFwOTGSykfz58yMhIQGA5b3CkvyWLVuGjz76CPXr17f6+8mTJ0+WzTCBt50VDhs2DH369BHtV8bDwwPz5s2zattS8fPzw5YtW6wauiCDi4uL7K3g7EUsiQEgOjaO4c23WrVqeu91m5mTcc6UyDh7AstHS9nIhg0b4OLiAi8vL4wdO1aRGJy5aNtW3t7eGDVqlN6YMFILCQnB1atXRfu9UKouieENY8eOHUZbsWVna9asAfB2+AKxRE23P5SRI0eibNmy+Pnnn/HFF1/g8ePHdovTGWSMkQW87eTSmThb608mMtlIvXr1cOvWLfz777+ig9yZ46+//kJYWBhOnz5t1fK6owSb290/yadSpUra/xuO7WMvhiNHV6hQAd7e3ujevTty5syJadOmKRKX3DI6ggT+17Psb7/9hnbt2uHQoUOiy3Tv3h23bt3CuXPnRIfxaN68OZYuXYpZs2Zph/Do3bs35s2bp7c9Mq1p06aYOnUqwsLCMo2TpUYTJkwA8LajU3MePauKQFZJS0sTbt68KaSlpSkdiqTk3q/U1FShZcuWwgcffCD8888/smxDTHb9vgBoX2Kio6OFxo0bCyNHjpQzTKPu378vGmdaWppw5coVITU1VahUqZIAQJgyZYpicUrB8PuaPn26UL9+feHUqVMKR2YbZz2/BMF59i0xMVFYv369EB0dLQiC8+yXIAiCRhCc6EGgHaWnp+POnTsoXbq0UzXv5H6pi6n90n2U58inulicuvsWHx+Py5cvo0aNGqp+PJldj0M1c9Z9c6b9Unf0ROQUNm7ciNKlS2P69Omi0728vFCzZk1VJzFEJA+2WiIixXXo0MHpKlQSkX2wRIbIibVv3x4AzO5QjYhIbVgiQ+TE1q1bhzNnzmTqT4SIyFkwkSFyYjly5GAzdyJyany0RERERKrFRIaIiIhUi4kMERERqRYTGSIiIlItJjJERESkWkxkiIiISLWYyBAREZFqMZEhIiIi1WIiQ0RERKrFRIaIiIhUi4kMERERqRYTGSIiIlItJjJERESkWkxkiIiISLU0giAISgdBREREZA2WyBAREZFqMZEhIiIi1WIiQ0RERKrFRIaIiIhUi4kMERERqRYTGSIiIlItJjJERESkWkxkiIiISLWYyBAREZFqMZGhbOnBgweoU6eO0mEQOSWeX2RPTGQAJCcnY9KkSQgJCUFgYCDCwsJw48YN7fSIiAg0adIEjRo1wvz585ExqkNqaipGjhyJ5s2bo3r16nj27Jneejt37oz69etrXzVq1MDq1avtum9ZadOmDT755BOlw5BFq1atcOHCBaXDkNSGDRvQvn171K1bF61atcLSpUuRlpZmdJnIyEgMGTLEThEaJ9c5dv/+fXz22Wdo2LAhmjdvjvDwcLvuV1Z4fqkLzy/x8+vBgwcYPHgwAgMD0b59exw9etSu+2UuJjIA0tLS4Ovri/DwcOzbtw8NGjTA8OHDAQCHDx/Gxo0bERERgQ0bNuDw4cPYvn27dtmqVati1qxZouvdsGEDDh06hEOHDiEyMhJubm4IDAy0yz4Zc+7cObx8+RJXr17F3bt3LV4+NTVVhqgoK8uXL0dERATGjRuHAwcO4Pvvv8fevXsxffp0pUMzm1zn2OzZs+Hr64s9e/Zg+fLlWL9+PY4fP26XfcoKzy914fmV9fk1btw4BAQEYO/evRg3bhzGjh2Lly9f2mOXLMJEBoC7uztCQ0NRtGhRuLq6okuXLnjw4AFevnyJnTt3omPHjihRogQKFSqEnj17YteuXQAANzc3dOvWDZUqVTK5jT179qBChQooWbKk3LtjUlRUFIKCglCjRg3tvmQUBa9btw5NmjRB69atceDAAe0yrVq1ws8//4z27dujU6dOSoVukYkTJyIiIkL73pF+QZnrzZs3CA8Px1dffYWqVavCzc0N5cqVw7fffott27bh9u3bePHiBcaOHYvg4GA0btwYCxcuxL179zB9+nQcP34c9evXR7du3RTdD7nOsYcPH+Ljjz+Gm5sbfH198eGHH+LmzZv23LVMeH6pB8+vrM+vuLg4XLhwAf369YObmxuqVq2K9957D/v377f37pnEREbE+fPnUbBgQeTPnx+3bt2Cv7+/dlq5cuWsulDu2rULzZo1kzJMq6SmpuKPP/5AcHAwgoODERUVpZ2WkpKCf/75Bzt27MDo0aMxfvx4vHjxQjv9wIEDWL58OdatW6dE6NnS+fPnkZqainr16un9vXz58vDx8cGpU6cwbtw4uLu7Y9u2bdixYwcCAwNRokQJfP3116hZsyYOHTqEtWvXKrQH4qQ6xzp16oTff/8dycnJ+Pfff3HhwgVUr15drrBN4vmlLjy/TMt4DJXxf6V/KIhhImMgNjYW06ZNw6BBgwAA8fHx8PLy0k739PREfHy8Ret88OABLl26hODgYEljtcbff/8NQRBQq1YtBAUFaWMD3h6kYWFhyJUrF+rUqYOKFSvir7/+0i7bvXt3FCxYELly5VIq/Gzn1atXyJ8/P1xdXTNNK1iwIF6+fImzZ89ixIgR8PDwQO7cufHBBx8oEKn5pDzHKleujAsXLqB+/fpo37492rRpo3fRtjeeX+rC8ytrnp6eqFixIlauXImUlBScPHkSp0+fRmJiomyxW4uJjI6kpCQMHz4c9erVQ5s2bQAAHh4eiI2N1c4TFxcHDw8Pi9YbFRWFmjVromDBgpLGa42dO3ciKCgIbm5uyJs3Lz766CPtr0YXFxcUKlRIO2/RokX1Kn8VKVLE7vFmd/ny5cPLly9FKx4+f/4crq6uKFiwIHLnzq1AdJaT8hxLS0vDF198gbZt2+Kvv/7C9u3bsWfPHuzZs0e2+E3h+aUuPL+M+/bbb3HlyhU0a9YMK1euROPGjVG4cGFZYrcFE5n/l5qaijFjxqBw4cIYOnSo9u9lypTRq/19/fp1+Pn5WbTuqKgoNG/eXKpQrRYXF4eDBw/ijz/+QNOmTdG0aVOcOnUKu3fvRnp6OtLT0/UurI8fP4a3t7f2vUajUSJsq7m7u+v9eoiJiVEwGut88MEHcHNzw+HDh/X+fu3aNTx8+BCVKlXC8+fPRX8lOdr3JfU59vr1azx9+hQdO3aEm5sbihcvjoYNG+LUqVNyhG8Szy+eX0qS4x5WokQJ/PDDD9i7dy8WLVqEhw8fIiAgQOrQbcZE5v9NnToVSUlJmDhxot4BGhISgk2bNuH+/ft49uwZ1qxZo5eUJCcnIykpCcDbZ+AZ/8+QcUI0bNjQLvthzP79+5E/f35s2rQJa9aswZo1a/Drr78iKSkJx48fh0ajwfLly5GcnIyjR4/iwoULqFu3rtJhW61cuXI4ePAgYmNjce/ePb2a+mqRJ08e9O3bFzNnzsTp06eRmpqK6OhojB8/Hq1bt0a1atXw4Ycf4vvvv0d8fDwSExO1TWMLFCiAx48fm2xGai9Sn2MFChRA0aJFsXXrVqSnp+Px48c4cOAA3n33Xfvu2P/j+cXzS0ly3MNu3bqFhIQEJCYmYu3atUhISHDIY9ZN6QAcwcOHDxEZGYlcuXIhKChI+/cFCxagXr16iI6ORu/evZGeno62bduidevW2nk6dOiAhw8fAnjb8gAATp48qZ0eFRWFwMBAuLu722lvsrZr1y60adNGr3gbeHugnz17Fjly5ECZMmUQEhICd3d3TJ482SEeh1lDo9EgJCQER48eRYsWLfDOO++gadOmuHjxotKhWSw0NBR58uTBlClT8OjRIxQsWBCtWrVC//79AQBTpkzBrFmz0KpVK2g0GrRr1w6VKlVCjRo1ULRoUTRp0gTFixfHmjVrFNsHuc6xmTNn4vvvv8fChQuRO3dufPzxx2jXrp0d9+x/eH7x/FKKXOfX4cOHERERgZSUFFSrVg1z5sxxuJIoANAIulWSKdt68OABOnbsiCNHjigdis0aN26M8PBwlCpVSulQiADw/CKSEx8tkVPJ+CXh4+OjcCREzofnFzkiPloipzF16lQcPXoUY8eORY4cOZQOh8ip8PwiR8VHS0RERKRafLREREREqsVEhoiIiFSLiQwRERGpFhMZIiIiUi0mMkRERKRaTGSIyGInT55E9erVUb16dTx48EDpcIgoG2M/MkSkp1WrVtouy7NSv359VKxYEQCQM2dOe4Rl0smTJ/Hpp58CALZv347ixYsrHBER2QMTGSLSU758ee2ozE+ePMGTJ08AvB0kMCNpCQwMRNu2bZUKkYhIix3iEVGWli5dimXLlgHQL+UQK/2YOHEifvvtN/j4+GDgwIFYvHgxYmNj0bp1a3z22Wf48ccfsX37duTJkwd9+vRBx44dtdt5+vQpFi1ahL///hsvX75E0aJF0apVK/Tp0wdubm9/b124cAGLFi3C9evXER8fjwIFCqB8+fIYPnw4duzYoY1TV8uWLTFx4kSsWrUKu3btwqNHjxAXF4e8efPiww8/xODBg1G6dGkAQGRkJCZNmgQAmDFjBlauXIk7d+6gWrVqmDRpEv78808sX74ciYmJCA4OxogRI7SxVa9eHQAwdOhQXL58GYcOHULu3LnRoUMHDBw40CEH2iNyFqwjQ0SSevbsGWbMmIEcOXIgLi4Oa9euRa9evbB9+3Z4eXnh0aNHmDVrFm7dugUAePnyJfr06YPIyEgkJCSgTJkyePToEZYsWYKpU6cCANLT0zF06FCcOHECbm5uKFOmDFJSUnDo0CE8evQIRYsWRZkyZbQxlCtXDhUrVkSJEiUAAKdOncLdu3fh7e2Nd955B69fv8b+/fsxaNAgJCUlZdqHCRMmIDk5GcnJyThy5AjCwsIwc+ZM5MqVC69evcLGjRuxbdu2TMstWrQIZ86cQZ48efDixQssX74c69evl+NjJqL/x0SGiCSVkpKCH374AZs3b0bRokUBAHfv3sXatWuxceNG5MqVC+np6Th16hQAYMOGDXj8+DG8vb2xdetWrF27FjNnzgQA/Pbbb7h79y5ev36NV69eAQDCw8Pxyy+/4I8//sD69evh5+eHtm3b4quvvtLG8N133yEiIgKhoaEAgCFDhmD//v349ddfsX79eixYsAAA8PjxY5w7dy7TPvTr1w8bN25Es2bNAAC3bt3ChAkTsHnzZnz44YcA/jeAoq73338fkZGR2L59O6pUqaKNl4jkwzoyRCSpjMc2AFCsWDE8fvwY7777rvaxVIECBfDo0SM8f/4cAHDp0iUAQExMDIKDg/XWJQgCLl68iObNm+ODDz7A+fPn0bFjR5QsWRLvvvsu6tWrp002jHn06BGmTZuGGzduID4+HrpP1J8+fZpp/gYNGgDQH+W5fv36AABfX1+cPXtWG7+uxo0bax83NW7cGGfOnEFMTAxevHiBAgUKmIyTiCzHRIaIJOXp6an9v6ura6a/ZdQXyUgmMv719PTUezyUIXfu3ADePraJiorCuXPncOvWLezduxe7d+/Gs2fP0Lt37yzjuXfvHkaMGIGUlBR4enrivffeQ2pqKq5fvw7g7WOrrPYhI34A8PLyEo2fiJTFRIaIFPX+++/jyJEjcHV1xbRp07QlN3Fxcdi/fz+CgoIgCALOnz+PVq1aaVtLTZ48Gdu3b8eZM2fQu3dvbcIDAAkJCdr/X7t2DSkpKQCAhQsX4oMPPsDvv/+OsWPHSr4ve/fu1VZi3rdvHwDA29ubpTFEMmIiQ0SK6ty5M7Zt24YnT56gQ4cOKFOmDOLi4vD48WOkpqaiZcuWSEtLw6BBg+Dp6YmiRYtCo9FoKwv7+/sDAEqUKAE3NzekpqZi0KBB8PHxQc+ePeHv7w9XV1ekpaVhyJAhKFasGGJiYmTZl6tXr6JVq1bQaDTaZuuffPKJLNsiordY2ZeIFFWgQAGEh4ejVatWyJcvH/755x8kJSWhSpUqGDZsGIC3j3g6dOiA4sWL48mTJ7h37x58fHzQq1cvDBgwAACQP39+jBgxAkWLFsXz589x8eJFxMTE4J133sH48ePh6+uL1NRU5M+fX9saSmqDBg1C9erVERsbi3z58qFfv37o2rWrLNsiorfYjwwRkY0y+pGZMGECWrVqpXA0RNkLS2SIiIhItZjIEBERkWrx0RIRERGpFktkiIiISLWYyBAREZFqMZEhIiIi1WIiQ0RERKrFRIaIiIhUi4kMERERqRYTGSIiIlItJjJERESkWv8HKEpSZWU/DxsAAAAASUVORK5CYII=",
+ "text/plain": [
+ "