Clarification on distr_output hyper-parameter and what to expect from DeepAR?

[rurjais]

I am trying to understand DeepAR in a very precise way, so I have a question regarding what does actually mean the distr_output attribute in DeepAR estimator:

• Is it the distribution we EXPECT our target time-series to follow, or;
• Is it the residual distribution y-\hat{y} that we want our residuals to follow?

I am guessing it is the first one, but if that is the case, then I have a doubt I would like someone to help me out with. I'll try to be clear :)

I also have a question about what is the avg_epoch_loss that shows during training. What loss is this...? I give the code below for further help :)

I created a simple linear array, onto which I add some Random Poisson noise:

import numpy as np
import pandas as pd
s = np.random.poisson(5, 469,) #the random noise
import copy
s_2 = copy.deepcopy(s)
for i in range(1,len(s_2)):
   s_2['S'][i]=s['S'][i]+i #create s_2 and replace its values with an increasing i+the respect value of the poisson noise`

Then, as I wanted to perform univariate analysis (although I understand that this is not the most proper task for DeepAR to be trained on) I create the train and test dataset:

from gluonts.dataset.common import ListDataset
start = pd.Timestamp(s_2[s_2.columns[0]][0])

# train dataset: cut the last window of length "prediction_length", add "target" and "start" fields
train_df = {'target': s_2[:int(0.8*(len(s_2)))][s_2.columns[1]], 'start': start}#80% dataset for training
train_df = ListDataset([train_df], freq='M')

# test dataset: use the whole dataset, add "target" and "start" fields
test_df = {'target': s_2[s_2.columns[1]], 'start': start}
test_df = ListDataset([test_df], freq='M')

The I created my estimator:

from gluonts.mx.model.deepar import DeepAREstimator
from gluonts.mx.trainer import Trainer
from gluonts.mx.distribution.poisson import PoissonOutput
import mxnet as mx
import numpy as np

np.random.seed(7)
mx.random.seed(7)

estimator_2 = DeepAREstimator(
    prediction_length=len(test_df[0]['target'])-len(train_df[0]['target']),
    context_length=94, #The number of time points that the model gets to see before making the prediction. The value for this parameter should be about the same as the prediction_length (https://docs.aws.amazon.com/sagemaker/latest/dg/deepar_hyperparameters.html)
    use_feat_dynamic_real=False,
    freq='M',
    distr_output=PoissonOutput(), #THE DOUBT IS  ALSO ABOUT THIS PARAMETER
    num_parallel_samples=100,
    trainer=Trainer(
        epochs=20, #I defined a small number of epochs so training was faster
        learning_rate=1e-3,
        num_batches_per_epoch=32)
)

After that I built my predictions:

predictor_2 = estimator_2.train(train_df) #DURING TRAINING, WHAT IS THIS AVG_EPOCH_LOSS? Does it depend on the distr_output function we use? So in this case would it be a Poisson_loss?
forecast_it, ts_it = make_evaluation_predictions(
    dataset=test_df, # test dataset
    predictor=predictor_2, # predictor
    num_samples=100,
) 

Then worked around the results to handle them better:

forecasts = list(forecast_it)
tss = list(ts_it)
ts_entry = tss[0]
forecast_entry = forecasts[0]
median_forecast=forecast_entry.median #I just wanted to evaluate point forecasts at this point, but I am not even sure if it is more correct to use forecast_entry.mean or forecast_entry.median... Can anyone help?

A plot for predictions is annexed (output.png file).

At this point, I checked the predictions were not good, along with residuals:

residuals = test_set.values-forecast_entry.median.reshape(94,1) #the issue with reshape is for plotting

As one can see, residuals are something else but normal (see plot annexed) (output_2.png file)

I performed a KS test on residuals (although I know it may not be proper for discrete data), but it just confirmed that residuals are not normal...

So now my questions are:

• Is distr_output actually modeling the target or the noise model? I assume it is the target, but why does my model behave so poorly if that is the case? Is it because the linear component is the one that dictates the most the behavior of s_2 series and so PoissonOutput() will not catch any predicted behavior?
• Is it just a matter of hard tune my hyperparameters to get a better model?
• Is it structurally an issue with DeepAR to make univariate predictions (and as such I should expect that using s as a covariate could help in getting good predictions of s_2?) and is it only competitive when we use hundreds of time-series as covariates?

Thank you all
Ricardo

[Serendipity31]

Hi @rurjais - I'm curious... did you learn the answer to your question(s)?

[satyrmipt]

@Serendipity31 , @rurjais
Please look at this example https://ts.gluon.ai/stable/tutorials/advanced_topics/howto_pytorch_lightning.html
May be it helps you to get what happens here. I'm not sure i got it even after read through the example.

[c1adrien]

@rurjais
The distr_output attribute is essentially a subclass of DistributionOutput, which constructs a distribution given the output of a network. To create a neural network that maps to the parameters of your distribution, you need to use the method get_args_proj, like this:

param_proj = distr_output.get_args_proj(hidden_size)

Thus, you can think of the distr_output attribute as a component that allows the neural network to project the parameters of the distribution from the outputs of the RNN.