n_val_weeks in the paper and code do not match

[mcvageesh]

Hi,

Will you please clarify the value for n_val_weeks used to replicate your results? Is it 52, or 42? In the paper, it is mentioned as 42 in several places, but in the example on the main page on github (to replicate results of NP dataset), it is 52. Has the value 42 been used in some datasets, and 52 in some, maybe by mistake?

Thank you.

[kdgutier]

Hi @mcvageesh,

Thanks for taking such a detailed look into the NBEATSx methodology, hope this can help:

We make a distinction between the early stopping set (42 weeks) and the validation set (52 weeks). As you can see from Figure 5 in the NBEATSx paper.

In the paper an experiment is divided in two stages:

1. Hyperparameter selection: Where we use the validation set to obtain the signal that guides hyperparameter selection.
2. Retraining: Where we shift the training data to include validation. And use the early stopping set to stop the optimization of the network as a regularization technique. The set can be random like in Figure 5 or be the last 42 weeks before test, as we mention in Section 4.3.5.

Finally, we are not maintaining this repository, you might consider switching to the NeuralForecast library.

[mcvageesh]

Hi @kdgutier

Thank you for the prompt response. I will also switch to the Neural Forecast library to follow your current work.

I guessed earlier that you may have meant 52 weeks for the validation set to be used for the hyperparameter selection, and 42 weeks for the early stopping set to be used for re training. But I noticed in the code that both the hyperparameter and retraining parts (the run_val_nbeatsx and the run_test_nbeatsx) use the same value of n_val_weeks, i.e., the args.n_val_weeks that is set by the user at the start of the code - the hyperopt_nbeatsx file sets n_val_weeks to args.n_val_weeks, and the utils_experiment file will use this value in all the functions part of this file.

If one sets args.n_val_weeks = 52 before running the code, then both the hyperparameter selection and re training parts use 52 weeks. Hence I had the confusion. Do let me know if I have misunderstood anything.

Thank you!

[kdgutier]

Thanks for catching it and letting us know @mcvageesh.