Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add comparison reducing series max length #1

Open
wants to merge 4 commits into
base: chronos-full-eval
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion experiments/amazon-chronos/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
# Extended comparison of Chronos against the statistical ensemble

We present an extension to the [original comparison by Nixtla](https://github.com/Nixtla/nixtla/tree/main/experiments/amazon-chronos) of Chronos [1] against the SCUM ensemble [2]. In this analysis on over 200K unique time series across 28 datasets from Benchmark II in the Chronos paper [1], we show that **zero-shot** Chronos models perform comparably to this strong ensemble of 4 statistical models while being significantly faster on average. We follow the original study as closely as possible, including loading task definitions from GluonTS and computing metrics using utilsforecast.

## Background
A few weeks ago, we presented a [fully reproducible experiment](https://github.com/Nixtla/nixtla/tree/main/experiments/amazon-chronos) showing that Amazon Chronos was 10% less accurate and 500% slower than training classical statistical models. The Amazon team kindly answered by extending our benchmarking efforts, confirming our results for the selected datasets, and showing a differentiated performance for new datasets.

Here we present an extension to the [original comparison by Nixtla](https://github.com/Nixtla/nixtla/tree/main/experiments/amazon-chronos) of Chronos [1] against the SCUM ensemble [2]. In this analysis on over 200K unique time series across 28 datasets from Benchmark II in the Chronos paper [1], we show that **zero-shot** Chronos models perform comparably to this strong ensemble of 4 statistical models while being significantly faster on average. We follow the original study as closely as possible, including loading task definitions from GluonTS and computing metrics using utilsforecast.

## Empirical Evaluation

Expand Down
Loading