Irregular time series support

[kashif]

Issue #, if available:

Description of changes:

Adding support to irregular time series where the datatimes are given by an INDEX key in the dataset

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Please tag this pr with at least one of these labels to make our release process faster: BREAKING, new feature, bug fix, other change, dev setup

[jaheba]

To understand it correctly, "irregular" means that the values are not spaced apart by the same time offset?

[kashif]

right! Which then implies we need to specify the time index for each point of the target... I believe i have training working... stuck at the inference part... I would appreciate some help! Let me push the latest changes and example to test...

[kashif]

@jaheba I am using the following snippet to test:

import random
import pandas as pd

from gluonts.dataset.common import Dataset, ListDataset
from gluonts.torch.model.deepar import DeepAREstimator
from gluonts.evaluation import make_evaluation_predictions, Evaluator

freq_str="1T"
prediction_length = 10

index = sorted(random.sample(list(pd.date_range("2021-01-01 02:12:06",
        periods=400, 
        freq=freq_str)), 50+prediction_length))

train_ds = ListDataset(
    [
        {
        "start": index[0], 
        "target": [float(i) for i in range(50)],
        "item_id": "0",
        "index": index[:50],
        },
    ],
    freq=freq_str,
) 

test_ds = ListDataset(
    data_iter=[
        {
        "start": index[0], 
        "target": [float(i) for i in range(50+prediction_length)],
        "item_id": "0",
        "index": index[:50+prediction_length],
        }
    ],
    freq=freq_str,
) 

estimator = DeepAREstimator(
    freq=freq_str,
    prediction_length=prediction_length,
    scaling=False,
    trainer_kwargs={"max_epochs": 1},
)

predictor = estimator.train(train_ds)

forecast_it, ts_it = make_evaluation_predictions(
    dataset=test_ds,
    predictor=predictor
)

forecasts = list(forecast_it)
tss = list(ts_it)

evaluator = Evaluator()
agg_metrics, ts_metrics = evaluator(iter(tss), iter(forecasts), num_series=len(test_ds))

[jaheba]

Can we add the snippet as a test?

[kashif]

sure... let me do that...

[jaheba]

Something I'm wondering: If we merge this PR we break the assumption that time-series are regular. Is that an assumption we rely on in other places? /cc @lostella

[kashif]

as long as there is no INDEX field the assumption of regular time interval is assumed and currently everything proceeds as before... if there is an INDEX then this assumption is broken and as far as I can tell it is used in:

• time features
• lags
• age feature
• padding in the Splitters? <- not sure about this yet...
• inference <- currently not working here

Apart from that I do not believe there is anything else relying on this assumption... (as far as I can tell)

[kashif]

so the one big missing piece is in the InstanceSplitter which currently is setting the wrong forecast_start_field field

[kashif]

@jaheba and @lostella I would be ready for a review. If you could check closely the padding in the splitters I would appreciate it! Thank you!

[kashif]

@rsnirwan would you have some time to quickly see how I can also use the new pandas datasets with irregular time indices in this PR?

[AjinkyaBankar]

@kashif @jaheba @lostella Is this resolved? Can we use irregular time series forecasting using DeepAR? This is highly required for some practical cases.

[kashif]

hmm @AjinkyaBankar kind-of, things were refactored and this branch kind of got left behind... i would love to get it up to speed but it might require a new PR

[AjinkyaBankar]

@kashif I am excited to use this as soon as possible. Let me know if I can help with this.