BUG: RecursionError: maximum recursion depth exceeded while calling a Python object

[Cameronwood611]

Description

Getting recursion depth exceeded with what I believe is coming from gluon calling get_batch (see stack trace "[Previous line repeated 964 more times]").

To Reproduce

This is an issue with the way the forcast_generator unpacks the batch. Only happens after calling model.predict several times and iterating through the returned generator.

This example isn't totally complete because you have to provide your own model, sorry.

import random
import numpy as np
import pandas as pd
from gluonts.torch import PyTorchPredictor
from gluonts.dataset.common import ListDataset, DataEntry
from gluonts.torch import TemporalFusionTransformerEstimator


estimator = TemporalFusionTransformerEstimator(
freq='S',
context_length=10,
prediction_length=5,
trainer_kwargs={'max_epochs': 5}
)
train = ListDataset([{'target': np.array([random.uniform(-1, 1) for _ in range(50)]), 'start': pd.Period('01-01-2023', freq='S')} for _ in range(20)], freq='S')

pred = ListDataset([{'target': np.array([random.uniform(-1, 1) for _ in range(50)]), 'start': pd.Period('01-01-2023', freq='S')} for _ in range(20)], freq='S')
model = estimator.train(train, pred)

x, y, z = np.array([x for x in range(10)]), np.array([y for y in range (10)]), np.array([z for z in range(10)]) #for example
start = pd.Period('01-01-2023', freq='S')
_input = ListDataset([{'target': x, 'start': start},
                          {'target': y, 'start': start},
                          {'target': z, 'start': start}], freq='S')
for i in range(5000):
    pred = model.predict(_input)
    output = [sig.mean for sig in pred]  #this calls on a generator which causes the recursion error 

Error message or code output

Exception in thread Controller:
Traceback (most recent call last):
  File "threading.py", line 980, in _bootstrap_inner
  File "project/event_loop.py", line 89, in run
  File "_project/_controller.py", line 54, in recv_msg
  File "project/controller.py", line 180, in recv_msg
  File "project/controller.py", line 320, in __predict_future_path
  File "project/predictor/periodic/predictor.py", line 38, in predict
    x, y, z = self.postprocess()
  File "project/predictor/periodic/predictor.py", line 87, in postprocess
    self.pred = [sig.mean_ts.to_numpy() for sig in self.pred] #type: ignore
  File "project/predictor/periodic/predictor.py", line 87, in <listcomp>
    self.pred = [sig.mean_ts.to_numpy() for sig in self.pred] #type: ignore
  File "gluonts/torch/model/predictor.py", line 89, in predict
  File "gluonts/model/forecast_generator.py", line 154, in __call__
  File "gluonts/transform/_base.py", line 111, in __iter__
  File "gluonts/transform/_base.py", line 132, in __call__
  File "gluonts/dataset/loader.py", line 50, in __call__
  File "gluonts/itertools.py", line 125, in get_batch
  File "gluonts/transform/_base.py", line 132, in __call__
  File "gluonts/transform/_base.py", line 132, in __call__
  File "gluonts/transform/_base.py", line 132, in __call__
  [Previous line repeated 964 more times]
  File "gluonts/transform/_base.py", line 186, in __call__
  File "gluonts/transform/_base.py", line 132, in __call__
  File "gluonts/transform/_base.py", line 132, in __call__
  File "gluonts/transform/_base.py", line 132, in __call__
  File "gluonts/transform/_base.py", line 136, in __call__
  File "gluonts/transform/_base.py", line 134, in __call__
  File "gluonts/transform/feature.py", line 367, in map_transform
  File "gluonts/transform/feature.py", line 367, in <listcomp>
  File "gluonts/time_feature/_base.py", line 30, in second_of_minute
  File "pandas/core/indexes/extension.py", line 77, in fget
  File "pandas/core/indexes/base.py", line 551, in __new__
  File "pandas/core/construction.py", line 527, in sanitize_array
  File "pandas/core/construction.py", line 441, in extract_array
  File "pandas/core/dtypes/generic.py", line 44, in _instancecheck
  File "pandas/core/dtypes/generic.py", line 38, in _check
RecursionError: maximum recursion depth exceeded while calling a Python object

Environment

• Operating system: WSL 2 - Ubuntu 20.04
• Python version: 3.9.16
• GluonTS version: 0.13.2
• MXNet version: N/A
• pandas version: 2.0.3

(Add as much information about your environment as possible, e.g. dependencies versions.)
I'm using the cpu based environment; no CUDA.

[lostella]

Can reproduce. Further reduced example to:

import numpy as np
import pandas as pd
from gluonts.torch import TemporalFusionTransformerEstimator

freq = "H"

estimator = TemporalFusionTransformerEstimator(
    freq=freq,
    context_length=10,
    prediction_length=5,
    num_batches_per_epoch=2,
    trainer_kwargs={"max_epochs": 1},
)

train = [
    {
        "target": np.random.random(size=(50,)),
        "start": pd.Period("01-01-2023", freq=freq),
    }
    for _ in range(20)
]

model = estimator.train(train)

pred = [
    {"target": np.arange(10), "start": pd.Period("01-01-2023", freq=freq)},
    {"target": np.arange(10), "start": pd.Period("01-01-2023", freq=freq)},
    {"target": np.arange(10), "start": pd.Period("01-01-2023", freq=freq)},
]

for i in range(5000):
    forecasts = model.predict(pred)
    list(forecasts)

[lostella]

Fixed in #2951, will be released shortly