Error: train_grafiti.py

[RichardShea]

Hi,

When I run the following example:

!python train_grafiti.py --epochs 200 --learn-rate 0.001 --batch-size 128 --attn-head 1 --latent-dim 128 --nlayers 4 --dataset physionet2012 --fold 0 -ct 36 -ft 12

I get the following error:

train_grafiti.py --epochs 200 --learn-rate 0.001 --batch-size 128 --attn-head 1 --latent-dim 128 --nlayers 4 --dataset physionet2012 --fold 0 -ct 36 -ft 12
Namespace(quiet=False, run_id=None, config=None, epochs=200, fold=0, batch_size=128, learn_rate=0.001, betas=(0.9, 0.999), weight_decay=0.001, hidden_size=32, kernel_init='skew-symmetric', note='', seed=None, nlayers=4, attn_head=1, latent_dim=128, dataset='physionet2012', forc_time=12, cond_time=36, nfolds=5) 2482694
Namespace(quiet=False, run_id=None, config=None, epochs=200, fold=0, batch_size=128, learn_rate=0.001, betas=(0.9, 0.999), weight_decay=0.001, hidden_size=32, kernel_init='skew-symmetric', note='', seed=None, nlayers=4, attn_head=1, latent_dim=128, dataset='physionet2012', forc_time=12, cond_time=36, nfolds=5)
2024-07-24 00:42:55.108815: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-07-24 00:42:55.108867: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-07-24 00:42:55.110423: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-07-24 00:42:56.736069: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/content/tsdm/viz/_config.py:54: UserWarning: matplotlib: no LaTeX rendering
  warnings.warn("matplotlib: no LaTeX rendering")
/root/.tsdm/rawdata/Physionet2012/set-a.tar.gz: 100% 6.33M/6.33M [00:00<00:00, 9.82MiB/s]
/root/.tsdm/rawdata/Physionet2012/set-b.tar.gz: 100% 6.34M/6.34M [00:00<00:00, 10.5MiB/s]
/root/.tsdm/rawdata/Physionet2012/set-c.tar.gz: 100% 6.29M/6.29M [00:00<00:00, 10.2MiB/s]
Loading patient data <TarInfo 'set-a/136047.txt' at 0x7d5897ffa980>: 100% 4001/4001 [05:24<00:00, 12.33it/s]
Loading patient data <TarInfo 'set-b/152811.txt' at 0x7d589ca6c280>: 100% 4001/4001 [05:24<00:00, 12.35it/s]
Loading patient data <TarInfo 'set-c/153918.txt' at 0x7d589ca24640>: 100% 4001/4001 [05:24<00:00, 12.32it/s]
/content/tsdm/datasets/base.py:278: UserWarning: File 'Physionet2012-set-A-sparse.tar' cannot be validated as no hash is stored in <class 'tsdm.datasets.physionet2012.Physionet2012'>.The filehash is '60dc2a5d4a98dec88b11e5967d59b9ff404b3fb90ed88d5a6261b5edeb5e2d3d'.
  warnings.warn(
/content/tsdm/datasets/base.py:278: UserWarning: File 'Physionet2012-set-B-sparse.tar' cannot be validated as no hash is stored in <class 'tsdm.datasets.physionet2012.Physionet2012'>.The filehash is '049682380ee56f31eed137a071192e1aacdc2abbb0aa2ed31abb410c0f6b7228'.
  warnings.warn(
/content/tsdm/datasets/base.py:278: UserWarning: File 'Physionet2012-set-C-sparse.tar' cannot be validated as no hash is stored in <class 'tsdm.datasets.physionet2012.Physionet2012'>.The filehash is '8416a80265bda0cb1a45cfbb1acc00fa1844e91c748b865896f95feb2341be48'.
  warnings.warn(
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexes/base.py", line 3653, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 147, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 176, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'Time'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/content/train_grafiti.py", line 90, in <module>
    TASK = Physionet2012(normalize_time=True, condition_time=ARGS.cond_time, forecast_horizon = ARGS.forc_time, num_folds=ARGS.nfolds)
  File "/content/tsdm/tasks/physionet2012.py", line 181, in __init__
    self.IDs = self.dataset.reset_index()["RecordID"].unique()
  File "/usr/lib/python3.10/functools.py", line 981, in __get__
    val = self.func(instance)
  File "/content/tsdm/tasks/physionet2012.py", line 199, in dataset
    self.encoder.fit(ts)
  File "/content/tsdm/utils/decorators/_decorators.py", line 625, in _wrapper
    result = func(*args, **kwargs)
File "/content/tsdm/encoders/_modular.py", line 432, in fit
    encoder.fit(index[group])
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/frame.py", line 3761, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexes/base.py", line 3655, in get_loc
    raise KeyError(key) from err
KeyError: 'Time'

Thank you,

Richard

[daweiTech]

Hi，I have the same problem，can you tell me how to solve it

[daweiTech]

@RichardShea

[charlieguo0610]

Same Problem + 1

[yalavarthivk]

Hi all, really sorry for the late reply. I will look into it and see where the issue is. It could be the version of pandas but let me double check this. Are you guys able to get USHCN run?

[daweiTech]

Hi all, really sorry for the late reply. I will look into it and see where the issue is. It could be the version of pandas but let me double check this. Are you guys able to get USHCN run?

Oh, I don't think it's the pandas library, it's more like some kind of data processing problem

[yalavarthivk]

Thanks, you are correct. There is an issue with the scaling of the time and it happens only when you run it for the first time when creating the datasets.
Try running it again and it should work.
I will correct the issue but it takes some time for me.

[RichardShea]

I run twice, but I get the following warnings, errors:

Loading patient data <TarInfo 'set-c/153918.txt' at 0x7feec6dcaec0>: 100% 4001/4001 [05:51<00:00, 11.37it/s] /content/tsdm/datasets/physionet2012.py:420: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation. time_series_df = pd.concat( /content/tsdm/datasets/base.py:278: UserWarning: File 'Physionet2012-set-A-sparse.tar' cannot be validated as no hash is stored in <class 'tsdm.datasets.physionet2012.Physionet2012'>.The filehash is '9082b90c11eb2cfda2f87179dc0149032b8e341dc53950d8264330a95cc903f2'. warnings.warn( /content/tsdm/datasets/base.py:278: UserWarning: File 'Physionet2012-set-B-sparse.tar' cannot be validated as no hash is stored in <class 'tsdm.datasets.physionet2012.Physionet2012'>.The filehash is '76cd2a2ad8f20d0c55d593acef9770b15f98694ba344b18916c2415691375d89'. warnings.warn(

`Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexes/base.py", line 3791, in get_loc
return self._engine.get_loc(casted_key)
File "index.pyx", line 152, in pandas._libs.index.IndexEngine.get_loc
File "index.pyx", line 181, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'Time'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/content/train_grafiti.py", line 90, in
TASK = Physionet2012(normalize_time=True, condition_time=ARGS.cond_time, forecast_horizon = ARGS.forc_time, num_folds=ARGS.nfolds)
File "/content/tsdm/tasks/physionet2012.py", line 181, in init
self.IDs = self.dataset.reset_index()["RecordID"].unique()
File "/usr/lib/python3.10/functools.py", line 981, in get
val = self.func(instance)
File "/content/tsdm/tasks/physionet2012.py", line 199, in dataset
self.encoder.fit(ts)
File "/content/tsdm/utils/decorators/_decorators.py", line 625, in _wrapper
result = func(*args, **kwargs)
File "/content/tsdm/encoders/_modular.py", line 432, in fit
encoder.fit(index[group])
File "/usr/local/lib/python3.10/dist-packages/pandas/core/frame.py", line 3893, in getitem
indexer = self.columns.get_loc(key)
File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexes/base.py", line 3798, in get_loc
raise KeyError(key) from err
KeyError: 'Time'
`