omegaconf.errors.UnsupportedInterpolationType: Unsupported interpolation type hydra

[zcx-language]

When using tensorboard logger and multi-gpus, got an error.

(yolov5) chengxin@lab3:~/OpenSource/LightningHydraTemplate$ python /home/chengxin/OpenSource/LightningHydraTemplate/train.py
/home/chengxin/OpenSource/LightningHydraTemplate/train.py:10: UserWarning:
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
@hydra.main(config_path="configs/", config_name="train.yaml")
/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/next/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
ret = run_job(
[2022-05-19 16:57:35,241][src.utils][INFO] - Disabling python warnings! <config.ignore_warnings=True>
[2022-05-19 16:57:35,241][src.utils][INFO] - Printing config tree with Rich! <config.print_config=True>

CONFIG
├── datamodule
│ └── target: src.datamodules.mnist_datamodule.MNISTDataModule
│ data_dir: /home/chengxin/OpenSource/LightningHydraTemplate/data/
│ batch_size: 64
│ train_val_test_split:
│ - 55000
│ - 5000
│ - 10000
│ num_workers: 0
│ pin_memory: false
│
├── model
│ └── target: src.models.mnist_module.MNISTLitModule
│ lr: 0.001
│ weight_decay: 0.0005
│ net:
│ target: src.models.components.simple_dense_net.SimpleDenseNet
│ input_size: 784
│ lin1_size: 256
│ lin2_size: 256
│ lin3_size: 256
│ output_size: 10
│
├── callbacks
│ └── model_checkpoint:
│ target: pytorch_lightning.callbacks.ModelCheckpoint
│ monitor: val/acc
│ mode: max
│ save_top_k: 1
│ save_last: true
│ verbose: false
│ dirpath: checkpoints/
│ filename: epoch_{epoch:03d}
│ auto_insert_metric_name: false
│ early_stopping:
│ target: pytorch_lightning.callbacks.EarlyStopping
│ monitor: val/acc
│ mode: max
│ patience: 100
│ min_delta: 0
│ model_summary:
│ target: pytorch_lightning.callbacks.RichModelSummary
│ max_depth: -1
│ rich_progress_bar:
│ target: pytorch_lightning.callbacks.RichProgressBar
│
├── logger
│ └── tensorboard:
│ target: pytorch_lightning.loggers.tensorboard.TensorBoardLogger
│ save_dir: tensorboard/
│ name: null
│ version: default
│ log_graph: false
│ default_hp_metric: true
│ prefix: ''
│
├── trainer
│ └── target: pytorch_lightning.Trainer
│ gpus: 2
│ min_epochs: 1
│ max_epochs: 10
│ resume_from_checkpoint: null
│
├── original_work_dir
│ └── /home/chengxin/OpenSource/LightningHydraTemplate
├── data_dir
│ └── /home/chengxin/OpenSource/LightningHydraTemplate/data/
├── print_config
│ └── True
├── ignore_warnings
│ └── True
├── train
│ └── True
├── test
│ └── True
├── seed
│ └── None
└── name
└── default
[2022-05-19 16:57:35,402][src.training_pipeline][INFO] - Instantiating datamodule <src.datamodules.mnist_datamodule.MNISTDataModule>
[2022-05-19 16:57:35,409][src.training_pipeline][INFO] - Instantiating model <src.models.mnist_module.MNISTLitModule>
[2022-05-19 16:57:35,433][src.training_pipeline][INFO] - Instantiating callback <pytorch_lightning.callbacks.ModelCheckpoint>
[2022-05-19 16:57:35,435][src.training_pipeline][INFO] - Instantiating callback <pytorch_lightning.callbacks.EarlyStopping>
[2022-05-19 16:57:35,436][src.training_pipeline][INFO] - Instantiating callback <pytorch_lightning.callbacks.RichModelSummary>
[2022-05-19 16:57:35,437][src.training_pipeline][INFO] - Instantiating callback <pytorch_lightning.callbacks.RichProgressBar>
[2022-05-19 16:57:35,438][src.training_pipeline][INFO] - Instantiating logger <pytorch_lightning.loggers.tensorboard.TensorBoardLogger>
[2022-05-19 16:57:35,440][src.training_pipeline][INFO] - Instantiating trainer <pytorch_lightning.Trainer>
[2022-05-19 16:57:35,470][pytorch_lightning.utilities.rank_zero][INFO] - Trainer already configured with model summary callbacks: [<class 'pytorch_lightning.callbacks.rich_model_summary.RichModelSummary'>]. Skipping setting a default ModelSummary callback.
[2022-05-19 16:57:35,470][pytorch_lightning.utilities.rank_zero][INFO] - GPU available: True, used: True
[2022-05-19 16:57:35,470][pytorch_lightning.utilities.rank_zero][INFO] - TPU available: False, using: 0 TPU cores
[2022-05-19 16:57:35,471][pytorch_lightning.utilities.rank_zero][INFO] - IPU available: False, using: 0 IPUs
[2022-05-19 16:57:35,471][pytorch_lightning.utilities.rank_zero][INFO] - HPU available: False, using: 0 HPUs
[2022-05-19 16:57:35,471][src.training_pipeline][INFO] - Logging hyperparameters!
[2022-05-19 16:57:35,479][src.training_pipeline][INFO] - Starting training!
/home/chengxin/OpenSource/LightningHydraTemplate/train.py:10: UserWarning:
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
@hydra.main(config_path="configs/", config_name="train.yaml")
/home/chengxin/OpenSource/LightningHydraTemplate/train.py:10: UserWarning:
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
@hydra.main(config_path="configs/", config_name="train.yaml")
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2
Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2

distributed_backend=nccl
All distributed processes registered. Starting with 2 processes

/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/pytorch_lightning/trainer/configuration_validator.py:307: LightningDeprecationWarning: The LightningModule.on_epoch_end hook was deprecated in v1.6 and will be removed in v1.8. Please use LightningModule.on_<train/validation/test>_epoch_end instead.
rank_zero_deprecation(
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
┏━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ ┃ Name ┃ Type ┃ Params ┃
┡━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ 0 │ net │ SimpleDenseNet │ 336 K │
│ 1 │ net.model │ Sequential │ 336 K │
│ 2 │ net.model.0 │ Linear │ 200 K │
│ 3 │ net.model.1 │ BatchNorm1d │ 512 │
│ 4 │ net.model.2 │ ReLU │ 0 │
│ 5 │ net.model.3 │ Linear │ 65.8 K │
│ 6 │ net.model.4 │ BatchNorm1d │ 512 │
│ 7 │ net.model.5 │ ReLU │ 0 │
│ 8 │ net.model.6 │ Linear │ 65.8 K │
│ 9 │ net.model.7 │ BatchNorm1d │ 512 │
│ 10 │ net.model.8 │ ReLU │ 0 │
│ 11 │ net.model.9 │ Linear │ 2.6 K │
│ 12 │ criterion │ CrossEntropyLoss │ 0 │
│ 13 │ train_acc │ Accuracy │ 0 │
│ 14 │ val_acc │ Accuracy │ 0 │
│ 15 │ test_acc │ Accuracy │ 0 │
│ 16 │ val_acc_best │ MaxMetric │ 0 │
└────┴──────────────┴──────────────────┴────────┘
Trainable params: 336 K
Non-trainable params: 0
Total params: 336 K
Total estimated model params size (MB): 1
Error executing job with overrides: []
Traceback (most recent call last):
File "/home/chengxin/OpenSource/LightningHydraTemplate/train.py", line 22, in main
return train(config)
File "/home/chengxin/OpenSource/LightningHydraTemplate/src/training_pipeline.py", line 86, in train
trainer.fit(model=model, datamodule=datamodule)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 768, in fit
self._call_and_handle_interrupt(
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 719, in _call_and_handle_interrupt
return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/spawn.py", line 78, in launch
mp.spawn(
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
while not context.join():
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 150, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
fn(i, *args)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/spawn.py", line 101, in _wrapping_function
results = function(*args, **kwargs)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in _fit_impl
results = self._run(model, ckpt_path=self.ckpt_path)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1222, in _run
self._log_hyperparams()
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1292, in _log_hyperparams
logger.save()
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/pytorch_lightning/utilities/rank_zero.py", line 32, in wrapped_fn
return fn(*args, **kwargs)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/pytorch_lightning/loggers/tensorboard.py", line 266, in save
save_hparams_to_yaml(hparams_file, self.hparams)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/pytorch_lightning/core/saving.py", line 386, in save_hparams_to_yaml
hparams = apply_to_collection(hparams, DictConfig, OmegaConf.to_container, resolve=True)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py", line 107, in apply_to_collection
v = apply_to_collection(
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py", line 99, in apply_to_collection
return function(data, *args, **kwargs)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/omegaconf.py", line 558, in to_container
return BaseContainer._to_content(
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 292, in _to_content
value = get_node_value(key)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 244, in get_node_value
conf._format_and_raise(key=key, value=None, cause=e)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 231, in _format_and_raise
format_and_raise(
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/_utils.py", line 873, in format_and_raise
_raise(ex, cause)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/_utils.py", line 771, in _raise
raise ex.with_traceback(sys.exc_info()[2]) # set env var OC_CAUSE=1 for full trace
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 242, in get_node_value
node = node._dereference_node()
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 246, in _dereference_node
node = self._dereference_node_impl(throw_on_resolution_failure=True)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 277, in _dereference_node_impl
return parent._resolve_interpolation_from_parse_tree(
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 584, in _resolve_interpolation_from_parse_tree
resolved = self.resolve_parse_tree(
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 764, in resolve_parse_tree
return visitor.visit(parse_tree)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 206, in accept
return visitor.visitConfigValue(self)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar_visitor.py", line 101, in visitConfigValue
return self.visit(ctx.getChild(0))
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 342, in accept
return visitor.visitText(self)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar_visitor.py", line 298, in visitText
return self.visitInterpolation(c)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar_visitor.py", line 125, in visitInterpolation
return self.visit(ctx.getChild(0))
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 921, in accept
return visitor.visitInterpolationNode(self)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar_visitor.py", line 158, in visitInterpolationNode
return self.node_interpolation_callback(inter_key, self.memo)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 745, in node_interpolation_callback
return self._resolve_node_interpolation(inter_key=inter_key, memo=memo)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 664, in _resolve_node_interpolation
parent, last_key, value = root_node._select_impl(
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 531, in _select_impl
value = root._maybe_resolve_interpolation(
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 719, in _maybe_resolve_interpolation
return self._resolve_interpolation_from_parse_tree(
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 584, in _resolve_interpolation_from_parse_tree
resolved = self.resolve_parse_tree(
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 764, in resolve_parse_tree
return visitor.visit(parse_tree)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 206, in accept
return visitor.visitConfigValue(self)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar_visitor.py", line 101, in visitConfigValue
return self.visit(ctx.getChild(0))
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 342, in accept
return visitor.visitText(self)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar_visitor.py", line 301, in visitText
return self._unescape(list(ctx.getChildren()))
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar_visitor.py", line 389, in _unescape
text = str(self.visitInterpolation(node))
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar_visitor.py", line 125, in visitInterpolation
return self.visit(ctx.getChild(0))
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 921, in accept
return visitor.visitInterpolationNode(self)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar_visitor.py", line 158, in visitInterpolationNode
return self.node_interpolation_callback(inter_key, self.memo)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 745, in node_interpolation_callback
return self._resolve_node_interpolation(inter_key=inter_key, memo=memo)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 664, in _resolve_node_interpolation
parent, last_key, value = root_node._select_impl(
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 531, in _select_impl
value = root._maybe_resolve_interpolation(
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 719, in _maybe_resolve_interpolation
return self._resolve_interpolation_from_parse_tree(
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 584, in _resolve_interpolation_from_parse_tree
resolved = self.resolve_parse_tree(
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 764, in resolve_parse_tree
return visitor.visit(parse_tree)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 206, in accept
return visitor.visitConfigValue(self)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar_visitor.py", line 101, in visitConfigValue
return self.visit(ctx.getChild(0))
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 342, in accept
return visitor.visitText(self)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar_visitor.py", line 298, in visitText
return self.visitInterpolation(c)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar_visitor.py", line 125, in visitInterpolation
return self.visit(ctx.getChild(0))
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 1041, in accept
return visitor.visitInterpolationResolver(self)
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/grammar_visitor.py", line 179, in visitInterpolationResolver
return self.resolver_interpolation_callback(
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 750, in resolver_interpolation_callback
return self._evaluate_custom_resolver(
File "/media/data1/miniconda3/envs/yolov5/lib/python3.8/site-packages/omegaconf/base.py", line 702, in _evaluate_custom_resolver
raise UnsupportedInterpolationType(
omegaconf.errors.UnsupportedInterpolationType: Unsupported interpolation type hydra
full_key: datamodule.data_dir
object_type=dict

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

[zhaisilong]

I encountered the same question, and it was solved by downgrading hydra. Please try: pip install hydra-optuna-sweeper==1.1.0 hydra-core==1.1.0