Nested Interpolation / Expansions

[m-gris]

It would be really nice to have the possibility to nest string interpolations / variables expansions, ex. interpolating in dvc.yaml a field of params.yaml that is itself an interpolation of another field...

For example:

In params.yaml

paths: 
  data: 
    root: path/to/data/dir
    raw: ${paths.data.root}/raw/
  scripts: 
    root: path/to/stages/dir
    load: ${paths.scripts.root}

and then dvc.yaml

stages:
  load:
    cmd: python ${paths.scripts.load}
    outs: 
      - ${paths.raw}/filename.extension

This would really help in avoiding cumbersome & "error prone" hardcoding.

Many thanks in advance
Regards
M

[d-walkama]

+1

[dberenbaum]

@m-gris You are using the hydra integration, right? Does it not solve your use case, or you just want to simplify by not needing hydra?

[m-gris]

@dberenbaum
Yes I do use the hydra integration indeed, but, afaik, it does not allow to do what I'm trying to do, i.e reduce hardcoding to the bare minimum by composing expanded variables as much as possible.

Here is a silly toy example that should suffice in illustrating what I have in mind:

Let's say that in config.yaml I have the following:

hydra_test:
    dir: 'raw'
    file: 'items.csv'
    path: ${hydra-test.dir}/${hydra_test.file}

The path key will indeed be properly parsed & evaluated if using OmegaConf in a python script.

But if, in dvc.yaml, I try to do the following:

stages:
    hydra_test:
        cmd: echo ${hydra_test.path}

It results in the following error:

/bin/bash: line 1: ${hydra_test.repo}/${hydra_test.file}: bad substitution

It would actually be really nice if dvc could interpret / evaluate those. :)

[daavoo]

The path key will indeed be properly parsed & evaluated if using OmegaConf in a python script.

This should be also working and the params.yaml should contain the interpolated value.
I will take a look to find out why it is not working

[m-gris]

It now works like a charm with Hydra!
Thx.