Hardcoded paths in MDIMs make future updates harder

[pabloarosado]

Context

Currently, mdim steps require a config yaml file, which includes full paths of indicators in tables of grapher steps.

Potential problems

• You could reference data in the YAML that wasn't in the DAG dependencies
  • This could lead to intermittent ETL failures, or the ETL not rebuilding the MDIM even though its data was stale
• The hard-coding of versions in the YAML (e.g. https://github.com/owid/etl/blob/master/etl/steps/export/multidim/energy/latest/energy.yml) means that updating MDIMs is more annoying

Possible solution

Treat the MDIM yaml as a template, where we fill in some variables at the time we ship it.

Ideally, we would avoid hardcoding paths (in steps and yaml files), and all dependencies would be specified in the DAG.

After a discussion with @lucasrodes we thought of a possible solution. The config yaml file (for example of the covid mdim step) could have a special placeholder for a dataset path, e.g. {ds:short_name} (e.g. {ds:covid_cases}), specifying the short name of a dataset listed as a dependency of the mdim step in the DAG.
Then, the function paths.load_mdim_config would read the config yaml file, and replace those placeholders by the full URIs of the corresponding dataset.

Possible rabbit holes or related issues

• But it is possible that multiple dependencies of an mdim step have the same short name. And we may also want to create an mdim that compares different versions of the same dataset. For such cases, we could define custom placeholders, e.g. {ds:custom_short_name}, and then pass a dictionary to paths.load_mdim_config mapping those custom short names to the corresponding dataset URI.
  • As a reference, see a parallel implementation of safe short names in a related project: https://github.com/larsyencken/shelf/blob/main/src/shelf/tables.py#L200
• We also noticed that it is inconvenient that Table does not have an URI, and we rely on Table.metadata.dataset.uri. Maybe tables should also have a URI attribute.
• We may need an additional function of paths to get the URI of a table in a dataset. Currently, the way we'd do that is by, e.g. `paths.load_dataset("dataset_path...") + "/table_name...".

Impact

We're not encountering this problem so much yet, but it's more that we are currently setting precedents on how a large amount of work will be done, so we're interested in saving ourselves future work by getting this right.

[larsyencken]

We discussed this in triage today.

@Marigold thought this could be good to tackle at a time when we are fixing a related issue, to make MDIMs on the ETL side...

• build from data://grapher/... steps (filesystem -> DB)
• rather than grapher://grapher/... steps (filesystem -> DB -> filesystem -> DB)

as it does now.

[pabloarosado]

To clarify, this should not include refactoring the currently existing mdims (e.g. covid). That should happen as a separate issue.

[Marigold]

This was fixed by #3945

It's now sufficient to specify table#indicator instead of the full catalog path. So grapher/energy/2024-06-20/energy_mix/energy_mix#coal__twh becomes energy_mix#coal__twh. Those paths get dynamically expanded in the last step in the upsert_multidim_data_page(...) function, you just have to pass extra argument dependencies=paths.dependencies so that the function knows dataset names.

[pabloarosado]

That sounds like a great solution, thanks a lot @Marigold!