Add a new configloader with OmegaConf as replacement for anyconf

[merelcht]

Introduction

As part of the work on improving configuration in Kedro we assessed alternatives for anyconfig. anyconfig was initially chosen because it supports reading different configuration formats. In practice, most users use yml files and so we want to use an alternative library that offers better functionality for yml configuration.

In #1657 it was decided to use OmegaConf as this replacement. More info about that conclusion can be found here: #1657 (comment)

We also did research to verify what config file formats users use in #1703

Implementation

In order to introduce this change in a non-breaking way we have to do the following:

1. Add a new configloader
2. Add OmegaConf as replacement for anyconf in this new configloader.
3. Turn off all built-in resolvers in OmegaConf in ConfigLoader. In the first iteration we want to keep these advanced features turned off https://omegaconf.readthedocs.io/en/latest/custom_resolvers.html#clear-resolver. It's easier to introduce them step by step, but removing them would be a breaking change. In Allow users to pass credentials through environment variables #1909 we will introduce environment variables for credentials.

[szczeles]

@MerelTheisenQB Do you know if OmegaConf implementation will keep the current configs merging behaviour? The challenge I've seen in the implementations is that top-level keys are overridden. For example having a conf/base/mlflow.yml like:

tracking:
  disable_tracking:
    pipelines:
    - on_exit_notification

  experiment:
    name: name-of-local-experiment

  params:
    long_params_strategy: tag

if I wanted to just change the experiment name, then this config located in conf/prod/mlflow.yml:

tracking:
  experiment:
    name: name-of-prod-experiment

overrides long_params_strategy and disable_tracking to the default values (in this case: fail and []) - inheritance doesn't work here. Basically, to inherit from base, I need to duplicate all the custom values.

In contrast, Helm package manager merges the configuration to the lowest level of the YAML tree, allowing to adjustment of only the necessary values. It would be great to be able to achieve this in Kedro too, but this change would be incompatible with the previous behavior.

[merelcht]

@MerelTheisenQB Do you know if OmegaConf implementation will keep the current configs merging behaviour? The challenge I've seen in the implementations is that top-level keys are overridden.

OmegaConf has a different merge strategy and will only replace the specified fields. So in your example if you merge the two files you'd end up with:

tracking:
  disable_tracking:
    pipelines:
    - on_exit_notification
  experiment:
    name: name-of-prod-experiment
  params:
    long_params_strategy: tag

See the OmegaConf docs for an example snippet you can run: https://omegaconf.readthedocs.io/en/2.2_branch/usage.html#merging-configurations

[szczeles]

@MerelTheisenQB Fantastic! Thanks a lot for the confirmation