Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add OmegaConfLoader #2085

Merged
merged 27 commits into from
Dec 20, 2022
Merged

Add OmegaConfLoader #2085

merged 27 commits into from
Dec 20, 2022

Conversation

merelcht
Copy link
Member

@merelcht merelcht commented Nov 30, 2022

Description

Added a new config loader class based on AbstractConfigLoader that uses OmegaConf to load and merge configuration.

Background 🔍

Development notes 🏗

Some differences between the OmegaConfLoader and the other config loader classes:

  • It's not recommended to use YAML anchors with OmegaConf, because they offer variable interpolation. Hence I didn't copy over the test test_ignore_hidden_keys
  • YAML parsing is done a bit differently. I added ScannerError with ParserError to catch this.

Other notes:

  • While writing a test for checking whether a file has the .yml or .yaml extension, I found out that OmegaConf can also handle .json files. So I've added .json as an acceptable extension for this loader as well.
  • After refactoring my initial implementation, I made sure not to use any of the helper methods from config/common.py. The methods in there are deeply nested and hard to follow. Usage of these methods makes it pretty much impossible for users to customise any configloader implementations.

Review notes ✍️

  • I added changes incrementally, so you can review this PR by checking out each individual commit.
  • All commits starting from 1c12a87 are a refactor to not use any methods from config/common.py

To Discuss 🗣

Which built-in resolvers should be turned off?

We decided to initially turn off all build-in resolvers. It's easier to introduce them step by step, but removing them would be a breaking change. In #1909 we will introduce environment variables for credentials. You can find a description of all built-in resolvers here: https://omegaconf.readthedocs.io/en/2.2_branch/custom_resolvers.html#built-in-resolvers
I feel like maybe it's okay to keep some resolvers turned on. For example oc.dict seems useful and not something we wouldn't want to add. I would like to hear what others think.

Is OmegaConfLoader the best name?

I first wanted to call this config loader YAMLConfigLoader but then I found out that OmegaConf can also load .json files. Any better name suggestions?

Will OmegaConfLoader replace ConfigLoader?

Follow up on the above question. I'm thinking that in 0.19.0 we should remove the ConfigLoader and TemplatedConfigLoader and rename this new OmegaConfLoader to ConfigLoader. If we do that, maybe the name of this new class doesn't matter so much for the 0.18 series.

Checklist ✅

  • Read the contributing guidelines
  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added a description of this change in the RELEASE.md file
  • Added tests to cover my changes

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Copy link
Contributor

@noklam noklam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leave 1st round of comments - haven't fully understood the code but since this is a relatively large PR, it's nice to get some early comments.

kedro/config/__init__.py Show resolved Hide resolved
kedro/config/common.py Outdated Show resolved Hide resolved
kedro/config/common.py Outdated Show resolved Hide resolved
kedro/config/omegaconf_config.py Show resolved Hide resolved
kedro/config/omegaconf_config.py Outdated Show resolved Hide resolved
kedro/config/omegaconf_config.py Outdated Show resolved Hide resolved
tests/config/test_omegaconf_config.py Outdated Show resolved Hide resolved
@merelcht merelcht marked this pull request as ready for review December 5, 2022 10:13
@merelcht merelcht requested a review from idanov as a code owner December 5, 2022 10:13
@merelcht merelcht self-assigned this Dec 5, 2022
@merelcht merelcht linked an issue Dec 5, 2022 that may be closed by this pull request
kedro/config/common.py Outdated Show resolved Hide resolved
kedro/config/omegaconf_config.py Outdated Show resolved Hide resolved
kedro/config/omegaconf_config.py Outdated Show resolved Hide resolved
kedro/config/omegaconf_config.py Outdated Show resolved Hide resolved
kedro/config/omegaconf_config.py Outdated Show resolved Hide resolved
merelcht and others added 11 commits December 8, 2022 11:13
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
…klabs/kedro into feature/add-omegaconf-loader
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
@merelcht merelcht requested review from idanov, noklam and SajidAlamQB and removed request for yetudada December 12, 2022 17:34
Copy link
Contributor

@SajidAlamQB SajidAlamQB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great thank you for this awesome work! Just had a few comments.

kedro/config/omegaconf_config.py Show resolved Hide resolved
kedro/config/omegaconf_config.py Outdated Show resolved Hide resolved
Copy link
Contributor

@SajidAlamQB SajidAlamQB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great! 🌟 I am happy for it to stay named OmegaConfLoader temporarily until 0.19.0 when we can rename it to ConfigLoader and remove the old TemplatedConfigLoader and ConfigLoader.

…loader instance

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Copy link
Contributor

@noklam noklam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is cleaner now with some of the methods moving into the class. I just re-iterate on one of the comment, generally look great!

kedro/config/omegaconf_config.py Outdated Show resolved Hide resolved
tests/config/test_omegaconf_config.py Show resolved Hide resolved
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
@merelcht merelcht requested a review from noklam December 15, 2022 10:08
Copy link
Contributor

@noklam noklam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for explaining! Nice work🌟

kedro/config/omegaconf_config.py Outdated Show resolved Hide resolved
kedro/config/omegaconf_config.py Outdated Show resolved Hide resolved
kedro/config/omegaconf_config.py Show resolved Hide resolved
kedro/config/omegaconf_config.py Outdated Show resolved Hide resolved
kedro/config/omegaconf_config.py Outdated Show resolved Hide resolved
kedro/config/omegaconf_config.py Outdated Show resolved Hide resolved
merelcht and others added 5 commits December 15, 2022 13:05
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a new configloader with OmegaConf as replacement for anyconf
5 participants