Allow users to pass credentials through environment variables

[jmholzer]

NOTE: Kedro datasets are moving from kedro.extras.datasets to a separate kedro-datasets package in
kedro-plugins repository. Any changes to the dataset implementations
should be done by opening a pull request in that repository.

Description

Resolves #1909.

Development notes

OmegaConf resolves DictConfig objects on read. In the following example (where credentials.yml contains environment variable grammar such as "key": "${oc.env:TEST_KEY}"), the oc.env resolver would have to be registered, or line 3 will raise an UnsupportedInterpolationType exception.

conf = OmegaConfLoader(...)
credentials = conf["credentials"]
credentials["test"]

However, the scope of the issue is to keep the oc.env resolver switched off. The way around this is to use OmegaConf.resolve, which resolves DictConfig objects in-place when these objects are created, using the oc.env resolver. This is done in the _resolve_environment_variables method.

Includes four automated tests:

1. Check that environment variables are resolved when the credentials is used as a key.
2. Check that environment variables are not resolved when credentials is not used as a key.
3. Check that the oc.env resolver is not registered after environment variables are read if it was not registered beforehand.
4. Check that the oc.env resolver remains registered after environment variables are read if it was registered beforehand.

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes

[antonymilne]

Stupid question... why can't we just leave oc.env switched on? Something to do with not wanting to allow environment variables except in credentials?

Just a quick comment on naming also: I think @idanov previously suggested that we might drop the oc. namespace for this and just have env instead.

[jmholzer]

Stupid question... why can't we just leave oc.env switched on? Something to do with not wanting to allow environment variables except in credentials?

Exactly, ideally we'd only load credentials in this way.

Just a quick comment on naming also: I think @idanov previously suggested that we might drop the oc. namespace for this and just have env instead.

Thanks for passing it on, I'll have a chat with him as I would personally be for keeping it.

[datajoely]

so happy this is coming!

[idanov]

Maybe we can drop the oc. or simply support both env and oc.env?

[jmholzer]

Do you mean drop the "oc." in:

1. name of the resolver (first argument to register_new_resolver)
   or in:
2. the name assigned to omegaconf.resolvers.oc.env (second argument to register_new_resolver)?

I personally wouldn't drop the oc. from 1., since this is the name assigned to the resolver by OmegaConf when it is instantiated. For 2. I would be ok with losing the namespace from the import: from omegaconf.resolvers.oc import env.

[idanov]

It would have been nice if OmegaConf did not rely on global state...

[merelcht]

This looks great! I'm so excited for this functionality to be added 🤩