How do I fill the credentials from environment variables

[HugoPerrier]

Issue description

I would like to set placeholders in my credential files and have them filled with environment variables.
Exemple of credentials.yml file (in this case connection to a postgre database):

dev_sql:
  con: "postgresql://${USERNAME}:${PASSWORD}@${SERVERNAME}:${PORT}/${NAME}"

My solution (thanks DmitriiDeriabinQB and documentation):

Edit the run.py file to :

1. Create a class inheriting from KedroContext (the default ProjectContext works fine)
2. If you don't use the ProjectContext class, adapt the .kedro.yml file
3. Override the KedroContext method _create_config_loader
4. Use a TemplatedConfigLoader instead of a ConfigLoader and pass the environment variable dict as globals_dict argument

My run.py file looks as follows:

class ProjectContext(KedroContext):
    """Users can override the remaining methods from the parent class here,
    or create new ones (e.g. as required by plugins)
    """

    project_name = "my_project"
    project_version = "0.16.1"
    package_name = "my_package"

    def _get_pipelines(self) -> Dict[str, Pipeline]:
        return create_pipelines()

    # Get credentials from environment variables
    def _create_config_loader(self, conf_paths):
        return TemplatedConfigLoader(conf_paths, globals_dict=os.environ)

Originally posted by @HugoPerrier in #49 (comment)

[mzjp2]

Hey @HugoPerrier -- thanks for creating this issue. It looks like you've gotten it solved with help from Dmitrii and have been kind enough to post your solution so others can benefit from it. I'll go ahead and close this as done. Thank you!

[bensdm]

Hi, i am trying to find the cleanest way to provide credentials as input to a node. Not sure if this is the best place to ask the question. Is there a way to load credentials.yml in the dataCatalog like the parameters.yml. Thanks

[hugo-quantmetry]

UPDATE for kedro >= 0.18.1

It seems like many people are still reading this issues so I'll post an update.

The solution proposed above is obsolete.
To inject credentials through environment variables, I suggest the following.

Modify the file project_name/src/package_name/settings.py as follows :

# Class that manages how configuration is loaded.
from kedro.config import TemplatedConfigLoader
import os
CONFIG_LOADER_CLASS = TemplatedConfigLoader
# Keyword arguments to pass to the `CONFIG_LOADER_CLASS` constructor.
CONFIG_LOADER_ARGS = {
#    "globals_pattern": "*globals.yml",
    "globals_dict": os.environ,
}

Then you will be able to use environment variables in your conf files.

Ex: You can set environment variables MY_LOGIN and MY_PASSWORD.
Modify the file project_name/conf/base/parameters.py as follows :

my_login : ${MY_LOGIN}
my_password : ${MY_PASSWORD}

And your kedro parameters params:my_login and params:my_password will take the value of the corresponding environment variables.

Explanation

By default, kedro uses a regular ConfigLoader object to load the conf files.
The settings file let's you change from a regular ConfigLoader class to a TemplatedConfigLoader class.
The TemplatedConfigLoader allows the usage of variables in the conf files with the syntax ${MY_VARIABLE}.
The variables that can be used in the conf can be specified in two ways:

• global_patterns : Variables specified in a yaml file "globals.yml" (ex: project_name/conf/base/globals.yml)
• global_dict : Variable specified as a python dict-like object.

By passing os.environ for the global_dict argument, environment variables are made available :)

Warning : There is a bug in version 0.18.0 so this solution would raise an error.

[schoobani]

How can we access os.environ when calling a custom dataset? The following doesn't work.

my_custom_dataset:
  type: {project_name}.extras.datasets.my_custom_dataset.CustomDataset
  args:
    arg1: "foo"
    arg2: ${VERY_SECRET_API_KEY}

[noklam]

@schoobani Hey! Sorry that you have to find this old issue. This should be the most up-to-date way to do so. https://docs.kedro.org/en/stable/configuration/advanced_configuration.html#how-to-load-credentials-through-environment-variables