Proposal: allow ConfigObj to transparently read settings from environment vars

[RobRuana]

Proposal

Add a use_env_vars mode in ConfigObj that allows ConfigObj to transparently read settings from environment variables, if they exist.

If use_env_vars is False – the default – then ConfigObj works the way it always has.

If use_env_vars is True then ConfigObj will attempt to read settings from the environment first, before reading settings loaded from a file.

Setting names will be converted to valid environment variable names when reading from the environment:

• non-alphanumeric, non-underscore characters will be converted to underscores
• section names and keywords will be concatenated with a double underscore
• setting names beginning with a number will be prefixed with an underscore
• setting names will be converted to all uppercase

This mode could potentially also be enabled by adding a special setting directly in a config file, thus enabling this feature without changing any existing application code:

[ConfigObj]
use_env_vars = True

Example

from configobj import ConfigObj
config = ConfigObj(filename, use_env_vars=True)

# First checks os.environ.get('KEYWORD1')
value1 = config['keyword1']

# First checks os.environ.get('DOTTED_KEYWORD2')
value2 = config['dotted.keyword2']

# Since 'section1' is a section, no checks are made in the environment
section1 = config['section1']

# First checks os.environ.get('SECTION1__KEYWORD3')
value3 = section1['keyword3']

# First checks os.environ.get('SECTION1__KEYWORD4')
value4 = section1['keyword4']

Rationale

Transparently reading settings from the environment should ease the containerization of any app that uses ConfigObj.

The use_env_vars setting is completely backwards compatible, so it shouldn't disrupt any apps that currently use ConfigObj.

Further Thoughts

The behavior of use_env_vars mode could be further modified by a series of settings. Each of these settings could also potentially be configured directly in a config file using the special [ConfigObj] section:

[ConfigObj]
use_env_vars = True
env_vars_section_separator = "__"
env_vars_prefix = ""
env_vars_uppercase = True

env_vars_section_separator

The env_vars_section_separator parameter sets the separator used to concatenate section names and keywords (defaults to "__").

config = ConfigObj(use_env_vars=True, env_vars_section_separator="__X__")
# First checks os.environ.get('SECTION1__X__KEYWORD1')
value1 = config['section1']['keyword1']

env_vars_prefix

To prevent environment variable collisions, an env_vars_prefix parameter can set a global prefix used when checking the environment (defaults to empty string).

config = ConfigObj(use_env_vars=True, env_vars_prefix="MYAPP_")
# First checks os.environ.get('MYAPP_KEYWORD1')
value1 = config['keyword1']

env_vars_uppercase

In general, environment variable names are always uppercased, but this behavior could be turned off using an env_vars_uppercase parameter (defaults to True).

config = ConfigObj(use_env_vars=True, env_vars_uppercase=False)
# First checks os.environ.get('keyword1')
value1 = config['keyword1']

[jcooter]

This change is pretty independent of any containerization setup. The same environment variables can be used in configuration management, or server orchestration setups using cloud-init + ansible or similar.

[RobRuana]

Well, if your legacy application already reads its config from a series of files, you have a few options for containerization:

1. Update the code to read from the environment
2. Have some orchestration process inside the container overwrite the config files when your config changes
3. Mount the config directory from the host system so you can overwrite the config files from outside the container

This proposal allows applications to read config from the environment without changing any application code 😃

[aellerton]

If anyone is looking for a pragmatic and somewhat gritty way of doing this, here's what I do.

If you're loading config like this:

config = ConfigObj(config_filename)

but would like a config file like this:

api_key = ljalfq9u3fi3jljfkaf

to be:

api_key = $API_KEY

change the Python loading code to:

config = ConfigObj(substitute_env(config_filename))

And add this function:

import io
import os
import re

def substitute_env(filename):
    """Reads filename, substitutes environment variables and returns a file-like
     object of the result.

    Substitution maps text like "$FOO" for the environment variable "FOO".
    """

    def lookup(match):
        """Replaces a match like $FOO with the env var FOO.
        """
        key = match.group(2)
        if key not in os.environ:
            raise Exception(f"Config env var '{key}' not set")  # or ignore
        return os.environ.get(key)

    pattern = re.compile(r'(\$(\w+))')
    with open(filename, 'r') as src:
        content = src.read()
        replaced = pattern.sub(lookup, content)

    return io.StringIO(replaced)

[jhermann]

Instead of the bool, this should use "envar_prefix=MYAPP" (default empty or None), and then use "{prefix}_{cfgkey}".upper() as the environ name.

[robdennis]

I think this worth considering for a new major version