Make registering custom resolvers easy

[merelcht]

Description

Omegaconf provides functionality to register custom config resolvers: https://omegaconf.readthedocs.io/en/2.3_branch/usage.html#resolvers
At the moment it's possible to do this by subclassing the OmegaConfigLoader. Ideally it should be possible to do this without having to make a custom class.

Context

This feature would allows users to do more advanced variable interpolation in their config.

Possible Implementation

Allow users to provide custom resolvers through settings.py, adding a new constructor arg like custom_resolvers : Optional[Dict[str, Callable]]

Possible Alternatives

Perhaps users can achieve this as well by using hooks.

[datajoely]

Awesome, here is a really scrappy version of what I think settings.py could look like:

import os
import polars as pl

CONFIG_LOADER_ARGS = {
   "custom_resolvers" : {
         "oc.env" : lambda x : os.environ[x],
         "polars" : lambda x : getattr(pl, x)
    }
}

• oc.env we may want to actually enable by default since there is a built-in version of this.
• polars is an attempt at addressing @astrojuanlu 's issue here where the user (I think could then do:

load_args:
     dtypes: 
       product_age: ${polars:Float64}
       group_identifier: ${polars:Utf8}

[marrrcin]

A few thoughts:

• should this setting enable all resolvers for all configs?
• should this setting allow to "clear all and use only my resolvers", or "add my resolvers to the existing ones"?

I totally agree with:

oc.env we may want to actually enable by default since there is a built-in version of this.

This API would obviously allow to register e.g. oc.select as a resolver, but there might be a friction among users, because the oc.*s are more like "built-in", so people might ask.

[datajoely]

Agreed on both points

• I think we should protect the 'oc.*' namespace
• We should have an additional option in CONFIG_LOADER_ARGS which handles the resolver overwriting question.

[datajoely]

Could you elaborate on how you imagine this would work?

should this setting enable all resolvers for all configs?

Do you mean like scoping by env or something?

[merelcht]

On the point of enabling oc.env by default: while I see valid reasons of using oc.env in other places than credentials, we also made a deliberate decision to not enable it for all config types, because we want to make it clear that you shouldn't just be using environment variables all over the place. Like for parameters etc.

When we conducted the config research in the summer (see #1847), the main use case for environment variables was credentials, and so we decided to enable it for just that to start with. An advanced user could overwrite this behaviour and just enable environment variables for everything, but I still believe it's a sensible choice not to enable it by default just to make sure users don't just abuse env vars because it's easy.

[marrrcin]

Could you elaborate on how you imagine this would work?

should this setting enable all resolvers for all configs?

Do you mean like scoping by env or something?

I've meant what @merelcht is talking about above - right now, the oc.env is scoped only to credentials.

---

An advanced user could overwrite this behaviour and just enable environment variables for everything, but I still believe it's a sensible choice not to enable it by default just to make sure users don't just abuse env vars because it's easy.

+1 to that!

[MatthiasRoels]

Why not enable the built-in resolvers for all configuration types? I mean, if a user wants to go ahead and use environment variables in their config, why not? I do see added value for e.g. configuring different paths in catalog entries based on the env (dev vs qa vs prod).

[noklam]

In hydra they use a specific _target_ key for Dynamic Injection.

data_fab:
  type: data_fab.extras.DataSet
  save_args:
    if_exists: append
    dtype:
      simulation_payload:
        _target_: sqlalchemy.types.JSON
        _partial_: False

[astrojuanlu]

By the way, registering custom resolvers does not even need subclassing OmegaConfigLoader:

# settings.py

if not OmegaConf.has_resolver("dummy"):
    OmegaConf.register_new_resolver("dummy", lambda x: x)

CONFIG_LOADER_CLASS = OmegaConfigLoader

apparently OmegaConf is a singleton 😬 or am I missing something?

[astrojuanlu]

(The if not OmegaConf.has_resolver(name) conditional is needed because apparently settings.py is executed twice by Kedro)

[noklam]

Why not enable the built-in resolvers for all configuration types? I mean, if a user wants to go ahead and use environment variables in their config, why not? I do see added value for e.g. configuring different paths in catalog entries based on the env (dev vs qa vs prod).

I guess this is exactly what Kedro's environment doing already.

[MatthiasRoels]

Why not enable the built-in resolvers for all configuration types? I mean, if a user wants to go ahead and use environment variables in their config, why not? I do see added value for e.g. configuring different paths in catalog entries based on the env (dev vs qa vs prod).

I guess this is exactly what Kedro's environment doing already.

I know, but if you can do it with an environment variable, you can simply add your catalog entry in conf/base without having to create "duplicates" in the different envs with only minor tweaks.

[noklam]

I see your point, however, we don't want to encourage users to abuse environment variables for everything. It's still a bit unclear how we should enable this feature.

I think for that purpose we should keep the discussion there. For this specific PR it's about how to enable registering custom resolver, and whether oc.env should be enabled by default is out of scope. I quite like @datajoely's suggestion.

• Add "globals" functionality to OmegaConfigLoader #2175

An advanced user could overwrite this behaviour and just enable environment variables for everything, but I still believe it's a sensible choice not to enable it by default just to make sure users don't just abuse env vars because it's easy.

[ankatiyar]

Summary of tech design session on 24/05:

Discussed two possible solutions -

1. Document how users can register resolvers using OmegaConf singleton in settings.py -> Make registering custom resolvers easy #2407 (comment)
   Problems with this approach (comments by @noklam)

1. It feels weird you need to import the underlying singleton from omegaconf to to change the settings, it works because it’s a singleton but doesn’t mean we should do it, in a way it’s a leaky abstraction
2. In the context of a standalone component, the Config class should be functioning once it’s constructed. With this solution you need to first construct the class, then register the resolvers in a separate step, which feels wrong.

2. Allow users to register custom resolvers using CONFIG_LOADER_ARGS -> Make registering custom resolvers easy #2407 (comment)
   This is the approach we'll go ahead with. The following things need to be done -

• Activate all built-in resolvers by default for OmegaConfigLoader except for oc.env
• Implement a very thin wrapper that will pass all the arguments straight to OmegaConf.register_new_resolver(). This will let users overwrite built-in resolvers using replace=True.
• On the documentation side, add a Kedro specific example for using custom resolvers in config.

[ankatiyar]

Closing this issue in favour of the following tickets -

• Allow registering of custom resolvers to OmegaConfigLoader through CONFIG_LOADER_ARGS #2622
• Activate all built-in resolvers for OmegaConfigLoader except oc.env #2623
• Document the use of custom resolvers with OmegaConfigLoader #2624