Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an oc.decode resolver to parse and evaluate strings #574

Closed
odelalleau opened this issue Feb 26, 2021 · 0 comments · Fixed by #606
Closed

Add an oc.decode resolver to parse and evaluate strings #574

odelalleau opened this issue Feb 26, 2021 · 0 comments · Fixed by #606
Assignees
Milestone

Comments

@odelalleau
Copy link
Collaborator

odelalleau commented Feb 26, 2021

Is your feature request related to a problem? Please describe.

With env being deprecated in favor of oc.env (#573), which always returns environment variables as strings, we need an easy way to convert strings to common types like bool, int, float, etc.

Describe the solution you'd like

Introduce a new resolver oc.decode that converts a string automatically, using the common OmegaConf grammar.
For instance:

cfg = OmegaConf.create(
    {
        "database": {
            "port": '${oc.decode:${oc.env:DB_PORT}}',
            "nodes": '${oc.decode:${oc.env:DB_NODES}}',
        }
    }
)

os.environ["DB_PORT"] = '3308'  # integer
os.environ["DB_NODES"] = '[host1, host2, host3]'  # list

print(repr(cfg.database.port))
print(repr(cfg.database.nodes))

should print:

3308
['host1', 'host2', 'host3']

In addition, ${oc.decode:null} is valid and returns None, in order to support for instance ${oc.decode:${oc.env:DB_PORT,null}}

Differences with the old env resolver

Note that ${oc.decode:${oc.env:VAR}} is not strictly equivalent to the old ${env:VAR} from OmegaConf 2.0. This is because oc.decode relies on the same parsing logic as used when parsing resolver arguments in interpolations, while env was using more basic custom parsing functions. Some key differences are:

  • oc.decode will raise an exception if the content of the environment variable is invalid according to the grammar (ex: an unquoted string containing invalid characters). One may use quotes (ex: VAR="'~!&*'") to avoid such exceptions (or just not use oc.decode to keep the original content of the variable)
  • oc.decode handles more data types like dictionaries, lists, null and even interpolations, which would have been kept as strings when using env
@odelalleau odelalleau added this to the OmegaConf 2.1 milestone Feb 26, 2021
@odelalleau odelalleau self-assigned this Feb 26, 2021
odelalleau added a commit to odelalleau/omegaconf that referenced this issue Mar 15, 2021
* Restore and deprecate the old `env` resolver for backward
  compatibility with OmegaConf 2.0

* The new `oc.env` resolver keeps the string representation of
  environment variables, and does not use the cache

* The new `oc.decode` resolver can be used to parse and evaluate strings
  according to the OmegaConf grammar

Fixes omry#383
Fixes omry#573
Fixes omry#574
odelalleau added a commit to odelalleau/omegaconf that referenced this issue Mar 15, 2021
* Restore and deprecate the old `env` resolver for backward
  compatibility with OmegaConf 2.0

* The new `oc.env` resolver keeps the string representation of
  environment variables, and does not use the cache

* The new `oc.decode` resolver can be used to parse and evaluate strings
  according to the OmegaConf grammar

Fixes omry#383
Fixes omry#573
Fixes omry#574
odelalleau added a commit to odelalleau/omegaconf that referenced this issue Mar 18, 2021
* Restore and deprecate the old `env` resolver for backward
  compatibility with OmegaConf 2.0

* The new `oc.env` resolver keeps the string representation of
  environment variables, and does not use the cache

* The new `oc.decode` resolver can be used to parse and evaluate strings
  according to the OmegaConf grammar

Fixes omry#383
Fixes omry#573
Fixes omry#574
odelalleau added a commit that referenced this issue Mar 18, 2021
* Introduce new `oc.env` and `oc.decode` resolvers

* Restore and deprecate the old `env` resolver for backward
  compatibility with OmegaConf 2.0

* The new `oc.env` resolver keeps the string representation of
  environment variables, does not use the cache, and accepts None as
  default value

* The new `oc.decode` resolver can be used to parse and evaluate strings
  according to the OmegaConf grammar

Fixes #383
Fixes #573
Fixes #574

* Allow `oc.decode` to evaluate interpolations

* Improve documentation of `oc.decode`

In particular it explains that dictionaries and lists are automatically
converted to transient config nodes.

* Update docs/source/usage.rst

Co-authored-by: Omry Yadan <omry@fb.com>

* Update news/573.api_change

Co-authored-by: Omry Yadan <omry@fb.com>

* Update omegaconf/_utils.py

Co-authored-by: Omry Yadan <omry@fb.com>

* Update omegaconf/_utils.py

Co-authored-by: Omry Yadan <omry@fb.com>

* Remove the USERID example

* Show example of quoted string as default value for `oc.env`

* Remove duplicated comments (# #)

* Validate default value of `oc.env` even when not used

* Simplify tests with recwarn

* Use convenience `show()` function in doc to show type and value

* Update doc on string interpolations

* More readable test formatting

* Improve comment formatting

* Restore interpolation examples

* Update docs/notebook/Tutorial.ipynb

Co-authored-by: Omry Yadan <omry@fb.com>

* Update docs/source/usage.rst

Co-authored-by: Omry Yadan <omry@fb.com>

* Update docs/source/usage.rst

Co-authored-by: Omry Yadan <omry@fb.com>

* Rephrasing in doc

* Use `show()` function in doc

* Raise a KeyError instead of ValidationError for missing env variables

* Remove handling of "null" as default in legacy env resolver

* Update news

* Explicit typing for the default value of the `oc.env` resolver

* Use a more appropriate exception type

* Update tests/test_interpolation.py

Co-authored-by: Omry Yadan <omry@fb.com>

* Safer markers for default values

* Fix coverage

* Use more appropriate TypeError

* Refactor: consistent use of _DEFAULT_MARKER_

Co-authored-by: Omry Yadan <omry@fb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants