Add "runtime_params" resolver to allow overriding of config with CLI params

[ankatiyar]

Description

Resolve #2531

Development notes

runtime_params resolver

• similar to globals - registered using _registed_runtime_params_resolver()
• Uses _get_runtime_value() to get the interpolated value, also similar to globals' _get_globals_value()

Changes to globals

• Read globals and populate self._globals in the __init__ function.
• When global values are read by _get_globals_value() it uses the self._globals dict.
• This is so when globals are first read, the runtime_params resolver is cleared and then registered again in the __getitem__ fn
• Also helps that we don't need to read self["globals"] each time to resolve wherever globals is used.

On the topic of nested resolvers

Need feedback from reviewers on this

• I disable the use of runtime_params in globals.yml by disabling the runtime_params in __getitem__ if the key is "globals" and then registering it again. Throws an UnsupportedInterpolationType error when runtime_params: is used in globals.
• Disabling globals within runtime_params is tricky, I haven't found a clean way to do it.

However, I think allowing globals to be used as a default value for parameters with runtime_params resolver is valid and might be useful and we probably should allow this -

my_param : "${runtime_params: y, ${globals: x, 34}}"

TODO

• Test
• Docs
• Release Notes

Checklist

• Read the contributing guidelines
• Signed off each commit with a Developer Certificate of Origin (DCO)
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[merelcht]

This looks great @ankatiyar ! I have also tried some simple use cases and it works perfectly 👍 I agree that it's should be allowed for users to use globals as a default inside the runtime_params resolver. But definitely the other way around doesn't make sense.

Are you adding docs as part of this PR or separately?

[merelcht]

This could potentially be raised because of something else right? So we should make sure that this specific message only shows when the user is trying to use runtime_params with globals.

[ankatiyar]

I've updated the condition to only show this message if runtime_params is used.

[ankatiyar]

I've also added docs to this PR itself since it was just one section.

[merelcht]

This is great! I think the docs are very clear too 👍

[stichbury]

This looks good to me, thanks @ankatiyar!

I made a commit to the branch to move the table of contents that links to the "How to " subsections to the top of the page (as there was some repetition with it in the place previously sited).

[noklam]

I find this a bit hard to follow - I am not sure if this is registered or not. If I follow the logic, it assumes that it is always on and only turn off if key==global. Maybe this should move to the beginning instead? Then the __init__ register_runtime_params_resolvers can be removed optionally.

In fact, I found this is inconsistent and I think we should treat this same as the read_environment_variable flag.

[ankatiyar]

Since replace=True is set in _register_runtime_params_resolver(), I didn't add an if condition here. It'll re-register it anyway.

[ankatiyar]

I've moved it to the beginning and removed the call from __init__.

[merelcht]

Btw the docs failure for the databricks was fixed in develop with https://docs.databricks.com/en/dev-tools/cli/authentication.html

[noklam]

I think we should have a comment explaining why we treat global differently.

[noklam]

Approved with minor comments. It's a great feature to add!