How to build a cascading settings resolver

[trondhindenes]

In our current config module we have the following logic:

• Determine some basic info about the app such as app name, and environment name (from environment variables)
• Then use these to optionally look up data from a settings store such as aws ssm parameter store or similar

This is done by invoking a series of config providers, from lowest-prioritoy up to highest. Each variable value can at any time be overwritten by the value from a higher-priority provider.

It seems to me that pydantic doesn't support this, because it seems to read top-down starting from the highest-priority provider. In my example I guess AWS SSM would be read first, and then any settings left unresolved would be left to the environment variable provider. That wouldn't work for us, since we need the env var provided-values to build the paths etc for aws ssm.

So I guess I'm asking:
Is it possible with pydantic-settings to build a hierarchy of "config sources" where one source relies on data provided by another?

[mnbbrown]

I was exploring this earlier this week.. from my grep of the codebase it's not possible currently because __call__ in PydanticBaseSettingsSource doesn't get the current state of the settings model as an argument. See https://github.com/pydantic/pydantic-settings/blob/main/pydantic_settings/main.py#L181

If the current model was passed in the source itself could:

a. retrieve the current value of attributes to help build new attributes
b. decided if it should overwrite or not.

This PR is an imperfect solution to this problem but goes some way to addressing it. Imperfect because the current_state is untyped and it's a breaking change.

[hramezani]

Thanks @trondhindenes for reporting this issue.

Yes, as @mnbbrown described it is not possible right now.

We have to postpone it to V3 because it will introduce a breaking change.
I will mark it as a feature request and we can include this in V3

[hramezani]

@mnbbrown @trondhindenes what do you think about #326?

Does this PR solve your problems?

[trondhindenes]

I believe it does, thank you!