env_prefix ignored if extra=allow

[iAnanich]

Problem: if model has both extra="allow" and env_prefix set, then model will have all envvars regardless of prefix, essentially ignorring env_prefix config.

Expected behavior: fields that match env_prefix can be added into model, even if they are not declared.

Noticed that it changed when upgraded, remember that in 2.1 it was clsoer to that, but also actually didn't include undeclared vars from .env . But having undeclared vars in .env that is shared between multiple nested settings files broke my code :)

Example code:

from pydantic_settings import BaseSettings

class MySettings(BaseSettings):
    class Config:
        env_prefix = "MY_PREFIX_"
        case_sensitive = False
        env_file = ".env"
        extra = "allow"
    not_in_a_file: str = "yes"
    inside_file: bool = False

obj = MySettings()
print(obj)

if not obj.inside_file:
    print(".env/MY_PREFIX_inside_file is not detected, even though it's declared.")
if "notaprefix" in obj.model_dump():
    print("undeclared var that doesn't match prefix got into object.")
if not "bar" in obj.model_dump():
    print("did not get undeclared but matching prefix envvar: .env/MY_PREFIX_BAR")
    if "my_prefix_bar" in obj.model_dump():
        print("\t... but it is there with it's prefix.")

respective .env to test:

NOTAPREFIX=True
MY_PREFIX_inside_file=True
MY_PREFIX_BAR=1

So latest main gives this output:

not_in_a_file='yes' inside_file=True notaprefix='True' my_prefix_bar='1'
undeclared var that doesn't match prefix got into object.
did not get undeclared but matching prefix envvar: .env/MY_PREFIX_BAR
	... but it is there with it's prefix.

and v2.1.0 gives this:

not_in_a_file='yes' inside_file=True
did not get undeclared but matching prefix envvar: .env/MY_PREFIX_BAR

[hramezani]

Thanks @iAnanich for reporting this.

Yeah, it it an expected behavior and we already documented it:

Pydantic settings loads all the values from dotenv file and passes it to the model, regardless of the model's env_prefix. So if you provide extra values in a dotenv file, whether they start with env_prefix or not, a ValidationError will be raised.

[iAnanich]

That removes usecase that I quite liked: having a bunch of Models read with different sets of variables, each with their own prefix, possibly nested, while allowing for extra fields to be added without disturbing.

That note makes it seem like env_prefix doesn't matter at all, which isn't what we see in practice. It's unexpected that once extra="allow" is used prefix is discarded.

[hramezani]

That note makes it seem like env_prefix doesn't matter at all, which isn't what we see in practice. It's unexpected that once extra="allow" is used prefix is discarded.

Agree, probably it is not clear. pydantic-settings passes all the extra values, but for the values that are related to the fields, it will remove the prefix and then forward it to pydantic.
I think we need more flag to control the extra values behavior in pydantic-settings. because we are using right now the flags from pydantic.

[iAnanich]

I believe this change is a consequence of #218 and #221 nuance of allowing undeclared variables with prefix or just ignoring them was lost.

In my head 3 flags were enough:

• forbid means that only declared fields can be present, and must follow env_prefix
• ignore means that only if defined envvars are present (and follow env_prefix) they influence model
• allow means that any envvar that follows env_prefix will influence model, being declared or not.

I haven't encountered other scenarios, but there might be a case for raising an error if there is an undeclared environment variable that matches the prefix, while ignoring undeclared variables that do not match the prefix. A potential solution could be a strict_env_prefix flag: when set to true, the model would only validate and get affected by environment variables that match the prefix. Otherwise, even using forbid wouldn't apply to variables that don't match. However, this seems unnecessary since the env_prefix is already defined - if all environment variables are required to have it, what is the purpose of having the flag?

[hramezani]

The reason that we collect all the values in .env is, pydantic has to handle these flags. it means if you define a model with extra='forbid', you expect to get a validation error if there is an extra value presented in the .env file. So, if we only collect the values start with prefix, we will ignore the other values and the model won't raise validation error.