Parsing nested argument of type list fails with cli_parse_args and alias_generator

[andlogreg]

Versions:

pydantic                       2.9.1
pydantic_core                  2.23.3
pydantic-settings              2.5.2

my_config.py:

from pydantic import Field
from pydantic_settings import BaseSettings, SettingsConfigDict
from pydantic import BaseModel, Field, ConfigDict
from typing import List


class SubModel(BaseModel):
    sub_list: List[str] = Field(
        default_factory=list, description="A list argument"
    )
    sub_int: int = Field(
        0,
        description="An int argument",
    )

    model_config = ConfigDict(
        alias_generator=lambda x: x.replace("_","__"),
    )


class MySettings(BaseSettings):
    sub_model: SubModel = SubModel()

    top_list: List[str] = Field(
        default_factory=list, description="A list argument"
    )

    top_int: int = Field(
        0,
        description="An int argument",
    )

    model_config = SettingsConfigDict(
        cli_parse_args=True,
        alias_generator=lambda x: x.replace("_","__")
    )



settings = MySettings()

print(settings)

✅ Help:

python my_config.py --help
usage: my_config.py [-h] [--sub__model JSON] [--sub__model.sub__list List[str]] [--sub__model.sub__int int] [--top__list List[str]] [--top__int int]

options:
  -h, --help            show this help message and exit
  --top__list List[str]
                        A list argument (default factory: list)
  --top__int int        An int argument (default: 0)

sub__model options:
  --sub__model JSON     set sub__model from JSON string
  --sub__model.sub__list List[str]
                        A list argument (default: [])
  --sub__model.sub__int int
                        An int argument (default: 0)

✅ Running without args ok:

# python my_config.py 
sub_model=SubModel(sub_list=[], sub_int=0) top_list=[] top_int=0

✅ Setting top args ok:

# python my_config.py --top__list=a,b,c --top__int=34
sub_model=SubModel(sub_list=[], sub_int=0) top_list=['a', 'b', 'c'] top_int=34

✅ Setting submodel int arg ok:

# python my_config.py --sub__model.sub__int=34
sub_model=SubModel(sub_list=[], sub_int=34) top_list=[] top_int=0

❌Setting submodel list arg fails:

# python my_config.py --sub__model.sub__list=a,b,c
Traceback (most recent call last):
  File "./my_config.py", line 40, in <module>
    settings = MySettings()
  File "/usr/local/lib/python3.10/site-packages/pydantic_settings/main.py", line 152, in __init__
    super().__init__(
  File "/usr/local/lib/python3.10/site-packages/pydantic/main.py", line 209, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for MySettings
sub__model.sub__list
  Input should be a valid list [type=list_type, input_value='["a","b","c"]', input_type=str]
    For further information visit https://errors.pydantic.dev/2.9/v/list_type

Comments

Similar behaviour happens when argument is of type dict. It seems alias generation conflicts with parsing complex types?

[hramezani]

@kschwab please take look when you have time

[kschwab]

@hramezani this is an issue in the EnvSettingsSource. The above will fail after removing CLI and using os.environ['sub__model.sub__list'] = '["a","b","c"]' with env_nested_delimiter='.' etc. It seems to only occur if SubModel has an alias generator set. e.g., os.environ['sub__model.sub_list'] = '["a","b","c"]' without alias will work.

[andlogreg]

@hramezani this is an issue in the EnvSettingsSource. The above will fail after removing CLI and using os.environ['sub__model.sub__list'] = '["a","b","c"]' with env_nested_delimiter='.' etc. It seems to only occur if SubModel has an alias generator set. e.g., os.environ['sub__model.sub_list'] = '["a","b","c"]' without alias will work.

Does it look like an easy fix? If you give me some pointers I can try to create a PR

[hramezani]

Thanks @kschwab for the investigation.

@andlogreg I think I need to investigate and I don't think it is an easy fix. probably the change it not too much but finding the place of change takes time.

I will check it later.

[hramezani]

@andlogreg
I created #400. Could you please confirm the fix?

[andlogreg]

@andlogreg I created #400. Could you please confirm the fix?

@hramezani thanks for the PR! Unfortunately, I tried the example above and I still get the same error. Did you try it on your end?

[andlogreg]

@andlogreg I created #400. Could you please confirm the fix?

@hramezani thanks for the PR! Unfortunately, I tried the example above and I still get the same error. Did you try it on your end?

Actually, please ignore this comment. It was something with my environment. I can confirm it is now working :)

[hramezani]

Thanks @andlogreg for confirming. it will be included in the next release!