Clarification Needed on pyproject_toml_table_header Logic in Pydantic Settings

[py-mu]

Issue Context

I tried to use pydantic-settings for project configuration management, but I couldn't understand why the pyproject_toml_table_header is restricted to a single block.

        self.toml_table_header: tuple[str, ...] = settings_cls.model_config.get(
            'pyproject_toml_table_header', ('tool', 'pydantic-settings')
        )
        self.toml_data = self._read_files(self.toml_file_path)
        for key in self.toml_table_header:
            self.toml_data = self.toml_data.get(key, {})
        super(TomlConfigSettingsSource, self).__init__(settings_cls, self.toml_data)

If multiple headers are provided, this logic seems to overwrite toml_data repeatedly, resulting in toml_data containing content from only one header. Is my understanding correct?

Are there any alternative logics to better handle this content? For instance, would it be more appropriate to use something like:

self.toml_data = {k:v for k, v in self.toml_data.items() if k in self.toml_table_header}

or other

[hramezani]

Thanks @py-mu for reporting this issue.

Could you please explain more? something like an example pyproject file and a setting model would be helpful.

[py-mu]

I want to use the configuration content from two blocks, but I see that it only returns the last [logging], while the [database] before it gets overwritten in the loop. Is this the intended design? It seems that it can only return one instead of filtering based on the input toml_table_header.

model_config = SettingsConfigDict(
        toml_file=project_conf_path / "test.toml",
        pyproject_toml_table_header=('database','logging'),
        case_sensitive=True,
        extra='allow'
    )

# conf/test.toml

[database]
db_uri = "postgresql://user:password@localhost/dbname"

[logging]
level = "DEBUG"

Is it because of an issue with the way I'm using the method?

[py-mu]

I made an operational error; I need to reopen the issue.

[py-mu]

@hramezani

class Settings(BaseSettings):

    project_path: ClassVar[Path] = Path(__file__).parent.parent.parent
    project_conf_path: ClassVar[Path] = project_path / 'conf'

    logging: LoggingConf
    database: DataBaseConf

    model_config = SettingsConfigDict(
        toml_file=project_conf_path / "test.toml",
        pyproject_toml_table_header=('database','logging'),
        case_sensitive=True,
        extra='allow'
    )

    @classmethod
    def settings_customise_sources(
            cls,
            settings_cls: Type[BaseSettings],
            init_settings: PydanticBaseSettingsSource,
            env_settings: PydanticBaseSettingsSource,
            dotenv_settings: PydanticBaseSettingsSource,
            file_secret_settings: PydanticBaseSettingsSource,
    ) -> Tuple[PydanticBaseSettingsSource, ...]:
        return (init_settings, env_settings,
                dotenv_settings, file_secret_settings,
                PyprojectTomlConfigSettingsSource(settings_cls),
                YamlConfigSettingsSource(settings_cls))

class PyprojectTomlConfigSettingsSource(TomlConfigSettingsSource):
    """
    A source class that loads variables from a `pyproject.toml` file.
    """

    def __init__(
        self,
        settings_cls: type[BaseSettings],
        toml_file: Path | None = None,
    ) -> None:
        self.toml_file_path = self._pick_pyproject_toml_file(
            toml_file, settings_cls.model_config.get('pyproject_toml_depth', 0)
        )
        self.toml_table_header: tuple[str, ...] = settings_cls.model_config.get(
            'pyproject_toml_table_header', ('tool', 'pydantic-settings')
        )
        self.toml_data = self._read_files(self.toml_file_path)
        for key in self.toml_table_header:
            self.toml_data = self.toml_data.get(key, {})
        super(TomlConfigSettingsSource, self).__init__(settings_cls, self.toml_data)

[py-mu]

i am e

[py-mu]

version: 2.5.2

[hramezani]

Thanks @py-mu for explaining. the pyproject_toml_table_header is used to load data from one section of pyproject.toml file. e.g. if we have pyproject_toml_table_header=('tool', 'my.tool', 'foo') and the file content is:

    [tool.pydantic-settings]
    foobar = "Hello"

    [tool.pydantic-settings.nested]
    nested_field = "world!"

    [tool."my.tool".foo]
    status = "success"

Then the settings source loads the last section [tool."my.tool".foo].

Right now it doesn't support loading from multiple sections.
You can inherit from PyprojectTomlConfigSettingsSource and override the __init__ to load data from multiple sections.

I will mark this issue as a feature. so you can work on this and create a PR if you want.

[braindevices]

I encountered the similar problem. But after 2nd thought, I think it is actually the correct behaviour. Because normally one tool should only just parse one section. It will be very hacky way to parse other tool's section, what if other tool decide to change the schema?

If one want to handle the entire project then one should just load the toml from root with model_config = SettingsConfigDict(extra='ignore', pyproject_toml_table_header=()) (according to the doc).

Also doc actually demonstrate the correct way to use pyproject_toml_table_header:

pyproject_toml_table_header=('tool', 'pydantic-settings') which will load variables from the [tool.pydantic-settings] table.

Bascially what @py-mu wanted should be achieved using RootSettings in the example. @hramezani I think this is already existing feature.