Use model class names as tags in format_as_xml and add option to include field titles and descriptions as attributes

[giacbrd]

The current helper format_as_xml allows to transform any Python object into a XML string, which is a preferable format for ingesting structured data into LLMs.

This PR adds an optional parameter to this helper for exploiting Pydantic Field metadata: attributes like title, description or alias. These can be serialized in the XML as element attributes.

This is an easy approach for the developer in order to help the LLM to understand the structured data fields, beyond their names.

Basic example:

class Person(BaseModel):
    name: str = Field(description="The person's name")
    age: int = Field(description='Years', title='Age', default=18)

person = Person(name="John", age=42)

person becomes

<name description="The person's name">John</name>
<age title="Age" description="Years">42</age>

Future developments could be:

• Setting attributes only down to a specific level in nested objects or avoiding repeating attributes in lists of objects.
• Creating a general natural language description of a model based on its definition (e.g. "A person data is made of a name and ...")

[DouweM]

@giacbrd Thanks Giacomo, it's a nice feature!

[DouweM]

As you suggested in the description, I'd really like to include these attributes only the first time the field is seen, so we don't unnecessarily flood the LLM context.

[giacbrd]

OK I have added a parameter: I would like to leave the option of adding attributes at each object occurrence. I imagine cases where I have a complex object A, with many fields and deep structure. In this deep structure an object B can occur in “distant” spots. For an LLM could be tricky to recognize the semantic of the object B at every occurrence, given it would be described only at the first occurrence (where "distance" is in terms of tokens)

[giacbrd]

@DouweM thanks for the review, I am currently on vacation, I will reply to your comments, and make the changes, next week

[DouweM]

These 2 calls end up calling _parse_data_structures twice, could we do it just once?

[DouweM]

Combined with my suggestion to always initialize _fields and _element_names as empty dicts, I think we can call self._parse_data_structures(self.data) when we see a BaseModel or dataclass and handle which (or both) of the two to populate in there

[giacbrd]

I have committed a solution for calling _parse_data_structures once. Before, I initialized these data structures with None for treating them as singletons, they must be created once. After they are populated they could be empty dictionaries. There are cases where not having a value that means "no initialization" could be tricky. E.g., a long list of models where fields have not attributes filled. We would call _parse_data_structures for each model and _fields would always remain an empty dictionary.

Now I use a flag _is_info_extracted so I make sure _parse_data_structures is called once and for all. We now call it for fields info even if we only have dataclasses, so no attributes to extract from any Pydantic Field. I have relaxed these checks because I expect to extract also dataclasses' field metadata in future developments.

The solution of an explicit method for the logics of initialization, even if trivial, looks clear to me. Moreover, ruff would complain of the code complexity if I keep these logics in _to_xml or in _parse_data_structures.

[DouweM]

This more like included_fields right?

[giacbrd]

changed

[DouweM]

Could we give self._element_names a default value of {} and always wriet directly into that instead of checking for None and passing element_names around as an arg?

Same for fields_map

[giacbrd]

see comment below

[DouweM]

Since _to_xml tag can be None, can we make that a default value so we can skip passing None here?

[giacbrd]

done

[DouweM]

If path should only be omitted for the root node, I think we should make it required and pass '' explicitly there

[giacbrd]

done

[DouweM]

@giacbrd Sorry for the delay in reviewing, thanks for the changes, we're almost there!

[DouweM]

@giacbrd Thanks a lot Giacomo!

[giacbrd]

@DouweM you're welcome!