Fix dynamic content types like audio and images

[davidberenstein1957]

Should be merged after #33

[davidberenstein1957]

@cfahlgren1, currently other modalities, like the openai_vision_example.py, do not work with the current message schema.

I am not too familiar with DuckDB but can you have a look at this schema definiton? The input is a bit dynaminc and I am not sure how to best tackle this besides converting to JSON (losing the nice array queryability etc).

tests/integration/observers/test_observers_examples.py hold some of the potential types we could be expecting and it should work for testing them.

[
      {
        role: "user",
        content: [
          {
            type: "text",
            text: "What's in this image?",
          },
          {
            type: "image_url",
            image_url: {
              url: imageUrl,
            },
          },
        ],
      },
]

[cfahlgren1]

Yeah very good point. Ollama has this format for images for their client library for vision

"messages": [
    {
      "role": "user",
      "content": "what is in this image?",
      "images": ["<base64-encoded image data>"]
    }

[cfahlgren1]

So for it to be a DuckDB Struct each row would have to have the same amount of keys.

The key difference is that DuckDB STRUCTs require the same keys in each row of a STRUCT column

We may have to make it a JSON type

[davidberenstein1957]

@cfahlgren1 Or we can rewrite the message structure to follow the following structure? I think they can be used interchangeably according to the typing. Then we avoid losing the queryability of the arrays. WDYT?
{"type": "text", "text": "hello", "image_url": None}
{"type": "text", "text": None, "image_url": "datauri"}

[davidberenstein1957]

These are the message formats. e.g.

ChatCompletionSystemMessageParam(content=""),
ChatCompletionUserMessageParam(
    content=ChatCompletionContentPartTextParam(text="test")
),
ChatCompletionAssistantMessageParam(content=""),
ChatCompletionUserMessageParam(
    content=ChatCompletionContentPartImageParam(image_url="image")
),
ChatCompletionAssistantMessageParam(content=""),
ChatCompletionUserMessageParam(
    content=ChatCompletionContentPartInputAudioParam(
        input_audio=InputAudio(data="audio", format="wav")
    )
),
ChatCompletionFunctionMessageParam(content=""),
ChatCompletionToolMessageParam(content="")

# the following are equal, more or less

ChatCompletionUserMessageParam(
    content=ChatCompletionContentPartTextParam(text="test")
) 
ChatCompletionUserMessageParam(
    content="test"
)

[davidberenstein1957]

Yes, that would be the down side, but querying etc would be easier I think.

[cfahlgren1]

Yeah I think that is fine and gets most of the benefits of having it in a sql store like that. Obviously when being send to HuggingFace we won't need to send image url etc if there are none.

[cfahlgren1]

The DuckDB union could be valuable here, but not sure if it would make anything more complicated

[cfahlgren1]

One the schema changes topic, how do we want to make things backwards compatible or for upgrading existing duckdb instances to work with new version?

One idea is some type of migrations to update columns etc? But don't want to add to much complexity 🤔

[cfahlgren1]

We will have the same problem with SQLite store. It may be better to just store as JSON. It adds a step (CASTING to STRUCT) for querying locally but I think it's a worthy trade off in favor of not introducing a lot of complexity or maintenence