High-level API for multimodality

[remixer-dec]

Is your feature request related to a problem? Please describe.

Current high-level implementation of multimodality is relying on a specific prompt format.

Describe the solution you'd like

Models like Obsidian work with llama.cpp server and have a different format. It would be nice to have a high-level API for multimodality in llama-cpp-python to be able to pass image/images as an argument after initializing Llama() with all the paths to required extra-models, without relying on a pre-defined prompt format such as Llava15ChatHandler.

Describe alternatives you've considered
Alternatively, a custom prompt format class that supports images can be implemented, where prompt string is passed as an argument.

[abetlen]

I'll consider adding the multi-modal args to the Llama class but I'm worried about growing the API surface too much too quickly.

For now, I can definitely implement an Obsidian chat handler and maybe abstract out some of the Llava stuff to make it easier to make this generic in the future.

Also thank you for bringing that model to my attention, definitely have to give it a try now!

[JoshuaFurman]

This seems like an apt place to ask this; How can I supply a local image file to use with multimodal images? Looks like the example assumes an image is being hosted at a url somewhere?

[abetlen]

@JoshuaFurman good question, I'll update the docs to include this. It works the same as the OpenAI gpt-4-vision-preview you can pass the image as a base64 encoded data url:

import base64

def image_to_base64_data_uri(file_path):
    with open(file_path, "rb") as img_file:
        base64_data = base64.b64encode(img_file.read()).decode('utf-8')
        return f"data:image/png;base64,{base64_data}"

# Replace 'file_path.png' with the actual path to your PNG file
file_path = 'file_path.png'
data_uri = image_to_base64_data_uri(file_path)

Then just pass that in place of the http url

[JoshuaFurman]

Oh fantastic thank you! And this can be done without running the server right? I'm looking to add this directly into an application

[abetlen]

Yup, you just need to pass the Llava15ChatHandler to Llama like in the README example so that the class knows how to format the chat with images.

[JoshuaFurman]

Great thanks so much! Happy thanksgiving :)

[XcerontangX]

@JoshuaFurman good question, I'll update the docs to include this. It works the same as the OpenAI gpt-4-vision-preview you can pass the image as a base64 encoded data url:

import base64

def image_to_base64_data_uri(file_path):
    with open(file_path, "rb") as img_file:
        base64_data = base64.b64encode(img_file.read()).decode('utf-8')
        return f"data:image/png;base64,{base64_data}"

# Replace 'file_path.png' with the actual path to your PNG file
file_path = 'file_path.png'
data_uri = image_to_base64_data_uri(file_path)

Then just pass that in place of the http url

Thanks this works

[abetlen]

Closed this by mistake, though will be solved in #1147