Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(multimodal): allow to template placeholders #3728

Merged
merged 1 commit into from
Oct 4, 2024

Conversation

mudler
Copy link
Owner

@mudler mudler commented Oct 4, 2024

Description

This PR allows to customize the placeholders for the images that are sent back to the model.

For context, some models/libraries have different way to express images, videos or audio placeholders. For example, llama.cpp backend expects images within an [img-ID] tag, but other backends/models (e.g. vLLM) use a different notation ( <|image_|>). This changeset allow to specify the notation in the template config section of the YAML file:

For example, to override defaults, now it is possible to set in the model configuration the following:

template:
  video: "<|video_{{.ID}}|> {{.Text}}"
  image: "<|image_{{.ID}}|> {{.Text}}"
  audio: "<|audio_{{.ID}}|> {{.Text}}"

Notes for Reviewers

This PR is related to #3670 #3602 #2318

Signed commits

  • Yes, I signed my commits.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler mudler added the enhancement New feature or request label Oct 4, 2024
Copy link

netlify bot commented Oct 4, 2024

Deploy Preview for localai ready!

Name Link
🔨 Latest commit 03b3ce9
🔍 Latest deploy log https://app.netlify.com/sites/localai/deploys/670006cb5994f80008c5e9ad
😎 Deploy Preview https://deploy-preview-3728--localai.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@mudler mudler changed the title feat(multimodal): allow to template image placeholders feat(multimodal): allow to template placeholders Oct 4, 2024
@mudler mudler merged commit 648ffdf into master Oct 4, 2024
32 of 33 checks passed
@mudler mudler deleted the feat/multimodal_template branch October 4, 2024 16:32
siddimore pushed a commit to siddimore/LocalAI that referenced this pull request Oct 6, 2024
feat(multimodal): allow to template image placeholders

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant