[Frontend] Support image object in llm.chat

[sfeng33]

Purpose

Fix 17551

Test Plan

Unit test:

pytest tests/entrypoints/test_chat_utils.py

Manual test:
Run a multimodal llm chat with hardcoded image object

python examples/offline_inference/mistral-small.py simple --format hf

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Summary of Changes

Hello @sfeng33, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces the capability to pass image objects directly within the multimodal message content for the llm.chat frontend API. Previously, only image URLs and image embeds were supported. This change addresses issue #17551 and provides a more flexible way to handle image inputs, particularly when the image is already available in memory as an object (like a PIL image).

Highlights

• New Multimodal Part Type: I've added support for a new message content part type, "image", allowing users to pass image objects directly in the llm.chat API.
• Parsing Logic: Implemented the necessary parsing logic in chat_utils.py to handle the new "image" type for both synchronous and asynchronous chat requests, storing the provided image object.
• Test Coverage: Updated existing unit tests in test_chat_utils.py to include examples of using the new "image" type with a PIL image object, ensuring this new functionality is tested.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configureGemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for directly passing image objects (e.g., PIL.Image) in chat messages, extending the existing multimodal capabilities. The changes include adding new parsing logic for these image objects in both synchronous and asynchronous content parsers, updating relevant type definitions, and modifying tests to cover this new functionality. My review primarily focuses on enhancing type specificity for the new image object parameters to improve code clarity, maintainability, and type safety.

[DarkLight1337]

Thanks for working on this, can you fix the pre-commit errors first?

[DarkLight1337]

Can you update the documentation and example files to showcase this way of inputting images?

[DarkLight1337]

Otherwise the implementation looks good to me. cc @Isotr0py @ywang96

[aarnphm]

I prefer to have

Suggested change

	"type": "image",
	"type": "image_pil",

given that we add image_embed

[DarkLight1337]

I think that HF format uses type: "image". Not 100% sure on whether it supports passing object at the same time like this though. cc @ywang96

[Isotr0py]

image_pil isn't a valid object type in HF chat template: https://huggingface.co/docs/transformers/main/en/chat_templating_multimodal?sampling=list+of+image+frames#image-inputs.

But since we will update all image messages to {"type": "image"} in _parse_chat_message_content_part, I think it's OK to use image_pil here.

[DarkLight1337]

Let's go with image_pil then

[ywang96]

image_pil is better IMO!

[ywang96]

Thanks for working on this and carrying #17919 to the finishing line! The code change is clean to me, but please also address the comments from other reviewers.

[ywang96]

image_pil is better IMO!

[sfeng33]

Thank you all for the feedback!

Updates in this revision:

• Replaced the type 'image' with 'image_pil'.
• Added a usage example in mistral-small.py.

I didn’t update the documentation since the current docs don’t seem to cover this area. That said, I’m happy to add something if you have suggestions for a good place to include it.

[DarkLight1337]

Would be great if you could update the documentation on this page! https://docs.vllm.ai/en/latest/features/multimodal_inputs.html

[sfeng33]

Addressed @DarkLight1337's feedback on documentation.