[Frontend] Multi-Modality Support for Loading Local Image Files

[chaunceyjiang]

FIX #8730
FIX #9742

Multi-Modality Support for Loading Local image Files

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[DarkLight1337]

For security reasons, let's require a CLI flag to enable loading local image files.

[chaunceyjiang]

For security reasons, let's require a CLI flag to enable loading local image files.

@DarkLight1337 What do you reckon would be a good name for a CLI flag?

How about --trust-local-files ?
Should this flag be added to the vllm serve command?

[DarkLight1337]

For security reasons, let's require a CLI flag to enable loading local image files.

@DarkLight1337 What do you reckon would be a good name for a CLI flag?

How about --trust-local-files ? Should this flag be added to the vllm serve command?

I think it is too easily confused with --trust-remote-code from HF.

Perhaps --trust-remote-access? Allowing API requests to read local image files basically does that.

[cedonley]

I would suggest that the flag would instead be the allowed path vs. allowing full filesystem access.

For example, --allow-local-media-path=/some/path

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. @chaunceyjiang please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[chaunceyjiang]

Test Code:

from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    # defaults to os.environ.get("OPENAI_API_KEY")
    api_key=openai_api_key,
    base_url=openai_api_base,
)

models = client.models.list()

model = models.data[0].id




# Single-image input inference
def run_single_image() -> None:

    # Use image url in the payload
    # "https://upload.wikimedia.org/wikipedia/commons/d/da/2015_Kaczka_krzy%C5%BCowka_w_wodzie_%28samiec%29.jpg"
    image_url = "file:///root/jxy/vllm-source/vllm/duck.jpg"
    chat_completion_from_url = client.chat.completions.create(
        messages=[{
            "role":
            "user",
            "content": [
                {
                    "type": "text",
                    "text": "What's in this image?"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": image_url
                    },
                },
            ],
        }],
        model=model,
        max_completion_tokens=64,
    )

    result = chat_completion_from_url.choices[0].message.content
    print("Chat completion output from image url:", result)


def main() -> None:
   run_single_image()


if __name__ == "__main__":
    main()

Test 1:

# vllm serve microsoft/Phi-3.5-vision-instruct --task generate \
 --trust-remote-code --max-model-len 4096 --limit-mm-per-prompt image=2 \
 --dtype=float16 --enforce-eager \
 --allowed-local-media-path=/root/jxy/vllm-source/vllm


# python run_single_image.py 
Chat completion output from image url:  The image shows a single mallard duck swimming on a body of water. The duck has a distinctive green head, yellow bill, white neck ring, and brown chest. The water appears calm, creating a clear reflection of the duck.

# vllm serve microsoft/Phi-3.5-vision-instruct --task generate \
    --trust-remote-code --max-model-len 4096 --limit-mm-per-prompt image=2 \
--dtype=float16 --enforce-eager



# python run_single_image.py                                                                   2 ↵
..
openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': "Invalid 'image_url': Cannot load local files without '--allowed-local-media-path'.", 'type': 'BadRequestError', 'param': None, 'code': 400}

[chaunceyjiang]

/cc @DarkLight1337 PTAL.

[DarkLight1337]

The implementation looks good, can you add some tests to verify this?

[DarkLight1337]

Thanks for implementing this!