[Bug]: qwen2.5-vl throw 500 Internal Server Error

[xlg-go]

How to resolve the issue where qwen2.5-vl returns a "500 Internal Server Error" when accessed via the OpenAI SDK?

1. env:

V1 LLM engine (v0.8.5.dev530+g0a9bbaa10)
ubuntu 22.04
cuda12.6
torch2.7.0
qwen2.5-vl-32b-instruct

2. my solution：
   pip install qwen-vl-utils

from qwen_vl_utils import process_vision_info

messages = [{"content": [{"type": "image", "image": f"{img_path}", }]}]
image_inputs, _ = process_vision_info(messages)
image_preprocessed = image_inputs[0]
# image_preprocessed to base64str
# via OpenAI SDK

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

Is anything being logged on the server?

[xlg-go]

Is anything being logged on the server?

1. request

{
    "model": "qwen2.5-vl-32b",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "ocr"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAIBAQEBAQIBAQECAgICAgQDAgICAgUEBAMEBgUGBgYFBgYGBwkIBgcJBwYGCAsICQoKCgoKBggLDAsKDAkKCgr/2wBDAQICAgICAgUDAwUKBwYHCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgr/wAARCAAUADgDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD6U1b/AIK3/wDBVj9pX4neOdE/YT/Y18P6r4c8A6/c6VfahrN7IGupLcFtvPrjH416D40/4Le+H/2Wf2eJPDP7a+jxaN8b4NHeW88DaJbh4bR2UmBXckBVZsAkkYB6ivP/AI6fGOw/4Ieft3eMfitrVq+rfBr41WT6vdaFpsu/UdP8RwqWjjEHZZWAG/tnNfMn7f8A+zf8Z/jp+xv4t/4KwfteeEF03xd4y8VaNp3hXwsbL5dA8Ku4VWn95EJz/vUAdj+wL/wVM/4YB/YJ8Y+PvEnww+JPirxZ4i8Rah4jMer+HZrTSrETzkwwrcSSsjqxKj5OcdK/U+0/bc+HPgf9jvw5+1r+0xrmn+A7PVPD1pqOpwXc/mCzkuIiI1Vf4icjA79K/JP/AIKhft1eH/20dJ+E/wDwTY/Yl1OO48Ix3uk2c/iz+zt2n3t+lumxIx3RFJLj0lNecft+eDP+CjV9+0J4S/4J+ftNfGjwJqEHg/SjqvhifxP/AKHo19YQQARtPL/FJG2NsffGKAPtb/gsv/wVH8Sn4PfDDwr+y78Ubjwr4X+LlncX178UoI/3dhZhPLNssgYFLlSS5XPKjr3PPf8ABL34G/BDR/GNr+37+zf/AMFEvGmv/D7wlpuowfFHSfG8krefcY3RXBiMpECbQ7KMHeJfevhr4paT8MvidZ/Bv/gnB4T/AGqLOP4d+EH1XUPEPxJ8UWHkaXJeyuDJNCP7qElUT/lrtA716z/wSh/4X38VfFfjz/gmN+zh4z0XxF8HdUaWXxP8WbXwsDJLaKCpwG4Mk3MUeeMkUAfZP7Pv/Bav4u/tW/t9eGvhB+zV8AptU+EN7qL2l/4kvrTF00SoVGqoe0P+qJPpJRXiv7BXjz4q/wDBL3/gphL/AMEtvhvaaT8U/CWs6lA39vR25TU/DFswm8pJXh4EaxIhKegIooA+zda/4IS/sUeJf2lbv9qH4ga58QfE+vyeIUvrWy1/xUJrGxmIU5hiWJSoz2LGvoLxp+y38LfiV8P7r4T+Ol1LU/DusWN1Z6jpV1flop4SdhQjHTaSKKKAOL0r/glH+xBofiDwfrOifCZbSbwFdyXfhcW92yrZyyDDkDvkVJ8dv+CYv7HX7VcVtq37RHw1Pi250wOthNq90ZGhT+4pxwvtRRQBx/xX/wCCKP8AwTr+M/iy01/x78EftMthFBp9pEmoyLFHbwwZjTZkggEdDxR+0T+y38LP2OP2OfEGn/stxah4FbzYE+0+H70xS4LbM5IIzg+lFFAHTfskf8Eyf2XP2ZNMn8a/DLTtdHibxOsd54k8WajrTz6lqjugJWabAJTkgKAOKKKKAP/Z"
                    }
                }
            ]
        }
    ]
}

2. log

TRACE:    192.168.11.202:61169 - HTTP connection made
TRACE:    192.168.11.202:61169 - ASGI [7] Started scope={'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('172.19.0.2', 8000), 'client': ('192.168.11.202', 61169), 'scheme': 'http', 'root_path': '', 'headers': '<...>', 'state': {}, 'method': 'POST', 'path': '/v1/chat/completions', 'raw_path': b'/v1/chat/completions', 'query_string': b''}
TRACE:    192.168.11.202:61169 - ASGI [7] Receive {'type': 'http.request', 'body': '<1108 bytes>', 'more_body': True}
TRACE:    192.168.11.202:61169 - ASGI [7] Receive {'type': 'http.request', 'body': '<1517 bytes>', 'more_body': False}
WARNING 05-12 03:34:36 [sampling_params.py:343] temperature 1e-06 is less than 0.01, which may cause numerical errors nan or inf in tensors. We have maxed it out to 0.01.
INFO 05-12 03:34:36 [logger.py:39] Received request chatcmpl-c0c0ecdadf854a0f9900314cdf4ecded: prompt: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nocr<|vision_start|><|image_pad|><|vision_end|><|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.01, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=4073, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO 05-12 03:34:36 [async_llm.py:415] Aborted request chatcmpl-c0c0ecdadf854a0f9900314cdf4ecded.
INFO 05-12 03:34:36 [async_llm.py:341] Request chatcmpl-c0c0ecdadf854a0f9900314cdf4ecded failed.
TRACE:    192.168.11.202:61169 - ASGI [7] Send {'type': 'http.response.start', 'status': <HTTPStatus.INTERNAL_SERVER_ERROR: 500>, 'headers': '<...>'}
INFO:     192.168.11.202:61169 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
TRACE:    192.168.11.202:61169 - ASGI [7] Send {'type': 'http.response.body', 'body': '<0 bytes>'}
TRACE:    192.168.11.202:61169 - ASGI [7] Completed
TRACE:    192.168.11.202:61169 - HTTP connection lost

[DarkLight1337]

Ok, perhaps you can try enabling more logging according to https://docs.vllm.ai/en/latest/getting_started/troubleshooting.html

[xlg-go]

Ok, perhaps you can try enabling more logging according to https://docs.vllm.ai/en/latest/getting_started/troubleshooting.html

1. env add VLLM_LOGGING_LEVEL=DEBUG

WARNING 05-13 09:20:19 [sampling_params.py:347] temperature 1e-06 is less than 0.01, which may cause numerical errors nan or inf in tensors. We have maxed it out to 0.01.
INFO 05-13 09:20:19 [logger.py:39] Received request chatcmpl-4c3f7088855a4c0a846482fa3757c17c: prompt: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nocr<|vision_start|><|image_pad|><|vision_end|><|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.01, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=4073, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO 05-13 09:20:19 [async_llm.py:411] Aborted request chatcmpl-4c3f7088855a4c0a846482fa3757c17c.
INFO 05-13 09:20:19 [async_llm.py:337] Request chatcmpl-4c3f7088855a4c0a846482fa3757c17c failed.
INFO:     192.168.11.202:59054 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error

[xlg-go]

Ok, perhaps you can try enabling more logging according to https://docs.vllm.ai/en/latest/getting_started/troubleshooting.html

Using a slow image processor as use_fast is unset and a slow processor was saved with this model. use_fast=True will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with use_fast=False.

#15614

[DarkLight1337]

Can you try using latest main? #18181 should allow the internal server error to be logged at top level

[xlg-go]

Can you try using latest main? #18181 should allow the internal server error to be logged at top level

@DarkLight1337 thank you so much!!!

0. vllm-0.9.1.dev32+g67da5720d-cp312-abi3-linux_x86_64.whl

1. postman:

{
    "object": "error",
    "message": "Failed to apply Qwen2_5_VLProcessor on data={'text': '<|image_pad|>', 'images': [<PIL.Image.Image image mode=RGB size=56x20 at 0x7F3781388EF0>]} with kwargs={}",
    "type": "BadRequestError",
    "param": null,
    "code": 400
}

2. vllm serve:

WARNING 05-19 02:02:06 [sampling_params.py:343] temperature 1e-06 is less than 0.01, which may cause numerical errors nan or inf in tensors. We have maxed it out to 0.01.
INFO 05-19 02:02:06 [logger.py:39] Received request chatcmpl-8f10697a055b4a74997b8db65e8ee954: prompt: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nocr<|vision_start|><|image_pad|><|vision_end|><|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.01, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=4073, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO 05-19 02:02:06 [async_llm.py:334] Request chatcmpl-8f10697a055b4a74997b8db65e8ee954 failed (bad request).
INFO:     192.168.11.202:52471 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request

[DarkLight1337]

It looks like your chat template is applied incorrectly. It should follow the prompt format shown in vllm serve

[xlg-go]

It looks like your chat template is applied incorrectly. It should follow the prompt format shown in vllm serve

chat template like this:

{
    "model": "qwen2.5-vl-32b",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "ocr"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAIBAQEBAQIBAQECAgICAgQDAgICAgUEBAMEBgUGBgYFBgYGBwkIBgcJBwYGCAsICQoKCgoKBggLDAsKDAkKCgr/2wBDAQICAgICAgUDAwUKBwYHCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgr/wAARCAAUADgDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD6U1b/AIK3/wDBVj9pX4neOdE/YT/Y18P6r4c8A6/c6VfahrN7IGupLcFtvPrjH416D40/4Le+H/2Wf2eJPDP7a+jxaN8b4NHeW88DaJbh4bR2UmBXckBVZsAkkYB6ivP/AI6fGOw/4Ieft3eMfitrVq+rfBr41WT6vdaFpsu/UdP8RwqWjjEHZZWAG/tnNfMn7f8A+zf8Z/jp+xv4t/4KwfteeEF03xd4y8VaNp3hXwsbL5dA8Ku4VWn95EJz/vUAdj+wL/wVM/4YB/YJ8Y+PvEnww+JPirxZ4i8Rah4jMer+HZrTSrETzkwwrcSSsjqxKj5OcdK/U+0/bc+HPgf9jvw5+1r+0xrmn+A7PVPD1pqOpwXc/mCzkuIiI1Vf4icjA79K/JP/AIKhft1eH/20dJ+E/wDwTY/Yl1OO48Ix3uk2c/iz+zt2n3t+lumxIx3RFJLj0lNecft+eDP+CjV9+0J4S/4J+ftNfGjwJqEHg/SjqvhifxP/AKHo19YQQARtPL/FJG2NsffGKAPtb/gsv/wVH8Sn4PfDDwr+y78Ubjwr4X+LlncX178UoI/3dhZhPLNssgYFLlSS5XPKjr3PPf8ABL34G/BDR/GNr+37+zf/AMFEvGmv/D7wlpuowfFHSfG8krefcY3RXBiMpECbQ7KMHeJfevhr4paT8MvidZ/Bv/gnB4T/AGqLOP4d+EH1XUPEPxJ8UWHkaXJeyuDJNCP7qElUT/lrtA716z/wSh/4X38VfFfjz/gmN+zh4z0XxF8HdUaWXxP8WbXwsDJLaKCpwG4Mk3MUeeMkUAfZP7Pv/Bav4u/tW/t9eGvhB+zV8AptU+EN7qL2l/4kvrTF00SoVGqoe0P+qJPpJRXiv7BXjz4q/wDBL3/gphL/AMEtvhvaaT8U/CWs6lA39vR25TU/DFswm8pJXh4EaxIhKegIooA+zda/4IS/sUeJf2lbv9qH4ga58QfE+vyeIUvrWy1/xUJrGxmIU5hiWJSoz2LGvoLxp+y38LfiV8P7r4T+Ol1LU/DusWN1Z6jpV1flop4SdhQjHTaSKKKAOL0r/glH+xBofiDwfrOifCZbSbwFdyXfhcW92yrZyyDDkDvkVJ8dv+CYv7HX7VcVtq37RHw1Pi250wOthNq90ZGhT+4pxwvtRRQBx/xX/wCCKP8AwTr+M/iy01/x78EftMthFBp9pEmoyLFHbwwZjTZkggEdDxR+0T+y38LP2OP2OfEGn/stxah4FbzYE+0+H70xS4LbM5IIzg+lFFAHTfskf8Eyf2XP2ZNMn8a/DLTtdHibxOsd54k8WajrTz6lqjugJWabAJTkgKAOKKKKAP/Z"
                    }
                }
            ]
        }
    ]
}

vllm/examples/online_serving/openai_chat_completion_client_for_multimodal.py

Line 97 in 221cfc2

chat_completion_from_base64 = client.chat.completions.create(

[DarkLight1337]

The chat template in vllm serve is correct, but not in postman

[xlg-go]

The chat template in vllm serve is correct, but not in postman

Could you show me how to use Postman?

not in postman:
code:

exception:

[DarkLight1337]

Sorry, I am not familiar with Postman