feat: support guided decoding for vllm async engine

[wxiwnd]

Support Guided Decoding for vllm async engine
waiting for vllm release, a version bump is needed.

#1562
vllm-project/vllm#8252

[qinxuye]

Which version is required?

[wxiwnd]

Which version is required?

latest version after 0.6.2, waiting for vllm to release new version

[qinxuye]

vllm has release v0.6.3, is this PR ready to work?

[wxiwnd]

I will do the test.

…

________________________________ 寄件者: Xuye Qin ***@***.***> 寄件日期: 星期四, 10月 17, 2024 5:38:13 下午 收件者: xorbitsai/inference ***@***.***> 副本: wxiwnd ***@***.***>; Author ***@***.***> 主旨: Re: [xorbitsai/inference] feat: support guided decoding for vllm async engine (PR #2391) vllm has release v0.6.3, is this PR ready to work? — Reply to this email directly, view it on GitHub<#2391 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJSDNXWKTHLIQE35VTQLT23Z36AQDAVCNFSM6AAAAABPJCVJQOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJZGA2DOMZWGE>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[wxiwnd]

vllm has release v0.6.3, is this PR ready to work?

Works on my machine now
Also This PR implemented response_format call, like:

curl --location --request POST 'http://ip:9997/v1/chat/completions' \
--header 'User-Agent: Apifox/1.0.0 (https://apifox.com)' \
--header 'Content-Type: application/json' \
--data '{
    "model": "qwen1.5-32b-chat-int4",
    "messages": [
        {
            "role": "user",
            "content": "“give me a recipe in json format"
        }
    ],
    "temperature": 0,
    "max_tokens": 1000,
    "stream": true,
    "response_format": {"type": "json_object"}
}'

[qinxuye]

vllm has release v0.6.3, is this PR ready to work?

Works on my machine now Also This PR implemented response_format call, like:

curl --location --request POST 'http://ip:9997/v1/chat/completions' \
--header 'User-Agent: Apifox/1.0.0 (https://apifox.com)' \
--header 'Content-Type: application/json' \
--data '{
    "model": "qwen1.5-32b-chat-int4",
    "messages": [
        {
            "role": "user",
            "content": "“give me a recipe in json format"
        }
    ],
    "temperature": 0,
    "max_tokens": 1000,
    "stream": true,
    "response_format": {"type": "json_object"}
}'

Can you confirm there is no exception if the vllm is an old version?

[wxiwnd]

vllm has release v0.6.3, is this PR ready to work?

Works on my machine now Also This PR implemented response_format call, like:

curl --location --request POST 'http://ip:9997/v1/chat/completions' \
--header 'User-Agent: Apifox/1.0.0 (https://apifox.com)' \
--header 'Content-Type: application/json' \
--data '{
    "model": "qwen1.5-32b-chat-int4",
    "messages": [
        {
            "role": "user",
            "content": "“give me a recipe in json format"
        }
    ],
    "temperature": 0,
    "max_tokens": 1000,
    "stream": true,
    "response_format": {"type": "json_object"}
}'

Can you confirm there is no exception if the vllm is an old version?

It now works properly even if vllm version < 0.6.3
All the guided encoding parameters will be ignored if vllm version is under 0.6.3

[qinxuye]

Is there any solution to this?

[wxiwnd]

It appears to be the same issue as Xinference Issue #2032, and I have not yet found a solution.

[wxiwnd]

I have developed a parameter parser for the RESTful API component to ensure that the functionality remains intact. Therefore, these parts can be safely ignored, even though parsing requests in the RESTful API part is somewhat "dirty".

[qinxuye]

IIRC, we solved by copying the openai pydantic model into xinference, refer to https://github.com/xorbitsai/inference/pull/2231/files

[wxiwnd]

While the error is Field name "schema" shadows a BaseModel attribute; use a different field name with "alias='schema'".
I found https://github.com/openai/openai-python/blob/main/src/openai/types/shared_params/response_format_json_schema.py#L11 maybe the source of conflict. Because another declaration of json_schema https://github.com/openai/openai-python/blob/main/src/openai/types/shared/response_format_json_schema.py#L27
use schema_ instead of schema.

I will try to fix this by define ResponseFormat myself.