Skip to content

feat: add ChatCompletion endpoint in OpenAI demo server. #330

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Jul 3, 2023
Merged

feat: add ChatCompletion endpoint in OpenAI demo server. #330

merged 3 commits into from
Jul 3, 2023

Conversation

@gesanqiu
Copy link
Contributor

@gesanqiu gesanqiu commented Jul 2, 2023 •
edited
Loading

@gesanqiu gesanqiu mentioned this pull request Jul 2, 2023
Closed
merrymercy
merrymercy suggested changes Jul 2, 2023
View reviewed changes
Copy link
Contributor

@merrymercy merrymercy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See also lm-sys/FastChat#1835

@merrymercy
Copy link
Contributor

merrymercy commented Jul 2, 2023 •
edited
Loading

The vLLM integration for OpenAI API server has been fixed by lm-sys/FastChat#1835.
Could you test it? It should be compatible with (completion, chat-completion) x (streaming, non-streaming)

With vLLM only, you get simplicity and continuous batching.
With FastChat + vllm_worker, you get a distributed multi-model multi-worker controller + continuous batching.

@gesanqiu
Copy link
Contributor Author

gesanqiu commented Jul 2, 2023

The vLLM integration for OpenAI API server has been fixed by lm-sys/FastChat#1835. Could you test it? It should be compatible with (completion, chat-completion) x (streaming, non-streaming)

With vLLM only, you get simplicity and continuous batching. With FastChat + vllm_worker, you get a distributed multi-model multi-worker controller + continuous batching.

I will try it tomorrow, thx.

zhuohan123
zhuohan123 approved these changes Jul 2, 2023
View reviewed changes
Copy link
Member

@zhuohan123 zhuohan123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution! The changes look good to me in general. Left some small comments to add advanced sampling functionality into the ChatCompletion API.

vllm/entrypoints/openai/protocol.py Outdated
object: str = "chat.completion.chunk"
created: int = Field(default_factory=lambda: int(time.time()))
model: str
choices: List[ChatCompletionResponseStreamChoice]
Copy link
Member

@zhuohan123 zhuohan123 Jul 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: new line

@zhuohan123 zhuohan123 merged commit 49b26e2 into vllm-project:main Jul 3, 2023
@gesanqiu gesanqiu deleted the chatcompletion branch July 7, 2023 02:49
hongxiayang pushed a commit to hongxiayang/vllm that referenced this pull request Feb 13, 2024
@gesanqiu
feat: add ChatCompletion endpoint in OpenAI demo server. (vllm-projec…
7ebf67b 
…t#330)
yukavio pushed a commit to yukavio/vllm that referenced this pull request Jul 3, 2024
@dhuangnm
bump version to 0.5.1 (vllm-project#330)
250f6d8 
Co-authored-by: dhuangnm <dhuang@MacBook-Pro-2.local>
jikunshang pushed a commit to jikunshang/vllm that referenced this pull request Sep 30, 2024
@kzawora-intel
Fix doc build warnings (vllm-project#330)
41217cf 
This PR fixes all the little warnings gaudi-installation.rst introduces
during documentation build ("WARNING: Title underline too short." etc.)
billishyahao pushed a commit to billishyahao/vllm that referenced this pull request Dec 31, 2024
@gshtras
Merge pull request vllm-project#330 from ROCm/upstream_merge_24_12_16
1f0f4c6 
Upstream merge 24 12 16
amy-why-3459 pushed a commit to amy-why-3459/vllm that referenced this pull request Sep 15, 2025
@shen-shanshan
[Doc] Fix bugs of installation doc and format tool (vllm-project#330)
441a62e 
### What this PR does / why we need it?
Fix bugs of installation doc and format tool.

### Does this PR introduce _any_ user-facing change?
no.

### How was this patch tested?
no.

Signed-off-by: shen-shanshan <467638484@qq.com>
jikunshang pushed a commit to jikunshang/vllm that referenced this pull request Sep 16, 2025
@yma11 @jikunshang
enable cutlass chunked-prefill (vllm-project#330)
fae4996 
* enable cutlass chunked-prefill

Signed-off-by: Yan Ma <yan.ma@intel.com>

* add required pkg for xpu-kernels compilation

Signed-off-by: Yan Ma <yan.ma@intel.com>

---------

Signed-off-by: Yan Ma <yan.ma@intel.com>

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
yma11 added a commit to yma11/vllm that referenced this pull request Sep 25, 2025
@yma11 @jikunshang
enable cutlass chunked-prefill (vllm-project#330)
82c8a17 
* enable cutlass chunked-prefill

Signed-off-by: Yan Ma <yan.ma@intel.com>

* add required pkg for xpu-kernels compilation

Signed-off-by: Yan Ma <yan.ma@intel.com>

---------

Signed-off-by: Yan Ma <yan.ma@intel.com>

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
yma11 added a commit to yma11/vllm that referenced this pull request Oct 10, 2025
@yma11 @jikunshang
enable cutlass chunked-prefill (vllm-project#330)
fe44f8c 
* enable cutlass chunked-prefill

Signed-off-by: Yan Ma <yan.ma@intel.com>

* add required pkg for xpu-kernels compilation

Signed-off-by: Yan Ma <yan.ma@intel.com>

---------

Signed-off-by: Yan Ma <yan.ma@intel.com>

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
jikunshang pushed a commit to jikunshang/vllm that referenced this pull request Oct 11, 2025
@yma11 @jikunshang
enable cutlass chunked-prefill (vllm-project#330)
8cacd1f 
* enable cutlass chunked-prefill

Signed-off-by: Yan Ma <yan.ma@intel.com>

* add required pkg for xpu-kernels compilation

Signed-off-by: Yan Ma <yan.ma@intel.com>

---------

Signed-off-by: Yan Ma <yan.ma@intel.com>

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
jikunshang pushed a commit to jikunshang/vllm that referenced this pull request Oct 27, 2025
@yma11 @jikunshang
enable cutlass chunked-prefill (vllm-project#330)
12da596 
* enable cutlass chunked-prefill

Signed-off-by: Yan Ma <yan.ma@intel.com>

* add required pkg for xpu-kernels compilation

Signed-off-by: Yan Ma <yan.ma@intel.com>

---------

Signed-off-by: Yan Ma <yan.ma@intel.com>

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
yma11 added a commit to yma11/vllm that referenced this pull request Nov 5, 2025
@1pikachu @jikunshang
@yma11 @wendyliu235
CI for vLLM with vllm-xpu-kernel (vllm-project#372)
302d078 
* layernorm use vllm_xpu_kernels

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* [ww34] switch silu_and_mul, reshape_and_cache_flash, rope to xpu kernel

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* update activation kernels

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* try remove ipex

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* switch to xpu kernel for w8a16 gemm (vllm-project#323)

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* enable cutlass chunked-prefill (vllm-project#330)

* enable cutlass chunked-prefill

Signed-off-by: Yan Ma <yan.ma@intel.com>

* add required pkg for xpu-kernels compilation

Signed-off-by: Yan Ma <yan.ma@intel.com>

---------

Signed-off-by: Yan Ma <yan.ma@intel.com>

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* enable topk/grouped_gemm based on llama4 (vllm-project#354)

* enable topk/grouped_gemm based on llama4

Signed-off-by: Yan Ma <yan.ma@intel.com>

* address comments

Signed-off-by: Yan Ma <yan.ma@intel.com>

---------

Signed-off-by: Yan Ma <yan.ma@intel.com>

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* enable CI

* replace lora kernels (vllm-project#347)

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* remove ipex (vllm-project#370)

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* update QA CI branch

* update QA CI yaml

* update QA CI yaml

* update QA CI yaml

* update QA CI yaml

* update QA CI yaml

* update QA CI yaml

* fix conflict

---------

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: Yan Ma <yan.ma@intel.com>
Co-authored-by: Liu, Wenjun <wenjun.liu@intel.com>
yma11 added a commit to yma11/vllm that referenced this pull request Nov 10, 2025
@yma11 @jikunshang
enable cutlass chunked-prefill (vllm-project#330)
125b400 
* enable cutlass chunked-prefill

Signed-off-by: Yan Ma <yan.ma@intel.com>

* add required pkg for xpu-kernels compilation

Signed-off-by: Yan Ma <yan.ma@intel.com>

---------

Signed-off-by: Yan Ma <yan.ma@intel.com>

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
yma11 added a commit to yma11/vllm that referenced this pull request Nov 10, 2025
@1pikachu @jikunshang
@yma11 @wendyliu235
CI for vLLM with vllm-xpu-kernel (vllm-project#372)
93c0b8d 
* layernorm use vllm_xpu_kernels

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* [ww34] switch silu_and_mul, reshape_and_cache_flash, rope to xpu kernel

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* update activation kernels

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* try remove ipex

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* switch to xpu kernel for w8a16 gemm (vllm-project#323)

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* enable cutlass chunked-prefill (vllm-project#330)

* enable cutlass chunked-prefill

Signed-off-by: Yan Ma <yan.ma@intel.com>

* add required pkg for xpu-kernels compilation

Signed-off-by: Yan Ma <yan.ma@intel.com>

---------

Signed-off-by: Yan Ma <yan.ma@intel.com>

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* enable topk/grouped_gemm based on llama4 (vllm-project#354)

* enable topk/grouped_gemm based on llama4

Signed-off-by: Yan Ma <yan.ma@intel.com>

* address comments

Signed-off-by: Yan Ma <yan.ma@intel.com>

---------

Signed-off-by: Yan Ma <yan.ma@intel.com>

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* enable CI

* replace lora kernels (vllm-project#347)

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* remove ipex (vllm-project#370)

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* update QA CI branch

* update QA CI yaml

* update QA CI yaml

* update QA CI yaml

* update QA CI yaml

* update QA CI yaml

* update QA CI yaml

* fix conflict

---------

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: Yan Ma <yan.ma@intel.com>
Co-authored-by: Liu, Wenjun <wenjun.liu@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhuohan123 zhuohan123 zhuohan123 approved these changes

+1 more reviewer

@merrymercy merrymercy merrymercy requested changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@gesanqiu @merrymercy @zhuohan123