[Doc] V1 user guide #13991

JenZhao · 2025-02-27T20:42:02Z

This PR adds the v1 User Guide as a living document, to be updated and expanded over time.

github-actions · 2025-02-27T20:42:13Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

docs/source/getting_started/v1_user_guide.md

robertgshaw2-redhat

Looking good! can you finish this off tomorrow? I want to include a link to this page in the logs

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

docs/source/getting_started/v1_user_guide.md

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

ywang96

Left some comments - PTAL

docs/source/getting_started/v1_user_guide.md

Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

Signed-off-by: Roger Wang <ywang@roblox.com>

WoosukKwon · 2025-03-04T10:31:00Z

Thanks for the PR! I will take a look tmr (Tue).

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

LiuXiaoxuanPKU

Thanks for the guide. It's great with lots of information.

Just some concerns:

On the high level, I feel it's a bit too negative? We talk a lot about features we don't support or need to be optimized. I feel we should highlight some optimized features, such as chunked prefill and prefix caching.
I feel we need to mention the scheduler (scheduling policy) somewhere, currently we batch prefill tokens and decoding tokens in the same batch, we don't prioritize prefill/decode.

docs/source/getting_started/v1_user_guide.md

LiuXiaoxuanPKU · 2025-03-05T23:08:53Z

docs/source/getting_started/v1_user_guide.md

+(e.g., see [PR #13096](https://github.com/vllm-project/vllm/pull/13096)).
+
+- **Spec Decode**: Currently, only ngram-based spec decode is supported in V1. There
+  will be follow-up work to support other types of spec decode (e.g., see [PR #13933](https://github.com/vllm-project/vllm/pull/13933)).


Suggested change

will be follow-up work to support other types of spec decode (e.g., see [PR #13933](https://github.com/vllm-project/vllm/pull/13933)).

will be follow-up work to support other types of spec decode (e.g., see [PR #13933](https://github.com/vllm-project/vllm/pull/13933)). We will prioritize the support for Eagle, MTP compared to draft model based spec decode.

Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>

docs/source/getting_started/v1_user_guide.md

Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

simon-mo

Thank you for working on this!!

docs/source/getting_started/v1_user_guide.md

simon-mo · 2025-03-13T17:36:36Z

Oh we need a small section on hardware support saying NVIDIA is natively supported, AMD and TPU are work in progress.

Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>

simon-mo

This is a great first verison!

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com> Signed-off-by: Roger Wang <ywang@roblox.com> Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com> Co-authored-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com> Co-authored-by: Jennifer Zhao <JenZhao@users.noreply.github.com> Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com> Co-authored-by: Roger Wang <ywang@roblox.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Signed-off-by: Louis Ulmer <ulmerlouis@gmail.com>

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com> Signed-off-by: Roger Wang <ywang@roblox.com> Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com> Co-authored-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com> Co-authored-by: Jennifer Zhao <JenZhao@users.noreply.github.com> Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com> Co-authored-by: Roger Wang <ywang@roblox.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com> Signed-off-by: Roger Wang <ywang@roblox.com> Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com> Co-authored-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com> Co-authored-by: Jennifer Zhao <JenZhao@users.noreply.github.com> Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com> Co-authored-by: Roger Wang <ywang@roblox.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>

mergify bot added the documentation Improvements or additions to documentation label Feb 27, 2025

JenZhao and others added 10 commits February 27, 2025 13:33

add vLLM V1 User Guide Template

904690e

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

wip update v1 user guide

c294f75

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

update logprob desc

2283371

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

update logprobs desc

1486cb8

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

update

f9bb563

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

fix

5002550

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

update

b395576

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

update

658957f

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

Update v1_user_guide.md

eacf90a

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

Merge branch 'vllm-project:main' into v1_guide

bf852c6

robertgshaw2-redhat reviewed Mar 2, 2025

View reviewed changes

docs/source/getting_started/v1_user_guide.md Outdated Show resolved Hide resolved

robertgshaw2-redhat reviewed Mar 2, 2025

View reviewed changes

docs/source/getting_started/v1_user_guide.md Outdated Show resolved Hide resolved

robertgshaw2-redhat reviewed Mar 2, 2025

View reviewed changes

docs/source/getting_started/v1_user_guide.md Outdated Show resolved Hide resolved

robertgshaw2-redhat reviewed Mar 2, 2025

View reviewed changes

docs/source/getting_started/v1_user_guide.md Show resolved Hide resolved

robertgshaw2-redhat requested changes Mar 2, 2025

View reviewed changes

update unsupported

560b267

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

shen-shanshan reviewed Mar 2, 2025

View reviewed changes

docs/source/getting_started/v1_user_guide.md Outdated Show resolved Hide resolved

address comments

33d759f

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

ywang96 reviewed Mar 2, 2025

View reviewed changes

JenZhao and others added 8 commits March 1, 2025 21:39

Apply suggestions from code review

2c28989

Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>

address comments

4e978c2

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

remove merged pr

32887cd

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

fix

fcffa3c

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

link pr #13361

047f46d

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

link pr #13997

d858c6a

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

update

54d51cf

Signed-off-by: Roger Wang <ywang@roblox.com>

pre-commit

b934056

Signed-off-by: Roger Wang <ywang@roblox.com>

JenZhao marked this pull request as ready for review March 2, 2025 20:50

JenZhao added 3 commits March 4, 2025 20:47

update

79549c9

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

update

aa7bed4

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

update

f80562b

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com>

LiuXiaoxuanPKU reviewed Mar 5, 2025

View reviewed changes

simon-mo added this to the v0.8.0 milestone Mar 12, 2025

JenZhao added 5 commits March 12, 2025 18:41

Merge branch 'vllm-project:main' into v1_guide

781a224

[Doc] Update v1 guide (#4)

6021f71

Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>

update

b4691c7

Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>

remove notes

7c48862

Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>

fix

e4c5e81

Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>

DarkLight1337 reviewed Mar 13, 2025

View reviewed changes

docs/source/getting_started/v1_user_guide.md Outdated Show resolved Hide resolved

DarkLight1337 reviewed Mar 13, 2025

View reviewed changes

docs/source/getting_started/v1_user_guide.md Outdated Show resolved Hide resolved

Apply suggestions from code review

4090ac9

Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

simon-mo reviewed Mar 13, 2025

View reviewed changes

docs/source/getting_started/v1_user_guide.md Show resolved Hide resolved

docs/source/getting_started/v1_user_guide.md Show resolved Hide resolved

docs/source/getting_started/v1_user_guide.md Outdated Show resolved Hide resolved

JenZhao added 6 commits March 13, 2025 18:51

update structured output

b1dad35

Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>

sort table, add benchmark placeholder

c5cc253

Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>

minor

5b1374d

Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>

add hardware support

35b7af7

Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>

address comments

16d9ce5

Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>

update

dedc925

Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>

simon-mo approved these changes Mar 15, 2025

View reviewed changes

simon-mo merged commit aaacf17 into vllm-project:main Mar 15, 2025
17 of 18 checks passed

ckhordiasma mentioned this pull request Apr 17, 2025

[do not merge] pr test for nm changes into 2.20 red-hat-data-services/vllm#107

Closed

	will be follow-up work to support other types of spec decode (e.g., see [PR #13933](https://github.com/vllm-project/vllm/pull/13933)).
	will be follow-up work to support other types of spec decode (e.g., see [PR #13933](https://github.com/vllm-project/vllm/pull/13933)). We will prioritize the support for Eagle, MTP compared to draft model based spec decode.

Uh oh!

[Doc] V1 user guide #13991

[Doc] V1 user guide #13991

Uh oh!

Conversation

JenZhao commented Feb 27, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 27, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robertgshaw2-redhat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ywang96 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

WoosukKwon commented Mar 4, 2025

Uh oh!

LiuXiaoxuanPKU left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LiuXiaoxuanPKU Mar 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

simon-mo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

simon-mo commented Mar 13, 2025

Uh oh!

simon-mo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

JenZhao commented Feb 27, 2025 •

edited by github-actions bot

Loading