[v1] Introduce KVCacheBlocks as interface between Scheduler and KVCacheManager #17479

heheda12345 · 2025-04-30T14:48:58Z

In the future hybrid allocator, the KVCacheManager output would be list[list[list[KVCacheBlocks]], which is much more complex than the current list[KVCacheBlocks].

To hide the complexity, this pr introduces KVCacheBlocks to save the KVCacheManager output so that scheduler do not need to parse the internal structure of KVCacheManager output.

Splitted from #16101

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

github-actions · 2025-04-30T14:49:06Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

mergify · 2025-04-30T19:38:04Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @heheda12345.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

WoosukKwon

LGTM! Left minor comments.

WoosukKwon · 2025-05-06T05:16:48Z

vllm/v1/core/kv_cache_manager.py

        request: Request,
        num_tokens: int,
-        new_computed_blocks: Optional[list[KVCacheBlock]] = None,
+        new_computed_blocks_obj: Optional[KVCacheBlocks] = None,


Q: Why rename this?

Becase it is with type KVCacheBlocks and we need to unwrap it to list[KVCacheBlock] before using. Prefer to call the object with type list[KVCacheBlock] "new_computed_blocks" so I rename this object to "new_computed_blocks_obj"

But when would we ever use list[KVCacheBlock] directly? The _obj suffix feels a bit awkward to me.

list[KVCacheBlock] is used inside allocate_slots

I thought we kinda decided to blur the distinction when introducing the KVCacheBlocks class. I don't understand why this particular variable name is a concern for you.

Otherwise I need to change new_computed_blocks in this function to another name.

I personally prefer that, because I really feel new_computed_blocks_obj awkward. But I'm fine if you want to stick to this.

OK. I can change it later.

vllm/v1/core/kv_cache_manager.py

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

heheda12345 · 2025-05-06T06:47:06Z

@WoosukKwon I've updated this PR. Can you take another look?

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

eryugey · 2025-05-06T14:01:16Z

vllm/v1/core/kv_cache_manager.py

            A tuple containing:
                - A list of blocks that are computed for the request.
                - The number of computed tokens.
        """


Update the doc string as well about the return type?

* [Model] Add GraniteMoeHybrid 4.0 model (vllm-project#17497) Signed-off-by: Thomas Ortner <boh@zurich.ibm.com> Signed-off-by: Stanislaw Wozniak <stw@zurich.ibm.com> Co-authored-by: Thomas Ortner <boh@zurich.ibm.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Tyler Michael Smith <tysmith@redhat.com> * [easy] Fix logspam on PiecewiseBackend errors (vllm-project#17138) Signed-off-by: rzou <zou3519@gmail.com> * [Bugfix] Fixed prompt length for random dataset (vllm-project#17408) Signed-off-by: Mikhail Podvitskii <podvitskiymichael@gmail.com> * [Doc] Update notes for H2O-VL and Gemma3 (vllm-project#17219) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> * [Misc] Fix ScalarType float4 naming (vllm-project#17690) Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> * Fix `dockerfilegraph` pre-commit hook (vllm-project#17698) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [Bugfix] Fix triton import with local TritonPlaceholder (vllm-project#17446) Signed-off-by: Mengqing Cao <cmq0113@163.com> * [V1] Enable TPU V1 backend by default (vllm-project#17673) Signed-off-by: mgoin <mgoin64@gmail.com> * [V1][PP] Support PP for MultiprocExecutor (vllm-project#14219) Signed-off-by: jiang1.li <jiang1.li@intel.com> Signed-off-by: jiang.li <jiang1.li@intel.com> * [v1] AttentionMetadata for each layer (vllm-project#17394) Signed-off-by: Chen Zhang <zhangch99@outlook.com> * [Feat] Add deprecated=True to CLI args (vllm-project#17426) Signed-off-by: Aaron Pham <contact@aarnphm.xyz> * [Docs] Use gh-file to add links to tool_calling.md (vllm-project#17709) Signed-off-by: windsonsea <haifeng.yao@daocloud.io> * [v1] Introduce KVCacheBlocks as interface between Scheduler and KVCacheManager (vllm-project#17479) Signed-off-by: Chen Zhang <zhangch99@outlook.com> * [doc] Add RAG Integration example (vllm-project#17692) Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com> * [Bugfix] Fix modality limits in vision language example (vllm-project#17721) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> * Make right sidebar more readable in "Supported Models" (vllm-project#17723) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [TPU] Increase block size and reset block shapes (vllm-project#16458) * [Misc] Add Next Edit Prediction (NEP) datasets support in `benchmark_serving.py` (vllm-project#16839) Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal> Signed-off-by: dtransposed <> Co-authored-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal> * [Bugfix] Fix for the condition to accept empty encoder inputs for mllama (vllm-project#17732) Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com> * [Kernel] Unified Triton kernel that doesn't distinguish between prefill + decode (vllm-project#16828) Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com> Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> --------- Signed-off-by: Thomas Ortner <boh@zurich.ibm.com> Signed-off-by: Stanislaw Wozniak <stw@zurich.ibm.com> Signed-off-by: rzou <zou3519@gmail.com> Signed-off-by: Mikhail Podvitskii <podvitskiymichael@gmail.com> Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Mengqing Cao <cmq0113@163.com> Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: jiang1.li <jiang1.li@intel.com> Signed-off-by: jiang.li <jiang1.li@intel.com> Signed-off-by: Chen Zhang <zhangch99@outlook.com> Signed-off-by: Aaron Pham <contact@aarnphm.xyz> Signed-off-by: windsonsea <haifeng.yao@daocloud.io> Signed-off-by: reidliu41 <reid201711@gmail.com> Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal> Signed-off-by: dtransposed <> Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com> Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com> Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Co-authored-by: Stan Wozniak <77159600+s3woz@users.noreply.github.com> Co-authored-by: Thomas Ortner <boh@zurich.ibm.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Tyler Michael Smith <tysmith@redhat.com> Co-authored-by: Richard Zou <zou3519@users.noreply.github.com> Co-authored-by: Mikhail Podvitskii <podvitskiymichael@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Mengqing Cao <cmq0113@163.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: Li, Jiang <jiang1.li@intel.com> Co-authored-by: Chen Zhang <zhangch99@outlook.com> Co-authored-by: Aaron Pham <contact@aarnphm.xyz> Co-authored-by: Michael Yao <haifeng.yao@daocloud.io> Co-authored-by: Reid <61492567+reidliu41@users.noreply.github.com> Co-authored-by: reidliu41 <reid201711@gmail.com> Co-authored-by: Jevin Jiang <jevin0change@gmail.com> Co-authored-by: d.transposed <damian.bogunowicz@gmail.com> Co-authored-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal> Co-authored-by: Gregory Shtrasberg <156009573+gshtras@users.noreply.github.com> Co-authored-by: Thomas Parnell <tpa@zurich.ibm.com> Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

…heManager (vllm-project#17479) Signed-off-by: Chen Zhang <zhangch99@outlook.com> Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>

…heManager (vllm-project#17479) Signed-off-by: Chen Zhang <zhangch99@outlook.com>

…heManager (vllm-project#17479) Signed-off-by: Chen Zhang <zhangch99@outlook.com> Signed-off-by: Yuqi Zhang <yuqizhang@google.com>

update test

e484ba9

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

heheda12345 requested review from WoosukKwon, alexm-redhat, comaniac, njhill, robertgshaw2-redhat and ywang96 as code owners April 30, 2025 14:48

mergify bot added the v1 label Apr 30, 2025

mergify bot added the needs-rebase label Apr 30, 2025

Merge branch 'main' of github.com:vllm-project/vllm into kv_cache_blocks

579c07a

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

mergify bot removed the needs-rebase label May 1, 2025

WoosukKwon approved these changes May 6, 2025

View reviewed changes

heheda12345 added 2 commits May 5, 2025 23:41

cls

93b985c

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

rename

4f1c40a

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

Merge branch 'main' of github.com:vllm-project/vllm into kv_cache_blocks

986f19c

WoosukKwon added the ready ONLY add when PR is ready to merge/full CI is needed label May 6, 2025

fix

7789dac

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

eryugey suggested changes May 6, 2025

View reviewed changes

WoosukKwon merged commit aabcd2c into vllm-project:main May 6, 2025
49 checks passed

mawong-amd pushed a commit to ROCm/vllm that referenced this pull request May 14, 2025

[v1] Introduce KVCacheBlocks as interface between Scheduler and KVCac…

909425e

…heManager (vllm-project#17479) Signed-off-by: Chen Zhang <zhangch99@outlook.com>

This was referenced May 14, 2025

[Bug]: V0 Scheduler is incapable of the newest KVCacheManager interface in vllm main branch code vllm-project/vllm-ascend#861

Closed

[WIP] Fixed: v0 style Scheduler broken issue in vllm main branch. vllm-project/vllm-ascend#862

Closed

Uh oh!

[v1] Introduce KVCacheBlocks as interface between Scheduler and KVCacheManager #17479

[v1] Introduce KVCacheBlocks as interface between Scheduler and KVCacheManager #17479

Uh oh!

Conversation

heheda12345 commented Apr 30, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 30, 2025

Uh oh!

mergify bot commented Apr 30, 2025

Uh oh!

WoosukKwon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WoosukKwon May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

heheda12345 commented May 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

heheda12345 commented Apr 30, 2025 •

edited by github-actions bot

Loading

WoosukKwon May 6, 2025 •

edited

Loading