[v1][KVCacheManager] pass num_new_computed_tokens to kv cache manager #18001
Conversation
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
heheda12345
requested review from
WoosukKwon,
alexm-redhat,
comaniac,
njhill,
robertgshaw2-redhat and
ywang96
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
|
This pull request has merge conflicts that must be resolved before it can be |
…mputed_tokens Signed-off-by: Chen Zhang <zhangch99@outlook.com>
|
The failing test looks related to KV cache |
|
@WoosukKwon Can you double-check the fix for NIXL compatibility? Thanks! |
vllm-project#18001 changed the behaviour subtly and broke some multi-connector cases. This change ensures we don't call the connector get_num_new_matched_tokens method a second time for a given request after an async load has completed. Signed-off-by: Nick Hill <nhill@redhat.com>
vllm-project#18001 changed the behaviour subtly and broke some multi-connector cases. This change ensures we don't call the connector get_num_new_matched_tokens method a second time for a given request after an async load has completed. Signed-off-by: Nick Hill <nhill@redhat.com>
* [BugFix] Fix handling of num_computed_tokens with connector vllm-project#18001 changed the behaviour subtly and broke some multi-connector cases. This change ensures we don't call the connector get_num_new_matched_tokens method a second time for a given request after an async load has completed. Signed-off-by: Nick Hill <nhill@redhat.com> * fix linting Signed-off-by: Nick Hill <nhill@redhat.com> * handle full cache hit on P/D decode worker case Signed-off-by: Nick Hill <nhill@redhat.com> * fix comment wording Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com> --------- Signed-off-by: Nick Hill <nhill@redhat.com> Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com>
…vllm-project#18001) Signed-off-by: Chen Zhang <zhangch99@outlook.com> Signed-off-by: Yuqi Zhang <yuqizhang@google.com>
4 participants
For hybrid allocator, the logic to get num_computed_tokens from a KVCacheBlocks will be complex. To avoid computing that value in allocate_slots, we pass it from the return value of get_computed_blocks to the argument of allocate_slots