Automatically configure KV cache size #6

WoosukKwon · 2023-03-03T10:05:40Z

This PR adds OPT memory analyzer to the system, and uses it to automatically determine the KV cache size.

Tested models:

OPT-125M
OPT-350M
OPT-1.3B
OPT-2.7B
OPT-6.7B
OPT-13B

Tested GPUs:

A100

* finish changing scheduler * finish merge * fix model * Fix (vllm-project#5) * fix problems * fix * delete unused params * remove redundant comments --------- Co-authored-by: Xiangyu Tian <109123695+xiangyuT@users.noreply.github.com>

…ect#6)

Add missing Python requirements

Co-authored-by: Mor Zusman <morz@ai21.com>

[CI/Build] Dockerfile.ubi : Remove test stage

FP8 on A100 for PHIMOE

Signed-off-by: remi <remi@mistral.ai>

commit 406d6bf Author: Rémi Delacourt <54138269+Flechman@users.noreply.github.com> Date: Fri Apr 11 00:47:40 2025 +0200 Add MLA support for v1 disagg connector (#6) Signed-off-by: remi <remi@mistral.ai> commit 1d8415d Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Thu Apr 10 21:59:54 2025 +0000 rename Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 9c4159c Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Thu Apr 10 21:41:20 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 54e1491 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Thu Apr 10 20:31:35 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 8e1eadc Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Thu Apr 10 20:26:37 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 05349a5 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 22:10:50 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 7f57f3c Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 17:13:31 2025 +0000 update lifecycle Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 7c31e29 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 17:03:55 2025 +0000 nits Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 74af233 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 16:44:01 2025 +0000 done with nits Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit e64f745 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 16:28:51 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 40e5d81 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 16:25:04 2025 +0000 refactor Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 25c9592 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 16:20:41 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit fc58dd5 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 16:13:39 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 20decdf Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 16:06:15 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 5145566 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:52:03 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 62e1421 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:47:40 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 689379e Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:39:40 2025 +0000 updaed Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit b1310fd Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:36:37 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 7b64acb Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:32:47 2025 +0000 clean up code Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 7766ca5 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:31:33 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit b0629bd Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:27:58 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit eca7a49 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:25:24 2025 +0000 cleaning Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 1881aa5 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:24:55 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 7833645 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:11:18 2025 +0000 updared Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit e72e5e4 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:08:34 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 48c2eb2 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:07:37 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit de1e487 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:01:14 2025 +0000 fix nit Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 0163070 Merge: e2ecc14 8b3f606 Author: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com> Date: Wed Apr 9 10:39:46 2025 -0400 Merge pull request #4 from robertgshaw2-redhat/rob-changes Rob changes commit 8b3f606 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 14:29:26 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 90e8c53 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 14:29:17 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit da019df Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 14:23:47 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 4ebcc3e Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 13:44:41 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 00df670 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 13:20:14 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit a73721a Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 13:19:17 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 31d807e Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Tue Apr 8 20:58:28 2025 +0000 stash Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 5accb53 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Tue Apr 8 16:00:29 2025 +0000 stash Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Signed-off-by: ApostaC <yihua98@uchicago.edu>

commit 3a24897 Author: ApostaC <yihua98@uchicago.edu> Date: Thu Apr 10 18:31:47 2025 -0700 [Fix] memory leak problem by proper clean up Signed-off-by: ApostaC <yihua98@uchicago.edu> commit 406d6bf Author: Rémi Delacourt <54138269+Flechman@users.noreply.github.com> Date: Fri Apr 11 00:47:40 2025 +0200 Add MLA support for v1 disagg connector (#6) Signed-off-by: remi <remi@mistral.ai> commit 1d8415d Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Thu Apr 10 21:59:54 2025 +0000 rename Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 9c4159c Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Thu Apr 10 21:41:20 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 54e1491 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Thu Apr 10 20:31:35 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 8e1eadc Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Thu Apr 10 20:26:37 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 05349a5 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 22:10:50 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 7f57f3c Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 17:13:31 2025 +0000 update lifecycle Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 7c31e29 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 17:03:55 2025 +0000 nits Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 74af233 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 16:44:01 2025 +0000 done with nits Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit e64f745 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 16:28:51 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 40e5d81 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 16:25:04 2025 +0000 refactor Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 25c9592 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 16:20:41 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit fc58dd5 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 16:13:39 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 20decdf Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 16:06:15 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 5145566 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:52:03 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 62e1421 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:47:40 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 689379e Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:39:40 2025 +0000 updaed Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit b1310fd Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:36:37 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 7b64acb Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:32:47 2025 +0000 clean up code Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 7766ca5 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:31:33 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit b0629bd Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:27:58 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit eca7a49 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:25:24 2025 +0000 cleaning Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 1881aa5 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:24:55 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 7833645 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:11:18 2025 +0000 updared Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit e72e5e4 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:08:34 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 48c2eb2 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:07:37 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit de1e487 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 15:01:14 2025 +0000 fix nit Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 0163070 Merge: e2ecc14 8b3f606 Author: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com> Date: Wed Apr 9 10:39:46 2025 -0400 Merge pull request #4 from robertgshaw2-redhat/rob-changes Rob changes commit 8b3f606 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 14:29:26 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 90e8c53 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 14:29:17 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit da019df Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 14:23:47 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 4ebcc3e Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 13:44:41 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 00df670 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 13:20:14 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit a73721a Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Wed Apr 9 13:19:17 2025 +0000 updated Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 31d807e Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Tue Apr 8 20:58:28 2025 +0000 stash Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit 5accb53 Author: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Date: Tue Apr 8 16:00:29 2025 +0000 stash Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Signed-off-by: ApostaC <yihua98@uchicago.edu>

Add support for mxfp4 through marlin experts

…-project#6 (KV cache parsing) Bug vllm-project#5: Fix JSON escaping in rope_scaling parameter - Line 379: Correct rope_scaling JSON format with proper escaping - Prevents malformed YAML in docker compose files Bug vllm-project#6: Update regex patterns to match actual log format - Lines 851-856: Update KV cache detection patterns - Match actual vLLM log output format All 6 grid search bugs now resolved (Missions 14a-14k) Grid search validation successful with 36 configurations tested Refs: Mission 14k, Mission 15

@wuhang2014

* # This is a combination of 6 commits. # This is the 1st commit message: mooncake store connector Signed-off-by: CHEN <116010019@link.cuhk.edu.cn> # This is the commit message vllm-project#2: mooncake store connector Signed-off-by: CHEN <116010019@link.cuhk.edu.cn> # This is the commit message vllm-project#3: mooncake store connector Signed-off-by: CHEN <116010019@link.cuhk.edu.cn> # This is the commit message vllm-project#4: mooncake store connector Signed-off-by: CHEN <116010019@link.cuhk.edu.cn> # This is the commit message vllm-project#5: mooncake store connector Signed-off-by: CHEN <116010019@link.cuhk.edu.cn> # This is the commit message vllm-project#6: mooncake store connector Signed-off-by: CHEN <116010019@link.cuhk.edu.cn> * mooncake store connector Signed-off-by: CHEN <116010019@link.cuhk.edu.cn> * mooncake store connector Signed-off-by: CHEN <116010019@link.cuhk.edu.cn> mooncake store connector Signed-off-by: CHEN <116010019@link.cuhk.edu.cn> mooncake store connector Signed-off-by: CHEN <116010019@link.cuhk.edu.cn> mooncake store connector Signed-off-by: CHEN <116010019@link.cuhk.edu.cn> mooncake store connector Signed-off-by: CHEN <116010019@link.cuhk.edu.cn> mooncake store connector Signed-off-by: CHEN <116010019@link.cuhk.edu.cn> mooncake store connector Signed-off-by: CHEN <116010019@link.cuhk.edu.cn> fix comments * Update vllm/distributed/ec_transfer/utils/tensor_memory_pool.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update vllm/distributed/ec_transfer/ec_lookup_buffer/mooncake_store.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update vllm/distributed/ec_transfer/ec_connector/mooncake_storage_connector.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @wuhang2014 line length format * Apply suggestion from @wuhang2014 remove extra empty line --------- Signed-off-by: CHEN <116010019@link.cuhk.edu.cn> Co-authored-by: wuhang <whlbx@hotmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* [kernel][DS-R1][linear] use default Fp8LinearMethod/Fp8MoEMethod Signed-off-by: Kunshang Ji <kunshang.ji@intel.com> * [kernel][DS-R1][Attention] enable Triton MLA attention Signed-off-by: Kunshang Ji <kunshang.ji@intel.com> * enable MHA for deepseek, need padding head_size to make flash attn kernel happy Signed-off-by: Kunshang Ji <kunshang.ji@intel.com> * not break fp8 path Signed-off-by: Kunshang Ji <kunshang.ji@intel.com> --------- Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

- Add complete Accounting ERP MCP server code (140+ lines) - Add complete File Storage MCP server code (40+ lines) - Add detailed Agent execution sequence (STEP 1-56) - Add comprehensive ROI analysis - 99.1% time savings, 726만원/year cost savings

WoosukKwon added 17 commits March 3, 2023 04:16

Fix a bug in 1D shape

e5a1fa8

Minor

342275f

Minor

b91a2fa

[WIP] Add memory analyzer

d78e2fb

Automatically config GPU/CPU blocks

2649eb5

Remove TODO

1ae7420

Merge branch 'main' into autoconfig

6654b34

Merge branch 'main' into autoconfig

fcbf027

Add max_num_batched_tokens argument

350ed27

Minor

6f5b41b

Minor

2d03918

Refactor model utils

8ec00fe

Re-implement memory analyzer

84203fc

Fix __init__

96b216c

Use memory analyzer in server.py

c89d440

Add psutil to README

f5d1e2c

Fix comment

cc63c24

WoosukKwon merged commit e9d3f2f into main Mar 12, 2023

WoosukKwon deleted the autoconfig branch March 12, 2023 07:23

TheBloke mentioned this pull request Jul 20, 2023

Can't launch OpenAI API server on newly installed vLLM in Docker - fastchat not found #537

Closed

shanshanpt mentioned this pull request Nov 17, 2023

Run long conetxt error : CUDA error: an illegal memory access was encountered #1700

Closed

junior-zsy mentioned this pull request Nov 20, 2023

Error with 32k Long Text in chatglm2-6b-32k Model #1725

Closed

hongxiayang pushed a commit to hongxiayang/vllm that referenced this pull request Feb 13, 2024

Add memory analyzer & utomatically configure KV cache size (vllm-proj…

de10960

…ect#6)

slyalin pushed a commit to slyalin/vllm that referenced this pull request Mar 21, 2024

Merge pull request vllm-project#6 from mzegla/extended_requirements

2922b06

Add missing Python requirements

mzusman added a commit to mzusman/vllm that referenced this pull request Apr 16, 2024

dtype (vllm-project#6)

00bce1f

Co-authored-by: Mor Zusman <morz@ai21.com>

dtrifiro referenced this pull request in dtrifiro/vllm Apr 26, 2024

Merge pull request #6 from z103cb/ibm_main_docker_ubi_updates

91e4a51

[CI/Build] Dockerfile.ubi : Remove test stage

dlopes78 mentioned this pull request May 8, 2024

[Bug]: VLLM + tritonserver #4695

Closed

Starmys pushed a commit to Starmys/vllm that referenced this pull request May 20, 2024

Merge pull request vllm-project#6 from wenxcs/wenxh/fp8-on-a100

4e56e27

FP8 on A100 for PHIMOE

oliver-li mentioned this pull request Jul 5, 2024

[Bug]: NCCL hangs and causes timeout #5484

Closed

southfreebird mentioned this pull request Apr 11, 2025

[Bug]: Medusa speculation hangs when tp > 1 #16477

Closed

1 task

QuanhuiGuan mentioned this pull request Apr 14, 2025

[Bug]: CUDA error: an illegal memory access was encountered #16398

Closed

1 task

maobaolong pushed a commit to maobaolong/vllm that referenced this pull request Apr 14, 2025

Add MLA support for v1 disagg connector (vllm-project#6)

406d6bf

Signed-off-by: remi <remi@mistral.ai>

hao-cold mentioned this pull request May 13, 2025

[Bug]: CUDA error: an illegal instruction was encountered #18045

Closed

1 task

markmc mentioned this pull request May 21, 2025

[Bug][Failing Test]: Distributed Comm Ops - distributed/test_shm_broadcast.py #18492

Closed

1 task

zerosurplus mentioned this pull request Jun 16, 2025

[Bug]: torch.distributed.DistNetworkError: The client socket has timed out after 600000ms while trying to connect to (172.17.0.9, 46229). #19670

Open

1 task

xiaocode337317439 mentioned this pull request Jun 27, 2025

[Bug]:RuntimeError: CUDA error: an illegal memory access was encountered #20170

Open

1 task

Chris113113 mentioned this pull request Jul 10, 2025

[Bug]: [V1][gpu_model_runner.py] CUDA memory error #19415

Open

1 task

shrijayan mentioned this pull request Jul 12, 2025

vLLM hangs after 10 minutes without any error message #1492

Closed

tyxiong23 mentioned this pull request Jul 30, 2025

[Bug]: GLM-4.1V-Thinking ValueError #21811

Closed

1 task

xiaomofang mentioned this pull request Jul 31, 2025

[Bug]: There is an issue with speculative inference in Eagle mode, where the context length of vLLM inference is constrained by the draft model. #21986

Open

1 task

devops724 mentioned this pull request Aug 3, 2025

[Bug]: vLLM engine crashes then restarts and loads the model on sleep if a chat request is made #15483

Open

1 task

fernandaspets mentioned this pull request Aug 8, 2025

[Bug]: --tensor-parallel-size 2 seems broken for Blackwell 6000 pro since version 10 #22479

Open

crischeng mentioned this pull request Aug 12, 2025

[Bug]: CUDA error during nsys profile : unspecified launch failure #22746

Closed

1 task

ruisearch42 mentioned this pull request Aug 22, 2025

[Bug]: VLLM_ALL2ALL_BACKEND=naive hangs/crashes on multi nodes when serving DeepSeekV3 #23448

Open

1 task

shaamil101-etched mentioned this pull request Aug 25, 2025

[Bug]: vLLM server timeout due to multiprocessing communication error #23582

Open

1 task

ZJY0516 mentioned this pull request Aug 31, 2025

[Bug]: CUDA error when serving MiniCPM-V model #23954

Closed

wyn1015 mentioned this pull request Sep 19, 2025

[Bug]: assortment of warnings / errors coming out of vllm basic python inference script #18634

Open

1 task

zhanghb55 mentioned this pull request Sep 25, 2025

[Bug]: Pipeline parallel (pp>1) crashes with CUDA illegal memory access #25650

Open

1 task

dcmaddix pushed a commit to dcmaddix/vllm that referenced this pull request Oct 10, 2025

Merge pull request vllm-project#6 from dcmaddix/marlin_experts_mxfp4

ac3a49e

Add support for mxfp4 through marlin experts

tina0852 mentioned this pull request Oct 11, 2025

[Bug]: Since version 0.9.2 comes with nccl built-in, using PCIE causes sys errors. How to disable nccl in vllm for versions after 0.9.2? #26607

Open

1 task

Michel-debug mentioned this pull request Oct 23, 2025

[Bug]: qwen3-vl-2b after ms-swift fine-tuning lance errors #27405

Closed

1 task

whwangovo mentioned this pull request Oct 23, 2025

[Bug]: vLLM (TP=8) on 235B model triggers "CUDA error: unspecified launch failure" and persistent "ERR!" state in nvidia-smi #27430

Open

1 task

acodercat mentioned this pull request Nov 10, 2025

[Bugfix] Add strong reference to CUDA pluggable allocator callbacks #23477

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Automatically configure KV cache size #6

Automatically configure KV cache size #6

WoosukKwon commented Mar 3, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Automatically configure KV cache size #6

Automatically configure KV cache size #6

Conversation

WoosukKwon commented Mar 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

WoosukKwon commented Mar 3, 2023 •

edited

Loading