[V0 Deprecation] Remove `VLLM_USE_V1` from tests #26341

DarkLight1337 · 2025-10-07T08:03:49Z

Purpose

Remove VLLM_USE_V1 flag from all tests, except for:

~~tests/basic_correctness/test_cumem.py: Should we delete the V0-specific code or just comment them out? cc @youkaichao @ywang96 ~~
tests/v1/test_oracle.py: For obvious reasons

Updates:

tests/v1/core/test_scheduler_e2e.py was previously not run. ~~Updated CPU model runner to avoid assert num_tokens <= self.scheduler_config.max_num_batched_tokens assertion error in dummy run. However now the test causes the process to be terminated, so I'm skipping this for CPU. cc @bigPYJ1151 ~~ Running this in the GPU CI instead of CPU CI
tests/v1/sample/test_sampling_params_e2e.py was previously not run. Removed test_priority because priority scheduling is now supported in [Core] feat: Implement Priority Scheduling in V1 Engine #19057.

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

gemini-code-assist

Code Review

This pull request effectively removes the VLLM_USE_V1 flag from a large number of test files, which is a crucial step in deprecating the V0 engine and solidifying V1 as the default. The changes are consistent and well-executed across the entire test suite, correctly removing environment variable settings, conditional logic, and fixtures related to the V0 engine. This simplifies the tests and aligns them with the V1-only architecture. I've reviewed the changes thoroughly and found no issues of high or critical severity. The cleanup is well done.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

chatgpt-codex-connector · 2025-10-07T08:08:35Z

tests/entrypoints/llm/test_accuracy.py

 @pytest.mark.parametrize("model", MODEL_NAMES)
-def test_lm_eval_accuracy_v1_engine(model, monkeypatch: pytest.MonkeyPatch):
+def test_lm_eval_accuracy_v1_engine(model):
    """Run with the V1 Engine."""

-    with monkeypatch.context() as m:
-        m.setenv("VLLM_USE_V1", "1")
+    more_args = None
+    if current_platform.is_tpu():
+        # Limit compilation time for TPU V1

-        more_args = None
-        if current_platform.is_tpu():
-            # Limit compilation time for TPU V1
+        more_args = "max_model_len=2048,max_num_seqs=64"

-            more_args = "max_model_len=2048,max_num_seqs=64"
+        # Add TP test (if provided)
+        if TPU_TP_TEST_STR:
+            more_args += ",{}".format(TPU_TP_TEST_STR)

-            # Add TP test (if provided)
-            if TPU_TP_TEST_STR:
-                more_args += ",{}".format(TPU_TP_TEST_STR)
+    run_test(model, more_args)

-        run_test(model, more_args)

-
-@pytest.mark.skipif(
-    not current_platform.is_cuda() and not current_platform.is_tpu(),
-    reason="V1 is currently only supported on CUDA and TPU",
-)
 @pytest.mark.parametrize("model", FP8_KV_MODEL_NAMES)
-def test_lm_eval_accuracy_v1_engine_fp8_kv_cache(
-    model, monkeypatch: pytest.MonkeyPatch
-):
+def test_lm_eval_accuracy_v1_engine_fp8_kv_cache(model):
    """Run with the V1 Engine."""

-    with monkeypatch.context() as m:
-        m.setenv("VLLM_USE_V1", "1")
-
-        more_args = None
-        if current_platform.is_tpu():
-            # Limit compilation time for TPU V1
-            more_args = "max_model_len=2048,max_num_seqs=128,kv_cache_dtype=fp8"
+    more_args = None
+    if current_platform.is_tpu():


Reinstate platform guard for LM-Eval accuracy tests

The two LM-Eval accuracy tests no longer have the @pytest.mark.skipif guard that previously skipped them when CUDA/TPU are unavailable. These tests spin up vLLM and run gsm8k with 500 concurrent requests, which assumes GPU or TPU support; running them on a CPU-only runner will make CI fail rather than skip as before. Please restore the skip condition (or an equivalent platform gate) so CPU environments don’t execute GPU-only workloads.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2025-10-07T08:08:35Z

tests/entrypoints/openai/correctness/test_lmeval.py

+def test_lm_eval_accuracy_v1_engine():
    """Run with the V1 Engine."""

-    with monkeypatch.context() as m:
-        m.setenv("VLLM_USE_V1", "1")
-        more_args = []
+    more_args = []

-        # Limit compilation time for V1
-        if current_platform.is_tpu():
-            more_args = ["--max-num-seqs", "64"]
+    # Limit compilation time for V1
+    if current_platform.is_tpu():
+        more_args = ["--max-num-seqs", "64"]


Accuracy correctness test now runs on unsupported platforms

The OpenAI correctness LM-Eval test also had its @pytest.mark.skipif removed and now runs unconditionally. It still launches a RemoteOpenAIServer and evaluates gsm8k with 500 concurrent requests, which requires CUDA/XPU/TPU. Executing this on CPU-only CI will fail rather than skip as intended earlier. A platform guard should be reintroduced to avoid running this GPU-only test on unsupported hardware.

Useful? React with 👍 / 👎.

hmellor

LGTM! One nit about a TODO which can be removed now that monkeypatch has been removed.

tests/v1/engine/test_async_llm.py

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

youkaichao · 2025-10-07T11:41:32Z

tests/basic_correctness/test_cumem.py: Should we delete the V0-specific code or just comment them out?

we can remove them.

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: xuebwang-amd <xuebwang@amd.com>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: Dhruvil Bhatt <bhattdbh@amazon.com>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: xuebwang-amd <xuebwang@amd.com>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

DarkLight1337 added 2 commits October 7, 2025 08:00

[V0 Deprecation] Remove VLLM_USE_V1 from tests

3ef6f48

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Remove comment

7a83947

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 7, 2025

DarkLight1337 added this to V0 Deprecation Oct 7, 2025

DarkLight1337 requested review from ApostaC, NickLucche, WoosukKwon, aarnphm, alexm-redhat, comaniac, heheda12345, mgoin, njhill, robertgshaw2-redhat, russellb, simon-mo, tlrmchlsmth, yewentao256, youkaichao and ywang96 as code owners October 7, 2025 08:03

mergify bot added multi-modality Related to multi-modality (#4194) structured-output speculative-decoding labels Oct 7, 2025

github-project-automation bot added this to Structured Output Oct 7, 2025

mergify bot added v1 tpu Related to Google TPUs kv-connector labels Oct 7, 2025

gemini-code-assist bot reviewed Oct 7, 2025

View reviewed changes

chatgpt-codex-connector bot reviewed Oct 7, 2025

View reviewed changes

DarkLight1337 moved this to In Progress in V0 Deprecation Oct 7, 2025

hmellor approved these changes Oct 7, 2025

View reviewed changes

tests/v1/engine/test_async_llm.py Outdated Show resolved Hide resolved

DarkLight1337 added 2 commits October 7, 2025 09:40

Remove outdated test

2b09bbd

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Fix test

7a6da1a

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

DarkLight1337 requested a review from bigPYJ1151 as a code owner October 7, 2025 09:45

Address comment

815db91

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

DarkLight1337 added 3 commits October 7, 2025 12:19

Skip and add FIXME

8bd512a

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Remove V0 code

8d47cdc

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Fix CPU test distribution

24747f3

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

DarkLight1337 force-pushed the remove-use-v1-tests branch from 665bc50 to 24747f3 Compare October 7, 2025 13:09

DarkLight1337 enabled auto-merge (squash) October 7, 2025 13:09

mergify bot added the ci/build label Oct 7, 2025

Fix typo

67fb7b4

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

DarkLight1337 merged commit 1e4ecca into vllm-project:main Oct 7, 2025
56 checks passed

DarkLight1337 deleted the remove-use-v1-tests branch October 7, 2025 15:42

github-project-automation bot moved this to Done in Structured Output Oct 7, 2025

github-project-automation bot moved this from In Progress to Done in V0 Deprecation Oct 7, 2025

mrasquinha-g pushed a commit to mrasquinha-g/vllm that referenced this pull request Oct 9, 2025

[V0 Deprecation] Remove VLLM_USE_V1 from tests (vllm-project#26341)

4e81c87

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request Oct 10, 2025

[V0 Deprecation] Remove VLLM_USE_V1 from tests (vllm-project#26341)

2072984

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: xuebwang-amd <xuebwang@amd.com>

Dhruvilbhatt pushed a commit to Dhruvilbhatt/vllm that referenced this pull request Oct 14, 2025

[V0 Deprecation] Remove VLLM_USE_V1 from tests (vllm-project#26341)

e5269e6

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: Dhruvil Bhatt <bhattdbh@amazon.com>

lywa1998 pushed a commit to lywa1998/vllm that referenced this pull request Oct 20, 2025

[V0 Deprecation] Remove VLLM_USE_V1 from tests (vllm-project#26341)

3757001

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

alhridoy pushed a commit to alhridoy/vllm that referenced this pull request Oct 24, 2025

[V0 Deprecation] Remove VLLM_USE_V1 from tests (vllm-project#26341)

d8c140d

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request Oct 24, 2025

[V0 Deprecation] Remove VLLM_USE_V1 from tests (vllm-project#26341)

cf92f84

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: xuebwang-amd <xuebwang@amd.com>

rtourgeman pushed a commit to rtourgeman/vllm that referenced this pull request Nov 10, 2025

[V0 Deprecation] Remove VLLM_USE_V1 from tests (vllm-project#26341)

d37b0ae

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[V0 Deprecation] Remove `VLLM_USE_V1` from tests #26341

[V0 Deprecation] Remove `VLLM_USE_V1` from tests #26341

Uh oh!

DarkLight1337 commented Oct 7, 2025 •

edited by github-actions bot

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

chatgpt-codex-connector bot Oct 7, 2025

Uh oh!

chatgpt-codex-connector bot Oct 7, 2025

Uh oh!

hmellor left a comment

Uh oh!

Uh oh!

youkaichao commented Oct 7, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

[V0 Deprecation] Remove VLLM_USE_V1 from tests #26341

[V0 Deprecation] Remove VLLM_USE_V1 from tests #26341

Uh oh!

Conversation

DarkLight1337 commented Oct 7, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector bot Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

hmellor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

youkaichao commented Oct 7, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[V0 Deprecation] Remove `VLLM_USE_V1` from tests #26341

[V0 Deprecation] Remove `VLLM_USE_V1` from tests #26341

DarkLight1337 commented Oct 7, 2025 •

edited by github-actions bot

Loading