Improve configs - the rest! #17562

hmellor · 2025-05-01T22:13:00Z

Converts the remaining configs to be compatible with @config
Enable Pydantic mypy checks and convert configs to Pydantic dataclasses #17599 (split into separate PR)
Automatically tell users that dict args must be valid JSON in CLI #17577 (split into separate PR)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

…cally Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

github-actions · 2025-05-01T22:13:09Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

aarnphm

quick review, but lgtm

vllm/config.py

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

mergify · 2025-05-02T12:38:00Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor · 2025-05-08T17:03:48Z

TPU is fixed
Remaining failures seem spec decode related

yaochengji

LGTM, thanks!

DarkLight1337 · 2025-05-09T03:12:38Z

Can you merge from main to ensure this doesn't break CI?

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor · 2025-05-09T12:03:43Z

Somehow there was a typo in the test that only got discovered when removing Pydantic and switching to @config 🙃

hmellor · 2025-05-09T12:10:16Z

The distributed 4 test has been failing on main for a while https://buildkite.com/vllm/ci/builds/19648/steps?jid=0196b442-64dd-4809-af32-ab6dadebaab8

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>

jinzhen-lin · 2025-05-10T08:49:51Z

hi @hmellor
After this PR is merged, initializing VllmConfig or even just reading its kwargs (https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py#L799 ) will instantiate ModelConfig, which attempts to download the default facebook/opt-125m model. This causes errors when running vllm in an offline environment.

hmellor · 2025-05-10T08:53:16Z

Thanks for letting me know @jinzhen-lin , I will hotfix this right away

hmellor · 2025-05-10T09:03:29Z

Fix is in #17943

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

ProExpertProg · 2025-05-14T18:54:14Z

vllm/config.py

-        return CompilationConfig.model_validate(dict_value)
-
-    def model_post_init(self, __context: Any) -> None:
+        return cls(**json.loads(cli_value))


Is there a reason you made this json instead of ast.literal_eval? This is a breaking change for CLI config passing: it enforces double quotes for property names and enforces true instead of True for booleans, etc.

Just for consistency with all the other JSON CLI args.

I also have a draft PR which enables Pydantic validation for this CLI arg (and the other JSON CLI args), not sure if Pydantic enforces quotes in its JSON parser.

Is it important that this arg accepts Python literal strings?

Jetbrains IDEs treat double quotes differently in run config params and I have a bunch of scripts using single quotes but I think it's fine. Was just curious why the change was needed, but no point changing it back. The only thing is right now the error message is not very helpful

The error message can definitely be improved before I figure out Pydantic (it's breaking the docs build...), so if I can't figure it out soon I'll do that in a separate PR

No rush, figuring out pydantic first seem like the right move, we can improve the error msg later!

With Pydantic we get the validation error directly in the CLI, it's super informative!

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Yuqi Zhang <yuqizhang@google.com>

hmellor added 9 commits May 1, 2025 23:23

Use Pydantic dataclass with arbitrary types enabled

3087d8e

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Make KVTransferConfig @config compliant

bbdba0f

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Make KVEventsConfig @config compliant

6a36bd0

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Make CompilationConfig compliant enough

b33ba00

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Make VllmConfig @config compliant

f6b5b90

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Move full config args to vllm_group

57f534f

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Support pydantic dataclasses in get_args

c33c85a

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Remove JSON tip from docstrings and add to all JSON CLI args automati…

4d31946

…cally Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Mapy mypy happy

8254635

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor changed the title ~~Improve-rest-configs~~ Improve configs - the rest! May 1, 2025

Small error message fix

ace86c0

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

aarnphm approved these changes May 1, 2025

View reviewed changes

vllm/config.py Outdated Show resolved Hide resolved

vllm/config.py Outdated Show resolved Hide resolved

vllm/config.py Outdated Show resolved Hide resolved

hmellor added 9 commits May 2, 2025 00:25

Fix usage of TypeAdapter

dc9ca53

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Review comment

18016ad

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Fix missing el

8a0228c

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Reorder arg default priority

6080705

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Fix forward references for pydantic

fae62fd

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

SKip validation for dynamic configs

45e3fb7

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Fix CompilationConfig

427fcb1

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Fix get_kwargs

1a55abf

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Enable pydantic mypy plugin

743db5a

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

mergify bot added the needs-rebase label May 2, 2025

hmellor added 3 commits May 2, 2025 15:02

Fix mypy

d0141ea

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Fix LogitsProcessorConstructor

2e24fec

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Merge branch 'main' into improve-rest-configs

9493728

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor requested review from mgoin and russellb as code owners May 2, 2025 14:37

mergify bot added the frontend label May 2, 2025

hmellor added 5 commits May 8, 2025 11:34

Fix v1 test

aa5afca

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Fix pytorch fullgraph smoke test

5aa1c1c

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Fix benchmarks CLI test

57aeb69

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Fix distributed tests

62f29a5

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Fix tpu test

45e3d46

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

mergify bot removed the needs-rebase label May 8, 2025

hmellor added 2 commits May 8, 2025 13:59

Merge branch 'main' into improve-rest-configs

fa018cd

Fix PassConfig not being a member of CompilationConfig

54d6705

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

yaochengji approved these changes May 8, 2025

View reviewed changes

hmellor added 2 commits May 9, 2025 08:08

Merge branch 'main' into improve-rest-configs

cd343b8

Fix typo in test...

d9997ab

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Merge branch 'main' into improve-rest-configs

44ff033

vllm-bot merged commit 4b2ed79 into vllm-project:main May 9, 2025
55 of 57 checks passed

github-project-automation bot moved this to Done in Structured Output May 9, 2025

princepride pushed a commit to princepride/vllm that referenced this pull request May 10, 2025

Improve configs - the rest! (vllm-project#17562)

7ee2365

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>

hmellor deleted the improve-rest-configs branch May 10, 2025 06:12

hmellor mentioned this pull request May 10, 2025

Don't default construct ModelConfig when default constructing VllmConfig #17943

Merged

RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request May 12, 2025

Improve configs - the rest! (vllm-project#17562)

f2d96f7

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>

mawong-amd pushed a commit to ROCm/vllm that referenced this pull request May 14, 2025

Improve configs - the rest! (vllm-project#17562)

b9cb4b6

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

ProExpertProg reviewed May 14, 2025

View reviewed changes

leeeizhang mentioned this pull request May 16, 2025

[Bugfix] tutorials broken by KVTransferConfig API changes in vLLM v0.9.0 LMCache/LMCache#664

Merged

zzzyq pushed a commit to zzzyq/vllm that referenced this pull request May 24, 2025

Improve configs - the rest! (vllm-project#17562)

8601a75

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Yuqi Zhang <yuqizhang@google.com>

Uh oh!

Improve configs - the rest! #17562

Improve configs - the rest! #17562

Uh oh!

Conversation

hmellor commented May 1, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented May 1, 2025

Uh oh!

aarnphm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mergify bot commented May 2, 2025

Uh oh!

hmellor commented May 8, 2025

Uh oh!

yaochengji left a comment

Choose a reason for hiding this comment

Uh oh!

DarkLight1337 commented May 9, 2025

Uh oh!

hmellor commented May 9, 2025

Uh oh!

hmellor commented May 9, 2025

Uh oh!

Uh oh!

jinzhen-lin commented May 10, 2025

Uh oh!

hmellor commented May 10, 2025

Uh oh!

hmellor commented May 10, 2025

Uh oh!

ProExpertProg May 14, 2025

Choose a reason for hiding this comment

Uh oh!

hmellor May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ProExpertProg May 14, 2025

Choose a reason for hiding this comment

Uh oh!

hmellor May 14, 2025

Choose a reason for hiding this comment

Uh oh!

ProExpertProg May 14, 2025

Choose a reason for hiding this comment

Uh oh!

hmellor May 14, 2025

Choose a reason for hiding this comment

Uh oh!

hmellor May 14, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

hmellor commented May 1, 2025 •

edited by github-actions bot

Loading

hmellor May 14, 2025 •

edited

Loading