Skip to content

[Core] feat: Implement Priority Scheduling in V1 Engine #18700

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 256 commits into from
Closed

[Core] feat: Implement Priority Scheduling in V1 Engine #18700

wants to merge 256 commits into from

Conversation

@amitm02
Copy link
Contributor

@amitm02 amitm02 commented May 26, 2025 •
edited by github-actions bot
Loading

This commit introduces priority scheduling capabilities to the V1 LLM engine.

Key changes include:

  1. EngineCoreRequest and Request updates:

    • Added a priority field to EngineCoreRequest and Request classes to carry priority information.

  2. Processor update:

    • Modified Processor.process_inputs to accept and pass the priority to EngineCoreRequest.

  3. V1 Scheduler modifications:

    • The scheduler now respects the --scheduling-policy argument.
    • When policy="priority", self.waiting is managed as a min-heap, prioritizing requests by their assigned priority value (lower value means higher priority) and then by arrival time (FCFS).
    • Preemption logic now correctly identifies and preempts the actual lowest-priority running request when space is needed for higher-priority or new requests.
    • FCFS behavior is maintained when policy="fcfs".

  4. Documentation:

    • Updated docs/usage/v1_guide.md and docs/serving/openai_compatible_server.md to reflect V1 engine's support for priority scheduling.

  5. Unit Tests:

    • Added a new test suite in tests/v1/core/test_scheduler.py.

This allows you to influence the order of request processing in the V1 engine by assigning priorities, which is particularly useful in scenarios with varying request importance.

FIX #14002

@github-actions
Copy link

github-actions bot commented May 26, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@mergify mergify bot added documentation Improvements or additions to documentation v1 labels May 26, 2025
This was referenced May 26, 2025
Closed
Open
@amitm02 amitm02 force-pushed the feat/v1-priority-scheduling branch from dbdfa5b to 8b54316 Compare May 27, 2025 12:08
@mergify mergify bot added ci/build frontend multi-modality Related to multi-modality (#4194) structured-output tpu Related to Google TPUs labels May 27, 2025
@mergify mergify bot added the tool-calling label May 27, 2025
@mergify mergify bot removed the tpu Related to Google TPUs label May 27, 2025
njhill and others added 14 commits June 1, 2025 17:57
@njhill @yiz-liu @amitm02
[BugFix] Fix multi-node offline data-parallel (vllm-project#18981)
21db17a 
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Yizhou Liu <liu_yizhou@outlook.com>
Signed-off-by: amit <amit.man@gmail.com>
@reidliu41 @amitm02
[Misc] add return token strs for tokenize (vllm-project#18941)
63f4c59 
Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>
Signed-off-by: amit <amit.man@gmail.com>
@ekagra-ranjan @amitm02
[Misc][Benchmark] Add support for CustomDataset (vllm-project#18511)
8882440 
Signed-off-by: amit <amit.man@gmail.com>
@benchislett @amitm02
[Bugfix] Fix EAGLE3 broken logits (vllm-project#18909)
2334fe9 
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Signed-off-by: amit <amit.man@gmail.com>
@DarkLight1337 @amitm02
[Core] Rework dtype resolution (vllm-project#18751)
5087dcc 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: amit <amit.man@gmail.com>
@Isotr0py @amitm02
[LoRA] Support dynamically initialize packed_modules_mapping for VL…
122b00a 
…M with arbitrary components (vllm-project#18987)

Signed-off-by: isotr0py <2037008807@qq.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: amit <amit.man@gmail.com>
@reidliu41 @amitm02
[doc] small fix - mkdocs (vllm-project#18996)
9311b97 
Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>
Signed-off-by: amit <amit.man@gmail.com>
@mgoin @amitm02
Let max_num_batched_tokens use human_readable_int for large numbers (v…
6c27d82 
…llm-project#18968)

Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: amit <amit.man@gmail.com>
@lengrongfu @amitm02
[BugFix] fix data parallel construct ipv6 url addres (vllm-project#18991
694899e 
)

Signed-off-by: rongfu.leng <rongfu.leng@daocloud.io>
Signed-off-by: amit <amit.man@gmail.com>
@njhill @amitm02
[BugFix] Fix incorrect metrics shutdown error log message (vllm-proje…
4e0185b 
…ct#18992)

Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: amit <amit.man@gmail.com>
@reidliu41 @amitm02
[doc] wrong output (vllm-project#19000)
5eab7d7 
Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>
Signed-off-by: amit <amit.man@gmail.com>
@amitm02
Update request.py
9228cd1 
Signed-off-by: amit <amit.man@gmail.com>
@amitm02
Update request.py
c062ede 
Signed-off-by: amit <amit.man@gmail.com>
@amitm02
Merge branch 'feat/v1-priority-scheduling' of https://github.com/amit…
ee3ab04 
…m02/vllm into feat/v1-priority-scheduling
@mergify mergify bot added speculative-decoding tpu Related to Google TPUs labels Jun 1, 2025
@mergify
Copy link

mergify bot commented Jun 1, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @amitm02.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Jun 1, 2025
@mergify mergify bot removed tpu Related to Google TPUs needs-rebase labels Jun 1, 2025
amitm02 added 3 commits June 1, 2025 18:08
@amitm02
minor
c289981 
Signed-off-by: amit <amit.man@gmail.com>
@amitm02
line too long
5e8b804 
Signed-off-by: amit <amit.man@gmail.com>
@amitm02
line too long
70ca4b2 
Signed-off-by: amit <amit.man@gmail.com>
@youkaichao
Copy link
Member

youkaichao commented Jun 2, 2025

the commit history is in a mess, can you clean it up? maybe open another PR?

@amitm02
Copy link
Contributor Author

amitm02 commented Jun 3, 2025

the commit history is in a mess, can you clean it up? maybe open another PR?

Re-submitted as #19057

@amitm02 amitm02 closed this Jun 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hmellor hmellor Awaiting requested review from hmellor

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@njhill njhill Awaiting requested review from njhill njhill is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

@comaniac comaniac Awaiting requested review from comaniac

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat alexm-redhat is a code owner

@jeejeelee jeejeelee Awaiting requested review from jeejeelee

@DarkLight1337 DarkLight1337 Awaiting requested review from DarkLight1337

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth

@simon-mo simon-mo Awaiting requested review from simon-mo

@mgoin mgoin Awaiting requested review from mgoin

@russellb russellb Awaiting requested review from russellb

@zhuohan123 zhuohan123 Awaiting requested review from zhuohan123

@youkaichao youkaichao Awaiting requested review from youkaichao

Assignees

No one assigned

Labels

ci/build documentation Improvements or additions to documentation frontend multi-modality Related to multi-modality (#4194) speculative-decoding structured-output tool-calling v1

Projects

Structured Output
Status: Done
Tool Calling
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature]: Implement Priority Scheduling In V1 Engine