Skip to content

[Core/DBO][3/N] Dual-Batch Overlap with speculative decode #24608

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[Core/DBO][3/N] Dual-Batch Overlap with speculative decode #24608

wants to merge 1 commit into from

Conversation

@MatthewBonanni
Copy link
Contributor

@MatthewBonanni MatthewBonanni commented Sep 10, 2025 •
edited by github-actions bot
Loading

Purpose

Enable speculative decoding with DBO.

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@mergify mergify bot added documentation Improvements or additions to documentation speculative-decoding v1 labels Sep 10, 2025
@mergify
Copy link

mergify bot commented Sep 10, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @MatthewBonanni.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Sep 10, 2025
@MatthewBonanni MatthewBonanni changed the title Enable MTP with DBO Enable Speculative Decoding with DBO Sep 10, 2025
@mergify mergify bot removed the needs-rebase label Sep 11, 2025
@MatthewBonanni MatthewBonanni force-pushed the dbo_mtp branch 2 times, most recently from f202b09 to aa9137c Compare September 12, 2025 00:06
@MatthewBonanni MatthewBonanni changed the title Enable Speculative Decoding with DBO [Core/DBO] Dual-Batch Overlap with speculative decoding Sep 15, 2025
@MatthewBonanni MatthewBonanni changed the title [Core/DBO] Dual-Batch Overlap with speculative decoding [Core/DBO][3/N] Dual-Batch Overlap with speculative decoding Sep 15, 2025
@mergify
Copy link

mergify bot commented Sep 15, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @MatthewBonanni.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Sep 15, 2025
@MatthewBonanni MatthewBonanni changed the title [Core/DBO][3/N] Dual-Batch Overlap with speculative decoding [Core/DBO][3/N] Dual-Batch Overlap with MLA multi-token decode Sep 15, 2025
@MatthewBonanni MatthewBonanni changed the title [Core/DBO][3/N] Dual-Batch Overlap with MLA multi-token decode [Core/DBO][3/N] Dual-Batch Overlap with speculative decode Sep 15, 2025
@mergify mergify bot removed the needs-rebase label Sep 17, 2025
@mergify
Copy link

mergify bot commented Sep 18, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @MatthewBonanni.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Sep 18, 2025
@MatthewBonanni
Handle shared experts correctly
7825421 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
@mergify mergify bot removed the needs-rebase label Sep 24, 2025
@MatthewBonanni
Copy link
Contributor Author

Closing as the issues have been addressed by #25904 and #26231

@MatthewBonanni MatthewBonanni deleted the dbo_mtp branch October 6, 2025 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mgoin mgoin Awaiting requested review from mgoin mgoin will be requested when the pull request is marked ready for review mgoin is a code owner

@pavanimajety pavanimajety Awaiting requested review from pavanimajety pavanimajety will be requested when the pull request is marked ready for review pavanimajety is a code owner

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation speculative-decoding v1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@MatthewBonanni