-
Notifications
You must be signed in to change notification settings - Fork 532
[0.11.0][MTP][Aclgraph] Fix the support aclgraph with MTP #3912
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: MengqingCao <cmq0113@163.com>
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request addresses two issues related to aclgraph support with MTP in vllm 0.11.0. It introduces a patch to avoid d2h synchronization in the MTP predictor's forward pass and adds the support_torch_compile decorator for deepseekmtp. The changes are targeted for the v0.11.0-dev branch.
Signed-off-by: MengqingCao <cmq0113@163.com>
|
@wangxiyuan @zzzzwwjj This pr is ready for merge, plz take a look, thanks! |
What this PR does / why we need it?
Fix 2 breaks of aclgraph with MTP:
support_torch_compiledecoratorAs we'll fix it in vllm main, this fix pr is only needed in branch v0.11.0-dev
The profling shows that MTP replays in aclgraph now:

How was this patch tested?