[BugFix] Fix accuracy bugs for unquantized deepseekv3 models #897
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: angazenn <zengyanjia@huawei.com>
Collaborator
|
Could you post the inference output of deepseek-v2-lite-chat after this pr? I come up with a precision issue with deepseek-v2-lite-chat now |
Angazenn
changed the title
Collaborator
|
still work in progress? or ready for review. |
Angazenn
changed the title
Angazenn
force-pushed
the
fix_acc
branch
2 times, most recently
from
bd9fc37 to
9b23577
Compare
Collaborator
Author
It is ready now. |
wangxiyuan
approved these changes
May 21, 2025
vllm_ascend/ops/fused_moe.py
Outdated
|
|
||
| ep_group = get_ep_group() | ||
| self.ep_size = ep_group.world_size | ||
| self.ep_group = get_ep_group() |
Collaborator
There was a problem hiding this comment.
Do not change the code if its not necessary
Collaborator
Author
There was a problem hiding this comment.
There are some redundant codes in fused_moe before. Now we maintain self.ep_group and use self.ep_group.world_size to get ep_size.
Angazenn
force-pushed
the
fix_acc
branch
2 times, most recently
from
fa81306 to
605fe9a
Compare
Signed-off-by: angazenn <zengyanjia@huawei.com>
momo609
pushed a commit
to momo609/vllm-ascend
that referenced
this pull request
May 30, 2025
…oject#897) ### What this PR does / why we need it? This PR fixes two accuracy bugs incurred by PR vllm-project#819 when running deepseekv3 series models: 1. vllm-project#819 adds `all_to_all` communication in quantized cases, but `all_gather` && `reduce_scatter` are removed in both of quantized and unquantized cases. When running unquantized deepseekv3 models with `ep_size == world_size`, the moe modules fail to communicate. Therefore, this PR adds `all_to_all` communication on unquantized situation to solve this accuracy issue. 2. Use `ep_size` rather than `dp_size` to decide whether to use `all_to_all` in moe. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI passed with new added/existing test. --------- Signed-off-by: angazenn <zengyanjia@huawei.com> Co-authored-by: angazenn <zengyanjia@huawei.com> Signed-off-by: wangxiaoxin (A) <w00664509@china.huawei.com>
zxdukki
pushed a commit
to zxdukki/vllm-ascend
that referenced
this pull request
Jun 3, 2025
…oject#897) ### What this PR does / why we need it? This PR fixes two accuracy bugs incurred by PR vllm-project#819 when running deepseekv3 series models: 1. vllm-project#819 adds `all_to_all` communication in quantized cases, but `all_gather` && `reduce_scatter` are removed in both of quantized and unquantized cases. When running unquantized deepseekv3 models with `ep_size == world_size`, the moe modules fail to communicate. Therefore, this PR adds `all_to_all` communication on unquantized situation to solve this accuracy issue. 2. Use `ep_size` rather than `dp_size` to decide whether to use `all_to_all` in moe. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI passed with new added/existing test. --------- Signed-off-by: angazenn <zengyanjia@huawei.com> Co-authored-by: angazenn <zengyanjia@huawei.com>
chopper0126
pushed a commit
to chopper0126/vllm-ascend
that referenced
this pull request
Oct 16, 2025
…oject#897) ### What this PR does / why we need it? This PR fixes two accuracy bugs incurred by PR vllm-project#819 when running deepseekv3 series models: 1. vllm-project#819 adds `all_to_all` communication in quantized cases, but `all_gather` && `reduce_scatter` are removed in both of quantized and unquantized cases. When running unquantized deepseekv3 models with `ep_size == world_size`, the moe modules fail to communicate. Therefore, this PR adds `all_to_all` communication on unquantized situation to solve this accuracy issue. 2. Use `ep_size` rather than `dp_size` to decide whether to use `all_to_all` in moe. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI passed with new added/existing test. --------- Signed-off-by: angazenn <zengyanjia@huawei.com> Co-authored-by: angazenn <zengyanjia@huawei.com>
Angazenn
added a commit
to Angazenn/vllm-ascend
that referenced
this pull request
Oct 21, 2025
…oject#897) ### What this PR does / why we need it? This PR fixes two accuracy bugs incurred by PR vllm-project#819 when running deepseekv3 series models: 1. vllm-project#819 adds `all_to_all` communication in quantized cases, but `all_gather` && `reduce_scatter` are removed in both of quantized and unquantized cases. When running unquantized deepseekv3 models with `ep_size == world_size`, the moe modules fail to communicate. Therefore, this PR adds `all_to_all` communication on unquantized situation to solve this accuracy issue. 2. Use `ep_size` rather than `dp_size` to decide whether to use `all_to_all` in moe. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI passed with new added/existing test. --------- Signed-off-by: angazenn <zengyanjia@huawei.com> Co-authored-by: angazenn <zengyanjia@huawei.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it?
This PR fixes two accuracy bugs incurred by PR #819 when running deepseekv3 series models:
all_to_allcommunication in quantized cases, butall_gather&&reduce_scatterare removed in both of quantized and unquantized cases. When running unquantized deepseekv3 models withep_size == world_size, the moe modules fail to communicate. Therefore, this PR addsall_to_allcommunication on unquantized situation to solve this accuracy issue.ep_sizerather thandp_sizeto decide whether to useall_to_allin moe.Does this PR introduce any user-facing change?
No.
How was this patch tested?
CI passed with new added/existing test.