Low E2E throughput performance on SM100

[yewentao256]

Hi DeepGemm team,

vllm-project/vllm#19820

Here I integrate the new version of DeepGemm On SM100

The performance test on kernel looks good, but for the e2e throughput performance, it look quite slow compared to triton.

VLLM_USE_DEEP_GEMM=1 vllm bench throughput  --model Qwen/Qwen3-30B-A3B-FP8 --load-format dummy --input-len 1000 --output-len 100 --trust_remote_code --enforce-eager --enable-expert-parallel --quantization fp8
Throughput: 26.34 requests/s, 28916.28 total tokens/s, 2634.07 output tokens/s
vllm bench throughput  --model Qwen/Qwen3-30B-A3B-FP8 --load-format dummy --input-len 1000 --output-len 100 --trust_remote_code --enforce-eager --enable-expert-parallel --quantization fp8
Throughput: 36.65 requests/s, 40270.79 total tokens/s, 3665.06 output tokens/s


VLLM_USE_DEEP_GEMM=1 vllm bench throughput  --model deepseek-ai/DeepSeek-R1 --load-format dummy --input-len 32 --output-len 128 --trust_remote_code --enforce-eager -tp 8 --enable-expert-parallel  --no-enable-prefix-caching
Throughput: 23.89 requests/s, 3821.89 total tokens/s, 3058.29 output tokens/s
# NO deepgemm
Throughput: 42.59 requests/s, 6811.08 total tokens/s, 5451.01 output tokens/s

Do you know why this would happen and how could we solve it?

[LyricZhao]

The main branch currently does not support archs other than SM90. For SM100, #112 supports SM100 but it is maintained by NVIDIA and still in active development/optimization.

[yewentao256]

The main branch currently does not support archs other than SM90. For SM100, #112 supports SM100 but it is maintained by NVIDIA and still in active development/optimization.

Thanks for the reply, I actually used this branch for SM100. Let's keep this issue and see whether they can give us some feedback or if there are something we can do to optimize it further.

[yewentao256]

@LyricZhao Hi, Tested on H100 with older version of DeepGemm, seems even worse

# Deepgemm for Qwen3
Throughput: 71.48 requests/s, 11429.83 total tokens/s, 9149.81 output tokens/s
# No deepgemm
Throughput: 142.95 requests/s, 22854.46 total tokens/s, 18297.18 output tokens/s

[LyricZhao]

The older version does not support SM100.

[yewentao256]

The older version does not support SM100.

Yes, I tested the older version on H100 (SM90).

I've found it is a known issue that a lot of people are working on. Like vllm-project/vllm#17934 or https://github.com/neuralmagic/vllm/tree/varun/ht-optimizations, so it has nothing to do with my integration of new DeepGemm.

We temporally don't need to discuss the E2E performance here. I will have other issues or PRs for that if needed in the future

[yewentao256]

Close this issue as fixed