[Kernel] Fuse computation of g and beta for Gated Delta Net

[ZJY0516]

Purpose

Fuse computation of g and beta into one triton kernel for Gated Delta Net and avoid rearrange

Perf test

vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct --enable-expert-parallel -tp 4

vllm bench serve \
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
--dataset-name random \
--num-prompts 32 \
--random-input-len 2048 \
--random-output-len 1024

hardware setup: H20
TLDR: throughput 6123.13 -> 6237.58
this pr

============ Serving Benchmark Result ============
Successful requests:                     32        
Failed requests:                         0         
Benchmark duration (s):                  15.54     
Total input tokens:                      65536     
Total generated tokens:                  31377     
Request throughput (req/s):              2.06      
Output token throughput (tok/s):         2019.51   
Peak output token throughput (tok/s):    2356.00   
Peak concurrent requests:                32.00     
Total Token throughput (tok/s):          6237.58   
---------------Time to First Token----------------
Mean TTFT (ms):                          1101.49   
Median TTFT (ms):                        1112.77   
P99 TTFT (ms):                           1848.29   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          14.09     
Median TPOT (ms):                        13.96     
P99 TPOT (ms):                           15.35     
---------------Inter-token Latency----------------
Mean ITL (ms):                           14.07     
Median ITL (ms):                         13.38     
P99 ITL (ms):                            14.00     
==================================================

main

============ Serving Benchmark Result ============
Successful requests:                     32        
Failed requests:                         0         
Benchmark duration (s):                  15.82     
Total input tokens:                      65536     
Total generated tokens:                  31310     
Request throughput (req/s):              2.02      
Output token throughput (tok/s):         1979.59   
Peak output token throughput (tok/s):    2325.00   
Peak concurrent requests:                32.00     
Total Token throughput (tok/s):          6123.13   
---------------Time to First Token----------------
Mean TTFT (ms):                          1109.19   
Median TTFT (ms):                        1120.40   
P99 TTFT (ms):                           1858.64   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          14.35     
Median TPOT (ms):                        14.22     
P99 TPOT (ms):                           15.31     
---------------Inter-token Latency----------------
Mean ITL (ms):                           14.34     
Median ITL (ms):                         13.66     
P99 ITL (ms):                            14.08     
==================================================

Accuracy test

lm_eval --model local-completions --model_args model=Qwen/Qwen3-Next-80B-A3B-Instruct,base_url=http://localhost:8000/v1/completions -t gsm8k --num_fewshot 5 --batch_size 250

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8499|±  |0.0098|
|     |       |strict-match    |     5|exact_match|↑  |0.8067|±  |0.0109|

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces an effective optimization by fusing the computation of g and beta for the Gated Delta Net into a single Triton kernel. This change reduces kernel launch overhead, leading to the performance improvements shown in the pull request description. The implementation correctly integrates the sigmoid calculation for beta into the fused_gdn_gating_kernel and updates the calling Python function fused_gdn_gating to handle the fused operation. The code is clean, and the logic appears sound. I have no further recommendations as the changes are well-implemented.

[vadiklyutiy]

@ZJY0516
could you check lm_eval to be sure in correctness.
could you add to descr what HW did you use for tests.

[ZJY0516]

@vadiklyutiy Done

[vadiklyutiy]

looks good to me

[vadiklyutiy]

PS torch.compile would be definitely better here. But as I know we still miss necessary interfaces.

[simon-mo]

Vadim stamped