[BugFix] Stopgap - Flashinfer Autotuner + GPT-OSS + DP/TP

[varun-sundar-rabindranath]

Purpose

Running the Flashinfer autotuner when,

• using data-parallel or tensor-parallel, and
• using a flashinfer mxfp4 backend, and
• eager-mode

Causes the engine startup to fail with,

(EngineCore_DP1 pid=3074289)   File "/home/varun-sundar-rabindranath/code/vllm/vllm-test/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP1 pid=3074289)     return forward_call(*args, **kwargs)
(EngineCore_DP1 pid=3074289)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=3074289)   File "/home/varun-sundar-rabindranath/code/vllm/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1168, in forward
(EngineCore_DP1 pid=3074289)     fused_out = self._fused_experts(
(EngineCore_DP1 pid=3074289)                 ^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=3074289)   File "/home/varun-sundar-rabindranath/code/vllm/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1021, in _fused_experts
(EngineCore_DP1 pid=3074289)     self.fused_experts.apply(
(EngineCore_DP1 pid=3074289)   File "/home/varun-sundar-rabindranath/code/vllm/vllm/model_executor/layers/fused_moe/trtllm_moe.py", line 135, in apply
(EngineCore_DP1 pid=3074289)     trtllm_fp4_block_scale_routed_moe(**kwargs)
(EngineCore_DP1 pid=3074289)   File "/home/varun-sundar-rabindranath/code/vllm/vllm-test/lib/python3.12/site-packages/flashinfer/fused_moe/core.py", line 1850, in trtllm_fp4_block_scale_routed_moe
(EngineCore_DP1 pid=3074289)     return get_trtllm_moe_sm100_module().trtllm_fp4_block_scale_moe(
(EngineCore_DP1 pid=3074289)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=3074289)   File "/home/varun-sundar-rabindranath/code/vllm/vllm-test/lib/python3.12/site-packages/flashinfer/fused_moe/core.py", line 1348, in trtllm_fp4_block_scale_moe_op
(EngineCore_DP1 pid=3074289)     _, tactic = tuner.choose_one(
(EngineCore_DP1 pid=3074289)                 ^^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=3074289)   File "/home/varun-sundar-rabindranath/code/vllm/vllm-test/lib/python3.12/site-packages/flashinfer/autotuner.py", line 457, in choose_one
(EngineCore_DP1 pid=3074289)     profiles = self._generate_optimization_profiles(tuning_config, inputs)
(EngineCore_DP1 pid=3074289)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=3074289)   File "/home/varun-sundar-rabindranath/code/vllm/vllm-test/lib/python3.12/site-packages/flashinfer/autotuner.py", line 643, in _generate_optimization_profiles
(EngineCore_DP1 pid=3074289)     assert len(opt_shapes) > 0, "Empty tuning buckets are not allowed"
(EngineCore_DP1 pid=3074289)            ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP1 pid=3074289) AssertionError: Empty tuning buckets are not allowed

This error was initially thought to be related to DP + certain choices for the MXFP4 backend. This PR updates the skip condition.

Test Plan and Test Result

B200 DP

VLLM_ALL2ALL_BACKEND="deepep_high_throughput" canhazgpu run -g2 -- vllm serve openai/gpt-oss-20b --data-parallel-size 2 --tensor-parallel-size 1 --enable-expert-parallel --no-enable-prefix-caching --port 9010

• PR Pass
• main Fail

VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8=1 canhazgpu run -g2 -- vllm serve openai/gpt-oss-20b --data-parallel-size 2 --tensor-parallel-size 1 --enable-expert-parallel --no-enable-prefix-caching --port 9010 --enforce-eager

• PR Pass
• main Pass

VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8_CUTLASS=1 canhazgpu run -g2 -- vllm serve openai/gpt-oss-20b --data-parallel-size 2 --tensor-parallel-size 1 --enable-expert-parallel --no-enable-prefix-caching --port 9010 --enforce-eager

• PR Pass
• main fail

VLLM_USE_FLASHINFER_MOE_MXFP4_BF16=1 canhazgpu run -g2 -- vllm serve openai/gpt-oss-20b --data-parallel-size 2 --tensor-parallel-size 1 --enable-expert-parallel --no-enable-prefix-caching --port 9010 --enforce-eager

• PR Pass
• main pass

B200 TP

VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8=1 canhazgpu run -g2 -- vllm serve openai/gpt-oss-20b --data-parallel-size 1 --tensor-parallel-size 2 --no-enable-prefix-caching --port 9010 --enforce-eager

• PR Pass
• main fail

VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8_CUTLASS=1 canhazgpu run -g2 -- vllm serve openai/gpt-oss-20b --data-parallel-size 1 --tensor-parallel-size 2 --no-enable-prefix-caching --port 9010 --enforce-eager

• PR Pass
• main fail

VLLM_USE_FLASHINFER_MOE_MXFP4_BF16=1 canhazgpu run -g2 -- vllm serve openai/gpt-oss-20b --data-parallel-size 1 --tensor-parallel-size 2 --no-enable-prefix-caching --port 9010 --enforce-eager

• PR Pass
• main fail

H100 DP

VLLM_USE_FLASHINFER_MOE_MXFP4_BF16=1 canhazgpu run -g2 -- vllm serve openai/gpt-oss-20b --data-parallel-size 2 --tensor-parallel-size 1 --enable-expert-parallel --no-enable-prefix-caching --port 9010 --enforce-eager

• PR pass
• main pass

H100 TP

VLLM_USE_FLASHINFER_MOE_MXFP4_BF16=1 canhazgpu run -g2 -- vllm serve openai/gpt-oss-20b --data-parallel-size 1 --tensor-parallel-size 2 --no-enable-prefix-caching --port 9010 --enforce-eager

• PR pass
• main fail

[gemini-code-assist]

Code Review

This pull request addresses a bug where the Flashinfer autotuner fails during engine startup under specific conditions: when using data or tensor parallelism with a flashinfer mxfp4 backend in eager mode. The fix correctly identifies this problematic configuration by checking for tensor or data parallelism, the usage of any flashinfer mxfp4 backend, and whether the execution mode is eager. The changes are well-implemented, using clearly named boolean variables to improve readability of the condition. The fix appears correct and complete based on the problem description and test results.

[varun-sundar-rabindranath]

@nvpohanh I verified that with the PR,

VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8=1   vllm serve openai/gpt-oss-20b --data-parallel-size 2 --tensor-parallel-size 1 --enable-expert-parallel   --no-enable-prefix-caching  --port 9010

flashinfer autotuning happens.

[varun-sundar-rabindranath]

cc @zyongye @mgoin PTAL! Thanks.

[nvpohanh]

@elvischenv has also verified this. Thanks for the quick fix!

[nvpohanh]

@mgoin It would be great if we can include this in v0.11.1 to avoid unexpected performance regression. Thanks!

[mgoin]

gpt-oss test looks flaky https://buildkite.com/organizations/vllm/analytics/suites/ci-1/tests/e242093d-1033-8042-a229-87b736d010dd?branch=main&period=7days