Support sequence parallel MOE after upstream #24982

[wuxun-zhang]

After vllm-project/vllm#24982 merged, sequence parallel MOE will be turned on when enable_expert_parallel=True, tp_size > 1 and dp_size > 1. Since for Gaudi, there is no choice for VLLM_ALL2ALL_BACKEND, we can not easily bypass it. So this PR aims to support the feature.

class ParallelConfig:

  @property
    def use_sequence_parallel_moe(self) -> bool:
        return (envs.VLLM_ALL2ALL_BACKEND
                in ("allgather_reducescatter", "naive",
                    "deepep_high_throughput", "deepep_low_latency")
                and self.enable_expert_parallel
                and self.tensor_parallel_size > 1
                and self.data_parallel_size > 1)

Update:
No hard requirement on vllm-project/vllm#25828

[github-actions]

✅ CI Passed

All checks passed successfully against the following vllm commit:
c242c98031b87d00999e07dbb4aa9b2a70798c6c

[xuechendi]

@wuxun-zhang , we are trying to make bs,seq_len, hidden_state not hard-requirement for HPU.
Please check with this new flag: https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/extension/features.py#L89

Please try with add GraniteMOE to flatten input_ids to 1D.

Meanwwhile, I also discussed with @kzawora-intel , since more and more models asserts on 2D input, we might change to 1D as default once performance validated.

[wuxun-zhang]

Please try with add GraniteMOE to flatten input_ids to 1D.

Thanks, it works. Just updated.

[github-actions]

✅ CI Passed

All checks passed successfully against the following vllm commit:
c242c98031b87d00999e07dbb4aa9b2a70798c6c