[XPU] Fix MOE DP accuracy issue on XPU

[faaany]

Purpose

• add dispatch and combine methods in XpuCommunicator to fix the MOE model accuracy issue on XPU
• add argument to let users disable expert_parallel
• make naive the default all2all_backend on XPU

Test Plan

VLLM_WORKER_MULTIPROC_METHOD=spawn python examples/offline_inference/data_parallel.py --enforce-eager --model="ibm-research/PowerMoE-3b" --dp-size=2 --tp-size=2 --disable-expert-parallel

Before:

DP rank 1, Prompt: 'Hello, my name is', Generated text: ' a boatatatatatatatatatatatatatatatatatat'
DP rank 1, Prompt: 'The president of the United States is', Generated text: ' from two of his the fact that it is the gratatatatatatatatat'
DP rank 1, Prompt: 'The capital of France is', Generated text: ' not necessarily to to the capital capital of India. India and France have the same capital capital'
DP rank 1, Prompt: 'The future of AI is', Generated text: '. The now is. Future of AI is. Future of AI is. Future of AI is.'
DP rank 1, Prompt: 'Hello, my name is', Generated text: ' my name,’’,’’,’’,’’,’’,’’'
DP rank 0, Prompt: 'Hello, my name is', Generated text: ' is my,, my name, my my my my my my my my my'
DP rank 0, Prompt: 'The president of the United States is', Generated text: ' is is is is is is a0 to the the the the the the the'
DP rank 0, Prompt: 'The capital of France is', Generated text: ' and France is is  58 and and and and and and\n\n'
DP rank 0, Prompt: 'The future of AI is', Generated text: '””””””””””””””’s'
DP rank 0, Prompt: 'Hello, my name is', Generated text: ' is is is’ my,, me, my\xa0, my way my way' 

After:

DP rank 0, Prompt: 'Hello, my name is', Generated text: ' [your name], and I would like to explore the possibility of a career change'
DP rank 0, Prompt: 'The president of the United States is', Generated text: ' the commander in chief of the armed forces of the United States,'
DP rank 0, Prompt: 'The capital of France is', Generated text: ' Paris. The head of state is a president, who is elected'
DP rank 0, Prompt: 'The future of AI is', Generated text: ' bright and exciting. We will use the lessons learned from our models to'
DP rank 0, Prompt: 'Hello, my name is', Generated text: ' Tristan.\nI am a backpacker who has been traveling for'
DP rank 1, Prompt: 'Hello, my name is', Generated text: ' [your name], and I would like to explore the possibility of a career change into the field of'
DP rank 1, Prompt: 'The president of the United States is', Generated text: ' the commander in chief of the armed forces of the United States, including all branches of'
DP rank 1, Prompt: 'The capital of France is', Generated text: ' Paris. The head of state is a president, who is elected by universal suffr'
DP rank 1, Prompt: 'The future of AI is', Generated text: ' bright and exciting. We will use the lessons learned from our models to further enhance our capability'
DP rank 1, Prompt: 'Hello, my name is', Generated text: ' Tristan.\nI am a backpacker who has been traveling for about three years now'

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[faaany]

cc @jikunshang @yma11 @chaojun-zhang

[gemini-code-assist]

Code Review

This pull request addresses an accuracy issue with Mixture-of-Experts (MoE) models on XPU devices when using data parallelism. The core of the fix involves implementing the dispatch and combine communication primitives in the XpuCommunicator, which are essential for expert parallelism. The changes correctly delegate these operations to an all2all_manager. Additionally, the PR makes the all2all backend on XPU robust by defaulting to the naive implementation, which is the only one currently supported, and warns the user if a different backend is configured. The modifications to the data parallelism example script to make enable_expert_parallel a configurable argument is also a good improvement for flexibility. The provided test results clearly demonstrate the effectiveness of the fix. The changes are well-implemented and follow existing patterns in the codebase. Overall, this is a solid contribution to improve XPU support.