[Qwen][ROCm] Flash Attention Rotary Embeddings

[vllmellm]

Purpose

Qwen VL models previously relied on a basic PyTorch implementation for applying rotary positional embeddings on ROCm architectures. This PR adds a ROCm specialisation to use flash_attn module's Triton-based operation which makes it much faster as seen in the benchmarking results below:

benchmark_serving script

python3 vllm/benchmarks/benchmark_serving.py  \
--backend openai-chat   \
--model Qwen/Qwen2-VL-7B-Instruct     \
--endpoint /v1/chat/completions   \
--dataset-name hf   \
--dataset-path lmarena-ai/VisionArena-Chat   \
--hf-split train   \
--num-prompts 2000 \

Metric	Qwen2.5-VL-7B-Instruct-After	Qwen2.5-VL-7B-Instruct-Before	Qwen2-VL-7B-Instruct-After	Qwen2-VL-7B-Instruct-Before
Successful requests	2000	2000	2000	2000
Benchmark duration (s)	1345.53	1813.95	1061.45	1245.72
Total input tokens	128725	128725	128725	128725
Total generated tokens	226206	226462	198558	197729
Request throughput (req/s)	1.49	1.10	1.88	1.61
Output token throughput (tok/s)	168.12	124.84	187.06	158.73
Total token throughput (tok/s)	263.78	195.81	308.34	262.06
Mean TTFT (ms)	518860.51	780409.88	393523.69	525409.53
Median TTFT (ms)	442684.40	734388.24	335331.87	524210.13
P99 TTFT (ms)	1339478.98	1784936.03	1034260.36	1217034.75
Mean TPOT (ms)	4134.86	5809.08	3351.70	4390.97
Median TPOT (ms)	4724.09	7592.68	3179.76	5423.96
P99 TPOT (ms)	9092.23	9980.45	8516.70	7897.34
Mean ITL (ms)	4826.60	6175.72	4119.97	4689.50
Median ITL (ms)	575.49	7872.18	470.59	520.93
P99 ITL (ms)	15527.60	15521.51	15152.06	14973.82

Test Plan

Evaluate accuracy with mistral-eval using chartqa dataset on Qwen2.5-VL-7B-Instruct and Qwen2-VL-7B-Instruct

Test Result

Qwen2.5-VL-7B-Instruct-Before

Metrics:
{
    "explicit_prompt_relaxed_correctness": 0.8676,
    "anywhere_in_answer_relaxed_correctness": 0.868
}

Qwen2-VL-7B-Instruct-Before

{
    "explicit_prompt_relaxed_correctness": 0.4552,
    "anywhere_in_answer_relaxed_correctness": 0.6968
}

Qwen2.5-VL-7B-Instruct-After

Metrics:
{
    "explicit_prompt_relaxed_correctness": 0.8632,
    "anywhere_in_answer_relaxed_correctness": 0.8632
}

Qwen2-VL-7B-Instruct-After

Metrics:
{
    "explicit_prompt_relaxed_correctness": 0.462,
    "anywhere_in_answer_relaxed_correctness": 0.7012
}

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[vllmellm]

@DarkLight1337 would appreciate if you can review.

[DarkLight1337]

Can you reorganize the imports to be in the correct order?

[vllmellm]

it is ordered using isort. Arranging it alphabetically causes isort to complain

[DarkLight1337]

Hmm weird, I thought isort would put it with the rest of the stdlib imports near L5

[xuechendi]

@DarkLight1337 @vllmellm , I found this PR introduced a bug to non_cuda/non_rocm, may you take a look of the fixing I provided here:
#26123