Fp8 paged attention update

[xiao-llm]

Purpose

Support fp8 mfma instruction with per wrap dynamic quantization for Query to improve performance, which reduces fp8 to fp16 data type conversion cost and improve mfma throughput for MI300x or later accelerators.

Test Plan

• Unit testing:
  export VLLM_ROCM_FP8_MFMA_PAGE_ATTN=1
  pytest -s tests/kernels/attention/test_attention.py
• Benchmark: lm-eval-harness
• LLM model: Meta-Llama-3.1-8B-Instruct
• Dataset: wikitext and gsm8k

Sample script:
export CUDA_VISIBLE_DEVICES=0,1,2,3
export MODEL_DIR=Meta-Llama-3.1-8B-Instruct
export VLLM_USE_V1=0

lm_eval --model vllm
--model_args pretrained=$MODEL_DIR
--tasks gsm8k
--trust_remote_code
--batch_size 8

Test Result

• GSM8K-Llama-3.1-8B

• Wikitext-Llama-3.3-70b

• Unitest

(Optional) Documentation Update

[mergify]

⚠️ The sha of the head commit of this PR conflicts with #22221. Mergify cannot evaluate rules on this PR. ⚠️

[gemini-code-assist]

Code Review

This pull request introduces updates for FP8 paged attention on ROCm. I've identified two critical compilation errors that need to be addressed. The first is a stray #endif directive that would break the build. The second is a variable scoping issue within a loop due to incorrect placement of preprocessor directives, which would also cause a compilation failure. Addressing these issues will ensure the code compiles and functions as intended.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[BowenBao]

cc @gshtras for review and comments

[xiao-llm]

@tlrmchlsmth Can you please help me review this PR?

[gshtras]

Solid job overall.
Could we have unit tests to cover the new fp8 path? Extend the additional rocm attention test.
Another major point is that we need a way to actually enable this, through command line/env/heuristic/etc., and not leave as a dead code.

[xiao-llm]

Solid job overall. Could we have unit tests to cover the new fp8 path? Extend the additional rocm attention test. Another major point is that we need a way to actually enable this, through command line/env/heuristic/etc., and not leave as a dead code.
I can add env setting like: USE_FP8_MFMA, what do you think?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @xiao-llm.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @xiao-llm.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork