[Kernel] Triton Configs for Fp8 Block Quantization

[robertgshaw2-redhat]

SUMMARY:

• configs for block quantized fp8 (for DeepSeekV3)
• adapted from add configs for block fp8 related kernels sgl-project/sglang#2628 (thanks @zhyncs and @HandH1998)
• updates with Update Triton configs for block fp8 kernels sgl-project/sglang#2641

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[NikolaBorisov]

@simon-mo This PR looks good an works well.
It is missing however this code

diff --git a/setup.py b/setup.py
index 7dfcec7f..308500a3 100644
--- a/setup.py
+++ b/setup.py
@@ -603,7 +603,11 @@ if _build_custom_ops():
     ext_modules.append(CMakeExtension(name="vllm._C"))
 
 package_data = {
-    "vllm": ["py.typed", "model_executor/layers/fused_moe/configs/*.json"]
+    "vllm": [
+        "py.typed",
+        "model_executor/layers/fused_moe/configs/*.json",
+        "model_executor/layers/quantization/utils/configs/*.json"
+    ]
 }
 
 if _no_device():

[tlrmchlsmth]

Looks good once the new json files are added to package_data in setup.py

[robertgshaw2-redhat]

On 8xH200:

• pr

python3 benchmark_throughput.py --model $MODEL --tensor-parallel-size 8 --max-model-len 8192 --dataset ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 100 --trust-remote-code --tokenizer deepseek-ai/DeepSeek-R1

>> Throughput: 2.27 requests/s, 1000.55 total tokens/s, 483.31 output tokens/s

• main

python3 benchmark_throughput.py --model $MODEL --tensor-parallel-size 8 --max-model-len 8192 --dataset ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 100 --trust-remote-code --tokenizer deepseek-ai/DeepSeek-R1

>> Throughput: 1.66 requests/s, 731.39 total tokens/s, 353.29 output tokens/s

[mgoin]

Excellent, LGTM

[robertgshaw2-redhat]

Mixtral looks good (performance and accuracy)

[robertgshaw2-redhat]

MODEL=/data/nm/models/DeepSeek-R1

lm_eval --model vllm \
    --model_args "pretrained=$MODEL,tokenizer=deepseek-ai/DeepSeek-R1,tensor_parallel_size=8,dtype=auto,gpu_memory_utilization=0.95,trust_remote_code=True,max_model_len=16384,trust_remote_code=True" \
    --task gsm8k --batch_size 100

• pr

vllm (pretrained=/data/nm/models/DeepSeek-R1,tokenizer=deepseek-ai/DeepSeek-R1,tensor_parallel_size=8,dtype=auto,gpu_memory_utilization=0.95,trust_remote_code=True,max_model_len=16384,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 100
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9538|±  |0.0058|
|     |       |strict-match    |     5|exact_match|↑  |0.9538|±  |0.0058|

• main

vllm (pretrained=/data/nm/models/DeepSeek-R1,tokenizer=deepseek-ai/DeepSeek-R1,tensor_parallel_size=8,dtype=auto,gpu_memory_utilization=0.95,trust_remote_code=True,max_model_len=16384,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 100
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9462|±  |0.0062|
|     |       |strict-match    |     5|exact_match|↑  |0.9462|±  |0.0062|