[Doc] Support kimi-k2-w8a8

docs/source/tutorials/index.md

@@ -13,4 +13,5 @@ multi_npu_qwen3_moe
 multi_npu_quantization
 single_node_300i
 multi_node
+multi_node_kimi
 :::

docs/source/tutorials/multi_node.md

@@ -90,12 +90,12 @@ docker run --rm \
 -it $IMAGE bash
 ```
 
+Run the following scripts on two nodes respectively
+
 :::{note}
-Before launch the inference server, ensure some environment variables are set for multi node communication
+Before launch the inference server, ensure the following environment variables are set for multi node communication
 :::
 
-Run the following scripts on two nodes respectively
-
 **node0**
 
 ```shell
@@ -178,7 +178,7 @@ vllm serve /root/.cache/ds_v3 \
 ```
 
 The Deployment view looks like:
-![alt text](../assets/multi_node_dp.png)
+![alt text](../assets/multi_node_dp_deepseek.png)
 
 Once your server is started, you can query the model with input prompts:
 

docs/source/tutorials/multi_node_kimi.md

@@ -0,0 +1,153 @@
+# Multi-Node-DP (Kimi-K2)
+
+## Verify Multi-Node Communication Environment
+
+referring to [multi_node.md](https://vllm-ascend.readthedocs.io/en/latest/tutorials/multi_node.html#verification-process)
+
+## Run with docker
+Assume you have two Atlas 800 A3(64G*16) nodes(or 4 *A2* 8), and want to deploy the `Kimi-K2-Instruct-W8A8` quantitative model across multi-node.
+
+```{code-block} bash
+   :substitutions:
+# Update the vllm-ascend image
+export IMAGE=m.daocloud.io/quay.io/ascend/vllm-ascend:|vllm_ascend_version|
+export NAME=vllm-ascend
+
+# Run the container using the defined variables
+# Note if you are running bridge network with docker, Please expose available ports for multiple nodes communication in advance
+docker run --rm \
+--name $NAME \
+--net=host \
+--device /dev/davinci0 \
+--device /dev/davinci1 \
+--device /dev/davinci2 \
+--device /dev/davinci3 \
+--device /dev/davinci4 \
+--device /dev/davinci5 \
+--device /dev/davinci6 \
+--device /dev/davinci7 \
+--device /dev/davinci8 \
+--device /dev/davinci9 \
+--device /dev/davinci10 \
+--device /dev/davinci11 \
+--device /dev/davinci12 \
+--device /dev/davinci13 \
+--device /dev/davinci14 \
+--device /dev/davinci15 \
+--device /dev/davinci_manager \
+--device /dev/devmm_svm \
+--device /dev/hisi_hdc \
+-v /usr/local/dcmi:/usr/local/dcmi \
+-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
+-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
+-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
+-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
+-v /etc/ascend_install.info:/etc/ascend_install.info \
+-v /mnt/sfs_turbo/.cache:/home/cache \
+-it $IMAGE bash
+```
+
+Run the following scripts on two nodes respectively
+
+:::{note}
+Before launch the inference server, ensure the following environment variables are set for multi node communication
+:::
+
+**node0**
+
+```shell
+#!/bin/sh
+
+# this obtained through ifconfig
+# nic_name is the network interface name corresponding to local_ip
+nic_name="xxxx"
+local_ip="xxxx"
+
+export HCCL_IF_IP=$local_ip
+export GLOO_SOCKET_IFNAME=$nic_name
+export TP_SOCKET_IFNAME=$nic_name
+export HCCL_SOCKET_IFNAME=$nic_name
+export OMP_PROC_BIND=false
+export OMP_NUM_THREADS=100
+export VLLM_USE_V1=1
+export HCCL_BUFFSIZE=1024
+
+# The w8a8 weight can obtained from https://www.modelscope.cn/models/vllm-ascend/Kimi-K2-Instruct-W8A8
+# If you want to the quantization manually, please refer to https://vllm-ascend.readthedocs.io/en/latest/user_guide/feature_guide/quantization.html
+vllm serve /home/cache/weights/Kimi-K2-Instruct-W8A8 \
+--host 0.0.0.0 \
+--port 8004 \
+--data-parallel-size 4 \
+--api-server-count 2 \
+--data-parallel-size-local 2 \
+--data-parallel-address $local_ip \
+--data-parallel-rpc-port 13389 \
+--seed 1024 \
+--served-model-name kimi \
+--quantization ascend \
+--tensor-parallel-size 8 \
+--enable-expert-parallel \
+--max-num-seqs 16 \
+--max-model-len 32768 \
+--max-num-batched-tokens 4096 \
+--trust-remote-code \
+--no-enable-prefix-caching \
+--gpu-memory-utilization 0.9 \
+--additional-config '{"ascend_scheduler_config":{"enabled":true},"torchair_graph_config":{"enabled":true}}'
+```
+
+**node1**
+
+```shell
+#!/bin/sh
+
+nic_name="xxxx"
+local_ip="xxxx"
+
+export HCCL_IF_IP=$local_ip
+export GLOO_SOCKET_IFNAME=$nic_name
+export TP_SOCKET_IFNAME=$nic_name
+export HCCL_SOCKET_IFNAME=$nic_name
+export OMP_PROC_BIND=false
+export OMP_NUM_THREADS=100
+export VLLM_USE_V1=1
+export HCCL_BUFFSIZE=1024
+
+vllm serve /home/cache/weights/Kimi-K2-Instruct-W8A8 \
+--host 0.0.0.0 \
+--port 8004 \
+--headless \
+--data-parallel-size 4 \
+--data-parallel-size-local 2 \
+--data-parallel-start-rank 2 \
+--data-parallel-address $node0_ip \
+--data-parallel-rpc-port 13389 \
+--seed 1024 \
+--tensor-parallel-size 8 \
+--served-model-name kimi \
+--max-num-seqs 16 \
+--max-model-len 32768 \
+--quantization ascend \
+--max-num-batched-tokens 4096 \
+--enable-expert-parallel \
+--trust-remote-code \
+--no-enable-prefix-caching \
+--gpu-memory-utilization 0.92 \
+--additional-config '{"ascend_scheduler_config":{"enabled":true},"torchair_graph_config":{"enabled":true}}'
+```
+
+The Deployment view looks like:
+![alt text](../assets/multi_node_dp_kimi.png)
+
+Once your server is started, you can query the model with input prompts:
+
+```shell
+curl http://{ node0 ip:8004 }/v1/completions \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "kimi",
+        "prompt": "The future of AI is",
+        "max_tokens": 50,
+        "temperature": 0
+    }'
+```

docs/source/user_guide/feature_guide/quantization.md

@@ -8,54 +8,57 @@ Since 0.9.0rc2 version, quantization feature is experimentally supported in vLLM
 
 To quantize a model, users should install [ModelSlim](https://gitee.com/ascend/msit/blob/master/msmodelslim/README.md) which is the Ascend compression and acceleration tool. It is an affinity-based compression tool designed for acceleration, using compression as its core technology and built upon the Ascend platform.
 
-Currently, only the specific tag [modelslim-VLLM-8.1.RC1.b020_001](https://gitee.com/ascend/msit/blob/modelslim-VLLM-8.1.RC1.b020_001/msmodelslim/README.md) of modelslim works with vLLM Ascend. Please do not install other version until modelslim master version is available for vLLM Ascend in the future.
-
 Install modelslim:
 
 ```bash
-git clone https://gitee.com/ascend/msit -b modelslim-VLLM-8.1.RC1.b020_001
+git clone https://gitee.com/ascend/msit
+# Optional, this commit has been verified
+git checkout f8ab35a772a6c1ee7675368a2aa4bafba3bedd1a
+
 cd msit/msmodelslim
 bash install.sh
 pip install accelerate
 ```
 
 ## Quantize model
 
-Take [DeepSeek-V2-Lite](https://modelscope.cn/models/deepseek-ai/DeepSeek-V2-Lite) as an example, you just need to download the model, and then execute the convert command. The command is shown below. More info can be found in modelslim doc [deepseek w8a8 dynamic quantization docs](https://gitee.com/ascend/msit/blob/modelslim-VLLM-8.1.RC1.b020_001/msmodelslim/example/DeepSeek/README.md#deepseek-v2-w8a8-dynamic%E9%87%8F%E5%8C%96).
-
-```bash
-cd example/DeepSeek
-python3 quant_deepseek.py --model_path {original_model_path} --save_directory {quantized_model_save_path} --device_type cpu --act_method 2 --w_bit 8 --a_bit 8  --is_dynamic True
-```
-
 :::{note}
-You can also download the quantized model that we uploaded. Please note that these weights should be used for test only. For example, https://www.modelscope.cn/models/vllm-ascend/DeepSeek-V2-Lite-W8A8
+You can choose to convert the model yourself or use the quantized model we uploaded,
+see https://www.modelscope.cn/models/vllm-ascend/Kimi-K2-Instruct-W8A8
+This conversion process will require a larger CPU memory, please ensure that the RAM size is greater than 2TB
 :::
 
-Once convert action is done, there are two important files generated.
+### Adapts and change
+1. Ascend does not support the `flash_attn` library. To run the model, you need to follow the [guide](https://gitee.com/ascend/msit/blob/master/msmodelslim/example/DeepSeek/README.md#deepseek-v3r1) and comment out certain parts of the code in `modeling_deepseek.py` located in the weights folder.
+2. The current version of transformers does not support loading weights in FP8 quantization format. you need to follow the [guide](https://gitee.com/ascend/msit/blob/master/msmodelslim/example/DeepSeek/README.md#deepseek-v3r1) and delete the quantization related fields from `config.json` in the weights folder
 
-1. [config.json](https://www.modelscope.cn/models/vllm-ascend/DeepSeek-V2-Lite-W8A8/file/view/master/config.json?status=1). Please make sure that there is no `quantization_config` field in it.
+### Generate the w8a8 weights
 
-2. [quant_model_description.json](https://www.modelscope.cn/models/vllm-ascend/DeepSeek-V2-Lite-W8A8/file/view/master/quant_model_description.json?status=1). All the converted weights info are recorded in this file.
+```bash
+cd example/DeepSeek
 
-Here is the full converted model files:
+export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:False
+export MODEL_PATH="/root/.cache/Kimi-K2-Instruct"
+export SAVE_PATH="/root/.cache/Kimi-K2-Instruct-W8A8"
+
+python3 quant_deepseek_w8a8.py --model_path $MODEL_PATH --save_path $SAVE_PATH --batch_size 4
+```
+
+Here is the full converted model files except safetensors:
 
 ```bash
 .
-├── config.json
-├── configuration_deepseek.py
-├── configuration.json
-├── generation_config.json
-├── quant_model_description.json
-├── quant_model_weight_w8a8_dynamic-00001-of-00004.safetensors
-├── quant_model_weight_w8a8_dynamic-00002-of-00004.safetensors
-├── quant_model_weight_w8a8_dynamic-00003-of-00004.safetensors
-├── quant_model_weight_w8a8_dynamic-00004-of-00004.safetensors
-├── quant_model_weight_w8a8_dynamic.safetensors.index.json
-├── README.md
-├── tokenization_deepseek_fast.py
-├── tokenizer_config.json
-└── tokenizer.json
+|-- config.json
+|-- configuration.json
+|-- configuration_deepseek.py
+|-- generation_config.json
+|-- modeling_deepseek.py
+|-- quant_model_description.json
+|-- quant_model_weight_w8a8_dynamic.safetensors.index.json
+|-- tiktoken.model
+|-- tokenization_kimi.py
+`-- tokenizer_config.json
 ```
 
 ## Run the model
@@ -90,10 +93,7 @@ for output in outputs:
 
 ### Online inference
 
-```bash
-# Enable quantization by specifying `--quantization ascend`
-vllm serve {quantized_model_save_path} --served-model-name "deepseek-v2-lite-w8a8" --max-model-len 2048 --quantization ascend --trust-remote-code
-```
+Enable quantization by specifying `--quantization ascend`, for more details, see DeepSeek-V3-W8A8 [tutorial](https://vllm-ascend.readthedocs.io/en/latest/tutorials/multi_node.html)
 
 ## FAQs
 

vllm_ascend/ascend_config.py

@@ -17,7 +17,7 @@
 
 from vllm.logger import logger
 
-TORCHAIR_MODEL_LIST = ["deepseek", "pangu"]
+TORCHAIR_MODEL_LIST = ["deepseek", "pangu", "kimi_k2"]
 
 
 def _check_torchair_supported(model_type: str):