[New Model]: Llama 4

[Yikun]

The model to consider.

https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct
https://www.modelscope.cn/models/LLM-Research/Llama-4-Scout-17B-16E-Instruct

The closest model vllm already supports.

No response

What's your difficulty of supporting the model you want?

No response

[Yikun]

11. torch_npu._npu_rotary_embedding timeout or trap error

Please refer to issue #693 to decrease the value of max_position_embeddings in config.json file

10. torch_npu._npu_rotary_embedding RuntimeError: ACL stream synchronize failed, error code:507034

Issue: #495 (comment)

9. NotImplementedError: Custom routing function is not supported now

Issue: #495
Try to support custom routing function: #495 (comment)

diff --git a/vllm_ascend/ops/fused_moe.py b/vllm_ascend/ops/fused_moe.py
index 2061b68..88aa52d 100644
--- a/vllm_ascend/ops/fused_moe.py
+++ b/vllm_ascend/ops/fused_moe.py
@@ -235,9 +235,6 @@ def select_experts(
     assert hidden_states.shape[0] == router_logits.shape[0], (
         "Number of tokens mismatch")

-    if custom_routing_function is not None:
-        raise NotImplementedError(
-            "Custom routing function is not supported now")

     if scoring_func == "softmax":
         # NOTE: vLLM use dtype=torch.float here
@@ -272,9 +269,15 @@ def select_experts(
                                                 k=top_k,
                                                 dim=-1,
                                                 sorted=False)
-    else:
+    elif custom_routing_function is None:
         topk_weights, topk_ids = topk_weights.topk(top_k, dim=-1)
         topk_weights = topk_weights.to(hidden_states.dtype)
+    else:
+        topk_weights, topk_ids = custom_routing_function(
+            hidden_states=hidden_states,
+            gating_output=router_logits,
+            topk=top_k,
+            renormalize=renormalize)

     # Required by npu_moe_init_routing
     topk_ids = topk_ids.to(torch.int32)

8. Try A2 (B3, 64GB * 8) machine:

vllm serve LLM-Research/Llama-4-Scout-17B-16E-Instruct --max-model-len 4096 -tp 8

Loading weights took 248.95 seconds,
45628/ 65536 per card
42339 * 8 ≈ 331GB mememory .

7. Try A2 (B4, 32GB * 8), raise a OOM, copying model to a A2 (B3, 64GB * 8) machine.

6. Do a hotfix on attention meta

• question in my mind: why not call attention v1 workflow?

diff --git a/vllm_ascend/attention/attention.py b/vllm_ascend/attention/attention.py
index 178d381..a7ca480 100644
--- a/vllm_ascend/attention/attention.py
+++ b/vllm_ascend/attention/attention.py
@@ -590,6 +590,7 @@ class AscendAttentionBackendImpl(AttentionImpl):
         blocksparse_params: Optional[Dict[str, Any]] = None,
         logits_soft_cap: Optional[float] = None,
         attn_type: str = AttentionType.DECODER,
+        use_irope: bool = False,
     ) -> None:
         self.num_heads = num_heads
         self.head_size = head_size

5. [Bug]: RuntimeError: operator torchvision::nms does not exist

#477

4. Running

vllm serve LLM-Research/Llama-4-Scout-17B-16E-Instruct --max-model-len 4096 -tp 8

3. Download models (203GB * 1024 / 10MB/s / 60 / 60 = 5.7H)

pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
pip install modelscope
modelscope download --model LLM-Research/Llama-4-Scout-17B-16E-Instruct

export VLLM_USE_MODELSCOPE=True
export PYTORCH_NPU_ALLOC_CONF=max_split_size_mb:256

2. Fail to install vllm ascend from source due to numpy VersionConflict

#473

1. Start with CANN image + vllm v0.8.3 + vLLM Ascend main

Follow on: https://vllm-ascend.readthedocs.io/en/latest/installation.html

export IMAGE=m.daocloud.io/quay.io/ascend/cann:8.0.0-910b-ubuntu22.04-py3.10
# export IMAGE=quay.io/ascend/cann:8.0.0-910b-ubuntu22.04-py3.10
docker run --rm \
--name vllm-ascend \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-p 8000:8000 \
-it $IMAGE bash

docker exec -it vllm-ascend /bin/bash

## 2. Install vLLM latest release
apt update  -y; apt install -y gcc g++ libnuma-dev vim git
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
pip install vllm

## 3. Install vLLM Ascend main
# Install vLLM Ascend
git clone --depth 1 --branch main https://ghfast.top/https://github.com/vllm-project/vllm-ascend
# git clone  --depth 1 --branch main  https://gitee.com/mirrors/vllm-ascend
cd vllm-ascend
pip install -e . --extra-index https://download.pytorch.org/whl/cpu/

mkdir pta
cd pta
wget https://pytorch-package.obs.cn-north-4.myhuaweicloud.com/pta/Daily/v2.5.1/20250320.3/pytorch_v2.5.1_py310.tar.gz
tar -xvf pytorch_v2.5.1_py310.tar.gz
pip install ./torch_npu-2.5.1.dev20250320-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

[cxcxflying]

11. torch_npu._npu_rotary_embedding timeout or trap error
Please refer to issue #693 to decrease the value of max_position_embeddings in config.json file

[wangxiyuan]

llama4 is supported in main branch now.