[New Model]: Qwen3-Next

[Stevepyl]

The model to consider.

Please add Qwen3-Next-80B-A3B-Instruct and Qwen3-Next-80B-A3B-Thinking support

The closest model vllm already supports.

No response

What's your difficulty of supporting the model you want?

No response

[yzy1996]

FYI: vllm-project/vllm#24526

[Yikun]

Thanks for feedback. We are still working on Qwen-3 Next support, it will be ready soon this week.

We will also release v0.10.2rc1 after the Qwen3 Next ready.

[MaoJianwei]

Thanks, detail logs here:

#2889

[Yikun]

20250915: We are still working on PR review on
PR：#2917 (include vllm-project/vllm#24486)
Doc: #2916

issue to be addressed: #2922

Guide

export IMAGE=quay.nju.edu.cn/ascend/vllm-ascend:main
docker run --rm \
--name vllm-ascend-qwen3 \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-p 8000:8000 \
-it $IMAGE bash

# Load model from ModelScope to speed up download
export VLLM_USE_MODELSCOPE=True

# Install triton
wget https://vllm-ascend.obs.cn-north-4.myhuaweicloud.com/vllm-ascend/Ascend-BiSheng-toolkit_aarch64.run
chmod a+x Ascend-BiSheng-toolkit_aarch64.run
./Ascend-BiSheng-toolkit_aarch64.run --install
source /usr/local/Ascend/8.3.RC1/bisheng_toolkit/set_env.sh

wget https://vllm-ascend.obs.cn-north-4.myhuaweicloud.com/vllm-ascend/triton_ascend-3.2.0.dev20250914-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl
pip install triton_ascend-3.2.0.dev20250914-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl

# cd source path
cd /vllm-workspace/vllm-ascend/

# checkout this PR:
https://github.com/vllm-project/vllm-ascend/pull/2917

# Run
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct --tensor-parallel-size 4 --enforce-eager

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-Next-80B-A3B-Instruct",
  "messages": [
    {"role": "user", "content": "Give me a short introduction to large language models."}
  ],
  "temperature": 0.6,
  "top_p": 0.95,
  "top_k": 20,
  "max_tokens": 4096
}'

[yingxudeng]

I am trying to run the Qwen3-Next model on an x86-based machine with the identifier 910b2. I have reviewed the available documentation and repository instructions (e.g., README), but they seem to be primarily focused on ARM architectures (like the 910b).