Skip to content

Commit 7b301e0

Browse files
wangxiyuanwangxiyuan
committed
[Doc] Add install doc
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
1 parent 46977f9 commit 7b301e0

File tree

3 files changed

+152
-26
lines changed

3 files changed

+152
-26
lines changed

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ By using vLLM Ascend plugin, popular open-source models, including Transformer-l
3939
* PyTorch >= 2.4.0, torch-npu >= 2.4.0
4040
* vLLM (the same version as vllm-ascend)
4141

42-
Find more about how to setup your environment step by step in [here](docs/installation.md).
42+
Find more about how to setup your environment step by step in [here](docs/source/installation.md).
4343

4444
## Getting Started
4545

@@ -68,7 +68,7 @@ Run the following command to start the vLLM server with the [Qwen/Qwen2.5-0.5B-I
6868
vllm serve Qwen/Qwen2.5-0.5B-Instruct
6969
curl http://localhost:8000/v1/models
7070
```
71-
**Please refer to [official docs](./docs/index.md) for more details.**
71+
**Please refer to [official docs](https://vllm-ascend.readthedocs.io/en/latest/) for more details.**
7272

7373
## Contributing
7474
See [CONTRIBUTING](docs/source/developer_guide/contributing.md) for more details, which is a step-by-step guide to help you set up development environment, build and test.

‎README.zh.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ vLLM 昇腾插件 (`vllm-ascend`) 是一个让vLLM在Ascend NPU无缝运行的
3939
* PyTorch >= 2.4.0, torch-npu >= 2.4.0
4040
* vLLM (与vllm-ascend版本一致)
4141

42-
在[此处](docs/installation.md)，您可以了解如何逐步准备环境。
42+
在[此处](docs/source/installation.md)，您可以了解如何逐步准备环境。
4343

4444
## 开始使用
4545

@@ -69,7 +69,7 @@ vllm serve Qwen/Qwen2.5-0.5B-Instruct
6969
curl http://localhost:8000/v1/models
7070
```
7171

72-
**请参阅 [官方文档](./docs/index.md)以获取更多详细信息**
72+
**请参阅 [官方文档](https://vllm-ascend.readthedocs.io/en/latest/)以获取更多详细信息**
7373

7474
## 贡献
7575
有关更多详细信息，请参阅 [CONTRIBUTING](docs/source/developer_guide/contributing.zh.md)，可以更详细的帮助您部署开发环境、构建和测试。

‎docs/source/installation.md‎

Lines changed: 148 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,51 @@
11
# Installation
22

3-
## Dependencies
4-
| Requirement | Supported version | Recommended version | Note |
5-
| ------------ | ------- | ----------- | ----------- |
6-
| Python | >= 3.9 | [3.10](https://www.python.org/downloads/) | Required for vllm |
7-
| CANN | >= 8.0.RC2 | [8.0.RC3](https://www.hiascend.com/developer/download/community/result?module=cann&cann=8.0.0.beta1) | Required for vllm-ascend and torch-npu |
8-
| torch-npu | >= 2.4.0 | [2.5.1rc1](https://gitee.com/ascend/pytorch/releases/tag/v6.0.0.alpha001-pytorch2.5.1) | Required for vllm-ascend |
9-
| torch | >= 2.4.0 | [2.5.1](https://github.com/pytorch/pytorch/releases/tag/v2.5.1) | Required for torch-npu and vllm required |
3+
This document describes how to install vllm-ascend manually.
104

11-
## Prepare Ascend NPU environment
5+
## Requirements
126

13-
Below is a quick note to install recommended version software:
7+
- OS: Linux
8+
- Python: 3.10 or higher
9+
- A hardware with Ascend NPU. It's usually the Atlas 800 A2 series.
10+
- Software:
1411

15-
### Containerized installation
12+
| Software | Supported version | Note |
13+
| ------------ | ----------------- | ---- |
14+
| CANN | >= 8.0.0.beta1 | Required for vllm-ascend and torch-npu |
15+
| torch-npu | >= 2.5.1rc1 | Required for vllm-ascend |
16+
| torch | >= 2.5.1 | Required for torch-npu and vllm |
1617

17-
You can use the [container image](https://hub.docker.com/r/ascendai/cann) directly with one line command:
18+
## Configure a new environment
19+
20+
Before installing the package, you need to make sure that the firmware and driver for NPU is installed correctly. i.e. `npu-smi` command is available.
21+
22+
> Tips: following the instructions provided in the [Ascend Installation Guide](https://ascend.github.io/docs/sources/ascend/quick_install.html) can help you to set up the environment easily.
23+
24+
Once it's done, you can read either **Set up using Python** or **Set up using Docker** section to install and use vllm-ascend.
25+
26+
If you want to install vllm-ascend on a bare environment by hand, you need install CANN first, otherwise, you can skip this step if using container environment.
1827

1928
```bash
29+
# Create a virtual environment
30+
python -m venv vllm-ascend-env
31+
source vllm-ascend-env/bin/activate
32+
33+
# Install required python packages.
34+
pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple attrs numpy==1.24.0 decorator sympy cffi pyyaml pathlib2 psutil protobuf scipy requests absl-py wheel typing_extensions
35+
36+
# Download and install the CANN package.
37+
wget https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%208.0.0/Ascend-cann-toolkit_8.0.0_linux-aarch64.run
38+
sh Ascend-cann-toolkit_8.0.0_linux-aarch64.run --full
39+
wget https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%208.0.0/Ascend-cann-kernels-910b_8.0.0_linux-aarch64.run
40+
sh Ascend-cann-kernels-910b_8.0.0_linux-aarch64.run --full
41+
```
42+
43+
## Set up using Python
44+
45+
Please make sure that CANN is installed. It can be done by **Configure a new environment** step. Or by using an CANN container directly.
46+
47+
```
48+
# Setup a CANN container using docker
2049
docker run \
2150
--name vllm-ascend-env \
2251
--device /dev/davinci1 \
@@ -28,28 +57,125 @@ docker run \
2857
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
2958
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
3059
-v /etc/ascend_install.info:/etc/ascend_install.info \
31-
-it quay.io/ascend/cann:8.0.rc3.beta1-910b-ubuntu22.04-py3.10 bash
60+
-it quay.io/ascend/cann:8.0.0.beta1-910b-ubuntu22.04-py3.10 bash
3261
```
3362

34-
You do not need to install `torch` and `torch_npu` manually, they will be automatically installed as `vllm-ascend` dependencies.
63+
Then you can install vllm-ascend from **pre-built wheel** or **source code**.
64+
65+
### Install from Pre-built wheels (Not support yet)
66+
67+
1. Install vllm
68+
69+
```bash
70+
git clone --depth 1 --branch v0.7.1 https://github.com/vllm-project/vllm
71+
cd vllm
72+
VLLM_TARGET_DEVICE=empty pip install . -f https://download.pytorch.org/whl/torch/
73+
```
74+
75+
2. Install vllm-ascend
76+
77+
```bash
78+
pip install vllm-ascend -f https://download.pytorch.org/whl/torch/
79+
```
3580

36-
### Manual installation
81+
### Install from source code
3782

38-
Or follow the instructions provided in the [Ascend Installation Guide](https://ascend.github.io/docs/sources/ascend/quick_install.html) to set up the environment.
83+
1. Install vllm
3984

40-
## Building
85+
```bash
86+
git clone https://github.com/vllm-project/vllm
87+
cd vllm
88+
VLLM_TARGET_DEVICE=empty pip install . -f https://download.pytorch.org/whl/torch/
89+
```
4190

42-
### Build Python package from source
91+
2. Install vllm-ascend
92+
93+
```bash
94+
git clone https://github.com/vllm-project/vllm-ascend.git
95+
cd vllm-ascend
96+
pip install -e . -f https://download.pytorch.org/whl/torch/
97+
```
98+
99+
## Set up using Docker
100+
101+
> Tips: CANN, torch, torch_npu, vllm and vllm_ascend are pre-installed in the Docker image already.
102+
103+
### Pre-built images (Not support yet)
104+
105+
Just pull the image and run it with bash.
43106

44107
```bash
45-
git clone https://github.com/vllm-project/vllm-ascend.git
46-
cd vllm-ascend
47-
pip install -e .
108+
docker pull quay.io/ascend/vllm-ascend:latest
109+
110+
docker run \
111+
--name vllm-ascend-env \
112+
--device /dev/davinci1 \
113+
--device /dev/davinci_manager \
114+
--device /dev/devmm_svm \
115+
--device /dev/hisi_hdc \
116+
-v /usr/local/dcmi:/usr/local/dcmi \
117+
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
118+
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
119+
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
120+
-v /etc/ascend_install.info:/etc/ascend_install.info \
121+
-it quay.io/ascend/vllm-ascend:0.7.1rc1 bash
48122
```
49123

50-
### Build container image from source
124+
### Build image from source
125+
126+
If you want to build the docker image from main branch, you can do it by following steps:
127+
51128
```bash
52129
git clone https://github.com/vllm-project/vllm-ascend.git
53130
cd vllm-ascend
54-
docker build -t vllm-ascend-dev-image -f ./Dockerfile .
131+
132+
docker build -t vllm-ascend-dev-image:latest -f ./Dockerfile .
133+
134+
docker run \
135+
--name vllm-ascend-env \
136+
--device /dev/davinci1 \
137+
--device /dev/davinci_manager \
138+
--device /dev/devmm_svm \
139+
--device /dev/hisi_hdc \
140+
-v /usr/local/dcmi:/usr/local/dcmi \
141+
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
142+
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
143+
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
144+
-v /etc/ascend_install.info:/etc/ascend_install.info \
145+
-it vllm-ascend-dev-image:latest bash
146+
```
147+
148+
## Extra information
149+
150+
### Verify installation
151+
152+
Create and Run a simple inference test. Then `example.py` is:
153+
154+
```
155+
from vllm import LLM, SamplingParams
156+
157+
prompts = [
158+
"Hello, my name is",
159+
"The president of the United States is",
160+
"The capital of France is",
161+
"The future of AI is",
162+
]
163+
164+
# Create a sampling params object.
165+
sampling_params = SamplingParams(max_tokens=100, temperature=0.0)
166+
# Create an LLM.
167+
llm = LLM(model="facebook/opt-125m")
168+
169+
# Generate texts from the prompts.
170+
outputs = llm.generate(prompts, sampling_params)
171+
for output in outputs:
172+
prompt = output.prompt
173+
generated_text = output.outputs[0].text
174+
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
175+
```
176+
177+
Then run:
178+
```bash
179+
# export VLLM_USE_MODELSCOPE=true to speed up download if huggingface is not reachable.
180+
python example.py
55181
```

0 commit comments

Comments
 (0)