[Performance]: vllm-ascend + mindie-turbo Performance Optimization

[shen-shanshan]

Doc: https://docs.google.com/document/d/1F4mnGa8XDmj37vCbNS6Zso6sg4TzXr4RTkihTbhjLYw/

Motivation

To achieve ultimate performance on vllm-asend v0.7.3 with mindie-turbo 2.0rc1, we have make efforts to optimize our codes, configs, etc.

Performance test:

• Qwen 2.5 7B

Separate single item optimizations

• v0.7.3 base image (vllm-ascend) + Ubuntu: m.daocloud.io/quay.io/ascend/vllm-ascend:v0.7.3
• Each improvement
• Performance test
  • Qwen2.5-7B-Instruct
• Doc

Overall optimizations

• e2e

1. Compiler Optimization (@MaskerPRC)

• python
• pytorch
• torch-npu

Depends on specific model:

• tiny enhancement (LTO)
• about 27% enhancement (PGO for specific model)
• (today) Step 1: Doc (must be ready)
• Step 2: Can be reproduce in dockerfile

2. OS Optimization (@celestialli)

• Mem allocator etc, performance
• (0514) Step 1: Doc (must be ready)
• Step 2: Can be reproduce in dockerfile (host / container)

3. torch-npu Optimization (@Potabk)

• Memory
• Scheduler

4. CANN Optimization (@Potabk)

• HCCL ENV
• mindie-turbo ENV

5. vllm-ascend Optimization (@shen-shanshan)

V1 Ascend Scheduler:

• Implementation: [V1] Add v0 style schedule into v1 engine. #512
• Usage: [Guide]: Usage on AscendScheduler in vLLM Ascend #788

Offline inference test using Qwen2.5-7B-Instruct:

• V1 without Ascend Scheduler: speed input: 8.05 toks/s, output: 146.39 toks/s
• V1 with Ascend Scheduler: speed input: 8.86 toks/s, output: 161.15 toks/s, but have accuracy problem need to be fixed.

6. Dockerfile (@MaskerPRC)

...

[celestialli]

OS Optimization update: we can switch to tcmalloc to allocate memory. I have tested ubuntu and openEuler. Sleep mode run normally with tcmalloc.

[shen-shanshan]

V1 Ascend Scheduler benchmark using #864:

python -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct \
--tensor-parallel-size 1 \
--swap-space 16 \
--disable-log-stats \
--disable-log-requests \
--load-format dummy \
--additional-config '{"ascend_scheduler_config":{}}'

Disable Ascend Scheduler:

============ Serving Benchmark Result ============
Successful requests:                     200       
Benchmark duration (s):                  187.89    
Total input tokens:                      40000     
Total generated tokens:                  25600     
Request throughput (req/s):              1.06      
Output token throughput (tok/s):         136.25    
Total Token throughput (tok/s):          349.14    
---------------Time to First Token----------------
Mean TTFT (ms):                          46.19     
Median TTFT (ms):                        46.47     
P99 TTFT (ms):                           65.37     
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          22.81     
Median TPOT (ms):                        22.76     
P99 TPOT (ms):                           24.57     
---------------Inter-token Latency----------------
Mean ITL (ms):                           22.81     
Median ITL (ms):                         22.47     
P99 ITL (ms):                            28.09     
==================================================

Enable Ascend Scheduler:

============ Serving Benchmark Result ============
Successful requests:                     200       
Benchmark duration (s):                  187.57    
Total input tokens:                      40000     
Total generated tokens:                  25600     
Request throughput (req/s):              1.07      
Output token throughput (tok/s):         136.48    
Total Token throughput (tok/s):          349.74    
---------------Time to First Token----------------
Mean TTFT (ms):                          41.91     
Median TTFT (ms):                        41.29     
P99 TTFT (ms):                           57.76     
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          21.24     
Median TPOT (ms):                        21.29     
P99 TPOT (ms):                           22.21     
---------------Inter-token Latency----------------
Mean ITL (ms):                           21.24     
Median ITL (ms):                         21.14     
P99 ITL (ms):                            25.03     
==================================================

Summary: we get lower latency of TTFT and TPOT with Ascend Scheduler.

[Potabk]

3. torch-npu optimization

3.1 Memory optimization

• PYTORCH_NPU_ALLOC_CONF
  Upper limit of memory block splitting allowed (MB), Setting this parameter can prevent large memory blocks from being split

export PYTORCH_NPU_ALLOC_CONF="max_split_size_mb:250"

• PYTORCH_NPU_ALLOC_CONF
  When operators on the communication stream have dependencies, they all need to be ended before being released for reuse. The logic of multi-stream reuse is to release the memory on the communication stream in advance so that the computing stream can be reused.

export PYTORCH_NPU_ALLOC_CONF="expandable_segments:True"

3.2 Schedule optimization

3.2.1 TASK_QUEUE_ENABLE

Optimize operator delivery queue, this will affect the memory peak value, and may degrade if the memory is tight.

export TASK_QUEUE_ENABLE=2

3.2.2 CPU_AFFINITY_CONF

Will greatly improve the CPU bottleneck model and ensure the same performance for the NPU bottleneck model.

export CPU_AFFINITY_CONF=1

4. CANN optimization

4.1 HCCL

These environment variables will improve performance in certain scenarios, please refer to https://www.hiascend.com
HCCL_INTRA_ROCE_ENABLE

HCCL_RDMA_TC

HCCL_RDMA_SL

HCCL_BUFFSIZE

4.2 mindie-turbo optimization

These environment variables will improve performance in certain scenarios, please refer to mindie_turbo

benchmark result

before:

============ Serving Benchmark Result ============
Successful requests:                     200       
Benchmark duration (s):                  188.76    
Total input tokens:                      40000     
Total generated tokens:                  25600     
Request throughput (req/s):              1.06      
Output token throughput (tok/s):         135.62    
Total Token throughput (tok/s):          347.53    
---------------Time to First Token----------------
Mean TTFT (ms):                          86.63     
Median TTFT (ms):                        84.45     
P99 TTFT (ms):                           126.74    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          32.15     
Median TPOT (ms):                        32.14     
P99 TPOT (ms):                           35.01     
---------------Inter-token Latency----------------
Mean ITL (ms):                           32.15     
Median ITL (ms):                         31.05     
P99 ITL (ms):                            63.87     
==================================================

============ Serving Benchmark Result ============
Successful requests:                     200       
Benchmark duration (s):                  188.76    
Total input tokens:                      40000     
Total generated tokens:                  25600     
Request throughput (req/s):              1.06      
Output token throughput (tok/s):         135.63    
Total Token throughput (tok/s):          347.54    
---------------Time to First Token----------------
Mean TTFT (ms):                          82.34     
Median TTFT (ms):                        82.51     
P99 TTFT (ms):                           97.76     
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          31.46     
Median TPOT (ms):                        31.31     
P99 TPOT (ms):                           34.48     
---------------Inter-token Latency----------------
Mean ITL (ms):                           31.46     
Median ITL (ms):                         30.46     
P99 ITL (ms):                            63.43     
==================================================

these optimization improvements in ttft

[shen-shanshan]

see #878

[Yikun]

Install python

RUN mv /tmp/*.so* /usr/local/lib && \
    tar -zxvf /tmp/py311_bisheng.*  -C /usr/local/  && \
    mv  /usr/local/py311_bisheng/  /usr/local/python && \
    sed -i "1c#\!/usr/local/python/bin/python3.11" /usr/local/python/bin/pip3 && \
    sed -i "1c#\!/usr/local/python/bin/python3.11" /usr/local/python/bin/pip3.11 && \
    ln -sf  /usr/local/python/bin/python3  /usr/bin/python && \
    ln -sf  /usr/local/python/bin/python3  /usr/bin/python3 && \
    ln -sf  /usr/local/python/bin/python3.11  /usr/bin/python3.11 && \
    ln -sf  /usr/local/python/bin/pip3  /usr/bin/pip3 && \ 
    ln -sf  /usr/local/python/bin/pip3  /usr/bin/pip && \
    rm -rf /tmp/*
    

ENV PATH=/usr/bin:/usr/local/python/bin:$PATH
ENV LANG C.UTF-8

[Yikun]

ENV TASK_QUEUE_ENABLE=2
ENV OMP_PROC_BIND=false
ENV LD_LIBRARY_PATH=/usr/local/python/lib/python3.11/site-packages/mindie_turbo:$LD_LIBRARY_PATH

ENV TASK_QUEUE_ENABLE=2
ENV OMP_PROC_BIND=false
ENV LD_LIBRARY_PATH=/usr/local/python/lib/python3.11/site-packages/mindie_turbo:$LD_LIBRARY_PATH

COPY --from=cann_installer /usr/local/Ascend /usr/local/Ascend
COPY --from=cann_installer /etc/Ascend /etc/Ascend
COPY --from=bisheng_installer /mindie_turbo/BiShengCompiler-4.1.0-aarch64-linux/lib/libomp.so  /usr/local/lib/libomp.so

[Yikun]

RUN pip install /tmp/torch_npu-*.whl --force-reinstall --no-deps && \
    pip install /tmp/torch-2.5.1*.whl --force-reinstall --no-deps && \
    pip install pandas gevent sacrebleu rouge_score pybind11 pytest && \
    pip cache purge && \
    rm -rf /tmp/*

[Yikun]

echo "deb http://mirrors.tools.huawei.com/ubuntu-ports/ jammy main restricted universe multiverse" > /etc/apt/sources.list && \
    echo "deb-src http://mirrors.tools.huawei.com/ubuntu-ports/ jammy main restricted universe multiverse" >> /etc/apt/sources.list && \
    echo "deb http://mirrors.tools.huawei.com/ubuntu-ports/ jammy-updates main restricted universe multiverse" >> /etc/apt/sources.list && \
    echo "deb-src http://repo.huaweicloud.com/ubuntu-ports/ jammy-updates main restricted universe multiverse" >> /etc/apt/sources.list && \
    echo "deb http://repo.huaweicloud.com/ubuntu-ports/ jammy-backports main restricted universe multiverse" >> /etc/apt/sources.list && \
    echo "deb-src http://repo.huaweicloud.com/ubuntu-ports/ jammy-backports main restricted universe multiverse" >> /etc/apt/sources.list && \
    echo "deb http://ports.ubuntu.com/ubuntu-ports/ jammy-security main restricted universe multiverse" >> /etc/apt/sources.list && \
    echo "deb-src http://ports.ubuntu.com/ubuntu-ports/ jammy-security main restricted universe multiverse" >> /etc/apt/sources.list