[Guide]: Usage on Speculative Decoding and MTP

[MengqingCao]

How to use Speculative Decoding and MTP on vLLM Ascend

Please refer to vLLM official doc on Speculative Decoding as a usage guide.

Note

When using Speculative Decoding and MTP, there are some limits on vllm-ascend compared with vllm:

1. When request preemption is triggered, there exsists precision issue with Speculative Decoding, except for MTP.
2. Speculative Decoding with multi-step preparation on npu is not supported, only support replacing by circle for on cpu to simulate multi-step preparation.
3. Only BatchExpansionTop1Scorer is supported now, MQAScorer is not supported.

[shen-shanshan]

Set VLLM_ASCEND_ACL_OP_INIT_MODE to 1 when using spec decode in V0 engine.

export VLLM_ASCEND_ACL_OP_INIT_MODE=1