[Feature]: Add Support for Guided Decoding (Structured Output)

[shen-shanshan]

Overview

In our roadmap, we plan to support guided decoding in 2025 Q1 as shown here (#71).

Feel free to feedback your issues when using guided decoding with vllm-ascend, and we will try to fix them if we can.

Roadmap

Latest Community News

• [Structured Outputs] Refactor bitmask construction into get_grammar_bitmask vllm#23361
• [PERF] Async structured outputs vllm#23224
• [Perf] Parallelize fill_bitmask to accelerate high-throughput guided decoding vllm#21862
• Frontend: Adding LM Format Enforcer support to V1 engine vllm#22564
• [RFC][core][V1] generalize structured output manager and backends vllm#17503
• [V1] Add structural_tag support using xgrammar vllm#17085
• [V0][V1][Core] Add outlines integration for V1, and update V0 integration. vllm#15975
• [V1][Experimental] Jump-forward decoding vllm#15490
• [V1][Feature] Enable Speculative Decoding with Structured Outputs vllm#14702
• Add support for xgrammar backend on aarch64:
  • [V1] Add regex structured output support with xgrammar vllm#14590
• Add support for reasoning model (DeepSeek-R1):
  • [Bugfix][Structured Output] Support outlines engine with reasoning outputs for DeepSeek R1 vllm#14114
  • [v0][structured output] Support reasoning output vllm#12955

Adaptation for vllm-ascend

• [7/N] Refactor for structured output module:
  • [V1][Structured Output] Minor modification to _validate_structured_output() vllm#16748
  • [V1][Structured Output] Move xgrammar related utils to backend_xgrammar.py vllm#16578
  • [V1][Platform] Remove supports_structured_output() in platform #531
  • [V1][Platform] Add supports_structured_output() method to Platform #475
  • [V1][Structured Output] Add supports_structured_output() method to Platform vllm#16148
  • [Structured Output][Refactor] Move apply_grammar_bitmask() method from ModelRunner to structured output utils vllm#21999
  • [Structured Output] Replace apply_grammar_bitmask() method with that in vllm to avoid maintenance #2524
• [3/N] Bugfix for xgrammar backend:
  • [V1][Structured Output] Clear xgrammar compiler object when engine core shut down to avoid nanobind leaked warning vllm#16954
  • [V1][Structured Output] Add apply_grammar_bitmask() method to model runner #555
  • [Bugfix] Fix grammar_bitmask IndexError caused by outdated apply_grammar_bitmask method #2314
• [2/N] Bugfix for guidance backend:
  • [V1][Structured Output] Update llguidance (>= 0.7.11) to avoid AttributeError (no StructTag)  vllm#17839
  • [v0.9.1][Bugfix] Fix guided decoding invalid backend #2645
• [2/N] CI & Test:
  • [CI] Update guided decoding ut #1312
  • [Structured Output][CI] Add test for outlines backend for structured output in CI #2283
• [3/N] Doc:
  • [Doc] Update Feature Support doc #234
  • [Doc] Update v0.8.4 release note, add contents for structured output feature #576
  • [Doc] Add Structured Output guide #1499
• [3/N] Others:
  • [Structured Output] Remove redundant check for grammar_bitmask #1459
  • [Structured Output] Make the output of structured output example more complete vllm#22481
  • [Structured Output][CI] Update structured output config to sync with upstream #3032

Learning Materials

I have written some posts for better understanding this feature:

• vLLM 学习笔记｜Guided Decoding (V0)
• vLLM 学习笔记｜Guided Decoding (V1)

[wangxiyuan]

Thanks for your investgation. If guided decoding works well with Ascend, please update the feature doc. Thanks

[ej-hw]

Guided Decoding seems to have subpar performance.
~10 tokens/s for normal completion vs ~1 tokens/s for guided decoding.

Steps to reproduce

export IMAGE = quay.io/ascend/vllm-ascend:v0.7.3rc1

docker run --rm \
--name vllm-ascend \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/  \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-p 5566:5566 \
-e PYTORCH_NPU_ALLOC_CONF=max_split_size_mb:256 \
-it $IMAGE bash -c "vllm serve Qwen/Qwen2.5-32B-Instruct --host=0.0.0.0 --port=5566 --api-key=Aicoe123? -tp=2 --gpu-memory-utilization 0.95 --max-model-len 32678 --enforce-eager --served-model-name=qwen2-5-32b-it"

Execution script

from openai import OpenAI
client = OpenAI(
    base_url="XXXX/v1",
    api_key="XXXX",
)

from pydantic import BaseModel
from enum import Enum

class CarType(str, Enum):
    sedan = "sedan"
    suv = "SUV"
    truck = "Truck"
    coupe = "Coupe"


class CarDescription(BaseModel):
    brand: str
    model: str
    story: str
    car_type: CarType


json_schema = CarDescription.model_json_schema()

completion = client.chat.completions.create(
    model="qwen2-5-32b-it",
    messages=[
        {
            "role": "user",
            "content": "Generate a JSON with the brand, model, story and car_type of the most iconic car from the 90's",
        }
    ],
    extra_body={"guided_json": json_schema, 
        "guided_decoding_backend": "xgrammar",},
)
print(completion.choices[0].message.content)

Other info:

Also, this warning may be relevant:
('Warning: torch.save with "_use_new_zipfile_serialization = False" is not recommended for npu tensor, which may bring unexpected errors and hopefully set "_use_new_zipfile_serialization = True"', 'if it is necessary to use this, please
convert the npu tensor to cpu tensor for saving')

[shen-shanshan]

@ej-hw

This is due to this PR vllm-project/vllm#13894 is not merged, you can modify the code locally, and use xgrammar>=0.1.14 for a try.

[wilschoo]

@ej-hw

This is due to this PR vllm-project/vllm#13894 is not merged, you can modify the code locally, and use xgrammar>=0.1.14 for a try.

This have been tested using xgrammar (0.1.14 and 0.1.15) and the fallback to outlines code block is commented out.

Outcome was that now xgrammar is used but an error stating "CUDA_HOME not set" was observed

[shen-shanshan]

@ej-hw
This is due to this PR vllm-project/vllm#13894 is not merged, you can modify the code locally, and use xgrammar>=0.1.14 for a try.

This have been tested using xgrammar (0.1.14 and 0.1.15) and the fallback to outlines code block is commented out.

Outcome was that now xgrammar is used but an error stating "CUDA_HOME not set" was observed

Which version of vllm do you use? note that V1 currently only supports xgrammar, find mote details here.

Please show more details for analysis.

[wilschoo]

@ej-hw
This is due to this PR vllm-project/vllm#13894 is not merged, you can modify the code locally, and use xgrammar>=0.1.14 for a try.

This have been tested using xgrammar (0.1.14 and 0.1.15) and the fallback to outlines code block is commented out.
Outcome was that now xgrammar is used but an error stating "CUDA_HOME not set" was observed

Which version of vllm do you use? note that V1 currently only supports xgrammar, find mote details here.

Please show more details for analysis.

xgrammar==0.1.14
torch==2.5.1
torch-npu==2.5.1.dev20250320
vllm==0.7.3+empty
vllm_ascend==0.7.3rc1

[shen-shanshan]

@wilschoo

I have tried this with your version, and I find xgrammar called apply_token_bitmask_inplace_cuda, which is so wierd (cuda is not available in NPU environment).

After commented the code show below in xgrammar/kernels/__init__.py, it can work:

"""The kernels for XGrammar."""

import torch

from .apply_token_bitmask_inplace_cpu import apply_token_bitmask_inplace_cpu

apply_token_bitmask_inplace_kernels = {"cpu": apply_token_bitmask_inplace_cpu}

__all__ = ["apply_token_bitmask_inplace_kernels"]

# if torch.cuda.is_available():
#     from .apply_token_bitmask_inplace_cuda import apply_token_bitmask_inplace_cuda

#     apply_token_bitmask_inplace_kernels["cuda"] = apply_token_bitmask_inplace_cuda

# try:
#     from .apply_token_bitmask_inplace_triton import (  # isort: skip
#         apply_token_bitmask_inplace_triton,
#     )

#     apply_token_bitmask_inplace_kernels["triton"] = apply_token_bitmask_inplace_triton
# except ImportError:
#     # If triton is not installed, we can still use the CPU and CUDA implementations.
#     pass

[shen-shanshan]

@wilschoo You can use main branch of vllm and vllm-ascend for a try, it works well with xgrammar 0.16.0.

[wilschoo]

@wilschoo You can use main branch of vllm and vllm-ascend for a try, it works well with xgrammar 0.16.0.

what are the vllm and vllm-ascend versions that you are using?

[shen-shanshan]

@wilschoo You can use main branch of vllm and vllm-ascend for a try, it works well with xgrammar 0.16.0.

what are the vllm and vllm-ascend versions that you are using?

vllm==0.8.3.dev168+g18ed3132d.empty
vllm-ascend==0.1.dev119+g7beb433.d20250331

[wilschoo]

@wilschoo You can use main branch of vllm and vllm-ascend for a try, it works well with xgrammar 0.16.0.

what are the vllm and vllm-ascend versions that you are using?

vllm==0.8.3.dev168+g18ed3132d.empty
vllm-ascend==0.1.dev119+g7beb433.d20250331

I have pulled and installed the main branch for both vllm and vllm_ascend:

vllm==0.1.dev1+g30d6a01.empty
vllm_ascend=0.1.dev1+g78083d4
xgrammar==0.1.16

By using my serve command

vllm serve Qwen/Qwen2.5-32B-Instruct -tp=2 --gpu-memory-utilization 0.95 --max-model-len 32678 --enforce-eager --guided-decoding-backend=xgrammar

my TPOT is now seeing ~12 tokens/s compared to ~1 token/s on previous vllm, vllm_ascend versions