[Bug]: Structured Output not working with MistralTokenizer (vLLM 0.8.2, V1)

[hibukipanim]

Despite #14625 which aims to support structured-outputs with mistral tokenizr, it doesn't seem to work with vllm 0.8.2 which includes this PR.

to reproduce:

vllm serve mistralai/Ministral-8B-Instruct-2410 --tokenizer_mode mistral --config_format mistral --load_format mistral --max-model-len 4096 --gpu-memory-utilization 0.95

(the memory limits args are just to make it run in RTX 4090)

and a simplified version of the Structured Outputs example from vllm docs:

from pydantic import BaseModel
from enum import Enum

from openai import OpenAI

class CarDescription(BaseModel):
    brand: str
    model: str
    car_type: str

json_schema = CarDescription.model_json_schema()

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="-",
)
completion = client.chat.completions.create(
    model="mistralai/Ministral-8B-Instruct-2410",
    messages=[
        {
            "role": "user",
            "content": "Generate a JSON with the brand, model and car_type of the most iconic car from the 90's",
        }
    ],
    extra_body={"guided_json": json_schema},
    seed=42
)
print(completion.choices[0].message.content)

runs for very long and then outputs broken json e.g.:

{
{ 	"bݨ" 	: 	"Ford", 	" 		il		od		od		od		od		od		od		od		od ...

b.t.w maybe it's expected but first request is blocked few seconds when the server prints:

[2025-03-26 16:13:24] INFO tekken.py:114: Adding special tokens <SPECIAL_20>, ..., <SPECIAL_999>
[2025-03-26 16:13:24] INFO tekken.py:293: Vocab size: 150000
[2025-03-26 16:13:24] INFO tekken.py:297: Cutting vocab to first 130072 tokens.
INFO 03-26 16:13:24 [loggers.py:80] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
/home/user/code/play-vllm/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2059: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(

INFO 03-26 16:13:34 [loggers.py:80] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 1 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%

[lukas-kuhl]

+1

I've tried vllm with V1 Engine 0 and 1. When using V1 engine, I get the same behavior as @hibukipanim, when the request completes, it takes really long, and if it completes, I get nonesense.

When using V0 I get this error for a request:

Unsupported tokenizer type: <class 'vllm.transformers_utils.tokenizers.mistral.MistralTokenizer'>

I'm using these parameters to start the vllm/vllm-openai:v0.8.2 container:

--host 0.0.0.0 --port 8000 --model mistralai/Mistral-Small-3.1-24B-Instruct-2503 --tokenizer-mode mistral --config-format mistral --config-format mistral --load-format mistral --limit-mm-per-prompt image=10,video=0 --max-model-len 64000 --tool-call-parser mistral --enable-auto-tool-choice

[mmoskal]

Out of curiosity, is there a particular reason --tokenizer_mode mistral needs to be used? At least the guidance backend seems to work correctly with --tokenizer-mode auto.

At least in case of llguidance, it only supports the "Fast Tokenizer" since it uses the same Rust tokenizers library under the hood. Currently, xgrammar only requires token_id -> bytes mapping for each token. In general, llguidance could also work with just this, but to support "fast-forward" tokens, it needs a real tokenization function, see https://github.com/guidance-ai/llguidance/blob/main/docs/fast_forward.md

While fast-forwarding is currently not supported in vLLM, hopefully it will be in future (it provides ~5-20% GPU saving for JSON schema generation).

[DavidOpVa]

I have the same issue with mistralai/Mistral-Small-3.1-24B-Instruct-2503.

curl "<my-vllm-address>/v1/chat/completions" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer XXX" \
    -d '{
        "model": "mistralai/Mistral-Small-3.1-24B-Instruct-2503",
        "messages": [
            {
                "role": "user",
                "content": "Classify this sentiment: vLLM is wonderful!"
            }
        ],
        "guided_choice": ["positive", "negative"]
    }'

Doesn't follow the guided proposition and return random garbage.

[hibukipanim]

@mmoskal regarding

Out of curiosity, is there a particular reason --tokenizer_mode mistral needs to be used? At least the guidance backend seems to work correctly with --tokenizer-mode auto.

Some mistral models are uploaded to HF in the mistral format (not HF one) so they need to be loaded with --tokenizer_mode mistral --config_format mistral --load_format mistral AFAIK. Specifically Mistral 3.1 is not yet supported in it's HF format but is supported in the mistral format.