[Feature] Apply structured output sampling after reasoning steps in Reasoning models

[xihuai18]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

Only apply constrained sampling only in the answer for reasoning model. i.e. for DeepSeek R1 only enforce grammar inside after </think>
This would make Reasoning models more useful in agent workflow expecting structured output.

Related resources

vllm-project/vllm#12619
vllm-project/vllm#12955

[gaocegege]

vllm-project/vllm#12955 This PR has problems I think. It does not support outlines. I am not sure if we use outlines in SGLang, just FYI.

Ref vllm-project/vllm#14113 & vllm-project/vllm#14114

[xihuai18]

@minleminzui any progress?

[minleminzui]

@minleminzui any progress?

wip，this branch

[thyecust]

I'm also working on this issue.

I'm using the test

import openai
from pydantic import BaseModel, Field
client = openai.Client(base_url=f"http://127.0.0.1:30000/v1", api_key="None")

model_name = "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"

# Define the schema using Pydantic
class Answer(BaseModel):
    name: str = Field(..., pattern=r"^\w+$", description="Name of the answer")
    ans: int = Field(..., description="integer anwser")

response_non_stream = client.chat.completions.create(
    model=model_name,
    messages=[
        {
            "role": "user",
            "content": "What is 1+3?",
        }
    ],
    temperature=0,
    top_p=0.95,
    stream=False,  # Non-streaming
    max_tokens=2048,
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "foo",
            # convert the pydantic model to json schema
            "schema": Answer.model_json_schema(),
        },
    },
    extra_body={"separate_reasoning": True},
)
print("==== Reasoning ====")
print(response_non_stream.choices[0].message.reasoning_content)
print("==== Text ====")
print(response_non_stream.choices[0].message.content)

{
  "name": "Start Server",
  "type": "debugpy",
  "request": "launch",
  "module": "sglang.launch_server",
  "args": [
    "--reasoning-parser",
    "deepseek-r1",
    "--model-path",
    "/root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1-Distill-Qwen-1.5B/snapshots/ad9f0ae0864d7fbcd1cd905e3c6c5b069cc8b562"
  ],
  "console": "integratedTerminal",
  "justMyCode": false,
}

The current behavior is that

• the syntax restriction will start with the first token and
• </think> won't appear, and
• the entire reply will be placed in the reasoning_content section, with
• the content section empty.

The expected result is that the reasoning_content part is the reasoning process and the content part is JSON.

I'm currently reading through the code to understand what the update_regex_vocab_mask function does.

[minleminzui]

I'm also working on this issue.

I'm using the test

import openai
from pydantic import BaseModel, Field
client = openai.Client(base_url=f"http://127.0.0.1:30000/v1", api_key="None")

model_name = "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"

Define the schema using Pydantic

class Answer(BaseModel):
name: str = Field(..., pattern=r"^\w+$", description="Name of the answer")
ans: int = Field(..., description="integer anwser")

response_non_stream = client.chat.completions.create(
model=model_name,
messages=[
{
"role": "user",
"content": "What is 1+3?",
}
],
temperature=0,
top_p=0.95,
stream=False, # Non-streaming
max_tokens=2048,
response_format={
"type": "json_schema",
"json_schema": {
"name": "foo",
# convert the pydantic model to json schema
"schema": Answer.model_json_schema(),
},
},
extra_body={"separate_reasoning": True},
)
print("==== Reasoning ====")
print(response_non_stream.choices[0].message.reasoning_content)
print("==== Text ====")
print(response_non_stream.choices[0].message.content)
{
"name": "Start Server",
"type": "debugpy",
"request": "launch",
"module": "sglang.launch_server",
"args": [
"--reasoning-parser",
"deepseek-r1",
"--model-path",
"/root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1-Distill-Qwen-1.5B/snapshots/ad9f0ae0864d7fbcd1cd905e3c6c5b069cc8b562"
],
"console": "integratedTerminal",
"justMyCode": false,
}
The current behavior is that

• the syntax restriction will start with the first token and
• </think> won't appear, and
• the entire reply will be placed in the reasoning_content section, with
• the content section empty.

The expected result is that the reasoning_content part is the reasoning process and the content part is JSON.

I'm currently reading through the code to understand what the update_regex_vocab_mask function does.

reason parser parsed the reponse, so the </think> won't appear, please see this pr

[thyecust]

The expected result is that the reasoning_content part is the reasoning process and the content part is JSON.
I'm currently reading through the code to understand what the update_regex_vocab_mask function does.

reason parser parsed the reponse, so the won't appear, please see this pr

Thank you, I've noticed that.

Two thoughts on the work you already have in place

1. in python/sglang/srt/managers/scheduler_output_processor_mixin.py we want to compare next_token and think_end, and in my tests this comparison always fails. That means we don't see think_end here.
2. instead of calling fill_vocab_mask in another file, can we make apply_mask_func a lambda function? It will decide whether to work on logits or not depending on whether it's in REASONING or not.

[minleminzui]

The expected result is that the reasoning_content part is the reasoning process and the content part is JSON.
I'm currently reading through the code to understand what the update_regex_vocab_mask function does.

reason parser parsed the reponse, so the won't appear, please see this pr

Thank you, I've noticed that.

Two thoughts on the work you already have in place

1. in python/sglang/srt/managers/scheduler_output_processor_mixin.py we want to compare next_token and think_end, and in my tests this comparison always fails. That means we don't see think_end here.
2. instead of calling fill_vocab_mask in another file, can we make apply_mask_func a lambda function? It will decide whether to work on logits or not depending on whether it's in REASONING or not.

1. the reason why you failed is that the status of Initializing and updating of the matcher issued, you should try to use try_jump_forward and jump_forward_str_state to update the matcher.
2. think_end_id does exist, you can set a log here to see the next_token_id.
3. a lambda function might also work, but you always remember to update the status of the matcher

[thyecust]

Now I have an option that might work, here it is: 78a634e#diff-27419e27638ba26c3b055e9daca00840a52779aa0e76b06c6a2a68d128abffb0.

The results of the test so far is

$ python demo.py
==== Reasoning ====
First, I recognize that the problem is asking for the sum of 1 and 4.

Next, I perform the addition: 1 plus 4 equals 5.

Therefore, the final answer is 5.

==== Text ====

text is 'First, I recognize that the problem is asking for the sum of 1 and 4.\n\nNext, I perform the addition: 1 plus 4 equals 5.\n\nTherefore, the final answer is 5.\n</think>\n\n'

[minleminzui]

Now I have an option that might work, here it is: 78a634e#diff-27419e27638ba26c3b055e9daca00840a52779aa0e76b06c6a2a68d128abffb0.

The results of the test so far is

$ python demo.py
==== Reasoning ====
First, I recognize that the problem is asking for the sum of 1 and 4.

Next, I perform the addition: 1 plus 4 equals 5.

Therefore, the final answer is 5.

==== Text ====
text is 'First, I recognize that the problem is asking for the sum of 1 and 4.\n\nNext, I perform the addition: 1 plus 4 equals 5.\n\nTherefore, the final answer is 5.\n</think>\n\n'

not this case，please test this one

What we need to test is the constraint decoding of the reasoning model, and we need to ensure that the decoding is not constrained in the thinking phase

[thyecust]

Now I have an option that might work, here it is: 78a634e#diff-27419e27638ba26c3b055e9daca00840a52779aa0e76b06c6a2a68d128abffb0.
The results of the test so far is
$ python demo.py
==== Reasoning ====
First, I recognize that the problem is asking for the sum of 1 and 4.
Next, I perform the addition: 1 plus 4 equals 5.
Therefore, the final answer is 5.
==== Text ====
text is 'First, I recognize that the problem is asking for the sum of 1 and 4.\n\nNext, I perform the addition: 1 plus 4 equals 5.\n\nTherefore, the final answer is 5.\n</think>\n\n'

not this case，please test this one

Now the text is

'Okay, so I need to generate a JSON object about the capital of France, which is Paris. Let me think about what information I should include. \n\nFirst, the basic info: the city\'s name is Paris, and it\'s the capital of France. That\'s straightforward. Next, the country it\'s the capital of, which is France, and the country\'s official name, which is the French Republic.\n\nNow, I should include some key facts. Paris is known as the "City of Light" because of its numerous bridges, like the Eiffel Tower and the bridges over the Seine. It\'s also famous for landmarks such as the Louvre Museum, the Arc de Triomphe, and Notre-Dame. I should mention that it\'s a major cultural, economic, and political center.\n\nDemographics-wise, Paris has a population of around 2.1 million, but it\'s one of the most densely populated cities in the world. The prefecture, which is the administrative center, is also Paris. The metropolitan area has a much larger population, maybe around 12 million.\n\nTransportation is another important aspect. Paris has an extensive public transportation system, including the Métro, which is a large subway network. The Château de Paris is a significant landmark, as is the Île de la Cité, which is a group of islands in the Seine.\n\nI should also note that Paris is a global city, attracting millions of visitors annually. It\'s a UNESCO World Heritage Site, which means it\'s recognized for its cultural heritage. The city has a rich history, with the Eiffel Tower being built in 1889, and it\'s been a center of art, literature, and fashion for centuries.\n\nPutting this all together, I\'ll structure the JSON with sections like "Name," "Country," "Official Name," "Key Facts," "Demographics," and "Transportation." Each section will have relevant details. I need to make sure the JSON is properly formatted with commas and brackets, and that the keys are in lowercase with underscores as per JSON standards.\n\nWait, I should double-check some numbers. The population of Paris is around 2.1 million, but the metropolitan area is much larger. The prefecture is indeed Paris, and the Eiffel Tower was completed in 1889. The Louvre is a major museum there. I think I\'ve covered all the main points. Now, I\'ll format this into a JSON structure.\n</think>\n\n'

There should be a bug here. I'll keep looking into it.