[GPT-OSS] Structure_Tag support for gpt-oss tool-call in cot #25515

Hanchenli · 2025-09-23T20:52:59Z

Purpose

The model sometimes generates function calls in CoT that have not been provided in tool_server. This PR uses structural tag to force model to 1. do not generate function calls when tool_server is not provided. 2. Only generate function that are present in the tool_server.

example custom structural tags: https://gist.github.com/frank-wei/4929118fe0881ee561ce48f15c1f80ed

Test Plan

Testing with @frank-wei

Test Result

vllm serve run: https://gist.github.com/frank-wei/86d37061685fd45e5b831b6583d865d6

see number of turns reduced:

without structural tags: https://gist.github.com/frank-wei/3c89e45e028cb0f61dc56c51f464252f
with structural tags: https://gist.github.com/frank-wei/45daec7615f0c75ab3d97594746324c2

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

mergify · 2025-09-23T20:53:45Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Hanchenli.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

gemini-code-assist

Code Review

This pull request introduces support for structural tags in gpt-oss tool calls, aiming to constrain the model's output to valid function calls. The changes span model configuration, engine arguments, the OpenAI protocol, and a new reasoning parser for gpt-oss. While the overall approach is sound, I've identified a critical bug in gptoss_reasoning_parser.py involving the mutation of a global dictionary, which could lead to incorrect behavior in subsequent requests. Additionally, there's a high-severity bug in a log statement within the same file that could cause runtime errors. I have provided detailed comments and code suggestions to address these issues. The rest of the changes, including new tests and dependency updates, appear to be in good order.

vllm/reasoning/gptoss_reasoning_parser.py

yeqcharlotte · 2025-10-15T17:39:30Z

@frank-wei ci failure is relevant

Signed-off-by: Wei Wei <wwei6@meta.com>

Frank oss try fix

Signed-off-by: Wei Wei <wwei6@meta.com>

Frank oss try fix

frank-wei · 2025-10-16T19:15:26Z

Looks like it is failed in trunk https://buildkite.com/vllm/ci/builds/35163/steps/canvas?sid=0199eb2d-ccad-4e6b-963d-ef868491a020?
Got this link from: https://app.hex.tech/533fe68e-dcd8-4a52-a101-aefba762f581/app/vLLM-CI-030kdEgDv6lSlh1UPYOkWP/latest?selectedStaticCellId=0199ac4b-8cf6-7ee8-8db1-2dff4166cbcf

hmellor · 2025-10-17T09:35:21Z

Yes this Phi3 test has been unreliable on main for a while. cc @DarkLight1337

DarkLight1337 · 2025-10-18T03:46:53Z

I should have fixed it on main already. Let me rebase this PR

…oject#25515) Signed-off-by: Hanchenli <lihanc2002@gmail.com> Signed-off-by: Hanchenli <61769611+Hanchenli@users.noreply.github.com> Signed-off-by: Wei Wei <wwei6@meta.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Wei Wei <wwei6@meta.com> Co-authored-by: Wei Wei <weiweinpu@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

…oject#25515) Signed-off-by: Hanchenli <lihanc2002@gmail.com> Signed-off-by: Hanchenli <61769611+Hanchenli@users.noreply.github.com> Signed-off-by: Wei Wei <wwei6@meta.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Wei Wei <wwei6@meta.com> Co-authored-by: Wei Wei <weiweinpu@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk> Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>

…oject#25515) Signed-off-by: Hanchenli <lihanc2002@gmail.com> Signed-off-by: Hanchenli <61769611+Hanchenli@users.noreply.github.com> Signed-off-by: Wei Wei <wwei6@meta.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Wei Wei <wwei6@meta.com> Co-authored-by: Wei Wei <weiweinpu@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

…oject#25515) Signed-off-by: Hanchenli <lihanc2002@gmail.com> Signed-off-by: Hanchenli <61769611+Hanchenli@users.noreply.github.com> Signed-off-by: Wei Wei <wwei6@meta.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Wei Wei <wwei6@meta.com> Co-authored-by: Wei Wei <weiweinpu@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk> Signed-off-by: xuebwang-amd <xuebwang@amd.com>

…oject#25515) Signed-off-by: Hanchenli <lihanc2002@gmail.com> Signed-off-by: Hanchenli <61769611+Hanchenli@users.noreply.github.com> Signed-off-by: Wei Wei <wwei6@meta.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Wei Wei <wwei6@meta.com> Co-authored-by: Wei Wei <weiweinpu@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk> Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>

…oject#25515) Signed-off-by: Hanchenli <lihanc2002@gmail.com> Signed-off-by: Hanchenli <61769611+Hanchenli@users.noreply.github.com> Signed-off-by: Wei Wei <wwei6@meta.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Wei Wei <wwei6@meta.com> Co-authored-by: Wei Wei <weiweinpu@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

Hanchenli requested review from LucasWilkinson, ProExpertProg, WoosukKwon, aarnphm, alexm-redhat, benchislett, chaunceyjiang, comaniac, hmellor, houseroad, mgoin, njhill, robertgshaw2-redhat, russellb, simon-mo, tlrmchlsmth, yewentao256, youkaichao and ywang96 as code owners September 23, 2025 20:53

mergify bot added ci/build frontend gpt-oss Related to GPT-OSS models structured-output labels Sep 23, 2025

mergify bot added the v1 label Sep 23, 2025

github-project-automation bot added this to gpt-oss Issues & Enhancements and Structured Output Sep 23, 2025

mergify bot added the needs-rebase label Sep 23, 2025

github-project-automation bot moved this to To Triage in gpt-oss Issues & Enhancements Sep 23, 2025

gemini-code-assist bot reviewed Sep 23, 2025

View reviewed changes

vllm/reasoning/gptoss_reasoning_parser.py Outdated Show resolved Hide resolved

vllm/reasoning/gptoss_reasoning_parser.py Outdated Show resolved Hide resolved

frank-wei added 4 commits October 15, 2025 11:09

merge main

7f36c11

Signed-off-by: Wei Wei <wwei6@meta.com>

fix the bug when StructuredOutputsParams is empty for constraints

07173f1

Signed-off-by: Wei Wei <wwei6@meta.com>

pre-commit

78642d0

Signed-off-by: Wei Wei <wwei6@meta.com>

Merge pull request #14 from frank-wei/frank_oss_try_fix

90b5a70

Frank oss try fix

auto-merge was automatically disabled October 15, 2025 21:43
Head branch was pushed to by a user without write access

frank-wei added 3 commits October 15, 2025 18:39

fix a bug when other constraints are not empty

ce880b1

Signed-off-by: Wei Wei <wwei6@meta.com>

pre-commit

a781978

Signed-off-by: Wei Wei <wwei6@meta.com>

Merge pull request #15 from frank-wei/frank_oss_try_fix

677364e

Frank oss try fix

Merge branch 'main' into lhc/oss-try

8be9e91

yeqcharlotte enabled auto-merge (squash) October 18, 2025 04:16

vllm-bot merged commit 7c57254 into vllm-project:main Oct 18, 2025
44 of 52 checks passed

github-project-automation bot moved this from Ready to Done in gpt-oss Issues & Enhancements Oct 18, 2025

github-project-automation bot moved this to Done in Structured Output Oct 18, 2025

Hanchenli mentioned this pull request Oct 20, 2025

[Feature] Support Latest Xgrammar Structural Tag #24708

Closed

5 tasks

alecsolder mentioned this pull request Nov 5, 2025

[Frontend] [gpt-oss] Chat format GD for tool calling with gptoss #28148

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[GPT-OSS] Structure_Tag support for gpt-oss tool-call in cot #25515

[GPT-OSS] Structure_Tag support for gpt-oss tool-call in cot #25515

Uh oh!

Hanchenli commented Sep 23, 2025 •

edited by github-actions bot

Loading

Uh oh!

mergify bot commented Sep 23, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

Uh oh!

yeqcharlotte commented Oct 15, 2025

Uh oh!

frank-wei commented Oct 16, 2025

Uh oh!

hmellor commented Oct 17, 2025

Uh oh!

DarkLight1337 commented Oct 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Uh oh!

[GPT-OSS] Structure_Tag support for gpt-oss tool-call in cot #25515

[GPT-OSS] Structure_Tag support for gpt-oss tool-call in cot #25515

Uh oh!

Conversation

Hanchenli commented Sep 23, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

mergify bot commented Sep 23, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

yeqcharlotte commented Oct 15, 2025

Uh oh!

frank-wei commented Oct 16, 2025

Uh oh!

hmellor commented Oct 17, 2025

Uh oh!

DarkLight1337 commented Oct 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Hanchenli commented Sep 23, 2025 •

edited by github-actions bot

Loading