Skip to content

[Bugfix] Fix Mistral guided generation using xgrammar #15704

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Mar 30, 2025
Merged

[Bugfix] Fix Mistral guided generation using xgrammar #15704

merged 5 commits into from
Mar 30, 2025

Conversation

@juliendenize
Copy link
Contributor

@juliendenize juliendenize commented Mar 28, 2025 •
edited by github-actions bot
Loading

This PR aims to fix guided generation for mistral models based on the tekken tokenizer.

FIX #15551

@juliendenize
Fix Mistral guided generation using xgrammar
402034e 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
@github-actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@mergify mergify bot added the v1 label Mar 28, 2025
mgoin
mgoin reviewed Mar 28, 2025
View reviewed changes
Copy link
Member

@mgoin mgoin left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@russellb do we not have a test case for Mistral on V1? @juliendenize maybe we could add a test case to tests/v1/entrypoints/llm/test_struct_output_generate.py?

Also a user in the issue reports a failure for V0 as well (#15551 (comment)), can this be resolved too?

@juliendenize
Add unit tests
f101eb8 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
@mergify
Copy link

mergify bot commented Mar 28, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @juliendenize.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Mar 28, 2025
@juliendenize
Merge branch 'main' into fix_mistral_guided_generation
c39d68c 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
@mergify mergify bot removed the needs-rebase label Mar 28, 2025
@juliendenize
Copy link
Contributor Author

@russellb do we not have a test case for Mistral on V1? @juliendenize maybe we could add a test case to tests/v1/entrypoints/llm/test_struct_output_generate.py?

Also a user in the issue reports a failure for V0 as well (#15551 (comment)), can this be resolved too?

I pushed tests for the V1 version, had to merge with main.

Concerning the V0 I'll investigate :)

@russellb russellb added the ready ONLY add when PR is ready to merge/full CI is needed label Mar 28, 2025
mgoin
mgoin approved these changes Mar 28, 2025
View reviewed changes
Copy link
Member

@mgoin mgoin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! We can cover the V0 issue in another PR if this is ready to merge

@mgoin mgoin enabled auto-merge (squash) March 28, 2025 19:49
@DarkLight1337
Copy link
Member

Can you fix the merge conflict?

@juliendenize
Merge branch 'main' into fix_mistral_guided_generation
dcadb2f 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
@juliendenize
Remove mistral tokenizer_mode for auto mode
d06c50a 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
auto-merge was automatically disabled March 29, 2025 15:27

Head branch was pushed to by a user without write access

@juliendenize
Copy link
Contributor Author

Can you fix the merge conflict?

Done, kuddos on whoever reduced the testing of the structured output BTW 😄

@vllm-bot vllm-bot merged commit 6909a76 into vllm-project:main Mar 30, 2025
29 of 32 checks passed
@hibukipanim
Copy link

thanks for fixing and merging 🙏 !

Alex4210987 pushed a commit to LeiWang1999/vllm-bitblas that referenced this pull request Apr 5, 2025
@juliendenize @Alex4210987
[Bugfix] Fix Mistral guided generation using xgrammar (vllm-project#1…
bbe8de4 
…5704)

Signed-off-by: Julien Denize <julien.denize@mistral.ai>
Signed-off-by: xinyuxiao <xinyuxiao2024@gmail.com>
lulmer pushed a commit to lulmer/vllm that referenced this pull request Apr 7, 2025
@juliendenize @lulmer
[Bugfix] Fix Mistral guided generation using xgrammar (vllm-project#1…
9366565 
…5704)

Signed-off-by: Julien Denize <julien.denize@mistral.ai>
Signed-off-by: Louis Ulmer <ulmerlouis@gmail.com>
@ckhordiasma ckhordiasma mentioned this pull request Apr 17, 2025
Closed
lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Apr 29, 2025
@juliendenize @lk-chen
[Bugfix] Fix Mistral guided generation using xgrammar (vllm-project#1…
be1356d 
…5704)

Signed-off-by: Julien Denize <julien.denize@mistral.ai>
shreyankg pushed a commit to shreyankg/vllm that referenced this pull request May 3, 2025
@juliendenize @shreyankg
[Bugfix] Fix Mistral guided generation using xgrammar (vllm-project#1…
9d8695d 
…5704)

Signed-off-by: Julien Denize <julien.denize@mistral.ai>
RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request May 12, 2025
@juliendenize
[Bugfix] Fix Mistral guided generation using xgrammar (vllm-project#1…
9bd2857 
…5704)

Signed-off-by: Julien Denize <julien.denize@mistral.ai>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mgoin mgoin mgoin approved these changes

@russellb russellb Awaiting requested review from russellb russellb is a code owner

Assignees

No one assigned

Labels

ready ONLY add when PR is ready to merge/full CI is needed v1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: Structured Output not working with MistralTokenizer (vLLM 0.8.2, V1)

6 participants

@juliendenize @DarkLight1337 @hibukipanim @mgoin @russellb @vllm-bot