[Core] Remove tokenizer group in vLLM #24078

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

zhuohan123 merged 24 commits into main from zhuohan/remove-token-group

Sep 17, 2025

Member

zhuohan123 commented Sep 2, 2025 •

edited by github-actions bot

Loading

Purpose

Tokenizer group is an abstraction introduced in early day vLLM to support the case where different LoRA adapters use different tokenizers. Looking back, LoRA is a niche feature among all vLLM users, and different tokenizers for different LoRAs is a "niche of the niche" feature. However, This niche of the niche feature spreads all around in vLLM code base, which becomes technical debt.

In the long term, I believe it's a good idea to eliminate the use of tokenizer in vLLM core and make most part of the core only works on token-IDs. This reduces our coupling with huggingface tokenizer and will make developing vLLM core easier.

Also see #23474 #23540

Test Plan

Make sure all the existing tests pass.

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.


          [WIP] Remove tokenizer group reference in the main codebase and pass …

349683f

…basic example

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>

mergify bot added documentation frontend structured-output v1 labels

github-project-automation bot added this to Structured Output

zhuohan123 added 5 commits

September 1, 2025 23:16


          fix all non-lora tests

d644792

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>


          mypy fix

49a304c

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>


          fix mypy

3cd8df1

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>


          fix mypy

fda07d7

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>


          remove get_lora_tokenizer

658d84d

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>

mergify bot added the performance label


          mypy

f350583

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>

zhuohan123 marked this pull request as ready for review

September 2, 2025 22:41

zhuohan123 requested review from DarkLight1337, WoosukKwon, aarnphm, alexm-redhat, comaniac, mgoin, njhill, robertgshaw2-redhat, russellb, simon-mo, youkaichao and ywang96 as code owners

September 2, 2025 22:42

zhuohan123 added 2 commits

September 2, 2025 17:24


          remove extra parametmer

ecfb457

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>


          fix mistral tokenizer

39a736e

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>

zhuohan123 requested a review from patrickvonplaten as a code owner

September 3, 2025 02:48

simon-mo added the ready label


          Merge branch 'main' into zhuohan/remove-token-group

4ca5ac9

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>

zhuohan123 enabled auto-merge (squash)

September 16, 2025 23:27

mergify bot removed the needs-rebase label


          fix test error

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>

mergify bot commented Sep 17, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @zhuohan123.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify bot added the needs-rebase label


          Merge branch 'main' into zhuohan/remove-token-group

3e0e590

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>

mergify bot removed the needs-rebase label

mergify bot commented Sep 17, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @zhuohan123.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify bot added the needs-rebase label


          Merge branch 'main' into zhuohan/remove-token-group

2ff7f95

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>

mergify bot removed the needs-rebase label

zhuohan123 merged commit 6c47f6b into main

56 checks passed

zhuohan123 deleted the zhuohan/remove-token-group branch

September 17, 2025 08:43

github-project-automation bot moved this to Done in Structured Output

github-project-automation bot moved this to Done in Tool Calling

adobrzyn mentioned this pull request

CI fix vllm-project/vllm-gaudi#186

Merged

xuechendi pushed a commit to vllm-project/vllm-gaudi that referenced this pull request


          CI fix (#186)

a3dce5c

vllm-project/vllm#24795 and
vllm-project/vllm#24615 and
vllm-project/vllm#24078

---------

Signed-off-by: Agata Dobrzyniewicz <adobrzyniewicz@habana.ai>

zhuohan123 mentioned this pull request

[Core] Remove lora additional vocabulary #23540

Open

mgoin mentioned this pull request

[CI Bugfix] Fix failing test_model_load_with_params tests due to tokenizer refactor #25086

Merged

5 tasks

slokesha pushed a commit to slokesha/vllm-gaudi that referenced this pull request


          CI fix (vllm-project#186)

77a28ac

vllm-project/vllm#24795 and
vllm-project/vllm#24615 and
vllm-project/vllm#24078

---------

Signed-off-by: Agata Dobrzyniewicz <adobrzyniewicz@habana.ai>
Signed-off-by: slokesha <slokeshappa@habana.ai>

FeiDaLI pushed a commit to FeiDaLI/vllm that referenced this pull request


          [Core] Remove tokenizer group in vLLM (vllm-project#24078)

6bd1664

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>

charlifu pushed a commit to ROCm/vllm that referenced this pull request


          [Core] Remove tokenizer group in vLLM (vllm-project#24078)

332a076

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
Signed-off-by: charlifu <charlifu@amd.com>

DarkLight1337 mentioned this pull request

[Performance] model_config.compute_hash is computed every time and introduce overhead in each new multi-modal req #25671

Closed

xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request


          [Core] Remove tokenizer group in vLLM (vllm-project#24078)

a065959

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
Signed-off-by: xuebwang-amd <xuebwang@amd.com>

choprahetarth pushed a commit to Tandemn-Labs/vllm that referenced this pull request


          [Core] Remove tokenizer group in vLLM (vllm-project#24078)

91c5278

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>

sducouedic pushed a commit to sducouedic/vllm that referenced this pull request


          [Core] Remove tokenizer group in vLLM (vllm-project#24078)

811f25e

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>

xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request


          [Core] Remove tokenizer group in vLLM (vllm-project#24078)

13ba472

Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
Signed-off-by: xuebwang-amd <xuebwang@amd.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

aarnphm aarnphm approved these changes

DarkLight1337 Awaiting requested review from DarkLight1337 DarkLight1337 is a code owner

robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

simon-mo Awaiting requested review from simon-mo simon-mo is a code owner

mgoin Awaiting requested review from mgoin mgoin is a code owner

russellb Awaiting requested review from russellb russellb is a code owner

WoosukKwon Awaiting requested review from WoosukKwon

njhill Awaiting requested review from njhill

ywang96 Awaiting requested review from ywang96

comaniac Awaiting requested review from comaniac

alexm-redhat Awaiting requested review from alexm-redhat

youkaichao Awaiting requested review from youkaichao

patrickvonplaten Awaiting requested review from patrickvonplaten patrickvonplaten is a code owner

jeejeelee Awaiting requested review from jeejeelee jeejeelee is a code owner

benchislett Awaiting requested review from benchislett benchislett is a code owner

NickLucche Awaiting requested review from NickLucche NickLucche is a code owner

chaunceyjiang Awaiting requested review from chaunceyjiang chaunceyjiang is a code owner

Labels

documentation frontend llama performance ready structured-output tool-calling v1