[Bugfix] Token type and position embeddings fail to be applied to `inputs_embeds` #25922

DarkLight1337 · 2025-09-30T03:28:05Z

Purpose

This PR makes sure that the semantics of input_embeds for BertEmbedding and RobertaEmbedding matches the definition in Transformers library.

Note: This by itself is not sufficient to enable prompt embedding inputs (i.e. user inputs inputs_embeds directly), because we still need the information of token_type_ids.

cc @maxdebayser @noooop

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

…puts_embeds` Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

gemini-code-assist

Code Review

This pull request correctly fixes an issue where inputs_embeds was not being handled properly in BertModel and RobertaModel, causing position and token type embeddings to be ignored. The changes align the behavior with the Hugging Face Transformers library.

However, I've identified a critical side effect introduced by these changes. The _decode_token_type_ids function, which is now called even when inputs_embeds are provided, modifies the input_ids tensor in-place. This can lead to silent data corruption and incorrect behavior if the input_ids tensor is reused. I've provided suggestions in both bert.py and roberta.py to fix this by avoiding the in-place modification when inputs_embeds are used. Addressing this is crucial for ensuring the correctness and predictability of the model's behavior.

vllm/model_executor/models/bert.py

vllm/model_executor/models/roberta.py

DarkLight1337 · 2025-09-30T04:22:53Z

cc @qthequartermasterman can you weigh in on whether this is the intended definition of inputs_embeds?

qthequartermasterman · 2025-09-30T11:03:22Z

I'll take a look later today. I'll need to review the transformers implementation.

maxdebayser

LGTM

qthequartermasterman

This looks right to me! Thanks.

…puts_embeds` (vllm-project#25922) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…puts_embeds` (#25922) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: yewentao256 <zhyanwentao@126.com>

…puts_embeds` (vllm-project#25922) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>

…puts_embeds` (vllm-project#25922) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: xuebwang-amd <xuebwang@amd.com>

…puts_embeds` (vllm-project#25922) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

### What this PR does / why we need it? This is the step 1 of refactoring code to adapt with vllm main, and this pr aligned with vllm-project/vllm@17c540a 1. refactor deepseek to the latest code arch as of vllm-project/vllm@17c540a 2. bunches of fixes due to vllm changes - Fix `AscendScheduler` `__post_init__`, caused by vllm-project/vllm#25075 - Fix `AscendScheduler` init got an unexpected arg `block_size`, caused by vllm-project/vllm#26296 - Fix `KVCacheManager` `get_num_common_prefix_blocks` arg, caused by vllm-project/vllm#23485 - Fix `MLAAttention` import,caused by vllm-project/vllm#25103 - Fix `SharedFusedMoE` import, caused by vllm-project/vllm#26145 - Fix `LazyLoader` improt, caused by vllm-project/vllm#27022 - Fix `vllm.utils.swap_dict_values` improt, caused by vllm-project/vllm#26990 - Fix `Backend` enum import, caused by vllm-project/vllm#25893 - Fix `CompilationLevel` renaming to `CompilationMode` issue introduced by vllm-project/vllm#26355 - Fix fused_moe ops, caused by vllm-project/vllm#24097 - Fix bert model because of `inputs_embeds`, caused by vllm-project/vllm#25922 - Fix MRope because of `get_input_positions_tensor` to `get_mrope_input_positions`, caused by vllm-project/vllm#24172 - Fix `splitting_ops` changes introduced by vllm-project/vllm#25845 - Fix multi-modality changes introduced by vllm-project/vllm#16229 - Fix lora bias dropping issue introduced by vllm-project/vllm#25807 - Fix structured ouput break introduced by vllm-project/vllm#26737 ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? CI passed with existing test. - vLLM version: v0.11.0rc3 - vLLM main: https://github.com/vllm-project/vllm/commit/v0.11.0 --------- Signed-off-by: MengqingCao <cmq0113@163.com> Signed-off-by: Icey <1790571317@qq.com> Co-authored-by: Icey <1790571317@qq.com>

…puts_embeds` (vllm-project#25922) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: xuebwang-amd <xuebwang@amd.com>

[Bugfix] Token type and position embeddings fail to be applied to `in…

f7453e9

…puts_embeds` Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

DarkLight1337 requested a review from Isotr0py September 30, 2025 03:28

DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 30, 2025

DarkLight1337 changed the title ~~[Bugfix] Token type and position embeddings fail to be applied to `in…~~ [Bugfix] Token type and position embeddings fail to be applied to inputs_embeds Sep 30, 2025

gemini-code-assist bot reviewed Sep 30, 2025

View reviewed changes

vllm/model_executor/models/bert.py Show resolved Hide resolved

vllm/model_executor/models/roberta.py Show resolved Hide resolved

Isotr0py approved these changes Sep 30, 2025

View reviewed changes

DarkLight1337 enabled auto-merge (squash) September 30, 2025 04:14

DarkLight1337 disabled auto-merge September 30, 2025 04:15

maxdebayser approved these changes Sep 30, 2025

View reviewed changes

qthequartermasterman approved these changes Sep 30, 2025

View reviewed changes

DarkLight1337 merged commit 9f1c4ec into vllm-project:main Sep 30, 2025
59 checks passed

DarkLight1337 deleted the fix-bert-input-embeds branch September 30, 2025 16:23

pdasigi pushed a commit to pdasigi/vllm that referenced this pull request Oct 2, 2025

[Bugfix] Token type and position embeddings fail to be applied to `in…

2ad110a

…puts_embeds` (vllm-project#25922) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

yewentao256 pushed a commit that referenced this pull request Oct 3, 2025

[Bugfix] Token type and position embeddings fail to be applied to `in…

e33579c

…puts_embeds` (#25922) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: yewentao256 <zhyanwentao@126.com>

wxsIcey mentioned this pull request Oct 17, 2025

[CI] Upgrade vllm to newest commit vllm-project/vllm-ascend#3423

Closed

lywa1998 pushed a commit to lywa1998/vllm that referenced this pull request Oct 20, 2025

[Bugfix] Token type and position embeddings fail to be applied to `in…

b747d45

…puts_embeds` (vllm-project#25922) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

MengqingCao mentioned this pull request Oct 22, 2025

[1/N][Refactor] Refactor code to adapt with vllm main vllm-project/vllm-ascend#3612

Merged

alhridoy pushed a commit to alhridoy/vllm that referenced this pull request Oct 24, 2025

[Bugfix] Token type and position embeddings fail to be applied to `in…

f6f5fd0

…puts_embeds` (vllm-project#25922) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Bugfix] Token type and position embeddings fail to be applied to `inputs_embeds` #25922

[Bugfix] Token type and position embeddings fail to be applied to `inputs_embeds` #25922

Uh oh!

DarkLight1337 commented Sep 30, 2025 •

edited

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 commented Sep 30, 2025

Uh oh!

qthequartermasterman commented Sep 30, 2025

Uh oh!

maxdebayser left a comment

Uh oh!

qthequartermasterman left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

[Bugfix] Token type and position embeddings fail to be applied to inputs_embeds #25922

[Bugfix] Token type and position embeddings fail to be applied to inputs_embeds #25922

Uh oh!

Conversation

DarkLight1337 commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 commented Sep 30, 2025

Uh oh!

qthequartermasterman commented Sep 30, 2025

Uh oh!

maxdebayser left a comment

Choose a reason for hiding this comment

Uh oh!

qthequartermasterman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[Bugfix] Token type and position embeddings fail to be applied to `inputs_embeds` #25922

[Bugfix] Token type and position embeddings fail to be applied to `inputs_embeds` #25922

DarkLight1337 commented Sep 30, 2025 •

edited

Loading