Truncation control for embedding models #14776

gmarinho2 · 2025-03-13T18:13:04Z

Enables truncation control for offline and online API for embedding models via the paramenter truncate_prompt_tokens. When truncate_prompt_tokens=-1 the maximum prompt length that the model supports is used. When it exceeds the maximum prompt length, an error is raised. When 0 <truncate_prompt_tokens < maximum prompt length, the prompt is truncated.

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

github-actions · 2025-03-13T18:13:13Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

maxdebayser · 2025-04-23T14:41:49Z

@DarkLight1337 , this is ready for review. Some CI step seems to be hanging, but all others have passed.

Signed-off-by: Max de Bayser <mbayser@br.ibm.com>

maxdebayser · 2025-04-29T19:05:51Z

@DarkLight1337 , all tests are passing now

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com> Signed-off-by: Max de Bayser <mbayser@br.ibm.com> Co-authored-by: Max de Bayser <mbayser@br.ibm.com>

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com> Signed-off-by: Max de Bayser <mbayser@br.ibm.com> Co-authored-by: Max de Bayser <mbayser@br.ibm.com> Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com> Signed-off-by: Max de Bayser <mbayser@br.ibm.com> Co-authored-by: Max de Bayser <mbayser@br.ibm.com> Signed-off-by: Yuqi Zhang <yuqizhang@google.com>

gmarinho2 and others added 28 commits January 21, 2025 11:18

[FEATURE] Enables offline /score for embedding models

9b0ce65

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Merge branch 'upstream_main'

74cd2dd

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Changes variable name and uses tuple instead of list.

590ab4d

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Separates scoring logic and makes small ajustments

f219024

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Separates scoring logic and makes small ajustments

47211e5

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Moves scoring functions declaration to llm class

1a89033

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Completes embedding_score function signature

db7919b

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Passes new parameters to self.encode() in embeddind_score

41b11be

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Adds type annotations for the parameters

b57e01a

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Minor adjustments

4956eae

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Minor adjustments in embedding_score

f8b8d8c

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

trigger ci

cd835de

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

trigger ci

0e95ead

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Merge branch 'vllm-project:main' into main

bba2ea6

Merge branch 'vllm-project:main' into main

635b8e8

Merge branch 'vllm-project:main' into main

9e1dbbe

Merge branch 'vllm-project:main' into main

2ed8d7b

Merge branch 'vllm-project:main' into main

7e05fb0

Merge branch 'vllm-project:main' into main

3931763

Merge branch 'vllm-project:main' into truncation-control

45dba65

Set truncation size to max_model_len when truncate_prompt_tokens is -1

8520fd3

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

put truncation control in score_utils

b3798a0

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

implement local truncation control

9035bfa

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

--amend

8ba7e22

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

fix typing errors

b3a8522

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

fix local truncation control

64b7d6d

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

implement tests for local truncation control

1db5069

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

implement tests for local truncation control

79a5693

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

mergify bot added the frontend label Mar 13, 2025

gmarinho2 and others added 3 commits April 17, 2025 13:32

Merge branch 'main' into truncation-control

f2ee840

Merge branch 'vllm-project:main' into main

83746e3

Merge branch 'main' into truncation-control

76829b5

russellb added this to Structured Output Apr 22, 2025

DarkLight1337 enabled auto-merge (squash) April 23, 2025 14:50

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Apr 23, 2025

DarkLight1337 approved these changes Apr 23, 2025

View reviewed changes

Merge branch 'upstream_main' into truncation-control

fe13004

Signed-off-by: Max de Bayser <mbayser@br.ibm.com>

auto-merge was automatically disabled April 24, 2025 17:19
Head branch was pushed to by a user without write access

DarkLight1337 enabled auto-merge (squash) April 24, 2025 17:27

Merge branch 'upstream_main' into truncation-control

37fa5c9

Signed-off-by: Max de Bayser <mbayser@br.ibm.com>

auto-merge was automatically disabled April 28, 2025 19:31
Head branch was pushed to by a user without write access

gmarinho2 and others added 2 commits April 29, 2025 13:43

Merge branch 'vllm-project:main' into main

13f98ac

Merge branch 'main' into truncation-control

7f53602

DarkLight1337 merged commit 1c2bc7e into vllm-project:main Apr 30, 2025
48 checks passed

github-project-automation bot moved this to Done in Structured Output Apr 30, 2025

gmarinho2 deleted the truncation-control branch April 30, 2025 15:54

maxdebayser mentioned this pull request Jun 10, 2025

[Feature]: Improve DX with respect to truncation in embedding and scoring tasks #13489

Closed

1 task

noooop mentioned this pull request Jul 10, 2025

[Feature]: Why vllm embedding just can't truncate longer text automatically? And a strange example here. #20744

Closed

1 task

noooop mentioned this pull request Jul 21, 2025

IRCOT复现失败 RUC-NLPIR/FlashRAG#185

Open

hmellor mentioned this pull request Aug 11, 2025

[Bug]: Possible mismatch in truncate_prompt_tokens value validation for -1 #22635

Closed

noooop mentioned this pull request Oct 10, 2025

Change the default value of truncate_prompt_tokens in the embedding/rerank/pooling model to -1 #24235

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Truncation control for embedding models #14776

Truncation control for embedding models #14776

Uh oh!

gmarinho2 commented Mar 13, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Mar 13, 2025

Uh oh!

maxdebayser commented Apr 23, 2025

Uh oh!

maxdebayser commented Apr 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Truncation control for embedding models #14776

Truncation control for embedding models #14776

Uh oh!

Conversation

gmarinho2 commented Mar 13, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 13, 2025

Uh oh!

maxdebayser commented Apr 23, 2025

Uh oh!

maxdebayser commented Apr 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gmarinho2 commented Mar 13, 2025 •

edited by github-actions bot

Loading