Skip to content

feat: add logits processor support for trtllm backend #2702

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
Aug 30, 2025
Merged

feat: add logits processor support for trtllm backend #2702

merged 9 commits into from
Aug 30, 2025

Conversation

@bhuvan002
Copy link
Contributor

@bhuvan002 bhuvan002 commented Aug 25, 2025 •
edited by coderabbitai bot
Loading

Overview:

Details:

Where should the reviewer start?

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

  • closes GitHub issue: #xxx

Summary by CodeRabbit

  • New Features

    • Support for custom logits processors in the TensorRT-LLM backend, enabling in-place logits modification during generation.
    • Added example processors (Temperature, Hello World) to illustrate usage.

  • Chores

    • Tokenizer now initializes by default when starting the TensorRT-LLM engine for improved out-of-the-box readiness.

@github-actions github-actions bot added the feat label Aug 25, 2025
milesial
milesial requested changes Aug 26, 2025
View reviewed changes
@bhuvan002 bhuvan002 force-pushed the bhuvan/add-trtllm-logits-processing branch from 421aa87 to f5744d8 Compare August 26, 2025 04:22
@bhuvan002 bhuvan002 force-pushed the bhuvan/add-trtllm-logits-processing branch from 3cbbd4c to cc025e0 Compare August 26, 2025 20:43
@bhuvan002 bhuvan002 requested a review from milesial August 26, 2025 20:47
@bhuvan002 bhuvan002 marked this pull request as ready for review August 27, 2025 06:01
@bhuvan002 bhuvan002 force-pushed the bhuvan/add-trtllm-logits-processing branch from ff6600e to 516903e Compare August 29, 2025 16:05
tanmayv25
tanmayv25 reviewed Aug 29, 2025
View reviewed changes
Copy link
Contributor

@tanmayv25 tanmayv25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add some documentation about this feature in backend readme?

@bhuvan002 bhuvan002 force-pushed the bhuvan/add-trtllm-logits-processing branch from 516903e to 3515141 Compare August 29, 2025 23:16
@bhuvan002
Copy link
Contributor Author

bhuvan002 commented Aug 29, 2025

Add some documentation about this feature in backend readme?

@tanmayv25 Added docs to the trtllm readme

@devactivity-app
Copy link

devactivity-app bot commented Aug 29, 2025

@bhuvan002, ⭐3 XP earned, 🏆First Comment Added completed, 💪Level 1 achieved! [Details]

icon

@bhuvan002 bhuvan002 requested a review from tanmayv25 August 29, 2025 23:27
bhuvan002 and others added 9 commits August 29, 2025 17:35
@bhuvan002
feat: add logits processor support for trtllm backend
e5071f4 
Signed-off-by: Bhuvan Agrawal <11240550+bhuvan002@users.noreply.github.com>
@bhuvan002
fix: use cuda stream provided by trtllm
7478710 
Signed-off-by: Bhuvan Agrawal <11240550+bhuvan002@users.noreply.github.com>
@bhuvan002
chore: refactor logits processors to be in place
553ca88 
Signed-off-by: Bhuvan Agrawal <11240550+bhuvan002@users.noreply.github.com>
@bhuvan002
chore: cleanup
9c4a10e 
Signed-off-by: Bhuvan Agrawal <11240550+bhuvan002@users.noreply.github.com>
@bhuvan002
chore: move around imports
fac26aa 
Signed-off-by: Bhuvan Agrawal <11240550+bhuvan002@users.noreply.github.com>
@bhuvan002 @coderabbitai
chore: Fail fast in helloworld processor if no eos token found
efd9dfa 
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: Bhuvan Agrawal <11240550+bhuvan002@users.noreply.github.com>
@bhuvan002
chore: add e2e test
1691e2a 
Signed-off-by: Bhuvan Agrawal <11240550+bhuvan002@users.noreply.github.com>
@bhuvan002
fix: only init tokenizer when needed for logits processors
54d0d84 
Signed-off-by: Bhuvan Agrawal <11240550+bhuvan002@users.noreply.github.com>
@bhuvan002
docs: add usage instructions on logits processors
d639382 
Signed-off-by: Bhuvan Agrawal <11240550+bhuvan002@users.noreply.github.com>
@bhuvan002 bhuvan002 force-pushed the bhuvan/add-trtllm-logits-processing branch from 3515141 to d639382 Compare August 30, 2025 00:35
@copy-pr-bot
Copy link

copy-pr-bot bot commented Aug 30, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@bhuvan002 bhuvan002 enabled auto-merge (squash) August 30, 2025 00:36
@bhuvan002 bhuvan002 merged commit 43a2695 into main Aug 30, 2025
8 checks passed
@bhuvan002 bhuvan002 deleted the bhuvan/add-trtllm-logits-processing branch August 30, 2025 01:06
jasonqinzhou pushed a commit that referenced this pull request Aug 30, 2025
@bhuvan002 @coderabbitai
feat: add logits processor support for trtllm backend (#2702)
fb9f092 
Signed-off-by: Bhuvan Agrawal <11240550+bhuvan002@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: Jason Zhou <jasonzho@jasonzho-mlt.client.nvidia.com>
KrishnanPrash pushed a commit that referenced this pull request Sep 2, 2025
@bhuvan002 @coderabbitai @KrishnanPrash
feat: add logits processor support for trtllm backend (#2702)
373f727 
Signed-off-by: Bhuvan Agrawal <11240550+bhuvan002@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
dillon-cullinan pushed a commit that referenced this pull request Sep 5, 2025
@bhuvan002 @coderabbitai @dillon-cullinan
feat: add logits processor support for trtllm backend (#2702)
8dd1dc9 
Signed-off-by: Bhuvan Agrawal <11240550+bhuvan002@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
nnshah1 pushed a commit that referenced this pull request Sep 8, 2025
@bhuvan002 @coderabbitai @nnshah1
feat: add logits processor support for trtllm backend (#2702)
c7669c8 
Signed-off-by: Bhuvan Agrawal <11240550+bhuvan002@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: nnshah1 <neelays@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments

@tanmayv25 tanmayv25 tanmayv25 approved these changes

@milesial milesial Awaiting requested review from milesial

@nnshah1 nnshah1 Awaiting requested review from nnshah1

@piotrm-nvidia piotrm-nvidia Awaiting requested review from piotrm-nvidia

@ptarasiewiczNV ptarasiewiczNV Awaiting requested review from ptarasiewiczNV

@ryanolson ryanolson Awaiting requested review from ryanolson

@grahamking grahamking Awaiting requested review from grahamking

@paulhendricks paulhendricks Awaiting requested review from paulhendricks

@biswapanda biswapanda Awaiting requested review from biswapanda

@tmonty12 tmonty12 Awaiting requested review from tmonty12

@GuanLuo GuanLuo Awaiting requested review from GuanLuo

@rmccorm4 rmccorm4 Awaiting requested review from rmccorm4

@tedzhouhk tedzhouhk Awaiting requested review from tedzhouhk

@alec-flowers alec-flowers Awaiting requested review from alec-flowers

@kkranen kkranen Awaiting requested review from kkranen

@ishandhanani ishandhanani Awaiting requested review from ishandhanani

@PeaBrane PeaBrane Awaiting requested review from PeaBrane

@hhzhang16 hhzhang16 Awaiting requested review from hhzhang16

Assignees

No one assigned

Labels

feat size/L

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@bhuvan002 @tanmayv25 @milesial