[Core] Remove prompt string from engine core data structures #17214
Conversation
By design, V1 engine core deals only with token ids. EngineCoreRequest contained an optional string prompt field, but we always set it to None before sending to the engine. It's used only by the output processor to set the prompt back in output messages returned to the user. It's best to remove it from EngineCoreRequest to make this separation clear/explicit. Signed-off-by: Nick Hill <nhill@redhat.com>
njhill
requested review from
WoosukKwon,
alexm-redhat,
comaniac,
robertgshaw2-redhat and
ywang96
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
| def __init__( | ||
| self, | ||
| request_id: str, | ||
| prompt: Optional[str], |
There was a problem hiding this comment.
This is always None anyhow
|
|
||
| req_id: str | ||
| prompt_token_ids: list[int] | ||
| prompt: Optional[str] |
There was a problem hiding this comment.
This is always None anyhow
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
…oject#17214) Signed-off-by: Nick Hill <nhill@redhat.com>
…oject#17214) Signed-off-by: Nick Hill <nhill@redhat.com>
…oject#17214) Signed-off-by: Nick Hill <nhill@redhat.com> Signed-off-by: Agata Dobrzyniewicz <adobrzyniewicz@habana.ai>
…oject#17214) Signed-off-by: Nick Hill <nhill@redhat.com> Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
…oject#17214) Signed-off-by: Nick Hill <nhill@redhat.com> Signed-off-by: Yuqi Zhang <yuqizhang@google.com>
3 participants
By design, V1 engine core deals only with token ids.
EngineCoreRequestcontained an optional string prompt field, but we always set it toNonebefore sending to the engine. It's used only by the output processor to set the prompt back in output messages returned to the user.It's best to remove it from EngineCoreRequest to make this separation clear/explicit.