[V1][Spec Decode] Eagle interface #15334
Conversation
LiuXiaoxuanPKU
requested review from
WoosukKwon,
alexm-redhat,
comaniac,
njhill,
robertgshaw2-redhat and
ywang96
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
| num_new_tokens = (request.num_tokens_with_spec + | ||
| self.num_lookahead_slots - |
There was a problem hiding this comment.
IIUC, this is not correct because num_lookahead_slots is only for memory allocation, not about how many tokens to be batched in this step.
There was a problem hiding this comment.
Can I ask how is self.num_lookahead_slots different from len(request.spec_token_ids)?
Isn't request.num_tokens_with_spec is taking into account the number of draft token for the req as it is def as len(request._all_token_ids) + len(request.spec_token_ids)?
There was a problem hiding this comment.
Ah, probably self.num_lookahead_slots is needed for kv cache of transformer block in eagle proposer? If so, then we need a bigger kvc for eagle module when doing prefill since the entire prompt is processed by the eagle module.
| @abstractmethod | ||
| def generate_draft_token_ids( | ||
| self, input_batch: InputBatch, sampled_token_ids: list[list[int]], | ||
| sampling_metadata: SamplingMetadata) -> list[list[int]]: |
There was a problem hiding this comment.
change it similar to *args
There was a problem hiding this comment.
add if statement in the model_runner to pass in different parameters
|
This pull request has merge conflicts that must be resolved before it can be |
3 participants
This PR
Some notes: