Vllm v1 eagle proposer #15346
Vllm v1 eagle proposer #15346
Conversation
sroy745
requested review from
WoosukKwon,
alexm-redhat,
comaniac,
njhill,
robertgshaw2-redhat and
ywang96
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
| self, *, target_model_input_ids: Tensor, | ||
| target_model_positions: Tensor, target_model_hidden_states: Tensor, | ||
| target_model_seq_lens: list[int], | ||
| sampled_token_ids: list[list[int]], | ||
| next_prompt_token_ids: list[list[int]], is_prefill: list[bool], | ||
| num_draft_tokens_to_propose: int, | ||
| attention_metadata: FlashAttentionMetadata) -> list[SamplerOutput]: |
There was a problem hiding this comment.
nit: please append , at the end of the input parameters and re-run the formatter, so that each input parameter can take a line. I really recommend this because otw adding/removing an input parameter can change the format again.
| self, *, target_model_input_ids: Tensor, | |
| target_model_positions: Tensor, target_model_hidden_states: Tensor, | |
| target_model_seq_lens: list[int], | |
| sampled_token_ids: list[list[int]], | |
| next_prompt_token_ids: list[list[int]], is_prefill: list[bool], | |
| num_draft_tokens_to_propose: int, | |
| attention_metadata: FlashAttentionMetadata) -> list[SamplerOutput]: | |
| self, | |
| *, | |
| target_model_input_ids: Tensor, | |
| target_model_positions: Tensor, | |
| target_model_hidden_states: Tensor, | |
| target_model_seq_lens: list[int], | |
| sampled_token_ids: list[list[int]], | |
| next_prompt_token_ids: list[list[int]], | |
| is_prefill: list[bool], | |
| num_draft_tokens_to_propose: int, | |
| attention_metadata: FlashAttentionMetadata, | |
| ) -> list[SamplerOutput]: |
| Generates speculative draft token IDs using the Eagle model. | ||
|
|
||
| This function aligns the Eagle model's KV cache with the target | ||
| model’s output before generating speculative tokens. It first |
There was a problem hiding this comment.
| model’s output before generating speculative tokens. It first | |
| model's output before generating speculative tokens. It first |
LiuXiaoxuanPKU
left a comment
LiuXiaoxuanPKU
left a comment
There was a problem hiding this comment.
Left some early partial comments, will finish one pass tomorrow.
| """ | ||
| self._vllm_config = vllm_config | ||
| self._model = model | ||
| self._sampling_metadata = sampling_metadata |
There was a problem hiding this comment.
why is this a field of eagle proposer? should it be passed in every proposing step?
| Tokens: [T12, T13, T14, T15, T22, T23, T24, T32] | ||
| Positions: [0, 1, 2, 3, 9, 10, 11, 44] | ||
| Previous Hidden States: [H11, H12, H13, H14, H21, H22, H23, H31] | ||
| Sampled Tokens: [[T16], [T25], [T33']] |
There was a problem hiding this comment.
Minor: why is this example different from the example of input? Maybe just use the same example?
There was a problem hiding this comment.
It is the same example IIUC. The above one shows all the information including inputs and outputs, while this part only shows the inputs.
| Note that for S1, we drop T11 (position 0). For S2 and S3, | ||
| T21 and T31 are skipped since they were processed earlier. | ||
| Eagle positions are always one less than the target model | ||
| due to dropping the first token. |
There was a problem hiding this comment.
| due to dropping the first token. | |
| due to dropping the first token. For example, T12 has position 1 when running the traget model, while its position is 0 when running the Eagle head. |
| model. | ||
| target_model_hidden_states: Hidden states from the target model. | ||
| target_model_seq_lens: Sequence lengths in the target model. | ||
| sampled_token_ids: Previously sampled/accepted tokens from the |
There was a problem hiding this comment.
| sampled_token_ids: Previously sampled/accepted tokens from the | |
| sampled_token_ids: Generated tokens from the previous generation step. |
| next_prompt_token_ids: The next prompt token for a sequence if it | ||
| is a partial prefill sequence and empty otherwise. | ||
| is_prefill: Boolean flags indicating prefill sequences. | ||
| num_draft_tokens_to_propose: Number of speculative tokens to |
There was a problem hiding this comment.
A shorter name? num_spec_tokens?
| # Determine expected sequence lengths in the Eagle model: | ||
| # - For prefill sequences, lengths remain unchanged. | ||
| # - For decoding sequences, lengths match the number of | ||
| # accepted tokens. |
There was a problem hiding this comment.
What is expected sequence length? A bit more context?
| Tokens: [T11, T12, T13, T14, T21, T22, T23, T31, T32, T33] | ||
| Positions: [0, 1, 2, 3, 9, 10, 11, 44, 45, 46] | ||
| Hidden States: [H11, H12, H13, H14, H21, H22, H23, H31, H32, H33] | ||
| Sampled Tokens: [[T15], [], [T32]] |
There was a problem hiding this comment.
I actually don't get this. Where is T33 from?
Do you mean this?
Input Tokens: [[T11, T12, T13, T14], [T21, T22, T23], [T31, T32 (draft), T33 (draft)]]
Sampled Tokens: [[T15], [], [T32' (recovered token)]]
| target_model_positions: Tensor, target_model_hidden_states: Tensor, | ||
| target_model_seq_lens: list[int], | ||
| sampled_token_ids: list[list[int]], | ||
| next_prompt_token_ids: list[list[int]], is_prefill: list[bool], |
There was a problem hiding this comment.
What does is_prefill mean?
There was a problem hiding this comment.
As you know, there's no "prefill" in V1.
|
hwo to load model by tp for eagle |
4 participants
No description provided.