[Model][Speculative Decoding] support k > 1 for MTP

[luccafong]

No description provided.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @luccafong.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[benchislett]

Hi @luccafong , could you take a look at #13626 ? I would appreciate if you could highlight any key differences between this implementation and the existing PR.

[luccafong]

Hi @luccafong , could you take a look at #13626 ? I would appreciate if you could highlight any key differences between this implementation and the existing PR.

Hi @benchislett , so I think #13626 is targeting on using EAGLE style to run forward on the same module, while this PR tartgeting on prediction for running forward on k MTP modules (k > 1) as described by the paper, so they are quite different

[benchislett]

I hope that we can orchestrate compatibility between these two features in the future so that either one is possible. I think there is a lot of overlap between the contributions of each feature.

[luccafong]

cc @LiuXiaoxuanPKU for early review. will need more cleanup and benchmark for publishing

[benchislett]

Is the multi-step logic omitted here, and self.mtp is just using TP1DraftModelRunner in is_fallback mode?

[benchislett]

Why is this block skipped?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @luccafong.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork