[Feature]: Multi-Token Prediction (MTP) #12181
Description
🚀 The feature, motivation and pitch
DeepSeek V3 is trained with MTP. This has potential to increase the throughput by 2-3x dependent on how many extra tokens are generated.
Paper: https://github.com/deepseek-ai/DeepSeek-V3/blob/main/DeepSeek_V3.pdf
Alternatives
No response
Additional context
No response
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.