[Feature]: [Perf] Optimize `reshape_and_cache` CUDA Kernel

[yewentao256]

🚀 The feature, motivation and pitch

Similar to #22036
We can optimize the reshape_and_cache Cuda kernel.

Pick it up if you are interested.

[ZJY0516]

I would like to help if possible.

[yewentao256]

@ZJY0516 Assigned, enjoy!

[Liu-congo]

hello, i would like to help if possible
here is my implementation [Perf] Optimize reshape_and_cache CUDA Kernel #26021

[mgoin]

I feel like the most impactful way to optimize this op is actually to look beyond just itself and instead fuse RoPE + reshape_and_cache (or other ops, deepseek is more complex than llama) - noted as "Fusion 3" in this profile from #24629

[ZJY0516]

I feel like the most impactful way to optimize this op is actually to look beyond just itself and instead fuse RoPE + reshape_and_cache (or other ops, deepseek is more complex than llama) - noted as "Fusion 3" in this profile from #24629

Agreed, but this approach requires a hand-written fusion kernel, which is orthogonal to the current problem.

[ZJY0516]

@Liu-congo Hi, thanks for your great work! To help us collaborate more efficiently and streamline the review process, would you be open to me cherry-picking your commits into my PR? Feel free to reach out on Slack if you'd like to discuss this more.

[Liu-congo]

@Liu-congo Hi, thanks for your great work! To help us collaborate more efficiently and streamline the review process, would you be open to me cherry-picking your commits into my PR? Feel free to reach out on Slack if you'd like to discuss this more.

sure, feel free to do that : )