Support for KV caching and batched inference#1934

Open

mseeger wants to merge 3 commits intoLightning-AI:mainfrom mseeger:kvcache3

+2,441-788

Commits on Feb 8, 2025

Support for KV caching and batched inference
mseeger
committed

Commits on Feb 14, 2025

Added test
mseeger
committed

Commits on Feb 17, 2025

Set enable_gqa flag in scaled_dot_product_attention
mseeger
committed