[RFC]: Support KV Cache sparse with common framework

[FangRun2]

Motivation.

With the increase of model size, the KV cache became larger and sparser, especially for long sequence requests. To reduce the GPU memory used, offload full KV to external storage and only keep partial or compressed KV in GPU memory became the popular direction. This can also reduce the GPU calculation, increase the sequence length and batch size of decoding.

Sparse KV cache have many different choices. Recently paper point out that there is no common way can fit all scenarios and all models. So better to build a common framework then different sparse algorithms can be plugin to it like KV connector for PC.

Proposed Change.

All gray boxes are current classes in 0.9.2. Green boxes are proposed to add. Light green ones show out the future sub classes base on this framework.

SpareKVBase is the base class of different algorithms. Just like KV connector design, it will hook few places of scheduler and layer.py to allow sparse algorithms do additional load, dump and calculate sparse KV blocks.

SparseKVManager provide different KV block allocation methods for different algorithms. To keep all implementation under SpareKVBase, it will call SparseKVBase and real implementation will happen in sub class of sparse algorithms.

KVStoreBase helps decoupling sparse algorithms and external storage. It defined the methods how to talk to external storage, so any sparse algorithms can work with any external storage. Concepts here is blocks identify by ID with offset. This is not only for sparse but also naturally for prefix cache also. KVStoreConnector connect it with current KVConnectorBase_V1 to provide PC function.

NFSStore is sample implementation here provide ability to store blocks in local file system or NFS mount point in multi-server case.

LocalCachedStore can refence any store to provide local DRAM read cache layer.

Feedback Period.

No response

CC List.

@james0zan, @ShawnD200, @chizhang118, @TKONIY, @JoelSeniorLiang, @heheda12345, @Zachary-ai-engineer

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Zachary-ai-engineer]

Very good. I'm currently working on a similar task in the PD separation for DeepSeek-R1, aiming to compress the length from 128K down to 16K, thereby achieving more than a 10x reduction in TTFT.
In my opinion, it's completely unnecessary to integrate so many algorithms — often, just one or two are enough to achieve the maximum sparsity effect.

[JewelChen2019]

Very good. I'm currently working on a similar task in the PD separation for DeepSeek-R1, aiming to compress the length from 128K down to 16K, thereby achieving more than a 10x reduction in TTFT. In my opinion, it's completely unnecessary to integrate so many algorithms — often, just one or two are enough to achieve the maximum sparsity effect.

UCM has already open-sourced, which provided the PD disaggregation ， https://github.com/ModelEngine-Group/unified-cache-management/tree/main