[RoadMap] Mooncake Store V2

[stmatengss]

This is a roadmap for Mooncake Store V2! If you're interested in contributing to any item, please let us know!

If any items you’re interested in are missing from the roadmap, your suggestions and input are highly encouraged! Please don’t hesitate to comment in this thread, submit a feature request, or establish an RFC.

---

Mooncake Store V2

1. LMCache Adaptor
   • KVCache Reuse between prefill nodes
   • HTTP notification protocols (KVAdmitMsg/KVEvictMsg) with LMCache Controller @xiaguan
   • Mooncake-based BufferAllocator in LMCache layer to allocate MemoryObj (zero-copy) [Core] perf: use CopyLessMemoryObj in mooncake connector for zero copy LMCache/LMCache#642
2. SGLang Adaptor Support l3 cache (mooncake store) for hiradix cache sgl-project/sglang#7211
3. HiCache support https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/mooncake_store/README.md
4. Mooncake Client
   • Client-side Buffer to reduce metadata footprint
   • Batch Put/Get interfaces for layer-wise. @zhaoyongke @xinranwang17 [RFC][Store]Provide BatchPut() and BatchGet() interfaces for fine-grained KV Slices #380
   • Soft-pin mechanism to pin KVCache in the indicated node/layers. [Store] Soft Pin for Important Object #587
   • Flexible APIs: removeAll, migrate fix(master): Fix negative storage size metrics after removeAll #355
   • Implementing the basic metrics system on the client side. feat(client): Add client-side metrics for transfer and RPC operations #733 [Transfer Engine] Metrics: Add total qp metrics #738
   • User-friendly configuration method (using global yaml config or avoid configurations)
5. Mooncake Master
   • Lease designs to avoid conflicts between get/remove [Store] Add features: lease and eviction #374
   • High availability service with extra protocols [Store] Soft Pin for Important Object #587 @ykwd
   • KVCache eviction mechanism [Store] Add features: lease and eviction #374 Add LRU in MasterService, complexity O(1) #287
   • 3FS backend for KVCache persistence. [Store] Enable Client SSD Offload And Storage Persistence #437 [Store]feat: Add 3fs native api plugin for KVCache storage persistence #610 [Store]feat: Migrate Persistence Metadata from Client to Master Service #690 @SgtPepperr
6. Multi-layer Storage
   • VRAM storage support (GDR-based zero-copy)
   • GPU-based VRAM pool @XucSh [WIP] Use VRAM as buffer #710
   • SSD storage support [Store] feat: store key-value data in buckets #968
7. Benchmark and CI/CD
   • KVCache workload emulation benchmark
   • Stress benchmark for high concurrency
8. Revise Website Documentation
   • Create the Python binding documentation.
   • Create the Mooncake Master documentation.
   • Create documentation to assist users in integrating with LMCache. @XucSh [Doc] Add document for integrating Mooncake Store to LMCache V1 #385
9. Deployment
   • User-friendly configuration method (using global yaml config or avoiding configurations)
   • native K8S helm chart for running store (including P, D and master). KEP 74: Mooncake integration sgl-project/rbg#75
   • Integrating etcd service into mooncake store
10. Fault-tolerance and High Availability (HA)
    • Master failover: multiple backup instances, elect new leader when old one fails [Store] High Availability 1: Master Failover #451
    • Master failover: kv metadata persistency [Store] Serialize/Deserialize Offset Allocator #760
    • Client failover: tolerant client crash and network partition [Store] High Availability V2: Client Failover  #501
    • Ensure correctness of get and put operations [Store] Add features: lease and eviction #374 [Store] Check If Get Completed Within Lease #778 [Store]: Cleanup processing objects if transferring timedout (#975) #993
    • Move stable HA-features (that do not depend on etcd) to non-HA mode feat(store): add client heartbeat support for non ha mode #845

[zhaoyongke]

Great!
BTW, is cache-aware scheduler in the roadmap?

[stmatengss]

Great! BTW, is cache-aware scheduler in the roadmap?

Good idea! Is the Cache-aware scheduler on the master side or another implementation?

[zhaoyongke]

Great! BTW, is cache-aware scheduler in the roadmap?

Good idea! Is the Cache-aware scheduler on the master side or another implementation?

Both, mooncake master provides key location information for global scheduler. I'll provide another RFC later.

[zhaoyongke]

Great! BTW, is cache-aware scheduler in the roadmap?

Good idea! Is the Cache-aware scheduler on the master side or another implementation?

Both, mooncake master provides key location information for global scheduler. I'll provide another RFC later.

support cache aware scheduler

[hzh0425]

• KVCache Reuse between prefill nodes

@stmatengss Hi, I wanna try it, thanks~

[stmatengss]

• KVCache Reuse between prefill nodes

@stmatengss Hi, I wanna try it, thanks~

Cool! Hope to see your PR soon!

[ykwd]

High availability service with extra protocols
Integrating etcd service into mooncake store

Hi, I wanna try the High availability feature. And as this feature may depend on ETCD server, these two could be solved in the same PR.

[zhaoyongke]

2. SGLang Adaptor

We'll take this @huangtingwei9988

[wwq2333]

High availability service with extra protocols
Integrating etcd service into mooncake store

Hi, I wanna try the High availability feature. And as this feature may depend on ETCD server, these two could be solved in the same PR.

Hi @ykwd @stmatengss, in the master's high availability design, is there a plan to implement persistence of various meta information?

[ykwd]

Hi @ykwd @stmatengss, in the master's high availability design, is there a plan to implement persistence of various meta information?

Yes. Supporting KV metadata persistence is on our roadmap.

[SpecterCipher]

Hi @ykwd @stmatengss, in the master's high availability design, is there a plan to implement persistence of various meta information?

Yes. Supporting KV metadata persistence is on our roadmap.

Hi, may I ask whether there has been any discussion or documentation concerning this approach? For example, would the metadata be stored in etcd? @ykwd

[ykwd]

Hi @ykwd @stmatengss, in the master's high availability design, is there a plan to implement persistence of various meta information?

Yes. Supporting KV metadata persistence is on our roadmap.

Hi, may I ask whether there has been any discussion or documentation concerning this approach? For example, would the metadata be stored in etcd? @ykwd

@SpecterCipher There’s no official documentation yet. In the meantime, we’ve had some informal discussions about the design. Using etcd to store KV metadata is not preferred due to performance concerns. Instead, we’re considering having each client store the metadata for the KVs it holds. This would make metadata persistence independent of etcd or any other external service. However, this approach hasn’t been finalized yet.