Skip to content

feat: standalone connector api #2076

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 143 commits into from
Closed

feat: standalone connector api #2076

wants to merge 143 commits into from

Conversation

@ryanolson
Copy link
Contributor

@ryanolson ryanolson commented Jul 23, 2025 •
edited by coderabbitai bot
Loading

Summary by CodeRabbit

  • New Features

    • Introduced KVBM (Key-Value Block Manager) integration for distributed and hierarchical KV cache management, including leader/worker roles and vLLM support.
    • Added a dedicated Dockerfile and build/run script support for KVBM environments.
    • Provided Python bindings for KVBM, including cache manager, request handling, and vLLM connector classes.
    • Implemented advanced block lifecycle, offload, and memory layout strategies (including per-layer storage).
    • Added controller APIs for querying and resetting KV cache pools at various cache levels.

  • Documentation

    • Added comprehensive guides, architecture diagrams, and test plans for KVBM and block manager components.

  • Bug Fixes

    • Improved error handling and validation for block allocation, transfer, and cache reset operations.

  • Refactor

    • Major redesign of block pool, offload manager, and block data abstractions to support locality and distributed operations.
    • Simplified and unified Python and Rust interfaces for block and cache management.

  • Tests

    • Introduced extensive unit and integration tests for KVBM, vLLM integration, and block manager functionality.

  • Chores

    • Updated workspace and dependency configurations for new features and compatibility.

ryanolson and others added 30 commits June 3, 2025 18:04
@ryanolson
enable kvbm
37c653e
@ryanolson
Merge remote-tracking branch 'origin/main' into ryan/vllm
5d4f677
@ryanolson
initial protocols
391750e
@ziqifan617
add initial impl for KVBM cache manager allocate_slots
0518da0
@ryanolson
merge stash
47e6a65
@ryanolson
handoff
5eba436
@ryanolson
wheel is building
fbb68e6
@jthomson04
Run clippy + fmt, Split out utils
0da33f1
@jthomson04
More fmt, remove nonexistent modules, add request id field kvrequest
3fc858e
@ryanolson
checkpoint
9629575
@ziqifan617
add test_kvbm_vllm_cache_manager.py and made it work
490dbab
@ziqifan617
rename test
b2c8a45
@ryanolson
back on track
84ac3bb
@ryanolson
comments
5903a61
@ryanolson
checkpoint
d64ef7c
@ryanolson
updating api; adding tests
0019d5f
@ryanolson
hand off
01a7e5f
@ryanolson
list of list for get_block_ids
467a9e6
@ryanolson
phase 3 tests
aafb1de
@ryanolson
phase 4 testing
8d2fda4
@ryanolson
adding doc for test plan
6061cec
@ryanolson
add has_slot method; refactor python to create a slot in allocate_slo…
092e0c3 
…ts if a slot does not exist for the request
@ziqifan617 @ryanolson
[KVBM] add more unit tests to test kvbm py bindings (#1442)
785210c 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: Ryan Olson <rolson@nvidia.com>
@ryanolson
adding worker side distributed object
3a7e6db
@ziqifan617
add more kvbm unit tests (#1448)
66e87d0
@ryanolson
move kv store to runtime
3809d31
@ziqifan617
add missing kvbm APIs
06ec03b
@ryanolson
updates
6ca2905
@ryanolson
stage 1 - presumptive gtg
12d4799
@ryanolson
adding missing file
730cb10
oandreeva-nv and others added 19 commits July 25, 2025 16:41
@oandreeva-nv
Leader / Worker integration from John's branch + small fixes (#2126)
5c01eb4
@oandreeva-nv
disabling device pool on BlockManager (#2129)
c8031c5
@ryanolson
building abstacting connector logic behind a trait
2d073bd
@ryanolson
improved debugging
fc63b9c
@ryanolson
leader and worker mostly wired up; next: link worker engine to transf…
95448ec 
…er engine, then issue leader actions on state change which will be scheduled by worker side event triggers
@ryanolson
move connector traits to llm::block_manager
45802de
@ryanolson
adding connector::protocols for coordinating leader <-> xfer engine <…
1b4c799 
…-> scheduler <-> worker
@ryanolson
updating connector: leader <-> xfer <-> scheduler <-> worker
b2191d8
@ryanolson
monday start
c5e959d
@ryanolson
expose block transfer handler on kvbm worker
66d1fd5
@ryanolson
establing more of the runtime purely in rust
0fa15bb
@ryanolson
leader initialized load - aka immediate ready to test
382ebf5
@ryanolson
onboarding functional - some cleanup is needed
f7c3a1d
@ryanolson
7/30 - am sync point (#2193)
f3a82e5
@ryanolson @ziqifan617
adding tests for scheduler <--> transfer task coordination (#2197)
ca3cc61 
Co-authored-by: Ziqi Fan <ziqif@nvidia.com>
@ryanolson
merge ziqi + ryan
e64314a
@oandreeva-nv
olga - trigger offlload on worker (#2211)
a16a8b8
@ryanolson @ziqifan617 @oandreeva-nv
working branch for 7/31 (#2219)
2a24c79 
Signed-off-by: Ryan Olson <ryanolson@users.noreply.github.com>
Co-authored-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: Olga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>
@ryanolson
connector/250801 (#2244)
1a78b06
@ryanolson ryanolson changed the base branch from jthomson04/kvbm-final to main August 2, 2025 19:37
@ryanolson ryanolson closed this Aug 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rmccorm4 rmccorm4 Awaiting requested review from rmccorm4

@tanmayv25 tanmayv25 Awaiting requested review from tanmayv25

@richardhuo-nv richardhuo-nv Awaiting requested review from richardhuo-nv

@ptarasiewiczNV ptarasiewiczNV Awaiting requested review from ptarasiewiczNV

@ishandhanani ishandhanani Awaiting requested review from ishandhanani

@alec-flowers alec-flowers Awaiting requested review from alec-flowers

@nnshah1 nnshah1 Awaiting requested review from nnshah1

@piotrm-nvidia piotrm-nvidia Awaiting requested review from piotrm-nvidia

@grahamking grahamking Awaiting requested review from grahamking

@paulhendricks paulhendricks Awaiting requested review from paulhendricks

@biswapanda biswapanda Awaiting requested review from biswapanda

@tmonty12 tmonty12 Awaiting requested review from tmonty12

@GuanLuo GuanLuo Awaiting requested review from GuanLuo

@tedzhouhk tedzhouhk Awaiting requested review from tedzhouhk

@kkranen kkranen Awaiting requested review from kkranen

@jthomson04 jthomson04 Awaiting requested review from jthomson04

@PeaBrane PeaBrane Awaiting requested review from PeaBrane

@indrajit96 indrajit96 Awaiting requested review from indrajit96

@krishung5 krishung5 Awaiting requested review from krishung5

@whoisj whoisj Awaiting requested review from whoisj

@hhzhang16 hhzhang16 Awaiting requested review from hhzhang16

@hutm hutm Awaiting requested review from hutm

@julienmancuso julienmancuso Awaiting requested review from julienmancuso

@mohammedabdulwahhab mohammedabdulwahhab Awaiting requested review from mohammedabdulwahhab

@atchernych atchernych Awaiting requested review from atchernych

@nealvaidya nealvaidya Awaiting requested review from nealvaidya

Assignees

No one assigned

Labels

feat size/XXL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@ryanolson @ziqifan617 @jthomson04 @oandreeva-nv @richardhuo-nv