Skip to content

Commit efe02fc

Browse files
committed
kv_cache_config fix for KV Routing Path
1 parent 82ffd54 commit efe02fc

File tree

1 file changed

+4
-2
lines changed
  • components/backends/trtllm/src/dynamo/trtllm

1 file changed

+4
-2
lines changed

components/backends/trtllm/src/dynamo/trtllm/main.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -104,8 +104,10 @@ async def init(runtime: DistributedRuntime, config: Config):
104104
kv_cache_config["event_buffer_max_size"] = DEFAULT_KV_EVENT_BUFFER_MAX_SIZE
105105
else:
106106
kv_cache_config = arg_map["kv_cache_config"]
107-
if not kv_cache_config.event_buffer_max_size:
108-
kv_cache_config.event_buffer_max_size = DEFAULT_KV_EVENT_BUFFER_MAX_SIZE
107+
if "event_buffer_max_size" not in kv_cache_config:
108+
kv_cache_config[
109+
"event_buffer_max_size"
110+
] = DEFAULT_KV_EVENT_BUFFER_MAX_SIZE
109111
arg_map["kv_cache_config"] = kv_cache_config
110112

111113
# Only pytorch backend is supported for now to publish events and metrics.

0 commit comments

Comments
 (0)