Skip to content

Commit 405e730

Browse files
authored
Disable auto enabling chunked prefill on ROCm platform on long contexts due to poor performance (#324)
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
1 parent 7efa6e0 commit 405e730

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

vllm/engine/arg_utils.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1063,7 +1063,8 @@ def create_engine_config(self,
10631063
if (is_gpu and not use_sliding_window and not use_spec_decode
10641064
and not self.enable_lora
10651065
and not self.enable_prompt_adapter
1066-
and model_config.task != "embedding"):
1066+
and model_config.task != "embedding"
1067+
and not current_platform.is_rocm()):
10671068
self.enable_chunked_prefill = True
10681069
logger.warning(
10691070
"Chunked prefill is enabled by default for models with "

0 commit comments

Comments
 (0)