-
-
Notifications
You must be signed in to change notification settings - Fork 10.8k
[deepseek] kernel block size for UniformTypeKVCacheSpecs #26559
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[deepseek] kernel block size for UniformTypeKVCacheSpecs #26559
Conversation
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request correctly adds support for UniformTypeKVCacheSpecs to address an issue with deepseek models. The approach of unpacking the spec and then handling it in the existing logic is sound. I've identified a minor improvement opportunity in the error reporting within gpu_model_runner.py to enhance debuggability for future unhandled spec types.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM; thanks for fixing!
…t#26559) Signed-off-by: Chen Zhang <zhangch99@outlook.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
…t#26559) Signed-off-by: Chen Zhang <zhangch99@outlook.com> Signed-off-by: Dhruvil Bhatt <bhattdbh@amazon.com>
…t#26559) Signed-off-by: Chen Zhang <zhangch99@outlook.com> Signed-off-by: bbartels <benjamin@bartels.dev>
…t#26559) Signed-off-by: Chen Zhang <zhangch99@outlook.com>
…t#26559) Signed-off-by: Chen Zhang <zhangch99@outlook.com>
…t#26559) Signed-off-by: Chen Zhang <zhangch99@outlook.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
…t#26559) Signed-off-by: Chen Zhang <zhangch99@outlook.com> Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
…t#26559) Signed-off-by: Chen Zhang <zhangch99@outlook.com> Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
Purpose
After #24486 , deepseek 3.2 will throw this error:
This PR fix it.
FIX #26524
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.