New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor: reduce the binary size of batch decode kernels #343

Merged

yzh119 merged 2 commits into main from reduce-decode-datatype

Jun 30, 2024

Collaborator

yzh119 commented Jun 30, 2024

This PR refactors the batch decode related kernels, and make the following breaking changes:

remove batch_decode_with_padded_kv_cache operator, we encourage user to use BatchDecodeWithPagedKVCacheWrapper.
Delete redundant DTypeQ * DTypeKV combinations, now we only support the following cases:
DTypeQ == DTypeKV
DTypeQ is a float16 and DTypeKV is a float8

The output data type follows the query data type.

yzh119 added 2 commits

June 30, 2024 06:53

upd

1915b8f


          add group size 2 back for decode kernels

2ebea3d

yzh119 merged commit 0d333ff into main

yzh119 deleted the reduce-decode-datatype branch

June 30, 2024 07:14

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet