New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce outer reduction for metal #17058

Merged

tqchen merged 1 commit into apache:main from Hzfengsy:low_batch_gemv

Jun 4, 2024

Member

Hzfengsy commented Jun 3, 2024

q4f16_0 may improve the performance of metal. To be specific, Llama-3 8B on M1 Pro:

Single batch 26.2 tok/s (q4f16_1) vs. 28.0 tok/s (q4f16_0)
low-batched serving: about 25% improvement


          Introduce outer reduction for metal

b43c030

Hzfengsy force-pushed the low_batch_gemv branch from b39fbbe to b43c030 Compare

June 3, 2024 11:55

tqchen approved these changes

View reviewed changes

tqchen merged commit 1c05902 into apache:main

19 checks passed

ysh329 mentioned this pull request

[Release] v0.17.0 Release Candidate Notes #17178

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet