-
Notifications
You must be signed in to change notification settings - Fork 79
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for head dimension 128 #26
Comments
That is on the eventual to-do list! |
Great! Thanks for the brilliant project!
Are there any pointers on what the block issues are? I would like to try to
fix this on my side.
来自手机回复
Benjamin Spector ***@***.***>于2024年5月16日 周四10:33写道:
… That is on the eventual to-do list!
—
Reply to this email directly, view it on GitHub
<#26 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAH4ZXZORXRM4YGACGHZGCLZCQLGNAVCNFSM6AAAAABHZHMMZ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJTHEYDEMJXGQ>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
looping in @Aaryan0404 for this |
Could you please explain why there is a limitation on the size of the head dimension? I'm not very clear about it.
Could you please explain why there is a limitation on the size of the head dimension? I'm not very clear about it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Most recent models use hdim=128, it would be great to see that ThunderKittens also support that.
ThunderKittens/examples/attn/h100/h100_train_atn.py
Lines 25 to 26 in a562ed2
The text was updated successfully, but these errors were encountered: