Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature]Support flashMLA decoding via flashAttn2 #29

Open
zhangjinnan opened this issue Feb 24, 2025 · 3 comments
Open

[Feature]Support flashMLA decoding via flashAttn2 #29

zhangjinnan opened this issue Feb 24, 2025 · 3 comments

Comments

@zhangjinnan
Copy link

zhangjinnan commented Feb 24, 2025 •
edited
Loading

As titled, please add support for MACA Platform via flashAttn2
@zhangjinnan zhangjinnan changed the title [Feature] Support flashMLA decoding on MXMACA Platform As titled, please add support for MACA Platform via flashAttn2 [Feature]Support flashMLA decoding via flashAttn2 Feb 24, 2025
@zhangjinnan zhangjinnan mentioned this issue Feb 24, 2025
Closed
@zhangjinnan
Copy link
Author

We provide the implementation of FlashMLA from FlashAttention-2(version 2.6.3), based on MACA toolkit and C500 chips.

FlashAttention-2 currently supports:

  1. Datatype fp16 and bf16.
  2. Multi-Token Parallelism = 1
  3. Paged kvcache with block size equal to 2^n (n >= 0)

@zhangjinnan zhangjinnan mentioned this issue Feb 25, 2025
Open
@QiuMike
Copy link

QiuMike commented Feb 25, 2025

Where is the source code？

@zhangjinnan
Copy link
Author

zhangjinnan commented Feb 25, 2025 •
edited
Loading

Where is the source code？

source code：#36

you can watch update in https://github.com/MetaX-MACA/FlashMLA repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@QiuMike @zhangjinnan