Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: support fused add rmsnorm #419

Merged
merged 2 commits into from
Aug 4, 2024
Merged

feat: support fused add rmsnorm #419

merged 2 commits into from
Aug 4, 2024

Conversation

zhyncs
Copy link
Member

@zhyncs zhyncs commented Aug 3, 2024

@zhyncs zhyncs requested a review from yzh119 August 3, 2024 22:47
@zhyncs zhyncs changed the title feat: support fused add rmsnorm WIP feat: support fused add rmsnorm Aug 3, 2024
@yzh119
Copy link
Collaborator

yzh119 commented Aug 3, 2024

Can you write a test for this function?

@zhyncs
Copy link
Member Author

zhyncs commented Aug 4, 2024

Can you write a test for this function?

OK

@zhyncs
Copy link
Member Author

zhyncs commented Aug 4, 2024

Hi @yzh119 I have added unit tests to that PR in SGLang, and the verification shows no issues. I need to figure out how to proceed with FlashInfer.
https://github.com/sgl-project/sglang/pull/907/files#diff-ffbd8966453a6660e47ec2a8b0f5a71fee45b9e0fd729577ef28cfb60d3ee774

python3 python/sglang/test/test_layernorm.py

test_rms_norm (__main__.TestRMSNorm.test_rms_norm) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.608s

OK

@yzh119
Copy link
Collaborator

yzh119 commented Aug 4, 2024

I think adding a test case in https://github.com/flashinfer-ai/flashinfer/blob/main/python/tests/test_norm.py would be great.

@zhyncs
Copy link
Member Author

zhyncs commented Aug 4, 2024

I think adding a test case in https://github.com/flashinfer-ai/flashinfer/blob/main/python/tests/test_norm.py would be great.

sure!

@zhyncs
Copy link
Member Author

zhyncs commented Aug 4, 2024

I think adding a test case in https://github.com/flashinfer-ai/flashinfer/blob/main/python/tests/test_norm.py would be great.

@yzh119 done.

pytest python/tests/test_norm.py
=================================================================================== test session starts ===================================================================================
platform linux -- Python 3.11.5, pytest-8.3.2, pluggy-1.5.0
rootdir: /root/flashinfer/python
plugins: anyio-4.4.0
collected 48 items

python/tests/test_norm.py ................................................                                                                                                          [100%]

=================================================================================== 48 passed in 6.86s ====================================================================================

@zhyncs zhyncs changed the title WIP feat: support fused add rmsnorm feat: support fused add rmsnorm Aug 4, 2024
yzh119
yzh119 approved these changes Aug 4, 2024
View reviewed changes
Copy link
Collaborator

@yzh119 yzh119 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you @zhyncs !

@yzh119 yzh119 merged commit b781513 into flashinfer-ai:main Aug 4, 2024
@zhyncs zhyncs deleted the fused-add branch August 4, 2024 07:48
@github-actions github-actions bot mentioned this pull request Aug 4, 2024
Merged
yzh119 added a commit that referenced this pull request Aug 9, 2024
@github-actions @yzh119
chore(main): release 0.1.4 (#415)
1969433 
🤖 I have created a release *beep* *boop*
---
##
[0.1.4](v0.1.3...v0.1.4)
(2024-08-09)


### Features

* append attention kernels for fp8 kv-cache
([#420](#420))
([906c2f5](906c2f5))
* support min_p sampling
([#422](#422))
([d52f2da](d52f2da))
* deterministic sampling
([#417](#417))
([0dd801d](0dd801d))
* more sampling operator options
([#431](#431))
([68df9c4](68df9c4))
* support fused add rmsnorm
([#419](#419))
([b781513](b781513))
* support fused silu mul
([#427](#427))
([ea0ba9a](ea0ba9a))

### Bug Fixes

* fix dispatch fp16 type when enable fp8
([#430](#430))
([daa5566](daa5566))
* improve numerical stability of sampling kernels
([#429](#429))
([898d8ea](898d8ea))

### Other improvements
* break up `_kernels` into multiple modules
([#428](#428))
([8e482d9](8e482d9))

### Acknowledgement

We thank contributions and feedbacks from the community:
[@comaniac](https://github.com/comaniac),
[@esmeetu](https://github.com/esmeetu),
[@LiuXiaoxuanPKU](https://github.com/LiuXiaoxuanPKU),
[@peng1999](https://github.com/peng1999),
[@xslingcn](https://github.com/xslingcn),
[@Yard1](https://github.com/Yard1),
[@zhyncs](https://github.com/zhyncs).

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Zihao Ye <expye@outlook.com>
@zhyncs zhyncs added the enhancement New feature or request label Aug 27, 2024
@zhyncs zhyncs mentioned this pull request Nov 8, 2024
Merged
@github-actions github-actions bot mentioned this pull request Dec 25, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yzh119 yzh119 yzh119 approved these changes

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@zhyncs @yzh119