You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Generalize SwiGLU related python code. Create base classes and generalized functions to reuse by SwiGLU and other GLU-like activation functions that could be implemented in the future.
Motivation
This and some other changes allow for other similar activation functions (like GeGLU) to be implemented using the same codebase.
Additional context
I plan to add GeGLU with fused ops and packed weights implementation based on the SwiGLU code. This change is necessary to avoid code duplication.
The text was updated successfully, but these errors were encountered:
🚀 Feature
Generalize SwiGLU related python code. Create base classes and generalized functions to reuse by SwiGLU and other GLU-like activation functions that could be implemented in the future.
Motivation
This and some other changes allow for other similar activation functions (like GeGLU) to be implemented using the same codebase.
Additional context
I plan to add GeGLU with fused ops and packed weights implementation based on the SwiGLU code. This change is necessary to avoid code duplication.
The text was updated successfully, but these errors were encountered: