Add StableLM-3B 4E1T to Keras Hub #2151
Conversation
|
@divyashreepathihalli Here is a comparison of numerics with Hugging Face in Colab. The results match with an absolute tolerance of 1e-3, but they do not match when using 1e-5. Could you please take a look and suggest some improvements or explanations for this discrepancy? |
sachinprasadhs
added
the
WIP
|
The numerics is good enough! |
| self._inv_norm_factor = 1.0 / math.sqrt(head_dim) | ||
|
|
||
| # Query projection (no bias ) | ||
| self._query_dense = keras.layers.EinsumDense( |
There was a problem hiding this comment.
Let's just name layers without the underscore at the beginning. So, self.query_dense.
| self.rotary_dim = int(head_dim * self.rotary_percentage) | ||
| self._inv_norm_factor = 1.0 / math.sqrt(head_dim) | ||
|
|
||
| # Query projection (no bias ) |
There was a problem hiding this comment.
Nit: Remove the space after bias?
| num_layers: int. The number of transformer layers (32 for | ||
| StableLM-3B4E1T). |
There was a problem hiding this comment.
We probably don't need (32 for StableLM-2B4E1T) here (and elsewhere too), since it'll be part of the config anyway. Let's remove.
|
@Bond099 let's sync this with the latest changes and make sure to run our format script. I'm not exactly sure why non of our CI is running, but I don't think it ran. |
This PR adds the StableLM-3B 4E1T model to Keras Hub. However, numerical matching with the Hugging Face implementation is still in progress.