activation function in BERTIntermediate #15
Comments
|
Yes, I hard coded that since the pre-trained models are all trained with gelu anyway. |
|
ok. but since config is there anyway, isn't it cleaner to use it (to avoid errors for people using configs that use a different activation for some reason) ? |
|
Yes we can, I'll change that in the coming first release (unless you would like to submit a PR which I would be happy to merge). |
|
yeah let me clean up and I'll PR |
Closed
2 tasks
rraminen
pushed a commit
to rraminen/transformers
that referenced
this issue
Oct 27, 2022
…r-2022-08-09 IFU-master-2022-08-09
jameshennessytempus
pushed a commit
to jameshennessytempus/transformers
that referenced
this issue
Jun 1, 2023
1 task
ocavue
pushed a commit
to ocavue/transformers
that referenced
this issue
Sep 13, 2023
…/webpack-5.76.0 Bump webpack from 5.75.0 to 5.76.0
LysandreJik
pushed a commit
that referenced
this issue
Mar 15, 2024
* Cohere Model Release (#1) Cohere Model Release * Remove unnecessary files and code (#2) Some cleanup * Delete cohere-model directory (#3) * Make Fix (#5) * Pr fixes (#6) * fixes for pr * pr fixes for the format * pr fixes for the format * src/transformers/models/auto/tokenization_auto.py * Tokenizer test (#8) * tokenizer test * format fix * Adding Docs and other minor changes (#7) * Add modeling tests (#9) * Smol Fix (#11) * tokenization tests are fixed * format fixes * fix pr doc tests * fix pr doc tests * fix pr doc tests * fix pr style check * small changes in cohere.md * FIX: Address final comments for transformers integration (#13) * fix modeling final nits and add proper test file * for now leave empty tests * add integration test * push new test * fix modeling cohere (#14) * Update chat templates to use the new API (#15) --------- Co-authored-by: ahmetustun <ahmetustun89@gmail.com> Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>
LysandreJik
pushed a commit
to LysandreJik/transformers
that referenced
this issue
Apr 10, 2024
ArthurZucker
pushed a commit
that referenced
this issue
Sep 25, 2024
Interpolate embeddings for 560 size and update integration tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
BERTConfig is not used for
BERTIntermediate's activation function.intermediate_act_fnis alwaysgelu. Is this normal?https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/modeling.py#L240
The text was updated successfully, but these errors were encountered: