-
Notifications
You must be signed in to change notification settings - Fork 339
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add adapter support to BEiT #428
Add adapter support to BEiT #428
Conversation
Tests initially failed, as adapter support for BEiT was implemented without accounting for latest changes to main branch (UniPELT). I adjusted the implementation and re-trained adapters on the above described model achieving similar accuracy (88 % after 20 epochs). |
On branch add-adapter-support-to-beit Changes to be committed: modified: src/transformers/__init__.py modified: src/transformers/adapters/__init__.py modified: src/transformers/adapters/head_utils.py new file: src/transformers/adapters/mixins/beit.py modified: src/transformers/adapters/models/auto/adapter_model.py new file: src/transformers/adapters/models/beit/__init__.py new file: src/transformers/adapters/models/beit/adapter_model.py modified: src/transformers/adapters/wrappers/configuration.py modified: src/transformers/models/beit/modeling_beit.py new file: tests_adapters/test_beit.py
On branch add-adapter-support-to-beit Changes to be committed: new file: adapter_docs/classes/models/beit.rst
On branch add-adapter-support-to-beit Changes to be committed: modified: src/transformers/adapters/mixins/beit.py modified: src/transformers/adapters/models/beit/adapter_model.py modified: src/transformers/models/beit/modeling_beit.py modified: tests_adapters/test_beit.py
54f3a00
to
dc58d79
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @jannik-brinkmann,
thank you very much for your contribution! The implementation looks very good so far, just pointing out some minor points from the contribution guide that are still missing:
Implementation:
- Add <model_type>AdapterModel to the ADAPTER_MODEL_MAPPING_NAMES mapping in src/transformers/adapters/models/auto.py
Testing:
- Append <model_type> to the list in check_adapters.py.
Documentation:
- [...] Finally, list the file in index.rst.
- Add a new row for the model in the model table of the overview page at adapter_docs/model_overview.md, listing all the methods implemented by the new model.
On branch add-adapter-support-to-beit Changes to be committed: modified: adapter_docs/index.rst modified: adapter_docs/model_overview.md modified: utils/check_adapters.py
Hi @calpt, thank you for your review! Please find below my response to each of your comments: Implementation:
Testing:
Documentation:
|
Thanks for the quick response.
You're absolutly correct, my apologies. Will update this in our docs.
Thanks for the feedback, will make this more clear. |
Thanks again for your contribution @jannik-brinkmann! We'd be happy to see any adapters you train with BEiT uploaded to the HuggingFace Hub. :) |
Integrates adapters into BEiT model. I ran adapter_tests/test_beit.py without errors and trained a BEiT-Base model with "houlsby" adapters on EuroSAT dataset achieving 86 % accuracy after 20 epochs with LR 5e-4 (still improving about 1 % per epoch) without any hyperparameter tuning. For comparison: With full fine-tuning of BEiT-Base on EuroSAT I achieved 89 % accuracy after 10 epochs with LR 3e-5.