Citation for ViT-Adapters

[arijitthegame]

Hello,

Thank you for the excellent repo. Can you please provide a citation for the adapters in ViT? I would love it for my research.

[lenglaender]

Hello @arijitthegame,

Thank you for your kind words about our repository! Regarding the adapters in ViT, can you provide a link to the specific adapters you're referring to?

[arijitthegame]

I am referring to the Houlsby config. More specifically, I am interested in finding a reference that talks about Houlsby adapters to different parts to a ViT layer and of course some baseline results that are already published would be amazing.
https://github.com/adapter-hub/adapter-transformers/blob/master/src/transformers/models/vit/modeling_vit.py#L378 (which paper talks about adding an adapter here ) and in here (https://github.com/adapter-hub/adapter-transformers/blob/master/src/transformers/models/vit/modeling_vit.py#L343)

Thank you so much!!

[lenglaender]

Our implementation isn't based on a specific ViT adapter paper but simply follows the implementation we used for other transformer architectures. Hence, there are currently no baseline results.

However, there are some papers showing that adapter-based approaches work well for transfer learning with ViT:

• "AdaptFormer: Adapting Vision Transformers for Few-shot Learning" by Chen and Ge (2022)
• "Towards a Unified View on Visual Transfer Learning" by Yu and Chang (2022)
• "Parameter-efficient Fine-tuning for Vision Transformers" by He and Li (2022)

For the concrete implementation details of our adapter-transformers ViT adapter, you can cite our AdapterHub paper or the soon-to-be-released "Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning" paper, see #584