Cannot use Compacter+LoRA adapters with Roberta model

[AWilcke]

Environment info

• adapter-transformers version: 3.2.0
• Platform: Linux-5.15.0-1025-gcp-x86_64-with-glibc2.29
• Python version: 3.8.10
• Huggingface_hub version: 0.13.1
• PyTorch version (GPU?): 1.10.2+cu113 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: Yes, A100
• Using distributed or parallel set-up in script?: No

Information

Model I am using (Bert, XLNet ...): Roberta

Language I am using the model on (English, Chinese ...): English

Adapter setup I am using (if any): Compacter+LoRA

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset: (give details below)

To reproduce

from transformers.models.roberta import RobertaModel
model = RobertaModel.from_pretrained("roberta-base")
model.add_adapter("adapter", "compacter|lora")

Raises the following error

File ~/env/lib/python3.8/site-packages/transformers/adapters/model_mixin.py:487, in ModelAdaptersMixin.add_adapter(self, adapter_name, config, overwrite_ok, set_active)
    485 self.config.adapters.add(adapter_name, config=config)
    486 try:
--> 487     self._add_adapter_weights(adapter_name)
    488 except ValueError as ex:
    489     self.delete_adapter(adapter_name)

File ~/env/lib/python3.8/site-packages/transformers/adapters/model_mixin.py:503, in ModelAdaptersMixin._add_adapter_weights(self, adapter_name)
    501     if isinstance(adapter_module, nn.ModuleList):
    502         adapter_module = adapter_module[0]
--> 503     self.base_model.shared_parameters[adapter_name] = adapter_module.adapter_down[0].init_shared_parameters()
    504 # Prefix Tuning
    505 for module in self.modules():

File ~/env/lib/python3.8/site-packages/torch/nn/modules/module.py:1177, in Module.__getattr__(self, name)
   1175     if name in modules:
   1176         return modules[name]
-> 1177 raise AttributeError("'{}' object has no attribute '{}'".format(
   1178     type(self).__name__, name))

AttributeError: 'LoRA' object has no attribute 'adapter_down'

Expected behavior

I would expect to be able to use Compacter+LoRA with a Roberta model, since it works for a T5 model for example. If this combination of adapters is not supported the error message should make that clearer, such as when combining Compacter and Compacter++.