How to load LoRA adapter??

[momozzing]

Thank you for the open-source adapter!!

I'm using LoRA Adapter from adapter-hub

but not a working load adapter...

train setting.

    tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
    config = LoRAConfig(r = 8, alpha = 8)
    model = GPT2LMHeadModel.from_pretrained('gpt2)
    model.resize_token_embeddings(len(tokenizer))
    model.add_adapter("LoRA", config=config)
    model.train_adapter('LoRA', train_embeddings=True)
    
    trainable_params = 0
    frozen_params = 0
    for name, param in model.named_parameters():
        if "LoRA" in name:
            param.requires_grad = True
            emb_params += param.numel()
        else:
            param.requires_grad = False
    
        if not param.requires_grad:
            print(f"🥶 Frozen layer '{name}'")
            frozen_params += param.numel()
        else:
            print(f"🚀 Trainable layer '{name}'")
            trainable_params += param.numel()
    
    print(f"Total frozen parameters: {frozen_params}")
    print(f"Total trainable parameters: {trainable_params}")

training finish

    model.save_pretrained('/model_save')
    model.save_adapter('/model_save_adapter', 'LoRA')

Load setting.

    tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
    config = LoRAConfig(r = 8, alpha = 8)
    # model = GPT2LMHeadModel.from_pretrained('model_save/gpt2').cuda()
    model = AutoAdapterModel.from_pretrained('model_save/gpt2').cuda()
    model.resize_token_embeddings(len(tokenizer))
    LoRA_adapter = model.load_adapter(adapter_name_or_path='model_save/gpt2_adapter', model_name='gpt2', set_active=True)
    model.set_active_adapters(LoRA_adapter)
    # model.load_state_dict(torch.load('model_save/gpt2_adapter', map_location="cpu"))
    model.train_adapter('LoRA', train_embeddings=True)

but not working load_adapter...

How shall I do it?

my error message
Some weights of the model checkpoint at model_save/gpt2-20 were not used when initializing GPT2AdapterModel: ['transformer.h.2.attn.c_attn.loras.LoRA.lora_B', 'transformer.h.11.attn.c_attn.loras.LoRA.lora_B', 'transformer.h.6.attn.c_attn.loras.LoRA.lora_A', 'transformer.h.10.attn.c_attn.loras.LoRA.lora_A', 'transformer.h.3.attn.c_attn.loras.LoRA.lora_B', 'transformer.h.0.attn.c_attn.loras.LoRA.lora_B', 'transformer.h.2.attn.c_attn.loras.LoRA.lora_A', 'transformer.h.5.attn.c_attn.loras.LoRA.lora_B', 'transformer.h.6.attn.c_attn.loras.LoRA.lora_B', 'transformer.h.9.attn.c_attn.loras.LoRA.lora_A', 'transformer.h.8.attn.c_attn.loras.LoRA.lora_A', 'transformer.h.1.attn.c_attn.loras.LoRA.lora_B', 'transformer.h.4.attn.c_attn.loras.LoRA.lora_B', 'transformer.h.11.attn.c_attn.loras.LoRA.lora_A', 'transformer.h.4.attn.c_attn.loras.LoRA.lora_A', 'transformer.h.0.attn.c_attn.loras.LoRA.lora_A', 'transformer.h.5.attn.c_attn.loras.LoRA.lora_A', 'transformer.h.9.attn.c_attn.loras.LoRA.lora_B', 'transformer.h.10.attn.c_attn.loras.LoRA.lora_B', 'transformer.h.8.attn.c_attn.loras.LoRA.lora_B', 'transformer.h.7.attn.c_attn.loras.LoRA.lora_B', 'transformer.h.1.attn.c_attn.loras.LoRA.lora_A', 'lm_head.weight', 'transformer.h.3.attn.c_attn.loras.LoRA.lora_A', 'transformer.h.7.attn.c_attn.loras.LoRA.lora_A']
- This IS expected if you are initializing GPT2AdapterModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing GPT2AdapterModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

[calpt]

Hi @momozzing, thanks for reporting this. I believe this is an issue with the current version of our library that should be solved by #378.

[Ch-rode]

Hello ! I have the same problem unfortunately. Is it the bug resolved or we should wait until some other checks?
Thank you very much

[momozzing]

Hi @momozzing, thanks for reporting this. I believe this is an issue with the current version of our library that should be solved by #378.

@calpt Wow~ Thank you for solving this problem!

Hello ! I have the same problem unfortunately. Is it the bug resolved or we should wait until some other checks?
Thank you very much

@Ch-rode The code has not been merged yet.
So I changed the code in #384.
src/transformers/adapters/configuration.py
src/transformers/adapters/model_mixin.py

Then it works well!

[calpt]

The mentioned fix is now merged into the master branch, therefore this issue should be resolved when installing from the latest master branch version.