Adapter weights are cast in float16 unexpectedly after load_adapter

[hiyouga]

System Info

Ubuntu + NVIDIA V100

transformers 4.34.1
peft 0.6.0

Who can help?

@pacman100 @younesbelkada

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder
• My own task or dataset (give details below)

Reproduction

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

tok = AutoTokenizer.from_pretrained("llama2-7b")
model = AutoModelForCausalLM.from_pretrained("llama2-7b", torch_dtype="auto", device_map="auto")

adapter_model = PeftModel.from_pretrained(model, "lora_weight", is_trainable=True)
for name, param in adapter_model.named_parameters():
    print(name, param.dtype) 

"""
base_model.model.model.layers.31.self_attn.v_proj.weight torch.float16
base_model.model.model.layers.31.self_attn.v_proj.lora_A.default.weight torch.float32 (correct)
base_model.model.model.layers.31.self_attn.v_proj.lora_B.default.weight torch.float32
"""

adapter_model.load_adapter("another_lora_weight", "reward", is_trainable=True)
for name, param in adapter_model.named_parameters():
    print(name, param.dtype)

"""
base_model.model.model.layers.31.self_attn.v_proj.weight torch.float16
base_model.model.model.layers.31.self_attn.v_proj.lora_A.default.weight torch.float16 (incorrect)
base_model.model.model.layers.31.self_attn.v_proj.lora_A.reward.weight torch.float16
base_model.model.model.layers.31.self_attn.v_proj.lora_B.default.weight torch.float16
base_model.model.model.layers.31.self_attn.v_proj.lora_B.reward.weight torch.float16
"""

Besides, add_adapter also triggers a similar problem for peft 0.6.0.

Expected behavior

The data type of adapter weights after load_adapter should remain float32

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.