AdaLoRA: self.ranknum[self.active_adapter] stays on CPU when training on GPU

[avostryakov]

adalora.py, line 443:

result += ( ( self.lora_dropout[self.active_adapter](x) @ (self.lora_A[self.active_adapter] * self.lora_E[self.active_adapter]).T @ self.lora_B[self.active_adapter].T ) * self.scaling[self.active_adapter] / (self.ranknum[self.active_adapter] + 1e-5) )

I got the following error because self.ranknum[self.active_adapter] is on CPU and other tensors on GPU:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

I used the following AdaLoraConfig:

config = AdaLoraConfig( r=8, target_r=8, init_r=16, tinit=200, tfinal=1000, lora_alpha=32, target_modules=target_modules, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM" )

[ShayDuane]

same problem here.....
can not put the ranknum into cuda
do you already have a solution?

[younesbelkada]

Hi @avostryakov
This should be fixed in #583
Also duplicate of : #581
Please re-install peft from source and the error should disappear, closing the issue for now - feel free to re-open in case you think that it has not been addressed

[avostryakov]

Thank you!