More fine-grained control on LoRA

[HenryCai11]

Feature request

In my case,

• I want to customize learning rates for different module I want to tune with LoRA. I think this one is relevant to issue Ability to switch between different adapter types and mix different adapter types #1006
• Also, I want to only train specific layers, say, something like model.layer.1.self_attn.q (not sure if the q_value is represented this way)

Motivation

There are some existing Knowledge Editing literature suggesting that we can find some optimal places (layers) to modify our models

Your contribution

I would be happy to help. But I'm not very familiar with the code. I'd like to discuss and make some PRs if you can point out which parts I can take a look at. Thanks in advance!

[HenryCai11]

Sorry to bother, just found that the second one is not an issue and can be solved using layers_to_transform