Add support for Task Arithmetics #698
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
6 tasks
…statements in python
lenglaender
changed the title
lenglaender
requested review from
calpt,
TimoImhof and
hSterz
and removed request for
calpt and
TimoImhof
calpt
reviewed
Jul 11, 2024
hSterz
reviewed
Jul 13, 2024
docs/adapter_composition.md
Outdated
| \Phi_{merged} = \sum_{i=0}^{N} \lambda_i \Phi_i | ||
| $$ | ||
|
|
||
| 2. `combine_strategy = "lora_linear_only_negate_b"` Following [Zhang et al. (2023)](https://proceedings.neurips.cc/paper_files/paper/2023/hash/299a08ee712d4752c890938da99a77c6-Abstract-Conference.html), this method only uses negative weights for the B-matrix if the weight is negative: |
There was a problem hiding this comment.
The only in the name is redundant. I would remove it to make it shorter
There was a problem hiding this comment.
I believe it is best to keep it. Because if we simply call it lora_linear_negate_b, it sounds like the B matrix is always negated. But this method means that when the weights are negative, then we only negate the B matrix and not the A matrix.
Co-authored-by: calpt <calpt@mail.de>
Co-authored-by: calpt <calpt@mail.de>
- move adapter merging to own docs page - move `average_head` method from `ModelAdaptersMixin` to `ModelWithHeadsAdaptersMixin` - In lora.py: Move SVD computation in helper function cause it was a bit to lengthy in the `average_adapter` function - test_lora.py: split test cases
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for various task arithmetic options for LoRA. Until now, our library supported averaging only by linearly combining different adapters. This may be insufficient, especially for LoRA — hence, several publications have proposed other ways to perform task arithmetic.
This PR: