forked from huggingface/peft
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
FIX: Sft train script FSDP QLoRA embedding mean resizing error (huggi…
…ngface#2151) Resizing the embedding layer with mean_resizing=True, which has been introduced in transformers > 4.45, will result in an error. This is because for FSDP + QLoRA the embedding matrix can be on meta device, in which case mean resizing fails. Therefore, if these conditions are detected, the script will set mean_resizing=False. Also updated the recommended package versions to newer versions that I have checked to be working.
- Loading branch information
1 parent
1c701dc
commit 0a8f42e
Showing
4 changed files
with
16 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters