Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix the fsdp peft autowrap policy #1694

Merged
merged 3 commits into from
May 1, 2024
Merged

fix the fsdp peft autowrap policy #1694

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
20ebeee
fix the fsdp peft autowrap policy
pacman100 Apr 30, 2024
40339ec
address comment wrt backwards compatibility
pacman100 Apr 30, 2024
cf239a2
Merge branch 'main' into smangrul/fix-fsdp-auto-wrap
pacman100 Apr 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion src/peft/utils/other.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -393,6 +393,11 @@ def fsdp_auto_wrap_policy(model):
import os

from accelerate import FullyShardedDataParallelPlugin

if hasattr(FullyShardedDataParallelPlugin, "get_module_class_from_name"):
get_module_class_from_name = FullyShardedDataParallelPlugin.get_module_class_from_name
else:
from accelerate.utils.dataclasses import get_module_class_from_name
from torch.distributed.fsdp.wrap import _or_policy, lambda_auto_wrap_policy, transformer_auto_wrap_policy

from ..tuners import PrefixEncoder, PromptEmbedding, PromptEncoder
Expand All @@ -405,7 +410,7 @@ def fsdp_auto_wrap_policy(model):
).split(",")
transformer_cls_to_wrap = {PrefixEncoder, PromptEncoder, PromptEmbedding}
for layer_class in transformer_cls_names_to_wrap:
transformer_cls = FullyShardedDataParallelPlugin.get_module_class_from_name(model, layer_class)
transformer_cls = get_module_class_from_name(model, layer_class)
if transformer_cls is None:
raise Exception("Could not find the transformer layer class to wrap in the model.")
else:
Expand Down
Loading