Skip to content

Commit

Permalink
Fix the problem of not using the decoding method corresponding to the…
Browse files Browse the repository at this point in the history
… base model in peft mode (lm-sys#2865)
  • Loading branch information
Jingsong-Yan authored and zhanghao.smooth committed Jan 26, 2024
1 parent 2914b62 commit f616540
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion fastchat/model/model_adapter.py
Original file line number Diff line number Diff line change
Expand Up @@ -375,10 +375,14 @@ def get_generate_stream_function(model: torch.nn.Module, model_path: str):
from fastchat.serve.inference import generate_stream

model_type = str(type(model)).lower()
is_peft = "peft" in model_type
if is_peft:
model.set_adapter(model_path)
model_type = str(type(model.base_model.model))

is_chatglm = "chatglm" in model_type
is_falcon = "rwforcausallm" in model_type
is_codet5p = "codet5p" in model_type
is_peft = "peft" in model_type
is_exllama = "exllama" in model_type
is_xft = "xft" in model_type

Expand Down

0 comments on commit f616540

Please sign in to comment.