Whether chatglm3 6b is supported by trl ?

[fjy01]

System Info

trl 0.11.4
transformers 4.46.1

File "/home/fjy/folders/ERNIE2.0/finetune_chatglm6b.py", line 42, in
model = AutoModelForCausalLMWithValueHead.from_pretrained(model.model, trust_remote_code=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fjy/anaconda3/envs/env/lib/python3.12/site-packages/trl/models/modeling_base.py", line 233, in from_pretrained
model = cls(pretrained_model, **trl_model_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fjy/anaconda3/envs/env/lib/python3.12/site-packages/trl/models/modeling_value_head.py", line 107, in init
raise ValueError("The model does not have a language model head, please use a model that has one.")
ValueError: The model does not have a language model head, please use a model that has one.

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder
• My own task or dataset (give details below)

Reproduction

loading tokenizer

tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm3-6b-128k", trust_remote_code=True)

loading base model

model = AutoModelForCausalLM.from_pretrained("THUDM/chatglm3-6b-128k", trust_remote_code=True)

loading adapter

model = PeftModel.from_pretrained(model, adapter_dir)

wraping model with a value head.

model = AutoModelForCausalLMWithValueHead.from_pretrained(model.model, trust_remote_code=True)

The system throws an error at AutoModelForCausalLMWithValueHead.from_pretrained, The model does not have a language model head, please use a model that has one. Is it because trl does not support chatglm3? Because chatglm3 should have a language header, right?

Expected behavior

How can trl work with chatglm3.

[qgallouedec]

Thanks for reporting, should be fixed in #2328