Qwen-Coder结束符是应该使用<endoftext>还是<im_end>? #1104
Unanswered
m-maoyanyu
asked this question in
Q&A
Replies: 1 comment
-
base 用 <|endoftext|>;instruct 用 <|im_end|> |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
请问,Qwen-Coder的config.json与generation.json中eos_token_id均为151643:"<|endoftext|>",但加载Qwen-Coder的tokenizer后,eos_token_id为151645:"<|endoftext|>" :"<|im_end|>"。直接使用Qwen-Coder推理时,也是输出"<|im_end|>"结尾。请问要以哪个作为结束符更好。
Beta Was this translation helpful? Give feedback.
All reactions