No chat template in evaluation

[xueyan-lii]

In LLAMA3 tokenizer, chat template is applied with the option to add system instruction. However, when using eleuther_eval.py, the tokenization process do not apply chat template. How can this be done?

[felipemello1]

@joecummings , mind taking a look?

[krammnic]

@felipemello1 It should be simple fix. We need to expose apply_chat_template argument in our wrapper (notice that it is not bool!). According to lm_eval: chat_template=getattr(lm, "apply_chat_template").

[pbontrager]

I explored using torchtune's apply_chat_template here but since Eleuther does it's own formatting, they'd conflict with each other. To echo what @krammnic already said, we need to

• override HFLM's "apply_chat_template" method in _LLMEvalWrapper shown here. This should stay mostly the same but grab the chat_template function that's inside the torchtune tokenizer here.
• We also need to verify that the apply_chat_template from HF tokenizer is doing the same thing as torchtune's apply template.

[krammnic]

@pbontrager Can I take this now?

[joecummings]

@pbontrager Can I take this now?

fire the lasers