Make chat templates part of ProcessorMixin #30744
Conversation
|
Hi @Rocketknight1, very nice! FYI, I'm working on standardizing the processors here #30511 and I was thinking how to make this compatible with apply_chat_template - thought you might want to take a look at the design here, and I'll keep an eye here as well :) |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
@molbap I didn't see your PR, but we should definitely use your approach! Supporting chat templates should be simple enough - all they require is moving |
|
I don't think you need to pause yours! I'll try and get mine merged soon - it's already approved, just need to make sure nothing breaks |
|
Cool! I'll probably wait until yours is merged, in that case, and then adding chat templates should be clean and straightforward. |
Rocketknight1
force-pushed
the
processor_chat_templates
branch
from
302cac4 to
e844dd3
Compare
Rocketknight1
force-pushed
the
processor_chat_templates
branch
from
9ef139d to
c440ca5
Compare
|
This should be ready - cc @amyeroberts for core maintainer review! I'm speeding things up because the lack of this feature is causing some user issues, and we have a lot of VLMs coming. |
* Let's try moving chat templates out of IDEFICS and into the generic ProcessorMixin * Chat templates should not be mandatory * Chat templates should not be mandatory * Not all classes will have default chat templates * stash commit * Add chat template docstring * Clean up docstring * Add chat templates to LLaVA/LLaVA-next * Docstring fixup * Quick IDEFICS2 fixup * Remove some old references to the Conversation class * make fixup
* Let's try moving chat templates out of IDEFICS and into the generic ProcessorMixin * Chat templates should not be mandatory * Chat templates should not be mandatory * Not all classes will have default chat templates * stash commit * Add chat template docstring * Clean up docstring * Add chat templates to LLaVA/LLaVA-next * Docstring fixup * Quick IDEFICS2 fixup * Remove some old references to the Conversation class * make fixup
* Let's try moving chat templates out of IDEFICS and into the generic ProcessorMixin * Chat templates should not be mandatory * Chat templates should not be mandatory * Not all classes will have default chat templates * stash commit * Add chat template docstring * Clean up docstring * Add chat templates to LLaVA/LLaVA-next * Docstring fixup * Quick IDEFICS2 fixup * Remove some old references to the Conversation class * make fixup
* Let's try moving chat templates out of IDEFICS and into the generic ProcessorMixin * Chat templates should not be mandatory * Chat templates should not be mandatory * Not all classes will have default chat templates * stash commit * Add chat template docstring * Clean up docstring * Add chat templates to LLaVA/LLaVA-next * Docstring fixup * Quick IDEFICS2 fixup * Remove some old references to the Conversation class * make fixup
* Let's try moving chat templates out of IDEFICS and into the generic ProcessorMixin * Chat templates should not be mandatory * Chat templates should not be mandatory * Not all classes will have default chat templates * stash commit * Add chat template docstring * Clean up docstring * Add chat templates to LLaVA/LLaVA-next * Docstring fixup * Quick IDEFICS2 fixup * Remove some old references to the Conversation class * make fixup
* Let's try moving chat templates out of IDEFICS and into the generic ProcessorMixin * Chat templates should not be mandatory * Chat templates should not be mandatory * Not all classes will have default chat templates * stash commit * Add chat template docstring * Clean up docstring * Add chat templates to LLaVA/LLaVA-next * Docstring fixup * Quick IDEFICS2 fixup * Remove some old references to the Conversation class * make fixup
Right now, IDEFICS2 supports chat templates, but in class-specific code. In this PR, we'll try to move that support to the generic
ProcessorMixinso that we don't have to keep duplicating it as more VLMs (like LLaVA) start to need chat templates.Draft PR for now until I make sure this doesn't break things!