Use jinja template for chat formatting (#730)

[nsarrazin]

• Upgrade transformers version and fix the types in transformersjs.ts

• Got rid of legacy parameters userMessageToken, userMessageEndToken, assistantMessageToken, assistantMessageEndToken so this is a breaking change but we've been pushing for using chatPromptTemplate for a while so I feel like it makes sense.

We support both chatPromptTemplate and just specifying the tokenizer. Priority goes chatPromptTemplate > tokenizer (more specific)

Did not change the templates in prod yet because they don't work well with system prompts

(#730)

[nsarrazin]

@mishig25 this should be good to go! am not using it in the prod config because most of the templates fail, but tested locally with some other models and it should work well.

[mishig25]

shouldnt the order of fallback be the other way around? First, try to use transfomersjs, then try to use chatPromptTemplate

[nsarrazin]

The downside of doing it this way is that for example in prod, we specify the tokenizer for token counting but we actually want to override the chat template with our own template.

I think in terms of specificity it makes sense to be custom chat template > tokenizer "default" template wdyt?

[mishig25]

yep sounds good. And if we want to change the order, let's do it in follow-up PR. Also, if we want to change the order, we would need to handle #744 (comment) first

[nsarrazin]

yes agreed, overall we should push towards using tokenizers rather than chat prompt templates, but we will need to handle the edge cases before indeed 😁

[mishig25]

right now, "system" role would case error on prompts that does not naturally support "system" right ?

[nsarrazin]

yep, I tried looking into it, but because the error messages are hardcoded in the jinja template, it's not obvious to me how to determine if the template is failing bc of a lack of system prompt or for some other reason

I guess you could retry without the "system" message and see if it builds the prompt then? we could do it in a second PR if that sounds good to you ?

[mishig25]

lgtm !

[flexchar]

So I pulled the latest changes and was met with the exciting greeting No tokenizer specified and no chat prompt template specified for model gpt-3.5-turbo-0125. Not so nice :)

I debugged my way to this PR introducing the error https://github.com/huggingface/chat-ui/blame/a9c711026b13672328ce93abc3865f1823524b0d/src/lib/server/models.ts#L133.

What is the migration path for this breaking change?

It would be nice to have such ones documented in a dedicated file. It would save a lot of time us all down the road. ✌️

[zacps]

I just ran into this as well, I'm not sure I understand why the default prompt template was removed?

It seems like keeping it would remove the need for a breaking change.

[nsarrazin]

Hey, apologies for the breaking change, fixed it in #985

My reasoning was that we want people to use either a tokenizer or chatPromptTemplate to ensure the chat is formatted correctly but of course if you use an endpoint that uses chat completion API, you don't need a chat template 🤦 So you would get an error complaining that something was missing even though you didn't need it.

I've set a default that should be sane (ChatML) which should fix the issue, and in the future I want to add a raw: true/false flag at the model level, so we can check if we need a prompt template at all and use the right API routes.

Seems like most people are moving away from text completion APIs and moving towards chat completion anyway 👀

[iChristGit]

Hey @nsarrazin
Ive been following for a while!
I had some issues that some models don't respond correctly to making up the "Title" and web search.
Does using the tokenizer instead fix the issue with the chatPromptTemplate being hard to set up for local models that are not on the prompt list?
https://github.com/huggingface/chat-ui/blob/main/PROMPTS.md

[nsarrazin]

Yes it should help! As long as your model has a chat_template in the tokenizer_config.json on the hub (example for google/gemma-1.1-7b-it) , you can just use the model name in the tokenizer field. (for example tokenizer: "google/gemma-1.1-7b-it" in this case)