Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate chat templates #5

Closed
amakropoulos opened this issue Jan 9, 2024 · 1 comment · Fixed by #94
Closed

Integrate chat templates #5

amakropoulos opened this issue Jan 9, 2024 · 1 comment · Fixed by #94
Labels
enhancement New feature or request

Comments

@amakropoulos
Copy link
Collaborator

HuggingFace has enabled chat templates alongside the model definitions.
"They specify how to convert conversations, represented as lists of messages, into a single tokenizable string in the format that the model expects."
This is part of a WIP PR from llama.cpp.
Once this is merged and integrated in llamafile it will be also incorporated here.

@amakropoulos amakropoulos added the enhancement New feature or request label Jan 9, 2024
@amakropoulos amakropoulos moved this from Todo to Blocked in LLM for Unity Roadmap Jan 9, 2024
@amakropoulos
Copy link
Collaborator Author

amakropoulos commented Feb 20, 2024

Chat templates seem to be blocked due to lack of simple c/c++ implementation.
jinja2 is a python engine and probably the only jinja compiler in c/c++ is Jinja2Cpp which is quite heavy.

references:
https://www.reddit.com/r/LocalLLaMA/comments/1afweyw/quick_headsup_about_using_codellama_70b_and/
ggerganov/llama.cpp#4216 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

1 participant