Generic Chat templating code with text/json file based config; main chat updated to drive its in-prefix, in-suffix and reverse-prompt from same; chat-apply-template equivalent c-api to allow use by other codes also #6834

Move [inst] [/inst] wrt llama2 from global to individual role specific parts. Avoid an extra \n wrt prefixes of llama3

Update the note Rename global-prefix|suffix to global-begin|end. Rename chat-apply-template to chat-apply-template-single, cas it handles only a single message. Add some debug log messages to the helper functions

Avoid adding global begin/end markers wrt ChatApplyTmplSingle. Add ChatApplyTmpl which goes through a vector of messages.

Llama2 seems to need it, so chaton-meta-json sample file updated to use same.

For a given template-id, cross check, all needed entries are there in the json.

rename because they return value of specified key. [main] update metaok to take template-id, so that one can cross check that all needed entries are there wrt that template-id in the chaton-meta-json file

Order the functions so that no need for seperate prototypes Also use kv_bool wrt boolean entries. Convert string to c char *

Avoid showing on screen the debug messages. meta-dump can either show on screen or not, based on how LOGXLN is defined.

Was looking at the tokenized vector, and noticed that the EOS mentioned by existing chat_apply_template of llama.cpp, is different from what I noticed in tokenizer_config.json of deepseek llm, so I have added two entries * "deepseek-alt" which matches llama.cpp's chat_apply_template and * "deepseek" which matches that in tokenizer_config.json. This impacts the assistant suffix and reverse prompt entries. CasOfThis: Need to look into other entries which I added previously at a later time. However as the default logic should be picking the EOS from model file, so I assume reverse-prompt being outofsync, may not matter beyond a limit, potentially.

Dump shows user->begin. chat-template-apply[-single] updated to work with begin and prefix TODO: need to wrap begin in a try-catch, so that irrespective of role, begin+prefix will work, irrespoective of whether that role has a begin entry or not.

so that even if a role doesnt contain begin, the logic will work fine.

Inturn Begin/BoS is added only for non 1st user messages in a system+user prompts chain.

Use the same for user role's begin and prefix entries.

Helps keep user prompt and chat-hs-template tag parts seperate, but in sequence

Now there is a simple and extended version of returning tagged message wrt a single role and its content. The extended version returns the tagged string, as well as the details of the parts that make up that tagged message interms of the type of parts and the lengths of the parts.

Now there is a simple and extended version of returning tagged messages. The extended version returns the tagged string, as well as the details of the parts that make up that tagged message interms of the type of parts and the lengths of the parts.

Cas of above reason, switch to directly accessing the keys in dump helper, which is inturn used by meta_ok check

Also fix a oversight wrt begin, when flag based begin adding control was introduced. NOTE: Currently system role suffix/end conditional adding always triggered, if 1st system prompt seen or additional system prompt is seen.

Because user messages dont normally need a EoS token.

Move helpers to the begining, so can avoid adding prototype declerations/function signatures to the begining Get the char * wrt string data in the c++ string.

Wrt llama2 * add bos wrt llama2 system and user begins, but not assistant * split system suffix into suffix and end, and add systemuser-system flags so that end can be avoided wrt system+user message combo * add eos wrt assistant end * With these potentially this should work with main and server flows Wrt llama3 * add empty begin, end fields and systemuser-system flags * This should potentially work with main and server flows

Fix a oversight wrt key name. Add a alert in case if passed meta json file contains begin(BoS) wrt assistant role, similar to check for end (EoS) wrt user role. Bcas normally both (ie EoS wrt User and BoS wrt Assistant) shouldnt be needed. Update main wrt begin & prefix and suffix & end addition.

Add a c api wrapper for a single message tagging scenario. Inturn to match convention followed by existing chat_apply_template code, make it return the size expected of the tagged message string buffer. Update internal single logic to help with same. Explicitly check if tmpl specified is available in the loaded json or not and then return a error if not found.

While sending the current chat session along with new user query to the model, many models expect that a tag be added at the end to indicate that user is expecting the model to respond, this flags allows for the same.

Initial skeletons Update existing logics to help with same. Also the inbetween helper was having a bad signature wrt returning status and data, thats also fixed.

As c doesnt have the concept of pass by reference, and inturn the existing c api uses pointers wrt llama chat message structure, so switching to same wrt chat_tmpl_apply logics. Also fix a oversight in previous commit and add the remaining logic.

to be in sync with the json namespace in server.

Had forgotten to update it before

Also as part of same add user role entry for system role also.

Pick eos from zephyr's tokenizer_config, which is different from what was hardcoded in the existing llama_chat_apply_template.

Add missing begin and end fields wrt deepseek-coder assistant in chaton_meta.json. Idea is to avoid json library dependency by adding a simple text based config file support.

Now one can pass the list/string of chars to trim at either end.

now that logic has been converted into a class, no need for this prefix

TODO: Have to look into C++ a bit later including these default container types. Not sure my current flow is efficient.

so that one can update the value-item's content, without needing to explicitly update/store the value-item back into map after the content has been updated. This should make these setting operations/helpers more efficient.

Also update the sample simpcfg file, to test for int and float values.

Warn if something not starting with double quote is being treated as a string. Show some examples of invalid floating point values wrt this logics floating point determination code

test-chat-template-chaton now tries to check if meta-ok is ok wrt the template-id being looked into. Log template-id info also, where it was previously missed out.

Cleanup the use of the variant Initialize and << op stringstream seperately.

TODO: string check wrt true/false, doesnt seem to be working after str_tolower was introduced. I seem to be doing some silly mistake not able to make out, moving in and out of sleep, need to check tomorrow. string == string-literal failed string == string-view failed string.compare(string-literal) failed Bit strange

Also the warning wrt is it string, now also logs the line number, group and key, to help user identify the line better. Misc: pass time last week Another life, Anchakkallakokkan, Deadloch

Bring the direct string comparison also back, as the issue with cmp was more to do with string.transform.

By having a seperate entry for each element of the array, the existing logic itself can be repurposed with minimal additions.

Good: Forgotten love, All the light we cant see, laapataa ladies

wcouts' involving 2nd wstring with non english char in it not showing up?

Without using imbue, I couldnt get non-english wstrings to print on mac. Need to check on linux also. Also avoid the uint8_t typecasting, given that wchar isnt 8bit

Seperate out the checks wrt different string types. Add a wstring_basic, which verifies that wstring iterator handles non english chars propery or atleast better.

The constructor method doesnt convert wstring to string, when it involves non-english chars which will encode to multibyte chars in utf8. even thou it does work for the already utf8 u8string. wcstombs doesnt seem to work for non english chars, when the locale is set to the default c, need to change to something like en_US.UTF-8, to allow it to do the conversion properly.

Update the notes to match the templated flow now and some of the nitty gritties involved. Update DumpHexString to be templated. Split check nonenglish flow wrt trim dumb and oversmart testing, so that things with work with one, but not the other can be differentiated in the flow.

So as to make the notes, more generic.

Also use NativeCharSize and MultiNativeCharSize wording to make the note more generic

Use the commonality between Indian languages to show mixup issue with the simple minded trim_dump logic and how trim_oversmart could potentially avoid that. Given that I am using valid strings to show the pitfalls of fixed native char size driven logic, so no need to keep the dump and oversmart flows seperate, so merge into a common loop.

template info picked from tokenizer config's default entry, Verified that same is used in the existing hardcoded chat apply template flow.

With this and past few commits, now there is simple yet sufficient support to help move multi-level-hierarchy config files into the SimpCfg's simple physically 1-level, but if reqd logically multi level hierarchy flow. B4 this series of commits also one could have still achieved this, but there would have been bit more effort needed.

Got from its tokenizer config json. Also same found in existing hardcoded template in default chat apply template logic of llamacpp

The first model seen, based on templates added till now into meta json file, that needs a Global Begin. From tokenizer_config json file, it appears like even system role should have a appropriate prefix, unlike what is seen in hardcoded default chat apply template of llama.cpp and chat jinja template.

Have looked at tokenizer_config.json, jinja file and default hardcoded template in llama.cpp. This is also one of the models where a Global BoS is needed. NOTE: Have taken the liberty to also add a SYSTEM: prefix wrt system message, even thou default vicuna doesnt seem to need, but vicuna-orca seems to need, so that both models can be driven from same chat template config. I am assuming the system prefix should not create any problem even in default vicuna, however if it does create a problem one can duplicate the existing vicuna block in chaton_meta.json and make the system prefix empty in it.

so that the size of the elements is explicit and fixed, so that it is inturn in sync with the fixed int size specified wrt the c-api, even with any c compilers with different idea about int. avoid some ununsed vars, need to update compile flags later to enable corresponding warnings.

If the tagged message will be of 0 length, ensure that the passed dest char* array, has null inserted appropriately. Check that user has passed a non-null pNumParts. Dont hard code int32_t size, pick using sizeof

Initial go

Have merged master branch has of 20240510IST12XY with chaton_v3 branch. As part of same had to update the flow in examples/main/main.cpp wrt conversion related commit in master branch and my chaton related commits in this branch.

It appears like std::format is not supported in older g++/lib still in wide use like current debian stable, so avoiding same wrt direct library use. Allow for empty VAARGS NOTE: However test program mode of the same uses cout and format

This should pave way for having a default chat templates dataset in the code, without needing to load it from a config file, if one doesnt want to. TODO: allow for loading config from json into simpcfg, so that a program which uses llama.cpp can decide, whether it is ok with what is already there in the internal dataset, or allow for loading template info at runtime using the simpcfg's simple text file or additionally include the json code to load template info at runtime from json file.

IE a minimal MapOfMapOfVariant, with some basic helpers. This can be the basis of a ChatTemplates object, as well as SimpCfg built on top of it.

Rename all the log messages to have GKV and not SC. The log messages in get_vector made conditional to GKV_DEBUG, this was missed out earlier in simpcfg itself.

So that no need to explicitly specify <int64_t> or LL wrt int literals, which dont need 64bit space by default. Which also means one shouldnt/cant mix up type of value stored and default type specified when getting.

Also change LWARN to LDBUG wrt previously GKV_DEBUG conditional code

Avoid defining GKV_TEST_PRG, used for self testing, by default Add it to common library

code used for exploring/testing commited just for future reference

Manually iterate the json object items using begin-end explicitly, because the implicit iteration for loop related helpers for the used json lib gives only the values and not a key-value pair.

WIP:NOTE: Initial go converting from json driven flow to ChatTemplatesGroupKV related flow done. Needs to be tested. A optional helper added to load ChatTemplates from a specified json file. Need to add a compile time initialized MapOfMapOfVariants wrt the chat template details of models/standards already known to the program. So that one can use the llama.cpp and this new chat template logic, even without json dependency, if one doesnt want to.

load the json file and put the template ids

Use repr to retain the escape sequences in the read string. And parallely skip the single quote around strings wrt repr. Bring in more k-v pairs wrt chaton_meta.json

Rename kv helpers to match their semantic. * whether working with string or bool value * whether two keys or a single key Add support for kv with bool value inturn add the kv boolean pairs used in the chaton_meta.json file Add the closing bracket

Also comment to indicate that the hpp file is auto converted from the chaton_meta.json file

This should allow for using this generic chat templating code flow along with the included chat template data, without needing to load any json file at runtime. However If user wants to change the already included chat template data, or add new chat template standard/model related data, one can explicitly load json file. TODO: Need to cross check this flow once, but logically should work

This should be ok, given that there is a version of the chat tmpl meta data already included with the library. So only if user wants to change the chat template info wrt a existing model/template-standard or add a new one, then there is need to pass a json file with info for that model/standard.

Also has dump was using get_value calls with fallback to default, so it wasnt identifying the missed field. Have fixed both of those. Also reconverted meta json file. Misc: interesting avesham and aattam

The json library generates less informative exception message, which doesnt help one identify which key is missing, so switch to the new json_get_str helper added in the last commit. It generates more informative exception message.

With this now even loading chaton_meta.json file will generate more informative exception, so that user can know which field is missing, if any.

Reuse the code already moved into GroupKV Add explicit get and set wrt int32_t, which was added after move to GroupKV wrt basic MapOfMapOfVariant logic.

Also set functions to inline or static as appropriate

The code which was using SC_DEBUG moved to GroupKV and inturn GKV_DEBUG

Also explicitly indicate that we are looking at a chain of keys

ChatTemplate directly supports these now, as well as the existing global instance based corresponding helpers depend on same.

To avoid having to duplicate any hardcoding in future, wrt any new model/chat-template-standard, at multiple locations, remove the single message templating code with a wrapper which does the same but using the multi-msg templating helper.

Given that now the multi chat templating logic itself is used to apply chat templating/tagging to a single chat message, so give flexibility of deciding whether global tags if any should be applied or not wrt the core tagging logic. examples/main inturn updated to not apply global tags if any wrt the system message. Also the user messages already dont apply global tags if any, as its currently implemented to build on the existing in-prefix/suffix and anitprompt flow.

Make it similar to user-begin+prefix control. ie only wrt 1st msg of respective type.

Use same to bypass any msg count based tagging behaviour for the single message tagging through its helper wrapper.

However still retain the wrappers, which work with a predefined global instance of ChatTemplates.

GroupKV dump adds needed ":" seperator on its own, so calling functions can just pass the tag string they want in the log without worrying about any demarkation.

Also add simple note wrt itself and its helper.

The initial version was rooted around a json object, while the new version is rooted around a MapOfMapOfVariant (GroupKV), which could be preloaded with chat templates info at compile time itself and used as is. Or optionally one could allow the configurable template data to be extended/updated at runtime from a text(/SimpCfg)/json file.

Any program which wants to use json file to update/extend the chaton's configurable template data, can include this new file chaton_json.hpp, to get the reqd functionality. Update chaton_meta_ok, _chaton_meta_validate_dump and chaton_meta_load_json to either work with a passed ChatTemplates instance, or fallback to the compiled-in global instance of same.

Merge upstream as of 20240515IST11XY

Rename chaton-meta hpp to cpp and include this cpp file which brings in the compile time built-in global chaton configurable template data into the common library, and avoid the nop hpp file references. Update chaton.hpp to not include the meta-cpp, instead just make a reference to the global ChatTemplates instance, so that the hpp can be used as a header file proper. Avoid pragma once in the chaton-meta.cpp, including the script, which helps create it.

C++17 provides a good enough variant as a standard feature, and chaton uses the same at its core, instead of rolling out its own struct of union based variant. And given that currently chaton is part of common library and not the base llama library, so limit the use of c++17 to common library. Initially while experimenting, had set the flag for full llama, limitting it for now. Also by now most embedded targets should be potentially having c++ compilers and libraries with support for c++17 features. So chances are it is a ok enough path to take.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generic Chat templating code with text/json file based config; main chat updated to drive its in-prefix, in-suffix and reverse-prompt from same; chat-apply-template equivalent c-api to allow use by other codes also #6834

Generic Chat templating code with text/json file based config; main chat updated to drive its in-prefix, in-suffix and reverse-prompt from same; chat-apply-template equivalent c-api to allow use by other codes also #6834

Commits on May 6, 2024

Commits on May 7, 2024

Commits on May 8, 2024

Commits on May 10, 2024

Commits on May 11, 2024

Commits on May 12, 2024

Commits on May 13, 2024

Commits on May 14, 2024

Commits on May 15, 2024

Commits on May 16, 2024

Generic Chat templating code with text/json file based config; main chat updated to drive its in-prefix, in-suffix and reverse-prompt from same; chat-apply-template equivalent c-api to allow use by other codes also #6834

Are you sure you want to change the base?

Generic Chat templating code with text/json file based config; main chat updated to drive its in-prefix, in-suffix and reverse-prompt from same; chat-apply-template equivalent c-api to allow use by other codes also #6834

Commits on May 6, 2024

Commits on May 7, 2024

Commits on May 8, 2024

Commits on May 10, 2024

Commits on May 11, 2024

Commits on May 12, 2024

Commits on May 13, 2024

Commits on May 14, 2024

Commits on May 15, 2024

Commits on May 16, 2024