SimpleChat: a simple and dumb web front end for testing /chat/completions and /completions end points and try chat

[hanishkvc]

Hi @ggerganov, @ngxson, @mofosyne

As mentioned in my other PR #6834 comment section, the web front end available with examples/server doesnt currently use the /chat/completions endpoint. It only generates requests for /completions endpoint. Also the web front end code is more involved/complicated for minimal testing.

This simple web frontend, allows triggering/testing the server's /completions or /chat/completions endpoints in a simple way with minimal code from a common code base. And also allows trying to maintain a basic back and forth chatting to an extent.

NOTE: Given that the idea is for basic minimal testing, it doesnt bother with any model context length and culling of old messages from the chat. Also currently I havent added input for a system prompt, but may add it [update: system prompt support added].

usage

first run examples/server

• bin/server -m path/model.gguf

next run this web front end in examples/server/public_simplechat

• ./simplechat.sh
• this uses python3's http.server to host this web front end

Open this simple web front end from your local browser as noted in the message printed when simplechat.sh is run

• by default it is http://127.0.0.1:9000/simplechat.html

[hanishkvc]

Have added support for system prompt.

[hanishkvc]

@mofosyne @ggerganov @ngxson

I have further structured the code and also added support for multiple independent chats, it is ready for merging now.

It implements a responsive design to adapt to laptop/mobile/tablet and their different screen resolutions and orientations implicitly by simply being structured around basic html+css's flex-column of flex-rows. It uses a scrolled view/div in normal usage, while parallely allowing unrolled multi page printing, by implicitly changing the document to have auto height (thus bypassing the overflow-scroll style entry) in print mode.

The whole idea of this is to keep things simple with out using any modules/frameworks/etal, and instead build on what is already available with basic html/js/css, which is good enough with current standards and browsers.

Currently this is placed independent of the existing web-frontend, in its own folder under examples/server/public_simplechat. We can either retain this there itself or move into public/chatui.

Lets keep this as it is with its simple and straight forward html/js/css use, without changing/refactoring it to use other modules like preact etal. Which should also keep it simple for anyone to modify it if required in future, indepedent of any framework conventions or changes or replacements to them.

However in future, if ngxson or anyone else wants to refactor it wrt preact/xyz etal, I would suggest that then parallely keep this pure html/css/js based version also, as is, in a seperate folder, like the current examples/server/public_simplechat.

NOTE: Input for anyone looking at refactoring, a copy, in future

• The core chat-[role+content]-set is maintained by the basic SimpleChat class including mapping it into prompt or messages api equivalent json. It also provides the logic for setting of system prompt either only-once-at-begining or anytime, as well as getting the latest system prompt wrt a chat session.

• While MultiChatUI class's handle_user_submit takes care of the getting the request json from current/needed SimpleChat instance, handshaking with server using fetch, extracting model response and updating the SimpleChat instance.

• MultiChatUI also provides the logic to maintain a map/dict/object of multiple chat sessions (SimpleChat instances) as well as manage the html UI elements, interms of taking care of switching between chat sessions. [This is the addition done in the last 24 hours, after ggerganov reviewed things].

[mofosyne]

CI failing noted

features/results.feature:75 consistent results with same seed and varying batch size -- @1.3

---

Replication attempt

cmake -B build
cmake --build build --config Release
./build/bin/server --hf-repo TheBloke/phi-2-GGUF --hf-file phi-2.Q6_K.gguf --path ./examples/server/public_simplechat

I'm suppose to put something in system line?

[mofosyne]

If the CI issue is more due to what's happening in the main branch and not within this PR, then I'll merge it after the CI in main branch is stable.

[hanishkvc]

@mofosyne

system prompt

Its up to the user if they want to put a system prompt or not.

If one enters a system prompt or changes it, it will be sent to the server and inturn the model, as a system role content, from the moment they entered it in the chat history. The image attached shows a model responding with and without system prompt. The 1st response is before a system prompt was set, the 2nd respond in the same chat session, is after a prompt was set.

with mistral-7b-instruct-v0.2.Q4_K_M.gguf

CI issue

I have fetched the upstream and verified this PR through examples/server from latest upstream, and everything is working as expected. That CI seems decoupled from this PR.

[mofosyne]

Fair enough, well I guess I was a bit confused by the statement that the system prompt must be filled in.

Something like an input placeholder attribute with the text you suggested above could help. @hanishkvc

e.g. "Optional system prompt. e.g. you are a drunk poet"

https://www.w3schools.com/TAGS/att_input_placeholder.asp

---

Thanks for confirming that the PR is most likely decoupled from main branch CI issue, will keep it in mind

[hanishkvc]

Hi @mofosyne @ngxson @ggerganov

possible issue wit CI test

If the phi2 line which mofosyne mentioned above is the one related to the CI issue
ie
./build/bin/server --hf-repo TheBloke/phi-2-GGUF --hf-file phi-2.Q6_K.gguf --path ./examples/server/public_simplechat

Then because currently llama.cpp's chat-apply-template doesnt have support for phi2 directly, so server and inturn its oaicompat_completion_params_parse will generate wrongly templated chat message which will be tokenized and sent to the ai llm model, this is independent of this PR. Rather any code be it server or some 3rd party program which uses llama.cpp's chat-apply-template will have same issue.

So also potentially the phi2 model will generate response which depending on the query may either be within sane limits beyond the required text completion/chatting, or it may even get into a runaway mode or so and get stuck for sometime or ...

Also given that this specific model is a base model and not a chat or instruct model, it eitherway will be unhinged wrt generating text beyond what is asked, which also can create problem.

ideal solution, independent of this CI issue

So either one will have to hard code phi2 chat template standard into chat-apply-template in current llama.cpp OR implement a new jinja template based flow OR if one is using my other PR #6834 , they will be able to specify phi2's chat template info in chaton_meta.json file and inturn get things working as needed.

Even this in itself may not be enough, wrt expecting a non-run-away answers, as this specific model is a base model and not chat/instruct model.

solution for CI issue at hand

Independent of anything else, rather if any CI code is currently using Phi2 to test llama.cpp's chat-templating, then it is invalid and needs to be fixed (unless it is a negative test or so), as it stands now. That is the test needs to be changed so that it doesnt use chat-templating-unsupported phi2 models.

Fair enough, well I guess I was a bit confused by the statement that the system prompt must be filled in.

Something like an input placeholder attribute with the text you suggested above could help. @hanishkvc

e.g. "Optional system prompt. e.g. you are a drunk poet"

https://www.w3schools.com/TAGS/att_input_placeholder.asp

Thanks for confirming that the PR is most likely decoupled from main branch CI issue, will keep it in mind

[hanishkvc]

Hi @mofosyne @ngxson @ggerganov

update / a correction

I need to correct myself partly, wrt my previous comment. I assume mofosyne was using phi-2 to test server in general, while the actual CI test was potentially using a different model. And inturn it seems to be testing both /chat/completions and /completions endpoints. So my comment wrt phi-2 and ci-testing if wrt /chat/completions doesnt hold.

HOWEVER my original comment that this specific CI failure issue has nothing to do with this PR and more of some issue between server in general and the test code around it still holds. So we can safely merge this PR with the main/master branch. Also this PR doesnt touch server code in any way and sits in its own subdirectory.

local direct check

Also wrt server on its own, I have tried testing the configuration which seems to be used by the ci-test which is failing.

cmake .. \
-DLLAMA_NATIVE=OFF
-DLLAMA_BUILD_SERVER=ON
-DLLAMA_CURL=ON
-DCMAKE_BUILD_TYPE=RelWithDebInfo
-DLLAMA_SANITIZE_THREAD=ON ;

cmake --build . --config RelWithDebInfo -j 4 --target server

Inturn testing the server against a bunch of models (llama3, phi-3, phi-2, mistral), things seem to be fine wrt server and this PR in general wrt chat and completion mode.

Hi @mofosyne @ngxson @ggerganov

possible issue wit CI test

If the phi2 line which mofosyne mentioned above is the one related to the CI issue ie ./build/bin/server --hf-repo TheBloke/phi-2-GGUF --hf-file phi-2.Q6_K.gguf --path ./examples/server/public_simplechat

Then because currently llama.cpp's chat-apply-template doesnt have support for phi2 directly, so server and inturn its oaicompat_completion_params_parse will generate wrongly templated chat message which will be tokenized and sent to the ai llm model, this is independent of this PR. Rather any code be it server or some 3rd party program which uses llama.cpp's chat-apply-template will have same issue.

So also potentially the phi2 model will generate response which depending on the query may either be within sane limits beyond the required text completion/chatting, or it may even get into a runaway mode or so and get stuck for sometime or ...

Also given that this specific model is a base model and not a chat or instruct model, it eitherway will be unhinged wrt generating text beyond what is asked, which also can create problem.

ideal solution, independent of this CI issue

So either one will have to hard code phi2 chat template standard into chat-apply-template in current llama.cpp OR implement a new jinja template based flow OR if one is using my other PR #6834 , they will be able to specify phi2's chat template info in chaton_meta.json file and inturn get things working as needed.

Even this in itself may not be enough, wrt expecting a non-run-away answers, as this specific model is a base model and not chat/instruct model.

solution for CI issue at hand

Independent of anything else, rather if any CI code is currently using Phi2 to test llama.cpp's chat-templating, then it is invalid and needs to be fixed (unless it is a negative test or so), as it stands now. That is the test needs to be changed so that it doesnt use chat-templating-unsupported phi2 models.

Fair enough, well I guess I was a bit confused by the statement that the system prompt must be filled in.
Something like an input placeholder attribute with the text you suggested above could help. @hanishkvc
e.g. "Optional system prompt. e.g. you are a drunk poet"
https://www.w3schools.com/TAGS/att_input_placeholder.asp
Thanks for confirming that the PR is most likely decoupled from main branch CI issue, will keep it in mind

[mofosyne]

Merged in now as @hanishkvc shown that this is unlikely to be an issue.
Regarding any further improvements, I think this is still a good base to start and it be ideal to see how it functions in the real world.

Any further changes could be in a new PR @hanishkvc but fingers crossed everyone understands it enough to not need to adjust the UI further (e.g. adding placeholder text to system input)

Thanks for your contribution. You may want to also consider adding a general guide/tutorial to the wiki if you so choose to do so (which is a bit more freeform than the readme in the codebase).

[hanishkvc]

@mofosyne

I had mentioned in the note shown to user that they can set the system prompt above. However I had avoided adding a placeholder, as sometimes it is not dull enough to distinguish that it is not something already entered. However now I have added the placeholder system prompt with e.g.

As well as changed the behaviour wrt Completion mode, such that the code doesnt automatically add the user: prefix, so that user has full flexibility wrt adding any role related prefix that they may want. As well as optionally insert a assistant related prefix even in the completion mode.

Shall I push these additions to this PR itself?

[mofosyne]

Already merged so best as a new PR. Also those kinds of changes are smaller quality of life changes so best as a separate PR now.

[hanishkvc]

Thanks @mofosyne,

Will create a new PR.

Wanted to be sure that I hadnt missed out something and inturn once I found a small difference compared to what I had commented, wanted to cross check things are fine or not and inturn inform you all. So missed out on pushing these earlier.

Already merged so best as a new PR. Also those kinds of changes are smaller quality of life changes so best as a separate PR now.

[mofosyne]

Continuation of minor changes and improvements #7480