server/public_simplechat - with reasoning and 0 setup builtin tool calls data store, calculator, javascript code runner, with simpleproxy helper web fetch, web search, pdf2text tool calls #16929
+3,499
−388
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Enable streaming by default, to check the handshake before going on to change the code, given that havent looked into this for more than a year now and have been busy with totally different stuff. Also updated the user messages used for testing a bit
Define the meta that needs to be passed to the GenAi Engine. Define the logic that implements the tool call, if called. Implement the flow/structure such that a single tool calls implementation file can define multiple tool calls.
Make tooljs structure and flow more generic Add a simple_calculator tool/function call logic Add initial skeleton wrt the main tools.mjs file.
Changed latestResponse type to an object instead of a string. Inturn it contains entries for content, toolname and toolargs. Added a custom clear logic due to the same and used it to replace the previously simple assigning of empty string to latestResponse. For now in all places where latestReponse is used, I have replaced with latestReponse.content. Next need to handle identifying the field being streamed and inturn append to it. Also need to add logic to call tool, when tool_call triggered by genai.
Update response_extract_stream to check for which field is being currently streamed ie is it normal content or tool call func name or tool call func args and then return the field name and extracted value. Previously it was always assumed that only normal content will be returned. Currently it is assumed that the server will only stream one of the 3 supported fields at any time and not more than one of them at the same time. TODO: Have to also add logic to extract the reasoning field later, ie wrt gen ai models which give out their thinking. Have updated append_response to expect both the key and the value wrt the latestResponse object, which it will be manipualted. Previously it was always assumed that content is what will be got and inturn appended.
I was wrongly checking for finish_reason to be non null, before trying to extract the genai content/toolcalls, have fixed this oversight with the new flow in progress. I had added few debug logs to identify the above issue, need to remove them later. Note: given that debug logs are disabled by replacing the debug function during this program's initialisation, which I had forgotten about, I didnt get the debug messages and had to scratch my head a bit, before realising this and the other issue ;) Also either when I had originally implemented simplechat 1+ years back, or later due to changes on the server end, the streaming flow sends a initial null wrt the content, where it only sets the role. This was not handled in my flow on the client side, so a null was getting prepended to the chat messages/responses from the server. This has been fixed now in the new generic flow.
Make latestResponse into a new class based type instance wrt ai assistant response, which is what it represents. Move clearing, appending fields' values and getting assistant's response info (irrespective of a content or toolcall response) into this new class and inturn use the same.
Switch oneshot handler to use AssistantResponse, inturn currenlty only handle the normal content in the response. TODO: If any tool_calls in the oneshot response, it is currently not handled. Inturn switch the generic/toplevel handle response logic to use AssistantResponse class, given that both oneshot and the multipart/streaming flows use/return it. Inturn add trimmedContent member to AssistantResponse class and make the generic handle response logic to save the trimmed content into this. Update users of trimmed to work with this structure.
As there could be failure wrt getting the response from the ai server some where in between a long response spread over multiple parts, the logic uses the latestResponse to cache the response as it is being received. However once the full response is got, one needs to transfer it to a new instance of AssistantResponse class, so that latestResponse can be cleared, while the new instance can be used in other locations in the flow as needed. Achieve the same now.
Previously if content was empty, it would have always sent the toolcall info related version even if there was no toolcall info in it. Fixed now to return empty string, if both content and toolname are empty.
The implementations of javascript and simple_calculator now use provided helpers to trap console.log messages when they execute the code / expression provided by GenAi and inturn store the captured log messages in the newly added result key in tc_switch This should help trap the output generated by the provided code or expression as the case maybe and inturn return the same to the GenAi, for its further processing.
Checks for toolname to be defined or not in the GenAi's response If toolname is set, then check if a corresponding tool/func exists, and if so call the same by passing it the GenAi provided toolargs as a object. Inturn the text generated by the tool/func is captured and put into the user input entry text box, with tool_response tag around it.
As output generated by any tool/function call is currently placed into the TextArea provided for End user (for their queries), bcas the GenAi (engine/LLM) may be expecting the tool response to be sent as a user role data with tool_response tag surrounding the results from the tool call. So also now at the end of submit btn click handling, the end user input text area is not cleared, if there was a tool call handled, for above reasons. Also given that running a simple arithmatic expression in itself doesnt generate any output, so wrap them in a console.log, to help capture the result using the console.log trapping flow that is already setup.
and inform the GenAi/LLM about the same
Should hopeful ensure that the GenAi/LLM will generate appropriate code/expression as the argument to pass to these tool calls, to some extent.
ie in vs code with ts-check
Move tool calling logic into tools module. Try trap async promise failures by awaiting results of tool calling and putting full thing in an outer try catch. Have forgotten the nitty gritties of JS flow, this might help, need to check.
So that when tool handler writes the result to the tc_switch, it can make use of the same, to write to the right location. NOTE: This also fixes the issue with I forgetting to rename the key in js_run wrt writing of result.
to better describe how it will be run, so that genai/llm while creating the code to run, will hopefully take care of any naunces required.
Also as part of same, wrap the request details in the assistant block using a similar tagging format as the tool_response in user block.
Instead of automatically calling the requested tool with supplied arguments, rather allow user to verify things before triggering the tool. NOTE: User already provided control over tool_response before submitting it to the ai assistant.
Instead of automatically calling any requested tool by the GenAi / llm, that is from the tail end of the handle user submit btn click, Now if the GenAi/LLM has requested any tool to be called, then enable the Tool Run related UI elements and fill them with the tool name and tool args. In turn the user can verify if they are ok with the tool being called and the arguments being passed to it. Rather they can even fix any errors in the tool usage like the arithmatic expr to calculate that is being passed to simple_calculator or the javascript code being passed to run_javascript_function_code If user is ok with the tool call being requested, then trigger the same. The results if any will be automatically placed into the user query text area. User can cross verify if they are ok with the result and or modify it suitabley if required and inturn submit the same to the GenAi/LLM.
Also avoid showing Tool calling UI elements, when not needed to be shown.
Copy validate_url and build initial skeleton
Check if the specified scheme is allowed or not. If allowed then call corresponding validator to check remaining part of the url is fine or not
Add --allowed.schemes config entry as a needed config. Setup the url validator. Use this wrt urltext, urlraw and pdf2text This allows user to control whether local file access is enabled or not. By default in the sample simpleproxy.json config file local file access is allowed.
Also trap any exceptions while handling and send exception info to the client requesting service
also move debug dump helper to its own module also remember to specify the Class name in quotes, similar to refering to a class within a member of th class wrt python type checking.
Its not necessary to request a page number range always. Take care of page number starting from 1 and underlying data having 0 as the starting index
Added logic to help get a file from either the local file system or from the web, based on the url specified. Update pdfmagic module to use the same, so that it can support both local as well as web based pdf. Bring in the debug module, which I had forgotten to commit, after moving debug helper code from simpleproxy.py to the debug module
This also indirectly adds support for local file system access through the web / fetch (ie urlraw and urltext) service request paths.
hanishkvc
changed the title
|
Hi @ggerganov @ericcurtin @ngxson @mofosyne With this PR the basic reasoning as well as builtin client side tool calling support with a useful bunch of ready to use tool calls, has been added to tools/server/public_simplechat. While the thinking / work on backend / server side mcp support etal for tool calling is a good thing for users targeting back end server deployment or custom device setups etal. For normal end users, I feel, the approach I have used above of using the browser flexibility and capability to expose a bunch of builtin tool calls with 0 additional setup, is a more practically and immidiately useful and usable way of enhancing GenAi/LLM value in productive ways. The note wrt this PR above has more details as also the readme.md. The PR chain going back starting from the one mentioned in the PR note above, will give a high level view on how this evolved. Or the git commits below have the more fine grained details. This PR series cleans up the existing tools/server/public_simplechat flow as well as adds more functionality and flexibility to the same. Please do merge this. |
Make it a details block and update the content a bit
Usage Note * Cleanup / fix some wording. * Pick chat history handshaked len from config Ensure the settings info is uptodate wrt available tool names by chaining a reshowing with tools manager initialisation.
Rename path and tags/identifiers from Pdf2Text to PdfText Rename the function call to pdf_to_text, this should also help indicate semantic more unambiguously, just in case, especially for smaller models.
Chances are for ai models which dont support tool calling, things will be such that the tool calls meta data shared will be silently ignored without much issue. So enabling tool calling feature by default, so that in case one is using a ai model with tool calling the feature is readily available for use. Revert SlidingWindow ChatHistory in Context from last 10 to last 5 (2 more then origianl, given more context support in todays models) by default, given that now tool handshakes go through the tools related side channel in the http handshake and arent morphed into normal user-assistant channel of the handshake.
helps ensure only service paths that can be serviced are enabled Use same to check for pypdf wrt pdftext
hanishkvc
changed the title
Allow the web tools handshake helper to pass additional header entries provided by its caller. Make use of this to send a list of tag and id pairs wrt web search tool. Which will be used to drop div's matching the specified id.
Rename search-drops to urltext-tag-drops, to indicate its more generic semantic. Rather search drops specified in UI by user will be mapped to urltext-tag-drops header entry of a urltext web fetch request. Implement a crude urltext-tag-drops logic in TextHtmlParser. If there is any mismatch with opening and closing tags in the html being parsed and inturn wrt the type of tag being targetted for dropping, things can mess up.
Update the initial skeleton wrt the tag drops logic * had forgotten to convert object to json string at the client end * had confused between js and python and tried accessing the dict elements using . notation rather than [] notation in python. * if the id filtered tag to be dropped is found, from then on track all other tags of the same type (independent of id), so that start and end tags can be matched. bcas end tag call wont have attribute, so all other tags of same type need to be tracked, for proper winding and unwinding to try find matching end tag * remember to reset the tracked drop tag type to None once matching end tag at same depth is found. should avoid some unnecessary unwinding. * set/fix the type wrt tagDrops explicitly to needed depth and ensure the dummy one and any explicitly got one is of right type. Tested with duckduckgo search engine and now the div based unneeded header is avoided in returned search result.
Update readme wrt searchDrops, auto settings ui creation Rename tools-auto to tools-autoSecs, to make it easy to realise that the value represents seconds.
Pretty print SimpleProxy gMe config Dont ignore the got http response status text. Update readme wrt why autoSecs
Allow user to clear the existing chat. The user does have the option to load the just cleared chat, if required. Add icons wrt clearing chat and settings.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[Updated 20251103IST0221]
The alternate tools/server/public_simplechat web client ui has been updated to support tool calls and reasoning for ai models that support the same.
The tool calls implicitly supported and in turn used by ai models (with tool calling support) when running this alternate web client ui without needing any additional setup include
direct builtin tool call supported (using your browser's web worker context)
additional builtin tool calls supported if running included simpleproxy.py helper
viewing of reasoning from ai models that support the same
One could get going with
build/bin/llama-server -m ../llama.cpp.models/gpt-oss-20b-mxfp4.gguf --jinja --path tools/server/public_simplechat/ --ctx-size 64000 --n-gpu-layers 12 -fa on
NOTE: even the default context size should be good enough for simple stuffs. Explicitly set the --ctx-size if working with many web site / pdf contents or so as needed.
if one additionally needs web search, web fetch and pdf related tool calls, then also run
cd tools/server/public_simplechat/local.tools; python3 ./simpleproxy.py --config simpleproxy.json
NOTE: Remember to edit simpleproxy.json with the list of sites you want to allow access to, as well as to disable local file access, if needed.
With this one can get the local ai to
and or answer queries around these or ... its up to you and the ai model ...
Cross check the tool calls before allowing their execution and responses before submitting them to the ai model, just to be on safe side.
Also remember to enable tools from the settings of this web client ui.
Look into included readme.md for additional info.
NOTE: Refer to the previous PR in this series #16852 - and the other PRs in this series to see the evolution. Or the git commits below for the evolution in more finer detail.
[ Previous / Older message below ]
This adds a simple pdf2text tool call, which given url (file:/// local paths for now) to a pdf file, will extract its text contents. It also allows one to optionally set a start and end page, to limit the amount of content fetched/processed.
This also includes a possible fix for the auto mode not working after the new ChatShow flow was added.
Also a new config variable added to the tools block to allow user to limit the amount of data from the tool call result, that should be submitted back to the ai engine server.
This is a work in progress, and the initial go at it and not yet ready for merging.
However the previous PRs in this series with the latest before this being #16852 - are ready for merging.
They add support for tool calling from the client /browser side, to the alternate public_simplechat client ui. And inturn they provide builtin tool call support (ie without any other additional setup) for calculator, javascript code runner and a data store.
They additionally also support fetching web pages and searching web, if the bundled simpleproxy.py is run.
The details are in the above mentioned PR and the others PRs before that and also in the readme to some extent.