Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

server: streaming of tool calls and thoughts when --jinja is on #12379

Draft
wants to merge 63 commits into
base: master
Choose a base branch
from
Draft
Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
16c9c63
add common_regex w/ support for partial final matches
ochafik Mar 12, 2025
6dcff43
add common_json w/ support for truncated json healing
ochafik Mar 12, 2025
a95fe78
renaming: string_find_partial_stop (moved to common.cpp)
ochafik Mar 12, 2025
ce2f593
add common_chat_msg_diff
ochafik Mar 12, 2025
cd3681d
partial common_chat_parse
ochafik Mar 12, 2025
9462365
refactor parser w/ optionals
ochafik Mar 12, 2025
6ed8a8f
server: wire chat diffs in stream mode
ochafik Mar 12, 2025
eaeed7d
fix trigger of thinking models (must happen after thoughts are closed)
ochafik Mar 13, 2025
d6e680a
nits + docs
ochafik Mar 14, 2025
64ea080
fix functionary v3.2 raw python!
ochafik Mar 14, 2025
c46d4da
rename: common_chat_syntax (now contains format)
ochafik Mar 14, 2025
4358d5d
rm common_regex.at_start
ochafik Mar 14, 2025
f477288
Merge remote-tracking branch 'origin/master' into tool-diffs
ochafik Mar 14, 2025
e0202b3
fix gcc compilation
ochafik Mar 14, 2025
f840e3a
fix unreachable code warning after [[noreturn]] annotation
ochafik Mar 14, 2025
af7391e
fix / refactor test-regex-partial
ochafik Mar 14, 2025
449917b
fix test-chat
ochafik Mar 14, 2025
b428b5c
rm spaces
ochafik Mar 14, 2025
668fc90
fix command r7b partial parsing (lacked args path)
ochafik Mar 14, 2025
b48ab23
Update test_chat_completion.py
ochafik Mar 14, 2025
aefc8a4
refactor + test chat parser (try_consume_json_with_dumped_args, liter…
ochafik Mar 15, 2025
22428a4
return partial msg from server
ochafik Mar 15, 2025
5b9c5a4
refactor partial json
ochafik Mar 15, 2025
3fbe84f
don't return empty <think></think>
ochafik Mar 15, 2025
d4cb7fe
test_tool_call: allow comment lines in now-multiline code strings (fo…
ochafik Mar 15, 2025
31f5eb2
accommodate yet another deepseek r1 distill fantasy syntax (<|tool▁ca…
ochafik Mar 15, 2025
bddc65a
rm space
ochafik Mar 15, 2025
ea3bf03
nit: fix python type
ochafik Mar 15, 2025
f3bfbc6
refactor test-chat-parser
ochafik Mar 15, 2025
bb7b9fe
fix QwQ 32B tool call parsing after thoughts (hermes2)
ochafik Mar 15, 2025
f0ea330
fix thinking models + tool calls (</think> not part of trigger's capt…
ochafik Mar 15, 2025
7856949
reinstate tool call id logic, keep track of previously generated ids
ochafik Mar 15, 2025
2412b5d
better logs for triggers
ochafik Mar 15, 2025
02913b0
fix msg diff test
ochafik Mar 15, 2025
c5c3482
try_consume_regex: basic tests + fix non-partial case
ochafik Mar 15, 2025
af79da0
chat-parser: test+fix finish, incomplete methods
ochafik Mar 15, 2025
562800f
normalize args in test-chat
ochafik Mar 15, 2025
ddeb318
consume spaces after parse_json_tool_calls
ochafik Mar 15, 2025
6c3f87e
Revert "fix thinking models + tool calls (</think> not part of trigge…
ochafik Mar 15, 2025
e2cef66
fix required tool calls w/ thinking models that have pre-opened think…
ochafik Mar 15, 2025
7a61eca
fix thinking model's initial trigger (take 2) + test qwq's template
ochafik Mar 15, 2025
2f55571
refactor chat parser (rm incomplete)
ochafik Mar 15, 2025
303f640
test groups of common_chat_msg_parser.try_consume_regex
ochafik Mar 15, 2025
e9540ad
run most test_tool_call tests in stream + non-stream modes
ochafik Mar 15, 2025
a818114
make functionary v3.2 parsing more strict (differentiate first match …
ochafik Mar 16, 2025
5031366
send final diff from server, to close off raw python arguments
ochafik Mar 16, 2025
dae6a28
nit: spaces
ochafik Mar 16, 2025
f026cb0
fix diff aggregation logic in make_any_request
ochafik Mar 16, 2025
e7f9d3e
fix test_chat_completion_with_timings_per_token & test_logprobs_stream
ochafik Mar 16, 2025
165b525
add missing functional import for gcc compilation
ochafik Mar 16, 2025
9d4a6f1
fix typo in test_calc_result
ochafik Mar 16, 2025
64b4039
fix thoughts parsing logic
ochafik Mar 16, 2025
fbba5da
support partial content streaming in Generic mode
ochafik Mar 16, 2025
4dcd653
strip reasoning (now that tags are strings and not regexes)
ochafik Mar 16, 2025
56156b7
run test_thoughts in stream mode too
ochafik Mar 16, 2025
5dfa2f7
r1: avoid partial call triggers from spaces
ochafik Mar 16, 2025
91a5084
fix test_thoughts / refactor expectations
ochafik Mar 16, 2025
4f78d44
fix partial json crashes
ochafik Mar 16, 2025
ea57e47
fix test-chat's unparsed thought expectation
ochafik Mar 16, 2025
1d25178
Merge remote-tracking branch 'origin/master' into tool-diffs
ochafik Mar 23, 2025
42cb16f
fix partial json crash after comma
ochafik Mar 23, 2025
37b4a3a
fix test-chat.cpp
ochafik Mar 23, 2025
13d725d
fix gcc build of test
ochafik Mar 23, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions common/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -58,19 +58,25 @@ add_library(${TARGET} STATIC
base64.hpp
chat.cpp
chat.h
chat-parser.cpp
chat-parser.h
common.cpp
common.h
console.cpp
console.h
json-schema-to-grammar.cpp
json.hpp
json-partial.h
json-partial.cpp
llguidance.cpp
log.cpp
log.h
minja/chat-template.hpp
minja/minja.hpp
ngram-cache.cpp
ngram-cache.h
regex-partial.cpp
regex-partial.h
sampling.cpp
sampling.h
speculative.cpp
Loading