[Feature] OpenAI-Compatible Tools API + Streaming for Hermes & Mistral models#5649

Merged

simon-mo merged 237 commits intovllm-project:mainfrom Constellate-AI:tool-use

Sep 4, 2024

+2,588-83

Commits on Jun 16, 2024

add hermes 2 pro function calling template
constellate
committed
feat(example): add example chat completion with tool usage
Kyle Mistele
committed

Commits on Jun 17, 2024

feat: add CLI argument for OpenAI API-style tool use system prompt jinja template
Kyle Mistele
committed
feat: add better validation for tool_choice, support for tool_choice = "auto"; set default tool_choice to "auto"
Kyle Mistele
committed

Commits on Jun 18, 2024

fix(cli): set OpenAI tool args in vllm/entrypoints/openai/cli_args.py instead of vllm/engine/arg_utils.py
Kyle Mistele
committed
fix(types): add "auto" as an option for tool_choice in pydantic models
Kyle Mistele
committed
fix: validation - guided decoding not valid with tool_choice = auto
Kyle Mistele
committed
feat: ensure guided deoding is only applied when tool_choice is NOT "none" AND NOT "auto"
Kyle Mistele
committed
feat(cli): update CLI args for auto tool choice and OpenAIServingChat to receive the arguments & validate them
Kyle Mistele
committed
feat: add loading in the system prompt jinja template if specified; along with validation
Kyle Mistele
committed

Commits on Jun 19, 2024

wip: add a case for tool choice = auto when handling chat completion requests
Kyle Mistele
committed
fix: hermes 2 pro prompt template to prevent newlines
Kyle Mistele
committed
feat: handle building the system prompt via template with --enable-auto-tool-choice and --tool-use-prompt-template are specified
Kyle Mistele
committed
fix: remove debugging log lines
Kyle Mistele
committed

Commits on Jun 20, 2024

fix(template): update hermes 2 pro template with newlines to get newlines as expected
Kyle Mistele
committed

Commits on Jul 1, 2024

merge: main into tool use & resolve conflicts
Kyle Mistele
committed

Commits on Jul 2, 2024

feat: add hermes 2 pro and mistral full tool use chat templates
Kyle Mistele
committed
chore: delete old template
Kyle Mistele
committed
feat: tool calls are now returned in the chat completion response
Kyle Mistele
committed
fix: mistral chat template. replace huggingface-suggested one with the mixtral tool_use chat template
Kyle Mistele
committed
feat: add mistral tool parser, and empty hermes tool parser. non-streaming ONLY
Kyle Mistele
committed
feat: update example
Kyle Mistele
committed
chore: update FunctionCall type to allow arguments as a Dict. non-auto tool-choice uses string, auto uses Dict
Kyle Mistele
committed
feat: clean up CLI arguments, engine. implement tool parser selection; tool call parsing for NON-STREAMING responses
Kyle Mistele
committed
feat: add methods to FunctionCall and ToolCall in protocol to make it easier to JSON-serialize
Kyle Mistele
committed
fix: ensure finish_reason = "tool_calls" when a tool call is generated
Kyle Mistele
committed
unfinished: work on hermes 2 tool call parser
Kyle Mistele
committed

Commits on Jul 3, 2024

partial: hermes 2 pro tool parsing
Kyle Mistele
committed
fix: hermes tool call parser, work on example
Kyle Mistele
committed
fix: tool call arguments should be returned as JSON string not as a literal dict/object
Kyle Mistele
committed
feat: enable both content and tool_calls if the model allows
Kyle Mistele
committed
feat: update example with tool call
Kyle Mistele
committed
fix: typing-related issues for chat messages
Kyle Mistele
committed

Commits on Jul 4, 2024

Commits on Jul 5, 2024

fix: mistral tool calling chat template
Kyle Mistele
committed
fix: make ChatMessage content Optional since it could be an assistant message with tool_calls and no content
Kyle Mistele
committed
refactor: move tool parsing into the right place in serving_chat, and update tool parsers accordingly
Kyle Mistele
committed

Commits on Jul 7, 2024

fix: finish_reason should NEVER be None; OpenAI defualt is "stop"
Kyle Mistele
committed
fix: typo introduced in earlier commit
Kyle Mistele
committed

Commits on Jul 8, 2024

feat: signature updates and refactoring to tool parser streaming; prepare for streaming tools
Kyle Mistele
committed
fix: kind of fixed mistral chat template
Kyle Mistele
committed
feat: make non-streaming tool parsing a static method so that streaming can have state
Kyle Mistele
committed
partial: work on streaming tool call parser for mistral
Kyle Mistele
committed

Commits on Jul 10, 2024

deps: add partial-json-parser for parsing streaming JSON
Kyle Mistele
committed
fix: protocol stuff, work on mistral streaming
Kyle Mistele
committed
feat: progress on mistral streaming parser
Kyle Mistele
committed
fix: some tool parser stuff. best its working yet
Kyle Mistele
committed
fix: major parsing logic issue when overlapping prefix & suffix due to json autocompletion
Kyle Mistele
committed
feat: implement mistral tool calling streaming for ONE TOOL ONLY RIGHT NOW
Kyle Mistele
committed
feat: update openai client to showcase streaming
Kyle Mistele
committed
fix: finish reason & debug logging
Kyle Mistele
committed
fix(docs): CLI argument description was bad
Kyle Mistele
committed

Commits on Jul 13, 2024

fix(parser): mistral tool parser issue that was giving me a stroke. Also, code cleanup & documentation
Kyle Mistele
committed
chore: update examples & logging
Kyle Mistele
committed
fix: accidentally broke non-tool streaming earlier; this fixes it
Kyle Mistele
committed
fix: some stuff in the example, and some mistral stuff
Kyle Mistele
committed
feat: work on hermes tool parser
Kyle Mistele
committed

Commits on Jul 17, 2024

Commits on Jul 18, 2024

fix(docs): some type issue that the doc CI check did not like
Kyle Mistele
committed
fix(docs): some type issue that the doc CI check did not like
Kyle Mistele
committed
fix(types): try Optional[str] = None
Kyle Mistele
committed
fix: refactor tool chat template and add docs
Kyle Mistele
committed

Commits on Aug 1, 2024

chore: annotate that parallel_tool_calls will be ignored
Kyle Mistele
committed
fix: handle un-handled "theoretically unreachable" case because such things are important
Kyle Mistele
committed
fix: hermes tool call template to omit tool-use system prompt if tools are not specified
Kyle Mistele
committed
fix: implement access to tool call token IDs via tokenizer vocab in tool call parser non-static methods
Kyle Mistele
committed
fix: hermes tool parser does not extract non-tool-call content the same way it does in streaming as in non-streaming
Kyle Mistele
committed
fix: grab the name properly from chat completions and fall back to empty string since its optional
Kyle Mistele
committed
fix: type annotation
Kyle Mistele
committed
fix: mistral tool extraction when dealing with poor precision
Kyle Mistele
committed
doc: indicate that temperature should be set to 0 when doing mistral tool calls
Kyle Mistele
committed

Commits on Aug 2, 2024

fix(conflicts): resolve a metric f-ck ton of merge conflicts
Kyle Mistele
committed
fix: log levels
Kyle Mistele
committed
fix: more logging changes
Kyle Mistele
committed
fix(ci): formatting
Kyle Mistele
committed
fix: formatting
Kyle Mistele
committed

Commits on Aug 3, 2024

fix: merge main+conflict resolution;fix --tool-call-parser being required by accident
Kyle Mistele
committed
fix: remove unnecessary case that was artifact from previous approach
Kyle Mistele
committed
fix: validation errors
Kyle Mistele
committed
fix: hermes prompt template issue that occurred when passing multiple tool calls in
Kyle Mistele
committed
fix: formatting & mypy fixes
Kyle Mistele
committed
fix: more types
Kyle Mistele
committed
fix: more mypy fixes
Kyle Mistele
committed
fix: formatting
Kyle Mistele
committed
fix: more mypy fixes
Kyle Mistele
committed
fix: final mypy fixes
Kyle Mistele
committed
fix: finish_reason behavior was broken for non-streaming calls
Kyle Mistele
committed

Commits on Aug 5, 2024

fix(test): ensure tool_choice="required" throws a BadRequestError, and that "auto" tool choice throws a BadRequestError without the proper CLI args
Kyle Mistele
committed
fix: remove deprecated CLI argument
Kyle Mistele
committed
fix: type
Kyle Mistele
committed
fix: remoev another assertion and replace with if/exception
Kyle Mistele
committed
fix: bad condition
Kyle Mistele
committed
fix: cleaner concat
Kyle Mistele
committed
fix: clean up vode
Kyle Mistele
committed
chore: more cleanup
Kyle Mistele
committed
chore: clean up conditional and document better
Kyle Mistele
committed

Commits on Aug 6, 2024

fix(ci): broken tool streaming when using guided decoding
Kyle Mistele
committed
fix: formatting
Kyle Mistele
committed
chore: refactor tool parsers structure to make it more maintainable
Kyle Mistele
committed
fix(tests): raise a valueError that was being passed instead of raised
Kyle Mistele
committed
fix(PEP8): openai chat completion client with tools
Kyle Mistele
committed
fix(PEP8): chat_utils
Kyle Mistele
committed
fix(PEP8): protocol.py
Kyle Mistele
committed

Commits on Aug 7, 2024

fix(PEP8): serving_chat & fix typo in protocol
Kyle Mistele
committed
fix(PEP8): Hermes Tool parser
Kyle Mistele
committed
fix(PEP8): format files with ./format --fix
Kyle Mistele
committed
fix: docs; allow specifying the tool_use huggingface template
Kyle Mistele
committed
chore: formatting
Kyle Mistele
committed
fix: mistral chat template formatting
Kyle Mistele
committed
feat: add official mistral 7B instruct v0.3 chat template
Kyle Mistele
committed
fix: patch official mistral template to handle vLLM-generated tool call IDs in an appropriate way
Kyle Mistele
committed
fix: replace unofficial mistral chat template with official one
Kyle Mistele
committed
chore(docs): update mistral tool calling docs to remove the notes about prompt limitations, since we now have a high-quality mistral prompt template
Kyle Mistele
committed
fix(test): transformers no longer supports using a default chat template when the model doesnt provide one
Kyle Mistele
committed
fix: add chat template path for opt-125m since not specifying this is not supported when the model does not have a template in the new version of transformers
Kyle Mistele
committed
fix(test): cast posix path to string
Kyle Mistele
committed
fix(test): updated expected token count because of applying chatml template
Kyle Mistele
committed
chore: remove print
Kyle Mistele
committed
chore: merge main into constellate-ai:vllm/tool-use
Kyle Mistele
committed
chore: fix more merge conflicts (I am going to jump off a bridge if I have to do this again)
Kyle Mistele
committed
fix: add chat template due to bumped transformers version
Kyle Mistele
committed
fix: tests
Kyle Mistele
committed
fix: yapf
Kyle Mistele
committed
fix: disable yapf for block conflicting with isort
Kyle Mistele
committed
fix: merge conflicts & formatting
Kyle Mistele
committed
fix: tool_call_id was accidentally message content
Kyle Mistele
committed
fix: use double quotes in example
Kyle Mistele
committed
fix: use double quotes in chat_utils
Kyle Mistele
committed
fix: use double quotes in api_server
Kyle Mistele
committed
fix: single quotes (that I added) in cli_args are now double quotes
Kyle Mistele
committed
fix: double quotes in protocol.py
Kyle Mistele
committed
fix: double quotes in serving_chat
Kyle Mistele
committed
fix: double quotes in abstracttoolparser
Kyle Mistele
committed
fix: double quotes in hermes tool parser
Kyle Mistele
committed
fix: double quotes in mistral tool parser
Kyle Mistele
committed
fix: remove todo
Kyle Mistele
committed
fix: remove deprecated to_dict method
Kyle Mistele
committed
fix: give comments their own line
Kyle Mistele
committed
fix: remove unused loader
Kyle Mistele
committed
fix: formatting
Kyle Mistele
committed

Commits on Aug 8, 2024

fix: indents in hermes tool parser by making cases better
Kyle Mistele
committed
fix: readability for hermes tool call parser
Kyle Mistele
committed
fix: more hermes tool parser readability
Kyle Mistele
committed
fix: more clarify updates to hermes parser
Kyle Mistele
committed
fix: last hermes tool parser formatting & logic tweaks
Kyle Mistele
committed
fix: mistral tool call parser formatting
Kyle Mistele
committed
fix: remove unnecessary else block in mistral tool call parser
Kyle Mistele
committed
fix: formatting and control flow for mistral tool parser
Kyle Mistele
committed
fix: formatting updates to mistral tool call parser
Kyle Mistele
committed
fix: catch a silent exception in hermes tool parser that was generating error lines but not causing problems
Kyle Mistele
committed
fix: refactoring & cleanup serving_chat
Kyle Mistele
committed
fix: two silent errors in mistral tool parser (not causing problems) and remove the replaceAll since its not necessary any more
Kyle Mistele
committed

Commits on Aug 9, 2024

fix: CLI args in docs & comments
Kyle Mistele
committed
fix: remove line about singlq quotes for mistral
Kyle Mistele
committed
fix: update doc in mistral tool parser as well
Kyle Mistele
committed
fix: update mistral chat templates and docs for mistral tool calling
Kyle Mistele
committed
format: serving_chat
Kyle Mistele
committed
doc: in example, explain how to start server for usage
Kyle Mistele
committed
chore: docs in serving_chat and swap or "" for the correct type annotation on the left-hand operand of assignment
Kyle Mistele
committed
merge vllm:main with constellate-ai/vllm:main with constellate-ai/vllm:tool-use
Kyle Mistele
committed

Commits on Aug 10, 2024

fix: patch the hermes chat template which was missing a quote (@nous bls fix in huggingface)
Kyle Mistele
committed
tests: add parametrized pytest fixture, and begin adding tests
Kyle Mistele
committed
fix: formatting
Kyle Mistele
committed
fix: test formatting that was causing format.sh to crash
Kyle Mistele
committed
test: add test to do tool choice
Kyle Mistele
committed

Commits on Aug 12, 2024

test: add test for tool calling with tool choice; very non-streaming and streaming outputs match
Kyle Mistele
committed

Commits on Aug 13, 2024

fix(tests): download models before starting openai server to prevent timeout
Kyle Mistele
committed
try restructuring tests to make it work
Kyle Mistele
committed
format: test_tools.py
Kyle Mistele
committed
try fixing tests by removing my specific dtype=half and kv-cache-dtype=fp8 options that I use when testing
Kyle Mistele
committed
fix: download model with hf_hub to prevent mistral timeout
Kyle Mistele
committed
merge: main into tool-use
Kyle Mistele
committed
test: parallel tool calls, re-trigger CI that was interrupted by huggingface outage
Kyle Mistele
committed

Commits on Aug 14, 2024

fix: print statements
Kyle Mistele
committed
fix(tests): allow passing in wait time to remote Open AI server and increase wait time
Kyle Mistele
committed
test: add final tests for providing tool call responses with parallel tool calling
Kyle Mistele
committed
refactor(tests): break tests out into multiple files for readability & in preparation of moving them out of entrypoints
Kyle Mistele
committed

Commits on Aug 15, 2024

fix: add consolidated.safetensor to ignore list for mistral in tool entrypoint tests
Kyle Mistele
committed
refactor(ci): move tool tests out of entrypoints fastcheck, and create new non-fastcheck stage for them
Kyle Mistele
committed
fix: refactor tool use with CI changes
Kyle Mistele
committed
doc: update docs to clarify recommended CLI options & available chat templates
Kyle Mistele
committed
fix: hermes tool template and set fixtures to be session-scoped
Kyle Mistele
committed
fix: formatting
Kyle Mistele
committed
chore(tests): move tool tests out of the fastcheck section
Kyle Mistele
committed
cleanup: range statement
Kyle Mistele
committed
cleanup: unnecessary backslash
Kyle Mistele
committed
cleanup: cli args
Kyle Mistele
committed

Commits on Aug 16, 2024

fix: examples
Kyle Mistele
committed

Commits on Aug 17, 2024

fix(tests): set max tokens to a lower number that is about 30-50% above what the test should require
Kyle Mistele
committed
fix: exceptions in tool validation should be raised, not returned, so that error messages are returned to the client
Kyle Mistele
committed
chore: cleanup debug lines in tool parsers to remove less-relevant ones
Kyle Mistele
committed
fix: remove comments
Kyle Mistele
committed
refactor: tool call parsers no longer use static methods
Kyle Mistele
committed
fix: remove cruft
Kyle Mistele
committed
fix: comment on its own line
Kyle Mistele
committed
fix: remove print()
Kyle Mistele
committed
refactor: types based on earlier refactor of hermes and mistral tool parsers
Kyle Mistele
committed

Commits on Aug 18, 2024

fix(tests): make max wait timeout for RemoteOpenAIServer an instance variable only
Kyle Mistele
committed
fix: refactor RemoteOpenAIServer to use a default class var to prevent breaking tests
Kyle Mistele
committed

Commits on Aug 21, 2024

fix(test): dtype in openai/test_oot_registration
Kyle Mistele
committed
merge: main into tool-use
Kyle Mistele
committed

Commits on Aug 22, 2024

fix: problems caused by resolution of merge conflicts
Kyle Mistele
committed
fix(tests): set max model len
Kyle Mistele
committed
fix: hermes system prompt in chat template was missing <|im_start|>system\n
Kyle Mistele
committed
fix(tests): double RemoteOpenAIServer start timeout suggested by @mgoin
Kyle Mistele
committed
fix: remove cruft
Kyle Mistele
committed
fix: spacing in doc strings
Kyle Mistele
committed
fix: make enable_auto_tools a boolean not optional
Kyle Mistele
committed
fix: whitespace
Kyle Mistele
committed

Commits on Aug 23, 2024

fix: formatting
Kyle Mistele
committed
delete: old file
Kyle Mistele
committed
refactor: tool parsers to use AnyTokenizer
Kyle Mistele
committed
refactor: util.py -> utils.py
Kyle Mistele
committed
refactor: utils
Kyle Mistele
committed
fix: cruft
Kyle Mistele
committed
fix: unnecessary union in type
Kyle Mistele
committed
fix: type
Kyle Mistele
committed
fix: yapf might not need to be disabled
Kyle Mistele
committed

Commits on Aug 25, 2024

fix: exception in hermes chat template thats unnecessary cc @interstellarninja
Kyle Mistele
committed
fix: need to check if tool calls is present regardless of "content" type; perform narrowing
Kyle Mistele
committed

Commits on Aug 30, 2024

fix: type narrowing & remove unused var
Kyle Mistele
committed
fix: format
Kyle Mistele
committed

Commits on Aug 31, 2024

refactor: tool arguments in assistant-role messages with tool_calls should parsed from a string into a dict before being passed into the chat template
Kyle Mistele
committed
fix: merge conflicts in main
Kyle Mistele
committed
fix: hermes tool parser issue
Kyle Mistele
committed
Merge branch 'main' into tool-use
Kyle Mistele
committed
fix: mypy stuff in tool parsers
Kyle Mistele
committed
fix: remove cruft
Kyle Mistele
committed

Commits on Sep 3, 2024

fix: merge
Kyle Mistele
committed
fix: merge conflict + type issues
Kyle Mistele
committed

Commits on Sep 4, 2024

Merge branch 'main' into tool-use
Kyle Mistele
committed
fix(tests): update pytest fixture for client based on vllm-project#7565
Kyle Mistele
committed
fix: docs
Kyle Mistele
committed
merge: main
Kyle Mistele
committed