Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stream tool calling #349

Merged
merged 2 commits into from
Oct 31, 2024
Merged

stream tool calling #349

merged 2 commits into from
Oct 31, 2024

Conversation

whisper-bye
Copy link
Contributor

Before submitting
  • Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?

What does this PR do?

According to openai's spec, ChatCompletionMessageToolCallChunk requires an index field, which is currently missing in litserve's ToolCall model, making it impossible to use it with ai sdk's streamText.

https://github.com/openai/openai-openapi/blob/master/openapi.yaml#L10749-L10774

class ToolCall(BaseModel):
    id: Optional[str] = None
    type: str = "function"
    function: FunctionCall
const result = await streamText({
  model: openai('gpt-3.5-turbo'),
  messages: ...
  tools: {
    ...
  }
});

https://sdk.vercel.ai/examples/node/generating-text/stream-text

my demo

import litserve as ls
from litserve import LitAPI
from litserve.specs.openai import ChatMessage, OpenAISpec


class TestAPI(LitAPI):
    def setup(self, device):
        self.model = None

    def predict(self, x):
        yield "This is a generated output"


class TestAPIWithToolCalls(TestAPI):
    def encode_response(self, output, context):
        yield ChatMessage(
            role="assistant",
            content="",
            tool_calls=[
                {
                    "id": "call_1",
                    "type": "function",
                    "function": {"name": "getTimeInformation", "arguments": '{}'},
                    "index": 0,
                }
            ],
        )


if __name__ == '__main__':
    api = TestAPIWithToolCalls()
    server = ls.LitServer(api, spec=OpenAISpec(), stream=True)
    server.run(port=8000)

this PR just add an index field

class ToolCall(BaseModel):
    index: int = 0
    id: Optional[str] = None
    type: str = "function"
    function: FunctionCall

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

@aniketmaurya
Copy link
Collaborator

Thanks for reporting and creating a fix @whisper-bye!! could you also please take a look at the tests? You can update the test to include index: 0 in the expected output.

async def test_openai_spec_with_tools(openai_request_data_with_tools):

@aniketmaurya aniketmaurya added the bug Something isn't working label Oct 30, 2024
@whisper-bye
Copy link
Contributor Author

Thanks for reporting and creating a fix @whisper-bye!! could you also please take a look at the tests? You can update the test to include index: 0 in the expected output.

async def test_openai_spec_with_tools(openai_request_data_with_tools):

Okay, I updated the test.

Copy link

codecov bot commented Oct 31, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96%. Comparing base (1f96db5) to head (87cd2e5).
Report is 1 commits behind head on main.

Additional details and impacted files
@@         Coverage Diff         @@
##           main   #349   +/-   ##
===================================
  Coverage    95%    96%           
===================================
  Files        22     22           
  Lines      1384   1394   +10     
===================================
+ Hits       1317   1332   +15     
+ Misses       67     62    -5     

Copy link
Collaborator

@aniketmaurya aniketmaurya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀 ! great work @whisper-bye

@aniketmaurya aniketmaurya merged commit 177f0ef into Lightning-AI:main Oct 31, 2024
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants