-
Notifications
You must be signed in to change notification settings - Fork 937
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide an interface similar to OpenAI API #334
Comments
@Pevernow |
Users want something like this https://github.com/lm-sys/FastChat/blob/main/docs/openai_api.md, so they can switch their apps from OpenAI models to TRT-LLM models easily without code change. |
@juney-nvidia Please take a look here, thank you |
Right now Python API still have a lot of issues to be fixed, I encapsulated one OpenAI API, but met #283, so you still need use C++ runtime, which means you need Triton. Spent weeks on TRT-LLM, it difficult to develop on python runtime. @juney-nvidia What's the position of TRT-LLM's Python runime? I mean, python is easier than C++, and batch manager doesn't open source right now. Most developors may not use Triton since they won't meet that large commercial demand. |
mark |
Sorry for replying late due to being trapped by other things.
@Pevernow @merrymercy well received, will discuss with prod about this. @ncomly-nvidia for vis.
@gesanqiu we have already released the Python binding of C++ runtime, including batch manager, does this fulfill your requirement here? |
vote up, monitoring |
More info about openai chat completion API spec here |
i need it too |
+1 for OpenAI API support |
1 similar comment
+1 for OpenAI API support |
+1 for OpenAI API support, it's been 9 months since the pr requested :( |
+1 OpenApi support |
+1 |
+1 for openai api support。 |
+1 for openai api support |
1 similar comment
+1 for openai api support |
+1 for openai api support |
Could you please provide a simple interface similar to OpenAI API?
The text was updated successfully, but these errors were encountered: