BootChatAI OpenAPI, OpenAI OpenAPI, AzureAI OpenAPI.... how many OpenAPI contracts ?

[agoncal]

BootChatAI has its own very simple OpenAPI contract:

• a /chat endpoint
• Request object
• Response object

But then, we have several OpenAI endpoints (/chat/completions, audio, moderation`...) and we might have other APIs that could be interested to support (AzureAI, etc.).

I find it interesting to have ChatBootAI supporting several APIs. We could start with, at least 2 (OpenAI and AzureAI) but we might end up with several depending on the contributions. If that's the case, we have several issues:

• Having a single OpenAPI contract does not make sense anymore (too big)
• Multiplying APIs (I can already see a Chat API for ChatBootAI, OpenAI, AzureAI...) and endpoints (we will clash all the /chat endpoints)
• The UI will be bloated (the advanced option will not be enough, we would need an OpenAI component, an AzureAI component....)
• Contribution could be more challenging if someone only wants to contribute on the OpenAI APIs, then she/he would have to go through lot of other code

But again, saying that, I quite like the idea for BootChatAI to be the main AI tool when it comes to chatting (being able to connect to dozens of different APIs would be awesome)

[sinedied]

Dropping some of my thoughts here:

1. We have the /chat endpoint as a contract between our backend and UI

• This is the "end-user" API
• It acts like an agent, with an implementation that is sample and language dependent.
• Implementation in samples already use some form of an abstraction layer, namely LangChain or SK, allowing to connect to different AI models: OpenAI, Azure, Anthropic etc.

2. The other endpoints can be seen in two different ways, and we have yet to decide how to approach it:

   1. As an implementation detail of the /chat API, used through LC tools for examples. As such, they do not need to be exposed to the end-user, but could be for educational/debug purposes. Since we already have an abstraction layer in the /chat implementation, I'm not sure of what would be their roles rather than a simple pass-through for education.

   2. As a lightweight abstraction layer for other models abstractions. For example, if we take OpenAI as a reference, there are already existing proxy projects that converts OpenAI calls to other models. As a benefit, this makes the implementation easier if we only have to consider usage of OpenAI API, but the benefit is small since we already have an abstraction layer (LC or SK). LC currently only abstracts text models (completion and embeddings), so audio and image might still be useful as an API.

My take: 1) is where we bring most of the value, and it's already a good target for an MVP.

For 2) I don't see much value in the first approach. Even for debug purpose, we could make a dedicated debug component that just directly call OpenAI APIs if the call is only educational/debugging.

The second approach brings additional complexity for LLMs that we don't need (LC already abstracts this), but with the proxy approach this makes a clean decoupling and removes the UI and /chat implementation of any concerns regarding model portability. We could also piggyback on other OSS projects such as https://github.com/BerriAI/litellm for example.

Going for the "OpenAI proxy" approach also brings other benefits for us:

• We don't need to care about it right now 🙂
• We can add support for it only when we need it, with not impact in the samples/ui implementation (as long as the base OpenAI url for the calls is an environment variable)
• We can rely on other community projects for the implementation

[sinedied]

Thanks for the proposal, you're an awesome job with LiteLLM!
For the chat completion and embedding use case, we should be good with what LiteLLM already propose.

But we'd like to go multi modal and also make use of OpenAI other APIs, including image generation and speech to text (for the first two we're interested in). Do you think that's something that could be in the scope of LiteLLM?

[ishaan-jaff]

we'd be open to the suggestion - why do you want to use a proxy server ?

[sinedied]

Our main use case would be to allow quickly switching image/audio models (like stable diffusion for images), or cloud providers without the need to change the underlying code.

[ishaan-jaff]

tracking this here: BerriAI/litellm#999

if we add this today, would you use our proxy and give us feedback ?

[agoncal]

@ishaan-jaff at the moment we are focusing on a 0.1 release with chat completion and documentation. Multi modal will come next. Give us a bit of time and we will investigate further how we could integrate LiteLLM