Feat/support new backend #138
Conversation
…entation Signed-off-by: csh <458761603@qq.com>
Signed-off-by: csh <458761603@qq.com>
Signed-off-by: csh <458761603@qq.com>
| options: LoadModelOptions, | ||
| tx: Sender<anyhow::Result<LoadModelResponse>>, | ||
| ) -> Self; | ||
| fn chat( |
There was a problem hiding this comment.
@L-jasmine Do we expect or plan to have a functionality in Moxin where it would be possible to chat with the model that is loaded behind a local web server? I'm kind of surprised we're adding code for this feature, since it is not what I thought when the local sever commands of the protocol were added.
In any case, I'm not against having this implemented, but I would double check if we really want to support such feature (which could not make sense, because the ability of spawning a local server seems to be more oriented to allow users to experiment with code snippets rather than interacting with the model using the Moxin UI.
Posting here the user interface of LMStudio for the local HTTP servers, just as a reference.
There was a problem hiding this comment.
Michael has just explained the plans in our weekly sync, so I better understand the motivations of these changes. No need to respond again here :)
There was a problem hiding this comment.
@L-jasmine @juntao Left a comment/question about the overall direction of this type of improvements, but the PR LGTM 🚢
We plan to use llama-api-server and llama-rag-server as the underlying wasm for a local web server. Therefore, I refactored the Backend code by defining a Trait to support multiple Backend implementations.
Currently, the api-server.wasm cannot be stopped, so the logic in use is still the original setup.