Support for exllama / self-hosted inference engines

[sundaraa-deshaw]

Please describe the feature you want
Tabby model spec, https://github.com/TabbyML/tabby/blob/main/MODEL_SPEC.md, says it supports only the .gguf files consumed by llama.cpp inference engine.

It will be good to be able to support other faster (on GPU) inference engines like exllama[v2].

Self-hosting a LLM usually means running a HTTP or WS based backend that runs off these engines (besides llama.cpp).

Are there ways to currently serve Tabby off such an engine? I checked the source and it seems only "vertex-ai", "fastchat" are supported to talk to an external API.

Additional context
Add any other context or screenshots about the feature request here.

---

Please reply with a 👍 if you want this feature.

[te0006]

https://github.com/mudler/LocalAI would be a very useful inference backend as well. It supports tons of open-source LLMs, is compatible to the OpenAI API, is able to switch models dynamically, can be configured to do CPU, GPU, and mixed execution, has official docker images and is MIT-licensed.

[wsxiaoys]

Close as dup of #795 (let’s continue the disucssion there)