Inquiry: Support for External Inference Endpoint in Tabby

[d0rc]

Hey Tabby Team,

Love what you're doing with Tabby, it's a game-changer! Quick question: is there a way to route Tabby's inference requests to an external endpoint? I've got a setup that batches requests to remote GPUs for better performance.

It could lower latency and let users like me use bigger, custom models. Plus, it might be a cool feature for others too. Just a thought – totally get it if it's not doable right now.

Thanks for all the awesome work!

[wsxiaoys]

Is this endpoint dedicate to tabby? If that's the case, you might consider deploy tabby to this machine directly?

[d0rc]

I was hoping I can give Tabby endpoint for inference, like with OpenAI protocol or something similar.

[wsxiaoys]

Related discussion: #795