Using HostedGPT with a Local LLM like Gemma 2 9B

[krschacht]

We recently added the ability for HostedGPT to point to any API server, including local servers. You can now run the app completely free if you download a model to your computer.

I've been trying out the new Gemma 2 models. My computer is not powerful enough to run the 27B model so I'm using Gemma 2 9B. Here are the steps, in case anyone would like to try it.

I recommend LM Studio but a lot of other people use Llama.cpp / Llamafiles so those should work too.

1. Download https://lmstudio.ai/
2. Search for "gemma-2-9b" within the app and download "Q6_K" (this is the full quality model, the others are "compressed" which lowers the quality.)
3. In the app, click the Local Server tab on the left (diamond arrows icon) and click "Start Server"
4. Downloaded HostedGPT: git clone https://github.com/AllYourBot/hostedgpt.git
5. cd hostedgpt and run docker compose up --build
6. Now open http://localhost:3000/ and you'll see a sign up screen for HostedGPT. Create your account.
7. After your logged in, click your name in bottom left and go to Settings
8. First: API Services > Add New >
   • Name: LM Studio
   • Driver: openai
   • URL: http://localhost:1234/
   • Token: (leave blank)
9. Then: Language Model > Add New >
   • Name: Gemma 2 9b
   • API Name: bartowski/gemma-2-9b-it-GGUF
   • API Service: select LM Studio
10. Finally: click "New Assistant"
    • Name: Gemma 2 9b
    • Language Model: select Gemma 2 9b
    • Save
11. Exit settings (X in top right) and try chatting with your new Gemma 2 model by selecting it on the left

If you try this, I'm eager to hear how well it works — both the setup and the model itself.

[alphabetek]

It works great!

If the same model is provided by multiple API services, the user could be confused about which one to choose from the list in the new assistant screen. Maybe adding a filter for "API services" on that screen or showing the API service in the list (e.g. Gemma 2 9b - LM Studio) could avoid this ambiguity.

[krschacht]

Glad it's working! And that's a really a good suggestion. Or at the least, even if there is no explicit filter, the list should at least include mention of the API service so that you can visually distinguish them.

[krschacht]

I plan to take a design pass on those settings screens soon so I added this as an item: #432