Use local LLM via Ollama

[Norfeldt]

Is it possible to use a local LLM via Ollama. If, what's the setup and what's the requirement for which LLM I can use (guessing it has to use openai api syntax)?

[netcaster1]

1. ai config
   a. change openai api endpoint to http://{your ollama ip:port}/v1
   b. if step a is done properly, then when you select 'Model ', you can see all models in your ollama, just select your prefer one
   (if not working, check if OLLAMA_ORIGINS=* has been setup correctly)

2. next will be a trick, looks still ai shell will look for 'gpt-3.5-turbo' model, then let's create one fake one in ollama to cheat ai shell
   for example:
   ollama show llama3.1:latest --modelfile > /tmp/llama3.1.modelfile
   ollama create gpt-3.5-turbo --file /tmp/llama3.1.modelfile
   ollama list

3. test
   ai list files

Done.

[Ajaymamtora]

It starts writing out the command but then cancels itself and says this:

Request to OpenAI failed with status 404:

{
"error": {
"message": "model "gpt-3.5-turbo" not found, try pulling it first",
"type": "api_error",
"param": null,
"code": null
}
}

[mxl]

@Ajaymamtora This was fixed in #115 but a new version is not released yet. @steve8708 could you help us with that?

[steve8708]

published!