Local models?

[FinnT730]

Will local models be supported one day as well?
(Unless they are, and I didn't find it in the readme XD)

[vyokky]

Will local models be supported one day as well? (Unless they are, and I didn't find it in the readme XD)

That's on our todo list ;)

[FinnT730]

👍 Thanks! Will wait for that.
Is it currently possible to use a program that emulates OpenAI's API? Something like Ollama?

[vyokky]

👍 Thanks! Will wait for that. Is it currently possible to use a program that emulates OpenAI's API? Something like Ollama?

It only supports GPT-V for now. We plan to incorporate more models in the future.

[calamity10110]

I tried to edit config file:
OPENAI_API_BASE: "http://127.0.0.1:11434/" # The the OpenAI API endpoint,
OPENAI_API_KEY: "Null" # The API key
OPENAI_API_MODEL: "Llava"
result
Error making API request: Extra data: line 1 column 5 (char 4)
Error occurs when calling LLM.
Error making API request: Extra data: line 1 column 5 (char 4)
Error occurs when calling LLM.

for url http://127.0.0.1:11434/v1/chat/completions
Error making API request: 400 Client Error: Bad Request for url: http://127.0.0.1:11434/v1/chat/completions
{'error': {'message': 'json: cannot unmarshal array into Go struct field Message.messages.content of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}}
Error making API request: 400 Client Error: Bad Request for url: http://127.0.0.1:11434/v1/chat/completions
{'error': {'message': 'json: cannot unmarshal array into Go struct field Message.messages.content of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}}
Error making API request: 400 Client Error: Bad Request for url: http://127.0.0.1:11434/v1/chat/completions
{'error': {'message': 'json: cannot unmarshal array into Go struct field Message.messages.content of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}}
Error occurs when calling LLM.

[vyokky]

Hi @calamity10110 , the current framework does not support non-openai model. We are working on it and will release a new feature for this soon.

[Justin-12138]

I am a non-subscriber of openai, can I still use UFO? I follow your instructions and my config file as following，I set the model to gpt3.5
version: 0.1
API_TYPE: "openai" # The API type, "openai" for the OpenAI API, "aoai" for the AOAI API.
OPENAI_API_BASE: "https://api.openai.com/v1/chat/completions" # The the OpenAI API endpoint, "https://api.openai.com/v1/chat/completions" for the OpenAI API.
OPENAI_API_KEY: "mykey" # The OpenAI API key
OPENAI_API_MODEL: "gpt-3.5-turbo-0301" # The only OpenAI model by now that accepts visual input
CONTROL_BACKEND: "uia" # The backend for control action

But I got error like this:

[FinnT730]

Well, I am waiting for when local models can be used.
Right now, yes you need access to the API of OpenAPI, which is not free.

[vyokky]

@FinnT730 @Justin-12138 You can now use models in Ollama for your local model deployment (in the pre-release branch). Please read https://github.com/microsoft/UFO/blob/pre-release/model_worker/README.md for details, and expect worse performance than GPT-V.

[vyokky]

I tried to edit config file: OPENAI_API_BASE: "http://127.0.0.1:11434/" # The the OpenAI API endpoint, OPENAI_API_KEY: "Null" # The API key OPENAI_API_MODEL: "Llava" result Error making API request: Extra data: line 1 column 5 (char 4) Error occurs when calling LLM. Error making API request: Extra data: line 1 column 5 (char 4) Error occurs when calling LLM.

for url http://127.0.0.1:11434/v1/chat/completions Error making API request: 400 Client Error: Bad Request for url: http://127.0.0.1:11434/v1/chat/completions {'error': {'message': 'json: cannot unmarshal array into Go struct field Message.messages.content of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}} Error making API request: 400 Client Error: Bad Request for url: http://127.0.0.1:11434/v1/chat/completions {'error': {'message': 'json: cannot unmarshal array into Go struct field Message.messages.content of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}} Error making API request: 400 Client Error: Bad Request for url: http://127.0.0.1:11434/v1/chat/completions {'error': {'message': 'json: cannot unmarshal array into Go struct field Message.messages.content of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}} Error occurs when calling LLM.

@calamity10110 You can now use models in Ollama for Llava deployment (in the pre-release branch). Please read https://github.com/microsoft/UFO/blob/pre-release/model_worker/README.md for details, and expect worse performance than GPT-V.

[FinnT730]

Thanks for the update! Have a good day, and thanks for working on this feature :)

…

On Wed, 10 Apr 2024, 14:09 vyokky, ***@***.***> wrote: @FinnT730 <https://github.com/FinnT730> @Justin-12138 <https://github.com/Justin-12138> You can now use models in Ollama for your local model deployment (in the pre-release branch). Please read https://github.com/microsoft/UFO/blob/pre-release/model_worker/README.md for details, and expect worse performance than GPT-V. — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYZB4VRGUHKKQFKRWHWUM3Y4UTXZAVCNFSM6AAAAABDJJ5YCCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBXGM3TOMRTGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[zsb87]

Hello Team,

I tried using local model llava in pre-release branch, but unfortunately got this error. Did I miss anything here? Thanks

[vyokky]

@Mac0q

[Mac0q]

@zsb87 It appears that your local model or API is refusing to respond. Usually this is because the model has limited functionality. Can you tell me your model version?

[zsb87]

@Mac0q . This is my model version:
{"name":"llava:latest","model":"llava:latest","modified_at":"2024-05-20T13:50:45.2323374-07:00","size":4733363377,"digest":"8dd30f6b0cb19f555f2c7a7ebda861449ea2cc76bf1f44e262931f45fc81d081","details":{"parent_model":"","format":"gguf","family":"llama","families":["llama","clip"],"parameter_size":"7B","quantization_level":"Q4_0"},"expires_at":"0001-01-01T00:00:00Z"}

[Mac0q]

@zsb87 I think llava:7b is still weak for this task, we will try it optimize the prompt to make it doable, but GPT-4V is for sure the best choice.