[extensions/openai] Failure to load instruction-following template for model

[MikeL1300]

Describe the bug

Always getting the following error for every model, despite the model having an instruction template correctly selected in the UI:

127.0.0.1 - - [25/Jul/2023 23:31:15] "POST /v1/chat/completions HTTP/1.1" 200 -
Exception: When loading characters/instruction-following/None.yaml: FileNotFoundError(2, 'No such file or directory')
Warning: Loaded default instruction-following template for model.

Getting this error even if the model has a correct template in "characters/instruction-following" that is automatically loaded and matches a pattern in "models/config.yaml".

Looking at the code, it seems the following line in "completions.py" doesn't work as expected, with "shared.settings['instruction_template']" always being "None":

instruct = yaml.safe_load(open(f"characters/instruction-following/{shared.settings['instruction_template']}.yaml", 'r'))

If it is replaced with (for example):

instruct = yaml.safe_load(open(f"characters/instruction-following/Alpaca.yaml", 'r'))

Then the template is loaded correctly.

Is there an existing issue for this?

• I have searched the existing issues

Reproduction

This error happens for every model once the API request is sent. For example the model: "TheBloke/Llama-2-13B-chat-GGML"

The model fits the pattern that exists in "config.yaml":

.*llama-(2|v2).*chat:
  mode: 'instruct'
  instruction_template: 'Llama-v2'

"Llama-v2" can also be selected as the instruction template in the UI (and is in fact automatically selected when loading the model).

However when sending an API request, the error appears.

Screenshot

No response

Logs

127.0.0.1 - - [25/Jul/2023 23:31:15] "POST /v1/chat/completions HTTP/1.1" 200 -
Exception: When loading characters/instruction-following/None.yaml: FileNotFoundError(2, 'No such file or directory')
Warning: Loaded default instruction-following template for model.

System Info

Windows 10

[matatonic]

Confirmed when loading the model via the UI. Thanks for the report, the shared settings are not being updated any more it seems.

As a workaround, It works if loading a model via the API or via the startup --model command.

[corallofrancesco]

I encountered the same problem: the openai extension does not refresh the shared.settings.
As @matatonic said, a partial workaround is to load the model at startup with the --model command, but if you change the model on the UI, the shared settings are NOT updated accordingly ( same issues in #2911 and #3153 ).

[MikeL1300]

Confirmed when loading the model via the UI. Thanks for the report, the shared settings are not being updated any more it seems.

As a workaround, It works if loading a model via the API or via the startup --model command.

Thanks for the response. Can you give an example (preferably in python) of how the model should be loaded through the API? Using "openai.ChatCompletion.create" with a model name doesn't seem to work. I tried the "/v1/engines/{model_name}" endpoint but got the following error when sending a request there:

OpenAIError ValueError('too many values to unpack (expected 2)')
Traceback (most recent call last):
File "D:\AI\text-generation-webui\oobabooga_windows\text-generation-webui\extensions\openai\script.py", line 98, in wrapper
func(self)
File "D:\AI\text-generation-webui\oobabooga_windows\text-generation-webui\extensions\openai\script.py", line 118, in do_GET
resp = OAImodels.load_model(model_name)
File "D:\AI\text-generation-webui\oobabooga_windows\text-generation-webui\extensions\openai\models.py", line 45, in load_model
shared.model, shared.tokenizer = load_model(shared.model_name)
ValueError: too many values to unpack (expected 2)

127.0.0.1 - - [26/Jul/2023 20:50:09] "GET /v1/engines/TheBloke_Llama-2-13B-chat-GGML HTTP/1.1" 500 -

[matatonic]

There is an example in the api-examples/ folder, which includes a few robust options for loading models. This is essentially what I use myself.

As you found, there is a bug in the (simple) model loaded hacked into the openai extension (which is also usable from the command line, like: openai api models.get -i TheBloke_WizardLM-7B-uncensored-GPTQ). That bug (#3305) is fixed in this PR: #3309

[MikeL1300]

Thanks, loading using the (non extension) API worked.

Great extension by the way, thank you for working on it! Makes it very easy to port an app using the openAI API to a local model. I just hope the issues with the UI are fixed soon, since a very common use case is to run the web UI to play with various options on the fly while sending requests through an app.

[github-actions]

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.