Unable to switch models via api(from litellm)

[iguy0]

Describe the bug

I have litellm configured as proxy according to their documentation(https://docs.litellm.ai/docs/providers/openai_compatible), but when attempting to use a specific model via the completions endpoint, it fails with the following error message IF the model isn't loaded already:

Exception in ASGI application
Traceback (most recent call last):
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/extensions/openai/script.py", line 137, in openai_chat_completions
    response = OAIcompletions.chat_completions(to_dict(request_data), is_legacy=is_legacy)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/extensions/openai/completions.py", line 538, in chat_completions
    return deque(generator, maxlen=1).pop()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/extensions/openai/completions.py", line 318, in chat_completions_common
    token_count = len(encode(prompt)[0])
                      ^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/modules/text_generation.py", line 115, in encode
    raise ValueError('No tokenizer is loaded')
ValueError: No tokenizer is loaded
Exception in ASGI application
Traceback (most recent call last):
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/extensions/openai/script.py", line 137, in openai_chat_completions
    response = OAIcompletions.chat_completions(to_dict(request_data), is_legacy=is_legacy)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/extensions/openai/completions.py", line 538, in chat_completions
    return deque(generator, maxlen=1).pop()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/extensions/openai/completions.py", line 318, in chat_completions_common
    token_count = len(encode(prompt)[0])
                      ^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/modules/text_generation.py", line 115, in encode
    raise ValueError('No tokenizer is loaded')
ValueError: No tokenizer is loaded
Exception in ASGI application
Traceback (most recent call last):
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/extensions/openai/script.py", line 137, in openai_chat_completions
    response = OAIcompletions.chat_completions(to_dict(request_data), is_legacy=is_legacy)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/extensions/openai/completions.py", line 538, in chat_completions
    return deque(generator, maxlen=1).pop()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/extensions/openai/completions.py", line 318, in chat_completions_common
    token_count = len(encode(prompt)[0])
                      ^^^^^^^^^^^^^^
  File "/home/v2/projects/text-generation-webui/modules/text_generation.py", line 115, in encode
    raise ValueError('No tokenizer is loaded')
ValueError: No tokenizer is loaded

In case the model is loaded in ooba, the completions query works for the loaded model. In case i attempt to use a different model, it routes to the model already loaded. Not respecting the request.

Please let me know if you need more info.

Is there an existing issue for this?

• I have searched the existing issues

Reproduction

.

Screenshot

No response

Logs

.

System Info

.

[Touch-Night]

Maybe #5720 can fix this

[St33lMouse]

I get the same thing.

[github-actions]

This issue has been closed due to inactivity for 2 months. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.