Ollama server resolves to host.docker.internal regardless of endpoint set in .env file

[lavinir]

Describe the bug
I'm trying to run letta in docker connected to an ollama service running on the same host. I'm using an .env file with the following vars:
LETTA_LLM_ENDPOINT=http://192.168.xx.xx:11434
LETTA_LLM_ENDPOINT_TYPE=ollama
LETTA_LLM_MODEL=llama3.2:3b-instruct-q8_0
LETTA_LLM_CONTEXT_WINDOW=8192
LETTA_EMBEDDING_ENDPOINT=http://192.168.xx.xx:11434
LETTA_EMBEDDING_ENDPOINT_TYPE=ollama
LETTA_EMBEDDING_MODEL=mxbai-embed-large
LETTA_EMBEDDING_DIM=512

The server loads fine. I configure an Agent and Persona via the web interface. Then when attempting to start a chat with the agent I receive the following error on the server:

urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='host.docker.internal', port=11434): Max retries exceeded with url: /api/generate (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x72f4fd942ba0>: Failed to resolve 'host.docker.internal' ([Errno -2] Name or service not known)"))

For some reason it's trying to use host.docker.internal even though the url is overridden in the .env file.
I'm also run inspect on the container when running and confirmed the correct env settings have been applied.

Please describe your setup

• Installed with docker compose
• Running on Linux (Ubuntu)

If you're not using OpenAI, please provide additional information on your local LLM setup:

Local LLM details

If you are trying to run Letta with local LLMs, please provide the following information:

• Model: llama3.2:3b-instruct-q8_0
• Backend: Ollama
• OS: Ubuntu Linux VM (CPU only)

[sarahwooders]

Can you try this with the latest version (0.5.0 - you can do pip install -U letta)? You no longer need to set all the LLM/embedding model related variables, just the following:

export OLLAMA_BASE_URL="http://192.168.xx.xx:11434/"
letta server

[lavinir]

Hi @sarahwooders ,

Thanks it seems now the server is communicating with the Ollama endpoint and I'm able to select my models. However, when attempting a chat I'm hitting a new error complaining an Unsupported Authentication Type (I have no auth for my Ollama Server and haven't seen anything in the docs that reference this)

The Error:

Letta.letta.server.server - ERROR - Error in server._step: Unsupport authentication type:
letta_server-1  | Traceback (most recent call last):
letta_server-1  |   File "/letta/server/server.py", line 431, in _step
letta_server-1  |     step_response = letta_agent.step(
letta_server-1  |                     ^^^^^^^^^^^^^^^^^
letta_server-1  |   File "/letta/agent.py", line 874, in step
letta_server-1  |     raise e
letta_server-1  |   File "/letta/agent.py", line 784, in step
letta_server-1  |     response = self._get_ai_reply(
letta_server-1  |                ^^^^^^^^^^^^^^^^^^^
letta_server-1  |   File "/letta/agent.py", line 495, in _get_ai_reply
letta_server-1  |     raise e
letta_server-1  |   File "/letta/agent.py", line 464, in _get_ai_reply
letta_server-1  |     response = create(
letta_server-1  |                ^^^^^^^
letta_server-1  |   File "/letta/llm_api/llm_api_tools.py", line 101, in wrapper
letta_server-1  |     raise e
letta_server-1  |   File "/letta/llm_api/llm_api_tools.py", line 70, in wrapper
letta_server-1  |     return func(*args, **kwargs)
letta_server-1  |            ^^^^^^^^^^^^^^^^^^^^^
letta_server-1  |   File "/letta/llm_api/llm_api_tools.py", line 347, in create
letta_server-1  |     return get_chat_completion(
letta_server-1  |            ^^^^^^^^^^^^^^^^^^^^
letta_server-1  |   File "/letta/local_llm/chat_completion_proxy.py", line 167, in get_chat_completion
letta_server-1  |     result, usage = get_ollama_completion(endpoint, auth_type, auth_key, model, prompt, context_window)
letta_server-1  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
letta_server-1  |   File "/letta/local_llm/ollama/api.py", line 61, in get_ollama_completion
letta_server-1  |     response = post_json_auth_request(uri=URI, json_payload=request, auth_type=auth_type, auth_key=auth_key)
letta_server-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
letta_server-1  |   File "/letta/local_llm/utils.py", line 38, in post_json_auth_request
letta_server-1  |     raise ValueError(f"Unsupport authentication type: {auth_type}")
letta_server-1  | ValueError: Unsupport authentication type:
letta_server-1  | None

[LVPRESec]

I'm getting the same error with the same docker configuration:

letta_server-1 | Letta.letta.server.server - DEBUG - Got input messages: [Message(id='message-eb7b2183-e1d9-4e52-8bad-944bfbc0a271', role=<MessageRole.user: 'user'>, text='{\n "type": "user_message",\n "message": "Hello?",\n "time": "2024-10-20 06:44:17 PM UTC+0000"\n}', user_id='user-00000000', agent_id='agent-7ff38507-e691-4ed4-8a1a-7ebee17136ad', model=None, name='human', created_at=datetime.datetime(2024, 10, 20, 18, 44, 17, 526818, tzinfo=datetime.timezone.utc), tool_calls=None, tool_call_id=None)]
letta_server-1 | Letta.letta.server.server - DEBUG - Checking for agent user_id=user-00000000 agent_id=agent-7ff38507-e691-4ed4-8a1a-7ebee17136ad
letta_server-1 | Letta.letta.server.server - DEBUG - Starting agent step
letta_server-1 | Added record with id message-3abdbb08-8aea-4e39-ae6d-d2f3320d9fd7
letta_server-1 | Letta.letta.server.server - ERROR - Error in server._step: Unsupport authentication type:
letta_server-1 | Traceback (most recent call last):
letta_server-1 | File "/letta/server/server.py", line 431, in _step
letta_server-1 | step_response = letta_agent.step(
letta_server-1 | ^^^^^^^^^^^^^^^^^
letta_server-1 | File "/letta/agent.py", line 874, in step
letta_server-1 | raise e
letta_server-1 | File "/letta/agent.py", line 784, in step
letta_server-1 | response = self._get_ai_reply(
letta_server-1 | ^^^^^^^^^^^^^^^^^^^
letta_server-1 | File "/letta/agent.py", line 495, in _get_ai_reply
letta_server-1 | raise e
letta_server-1 | File "/letta/agent.py", line 464, in _get_ai_reply
letta_server-1 | response = create(
letta_server-1 | ^^^^^^^
letta_server-1 | File "/letta/llm_api/llm_api_tools.py", line 101, in wrapper
letta_server-1 | raise e
letta_server-1 | File "/letta/llm_api/llm_api_tools.py", line 70, in wrapper
letta_server-1 | return func(*args, **kwargs)
letta_server-1 | ^^^^^^^^^^^^^^^^^^^^^
letta_server-1 | File "/letta/llm_api/llm_api_tools.py", line 347, in create
letta_server-1 | return get_chat_completion(
letta_server-1 | ^^^^^^^^^^^^^^^^^^^^
letta_server-1 | File "/letta/local_llm/chat_completion_proxy.py", line 167, in get_chat_completion
letta_server-1 | result, usage = get_ollama_completion(endpoint, auth_type, auth_key, model, prompt, context_window)
letta_server-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
letta_server-1 | File "/letta/local_llm/ollama/api.py", line 61, in get_ollama_completion
letta_server-1 | response = post_json_auth_request(uri=URI, json_payload=request, auth_type=auth_type, auth_key=auth_key)
letta_server-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
letta_server-1 | File "/letta/local_llm/utils.py", line 38, in post_json_auth_request
letta_server-1 | raise ValueError(f"Unsupport authentication type: {auth_type}")
letta_server-1 | ValueError: Unsupport authentication type:
letta_server-1 | None
letta_server-1 | Letta.letta.server.server - DEBUG - Calling step_yield()
letta_db-1 | 2024-10-20 18:45:46.268 UTC [27] LOG: checkpoint starting: time
letta_db-1 | 2024-10-20 18:45:47.984 UTC [27] LOG: checkpoint complete: wrote 18 buffers (0.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=1.706 s, sync=0.003 s, total=1.717 s; sync files=12, longest=0.002 s, average=0.001 s; distance=99 kB, estimate=99 kB

[stracker-phil]

Same for me: Letta works, local models are available for selection, but chat messages yield an error message in the log.
Getting the same error with Ollama models in the latest letta server UI

Log output

...
letta_server-1  | Letta.letta.server.server - DEBUG - Checking for agent user_id=user-00000000 agent_id=agent-b9e504d9-5b64-4833-a819-7b0f77f27247
letta_server-1  | Letta.letta.server.server - DEBUG - Starting agent step
letta_server-1  | Added record with id message-49cdbad1-e5e5-4376-9fc9-8c1ee386cccd
letta_server-1  | Letta.letta.server.server - ERROR - Error in server._step: Unsupport authentication type: 
letta_server-1  | Traceback (most recent call last):
letta_server-1  |   File "/letta/server/server.py", line 431, in _step
letta_server-1  |     step_response = letta_agent.step(
letta_server-1  |                     ^^^^^^^^^^^^^^^^^
letta_server-1  |   File "/letta/agent.py", line 874, in step
letta_server-1  |     raise e
letta_server-1  |   File "/letta/agent.py", line 784, in step
letta_server-1  |     response = self._get_ai_reply(
letta_server-1  |                ^^^^^^^^^^^^^^^^^^^
letta_server-1  |   File "/letta/agent.py", line 495, in _get_ai_reply
letta_server-1  |     raise e
letta_server-1  |   File "/letta/agent.py", line 464, in _get_ai_reply
letta_server-1  |     response = create(
letta_server-1  |                ^^^^^^^
letta_server-1  |   File "/letta/llm_api/llm_api_tools.py", line 101, in wrapper
letta_server-1  |     raise e
letta_server-1  |   File "/letta/llm_api/llm_api_tools.py", line 70, in wrapper
letta_server-1  |     return func(*args, **kwargs)
letta_server-1  |            ^^^^^^^^^^^^^^^^^^^^^
letta_server-1  |   File "/letta/llm_api/llm_api_tools.py", line 347, in create
letta_server-1  |     return get_chat_completion(
letta_server-1  |            ^^^^^^^^^^^^^^^^^^^^
letta_server-1  |   File "/letta/local_llm/chat_completion_proxy.py", line 167, in get_chat_completion
letta_server-1  |     result, usage = get_ollama_completion(endpoint, auth_type, auth_key, model, prompt, context_window)
letta_server-1  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
letta_server-1  |   File "/letta/local_llm/ollama/api.py", line 61, in get_ollama_completion
letta_server-1  |     response = post_json_auth_request(uri=URI, json_payload=request, auth_type=auth_type, auth_key=auth_key)
letta_server-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
letta_server-1  |   File "/letta/local_llm/utils.py", line 38, in post_json_auth_request
letta_server-1  |     raise ValueError(f"Unsupport authentication type: {auth_type}")
letta_server-1  | ValueError: Unsupport authentication type: 
letta_server-1  | None
letta_server-1  | Letta.letta.server.server - DEBUG - Calling step_yield()

[Asamsodien]

experiencing the same issue
Unsupport authentication type: {auth_type}

currently using container letta/letta:nightly (9a819e78d7b9)

[lavinir]

Hit a different issue with the UI after that but for what it's worth, based on a suggestion from someone on the discord server, this issue can be worked around by moving the OLLAMA_BASE_URL from the .env file directly into the docker compose file.

[sarahwooders]

The fix for the authentication type error should be in #2018 - we'll rebuild the letta/letta:nightly image once it's merged.

If you're running Ollama locally, remember to use run export OLLAMA_BASE_URL="http://host.docker.internal:11434".

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.