OLLAMA models not working at all with LiteLLM

[ArindamRoy23]

I am trying to run the basic tutorial. Only changes I have made are: calling the Ollama models thru LiteLLM instead of standard API calls. The same model is working ok thru LiteLLM when calling thru Groq API. When I call the model from Ollama, there are a myriad of errors. I have tried:

• Llama 3.1 7B
• Phi 4 mini
• GPT OSS 20B
• Llama 3 7B
• Gemma 240M

Errors I have faced:

- No response:

adk run _agents/
Log setup complete: /tmp/agents_log/agent.20250903_110925.log
To access latest log: tail -F /tmp/agents_log/agent.latest.log
/home/vscode/.cache/pypoetry/virtualenvs/tidyworld-NI6vCWsz-py3.10/lib/python3.10/site-packages/google/adk/cli/cli.py:143: UserWarning: [EXPERIMENTAL] InMemoryCredentialService: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
  credential_service = InMemoryCredentialService()
/home/vscode/.cache/pypoetry/virtualenvs/tidyworld-NI6vCWsz-py3.10/lib/python3.10/site-packages/google/adk/auth/credential_service/in_memory_credential_service.py:33: UserWarning: [EXPERIMENTAL] BaseCredentialService: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
  super().__init__()
Running agent weather_time_agent, type exit to exit.
[user]: Whats the weather in NY> ?
11:09:34 - LiteLLM:INFO: utils.py:3085 - 
LiteLLM completion() model= gpt-oss:20b; provider = ollama_chat
11:10:59 - LiteLLM:INFO: cost_calculator.py:636 - selected model name for cost calculation: ollama_chat/gpt-oss:20b
11:10:59 - LiteLLM:INFO: cost_calculator.py:636 - selected model name for cost calculation: ollama_chat/gpt-oss:20b
[user]: 

- Tool Metadata as response

[user]: What's the weather in NY?
07:20:02 - LiteLLM:INFO: utils.py:3085 - 
LiteLLM completion() model= llama3.1:8b; provider = ollama_chat
07:20:12 - LiteLLM:INFO: cost_calculator.py:636 - selected model name for cost calculation: ollama_chat/llama3.1:8b
07:20:12 - LiteLLM:INFO: cost_calculator.py:636 - selected model name for cost calculation: ollama_chat/llama3.1:8b
[weather_time_agent]: {"name": "get_weather", "arguments":{"city": "NY"}}

- Hallucinated random structured output (Phi Model)
Gives a random structured output response without using tool.

Code

import datetime

from google.adk.agents import Agent

# from _llm._default import DefaultLLMService
from google.adk.models.lite_llm import LiteLlm
from zoneinfo import ZoneInfo


def get_weather(city: str) -> dict:
    """Retrieves the current weather report for a specified city.

    Args:
        city (str): The name of the city for which to retrieve the weather report.

    Returns:
        dict: status and result or error msg.
    """
    print(f"*************Getting weather for {city}")
    if city.lower() == "new york":
        return {
            "status": "success",
            "report": (
                "The weather in New York is sunny with a temperature of 25 degrees" " Celsius (77 degrees Fahrenheit)."
            ),
        }
    else:
        return {
            "status": "error",
            "error_message": f"Weather information for '{city}' is not available.",
        }


def get_current_time(city: str) -> dict:
    """Returns the current time in a specified city.

    Args:
        city (str): The name of the city for which to retrieve the current time.

    Returns:
        dict: status and result or error msg.
    """
    if city.lower() == "new york":
        tz_identifier = "America/New_York"
    else:
        return {
            "status": "error",
            "error_message": (f"Sorry, I don't have timezone information for {city}."),
        }

    tz = ZoneInfo(tz_identifier)
    now = datetime.datetime.now(tz)
    report = f'The current time in {city} is {now.strftime("%Y-%m-%d %H:%M:%S %Z%z")}'
    return {"status": "success", "report": report}


# llm = DefaultLLMService(
#     model="groq/openai/gpt-oss-20b"  # , config={"api_base": "http://host.docker.internal:11434"}
# ).get_model()
import os

llm = LiteLlm(model="groq/openai/gpt-oss-20b")
# llm = LiteLlm(model="ollama_chat/gpt-oss:20b", api_base="http://host.docker.internal:11434")
root_agent = Agent(
    name="weather_time_agent",
    model=llm,
    description=("Agent to answer questions about the time and weather in a city."),
    instruction=("You are a helpful agent who can answer user questions about the time and weather in a city."),
    tools=[get_weather, get_current_time],
)

Config

• Ubuntu Docker dev container
• google-ADK: 1.13.0
• LiteLLM: 1.65.8
• Ollama: 0.11.8

[gabrik]

I'm having the same issue, is there any workaround/solution?

[desolatorxxl]

@ArindamRoy23 the docs recommend to export OLLAMA_API_BASE in your shell, e.g.:

export OLLAMA_API_BASE="http://localhost:11434"

Have you tried that?

[gabrik]

@ArindamRoy23 the docs recommend to export OLLAMA_API_BASE in your shell, e.g.:

export OLLAMA_API_BASE="http://localhost:11434"
Have you tried that?

I'm actually creating the client

client = OpenAI(
    base_url="http://localhost:11434", # here tried with both /v1 path and without, same result
    api_key="ollama"  # Dummy key, but required
)

root_agent = Agent(
    name="weather_time_agent",
    model=LiteLlm(model="ollama/gpt-oss:20b", client=client),
    description=(
        "Agent to answer questions about the time and weather in a city."
    ),
    instruction=(
        "You are a helpful agent who can answer user questions about the time and weather in a city."
    ),
    tools=[get_weather, get_current_time],

If it can help, this is the output with litellm._turn_on_debug()

[user]: hi!
10:05:08 - LiteLLM:DEBUG: utils.py:349 - 

10:05:08 - LiteLLM:DEBUG: utils.py:349 - Request to litellm:
10:05:08 - LiteLLM:DEBUG: utils.py:349 - litellm.acompletion(model='ollama/gpt-oss:20b', messages=[{'role': 'developer', 'content': 'You are a helpful agent who can answer user questions about the time and weather in a city.\n\nYou are an agent. Your internal name is "weather_time_agent".\n\n The description about you is "Agent to answer questions about the time and weather in a city."'}, {'role': 'user', 'content': 'hhi!'}], tools=[{'type': 'function', 'function': {'name': 'get_weather', 'description': 'Retrieves the current weather report for a specified city.\n\n    Args:\n        city (str): The name of the city for which to retrieve the weather report.\n\n    Returns:\n        dict: status and result or error msg.\n    ', 'parameters': {'type': 'object', 'properties': {'city': {'type': 'string'}}, 'required': ['city']}}}, {'type': 'function', 'function': {'name': 'get_current_time', 'description': 'Returns the current time in a specified city.\n\n    Args:\n        city (str): The name of the city for which to retrieve the current time.\n\n    Returns:\n        dict: status and result or error msg.\n    ', 'parameters': {'type': 'object', 'properties': {'city': {'type': 'string'}}, 'required': ['city']}}}], response_format=None, client=<openai.OpenAI object at 0x13fe88280>)
10:05:08 - LiteLLM:DEBUG: utils.py:349 - 

10:05:08 - LiteLLM:DEBUG: litellm_logging.py:474 - self.optional_params: {}
10:05:08 - LiteLLM:DEBUG: utils.py:349 - ASYNC kwargs[caching]: False; litellm.cache: None; kwargs.get('cache'): None
10:05:08 - LiteLLM:DEBUG: caching_handler.py:234 - CACHE RESULT: None
10:05:08 - LiteLLM:DEBUG: base_utils.py:219 - Translating developer role to system role for non-OpenAI providers.
10:05:08 - LiteLLM:INFO: utils.py:3338 - 
LiteLLM completion() model= gpt-oss:20b; provider = ollama
10:05:08 - LiteLLM:DEBUG: utils.py:3341 - 
LiteLLM: Params passed to completion() {'functions': None, 'function_call': None, 'temperature': None, 'top_p': None, 'stream': None, 'stream_options': None, 'stop': None, 'max_tokens': None, 'max_completion_tokens': None, 'modalities': None, 'prediction': None, 'audio': None, 'presence_penalty': None, 'frequency_penalty': None, 'logit_bias': None, 'user': None, 'response_format': None, 'seed': None, 'tools': [{'type': 'function', 'function': {'name': 'get_weather', 'description': 'Retrieves the current weather report for a specified city.\n\n    Args:\n        city (str): The name of the city for which to retrieve the weather report.\n\n    Returns:\n        dict: status and result or error msg.\n    ', 'parameters': {'type': 'object', 'properties': {'city': {'type': 'string'}}, 'required': ['city']}}}, {'type': 'function', 'function': {'name': 'get_current_time', 'description': 'Returns the current time in a specified city.\n\n    Args:\n        city (str): The name of the city for which to retrieve the current time.\n\n    Returns:\n        dict: status and result or error msg.\n    ', 'parameters': {'type': 'object', 'properties': {'city': {'type': 'string'}}, 'required': ['city']}}}], 'tool_choice': None, 'max_retries': None, 'logprobs': None, 'top_logprobs': None, 'extra_headers': None, 'api_version': None, 'parallel_tool_calls': None, 'allowed_openai_params': None, 'reasoning_effort': None, 'additional_drop_params': None, 'messages': [{'role': 'system', 'content': 'You are a helpful agent who can answer user questions about the time and weather in a city.\n\nYou are an agent. Your internal name is "weather_time_agent".\n\n The description about you is "Agent to answer questions about the time and weather in a city."'}, {'role': 'user', 'content': 'hhi!'}], 'thinking': None, 'web_search_options': None, 'custom_llm_provider': 'ollama', 'drop_params': None, 'model': 'gpt-oss:20b', 'n': None}
10:05:08 - LiteLLM:DEBUG: utils.py:3344 - 
LiteLLM: Non-Default params passed to completion() {}
10:05:08 - LiteLLM:DEBUG: utils.py:349 - Final returned optional params: {'format': 'json', 'functions_unsupported_model': [{'type': 'function', 'function': {'name': 'get_weather', 'description': 'Retrieves the current weather report for a specified city.\n\n    Args:\n        city (str): The name of the city for which to retrieve the weather report.\n\n    Returns:\n        dict: status and result or error msg.\n    ', 'parameters': {'type': 'object', 'properties': {'city': {'type': 'string'}}, 'required': ['city']}}}, {'type': 'function', 'function': {'name': 'get_current_time', 'description': 'Returns the current time in a specified city.\n\n    Args:\n        city (str): The name of the city for which to retrieve the current time.\n\n    Returns:\n        dict: status and result or error msg.\n    ', 'parameters': {'type': 'object', 'properties': {'city': {'type': 'string'}}, 'required': ['city']}}}]}
10:05:08 - LiteLLM:DEBUG: litellm_logging.py:474 - self.optional_params: {'tools': [{'type': 'function', 'function': {'name': 'get_weather', 'description': 'Retrieves the current weather report for a specified city.\n\n    Args:\n        city (str): The name of the city for which to retrieve the weather report.\n\n    Returns:\n        dict: status and result or error msg.\n    ', 'parameters': {'type': 'object', 'properties': {'city': {'type': 'string'}}, 'required': ['city']}}}, {'type': 'function', 'function': {'name': 'get_current_time', 'description': 'Returns the current time in a specified city.\n\n    Args:\n        city (str): The name of the city for which to retrieve the current time.\n\n    Returns:\n        dict: status and result or error msg.\n    ', 'parameters': {'type': 'object', 'properties': {'city': {'type': 'string'}}, 'required': ['city']}}}]}
10:05:08 - LiteLLM:DEBUG: utils.py:4710 - checking potential_model_names in litellm.model_cost: {'split_model': 'gpt-oss:20b', 'combined_model_name': 'ollama/gpt-oss:20b', 'stripped_model_name': 'gpt-oss:20b', 'combined_stripped_model_name': 'ollama/gpt-oss:20b', 'custom_llm_provider': 'ollama'}
10:05:08 - LiteLLM:DEBUG: litellm_logging.py:929 - 

POST Request Sent from LiteLLM:
curl -X POST \
http://localhost:11434/api/generate \
-d '{'model': 'gpt-oss:20b', 'prompt': '### System:\nYou are a helpful agent who can answer user questions about the time and weather in a city.\n\nYou are an agent. Your internal name is "weather_time_agent".\n\n The description about you is "Agent to answer questions about the time and weather in a city." Produce JSON OUTPUT ONLY! Adhere to this format {"name": "function_name", "arguments":{"argument_name": "argument_value"}} The following functions are available to you:\n{\'type\': \'function\', \'function\': {\'name\': \'get_weather\', \'description\': \'Retrieves the current weather report for a specified city.\\n\\n    Args:\\n        city (str): The name of the city for which to retrieve the weather report.\\n\\n    Returns:\\n        dict: status and result or error msg.\\n    \', \'parameters\': {\'type\': \'object\', \'properties\': {\'city\': {\'type\': \'string\'}}, \'required\': [\'city\']}}}\n\n{\'type\': \'function\', \'function\': {\'name\': \'get_current_time\', \'description\': \'Returns the current time in a specified city.\\n\\n    Args:\\n        city (str): The name of the city for which to retrieve the current time.\\n\\n    Returns:\\n        dict: status and result or error msg.\\n    \', \'parameters\': {\'type\': \'object\', \'properties\': {\'city\': {\'type\': \'string\'}}, \'required\': [\'city\']}}}\n\n\n### User:\nhhi!\n\n', 'options': {}, 'stream': False, 'format': 'json', 'images': []}'


10:05:08 - LiteLLM:DEBUG: http_handler.py:579 - Using AiohttpTransport...
10:05:08 - LiteLLM:DEBUG: http_handler.py:636 - Creating AiohttpTransport...
10:05:08 - LiteLLM:DEBUG: litellm_logging.py:2832 - Filtered callbacks: []
10:05:08 - LiteLLM:DEBUG: cost_calculator.py:669 - selected model name for cost calculation: ollama/gpt-oss:20b
10:05:08 - LiteLLM:DEBUG: utils.py:4710 - checking potential_model_names in litellm.model_cost: {'split_model': 'gpt-oss:20b', 'combined_model_name': 'ollama/gpt-oss:20b', 'stripped_model_name': 'gpt-oss:20b', 'combined_stripped_model_name': 'ollama/gpt-oss:20b', 'custom_llm_provider': 'ollama'}
10:05:08 - LiteLLM:DEBUG: cost_calculator.py:388 - Returned custom cost for model=ollama/gpt-oss:20b - prompt_tokens_cost_usd_dollar: 0, completion_tokens_cost_usd_dollar: 0
10:05:08 - LiteLLM:DEBUG: litellm_logging.py:1231 - response_cost: 0.0
[user]: 

[rfdickerson]

I can't get it to work either. I took code that works with Gemini Flash, but switching to LiteLLM with Ollama causes the subagent referral not to work anymore.

Should I modify the system prompt to not just expose tools but subagents? I have tested with llama3.2 and gemma3.

[marreapato]

Also facing a similar issue here:

)

litellm.exceptions.APIConnectionError: litellm.APIConnectionError: Ollama_chatException - {"error":"json: cannot unmarshal array into Go struct field ChatRequest.messages.content of type string"}