[Bug]: IBM watsonx.ai provider fails with model_no_support_for_function error despite direct API calls working

[mvrodrig]

What happened?

Hi @krrishdholakia
A bug happened!

Description:

I am encountering an issue when trying to use the IBM watsonx.ai model provider via LiteLLM. When I call the IBM watsonx.ai API directly using curl, the response is correct. However, when I attempt to perform the same request through LiteLLM, I receive an error indicating that the model does not support the required function.

1. Direct IBM watsonx.ai API Call (Working Example)

Request:

curl --request POST \
  --url 'https://us-south.ml.cloud.ibm.com/ml/v1/text/generation?version=2023-05-29' \
  --header 'Accept: application/json' \
  --header 'Authorization: Bearer ey****' \ #IAM Access Token
  --header 'Content-Type: application/json' \
  --data '{
    "model_id": "ibm/granite-13b-chat-v2",
    "input": "hello, how are you?",
    "parameters": {
      "decoding_method": "greedy",
      "max_new_tokens": 100,
      "min_new_tokens": 0,
      "stop_sequences": [],
      "repetition_penalty": 1
    },
    "project_id": "2e078*********"
}'

Response:

{
  "model_id": "ibm/granite-13b-chat-v2",
  "model_version": "2.1.0",
  "created_at": "2024-12-13T14:00:57.066Z",
  "results": [
    {
      "generated_text": "\n\nA: I'm doing well, thank you. And you?\n\nB: I'm doing great, thank you. ...",
      "generated_token_count": 100,
      "input_token_count": 6,
      "stop_reason": "max_tokens"
    }
  ],
  "system": {
    "warnings": [
      {
        "message": "Model 'ibm/granite-13b-chat-v2' is in deprecated state from 2024-11-04. It will be in withdrawn state from 2025-02-03.",
        "id": "deprecation_warning",
        "more_info": "https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-model-lifecycle.html?context=wx&audience=wdp"
      }
    ]
  }
}

2. LiteLLM Call (Failing Example)

Request:

curl --request POST \
  --url http://localhost:4000/v1/chat/completions \
  --header 'Content-Type: application/json' \
  --header 'User-Agent: insomnia/9.3.3' \
  --data '{
    "model": "watsonx/ibm/granite-13b-chat-v2",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant. Answer the user'\''s question."
      },
      {
        "role": "user",
        "content": "hello how are you?"
      }
    ],
    "stream": false
  }'

Response:

{
  "error": {
    "message": "litellm.BadRequestError: watsonxException - {\"errors\":[{\"code\":\"model_no_support_for_function\",\"message\":\"Model 'ibm/granite-13b-chat-v2' does not support function 'function_text_chat'\",\"more_info\":\"https://cloud.ibm.com/apidocs/watsonx-ai\"}],\"trace\":\"1621c4e6cda726cdb5e09fb60ec237c2\",\"status_code\":400}\nReceived Model Group=watsonx/ibm/granite-13b-chat-v2\nAvailable Model Group Fallbacks=None",
    "type": null,
    "param": null,
    "code": "400"
  }
}

Environment Variables Used:

WATSONX_URL=https://us-south.ml.cloud.ibm.com/ml/v1/text/generation
WATSONX_API_KEY=<my_watsonx_api_key>
WATSONX_TOKEN=<my_watsonx_iam_access_token> #expires every hour
WATSONX_PROJECT_ID=<my_watsonx_project_id>

I also tried using https://us-south.ml.cloud.ibm.com/ml as the WATSONX_URL and got the same error.

Configuration in config.yaml:

# IBM WATSONX
- model_name: watsonx/*
  litellm_params:
    model: watsonx/*
    api_key: os.environ/WATSONX_TOKEN

Expected Behavior: The LiteLLM call should return a successful completion response similar to the direct curl request to the IBM watsonx.ai API.

Actual Behavior: LiteLLM returns an error stating that the model does not support function_text_chat, even though direct API calls work correctly.

Additional Context:

• I have tested this with ibm/granite-13b-chat-v2 as well as other Granite models, and the same model_no_support_for_function error occurs.
• The model ibm/granite-13b-chat-v2 is currently in a deprecated state (as per the warning in the direct API call), but direct calls still succeed without issues.
• I have verified that my IAM token and project ID are correct and that the environment variables are properly set before running LiteLLM.

Any guidance or assistance to get the IBM watsonx.ai model integration working through LiteLLM would be greatly appreciated.

Relevant log output

{
  "error": {
    "message": "litellm.BadRequestError: watsonxException - {\"errors\":[{\"code\":\"model_no_support_for_function\",\"message\":\"Model 'ibm/granite-13b-chat-v2' does not support function 'function_text_chat'\",\"more_info\":\"https://cloud.ibm.com/apidocs/watsonx-ai\"}],\"trace\":\"1621c4e6cda726cdb5e09fb60ec237c2\",\"status_code\":400}\nReceived Model Group=watsonx/ibm/granite-13b-chat-v2\nAvailable Model Group Fallbacks=None",
    "type": null,
    "param": null,
    "code": "400"
  }
}

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.54.1

Twitter / LinkedIn details

No response

[krrishdholakia]

LiteLLM calls the IBM Watsonx /text/chat endpoint on the watsonx route. It looks like this model doesn't support that endpoint.

Confirmed this works when calling with watsonx_text/ route which calls the corresponding /text/generation endpoint.

@mvrodrig for future debugging, please try running the proxy with --detailed_debug. it will show you the raw request made by litellm. I was able to use that to identify the diff in url's and confirm the issue.

[krrishdholakia]

To close this ticket, i'll add handling for this error to raise the a helpful message indicating which route to use.