[Bug] Claude图片上传后提问报错

[yuwuxing]

📦 部署环境

Docker

📌 软件版本

v1.15.26

💻 系统环境

Ubuntu

🌐 浏览器

Chrome

🐛 问题描述

使用Claude sonnet 3.5模型和google Gemini 模型提问，上传一张图片，显示上传成功，但是返回报错信息：

Claude 报错

{
  "error": {
    "error": {
      "type": "error",
      "error": {
        "type": "invalid_request_error",
        "message": "messages.0.content.1.image.source.base64.media_type: Input should be 'image/jpeg', 'image/png', 'image/gif' or 'image/webp'"
      }
    },
    "status": 400,
    "headers": {
      "via": "1.1 google",
      "date": "Mon, 09 Sep 2024 09:37:20 GMT",
      "cf-ray": "8c0632f70cee0906-LAX",
      "server": "cloudflare",
      "connection": "keep-alive",
      "request-id": "req_01BPu9sdKmLUNny7x43R1joy",
      "content-type": "application/json",
      "x-robots-tag": "none",
      "content-length": "193",
      "x-should-retry": "false",
      "cf-cache-status": "DYNAMIC",
      "x-cloud-trace-context": "4f50bc854b960317dd3fffd90bf36fb6"
    },
    "request_id": "req_01BPu9sdKmLUNny7x43R1joy"
  },
  "endpoint": "https://api.anthropic.com",
  "provider": "anthropic"
}

Gemini Flash 1.5 报错

{
"error": {
"message": "currently we don't support image url: https://s3-lobechat.xxxxx.com/lobechat/files/479410/99270fd2-1643-41eb-93ca-684d1d6934ed.jpg"
},
"provider": "google"
}

控制台查询图片路径正常上传，图片可以访问
换了本地部署的minio和R2都是同样报错

📷 复现步骤

1. 上传一张图片
2. 选择Claude Sonnet 3.5或者Google Gemini模型
3. 提问
4. 返回报错信息
   
   

🚦 期望结果

能够正常对话

📝 补充信息

GPT相关模型可以回答成功

[lobehubbot]

👀 @yuwuxing

Thank you for raising an issue. We will investigate into the matter and get back to you as soon as possible.
Please make sure you have given us as much context as possible.
非常感谢您提交 issue。我们会尽快调查此事，并尽快回复您。 请确保您已经提供了尽可能多的背景信息。

[dosubot]

Hello @yuwuxing! I'm here to help you with bug fixes, answer questions, and assist with contributions. Let's tackle this issue together!

I found a similar unsolved discussion regarding the Gemini model not receiving images ^[1].

To resolve the error messages when uploading an image and using Claude Sonnet 3.5 or Google Gemini models in a Docker-deployed environment on Ubuntu with Chrome, ensure that the necessary environment variables for these models are correctly set in your Dockerfile. Here is the relevant section of the Dockerfile where you can set these variables:

# Model Variables
ENV \
    # Google
    GOOGLE_API_KEY="" GOOGLE_PROXY_URL="" \
    # Anthropic
    ANTHROPIC_API_KEY="" ANTHROPIC_PROXY_URL=""

Replace the empty strings with your actual API keys and proxy URLs if needed. For example:

# Model Variables
ENV \
    # Google
    GOOGLE_API_KEY="your_google_api_key" GOOGLE_PROXY_URL="your_google_proxy_url" \
    # Anthropic
    ANTHROPIC_API_KEY="your_anthropic_api_key" ANTHROPIC_PROXY_URL="your_anthropic_proxy_url"

Additionally, ensure that the sharp library, which is used for image processing, is correctly installed and included in your build process as shown in the Dockerfile:

# Add sharp dependencies
&& mkdir -p /sharp \
&& pnpm add sharp --prefix /sharp

For the specific error messages you encountered:

1. Claude Sonnet 3.5 Error:

   • Ensure the image is in one of the supported formats: image/jpeg, image/png, image/gif, or image/webp.

2. Google Gemini Flash 1.5 Error:

   • The error indicates that the model currently does not support image URLs. This might be a limitation of the model itself.

If you have set these variables correctly and still encounter issues, check the logs for more specific error messages that could provide further insights into what might be going wrong.

It's also important to note that based on recent evaluations, the Google Gemini models, particularly Gemini 1.5 Pro and Gemini 1.5 Flash, have shown poor performance in tool calling capabilities. For instance, Gemini 1.5 Pro has issues with recognizing plugin names correctly and throws errors for complex instructions. Gemini 1.5 Flash does not output tool calling data at all, making it essentially unusable for such tasks ^[2][3].

To continue talking to Dosu, mention @dosu.

[arvinxx]

Google Gemini 在 #3370 修复

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Fixed by Google Gemini in #3370

[arvinxx]

已在 #3870 修复

[lobehubbot]

✅ @yuwuxing

This issue is closed, If you have any questions, you can comment and reply.
此问题已经关闭。如果您有任何问题，可以留言并回复。

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Fixed in #3870