[Bug] Issues with Media Uploads for Gemini and Claude

[samkurd]

📦 Environment

Vercel

📌 Version

v1.10.1

💻 Operating System

Windows

🌐 Browser

Edge

🐛 Bug Description

Hello,

I have been experiencing issues with media uploads in the lobe-chat repository. I have deployed the application using Vercel and have set up the database using Neon. Additionally, I am using Cloudflare for DNS handling.

Details:

Deployment Environment:

Platform: Vercel
Database: Neon
DNS Handling: Cloudflare
Issue Description:

Previous Behavior: Before the v1.10.1 update, there were errors with media uploads across all language models.
Current Behavior: After updating to v1.10.1, media uploads work fine with OpenAI. However, I am encountering errors when using the Gemini and Claude language models.
Error Messages:

Gemini:

"Error requesting Google service, please troubleshoot or retry based on the following information"

{ "error": { "message": "currently we don't support image url: https://mydomain.com/files/478784/8e3cec19-9eae-442d-91b4-4791cc58f4bb.jpg" }, "provider": "google" }

Claude:

"Error requesting Anthropic service, please troubleshoot or retry based on the following information"

{ "error": { "error": { "type": "error", "error": { "type": "invalid_request_error", "message": "messages.0.content.1.image.source.base64.media_type: Input should be 'image/jpeg', 'image/png', 'image/gif' or 'image/webp'" } }, "status": 400, "headers": { "via": "1.1 google", "date": "Wed, 14 Aug 2024 07:34:00 GMT", "connection": "keep-alive", "request-id": "req_01LzcGdawx4398ndPL78KsKg", "content-type": "application/json", "x-robots-tag": "none", "content-length": "193", "x-should-retry": "false", "x-cloud-trace-context": "6a3fcc084a2ee937518088998956b28c" } }, "endpoint": "https://api.anthropic.com", "provider": "anthropic" }

This is the error I get on my Browser console:
https://mydomain.com/api/chat/google 471

📷 Recurrence Steps

1. Deploy the lobe-chat application using Vercel.
2. Set up the database using Neon.
3. Configure DNS handling with Cloudflare and R2 Storage.
4. Attempt to upload media using the Gemini and Claude language models.

🚦 Expected Behavior

Media uploads should work seamlessly across all language models, including Gemini and Claude.

📝 Additional Information

No response

[lobehubbot]

👀 @samkurd

Thank you for raising an issue. We will investigate into the matter and get back to you as soon as possible.
Please make sure you have given us as much context as possible.
非常感谢您提交 issue。我们会尽快调查此事，并尽快回复您。 请确保您已经提供了尽可能多的背景信息。

[Zolyn]

#3370 is addressing this issue.

[samkurd]

#3370 seems to address only Gemini, but I'm facing issues with Claude too. Does that fix also solves the issue with Claude?

[adminewacc]

trying to solve this problem too, if i can, i will say here how

[samkurd]

I'm looking forward to your solution, thanks @adminewacc

[adminewacc]

almost solved ( claude lol )

[samkurd]

That's a great news

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

That's a great news

[adminewacc]

solved, i later will try to request it

[adminewacc]

tommorrow i solved it but i forgot to post here lol

[samkurd]

Oh that's amazing, thanks @adminewacc

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Oh that's amazing, thanks @adminewacc

[lobehubbot]

✅ @samkurd

This issue is closed, If you have any questions, you can comment and reply.
此问题已经关闭。如果您有任何问题，可以留言并回复。

[lobehubbot]

🎉 This issue has been resolved in version 1.15.30 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[samkurd]

unfortunately the error is still available and unresolved.

[arvinxx]

@samkurd please share more info about the error you meet.

[samkurd]

This is the error I'm facing:

Error requesting Anthropic service, please troubleshoot or retry based on the following information

{
"error": {
"error": {
"type": "error",
"error": {
"type": "invalid_request_error",
"message": "messages.0.content.1.image.source.base64.data: The image was specified using the image/png media type, but does not appear to be a valid png image"
}
},
"status": 400,
"headers": {
"via": "1.1 google",
"date": "Tue, 10 Sep 2024 14:53:31 GMT",
"connection": "keep-alive",
"request-id": "req_019vWkbWd7H9i6wDvLcRbfLG",
"content-type": "application/json",
"x-robots-tag": "none",
"content-length": "216",
"x-should-retry": "false",
"x-cloud-trace-context": "7b9beefcef835dcdec7c555695c927e1"
},
"request_id": "req_019vWkbWd7H9i6wDvLcRbfLG"
},
"endpoint": "https://api.anthropic.com",
"provider": "anthropic"
}

[arvinxx]

@samkurd what's the image you use?

[samkurd]

so I did a little digging, it only accepts png files and in my tests no other image format is supported.

[samkurd]

all fixed and working with OpenAI, Gemini and Claude.

[ous50]

Is it possible to adapt this method to all of the model providers? I have built an One-API server, from which I may use Claude, Gemini, ChatGPT, etc. by simply configuring the OpenAI's API. When I was using Gemini with OpenAI config, it just returns:

{
"error": {
"code": "image_url_invalid",
"type": "one_hub_error",
"message": "请求上游地址失败 (request id: 202409150736377660162153tPnPuIK)"
},
"endpoint": "https://www/v1",
"provider": "openai"
}

and with google provider it works perfectly: