[Bug]: Giving Gemini an image whose URL does not end with an extension causes a 500 error

[wwwcojp]

What happened?

Hi there!

I sent a request to Gemini on Litellm proxy with the image in URL format instead of BASE64 format.

In that case, if the image URL ended with an extension, it succeeded without any problem.
However, when the image URL did not end with an extension, a 500 error occurred.

Request Format

{
    "model": "gemini-1.5-flash",
    "messages": [
        {
        "role": "user",
        "content": [{"type": "text", "text": "describe this image"},
                    {"type": "image_url", "image_url": {"url": "https://images.pexels.com/photos/1319515/pexels-photo-1319515.jpeg?auto=compress&cs=tinysrgb&w=1260&h=750&dpr=1" }}
                ]
            }
    ],
}

Error Message

{"error":{"message":"litellm.InternalServerError: VertexAIException InternalServerError - Image url not in expected format. Example Expected input - \"image_url\": \"data:image/jpeg;base64,{base64_image}\". Supported formats - ['image/jpeg', 'image/png', 'image/gif', 'image/webp'] ","type":null,"param":null,"code":500}

From the error message, I think the cause is that the following process determines the file type at the end of the URL.

https://github.com/BerriAI/litellm/blob/main/litellm/llms/prompt_templates/factory.py#L666-L678

There are many cases where the URL of an image does not end with an extension, such as a signed URL in S3 or a SAS URL in Azure BLOB Storage.
Therefore, we would appreciate it if you could add a process to properly determine the file type in order to combine this with a mechanism to securely publish sensitive data.

Relevant log output

No response

Twitter / LinkedIn details

No response

[krrishdholakia]

looks like we can get image type from response headers

[wwwcojp]

Big thanks for the quick turnaround on that image URL issue. Awesome work!

[ishaan-jaff]

@wwwcojp any chance we can hop on a call ? I'd love to learn how how we can improve litellm for you.

Sharing a link to my cal for your convenience: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
My linkedin if you prefer DM: https://www.linkedin.com/in/reffajnaahsi/