openAI API not supporting Image processing

[AhzamHassan]

I am trying to upload an image to get response from the openAI using its API but the response says, i am unable to process images.
my code:

const completion = await openai.chat.completions.create({
      model: "gpt-4o",
messages: [
        {
          role: "system",
          content: "You are a helpful assistant, i am sharing an image with you please give me the solution for this math problem.",
        },
        {
          role: "user",
          content: JSON.stringify({
            type: "image_url",
            image_url: "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
          }),
        },
      ],
    });

Response (Postman):

{
    "data": {
        "role": "assistant",
        "content": "I'm sorry, but as a text-based AI, I'm unable to view or interpret images. However, if you describe the math problem to me or type it out, I'd be more than happy to assist you in solving it.",
        "refusal": null
    },
    "message": "Success",
    "success": true
}

[erenakbay]

The GPT-4 API doesn’t support image processing directly, as it handles only text inputs. To resolve this, use an OCR tool like Tesseract to extract text from the image, then pass the extracted text to the GPT-4 API for analysis or problem-solving.

[Andyple]

GPT-4o does accept images as input. Try this code and see if this works.

response = client.chat.completions.create(
  model="gpt-4o",
  messages=[
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "You are a helpful assistant, i am sharing an image with you please give me the solution for this math problem."
        }
      ]
    },
    {
      "role": "user",
      "content": [
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/png;base64,..."
          }
        },
        {
          "type": "text",
          "text": "Explain this image"
        }
      ]
    }
  ],
  response_format={
    "type": "text"
  },
)

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This issue was closed because it has been stalled for 10 days with no activity.