Does the vision version of GPT4 has json content?

[Danya25]

Hello everyone, i just want to clarify one thing.

In OpenAI documentation they talking about possibility to send base64 of image and get answer.

But when i am trying to send json

string jsonString = @$"{{ 
              ""content"": [ 
                {{ ""type"": ""text"", ""text"": ""{prompt}"" }}, 
                {{ 
                  ""type"": ""image_url"", 
                  ""image_url"": {{ 
                    ""url"": f""data:image/jpeg;base64,{base64Image}"" 
                  }} 
                }} 
              ] 
            }}";
           
 var completionResult = await OpenAI.ChatCompletion.CreateCompletion(new ChatCompletionCreateRequest
  {
      Messages = new List<ChatMessage>
      {
          ChatMessage.FromUser(jsonString),
      },
      Model = ModelsGpt.Gpt_4_vision_preview,
      MaxTokens = 300,
  });

I always getting exception about
"Request too large for gpt-4-vision-preview in organization "ID" on tokens per min (TPM): Limit 10000, Requested 72977. The input or output tokens must be reduced in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more."

What i did wrong?

[belaszalontai]

In this issue I explained why it is not working currently. #426

But the API is already extended, the pull request is waiting for review.. Be patient..

[George-Smy]

I'm using 7.4.3 (had been on 7.4.0 for sometime) and I'm having an issue with the image preview when trying to send Base64. The same image works when I use the Url version but I get the same message about TPM limits when I switch to Base64. My only major change is the Json section for passing the image in-line. I've reviewed the discussion under #426 and it sounds like 7.4.3 has the fix. Here's my building code prior to the CreateCompletion call.

`
' For this call there's only one model at the moment so we'll just use it.
Dim sModelToUse As String = Models.Gpt_4_vision_preview

        ' Call LogFile($"{vbTab}In call with {sModelToUse}")
        apiService.SetDefaultModelId(sModelToUse)

        ' And since we're sending the image directly it needs to be Base64 encoded.
        ' Dim bBytes As Byte() = brThumbnail.ReadBytes(infoReader.Length)
        Dim sBase64Image As String = Convert.ToBase64String(File.ReadAllBytes(sImageToSend))

        ' Suggestion noted from API forums 11/8/2023 - GLS. For later research.
        ' Then good parameters for coding are realistically top-p: 0.1 and temperature 0.1. You don’t want the AI to be “creative” in its token generation.

        ' The sample call says that images can't go to 'system' first so 'user' it is.
        ' Dim msgMessage As New ChatMessage("user", strSummarize)
        ' https://platform.openai.com/docs/guides/vision
        Dim sJSONVersion As String = "[ { 'role': 'user', 'content': [ { 'type': 'text', 'text': '" & strPrompt & "' }, { 'type': 'image_url', 'image_url': { 'url': 'data:image/gif;base64," & sBase64Image & "' } } ] } ]"

        ' Call LogFile($"Start of JSON>{sJSONVersion}<")

        ' Note that unlike the other calls we need to tell ChatGPT the maximum number of tokens to return otherwise it tends to return nothing useful.
        sJSONVersion = sJSONVersion.Replace("'", Chr(34))
        Call LogFile($"{vbTab}The json version is {sJSONVersion}")
        Dim msgContent As New ChatMessage("user", sJSONVersion)

        Dim objChatRequest As New ChatCompletionCreateRequest With {
            .Model = sModelToUse,
            .Messages = New List(Of ChatMessage) From {
            msgContent
            },
            .MaxTokens = 4096
        }`

I've double-checked that json as best I can. Am I doing something wrong or is that fix still pending.

Thanks for all of the work on this DLL!