Add "gpt-4-vision-preview" as default parser #1417
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Mar 7, 2024
rholinshead
requested review from
saqadri,
suyoglastmileai,
Ankush-lastmile,
jonathanlastmileai and
rossdanlm
as code owners
# Update openai to 1.13.3 Summary: Updating the openai version to latest (1.13.3) so that we can use the image messages for gpt-4-v in the next PR Test Plan: - pytest works without errors - WIP: Testing all relevant cookbooks
rossdanlm
reviewed
Mar 8, 2024
Comment on lines
+46
to
+48
| ModelParserRegistry.register_model_parser( | ||
| OpenAIVisionParser("gpt-4-vision-preview") | ||
| ) |
There was a problem hiding this comment.
cc @tanya-rai this would be the 3rd position from drop-down. If you want to change it, we can easily change in a future PR
rossdanlm
approved these changes
Mar 8, 2024
added 2 commits
March 8, 2024 13:34
# Implement OpenAIVisionParser Summary: Using the default openai parser as the basis (and extending it), implement a parser for text-and-image openai models (e.g. gpt-4-v). The key differences are: - vision models don't support function calling or tools, so exclude from the completion params - serialize/deserialize need to be a bit different to construct the user messages' content with text and image urls One thing to note is the default max tokens for GPT-4V is quite small. Bumping it to 100 in #1419 Another thing I found while testing is that the model gets confused sometimes when you refer to the attached image as an image. For example, I had a prompt as "What is this image of" and it responded with "I can't provide information on any images until you upload one". Then I reran a handful of times and it finally gave expected result. Changing the prompt to "What is this?" seems to provide much better results. Test Plan: https://github.com/lastmile-ai/aiconfig/assets/5060851/47e0ae90-fab7-4d4f-98e9-163cd992aeed
# Add "gpt-4-vision-preview" as default parser Summary: Just adding this model/parser as a default alongside the other openai defaults Test Plan: Can use the parser in the editor
rholinshead
added a commit
that referenced
this pull request
Mar 8, 2024
Default GPT-4V max_tokens to 100 # Default GPT-4V max_tokens to 100 Summary: The default max_tokens is really low (like < 20) so bump to 100 to make it more useable Test Plan:  --- [//]: # (BEGIN SAPLING FOOTER) Stack created with [Sapling](https://sapling-scm.com). Best reviewed with [ReviewStack](https://reviewstack.dev/lastmile-ai/aiconfig/pull/1419). * __->__ #1419 * #1418 * #1417 * #1416 * #1415
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add "gpt-4-vision-preview" as default parser
Add "gpt-4-vision-preview" as default parser
Summary: Just adding this model/parser as a default alongside the other openai defaults
Test Plan: Can use the parser in the editor
Stack created with Sapling. Best reviewed with ReviewStack.