initial checkin for vision support with tests #2240
+138
−27
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cpacker
reviewed
Dec 12, 2024
letta/client/client.py
Outdated
| @@ -152,10 +152,11 @@ def send_message( | |||
| stream: Optional[bool] = False, | |||
| stream_steps: bool = False, | |||
| stream_tokens: bool = False, | |||
| image: Optional[str] = None, | |||
There was a problem hiding this comment.
I think the ideal way to do this is we actually make message (currently str) be type MultiMediaContent, which itself is Union[str, List[MultiMediaContentPart]]
Then we will be able to support multiple images and future modalities like audio.
There was a problem hiding this comment.
sounds good, updated. see message in your other comment
cpacker
reviewed
Dec 12, 2024
letta/schemas/message.py
Outdated
| @@ -421,8 +423,22 @@ def to_openai_dict( | |||
|
|
|||
| elif self.role == "user": | |||
| assert all([v is not None for v in [self.text, self.role]]), vars(self) | |||
|
|
|||
| if self.image is not None: | |||
There was a problem hiding this comment.
So inside of Message, .text becomes .content, which basically is a 1:1 mapping to OpenAI's format:
There was a problem hiding this comment.
agree with this and made the initial typing changes/enough to pass tests. However left off before two major changes:
- Renaming Message.text -> Message.content which is used in a lot of places
- Updating the ORM Message object which is typed str and requires migration. Tricky because images probably shouldn't be stored raw in the DB
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please describe the purpose of this pull request.
Add vision support when using OpenAI models
Remaining work before merge:
Extensions:
How to test
How can we test your PR during review? What commands should we run? What outcomes should we expect?
Unit tests included
Have you tested this PR?
Have you tested the latest commit on the PR? If so please provide outputs from your tests.
Unit tests pass
Related issues or PRs
Please link any related GitHub issues or PRs.
Is your PR over 500 lines of code?
If so, please break up your PR into multiple smaller PRs so that we can review them quickly, or provide justification for its length.
Additional context
Add any other context or screenshots about the PR here.