feat: Add anthropic image description #1436
Conversation
| @@ -97,11 +98,13 @@ export class ImageDescriptionService | |||
|
|
|||
| if (model === models[ModelProviderName.LLAMALOCAL]) { | |||
| await this.initializeLocalModel(); | |||
| } else if (model === models[ModelProviderName.ANTHROPIC]) { | |||
| this.modelId = "claude-3-haiku-20240307"; | |||
There was a problem hiding this comment.
Can we make this a env var and add this hard coded value as a default?
| imageData, | ||
| 400, | ||
| 400 |
There was a problem hiding this comment.
You should be able to configure the dimensions of this can we pass that in as a param
| // Detect MIME type | ||
| const fileTypeResult = await FileType.fileTypeFromBuffer(imageBuffer); | ||
| if (!fileTypeResult || !fileTypeResult.mime.startsWith("image/")) { | ||
| throw new Error("Invalid image format"); |
There was a problem hiding this comment.
please elizaLogger before errors thanks
| }, | ||
| }, | ||
| }; | ||
| } catch (error) { |
odilitime
changed the title
| if (model === models[ModelProviderName.LLAMALOCAL]) { | ||
| await this.initializeLocalModel(); | ||
| } else if (model === models[ModelProviderName.ANTHROPIC]) { | ||
| this.modelId = "claude-3-haiku-20240307"; | ||
| this.device = "cloud"; | ||
| } else { | ||
| this.modelId = "gpt-4o-mini"; | ||
| this.device = "cloud"; |
There was a problem hiding this comment.
The model selection logic lacks a default case for unrecognized models. This could lead to unexpected behavior if a new model is added but not handled. Consider adding a default case that logs an error or throws an exception to ensure that all models are accounted for.
| const prompt = | ||
| "Describe this image and give it a title. The first line should be the title, and then a line break, then a detailed description of the image. Respond with the format 'title\ndescription'"; | ||
| const text = await this.requestOpenAI( | ||
| imageUrl, | ||
| imageData, | ||
| prompt, | ||
| isGif | ||
| ); | ||
| const text = | ||
| this.runtime.imageModelProvider === ModelProviderName.ANTHROPIC | ||
| ? await this.requestAnthropic(imageData, prompt) | ||
| : await this.requestOpenAI( | ||
| imageUrl, | ||
| imageData, | ||
| prompt, | ||
| isGif | ||
| ); |
There was a problem hiding this comment.
The conditional logic for selecting the request method is not clear. If this.runtime.imageModelProvider is not set correctly, it could lead to unexpected behavior. Ensure that this property is validated before use, or provide a fallback mechanism.
| private async requestAnthropic( | ||
| imageData: Buffer, | ||
| prompt: string | ||
| ): Promise<string> { | ||
| for (let attempt = 0; attempt < 3; attempt++) { | ||
| try { | ||
| const endpoint = | ||
| models[this.runtime.imageModelProvider].endpoint ?? | ||
| "https://api.anthropic.com/v1"; | ||
|
|
||
| // Resize image to 400x400 max, keeping the token count ~ 213 | ||
| const resizedImage = await resizeImageBuffer( | ||
| imageData, | ||
| 400, | ||
| 400 | ||
| ); | ||
|
|
||
| const response = await fetch(endpoint + "/messages", { | ||
| method: "POST", | ||
| headers: { | ||
| "Content-Type": "application/json", | ||
| "x-api-key": `${this.runtime.getSetting("ANTHROPIC_API_KEY")}`, | ||
| "anthropic-version": "2023-06-01", | ||
| }, | ||
| body: JSON.stringify({ | ||
| model: this.modelId, | ||
| max_tokens: 300, | ||
| messages: [ | ||
| { | ||
| role: "user", | ||
| content: [ | ||
| { | ||
| type: "image", | ||
| source: { | ||
| type: "base64", | ||
| media_type: resizedImage.mimeType, | ||
| data: resizedImage.buffer.toString( | ||
| "base64" | ||
| ), | ||
| }, | ||
| }, | ||
| { | ||
| type: "text", | ||
| text: prompt, | ||
| }, | ||
| ], | ||
| }, | ||
| ], | ||
| }), | ||
| }); | ||
|
|
||
| if (!response.ok) { | ||
| throw new Error( | ||
| `HTTP error! status: ${await response.text()}` | ||
| ); | ||
| } | ||
|
|
||
| const data = await response.json(); | ||
| return data.content[0].text; | ||
| } catch (error) { | ||
| elizaLogger.error( | ||
| `Anthropic request failed (attempt ${attempt + 1}):`, | ||
| error | ||
| ); | ||
| if (attempt === 2) throw error; | ||
| } | ||
| } | ||
| throw new Error( | ||
| "Failed to recognize image with Anthropic after 3 attempts" |
There was a problem hiding this comment.
The requestAnthropic method has a retry mechanism, but it does not handle specific error types. Consider implementing more granular error handling to differentiate between transient errors (which should be retried) and permanent errors (which should not). This will improve the robustness of the error handling.
| if (!response.ok) { | ||
| throw new Error( | ||
| `HTTP error! status: ${await response.text()}` | ||
| ); | ||
| } | ||
|
|
||
| const data = await response.json(); | ||
| return data.content[0].text; | ||
| } catch (error) { | ||
| elizaLogger.error( | ||
| `Anthropic request failed (attempt ${attempt + 1}):`, | ||
| error | ||
| ); | ||
| if (attempt === 2) throw error; |
There was a problem hiding this comment.
Logging the error in the retry mechanism is good, but consider adding more context to the log message, such as the endpoint being called or the parameters used. This will help in debugging issues when they arise.
| // Detect MIME type | ||
| try { | ||
| // Detect MIME type | ||
| const fileTypeResult = await FileType.fileTypeFromBuffer(imageBuffer); |
There was a problem hiding this comment.
The method 'fileTypeFromBuffer' is not guaranteed to return a valid result. Consider adding a check for 'fileTypeResult.ext' to ensure the file type is supported before proceeding.
| const metadata = await sharp(imageBuffer).metadata(); | ||
| if (!metadata.width || !metadata.height) { | ||
| throw new Error("Could not get image dimensions"); |
There was a problem hiding this comment.
The error handling for metadata retrieval is too generic. Consider logging the error or providing more context about the failure to aid in debugging.
| const resizedBuffer = await sharp(imageBuffer) | ||
| .resize(width, height, { | ||
| fit: "inside", | ||
| withoutEnlargement: true, | ||
| }) | ||
| .toBuffer(); |
There was a problem hiding this comment.
The image processing could lead to performance issues if the input image is large. Consider implementing a size limit check before processing to avoid excessive memory usage.
|
Vercel aisdk that we use in generation.ts, support vision models. Why don't we use aigeneratetext instead of manually implementing fetch for description services? I think it can keep integrity overall repo. I created an example how it can be used for image descriptions. https://github.com/denizekiz/OllamaVisionExample/blob/main/src/index.ts |
Relates to:
Risks
Background
What does this PR do?
What kind of change is this?
Documentation changes needed?
Testing
Where should a reviewer start?
Detailed testing steps