-
Notifications
You must be signed in to change notification settings - Fork 101
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[JS] VertexAI.ClientError when providing context from retriever and Gemini calling a provided tool #452
Comments
I am hitting the same issue. |
It appears the issue is form Gemini. Here is the place in the gemini code where the error message comes from and seems like it is not supported at this point: |
Hi, just to clarify, is this using the Vertex AI plugin? The package you linked is from Google AI, but the title of the issue mentions Vertex. Could you provide your configureGenkit options? (redacting anything sensitive of course) |
But the model in my dotprompt file refers to gemini-1.5-flash-preview (and I've tried with gemini-1.0-pro as well).
|
Just noting down what i've found so far: Seems like the issue is in this method interacting with the Vertex models, in that a function response part can't also have parts of other type in it. Note that all the Vertex AI plugin models do not contain |
OK so appears to be exactly what it's logging actually, a FunctionResponse is being returned to Gemini together with other types and it doesn't like that. My theory is that the ToolResponse contains the context still, and this will be added as text parts to the gemini request, causing it to complain. In fact this line https://github.com/firebase/genkit/blob/main/js/plugins/vertexai/src/gemini.ts#L515 is likely the issue, as we send all ToolResponses and other type parts back together. |
I believe this is now fixed, as of #591 |
This is working as expected for us now. Thanks! |
Describe the bug
When calling generate with an array of available tools and providing the context parameter the results of a Firestore Retriever call, the call errors out with the following error:
To Reproduce
Here is some simplified code that shows the setup that produces the error for me every time. When I comment out passing in the context, it works as expected.
The text was updated successfully, but these errors were encountered: