Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(js): split tool response and other parts in vertexai gemini #560

Closed
wants to merge 1 commit into from

Conversation

cabljac
Copy link
Contributor

@cabljac cabljac commented Jul 9, 2024

Fixes #452

Only manually tested so far in a couple of specific cases.

Checklist (if applicable):

  • Tested (manually, unit tested, etc.)
  • Changelog updated
  • Docs updated

@cabljac cabljac requested a review from huangjeff5 July 10, 2024 08:23
@cabljac cabljac marked this pull request as ready for review July 10, 2024 08:23
@cabljac cabljac force-pushed the @invertase/vertexai-gemini-tool-response branch 3 times, most recently from d9318f5 to d3e3c97 Compare July 11, 2024 11:07
@cabljac cabljac force-pushed the @invertase/vertexai-gemini-tool-response branch from d3e3c97 to b8e1e41 Compare July 11, 2024 11:13
@MichaelDoyle
Copy link
Member

MichaelDoyle commented Jul 11, 2024

Discussing on Discord, but I am of the opinion this should be fixed further upstream. That could be in the middleware. But I am not 💯 about this interaction.

  1. The issue applies to both Google AI and Vertex AI
  2. [Prior to this fix] the context appears to "move" from the original user message down into the toolResponse during the tool calling loop. This is definitely not what I would expect. I would expect the original history to be maintained, and a toolResponse simply tacked on. IOW, there should be no need to "move it back".

Reproduced in a trace for Google AI:
https://gist.github.com/MichaelDoyle/a5e4336d92e4353b2cd5237f97200690

@cabljac
Copy link
Contributor Author

cabljac commented Jul 12, 2024

closing in favour of #591

@cabljac cabljac closed this Jul 12, 2024
@cabljac cabljac deleted the @invertase/vertexai-gemini-tool-response branch July 16, 2024 16:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[JS] VertexAI.ClientError when providing context from retriever and Gemini calling a provided tool
3 participants