-
Notifications
You must be signed in to change notification settings - Fork 65
-
Notifications
You must be signed in to change notification settings - Fork 65
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Verify new tools api code with GoogleAI #107
Comments
@brainlid sure i should be able to run |
The big change is around function calls and function results.
An assistant message can contain 0 to many ToolCalls. A Tool message
contains 1 or more ToolResults.
…On Sat, Apr 27, 2024 at 12:32 AM John Dengis ***@***.***> wrote:
@brainlid <https://github.com/brainlid> sure i should be able to run main
against my setup and see if it breaks anything. Are there any breaking
changes to the API i'd need to integrate?
—
Reply to this email directly, view it on GitHub
<#107 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAGFQGAKV7OYQCSYAYZFTNLY7NBBVAVCNFSM6AAAAABG3WO4RSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBQGM4DMOBZGI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Looking forward to this as Gemini Pro APIs have become publicly available. Here are the Function Calling docs for Gemini models: https://ai.google.dev/gemini-api/docs/function-calling |
@nileshtrivedi if you are interested I would appreciate some help testing out the above change. I tried it out in my project but ran into some errors with the new code and I couldn't get it working. I haven't had time to go back an fix the issues however. |
@jadengis I thought of modifying the existing
I don't know whether it's a bug in the test code itself or something else. Unable to test further. 🫤 For my own work, even Autogen itself currently seems broken for Gemini models. I ended up using Python example code using Google's own SDK. |
@nileshtrivedi I updated and fixed the Calculator tool and tests. Thanks for pointing that out! |
@jadengis please let me know what errors you're getting! We can hopefully get it ironed out quickly. |
For migrating, I tried to document what would be needed in the CHANGELOG. Let me know if you're finding gaps! |
I submitted #135 as a failing test. While |
I think there are multiple errors in calling Gemini model APIs properly:
There may be more issues. EDITED: Noticed this open PR for fixing the endpoint: #118 Seems like there are subtle difference between Gemini API and VertexAI API which are causing these. |
@nileshtrivedi We split out ChatVertexAI from ChatGoogleAI because the differences were subtle but throughout. In the published RC, the callbacks have been updated as well. Thank you for looking at it before. How does it look now? |
Gemini API still seems to fail when testing with I tested after making this change in
This is the test failure I get:
I confirmed that my actual api_key was printed where it says I think it might be easier if you can signup on https://ai.google.dev/ to get an API key to help with testing? |
Hello @nileshtrivedi, If you change these lines: langchain/lib/chat_models/chat_google_ai.ex Lines 27 to 29 in d63e11a
by @default_endpoint "https://generativelanguage.googleapis.com"
@default_api_version "v1beta" Do the tests pass? |
@ljgago No, it fails but with a different error:
I am happy to get on a call with any devs to work this out. |
@nileshtrivedi I setup a GoogleAI account and got an API key. There are multiple issues with the ChatGoogleAI implementation. I've fixed a couple locally (not pushed yet), but there's an issue with the ToolResult handling that I'm still trying to figure out. A big issue is that Google's API docs are really poor. Their API also does things I haven't seen any other API do (aka. odd behaviors). All in all, I don't like Google's service! 😬 Still, I acknowledge your issue, it is valid, and I hope to have a resolution sometime soon. Thanks! |
@nileshtrivedi This is hopefully fixed now! 🤞 Just merged PR #152 to |
Thanks @brainlid ! This definitely seems to have improved as now ChatGoogleAI responses are being collected. To the user's message However, there seem to be differences between ChatOpenAI and ChatGoogleAI return values.
With ChatGoogleAI, instead of Also, Gemini's response is simply |
I think it may require setting the function calling mode to |
My bad, I didn't notice the new tests added (including the one for tool calling). Kudos @brainlid for following up and fixing this! 👏 |
@nileshtrivedi I'm pretty annoyed by the GoogleAI honestly. It's such an oddball compared to others. The API docs are sparse and difficult to use too. Ugh. One odd thing that you noticed is the assistant returns the contents in parts every time. That's just a decision they made. We could pattern match on that and if there is only a single text part, flatten it to be I haven't used it enough to see it return anything else though. Have you? If it makes sense, then doing that would make it easier to swap out the backend AI without impacting an application. I updated the |
The PR #105 was merged to
main
, and the tests pass, but I didn't run it against a GoogleAI LLM. I would appreciate help validating that.@jadengis, this impacts your code the most. Just a heads-up.
The text was updated successfully, but these errors were encountered: