-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Single string output adapter occasionally comes back empty or generic #43
Comments
few more notes on this for posterity: gpt 4 50/50 when renaming the function name in the api calls to "formatter" or "format_response" 4o behaves very well. BUT blueprints begins to behave badly. when renaming the function to something generic like "response" or "function". Blueprints continues to work well. but historical event finder gets even worse. local: ^both work with blueprints |
Hmm if you have the llama models set up, want to see how it plays with the |
sorry took a bit to rebase it had some conflicts for historical event finder / llama3:8b: for historical event finder / llama3.1:8b: |
I could revert back to the xml approach to test this out if we would like to see how this would effect it! |
nah it seems like everyone is converging on this json spec version of tool calling so I think its fine, was more curious to see if that theory of more complex data types could be a fruitful path... When we have the universal json spec formatter, we could change single string into something like the parameter itself and then an additional ignored parameter like "explanation" or "notes" or something...which may also end up increasing the quality of the output anyway.. |
Looks like due to the nature of the way we're using function calling to get structured outputs back, some models occasionally return empty strings for the parameters or just return the description/prompt they were given.
A couple initial thoughts:
The text was updated successfully, but these errors were encountered: