Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Single string output adapter occasionally comes back empty or generic #43

Open
swerner opened this issue Jul 17, 2024 · 5 comments · Fixed by #47
Open

Single string output adapter occasionally comes back empty or generic #43

swerner opened this issue Jul 17, 2024 · 5 comments · Fixed by #47
Labels
bug Something isn't working

Comments

@swerner
Copy link
Contributor

swerner commented Jul 17, 2024

Looks like due to the nature of the way we're using function calling to get structured outputs back, some models occasionally return empty strings for the parameters or just return the description/prompt they were given.

A couple initial thoughts:

@AndrewBKang
Copy link
Contributor

AndrewBKang commented Jul 26, 2024

few more notes on this for posterity:
the tutorial demo of historical event finder has the following issues with the models below:

gpt 4 50/50
gpt 4o 👎

when renaming the function name in the api calls to "formatter" or "format_response" 4o behaves very well. BUT blueprints begins to behave badly.

when renaming the function to something generic like "response" or "function". Blueprints continues to work well. but historical event finder gets even worse.


local:
llama3:8b does not play well with event finder either (this was done through xml)
llama3.1:8b DOES play well with event finder.

^both work with blueprints

@swerner
Copy link
Contributor Author

swerner commented Jul 26, 2024

Hmm if you have the llama models set up, want to see how it plays with the list_of_strings output adapter? I'm curious if the single string output adapter might just be too simple, but if we have a more complex data type it performs better...

@AndrewBKang
Copy link
Contributor

sorry took a bit to rebase it had some conflicts

for historical event finder / llama3:8b:
{"error"=>"llama3 does not support tools"}
bah humbug

for historical event finder / llama3.1:8b:
["First Landing by Vikings in North America", "Independence Day of Chile", "Death of Joseph Stalin"]
(just change historical event finder to 3 list long)

@AndrewBKang
Copy link
Contributor

I could revert back to the xml approach to test this out if we would like to see how this would effect it!

@swerner
Copy link
Contributor Author

swerner commented Jul 26, 2024

nah it seems like everyone is converging on this json spec version of tool calling so I think its fine, was more curious to see if that theory of more complex data types could be a fruitful path...

When we have the universal json spec formatter, we could change single string into something like the parameter itself and then an additional ignored parameter like "explanation" or "notes" or something...which may also end up increasing the quality of the output anyway..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
2 participants