Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix/multiline few shot colang 2 #765

Merged

Conversation

radinshayanfar
Copy link
Contributor

@radinshayanfar radinshayanfar commented Sep 19, 2024

This PR fixes the issue of multiline bot say parameter, which could be a follow-up problem of the #748. Consider the following application:

import core
import llm

flow main
  activate llm continuation

  bot say "Hello! What movie preferences or needs do you have?"

  user said wants to know about kid friendly movies

  bot say "1. Toy Story (1995)\n2. Finding Nemo (2003)\n3. The Lion King (1994)\n4. Moana (2016)\n5. Frozen (2013)\n6. Zootopia (2016)\n7. Despicable Me (2010)\n8. The Incredibles (2004)\n9. 
Shrek (2001)\n10. Paddington (2014)"

  user said wants to know age suitability

flow user said wants to know about kid friendly movies
  user said "i want to learn about kid friendly movies"

flow user said wants to know age suitability
  user said "for what age are these movies suitable?"

After the kid-friendly movies are printed, if the user intent is undefined, the call to LLM's prompt would include this:

User                                                                                                                                                                                                        
user action: user said "recommend some movies for kids"                                                                                                                                                     
Bot                                                                                                                                                                                                         
user intent: user said wants to know about kid friendly movies                                                                                                                                              
bot action: bot say "Here are some movies for kids:"                                                                                                                                                        
bot action: bot say "1. Toy Story (1995)                                                                                                                                                                    
2. Finding Nemo (2003)                                                                                                                                                                                      
3. The Lion King (1994)                                                                                                                                                                                     
4. Moana (2016)                                                                                                                                                                                             
5. Frozen (2013)                                                                                                                                                                                            
6. Zootopia (2016)                                                                                                                                                                                          
7. Despicable Me (2010)                                                                                                                                                                                     
8. The Incredibles (2004)                                                                                                                                                                                   
9. Shrek (2001)                                                                                                                                                                                             
10. Paddington (2014)"

which is misleading for few-shot learning. This leads to a generation like:

LLM Completion
bot action: bot say "Sure! Here are some kid-friendly movies released after 2010:"                                                                                                                          
bot action: bot say "1. Moana (2016)                                                                                                                                                                        
2. Zootopia (2016)                                                                                                                                                                                          
3. Finding Dory (2016)                                                                                                                                                                                      
4. The Lego Movie (2014)                                                                                                                                                                                    
5. Inside Out (2015)                                                                                                                                                                                        
6. Coco (2017)                                                                                                                                                                                              
7. Paddington 2 (2017)                                                                                                                                                                                      
8. Ralph Breaks the Internet (2018)                                                                                                                                                                         
9. Toy Story 4 (2019)                                                                                                                                                                                       
10. Onward (2020)"

First of all, the first line which is considered a flow name is syntactically incorrect. This PR escapes non-word chars in generated flow names. Secondly, the correct generation should be like:

LLM Completion
bot intent: bot provide a list of kids' movies released after 2010                                                                                                                                          
bot action: bot say "Sure! Here are some kid-friendly movies released after 2010:\n1. Frozen (2013)\n2. Moana (2016)\n3. Zootopia (2016)\n4. Finding Dory (2016)\n5. The Lego Movie (2014)\n6. Inside Out 
(2015)\n7. Coco (2017)\n8. Ralph Breaks the Internet (2018)\n9. Toy Story 4 (2019)\n10. Onward (2020)"                                                                                                      

This PR converts new lines in params to \n tokens to fix this problem.

@radinshayanfar
Copy link
Contributor Author

Hi @Pouyanpi, @drazvan, @schuellc-nvidia.
Can you take a look and merge this if it's okay? I think this problem can occur frequently in a real-world application built with Colang, and this PR will fix it.

Copy link
Collaborator

@schuellc-nvidia schuellc-nvidia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you a lot for your fix. Only some small improvements needed.

nemoguardrails/actions/llm/utils.py Outdated Show resolved Hide resolved
nemoguardrails/actions/llm/utils.py Outdated Show resolved Hide resolved
@radinshayanfar
Copy link
Contributor Author

Hi @schuellc-nvidia,
Thank you for the comments. They are resolved now.

@schuellc-nvidia
Copy link
Collaborator

Great! Thanks!

@schuellc-nvidia schuellc-nvidia merged commit b25eaec into NVIDIA:develop Nov 4, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants