Completion vs. Chat - clarification request

[aitamar]

Hey,
I wanted to start off by saying a huge thanks for the awesome work your team has done, it is really an amazing project.

I've been tinkering around with Nemo and I have a question I'm hoping you can help me with,

During my usage of Nemo with chat models (such as GPT-3.5/4), I have encountered an issue where the process deviates from the intended flow during bot message generation. Although user intent is being correctly identified, the bot message generated employs the language model (LLM) to create a more general bot response instead of adhering to the predefined conversational flow.

When using Nemo with a completion model (specifically davinci-003), I've observed that it faithfully follows the conversation flows. however, it appears to lack the creative element and capabilities of a language model when generating bot messages.

Could you please provide clarification on the distinctions between running Nemo in a chat model vs. a completion model? Additionally, if its not too much trouble, I would appreciate guidance on the necessary adjustments when utilizing each of these models. (reading the documentation, I couldn't figure out how to make both work properly)

Getting some insights on this would be a game-changer for me.
Thanks again for the hard work,
Ari

[baravit]

+1.
I'll add that it seems like even simple things including "context" messages / register_prompt_context action, are not working when running Nemo with main chat model (gpt-3.5/4)..

[drazvan]

Hi @aitamar and @baravit!

In theory, the same guardrails configuration should work with both completion and chat models. The completion mode works better (e.g. text-davinci-003) because that's what we started with. We still need to experiment some more with chat models, to figure out the best way to prompt them. We're also actively testing a way of prompting them with a single call, rather than making three calls. So, in the next month or so, we should see some improvements.

That being said. Can you share some quick examples here? I'd like to debug this a bit with you if you have time. Just the config + the dialog is enough. And just in case you're not doing this already, you should use nemoguardrails chat --config=path/to/config --verbose to debug.

Thanks!

[baravit]

Hi @drazvan!
Thank you for your quick reply:)
We would be honored to have a debug session with you!

In the meantime, here is an example of Nemo breaking out of a flow after generating user intent:

from nemoguardrails import LLMRails, RailsConfig
from dotenv import load_dotenv


load_dotenv()

YAML_CONFIG = """
models:
  - type: main
    engine: openai
    model: gpt-3.5-turbo
"""
COLANG_CONFIG = """
define user ask question
  "What is the meaning of life?"

define flow
  user express greeting
  bot express greeting

  
define flow
  priority 1000
  user ask question
  bot reply "Yo Yo Yo"

"""

def main():
    config = RailsConfig.from_content(COLANG_CONFIG, YAML_CONFIG)
    APP = LLMRails(config, verbose=True)

    history = [
        {"role": "user", "content": "What is the meaning of life?"}
    ]
    result = APP.generate(messages=history)
    print(result)

if __name__ == "__main__":
    main()

Just run it with gpt-3.5-turbo and then switch to text-davinci-003..

Regarding the context message and register_prompt_context, I've tested these in an isolated environment and they seem to work just fine. I need to conduct further testing on the issue within our infrastructure. I'll keep you posted if anything comes up.

We are familiar with the debugging options and use them quite often, but our backend includes some other core-logic components between Nemo and the client. Therefore, in some cases, we simply can't employ that approach:)

Thank you again for your help!
Bar.

[alvaroNicasource]

+1, I'm having the same issue.
model: gpt-3.5-turbo

[baravit]

Hi @drazvan, hope you are doing well.
Following up on this, wanted to ask if there is anything new regarding our issue? Were you able to reproduce it?
From our latest research, it seems like the problem occurs when sending nemo messages that it doesn't have the cached event for (i.e server restart / send last n messages), as pointed out in our other discussion here: #63

Also in that discussion, you told us about the state feature that you guys are working on..
Wanted to see if there are any updates regarding it.

Thank you again for all your work, let us know if there is anything you need from us or if we can help in any way.

Bar.

[austinmw]

Hi, does setting model: gpt-4 work? I set this, but when I ask the LLM what model it is, it replies, "I am an AI language model developed by OpenAI. More specifically, I'm powered by gpt-3.5-turbo."

[sinderwing]

I get 2x "Parameter temperature does not exist for OpenAIChat" when trying to use chat models like gpt-4.

openai==0.28.1
nemoguardrails==0.5.0

config.yaml

models:
- type: main
  engine: openai
  model: gpt-4

# Load a guardrails configuration from the specified path.
config = RailsConfig.from_path("config")
rails = LLMRails(config)

def ask(prompt):
    completion = rails.generate(
        messages=[
            {"role": "context", "content": {"name": user}},
            {"role": "user", "content": prompt}
            ]
    )
    response = completion['content'].replace("__User__", user)

    return f"User:\n{prompt}\n----------\nSystem:\n{response}"

subject = "the British prime minister"
prompt = f"Give me one common critique regarding {subject}"

print(ask(prompt))

Returns

Parameter temperature does not exist for OpenAIChat
Parameter temperature does not exist for OpenAIChat
User:
Give me one common critique regarding the British prime minister
----------
System:
One common critique ...

Trying to use guardrails to prevent it from discussing politics as it's a common default objective.

Solved by bypassing the NeMo yaml by adding

from langchain.chat_models import ChatOpenAI
rails = LLMRails(config, llm=ChatOpenAI(model='gpt-4'))

[drazvan]

@Johnnil: can you check again with 0.6.0 which was released yesterday? I've just tested and it seems the issue is no longer there.