GraphTOD - Reecall

Training task-oriented dialogue systems is both costly and time-consuming, due to the need for high-quality datasets encompassing diverse intents. Traditional methods depend on extensive human annotation, while recent advancements leverage large language models (LLMs) to generate synthetic data. However, these approaches often require custom prompts or code, limiting accessibility for non-technical users. We introduce GraphTOD, an end-to-end framework that simplifies the generation of task-oriented dialogues. Users can create dialogues by specifying transition graphs in JSON format. Our evaluation demonstrates that GraphTOD generates high-quality dialogues across various domains, significantly lowering the cost and complexity of dataset creation.

Demo at : https://graphtod.reecall.com/

Video at : https://www.youtube.com/watch?v=5pXa4yGcc58

Introduction

The framework allows the construction of state-transition graphs to model user-agent interactions in dialogue systems or other event-based applications. It uses explicit nodes and transitions to define actions and reactions in the system, facilitating integration with APIs and custom functions.
Additionally, this framework enables the generation of dialogue datasets for training natural language models.

The dialogues are generated by following the steps of the conversation. These steps are defined through a random walk on the state-transition graph. It is generated at each step of the conversation. Among the transitions accessible from the current node, one is chosen randomly. The LLM, acting either as the agent or the user, will generate a sentence using the selected transition.

Environment variables

To use the program with command lines, you need to set the environment variables. You can do this by creating a .env file in the root directory of the project and adding the following variables:

DEFAULT_OPENAI_TYPE =
DEFAULT_OPENAI_DEPLOYMENT_NAME=
DEFAULT_OPENAI_ENDPOINT=
DEFAULT_OPENAI_API_KEY=

For DEFAULT_OPENAI_TYPE, you can choose between 'openai' and 'azure_openai'. Also, DEFAULT_OPENAI_ENDPOINT is needed only for 'azure_openai' type.

Getting Started

Requirements:

pip install -r requirements.txt

Conversation generation:

python generate_conversations.py -m hotel -n 5 -p

Conversation evaluation

cd unieval
python eval_conv.py -f conversations.jsonl -m full -t graphtod

Visualize graph

python show_graph.py -m hotel

Minimum Requirements

To use this framework, the following elements are necessary:

1. Starting Sentence: The initial sentence to start the interaction.
2. Initial Node: The node from which the interaction begins.
3. Graph of Possible Actions: The definition of transitions between nodes.
4. APIs Associated with Transitions: The APIs to call during transitions if needed.

Graph Formalism

Graph Definition

The state-transition graph is a data structure that defines the possible actions of the user and the corresponding responses of the agent. It consists of nodes and transitions that describe the interactions between the user and the agent. In the code, the graph is defined as a dictionary of nodes and transitions.

Node Structure

A node is defined by a name followed by its possible transitions. Each transition is described by a user action and the corresponding agent action, leading to another node. Nodes should be named in CamelCase and transitions in snake_case. By default, the initial node is named "InitialState." A node is an action of the agent, using an action verb and a complement that describes how to apply the verb.

The name of a node is written as follows: "VerbComplement".

Example

"CollectLicensePlateNumber", "ValidateReservation", "AskAppointmentDate", ...

"Node" : { transitions }

Defining a Transition

A transition is the user's action. Transitions from one node to another are defined by a pair user_action: AgentAction, where AgentAction is the name of the next node. To name the transition, the complement works as it does for nodes.

The name of a transition is written as follows: "verb_complement"

Example

"cancel_reservation", "select_doctor", "schedule_appointment", ...

"user_action" : "AgentAction"

Building a Graph

A graph is built as a Python dictionary. It consists of nodes and transitions that will be written following the definitions above.

myDict = {
    "Node1": {
        "edge": "Node2"
        }
}

myDict = {
    "VerbComplement1": {
        "verb_complement": "VerbComplement2",
        "verb_complement": "VerbComplement1",
    },
    "VerbComplement2": {
    ...
    }
}

Naming Conventions for Nodes and Transitions

• Nodes and transitions must be named using action verbs such as "ask," "schedule," "search," "show," etc.
• Naming should be explicit to clearly indicate the action performed in the prompt.

Example of Action Verbs:

Ask, Schedule, Edit, Identify, Reschedule, Cancel, Remind, Search, Select, Check, Show, Pick-up, Extend, Book, Extend, Make, Provide, Return, Collect, Validate, Explain, Send, Give, Confirm

Using Functions in Transitions

Transitions can trigger functions whose results will be used in the prompt of the next node. These functions can contain API calls to retrieve the information necessary for the action. This is how additional capabilities can be added to the agent.

Example of a Function

def select_i(api_url, parameters):
    # Implementation here

The function select_i is used to have the LLM select from a list of items.

Example of Function Use in an Agent

Here, we call an API for the action see_vehicles and use the function select_i for the action select_vehicle.

graph = {
            "AskVehicleType": {
                "see_vehicles": "VehiclesAvailables",
            },
            "VehiclesAvailables": {
                "select_vehicle": "AskNameToValidateReservation",
            }
}
function_call = {
    "select_vehicle": self.select_i,
    "see_vehicles": "/car/search",
}

Example of a State-Transition Graph

Here is an example of defining a state-transition graph:

{
	"Start": {
		"ask_question": "Faq",
		"schedule_appointment": "AskPatientInfo",
		"edit_appointment": "AskPatientName"
	},
	"AskPatientInfo": {
		"search_doctor_list": "ShowDoctorList"
	}
}

Persona

The framework automatically generates user personas based on the content of the agent, created by a language model (LLM). The agent's content is summarized by an LLM and then used as context to generate user personalities that are relevant to the agent.

Each persona has an automatically generated name, age, and their preferences.

Example: For a medical appointment booking agent, the generated personas will have preferences for doctors, schedules, etc.

Evaluation

Unieval is a T5 based tool to evaluate the generated conversations. Details in Unieval folder.

Additional Explanations

The transitions generated by the random walk will not be followed when a transition that logically leads to the next step is missing or when an illogical request is made. When writing the graph, the logic of things must be respected: a state does one thing only, and we don't assume that an additional action will be performed within it. For example: having an appointment date only means giving the appointment dates, it does not include interaction with the user (their response will be a transition to another node). We will break it down into requesting an appointment date, responding with the appointment dates, and then selecting the appointment dates.

Only one action will be made at each time step, either by the user or the agent. The transitions will be followed in the order of the graph, respecting the logic of the actions.

To help the persona generation, the graph should be written in such a way that the critical information for the agent's actions is in the first nodes of the graph. A possible explanation is that LLMs are more sensitive to information at the beginning of the context.

Full Example of an Agent Graph

This graph defines an agent managing hotel bookings.

It uses a function to select from a list and two API calls to check available hotels. It is able to book a room, or update your booking.

{
    "InitialState": {
        "search_hotels": "DisplayHotels",
        "ask_cancelling_or_modifying_reservation": "BookingFound"
    },
    "DisplayHotels": {
        "ask_for_more_hotels": "DisplayHotels",
        "select_hotel": "AskPaymentInfo",
        "ask_for_details": "DisplayHotelDetails"
    },
    "DisplayHotelDetails": {
        "select_hotel": "AskPaymentInfo",
        "ask_for_more_hotel": "DisplayHotels"
    },
    "AskPaymentInfo": {
        "check_payment_type": "AskPaymentConfirmation"
    },
    "AskPaymentConfirmation": {
        "process_payment": "PaymentAccepted"
    },
    "PaymentAccepted": {
        "request_invoice": "SendInvoice",
        "end": "Stop"
    },
    "SendInvoice": {
        "end": "Stop"
    },
    "BookingFound": {
        "criteria_to_modify": "ModificationPossible",
        "refund": "AnswerForRefund"
    },
    "ModificationPossible": {
        "add_criteria": "OtherCriteriaAdded"
    },
    "OtherCriteriaAdded": {
        "end": "Stop"
    },
    "AnswerForRefund": {
        "contest_refund_decision": "DetailsRefundDecision",
        "accept_decision": "Stop"
    },
    "DetailsRefundDecision": {
        "ask_for_another_compensation": "AlternativeCompensation",
        "accept_decision": "Stop"
    },
    "AlternativeCompensation": {
        "user_accepts": "CompensationAccepted",
        "user_refuses": "CompensationRefused"
    },
    "CompensationAccepted": {
        "end": "Stop"
    },
    "CompensationRefused": {
        "negociate_compensation": "AlternativeCompensation",
        "end": "Stop"
    },
    "Stop": {}
}

function_call = {
            "select_hotel": self.select_i,
            "search_hotels": "/hotel/search",
            "ask_for_more_hotels": "/hotel/search",
        }