This repository contains the data, scripts and baseline codes for DSTC11 Track 5.
💬 For the description and instructions of our team's approaches, please see CLTeamL.
This challenge track aims to support more informative and engaging task-oriented conversations by utilizing the subjective knowledge from review posts. Track participants will develop dialogue systems to understand relevant review posts, and generate system responses grounded on the selected review knowledge snippets.
Organizers: Seokhwan Kim, Spandana Gella, Chao Zhao, Di Jin, Alexandros Papangelis, Behnam Hedayatnia, Yang Liu, Dilek Hakkani-Tur
This challenge track distinguishes between turns that could be handled by the existing task-oriented conversational models with no extra knowledge and turns that require external subjective knowledge to be answered by the dialogue system. We focus on the turns that require knowledge access as the evaluation target in this track by the following three tasks:
| Task #1 | Knowledge-seeking Turn Detection |
|---|---|
| Goal | To decide whether to continue the existing scenario or trigger the knowledge access branch for a given utterance and dialogue history |
| Input | Current user utterance, Dialogue context, Knowledge snippets |
| Output | Binary class (requires knowledge access or not) |
| Task #2 | Knowledge Selection |
|---|---|
| Goal | To select proper subjective knowledge sources given a dialogue state at each turn with knowledge access |
| Input | Current user utterance, Dialogue context, Knowledge snippets |
| Output | List of relevant knowledge candidates |
| Task #3 | Knowledge-grounded Response Generation |
|---|---|
| Goal | To take a triple of input utterance, dialog context, and the selected knowledge snippets and generate a system response |
| Input | Current user utterance, Dialogue context, and Selected knowledge snippets |
| Output | Generated system response |
Participants will develop systems to generate the outputs for each task. They can leverage the annotations and the ground-truth responses available in the training and validation datasets.
In the test phase, participants will be given a set of unlabeled test instances. And they will submit up to 5 system outputs for all three tasks.
NOTE: For teams who are interested in only one or two of the tasks, we recommend to use our baseline system for the remaining tasks to complete the system outputs.
Each submission will be evaluated in the following task-specific automated metrics first:
| Task | Automated Metrics |
|---|---|
| Knowledge-seeking Turn Detection | Precision/Recall/F-measure |
| Knowledge Selection | Precision/Recall/F-measure, Accuracy |
| Knowledge-grounded Response Generation | BLEU, ROUGE, METEOR |
To consider the dependencies between the tasks, the scores for knowledge selection and knowledge-grounded response generation are weighted by knowledge-seeking turn detection performances. Please find more details from scores.py.
The final ranking will be based on human evaluation results only for selected systems according to automated evaluation scores. It will address the following aspects: appropriateness and relevance to given knowledge.
In this challenge track, participants will use an augmented version of MultiWoz 2.1 which includes newly introduced subjective knowledge-seeking turns. All the ground-truth annotations for Knowledge-seeking Turn Detection and Knowledge Selection tasks as well as the agent's responses for Knowledge-grounded Response Generation task are available to develop the components on the training and validation sets. In addition, relevant knowledge snippets for each domain and entity are also provided in knowledge.json.
In the test phase, participants will be evaluated on the results generated by their models for the unlabeled test set. To evaluate the generalizability and the portability of each model, the unseen test set may include different domains, entities and locales than MultiWoz.
Data and system output format details can be found from data/README.md.
- Training data released: Dec 19, 2022
- Test data released: Mar 24, 2023
- Entry submission deadline: Mar 31, 2023
- Objective evaluation completed: Apr 3, 2023
- Human evaluation completed: Apr 10, 2023
- Participation is welcome from any team (academic, corporate, non profit, government).
- The identity of participants will NOT be published or made public. In written results, teams will be identified as team IDs (e.g. team1, team2, etc). The organizers will verbally indicate the identities of all teams at the workshop chosen for communicating results.
- Participants may identify their own team label (e.g. team5), in publications or presentations, if they desire, but may not reveal the identities of other teams.
- Participants are allowed to use any external datasets, resources or pre-trained models.
-
To join the mailing list: visit https://groups.google.com/a/dstc.community/forum/#!forum/list/join
-
To post a message: send your message to list@dstc.community
-
To leave the mailing list: visit https://groups.google.com/a/dstc.community/forum/#!forum/list/unsubscribe
Please feel free to contact: seokhwk (at) amazon (dot) com
The code is licensed under Apache 2.0 (see SOFTWARELICENSE) and the data files are licensed under CDLA-Sharing 1.0 (see DATALICENSE).