This repo is a dataset and methods survey for Task-oriented Dialogue.
We investigated most existing dialogue datasets and summarized their basic information, such as brief, download link and size.
We also included leader boards of popular dataset to present research progress in the task oriented dialogue fields.
A Chinese intro & news for this project is available here
@misc{MAML_Pytorch,
author = {Yutai Hou},
title = {Task-Oriented Dialogue Research Progress Survey},
year = {2018},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/AtmaHou/Task-Oriented-Dialogue-Research-Progress-Survey/}},
commit = {master}
}
This section records big updates to ease refer (See ./release_detail.md or click links below):
- Updates 2020.7.30: Add 3 new datasets, re-claim 6 abnormal NLU results, refine layout.
- Updates 2020.7.27: Add 10 new datasets, 5 new SOTA results, 1 correctness.
Contributions are welcomed, you are encouraged to:
- Directly pull request
- Send me new dataset info
- Send me new experiment results from published paper or public code implements.
The ranking is depended on published results of related papers. We are trying to keep it up-to-date. The ranking may be unfair because features used and train/dev set splitting in those papers may be different. However, it shows a trend of research, and would be helpful for someone to start a project about task-oriented dialogue.
Dialogue state tacking task aims to predict or give representation of dialogue state, which usually contains a goal constraint, a set of requested slots, and the user's dialogue act.
MultiWOZ 2.0 - Dialogue State Tracking
Multi-Domain Wizard-of-Oz dataset (MultiWOZ), a fully-labeled collection of human-human written conversations spanning over multiple domains and topics. At a size of 10k dialogues, it is at least one order of magnitude larger than all previous annotated task-oriented corpora.
The new, corrected versions of the dataset are available at MultiWOZ 2.1 (2019), MultiWOZ 2.2 (2020).
Notice: Models marked with * are open-vocabulary based models.`
Clarification of dataset types:
The main results we list here are obtained from pure DSTC2 dataset (ASR n-best).
However, we don't list other kinds of DSTC2 data source results such as DSTC2-text (It formulates the dialog state tracking as a machine reading problem which read the dialog transcriptions multiple times and answer the questions about each of the slot, for more info please refer to paper) and DSTC-cleaned (It is used by the NBT paper and fixes ASR noise and typo during training and include ASR noise during testing, The cleaned version is available at here),
| Model | Area | Food | Price | Joint | Paper / Source |
|---|---|---|---|---|---|
| Liu et al. (2018) | 90 | 84 | 92 | 72 | Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems |
| Neural belief tracker (Mrkšić et al., 2017) | 90 | 84 | 94 | 72 | Neural Belief Tracker: Data-Driven Dialogue State Tracking |
| RNN (Henderson et al., 2014) | 92 | 86 | 86 | 69 | Robust dialog state tracking using delexicalised recurrent neural networks and unsupervised gate |
Slot filling task aims to recognize key entity within user utterance, such as position and time.
Notice: The following works have abnormal high-scores, because they are considered to exploit special pre-processing steps: Bi-model-Decoder (Wang et al., 2018), Intent Gating + Self-atten. (Li et al., 2018), Atten.-Based (Liu and Lane, 2016)
Intent detection task aims to classify user utterance into different domain or intents.
| Model | Acc. | Paper / Source |
|---|---|---|
| ELMo + BLSTM-CRF (Siddhant et al., 2018) | 99.29 | Unsupervised Transfer Learning for Spoken Language Understanding in Intelligent Agents |
| Enc-dec (focus) + ELMo | 99.14 | Code |
| Stack-Propagation + BERT (Qin et al., 2019) | 99.0 | A Stack-Propagation Framework with Token-level Intent Detection for Spoken Language Understanding |
| Joint BERT (Chen et al., 2019) | 98.6 | BERT for Joint Intent Classification and Slot Filling |
| Stack-Propagation (Qin et al., 2019) | 98.0 | A Stack-Propagation Framework with Token-level Intent Detection for Spoken Language Understanding |
| Capsule Neural Networks (Zhang et al., 2018) | 97.7 | Joint Slot Filling and Intent Detection via Capsule Neural Networks |
| Slot-Gated (Full Atten.) (Goo et al., 2018) | 97.0 | Slot-Gated Modeling for Joint Slot Filling and Intent Prediction |
| Slot-Gated (Intent Atten.) (Goo et al., 2018) | 96.8 | Slot-Gated Modeling for Joint Slot Filling and Intent Prediction |
Notice-1: The following works have abnormal high-scores, because they are considered to exploit special pre-processing steps: Bi-model-Decoder (Wang et al., 2018), Intent Gating + Self-atten. (Li et al., 2018), Atten.-Based (Liu and Lane, 2016), BLSTM (Zhang et al., 2016)
See the data details Here or in Excel File
Following information is included for each dataset:
- Name
- Introduction
- Link (Download & Paper)
- Multi or single turn
- Task detail
- Whether Public Accessible
- Size & Stats
- Included Label
- Missing Label
Tips: The table below may not be displayed completely, scroll right to see more~
| Name | Introduction | Links | Multi/Single Turn | Task Detail | Public Accessible | Size & Stats | Included Label | Missing Label |
|---|---|---|---|---|---|---|---|---|
| Few-shot Slot Tagging Benchmark | 1. Dialogue slot tagging dataset for few-shot learning setting 2. First few-shot sequence labeling benchmark (Meta-episode style data format) 3. Also include 5 NER dataset for few-shot sequence labeling evaluation. |
Download:https://atmahou.github.io/attachments/ACL2020data.zip Paper: https://arxiv.org/pdf/2006.05702.pdf |
S | 7 dialogue task: Weather,play music, search, add to list, book, moive 5 NER task |
Yes | For each task, it contains 100 episodes. Each episode contains a query set (20 samples) and a support set (1-shot & 5-shot) |
Slots | Intent |
| Taskmaster-2 (2020) | 1. Unlike Taskmaster-1, which includes both written "self-dialogs" and spoken two-person dialogs, Taskmaster-2 consists entirely of spoken two-person dialogs. 2. Users were led to believe they were interacting with an automated system that “spoke” using text-to-speech (TTS) 3. Intents are labeled on slots |
Download: https://github.com/google-research-datasets/Taskmaster/tree/master/TM-2-2020/data Homepage: https://github.com/google-research-datasets/Taskmaster/tree/master/TM-2-2020 |
M | 7 domains: restaurants, food ordering, movies, hotels, flights, music, sports |
Yes | 17,289 dialogs: restaurants (3276) food ordering (1050) movies (3047) hotels (2355) flights (2481) music (1602) sports (3478) |
NLU(Intent, Slots) | |
| JDDC Corpus 2020 | 1. A large-scale Multimodal Chinese E-commerce conversation corpus. 2. Human2Human conversations |
Download: https://jddc.jd.com/auth\_environment Homepage: https://jddc.jd.com/description |
M | Multimodal E-commerce conversation | Yes | Electronic: 130k dialogues, 950k utterances, 215k images. Clothing: 116k dialogues, 810k utterances, 200k images. |
Intents (Only on images), Database |
NLU(Intent, Slots) |
| CrossWOZ | 1. CrossWOZ, the first large-scale Chinese Cross-Domain Wizard-of-Oz taskoriented dataset. 2. Encourage natural transition across domains in conversation. 3. Provide a user simulator 4. Human2Human |
Download: https://github.com/thu-coai/CrossWOZ Paper: https://arxiv.org/pdf/2002.11893.pdf |
M | 5 domains, including hotel, restaurant, attraction, metro, and taxi. | Yes | 5,012 dialogues, 84,692 turns, 16.9 Avg. turns, Annotation: 72 slots, 7,871 vlaues, 6 intents |
User Goals, State (Intent, Slots), Database |
API calls |
| JDDC Corpus 2019 | 1. A large-scale real scenario Chinese E-commerce conversation corpus. 2. Human2Human conversations covers: task-oriented, chitchat and question-answering. 3. Large scale: 1 million multi-turn dialogues, 20 million utterances. 4. Main task: dialogue generation |
Download: http://jddc.jd.com/auth\_environment Paper: https://arxiv.org/pdf/1911.09969.pdf |
M | E-commerce conversation | Yes | Totoal: 1 million dialogues, 20 million utterances. Annotation: 289 different intents Challenge1: 300 dialogues, 300 questions; Challenge2: 15 dialogues, 168 questions; Challenge3: 108 dialogues, 500 questions; |
Intent (Machine Labeled), Database |
Slot |
| CAIS | 1. Dialogue utterances from the Chinese Artificial Intelligence Speakers (CAIS) annotated with slot tags and intent labels. | Download: https://github.com/Adaxry/CM-Net Paper: https://www.aclweb.org/anthology/D19-1097.pdf |
S | Most are music related tasks. | Yes | Train 7995; Dev 994; Test 1012; 11 Intents, 75 Slots |
Intent Slots |
|
| Multimodal Dialogs (MMD) Dataset | 1. Multimodal conversations in the fashion domain. 2. Human-to-human 3. Contain annotation of query type (Similar to intent) 4. Large size: 150K conversation |
Download: https://amritasaha1812.github.io/MMD/ Paper: https://arxiv.org/abs/1704.00200 |
M | Shopping Assistant | Yes | 150K conversation session | question-type (Intent) State Type (17 type of dialogue state class) |
Slot |
| Taskmaster-1 (2019) | 1. A task-based dataset collect with two different procedures: Wizard of Oz and self conversation. 2. Encourage realistic and diversity by giving up restrict speaker with knowledage base. 3. Both Human2machine and Human2Human dialogues |
Download: https://g.co/dataset/taskmaster-1 Paper: https://arxiv.org/pdf/1909.05358.pdf |
M | 6 domains: ordering pizza, creating auto repair appointments, setting up ride service, ordering movie tickets, ordering coffee drinks and making restaurant reservations. | Yes | Human-Human: 7,708 dialogues, 169,469 utterances Human-Machine: 10,438 dialogues, 132,610 utterances |
API calls, Argument (Slot) |
Intent |
| MetaLWOz | 1. Dialogue dataset for developing fast adaptation methods for conversation models. (Track in DSTC 8) 2. Lots of domains and tasks: 47 domains and 227 tasks. 3. Suitable for meta learning. 4. Main task: dialogue generation |
Download: https://www.microsoft.com/en-us/download/58389 Homepage: https://www.microsoft.com/en-us/research/project/metalwoz/ |
M | 47 domains and 227 tasks. | Yes | 37,884 dialogues, ( >10 turns long), 47 domains and 227 tasks. |
Only utterance | NLU(Intent, Slots) |
| Minecraft Dialogue corpus | 1. The goal of this project is to develop systems that can collaborate and communicate with each other to solve tasks in a 3D environment. 2. Human2Human 3. Main task: given context and 3D scenes, generating response. |
Download: http://juliahmr.cs.illinois.edu/Minecraft/ Paper: https://www.aclweb.org/anthology/P19-1537.pdf |
M | "Architect" instruct the "Builder" to build a 3D structure. | Yes | 509 human-human dialogues; 15,926 utterances (train 6,548, dev 2,855, test 2,251 Architect utterances); |
Golden utterance, game log, screenshots |
NLU(Intent, Slots) |
| E-commerce Dialogue Corpus (EDC) | 1. Real-world conversations between customers and customer service staff from our E-commerce partners in Taobao 2. Main task: response selection |
Download: https://github.com/cooelf/ Paper: https://arxiv.org/pdf/1806.09102.pdf |
M | Contains 5 types of conversations: commodity consultation, logistics express, recommendation, negotiation and chitchat based on over 20 commodities. | Yes | Dialogues 1,020,000, Utterance 7,500,000. |
Only utterance | NLU(Intent, Slots) |
| Schema-Guided Dialogue State Tracking(DST8) | 1. Largest by now & containing over 16k multi-domain conversations spanning 16 domains 2. Present a schema-guided paradigm 3. Enable zero-shot generalization to new APIs |
Download: https://github.com/google-research-datasets/dstc8-schema-guided-dialogue Paper: https://arxiv.org/pdf/1909.05855.pdf |
M | 16 domains Alarm, Banks, Buses, Calendar, Events, Flights, Homes, Media, Messaging, Movies, etc. | Yes | Over 16k dialogues average number of turns are 20.44 multi-domain dialogues. 329,964 turns in total. | Schema for each service contains: service_name and description, slots, intents |
_ |
| MultiWOZ 2.0 | 1. Proposed by EMNLP 2018 best paper. 2. Largest by now & contain multi-domains. 3. Human2human 4. goal changes are encouraged |
Download: http://dialogue.mi.eng.cam.ac.uk/index.php/corpus/ Paper: https://arxiv.org/pdf/1810.00278.pdf |
M | 7 domains Attraction, Hospital, Police, Hotel, Restaurant, Taxi, Train. |
Yes | Total 10438 dialogues average number of turns are 8.93 and 15.39 for single and multi-domain dialogues respectively. 115, 434 turns in total. |
Belief state User Act(inform, request slots) Agent Act(inform, request slots) |
NLU(Intent, Slots) |
| Facebook Multilingual Task Oriented Dataset | 1. (Faceboook) We release a dataset of around 57k annotated utterances in English (43k), Spanish (8.6k) and Thai (5k) for three task oriented domains … ALARM, REMINDER, and WEATHER. 2. For cross-lingual natural language understanding |
Download: https://fb.me/multilingual\_task\_oriented\_data Paper: https://arxiv.org/pdf/1810.13327.pdf |
S | 3 Domains: Alarm, Reminder, Weather 3 Languages: English, Spanish, Thai |
Yes | English Train: 30,521 English Dev: 4,181 English Test: 8,621 Spanish Train: 3,617 Spanish Dev: 1,983 Spanish Test: 3,043 Thai Train: 2,156 Thai Dev: 1,235 Thai Test: 1,692 |
Slot Intent |
|
| Medical DS | 1. Our dataset is collected from the pediatric department in a Chinese online healthcare community 2. Task-oriented Dialogue System for Automatic Diagnosis |
Download: http://www.sdspeople.fudan.edu.cn/zywei/data/acl2018-mds.zip Paper: http://www.sdspeople.fudan.edu.cn/zywei/paper/liu-acl2018.pdf |
M | Automatic Diagnosis | Yes | 4 Disease 67 symptoms |
Slot Action |
|
| Snips | 1. Collected by Snips for model evaluation. 2. For natural language understanding 3. Homepage: https://medium.com/snips-ai/benchmarking-natural-language-understanding-systems-google-facebook-microsoft-and-snips-2b8ddcf9fb19 |
Download: https://github.com/snipsco/ nlu-benchmark/tree/master/ 2017-06-custom-intent-engines |
S | 7 task: Weather,play music, search, add to list, book, moive |
Yes | Train:13,084 Test:700 7 intent 72 slot labels |
Intent Slots |
|
| MIT Restaurant Corpus | 1. The MIT Restaurant Corpus is a semantically tagged training and test corpus in BIO format. 2. For natural language understanding |
Download: https://groups.csail.mit.edu/sls/downloads/restaurant/ |
S | Restaurant | Yes | Train, Dev, Test 6,894 766 1,521 |
Slot | Intent |
| MIT Movie Corpus | 1. The MIT Movie Corpus is a semantically tagged training and test corpus in BIO format. The eng corpus are simple queries, and the trivia10k13 corpus are more complex queries. 2. For natural language understanding |
Download: https://groups.csail.mit.edu/sls/downloads/movie/ |
S | Movie | Yes | Train, Dev, Test MIT Movie Eng 8,798 977 2,443 MIT Movie Trivia 7,035 781 1,953 Refer to: Data Augmentation for Spoken Language Understanding via Joint Variational Generation |
Slot | Intent |
| ATIS | 1. The ATIS (Airline Travel Information Systems) dataset (Tur et al., 2010) is widely used in SLU research 2. For natural language understanding |
Download: 1. https://github.com/AtmaHou/Bi-LSTM\_PosTagger/tree/master/data 2.https://github.com/yvchen/JointSLU/tree/master/data |
S | Airline Travel Information | Yes | Train: 4478 Test: 893 120 slot and 21 intent |
Intent Slots |
|
| Microsoft Dialogue Challenge | 1. Containing human-annotated conversational data in three domains an 2. Experiment platform with built-in simulators in each domain, for training and evaluation purposes. |
Paper： https://arxiv.org/pdf/1807.11125.pdf |
M | Movie-Ticket Booking Restaurant Reservation Taxi Ordering |
Yes | Task Intents Slots Dialogues Movie-Ticket Booking 11 29 2890 Restaurant Reservation 11 30 4103 Taxi Ordering 11 29 3094 |
Intent Slots |
Database API-call |
| CamRest676 | CamRest676 Human2Human dataset contains the following three json files: 1. CamRest676.json: the woz dialogue dataset, which contains the conversion from users and wizards, as well as a set of coarse labels for each user turn. 2. CamRestDB.json: the Cambridge restaurant database file, containing restaurants in the Cambridge UK area and a set of attributes. 3. The ontology file, specific all the values the three informable slots can take. |
Download: https://www.repository.cam.ac.uk/handle/1810/260970 Paper: https://arxiv.org/abs/1604.04562 |
M | Booking restaurant | Yes | Total 676 Dialogues Total 1500 Turns Train:Dev:Test 3:1:1 (Test set not given) |
Slot User Act(inform, request slots) Agent Act(inform, request slots) |
Intent API call Database |
| Human-human goal oriented dataset | 1. Maluuba reased a travel booking dataset 2. Design for new task: frame tracking (allow comparing between history entities) 3. Homepage: https://datasets.maluuba.com/Frames 4. Human2Human |
Download: https://datasets.maluuba.com/Frames/dl Paper: https://arxiv.org/abs/1706.01690 https://1drv.ms/b/s!Aqj1OvgfsHB7dsg42yp2BzDUK6U |
M | Travel Booking | Yes | Dialogues 1369 Turns 19986 Average user satisfaction (from 1-5) 4.58 |
Frame User agenda User Act(inform, request slots) Agent Act(inform, request slots) API Call User's satisfaction Task successful Database Entity reference |
Intent |
| Dialog bAbI tasks data | 1. Facebook's 6 task-oriented dialogues data set consist of 6 different tasks. 2. Dataset for task 1-5 is constucted automaticly from bots' chat(Bot2Bot). And dataset for task 6 is simply reformated dstc2 dataset. 3. A Shared database is included. 4. This is the only task-oriented dataset among bAbI tasks. 5. The goal of it is to evaluate end2end tasks, so there is not intents and slots. |
Download: https://research.fb.com/downloads/babi/ Paper: http://arxiv.org/abs/1605.07683 |
M | Book a table at a restaurant | Yes | For each task, training 1000 develop 1000 test 1000 For tasks 1-5, second test set (with suffix -OOV.txt) that contains dialogs including entities not present. |
API call Full Database |
Slot Intent User Act Agent Act |
| Stanford Dialog Dataset | 1. Standford NLP group's data of car autopilot agent. 2. Human2Human 3. A quick intro http://m.sohu.com/n/499803391/ |
Download: http://nlp.stanford.edu/projects/kvret/kvret\_dataset\_public.zip Paper: https://arxiv.org/abs/1705.05414 |
M | car autopilot agent: schedule, weather, navigation | Yes | Training Dialogues 2,425 Validation Dialogues 302 Test Dialogues 304 Avg. # of Utterances Per Dialogue 5.25 |
Dialogue level database User Act(inform, request slots) Agent Act(inform, request slots) |
API call Intent Slot |
| Stanford Dialog Dataset LU | 1. Stanford data labeled by HIT, relabel slot & intent 2. Human2Human 3. A quick intro http://m.sohu.com/n/499803391/ to stanford data 4. Annotation handbook: https://docs.google.com/document/d/1ROARKf8AJNnG2\_nPINe1Xm5Rza7V0jPnQV8io09hcFY/edit |
N/A |
M | car autopilot agent: schedule, weather, navigation | No | Training Dialogues 2,425 Validation Dialogues 302 Test Dialogues 304 Avg. # of Utterances Per Dialogue 5.25 |
Slot Intent |
API call Need to do sample alignment to get the following: Dialogue level database User Act(inform, request slots) Agent Act(inform, request slots) Agent Reply |
| DSTC-2 | 1. Human2Bot restaurant booking dataset 2. For usage refer to: http://camdial.org/~mh521/dstc/downloads/handbook.pdf 3. Each dialofue is stored in different folder, which contains log and label. |
http://camdial.org/~mh521/dstc/ | M | Booking restaurant | Yes | Train 1612 calls Dev 506 calls Test 1117 dialogs |
Slot User Act(inform, request slots) Agent Act(inform, request slots) |
Intent API call Database |
| DSTC4 | 1. Data name as TourSG consists of 35 dialog sessions on touristic information for Singapore collected from Skype calls between three tour guides and 35 tourists 2. All the recorded dialogs with the total length of 21 hours have been manually transcribed and annotated with speech act and semantic labels for each turn level. 3. Homepage: http://www.colips.org/workshop/dstc4/data.html 4. Human2Human |
N/A | M | Query touristic information | No | Train 20 dialogs Test 15 dialogs |
speech act (User & Agent) semantic labels(Intent? User & Agent) topic for turn (Intent?) |
N/A |
| Movie Booking Dataset | 1. (Microsoft) Raw conversational data collected via Amazon Mechanical Turk, with annotations provided by domain experts. 2. Human2Human |
Download: https://github.com/MiuLab/TC-Bot#data Paper: TC-bot |
M | Booking Movie | Yes | 280 dialogues turns per dialogue is approximately 11 |
User Act(inform, request slots) Agent Act(inform, request slots) Intent Slots |
Database API-call |
| Lingxi | 1. The data is all single round user input divided into good words. There is more noise. 2. Completed part of speech tagging and slot labeling 3. Language: Chinese |
N/A | S | conversational robot service user log | No | Utterance: 5132 | Slot POS |
Agent reply Intent API call Database |
| TOP semantic parsing | 1. (Facebook) A hierarchical semantic representation for task oriented dialog systems that can model compositional and nested queries. (hierarchical intent and slot) 2. For natural language understanding 3. Human2bot |
Download: http://fb.me/semanticparsingdialog Paper: https://arxiv.org/pdf/1810.07942.pdf |
S | Navigation and event | Yes | Train 31279 utterances Dev 4462 utterances Test 9042 utterances |
Hierarchical intents Slots |
|
| SwDA | 1. The Switchboard Dialog Act Corpus (SwDA) extends the Switchboard-1 Telephone Speech Corpus, Release 2, with turn/utterance-level dialog-act tags. 2. The tags summarize syntactic, semantic, and pragmatic information about the associated turn. The SwDA project was undertaken at UC Boulder in the late 1990s. |
Download: http://compprag.christopherpotts.net/swda.html Instruction: https://web.stanford.edu/~jurafsky/ws97/manual.august1.html |
S | Switchboard Dialog | Yes | Train: 197,489 training-set utterances, 1115 conversations Test: 40 conversations Annotation: 42 Classes |
Act | Slot |
Thanks for supports from my adviser Wanxiang Che.
Thanks for public contributions from: Shuai Lin, JiAnge, Su Zhu, seeledu, Tony Lin, Jason Krone, Libo Qin, HariiHe, Jelle Bosscher, .