diff --git a/projects/msc/README.md b/projects/msc/README.md new file mode 100644 index 00000000000..0721e15cac9 --- /dev/null +++ b/projects/msc/README.md @@ -0,0 +1,64 @@ +# Beyond Goldfish Memory: Long-Term Open-Domain Conversation + +Jing Xu, Arthur Szlam, Jason Weston + +## Abstract + +Despite recent improvements in open-domain dialogue models, state of the art models are trained and evaluated on short conversations with little context. +In contrast, the long-term conversation setting has hardly been studied. In this work we collect and release a human-human dataset consisting of multiple chat sessions whereby the speaking partners learn about each other's interests and discuss the things they have learnt from past sessions. +We show how existing models trained on existing datasets perform poorly in this long-term conversation setting in both automatic and human evaluations, and we study long-context models that can perform much better. In particular, we find retrieval-augmented methods and methods with an ability to summarize and recall previous conversations outperform the standard encoder-decoder architectures currently considered state of the art. + +## Paper + +[Link](msc.pdf) + +## Example Conversation Session +(Human-human conversation from the dataset) + +
+ + +## Data + +We release the Multi-Session Chat and Summarization tasks at `parlai/tasks/msc`. + +To view the multi-session chat data (session 1-4, with 237k training examples and 25k valid examples), run: +``` +parlai display_data -t msc +``` + +To view the multi-session chat data (session 1-5, with extra 6k valid examples), run: + +``` +parlai display_data -t msc --include-last-session True +``` + +To view the dialog summary for multi-session chat data (session 1-4, with 130k train examples and 25k valid examples), run: + +``` +parlai display_data -t msc:PersonaSummary --include-last-session True +``` + +See the [ParlAI quickstart for help](http://www.parl.ai/docs/tutorial_quick.html). + + +## Models + +- The model MSC2.7B (truncate 1024) can be downloaded and evaluated with the following command: +``` +parlai eval_model -mf zoo:msc/msc3B_1024/model -t msc -dt valid -bs 16 --previous-persona-type raw_history +``` + +- The model SumMem-MSC 2.7B (Fid-RAG) can be downloaded and evaluated with the following command: +``` +parlai eval_model -mf zoo:msc/summsc_fidrag3B/model -t msc -dt valid -bs 16 --previous-persona-type predsum_both +``` + +- The Dialogue Summarization Model (for summarizing dialogue and writing memories) can be downloaded and evaluated with the following command: +``` +parlai eval_model -mf zoo:msc/dialog_summarizer/model -t msc:PersonaSummary -dt valid -bs 16 +``` + +## Human Evaluations + +Human evaluations were performed using the Mephisto task [given here](https://github.com/facebookresearch/ParlAI/tree/master/parlai/crowdsourcing/tasks/model_chat). diff --git a/projects/msc/msc.pdf b/projects/msc/msc.pdf new file mode 100644 index 00000000000..a8ee9b49afb Binary files /dev/null and b/projects/msc/msc.pdf differ diff --git a/projects/msc/msc_conv_example.png b/projects/msc/msc_conv_example.png new file mode 100644 index 00000000000..30141e3e1e6 Binary files /dev/null and b/projects/msc/msc_conv_example.png differ