LLMs-Enhanced-Long-Text-Generation-Survey

Long Form NLG Generation Based on Large Language Models

Resource

A. Task Perspective

A.1 Long Form Open Domain Dialogue

1. MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversation. Junru Lu, Siyu An, Mingbao Lin, Gabriele Pergola, Yulan He, Di Yin, Xing Sun and Yunsheng Wu. [pdf]. arXiv Aug 16, 2023.
2. Prompted LLMs as Chatbot Modules for Long Open-domain Conversation. Gibbeum Lee，Volker Hartmann， Jongho Park，Dimitris Papailiopoulos and Kangwook Lee. [pdf]. Findings of ACL 2023.

A.2 Long Dialogue Summarization

1. An Exploratory Study on Long Dialogue Summarization: What Works and What’s Next.Yusen Zhang, Ansong Ni, Tao Yu, Rui Zhang, Chenguang Zhu, Budhaditya Deb, Asli Celikyilmaz, Ahmed Hassan Awadallah, Dragomir Radev, [pdf]Arxiv 10 Sep 2021
2. Improving Long Dialogue Summarization with Semantic Graph Representation.Yilun Hua, Zhaoyuan Deng, Kathleen McKeown, [pdf]ACL July 9-14 2023
3. DIALOGLM: Pre-trained Model for Long Dialogue Understanding and Summarization.Ming Zhong, Yang Liu, Yichong Xu, Chenguang Zhu, Michael Zeng, [pdf]Arxiv 6 Jan 2022
4. Negative Guided Abstractive Dialogue Summarization.Junpeng Liu, Yanyan Zou, Yuxuan Xi, Shengjie Li, Mian Ma, Zhuoye Ding, Bo Long, [pdf]Interspeech 18-22 Sept 2022
5. Improving Abstractive Dialogue Summarization with Graph Structures and Topic Words.Lulu Zhao, Weiran Xu, Jun Guo, [pdf]ICCL Dec 8-13 2020
6. Improving Long Dialogue Summarization with Semantic Graph Representation.Bobby Yilun Hua, Zhaoyuan Deng and K. McKeown.[pdf]ACL July 2023

A.3 Long Document Summarization

1. Two-Stage Movie Script Summarization: An Efficient Method For Low-Resource Long Document Summarization.Dongqi Pu, Xudong Hong, Pin-Jie Lin, Ernie Chang, Vera Demberg, [pdf].ACL July 2020
2. Efficient Attentions for Long Document Summarization.Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, Lu Wang, [pdf]NAACL June 6-11 2021.
3. HEGEL: Hypergraph Transformer for Long Document Summarization.Haopeng Zhang, Xiao Liu, Jiawei Zhang, [pdf]NLP Dec 7-11 2022
4. Long Document Summarization with Top-down and Bottom-up Inference.Bo Pang, Erik Nijkamp, Wojciech Kryscinski, Silvio Savarese, Yingbo Zhou, Caiming Xiong, [pdf]EACL May 2-6 2023
5. Globalizing BERT-based Transformer Architectures for Long Document.Quentin Grail, Julien Perez, [pdf]EACL April 19-23 2021
6. PLSGA：阶段式长文本摘要生成方法.方缙, 李宝安, 游新冬 and 吕学强.[pdf]CNKI Nov 2023
7. 关于中文长文本的自动文本摘要算法研究.李永星.[pdf]CNKI Jan 2023
8. V2P: Vision-to-Prompt based Multi-Modal Product Summary Generation.Xuemeng Song, Liqiang Jing, Dengtian Lin, Zhongzhou Zhao, Haiqing Chen and Liqiang Nie.[doi.org]ACM 7 July 2022

A.4 Long-Form Narrative Text

1. EIPE-text: Evaluation-Guided Iterative Plan Extraction for Long-Form Narrative Text Generation.Wang You, WenshanWu, Yaobo Liang, Shaoguang Mao, Chenfei Wu, Maosong Cao, Yuzhe Cai, Yiduo Guo, Yan Xia, Furu Wei, Nan Duan, [pdf].Arxiv 12 Oct 2023.

A.5 Story

1. A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation.Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, Minlie Huang, [pdf]TACL 1 Jan 2020
2. Open-ended Long Text Generation via Masked Language Modeling.Xiaobo Liang, Zecheng Tang, Juntao Li, Min Zhang, [pdf]ACL July 9-14 2023
3. Improving Chinese Story Generation via Awareness of Syntactic Dependencies and Semantics.Henglin Huang, Chen Tang, Tyler Loakman, Frank Guerin, Chenghua Lin, [pdf]AACL 19 Oct 2022
4. NGEP: A Graph-based Event Planning Framework for Story Generation.Chen Tang, Zhihao Zhang, Tyler Loakman, Chenghua Lin and Frank Guerin.[pdf]ACL 2022
5. Improving Chinese Story Generation via Awareness of Syntactic Dependencies and Semantics.Henglin Huang, Chen Tang, Tyler Loakman, Frank Guerin1 and Chenghua Lin.[pdf]ACL 2022
6. Make-A-Story: Visual Memory Conditioned Consistent Story Generation.Tanzila Rahman, Hsin-Ying Lee, Jian Ren, S. Tulyakov, Shweta Mahajan, L. Sigal.[pdf]arXiv 6 May 2023
7. EtriCA: Event-Triggered Context-Aware Story Generation Augmented by Cross Attention.Chen Tang, Chenghua Lin, Hen-Hsen Huang, Frank Guerin and Zhihao Zhang.[pdf]arXiv 22 Oct 2022
8. GraphPlan: Story Generation by Planning with Event Graph.Hong Chen, Raphael Shu, Hiroya Takamura, Hideki Nakayama.[pdf]INLG Sept 2021
9. Controllable Story Generation with External Knowledge Using Large-Scale Language Models.Peng Xu, M. Patwary, M. Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar and Bryan Catanzaro.[pdf]NLP 2 Oct 2020
10. A Temporal Variational Model for Story Generation.David Wilmot and Frank Keller.[pdf]arXiv 14 Sept 2021
11. Chinese Story Generation with FastText Transformer Network.Jhe-Wei Lin, Yunwen Gao and Rong-Guey Chang.[doi.org]IEEE Feb 2019
12. Improving Pacing in Long-Form Story Planning.Yichen Wang, Kevin Yang, Xiaoming Liu and Dan Klein.[pdf]arXiv 8 Nov 2023

A.6 Reviews

1. Towards coherent and cohesive long-form text generation.Woon Sang Cho, Pengchuan Zhang, Yizhe Zhang, Xiujun Li, Michel Galley, Chris Brockett, Mengdi Wang, Jianfeng Gao, [pdf]ACL 1 Nov 2018

A.7 Steganography

1. Generative Steganography Based on Long Readable Text Generation. Yi Cao, Zhili Zhou, Chinmay Chakraborty, Meimin Wang, Q.M.Jonathan Wu, Xingming Sun and Keping Yu.[pdf]IEEE 19 May 2022

A.8 Table-to-Text

1. Two-Level Model for Table-to-Text Generation.Juan Cao, Junpeng Gong and Pengzhou Zhang.[pdf]ACM Sept 2019
2. Three-stage Logical Table-to-Text Generation based on Type Control.Weiwei Shi, Yubo Liu, Jie Wu and Jianming Liao.[pdf]ACM Dec 2022

A.9 Patent

1. Controlling Patent Text Generation by Structural Metadata.Jieh-Sheng Lee.[pdf]ACM Oct 2020

A.10 Multilingual abstract

1. XWikiGen: Cross-lingual Summarization for Encyclopedic Text Generation in Low Resource Languages.Dhaval Taunk, Shivprasad Sagare, Anupam Patil, Shivansh Subramanian and Manish Gupta.[pdf]ACM Apr 2023
2. Long-Document Cross-Lingual Summarization.Shaohui Zheng, Zhixu Li, Jiaan Wang, Jianfeng Qu, An Liu, Lei Zhao and Zhigang Chen.[pdf]ACM Feb 2023
3. mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences.David C. Uthus, Santiago Ontan'on, J. Ainslie and Mandy Guo.[pdf]arXiv 26 Oct 2023

A.11 Persuasive Text

1. PersuAIDE! An Adaptive Persuasive Text Generation System for Fashion Domain.Vitobha Munigala, Abhijit Mishra, Srikanth G. Tamilselvam, Shreya Khare, Riddhiman Dasgupta and Anush Sankaran.[pdf]ACM Apr 2018

A.12 Hierarchical Topic-to-Essay

1. Transformer-based Hierarchical Topic-to-Essay Generation.Wangbo He and Yuan Rao.[pdf]ScienceDirect May 2022

A.13 Expository Text

1. Expository Text Generation: Imitate, Retrieve, Paraphrase.Nishant Balepur, Jie Huang and K. Chang.[pdf]arXiv 2023

A.14 Open-ended Text

1. KNN-LM Does Not Improve Open-ended Text Generation.Shufan Wang, Yixiao Song, Andrew Drozdov, Aparna Garimella, Varun Manjunatha and Mohit Iyyer.[pdf]arXiv 2023
2. Towards Informative Open-ended Text Generation with Dynamic Knowledge Triples.Zixuan Ren, Yang Zhao and Chengqing Zong.[pdf]EMNLP 2023
3. Open-ended Long Text Generation via Masked Language Modeling.Xiaobo Liang, Zecheng Tang, Juntao Li and Min Zhang.[pdf]ACL July 2023

A.15 Poetry

1. ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models.Jonas Belouadi and Steffen Eger.[pdf]ACL 20 Dec 2022
2. Chinese poetry generation model with UniLM.Zhangmin Ling and Lin Zhang[doi.org]IEEE Jan 2022
3. Modern French Poetry Generation with RoBERTa and GPT-2.Mika Hämäläinen, Khalid Alnajjar and T. Poibeau.[pdf]arXiv Dec 2022
4. Ancient poetry generation with an unsupervised method.Zhanjun Zhang, Haoyu Zhang, Qian Wan, Xiangyu Jia, Zhe Zhang and Jie Liu.[pdf]Springer 12 Mar 2022
5. A Sentiment and Style Controllable Approach for Chinese Poetry Generation.Yizhan Shao, Tong Shao, Minghao Wang, Peng Wang and Jie Gao.[doi.org]ACM 30 Oct 2021
6. SP-GPT2: Semantics Improvement in Vietnamese Poetry Generation._Tuan-Duy H. Nguyen, H. Pham, T. Bui, Tan-Minh Nguyen, D. Luong and Phong Nguyen.[pdf]arXiv Oct 2021
7. Classical Chinese Poetry Generation based on Transformer-XL.Jianli Zhao and H. Lee.[doi.org]IEEE Aug 2021
8. AfriKI: Machine-in-the-Loop Afrikaans Poetry Generation.Imke van Heerden and Anil Bas.[pdf]HCINLP 30 March 2021
9. Lingxi: A Diversity-aware Chinese Modern Poetry Generation System.Xinran Zhang, Maosong Sun, Jiafeng Liu, Xiaobing Li.[pdf]ACL 27 Aug 2021
10. Jiuge: A Human-Machine Collaborative Chinese Classical Poetry Generation System.Zhipeng Guo, Xiaoyuan Yi, Maosong Sun, Wenhao Li, Cheng Yang, Jian-na Liang, Huimin Chen, Yuhui Zhang and Ruoyu Li.[pdf]ACL 1 July 2019
11. Compose Like Humans: Jointly Improving the Coherence and Novelty for Modern Chinese Poetry Generation.Lei Shen, Xiaoyu Guo and Meng Chen.[pdf]arXiv 4 May 2020
12. Image Inspired Poetry Generation in XiaoIce.Wen-Feng Cheng, Chao-Chung Wu, Ruihua Song, Jianlong Fu, Xing Xie and Jian-Yun Nie.[pdf]arXiv 9 Aug 2018
13. TPoet: Topic-Enhanced Chinese Poetry Generation.Liang Yang, Zhexu Shen, Fengqing Zhou, Hongfei Lin and Junpeng Li.[doi.org]ACM 19 June 2023
14. GPT-based Generation for Classical Chinese Poetry.Yi Liao, Yasheng Wang, Qun Liu and Xin Jiang.[pdf]arXiv 29 June 2019
15. Automatic Generation Method of Ancient Poetry Based on LSTM.Hanshuang Zhang and Zhi Zhang.[doi.org]IEEE Nov 2020

A.16 Script

1. GPT-2-based Human-in-the-loop Theatre Play Script Generation.Rudolf Rosa, Patrícia Schmidtová, Ondrej Dusek, Tomáš Musil, D. Mareček, Saad Obaid, Marie Nováková, Klára Vosecká and Josef Doležal.[pdf]WNU 2022
2. The Film Script Generation Analysis Based on the Fiction Book Text Using Machine Learning.Danylo Ivanchyshyn, V. Vysotska and S. Albota[doi.org]IEEE Sept 2021
3. Leveraging Narrative to Generate Movie Script.Yutao Zhu, Ruihua Song, J. Nie, Pan Du, Zhicheng Dou and Jin Zhou.[doi.org]ACM 9 March 2022
4. "Kurosawa": A Script Writer's Assistant.Prerak Gandhi, Vishal Pramanik and P. Bhattacharyya.[pdf]arXiv 6 Aug 2023

A.17 News

1. Fact-Enhanced Synthetic News Generation.Kai Shu, Yichuan Li, Kaize Ding and Huan Liu.[pdf]arXiv 13 Dec 2020
2. Topic-Preserving Synthetic News Generation: An Adversarial Deep Reinforcement Learning Approach.Ahmadreza Mosallanezhad, Kai Shu and Huan Liu.[pdf]arXiv 30 Oct 2020
3. Data-Driven News Generation for Automated Journalism.Leo Leppänen, Myriam Munezero, Mark Granroth-Wilding and Hannu (TT) Toivonen.[pdf]NLG 1 Sept 2017
4. SHEG: summarization and headline generation of news articles using deep learning.R. Singh, Sonia Khetarpaul, R. Gorantla, Sai Giridhar Allada.[pdf]Springer 23 July 2020

A.18 Paper

1. Neural Academic Paper Generation.Samet Demir, Uras Mutlu and Özgür Özdemir.[pdf]arXiv 2 Dec 2019

A.19 Advertising Slogan

1. Japanese Advertising Slogan Generator using Case Frame and Word Vector.Kango Iwama and Yoshinobu Kano.[pdf]INLG Nov 2018
2. Smart Generation System of Personalized Advertising Copy and Its Application to Advertising Practice and Research.Shasha Deng, Chee‐Wee Tan, Weijun Wang and Yu Pan.[pdf]Journal of Advertising 8 Aug 2019
3. Scenario-based Multi-product Advertising Copywriting Generation for E-Commerce.Xueying Zhang, Kai Shen, Chi Zhang, Xiaochuan Fan, Yun Xiao, Zhen He, Bo Long and Lingfei Wu.[pdf]arXiv 21 May 2022

A.20 Email

1. Template-based Contact Email Generation for Job Recommendation.Qiuchi Li and C. Lioma.[pdf]IEEE 6 Dec 2022
2. Automated email Generation for Targeted Attacks using Natural Language.Avisha Das and Rakesh M. Verma.[pdf]arXiv 19 Aug 2019

A.21 Description

1. Towards Knowledge-Based Personalized Product Description Generation in E-commerce.Qibin Chen, Junyang Lin, Yichang Zhang, Hongxia Yang, Jingren Zhou and Jie Tang.[pdf]arXiv 5 Jun 2019
2. Probing Product Description Generation via Posterior Distillation.Haolan Zhan, Hainan Zhang, Hongshen Chen, Lei Shen, Zhuoye Ding, Yongjun Bao, Weipeng P. Yan and Yanyan Lan.[pdf]arXiv 2 Mar 2021
3. Automatic Generation of Pattern-controlled Product Description in E-commerce.Zhang Tao, Jin Zhang, Chengfu Huo and Weijun Ren.[doi.org]ACM 13 May 2019
4. Description Generation for Points of Interest.Meng Zhou, Jingbo Zhou, Yanjie Fu, Z. Ren, Xiaoli Wang and Hui Xiong.[doi.org]IEEE Apr 2021

A.22 Code

B. Constraints Perspective

1. Fixed global memory for controllable long text generation. Zheng Chen, Zhejun Liu. [pdf]. Applied Intelligence, 2022.
2. Critic-Guided Decoding for Controlled Text Generation.Minbeom Kim, Hwanhee Lee, Kang Min Yoo, Joonsuk Park, Hwaran Lee, Kyomin Jung, [pdf]Arxiv 21 Dec 2022
3. MDM: Meta diffusion model for hard-constrained text generation.Wenjun Ke, Yikai Guo, Qi Liu, Wanyi Chen, Peng Wang, Haoran Luo and Zhizhao Luo.[pdf]ScienceDirect Nov 2023

C. Technique (Method) Perspective

C.1 Data Augmentation

1. Data augmentation in natural language processing: a novel text generation approach for long and short text classifers.Markus Bayer, Marc‑André Kaufhold, Björn Buchhold, Marcel Keller, Jörg Dallmeyer and Christian Reuter.[pdf] International Journal of Machine Learning and Cybernetics (2023).
2. Augmented Language Models: a Survey Grégoire Mialon， Roberto Dessì， Maria Lomel [pdf] Arxiv Feb 15 2023

C.2 Detector

1. A Survey on LLM-generated Text Detection: Necessity, Methods, and Future Directions.Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Derek F. Wong Senior Member, IEEE and Lidia S. Chao Member, IEEE. [pdf] Arxiv Oct 24, 2023.

C.3 Instruction Tuning

1. LongForm: Optimizing Instruction Tuning for Long Text Generation with Corpus Extraction.Abdullatif Köksal, Timo Schick, Anna Korhonen, Hinrich Schütze [pdf] Arxiv *17 Apr 2023*.

C.4 Adversarial Training

1. Long Text Generation via Adversarial Training with Leaked Information.Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, Jun Wang, [pdf]Arxiv 8 Dec 2017.
2. Improving Adversarial Text Generation by Modeling the Distant Future.Ruiyi Zhang, Changyou Chen, Zhe Gan, Wenlin Wang, Dinghan Shen, GuoyinWang, Zheng Wen, Lawrence Carin, [pdf]ACL 4 May 2020.
3. Diversity regularized autoencoders for text generation.Hyeseon Ko, Junhyuk Lee, Jinhong Kim, Jongwuk Lee and Hyunjung Shim.[pdf]ACM Mar 2020
4. 融合自注意力机制的长文本生成对抗网络模型.夏鸿斌， 肖奕飞 and 刘渊.[pdf]计算机科学与探索 2022 16(7)
5. Feature-aware conditional GAN for category text generation.Xinze Li, Kezhi Mao, Fanfan Lin and Zijian Feng.[pdf]ScienceDirect May 2023
6. WordIllusion: An adversarial text generation algorithm based on human cognitive system.Haoran Fu, Chundong Wang, Jiaqi Sun, Yumeng Zhao, Hao Lin, Junqing Sun and Baixue Zhang.[pdf]ScienceDirect Oct 2023

C.5 Task-adaptive Tokenization

1. Enhancing Long-form Text Generation in Mental Health with Task-adaptive Tokenization.Siyang Liu, Naihao Deng, Sahand Sabour, Yilin Jia, Minlie Huang, Rada Mihalcea, [pdf]Arxiv 23 Oct 2023.

C.6 Graph-based

1. Graph-based Multi-hop Reasoning for Long Text Generation.Liang Zhao, Jingjing Xu, Junyang Lin, Yichang Zhang, Hongxia Yang, Xu Sun, [pdf]Arxiv 28 Sep 2020.
2. Text Generation from Knowledge Graphs with Graph Transformers.Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi, [pdf]ACL 1 Apr 2019.
3. GGP: A Graph-based Grouping Planner for Explicit Control of Long Text Generation. Xuming Lin, Shaobo Cui, Zhongzhou Zhao, Wei Zhou, Ji Zhang and Haiqing Chen.[pdf]ACM Oct 2021
4. NGEP: A Graph-based Event Planning Framework for Story Generation.Chen Tang, Zhihao Zhang, Tyler Loakman, Chenghua Lin and Frank Guerin.[pdf]ACL 2022

C.7 Active Learning

1. Active Learning for Natural Language Generation.Yotam Perlitz, Ariel Gera, Michal Shmueli-Scheuer, Dafna Sheinwald, Noam Slonim, Liat Ein-Dor, [pdf]Arxiv 17 Oct 2023.

C.8 Model Criticism

1. Model Criticism for Long-Form Text Generation.Yuntian Deng, Volodymyr Kuleshov, Alexander M. Rush, [pdf]ACL 16 Oct 2022.

C.9 Planning

1. DYPLOC: Dynamic Planning of Content Using Mixed Language Models for Text Generation. Xinyu Hua, Ashwin Sreevatsa and Lu Wang. [pdf]. ACL 2021.
2. Improving Text Generation via Neural Discourse Planning.Alexander Chernyavskiy.[pdf]ACM Feb 2022
3. Knowledge-based Review Generation by Coherence Enhanced Text Planning.Junyi Li, Wayne Xin Zhao, Zhicheng Wei, Nicholas Jing Yuan and Ji-Rong Wen.[pdf]ACM July 2021

C.10 Text Embedding for Long Input Tokens

1. Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents.Michael Günther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, Maximilian Werk, Nan Wang, Han Xiao.[pdf]ArXiv 30 Oct 2023.

C.11 Diffusion

1. AR-DIFFUSION: Auto-Regressive Diffusion Model for Text Generation. Tong Wu, Zhihao Fan, Xiao Liu, Yeyun Gong, Yelong Shen, Jian Jiao, Hai-Tao Zheng, Juntao Li, Zhongyu Wei, Jian Guo, Nan Duan and Weizhu Chen. [pdf]. NeurIPS 2023.

D. Model Perspective

D.1 Language Models

1. A Survey of Large Language Models. Wayne Xin Zhao, Kun Zhou and Junyi Li. [pdf]. arXiv 31 Mar, 2023.
2. Long and Diverse Text Generation with Planning-based Hierarchical Variational Model.Zhihong Shao, Minlie Huang, Jiangtao Wen, Wenfei Xu, Xiaoyan Zhu, [pdf]Arxiv *25 Aug 2019*.
3. Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence.Jian Guan, Xiaoxi Mao, Changjie Fan, Zitao Liu, Wenbiao Ding, and Minlie Huang, [pdf]Arxiv 19 May 2021.
4. Coherent Long Text Generation by Contrastive Soft Prompt.Guandan Chen, Jiashu Pu, Yadong Xi, Rongsheng Zhang, [pdf]ACL 7 Dec 2022.
5. LONGLORA: EFFICIENT FINE-TUNING OF LONG CONTEXT LARGE LANGUAGE MODELS. Yukang Chen, Shengju Qian, Zhijian Liu, Haotian Tang, Song Han and Jiaya Jia.[pdf]arXiv 5 Dec 2023

D.2 Pretrained Models

1. Progressive Generation of Long Text with Pretrained Language Models.Bowen Tan, Zichao Yang, Maruan Al-Shedivat, Eric P. Xing, Zhiting Hu, [pdf]NAACL 11 Jun 2021.
2. A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation.Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, Minlie Huang, [pdf]TACL 1 Jan 2020.
3. DIALOGLM: Pre-trained Model for Long Dialogue Understanding and Summarization.Ming Zhong, Yang Liu, Yichong Xu, Chenguang Zhu, Michael Zeng, [pdf]Arxiv 6 Jan 2022.
4. Adapting Pretrained Text-to-Text Models for Long Text Sequences.Wenhan Xiong, Anchit Gupta, Shubham Toshniwal, Yashar Mehdad and Wen-tau Yih.[pdf]arXiv 2022

D.3 Combination of RNN and Transformer

1. RECURRENTGPT:Interactive Generation of (Arbitrarily) Long Text.Wangchunshu Zhou, Yuchen Eleanor Jiang, Peng Cui Tiannan Wang, Zhenxin Xiao, Yifan Hou, Ryan Cotterell, Mrinmaya Sachan, ETH Zürich, [pdf]Arxiv 22 May 2023

D.4 Transformer

1. Text Generation from Knowledge Graphs with Graph Transformers.Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi, [pdf]ACL 1 Apr 2019.
2. LongT5 Efficient Text-To-Text Transformer for Long Sequences.Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontañón, Jianmo Ni, Yun-Hsuan Sung, Yinfei Yang, [pdf]Arxiv 3 May 2022.
3. DISCODVT: Generating Long Text with Discourse-Aware Discrete Variational Transformer.Haozhe Ji, Minlie Huang, [pdf]ACL 12 Oct 2021.
4. Generating Long Sequences with Sparse Transformers.Rewon Child, Scott Gray, Alec Radford and Ilya Sutskever.[pdf]arXiv 2019
5. Segmented Recurrent Transformer: An Efficient Sequence-to-Sequence Model.Yinghan Long, Sayeed Shafayet Chowdhury and Kaushik Roy.[pdf]arXiv 23 Oct 2023
6. Fixed global memory for controllable long text generation.Zheng Chen and Zhejun Liu.[pdf]Springer 20 Oct 2022

D.5 RNN (LSTM)

1. Research on Text Generation Based on LSTM.Lifen Li, Tianyu Zhang, [pdf]International Core Journal of Engineering Volume 7 Issue 5, 2021.
2. A method of automatic text summarisation based on long short-term memory.Wei Fang, TianXiao Jiang, Ke Jiang, Feihong Zhang, Yewen Ding and Jack Sheng, [pdf]International Journal of Computational Science and Engineering 4 May 2020.

D.6 CNN

1. A Hybrid Convolutional Variational Autoencoder for Text Generation. Stanislau Semeniuta, Aliaksei Severyn and Erhardt Barth. [pdf]. EMNLP 2017.

D.7 Different semantic granularity

1. 面向不同语义粒度约束的文本生成方法研究.潘囿丞[pdf]CNKI July 2022

E. Reading Papers

1. Long Text Generation Challenge. Nikolay Mikhaylovskiy. [pdf].arXiv 4 June 2023.

F. Long Text Generation Evaluation Methods and Metrics

1. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer and Hannaneh Hajishirzi. [pdf]. [code]. EMNLP 2023.

G. Research Hotpot, Trend and Challenge of Long Text Generation

G.1 The Combination of Agent and Long Text Generation

G.2 Long Text Generation with LLMs (above 13B)

G.2 Hallucination in Long Text Generation

G.3 Model Lossless Compression for Long Text Generation

i. Quantization
ii. Distillation
iii. Pruning

G.4 Efficient Parameters Learning for Long Text Generation

i. LoRA
ii. Flash Attention
iii. Sparse Attenation
iv. KV Quantization & Storage

G.5 Context Size Extension (Input Token Length) for Long Text Generation

G.6 Reasoning in Long Text Generation

G.7 Chain-of-Thought in Long Text Generation

i. Reflect CoT
ii. Tree-of-Thought
iii. Graph-of-Thought

G.8 Long Text Contraint Generation

i. Multi-Styles Long Text Generation
ii. Multi-Tasks Long Text Generation
iii. Controllable Long Text Generation

G.9 Interactive Long Text Generation

i. Editable Long Text Generation
ii. Chat to Generate Long Text

G.10 Mutli-modal Text Generation

i. Image to Long Text Generation
ii. Video to Long Text Generation

G.11 Align with Human for Long Text Generation

i. RLHF
ii RLAIF
iii. RLMoEF

G.12 RAG Enhanced Long Text Generation

Project Maintainers & Contributors

• Junwen Zhang (@Fendi)
• Yuanhao Lou (@Yuanhao)
• Shuang Chen (@Chen Shuang)

Contact

• Junwen Zhang: junwenzhang@zju.edu.cn