Awesome LLMs for World Models and Physical Understanding
Explore the docs »

Topics covered

• About
• World Models
• Large Language Models for World Understanding
• Multimodal Generative World Models
• Benchmarks
• Miscellaneous
• Acknowledgement
• Contributing
• Authors & contributors
• License

---

About

With the recent success of Large Language Models (LLMs) in Natural Language Understanding (NLU) and Natural Language Generation (NLG) tasks, there is growing interest in evaluating their capabilities beyond traditional NLP applications. One key area of exploration is assessing how well LLMs understand the world around us (can LLMs act as world models?). This understanding enables the use of LLMs for general planning and physical reasoning tasks. Below, you'll find a curated list of papers and code repositories focused on this topic.

World Models

Paper	Link	Code	Venue	Date	Other
World Models	arXiv	--	NeurIPS Oral	27 Mar 2018	project page
Learning to Model the World with Language	arXiv	GitHub	ICML Oral	31 May 2024	project page
Mastering Memory Tasks with World Models	arXiv	GitHub	ICLR Oral	7 Mar 2024
[Dreamer-V3] Mastering Diverse Domains through World Models	arXiv	GitHub	arxiv	10 Jan 2023	project page

Large Language Models for World Understanding

Paper	Link	Code	Venue	Date	Other
Language Models Meet World Models: Embodied Experiences Enhance Language Models	arXiv	GitHub	NeurIPS	28 Oct 2023
Evaluating the World Model Implicit in a Generative Model	arXiv	--	arXiv	22 Jun 2024
Reasoning with Language Model is Planning with World Model	arXiv	GitHub	EMNLP	23 Oct 2023
Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning	arXiv	GitHub	NeurIPS	2 Nov 2023	project page
Can Language Models Serve as Text-Based World Simulators?	arXiv	--	--	10 Jun 2024
Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task	arXiv	GitHub	ICLR Top 5%	26 Jun 2024
From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought	arXiv	GitHub	arXiv	23 Jun 2023	Youtube - Wong and Gabriel YouTube - Josh
Learning adaptive planning representations with natural language guidance	arXiv	--	ICLR	13 Dec 2023

Multimodal Generative World Models

Paper	Link	Code	Venue	Date	Other
Pandora: Towards General World Model with Natural Language Actions and Video States	arXiv	GitHub	arXiv	12 Jun 2024
Learning Interactive Real-World Simulators	arXiv	--	ICLR outstanding paper	13 Jan 2024	project page

Benchmarks

Paper	Link	Code	Venue	Date	Other
Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answering	arXiv	--	arXiv	2 Jun 2024
ContPhy: Continuum Physical Concept Learning and Reasoning from Videos	arXiv	GitHub	ICML	9 Feb 2024	project page
STAR: A Benchmark for Situated Reasoning in Real-World Videos	arXiv	GitHub	NeurIPS	2021
MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities	arxiv	GitHub	ICML 2024	4 Aug 2023
OpenEQA: From word models to world models	Paper	GitHub	Preprint	2024	project page, Meta blog
CityBench: Evaluating the Capabilities of Large Language Model as World Model	arXiv	--	arXiv	20 Jun 2024

Miscellaneous

Paper	Link	Code	Venue	Date	Other
OpenVLA: An Open-Source Vision-Language-Action Model	arXiv	GitHub	arXiv	13 Jun 2024	project page
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models	arXiv	GitHub	ICML	06 Oct 2023
Building Cooperative Embodied Agents Modularly with Large Language Models	arXiv	GitHub	ICLR	05 Jul 2023	project page
True Knowledge Comes from Practice: Aligning LLMs with Embodied Environments via Reinforcement Learning	arxiv	GitHub	ICLR	25 Jan 2024

Acknowledgement

If you want to say thank you or/and support active development of Awesome LLMs for World Models and Physical Understanding:

• Add a GitHub Star to the project.
• Tweet about the Awesome LLMs for World Models and Physical Understanding.
• Write interesting articles about the project on Dev.to, Medium or your personal blog.

Together, we can make Awesome LLMs for World Models and Physical Understanding better!

Contributing

First off, thanks for taking the time to contribute! Contributions are what make the open-source community such an amazing place to learn, inspire, and create. Any contributions you make will benefit everybody else and are greatly appreciated.

Authors & contributors

The original setup of this repository is by Shubham Parashar.

For a full list of all authors and contributors, see the contributors page.

License

This project is licensed under the MIT license.

See LICENSE for more information.