Large Driving Models

While most existing methods focus on adapting driving tasks to pre-trained large language models or vision-language models (Large Models for Autonomous Driving), we design a series of Large Driving Models specifically for autonomous driving.

Large Driving Model Zoo (Literally!)

Model	Function	Task	Core Contributor	Code	Release Data	Why the name?
Stereo Anything	Large Stereo Model	Stereo-based Depth Estimation	Xianda Guo	https://github.com/XiandaGuo/OpenStereo	2024/11/22	Stereo Anything
Stag-1	Large Simulation Model	4D Photorealistic Simulation	Lening Wang	https://github.com/wzzheng/Stag	2024/12/9	Spatial-Temporal simulAtion for drivinG
Driv3R	Large Reconstruction Model	Pose-free Dense Reconstruction	Xin Fei	https://github.com/Barrybarry-Smith/Driv3R	2024/12/10	DRIVing 3d Reconstruction
GPD-1	Latent World Model	Close-Loop Simulation, Planning, Scene Generation...	Zixun Xie	https://github.com/wzzheng/GPD	2024/12/12	Generative Pre-training for Driving
Doe-1	Large World Model	End-to-End Perception, Prediction, Planning...	Zetian Xia	https://github.com/wzzheng/Doe	2024/12/13	Driving wOrld modEl
DrivingRecon	Large Gaussian Model	Feed-Forward 4D Gaussian Reconstruction	Hao Lu	https://github.com/EnVision-Research/DriveRecon	2024/12/13	Driving Reconstruction
Owl-1	Video Generation Model	End-to-End Planning and Generation	Yuanhui Huang	https://github.com/huang-yh/Owl	2024/12/13	Omni World modeL

Object-Centric Autonomous System

Model	Scenario	Task	Core Contributor	Code	Release Data
GaussianFormer	Outdoor	Multi-View 3D Occupancy Prediction	Yuanhui Huang	https://github.com/huang-yh/GaussianFormer	2024/5/27
GaussianFormer-2	Outdoor	Multi-View 3D Occupancy Prediction	Yuanhui Huang	https://github.com/huang-yh/GaussianFormer	2024/12/6
EmbodiedOcc	Indoor	Embodied 3D Occupancy Prediction	Yuqi Wu	https://github.com/YkiWu/EmbodiedOcc	2024/12/6
GaussianWorld	Outdoor	Streaming 3D Occupancy Prediction	Sicheng Zuo	https://github.com/zuosc19/GaussianWorld	2024/12/16
GaussianAD	Outdoor	End-to-End Autonomous Driving	Junjie Wu	https://github.com/wzzheng/GaussianAD	2024/12/16

Demos

Stag-1: Feed-Forward 4D Photorealistic Simulation

Freeze Time

Freeze View

Demo 3: Multi-View

Driv3R: Pose-free Dense Reconstruction

DrivingRecon: Feed-Forward 4D Gaussian Reconstruction

GPD-1: All-in-One Model for Autonomous Driving Simulation

Doe-1: Closed-Loop Autonomous Driving

EmbodeidOcc: Online Embodied 3D Occupancy Prediction

Citations

If you find this project helpful, please consider citing the following papers:

### Stereo Anything
@article{guo2024stereo,
  title={Stereo Anything: Unifying Stereo Matching with Large-Scale Mixed Data},
  author={Guo, Xianda and Zhang, Chenming and Zhang, Youmin and Nie, Dujun and Wang, Ruilin and Zheng, Wenzhao and Poggi, Matteo and Chen, Long},
  journal={arXiv preprint arXiv:2411.14053},
  year={2024}
}

### Stag-1
@article{stag-1,
    title={Stag-1: Towards Realistic 4D Driving Simulation with Video Generation Model},
    author={Wang, Lening and Zheng, Wenzhao and Du, Dalong and Zhang, Yunpeng and Ren, Yilong and Jiang, Han and Cui, Zhiyong and Yu, Haiyang and Zhou, Jie and Lu, Jiwen and Zhang, Shanghang},
    journal={arXiv preprint arXiv:},
    year={2024}
	}

### Driv3R
@article{driv3r,
  title={Driv3R: Learning Dense 4D Reconstruction for Autonomous Driving}, 
  author={Fei, Xin and Zheng, Wenzhao and Duan, Yueqi and Zhan, Wei and Tomizuka, Masayoshi and Keutzer, Kurt and Lu, Jiwen},
  journal={arXiv preprint arXiv:2412.06777},
  year={2024}
}

### GPD-1
  @article{gpd-1,
    title={GPD-1: Generative Pre-training for Driving},
    author={Xie, Zixun and Zuo, Sicheng and Zheng, Wenzhao and Zhang, Yunpeng and Du, Dalong and Zhou, Jie and Lu, Jiwen and Zhang, Shanghang},
    journal={arXiv preprint arXiv:2412.08643},
    year={2024}
}

### Doe-1
@article{doe,
    title={Doe-1: Closed-Loop Autonomous Driving with Large World Model},
    author={Zheng, Wenzhao and Xia, Zetian and Huang, Yuanhui and Zuo, Sicheng and Zhou, Jie and Lu, Jiwen},
    journal={arXiv preprint arXiv:},
    year={2024}
}

### DrivingRecon
@article{Lu2024DrivingRecon,
        title={DrivingRecon: Large 4D Gaussian Reconstruction Model For Autonomous Driving},
        author={Hao LU, Tianshuo XU, Wenzhao ZHENG, Yunpeng ZHANG, Wei ZHAN, Dalong DU, Masayoshi Tomizuka, Kurt Keutzer, Yingcong CHEN},
        journal={arXiv preprint arXiv:2412.09043},
        year={2024}
      }

### Owl-1
@article{owl-1,
    title={Owl-1: Omni World Model for Consistent Long Video Generation}, 
    author={Huang, Yuanhui and Zheng, Wenzhao and Gao, Yuan and Tao, Xin and Wan, Pengfei and Zhang, Di and Zhou, Jie and Lu, Jiwen},
    journal={arXiv preprint arXiv:2412.09600},
    year={2024},
}

### GaussianFormer-1
@article{gaussianformer,
    title={GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction},
    author={Huang, Yuanhui and Zheng, Wenzhao and Zhang, Yunpeng and Zhou, Jie and Lu, Jiwen},
    journal={arXiv preprint arXiv:2405.17429},
    year={2024}
}

### GaussianFormer-2
@article{gaussianformer-2,
      title={GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Prediction}, 
      author={Yuanhui Huang and Amonnut Thammatadatrakoon and Wenzhao Zheng and Yunpeng Zhang and Dalong Du and Jiwen Lu},
      journal={arXiv preprint arXiv:2412.04384},
      year={2024}
}
	
### EmbodiedOcc
@article{embodiedocc,
      title={EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understanding}, 
      author={Wu, Yuqi and Zheng, Wenzhao and Zuo, Sicheng and Huang, Yuanhui and Zhou, Jie and Lu, Jiwen},
      journal={arXiv preprint arXiv:2412.04380},
      year={2024}
}