We build object-centric autonomous driving systems based on Gaussians!

This repository contains the implementation of the following methods.

GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction
Yuanhui Huang, Wenzhao Zheng^*, Yunpeng Zhang, Jie Zhou, Jiwen Lu⁺
European Conference on Computer Vision (ECCV), 2024
Paper | Project Page

GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Prediction
Yuanhui Huang, Amonnut Thammatadatrakoon, Wenzhao Zheng^*, Yunpeng Zhang, Dalong Du, Jiwen Lu
Paper

^* Project leader. ⁺ Corresponding author.

News.

• [2024/12/05] GaussianFormer-2 code release.
• [2024/09/30] Occupancy and Gaussian visualization code release.
• [2024/09/12] Training code release.
• [2024/09/05] An updated version of GaussianFormer modeling only the occupied area.
• [2024/09/05] Model weights and evaluation code release.
• [2024/07/01] GaussianFormer is accepted to ECCV24!
• [2024/05/28] Paper released on arXiv.
• [2024/05/28] Demo release.

GaussianFormer

GaussianFormer proposes the 3D semantic Gaussians as a more efficient object-centric representation for driving scenes compared with 3D occupancy.

Considering the universal approximating ability of Gaussian mixture, we propose an object-centric 3D semantic Gaussian representation to describe the fine-grained structure of 3D scenes without the use of dense grids. We propose a GaussianFormer model consisting of sparse convolution and cross-attention to efficiently transform 2D images into 3D Gaussian representations. To generate dense 3D occupancy, we design a Gaussian-to-voxel splatting module that can be efficiently implemented with CUDA. With comparable performance, our GaussianFormer reduces memory consumption of existing 3D occupancy prediction methods by 75.2% - 82.2%.

GaussianFormer-2

We propose a probabilistic Gaussian superposition model which interprets each Gaussian as a probability distribution of its neighborhood being occupied and conforms to probabilistic multiplication to derive the overall geometry. Furthermore, we adopt the exact Gaussian mixture model for semantics calculation to avoid unnecessary overlapping of Gaussians. To effectively initialize Gaussians in non-empty region, we design a distribution-based initialization module which learns the pixel-aligned occupancy distribution instead of the depth of surfaces. We conduct extensive experiments on nuScenes and KITTI-360 datasets and our GaussianFormer-2 achieves state-of-the-art performance with high efficiency.

Getting Started

Installation

Follow instructions HERE to prepare the environment.

Data Preparation

1. Download nuScenes V1.0 full dataset data HERE.

2. Download the occupancy annotations from SurroundOcc HERE and unzip it.

3. Download pkl files HERE.

Folder structure

GaussianFormer
├── ...
├── data/
│   ├── nuscenes/
│   │   ├── maps/
│   │   ├── samples/
│   │   ├── sweeps/
│   │   ├── v1.0-test/
|   |   ├── v1.0-trainval/
│   ├── nuscenes_cam/
│   │   ├── nuscenes_infos_train_sweeps_occ.pkl
│   │   ├── nuscenes_infos_val_sweeps_occ.pkl
│   │   ├── nuscenes_infos_val_sweeps_lid.pkl
│   ├── surroundocc/
│   │   ├── samples/
│   │   |   ├── xxxxxxxx.pcd.bin.npy
│   │   |   ├── ...

Inference

We provide the following checkpoints trained on the SurroundOcc dataset:

Name	Type	#Gaussians	mIoU	Config	Weight
Baseline	GaussianFormer	144000	19.10	config	weight
NonEmpty	GaussianFormer	25600	19.31	config	weight
Prob-64	GaussianFormer-2	6400	20.04	config	weight
Prob-128	GaussianFormer-2	12800	20.08	config	weight
Prob-256	GaussianFormer-2	25600	20.33	config	weight

python eval.py --py-config config/xxxx.py --work-dir out/xxxx/ --resume-from out/xxxx/state_dict.pth

Train

Download the pretrained weights for the image backbone HERE and put it inside ckpts.

python train.py --py-config config/xxxx.py --work-dir out/xxxx

Stay tuned for more exciting work and models!🤗

Visualize

Install packages for visualization according to the documentation. Here is an example command where you can change --num-samples and --vis-index.

CUDA_VISIBLE_DEVICES=0 python visualize.py --py-config config/nuscenes_gs25600_solid.py --work-dir out/nuscenes_gs25600_solid --resume-from out/nuscenes_gs25600_solid/state_dict.pth --vis-occ --vis-gaussian --num-samples 3 --model-type base

Related Projects

Our work is inspired by these excellent open-sourced repos: TPVFormer PointOcc SelfOcc SurroundOcc OccFormer BEVFormer

Our code is originally based on Sparse4D and migrated to the general framework of SelfOcc.

Citation

If you find this project helpful, please consider citing the following paper:

@article{huang2024gaussian,
    title={GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction},
    author={Huang, Yuanhui and Zheng, Wenzhao and Zhang, Yunpeng and Zhou, Jie and Lu, Jiwen},
    journal={arXiv preprint arXiv:2405.17429},
    year={2024}
}
@article{huang2024probabilisticgaussiansuperpositionefficient,
      title={GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Prediction}, 
      author={Yuanhui Huang and Amonnut Thammatadatrakoon and Wenzhao Zheng and Yunpeng Zhang and Dalong Du and Jiwen Lu},
      journal={arXiv preprint arXiv:2412.04384},
      year={2024}
}