S³Gaussian: Self-Supervised Street Gaussians for Autonomous Driving

Paper | Project Page

S³Gaussian: Self-Supervised Street Gaussians for Autonomous Driving

Nan Huang*, Xiaobao Wei, Wenzhao Zheng$^\dagger$, Pengju An, Ming Lu, Wei Zhan, Masayoshi Tomizuka, Kurt Keutzer, Shanghang Zhang$^\ddagger$

* Work done while interning at UC Berkeley $\dagger$ Project leader $\ddagger$ Corresponding author

S³Gaussian employs 3D Gaussians to model dynamic scenes for autonomous driving without other supervisions (e.g., 3D bounding boxes).

News

• [2023/5/31] Training & evaluation code release!
• [2024/5/31] Paper released on arXiv.

Demo

Overview

To tackle the challenges in self-supervised street scene decomposition, we propose a multi-resolution hexplane-based encoder to encode 4D grid into feature planes and a multi-head Gaussian decoder to decode them into deformed 4D Gaussians. We optimize the overall model without extra annotations in a self-supervised manner and achieve superior scene decomposition ability and rendering quality.

Results

Getting Started

Environmental Setups

Our code is developed on Ubuntu 22.04 using Python 3.9 and pytorch=1.13.1+cu116. We also tested on pytorch=2.2.1+cu118. We recommend using conda for the installation of dependencies.

git clone https://github.com/nnanhuang/S3Gaussian.git --recursive
cd S3Gaussian
conda create -n S3Gaussian python=3.9 
conda activate S3Gaussian

pip install -r requirements.txt
pip install -e submodules/depth-diff-gaussian-rasterization
pip install -e submodules/simple-knn

Preparing Dataset

Follow detailed instructions in Prepare Dataset.

We only use dynamic32 and static32 split.

Training

For training first clip (eg. 0-50 frames), run

python train.py -s $data_dir --port 6017 --expname "waymo" --model_path $model_path 

If you want to try novel view synthesis, use

--configs "arguments/nvs.py"

For instance, you can try:

python train.py -s "./data/processed/dynamic32/training/022" --expname "waymo" --model_path "./work_dirs/phase1/dynamic/recon/022"

For training next clip (eg. 51-100 frames), run

python train.py -s $data_dir --port 6017 --expname "waymo" --model_path $model_path --prior_checkpoint "$prior_dir/chkpnt_fine_50000.pth" --configs "arguments/stage2.py"

For instance, you can try:

python train.py -s "./data/processed/dynamic32/training/022" --expname "waymo" --model_path "./work_dirs/phase1/dynamic/recon/p2/022" --prior_checkpoint "./work_dirs/phase1/dynamic/recon/022/chkpnt_fine_50000.pth" --configs "arguments/stage2.py"

Also, you can load an existing checkpoint with:

python train.py -s $data_dir --port 6017 --expname "waymo" --start_checkpoint "$ckpt_dir/chkpnt_fine_30000.pth" --model_path $model_path 

For more scripts examples, please check here.

Evaluation and Visualization

You can visualize and eval a checkpoints follow:

python train.py -s $data_dir --port 6017 --expname "waymo" --start_checkpoint "$ckpt_dir/chkpnt_fine_50000.pth" --model_path $model_path --eval_only

If you use different configs, you will need to add them as well:

--configs "arguments/nvs.py"

Then you can get rendering RGB videos, ground truth RGB videos, depth videos, dynamic rgb videos and static rgb videos.

Acknowledgments

Credits to @Korace0v0 for building 3D Gaussians for street scenes. Many thanks!

Special thanks to StreetGaussians for sharing visualization results!

Our code is based on 4D Gaussians and EmerNeRF.

Thanks to these excellent open-sourced repos!

Citation

If you find this project helpful, please consider citing the following paper:

@article{huang2024s3gaussian,
        title={S3Gaussian: Self-Supervised Street Gaussians for Autonomous Driving},
        author={Huang, Nan and Wei, Xiaobao and Zheng, Wenzhao and An, Pengju and Lu, Ming and Zhan, Wei and Tomizuka,    Masayoshi and Keutzer, Kurt and Zhang, Shanghang},
        journal={arXiv preprint arXiv:2405.20323},
        year={2024}
      }