Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting

Project Page | Paper | Unofficial Implementation

Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting
Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, Sida Peng
ECCV 2024

street_gaussians.mp4

Installation

Clone this repository

git clone https://github.com/zju3dv/street_gaussians.git

Set up the python environment

# Set conda environment
conda create -n street-gaussian python=3.8
conda activate street-gaussian

# Install torch (corresponding to your CUDA version)
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116

# Install requirements
pip install -r requirements.txt

# Install submodules
pip install ./submodules/diff-gaussian-rasterization
pip install ./submodules/simple-knn
pip install ./submodules/simple-waymo-open-dataset-reader
python script/test_gaussian_rasterization.py

Prepare Waymo Open Dataset.

We provide the example scenes here. You can directly download the data and skip the following steps for a quick start.

Download the training and validation set of Waymo Open Dataset.

We provide the split file following EmerNeRF. You can refer to this document for download details.

Preprocess the data

Download the tracking predictions on validation set, We provide the processed results here.

Preprocess the example scenes

python script/waymo/waymo_converter.py --root_dir TRAINING_SET_DIR --save_dir SAVE_DIR --split_file script/waymo/waymo_splits/demo.txt --segment_file script/waymo/waymo_splits/segment_list_train.txt

Preprocess the experiment scenes

python script/waymo/waymo_converter.py --root_dir VALIDATION_SET_DIR --save_dir SAVE_DIR --split_file script/waymo/waymo_splits/val_dynamic.txt --segment_file script/waymo/waymo_splits/segment_list_val.txt
--track_file TRACKER_PATH

Generating LiDAR depth

python script/waymo/generate_lidar_depth.py --datadir DATA_DIR

Generating sky mask

Install GroundingDINO following this repo and download SAM checkpoint from this link.

python script/waymo/generate_sky_mask.py --datadir DATA_DIR --sam_checkpoint SAM_CKPT

Prepare Custom Dataset.

TODO

Training

python train.py --config configs/xxxx.yaml

Training on example scenes

bash script/waymo/train_waymo_expample.sh

Training on experiment scenes

bash script/waymo/train_waymo_exp.sh

Rendering

python render.py --config configs/xxxx.yaml mode {evaluate, trajectory}

Rendering on example scenes

bash script/waymo/render_waymo_expample.sh

Rendering on experiment scenes

bash script/waymo/render_waymo_exp.sh

Visualization

You can convert the scene at one certain frame into the format that can be viewed in SIBR_viewers.

python make_ply.py --config configs/xxxx.yaml viewer.frame_id {frame_idx} mode evaluate

Pipeline

Citation

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{yan2024street,
    title={Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting}, 
    author={Yunzhi Yan and Haotong Lin and Chenxu Zhou and Weijie Wang and Haiyang Sun and Kun Zhan and Xianpeng Lang and Xiaowei Zhou and Sida Peng},
    booktitle={ECCV},
    year={2024}
}