Superpoint Gaussians Splatting for Real-Time High-Fidelity Monocular Dynamic Scene Reconstruction

Project page | Paper

This repository contains the official implementation associated with the paper "Superpoint Gaussians Splatting for Real-Time High-Fidelity Monocular Dynamic Scene Reconstruction".

Dataset

In our paper, we use:

• synthetic dataset from D-NeRF
• real-world dataset from NeRF-DS and Hyper-NeRF .

We organize the datasets as follows:

├── data
│   ├── D-NeRF 
│   │  ├── hook
│   │  ├── standup 
│   │  ├── ...
│   ├── NeRF-DS
│   │  ├── as
│   │  ├── basin
│   │  ├── ...
│   ├── HyperNeRF
│   │  ├── interp
│   │  ├── misc
│   │  ├── virg

Pipeline

Run

Environment

git clone https://github.com/dnvtmf/SP_GS --recursive
cd SP_GS

conda create -n sp_gs_env  # python>=3.8
conda activate sp_gs_env

# install pytorch
pip install torch torchvision torchaudio

# install dependencies
pip install -r requirements.txt

Train

python train.py -s <path/to/your/dataset> -m <output/exp-name> --eval

Render & Evaluation

python render.py -m <output/exp-name>
python metrics.py <-m output/exp-name>

We provide several modes for rendering:

• render: render all the test images
• time: time interpolation tasks for D-NeRF dataset
• all: time and view synthesis tasks for D-NeRF dataset
• view: view synthesis tasks for real-world dataset
• original: time and view synthesis tasks for real-world dataset

Visualization

python gui_edit.py

Then, click Add Model button to choose trained model, you can choose

• ".ply" file (e.g., "output/NeRF_DS/as/point_cloud/iteration_40000/point_cloud.ply),
• ".pth" file (e.g., "output/NeRF_DS/as/deform/iteration_40000/deform.pth")

Next, to use the camera of training dataset, first click on the heading camera, and then click on the button use.

BibTex

@InProceedings{SP-GS,
  title = 	 {Superpoint Gaussian Splatting for Real-Time High-Fidelity Dynamic Scene Reconstruction},
  author =       {Wan, Diwen and Lu, Ruijie and Zeng, Gang},
  booktitle = 	 {Proceedings of the 41st International Conference on Machine Learning},
  pages = 	 {49957--49972},
  year = 	 {2024},
}

And thanks to the authors of Deformable-3D-Gaussians and 3D Gaussians for their excellent code, please consider also cite this repository:

@article{yang2023deformable3dgs,
    title={Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction},
    author={Yang, Ziyi and Gao, Xinyu and Zhou, Wen and Jiao, Shaohui and Zhang, Yuqing and Jin, Xiaogang},
    journal={arXiv preprint arXiv:2309.13101},
    year={2023}
}

@Article{kerbl3Dgaussians,
      author       = {Kerbl, Bernhard and Kopanas, Georgios and Leimk{\"u}hler, Thomas and Drettakis, George},
      title        = {3D Gaussian Splatting for Real-Time Radiance Field Rendering},
      journal      = {ACM Transactions on Graphics},
      number       = {4},
      volume       = {42},
      month        = {July},
      year         = {2023},
      url          = {https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/}
}