Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis

Point Cloud Visualization

 

Training Process Visualization

 

Introduction

This is the official implementation of our ECCV 2024 paper "Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis". Our X-Gaussian is SfM-free. If you find this repo useful, please give it a star ⭐ and consider citing our paper. Thank you.

News

• 2024.09.01 : Code, models, and training logs have been released. Welcome to have a try 😆
• 2024.07.01 : Our X-Gaussian has been accepted by ECCV 2024! Code will be released before the start date of the conference (2024.09.29). Stay tuned. 🚀
• 2024.06.03 : Code for traditional methods has been released at SAX-NeRF. ✨
• 2024.06.03 : Code for fancy visualization and data generation has been released at SAX-NeRF. 🚀
• 2024.06.03 : Data, code, models, and training logs of our CVPR 2024 work SAX-NeRF have been released. Feel free to use them :)
• 2024.06.03 : The datasets have been released on Google Drive. Feel free to use them. 🚀
• 2024.03.07 : Our paper is on arxiv now. Code, models, and training logs will be released. Stay tuned. 💫

Performance

Novel View Synthesis

CT Reconstruction

Coordinate System

The coordinate system in circular cone-beam X-ray scanning follows the OpenCV standards. The transformation between the camera, world, and image coordinate systems is shown below.

 

1. Create Environment:

We recommend using Conda to set up the environment.

# cloning our repo
git clone https://github.com/caiyuanhao1998/X-Gaussian --recursive


SET DISTUTILS_USE_SDK=1 # Windows only

# install the official environment of 3DGS
cd X-Gaussian
conda env create --file environment.yml
conda activate x_gaussian

# Use our X-ray rasterizer package to replace the original RGB rasterizer
rm -rf submodules/diff-gaussian-rasterization/cuda_rasterizer
mv cuda_rasterizer submodules/diff-gaussian-rasterization/

# re-install the diff-gaussian-rasterization package
pip install submodules/diff-gaussian-rasterization

 

2. Prepare Dataset

Download our processed datasets from Google drive or Baidu disk. Then put the downloaded datasets into the folder data/ as

  |--data
      # The first five datasets are used in our paper
      |--chest_50.pickle
      |--abdomen_50.pickle
      |--foot_50.pickle
      |--head_50.pickle
      |--pancreas_50.pickle
      # The rest datasets are from the X3D benchmark
      |--aneurism_50.pickle
      |--backpack_50.pickle
      |--bonsai_50.pickle
      |--box_50.pickle
      |--carp_50.pickle
      |--engine_50.pickle
      |--leg_50.pickle
      |--pelvis_50.pickle
      |--teapot_50.pickle
      |--jaw_50.pickle

Note: The first five datasets are used to do experiments in our paper. The rest datasets are from the X3D benchmark. Please also note that the pickle data used by X-Gaussian is dumped/read by pickle protocol 4, which is supported by python < 3.8. The original X3D data is processed by pickle protocol 5, which is supported by python >= 3.8. I have re-dumped the pickle data from the original X3D datasets to make sure you can run our code without extra effort. If you want to re-dump the pickle data, please run

python pickle_redump.py

 

3. Training and Testing

You can download our trained Gaussian point clouds from Google Drive or Baidu Disk (code: cyh2) as

We share the training log for your convienience to debug. Please download them from Google Drive or Baidu Disk (code: cyh2). To make the training and evaluation easier, your can directly run the train.sh file by

bash train.sh

Or you can separately train on each scene like

python3 train.py --config config/chest.yaml --eval

python3 train.py --config config/foot.yaml --eval

python3 train.py --config config/abdomen.yaml --eval

python3 train.py --config config/head.yaml --eval

python3 train.py --config config/pancreas.yaml --eval

python3 train.py --config config/jaw.yaml --eval

python3 train.py --config config/pelvis.yaml --eval

python3 train.py --config config/aneurism.yaml --eval

python3 train.py --config config/carp.yaml --eval

python3 train.py --config config/bonsai.yaml --eval

python3 train.py --config config/box.yaml --eval

python3 train.py --config config/backpack.yaml --eval

python3 train.py --config config/engine.yaml --eval

python3 train.py --config config/leg.yaml --eval

python3 train.py --config config/teapot.yaml --eval

 

4. Visualization

We also provide code for the visualization of rotating the Gaussian point clouds

python point_cloud_vis.py

 

5. Citation

# X-Gaussian
@inproceedings{x_gaussian,
  title={Radiative gaussian splatting for efficient x-ray novel view synthesis},
  author={Yuanhao Cai and Yixun Liang and Jiahao Wang and Angtian Wang and Yulun Zhang and Xiaokang Yang and Zongwei Zhou and Alan Yuille},
  booktitle={ECCV},
  year={2024}
}

# sax-nerf
@inproceedings{sax_nerf,
  title={Structure-Aware Sparse-View X-ray 3D Reconstruction},
  author={Yuanhao Cai and Jiahao Wang and Alan Yuille and Zongwei Zhou and Angtian Wang},
  booktitle={CVPR},
  year={2024}
}

 

Acknowledgement

Our code and data are heavily borrowed from SAX-NeRF and 3DGS