BeNeRF: Neural Radiance Fields from a Single Blurry Image and Event Stream

Wenpu Li^1,5*    Pian Wan ^1,2*    Peng Wang^1,3*    Jinghang Li⁴    Yi Zhou⁴    Peidong Liu^1†

^*equal contribution    ^† denotes corresponding author.

¹Westlake University    ²EPFL    ³Zhejiang University    ⁴Hunan University   
⁵Guangdong University of Technology

---

This paper was accepted by European Conference on Computer Vision (ECCV) 2024.

If you like our project, please give us a star ⭐ on GitHub.

This repository is an official PyTorch implementation of the paper "BeNeRF: Neural Radiance Fields from a Single Blurry Image and Event Stream". We explore the possibility of recovering the neural radiance fields and camera motion trajectory from a single blurry image. This allows BeNeRF to decode the underlying sharp video from a single blurred image.

📢 News

• 2024.09.24 We have completed the construction of the project homepage. You can access it by clicking this link.
• 2024.08.25 Testing and evaluation Code have been released.
• 2024.08.20 Training Code and datasets have been released.
• 2024.07.01 Our paper was accepted by ECCV2024!! Congratulates to all collaborators!!

📋 Overview

Given a single blurry image and its corresponding event stream, BeNeRF recovers the underlying 3D scene representation and the camera motion trajectory jointly. In particular, we represent the 3D scene with neural radiance fields and the camera motion trajectory with a cubic B-Spline in SE(3) space. Both the blurry image and accumulated events within a time interval can thus be synthesized from the 3D scene representation providing the camera poses. The camera trajectory, NeRF, are then optimized by minimizing the difference between the synthesized data and the real measurements.

🔥 QuickStart

1. Installation

In the path where your want to store code, enter the following terminal command:

git clone https://github.com/WU-CVGL/BeNeRF.git
cd BeNeRF
conda create -n benerf python=3.9
conda activate benerf
pip install -r requirements.txt

If the network speed is slow when using pip to download dependencies, you may consider changing the pip source:

python -m pip install --upgrade pip
pip config set global.index-url https://mirrors.bfsu.edu.cn/pypi/web/simple

2. Download Datasets

You can download BeNeRF_Datasets by clicking this link.

It contains real-world dataset(e.g. e2nerf_real) and synthetic dataset(benerf_blender, benerf_unreal and e2nerf_synthetic). For the every scene, the folder images includes blurry images and folder events includes event stream for entire sequence. The timestamps of the start and end of exposure for each image are stored in a txt file. These timestamps are used to segment the entire event stream into individual event streams corresponding to each image. Additionally, we provide grounth sharp images in folder images_test for syntehtic dataset in order to evaluate metrics.

For the scenes of benerf_blender and benerf_unreal, We provide two versions of the images: one in color and one in grayscale. The grayscale version is used for quantitative evaluation, while the color version is used for qualitative evaluation. Since event-enhanced baseline methods we compared to can only run with gray image due to the single channel event stream we synthesized, we compute the metrics of all methods with gray images for consistency.

3. Train

First, you need to modify the path of datasets in config file:

datadir = XXXXX/BeNeRF_Datasets/real or synthetic/<dataset>/<scene>

Then, You can train model by entering terminal command as follow:

python train.py --device <cuda_id> --config ./configs/<dataset>/<scene>.txt --index <img_id>

We use wandb as a viewr to moniter the training process by defalut:

After training, all results including render image, render video, camera trajectory and checkpoint file will be saved in the path specified by logdir/<img_id>/ in the config file.

4. Test

You can test the model by loading the checkpoint file saved in the logdir/<img_id>/ path. We provide three options to test model.

Extract poses

python test.py --device <cuda_id> --config ./configs/<dataset>/<scene>.txt --index <img_id> \
               --extract_poses --num_extract_poses <the number of poses you want to extract>

We have set the default number of extracted poses to 19, i.e., num_extract_poses = 19.

Render images

python test.py --device <cuda_id> --config ./configs/<dataset>/<scene>.txt --index <img_id> \ 
               --render_images --num_render_images <the number of images you want to render>

We have set the default number of render images to 19, i.e., num_render_images = 19.

Render video

python test.py --device <cuda_id> --config ./configs/<dataset>/<scene>.txt --index <img_id> --render_video

Of course, you can choose all three options simultaneously to test the model.

python test.py --device <cuda_id> --config ./configs/<dataset>/<scene>.txt --index <img_id> \
               --extract_poses --num_extract_poses <the number of poses you want to extract> \
               --render_images --num_render_images <the number of images you want to render> \
               --render_video                                                           

The test results will be saved in logdir/<img_id>/test_results path.

5. Evaluation

To compute metrics like PSNR, SSIM, LPIPS for evaluating model performance on synthetic dataset, you can run script as follow:

python evaluate.py --dataset <dataset name> --scene <scene name> \
                   --result <path to folder of image results> --groundtruth <path to folder of groundtruth image>

For real-world datasets, since sharp ground truth images are not available, we choose to use the no-reference image quality metric BRISQUE to quantitatively evaluate the model's performance on real-world datasets. You can run the following script in MATLAB.

matlab -r "eval_brisque('<path to folder of image results>')"

Results

You can check our results at the following link.

• https://arxiv.org/abs/2407.02174
• https://akawincent.github.io/BeNeRF/

✒️ Citation

If you find this repository useful, please consider citing our paper:

@inproceedings{li2024benerf,
    author = {Wenpu Li and Pian Wan and Peng Wang and Jinghang Li and Yi Zhou and Peidong Liu},
    title = {BeNeRF: Neural Radiance Fields from a Single Blurry Image and Event Stream},
    booktitle = {European Conference on Computer Vision (ECCV)},
    year = {2024}
} 

🙏 Acknowledgment

In our work, the camera optimization and event stream integration into NeRF were inspired by BAD-NeRF and E-NeRF, respectively. The overall code framework is based on nerf-pytorch. We appreciate the effort of the contributors to these amazing repositories.