Skip to content

Latest commit

 

History

History
161 lines (125 loc) · 6.13 KB

readme.md

File metadata and controls

161 lines (125 loc) · 6.13 KB
Raw

Deep Hybrid Camera Deblurring for Smartphone Cameras

Project | Paper | Supple

Official Implementation of SIGGRAPH Paper

Deep Hybrid Camera Deblurring for Smartphone Cameras
Jaesung Rim1, Junyong Lee2, Heemin Yang1, Sunghyun Cho1.
1POSTECH, 2Samsung AI Center Toronto
ACM SIGGRAPH 2024 Conference Papers

Installation

pip install -r requirements.txt
python setup.py develop --no_cuda_ext
pip install git+https://github.com/cheind/pytorch-debayer

Download

Dataset [Google Drive]

Descriptions (click)
  • HCBlur-Syn
    • HCBlur_Syn_train : 5,795 samples for training.
    • HCBlur_Syn_val : 880 samples for validation.
    • HCBlur_Syn_test : 1,731 samples for evaluation.
  • HCBlur-Real
    • 471 pairs of real-world blurred W and U

The HCBlur-Syn dataset

# HCBlur_Syn_test.zip
HCBlur_Syn_test
├── longW # long-exposure wide images
│   ├── 0908/20230908_10_32_05/000001
│   │   ├── longW/blur # folder of a blurred image
│   │   ├── longW/gt # folder of a gt sharp image
│   ...
├── shortUW # short-exposure ultra-wide images
│   ├── 0908/20230908_10_32_05/000001
│   │   ├── UWseqs/000001 # ultra-wide sequnece corresponding to longW/0908/20230908_10_32_05/000001
│   ...
├── shortUW_depth # estimated depth from the FOV alignment step.
│   ├── 0908/20230908_10_32_05_depth.txt # estimated depth values
│   ...
├── shortUW_flows # estimated optical flows from ultra-wide images.
│   ├── 0908/20230908_10_32_05/000001
│   │   ├── UWflows/000001 # estimate optical flows
│   ...
...

The HCBlur-Real dataset

# HCBlur_Real.zip
HCBlur_Real
├── longW # long-exposure wide images
│   ├── 1780013444228916_1780013544228916.png 
│   ...
├── shortUW 
│   ├── 1780013444228916_1780013544228916 # ultra-wide sequnece corresponding to 1780013444228916_1780013544228916.png
│   │   ├── 1780013434097457_1780013442430791.jpg 
│   ...
...

Dataset splits [link]

Pre-trained models [Google Drive]

Descriptions (click)
  • HC-DNet.pth: Weight of HC-DNet trained on HCBlur.
  • HC-FNet.pth: Weight of HC-FNet trained on HCBlur.
  • raft-sintel.pth: Weight of RAFT.
  • raft-small.pth: Weight of RAFT_small.

Demo

# ./HCDeblur

# demo of two samples of HCBlur-Real
python test_HCBlur_Real.py --dataset_root=demo --out_dir=results_real/demo

Testing

# ./HCDeblur
# datasets should be located in datasets
# pre-trained weights should be located in pretrained_models

## test on HCBlur-Syn
python -m torch.distributed.launch --nproc_per_node=1 --master_port=4114 basicsr/test.py -opt options/test/HCDNet-test.yml --launcher pytorch
python -m torch.distributed.launch --nproc_per_node=1 --master_port=4115 basicsr/test.py -opt options/test/HCFNet-test.yml --launcher pytorch

## test on HCBlur-Real
python test_HCBlur_Real.py --dataset_root=datasets/HCBlur-Real --out_dir=results/HCBlur-Real

Evaluation

# ./HCDeblur

# compute PSNR and SSIM on HCBlur-Syn
python evaluation/evaluate_HCBlur.py --input_dir=results/HCDNet/visualization/HCBlur-test --out_txt=HCDNet.txt
python evaluation/evaluate_HCBlur.py --input_dir=results/HCFNet/visualization/HCBlur-test --out_txt=HCFNet.txt

# compute non-reference metrics on HCBlur-Real
bash evaluation/evaluation_NR_metrics.sh "results/HCBlur-Real/HCDNet/*_HCDNet.png" HCDNet
bash evaluation/evaluation_NR_metrics.sh "results/HCBlur-Real/HCFNet/*_HCFNet.png" HCFNet

Training

# ./HCDeblur
# datasets should be located in datasets

# Step 1: pre-compute optical flows for training # requires 433 GB
python compute_flows.py --root_path=datasets/HCBlur_Syn_train --out_path=datasets/HCBlur_Syn_train/shortUW_flows;
python compute_interpolated_flows_trainset.py

# Step 2: training HC-DNet
python -m torch.distributed.launch --nproc_per_node=4 --master_port=4110 basicsr/train.py -opt options/train/HCDNet-train.yml --launcher pytorch

# Step 3: save results of HC-DNet
python -m torch.distributed.launch --nproc_per_node=1 --master_port=4111 basicsr/test.py -opt options/test/HCDNet-save-trainset.yml --launcher pytorch
python -m torch.distributed.launch --nproc_per_node=1 --master_port=4112 basicsr/test.py -opt options/test/HCDNet-save-valset.yml --launcher pytorch

# Step 4: training HC-FNet
python -m torch.distributed.launch --nproc_per_node=2 --master_port=4113 basicsr/train.py -opt options/train/HCFNet-train.yml --launcher pytorchl

License

The HCBlur dataset is released under CC BY 4.0 license.

Acknowledment

The code is based on BasicSR, NAFNet, RAFT, EDVR and IQA-PyTorch.

Citation

If you use our dataset for your research, please cite our paper.

@inproceedings{HCDeblur_rim,
 title={Deep Hybrid Camera Deblurring for Smartphone Cameras},
 author={Rim, Jaesung and Lee, Junyong and Yang, Heemin and Cho, Sunghyun},
 booktitle={ACM SIGGRAPH 2024 Conference Papers},
 year={2024}
}