Selective Frequency Network for Image Restoration (ICLR2023)

Yuning Cui, Yi Tao, Zhenshan Bing, Wenqi Ren, Xinwei Gao, Xiaochun Cao, Kai Huang, Alois Knoll

Update

Our extended paper has been accepted by T-PAMI (Image Restoration via Frequency Selection)

• 2023.08.04 🎉 Release the model and code for image deraining
• 2023.04.19 🎉 Release our results on RSBlur
• 2023.04.14 :tada: Release pre-trained models, code for three tasks (dehazing, motion deblurring, desnowing), and resulting images on most datasets.

Image restoration aims to reconstruct the latent sharp image from its corrupted counterpart. Besides dealing with this long-standing task in the spatial domain, a few approaches seek solutions in the frequency domain in consideration of the large discrepancy between spectra of sharp/degraded image pairs. However, these works commonly utilize transformation tools, e.g., wavelet transform, to split features into several frequency parts, which is not flexible enough to select the most informative frequency component to recover. In this paper, we exploit a multi-branch and content-aware module to decompose features into separate frequency subbands dynamically and locally, and then accentuate the useful ones via channel-wise attention weights. In addition, to handle large-scale degradation blurs, we propose an extremely simple decoupling and modulation module to enlarge the receptive field via global and window-based average pooling. Integrating two developed modules into a U-Net backbone, the proposed Selective Frequency Network (SFNet) performs favorably against state-of-the-art algorithms on five image restoration tasks, including single-image defocus deblurring, image dehazing, image motion deblurring, image desnowing, and image deraining.

Architecture

Installation

The project is built with PyTorch 3.8, PyTorch 1.8.1. CUDA 10.2, cuDNN 7.6.5 For installing, follow these instructions:

conda install pytorch=1.8.1 torchvision=0.9.1 -c pytorch
pip install tensorboard einops scikit-image pytorch_msssim opencv-python

Install warmup scheduler:

cd pytorch-gradual-warmup-lr/
python setup.py install
cd ..

Training and Evaluation

Please refer to respective directories.

Results

Task	Dataset	PSNR	SSIM	Resulting Images
Motion Deblurring	GoPro	33.27	0.963	gdrive,Baidu
	HIDE	31.10	0.941	gdrive,Baidu
	RSBlur	34.35	0.872	gdrive,Baidu
Image Dehazing	SOTS-Indoor	41.24	0.996	gdrive,Baidu
	SOTS-Outdoor	40.05	0.996	gdrive,Baidu
	Dense-Haze	17.46	0.578	gdrive,Baidu
Image Desnowing	CSD	38.41	0.99	gdrive,Baidu
	SRRS	32.40	0.98	gdrive,Baidu
	Snow100K	33.79	0.95	gdrive,Baidu
Image Deraining	Average	33.56	0.929	gdrive,Baidu
Defocus Deblurring	DPDDsingle	26.23	0.811	gdrive,Baidu
	DPDDdual	26.34	0.817	gdrive,Baidu

Citation

If you find this project useful for your research, please consider citing:

@inproceedings{cui2023selective,
  title={Selective Frequency Network for Image Restoration},
  author={Cui, Yuning and Tao, Yi and Bing, Zhenshan and Ren, Wenqi and Gao, Xinwei and Cao, Xiaochun and Huang, Kai and Knoll, Alois},
  booktitle={The Eleventh International Conference on Learning Representations},
  year={2023}
}

Acknowledgement

This project is mainly based on MIMO-UNet.

Contact

Should you have any question, please contact Yuning Cui.