nighttime_dehaze (ACMMM'2023)

Introduction

Enhancing Visibility in Nighttime Haze Images Using Guided APSF and Gradient Adaptive Convolution
ACM International Conference on Multimedia (ACMMM2023)
Yeying Jin, Beibei Lin, Wending Yan, Yuan Yuan, Wei Ye, and Robby T. Tan

[Paper]

Prerequisites

git clone https://github.com/jinyeying/nighttime_dehaze.git
cd nighttime_dehaze/
conda create -n dehaze python=3.7
conda activate dehaze
conda install pytorch=1.10.2 torchvision torchaudio cudatoolkit=11.3 -c pytorch
python3 -m pip install scipy==1.7.3
python3 -m pip install opencv-python==4.4.0.46

Nighttime Haze Data

Data	Dropbox	BaiduPan	Number & Type
RealNightHaze	Dropbox	BaiduPan code:r5mi	443, Haze
Internet_night_clean1	Dropbox	BaiduPan code:m7k1	411, Clean Reference
Internet_night_clean2	Dropbox	BaiduPan code:8f13	50, Clean Reference
GTA5 nighttime fog	Dropbox	BaiduPan code:67ml	Train:787,Test:77, Synthetic

Synthetic Nighttime Haze and Clean Ground Truth

• ECCV2020 Nighttime Defogging Using High-Low Frequency Decomposition and Grayscale-Color Networks [Paper]
  Wending Yan, Robby T. Tan and Dengxin Dai

Nighttime Dehazing Results Dropbox | BaiduPan code:oovt

Model	Dropbox	BaiduPan	Model Put in Path	Results Dropbox	Results BaiduPan
dehaze.pt	dehaze.pt	dehaze.pt code:n3t8	results/dehaze/model	RealNightHaze	RealNightHaze code:i43f
GTA5.pt	GTA5.pt	GTA5.pt code:fk29	results/GTA5/model	GTA5	GTA5 code:ufen
NHR.pt	NHR.pt	NHR.pt code:dnhf	results/NHR/model	NHR	NHR code:0nma
NHM.pt	NHM.pt	NHM.pt code:d7oj	results/NHM/model	NHM	NHM code：4gt0
NHC.pt	NHC.pt	NHC.pt code:yryp	results/NHC/model	NHC	NHC code：njf9

We provide the visualization results in 0_ACMMM23_RESULTS/NHR/index.html,
inside the directory 0_ACMMM23_RESULTS/NHR/img_0/ are hazy inputs, img_1 are ground truths, img_2 are our results.
For results corresponding to GTA5, NHM or NHC, please refer to the respective directories.

• For the RealNightHaze Dataset

1. Set the datasetpath to RealNightHaze,
2. Download the checkpoint dehaze.pt Dropbox| BaiduPan code:n3t8 put in results/dehaze/model,
3. Run the Python code, results are in results/dehaze/output.

CUDA_VISIBLE_DEVICES=1 python main_test.py --dataset dehaze --datasetpath /diskc/yeying/night_dehaze/dataset/Internet_night_fog/

• For the Synthetic Dataset

1. Set Line18 --have_gt to True, set the datasetpath to GTA5 or NHR or NHM or NHC,
2. Download the checkpoint GTA5.pt, put in results/GTA5/model. Similarly, for NHR.pt, NHM.pt, NHC.pt,
3. Run the Python code,

CUDA_VISIBLE_DEVICES=1 python main_test.py --dataset NHM --datasetpath /diskc/yeying/night_dehaze/dataset/middlebury/testA/ 
CUDA_VISIBLE_DEVICES=1 python main_test.py --dataset NHC --datasetpath /diskc/yeying/night_dehaze/dataset/Cityscape/testA/ 
CUDA_VISIBLE_DEVICES=1 python main_test.py --dataset NHR --datasetpath /diskc/yeying/night_dehaze/dataset/NHR/testA/ 
CUDA_VISIBLE_DEVICES=1 python main_test.py --dataset GTA5 --datasetpath /diskc/yeying/night_dehaze/GTA5/testA/

• Evaluation: Set the dataset_name GTA5 or NHR or NHM or NHC, and run the Python code:

python calculate_psnr_ssim_NH_GTA5.py

Dataset	PSNR	SSIM
GTA5	30.383	0.9042
NHR	26.56	0.89
NHM	33.76	0.92
NHC	38.86	0.97

APSF-Guided Nighttime Glow Rendering

Run the Matlab code to obtain the clean and glow pairs:

APSF_GLOW_RENDER_CODE/synthetic_glow_pairs.m

Change the datapath nighttime_dehaze/paired_data/clean_data/,
the paired clean and glow results are saved in nighttime_dehaze/paired_data/clean/ and nighttime_dehaze/paired_data/glow/,
the visualization of light source maps are in nighttime_dehaze/paired_data/glow_render_visual/light_source/.

Run the Matlab code to visualize Fig.3 in the main paper:

APSF_GLOW_RENDER_CODE/synthetic_glow_fig3_visualization.m

APSF and Alpha Matting are the implementations of the papers:

• CVPR03 Shedding Light on the Weather [Paper]
• CVPR06 A Closed-Form Solution to Natural Image Matting [Paper]

Edge

Run the Python code to visualize Fig.6, the environment is Pytorch 1.9 with cuda 10.1 and cudnn 7.5, results are in EDGE/results.

conda install pytorch==1.9.0 torchvision==0.10.0 torchaudio==0.9.0 cudatoolkit=10.1 -c pytorch

python main.py --sa --dil --gpu 1 --datadir ./Input/ --evaluate-converted

Enhancement

Run the Matlab code to visualize Fig.8, results are in ENHANCEMENT/attention_map.

Run the Matlab code to visualize Fig.10.

License

The code and models in this repository are licensed under the MIT License for academic and other non-commercial uses.
For commercial use of the code and models, separate commercial licensing is available. Please contact:

• Yeying Jin (jinyeying@u.nus.edu)
• Robby T. Tan (tanrobby@gmail.com)
• Jonathan Tan (jonathan_tano@nus.edu.sg)

Citation

If this work or the Internet data is useful for your research, please cite our paper.

@inproceedings{jin2023enhancing,
  title={Enhancing visibility in nighttime haze images using guided apsf and gradient adaptive convolution},
  author={Jin, Yeying and Lin, Beibei and Yan, Wending and Yuan, Yuan and Ye, Wei and Tan, Robby T},
  booktitle={Proceedings of the 31st ACM International Conference on Multimedia},
  pages={2446--2457},
  year={2023}
}

@inproceedings{jin2022unsupervised,
  title={Unsupervised night image enhancement: When layer decomposition meets light-effects suppression},
  author={Jin, Yeying and Yang, Wenhan and Tan, Robby T},
  booktitle={European Conference on Computer Vision},
  pages={404--421},
  year={2022},
  organization={Springer}
}