This is official Pytorch implementation of DRMF: Degradation-Robust Multi-Modal Image Fusion via Composable Diffusion Prior (ACM MM 2024).
@inproceedings{Tang2024DRMF,
title={DRMF: Degradation-Robust Multi-Modal Image Fusion via Composable Diffusion Prior},
author={Tang, Linfeng and Deng, Yuxin and Yi, Xunpeng and Yan, Qinglong and Yuan, Yixuan and Ma, Jiayi},
booktitle={Proceedings of the ACM International Conference on Multimedia},
pages={8546--8555},
year={2024}
}
The overall framework of the proposed DRMF
conda create -n DRMF python=3.9
conda install -r requirement.txt
- Downloading the pre-trained model and placing them in ./experiments .
- Run the following script for fusion testing:
## for practical infrared and visible image fusion
python test_fusion.py --config 'Fusion.yml' \
--phase 'fusion' \
--fusion_type 'IVIF' \
--name 'IVIF_norm' \
--data_dir 'data/IVIF_norm/val' \
--resume_ir 'experiments/IVIF_norm/IR_MSRS.pth' \
--resume_vi 'experiments/IVIF_norm/VI_LOL.pth' \
--resume_weight 'experiments/Fusion/Fusion_DPCM.pth' \
--save_folder './Results' \
--sampling_timesteps 5
## for challenging infrared and visible image fusion
python test_fusion.py --config 'Fusion.yml' \
--phase 'fusion' \
--fusion_type 'IVIF' \
--name 'IVIF_degraded' \
--data_dir 'data/IVIF_degraded/val' \
--resume_ir 'experiments/IVIF_degraded/IR_MSRS.pth' \
--resume_vi 'experiments/IVIF_degraded/VI_LOL.pth' \
--resume_weight 'experiments/Fusion/Fusion_DPCM.pth' \
--save_folder './Results' \
--sampling_timesteps 5
## for practical medical image fusion
python test_fusion.py --config 'Fusion.yml' \
--phase 'fusion' \
--fusion_type 'MIF' \
--name 'MIF_norm' \
--data_dir 'data/MIF_norm/val' \
--resume_ir 'experiments/MIF_norm/CT.pth' \
--resume_vi 'experiments/MIF_norm/MRI.pth' \
--resume_weight 'experiments/Fusion/Fusion_DPCM.pth' \
--save_folder './Results' \
--sampling_timesteps 5
## for challenging medical image fusion
python test_fusion.py --config 'Fusion.yml' \
--phase 'fusion' \
--fusion_type 'MIF' \
--name 'MIF_degraded' \
--data_dir 'data/MIF_degraded/val' \
--resume_ir 'experiments/MIF_degraded/CT.pth' \
--resume_vi 'experiments/MIF_degraded/MRI.pth' \
--resume_weight 'experiments/Fusion/Fusion_DPCM.pth' \
--save_folder './Results' \
--sampling_timesteps 5
Construct pairs of degraded images and their corresponding high-quality version. (For example, '.\data\LOL\train\high' and '.\data\LOL\train\low' for low-light image enhancement)
- Edite ./configs/Restoration.yml for setting hyper-parameters.
- Run the following script for training the DRCDMs:
## for training DRCDM for visible images on the LOL dataset
python train_restoration.py --config 'Restoration.yml' \
--phase 'train' \
--data_type 'LOL' \
--name 'Restoration_VI_LOL' \
--data_dir 'data/LOL' \
--sampling_timesteps 20
## for training DRCDM for infrared images on the MSRS dataset
python train_restoration.py --config 'Restoration.yml' \
--phase 'train' \
--data_type 'IR' \
--name 'Restoration_IR_MSRS' \
--data_dir 'data/IVIF_degraded' \
--sampling_timesteps 20
## for training DRCDM for CT images on the Harvard dataset
python train_restoration.py --config 'Restoration.yml' \
--phase 'train' \
--data_type 'CT' \
--name 'Restoration_CT' \
--data_dir 'data/MIF_degraded' \
--sampling_timesteps 20
## for training DRCDM for MRI images on the Harvard dataset
python train_restoration.py --config 'Restoration.yml' \
--phase 'train' \
--data_type 'MRI' \
--name 'Restoration_MRI' \
--data_dir 'data/MIF_degraded' \
--sampling_timesteps 20
- Run the following script to generate high quality reference images if high quality GTs are missing, such as normal visible images in the MSRS dataset:
python test_restoration.py --config 'Restoration.yml' \
--phase 'test' \
--data_type 'VI' \
--name 'Restoration_VI_MSRS' \
--data_dir 'data/IVIF_degraded/train' \
--resume 'experiments/IVIF_degraded/VI_LOL.pth' \
--save_folder './Restoration' \
--sampling_timesteps 20 \
- Run the following script to generate high quality reference images if high quality GTs are missing, such as normal visible images in the MSRS dataset:
python test_restoration.py --config 'Restoration.yml' \
--phase 'test' \
--data_type 'VI' \
--name 'Restoration_VI_MSRS' \
--data_dir 'data/IVIF_degraded/train' \
--resume 'experiments/IVIF_degraded/VI_LOL.pth' \
--save_folder './Restoration' \
--sampling_timesteps 20
- Construct pairs of degraded images and their corresponding high-quality version.
- Editing ./configs/Fusion.yml for setting hyper-parameters.
- Run the following script for training the DPCM:
## for training DPCM for information aggragation on the MSRS dataset
python your_script.py --config 'Fusion.yml' \
--phase 'train' \
--fusion_type 'IVIF' \
--name 'Fusion_DPCM' \
--data_dir 'data/IVIF_degraded' \
--resume_ir 'experiments/IVIF_degraded/IR_MSRS.pth' \
--resume_vi 'experiments/IVIF_degraded/VI_LOL.pth' \
--sampling_timesteps 5
Fusion schematic in challenging scenarios for MMIF tasks. DDFM and DIVFusion are Diffusion-based and illumination-robust image fusion method, respectively. DiffLL and CAT are SOTA image restoration approeches
Visual comparison of DRMF with state-of-the-art approaches on practical and challenging fusion scenarios for IVIF
Visual comparison of our DRMF with state-of-the-art approaches on normal and challenging scenarios for MIF
Quantitative comparison of DRMF with state-of-the-art methods on IVIF and MIF tasks
Pre-enhancement for existing approaches
Visual comparison of object detection on the LLVIP dataset
If this work is helpful to you, please cite it as:
@inproceedings{Tang2024DRMF,
title={DRMF: Degradation-Robust Multi-Modal Image Fusion via Composable Diffusion Prior},
author={Tang, Linfeng and Deng, Yuxin and Yi, Xunpeng and Yan, Qinglong and Yuan, Yixuan and Ma, Jiayi},
booktitle={Proceedings of the ACM International Conference on Multimedia},
pages={8546--8555},
year={2024}
}
This code is built on WeatherDiff.