Simple-SR

The repository includes MuCAN, LAPAR, Beby-GAN and etc. It is designed for simple training and evaluation.

---

Update

The training code of BebyGAN and LAPAR is released.

The well-trained models and all visual examples (including BebyGAN and other SOTAs) are available here.

---

Paper

MuCAN: Multi-Correspondence Aggregation Network for Video Super-Resolution

[ECCV] [arXiv]

LAPAR: Linearly-Assembled Pixel-Adaptive Regression Network for Single Image Super-resolution and Beyond

[NeurIPS] [arXiv]

Best-Buddy GANs for Highly Detailed Image Super-Resolution

[arXiv]

Please find supplementary files of MuCAN and LAPAR here.

---

Usage

1. Clone the repository

   git clone https://github.com/Jia-Research-Lab/Simple-SR.git

2. Install the dependencies

   • Python >= 3.6
   • PyTorch >= 1.2
   • spatial-correlation-sampler

   pip install spatial-correlation-sampler

   • Other packages

   pip install -r requirements.txt

3. Download pretrained models from Google Drive. We re-trained the LAPAR models and their results are slightly different from the ones reported in paper.

   • MuCAN
     • MuCAN_REDS.pth: trained on REDS dataset, 5-frame input, x4 scale
     • MuCAN_Vimeo90K.pth: trained on Vimeo90K dataset, 7-frame input, x4 scale
   • LAPAR: trained on DIV2K+Flickr2K datasets

     Scale x2	Scale x3	Scale x4
     LAPAR_A_x2.pth	LAPAR_A_x3.pth	LAPAR_A_x4.pth
     LAPAR_B_x2.pth	LAPAR_B_x3.pth	LAPAR_B_x4.pth
     LAPAR_C_x2.pth	LAPAR_C_x3.pth	LAPAR_C_x4.pth

   • BebyGAN
     • RRDB_warmup.pth: provided for initialization
     • BebyGAN_x4.pth: trained on DIV2K+Flickr2K datasets, x4 scale

4. Quick test

   You have to define the output_path or gt_path or both. If output_path is given, outputs will be saved. If gt_path is given, PSNR/SSIM will be calculated. If you want to calculate LPIPS results, please install lpips library.

   • For SISR,

   python3 test_sample.py --sr_type SISR --model_path /model/path --input_path ./demo/SISR/LR_imgs --output_path ./demo/SISR/output --gt_path ./demo/SISR/HR_imgs

   • For VSR,

   python3 test_sample.py --sr_type VSR --model_path /model/path --input_path ./demo/VSR/LR_imgs --output_path ./demo/VSR/output --gt_path ./demo/VSR/HR_imgs

Prepare Data

1. Training Datasets

   Download DIV2K and Flickr2K. You may crop the HR and LR images to sub-images for fast reading referring to .utils/data_prep/extract_subimage.py.

2. Evaluation Datasets

   Download Set5, Set14, Urban100, BSDS100 and Manga109 from Google Drive uploaded by BasicSR.

3. Update the dataset location in .dataset/__init__.py.

4. (Optional) You can convert images to lmdb files for fast loading referring to BasicSR. And you need to modify the data reading logics in .dataset/*dataset.py accordingly.

Train

1. Create a log folder as

   mkdir logs

2. Create a new experiment folder in .exps/. You just need to prepare the config.py and network.py, while the train.py and validate.py are universal. For example, for LAPAR_A_x2, run

   cd exps/LAPAR_A_x2/
   bash train.sh $GPU_NUM $PORT

   Please modify the path of initialization model of BebyGAN (found in Google Drive) in config.py before training.

   Notice that you can find the checkpoints, log files and visualization images in either .exps/LAPAR_A_x2/log/ (a soft link) or .logs/LAPAR_A_x2/.

Test

Please refer to validate.py in each experiment folder or quick test above.

---

Acknowledgement

We refer to BasicSR for some details.

---

Bibtex

@inproceedings{li2020mucan,
  title={MuCAN: Multi-correspondence Aggregation Network for Video Super-Resolution},
  author={Li, Wenbo and Tao, Xin and Guo, Taian and Qi, Lu and Lu, Jiangbo and Jia, Jiaya},
  booktitle={European Conference on Computer Vision},
  pages={335--351},
  year={2020},
  organization={Springer}
}
@article{li2020lapar,
  title={LAPAR: Linearly-Assembled Pixel-Adaptive Regression Network for Single Image Super-resolution and Beyond},
  author={Li, Wenbo and Zhou, Kun and Qi, Lu and Jiang, Nianjuan and Lu, Jiangbo and Jia, Jiaya},
  journal={Advances in Neural Information Processing Systems},
  volume={33},
  year={2020}
}
@article{li2021best,
  title={Best-Buddy GANs for Highly Detailed Image Super-Resolution},
  author={Li, Wenbo and Zhou, Kun and Qi, Lu and Lu, Liying and Jiang, Nianjuan and Lu, Jiangbo and Jia, Jiaya},
  journal={arXiv preprint arXiv:2103.15295},
  year={2021}
}