Skip to content

Latest commit

 

History

History
108 lines (65 loc) · 4.84 KB

README.md

File metadata and controls

108 lines (65 loc) · 4.84 KB
Raw

Image Super-Resolution with Text Prompt Diffusion

Zheng Chen, Yulun Zhang, Jinjin Gu, Xin Yuan, Linghe Kong, Guihai Chen, and Xiaokang Yang, "Image Super-Resolution with Text Prompt Diffusion", arXiv, 2023

[arXiv] [supplementary material] [visual results] [pretrained models]

🔥🔥🔥 News

  • 2023-11-25: This repo is released.

Abstract: Image super-resolution (SR) methods typically model degradation to improve reconstruction accuracy in complex and unknown degradation scenarios. However, extracting degradation information from low-resolution images is challenging, which limits the model performance. To boost image SR performance, one feasible approach is to introduce additional priors. Inspired by advancements in multi-modal methods and text prompt image processing, we introduce text prompts to image SR to provide degradation priors. Specifically, we first design a text-image generation pipeline to integrate text into the SR dataset through the text degradation representation and degradation model. The text representation applies a discretization manner based on the binning method to describe the degradation abstractly. This method maintains the flexibility of the text and is user-friendly. Meanwhile, we propose the PromptSR to realize the text prompt SR. The PromptSR utilizes the pre-trained language model (e.g., T5 or CLIP) to enhance restoration. We train the model on the generated text-image dataset. Extensive experiments indicate that introducing text prompts into SR, yields excellent results on both synthetic and real-world images.

LR Bicubic Prompt: [Light Noise] Prompt: [Heavy Noise]

⚒️ TODO

  • Release code and pretrained models

🔗 Contents

  1. Datasets
  2. Models
  3. Training
  4. Testing
  5. Results
  6. Citation
  7. Acknowledgements

🔎 Results

We achieve state-of-the-art performance on synthetic and real-world dataset. Detailed results can be found in the paper.

Evaluation on Synthetic Datasets (click to expand)
  • quantitative comparison

  • visual comparison

Evaluation on Real-World Datasets (click to expand)
  • quantitative comparison

  • visual comparison

📎 Citation

If you find the code helpful in your research or work, please cite the following paper(s).

@article{chen2023image,
  title={Image Super-Resolution with Text Prompt Diffusion},
  author={Chen, Zheng and Zhang, Yulun and Gu, Jinjin and Yuan, Xin and Kong, Linghe and Chen, Guihai and Yang, Xiaokang},
  journal={arXiv preprint arXiv:2303.06373},
  year={2023}
}

💡 Acknowledgements

This code is built on BasicSR, Image-Super-Resolution-via-Iterative-Refinement.