Learning Degradation-unaware Representation with Prior-based Latent Transformations for Blind Face Restoration

A pytorch implementation for the paper: PLTrans

---

---

🎉 News

• [2025.01.16] 🚀🚀 Release the code and weight of PLTrans.

---

Introduction

Blind face restoration focuses on restoring high-fidelity details from images subjected to complex and unknown degradations, while preserving identity information. In this paper, we present a Prior-based Latent Transformation approach (PLTrans), which is specifically designed to learn a degradation-unaware representation, thereby allowing the restoration network to effectively generalize to real-world degradation. Toward this end, PLTrans learns a degradation-unaware query via a latent diffusion-based regularization module. Furthermore, conditioned on the features of a degraded face image, a latent dictionary that captures the priors of HQ face images is leveraged to refine the features by mapping the top-$d$ nearest elements. The refined version will be used to build key and value for the cross-attention computation, which is tailored to each degraded image and exhibits reduced sensitivity to different degradation factors. Conditioned on the resulting representation, we train a decoding network that synthesizes face images with authentic details and identity preservation. Through extensive experiments, we verify the effectiveness of the design elements and demonstrate the generalization ability of our proposed approach for both synthetic and unknown degradations. We finally demonstrate the applicability of PLTrans in other vision tasks.

---

Getting Started

1.Environment

Please first clone the repo and install the required environment, which can be done by running the following commands:

conda create -n PLTrans python=3.8
conda activate PLTrans

pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r requirements.txt

2. Inference

Download the checkpoint file from huggingface and put it into directory [weights].

Blind Face Restoration

python scripts/guide_sample.py --indir {}  --outdir results/{} --start_t 200  --end_t 199 --range_t 0 --down_N 4

🚀🚀 Beyond Restoration: One Model, Limitless Possibilities !! 🚀🚀

Thanks to the diffusion module integrated into PLTrans, you can perform a range of tasks, such as interpolating between two real face images, transforming any face-like image into a real face, generating faces from noise, inpainting, or editing one face based on another reference. All of these functions can be achieved with a well-trained model.

However, the limitation is that you may need to spend significant time fine-tuning parameters during the inference stage, such as --start_t 150, --end_t 145, --range_t 0, and --lerp_w 0.1.

Anything -> Real Face

• Noise -> Face

• Cartoon -> Face

• Scribble -> Face

python scripts/guide_sample_q_replace_UTinterploate_single.py --indir {}  --outdir results/{} --start_t 200  --end_t 150 --range_t 0 --lerp_w 0.1
bash scripts/multi_interpolation.sh

Inpainting

python scripts/guide_sample_inpaint.py  --indir inputs/inpainting/image/ --outdir results/inpainting --start_t 600  --end_t 100 --range_t 0

Inpainting directory structure:

• image
• mask (npy format)

Face Edition

python scripts/guide_sample_q_replace_edit_couple.py --path1 inputs/edit/image/116597.png --path2 inputs/edit/047992.jpg  --outdir results/edit/047992-edited.png --start_t 200  --end_t 190 --lerp_w 0.1

Edition directory structure:

• image
• mask (npy format)

Interpolation

python scripts/guide_sample_q_replace_UTinterploate_couple.py --path1 {img1_path} --path2 {img2_path}  --outdir results/{output name} --start_t 150  --end_t 145 --range_t 0 --lerp_w 0.1
bash scripts/multi_interpolation.sh

Citation

If you find PLTrans useful in your research or applications, please kindly cite:

@InProceedings{Xie_2024_CVPR,
    author    = {Xie, Lianxin and Zheng, Csbingbing and Xue, Wen and Jiang, Le and Liu, Cheng and Wu, Si and Wong, Hau San},
    title     = {Learning Degradation-unaware Representation with Prior-based Latent Transformations for Blind Face Restoration},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {9120-9129}
}

Acknowledgments

This project is built upon the open-source framework RestorFormer and LDM. We would like to express our sincere gratitude to the authors and contributors of Restoration and LDM for providing such a valuable resource. Their work has greatly influenced and supported the development of this project. You can find the original RestorFormer and LDM repository here: RestorFormer Repository, LDM