Deep Face Restoartion: Denoise, Super-Resolution, Deblur and Artifact Removal

A comprehensive list of resources for Deep Face Restoartion


Suggest new item
Report Bug

This repository provides a summary of deep learning-based face restoration algorithms. Our classification is based on the review paper "A Survey of Deep Face Restoration: Denoise, Super-Resolution, Deblur, Artifact Removal".

Blind Face Restoration	Face Super-Resolution	Face Deblurring	Face Denoising	Face Artifact Removal

Survey paper

T. Wang, k. Zhang, X. Chen, W. Luo, J. Deng, T. Lu, X. Cao, W. Liu, H. Li, and S. Zafeiriou “A Survey of Deep Face Restoration: Denoise, Super-Resolution, Deblur, Artifact Removal,” arXiv preprint arXiv:2211.02831, 2022. [pdf]

@article{wang2022survey
  title={A Survey of Deep Face Restoration: Denoise, Super-Resolution, Deblur, Artifact Removal},
  author={Wang, Tao, and Zhang, kaihao, and Chen, Xuanxi and Luo, Wenhan and Deng, Jiankang and Lu, Tong and Cao, Xiaochun and Liu, Wei and Li, Hongdong and Zafeiriou, Stefanos},
  journal={arXiv preprint arXiv:2211.02831},
  year={2022}
}

See our paper for more details. If you have any suggestions, feel free to contact me (e-mail: taowangzj@gmail.com). Thanks.

Table of contents

• Surveys
• Deep Blind Face Restoration
• Deep Face Super-Resolution
• Deep Face Deblurring
• Deep Face Denoising
• Deep Face Artifact Removal
• Other Related Works
• Image Quality Assessment
• Benchmark Datasets
  • Recommended Datasets
  • All Datasets

Surveys

year	Pub	Title	Link
2022	Arxiv	A Survey of Deep Face Restoration: Denoise, Super-Resolution, Deblur, Artifact Removal	[paper]

Deep Blind Face Restoration

Year	Pub	Title	Links	Arch
2018	ECCV	Learning Warped Guidance for Blind Face Restoration	[paper][code]	CNN
2018	CVPR	FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors	[paper][code]	CNN
2019	IJCV	Identity-preserving Face Recovery from Stylized Portraits	[paper][code]	GAN
2020	ECCV	Blind Face Restoration via Deep Multi-scale Component Dictionaries	[paper][code]	CNN
2020	CVPR	Enhanced Blind Face Restoration with Multi-Exemplar Images and Adaptive Spatial Feature Fusion	[paper][code]	CNN
2020	MM	HiFaceGAN: Face Renovation via Collaborative Suppression and Replenishment	[paper][code]	GAN
2020	CVPR	Image Processing Using Multi-Code GAN Prior	[paper][code]	GAN
2021	TPAMI	Face Restoration via Plug-and-Play 3D Facial Priors	[paper][code]	CNN
2021	CVPR	Progressive Semantic-Aware Style Transformation for Blind Face Restoration	[paper][code]	GAN
2021	CVPR	Towards Real-World Blind Face Restoration with Generative Facial Prior	[paper][code]	GAN
2021	CVPR	GAN Prior Embedded Network for Blind Face Restoration in the Wild	[paper][code]	GAN
2022	Arxiv	Multi-prior learning via neural architecture search for blind face restoration	[paper][code]	CNN
2022	CVPR	Blind Face Restoration via Integrating Face Shape and Generative Priors	[paper][code]	GAN
2022	CVPR	RestoreFormer: High-Quality Blind Face Restoration from Undegraded Key-Value Pairs	[paper][code]	ViT
2022	NeurIPS	Towards Robust Blind Face Restoration with Codebook Lookup Transformer	[paper][code]	ViT
2022	AAAI	Panini-Net: GAN Prior Based Degradation-Aware Feature Interpolation for Face Restoration	[paper][code]	GAN
2022	Arxiv	FaceFormer: Scale-aware Blind Face Restoration with Transformers	[paper][code]	ViT
2022	Arxiv	Blind Face Restoration: Benchmark Datasets and a Baseline Model	[paper][code]	ViT
2022	ECCV	VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder	[paper][code]	CNN
2022	TPAMI	Learning Dual Memory Dictionaries for Blind Face Restoration	[paper][code]	CNN
2022	Arxiv	Difface: Blind Face Restoration with Diffused Error Contraction	[paper][code]	Diffusion
2023	CVPR	DR2: Diffusion-based Robust Degradation Remover for Blind Face Restoration	[paper][code]	Diffusion
2023	CVPR	TFRGAN: Leveraging Text Information for Blind Face Restoration with Extreme Degradation	[paper][code]	GAN
2023	TCSVT	DEAR-GAN: Degradation-Aware Face Restoration With GAN Prior	[paper][code]	GAN
2023	FSP	Degradation Learning and Skip-Transformer for Blind Face Restoration	[paper][code]	GAN
2023	WACV	AT-DDPM: Restoring Faces degraded by Atmospheric Turbulence using Denoising Diffusion Probabilistic Models	[paper][code]	Diffusion
2023	ACMMM	DiffBFR: Bootstrapping Diffusion Model for Blind Face Restoration	[paper][code]	Diffusion
2024	TCSVT	Towards Real-World Blind Face Restoration with Generative Diffusion Prior	[paper][code]	Diffusion
2024	WACV	Diffuse and Restore: A Region-Adaptive Diffusion Model for Identity-Preserving Blind Face Restoration	[paper][code]	Diffusion
2024	AAAI	Blind Face Restoration under Extreme Conditions: Leveraging 3D-2D Prior Fusion for Superior Structural and Texture Recovery	[paper][code]	Diffusion

Deep Face Super-Resolution

Year	Pub	Title	Links	Arch
2015	AAAI	Learning Face Hallucination in the Wild	[paper][code]	CNN
2016	ECCV	Deep Cascaded Bi-Network for Face Hallucination	[paper][code]	CNN
2016	ECCV	Ultra-Resolving Face Images by Discriminative Generative Networks	[paper][code]	GAN
2017	CVPR	Attention-Aware Face Hallucination via Deep Reinforcement Learning	[paper][code]	CNN
2017	CVPR	Hallucinating Very Low-Resolution Unaligned and Noisy Face Images by Transformative Discriminative Autoencoders	[paper][code]	CNN
2017	ICCV	Learning to Super-Resolve Blurry Face and Text Images	[paper][code]	GAN
2017	AAAI	Face Hallucination with Tiny Unaligned Images by Transformative Discriminative Neural Networks	[paper][code]	GAN
2018	ECCV	Face Super-resolution Guided by Facial Component Heatmaps	[paper][code]	CNN
2018	CVPR	Super-FAN: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans	[paper][code]	GAN
2018	ECCV	To learn image super-resolution, use a GAN to learn how to do image degradation first	[paper][code]	GAN
2019	CVPRW	Exemplar Guided Face Image Super-Resolution without Facial Landmarks	[paper][code]	CNN
2019	BMVC	Progressive Face Super-Resolution via Attention to Facial Landmark	[paper][code]	CNN
2020	WACV	Component Attention Guided Face Super-Resolution Network: CAGFace	[paper][code]	CNN
2020	TNNLS	Dual-Path Deep Fusion Network for Face Image Hallucination	[paper][code]	CNN
2019	NEUCOM	On potentials of regularized Wasserstein generative adversarial networks for realistic hallucination of tiny faces	[paper][code]	GAN
2020	CVPR	PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models	[paper][code]	GAN
2021	TBBIS	E-ComSupResNet: Enhanced Face Super-Resolution Through Compact Network	[paper][code]	CNN
2021	MM	Face Hallucination via Split-Attention in Split-Attention Network	[paper][code]	CNN
2021	ICIP	Progressive Face Super-Resolution with Non-Parametric Facial Prior Enhancement	[paper][code]	GAN
2022	CVPR	GCFSR: a Generative and Controllable Face Super Resolution Method Without Facial and GAN Priors	[paper][code]	GAN
2022	TPAMI	EDFace-Celeb-1 M: Benchmarking Face Hallucination with a Million-scale Dataset	[paper][code]	CNN
2023	TIP	Semi-Cycled Generative Adversarial Networks for Real-World Face Super-Resolution	[paper][code]	GAN
2023	AAAI	GAN Prior based Null-Space Learning for Consistent Super-Resolution	[paper][code]	GAN
2023	CVPR	Spatial-Frequency Mutual Learning for Face Super-Resolution	[paper][code]	CNN
2023	TMM	Sctanet: A Spatial Attention-guided Cnn-transformer Aggregation Network for Deep Face Image Super-Resolution	[paper][code]	ViT
2023	TMM	An Efficient Latent Style Guided Transformer-CNN Framework for Face Super-Resolution	[paper][code]	ViT
2023	TMM	Exploiting Multi-scale Parallel Self-attention and Local Variation via Dual-branch Transformer-CNN Structure for Face Super-resolution	[paper][code]	ViT
2023	PR	A Composite Network Model for Face Super-Resolution with Multi-Order Head Attention Facial Priors	[paper][code]	ViT
2023	TIP	Semi-Cycled Generative Adversarial Networks for Real-World Face Super-Resolution	[paper][code]	GAN
2023	AS	A Multi-Scale Deep Back-Projection Backbone for Face Super-Resolution with Diffusion Models	[paper][code]	Diffusion
2023	TIM	Deep HyFeat Based Attention in Attention Model for Face Super-Resolution	[paper][code]	CNN
2023	PRL	Attentive ExFeat based Deep Generative Adversarial Network for Noise Robust Face Super-resolution	[paper][code]	GAN
2023	NN	Self-attention Learning Network for Face Super-resolution	[paper][code]	CNN

Deep Face Deblurring

Year	Pub	Title	Links	Arch
2017	Arxiv	DeepDeblur: Fast one-step blurry face images restoration	[paper][code]	CNN
2018	CVPR	Deep Semantic Face Deblurring	[paper][code]	CNN
2020	IJCV	Exploiting Semantics for Face Image Deblurring	[paper][code]	CNN
2020	TIP	Deblurring Face Images using Uncertainty Guided Multi-Stream Semantic Networks	[paper][code]	CNN
2020	MM	HiFaceGAN: Face Renovation via Collaborative Suppression and Replenishment	[paper][code]	GAN
2020	AAAI	Learning to deblur face images via sketch synthesis	[paper][code]	CNN
2022	TOG	Face Deblurring using Dual Camera Fusion on Mobile Phones	[paper][code]	CNN
2022	WACV	Deep Feature Prior Guided Face Deblurring	[paper][code]	CNN
2022	Arxiv	Multi-prior learning via neural architecture search for blind face restoration	[paper][code]	CNN
2022	Arxiv	Blind Face Restoration: Benchmark Datasets and a Baseline Model	[paper][code]	ViT
2022	SIGGRAPH	Face Deblurring using Dual Camera Fusion on Mobile Phones	[paper][code]	CNN

Deep Face Denoising

Year	Pub	Title	Links	Arch
2020	MM	HiFaceGAN: Face Renovation via Collaborative Suppression and Replenishment	[paper][code]	GAN
2022	Arxiv	Multi-prior learning via neural architecture search for blind face restoration	[paper][code]	CNN
2022	Arxiv	Blind Face Restoration: Benchmark Datasets and a Baseline Model	[paper][code]	ViT

Deep Face Artifact Removal

Year	Pub	Title	Links	Arch
2020	MM	HiFaceGAN: Face Renovation via Collaborative Suppression and Replenishment	[paper][code]	GAN
2022	Arxiv	Blind Face Restoration: Benchmark Datasets and a Baseline Model	[paper][code]	ViT
2022	Arxiv	Multi-prior learning via neural architecture search for blind face restoration	[paper][code]	CNN

Other Related Works

• CG-GAN: Class-Attribute Guided Generative Adversarial Network for Old Photo Restoration, Liu et al., ACMM 2021. Link
• Old Photo Restoration via Deep Latent Space Translation, Wan et al., TPAMI 2022. Link
• Demeshnet: Blind Face Inpainting for Deep Meshface Verification, Zhang et al. TIP 2017. Link
• Face Inpainting based on High-level Facial Attributes, Jampour et al., CVIU 2017. Link
• ABPN: Adaptive Blend Pyramid Network for Real-Time Local Retouching of Ultra High-Resolution Photo, Lei et al., CVPR 2022. Link
• Autoretouch: Automatic Professional Face Retouching, Shafaei et al., WACV 2021. Link
• Network Architecture Search for Face Enhancement, Yasarla et al., Arxiv 2021. Link
• Towards Automatic Face-to-face Translation, KR, R et al., ACMMM 2019. Link
• A Sketchtransformer Network for Face Photo-sketch Synthesis, Zhu et al., IJCAI 2021. Link

Image Quality Assessment

Method	Type	Code/Ref
PSNR (Peak Signal-to-Noise Ratio)	Full-Reference	Code
SSIM (Structural Similarity Index Measurement)	Full-Reference	Code
MS-SSIM (Multi-scale Structural Similarity Index Measurement)	Full-Reference	Code
LPIPS (Learned Perceptual Image Patch Similarity)	Full-Reference	Code
NIQE (Naturalness Image Quality Evaluator)	Non-Reference	Code
FID (Fréchet Inception Distance)	Non-Reference	Code
PI (Perceptual Index)	Non-Reference	Code
MOS (Mean Opinion Score)	Subject-Metric	Ref
iPrecision	Task Driven-Metric	Ref
iRecall	Task Driven-Metric	Ref
LLE (Landmark Localization Error)	Task Driven-Metric	Code
Deg (Identity Distance)	Task Driven-Metric	Code
AFLD (Average Face Landmark Distance)	Task Driven-Metric	Code
AFICS (Average Face ID Cosine Similarity)	Task Driven-Metric	Code

Benchmark Datasets

Recommended Datasets

Dataset	Usage	Quantity	Type
EDFace-Celeb	training&testing	>1M	paired dataset
PFHQ	training	60,000	paired dataset
FFHQ	training	70,000	non-paired dataset
CelebChild-Test	testing	180	non-paired real-world dataset
WebPhoto-Test	testing	407	non-paired real-world dataset
LFW-Test	testing	1,711	non-paired real-world dataset

All Datasets

Dataset	Paper	Year
BioID	Robust face detection using the hausdorff distance	2001
LFW	Labeled faces in the wild: A database forstudying face recognition in unconstrainedenvironments	2008
Pubfig	Attribute and similar classifiers for face verification	2009
Multi-PIE	Multi-PIE	2010
AFLW	Annotated facial landmarks in the wild: A large-scale, real-world database for facial landmark localization	2011
Helen	Interactive Facial Feature Localization	2012
300W	300 Faces in-the-Wild Challenge: The first facial landmark localization Challenge	2013
CASIA-WebFace (Password: y3wj)	Learning Face Representation from Scratch	2014
CelebA	Deep Learning Face Attributes in the Wild	2015
IMDB-WIKI	DEX: Deep EXpectation of apparent age from a single image	2015
LSUN	LSUN: Construction of a Large-Scale Image Dataset using Deep Learning with Humans in the Loop	2015
VGGFace	Deep Face Recognition	2015
300W-LP	Face Alignment Across Large Poses: A 3D Solution	2016
VoxCeleb1	VoxCeleb: a large-scale speaker identification dataset	2017
LS3D-W	How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)	2017
LS3D-W balanced	How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)	2017
Menpo	The Menpo Facial Landmark Localisation Challenge: A step towards the solution	2017
VGGFace2	VGGFace2: A dataset for recognising faces across pose and age	2018
VoxCeleb2	Voxceleb2: Deep speaker recognition	2018
FFHQ	A Style-Based Generator Architecture for Generative Adversarial Networks	2019
CelebChild-Test	Towards Real-World Blind Face Restoration with Generative Facial Prior	2021
WebPhoto-Test	Towards Real-World Blind Face Restoration with Generative Facial Prior	2021
CelebA-Test	Towards Real-World Blind Face Restoration with Generative Facial Prior	2021
LFW-Test	Towards Real-World Blind Face Restoration with Generative Facial Prior	2021
VFHQ	VFHQ: A High-Quality Dataset and Benchmark for Video Face Super-Resolution	2022
EDFace-Celeb	EDFace-Celeb-1M: Benchmarking Face Hallucination with a Million-scale Dataset	2022
EDFace-Celeb-1M (BFR128)	Blind Face Restoration: Benchmark Datasets and a Baseline Model	2022
EDFace-Celeb-150K (BFR512)	Blind Face Restoration: Benchmark Datasets and a Baseline Model	2022

statistics