Intrinsic Decomposition & Inverse Rendering

A Survey on Intrinsic Images: Delving Deep Into Lambert and Beyond

Table of contents

• Intrinsic Decomposition (non-DL)
• Intrinsic Decomposition (DL)
• Inverse Rendering (non-DL)
• Inverse Rendering (DL)
• Portrait Relighting (DL)
• Dataset

Intrinsic Decomposition (non-DL)

Year	Pub	Paper	Repo
2016	SIGGRAPH	Live Intrinsic Video
2021	SIGGRAPH	Real-time Global Illumination Decomposition of Videos
2021	Eurographics	Interactive Photo Editing on Smartphones via Intrinsic Decomposition

Intrinsic Decomposition (DL)

Year	Pub	Paper	Repo
2017	Eurographics	Intrinsic Decompositions for Image Editing	Code Dataset
2017	NeurIPS	Self-Supervised Intrinsic Image Decomposition	Code / Dataset
2018	CVPR	Intrinsic Image Transformation via Scale Space Decomposition	Code / Dataset
2018	CVPR	Revisiting Deep Intrinsic Image Decompositions	Code
2018	CVPR	Learning Intrinsic Image Decomposition from Watching the World	Code / Dataset
2018	Pacific Graphics	Unsupervised Deep Single-Image Intrinsic Decomposition using Illumination-Varying Image Sequences	Code
2018	ECCV	CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering	Code / Dataset
2018	ECCV	Single Image Intrinsic Decomposition without a Single Intrinsic Image
2019	CVPR	InverseRenderNet: Learning single image inverse rendering	Code / Dataset
2019	CVPR	Learning to Separate Multiple Illuminants in a Single Image	Code & Dataset
2019	ICCV	Neural Inverse Rendering of an Indoor Scene From a Single Image
2019	ICCV	GLoSH: Global-Local Spherical Harmonics for Intrinsic Image Decomposition
2020	CVPR	Unsupervised Learning for Intrinsic Image Decomposition from a Single Image
2022	CVPR	PIE-Net: Photometric Invariant Edge Guided Network for Intrinsic Image Decomposition	Code

Inverse Rendering (non-DL)

L:Lighting Estimation, G:Geometry Estimation, M:Material Estimation

Year	Pub	Paper	Repo	L	G	M
2010	SIGGRAPH	Manifold bootstrapping for SVBRDF capture			✓	✓
2013	SIGGRAPH	Practical SVBRDF Capture in the Frequency Domain			✓	✓
2014	SIGGRAPH ASIA	Appearance-from-Motion: Recovering Spatially Varying Surface Reflectance under Unknown Lighting		✓	✓	✓
2015	CGF	Mobile Surface Reflectometry			✓	✓
2015	SIGGRAPH	Two-Shot SVBRDF Capture for Stationary Materials	Code & Dataset		✓	✓
2016	SIGGRAPH ASIA	Simultaneous Acquisition of Microscale Reflectance and Normals			✓	✓
2016	SIGGRAPH ASIA	Recovering Shape and Spatially-Varying Surface Reflectance under Unknown Illumination		✓	✓	✓
2017	SIGGRAPH ASIA	Polarization imaging reflectometry in the wild			✓	✓
2018	SIGGRAPH ASIA	Simultaneous Acquisition of Polarimetric SVBRDF and Normals			✓	✓
2021	Eurographics	Progressive Acquisition of SVBRDF and Shape in Motion			✓	✓

Inverse Rendering (DL)

L:Lighting Estimation, G:Geometry Estimation, M:Material Estimation

Year	Pub	Paper	Repo	L	G	M
2016	SIGGRAPH	Reflectance Modeling by Neural Texture Synthesis			✓	✓
2017	CVPR	Deep Outdoor Illumination Estimation	Dataset	✓
2017	SIGGRAPH	Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks	Code		✓	✓
2017	ICCV	Reflectance Capture using Univariate Sampling of BRDFs			✓	✓
2017	SIGGRAPH ASIA	Learning to Predict Indoor Illumination from a Single Image	Dataset	✓
2018	SIGGRAPH	Single-Image SVBRDF Capture with a Rendering-Aware Deep Network	Code & Dataset		✓	✓
2018	3DV	Learning to Estimate Indoor Lighting from 3D Objects	Code & Dataset	✓
2018	SIGGRAPH ASIA	Relighting Humans: Occlusion-Aware Inverse Rendering for Full-Body Human Images	Code & Dataset	✓	✓	✓
2018	SIGGRAPH ASIA	Learning to Reconstruct Shape and Spatially-Varying Reflectance from a Single Image	Code / Dataset		✓	✓
2019	CVPR	Fast Spatially-Varying Indoor Lighting Estimation	Dataset	✓		✓
2019	CVPR	All-Weather Deep Outdoor Lighting Estimation	Dataset	✓
2019	CVPR	DeepLight: Learning Illumination for Unconstrained Mobile Mixed Reality		✓
2019	CVPR	Neural Illumination: Lighting Prediction for Indoor Environments	Dataset	✓	✓
2019	ICCV	Deep Parametric Indoor Lighting Estimation	Dataset	✓
2020	CVPR	Lighthouse: Predicting Lighting Volumes for Spatially-Coherent Illumination	Code / Dataset	✓
2020	CVPR	Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image	Code / Dataset	✓	✓	✓
2020	ECCV	Object-Based Illumination Estimation with Rendering-Aware Neural Networks		✓		✓
2020	ECCV	Single-Shot Neural Relighting and SVBRDF Estimation	Code / Dataset		✓	✓
2020	SIGGRAPH ASIA	MaterialGAN: Reflectance Capture using a Generative SVBRDF Model	Code		✓	✓
2021	SIGGRAPH	Highlight-Aware Two-Stream Network for Single-Image SVBRDF Acquisition			✓	✓
2021	CVPR	PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting	Code	✓	✓	✓
2021	CVPR	Deep Polarization Imaging for 3D Shape and SVBRDF Acquisition	Dataset		✓	✓
2021	CVPR	Shape and Material Capture at Home	Code & Dataset		✓	✓
2021	ICCV	Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting		✓	✓
2021	ICCV	SurfaceNet: Adversarial SVBRDF Estimation from a Single Image	Code		✓	✓
2021	AAAI	EMLight: Lighting Estimation via Spherical Distribution Approximation		✓
2021	EGSR	Material and Lighting Reconstruction for Complex Indoor Scenes with Texture-space Differentiable Rendering		✓		✓
2021	SIGGRAPH ASIA	Generative Modelling of BRDF Textures from Flash Images	Code		✓	✓
2022	CVPR	IRISformer: Dense Vision Transformers for Single-Image Inverse Rendering in Indoor Scenes	Code	✓	✓	✓

Portrait Relighting (DL)

Year	Pub	Paper	Repo
2019	SIGGRAPH	Single Image Portrait Relighting
2020	CVPR	Learning Physics-Guided Face Relighting Under Directional Light
2020	TOG	Single Image Portrait Relighting via Explicit Multiple Reflectance Channel Modeling	Dataset
2021	WACV	Relighting Images in the Wild with a Self-Supervised Siamese Auto-Encoder
2021	CVPR	Monocular Reconstruction of Neural Face Reflectance Fields
2021	SIGGRAPH	Learning to Relight Portraits for Background Replacement
2021	ICCV	Neural Video Portrait Relighting in Real-time via Consistency Modeling	Code / Dataset

Dataset

Year	Pub	Paper	Repo
2014	SIGGRAPH	Intrinsic Images in the Wild	Dataset
2017	CVPR	Shading Annotations in the Wild	Dataset
2019	ICCV	A Dataset of Multi-Illumination Images in the Wild	Code / Dataset
2021	CVPR	OpenRooms: An Open Framework for Photorealistic Indoor Scene Datasets	Dataset
2021	ICCV	Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding	Dataset