Code for the paper: Intrinsic Harmonization for Illumination-Aware Compositing, Chris Careaga, S. Mahdi H. Miangoleh Yağız Aksoy, Proc. SIGGRAPH Asia, 2023
Project Page | Paper | Video | Supplementary
We propose an illumination-aware image harmonization approach for in-the-wild imagery. Our method is formulated in the intrinsic image domain. We use off-the-shelf networks to generate albedo, shading and surface normals for the input composite and background image. We first harmonize the albedo of the background and foreground by predicting image editing parameters. Using normals and shading we estimate a simple lighting model for the background illumination. With this lighting model, we render Lambertian shading for the foreground and refine it using a network trained on segmentation datasets via self-supervision. When compared to prior works we are the only method that is capable of modeling realistic lighting effects.
Compositing is a crucial image editing task requiring realistic integration of objects into new backgrounds. Achieving a natural composition requires adjusting the appearance of the inserted object through a process called image harmonization. While existing literature addresses color harmonization, relighting, an equally vital aspect, is often overlooked due to the challenges in realistically adjusting object illumination in diverse environments.
In this project, we tackle image harmonization in the intrinsic domain, decomposing images into reflectance (albedo) and illumination (shading). We employ a two-step approach: first, harmonizing color in the albedo space, and then addressing the challenging relighting problem in the shading domain. Our goal is to generate realistic shading for the composited object, reflecting the new illumination environment.
More specifically, we initially render an initial shading using the Lambertian model and surface normals for the background and inserted object. A re-shading network then refines this shading for the composited object in a self-supervised manner. Our method is able to generate novel reshadings of the foreground region that reflect the illumination conditions of the background scene.
Our method outperforms prior works, producing realistic composite images that not only match color but also exhibit realistic illumination in diverse scenarios.
Our re-shading network learns to predict spatially-varying lighting effects in-context due to our self-supervised training approach
Depending on how you would like to use the code in this repository there are two options to setup the code.
In either case, you should first create a fresh virtual environment (python3 -m venv intrinsic_env) and start it (source intrinsic_env/bin/activate)
You can install this repository as a package using pip:
git clone https://github.com/compphoto/IntrinsicCompositing
cd IntrinsicCompositing
pip install .
If you want to make changes to the code and have it reflected when you import the package use pip install --editable
Or perform the same action without cloning the code using:
pip install https://github.com/compphoto/IntrinsicCompositing/archive/main.zip
This will allow you to import the repository as a Python package, and use our pipeline as part of your codebase.
The best way to run our pipeline is by using our interactive interface. We provide some example backgrounds and foregrounds in interface/examples:
$ cd interface
$ python interface.py --bg examples/bgs/lamp.jpeg --fg examples/fgs/soap.png --mask examples/masks/soap.png
The first time you run the interface multiple pretrained checkpoints will be downloaded (the method makes use of multiple off-the-shelf models) which may take some time. Subsequent runs will use the cached weights, but there is still a bit of preprocessing that is required when the interface is started. Once the preprocessing is done the interface window will appear and the input composite can be edited. After editing the composite, harmonizing only requires running our albedo and shading networks which should only take a second or two. These are the keybinds for the interface:
| Key | Action |
|---|---|
| r | run the harmonization of the current composite |
| s | save inputs, outputs and intermediate images |
| 1-5 | view various intermediate representations (shading, normals, etc) |
| scroll up/down | scale foreground region up or down |
The interface has been tested on an RTX2060 with 8 gb of VRAM which should be able to handle inference at a 1024 pixel resolution.
@INPROCEEDINGS{careagaCompositing,
author={Chris Careaga and S. Mahdi H. Miangoleh and Ya\u{g}{\i}z Aksoy},
title={Intrinsic Harmonization for Illumination-Aware Compositing},
booktitle={Proc. SIGGRAPH Asia},
year={2023},
}
This implementation is provided for academic use only. Please cite our paper if you use this code or any of the models.
The methodology presented in this work is safeguarded under intellectual property protection. For inquiries regarding licensing opportunities, kindly reach out to SFU Technology Licensing Office <tlo_dir ατ sfu δøτ ca> and Dr. Yağız Aksoy <yagiz ατ sfu δøτ ca>.