This is the official repository for the following paper:
Deep Image Harmonization with Globally Guided Feature Transformation and Relation Distillation [arXiv]
Li Niu, Linfeng Tan, Xinhao Tao, Junyan Cao, Fengjun Guo, Teng Long, Liqing Zhang
Accepted by ICCV 2023.
Our ccHarmony dataset can better reflect natural illumination variation, with closer data distribution to Hday2night in iHarmony4. In the meanwhile, the collection cost of ccHarmony is much lower than Hday2night. Therefore, ccHarmony strikes a good balance between natural illumination variation and collection cost.
You can augment ccHarmony dataset using our SycoNet, by synthesizing high-quality composite images for real images.
Our dataset ccHarmony is a color checker (cc) based image harmonization dataset. In previous datasets like NUS dataset and Gehler dataset, images are captured with a color checker placed in the scene that provides ground truth reference for illumination estimation, as shown in (a) and (b) in the figure below. Based on these datasets, we design a novel transitive way to construct image harmonization dataset (see (c) in the figure below). Specifically, we convert the foreground in a real image to the standard illumination condition, and then convert it to another illumination condition, arriving at a synthetic composite image. In this way, we obtain 4260 pairs of synthetic composite images and ground-truth real images. More details can be found in our paper.
Our dataset contains 350 real images and 426 segmented foregrounds, in which each real image has one or two segmented foregrounds. Each foreground is associated with 10 synthetic composite images. Therefore, our dataset has in total 4260 pairs of synthetic composite images and ground-truth real images. We split all pairs into 3080 training pairs and 1180 test pairs. Our dataset can be downloaded from Baidu Cloud (access code: bulf) or OneDrive. Several example real images and their corresponding synthetic composite images are show below.
We evaluate several existing image harmonization methods on our ccHarmony dataset. Specifically, given their released models pretrained on iHarmony4 dataset, we finetune their models on the training set of ccHarmony and evaluate on the test set of ccHarmony.
| MSE | fMSE | PSNR | fSSIM | |
|---|---|---|---|---|
| DoveNet | 110.84 | 880.94 | 31.61 | 0.8231 |
| RainNet | 58.11 | 519.32 | 34.78 | 0.8665 |
| IIH | 83.72 | 636.28 | 33.64 | 0.7640 |
| IHT | 55.73 | 514.47 | 35.07 | 0.8203 |
| iSSAM | 28.83 | 264.84 | 36.05 | 0.9096 |
| CDTNet | 27.87 | 264.51 | 36.62 | 0.9225 |
| Harmonizer | 43.31 | 402.09 | 34.68 | 0.8951 |
| DCCF | 29.25 | 259.83 | 36.62 | 0.9094 |
| GiftNet(Ours) | 24.55 | 235.20 | 37.59 | 0.9322 |
Due to commercial collaboration, the code is not allowed to be released. However, an improved version can be found here.