Download Stable Diffusion weights and save them as models/sd/models/ldm/stable-diffusion-v1/model.ckpt (create the stable-diffusion-v1 folder if needed).
| Dataset | Description | Attribute Used | Google Drive |
|---|---|---|---|
| CelebA | Real face images | 40 binary facial attributes | Link |
| FairFace | Real face images | Age with 9 categories | Link |
| FAIR | Synthetic face images | Skin tone with 6 categories | Link |
| LHQ | Natural scenes | 11 global scene attributes | Link |
Download the above datasets and unzip them as follows
|-- data
| |-- celeba
| | |-- 5_o_Clock_Shadow
| | |-- Bald
| | |-- ...
| |-- FAIR_benchmark
| | |-- Skin_tone
| |-- fairface
| | |-- Age
| |-- lhq
| | |-- Bright
| | |-- Colorful
| | |-- ...
conda env create --name iti-gen --file=environment.yml
source activate iti-gen
cd models/sd/
pip install -e .
cd ../../All of our experiments are reproducible, and we include bash scripts to run them in the jobfiles folder. The bash scripts are generated automatically by Python scripts that live in the generate_jobfiles folder. You might be interested in modifying the script-creating scripts (e.g. to include SLURM directives, modify the batch size, etc.). If you do so, you can re-generate the bash scripts by running
find generate_jobfiles/ -name "*.py" | xargs -n1 pythonThis experiment compares the diversity and image quality of vanilla Stable Diffusion, ITI-GEN, HPS and HPS with negative prompting. It computes the FID score and KL divergences (using CLIP as classifier) for all 40 attributes of CelebA. See instructions here.
This experiment demonstrates how ITI-GEN works on multiple domains, such as headshots and landscapes. See instructions here.
This experiment shows the plug-and-play capabilities of ITI-GEN: the text prompt used for training need not be the same as the one used for generation. We train "Age" and "Gender" using "a headshot of a person", and apply them using "a headshot of a firefighter" and "a headshot of a doctor", respectively. See instructions here.
This experiment proves the plug-and-play capabilities of ITI-GEN when it is used with ControlNet. See instructions here.
This experiment varies the number of reference images per category to prove that ITI-GEN is robust in the low-data regime. See instructions here.
This experiment shows how ITI-GEN uses gender as a proxy attribute when learning other attributes, such as "Bald", "Mustache" or "Beard". See instructions here.
This experiment compares the diversity and image quality of vanilla Stable Diffusion, ITI-GEN, HPS and HPS with negative prompting. It generates images for some attribute combinations and computes the KL divergence (using CLIP as classifier). See instructions here.
This experiment compares the performance of ITI-GEN on the attribute combination Male x Eyeglasses when using different reference datasets for the Eyeglasses attribute. See instructions here
This experiments shows that the training time is proportional to the size of the joint distribution, that grows exponentially with the number of attributes. See instructionshere
This experiment combines ITI-GEN with HPSn. See instructions here