TCDiff

TCDiff: Triple Condition Diffusion Model with 3D Constraints for Stylizing Synthetic Faces
Bernardo Biesseck, Pedro Vidal, Luiz Coelho, Roger Granada, David Menotti
In SIBGRAPI 2024

Paper     Arxiv

We propose a Triple Condition Diffusion Model (TCDiff) to improve face style transfer from real to synthetic faces through 2D and 3D facial constraints, enhancing face identity consistency while keeping the necessary high intra-class variance for training face recognition models with synthetic data.

1. Main requirements

• Python==3.8
• CUDA==11.2
• numpy==1.24.2
• mxnet==1.9.1
• torch>=2.2.0
• torchvision==0.12.0
• pytorch-lightning==1.7.1
• opencv-python>=4.8.1.78

2. Create environment

CONDA_ENV=tcdiff
conda create -y --name $CONDA_ENV python=3.9
conda activate $CONDA_ENV

conda env config vars set CUDA_HOME="/usr/local/cuda-11.2"; conda deactivate; conda activate $CONDA_ENV
conda env config vars set LD_LIBRARY_PATH="$CUDA_HOME/lib64"; conda deactivate; conda activate $CONDA_ENV
conda env config vars set PATH="$CUDA_HOME:$CUDA_HOME/bin:$LD_LIBRARY_PATH:$PATH"; conda deactivate; conda activate $CONDA_ENV

3. Clone this repository and install requirements

git clone https://github.com/BOVIFOCR/tcdiff.git
cd tcdiff
./install.sh   # install dependencies and download needed pre-trained models

4. Train model

cd tcdiff
bash src/scripts/train_with_3DMM_consistency_constraints.sh

Model .ckpt will be saved at folder experiments_WITH_3DMM_CONSISTENCY_CONSTRAINTS/tcdiff/checkpoints/

5. Create new synthetic face dataset

For simplicity, we provide here the 10k synthetic identities generated and used by DCFace

• 10k synthetic identities