Skip to content

The Implementation of paper "Controllable Person Image Synthesis with Attribute-Decomposed GAN" CVPR 2020 (Oral); Pose and Appearance Attributes Transfer;

Notifications You must be signed in to change notification settings

menyifang/ADGAN

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

24 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ADGAN

PyTorch implementation for controllable person image synthesis.

ADGAN: Controllable Person Image Synthesis with Attribute-Decomposed GAN,
Yifang Men, Yiming Mao, Yuning Jiang, Wei-ying Ma, Zhouhui Lian
In: Proc. Computer Vision and Pattern Recognition (CVPR Oral), 2020 arXiv preprint (arXiv 2003.12267)

ADGAN++: Controllable Image Synthesis with Attribute-Decomposed GAN,
Guo Pu*, Yifang Men*, Yiming Mao, Yuning Jiang, Wei-ying Ma, Zhouhui Lian
In: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022. arXiv preprint (comming soon) Code

Updates

-(03/22/2022)ADGAN++, an extension vision with improved methods and more applications will be released soon.

Demo

Component Attribute Transfer

Pose Transfer

Requirement

  • python 3
  • pytorch(>=1.0)
  • torchvision
  • numpy
  • scipy
  • scikit-image
  • pillow
  • pandas
  • tqdm
  • dominate

Getting Started

You can directly download our generated images (in Deepfashion) from Google Drive.

Installation

  • Clone this repo:
git clone https://github.com/menyifang/ADGAN.git
cd ADGAN

Data Preperation

We use DeepFashion dataset and provide our dataset split files, extracted keypoints files and extracted segmentation files for convience.

The dataset structure is recommended as:

+—deepfashion
|   +—fashion_resize
|       +--train (files in 'train.lst')
|          +-- e.g. fashionMENDenimid0000008001_1front.jpg
|       +--test (files in 'test.lst')
|          +-- e.g. fashionMENDenimid0000056501_1front.jpg
|       +--trainK(keypoints of person images)
|          +-- e.g. fashionMENDenimid0000008001_1front.jpg.npy
|       +--testK
|          +-- e.g. fashionMENDenimid0000056501_1front.jpg.npy
|   +—semantic_merge
|   +—fashion-resize-pairs-train.csv
|   +—fashion-resize-pairs-test.csv
|   +—fashion-resize-annotation-pairs-train.csv
|   +—fashion-resize-annotation-pairs-test.csv
|   +—train.lst
|   +—test.lst
|   +—vgg19-dcbb9e9d.pth
|   +—vgg_conv.pth
...
  1. Person images
python tool/generate_fashion_datasets.py

Note: In our settings, we crop the images of DeepFashion into the resolution of 176x256 in a center-crop manner.

  1. Keypoints files
  • Download train/test pairs and train/test key points annotations from Google Drive, including fashion-resize-pairs-train.csv, fashion-resize-pairs-test.csv, fashion-resize-annotation-train.csv, fashion-resize-annotation-train.csv. Put these four files under the deepfashion directory.
  • Generate the pose heatmaps. Launch
python tool/generate_pose_map_fashion.py
  1. Segmentation files
  • Extract human segmentation results from existing human parser (e.g. Look into Person) and merge into 8 categories. Our segmentation results are provided in Google Drive, including ‘semantic_merge2’ and ‘semantic_merge3’ in different merge manner. Put one of them under the deepfashion directory.

Optionally, you can also generate these files by yourself.

  1. Keypoints files

We use OpenPose to generate keypoints.

  • Download pose estimator from Google Drive. Put it under the root folder ADGAN.
  • Change the paths input_folder and output_path in tool/compute_coordinates.py. And then launch
python2 compute_coordinates.py
  1. Dataset split files
python2 tool/create_pairs_dataset.py

Train a model

bash ./scripts/train.sh 

Test a model

Download our pretrained model from Google Drive. Modify your data path and launch

bash ./scripts/test.sh 

Evaluation

We adopt SSIM, IS, DS, CX for evaluation. This part is finished by Yiming Mao.

1) SSIM

For evaluation, Tensorflow 1.4.1(python3) is required.

python tool/getMetrics_market.py

2) DS Score

Download pretrained on VOC 300x300 model and install propper caffe version SSD. Put it in the ssd_score forlder.

python compute_ssd_score_fashion.py --input_dir path/to/generated/images

3) CX (Contextual Score)

Refer to folder ‘cx’ to compute contextual score.

Citation

If you use this code for your research, please cite our paper:

@inproceedings{men2020controllable,
  title={Controllable Person Image Synthesis with Attribute-Decomposed GAN},
  author={Men, Yifang and Mao, Yiming and Jiang, Yuning and Ma, Wei-Ying and Lian, Zhouhui},
  booktitle={Computer Vision and Pattern Recognition (CVPR), 2020 IEEE Conference on},
  year={2020}
}

@inproceedings{pu2022controllable,
  title={Controllable Image Synthesis with Attribute-Decomposed GAN},
  author={Pu, Guo and Men, Yifang and Mao, Yiming and Jiang, Yuning and Ma, Wei-Ying and Lian, Zhouhui},
  booktitle={Pattern Analysis and Machine Intelligence (TPAMI), 2022 IEEE Transactions on},
  year={2022}
}

Acknowledgments

Our code is based on PATN and thanks for their great work.

About

The Implementation of paper "Controllable Person Image Synthesis with Attribute-Decomposed GAN" CVPR 2020 (Oral); Pose and Appearance Attributes Transfer;

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published