Customizing Synthetic data for Data-Free Student Learning

This is the implementation for the ICME2023 paper: Customizing Synthetic Data for Data-Free Student Learning.

1. Environment

This repository is tested with Ubuntu 18.04.5 LTS, python 3.6.5, pytorch 1.7.1 and cuda 11.4.

All experiments are conducted on one NVIDIA GeForce RTX 2080Ti GPU.

2. Get the pre-trained teacher model

python train_scratch.py --model wrn40_2 --dataset cifar10

After the training is completed, the teacher model will be saved as checkpoints/pretrained/cifar10_wrn40_2.pth.

Or you can directly download pre-trained teacher models from Dropbox-Models (266 MB) and extract them as checkpoints/pretrained.

3. Initialize image bank

To prevent the student from overfitting to data generated by early training rounds, it is necessary to synthesize some data firstly to initialize image bank by removing $\mathcal{L}_{csd}$ , i.e., setting the flag --csd to 0, and running 400 synthesis batches with each one containing 200 samples.

bash scripts/csd/csd_cifar10_initBank_wrn402.sh

4. Train student with CSD

bash scripts/csd/csd_cifar10_wrn402_wrn161.sh

5. Train student with comparison methods

bash scripts/xxx/xxx.sh # e.g. scripts/zskt/zskt_cifar10_wrn402_wrn161.sh

The implement of the comparison methods are based on a public library developed by CMI [1].

[1] Gongfan Fang, Jie Song, Xinchao Wang, Chengchao Shen, Xingen Wang, and Mingli Song. Contrastive model invertion for data-free knolwedge distillation. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 2374–2380, 2021.