Official implementation of Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning (IJCAI 2019) in pytorch.
Teacher Model | Dataset | num_classes | Acc |
---|---|---|---|
ResNet18 | CUB200 | 200 | 0.7411 |
ResNet34 | StanfordDogs | 120 | 0.8663 |
Target Model | KD | CFL |
---|---|---|
ResNet34 | 0.7684 | 0.7721 |
ResNet50 | 0.7965 | 0.7997 |
DenseNet121 | 0.7769 | 0.7815 |
see logs for more information
Feature Space
: space constructed with intermediate outputs.
Common Space
: common feature space in CFL Blocks.
Some Feature spaces are None because of different feature dimensions (e.g. 2048 for ResNet50 but 512 for ResNet34)
Target Model | Common Space | Feature Space |
---|---|---|
ResNet34 | ||
ResNet50 | None | |
DenseNet121 | None |
python download_data.py
ResNet18 & ResNet34, 242.9 MB
Google Drive
BaiDu Yun
python amal.py --model resnet34 --gpu_id 0 --lr 1e-4 --cfl_lr 5e-4
python kd.py --model resnet34 --gpu_id 0 --lr 1e-4
or
bash run_all.sh
cd logs/
python draw_acc_curve.py
TSNE results will be saved at tsne_results/MODEL_NAME/
# ResNet34
python tsne_common_space.py --ckpt checkpoints/amal_resnet34_best.pth --t1_ckpt checkpoints/cub200_resnet18_best.pth --t2_ckpt checkpoints/dogs_resnet34_best.pth --gpu_id 0
# ResNet50
python tsne_common_space.py --ckpt checkpoints/amal_resnet50_best.pth --t1_ckpt checkpoints/cub200_resnet18_best.pth --t2_ckpt checkpoints/dogs_resnet34_best.pth --gpu_id 0
# DenseNet121
python tsne_common_space.py --ckpt checkpoints/amal_densenet121_best.pth --t1_ckpt checkpoints/cub200_resnet18_best.pth --t2_ckpt checkpoints/dogs_resnet34_best.pth --gpu_id 0
@inproceedings{luo2019knowledge,
title={Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning},
author={Luo, Sihui and Wang, Xinchao and Fang, Gongfan and Hu, Yao and Tao, Dapeng and Song, Mingli},
booktitle={Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI)},
year={2019},
}
Sihui Luo | sihuiluo829@zju.edu.cn |
Gongfan Fang | fgfvain97@zju.edu.cn |
Homepage of VIPA Group, Zhejiang University, China