Skip to content

Latest commit

 

History

History
 
 

Mini-DeiT

README.md

Mini-DeiT

This repo is for MiniViT for DeiTs.

Model Zoo

Model Params. Input Top-1 Acc. % Top-5 Acc. % Download link
Mini-DeiT-Ti 3M 224x224 73.0 91.6 model, log
Mini-DeiT-S 11M 224x224 80.9 95.6 model, log
Mini-DeiT-B 44M 224x224 83.2 96.5 model, log
Mini-DeiT-B 44M 384x384 84.9 97.2 model, log

Usage

Create the environment:

pip install -r requirements.txt

Compile operations:

cd rpe_ops
python setup.py install --user

Data Preparation

You can download the ImageNet-1K dataset from http://www.image-net.org/.

The train set and validation set should be saved as the *.tar archives:

ImageNet/
├── train.tar
└── val.tar

Our code also supports storing images as individual files as follow:

ImageNet/
├── train
│   ├── n01440764
│   │   ├── n01440764_10026.JPEG
│   │   ├── n01440764_10027.JPEG
...
├── val
│   ├── n01440764
│   │   ├── ILSVRC2012_val_00000293.JPEG

Training

Training Mini-DeiT-Ti

python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --model mini_deit_tiny_patch16_224 --batch-size 128 --data-path ./ImageNet --output_dir ./outputs  --teacher-model regnety_160 --distillation-type soft --distillation-alpha 1.0 --drop-path 0.0
Training Mini-DeiT-S 
python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --model mini_deit_small_patch16_224 --batch-size 128 --data-path ./ImageNet --output_dir ./outputs  --teacher-model regnety_160 --distillation-type soft --distillation-alpha 1.0 --drop-path 0.0
Training Mini-DeiT-B 
python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --model mini_deit_base_patch16_224 --batch-size 128 --data-path ./ImageNet --output_dir ./outputs  --teacher-model regnety_160 --distillation-type soft --distillation-alpha 1.0 --drop-path 0.1
Finetune Mini-DeiT-B with resolution 384 
python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --model mini_deit_base_patch16_384 --batch-size 32 --data-path ./ImageNet --output_dir ./outputs --finetune checkpoints/mini_deit_base_patch16_224.pth --input-size 384 --lr 5e-6 --min-lr 5e-6 --weight-decay 1e-8 --epochs 30

Evaluation

Run the following commands for evaluation:

Evaluate Mini-DeiT-Ti

python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --model mini_deit_tiny_patch16_224 --batch-size 128 --data-path ./ImageNet --output_dir ./outputs  --resume ./checkpoints/mini_deit_tiny_patch16_224.pth --eval
Evaluate Mini-DeiT-S 
python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --model mini_deit_small_patch16_224 --batch-size 128 --data-path ./ImageNet --output_dir ./outputs  --resume ./checkpoints/mini_deit_small_patch16_224.pth --eval
Evaluate Mini-DeiT-B 
python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --model mini_deit_base_patch16_224 --batch-size 128 --data-path ./ImageNet --output_dir ./outputs  --resume ./checkpoints/mini_deit_base_patch16_224.pth --eval
Evaluate Mini-DeiT-B-384 
python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --model mini_deit_base_patch16_384 --batch-size 32 --data-path ./ImageNet --output_dir ./outputs  --resume ./checkpoints/mini_deit_base_patch16_384.pth --input-size 384 --eval

Bibtex

If this repo is helpful for you, please consider to cite it. Thank you! :)

@article{MiniViT,
  title={MiniViT: Compressing Vision Transformers with Weight Multiplexing},
  author={Zhang, Jinnian and Peng, Houwen and Wu, Kan and Liu, Mengchen and Xiao, Bin and Fu, Jianlong and Yuan, Lu},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2022}
}

License

Our code is based on DeiT. Thank you!

Apache License