Skip to content

Latest commit

 

History

History
135 lines (110 loc) · 7.51 KB

README.md

File metadata and controls

135 lines (110 loc) · 7.51 KB
Raw

Fully Attentional Networks

PWC PWC PWC

Project Page | Paper | Slides | Poster

Understanding The Robustness in Vision Transformers.
Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Anima Anandkumar, Jiashi Feng and Jose M. Alvarez.
International Conference on Machine Learning, 2022.

This repository contains the official Pytorch implementation of the training/evaluation code and the pretrained models of Fully Attentional Network (FAN).

FAN is a family of general-purpose Vision Transformer backbones that are highly robust to unseen natural corruptions in various visual recognition tasks.

Catalog

  • ImageNet-22K Fine-tuning Code Release
  • Cityscape-C and COCO-C Dataset Release
  • Pre-trained Model Release
  • Cityscape-C and COCO-C Dataset Generation Script
  • Downstream Transfer (Detection, Segmentation) Code Release
  • ImageNet-1K Training & Fine-tuning Code Release
  • Init Repo

Dependencies

The repo is built based on timm library, which can be installed via: pip3 install timm==0.5.4 pip3 install torchvision==0.9.0

Dataset preparation

Download ImageNet clean dataset and ImageNet-C dataset and structure the datasets as follows:

/path/to/imagenet-C/
  clean/
    class1/
      img3.jpeg
    class2/
      img4.jpeg
  corruption1/
    severity1/
      class1/
        img3.jpeg
      class2/
        img4.jpeg
    severity2/
      class1/
        img3.jpeg
      class2/
        img4.jpeg

For other out-of-distribution shift benchmarks, we use ImageNet-A or ImageNet-R for evaluation.

Results and Pre-trained Models

FAN-ViT ImageNet-1K trained models

Model Resolution IN-1K IN-C IN-A IN-R #Params Download
FAN-T-ViT 224x224 79.2 57.5 15.6 42.5 7.3M model
FAN-S-ViT 224x224 82.5 64.5 29.1 50.4 28.0M model
FAN-B-ViT 224x224 83.6 67.0 35.4 51.8 54.0M model
FAN-L-ViT 224x224 83.9 67.7 37.2 53.1 80.5M model

FAN-Hybrid ImageNet-1K trained models

Model Resolution IN-1K / IN-C City / City-C COCO / COCO-C #Params Download
FAN-T-Hybrid 224x224 80.1/57.4 81.2/57.1 50.2/33.1 7.4M model
FAN-S-Hybrid 224x224 83.5/64.7 81.5/66.4 53.3/38.7 26.3M model
FAN-B-Hybrid 224x224 83.9/66.4 82.2/66.9 54.2/40.6 50.4M model
FAN-L-Hybrid 224x224 84.3/68.3 82.3/68.7 55.1/42.0 76.8M model

FAN-Hybrid ImageNet-22K trained models

Model Resolution IN-1K/IN-C #Params Download
FAN-B-Hybrid 224x224 85.3/70.5 50.4M model
FAN-B-Hybrid 384x384 85.6/- 50.4M model
FAN-L-Hybrid 224x224 86.5/73.6 76.8M model
FAN-L-Hybrid 384x384 87.1/- 76.8M model

The pre-trained model weights for FAN-B-Hybrid and FAN-L-Hybrid on ImageNet22K without fine-tuning on ImageNet-1k are also uploaded. Checkpoints cabn be downloaded by clicking on the model name.

Demos

Semantic Segmentation on Cityscapes-C

animated

ImageNet-1K Training

FAN-T training on ImageNet-1K with 4 8-GPU nodes:

python3 -m torch.distributed.launch --nproc_per_node=8 --nnodes=$rank_num \
	--node_rank=$rank_index --master_addr="ip.addr" --master_port=$MASTER_PORT \
	 main.py  /PATH/TO/IMAGENET/ --model fan_tiny_8_p4_hybrid -b 32 --sched cosine --epochs 300 \
	--opt adamw -j 16 --warmup-epochs 5  \
	--lr 10e-4 --drop-path .1 --img-size 224 \
	--output ../fan_tiny_8_p4_hybrid/ \
	--amp --model-ema \

Robustness on ImageNet-C

bash scripts/imagenet_c_val.sh $model_name $ckpt

Measurement on ImageNet-A

bash scripts/imagenet_a_val.sh $model_name $ckpt

Measurement on ImageNet-R

bash scripts/imagenet_r_val.sh $model_name $ckpt

Acknowledgement

This repository is built using the timm library, DeiT, PVT and SegFormer repositories.

Citation

If you find this repository helpful, please consider citing:

@inproceedings{zhou2022understanding,
  title   = {Understanding The Robustness in Vision Transformers},
  author  = {Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Anima Anandkumar, Jiashi Feng, Jose M. Alvarez},
  booktitle = {International Conference on Machine Learning (ICML)},
  year    = {2022},
}