MobileViTv3 : Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Features [arXiv]

This repository contains MobileViTv3's source code for training and evaluation. It uses the CVNets library and is inspired by MobileViT (paper, code).

Installation and Training Models:

We recommend to use Python 3.8+ and PyTorch (version >= v1.8.0) with conda environment. For setting-up the python environment with conda, see here.

MobileViTv3-S,XS,XXS

Download MobileViTv1 and replace the files provided in MobileViTv3-v1. Conda environment used for training: environment_cvnet.yml. Then install according to instructions provided in the downloaded repository. For training, use training-and-evaluation readme provided in the downloaded repository.

MobileViTv3-1.0,0.75,0.5

Download MobileViTv2 and replace the files provided in MobileViTv3-v2. Conda environment used for training: environment_mbvt2.yml Then install according to instructions provided in the downloaded repository. For training, use training-and-evaluation readme provided in the downloaded repository.

Trained models:

Download the trained MobileViTv3 models from here. checkpoint_ema_best.pt files inside the model folder is used to generated the accuracy of models. Low-latency models are build by reducing the number of MobileViTv3-blocks in 'layer4' from 4 to 2. Please refer to the paper for more details. Note that for the segmentation and detection, only the backbone architecture parameters are listed.

Classification

ImageNet-1K:

Model name	Accuracy (%)	Parameters (Million)	FLOPs (Million)	Foldername
MobileViTv3-S	79.3	5.8	1841	mobilevitv3_S_e300_7930
MobileViTv3-XS	76.7	2.5	927	mobilevitv3_XS_e300_7671
MobileViTv3-XXS	70.98	1.2	289	mobilevitv3_XXS_e300_7098
MobileViTv3-1.0	78.64	5.1	1876	mobilevitv3_1_0_0
MobileViTv3-0.75	76.55	3.0	1064	mobilevitv3_0_7_5
MobileViTv3-0.5	72.33	1.4	481	mobilevitv3_0_5_0

ImageNet-1K using low-latency models:

Model name	Accuracy (%)	Parameters (Million)	FLOPs (Million)	Foldername
MobileViTv3-S-L2	79.06	5.2	1651	mobilevitv3_S_L2_e300_7906
MobileViTv3-XS-L2	76.10	2.3	853	mobilevitv3_XS_L2_e300_7610
MobileViTv3-XXS-L2	70.23	1.1	256	mobilevitv3_XXS_L2_e300_7023

Segmentation

PASCAL VOC 2012:

Model name	mIoU	Parameters (Million)	Foldername
MobileViTv3-S	79.59	7.2	mobilevitv3_S_voc_e50_7959
MobileViTv3-XS	78.77	3.3	mobilevitv3_XS_voc_e50_7877
MobileViTv3-XXS	74.04	2.0	mobilevitv3_XXS_voc_e50_7404
MobileViTv3-1.0	80.04	13.6	mobilevitv3_voc_1_0_0
MobileViTv3-0.5	76.48	6.3	mobilevitv3_voc_0_5_0

ADE20K:

Model name	mIoU	Parameters (Million)	Foldername
MobileViTv3-1.0	39.13	13.6	mobilevitv3_ade20k_1_0_0
MobileViTv3-0.75	36.43	9.7	mobilevitv3_ade20k_0_7_5
MobileViTv3-0.5	33.57	6.4	mobilevitv3_ade20k_0_5_0

Detection MS-COCO:

Model name	mAP	Parameters (Million)	Foldername
MobileViTv3-S	27.3	5.5	mobilevitv3_S_coco_e200_2730
MobileViTv3-XS	25.6	2.7	mobilevitv3_XS_coco_e200_2560
MobileViTv3-XXS	19.3	1.5	mobilevitv3_XXS_coco_e200_1930
MobileViTv3-1.0	27.0	5.8	mobilevitv3_coco_1_0_0
MobileViTv3-0.75	25.0	3.7	mobilevitv3_coco_0_7_5
MobileViTv3-0.5	21.8	2.0	mobilevitv3_coco_0_5_0

Citation

If you find this repository useful, please consider giving a star ⭐ and citation 📣:

@inproceedings{wadekar2022mobilevitv3,
  title = {MobileViTv3: Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Features},
  author = {Wadekar, Shakti N. and Chaurasia, Abhishek},
  doi = {10.48550/ARXIV.2209.15159},
  year = {2022}
}