Skip to content

[ICML 2024] This repository includes the official implementation of our paper "Rejuvenating image-GPT as Strong Visual Representation Learners"

Notifications You must be signed in to change notification settings

OliverRensu/D-iGPT

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

23 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

D-iGPT

This repository is the official implementation of our Rejuvenating image-GPT as Strong Visual Representation Learners (accepted by ICML2024)

Introduction

This paper enhances image iGPT, one of the pioneering works that introduce autoregressive pretraining to predict next pixels for visual representation learning. Two simple yet essential changes are made. First, we shift the prediction target from raw pixels to semantic tokens, enabling a higher-level understanding of visual content. Second, we supplement the autoregressive modeling by instructing the model to predict not only the next tokens but also the visible tokens. This pipeline is particularly effective when semantic tokens are encoded by discriminatively trained models, such as CLIP. We introduce this novel approach as D-iGPT. Extensive experiments showcase that D-iGPT excels as a strong learner of visual representations: A notable achievement is its compelling performance on the ImageNet-1K dataset --- by training on publicly available datasets, D-iGPT unprecedentedly achieves 90.0% top-1 accuracy with a vanilla ViT-H. Additionally, D-iGPT shows strong generalization on the downstream task.

method

Model Training

We train our base size D-iGPT on torch+GPU and large/huge size D-iGPT on torch_xla+TPU. Please refer D-iGPT and D-iGPT_torchxla respectively.

Performance

Performance comparison on ImageNet-1K classification and ADE20K Semantic Segmentation.

Method Model Size Top-1 mIoU
MAE ViT-B 83.6 48.1
RandSac ViT-B 83.7 -
EVA ViT-B 85.5 53.3
D-iGPT ViT-B 86.2 53.8

The torch+GPU code produces better results. This is likely caused by the system difference between torch+GPU and torchxla+TPU.

ViT-Base
torch+GPU 86.2
torchxla+TPU 85.9

Performance comparison on ImageNet-1K classification with IN-21K as training data (if avaiable).

Method Model Size Top-1
MAE ViT-L 85.9
BEiT ViT-L 88.6
D-iGPT ViT-L 89.5
OpenClip ViT-H 88.6
D-iGPT ViT-H 90.0

Checkpoint

The pretrained models are available at [huggingface🤗]

Acknowledgement

We are very grateful that this work is supported by TPU Research Cloud (TRC) program and Google Cloud Research Credits program.

✍ Citation

@inproceedings{ren2024digpt,
   title = {Rejuvenating i-GPT for Scalable Visual Representation Learning},
   author = {Ren, Sucheng and Wang, Zeyu and Zhu, Hongru and Xiao, Junfei and Yuille, Alan and Xie, Cihang},
   booktitle = {ICML},
   year = {2024}
}

If you have any question, feel free to contact Sucheng Ren :)

About

[ICML 2024] This repository includes the official implementation of our paper "Rejuvenating image-GPT as Strong Visual Representation Learners"

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published