Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models (NeurIPS 2024)

Technical details

[Paper link] [Poster link] [Video link]

Plan for release

• Release the concept decriptions we used [in /concept_description].
• Release the model checkpoints and inference codes.
• Release the training scripts.

Requirement

1. Clone this repository and navigate to DRM folder
git clone https://github.com/vaynexie/DRM.git
cd DRM

2. Install Package
conda create -n drm python=3.10 -y
conda activate drm
pip install -r requirements.txt

Inference

	Dataset [please download and unzip the dataset under the data folder]	Checkpoint [please download the checkpoints and put them under the ckpts folder ]	Inference
iWildCam	See link in data/readme.md	Link CLIP ViT-B/16 (ckpts/iwildcam_vit_b16.pt) [ID tesing F1: 0.5353, OOD tesing F1: 0.4049] CLIP ViT-L/14 (ckpts/iwildcam_vit_l14.pt) [ID tesing F1: 0.6222, OOD tesing F1: 0.4875] CLIP ViT-L/14@336px (ckpts/iwildcam_vit_l14_336.pt) [ID tesing F1: 0.6273 , OOD tesing F1: 0.5139]	bash src/eval_iwildcam.sh
FMoW	See link in data/readme.md	Link CLIP ViT-B/16 (ckpts/fmow_vit_b16.pt) [ID tesing acc: 0.6857, OOD tesing worst-region acc: 0.4566] CLIP ViT-L/14 (ckpts/fmow_vit_l14.pt) [ID tesing acc: 0.7093, OOD tesing worst-region acc: 0.5137] CLIP ViT-L/14@336px (ckpts/fmow_vit_l14_336.pt) [ID tesing acc: 0.7389 , OOD tesing worst-region acc: 0.5253]	bash src/eval_fmow.sh
ImageNet	See links in data/readme.md	Link CLIP ViT-B/16 (ckpts/imagenet_vit_b16.pt) [Acc - Val: 0.8205, V2: 0.7343, R: 0.7782, A: 0.5353, Sketch: 0.5248]	bash src/eval_imagenet.sh

Cite us

@inproceedings{
li2024dual,
title={Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models},
author={Kaican Li and Weiyan Xie and Yongxiang Huang and Didan Deng and Lanqing HONG and Zhenguo Li and Ricardo Silva and Nevin L. Zhang},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=p50Dyqk0GX}
}

Acknowledgement

Our code is modified based on FLYP, WILDS, CLIP, CoOp, thanks to all the contributors!

---

Correspondence to: Vayne Xie (wxieai@cse.ust.hk）