Amir Bar, Xin Wang, Vadim Kantorov, Colorado J Reed, Roei Herzig, Gal Chechik, Anna Rohrbach, Trevor Darrell, Amir Globerson
This repository is the implementation of DETReg, see Project Page.
Recent self-supervised pretraining methods for object detection largely focus on pretraining the backbone of the object detector, neglecting key parts of detection architecture. Instead, we introduce DETReg, a new self-supervised method that pretrains the entire object detection network, including the object localization and embedding components. During pretraining, DETReg predicts object localizations to match the localizations from an unsupervised region proposal generator and simultaneously aligns the corresponding feature embeddings with embeddings from a self-supervised image encoder. We implement DETReg using the DETR family of detectors and show that it improves over competitive baselines when finetuned on COCO, PASCAL VOC, and Airbus Ship benchmarks. In low-data regimes, including semi-supervised and few-shot learning settings, DETReg establishes many state-of-the-art results, e.g., on COCO we see a +6.0 AP improvement for 10-shot detection and +3.5 AP improvement when training with only 1% of the labels.
Interact with the DETReg pretrained model in a Google Colab!
-
Linux, CUDA>=9.2, GCC>=5.4
-
Python>=3.7
We recommend you to use Anaconda to create a conda environment:
conda create -n detreg python=3.7 pip
Then, activate the environment:
conda activate detreg
Installation: (change cudatoolkit to your cuda version. For detailed pytorch installation instructions click here)
conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=10.2 -c pytorch
-
Other requirements
pip install -r requirements.txt
cd ./models/ops
sh ./make.sh
# unit test (should see all checking is True)
python test.pyDownload ImageNet and organize it in the following structure:
code_root/
└── data/
└── ilsvrc/
├── train/
└── val/
Note that in this work we also used the ImageNet100 dataset, which is x10 smaller than ImageNet. To create ImageNet100 run the following command:
mkdir -p data/ilsvrc100/train
mkdir -p data/ilsvrc100/val
code_root=/path/to/code_root
while read line; do ln -s "${code_root}/data/ilsvrc/train/$line" ${code_root}/data/ilsvrc100/train/$line"; done < "${code_root}/datasets/category.txt"
while read line; do ln -s "${code_root}/data/ilsvrc/val/$line" "${code_root}/data/ilsvrc100/val/$line"; done < "${code_root>/datasets/category.txt"This should results with the following structure:
code_root/
└── data/
├── ilsvrc/
├── train/
└── val/
└── ilsvrc100/
├── train/
└── val/
Please download COCO 2017 dataset and organize it in the following structure:
code_root/
└── data/
└── MSCoco/
├── train2017/
├── val2017/
└── annotations/
├── instances_train2017.json
└── instances_val2017.json
Download Pascal VOC dataset (2012trainval, 2007trainval, and 2007test):
mkdir -p data/pascal
cd data/pascal
wget http://host.robots.ox.ac.uk:8080/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar
wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar
wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar
tar -xvf VOCtrainval_11-May-2012.tar
tar -xvf VOCtrainval_06-Nov-2007.tar
tar -xvf VOCtest_06-Nov-2007.tarThe files should be organized in the following structure:
code_root/
└── data/
└── pascal/
└── VOCdevkit/
├── VOC2007
└── VOC2012
The command for pretraining DETReg, based on Deformable-DETR, on 8 GPUs on ImageNet is as follows:
GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_top30_in.sh --batch_size 24 --num_workers 8Using underlying DETR architecture:
GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_top30_in_detr.sh --batch_size 24 --num_workers 8The command for pretraining DETReg on 8 GPUs on ImageNet100 is as following:
GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_top30_in100.sh --batch_size 24 --num_workers 8Training takes around 1.5 days with 8 NVIDIA V100 GPUs, you can download a pretrained model (see below) if you want to skip this step.
After pretraining, a checkpoint is saved in exps/DETReg_top30_in/checkpoint.pth. To fine tune it over different coco settings use the following commands:
The command for pretraining DETReg on 8 GPUs on MSCoco is as following:
GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_top30_coco.sh --batch_size 24 --num_workers 8Fine tuning on full COCO (should take 2 days with 8 NVIDIA V100 GPUs):
GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_fine_tune_full_coco.shThis assumes a checkpoint exists in exps/DETReg_top30_in/checkpoint.pth.
Finetuning on MSCoco low-data regime, from full MSCoco pretraining (Semi-Supervised Learning setting)
Fine tuning on 1%
GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_fine_tune_1pct_coco.sh --batch_size 3Fine tuning on 2%
GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_fine_tune_2pct_coco.sh --batch_size 3Fine tuning on 5%
GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_fine_tune_5pct_coco.sh --batch_size 3Fine tuning on 10%
GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_fine_tune_10pct_coco.sh --batch_size 3Fine tune on full Pascal:
GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_fine_tune_full_pascal.sh --batch_size 4 --epochs 100 --lr_drop 70Fine tune on 10% of Pascal:
GPUS_PER_NODE=2 ./tools/run_dist_launch.sh 2 ./configs/DETReg_fine_tune_10pct_pascal.sh --batch_size 4 --epochs 200 --lr_drop 150For few-shot, please follow this code base for the dataloaders, classes, datasplits, etc. We used the few-shot dataset generated with seed = 0.
Finetune DETReg on base classes (60 classes, 99k labeled images). Similar hyperparams as in MSCoco finetuning. Then fine-tune it on few-shot labeled images (80 classes, every class has 10 or 30 instances) for 150 epochs, lr drop after 140 epochs.
Fine-tune DETReg on few-shot labeled images (80 classes, every class has 10 or 30 instances) for 1000 epochs, lr drop after 990 epochs.
To evaluate a finetuned model, use the following command from the project basedir:
./configs/<config file>.sh --resume exps/<config file>/checkpoint.pth --eval| Model | Type | Architecture | Dataset | Epochs | Checkpoint |
|---|---|---|---|---|---|
| DETReg | Pretraining | Deformable DETR | ImageNet | 5 | link |
| DETReg | Pretraining | DETR | ImageNet | 60 | link |
| DETReg | Pretraining | Deformable DETR | MSCoco | 50 | link |
| DETReg | Finetuned | Deformable DETR | MSCoco | 50 | link |
| DETReg | 10 Shot (w/ baseclass) | Deformable DETR | MSCoco | 150 | link |
| DETReg | 30 Shot (w/ baseclass) | Deformable DETR | MSCoco | 150 | link |
| DETReg | 10 Shot (no baseclass) | Deformable DETR | MSCoco | 1000 | link |
| DETReg | 30 Shot (no baseclass) | Deformable DETR | MSCoco | 1000 | link |
If you found this code helpful, feel free to cite our work:
@misc{bar2021detreg,
title={DETReg: Unsupervised Pretraining with Region Priors for Object Detection},
author={Amir Bar and Xin Wang and Vadim Kantorov and Colorado J Reed and Roei Herzig and Gal Chechik and Anna Rohrbach and Trevor Darrell and Amir Globerson},
year={2021},
eprint={2106.04550},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
If you found DETReg useful, consider checking out these related works as well: ReSim, SwAV, DETR, UP-DETR, and Deformable DETR.
- 07/17/23 - Release DETReg few-shot learning checkpoints pretrained finetuned on baseclass and from scratch (Table 3,4).
- 07/17/23 - Update few-shot learning results. New paper version.
- 04/28/22 - Bug fix in multiprocessing, affects Table 5 results. Up-to-date results here, new paper version will be uploaded tonight.
- 12/13/21 - Add DETR architecture
- 12/12/21 - Update experiments hyperparams in accordance with new paper version
- 12/12/21 - Avoid box caching on TopK policy (bug fix)
- 9/19/21 - Fixed Pascal VOC training with %X of training data
DETReg builds on previous works code base such as Deformable DETR and UP-DETR. If you found DETReg useful please consider citing these works as well.
DETReg is released under the Apache 2.0 license. Please see the LICENSE file for more information.
