Self-Support Few-Shot Semantic Segmentation

Qi Fan, Wenjie Pei, Yu-Wing Tai, Chi-Keung Tang

The codebase contains the official code of our paper Self-Support Few-Shot Semantic Segmentation, ECCV 2022.

中文解读: ECCV 2022 | SSP: 自支持匹配的小样本任务新思想

Data preparation

Download

Pretrained model: ResNet-50 | ResNet-101

Dataset: Pascal images and ids | Semantic segmentation annotations

File Organization

You can follow HSNet to prepare the MS COCO and FSS-1000 datasets.

    ../                         # parent directory
    ├── ./SSP                   # current (project) directory
    |   ├── codes               # various codes
    |   └── ./pretrained        # pretrained model directory
    |            ├── resnet50.pth
    |            └── resnet101.pth
    └── Datasets_SSP/
        ├── VOC2012/            # PASCAL VOC2012 devkit
        │   ├── Annotations/
        │   ├── ImageSets/
        │   ├── ...
        │   └── SegmentationClassAug/
        ├── COCO2014/           
        │   ├── annotations/
        │   │   ├── train2014/  # (dir.) training masks (from Google Drive) 
        │   │   ├── val2014/    # (dir.) validation masks (from Google Drive)
        │   │   └── ..some json files..
        │   ├── train2014/
        │   └── val2014/
        └── FSS-1000/           # (dir.) contains 1000 object classes
            ├── abacus/   
            ├── ...
            └── zucchini/

Run the code

You can adapt the scripts of all.sh, test.sh and hsnet_test.sh (for the HSNet evaluation protocol) to train and evaluate your models.

CUDA_VISIBLE_DEVICES=0,1 python -W ignore main.py \
  --dataset pascal --data-root [Your Pascal Path] \
  --backbone resnet50 --fold 0 --shot 1

You may change the backbone from resnet50 to resnet101, change the fold from 0 to 1/2/3, or change the shot from 1 to 5 for other settings.

You can add/remove --refine to enable/disable the self-support refinement.

Performance and Trained Models

Pascal VOC

Method	Setting	Backbone	SSP Refine	Fold 0	Fold 1	Fold 2	Fold 3	Mean
Baseline	1-shot	ResNet-50	No	54.9	66.5	61.7	48.3	57.9
Baseline	1-shot	ResNet-101	No	57.2	68.5	61.3	53.3	60.1
Baseline	5-shot	ResNet-50	No	61.6	70.3	70.5	56.4	64.7
Baseline	5-shot	ResNet-101	No	64.2	74.0	71.5	61.3	67.8
SSP (Ours)	1-shot	ResNet-50	Yes	61.4	67.8	66.5	50.9	61.7
SSP (Ours)	1-shot	ResNet-101	Yes	63.2	70.4	68.5	56.3	64.6
SSP (Ours)	5-shot	ResNet-50	Yes	67.5	72.3	75.2	62.1	69.3
SSP (Ours)	5-shot	ResNet-101	Yes	70.9	77.1	78.9	66.1	73.3

MS COCO

Method	Setting	Backbone	Eval Protocol	Fold 0	Fold 1	Fold 2	Fold 3	Mean
SSP (Ours)	1-shot	ResNet-50	Ours	46.4	35.2	27.3	25.4	33.6
SSP (Ours)	1-shot	ResNet-101	Ours	50.4	39.9	30.6	30.0	37.7
SSP (Ours)	5-shot	ResNet-50	Ours	53.9	42.0	36.0	33.7	41.4
SSP (Ours)	5-shot	ResNet-101	Ours	57.8	47.0	40.2	39.9	46.2
SSP (Ours)	1-shot	ResNet-50	HSNet	35.5	39.6	37.9	36.7	37.4
SSP (Ours)	1-shot	ResNet-101	HSNet	39.1	45.1	42.7	41.2	42.0
SSP (Ours)	5-shot	ResNet-50	HSNet	40.6	47.0	45.1	43.9	44.1
SSP (Ours)	5-shot	ResNet-101	HSNet	47.4	54.5	50.4	49.6	50.2

Acknowledgement

This codebase is built based on MLC's baseline code and we borrow HSNet's evaluation protocol for the MS COCO dataset. We thank MLC and other FSS works for their great contributions.

Other related repos

Few-shot image/video object detection: FewX

Other related papers

@inproceedings{fan2021fsvod,
  title={Few-Shot Video Object Detection},
  author={Fan, Qi and Tang, Chi-Keung and Tai, Yu-Wing},
  booktitle={ECCV},
  year={2022}
}
@inproceedings{fan2020cpmask,
  title={Commonality-Parsing Network across Shape and Appearance for Partially Supervised Instance Segmentation},
  author={Fan, Qi and Ke, Lei and Pei, Wenjie and Tang, Chi-Keung and Tai, Yu-Wing},
  booktitle={ECCV},
  year={2020}
}
@inproceedings{fan2020fsod,
  title={Few-Shot Object Detection with Attention-RPN and Multi-Relation Detector},
  author={Fan, Qi and Zhuo, Wei and Tang, Chi-Keung and Tai, Yu-Wing},
  booktitle={CVPR},
  year={2020}
}

Citation

@inproceedings{fan2022ssp,
  title={Self-Support Few-Shot Semantic Segmentation},
  author={Fan, Qi and Pei, Wenjie and Tai, Yu-Wing and Tang, Chi-Keung},
  journal={ECCV},
  year={2022}
}