Visual Recognition by Request

Code for the paper "Visual Recognition by Request" [arXiv].

Contact: chufeng.t@foxmail.com

NOTE: This release is currently a preliminary version, which could help you understand how the proposed algorithm works. We will release the complete version as well as the checkpoints in the near future.

Installation

This project is built upon several open-source toolboxes, follow the default instruction to install:

• MMSegmentation for whole-to-part semantic segmentation (Type-I requests): follow INSTALL.md to install the required packages and build the project locally (under the folder whole-to-part-semantic-segmentation).

• AdelaiDet for instance segmentation (Type-II requests): follow INSTALL.md to install the required packages and build the project locally (under the folder instance-segmentation).

• CLIP for text features: INSTALL.md.

Other requirements:

pip install cityscapesscripts
pip install panoptic_parts

Data Preparation

• Cityscapes-Panoptic-Parts (CPP): Download

• ADE20K (with Parts): Download (images, semantic and instance annotations)

Code for data processing will be coming soon.

Training and Inference

• Whole-to-part semantic segmentation (Type-I requests): follow train.md and inference.md. See available configs (whole-to-part-semantic-segmentation/configs/segmentation-by-request/).

• Instance segmentation (Type-II requests): follow Quick-Start.md. See available configs (instance-segmentation/configs/segmentation-by-request/).

Checkpoints will be coming soon.

Evaluation

Code for evaluation (e.g., HPQ computation) will be coming soon.

Reference

If this project is useful to your research, please consider cite:

@article{tang2022request,
  title={Visual Recognition by Request},
  author={Tang, Chufeng and Xie, Lingxi and Zhang, Xiaopeng and Hu, Xiaolin and Tian, Qi},
  journal={arXiv preprint arXiv:2207.14227},
  year={2022}
}