TAPPS: Task-aligned Part-aware Panoptic Segmentation (CVPR 2024)

[Project page] [Paper]

Code for 'Task-aligned Part-aware Panoptic Segmentation through Joint Object-Part Representations', Daan de Geus and Gijs Dubbelman, CVPR 2024.

Installation

• Follow the installation instructions.

Getting Started

• Prepare the datasets for TAPPS.
• To use COCO panoptic pre-training, download the checkpoints following these instructions.

Training

To train a model, you need a configuration file. We provide default configuration files for the models presented in our work. Our configs are designed to be trained on 4 GPUs, using a different number of GPUs likely requires changes to the learning rate.

python train_net.py --num-gpus 4 \
                    --config-file /PATH/TO/CONFIG/FILE.yaml 

Example:

python train_net.py --num-gpus 4 \
                    --config-file configs/pascal/pps/tapps_pascal_r50_cocoinit.yaml

Evaluation

Evaluating a model on the PartPQ metric requires two steps: (1) making and saving the predictions, (2) evaluating the predictions and calculating the PartPQ.

1. Making predictions and saving them

This step requires a configuration file and the model weights. We provide default configuration files and weights for the models presented in our work. You also need to define a directory where you wish to store the predictions.

python train_net.py --num-gpus 4 \
                    --config-file /PATH/TO/CONFIG/FILE.yaml \
                    --eval-only \
                    MODEL.WEIGHTS /PATH/TO/MODEL/WEIGHTS.bin \
                    SAVE_PREDICTIONS True \
                    SAVE_DIR /PATH/WHERE/PREDICTIONS/ARE/STORED/

Example:

python train_net.py --num-gpus 4 \
                    --config-file configs/pascal/pps/tapps_pascal_r50_cocoinit.yaml \
                    --eval-only \
                    MODEL.WEIGHTS checkpoints/tapps_pascal_r50_cocoinit.bin \
                    SAVE_PREDICTIONS True \
                    SAVE_DIR predictions/

2. Evaluating the predictions and calculating the PartPQ

Provide the path where the predictions are stored and specify the dataset.

python eval/eval_partpq.py --save_dir /PATH/WHERE/PREDICTIONS/ARE/STORED/ \
                           --dataset DATASET_NAME  # 'pascal', 'cityscapes' or 'pascal107'

Example:

python eval/eval_partpq.py --save_dir predictions/ \
                           --dataset pascal 

Visualization

To visualize the part-aware panoptic segmentation predictions that you stored (see Evaluation step 1), run:

python eval/visualize_pps.py --pred_dir /PATH/WHERE/PREDICTIONS/ARE/STORED/ \
                             --save_dir /PATH/WHERE/VISUALIZATIONS/WILL/BE/STORED/ \
                             --dataset DATASET_NAME  # 'pascal', 'cityscapes' or 'pascal107'

Models

Check this page for trained models and associated config files.

License

Shield:

This code builds upon the official Mask2Former code. The majority of Mask2Former is licensed under an MIT License. However, portions of the Mask2Former project are available under separate license terms: Swin-Transformer-Semantic-Segmentation is licensed under the MIT license, Deformable-DETR is licensed under the Apache-2.0 License.

This code also uses the panoptic_parts code, which is licensed under the Apache-2.0 license.

The remaining code, specifically added for TAPPS, is licensed under an MIT license.

Citing us

Please consider citing our work if it is useful for your research.

@inproceedings{degeus2024tapps,
  title={{Task-aligned Part-aware Panoptic Segmentation through Joint Object-Part Representations}},
  author={{de Geus}, Daan and Dubbelman, Gijs},
  booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2024}
}

If you use the part-aware panoptic segmentation task in your work, consider citing:

@inproceedings{degeus2021pps,
  title = {{Part-aware Panoptic Segmentation}},
  author = {{de Geus}, Daan and Meletis, Panagiotis and Lu, Chenyang and Wen, Xiaoxiao and Dubbelman, Gijs},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2021},
}

Acknowledgement

This project is built on top of the official code of Mask2Former, we thank the authors for their great work and useful code.