Jiho Choi1 *, Seonho Lee1 *, Seungho Lee2, Minhyun Lee2, Hyunjung Shim1
(* indicates equal contributions)
1Graduate School of Artificial Intelligence, KAIST, Republic of Korea
2School of Integrated Technology, Yonsei University, Republic of Korea
{jihochoi, glanceyes, kateshim}@kaist.ac.kr
, {seungholee, lmh315}@yonsei.ac.kr
PartCLIPSeg is a framework designed to enhance open-vocabulary part segmentation by integrating generalized parts with object-level contexts and attention controls, improving the model's ability to generalize and segment fine-grained parts accurately.
- [2024.10.10] 👨💻 The official codes have been released!
- [2024.09.26] 🎉 Our paper has been accepted to NeurIPS 2024!
- [2024.07.17] 📄 Our paper is now available! You can find the paper here.
# ------------------
# Init conda
# ------------------
conda create --name partclipseg python=3.8 -y
conda activate partclipseg
pip install --upgrade pip
conda install cuda=12.4.1 -c nvidia -y
pip install torch==2.2.2 torchvision==0.17.2 --index-url https://download.pytorch.org/whl/cu121
pip install timm==0.9.1
pip install scikit-image==0.21.0
pip install scikit-learn==0.24.2
pip install opencv-python==4.5.5.64
pip install hydra-core==1.3.2
pip install openmim==0.3.6
pip install mmsegmentation==0.29.1
pip install tokenizers==0.11.1
pip install Pillow~=9.5
pip install numpy==1.23.0
pip install einops ftfy regex fire ninja psutil gdown
# --------------------------
# Install Detectron2
# --------------------------
pip install 'git+https://github.com/facebookresearch/detectron2.git'
python -c "import detectron2; print(detectron2.__version__)" # 0.6
# --------------------------
# Install mmcv
# --------------------------
# pip install mmcv-full==1.7.1
# => if an error occurs
pip install mmcv-full==1.7.1 -f https://download.openmmlab.com/mmcv/dist/cu110/torch1.7.0/index.html
python -c "import mmcv; print(mmcv.__version__)" # 1.7.1
cd datasets
- You can find further information in the OV-PARTS GitHub repository.
gdown https://drive.google.com/uc?id=1QF0BglrcC0teKqx15vP8qJNakGgCWaEH
tar -xzf PascalPart116.tar.gz
find datasets/PascalPart116/images/val/ -name '._*' -delete
find datasets/PascalPart116/ -name '._*' -delete
gdown https://drive.google.com/uc?id=1EBVPW_tqzBOQ_DC6yLcouyxR7WrctRKi
tar -xzf ADE20KPart234.tar.gz
- Download the
LOC_synset_mapping.txt
file from this link and place it in thedatasets
folder. - Download
PartImageNet_Seg
from PartImageNet and extract it into thedatasets
folder.
- PascalPart116
- ADE20KPart234
- PartImageNet
# PascalPart116
python baselines/data/datasets/mask_cls_collect.py \
datasets/PascalPart116/annotations_detectron2_part/val \
datasets/PascalPart116/annotations_detectron2_part/val_part_label_count.json
python baselines/data/datasets/mask_cls_collect.py \
datasets/PascalPart116/annotations_detectron2_obj/val \
datasets/PascalPart116/annotations_detectron2_part/val_obj_label_count.json
# ADE20KPart234
# (no preprocessing required)
# PartImageNet
cd datasets
python partimagenet_preprocess.py --data_dir PartImageNet
# Make sure to have LOC_synset_mapping.txt in the datasets folder mentioned above.
Model | Setting | Dataset | Checkpoint |
---|---|---|---|
PartCLIPSeg | zero-shot | Pascal-Part-116 | model |
PartCLIPSeg | zero-shot | ADE20K-Part-234 | model |
mkdir weights && cd weights
# Pascal-Part-116
# PartCLIPSeg + Zero-shot (partclipseg_voc_oracle.pth (829M))
gdown https://drive.google.com/uc?id=1ytZpfutk__67GLwB-4jMClehJlnAzYFS
# ADE20K-Part-234
# PartCLIPSeg + Zero-shot (partclipseg_voc_oracle.pth (829M))
gdown https://drive.google.com/uc?id=10UTxrlamYq2rq3Uuv1GdOxpLy8f-QXoz
# -------------
# Train
# -------------
python train_net.py \
--num-gpus 8 \
--config-file configs/zero_shot/partclipseg_voc_oracle.yaml
# -----------------
# Inference
# -----------------
python train_net.py \
--num-gpus 8 \
--config-file configs/zero_shot/partclipseg_voc_oracle.yaml \
--eval-only MODEL.WEIGHTS ./weights/partclipseg_voc_oracle.pth
We would like to express our gratitude to the open-source projects and their contributors, including OV-PARTS, CLIPSeg, Mask2Former, CLIP, and OV-DETR.