Skip to content

Latest commit

 

History

History
300 lines (249 loc) · 16.9 KB

README.md

File metadata and controls

300 lines (249 loc) · 16.9 KB
Raw

PointGPT

PointGPT: Auto-regressively Generative Pre-training from Point Clouds ArXiv

In this work, we present PointGPT, a novel approach that extends the concept of GPT to point clouds, utilizing a point cloud auto-regressive generation task for pre-training transformer models. In object classification tasks, our PointGPT achieves 94.9% accuracy on the ModelNet40 dataset and 93.4% accuracy on the ScanObjectNN dataset, outperforming all other transformer models. In few-shot learning tasks, our method also attains new SOTA performance on all four benchmarks.

News

[2023.09.22] PointGPT has been accepted by NeurIPS 2023!

[2023.09.08] Unlabeled hybrid dataset and labeled hybrid dataset have been released!

[2023.08.19] Code has been updated; PointGPT-B and PointGPT-L models have been released!

[2023.06.20] Code and the PointGPT-S models have been released!

1. Requirements

PyTorch >= 1.7.0; python >= 3.7; CUDA >= 9.0; GCC >= 4.9; torchvision;

pip install -r requirements.txt

# Chamfer Distance & emd
cd ./extensions/chamfer_dist
python setup.py install --user
cd ./extensions/emd
python setup.py install --user
# PointNet++
pip install "git+https://github.com/erikwijmans/Pointnet2_PyTorch.git#egg=pointnet2_ops&subdirectory=pointnet2_ops_lib"
# GPU kNN
pip install --upgrade https://github.com/unlimblue/KNN_CUDA/releases/download/0.2/KNN_CUDA-0.2-py3-none-any.whl

2. Datasets

Our training data for the PointGPT-S model encompasses ShapeNet, ScanObjectNN, ModelNet40, and ShapeNetPart datasets. For detailed information, please refer to DATASET.md.

To pretrain the PointGPT-B and PointGPT-L models, we employ both unlabeled hybrid dataset and labeled hybrid dataset, available for download here.

3. PointGPT Models

PointGPT-S Models

Task Dataset Config Acc. Download
Pre-training ShapeNet pretrain.yaml N.A. here
Classification ScanObjectNN finetune_scan_hardest.yaml 86.9% here
Classification ScanObjectNN finetune_scan_objbg.yaml 91.6% here
Classification ScanObjectNN finetune_scan_objonly.yaml 90.0% here
Classification ModelNet40(1k) finetune_modelnet.yaml 94.0% here
Classification ModelNet40(8k) finetune_modelnet_8k.yaml 94.2% here
Part segmentation ShapeNetPart segmentation 86.2% mIoU here
Task Dataset Config 5w10s Acc. (%) 5w20s Acc. (%) 10w10s Acc. (%) 10w20s Acc. (%)
Few-shot learning ModelNet40 fewshot.yaml 96.8 ± 2.0 98.6 ± 1.1 92.6 ± 4.6 95.2 ± 3.4

PointGPT-B Models

Task Dataset Config Acc. Download
Pre-training UnlabeledHybrid pretrain.yaml N.A. here
Post-pre-training LabeledHybrid post_pretrain.yaml N.A. here
Classification ScanObjectNN finetune_scan_hardest.yaml 91.9% here
Classification ScanObjectNN finetune_scan_objbg.yaml 95.8% here
Classification ScanObjectNN finetune_scan_objonly.yaml 95.2% here
Classification ModelNet40(1k) finetune_modelnet.yaml 94.4% here
Classification ModelNet40(8k) finetune_modelnet_8k.yaml 94.6% here
Part segmentation ShapeNetPart segmentation 86.5% mIoU here
Task Dataset Config 5w10s Acc. (%) 5w20s Acc. (%) 10w10s Acc. (%) 10w20s Acc. (%)
Few-shot learning ModelNet40 fewshot.yaml 97.5 ± 2.0 98.8 ± 1.0 93.5 ± 4.0 95.8 ± 3.0

PointGPT-L Models

Task Dataset Config Acc. Download
Pre-training UnlabeledHybrid pretrain.yaml N.A. here
Post-pre-training LabeledHybrid post_pretrain.yaml N.A. here
Classification ScanObjectNN finetune_scan_hardest.yaml 93.4% here
Classification ScanObjectNN finetune_scan_objbg.yaml 97.2% here
Classification ScanObjectNN finetune_scan_objonly.yaml 96.6% here
Classification ModelNet40(1k) finetune_modelnet.yaml 94.7% here
Classification ModelNet40(8k) finetune_modelnet_8k.yaml 94.9% here
Part segmentation ShapeNetPart segmentation 86.6% mIoU here
Task Dataset Config 5w10s Acc. (%) 5w20s Acc. (%) 10w10s Acc. (%) 10w20s Acc. (%)
Few-shot learning ModelNet40 fewshot.yaml 98.0 ± 1.9 99.0 ± 1.0 94.1 ± 3.3 96.1 ± 2.8

4. PointGPT Pre-training

To pretrain PointGPT, run the following command.

CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/<MODEL_NAME>/pretrain.yaml --exp_name <output_file_name>

To post-pretrain PointGPT, run the following command.

CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/<MODEL_NAME>/post_pretrain.yaml --exp_name <output_file_name> --finetune_model

5. PointGPT Fine-tuning

Fine-tuning on ScanObjectNN, run the following command:

CUDA_VISIBLE_DEVICES=<GPUs> python main.py --config cfgs/<MODEL_NAME>/finetune_scan_hardest.yaml \
--finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>

Fine-tuning on ModelNet40, run the following command:

CUDA_VISIBLE_DEVICES=<GPUs> python main.py --config cfgs/<MODEL_NAME>/finetune_modelnet.yaml \
--finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>

Voting on ModelNet40, run the following command:

CUDA_VISIBLE_DEVICES=<GPUs> python main.py --test --config cfgs/<MODEL_NAME>/finetune_modelnet.yaml \
--exp_name <output_file_name> --ckpts <path/to/best/fine-tuned/model>

Few-shot learning, run the following command:

CUDA_VISIBLE_DEVICES=<GPUs> python main.py --config cfgs/<MODEL_NAME>/fewshot.yaml --finetune_model \
--ckpts <path/to/pre-trained/model> --exp_name <output_file_name> --way <5 or 10> --shot <10 or 20> --fold <0-9>

Part segmentation on ShapeNetPart, run the following command:

cd segmentation
python main.py --ckpts <path/to/pre-trained/model> --root path/to/data --learning_rate 0.0002 --epoch 300 --model_name <MODEL_NAME>

6. Visualization

Visulization of pre-trained model on validation set, run:

python main_vis.py --test --ckpts <path/to/pre-trained/model> --config cfgs/<MODEL_NAME>/pretrain.yaml --exp_name <name>

7. Ablation studies on post-pre-training stage

Methods ScanObjectNN ModelNet40 ShapeNetPart
OBJ_BG OBJ_ONLY PB_T50_RS 1k P 8k P Cls.mIoU Inst.mIoU
without post-pre-training
PointGPT-B 93.6 92.5 89.6 94.2 94.4 84.5 86.4
PointGPT-L 95.7 94.1 91.1 94.5 94.7 84.7 86.5
with post-pre-training
PointGPT-B 95.8 (+2.2) 95.2 (+2.7) 91.9 (+2.3) 94.4 (+0.2) 94.6 (+0.2) 84.5 (+0.0) 86.5 (+0.1)
PointGPT-L 97.2 (+1.5) 96.6 (+2.5) 93.4 (+2.3) 94.7 (+0.2) 94.9 (+0.2) 84.8 (+0.1) 86.6 (+0.1)

Acknowledgements

Our codes are built upon Point-MAE, Point-BERT, Pointnet2_PyTorch and Pointnet_Pointnet2_pytorch

The unlabeled hybrid dataset and labeled hybrid dataset are built upon ModelNet40, PartNet, ShapeNet, S3DIS, ScanObjectNN, SUN RGB-D, and Semantic3D

Reference

@article{chen2024pointgpt,
  title={Pointgpt: Auto-regressively generative pre-training from point clouds},
  author={Chen, Guangyan and Wang, Meiling and Yang, Yi and Yu, Kai and Yuan, Li and Yue, Yufeng},
  journal={Advances in Neural Information Processing Systems},
  volume={36},
  year={2024}
}

For unlabeled hybrid dataset or labeled hybrid dataset, please also cite the following work.

@inproceedings{wu20153d,
  title={3d shapenets: A deep representation for volumetric shapes},
  author={Wu, Zhirong and Song, Shuran and Khosla, Aditya and Yu, Fisher and Zhang, Linguang and Tang, Xiaoou and Xiao, Jianxiong},
  booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
  pages={1912--1920},
  year={2015}
}

@inproceedings{mo2019partnet,
  title={Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding},
  author={Mo, Kaichun and Zhu, Shilin and Chang, Angel X and Yi, Li and Tripathi, Subarna and Guibas, Leonidas J and Su, Hao},
  booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
  pages={909--918},
  year={2019}
}

@article{chang2015shapenet,
  title={Shapenet: An information-rich 3d model repository},
  author={Chang, Angel X and Funkhouser, Thomas and Guibas, Leonidas and Hanrahan, Pat and Huang, Qixing and Li, Zimo and Savarese, Silvio and Savva, Manolis and Song, Shuran and Su, Hao and others},
  journal={arXiv preprint arXiv:1512.03012},
  year={2015}
}

@inproceedings{armeni20163d,
  title={3d semantic parsing of large-scale indoor spaces},
  author={Armeni, Iro and Sener, Ozan and Zamir, Amir R and Jiang, Helen and Brilakis, Ioannis and Fischer, Martin and Savarese, Silvio},
  booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
  pages={1534--1543},
  year={2016}
}

@inproceedings{uy-scanobjectnn-iccv19,
  title = {Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data},
  author = {Mikaela Angelina Uy and Quang-Hieu Pham and Binh-Son Hua and Duc Thanh Nguyen and Sai-Kit Yeung},
  booktitle = {International Conference on Computer Vision (ICCV)},
  year = {2019}
}

@inproceedings{song2015sun,
  title={Sun rgb-d: A rgb-d scene understanding benchmark suite},
  author={Song, Shuran and Lichtenberg, Samuel P and Xiao, Jianxiong},
  booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
  pages={567--576},
  year={2015}
}

@article{hackel2017semantic3d,
  title={Semantic3d. net: A new large-scale point cloud classification benchmark},
  author={Hackel, Timo and Savinov, Nikolay and Ladicky, Lubor and Wegner, Jan D and Schindler, Konrad and Pollefeys, Marc},
  journal={arXiv preprint arXiv:1704.03847},
  year={2017}
}