🤖 OccRWKV

Rethinking Efficient 3D Semantic Occupancy Prediction with Linear Complexity

📢 News

• [2024/09.28]: OccRWKV's logs are available for download:

OccRWKV Results	Experiment Log
OccRWKV on the SemanticKITTI hidden official test dataset	link
OccRWKV train log	link

• [2024/09.15]: The pre-trained model can be downloaded at OneDrive.
• [2024/09.01]: 🔥 We released the code of OccRWKV. The First Receptance Weighted Key Value (RWKV)-based 3D Semantic Occupancy Network

📜 Introduction

OccRWKV introduces an efficient semantic occupancy prediction network that smartly decouples semantics and occupancy with specialized RWKV blocks, achieving superior accuracy and real-time performance for 3D semantic scene understanding. Its use of BEV space and sparse feature projection significantly reduces computational demands, making it highly suitable for enhancing real-time autonomous navigation in robotic systems.

🚀 Citing

@article{wang2024occrwkv,
  title={OccRWKV: Rethinking Efficient 3D Semantic Occupancy Prediction with Linear Complexity},
  author={Wang, Junming and Yin, Wei and Long, Xiaoxiao and Zhang, Xingyu and Xing, Zebin and Guo, Xiaoyang and Zhang, Qian},
  journal={arXiv preprint arXiv:2409.19987},
  year={2024}
}

Please kindly star ⭐️ this project if it helps you. We take great efforts to develop and maintain it 😁.

🛠️ Installation

conda create -n occ_rwkv python=3.10 -y
conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia -y
pip install spconv-cu120
pip install tensorboardX
pip install dropblock
pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f https://data.pyg.org/whl/torch-2.1.0+cu121.html
pip install -U openmim
mim install mmcv-full
pip install mmcls==0.25.0

💽 Dataset

Please download the Semantic Scene Completion dataset (v1.1) from the SemanticKITTI website and extract it.

Or you can use voxelizer to generate ground truths of semantic scene completion.

The dataset folder should be organized as follows.

SemanticKITTI
├── dataset
│   ├── sequences
│   │  ├── 00
│   │  │  ├── labels
│   │  │  ├── velodyne
│   │  │  ├── voxels
│   │  │  ├── [OTHER FILES OR FOLDERS]
│   │  ├── 01
│   │  ├── ... ...

🤗 Getting Start

Clone the repository:

https://github.com/jmwang0117/OccRWKV.git

Train OccRWKV Net

$ cd <root dir of this repo>
$ bash scripts/run_train.sh

Validation

$ cd <root dir of this repo>
$ bash scripts/run_val.sh

Test

Since SemantiKITTI contains a hidden test set, we provide test routine to save predicted output in same format of SemantiKITTI, which can be compressed and uploaded to the SemanticKITTI Semantic Scene Completion Benchmark. You can provide which checkpoints you want to use for testing. We used the ones that performed best on the validation set during training. For testing, you can use the following command.

$ cd <root dir of this repo>
$ bash scripts/run_test.sh

🏆 Acknowledgement

Many thanks to these excellent open source projects:

• RWKV
• Vision-RWKV
• SSC-RS
• Co-Occ
• AGRNav
• HE-Nav