Skip to content

Latest commit

 

History

History
77 lines (60 loc) · 3.37 KB

README.md

File metadata and controls

77 lines (60 loc) · 3.37 KB
Raw

QTNet

[NeurIPS 2023] Query-based Temporal Fusion with Explicit Motion for 3D Object Detection

paper

Introduction

This repository is an official implementation of QTNet.

In this paper, we propose a simple and effective Query-based Temporal Fusion Network (QTNet). The main idea is to exploit the object queries in previous frames to enhance the representation of current object queries by the proposed Motion-guided Temporal Modeling (MTM) module, which utilizes the spatial position information of object queries along the temporal dimension to construct their relevance between adjacent frames reliably. Our method can be integrated into some advanced LiDAR-only or multi-modality 3D detectors and even brings new SOTA performance with negligible computation cost and latency on the nuScenes dataset.

News

  • [2023-12-06] The memory bank training code is released.
  • [2023-09-22] QTNet is accepted by NeurIPS 2023. 🔥

Preparation

  • Memory Bank Data

    Please install the memory bank of TransFusion-L, which contains query features and detection results.

  • Training and Validation Infos

    You need to run the data conver script in this repo (create_data). We also provided the processed training and validation infos.

After preparation, you will be able to see the following directory structure:

QTNet
├── configs
│   ├── qtnet.py
├── mmdet3d
├── tools
├── data
│   ├── nuscenes
│     ├── memorybank
│     ├── ...
├── ...
├── README.md

Train & inference

You can train the model following:

tools/dist_train.sh configs/qtnet.py 8 --work-dir work_dirs/qtnet_4frames/

You can evaluate the model following:

tools/dist_test.sh configs/qtnet.py work_dirs/qtnet_4frames/latest.pth 8 --eval mAP

Results on NuScenes Val Set.

Model Setting mAP NDS Config Download
QTNet LiDAR - 4 frames 66.5 70.9 config model

Results on NuScenes Test Set.

Model Setting mAP NDS
QTNet LiDAR - 4 frames 68.4 72.2

TODO

  • Release the paper.
  • Release the code of QTNet (Memory Bank Training).

Acknowledgements

We thank these great works and open-source repositories: TransFusion, DeepInteraction, and MMDetection3d.

Citation

@inproceedings{hou2023querybased,
  title={Query-based Temporal Fusion with Explicit Motion for 3D Object Detection},
  author={Jinghua Hou and Zhe Liu and dingkang liang and Zhikang Zou and Xiaoqing Ye and Xiang Bai},
  booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
  year={2023},
}