English | 简体中文

Lingdong Kong¹  Shaoyuan Xie²  Hanjiang Hu³  Benoit Cottereau⁴  Lai Xing Ng⁵  Wei Tsang Ooi^1 
1National University of Singapore   ²Huazhong Univerisity of Science and Technology
³Carnegie Mellon University   ⁴CNRS   ⁵A*STAR

RoboDepth is a comprehensive evaluation benchmark designed for probing the robustness of monocular depth estimation algorithms. It includes 18 common corruption types, ranging from weather and lighting conditions, sensor failure and movement, and noises during data processing.

• [2023.01] - The NYUDepth2-C dataset is ready to be downloaded! See here for more details.
• [2023.01] - Evaluation server for Track 2 (fully-supervised depth estimation) is available on this page.
• [2023.01] - Evaluation server for Track 1 (self-supervised depth estimation) is available on this page.
• [2022.11] - We are organizing the 1st RoboDepth Competition at ICRA 2023. Join the challenge today! 🙋
• [2022.11] - The KITTI-C dataset is ready to be downloaded! See here for more details.

• Installation
• Data Preparation
• Getting Started
• Model Zoo
• Benchmark
• Idiosyncrasy Analysis
• Create Corruption Sets
• TODO List
• Citation
• License
• Acknowledgements

Kindly refer to INSTALL.md for the installation details.

Kindly refer to DATA_PREPARE.md for the details to prepare the ¹KITTI, ²KITTI-C, ³Cityscapes, ⁴NYUDepth2, and ⁵NYUDepth2-C datasets.

Kindly refer to this page for the details to prepare the training and evaluation data associated with the 1st RoboDepth Competition at the 40th IEEE Conference on Robotics and Automation (ICRA 2023).

Kindly refer to GET_STARTED.md to learn more usage about this codebase.

The following metrics are consistently used in our benchmark:

• Absolute Relative Difference (the lower the better): $\text{Abs Rel} = \frac{1}{|D|}\sum_{pred\in D}\frac{|gt - pred|}{gt}$ .
• Accuracy (the higher the better): $\delta_t = \frac{1}{|D|}|{\ pred\in D | \max{(\frac{gt}{pred}, \frac{pred}{gt})< 1.25^t}}| \times 100\%$ .
• Depth Estimation Error (the lower the better):
  • $\text{DEE}_1 = \text{Abs Rel} - \delta_1 + 1$ ;
  • $\text{DEE}_2 = \frac{\text{Abs Rel} - \delta_1 + 1}{2}$ ;
  • $\text{DEE}_3 = \frac{\text{Abs Rel}}{\delta_1}$ .

The second Depth Estimation Error term ($\text{DEE}_2$) is adopted as the main indicator for evaluating model performance in our RoboDepth benchmark. The following two metrics are adopted to compare between models' robustness:

• mCE (the lower the better): The average corruption error (in percentage) of a candidate model compared to the baseline model, which is calculated among all corruption types across five severity levels.
• mRR (the higher the better): The average resilience rate (in percentage) of a candidate model compared to its "clean" performance, which is calculated among all corruption types across five severity levels.

Symbol ^⭐ denotes the baseline model adopted in mCE calculation.

For more detailed benchmarking results and to access the pretrained weights used in robustness evaluation, kindly refer to RESULT.md.

You can manage to create your own "RoboDepth" corrpution sets! Follow the instructions listed in CREATE.md.

• Initial release. 🚀
• Add scripts for creating common corruptions.
• Add download link of KITTI-C and NYUDepth2-C.
• Add competition data.
• Add benchmarking results.
• Add evaluation scripts on corruption sets.

If you find this work helpful, please kindly consider citing our paper:

@ARTICLE{kong2023robodepth,
  title={RoboDepth: Robust Out-of-Distribution Depth Estimation under Corruptions},
  author={Kong, Lingdong and Xie, Shaoyuan and Hu, Hanjiang and Cottereau, Benoit and Ng, Lai Xing and Ooi, Wei Tsang},
  journal={arXiv preprint arXiv:23xx.xxxxx}, 
  year={2023},
}

This work is under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

This project is supported by DesCartes, a CNRS@CREATE program on Intelligent Modeling for Decision-Making in Critical Urban Systems.