P2ORM: Formulation, Inference & Application

This repository contains the official PyTorch implementation of our paper: "Pixel-Pair Occlusion Relationship Map (P2ORM): Formulation, Inference & Application" (ECCV 2020 Spotlight).
It contains two parts: the estimation of occlusion boundaries with directions and the refinement on depth maps using the estimated occlusions. By using this repository, you can evaluate the pretrained models, train your own models and generate new occlusion annotations with the proposed method. For more details such as videos and introduction slides, please refer to our project webpage.

[Project webpage] [Paper] [Supp.]

News (2021-02-01): The code for occlusion label generation is released!

News (2020-08-28): The code for training and test is released!

If our project is helpful for your research, please consider citing :

@inproceedings{qiu2020pixel,
  title     = {Pixel-{P}air Occlusion Relationship Map ({P2ORM}): Formulation, Inference \& Application},
  author    = {Xuchong Qiu and Yang Xiao and Chaohui Wang and Renaud Marlet},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2020}
}

Table of contents

• 1. Visual Results
• 2. Installation
• 3. Evaluation
• 4. Train
• 5. Generate Occlusion Labels

1. Visual Results

1.1 Occlusion boundary and orientation estimation

Given an image as input, the model predicts occlusion status between neighbor pixels which can be converted to occlusion boundaries with orientations (the left-side of a green arrow is the occlusion foreground). In the predictions, green color indicates the correctly estimated boundaries.

1.2 Depth map refinement

Given an initial depth prediction and the estimated occlusions as input, the model predicts a refined depth map.

2. Installation

2.1 Dependencies

The code has been tested on ubuntu 16.04 with a single GeForce GTX TITAN X (12GB). The main dependencies are: Python=3.6, Pytorch>=1.1.0 and Torchvision>=0.3.0. Please install the PyTorch version based on your CUDA version. We recommend to utilize conda environment to install all dependencies and test the code.

# Download the repository
git clone 'https://github.com/tim885/P2ORM'
cd P2ORM

# Create python env with relevant dependencies
conda create --name P2ORM --file spec-file.txt
conda activate P2ORM
/your_conda_root/envs/P2ORM/bin/pip install tensorboardX Cython 

2.2 Data preparation

Download the relevant dataset and unzip it in folder ./data/.

2.2.1 Download datasets with occlusion labels for train/test.

Download BSDS300.zip here.

Download InteriorNet_OR.zip here.

Download iBims1_OR.zip here.

Download NYUv2_OR.zip here.

2.2.2 Generate new occlusion labels on a dataset (e.g., InteriorNet).

Download InteriorNet.zip here

N.B. The file is used only for a demonstration, please refer to here for the whole dataset.

3. Evaluation

3.1 Pixel-Pair Occlusion Detection

3.1.1 Pretrained models

Download pretrained model for BSDSownership here.

Download pretrained model for iBims1_OR here.

Download pretrained model for NYUv2_OR here.

3.1.2 Test and evaluation

Use the pretrained models and test them on three datasets. The models predict occlusion status between neighbor pixels for each image and save them in the corresponding files.

# create data dir symbole link for occlusion detection and exp dir 
cd DetectOcclusion/ && ln -s ../data/ data/
mkdir DetectOcclusion/output/  && cd detect_occ/ 

# test on BSDSownership dataset and save results in ./DetectOcclusion/experiments/output/BSDSownership_pretrained/
mkdir DetectOcclusion/output/BSDSownership_pretrained/  # and put pretrained model here
python train_val.py --config ../experiments/configs/BSDSownership_order_myUnet_CCE_1.yaml --evaluate --resume BSDSownership_pretrained/BSDSownership_epoch19.pth.tar --gpus 0

# test on iBims1_OR dataset and save results in ./DetectOcclusion/experiments/output/iBims1_OR_pretrained/
mkdir DetectOcclusion/output/iBims1OR_pretrained/  # and put pretrained model here
python train_val.py --config ../experiments/configs/ibims_order_myUnet_CCE_1.yaml --evaluate --resume iBims1OR_pretrained/iBims1OR_epoch179.pth.tar --gpus 0

# test on NYUv2_OR dataset and save results in ./DetectOcclusion/experiments/output/NYUv2_OR_pretrained/
mkdir DetectOcclusion/output/NYUv2OR_pretrained/  # and put pretrained model here
python train_val.py --config ../experiments/configs/nyuv2_order_myUnet_CCE_1.yaml --evaluate --resume NYUv2OR_pretrained/NYUv2OR_epoch74.pth.tar --gpus 0

# (Optional) we use cython to boost the speed of NMS. Run this if you can't run the test code. 
cd DetectOcclusion/lib/dataset
python setup_cython.py build_ext --inplace

# evaluate the predictions with matlab (default: BSDSownership)
# please refer to evaluation_matlab/README.md for more details
cd ../evaluation_matlab/evaluation/
matlab -nodisplay
run EvaluateOcc.m

The pairwise occlusion predictions (.npz files) for a dataset (e.g., iBims1OR) are stored in the following directory:

./DetectOcclusion/experiments/output/iBims1OR_pretrained/results_vis/test_179_ibims/res_mat/test_order_nms_pred/

N.B. Each .npz file contains the first channel for occlusion boundary probability (0~127) and next eight channels of occlusion relationship w.r.t each pixel's eight neighbor pixels with label (1: occludes, -1: occluded, 0: no occlusion). The mapping from channel index to pixel neighbor is:
1 2 3
4    5
6 7 8

# Please load the .npz file using:
occ = numpy.load('file_path')['order']

We also offer pairwise occlusion predictions directly to encourage using them in downstream tasks:

Download pairwise occlusion predictions for iBims-1 here.

Download pairwise occlusion predictions for NYUv2 here.

3.2 Depth Refinement with Detected Occlusion

Change to the directory: cd ./DepthRefine/.

Pretrained model is saved at pretrained/ckpt.pth.

To refine the depth map on iBims1_OR:

python refine_ibims.py --checkpoint #model_path --depth_pred_method #method_name

where the initial depth maps are predicted by #method_name, which should be chosen from: [ eigen, fcrn, fayao, junli, planenet, sharpnet ].

To refine and evaluate depth on NYUv2_OR:

python refine_nyu.py --checkpoint #model_path --result_dir #refined_depths
python eval_nyu.py #refined_depths

the refined depths would be saved in #refined_depths and evaluation results would be logged in file #refined_depths/eval.txt. The initial depth maps are predicted by: [ eigen, laina, dorn, sharpnet, jiao, vnl ].

4. Train

4.1 Pixel-Pair Occlusion Detection

# create data dir symbole link for occlusion detection and exp dir 
cd DetectOcclusion/ && ln -s ../data/ data/
mkdir DetectOcclusion/output/ && cd detect_occ/

# train for BSDSownership dataset
python train_val.py --config ../experiments/configs/BSDSownership_order_myUnet_CCE_1.yaml --gpus 1   

# train for iBims1_OR dataset
python train_val.py --config ../experiments/configs/ibims_order_myUnet_CCE_1.yaml --gpus 1

# train for NYUv2_OR dataset
python train_val.py --config ../experiments/configs/nyuv2_order_myUnet_CCE_1.yaml --gpus 1

4.2 Depth Refinement

Our model is trained on 10,160 images of InteriorNet_OR for 30 epochs:

cd ./DepthRefine/
python train.py --save_dir # save_model_path

The whole training procedure can be finished in ~10 hours with a single TitanX GPU.

5. Generate Occlusion Labels

You can generate the occlusion relationship label between immediate neighbor pixels using our method proposed in the paper. Please follow the instructions presented below:

# create data dir symbole link  
cd DetectOcclusion/ && ln -s ../data/ data/ && cd utils

# generate occlusion labels on InteriorNet (please refer to the script for more details)
python gen_InteriornetOR_label.py