Synthetic2Realistic

This repository implements the training and testing of T2Net for "T2Net: Synthetic-to-Realistic Translation for Depth Estimation Tasks" by Chuanxia Zheng, Tat-Jen Cham and Jianfei Cai at NTU. A video is available on YouTube. The repository offers the implementation of the paper in Pytoch.

• Outdoor Translation

• Indoor Translation

• Extension (WS-GAN, unpaired Image-to-Image Translation, horse2zebra)

This repository can be used for training and testing of

• Unpaired image-to-image Translation
• Single depth Estimation

Getting Started

Installation

This code was tested with Pytoch 0.4.0, CUDA 8.0, Python 3.6 and Ubuntu 16.04

• Install Pytoch 0.4, torchvision, and other dependencies from http://pytorch.org
• Install python libraries visdom and dominate for visualization

pip install visdom dominate

• Clone this repo:

git clone https://github.com/lyndonzheng/Synthetic2Realistic
cd Synthetic2Realistic

Datasets

The indoor Synthetic Dataset renders from SUNCG and indoor Realistic Dataset comes from NYUv2. The outdooe Synthetic Dataset is vKITTI and outdoor Realistic dataset is KITTI

Training

Warning: The input sizes need to be muliples of 64. The feature GAN model needs to be change for different scale

• Train a model with multi-domain datasets:

python train.py --name Outdoor_nyu_wsupervised --model wsupervised
--img_source_file /dataset/Image2Depth31_KITTI/trainA_SYN.txt
--img_target_file /dataset/Image2Depth31_KITTI/trainA.txt
--lab_source_file /dataset/Image2Depth31_KITTI/trainB_SYN.txt
--lab_target_file /dataset/Image2Depth31_KITTI/trainB.txt
--shuffle --flip --rotation

• To view training results and loss plots, run python -m visdom.server and copy the URL http://localhost:8097.
• Training results will be saved under the checkpoints folder. The more training options can be found in options.

Testing

• Test the model

python test.py --name Outdoor_nyu_wsupervised --model test
--img_source_file /dataset/Image2Depth31_KITTI/testA_SYN80
--img_target_file /dataset/Image2Depth31_KITTI/testA

Estimation

• Depth Estimation, the code based on monodepth

python evaluation.py --split eigen --file_path ./datasplit/
--gt_path ''your path''/KITTI/raw_data_KITTI/
--predicted_depth_path ''your path''/result/KITTI/predicted_depth_vk
--garg_crop

Trained Models

The pretrained model for indoor scene weakly wsupervised.

The pretrained model for outdoor scene weakly wsupervised

Note: Since our orginal model in the paper trained on single-GPU, this pretrained model is for multi-GPU version.

Citation

If you use this code for your research, please cite our papers.

@inproceedings{zheng2018t2net,
  title={T2Net: Synthetic-to-Realistic Translation for Solving Single-Image Depth Estimation Tasks},
  author={Zheng, Chuanxia and Cham, Tat-Jen and Cai, Jianfei},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
  pages={767--783},
  year={2018}
}

Acknowledgments

Code is inspired by Pytorch-CycleGAN