Skip to content

Code for paper "Cross-Domain Self-supervised Multi-task Feature Learning using Synthetic Imagery", Ren et al., CVPR'18

Notifications You must be signed in to change notification settings

jason718/game-feature-learning

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

30 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

game-feature-learning

[Project] [Paper]

If you feel this useful, please consider cite:

@inproceedings{ren-cvpr2018,
  title = {Cross-Domain Self-supervised Multi-task Feature Learning using Synthetic Imagery},
  author = {Ren, Zhongzheng and Lee, Yong Jae},
  booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2018}
}

Feel free to contact Jason Ren (zr5@illinois.edu) if you have any questions!

Prerequisites

  • Pytorch-0.4 (some evaluation code borrowed from other places requiring Caffe)
  • Python2 (One evaluation code requiring Python3)
  • NVIDIA GPU + CUDA CuDNN (Sorry no CPU version)

Getting Started

Installation

  • Install PyTorch 0.4 and torchvision from http://pytorch.org
  • Python packages
    • torchvision
    • numpy
    • opencv
    • tensorflow (necesary for the use of tensorboard. Will change it to tensorboardX)
    • scikit-image
  • Clone this repo:
git clone https://github.com/jason718/game-feature-learning
cd game-feature-learning
  • Change the config files under configs.

Pre-trained models:

  • Caffemodel(Caffenet): Coming in second update
  • Pytorch model: Coming in second update

Since I greatly changed the code structure, I am retraining using the new code to reproduce the paper results.

Dataset:

  • SUNCG: Download the SUNCG images from suncg website. And make sure to put the files as the following structure:

        suncg_image
        ├── depth
           ├── room_id1
           ├── ...
        ├── normal
           ├── room_id1
           ├── ...
        ├── edge
           ├── room_id1
           ├── ...
        └── lab
            ├── room_id1
            ├── ...
    
  • SceneNet: Download the SceneNet images from scenenet website. And make sure to put the files as the following structure:

        scenenet_image
        └── train
           ├── 1
           ├── 2
           ├── ...
    

    Please check scripts/surface_normals_code to generate surface normals from depth maps.

  • Dataset For Domain Adaptation:

    • Places-365: Download the Places images from places website.
    • Or you can choose other dataset for DA such ImageNet...

Train/Test

  • Train a model:
sh ./scripts/train.sh
  • Evaluate on feature learning

    • Read each README.md under folder "eval-3rd-party"
  • Evaluate on three tasks

    • Coming in Third Update

Useful Resources

There are lots of awesome papers studying self-supervision for various tasks such as Image/Video Representation learning, Reinforcement learning, and Robotics. I am maintaining a paper list [awesome-self-supervised-learning] on Github. You are more than welcome to contribute and share :)

Supervised Learning is awesome but limited. Un-/Self-supervised learning generalizes better and sometimes also works better (which is already true in some geometry tasks)!

Acknowledgement

This work was supported in part by the National Science Foundation under Grant No. 1748387, the AWS Cloud Credits for Research Program, and GPUs donated by NVIDIA. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

About

Code for paper "Cross-Domain Self-supervised Multi-task Feature Learning using Synthetic Imagery", Ren et al., CVPR'18

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published