Skip to content

Latest commit

 

History

History
122 lines (95 loc) · 4.2 KB

README.md

File metadata and controls

122 lines (95 loc) · 4.2 KB
Raw

Setup for deep learning baseline with real-world data

This component of Jax3DP3 runs end-to-end PoseCNN (segmentation) and DenseFusion (pose estimation) to detect and localize objects given a single RGB-D image. (Note: The model-training capability of this module has not been tested, and the scripts assume the existence of pretrained .pth models for PoseCNN and DenseFusion.)

Required environment

  • Ubuntu 16.04 or above
  • PyTorch 0.4.1 or above
  • CUDA 9.1 or above
  • Python3

Installation (Reference)

  1. Install PyTorch

  2. Install Eigen from the Github source code here

        # change eigen version
    apt remove libeigen3-dev
    cd /opt
    git clone https://gitlab.com/libeigen/eigen.git
    cd eigen
    mkdir build
    cd build
    cmake ..
    make
    make install
    # make a symbolic link
    cd /usr/local/include
    sudo ln -sf eigen3/Eigen Eigen
    sudo ln -sf eigen3/unsupported unsupported
    sudo ln -sf eigen3/Eigen /usr/include/Eigen
    sudo cp -r /usr/local/include/eigen3 /usr/include/eigen3

  3. Install Sophus from the Github source code here

        cd /opt
    git clone https://github.com/strasdat/Sophus
    cd Sophus
    git reset --hard ceb6380a1584b300e687feeeea8799353d48859f
    # TODO include patch in CMake  (do we still need?)
    #-find_package(Eigen3 REQUIRED)
    #+find_package(PkgConfig)
    #+pkg_search_module(Eigen3 REQUIRED eigen3)
    # add -Wno-error=deprecated-copy
    mkdir build
    cd build
    cmake ..
    make
    make install

  4. Install python packages

    pip install -r requirement.txt

  5. Initialize the submodules in ycb_render

    git submodule update --init --recursive

  6. Compile the new layers under $ROOT/lib/layers (for PoseCNN)

    cd $ROOT/lib/layers
python setup.py install

  7. Compile cython components (for PoseCNN)

    cd $ROOT/lib/utils
python setup.py build_ext --inplace

  8. Compile the ycb_render in $ROOT/ycb_render (for PoseCNN)

    cd $ROOT/ycb_render
python setup.py develop

Download (Also see: PoseCNN, DenseFusion)

  • 3D models of YCB Objects here (3G). Save as $ROOT/data/models or use a symbol link.

  • Pre-trained PoseCNN checkpoints here (4G). Save files as $ROOT/data/trained_checkpoints/posecnn or use a symbol link.

  • Pre-trained DenseFusion checkpoints here. Save trained_checkpoints.zip into $ROOT/trained_checkpoints/densefusion or use a symbol link.

PoseCNN Demo

  • First test the PoseCNN model with the provided demo.

run the following script Shell ./experiments/scripts/posecnn/demo.sh

  • If you get an error, try rebuilding libassimp 4.1.0 from source, following the instructions here
  • Results will be in $ROOT/datasets/posecnn/demo.

PoseCNN + DenseFusion Demo

  • Now test the full object detection and pose estimation workflow.

run the following python script Shell python tools/test_image_pandas_ycb_full.py # TODO parse args

  • Results will be in experiments/eval_result/pandas. # TODO

Testing on custom data containing YCB objects

  • This is to run the posecnn+densefusion test with your own data
  • For each hxw image frame you wish to test, create a .pik file containing the fields:
    • rgb: (h,w,3) rgb data
    • depth: (h,w) depth data
    • factor_depth: a (divisive, not multiplicative) scaling factor for the depth data
    • intrinsics : a 3x3 camera intrinsics matrix
  • Place the folder containing these .pik files as a subfolder within $ROOT/datasets/pandas/data.