This repo contains an implementation of the CVPR 2020 paper:
DualSDF: Semantic Shape Manipulation using a Two-Level Representation,
by Zekun Hao, Hadar Averbuch-Elor, Noah Snavely, Serge Belongie.
| Adjust the wheelbase of the car by changing the distance between the two selected spheres |
|
| Adjust the fuselage diameter of the airplane by changing the diameter of the selected sphere | |
| Adjust the height of the chair by moving the selected sphere located on one of the legs |
The WebGL demo is largely based on three.js examples.
If you find our code useful, please consider citing our paper:
@inproceedings{hao2020dualsdf,
title={DualSDF: Semantic Shape Manipulation using a Two-Level Representation},
author={Hao, Zekun and Averbuch-Elor, Hadar and Snavely, Noah and Belongie, Serge},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2020}
}
We provide pretrained models for the chair and airplane categories. Each model is trained on a subset of shapes from the corresponding ShapeNet category. The list of shapes can be found under datasets/splits, which are taken from DeepSDF.
# Host the chair demo on port 1234 (default port) using GPU 0
python3.6 demo.py ./config/dualsdf_chairs_demo.yaml --pretrained ./pretrained/dualsdf_chairs_demo/epoch_2799.pth
# Host the airplane demo on port 4321 using GPU 1
CUDA_VISIBLE_DEVICES=1 python3.6 demo.py ./config/dualsdf_airplanes_demo.yaml --pretrained ./pretrained/dualsdf_airplanes_demo/epoch_2799.pth --port 4321
# Visit localhost:1234 or localhost:4321 on a browser that supports WebGL.
# We have tested Firefox and Chrome.We provide pre-sampled SDF dataset for ShapeNet chairs and airplanes on Google Drive. They are in SquashFS format so that you can mount it directly in Linux without extraction.
To mount a SquashFS file with squashfuse:
squashfuse <path_to_sqfs_file> <path_to_mount_point>Alternatively, with mount:
mount <path_to_sqfs_file> <path_to_mount_point> -t squashfs -o loopAlternatively, extract a SquashFS file:
unsquashfs -f -d <destination_path> <path_to_sqfs_file>Our SDF dataset contains SDF samples from all the shapes in the ShapeNetCore v2 chair and airplane categories. Each shape is normalized to be within a unit sphere prior to sampling.
There are two directories for each shape category:
<category_id>_spherecontains sdf values sampled uniformly within the unit sphere. There are 250000 samples for each shape.<category_id>_surfacecontains sdf values sampled near the surface. There are at least 500000 samples for each shape. The first half of them are inside the shape (negative sign) while the second half of them are outside (positive sign).
SDF samples are stored as N-by-4 numpy arrays and saved as individual npy files <shape_id>.npy. N samples are given and each sample takes the form of (x, y, z, d).
Use the following instructions if you want to sample your own SDF dataset.
For ease of process, we convert meshes to numpy arrays before sampling SDFs.
Tools are widely available to load ShapeNet obj files. Each npy file should contain a float32 array with a shape of #triangles x 3 (vertices) x 3 (xyz coordinate of each vertex), representing a triangle mesh. The scale and center of the mesh is not important, as it will be normalized to a unit sphere right before sampling SDFs.
The naming convention is as follows:
<category_id>/
<shape_id>.npy
Other shape datasets (i.e. ABC Dataset, ModelNet, PartNet) can be used instead of ShapeNet, as long as they are also converted to npy format and the npy files are stored using the naming convention described above.
Compile the CUDA kernel for computing SDF:
cd extensions/mesh2sdf2_cuda
makeSample SDFs using the provided script:
python3.6 sample_sdfs.py <path_to_mesh_npy_folder> <path_to_save_sampled_results>This is what the result directory should look like:
<path_to_save_sampled_results>/
<category_id>_sphere/
<shape_id>.npy
<category_id>_surface/
<shape_id>.npy
It is a good idea to start with an existing config file (i.e. config/dualsdf_airplanes_demo.yaml). Edit the data section to reflex your dataset configuration. You will need to build and specify your own split_files for new datasets. It is generally a good idea to exclude shapes with large non-manifold regions, as they cannot be represented well with SDFs.
data:
...
cate_id: "02691156" # Replace with your category id
split_files:
train: ./datasets/splits/sv2_planes_all.json # Replace with your training split file
test: ./datasets/splits/sv2_planes_all.json # Replace with your testing split file
sdf_data_dir:
surface: /mnt/data3/ShapeNet_sdf_even/02691156_surface # Replace with your path
sphere: /mnt/data3/ShapeNet_sdf_even/02691156_sphere # Replace with your path
...
...
python3.6 train.py ./config/dualsdf_airplanes_demo.yamlTensorboard databases and checkpoints will appear under logs directory.
To render shape reconstruction results on the training set, run the following command with properly set paths to config file and checkpoint file:
python3.6 train.py ./config/dualsdf_airplanes_demo.yaml --resume --pretrained ./pretrained/dualsdf_airplanes_demo/epoch_2799.pth --special render_known_shapesThe rendered images of both primitive-based representation and high-resolution representation will appear under the logs/special_render_known_shapes_dualsdf_airplanes_demo_<datetime> directory.
A comprehensive set of options related to rendering, such as resolution, color scheme, etc. can be modified in the config file.
The code was developed under Python 3.6 and Pytorch 1.4. Other dependencies are:
opencv-python(Only for running the demo)tensorboardX(Only for training)
MIT License