3D Shape Reconstruction from Free-Hand Sketches

[Project] [Paper]

Overview

This is authors' re-implementation of the paper described in:
"3D Shape Reconstruction from Free-Hand Sketches"
Jiayun Wang,  Jierui Lin,  Qian Yu  Runtao Liu,  Yubei Chen,  Stella X. Yu  (UC Berkeley/ICSI/BUAA)  in arXiv.

Requirements

• Tensorflow (version >= 1.12.0)

Overall architecture

data
  |--Data Folder #please update corresponding path in depthestimate//BatchFetcher.py
  |--depthestimate # 3D reconstruction module
    |--BatchFetcher.py
    |--train_nn_trans.py
    |--visualizeptexample.v.py
    |--...
  |--sketchstd # sketch standarization module

Training the network

Download the dataset

We use the ShapeNet dataset in our experiments, which are available below:

• ShapeNet rendering images: http://cvgl.stanford.edu/data2/ShapeNetRendering.tgz
• ShapeNet voxelized models: http://cvgl.stanford.edu/data2/ShapeNetVox32.tgz (optional, for comparison with voxelized output only)

Make sure you update the data path in the BatchFetcher.py.

Generate synthetic sketches from rendered images

We use a previous work Unsupervised Sketch to Photo Synthesis for generating synthetic sketches. Please refer to their code to generate data for training.

You can use CycleGAN as an alternative to generate synthetic sketches from rendered images. This may lead to a worse result.

Compiling CUDA code

You will need to first edit makefile to match your nvcc, cudalib, and tensorflow path.

Then compile CUDA code

$ make

Training the sketch standarization module

We use CycleGAN to train the sketch standarization module. Please refer to sketchstd.

Training the reconstruction network

Update path to STANDARIZED sketch and point cloud dataset in depthestimate//BatchFetcher.py first.

$ python depthestimate/train_nn_trans.py data=. dump=. train

Predict with trained network

$ python depthestimate/train_nn_trans.py data=. dump=. predict

Visualualize dumpped prediction (press space to view the next one)

$ python depthestimate/visualizeptexample.v.py <path>/train_nn.v.pkl
example: $ python depthestimate/visualizeptexample.v.py dump/train_nn.v.pkl

Note

• The repo is adapted from PointSetGeneration.
• The current code does not support multi-GPU settings.

Q & A

Please raise issues if you encounter any problem.

License and Citation

The use of this software is released under BSD-3.

@article{wang20203d,
  title={3D Shape Reconstruction from Free-Hand Sketches},
  author={Wang, Jiayun and Lin, Jierui and Yu, Qian and Liu, Runtao and Chen, Yubei and Yu, Stella X},
  journal={arXiv preprint arXiv:2006.09694},
  year={2020}
}