readme.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hand Pose Estimation with Keras\n",
    "\n",
    "## Zane Geiger"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Introduction\n",
    "\n",
    "Hand pose estimation is a difficult problem with many potential applications in virtual reality and augmented reality. Depth information makes it a more tractable task, but there are still many challenges involved due to the high dimensionality of hand pose information, self-similarity, and self-occlusion. Recent papers have applied deep convolutional neural networks to this problem, with high levels of success. Enforcing a prior model on the hand pose further improves predictions, as well as refining joint locations using independent networks. Other have approached the problem by applying random decision forests.\n",
    "\n",
    "Augmented and virtual reality hardware is becoming increasingly dependent on mobile architectures. Smartphones have become a cost effective way to bring virtual reality to a wider audience, using head-mounted cases such as Samsung's gear VR and Google's Daydream. Newer augmented reality devices like Microsoft's HoloLens and other untethered headsets under development also rely on smartphone hardware. Hand gestures are a natural and intuitive method of interaction in virtual and augmented reality scenarios.  Bringing gesture support to these systems without the need for bulky gloves or visual markers has the potential to expand their use cases, making them easier to use and more flexible. Gestures in augmented and virtual reality can be used to perform many actions, such as direct interaction with the virtual environment and objects, without requiring users to learn a unique set of controls for each application.\n",
    "\n",
    "The human hand possesses several properties which make pose estimation challenging. The first is the large number of degrees of freedom; models of the hand can have from [25 to 50 degrees of freedom](data#input-images), though these models do not include joint angle constraints or the relationships between adjacent finger angles. The second is self-similarity; different fingers are difficult to distinguish, which can lead to incorrect inverse kinematics solutions. The last is self-occlusion; fingers may pass in front of one another, and the palm is concave, resulting in many parts of the hand becoming occluded and thus impossible to segment. \n",
    "\n",
    "On top of these inherent properties of the hand, the typical use cases for hand pose estimation introduce additional difficulties. In augmented and virtual reality applications, environmental conditions such as lighting and background color can vary wildly, compounding the difficulty of detecting hand features. Such applications may also require the tracking of hands at relatively high speeds in order to be useful, while most mobile cameras are only capable of 30 to 60 frames per second, even without taking processing speed into account. Finally, such mobile applications will be severely resource-constrained when compared to desktop or server hardware, limiting maximum frame rates and imposing a limit on latency.\n",
    "\n",
    "Modern deep learning toolkits such as TensorFlow and Keras enable rapid prototyping of network architectures and hyperparameters. Importantly, TensorFlow makes it relatively easy to run models on low-powered hardware, providing libraries for Android and ARM devices and accelerating computations by offloading them to the mobile GPU. With this in mind, we have taken an existing approach to hand pose estimation, first introduced by Oberweger, _et al._, and created an implementation in Keras using TensorFlow as a backend.\n",
    "\n",
    "### Existing work\n",
    "\n",
    "Several previous papers have looked at recovering hand poses using convolutional and deep networks, and these are what I will be basing the bulk of my research on. Tompson, _et al._ were the first to use convolutional networks for this particular task, and were able to improve on existing techniques. Previous works relied on methods such as randomized decision forests (Shotton _et al._ 2011, Keskin _et al._ 2011/2012), energy minimization for model fitting (Li _et al._ 2013, Ballan _et al._ 2012), and constraint solving (Melax _et al._ 2013). More recent approaches made use of particle swarm optimization (Oikonomidis _et al._ 2011, Qian _et al._ 2014) and regression forests (Tang _et al._ 2014).\n",
    "\n",
    "There are several hand pose datasets in existence, which typically provide a set of depth images, corresponding RGB images, and ground truth hand pose labels. The NYU dataset is one such example, consisting of 3 different viewpoints captured by Kinect sensors and ground truth poses approximated using manual labeling combined with a particle swarm based estimation algorithm. The ICVL dataset is similar, captured using a time of flight camera. The former has more pose variability but less clean depth information, while the latter has more imagery but less accurate labels. My research will depend primarily on the NYU dataset, as I will primarily be attempting to reproduce the results of Oberweger, _et al._\n",
    "\n",
    "### Replication\n",
    "\n",
    "This project will attempt to reproduce the results of Oberweger, _et al._ in [Hands Deep in Deep Learning for Hand Pose Estimation](https://arxiv.org/abs/1502.06807) using [Keras](https://keras.io) with the [TensorFlow](https://www.tensorflow.org/) backend using the [NYU hand pose dataset](http://cims.nyu.edu/~tompson/NYU_Hand_Pose_Dataset.htm). This initial implementation is based heavily on the work of [James Supancic](http://www.ics.uci.edu/~jsupanci/) in his [Deep Hand Pose](https://github.com/jsupancic/deep_hand_pose) project, which is implemented in the [Caffe](http://caffe.berkeleyvision.org/) framework, but is likely to diverge substantially as my research continues.\n",
    "\n",
    "The novel approach used in Hands Deep in Deep Learning is the enforcement of a hand pose prior, which learns and incorporates constraints on the predicted hand pose, allowing the model to eliminate unlikely poses. This is implemented by reducing the dimensionality of the predicted joint locations, and then reprojecting these predictions into the original coordinate space using principal component analysis. More specifically, the penultimate layer of their neural network contains only 22 neurons, while the final layer contains 28 (2 per joint location). This initial set of predictions is used to generate a heatmap of estimated joint positions, each of which is then refined by a separately trained neural network, incorporating information from several scales for context and precision. Thus far, our network implements the first stage convolutional network, including the hand pose dimensionality reduction step.\n",
    "\n",
    "Our reimplementation consists of two major components:\n",
    "\n",
    "* [A data processing pipeline](data)\n",
    "* [A deep learning model](model)\n",
    "\n",
    "Each is a thoroughly documented IPython notebook, intended to be read as an integral part of this thesis, as they contain the technical information relevant to this report. The data processing script takes the NYU Hand Pose Dataset as input and produces a single [.hdf5](https://support.hdfgroup.org/HDF5/) archive, containing images and labels suitable for input to the deep learning model, performing data conversion and normalization during processing. The deep learning model builds a deep convolutional neural network, training and evaluating it on the processed dataset.\n",
    "\n",
    "### Mobile\n",
    "\n",
    "### Future work"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}