t81_558_class8_cnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Class 8: Convolutional Neural Networks.**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), School of Engineering and Applied Science, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "This class will focus on computer vision.  There are some important differences and similarities with previous neural networks.\n",
    "\n",
    "* We will usually use classification, though regression is still an option.\n",
    "* The input to the neural network is now 3D (height, width, color)\n",
    "* Data are not transformed, no zscores or dummy variables.\n",
    "* Processing time is much longer.\n",
    "* We now have different layer times: dense layers (just like before), convolution layers and max pooling layers.\n",
    "* Data will no longer arrive as CSV files. TensorFlow provides some utilities for going directly from image to the input fo a neural network.\n",
    "\n",
    "\n",
    "# Computer Vision Data Sets\n",
    "\n",
    "There are many data sets for computer vision.  Two of the most popular are the MNIST digits data set and the CIFAR image data sets.\n",
    "\n",
    "## MNIST Digits Data Set\n",
    "\n",
    "The [MNIST Digits Data Set](http://yann.lecun.com/exdb/mnist/) is very popular in the neural network research community.  A sample of it can be seen here:\n",
    "\n",
    "![MNIST Data Set](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_mnist.png \"MNIST Data Set\")\n",
    "\n",
    "This data set was generated from scanned forms.\n",
    "\n",
    "![Exam](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_exam.png \"Exam\")\n",
    "\n",
    "\n",
    "## CIFAR Data Set\n",
    "\n",
    "The [CIFAR-10 and CIFAR-100](https://www.cs.toronto.edu/~kriz/cifar.html) datasets are also frequently used by the neural network research community.\n",
    "\n",
    "![cifar-10](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_cifar.png \"cifar-10\")\n",
    "\n",
    "The CIFAR-10 data set contains low-rez images that are divided into 10 classes.  The CIFAR-100 data set contains 100 classes in a hierarchy. \n",
    "\n",
    "# Other Resources\n",
    "\n",
    "* [Imagenet:Large Scale Visual Recognition Challenge 2014](http://image-net.org/challenges/LSVRC/2014/index)\n",
    "* [Andrej Karpathy](http://cs.stanford.edu/people/karpathy/) - PhD student/instructor at Stanford.\n",
    "    * [CS231n Convolutional Neural Networks for Visual Recognition](http://cs231n.stanford.edu/) - Stanford course on computer vision/CNN's.\n",
    "        * [CS231n - GitHub](http://cs231n.github.io/)\n",
    "    * [ConvNetJS](http://cs.stanford.edu/people/karpathy/convnetjs/) - Javascript library for deep learning.\n",
    " \n",
    "\n",
    "# Convolutional Neural Networks (CNNs)\n",
    "\n",
    "The convolutional neural network (CNN) is a neural network technology that has profoundly impacted the area of computer vision (CV). Fukushima (1980) introduced the original concept of a convolutional neural network, and LeCun, Bottou, Bengio & Haffner (1998) greatly improved this work. From this research, Yan LeCun introduced the famous LeNet-5 neural network architecture. This class follows the LeNet-5 style of convolutional neural network.\n",
    "\n",
    "**A LeNET-5 Network (LeCun, 1998)**\n",
    "![LENET5](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_lenet5.png \"LENET5\")\n",
    "\n",
    "So far we have only seen one layer type (dense layers).  By the end of this course we will have seen:\n",
    "\n",
    "* **Dense Layers** - Fully connected layers.  (introduced previously)\n",
    "* **Convolution Layers** - Used to scan across images. (introduced this class)\n",
    "* **Max Pooling Layers** - Used to downsample images. (introduced this class)\n",
    "* **Dropout Layer** - Used to add regularization. (introduced next class)\n",
    "\n",
    "## Convolution Layers\n",
    "\n",
    "The first layer that we will examine is the convolutional layer. We will begin by looking at the hyper-parameters that you must specify for a convolutional layer in most neural network frameworks that support the CNN:\n",
    "\n",
    "* Number of filters\n",
    "* Filter Size\n",
    "* Stride\n",
    "* Padding\n",
    "* Activation Function/Non-Linearity\n",
    "\n",
    "The primary purpose for a convolutional layer is to detect features such as edges, lines, blobs of color, and other visual elements. The filters can detect these features. The more filters that we give to a convolutional layer, the more features it can detect.\n",
    "\n",
    "A filter is a square-shaped object that scans over the image. A grid can represent the individual pixels of a grid. You can think of the convolutional layer as a smaller grid that sweeps left to right over each row of the image. There is also a hyper parameter that specifies both the width and height of the square-shaped filter. Figure 10.1 shows this configuration in which you see the six convolutional filters sweeping over the image grid:\n",
    "\n",
    "A convolutional layer has weights between it and the previous layer or image grid. Each pixel on each convolutional layer is a weight. Therefore, the number of weights between a convolutional layer and its predecessor layer or image field is the following:\n",
    "\n",
    "```\n",
    "[FilterSize] * [FilterSize] * [# of Filters]\n",
    "```\n",
    "\n",
    "For example, if the filter size were 5 (5x4) for 10 filters, there would be 250 weights.\n",
    "\n",
    "You need to understand how the convolutional filters sweep across the previous layer’s output or image grid. Figure 10.2 illustrates the sweep:\n",
    "\n",
    "![CNN](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_cnn_grid.png \"CNN\")\n",
    "\n",
    "The above figure shows a convolutional filter with a size of 4 and a padding size of 1. The padding size is responsible for the boarder of zeros in the area that the filter sweeps. Even though the image is actually 8x7, the extra padding provides a virtual image size of 9x8 for the filter to sweep across. The stride specifies the number of positions at which the convolutional filters will stop. The convolutional filters move to the right, advancing by the number of cells specified in the stride. Once the far right is reached, the convolutional filter moves back to the far left, then it moves down by the stride amount and\n",
    "continues to the right again.\n",
    "\n",
    "Some constraints exist in relation to the size of the stride. Obviously, the stride cannot be 0. The convolutional filter would never move if the stride were set to 0. Furthermore, neither the stride, nor the convolutional filter size can be larger than the previous grid. There are additional constraints on the stride (s), padding (p) and the filter width (f) for an image of width (w). Specifically, the convolutional filter must be able to start at the far left or top boarder, move a certain number of strides, and land on the far right or bottom boarder. The following equation shows the number of steps a convolutional operator\n",
    "must take to cross the image:\n",
    "\n",
    "$ steps = \\frac{w - f + 2p}{s+1} $\n",
    "\n",
    "The number of steps must be an integer. In other words, it cannot have decimal places. The purpose of the padding (p) is to be adjusted to make this equation become an integer value.\n",
    "\n",
    "## Max Pooling Layers\n",
    "\n",
    "Max-pool layers downsample a 3D box to a new one with smaller dimensions. Typically, you can always place a max-pool layer immediately following convolutional layer. The LENET shows the max-pool layer immediately after layers C1 and C3. These max-pool layers progressively decrease the size of the dimensions of the 3D boxes passing through them. This technique can avoid overfitting (Krizhevsky, Sutskever & Hinton, 2012).\n",
    "\n",
    "A pooling layer has the following hyper-parameters:\n",
    "\n",
    "* Spatial Extent (f )\n",
    "* Stride (s)\n",
    "\n",
    "Unlike convolutional layers, max-pool layers do not use padding. Additionally, max-pool layers have no weights, so training does not affect them. These layers simply downsample their 3D box input. The 3D box output by a max-pool layer will have a width equal to this equation:\n",
    "\n",
    "$ w_2 = \\frac{w_1 - f}{s + 1} $\n",
    "\n",
    "The height of the 3D box produced by the max-pool layer is calculated similarly with this equation:\n",
    "\n",
    "$ h_2 = \\frac{h_1 - f}{s + 1} $\n",
    "\n",
    "The depth of the 3D box produced by the max-pool layer is equal to the depth the 3D box received as input. The most common setting for the hyper-parameters of a max-pool layer are f =2 and s=2. The spatial extent (f) specifies that boxes of 2x2 will be scaled down to single pixels. Of these four pixels, the pixel with the maximum value will represent the 2x2 pixel in the new grid. Because squares of size 4 are replaced with size 1, 75% of the pixel information is lost. The following figure shows this transformation as a 6x6 grid becomes a 3x3:\n",
    "\n",
    "![MaxPool](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_conv_maxpool.png \"MaxPool\")\n",
    "\n",
    "Of course, the above diagram shows each pixel as a single number. A grayscale image would have this characteristic. For an RGB image, we usually take the average of the three numbers to determine which pixel has the maximum value.\n",
    "\n",
    "[More information on CNN's](http://cs231n.github.io/convolutional-networks/)\n",
    "\n",
    "# TensorFlow with CNNs\n",
    "\n",
    "The following sections describe how to use TensorFlow/SKFlow with CNNs.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Access to Data Sets\n",
    "\n",
    "TensorFlow provides built in access classes for MNIST.  It is important to note that MNIST data arrives already separated into three sets:\n",
    "\n",
    "* **train** - Neural network will be trained with this.\n",
    "* **test** - Used for early stopping.\n",
    "* **validation** - Used to evaluate the final network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"Functions for downloading and reading MNIST data.\"\"\"\n",
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import gzip\n",
    "import os\n",
    "import tempfile\n",
    "\n",
    "import numpy\n",
    "from six.moves import urllib\n",
    "from six.moves import xrange  # pylint: disable=redefined-builtin\n",
    "import tensorflow as tf\n",
    "from tensorflow.contrib.learn.python.learn.datasets.mnist import read_data_sets\n",
    "\n",
    "# Loading MNIST data\n",
    "mnist = read_data_sets('MNIST_data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Define CNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "import tensorflow.contrib.learn as skflow\n",
    "from sklearn import datasets, metrics\n",
    "\n",
    "def max_pool_2x2(tensor_in):\n",
    "    return tf.nn.max_pool(tensor_in, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1],\n",
    "        padding='SAME')\n",
    "\n",
    "def conv_model(X, y):\n",
    "    # reshape X to 4d tensor with 2nd as image width, 3rd dimension \n",
    "    # as image height, 4th dimension as color channels.\n",
    "    X = tf.reshape(X, [-1, 28, 28, 1])\n",
    "    # Conv Layer #1: 32 channels/neurons for each 5x5 patch\n",
    "    with tf.variable_scope('conv_layer1'):\n",
    "        h_conv1 = skflow.ops.conv2d(X, n_filters=32, filter_shape=[5, 5], \n",
    "                                    bias=True, activation=tf.nn.relu)\n",
    "        h_pool1 = max_pool_2x2(h_conv1)\n",
    "    # second conv layer will compute 64 channels for each 5x5 patch\n",
    "    with tf.variable_scope('conv_layer2'):\n",
    "        h_conv2 = skflow.ops.conv2d(h_pool1, n_filters=64, filter_shape=[5, 5], \n",
    "                                    bias=True, activation=tf.nn.relu)\n",
    "        h_pool2 = max_pool_2x2(h_conv2)\n",
    "        # Reshape tensor into a batch of vectors\n",
    "        h_pool2_flat = tf.reshape(h_pool2, [-1, 7 * 7 * 64])\n",
    "    # densely connected layer with 256 neurons\n",
    "    h_fc1 = skflow.ops.dnn(h_pool2_flat, [256], activation=tf.nn.relu)\n",
    "    return skflow.models.logistic_regression(h_fc1, y)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training/Fitting CNN\n",
    "\n",
    "Two methods are provided, run only one, both are not needed.  Once complete, go to the next section to test accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step #99, avg. train loss: 2.25026\n",
      "Step #199, avg. train loss: 2.01235\n",
      "Step #299, avg. train loss: 1.71387\n",
      "Step #399, avg. train loss: 1.35640\n",
      "Step #499, avg. train loss: 1.02610\n",
      "Step #600, epoch #1, avg. train loss: 0.80594\n",
      "Step #700, epoch #1, avg. train loss: 0.66509\n",
      "Step #800, epoch #1, avg. train loss: 0.58332\n",
      "Step #900, epoch #1, avg. train loss: 0.51632\n",
      "Step #1000, epoch #1, avg. train loss: 0.48027\n",
      "Step #1100, epoch #2, avg. train loss: 0.44487\n",
      "Step #1200, epoch #2, avg. train loss: 0.42161\n",
      "Step #1300, epoch #2, avg. train loss: 0.39758\n",
      "Step #1400, epoch #2, avg. train loss: 0.36532\n",
      "Step #1500, epoch #2, avg. train loss: 0.35838\n",
      "Step #1600, epoch #2, avg. train loss: 0.34282\n",
      "Step #1700, epoch #3, avg. train loss: 0.34383\n",
      "Step #1800, epoch #3, avg. train loss: 0.32769\n",
      "Step #1900, epoch #3, avg. train loss: 0.30998\n",
      "Step #2000, epoch #3, avg. train loss: 0.31082\n",
      "Step #2100, epoch #3, avg. train loss: 0.29584\n",
      "Step #2200, epoch #4, avg. train loss: 0.29659\n",
      "Step #2300, epoch #4, avg. train loss: 0.28404\n",
      "Step #2400, epoch #4, avg. train loss: 0.26940\n",
      "Step #2500, epoch #4, avg. train loss: 0.28359\n",
      "Step #2600, epoch #4, avg. train loss: 0.27643\n",
      "Step #2700, epoch #4, avg. train loss: 0.26371\n",
      "Step #2800, epoch #5, avg. train loss: 0.26581\n",
      "Step #2900, epoch #5, avg. train loss: 0.25664\n",
      "Step #3000, epoch #5, avg. train loss: 0.25396\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TensorFlowEstimator(batch_size=100, class_weight=None, clip_gradients=5.0,\n",
       "          config=None, continue_training=False, learning_rate=0.001,\n",
       "          model_fn=<function conv_model at 0x7f472f41d1e0>, n_classes=10,\n",
       "          optimizer='Adagrad', steps=3000, verbose=1)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To fit/train use either the simple train (this box) or the early stop (next box)\n",
    "# Do not use both.  This box is faster, but no early stop\n",
    "classifier = skflow.TensorFlowEstimator(\n",
    "    model_fn=conv_model, n_classes=10, batch_size=100, steps=3000,\n",
    "    learning_rate=0.001)\n",
    "\n",
    "classifier.fit(mnist.train.images, mnist.train.labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Early stopping - WARNING, this is slow on Data Scientist Workbench\n",
    "# Do not run both this (and previous box)  Choose one or the other.\n",
    "\n",
    "# Training and predicting\n",
    "classifier = skflow.TensorFlowEstimator(\n",
    "    model_fn=conv_model, n_classes=10, batch_size=100, steps=500,\n",
    "    learning_rate=0.001)\n",
    "\n",
    "\n",
    "early_stop = skflow.monitors.ValidationMonitor(mnist.validation.images, \n",
    "    mnist.validation.labels, n_classes=10,\n",
    "    early_stopping_rounds=200, print_steps=50)\n",
    "\n",
    "# Fit/train neural network\n",
    "classifier.fit(mnist.train.images, mnist.train.labels, monitor=early_stop)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy score: 0.9355\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "# Evaluate success using accuracy\n",
    "pred = classifier.predict(mnist.test.images)\n",
    "score = metrics.accuracy_score(pred, mnist.test.labels)\n",
    "print(\"Accuracy score: {}\".format(score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}