2.2-Review-CNN-MNIST-Dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.bigdatauniversity.com\"><img src=\"https://ibm.box.com/shared/static/qo20b88v1hbjztubt06609ovs85q8fau.png\" width=\"400px\"  align=\"center\"></a>\n",
    "<h1 align=\"center\"><font size=\"5\">CONVOLUTIONAL NEURAL NETWORK APPLICATION</font></h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Introduction</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will use the famous [MNIST Dataset](http://yann.lecun.com/exdb/mnist/) to build two Neural Networks capable to perform handwritten digits classification. The first Network is a simple Multi-layer Perceptron (MLP) and the second one is a Convolutional Neural Network (CNN from now on). In other words, when given an input our algorithm will say, with some associated error, what type of digit this input represents."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
    "<font size = 3><strong>Click on the links to go to the following sections:</strong></font>\n",
    "<br>\n",
    "<h2>Table of Contents</h2>\n",
    "<ol>\n",
    "    <li><a href=\"#ref1\">What is Deep Learning</a></li>\n",
    "    <li><a href=\"#ref2\">Simple test: Is TensorFlow working?</a></li>\n",
    "    <li><a href=\"#ref3\">1st part: classify MNIST using a simple model</a></li>\n",
    "    <li><a href=\"#ref4\">Evaluating the final result</a></li>\n",
    "    <li><a href=\"#ref5\">How to improve our model?</a></li>\n",
    "    <li><a href=\"#ref6\">2nd part: Deep Learning applied on MNIST</a></li>\n",
    "    <li><a href=\"#ref7\">Summary of the Deep Convolutional Neural Network</a></li>\n",
    "    <li><a href=\"#ref8\">Define functions and train the model</a></li>\n",
    "    <li><a href=\"#ref9\">Evaluate the model</a></li>\n",
    "</ol>    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref1\"></a>\n",
    "<h2>What is Deep Learning?</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Brief Theory:</b> Deep learning (also known as deep structured learning, hierarchical learning or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using multiple processing layers, with complex structures or otherwise, composed of multiple non-linear transformations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://ibm.box.com/shared/static/gcbbrh440604cj2nksu3f44be87b8ank.png\" alt=\"HTML5 Icon\" style=\"width: 600px; height: 450px;\">\n",
    "<div style=\"text-align: center\">It's time for deep learning. Our brain doesn't work with only one or three layers. Why it would be different with machines?. </div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>In Practice, defining the term \"Deep\":</b> in this context, deep means that we are studying a Neural Network which has several hidden layers (more than one), no matter what type (convolutional, pooling, normalization, fully-connected etc). The most interesting part is that some papers noticed that Deep Neural Networks with the right architectures/hyper-parameters achieve better results than shallow Neural Networks with the same computational power (e.g. number of neurons or connections). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>In Practice, defining \"Learning\":</b> In the context of supervised learning, digits recognition in our case, the learning part consists of a target/feature which is to be predicted using a given set of observations with the already known final prediction (label). In our case, the target will be the digit (0, 1, 2, 3, 4, 5, 6, 7, 8, 9) and the observations are the intensity and relative position of the pixels. After some training, it is possible to generate a \"function\" that map inputs (digit image) to desired outputs(type of digit). The only problem is how well this map operation occurs. While trying to generate this \"function\", the training process continues until the model achieves a desired level of accuracy on the training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Notice:</b> This notebook has been created with TensorFlow version 1.8, and might not work with other versions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we first classify MNIST using a simple Multi-layer perceptron and then, in the second part, we use deeplearning to improve the accuracy of our results.\n",
    "\n",
    "<a id=\"ref3\"></a>\n",
    "<h2>1st part: classify MNIST using a simple model.</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to create a simple Multi-layer perceptron, a simple type of Neural Network, to perform classification tasks on the MNIST digits dataset. If you are not familiar with the MNIST dataset, please consider to read more about it: <a href=\"http://yann.lecun.com/exdb/mnist/\">click here</a> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>What is MNIST?</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to LeCun's website, the MNIST is a: \"database of handwritten digits that has a training set of 60,000 examples, and a test set of 10,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Import the MNIST dataset using TensorFlow built-in feature</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's very important to notice that MNIST is a high optimized data-set and it does not contain images. You will need to build your own code if you want to see the real digits. Another important side note is the effort that the authors invested on this data-set with normalization and centering operations.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The <span style=\"background-color:#dcdcdc\"> one-hot = True</span> argument only means that, in contrast to Binary representation, the labels will be presented in a way that to represent a number N, the $N^{th}$ bit is 1 while the the other bits are 0. For example, five and zero in a binary code would be:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre>\n",
    "Number representation:    0\n",
    "Binary encoding:        [2^5]  [2^4]   [2^3]   [2^2]   [2^1]   [2^0]  \n",
    "Array/vector:             0      0       0       0       0       0 \n",
    "\n",
    "Number representation:    5\n",
    "Binary encoding:        [2^5]  [2^4]   [2^3]   [2^2]   [2^1]   [2^0]  \n",
    "Array/vector:             0      0       0       1       0       1  \n",
    "</pre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using a different notation, the same digits using one-hot vector representation can be show as: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre>\n",
    "Number representation:    0\n",
    "One-hot encoding:        [5]   [4]    [3]    [2]    [1]   [0]  \n",
    "Array/vector:             0     0      0      0      0     1   \n",
    "\n",
    "Number representation:    5\n",
    "One-hot encoding:        [5]   [4]    [3]    [2]    [1]    [0]  \n",
    "Array/vector:             1     0      0      0      0      0   \n",
    "</pre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Understanding the imported data</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The imported data can be divided as follow:\n",
    "\n",
    "- Training (mnist.train) >>  Use the given dataset with inputs and related outputs for training of NN. In our case, if you give an image that you know that represents a \"nine\", this set will tell the neural network that we expect a \"nine\" as the output.  \n",
    "        - 55,000 data points\n",
    "        - mnist.train.images for inputs\n",
    "        - mnist.train.labels for outputs\n",
    "  \n",
    "   \n",
    "- Validation (mnist.validation) >> The same as training, but now the data is used to generate model properties (classification error, for example) and from this, tune parameters like the optimal number of hidden units or determine a stopping point for the back-propagation algorithm  \n",
    "        - 5,000 data points\n",
    "        - mnist.validation.images for inputs\n",
    "        - mnist.validation.labels for outputs\n",
    "  \n",
    "  \n",
    "- Test (mnist.test) >> the model does not have access to this informations prior to the testing phase. It is used to evaluate the performance and accuracy of the model against \"real life situations\". No further optimization beyond this point.  \n",
    "        - 10,000 data points\n",
    "        - mnist.test.images for inputs\n",
    "        - mnist.test.labels for outputs\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Creating an interactive section</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have two basic options when using TensorFlow to run your code:\n",
    "\n",
    "- [Build graphs and run session] Do all the set-up and THEN execute a session to evaluate tensors and run operations (ops) \n",
    "- [Interactive session] create your coding and run on the fly. \n",
    "\n",
    "For this first part, we will use the interactive session that is more suitable for environments like Jupyter notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "sess = tf.InteractiveSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating placeholders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It iss a best practice to create placeholders before variable assignments when using TensorFlow. Here we'll create placeholders for inputs (\"Xs\") and outputs (\"Ys\").   \n",
    "\n",
    "<b>Placeholder 'X':</b> represents the \"space\" allocated input or the images. \n",
    "<ul>\n",
    "    <li>Each input has 784 pixels distributed by a 28 width x 28 height matrix</li>   \n",
    "    <li>The 'shape' argument defines the tensor size by its dimensions.</li>     \n",
    "    <li>1st dimension = None. Indicates that the batch size, can be of any size.</li>     \n",
    "    <li>2nd dimension = 784. Indicates the number of pixels on a single flattened MNIST image.</li>    \n",
    "</ul>\n",
    "    \n",
    "<b>Placeholder 'Y':</b> represents the final output or the labels.\n",
    "<ul>\n",
    "    <li>10 possible classes (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)</li>  \n",
    "    <li>The 'shape' argument defines the tensor size by its dimensions.</li>    \n",
    "    <li>1st dimension = None. Indicates that the batch size, can be of any size.</li>     \n",
    "    <li>2nd dimension = 10. Indicates the number of targets/outcomes</li>   \n",
    "</ul>\n",
    "<b>dtype for both placeholders:</b> if you not sure, use tf.float32. The limitation here is that the later presented softmax function only accepts float32 or float64 dtypes. For more dtypes, check TensorFlow's documentation <a href=\"https://www.tensorflow.org/api_docs/python/tf/DType\">here</a>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "x  = tf.placeholder(tf.float32, shape=[None, 784])\n",
    "y_ = tf.placeholder(tf.float32, shape=[None, 10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Assigning bias and weights to null tensors</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are going to create the weights and biases, for this purpose they will be used as arrays filled with zeros. The values that we choose here can be critical, but we'll cover a better way on the second part, instead of this type of initialization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "    # Weight tensor\n",
    "    W = tf.Variable(tf.zeros([784, 10],tf.float32))\n",
    "    # Bias tensor\n",
    "    b = tf.Variable(tf.zeros([10],tf.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Execute the assignment operation</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before, we assigned the weights and biases but we did not initialize them with null values. For this reason, TensorFlow need to initialize the variables that you assign.  \n",
    "Please notice that we're using this notation \"sess.run\" because we previously started an interactive session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# run the op initialize_all_variables using an interactive session\n",
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Adding Weights and Biases to input</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only difference for our next operation to the picture below is that we are using the mathematical convention for what is being executed in the illustration. The tf.matmul operation performs a matrix multiplication between x (inputs) and W (weights) and after the code add biases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img src=\"https://ibm.box.com/shared/static/88ksiymk1xkb10rgk0jwr3jw814jbfxo.png\" alt=\"HTML5 Icon\" style=\"width:400px;height:350px;\"> \n",
    "<div style=\"text-align:center\">Illustration showing how weights and biases are added to neurons/nodes. </div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# mathematical operation to add weights and biases to the inputs\n",
    "tf.matmul(x,W) + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Softmax Regression</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Softmax is an activation function that is normally used in classification problems. It generate the probabilities for the output. For example, our model will not be 100% sure that one digit is the number nine, instead, the answer will be a distribution of probabilities where, if the model is right, the nine number will have a larger probability than the other other digits.\n",
    "\n",
    "For comparison, below is the one-hot vector for a nine digit label:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "0 --> 0  \n",
    "1 --> 0 \n",
    "2 --> 0\n",
    "3 --> 0\n",
    "4 --> 0\n",
    "5 --> 0\n",
    "6 --> 0\n",
    "7 --> 0\n",
    "8 --> 0\n",
    "9 --> 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A machine does not have all this certainty, so we want to know what is the best guess, but we also want to understand how sure it was and what was the second better option. Below is an example of a hypothetical distribution for a nine digit:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "0 -->0.01  \n",
    "1 -->0.02  \n",
    "2 -->0.03  \n",
    "3 -->0.02  \n",
    "4 -->0.12  \n",
    "5 -->0.01  \n",
    "6 -->0.03\n",
    "7 -->0.06\n",
    "8 -->0.1\n",
    "9 -->0.6  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "y = tf.nn.softmax(tf.matmul(x,W) + b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Logistic function output is used for the classification between two target classes 0/1. Softmax function is generalized type of logistic function. That is, Softmax can output a multiclass categorical probability distribution. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Cost function</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is a function that is used to minimize the difference between the right answers (labels) and estimated outputs by our Network. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "    cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Type of optimization: Gradient Descent</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the part where you configure the optimizer for your Neural Network. There are several optimizers available, in our case we will use Gradient Descent because it is a well established optimizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Training batches</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train using minibatch Gradient Descent.\n",
    "\n",
    "In practice, Batch Gradient Descent is not often used because is too computationally expensive. The good part about this method is that you have the true gradient, but with the expensive computing task of using the whole dataset in one time. Due to this problem, Neural Networks usually use minibatch to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#Load 50 training examples for each training iteration   \n",
    "for i in range(1000):\n",
    "    batch = mnist.train.next_batch(50)\n",
    "    train_step.run(feed_dict={x: batch[0], y_: batch[1]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Test</h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "acc = accuracy.eval(feed_dict={x: mnist.test.images, y_: mnist.test.labels}) * 100\n",
    "print(\"The final accuracy for the simple ANN model is: {} % \".format(acc) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "sess.close() #finish the session"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref4\"></a>\n",
    "<h2>Evaluating the final result</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is the final result good?\n",
    "\n",
    "Let's check the best algorithm available out there (10th June 2016):\n",
    "  \n",
    "_Result:_ 0.21% error (99.79% accuracy)  \n",
    "<a href=\"http://cs.nyu.edu/~wanli/dropc/\">Reference here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref5\"></a>\n",
    "<h2>How to improve our model?</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Several options as follow:</h4>\n",
    "<ul>\n",
    "    <li>Regularization of Neural Networks using DropConnect</li>\n",
    "    <li>Multi-column Deep Neural Networks for Image Classification</li> \n",
    "    <li>APAC: Augmented Pattern Classification with Neural Networks</li>\n",
    "    <li>Simple Deep Neural Network with Dropout</li>\n",
    "</ul>\n",
    "<h4>In the next part we are going to explore the option:</h4>\n",
    "<ul>\n",
    "    <li>Simple Deep Neural Network with Dropout (more than 1 hidden layer)</li>\n",
    "</ul> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref6\"></a>\n",
    "<h2>2nd part: Deep Learning applied on MNIST</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the first part, we learned how to use a simple ANN to classify MNIST. Now we are going to expand our knowledge using a Deep Neural Network. \n",
    "\n",
    "\n",
    "Architecture of our network is:\n",
    "    \n",
    "- (Input) -> [batch_size, 28, 28, 1]  >> Apply 32 filter of [5x5]\n",
    "- (Convolutional layer 1)  -> [batch_size, 28, 28, 32]\n",
    "- (ReLU 1)  -> [?, 28, 28, 32]\n",
    "- (Max pooling 1) -> [?, 14, 14, 32]\n",
    "- (Convolutional layer 2)  -> [?, 14, 14, 64] \n",
    "- (ReLU 2)  -> [?, 14, 14, 64] \n",
    "- (Max pooling 2)  -> [?, 7, 7, 64] \n",
    "- [fully connected layer 3] -> [1x1024]\n",
    "- [ReLU 3]  -> [1x1024]\n",
    "- [Drop out]  -> [1x1024]\n",
    "- [fully connected layer 4] -> [1x10]\n",
    "\n",
    "\n",
    "The next cells will explore this new architecture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Starting the code</h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "# finish possible remaining session\n",
    "sess.close()\n",
    "\n",
    "#Start interactive session\n",
    "sess = tf.InteractiveSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>The MNIST data</h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Initial parameters</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create general parameters for the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "width = 28 # width of the image in pixels \n",
    "height = 28 # height of the image in pixels\n",
    "flat = width * height # number of pixels in one image \n",
    "class_output = 10 # number of possible classifications for the problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Input and output</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create place holders for inputs and outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "x  = tf.placeholder(tf.float32, shape=[None, flat])\n",
    "y_ = tf.placeholder(tf.float32, shape=[None, class_output])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Converting images of the data set to tensors</h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The input image is 28 pixels by 28 pixels, 1 channel (grayscale). In this case, the first dimension is the <b>batch number</b> of the image, and can be of any size (so we set it to -1). The second and third dimensions are width and height, and the last one is the image channels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "x_image = tf.reshape(x, [-1,28,28,1])  \n",
    "x_image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Convolutional Layer 1</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Defining kernel weight and bias</h4>\n",
    "We define a kernel here. The Size of the filter/kernel is 5x5;  Input channels is 1 (grayscale);  and we need 32 different feature maps (here, 32 feature maps means 32 different filters are applied on each image. So, the output of convolution layer would be 28x28x32). In this step, we create a filter / kernel tensor of shape <code>[filter_height, filter_width, in_channels, out_channels]</code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "W_conv1 = tf.Variable(tf.truncated_normal([5, 5, 1, 32], stddev=0.1))\n",
    "b_conv1 = tf.Variable(tf.constant(0.1, shape=[32])) # need 32 biases for 32 outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://ibm.box.com/shared/static/vn26neef1nnv2oxn5cb3uueowcawhkgb.png\" style=\"width: 800px; height: 400px;\" alt=\"HTML5 Icon\" >\n",
    "\n",
    "</h4>Convolve with weight tensor and add biases.</h4>\n",
    "\n",
    "To create convolutional layer, we use <b>tf.nn.conv2d</b>. It computes a 2-D convolution given 4-D input and filter tensors.\n",
    "\n",
    "Inputs:\n",
    "- tensor of shape [batch, in_height, in_width, in_channels]. x of shape [batch_size,28 ,28, 1]\n",
    "- a filter / kernel tensor of shape [filter_height, filter_width, in_channels, out_channels]. W is of size [5, 5, 1, 32]\n",
    "- stride which is  [1, 1, 1, 1]. The convolutional layer, slides the \"kernel window\" across the input tensor. As the input tensor has 4 dimensions:  [batch, height, width, channels], then the convolution operates on a 2D window on the height and width dimensions. __strides__ determines how much the window shifts by in each of the dimensions. As the first and last dimensions are related to batch and channels, we set the stride to 1. But for second and third dimension, we could set other values, e.g. [1, 2, 2, 1]\n",
    "    \n",
    "    \n",
    "Process:\n",
    "- Change the filter to a 2-D matrix with shape [5\\*5\\*1,32]\n",
    "- Extracts image patches from the input tensor to form a *virtual* tensor of shape `[batch, 28, 28, 5*5*1]`.\n",
    "- For each batch, right-multiplies the filter matrix and the image vector.\n",
    "\n",
    "Output:\n",
    "- A `Tensor` (a 2-D convolution) of size tf.Tensor 'add_7:0' shape=(?, 28, 28, 32)- Notice: the output of the first convolution layer is 32 [28x28] images. Here 32 is considered as volume/depth of the output image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "convolve1= tf.nn.conv2d(x_image, W_conv1, strides=[1, 1, 1, 1], padding='SAME') + b_conv1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://ibm.box.com/shared/static/iizf4ui4b2hh9wn86pplqxu27ykpqci9.png\" style=\"width: 800px; height: 400px;\" alt=\"HTML5 Icon\" >\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Apply the ReLU activation Function</h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step, we just go through all outputs convolution layer, <b>convolve1</b>, and wherever a negative number occurs, we swap it out for a 0. It is called ReLU activation Function.<br> Let f(x) is a ReLU activation function $f(x) = max(0,x)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "h_conv1 = tf.nn.relu(convolve1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Apply the max pooling</h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>max pooling</b> is a form of non-linear down-sampling. It partitions the input image into a set of rectangles and, and then find the maximum value for that region. \n",
    "\n",
    "Lets use <b>tf.nn.max_pool</b> function to perform max pooling. \n",
    "<b>Kernel size:</b> 2x2 (if the window is a 2x2 matrix, it would result in one output pixel)  \n",
    "<b>Strides:</b> dictates the sliding behaviour of the kernel. In this case it will move 2 pixels everytime, thus not overlapping. The input is a matrix of size 28x28x32, and the output would be a matrix of size 14x14x32.\n",
    "\n",
    "<img src=\"https://ibm.box.com/shared/static/kmaja90mn3aud9mro9cn8pbbg1h5pejy.png\" alt=\"HTML5 Icon\" style=\"width: 800px; height: 400px;\"> \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "conv1 = tf.nn.max_pool(h_conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME') #max_pool_2x2\n",
    "conv1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First layer completed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Convolutional Layer 2</h3>\n",
    "<h4>Weights and Biases of kernels</h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We apply the convolution again in this layer. Lets look at the second layer kernel:  \n",
    "- Filter/kernel: 5x5 (25 pixels) \n",
    "- Input channels: 32 (from the 1st Conv layer, we had 32 feature maps) \n",
    "- 64 output feature maps  \n",
    "\n",
    "<b>Notice:</b> here, the input image is [14x14x32], the filter is [5x5x32], we use 64 filters of size [5x5x32], and the output of the convolutional layer would be 64 convolved image, [14x14x64].\n",
    "\n",
    "<b>Notice:</b> the convolution result of applying a filter of size [5x5x32] on image of size [14x14x32] is an image of size [14x14x1], that is, the convolution is functioning on volume."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "W_conv2 = tf.Variable(tf.truncated_normal([5, 5, 32, 64], stddev=0.1))\n",
    "b_conv2 = tf.Variable(tf.constant(0.1, shape=[64])) #need 64 biases for 64 outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Convolve image with weight tensor and add biases.</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "convolve2= tf.nn.conv2d(conv1, W_conv2, strides=[1, 1, 1, 1], padding='SAME') + b_conv2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Apply the ReLU activation Function</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "h_conv2 = tf.nn.relu(convolve2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Apply the max pooling</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "conv2 = tf.nn.max_pool(h_conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME') #max_pool_2x2\n",
    "conv2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second layer completed. So, what is the output of the second layer, layer2?\n",
    "- it is 64 matrix of [7x7]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Fully Connected Layer</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You need a fully connected layer to use the Softmax and create the probabilities in the end. Fully connected layers take the high-level filtered images from previous layer, that is all 64 matrices, and convert them to a flat array.\n",
    "\n",
    "So, each matrix [7x7] will be converted to a matrix of [49x1], and then all of the 64 matrix will be connected, which make an array of size [3136x1]. We will connect it into another layer of size [1024x1]. So, the weight between these 2 layers will be [3136x1024]\n",
    "\n",
    "\n",
    "<img src=\"https://ibm.box.com/shared/static/pr9mnirmlrzm2bitf1d4jj389hyvv7ey.png\" alt=\"HTML5 Icon\" style=\"width: 800px; height: 400px;\"> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Flattening Second Layer</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "layer2_matrix = tf.reshape(conv2, [-1, 7 * 7 * 64])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Weights and Biases between layer 2 and 3</h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Composition of the feature map from the last layer (7x7) multiplied by the number of feature maps (64); 1027 outputs to Softmax layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "W_fc1 = tf.Variable(tf.truncated_normal([7 * 7 * 64, 1024], stddev=0.1))\n",
    "b_fc1 = tf.Variable(tf.constant(0.1, shape=[1024])) # need 1024 biases for 1024 outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Matrix Multiplication (applying weights and biases)</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "fcl = tf.matmul(layer2_matrix, W_fc1) + b_fc1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Apply the ReLU activation Function</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "h_fc1 = tf.nn.relu(fcl)\n",
    "h_fc1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Third layer completed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Dropout Layer, Optional phase for reducing overfitting</h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is a phase where the network \"forget\" some features. At each training step in a mini-batch, some units get switched off randomly so that it will not interact with the network. That is, it weights cannot be updated, nor affect the learning of the other network nodes.  This can be very useful for very large neural networks to prevent overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "keep_prob = tf.placeholder(tf.float32)\n",
    "layer_drop = tf.nn.dropout(h_fc1, keep_prob)\n",
    "layer_drop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Readout Layer (Softmax Layer)</h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Type: Softmax, Fully Connected Layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Weights and Biases</h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In last layer, CNN takes the high-level filtered images and translate them into votes using softmax.\n",
    "Input channels: 1024 (neurons from the 3rd Layer); 10 output features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "W_fc2 = tf.Variable(tf.truncated_normal([1024, 10], stddev=0.1)) #1024 neurons\n",
    "b_fc2 = tf.Variable(tf.constant(0.1, shape=[10])) # 10 possibilities for digits [0,1,2,3,4,5,6,7,8,9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Matrix Multiplication (applying weights and biases)</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fc=tf.matmul(layer_drop, W_fc2) + b_fc2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Apply the Softmax activation Function</h4>\n",
    "<b>softmax</b> allows us to interpret the outputs of <b>fcl4</b> as probabilities. So, <b>y_conv</b> is a tensor of probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "y_CNN= tf.nn.softmax(fc)\n",
    "y_CNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref7\"></a>\n",
    "<h2>Summary of the Deep Convolutional Neural Network</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now is time to remember the structure of  our network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 0) Input - MNIST dataset\n",
    "#### 1) Convolutional and Max-Pooling\n",
    "#### 2) Convolutional and Max-Pooling\n",
    "#### 3) Fully Connected Layer\n",
    "#### 4) Processing - Dropout\n",
    "#### 5) Readout layer - Fully Connected\n",
    "#### 6) Outputs - Classified digits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref8\"></a>\n",
    "<h2>Define functions and train the model</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Define the loss function</h4>\n",
    "\n",
    "We need to compare our output, layer4 tensor, with ground truth for all mini_batch. we can use <b>cross entropy>/b> to see how bad our CNN is working - to measure the error at a softmax layer.\n",
    "\n",
    "The following code shows an toy sample of cross-entropy for a mini-batch of size 2 which its items have been classified. You can run it (first change the cell type to <b>code</b> in the toolbar) to see how cross entropy changes."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "import numpy as np\n",
    "layer4_test =[[0.9, 0.1, 0.1],[0.9, 0.1, 0.1]]\n",
    "y_test=[[1.0, 0.0, 0.0],[1.0, 0.0, 0.0]]\n",
    "np.mean( -np.sum(y_test * np.log(layer4_test),1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>reduce_sum</b> computes the sum of elements of <b>(y_ * tf.log(layer4)</b> across second dimension of the tensor, and <b>reduce_mean</b> computes the mean of all elements in the tensor.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y_CNN), reduction_indices=[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Define the optimizer</h4>\n",
    "\n",
    "It is obvious that we want minimize the error of our network which is calculated by cross_entropy metric. To solve the problem, we have to compute gradients for the loss (which is minimizing the cross-entropy) and apply gradients to variables. It will be done by an optimizer: GradientDescent or Adagrad. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Define prediction</h4>\n",
    "Do you want to know how many of the cases in a mini-batch has been classified correctly? lets count them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "correct_prediction = tf.equal(tf.argmax(y_CNN, 1), tf.argmax(y_, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Define accuracy</h4>\n",
    "It makes more sense to report accuracy using average of correct cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Run session, train</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>If you want a fast result (<b>it might take sometime to train it</b>)</i>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "for i in range(1100):\n",
    "    batch = mnist.train.next_batch(50)\n",
    "    if i%100 == 0:\n",
    "        train_accuracy = accuracy.eval(feed_dict={x:batch[0], y_: batch[1], keep_prob: 1.0})\n",
    "        print(\"step %d, training accuracy %g\"%(i, float(train_accuracy)))\n",
    "    train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success alertsuccess\" style=\"margin-top: 20px\">\n",
    "<font size = 3><strong><i>You can run this cell if you REALLY have time to wait, or you are running it using PowerAI (<b>change the type of the cell to code</b>)</i></strong></font>"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "for i in range(20000):\n",
    "    batch = mnist.train.next_batch(50)\n",
    "    if i%100 == 0:\n",
    "        train_accuracy = accuracy.eval(feed_dict={\n",
    "            x:batch[0], y_: batch[1], keep_prob: 1.0})\n",
    "        print(\"step %d, training accuracy %g\"%(i, train_accuracy))\n",
    "    train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>PS. If you have problems running this notebook, please shutdown all your Jupyter runnning notebooks, clear all cells outputs and run each cell only after the completion of the previous cell.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref9\"></a>\n",
    "<h2>Evaluate the model</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the evaluation to the user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# evaluate in batches to avoid out-of-memory issues\n",
    "n_batches = mnist.test.images.shape[0] // 50\n",
    "cumulative_accuracy = 0.0\n",
    "for index in range(n_batches):\n",
    "    batch = mnist.test.next_batch(50)\n",
    "    cumulative_accuracy += accuracy.eval(feed_dict={x: batch[0], y_: batch[1], keep_prob: 1.0})\n",
    "print(\"test accuracy {}\".format(cumulative_accuracy / n_batches))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Visualization</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do you want to look at all the filters?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "kernels = sess.run(tf.reshape(tf.transpose(W_conv1, perm=[2, 3, 0,1]),[32, -1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "!wget --output-document utils1.py http://deeplearning.net/tutorial/code/utils.py\n",
    "import utils1\n",
    "from utils1 import tile_raster_images\n",
    "import matplotlib.pyplot as plt\n",
    "from PIL import Image\n",
    "%matplotlib inline\n",
    "image = Image.fromarray(tile_raster_images(kernels, img_shape=(5, 5) ,tile_shape=(4, 8), tile_spacing=(1, 1)))\n",
    "### Plot image\n",
    "plt.rcParams['figure.figsize'] = (18.0, 18.0)\n",
    "imgplot = plt.imshow(image)\n",
    "imgplot.set_cmap('gray')  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do you want to see the output of an image passing through first convolution layer?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "plt.rcParams['figure.figsize'] = (5.0, 5.0)\n",
    "sampleimage = mnist.test.images[1]\n",
    "plt.imshow(np.reshape(sampleimage,[28,28]), cmap=\"gray\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "ActivatedUnits = sess.run(convolve1,feed_dict={x:np.reshape(sampleimage,[1,784],order='F'),keep_prob:1.0})\n",
    "filters = ActivatedUnits.shape[3]\n",
    "plt.figure(1, figsize=(20,20))\n",
    "n_columns = 6\n",
    "n_rows = np.math.ceil(filters / n_columns) + 1\n",
    "for i in range(filters):\n",
    "    plt.subplot(n_rows, n_columns, i+1)\n",
    "    plt.title('Filter ' + str(i))\n",
    "    plt.imshow(ActivatedUnits[0,:,:,i], interpolation=\"nearest\", cmap=\"gray\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about second convolution layer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "ActivatedUnits = sess.run(convolve2,feed_dict={x:np.reshape(sampleimage,[1,784],order='F'),keep_prob:1.0})\n",
    "filters = ActivatedUnits.shape[3]\n",
    "plt.figure(1, figsize=(20,20))\n",
    "n_columns = 8\n",
    "n_rows = np.math.ceil(filters / n_columns) + 1\n",
    "for i in range(filters):\n",
    "    plt.subplot(n_rows, n_columns, i+1)\n",
    "    plt.title('Filter ' + str(i))\n",
    "    plt.imshow(ActivatedUnits[0,:,:,i], interpolation=\"nearest\", cmap=\"gray\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "sess.close() #finish the session"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Want to learn more?\n",
    "\n",
    "Running deep learning programs usually needs a high performance platform. __PowerAI__ speeds up deep learning and AI. Built on IBM’s Power Systems, __PowerAI__ is a scalable software platform that accelerates deep learning and AI with blazing performance for individual users or enterprises. The __PowerAI__ platform supports popular machine learning libraries and dependencies including TensorFlow, Caffe, Torch, and Theano. You can use [PowerAI on IMB Cloud](https://cocl.us/ML0120EN_PAI).\n",
    "\n",
    "Also, you can use __Watson Studio__ to run these notebooks faster with bigger datasets.__Watson Studio__ is IBM’s leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, __Watson Studio__ enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of __Watson Studio__ users today with a free account at [Watson Studio](https://cocl.us/ML0120EN_DSX).This is the end of this lesson. Thank you for reading this notebook, and good luck on your studies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Thanks for completing this lesson!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Created by <a href=\"https://linkedin.com/in/saeedaghabozorgi\"> Saeed Aghabozorgi </a>, <a href=\"https://linkedin.com/in/luisotsm\">Luis Otavio Silveira Martins</a>, <a href=\"https://linkedin.com/in/erich-natsubori-sato\"> Erich Natsubori Sato </a></h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References:\n",
    "\n",
    "https://en.wikipedia.org/wiki/Deep_learning    \n",
    "http://ruder.io/optimizing-gradient-descent/<br> \n",
    "http://yann.lecun.com/exdb/mnist/  \n",
    "https://www.quora.com/Artificial-Neural-Networks-What-is-the-difference-between-activation-functions  \n",
    "https://www.tensorflow.org/versions/r0.9/tutorials/mnist/pros/index.html  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "Copyright &copy; 2018 [Cognitive Class](https://cocl.us/DX0108EN_CC). This notebook and its source code are released under the terms of the [MIT License](https://bigdatauniversity.com/mit-license/)."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}