hw2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\">**Note to the TA: Ask the instructor if he wants to provide the section 5.1 mathematical formulae for backward propagation. In 2021, we had initially not provided them with it. But there were too many doubts. So, I had to finally provide them later.**\n",
    "</span>\n",
    "\n",
    "\n",
    "# Basic Instructions\n",
    "\n",
    "1. Enter your Name and UID in the provided space.\n",
    "2. Do the assignment in the notebook itself.\n",
    "3. you are free to use Google Colab.\n",
    "4. Upload to Google Drive.\n",
    "5. Now enter the Google Drive link in the provided space. (you can do this by opening the iPython notebook uploaded using Google Collab).\n",
    "6. Submit the assignment to Gradescope.\n",
    "\n",
    "**Note** - You are NOT supposed to use Pytorch library for this assignment.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Name:  **Name Here**  \n",
    "UID:  **UID Here**\n",
    "\n",
    "Link to Google Drive : **Link Here (make sure it works)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the first part, you will implement all the functions required to build a two layer neural network.\n",
    "In the next part, you will use these functions for image and text classification. Provide your code at the appropriate placeholders."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import h5py\n",
    "import scipy\n",
    "from PIL import Image\n",
    "from scipy import ndimage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Layer Initialization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Create and initialize the parameters of the 2-layer neural network. Use random initialization for the weight matrices (use `0.01*np.random.randn()` ) and zero initialization for the biases. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def initialize_parameters(n_x, n_h, n_y):\n",
    "    \"\"\"\n",
    "    Argument:\n",
    "    n_x -- size of the input layer\n",
    "    n_h -- size of the hidden layer\n",
    "    n_y -- size of the output layer\n",
    "    \n",
    "    Returns:\n",
    "    parameters -- python dictionary containing your parameters:\n",
    "                    W1 -- weight matrix of shape (n_h, n_x)\n",
    "                    b1 -- bias vector of shape (n_h, 1)\n",
    "                    W2 -- weight matrix of shape (n_y, n_h)\n",
    "                    b2 -- bias vector of shape (n_y, 1)\n",
    "    \"\"\"\n",
    "    \n",
    "    np.random.seed(1)\n",
    "    \n",
    "    ### START CODE HERE ### (≈ 4 lines of code)\n",
    "\n",
    "    \n",
    "    \n",
    "    \n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    assert(W1.shape == (n_h, n_x))\n",
    "    assert(b1.shape == (n_h, 1))\n",
    "    assert(W2.shape == (n_y, n_h))\n",
    "    assert(b2.shape == (n_y, 1))\n",
    "    \n",
    "    parameters = {\"W1\": W1,\n",
    "                  \"b1\": b1,\n",
    "                  \"W2\": W2,\n",
    "                  \"b2\": b2}\n",
    "    \n",
    "    return parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters = initialize_parameters(3,2,1)\n",
    "print(\"W1 = \" + str(parameters[\"W1\"]))\n",
    "print(\"b1 = \" + str(parameters[\"b1\"]))\n",
    "print(\"W2 = \" + str(parameters[\"W2\"]))\n",
    "print(\"b2 = \" + str(parameters[\"b2\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected output**:\n",
    "       \n",
    "<table style=\"width:80%\">\n",
    "  <tr>\n",
    "    <td> **W1** </td>\n",
    "    <td> [[ 0.01624345 -0.00611756 -0.00528172]\n",
    " [-0.01072969  0.00865408 -0.02301539]] </td> \n",
    "  </tr>\n",
    "\n",
    "  <tr>\n",
    "    <td> **b1**</td>\n",
    "    <td>[[ 0.]\n",
    " [ 0.]]</td> \n",
    "  </tr>\n",
    "  \n",
    "  <tr>\n",
    "    <td>**W2**</td>\n",
    "    <td> [[ 0.01744812 -0.00761207]]</td>\n",
    "  </tr>\n",
    "  \n",
    "  <tr>\n",
    "    <td> **b2** </td>\n",
    "    <td> [[ 0.]] </td> \n",
    "  </tr>\n",
    "  \n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Forward Propagation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have initialized your parameters, you will do the forward propagation module. You will start by implementing some basic functions that you will use later when implementing the model. You will complete three functions in this order:\n",
    "\n",
    "- LINEAR\n",
    "- LINEAR -> ACTIVATION where ACTIVATION will be either ReLU or Sigmoid.\n",
    "\n",
    "The linear module computes the following equation:\n",
    "\n",
    "$$Z = WA+b\\tag{4}$$\n",
    "\n",
    "### 3.1 Exercise: Build the linear part of forward propagation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def linear_forward(A, W, b):\n",
    "    \"\"\"\n",
    "    Implement the linear part of a layer's forward propagation.\n",
    "\n",
    "    Arguments:\n",
    "    A -- activations from previous layer (or input data): (size of previous layer, number of examples)\n",
    "    W -- weights matrix: numpy array of shape (size of current layer, size of previous layer)\n",
    "    b -- bias vector, numpy array of shape (size of the current layer, 1)\n",
    "\n",
    "    Returns:\n",
    "    Z -- the input of the activation function, also called pre-activation parameter \n",
    "    cache -- a python dictionary containing \"A\", \"W\" and \"b\" ; stored for computing the backward pass efficiently\n",
    "    \"\"\"\n",
    "    \n",
    "    ### START CODE HERE ### (≈ 1 line of code)\n",
    "\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    assert(Z.shape == (W.shape[0], A.shape[1]))\n",
    "    cache = (A, W, b)\n",
    "    \n",
    "    return Z, cache"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(1)\n",
    "\n",
    "A = np.random.randn(3,2)\n",
    "W = np.random.randn(1,3)\n",
    "b = np.random.randn(1,1)\n",
    "\n",
    "Z, linear_cache = linear_forward(A, W, b)\n",
    "print(\"Z = \" + str(Z))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected output**:\n",
    "\n",
    "<table style=\"width:35%\">\n",
    "  \n",
    "  <tr>\n",
    "    <td> **Z** </td>\n",
    "    <td> [[ 3.26295337 -1.23429987]] </td> \n",
    "  </tr>\n",
    "  \n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 - Linear-Activation Forward\n",
    "\n",
    "In this notebook, you will use two activation functions:\n",
    "\n",
    "- **Sigmoid**: $\\sigma(Z) = \\sigma(W A + b) = \\frac{1}{ 1 + e^{-(W A + b)}}$. Write the code for the `sigmoid` function. This function returns **two** items: the activation value \"`a`\" and a \"`cache`\" that contains \"`Z`\" (it's what we will feed in to the corresponding backward function). To use it you could just call: \n",
    "``` python\n",
    "A, activation_cache = sigmoid(Z)\n",
    "```\n",
    "\n",
    "- **ReLU**: The mathematical formula for ReLu is $A = RELU(Z) = max(0, Z)$. Write the code for the  `relu` function. This function returns **two** items: the activation value \"`A`\" and a \"`cache`\" that contains \"`Z`\" (it's what we will feed in to the corresponding backward function). To use it you could just call:\n",
    "``` python\n",
    "A, activation_cache = relu(Z)\n",
    "\n",
    "**Exercise**: \n",
    "- Implement the activation functions\n",
    "- Build the linear activation part of forward propagation. Mathematical relation is: $A = g(Z) = g(WA_{prev} +b)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(Z):\n",
    "    \"\"\"\n",
    "    Implements the sigmoid activation in numpy\n",
    "    \n",
    "    Arguments:\n",
    "    Z -- numpy array of any shape\n",
    "    \n",
    "    Returns:\n",
    "    A -- output of sigmoid(z), same shape as Z\n",
    "    cache -- returns Z, useful during backpropagation\n",
    "    \"\"\"\n",
    "    ### START CODE HERE ### (≈ 2 line of code)\n",
    "\n",
    "    \n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return A, cache\n",
    "\n",
    "def relu(Z):\n",
    "    \"\"\"\n",
    "    Implement the RELU function.\n",
    "\n",
    "    Arguments:\n",
    "    Z -- Output of the linear layer, of any shape\n",
    "\n",
    "    Returns:\n",
    "    A -- Post-activation parameter, of the same shape as Z\n",
    "    cache --  returns Z, useful during backpropagation\n",
    "    \"\"\"\n",
    "    \n",
    "    ### START CODE HERE ### (≈ 2 line of code)\n",
    "\n",
    "    \n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    assert(A.shape == Z.shape) \n",
    "    return A, cache"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def linear_activation_forward(A_prev, W, b, activation):\n",
    "    \"\"\"\n",
    "    Implement the forward propagation for the LINEAR->ACTIVATION layer\n",
    "\n",
    "    Arguments:\n",
    "    A_prev -- activations from previous layer (or input data): (size of previous layer, number of examples)\n",
    "    W -- weights matrix: numpy array of shape (size of current layer, size of previous layer)\n",
    "    b -- bias vector, numpy array of shape (size of the current layer, 1)\n",
    "    activation -- the activation to be used in this layer, stored as a text string: \"sigmoid\" or \"relu\"\n",
    "\n",
    "    Returns:\n",
    "    A -- the output of the activation function, also called the post-activation value \n",
    "    cache -- a python dictionary containing \"linear_cache\" and \"activation_cache\";\n",
    "             stored for computing the backward pass efficiently\n",
    "    \"\"\"\n",
    "    \n",
    "    if activation == \"sigmoid\":\n",
    "        # Inputs: \"A_prev, W, b\". Outputs: \"A, activation_cache\".\n",
    "        ### START CODE HERE ### (≈ 2 lines of code)\n",
    "        \n",
    "        \n",
    "        ### END CODE HERE ###\n",
    "    \n",
    "    elif activation == \"relu\":\n",
    "        # Inputs: \"A_prev, W, b\". Outputs: \"A, activation_cache\".\n",
    "        ### START CODE HERE ### (≈ 2 lines of code)\n",
    "        \n",
    "        \n",
    "        ### END CODE HERE ###\n",
    "    \n",
    "    assert (A.shape == (W.shape[0], A_prev.shape[1]))\n",
    "    cache = (linear_cache, activation_cache)\n",
    "\n",
    "    return A, cache"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(2)\n",
    "A_prev = np.random.randn(3,2)\n",
    "W = np.random.randn(1,3)\n",
    "b = np.random.randn(1,1)\n",
    "\n",
    "A, linear_activation_cache = linear_activation_forward(A_prev, W, b, activation = \"sigmoid\")\n",
    "print(\"With sigmoid: A = \" + str(A))\n",
    "\n",
    "A, linear_activation_cache = linear_activation_forward(A_prev, W, b, activation = \"relu\")\n",
    "print(\"With ReLU: A = \" + str(A))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected output**:\n",
    "       \n",
    "<table style=\"width:35%\">\n",
    "  <tr>\n",
    "    <td> **With sigmoid: A ** </td>\n",
    "    <td > [[ 0.96890023  0.11013289]]</td> \n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td> **With ReLU: A ** </td>\n",
    "    <td > [[ 3.43896131  0.        ]]</td> \n",
    "  </tr>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4 - Loss function\n",
    "\n",
    "Now you will implement forward and backward propagation. You need to compute the loss, because you want to check if your model is actually learning.\n",
    "\n",
    "**Exercise**: Compute the cross-entropy loss $J$, using the following formula: $$-\\frac{1}{m} \\sum\\limits_{i = 1}^{m} (y^{(i)}\\log\\left(a^{ (i)}\\right) + (1-y^{(i)})\\log\\left(1- a^{(i)}\\right)) \\tag{7}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GRADED FUNCTION: compute_loss\n",
    "\n",
    "def compute_loss(A, Y):\n",
    "    \"\"\"\n",
    "    Implement the loss function defined by equation (7).\n",
    "\n",
    "    Arguments:\n",
    "    A -- probability vector corresponding to your label predictions, shape (1, number of examples)\n",
    "    Y -- true \"label\" vector (for example: containing 0 if non-cat, 1 if cat), shape (1, number of examples)\n",
    "\n",
    "    Returns:\n",
    "    loss -- cross-entropy loss\n",
    "    \"\"\"\n",
    "    \n",
    "    m = Y.shape[1]\n",
    "\n",
    "    # Compute loss from aL and y.\n",
    "    ### START CODE HERE ### (≈ 1 lines of code)\n",
    "    \n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    loss = np.squeeze(loss)      # To make sure your loss's shape is what we expect (e.g. this turns [[17]] into 17).\n",
    "    assert(loss.shape == ())\n",
    "    \n",
    "    return loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = np.asarray([[1, 1, 1]])\n",
    "A = np.array([[.8,.9,0.4]])\n",
    "\n",
    "print(\"loss = \" + str(compute_loss(A, Y)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected Output**:\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "    <td>**loss** </td>\n",
    "    <td> 0.41493159961539694</td> \n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5 - Backward propagation module\n",
    "\n",
    "Just like with forward propagation, you will implement helper functions for backpropagation. Remember that back propagation is used to calculate the gradient of the loss function with respect to the parameters. \n",
    "\n",
    "Now, similar to forward propagation, you are going to build the backward propagation in two steps:\n",
    "- LINEAR backward\n",
    "- LINEAR -> ACTIVATION backward where ACTIVATION computes the derivative of either the ReLU or sigmoid activation\n",
    "\n",
    "\n",
    "Following are the relationships - \n",
    "\\begin{align}\n",
    "dA_{prev} &= W^{T}dZ\\\\\n",
    "dW &= dZ A_{prev}^{T} \\\\\n",
    "db &= \\sum_{i=1}^m dZ^{(i)} \\: \\text{       where m is the number of samples} \\\\\n",
    "    &= dZ \\hat I \\: \\text{    where $\\hat I$ is a column vector of size(m,1) with all entries 1 }\n",
    "\\end{align}\n",
    "\n",
    "### 5.1 - Linear backward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GRADED FUNCTION: linear_backward\n",
    "\n",
    "def linear_backward(dZ, cache):\n",
    "    \"\"\"\n",
    "    Implement the linear portion of backward propagation for a single layer (layer l)\n",
    "\n",
    "    Arguments:\n",
    "    dZ -- Gradient of the loss with respect to the linear output (of current layer l)\n",
    "    cache -- tuple of values (A_prev, W, b) coming from the forward propagation in the current layer\n",
    "\n",
    "    Returns:\n",
    "    dA_prev -- Gradient of the loss with respect to the activation (of the previous layer l-1), same shape as A_prev\n",
    "    dW -- Gradient of the loss with respect to W (current layer l), same shape as W\n",
    "    db -- Gradient of the loss with respect to b (current layer l), same shape as b\n",
    "    \"\"\"\n",
    "    A_prev, W, b = cache\n",
    "    m = A_prev.shape[1]\n",
    "\n",
    "    ### START CODE HERE ### (≈ 3 lines of code)\n",
    "\n",
    "    \n",
    "    \n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    assert (dA_prev.shape == A_prev.shape)\n",
    "    assert (dW.shape == W.shape)\n",
    "    assert (db.shape == b.shape)\n",
    "    \n",
    "    return dA_prev, dW, db"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(1)\n",
    "dZ = np.random.randn(1,2)\n",
    "A = np.random.randn(3,2)\n",
    "W = np.random.randn(1,3)\n",
    "b = np.random.randn(1,1)\n",
    "linear_cache = (A, W, b)\n",
    "\n",
    "dA_prev, dW, db = linear_backward(dZ, linear_cache)\n",
    "print (\"dA_prev = \"+ str(dA_prev))\n",
    "print (\"dW = \" + str(dW))\n",
    "print (\"db = \" + str(db))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected Output**: \n",
    "\n",
    "<table style=\"width:90%\">\n",
    "  <tr>\n",
    "    <td> **dA_prev** </td>\n",
    "    <td > [[ 0.51822968 -0.19517421]\n",
    " [-0.40506361  0.15255393]\n",
    " [ 2.37496825 -0.89445391]] </td> \n",
    "  </tr> \n",
    "    <tr>\n",
    "        <td> **dW** </td>\n",
    "        <td > [[-0.2015379   2.81370193  3.2998501 ]] </td> \n",
    "    </tr> \n",
    "    <tr>\n",
    "        <td> **db** </td>\n",
    "        <td> [[1.01258895]] </td> \n",
    "    </tr>     \n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 - Linear Activation backward\n",
    "\n",
    "Next, you will create a function that merges the two helper functions: **`linear_backward`** and the backward step for the activation **`linear_activation_backward`**. \n",
    "\n",
    "Before implementing `linear_activation_backward`, you need to implement two backward functions for each activations:\n",
    "- **`sigmoid_backward`**: Implements the backward propagation for SIGMOID unit. You can call it as follows:\n",
    "\n",
    "```python\n",
    "dZ = sigmoid_backward(dA, activation_cache)\n",
    "```\n",
    "\n",
    "- **`relu_backward`**: Implements the backward propagation for RELU unit. You can call it as follows:\n",
    "\n",
    "```python\n",
    "dZ = relu_backward(dA, activation_cache)\n",
    "```\n",
    "\n",
    "If $g(.)$ is the activation function, \n",
    "`sigmoid_backward` and `relu_backward` compute $$dZ^{[l]} = dA^{[l]} * g'(Z^{[l]}) \\tag{11}$$.  \n",
    "\n",
    "**Exercise**: \n",
    "- Implement the backward functions for the relu and sigmoid activation layer.\n",
    "- Implement the backpropagation for the *LINEAR->ACTIVATION* layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def relu_backward(dA, cache):\n",
    "    \"\"\"\n",
    "    Implement the backward propagation for a single RELU unit.\n",
    "\n",
    "    Arguments:\n",
    "    dA -- post-activation gradient, of any shape\n",
    "    cache -- 'Z' where we store for computing backward propagation efficiently\n",
    "\n",
    "    Returns:\n",
    "    dZ -- Gradient of the loss with respect to Z\n",
    "    \"\"\"\n",
    "    \n",
    "    Z = cache\n",
    "    dZ = np.array(dA, copy=True) # just converting dz to a correct object.\n",
    "    \n",
    "    ### START CODE HERE ### (≈ 1 line of code)\n",
    "\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    assert (dZ.shape == Z.shape)\n",
    "    \n",
    "    return dZ\n",
    "\n",
    "def sigmoid_backward(dA, cache):\n",
    "    \"\"\"\n",
    "    Implement the backward propagation for a single SIGMOID unit.\n",
    "\n",
    "    Arguments:\n",
    "    dA -- post-activation gradient, of any shape\n",
    "    cache -- 'Z' where we store for computing backward propagation efficiently\n",
    "\n",
    "    Returns:\n",
    "    dZ -- Gradient of the loss with respect to Z\n",
    "    \"\"\"\n",
    "    \n",
    "    Z = cache\n",
    "    \n",
    "    ### START CODE HERE ### (≈ 2 line of code)\n",
    "\n",
    "    \n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    assert (dZ.shape == Z.shape)\n",
    "    \n",
    "    return dZ\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GRADED FUNCTION: linear_activation_backward\n",
    "\n",
    "def linear_activation_backward(dA, cache, activation):\n",
    "    \"\"\"\n",
    "    Implement the backward propagation for the LINEAR->ACTIVATION layer.\n",
    "    \n",
    "    Arguments:\n",
    "    dA -- post-activation gradient for current layer l \n",
    "    cache -- tuple of values (linear_cache, activation_cache) we store for computing backward propagation efficiently\n",
    "    activation -- the activation to be used in this layer, stored as a text string: \"sigmoid\" or \"relu\"\n",
    "    \n",
    "    Returns:\n",
    "    dA_prev -- Gradient of the loss with respect to the activation (of the previous layer l-1), same shape as A_prev\n",
    "    dW -- Gradient of the loss with respect to W (current layer l), same shape as W\n",
    "    db -- Gradient of the loss with respect to b (current layer l), same shape as b\n",
    "    \"\"\"\n",
    "    linear_cache, activation_cache = cache\n",
    "    \n",
    "    if activation == \"relu\":\n",
    "        ### START CODE HERE ### (≈ 2 lines of code)\n",
    "        \n",
    "        \n",
    "        ### END CODE HERE ###\n",
    "        \n",
    "    elif activation == \"sigmoid\":\n",
    "        ### START CODE HERE ### (≈ 2 lines of code)\n",
    "        \n",
    "        \n",
    "        ### END CODE HERE ###\n",
    "    \n",
    "    return dA_prev, dW, db"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(2)\n",
    "dA = np.random.randn(1,2)\n",
    "A = np.random.randn(3,2)\n",
    "W = np.random.randn(1,3)\n",
    "b = np.random.randn(1,1)\n",
    "Z = np.random.randn(1,2)\n",
    "linear_cache = (A, W, b)\n",
    "activation_cache = Z\n",
    "linear_activation_cache = (linear_cache, activation_cache)\n",
    "\n",
    "dA_prev, dW, db = linear_activation_backward(dA, linear_activation_cache, activation = \"sigmoid\")\n",
    "print (\"sigmoid:\")\n",
    "print (\"dA_prev = \"+ str(dA_prev))\n",
    "print (\"dW = \" + str(dW))\n",
    "print (\"db = \" + str(db) + \"\\n\")\n",
    "\n",
    "dA_prev, dW, db = linear_activation_backward(dA, linear_activation_cache, activation = \"relu\")\n",
    "print (\"relu:\")\n",
    "print (\"dA_prev = \"+ str(dA_prev))\n",
    "print (\"dW = \" + str(dW))\n",
    "print (\"db = \" + str(db))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected output with sigmoid:**\n",
    "\n",
    "<table style=\"width:100%\">\n",
    "  <tr>\n",
    "    <td > dA_prev </td> \n",
    "           <td >[[ 0.11017994  0.01105339]\n",
    " [ 0.09466817  0.00949723]\n",
    " [-0.05743092 -0.00576154]] </td> \n",
    "  </tr> \n",
    "    <tr>\n",
    "    <td > dW </td> \n",
    "           <td > [[ 0.20533573  0.19557101 -0.03936168]] </td> \n",
    "  </tr> \n",
    "    <tr>\n",
    "    <td > db </td> \n",
    "           <td > [[-0.11459244]] </td> \n",
    "  </tr> \n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected output with relu:**\n",
    "\n",
    "<table style=\"width:100%\">\n",
    "  <tr>\n",
    "    <td > dA_prev </td> \n",
    "           <td > [[ 0.44090989  0.        ]\n",
    " [ 0.37883606  0.        ]\n",
    " [-0.2298228   0.        ]] </td> \n",
    "  </tr> \n",
    "    <tr>\n",
    "    <td > dW </td> \n",
    "           <td > [[ 0.89027649  0.74742835 -0.20957978]] </td> \n",
    "  </tr> \n",
    "    <tr>\n",
    "    <td > db </td> \n",
    "           <td > [[-0.41675785]] </td> \n",
    "  </tr> \n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6 - Update Parameters\n",
    "\n",
    "In this section you will update the parameters of the model, using gradient descent: \n",
    "\n",
    "$$ W^{[1]} = W^{[1]} - \\alpha \\text{ } dW^{[1]} \\tag{16}$$\n",
    "$$ b^{[1]} = b^{[1]} - \\alpha \\text{ } db^{[1]} \\tag{17}$$\n",
    "$$ W^{[2]} = W^{[2]} - \\alpha \\text{ } dW^{[2} \\tag{16}$$\n",
    "$$ b^{[2]} = b^{[2]} - \\alpha \\text{ } db^{[2]} \\tag{17}$$\n",
    "\n",
    "where $\\alpha$ is the learning rate. After computing the updated parameters, store them in the parameters dictionary. \n",
    "\n",
    "**Exercise**: Implement `update_parameters()` to update your parameters using gradient descent.\n",
    "\n",
    "**Instructions**:\n",
    "Update parameters using gradient descent.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GRADED FUNCTION: update_parameters\n",
    "\n",
    "def update_parameters(parameters, grads, learning_rate):\n",
    "    \"\"\"\n",
    "    Update parameters using gradient descent\n",
    "    \n",
    "    Arguments:\n",
    "    parameters -- python dictionary containing your parameters \n",
    "    grads -- python dictionary containing your gradients, output of L_model_backward\n",
    "    \n",
    "    Returns:\n",
    "    parameters -- python dictionary containing your updated parameters \n",
    "                  parameters[\"W\" + str(l)] = ... \n",
    "                  parameters[\"b\" + str(l)] = ...\n",
    "    \"\"\"\n",
    "    # Update rule for each parameter. Use a for loop.\n",
    "    ### START CODE HERE ### (≈ 4 lines of code)\n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "    ### END CODE HERE ###\n",
    "    return parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(2)\n",
    "W1 = np.random.randn(3,4)\n",
    "b1 = np.random.randn(3,1)\n",
    "W2 = np.random.randn(1,3)\n",
    "b2 = np.random.randn(1,1)\n",
    "parameters = {\"W1\": W1,\n",
    "              \"b1\": b1,\n",
    "              \"W2\": W2,\n",
    "              \"b2\": b2}\n",
    "np.random.seed(3)\n",
    "dW1 = np.random.randn(3,4)\n",
    "db1 = np.random.randn(3,1)\n",
    "dW2 = np.random.randn(1,3)\n",
    "db2 = np.random.randn(1,1)\n",
    "grads = {\"dW1\": dW1,\n",
    "         \"db1\": db1,\n",
    "         \"dW2\": dW2,\n",
    "         \"db2\": db2}\n",
    "parameters = update_parameters(parameters, grads, 0.1)\n",
    "\n",
    "print (\"W1 = \"+ str(parameters[\"W1\"]))\n",
    "print (\"b1 = \"+ str(parameters[\"b1\"]))\n",
    "print (\"W2 = \"+ str(parameters[\"W2\"]))\n",
    "print (\"b2 = \"+ str(parameters[\"b2\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected Output**:\n",
    "\n",
    "<table style=\"width:100%\"> \n",
    "    <tr>\n",
    "    <td > W1 </td> \n",
    "           <td > [[-0.59562069 -0.09991781 -2.14584584  1.82662008]\n",
    " [-1.76569676 -0.80627147  0.51115557 -1.18258802]\n",
    " [-1.0535704  -0.86128581  0.68284052  2.20374577]] </td> \n",
    "  </tr> \n",
    "    <tr>\n",
    "    <td > b1 </td> \n",
    "           <td > [[-0.04659241]\n",
    " [-1.28888275]\n",
    " [ 0.53405496]] </td> \n",
    "  </tr> \n",
    "  <tr>\n",
    "    <td > W2 </td> \n",
    "           <td > [[-0.55569196  0.0354055   1.32964895]]</td> \n",
    "  </tr> \n",
    "    <tr>\n",
    "    <td > b2 </td> \n",
    "           <td > [[-0.84610769]] </td> \n",
    "  </tr> \n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## 7 - Conclusion\n",
    "\n",
    "Congrats on implementing all the functions required for building a deep neural network! \n",
    "\n",
    "We know it was a long assignment but going forward it will only get better. The next part of the assignment is easier. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2:\n",
    "\n",
    "In the next part you will put all these together to build a two-layer neural networks for image classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (5.0, 4.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "np.random.seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Problem Statement**: You are given a dataset (\"data/train_catvnoncat.h5\", \"data/test_catvnoncat.h5\") containing:\n",
    "    - a training set of m_train images labelled as cat (1) or non-cat (0)\n",
    "    - a test set of m_test images labelled as cat and non-cat\n",
    "    - each image is of shape (num_px, num_px, 3) where 3 is for the 3 channels (RGB).\n",
    "\n",
    "Let's get more familiar with the dataset. Load the data by completing the function and run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_data(train_file, test_file):\n",
    "    # Load the training data\n",
    "    train_dataset =\n",
    "    \n",
    "    # Separate features(x) and labels(y) for training set\n",
    "    train_set_x_orig = \n",
    "    train_set_y_orig = \n",
    "\n",
    "    # Load the test data\n",
    "    test_dataset = \n",
    "    \n",
    "    # Separate features(x) and labels(y) for training set\n",
    "    test_set_x_orig = \n",
    "    test_set_y_orig =\n",
    "\n",
    "    classes = np.array(test_dataset[\"list_classes\"][:]) # the list of classes\n",
    "    \n",
    "    train_set_y_orig = train_set_y_orig.reshape((1, train_set_y_orig.shape[0]))\n",
    "    test_set_y_orig = test_set_y_orig.reshape((1, test_set_y_orig.shape[0]))\n",
    "    \n",
    "    return train_set_x_orig, train_set_y_orig, test_set_x_orig, test_set_y_orig, classes\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_file=\"data/train_catvnoncat.h5\"\n",
    "test_file=\"data/test_catvnoncat.h5\"\n",
    "train_x_orig, train_y, test_x_orig, test_y, classes = load_data(train_file, test_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code will show you an image in the dataset. Feel free to change the index and re-run the cell multiple times to see other images. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example of a picture\n",
    "index = 10\n",
    "plt.imshow(train_x_orig[index])\n",
    "print (\"y = \" + str(train_y[0,index]) + \". It's a \" + classes[train_y[0,index]].decode(\"utf-8\") +  \" picture.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Explore your dataset \n",
    "m_train = train_x_orig.shape[0]\n",
    "num_px = train_x_orig.shape[1]\n",
    "m_test = test_x_orig.shape[0]\n",
    "\n",
    "print (\"Number of training examples: \" + str(m_train))\n",
    "print (\"Number of testing examples: \" + str(m_test))\n",
    "print (\"Each image is of size: (\" + str(num_px) + \", \" + str(num_px) + \", 3)\")\n",
    "print (\"train_x_orig shape: \" + str(train_x_orig.shape))\n",
    "print (\"train_y shape: \" + str(train_y.shape))\n",
    "print (\"test_x_orig shape: \" + str(test_x_orig.shape))\n",
    "print (\"test_y shape: \" + str(test_y.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual, you reshape and standardize the images before feeding them to the network.\n",
    "\n",
    "<img src=\"imvectorkiank.png\" style=\"width:450px;height:300px;\">\n",
    "\n",
    "<caption><center> <u>Figure 1</u>: Image to vector conversion. <br> </center></caption>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reshape the training and test examples \n",
    "train_x_flatten = train_x_orig.reshape(train_x_orig.shape[0], -1).T   # The \"-1\" makes reshape flatten the remaining dimensions\n",
    "test_x_flatten = test_x_orig.reshape(test_x_orig.shape[0], -1).T\n",
    "\n",
    "# Standardize data to have feature values between 0 and 1.\n",
    "train_x = train_x_flatten/255.\n",
    "test_x = test_x_flatten/255.\n",
    "\n",
    "print (\"train_x's shape: \" + str(train_x.shape))\n",
    "print (\"test_x's shape: \" + str(test_x.shape))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 - Architecture of your model\n",
    "\n",
    "Now that you are familiar with the dataset, it is time to build a deep neural network to distinguish cat images from non-cat images.\n",
    "\n",
    "###  2-layer neural network\n",
    "\n",
    "<img src=\"2layerNN_kiank.png\" style=\"width:650px;height:400px;\">\n",
    "<caption><center> <u>Figure 2</u>: 2-layer neural network. <br> The model can be summarized as: ***INPUT -> LINEAR -> RELU -> LINEAR -> SIGMOID -> OUTPUT***. </center></caption>\n",
    "\n",
    "<u>Detailed Architecture of figure 2</u>:\n",
    "- The input is a (64,64,3) image which is flattened to a vector of size $(12288,1)$. \n",
    "- The corresponding vector: $[x_0,x_1,...,x_{12287}]^T$ is then multiplied by the weight matrix $W^{[1]}$ of size $(n^{[1]}, 12288)$.\n",
    "- You then add a bias term and take its relu to get the following vector: $[a_0^{[1]}, a_1^{[1]},..., a_{n^{[1]}-1}^{[1]}]^T$.\n",
    "- You multiply the resulting vector by $W^{[2]}$ and add your intercept (bias). \n",
    "- Finally, you take the sigmoid of the result. If it is greater than 0.5, you classify it to be a cat.\n",
    "\n",
    "\n",
    "###  General methodology\n",
    "\n",
    "As usual you will follow the Deep Learning methodology to build the model:\n",
    "    1. Initialize parameters / Define hyperparameters\n",
    "    2. Loop for num_iterations:\n",
    "        a. Forward propagation\n",
    "        b. Compute loss function\n",
    "        c. Backward propagation\n",
    "        d. Update parameters (using parameters, and grads from backprop) \n",
    "    4. Use trained parameters to predict labels\n",
    "\n",
    "Let's now implement those the model!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "**Question**:  Use the helper functions you have implemented in the previous assignment to build a 2-layer neural network with the following structure: *LINEAR -> RELU -> LINEAR -> SIGMOID*. The functions you may need and their inputs are:\n",
    "```python\n",
    "def initialize_parameters(n_x, n_h, n_y):\n",
    "    ...\n",
    "    return parameters \n",
    "def linear_activation_forward(A_prev, W, b, activation):\n",
    "    ...\n",
    "    return A, cache\n",
    "def compute_loss(AL, Y):\n",
    "    ...\n",
    "    return loss\n",
    "def linear_activation_backward(dA, cache, activation):\n",
    "    ...\n",
    "    return dA_prev, dW, db\n",
    "def update_parameters(parameters, grads, learning_rate):\n",
    "    ...\n",
    "    return parameters\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### CONSTANTS DEFINING THE MODEL ####\n",
    "n_x = 12288     # num_px * num_px * 3\n",
    "n_h = 7\n",
    "n_y = 1\n",
    "layers_dims = (n_x, n_h, n_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def two_layer_model(X, Y, layers_dims, learning_rate = 0.0075, num_iterations = 3000, print_loss=False):\n",
    "    \"\"\"\n",
    "    Implements a two-layer neural network: LINEAR->RELU->LINEAR->SIGMOID.\n",
    "    \n",
    "    Arguments:\n",
    "    X -- input data, of shape (n_x, number of examples)\n",
    "    Y -- true \"label\" vector (containing 0 if cat, 1 if non-cat), of shape (1, number of examples)\n",
    "    layers_dims -- dimensions of the layers (n_x, n_h, n_y)\n",
    "    num_iterations -- number of iterations of the optimization loop\n",
    "    learning_rate -- learning rate of the gradient descent update rule\n",
    "    print_loss -- If set to True, this will print the loss every 100 iterations \n",
    "    \n",
    "    Returns:\n",
    "    parameters -- a dictionary containing W1, W2, b1, and b2\n",
    "    \"\"\"\n",
    "    \n",
    "    np.random.seed(1)\n",
    "    grads = {}\n",
    "    losses = []                              # to keep track of the loss\n",
    "    m = X.shape[1]                           # number of examples\n",
    "    (n_x, n_h, n_y) = layers_dims\n",
    "    \n",
    "    # Initialize parameters dictionary, by calling one of the functions you'd previously implemented\n",
    "    ### START CODE HERE ### (≈ 1 line of code)\n",
    "    \n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    # Get W1, b1, W2 and b2 from the dictionary parameters.\n",
    "    W1 = parameters[\"W1\"]\n",
    "    b1 = parameters[\"b1\"]\n",
    "    W2 = parameters[\"W2\"]\n",
    "    b2 = parameters[\"b2\"]\n",
    "    \n",
    "    # Loop (gradient descent)\n",
    "\n",
    "    for i in range(0, num_iterations):\n",
    "\n",
    "        # Forward propagation: LINEAR -> RELU -> LINEAR -> SIGMOID. Inputs: \"X, W1, b1, W2, b2\". Output: \"A1, cache1, A2, cache2\".\n",
    "        ### START CODE HERE ### (≈ 2 lines of code)\n",
    "        \n",
    "        \n",
    "        ### END CODE HERE ###\n",
    "        \n",
    "        # Compute loss\n",
    "        ### START CODE HERE ### (≈ 1 line of code)\n",
    "        \n",
    "        ### END CODE HERE ###\n",
    "        \n",
    "        # Initializing backward propagation\n",
    "        dA2 = - (np.divide(Y, A2) - np.divide(1 - Y, 1 - A2)) / m\n",
    "        \n",
    "        # Backward propagation. Inputs: \"dA2, cache2, cache1\". Outputs: \"dA1, dW2, db2; also dA0 (not used), dW1, db1\".\n",
    "        ### START CODE HERE ### (≈ 2 lines of code)\n",
    "        \n",
    "        \n",
    "        ### END CODE HERE ###\n",
    "        \n",
    "        # Set grads['dWl'] to dW1, grads['db1'] to db1, grads['dW2'] to dW2, grads['db2'] to db2\n",
    "        ### START CODE HERE ### (≈ 4 lines of code)\n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        ### END CODE HERE ###\n",
    "        \n",
    "    \n",
    "        # Update parameters.\n",
    "        ### START CODE HERE ### (approx. 1 line of code)\n",
    "        \n",
    "        ### END CODE HERE ###\n",
    "\n",
    "        # Retrieve W1, b1, W2, b2 from parameters\n",
    "        W1 = parameters[\"W1\"]\n",
    "        b1 = parameters[\"b1\"]\n",
    "        W2 = parameters[\"W2\"]\n",
    "        b2 = parameters[\"b2\"]\n",
    "        \n",
    "        # Print the loss every 100 training example\n",
    "        if print_loss and i % 100 == 0:\n",
    "            print(\"Loss after iteration {}: {}\".format(i, np.squeeze(loss)))\n",
    "        if print_loss and i % 100 == 0:\n",
    "            losses.append(loss)\n",
    "       \n",
    "    # plot the loss\n",
    "\n",
    "    plt.plot(np.squeeze(losses))\n",
    "    plt.ylabel('loss')\n",
    "    plt.xlabel('iterations (per tens)')\n",
    "    plt.title(\"Learning rate =\" + str(learning_rate))\n",
    "    plt.show()\n",
    "    \n",
    "    return parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters = two_layer_model(train_x, train_y, layers_dims = (n_x, n_h, n_y), num_iterations = 2500, print_loss=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected Output**:\n",
    "<table> \n",
    "    <tr>\n",
    "        <td> **Loss after iteration 0**</td>\n",
    "        <td> 0.6930497356599888 </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td> **Loss after iteration 100**</td>\n",
    "        <td> 0.6464320953428849 </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td> **...**</td>\n",
    "        <td> ... </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td> **Loss after iteration 2400**</td>\n",
    "        <td> 0.048554785628770206 </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Good thing you built a vectorized implementation! Otherwise it might have taken 10 times longer to train this.\n",
    "\n",
    "Now, you can use the trained parameters to classify images from the dataset. \n",
    "\n",
    "***Exercise:*** \n",
    "- Implement the forward function\n",
    "- Implement the predict function below to make prediction on test_images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def two_layer_forward(X, parameters):\n",
    "    \"\"\"\n",
    "    Implement forward propagation for the [LINEAR->RELU]*(L-1)->LINEAR->SIGMOID computation\n",
    "    \n",
    "    Arguments:\n",
    "    X -- data, numpy array of shape (input size, number of examples)\n",
    "    parameters -- output of initialize_parameters_deep()\n",
    "    \n",
    "    Returns:\n",
    "    AL -- last post-activation value\n",
    "    caches -- list of caches containing:\n",
    "                every cache of linear_relu_forward() (there are L-1 of them, indexed from 0 to L-2)\n",
    "                the cache of linear_sigmoid_forward() (there is one, indexed L-1)\n",
    "    \"\"\"\n",
    "\n",
    "    caches = []\n",
    "    A = X\n",
    "    \n",
    "    # Implement LINEAR -> RELU. Add \"cache\" to the \"caches\" list.\n",
    "    ### START CODE HERE ### (approx. 3 line of code)\n",
    "        \n",
    "    \n",
    "    \n",
    "    ### END CODE HERE ###\n",
    "\n",
    "    # Implement LINEAR -> SIGMOID. Add \"cache\" to the \"caches\" list.\n",
    "    ### START CODE HERE ### (approx. 3 line of code)\n",
    "        \n",
    "    \n",
    "    \n",
    "    ### END CODE HERE ###\n",
    "\n",
    "    assert(A2.shape == (1,X.shape[1]))\n",
    "            \n",
    "    return A2, caches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(X, y, parameters):\n",
    "    \"\"\"\n",
    "    This function is used to predict the results of a  L-layer neural network.\n",
    "    \n",
    "    Arguments:\n",
    "    X -- data set of examples you would like to label\n",
    "    parameters -- parameters of the trained model\n",
    "    \n",
    "    Returns:\n",
    "    p -- predictions for the given dataset X\n",
    "    \"\"\"\n",
    "    \n",
    "    m = X.shape[1]\n",
    "    n = len(parameters) // 2 # number of layers in the neural network\n",
    "    p = np.zeros((1,m))\n",
    "    \n",
    "    # Forward propagation\n",
    "    ### START CODE HERE ### (≈ 1 lines of code)\n",
    "    probas, caches =  \n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    # convert probas to 0/1 predictions\n",
    "    for i in range(0, probas.shape[1]):\n",
    "        ### START CODE HERE ### (≈ 4 lines of code)\n",
    "        \n",
    "        \n",
    "        \n",
    "        ### END CODE HERE ###\n",
    "    \n",
    "    print(\"Accuracy: \"  + str(np.sum((p == y)/m)))\n",
    "        \n",
    "    return p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions_train = predict(train_x, train_y, parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions_test = predict(test_x, test_y, parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Exercise:***\n",
    "Identify the hyperparameters in the model and For each hyperparameter\n",
    "- Briefly explain its role\n",
    "- Explore a range of values and describe their impact on (a) training loss and (b) test accuracy\n",
    "- Report the best hyperparameter value found.\n",
    "\n",
    "Note: Provide your results and explanations in the answer for this question."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Type your answer here**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Results Analysis\n",
    "\n",
    "First, let's take a look at some images the 2-layer model labeled incorrectly. This will show a few mislabeled images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_mislabeled_images(classes, X, y, p):\n",
    "    \"\"\"\n",
    "    Plots images where predictions and truth were different.\n",
    "    X -- dataset\n",
    "    y -- true labels\n",
    "    p -- predictions\n",
    "    \"\"\"\n",
    "    a = p + y\n",
    "    mislabeled_indices = np.asarray(np.where(a == 1))\n",
    "    plt.rcParams['figure.figsize'] = (40.0, 40.0) # set default size of plots\n",
    "    num_images = len(mislabeled_indices[0])\n",
    "    for i in range(num_images):\n",
    "        index = mislabeled_indices[1][i]\n",
    "        \n",
    "        plt.subplot(2, num_images, i + 1)\n",
    "        plt.imshow(X[:,index].reshape(64,64,3), interpolation='nearest')\n",
    "        plt.axis('off')\n",
    "        plt.title(\"Prediction: \" + classes[int(p[0,index])].decode(\"utf-8\") + \" \\n Class: \" + classes[y[0,index]].decode(\"utf-8\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_mislabeled_images(classes, test_x, test_y, predictions_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Exercise:*** Identify a few types of images that tends to perform poorly on the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, lets use the same architecture to predict sentiment of movie reviews. In this section, most of the implementation is already provided. The exercises are mainly to understand what the workflow is when handling the text data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Problem Statement**: You are given a dataset (\"train_imdb.txt\", \"test_imdb.txt\") containing:\n",
    "    - a training set of m_train reviews \n",
    "    - a test set of m_test reviews \n",
    "    - the labels for the training examples are such that the first 50% belong to class 1 (positive) and the rest 50% of the data belong to class 0(negative)\n",
    "    \n",
    "Let's get more familiar with the dataset. Load the data by completing the function and run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_data(train_file, test_file):\n",
    "    train_dataset = []\n",
    "    test_dataset = []\n",
    "    \n",
    "    # Read the training dataset file line by line\n",
    "    for line in open(train_file, 'r'):\n",
    "        train_dataset.append(line.strip())\n",
    "        \n",
    "    for line in open(test_file, 'r'):\n",
    "        test_dataset.append(line.strip())\n",
    "    return train_dataset, test_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_file = \"data/train_imdb.txt\"\n",
    "test_file = \"data/test_imdb.txt\"\n",
    "train_dataset, test_dataset = load_data(train_file, test_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is just how the data is organized. The first 50% data is positive and the rest 50% is negative for both train and test splits.\n",
    "y = [1 if i < len(train_dataset)*0.5 else 0 for i in range(len(train_dataset))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual, lets check our dataset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example of a review\n",
    "index = 10\n",
    "print(train_dataset[index])\n",
    "print (\"y = \" + str(y[index]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Explore your dataset \n",
    "m_train = len(train_dataset)\n",
    "m_test = len(test_dataset)\n",
    "\n",
    "print (\"Number of training examples: \" + str(m_train))\n",
    "print (\"Number of testing examples: \" + str(m_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the example review, you can see that the raw data is really noisy! This is generally the case with the text data. Hence, Preprocessing the raw input and cleaning the text is  essential. Please run the code snippet provided below.\n",
    "\n",
    "**Exercise**: Explain what pattern the model is trying to capture using re.compile. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Type your answer here**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "REPLACE_NO_SPACE = re.compile(\"(\\.)|(\\;)|(\\:)|(\\!)|(\\')|(\\?)|(\\,)|(\\\")|(\\()|(\\))|(\\[)|(\\])|(\\d+)\")\n",
    "REPLACE_WITH_SPACE = re.compile(\"(<br\\s*/><br\\s*/>)|(\\-)|(\\/)\")\n",
    "NO_SPACE = \"\"\n",
    "SPACE = \" \"\n",
    "\n",
    "def preprocess_reviews(reviews):\n",
    "    \n",
    "    reviews = [REPLACE_NO_SPACE.sub(NO_SPACE, line.lower()) for line in reviews]\n",
    "    reviews = [REPLACE_WITH_SPACE.sub(SPACE, line) for line in reviews]\n",
    "    \n",
    "    return reviews\n",
    "\n",
    "train_dataset_clean = preprocess_reviews(train_dataset)\n",
    "test_dataset_clean = preprocess_reviews(test_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example of a clean review\n",
    "index = 10\n",
    "print(train_dataset_clean[index])\n",
    "print (\"y = \" + str(y[index]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vectorization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets create a feature vector for our reviews based on a simple bag of words model. So, given an input text, we need to create a numerical vector which is simply the vector of word counts for each word of the vocabulary. Run the code below to get the feature representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "cv = CountVectorizer(binary=True, stop_words=\"english\", max_features=2000)\n",
    "cv.fit(train_dataset_clean)\n",
    "X = cv.transform(train_dataset_clean)\n",
    "X_test = cv.transform(test_dataset_clean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CountVectorizer provides a sparse feature representation by default which is reasonable because only some words occur in individual example. However, for training neural network models, we generally use a dense representation vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.array(X.todense()).astype(float)\n",
    "X_test = np.array(X_test.todense()).astype(float)\n",
    "y = np.array(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_val, y_train, y_val = train_test_split(\n",
    "    X, y, train_size = 0.80\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is just to correct the shape of the arrays as required by the two_layer_model\n",
    "X_train = X_train.T\n",
    "X_val = X_val.T\n",
    "y_train = y_train.reshape(1,-1)\n",
    "y_val = y_val.reshape(1,-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### CONSTANTS DEFINING THE MODEL ####\n",
    "n_x = X_train.shape[0]    \n",
    "n_h = 200\n",
    "n_y = 1\n",
    "layers_dims = (n_x, n_h, n_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the same two layer model that you completed in the previous section for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters = two_layer_model(X_train, y_train, layers_dims = (n_x, n_h, n_y), num_iterations = 2500, print_loss=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predict the review for our movies!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions_train = predict(X_train, y_train, parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions_val = predict(X_val, y_val, parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Results Analysis\n",
    "\n",
    "Let's take a look at some examples the 2-layer model labeled incorrectly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_mislabeled_reviews(X, y, p):\n",
    "    \"\"\"\n",
    "    Plots images where predictions and truth were different.\n",
    "    X -- dataset\n",
    "    y -- true labels\n",
    "    p -- predictions\n",
    "    \"\"\"\n",
    "    a = p + y\n",
    "    mislabeled_indices = np.asarray(np.where(a == 1))\n",
    "    plt.rcParams['figure.figsize'] = (40.0, 40.0) # set default size of plots\n",
    "    num_reviews = len(mislabeled_indices[0])\n",
    "    for i in range(num_reviews):\n",
    "        index = mislabeled_indices[1][i]\n",
    "        \n",
    "        print((\" \").join(cv.inverse_transform(X[index])[0]))\n",
    "        print(\"Prediction: \" + str(int(p[0,index])) + \" \\n Class: \" + str(y[0,index]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_mislabeled_reviews(X_val.T, y_val, predictions_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**: Provide explanation as to why these examples were misclassified below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Type your answer here**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}