Very_Basic_Perceptron_NN_in_Python_v4.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Very Basic Perceptron NN in Python v4.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OBhVKp2zRTF6",
        "colab_type": "text"
      },
      "source": [
        "# Perceptron:\n",
        "\n",
        "# A Very Basic (A)NN (Artificial) Neural Network \n",
        "## from scratch in Python"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X-OcABJwRmO4",
        "colab_type": "text"
      },
      "source": [
        "### Basic Parts of This Perceptron: \n",
        "\n",
        "1. The input data (X) \n",
        "2. The weights (in memory as `weights`)\n",
        "3. The initial random state of weights\n",
        "4. The activation function (sigmoid) and its derivative\n",
        "5. The set number of iterations\n",
        "6. The calculated weighted sum of inputs\n",
        "7. The Back-propagation process (going back to check results and weights) \n",
        "8. The calculation of error\n",
        "9. The calculation of adjustments to weights\n",
        "10. The adjustment of the weights\n",
        "11. The output\n",
        "\n",
        "Note: this perceptron would not be able to model a non-linear function (such as an Xor gate)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dUYCBqg2xrTA",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "## These are the changes that will be made to the weights:\n",
        "adjustments = error * sigmoid_derivative(activated_output)\n",
        "\n",
        "\n",
        "## This is the probability that an answer will be zero or one:\n",
        "activated_output = sigmoid(weighted_sum)\n",
        "\n",
        "error = corrected_output = activated_output\n",
        "\n",
        "weighted_sum = np.dot(inputs, weights)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8enuBHby5tHu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Notes:\n",
        "# Data-Set(X) & Target(y)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-RtELeKNK14l",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# import libraries\n",
        "import numpy as np\n",
        "import pandas as pd"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fqO21O40ZV6w",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# set a random seed\n",
        "np.random.seed(42)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZRYlrtxm5tKP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Data as 2 arrays\n",
        "\n",
        "# X\n",
        "# numpy array\n",
        "# analogy: person data\n",
        "inputs = np.array([\n",
        "                   [0,0,1],\n",
        "                   [1,1,1],\n",
        "                   [1,0,1],\n",
        "                   [0,1,1]\n",
        "])\n",
        "\n",
        "# analogy: blood types\n",
        "# Y\n",
        "# list of corrected outputs\n",
        "correct_outputs = [[0],\n",
        "                   [1],\n",
        "                   [1],\n",
        "                   [0]]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XF7X95sRVVnP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# data as one mask-array for a dataframe\n",
        "data = np.array([\n",
        "                   [0,0,0,1],\n",
        "                   [1,1,1,1],\n",
        "                   [1,1,0,1],\n",
        "                   [0,0,1,1]\n",
        "])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aiKsNBG0WDQZ",
        "colab_type": "code",
        "outputId": "9f9b07de-3fd5-4405-916a-f1793156a010",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "inputs, correct_outputs"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([[0, 0, 1],\n",
              "        [1, 1, 1],\n",
              "        [1, 0, 1],\n",
              "        [0, 1, 1]]), [[0], [1], [1], [0]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lgGlTPqYU_nw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df = pd.DataFrame(data, columns = ['correct_outputs', 'input1' ,'input2','input3']) "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "crinoeevWI1E",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "X = df.iloc[:,1:].values"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kBzCh2eKWKmH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "y = df.iloc[0,:].values"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IEObTXaxXHTs",
        "colab_type": "code",
        "outputId": "ba7dd0aa-38c9-489d-daf9-f0523f2dabf5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "X, y"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([[0, 0, 1],\n",
              "        [1, 1, 1],\n",
              "        [1, 0, 1],\n",
              "        [0, 1, 1]]), array([0, 0, 0, 1]))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "EEr9JbT3555_",
        "colab": {}
      },
      "source": [
        "# back propagation: adjusting weights to minimize error\n",
        "# gradient descent: use the derivative to find the minimum of a function ()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gSbq-WFmyig-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# set the initial random starting weights\n",
        "weights = 2 * np.random.random((3, 1)) -1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bmuUuh0ndC0q",
        "colab_type": "code",
        "outputId": "bef2f59c-bf03-418b-830c-49b537e25188",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "weights"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[-0.25091976],\n",
              "       [ 0.90142861],\n",
              "       [ 0.46398788]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rTYVLSVnyT-p",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# this is the activiation function to be used in the NN\n",
        "# creating a sigmoid function\n",
        "def sigmoid(x):\n",
        "  return 1 / (1 + np.exp(-x))\n",
        "\n",
        "  # caused error:\n",
        "  #return float(1 / (1 + np.exp(-x)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1PERka4p1kif",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def sigmoid_derivative(x):# inspecting/testing\n",
        "  sx = sigmoid(x)\n",
        "  return sx * (1 - sx)  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a4XoNiOJ5tVJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Steps done before: \n",
        "# 1. Randomly initialize weights, in memory as `weights`\n",
        "# 2. Get input data (X) \n",
        "# 3. Get correct_outputs (y) target\n",
        "# 4. Set activation function (sigmoid) and its derivative\n",
        "\n",
        "# Steps done here:\n",
        "# (Forward propagation: through all layers from input to output)\n",
        "# 5. set number of iterations\n",
        "# 6. calculate weighted sum of inputs \n",
        "# 7. Activate: sigmoid activation function\n",
        "# (Back propagation: going backwards through the weights/wires)\n",
        "# 8. Calculating the error\n",
        "# 9. Calculate adjustents\n",
        "# 10. apply adjustments to weights\n",
        "\n",
        "# Each set of wires has its own weights...\n",
        "\n",
        "# 5 Update our weights X times e.g. 10,000\n",
        "for iteration in range(10):\n",
        "    \n",
        "    # 6 Calculating the weighted sum of inputs\n",
        "    weighted_sum = np.dot(inputs, weights)\n",
        "    \n",
        "    # 7 Activate: sigmoid activation function\n",
        "    activated_output = sigmoid(weighted_sum)\n",
        "    \n",
        "    # 8 Calculating the error\n",
        "    error = correct_outputs - activated_output\n",
        "    \n",
        "    # 9 these are the changes that will later be made to the weights\n",
        "    adjustments = error * sigmoid_derivative(activated_output)\n",
        "    \n",
        "    # 10 Update the Weights: here the weights are changed (happens in the 'wire')\n",
        "    weights += np.dot(inputs.T, adjustments)\n",
        "  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J2gslwP_LsDb",
        "colab_type": "code",
        "outputId": "04b87829-04de-4649-ad0b-b6c26f9a03a7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        }
      },
      "source": [
        "print(\"Weights after training\")\n",
        "print(weights)\n",
        "\n",
        "print(\"Output after training\")\n",
        "print(activated_output)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Weights after training\n",
            "[[ 1.29240031]\n",
            " [ 0.16483768]\n",
            " [-0.50464319]]\n",
            "Output after training\n",
            "[[0.39136975]\n",
            " [0.7154103 ]\n",
            " [0.67140113]\n",
            " [0.44169656]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R1XbAYVp5tX5",
        "colab_type": "code",
        "outputId": "7e7f43b5-a623-409b-a26f-7f3a69f8a9e4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "correct_outputs"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[0], [1], [1], [0]]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    }
  ]
}