ML_ANN_BackOrders_Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem Statement\n",
    "\n",
    "    Identify products at risk of backorder before the event occurs so that business has time to react."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is a Backorder?\n",
    "Backorders are products that are temporarily out of stock, but a customer is permitted to place an order against future inventory. A backorder generally indicates that customer demand for a product or service exceeds a company’s capacity to supply it. Back orders are both good and bad. Strong demand can drive back orders, but so can suboptimal planning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data\n",
    "\n",
    "Data file contains the historical data for the 8 weeks prior to the week we are trying to predict. The data was taken as weekly snapshots at the start of each week. Columns are defined as follows:\n",
    "\n",
    "    sku - Random ID for the product\n",
    "\n",
    "    national_inv - Current inventory level for the part\n",
    "\n",
    "    lead_time - Transit time for product (if available)\n",
    "\n",
    "    in_transit_qty - Amount of product in transit from source\n",
    "\n",
    "    forecast_3_month - Forecast sales for the next 3 months\n",
    "\n",
    "    forecast_6_month - Forecast sales for the next 6 months\n",
    "\n",
    "    forecast_9_month - Forecast sales for the next 9 months\n",
    "\n",
    "    sales_1_month - Sales quantity for the prior 1 month time period\n",
    "\n",
    "    sales_3_month - Sales quantity for the prior 3 month time period\n",
    "\n",
    "    sales_6_month - Sales quantity for the prior 6 month time period\n",
    "\n",
    "    sales_9_month - Sales quantity for the prior 9 month time period\n",
    "\n",
    "    min_bank - Minimum recommend amount to stock\n",
    "\n",
    "    potential_issue - Source issue for part identified\n",
    "\n",
    "    pieces_past_due - Parts overdue from source\n",
    "\n",
    "    perf_6_month_avg - Source performance for prior 6 month period\n",
    "\n",
    "    perf_12_month_avg - Source performance for prior 12 month period\n",
    "\n",
    "    local_bo_qty - Amount of stock orders overdue\n",
    "\n",
    "    deck_risk - Part risk flag\n",
    "\n",
    "    oe_constraint - Part risk flag\n",
    "\n",
    "    ppap_risk - Part risk flag\n",
    "\n",
    "    stop_auto_buy - Part risk flag\n",
    "\n",
    "    rev_stop - Part risk flag\n",
    "\n",
    "    went_on_backorder - Product actually went on backorder. This is the target value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Pre-processing\n",
    "#### Loading the required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n",
      "/anaconda3/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix, roc_curve, auc\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.utils import to_categorical\n",
    "from keras import optimizers\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline  \n",
    "import os\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loading the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/seemabasantani/Documents/Academics/Batch 56/CSE7321c/ANN/Lab/Batch56'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#For setting working directory, if required\n",
    "#os.chdir('path to file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"BackOrders.csv\",header=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Understand the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the number row and columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(61589, 23)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['sku', 'national_inv', 'lead_time', 'in_transit_qty',\n",
       "       'forecast_3_month', 'forecast_6_month', 'forecast_9_month',\n",
       "       'sales_1_month', 'sales_3_month', 'sales_6_month', 'sales_9_month',\n",
       "       'min_bank', 'potential_issue', 'pieces_past_due', 'perf_6_month_avg',\n",
       "       'perf_12_month_avg', 'local_bo_qty', 'deck_risk', 'oe_constraint',\n",
       "       'ppap_risk', 'stop_auto_buy', 'rev_stop', 'went_on_backorder'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=61589, step=1)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the top rows of the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sku</th>\n",
       "      <th>national_inv</th>\n",
       "      <th>lead_time</th>\n",
       "      <th>in_transit_qty</th>\n",
       "      <th>forecast_3_month</th>\n",
       "      <th>forecast_6_month</th>\n",
       "      <th>forecast_9_month</th>\n",
       "      <th>sales_1_month</th>\n",
       "      <th>sales_3_month</th>\n",
       "      <th>sales_6_month</th>\n",
       "      <th>...</th>\n",
       "      <th>pieces_past_due</th>\n",
       "      <th>perf_6_month_avg</th>\n",
       "      <th>perf_12_month_avg</th>\n",
       "      <th>local_bo_qty</th>\n",
       "      <th>deck_risk</th>\n",
       "      <th>oe_constraint</th>\n",
       "      <th>ppap_risk</th>\n",
       "      <th>stop_auto_buy</th>\n",
       "      <th>rev_stop</th>\n",
       "      <th>went_on_backorder</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1888279</td>\n",
       "      <td>117</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>-99.00</td>\n",
       "      <td>-99.00</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1870557</td>\n",
       "      <td>7</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.28</td>\n",
       "      <td>0</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1475481</td>\n",
       "      <td>258</td>\n",
       "      <td>15.0</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>77</td>\n",
       "      <td>184</td>\n",
       "      <td>46</td>\n",
       "      <td>132</td>\n",
       "      <td>256</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0.54</td>\n",
       "      <td>0.70</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       sku  national_inv  lead_time  in_transit_qty  forecast_3_month  \\\n",
       "0  1888279           117        NaN               0                 0   \n",
       "1  1870557             7        2.0               0                 0   \n",
       "2  1475481           258       15.0              10                10   \n",
       "\n",
       "   forecast_6_month  forecast_9_month  sales_1_month  sales_3_month  \\\n",
       "0                 0                 0              0              0   \n",
       "1                 0                 0              0              0   \n",
       "2                77               184             46            132   \n",
       "\n",
       "   sales_6_month        ...         pieces_past_due  perf_6_month_avg  \\\n",
       "0             15        ...                       0            -99.00   \n",
       "1              0        ...                       0              0.50   \n",
       "2            256        ...                       0              0.54   \n",
       "\n",
       "  perf_12_month_avg  local_bo_qty  deck_risk  oe_constraint  ppap_risk  \\\n",
       "0            -99.00             0         No             No        Yes   \n",
       "1              0.28             0        Yes             No         No   \n",
       "2              0.70             0         No             No         No   \n",
       "\n",
       "  stop_auto_buy rev_stop went_on_backorder  \n",
       "0           Yes       No                No  \n",
       "1           Yes       No                No  \n",
       "2           Yes       No                No  \n",
       "\n",
       "[3 rows x 23 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shows a quick statistic summary of your data using describe.\n",
    "\n",
    "For object data (e.g. strings or timestamps), the result’s index will include count, unique, top, and freq. \n",
    "\n",
    "The top is the most common value.\n",
    "\n",
    "The freq is the most common value’s frequency.\n",
    "\n",
    "Timestamps also include the first and last items."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sku</th>\n",
       "      <th>national_inv</th>\n",
       "      <th>lead_time</th>\n",
       "      <th>in_transit_qty</th>\n",
       "      <th>forecast_3_month</th>\n",
       "      <th>forecast_6_month</th>\n",
       "      <th>forecast_9_month</th>\n",
       "      <th>sales_1_month</th>\n",
       "      <th>sales_3_month</th>\n",
       "      <th>sales_6_month</th>\n",
       "      <th>...</th>\n",
       "      <th>pieces_past_due</th>\n",
       "      <th>perf_6_month_avg</th>\n",
       "      <th>perf_12_month_avg</th>\n",
       "      <th>local_bo_qty</th>\n",
       "      <th>deck_risk</th>\n",
       "      <th>oe_constraint</th>\n",
       "      <th>ppap_risk</th>\n",
       "      <th>stop_auto_buy</th>\n",
       "      <th>rev_stop</th>\n",
       "      <th>went_on_backorder</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>6.158900e+04</td>\n",
       "      <td>61589.000000</td>\n",
       "      <td>58186.000000</td>\n",
       "      <td>61589.000000</td>\n",
       "      <td>6.158900e+04</td>\n",
       "      <td>6.158900e+04</td>\n",
       "      <td>6.158900e+04</td>\n",
       "      <td>61589.000000</td>\n",
       "      <td>61589.000000</td>\n",
       "      <td>6.158900e+04</td>\n",
       "      <td>...</td>\n",
       "      <td>61589.000000</td>\n",
       "      <td>61589.000000</td>\n",
       "      <td>61589.000000</td>\n",
       "      <td>61589.000000</td>\n",
       "      <td>61589</td>\n",
       "      <td>61589</td>\n",
       "      <td>61589</td>\n",
       "      <td>61589</td>\n",
       "      <td>61589</td>\n",
       "      <td>61589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>48145</td>\n",
       "      <td>61577</td>\n",
       "      <td>53792</td>\n",
       "      <td>59303</td>\n",
       "      <td>61569</td>\n",
       "      <td>50296</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>2.037188e+06</td>\n",
       "      <td>287.721882</td>\n",
       "      <td>7.559619</td>\n",
       "      <td>30.192843</td>\n",
       "      <td>1.692728e+02</td>\n",
       "      <td>3.150413e+02</td>\n",
       "      <td>4.535760e+02</td>\n",
       "      <td>44.742957</td>\n",
       "      <td>150.732631</td>\n",
       "      <td>2.835465e+02</td>\n",
       "      <td>...</td>\n",
       "      <td>1.605400</td>\n",
       "      <td>-6.264182</td>\n",
       "      <td>-5.863664</td>\n",
       "      <td>1.205361</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>6.564178e+05</td>\n",
       "      <td>4233.906931</td>\n",
       "      <td>6.498952</td>\n",
       "      <td>792.869253</td>\n",
       "      <td>5.286742e+03</td>\n",
       "      <td>9.774362e+03</td>\n",
       "      <td>1.420201e+04</td>\n",
       "      <td>1373.805831</td>\n",
       "      <td>5224.959649</td>\n",
       "      <td>8.872270e+03</td>\n",
       "      <td>...</td>\n",
       "      <td>42.309229</td>\n",
       "      <td>25.537906</td>\n",
       "      <td>24.844514</td>\n",
       "      <td>29.981155</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.068628e+06</td>\n",
       "      <td>-2999.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-99.000000</td>\n",
       "      <td>-99.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.498574e+06</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.620000</td>\n",
       "      <td>0.640000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.898033e+06</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>4.000000e+00</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.820000</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>2.314826e+06</td>\n",
       "      <td>57.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.200000e+01</td>\n",
       "      <td>2.500000e+01</td>\n",
       "      <td>3.600000e+01</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>17.000000</td>\n",
       "      <td>3.400000e+01</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.960000</td>\n",
       "      <td>0.950000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>3.284895e+06</td>\n",
       "      <td>673445.000000</td>\n",
       "      <td>52.000000</td>\n",
       "      <td>170976.000000</td>\n",
       "      <td>1.126656e+06</td>\n",
       "      <td>2.094336e+06</td>\n",
       "      <td>3.062016e+06</td>\n",
       "      <td>295197.000000</td>\n",
       "      <td>934593.000000</td>\n",
       "      <td>1.799099e+06</td>\n",
       "      <td>...</td>\n",
       "      <td>7392.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2999.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>11 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 sku   national_inv     lead_time  in_transit_qty  \\\n",
       "count   6.158900e+04   61589.000000  58186.000000    61589.000000   \n",
       "unique           NaN            NaN           NaN             NaN   \n",
       "top              NaN            NaN           NaN             NaN   \n",
       "freq             NaN            NaN           NaN             NaN   \n",
       "mean    2.037188e+06     287.721882      7.559619       30.192843   \n",
       "std     6.564178e+05    4233.906931      6.498952      792.869253   \n",
       "min     1.068628e+06   -2999.000000      0.000000        0.000000   \n",
       "25%     1.498574e+06       3.000000      4.000000        0.000000   \n",
       "50%     1.898033e+06      10.000000      8.000000        0.000000   \n",
       "75%     2.314826e+06      57.000000      8.000000        0.000000   \n",
       "max     3.284895e+06  673445.000000     52.000000   170976.000000   \n",
       "\n",
       "        forecast_3_month  forecast_6_month  forecast_9_month  sales_1_month  \\\n",
       "count       6.158900e+04      6.158900e+04      6.158900e+04   61589.000000   \n",
       "unique               NaN               NaN               NaN            NaN   \n",
       "top                  NaN               NaN               NaN            NaN   \n",
       "freq                 NaN               NaN               NaN            NaN   \n",
       "mean        1.692728e+02      3.150413e+02      4.535760e+02      44.742957   \n",
       "std         5.286742e+03      9.774362e+03      1.420201e+04    1373.805831   \n",
       "min         0.000000e+00      0.000000e+00      0.000000e+00       0.000000   \n",
       "25%         0.000000e+00      0.000000e+00      0.000000e+00       0.000000   \n",
       "50%         0.000000e+00      0.000000e+00      0.000000e+00       0.000000   \n",
       "75%         1.200000e+01      2.500000e+01      3.600000e+01       6.000000   \n",
       "max         1.126656e+06      2.094336e+06      3.062016e+06  295197.000000   \n",
       "\n",
       "        sales_3_month  sales_6_month        ...         pieces_past_due  \\\n",
       "count    61589.000000   6.158900e+04        ...            61589.000000   \n",
       "unique            NaN            NaN        ...                     NaN   \n",
       "top               NaN            NaN        ...                     NaN   \n",
       "freq              NaN            NaN        ...                     NaN   \n",
       "mean       150.732631   2.835465e+02        ...                1.605400   \n",
       "std       5224.959649   8.872270e+03        ...               42.309229   \n",
       "min          0.000000   0.000000e+00        ...                0.000000   \n",
       "25%          0.000000   0.000000e+00        ...                0.000000   \n",
       "50%          2.000000   4.000000e+00        ...                0.000000   \n",
       "75%         17.000000   3.400000e+01        ...                0.000000   \n",
       "max     934593.000000   1.799099e+06        ...             7392.000000   \n",
       "\n",
       "        perf_6_month_avg perf_12_month_avg  local_bo_qty  deck_risk  \\\n",
       "count       61589.000000      61589.000000  61589.000000      61589   \n",
       "unique               NaN               NaN           NaN          2   \n",
       "top                  NaN               NaN           NaN         No   \n",
       "freq                 NaN               NaN           NaN      48145   \n",
       "mean           -6.264182         -5.863664      1.205361        NaN   \n",
       "std            25.537906         24.844514     29.981155        NaN   \n",
       "min           -99.000000        -99.000000      0.000000        NaN   \n",
       "25%             0.620000          0.640000      0.000000        NaN   \n",
       "50%             0.820000          0.800000      0.000000        NaN   \n",
       "75%             0.960000          0.950000      0.000000        NaN   \n",
       "max             1.000000          1.000000   2999.000000        NaN   \n",
       "\n",
       "        oe_constraint  ppap_risk stop_auto_buy rev_stop went_on_backorder  \n",
       "count           61589      61589         61589    61589             61589  \n",
       "unique              2          2             2        2                 2  \n",
       "top                No         No           Yes       No                No  \n",
       "freq            61577      53792         59303    61569             50296  \n",
       "mean              NaN        NaN           NaN      NaN               NaN  \n",
       "std               NaN        NaN           NaN      NaN               NaN  \n",
       "min               NaN        NaN           NaN      NaN               NaN  \n",
       "25%               NaN        NaN           NaN      NaN               NaN  \n",
       "50%               NaN        NaN           NaN      NaN               NaN  \n",
       "75%               NaN        NaN           NaN      NaN               NaN  \n",
       "max               NaN        NaN           NaN      NaN               NaN  \n",
       "\n",
       "[11 rows x 23 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.describe(include='all')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display data type of each variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sku                    int64\n",
       "national_inv           int64\n",
       "lead_time            float64\n",
       "in_transit_qty         int64\n",
       "forecast_3_month       int64\n",
       "forecast_6_month       int64\n",
       "forecast_9_month       int64\n",
       "sales_1_month          int64\n",
       "sales_3_month          int64\n",
       "sales_6_month          int64\n",
       "sales_9_month          int64\n",
       "min_bank               int64\n",
       "potential_issue       object\n",
       "pieces_past_due        int64\n",
       "perf_6_month_avg     float64\n",
       "perf_12_month_avg    float64\n",
       "local_bo_qty           int64\n",
       "deck_risk             object\n",
       "oe_constraint         object\n",
       "ppap_risk             object\n",
       "stop_auto_buy         object\n",
       "rev_stop              object\n",
       "went_on_backorder     object\n",
       "dtype: object"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Observations\n",
    "\n",
    "sku is Categorical but is interpreted as int64 \n",
    "potential_issue, deck_risk, oe_constraint, ppap_risk, stop_auto_buy, rev_stop, and went_on_backorder are also \n",
    "categorical but is interpreted as object. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert all the attributes to appropriate type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data type conversion\n",
    "\n",
    "    Using astype('category') to convert potential_issue, deck_risk, oe_constraint, ppap_risk, stop_auto_buy, rev_stop, and went_on_backorder attributes to categorical attributes.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "for col in ['sku', 'potential_issue', 'deck_risk', 'oe_constraint', 'ppap_risk', 'stop_auto_buy', 'rev_stop', 'went_on_backorder']:\n",
    "    data[col] = data[col].astype('category')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display data type of each variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sku                  category\n",
       "national_inv            int64\n",
       "lead_time             float64\n",
       "in_transit_qty          int64\n",
       "forecast_3_month        int64\n",
       "forecast_6_month        int64\n",
       "forecast_9_month        int64\n",
       "sales_1_month           int64\n",
       "sales_3_month           int64\n",
       "sales_6_month           int64\n",
       "sales_9_month           int64\n",
       "min_bank                int64\n",
       "potential_issue      category\n",
       "pieces_past_due         int64\n",
       "perf_6_month_avg      float64\n",
       "perf_12_month_avg     float64\n",
       "local_bo_qty            int64\n",
       "deck_risk            category\n",
       "oe_constraint        category\n",
       "ppap_risk            category\n",
       "stop_auto_buy        category\n",
       "rev_stop             category\n",
       "went_on_backorder    category\n",
       "dtype: object"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Delete sku attribute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "61589"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.size(np.unique(data.sku))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.drop('sku', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Missing Data\n",
    "\n",
    "Missing value analysis and dropping the records with missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "national_inv            0\n",
       "lead_time            3403\n",
       "in_transit_qty          0\n",
       "forecast_3_month        0\n",
       "forecast_6_month        0\n",
       "forecast_9_month        0\n",
       "sales_1_month           0\n",
       "sales_3_month           0\n",
       "sales_6_month           0\n",
       "sales_9_month           0\n",
       "min_bank                0\n",
       "potential_issue         0\n",
       "pieces_past_due         0\n",
       "perf_6_month_avg        0\n",
       "perf_12_month_avg       0\n",
       "local_bo_qty            0\n",
       "deck_risk               0\n",
       "oe_constraint           0\n",
       "ppap_risk               0\n",
       "stop_auto_buy           0\n",
       "rev_stop                0\n",
       "went_on_backorder       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observing the number of records before and after missing value records removal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(61589, 22)\n"
     ]
    }
   ],
   "source": [
    "print (data.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Since the number of missing values is about 5%. For initial analysis we ignore all these records\n",
    "data = data.dropna(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "national_inv         0\n",
      "lead_time            0\n",
      "in_transit_qty       0\n",
      "forecast_3_month     0\n",
      "forecast_6_month     0\n",
      "forecast_9_month     0\n",
      "sales_1_month        0\n",
      "sales_3_month        0\n",
      "sales_6_month        0\n",
      "sales_9_month        0\n",
      "min_bank             0\n",
      "potential_issue      0\n",
      "pieces_past_due      0\n",
      "perf_6_month_avg     0\n",
      "perf_12_month_avg    0\n",
      "local_bo_qty         0\n",
      "deck_risk            0\n",
      "oe_constraint        0\n",
      "ppap_risk            0\n",
      "stop_auto_buy        0\n",
      "rev_stop             0\n",
      "went_on_backorder    0\n",
      "dtype: int64\n",
      "----------------------------------\n",
      "(58186, 22)\n"
     ]
    }
   ],
   "source": [
    "print(data.isnull().sum())\n",
    "print(\"----------------------------------\")\n",
    "print(data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Converting Categorical to Numeric\n",
    "\n",
    "For some of the models all the independent attribute should be of type numeric and ANN model is one among them.\n",
    "But this data set has some categorial attributes.\n",
    "\n",
    "'pandas.get_dummies' To convert convert categorical variable into dummy/indicator variables\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['national_inv', 'lead_time', 'in_transit_qty', 'forecast_3_month',\n",
      "       'forecast_6_month', 'forecast_9_month', 'sales_1_month',\n",
      "       'sales_3_month', 'sales_6_month', 'sales_9_month', 'min_bank',\n",
      "       'potential_issue', 'pieces_past_due', 'perf_6_month_avg',\n",
      "       'perf_12_month_avg', 'local_bo_qty', 'deck_risk', 'oe_constraint',\n",
      "       'ppap_risk', 'stop_auto_buy', 'rev_stop', 'went_on_backorder'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "print (data.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Creating dummy variables.\n",
    "\n",
    "If we have k levels in a category, then we create k-1 dummy variables as the last one would be redundant. So we use the parameter drop_first in pd.get_dummies function that drops the first level in each of the category\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_Attributes = data.select_dtypes(include=['category']).columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.get_dummies(columns=categorical_Attributes, data=data, prefix=categorical_Attributes, prefix_sep=\"_\",drop_first=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['national_inv', 'lead_time', 'in_transit_qty', 'forecast_3_month',\n",
      "       'forecast_6_month', 'forecast_9_month', 'sales_1_month',\n",
      "       'sales_3_month', 'sales_6_month', 'sales_9_month', 'min_bank',\n",
      "       'pieces_past_due', 'perf_6_month_avg', 'perf_12_month_avg',\n",
      "       'local_bo_qty', 'potential_issue_Yes', 'deck_risk_Yes',\n",
      "       'oe_constraint_Yes', 'ppap_risk_Yes', 'stop_auto_buy_Yes',\n",
      "       'rev_stop_Yes', 'went_on_backorder_Yes'],\n",
      "      dtype='object') (58186, 22)\n"
     ]
    }
   ],
   "source": [
    "print (data.columns, data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Target attribute distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    47217\n",
       "1    10969\n",
       "Name: went_on_backorder_Yes, dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.value_counts(data['went_on_backorder_Yes'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train-Test Split\n",
    "\n",
    "Using sklearn.model_selection.train_test_split\n",
    "\n",
    "    Split arrays or matrices into train and test subsets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Performing train test split on the data\n",
    "X, y = data.loc[:,data.columns!='went_on_backorder_Yes'].values, data.loc[:,'went_on_backorder_Yes'].values\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=123, stratify = data['went_on_backorder_Yes'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    33052\n",
      "1     7678\n",
      "dtype: int64\n",
      "0    14165\n",
      "1     3291\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "#To get the distribution in the target in train and test\n",
    "print(pd.value_counts(y_train))\n",
    "print(pd.value_counts(y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------------------------------------------------------------------------------------------------------------------\n",
    "-------------------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic Regression Modelling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Building a logistic regression model using sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier = LogisticRegression(random_state=123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=123, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predictions on train data\n",
    "train_pred = classifier.predict(X_train)\n",
    "# Predictions on test data\n",
    "test_pred = classifier.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix - Train Data: \n",
      " [[32927   125]\n",
      " [ 7094   584]]\n",
      "Confusion Matrix - Test Data: \n",
      " [[14118    47]\n",
      " [ 3036   255]]\n"
     ]
    }
   ],
   "source": [
    "# Train data\n",
    "confusion_matrix_train = confusion_matrix(y_train, train_pred)\n",
    "print(\"Confusion Matrix - Train Data: \\n\", confusion_matrix_train)\n",
    "# Test data\n",
    "confusion_matrix_test= confusion_matrix(y_test, test_pred)\n",
    "print(\"Confusion Matrix - Test Data: \\n\", confusion_matrix_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Error Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Specificity:  0.9962180806002663\n",
      "Train Recall:  0.07606147434227663\n",
      "Train Precision:  0.8236953455571228\n",
      "Train Accuracy:  0.8227596366314756\n"
     ]
    }
   ],
   "source": [
    "# Metrics on train data for logistic regression model\n",
    "#Accuracy\n",
    "accuracy_Train_logReg = (confusion_matrix_train[0,0]+confusion_matrix_train[1,1])/(confusion_matrix_train[0,0]+confusion_matrix_train[0,1]+confusion_matrix_train[1,0]+confusion_matrix_train[1,1])\n",
    "#specificity or true negative rate (TNR)\n",
    "specificity_Train_logReg = confusion_matrix_train[0,0]/(confusion_matrix_train[0,0]+confusion_matrix_train[0,1])\n",
    "#sensitivity, recall, hit rate, or true positive rate (TPR)\n",
    "recall_Train_logReg = confusion_matrix_train[1,1]/(confusion_matrix_train[1,0]+confusion_matrix_train[1,1])\n",
    "#precision\n",
    "precision_Train_logReg = confusion_matrix_train[1,1]/(confusion_matrix_train[0,1]+confusion_matrix_train[1,1])\n",
    "\n",
    "print(\"Train Specificity: \",specificity_Train_logReg)\n",
    "print(\"Train Recall: \",recall_Train_logReg)\n",
    "print(\"Train Precision: \",precision_Train_logReg)\n",
    "print(\"Train Accuracy: \",accuracy_Train_logReg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Specificity:  0.9966819625838333\n",
      "Test Recall:  0.07748404740200547\n",
      "Test Precision:  0.8443708609271523\n",
      "Test Accuracy:  0.823384509624198\n"
     ]
    }
   ],
   "source": [
    "# Metrics on test data\n",
    "#Accuracy\n",
    "accuracy_Test_logReg = (confusion_matrix_test[0,0]+confusion_matrix_test[1,1])/(confusion_matrix_test[0,0]+confusion_matrix_test[0,1]+confusion_matrix_test[1,0]+confusion_matrix_test[1,1])\n",
    "#specificity or true negative rate (TNR)\n",
    "specificity_Test_logReg = confusion_matrix_test[0,0]/(confusion_matrix_test[0,0]+confusion_matrix_test[0,1])\n",
    "#sensitivity, recall, hit rate, or true positive rate (TPR)\n",
    "recall_Test_logReg = confusion_matrix_test[1,1]/(confusion_matrix_test[1,0]+confusion_matrix_test[1,1])\n",
    "#precision\n",
    "precision_Test_logReg = confusion_matrix_test[1,1]/(confusion_matrix_test[0,1]+confusion_matrix_test[1,1])\n",
    "\n",
    "print(\"Test Specificity: \",specificity_Test_logReg)\n",
    "print(\"Test Recall: \",recall_Test_logReg)\n",
    "print(\"Test Precision: \",precision_Test_logReg)\n",
    "print(\"Test Accuracy: \",accuracy_Test_logReg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------------------------------------------------------------------------------------------------------------------\n",
    "-------------------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/fc_dense_layers_keras.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### A. The core data structure of Keras is a model, a way to organize layers. The simplest type of model is the Sequential model, a linear stack of layers. \n",
    "\n",
    "* The keras sequential api enables us to build common yet complex neural network architectures flexibly\n",
    "\n",
    "* Objects of the Keras sequential class, can have multiple neural network layers stacked on top of one another\n",
    "\n",
    "![](img/keras_sequential_api.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "\n",
    "model = Sequential()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Dense\n",
    "\n",
    "model.add(Dense(units=64, input_dim=21, activation='sigmoid'))\n",
    "model.add(Dense(units=1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 64)                1408      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 65        \n",
      "=================================================================\n",
      "Total params: 1,473\n",
      "Trainable params: 1,473\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### B. Compilation\n",
    "Before training a model, you need to configure the learning process, which is done via the compile method. receives three arguments\n",
    "\n",
    "* optimizer - An optimizer. An optimizer is an algorithm that uses this feedback signal, to actually update the weights so that the output from the network gets closer to the ground truth.\n",
    "* loss - A loss function. This is the objective function that the model will try to minimize.\n",
    "* metrics - A list of error metrics. This is for users reference and does not add value to the weights calculations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss = 'binary_crossentropy', optimizer = 'sgd', metrics = ['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### C. Training\n",
    "Keras models are trained on Numpy arrays of input data and labels. For training a model, you will typically use the  fit function\n",
    "\n",
    "* epoch = one forward pass and one backward pass of all the training examples\n",
    "* batch size = the number of training examples in one forward/backward pass. The higher the batch size, the more memory space you'll need.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "40730/40730 [==============================] - 1s 27us/step - loss: 0.3396 - acc: 0.8408\n",
      "Epoch 2/100\n",
      "40730/40730 [==============================] - 1s 23us/step - loss: 0.2805 - acc: 0.8788: 0s - loss: 0.2804 - acc:\n",
      "Epoch 3/100\n",
      "40730/40730 [==============================] - 1s 20us/step - loss: 0.2676 - acc: 0.8871\n",
      "Epoch 4/100\n",
      "40730/40730 [==============================] - 1s 24us/step - loss: 0.2577 - acc: 0.8902\n",
      "Epoch 5/100\n",
      "40730/40730 [==============================] - 1s 21us/step - loss: 0.2552 - acc: 0.8922\n",
      "Epoch 6/100\n",
      "40730/40730 [==============================] - 1s 24us/step - loss: 0.2537 - acc: 0.8938\n",
      "Epoch 7/100\n",
      "40730/40730 [==============================] - 1s 21us/step - loss: 0.2536 - acc: 0.8931\n",
      "Epoch 8/100\n",
      "40730/40730 [==============================] - 1s 22us/step - loss: 0.2532 - acc: 0.8939\n",
      "Epoch 9/100\n",
      "40730/40730 [==============================] - 1s 23us/step - loss: 0.2517 - acc: 0.8940\n",
      "Epoch 10/100\n",
      "40730/40730 [==============================] - 1s 24us/step - loss: 0.2518 - acc: 0.8946\n",
      "Epoch 11/100\n",
      "40730/40730 [==============================] - 1s 29us/step - loss: 0.2514 - acc: 0.8949\n",
      "Epoch 12/100\n",
      "40730/40730 [==============================] - 1s 26us/step - loss: 0.2501 - acc: 0.8940\n",
      "Epoch 13/100\n",
      "40730/40730 [==============================] - 1s 22us/step - loss: 0.2481 - acc: 0.8957\n",
      "Epoch 14/100\n",
      "40730/40730 [==============================] - 1s 19us/step - loss: 0.2499 - acc: 0.8944\n",
      "Epoch 15/100\n",
      "40730/40730 [==============================] - 1s 22us/step - loss: 0.2489 - acc: 0.8953\n",
      "Epoch 16/100\n",
      "40730/40730 [==============================] - 1s 19us/step - loss: 0.2490 - acc: 0.8951\n",
      "Epoch 17/100\n",
      "40730/40730 [==============================] - 1s 21us/step - loss: 0.2506 - acc: 0.8948\n",
      "Epoch 18/100\n",
      "40730/40730 [==============================] - 1s 20us/step - loss: 0.2460 - acc: 0.8970\n",
      "Epoch 19/100\n",
      "40730/40730 [==============================] - 1s 20us/step - loss: 0.2470 - acc: 0.8960\n",
      "Epoch 20/100\n",
      "40730/40730 [==============================] - 1s 18us/step - loss: 0.2469 - acc: 0.8971\n",
      "Epoch 21/100\n",
      "40730/40730 [==============================] - 1s 20us/step - loss: 0.2451 - acc: 0.8971\n",
      "Epoch 22/100\n",
      "40730/40730 [==============================] - 1s 18us/step - loss: 0.2463 - acc: 0.8968\n",
      "Epoch 23/100\n",
      "40730/40730 [==============================] - 1s 20us/step - loss: 0.2488 - acc: 0.8957\n",
      "Epoch 24/100\n",
      "40730/40730 [==============================] - 1s 18us/step - loss: 0.2484 - acc: 0.8962\n",
      "Epoch 25/100\n",
      "40730/40730 [==============================] - 1s 21us/step - loss: 0.2471 - acc: 0.8972: 0s - loss: 0.2569 - a\n",
      "Epoch 26/100\n",
      "40730/40730 [==============================] - 1s 17us/step - loss: 0.2477 - acc: 0.8961\n",
      "Epoch 27/100\n",
      "40730/40730 [==============================] - 1s 22us/step - loss: 0.2490 - acc: 0.8958\n",
      "Epoch 28/100\n",
      "40730/40730 [==============================] - 1s 18us/step - loss: 0.2475 - acc: 0.8974\n",
      "Epoch 29/100\n",
      "40730/40730 [==============================] - 1s 23us/step - loss: 0.2483 - acc: 0.8975: 0s - loss: 0.2501 - acc:\n",
      "Epoch 30/100\n",
      "40730/40730 [==============================] - 1s 18us/step - loss: 0.2476 - acc: 0.8971\n",
      "Epoch 31/100\n",
      "40730/40730 [==============================] - 1s 21us/step - loss: 0.2444 - acc: 0.8984\n",
      "Epoch 32/100\n",
      "40730/40730 [==============================] - 1s 18us/step - loss: 0.2426 - acc: 0.8993\n",
      "Epoch 33/100\n",
      "40730/40730 [==============================] - 1s 20us/step - loss: 0.2433 - acc: 0.8986\n",
      "Epoch 34/100\n",
      "40730/40730 [==============================] - 1s 18us/step - loss: 0.2416 - acc: 0.9000\n",
      "Epoch 35/100\n",
      "40730/40730 [==============================] - 1s 21us/step - loss: 0.2419 - acc: 0.8989\n",
      "Epoch 36/100\n",
      "40730/40730 [==============================] - 1s 18us/step - loss: 0.2475 - acc: 0.8974\n",
      "Epoch 37/100\n",
      "40730/40730 [==============================] - 1s 21us/step - loss: 0.2474 - acc: 0.8971\n",
      "Epoch 38/100\n",
      "40730/40730 [==============================] - 1s 18us/step - loss: 0.2407 - acc: 0.9006\n",
      "Epoch 39/100\n",
      "40730/40730 [==============================] - 1s 21us/step - loss: 0.2419 - acc: 0.8995\n",
      "Epoch 40/100\n",
      "40730/40730 [==============================] - 1s 19us/step - loss: 0.2408 - acc: 0.9006\n",
      "Epoch 41/100\n",
      "40730/40730 [==============================] - 1s 20us/step - loss: 0.2474 - acc: 0.8976\n",
      "Epoch 42/100\n",
      "40730/40730 [==============================] - 1s 18us/step - loss: 0.2468 - acc: 0.8975\n",
      "Epoch 43/100\n",
      "40730/40730 [==============================] - 1s 20us/step - loss: 0.2456 - acc: 0.8991\n",
      "Epoch 44/100\n",
      "40730/40730 [==============================] - 1s 18us/step - loss: 0.2442 - acc: 0.8987\n",
      "Epoch 45/100\n",
      "40730/40730 [==============================] - 1s 20us/step - loss: 0.2440 - acc: 0.9001\n",
      "Epoch 46/100\n",
      "40730/40730 [==============================] - 1s 19us/step - loss: 0.2444 - acc: 0.8989\n",
      "Epoch 47/100\n",
      "40730/40730 [==============================] - 1s 19us/step - loss: 0.2448 - acc: 0.8981\n",
      "Epoch 48/100\n",
      "40730/40730 [==============================] - 1s 21us/step - loss: 0.2500 - acc: 0.8959: 0s - loss: 0.2438 - ac\n",
      "Epoch 49/100\n",
      "40730/40730 [==============================] - 1s 19us/step - loss: 0.2467 - acc: 0.8988\n",
      "Epoch 50/100\n",
      "40730/40730 [==============================] - 1s 20us/step - loss: 0.2459 - acc: 0.8986\n",
      "Epoch 51/100\n",
      "40730/40730 [==============================] - 1s 19us/step - loss: 0.2435 - acc: 0.9013\n",
      "Epoch 52/100\n",
      "40730/40730 [==============================] - 1s 19us/step - loss: 0.2561 - acc: 0.8934\n",
      "Epoch 53/100\n",
      "40730/40730 [==============================] - 1s 20us/step - loss: 0.2531 - acc: 0.8957\n",
      "Epoch 54/100\n",
      "40730/40730 [==============================] - 1s 20us/step - loss: 0.2498 - acc: 0.8967\n",
      "Epoch 55/100\n",
      "40730/40730 [==============================] - 1s 19us/step - loss: 0.2427 - acc: 0.8998\n",
      "Epoch 56/100\n",
      "40730/40730 [==============================] - 1s 25us/step - loss: 0.2498 - acc: 0.8960\n",
      "Epoch 57/100\n",
      "40730/40730 [==============================] - 2s 37us/step - loss: 0.2460 - acc: 0.8992\n",
      "Epoch 58/100\n",
      "40730/40730 [==============================] - 1s 22us/step - loss: 0.2417 - acc: 0.9011\n",
      "Epoch 59/100\n",
      "40730/40730 [==============================] - 1s 26us/step - loss: 0.2413 - acc: 0.9007\n",
      "Epoch 60/100\n",
      "40730/40730 [==============================] - 1s 29us/step - loss: 0.2428 - acc: 0.9001\n",
      "Epoch 61/100\n",
      "40730/40730 [==============================] - 1s 20us/step - loss: 0.2403 - acc: 0.9011\n",
      "Epoch 62/100\n",
      "40730/40730 [==============================] - 1s 19us/step - loss: 0.2461 - acc: 0.8993\n",
      "Epoch 63/100\n",
      "40730/40730 [==============================] - 1s 22us/step - loss: 0.2518 - acc: 0.8967\n",
      "Epoch 64/100\n",
      "40730/40730 [==============================] - 1s 24us/step - loss: 0.2496 - acc: 0.8961\n",
      "Epoch 65/100\n",
      "40730/40730 [==============================] - 1s 25us/step - loss: 0.2463 - acc: 0.8967\n",
      "Epoch 66/100\n",
      "40730/40730 [==============================] - 1s 26us/step - loss: 0.2436 - acc: 0.8988\n",
      "Epoch 67/100\n",
      "40730/40730 [==============================] - 1s 24us/step - loss: 0.2428 - acc: 0.8998\n",
      "Epoch 68/100\n",
      "40730/40730 [==============================] - 1s 28us/step - loss: 0.2427 - acc: 0.9005\n",
      "Epoch 69/100\n",
      "40730/40730 [==============================] - 1s 26us/step - loss: 0.2441 - acc: 0.8989\n",
      "Epoch 70/100\n",
      "40730/40730 [==============================] - 1s 25us/step - loss: 0.2510 - acc: 0.8952\n",
      "Epoch 71/100\n",
      "40730/40730 [==============================] - 1s 21us/step - loss: 0.2456 - acc: 0.8982\n",
      "Epoch 72/100\n",
      "40730/40730 [==============================] - 1s 26us/step - loss: 0.2484 - acc: 0.8972\n",
      "Epoch 73/100\n",
      "40730/40730 [==============================] - 1s 24us/step - loss: 0.2467 - acc: 0.8982\n",
      "Epoch 74/100\n",
      "40730/40730 [==============================] - 1s 28us/step - loss: 0.2440 - acc: 0.9002\n",
      "Epoch 75/100\n",
      "40730/40730 [==============================] - 1s 27us/step - loss: 0.2435 - acc: 0.9009: 0s - loss: 0.2440\n",
      "Epoch 76/100\n",
      "40730/40730 [==============================] - 1s 24us/step - loss: 0.2426 - acc: 0.9018\n",
      "Epoch 77/100\n",
      "40730/40730 [==============================] - 1s 27us/step - loss: 0.2468 - acc: 0.8977\n",
      "Epoch 78/100\n",
      "40730/40730 [==============================] - 1s 34us/step - loss: 0.2517 - acc: 0.8946\n",
      "Epoch 79/100\n",
      "40730/40730 [==============================] - 1s 28us/step - loss: 0.2534 - acc: 0.8968\n",
      "Epoch 80/100\n",
      "40730/40730 [==============================] - 1s 32us/step - loss: 0.2478 - acc: 0.8981\n",
      "Epoch 81/100\n",
      "40730/40730 [==============================] - 1s 31us/step - loss: 0.2436 - acc: 0.8994\n",
      "Epoch 82/100\n",
      "40730/40730 [==============================] - 1s 31us/step - loss: 0.2481 - acc: 0.8966\n",
      "Epoch 83/100\n",
      "40730/40730 [==============================] - 1s 23us/step - loss: 0.2528 - acc: 0.8961\n",
      "Epoch 84/100\n",
      "40730/40730 [==============================] - 1s 18us/step - loss: 0.2459 - acc: 0.8998\n",
      "Epoch 85/100\n",
      "40730/40730 [==============================] - 1s 21us/step - loss: 0.2439 - acc: 0.8999\n",
      "Epoch 86/100\n",
      "40730/40730 [==============================] - 1s 18us/step - loss: 0.2505 - acc: 0.8951\n",
      "Epoch 87/100\n",
      "40730/40730 [==============================] - 1s 21us/step - loss: 0.2427 - acc: 0.9010\n",
      "Epoch 88/100\n",
      "40730/40730 [==============================] - 1s 20us/step - loss: 0.2450 - acc: 0.8987\n",
      "Epoch 89/100\n",
      "40730/40730 [==============================] - 1s 25us/step - loss: 0.2426 - acc: 0.9012\n",
      "Epoch 90/100\n",
      "40730/40730 [==============================] - 1s 26us/step - loss: 0.2436 - acc: 0.9006\n",
      "Epoch 91/100\n",
      "40730/40730 [==============================] - 1s 24us/step - loss: 0.2553 - acc: 0.8973\n",
      "Epoch 92/100\n",
      "40730/40730 [==============================] - 1s 21us/step - loss: 0.2518 - acc: 0.8961\n",
      "Epoch 93/100\n",
      "40730/40730 [==============================] - 1s 27us/step - loss: 0.2543 - acc: 0.8949\n",
      "Epoch 94/100\n",
      "40730/40730 [==============================] - 1s 26us/step - loss: 0.2584 - acc: 0.8914\n",
      "Epoch 95/100\n",
      "40730/40730 [==============================] - 1s 36us/step - loss: 0.2538 - acc: 0.8935\n",
      "Epoch 96/100\n",
      "40730/40730 [==============================] - 1s 29us/step - loss: 0.2628 - acc: 0.8910\n",
      "Epoch 97/100\n",
      "40730/40730 [==============================] - 1s 35us/step - loss: 0.2576 - acc: 0.8924\n",
      "Epoch 98/100\n",
      "40730/40730 [==============================] - 1s 32us/step - loss: 0.2570 - acc: 0.8925\n",
      "Epoch 99/100\n",
      "40730/40730 [==============================] - 1s 35us/step - loss: 0.2542 - acc: 0.8951\n",
      "Epoch 100/100\n",
      "40730/40730 [==============================] - 1s 25us/step - loss: 0.2518 - acc: 0.8967\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1a2259be48>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train, epochs=100, batch_size=64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ANN Model Building"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A. Base model (Perceptron)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "perceptron_model = Sequential()\n",
    "\n",
    "perceptron_model.add(Dense(1, input_dim=21, activation='sigmoid', kernel_initializer='normal'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "perceptron_model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "40730/40730 [==============================] - 2s 37us/step - loss: 0.5324 - acc: 0.8216\n",
      "Epoch 2/30\n",
      "40730/40730 [==============================] - 1s 28us/step - loss: 0.4214 - acc: 0.8419\n",
      "Epoch 3/30\n",
      "40730/40730 [==============================] - 1s 27us/step - loss: 0.4069 - acc: 0.8465\n",
      "Epoch 4/30\n",
      "40730/40730 [==============================] - 1s 32us/step - loss: 0.4198 - acc: 0.8493\n",
      "Epoch 5/30\n",
      "40730/40730 [==============================] - 1s 29us/step - loss: 0.3987 - acc: 0.8567\n",
      "Epoch 6/30\n",
      "40730/40730 [==============================] - 1s 36us/step - loss: 0.4166 - acc: 0.8564\n",
      "Epoch 7/30\n",
      "40730/40730 [==============================] - 1s 28us/step - loss: 0.4067 - acc: 0.8577\n",
      "Epoch 8/30\n",
      "40730/40730 [==============================] - 1s 32us/step - loss: 0.4425 - acc: 0.8597\n",
      "Epoch 9/30\n",
      "40730/40730 [==============================] - 1s 33us/step - loss: 0.3931 - acc: 0.8645\n",
      "Epoch 10/30\n",
      "40730/40730 [==============================] - 1s 23us/step - loss: 0.3992 - acc: 0.8611\n",
      "Epoch 11/30\n",
      "40730/40730 [==============================] - 1s 29us/step - loss: 0.3942 - acc: 0.8609\n",
      "Epoch 12/30\n",
      "40730/40730 [==============================] - 1s 22us/step - loss: 0.4198 - acc: 0.8591\n",
      "Epoch 13/30\n",
      "40730/40730 [==============================] - 1s 23us/step - loss: 0.4085 - acc: 0.8614\n",
      "Epoch 14/30\n",
      "40730/40730 [==============================] - 1s 20us/step - loss: 0.4019 - acc: 0.8662\n",
      "Epoch 15/30\n",
      "40730/40730 [==============================] - 1s 24us/step - loss: 0.4039 - acc: 0.8661\n",
      "Epoch 16/30\n",
      "40730/40730 [==============================] - 1s 22us/step - loss: 0.4137 - acc: 0.8647\n",
      "Epoch 17/30\n",
      "40730/40730 [==============================] - 1s 24us/step - loss: 0.3967 - acc: 0.8667\n",
      "Epoch 18/30\n",
      "40730/40730 [==============================] - 1s 25us/step - loss: 0.4062 - acc: 0.8654\n",
      "Epoch 19/30\n",
      "40730/40730 [==============================] - 1s 25us/step - loss: 0.3950 - acc: 0.8658\n",
      "Epoch 20/30\n",
      "40730/40730 [==============================] - 1s 26us/step - loss: 0.4272 - acc: 0.8631\n",
      "Epoch 21/30\n",
      "40730/40730 [==============================] - 1s 22us/step - loss: 0.4090 - acc: 0.8644: 0s - loss: 0.4010 - acc: 0.8\n",
      "Epoch 22/30\n",
      "40730/40730 [==============================] - 1s 22us/step - loss: 0.4062 - acc: 0.8663\n",
      "Epoch 23/30\n",
      "40730/40730 [==============================] - 1s 21us/step - loss: 0.4057 - acc: 0.8662\n",
      "Epoch 24/30\n",
      "40730/40730 [==============================] - 1s 23us/step - loss: 0.4012 - acc: 0.8653\n",
      "Epoch 25/30\n",
      "40730/40730 [==============================] - 1s 20us/step - loss: 0.3925 - acc: 0.8660\n",
      "Epoch 26/30\n",
      "40730/40730 [==============================] - 1s 23us/step - loss: 0.4226 - acc: 0.8640\n",
      "Epoch 27/30\n",
      "40730/40730 [==============================] - 1s 20us/step - loss: 0.4007 - acc: 0.8663\n",
      "Epoch 28/30\n",
      "40730/40730 [==============================] - 1s 23us/step - loss: 0.4065 - acc: 0.8660\n",
      "Epoch 29/30\n",
      "40730/40730 [==============================] - 1s 32us/step - loss: 0.4134 - acc: 0.8663\n",
      "Epoch 30/30\n",
      "40730/40730 [==============================] - 1s 24us/step - loss: 0.4053 - acc: 0.8669\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x107e01ef0>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "perceptron_model.fit(X_train, y_train, epochs=30, batch_size=64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[31994  1058]\n",
      " [ 4108  3570]]\n",
      "[[13665   500]\n",
      " [ 1759  1532]]\n"
     ]
    }
   ],
   "source": [
    "test_pred=perceptron_model.predict_classes(X_test)\n",
    "train_pred=perceptron_model.predict_classes(X_train)\n",
    "\n",
    "confusion_matrix_test = confusion_matrix(y_test, test_pred)\n",
    "confusion_matrix_train = confusion_matrix(y_train, train_pred)\n",
    "\n",
    "print(confusion_matrix_train)\n",
    "print(confusion_matrix_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Train Test Accuracy, True Negative Rate and True Positive Rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train TNR:  0.9679898342006535\n",
      "Train TPR:  0.4649648345923418\n",
      "Train Accuracy:  0.8731647434323594\n",
      "-----------------------\n",
      "Test TNR:  0.9647017296152488\n",
      "Test TPR:  0.4655120024308721\n",
      "Test Accuracy:  0.8705889092575618\n"
     ]
    }
   ],
   "source": [
    "Accuracy_Train=(confusion_matrix_train[0,0]+confusion_matrix_train[1,1])/(confusion_matrix_train[0,0]+confusion_matrix_train[0,1]+confusion_matrix_train[1,0]+confusion_matrix_train[1,1])\n",
    "TNR_Train= confusion_matrix_train[0,0]/(confusion_matrix_train[0,0]+confusion_matrix_train[0,1])\n",
    "TPR_Train= confusion_matrix_train[1,1]/(confusion_matrix_train[1,0]+confusion_matrix_train[1,1])\n",
    "\n",
    "print(\"Train TNR: \",TNR_Train)\n",
    "print(\"Train TPR: \",TPR_Train)\n",
    "print(\"Train Accuracy: \",Accuracy_Train)\n",
    "print(\"-----------------------\")\n",
    "\n",
    "Accuracy_Test=(confusion_matrix_test[0,0]+confusion_matrix_test[1,1])/(confusion_matrix_test[0,0]+confusion_matrix_test[0,1]+confusion_matrix_test[1,0]+confusion_matrix_test[1,1])\n",
    "TNR_Test= confusion_matrix_test[0,0]/(confusion_matrix_test[0,0] +confusion_matrix_test[0,1])\n",
    "TPR_Test= confusion_matrix_test[1,1]/(confusion_matrix_test[1,0] +confusion_matrix_test[1,1])\n",
    "\n",
    "print(\"Test TNR: \",TNR_Test)\n",
    "print(\"Test TPR: \",TPR_Test)\n",
    "print(\"Test Accuracy: \",Accuracy_Test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Adding a hidden layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlp_model = Sequential()\n",
    "\n",
    "mlp_model.add(Dense(12, input_dim=21, activation='sigmoid', kernel_initializer='normal'))\n",
    "mlp_model.add(Dense(1, activation='sigmoid', kernel_initializer='normal'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_4 (Dense)              (None, 12)                264       \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 1)                 13        \n",
      "=================================================================\n",
      "Total params: 277\n",
      "Trainable params: 277\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "mlp_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlp_model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 32584 samples, validate on 8146 samples\n",
      "Epoch 1/30\n",
      "32584/32584 [==============================] - 1s 39us/step - loss: 0.4122 - acc: 0.8271 - val_loss: 0.3148 - val_acc: 0.8512\n",
      "Epoch 2/30\n",
      "32584/32584 [==============================] - 1s 26us/step - loss: 0.2840 - acc: 0.8737 - val_loss: 0.2757 - val_acc: 0.8808\n",
      "Epoch 3/30\n",
      "32584/32584 [==============================] - 1s 28us/step - loss: 0.2622 - acc: 0.8926 - val_loss: 0.2603 - val_acc: 0.8954\n",
      "Epoch 4/30\n",
      "32584/32584 [==============================] - 1s 35us/step - loss: 0.2503 - acc: 0.8980 - val_loss: 0.2534 - val_acc: 0.8986\n",
      "Epoch 5/30\n",
      "32584/32584 [==============================] - 1s 31us/step - loss: 0.2463 - acc: 0.8982 - val_loss: 0.2467 - val_acc: 0.8984\n",
      "Epoch 6/30\n",
      "32584/32584 [==============================] - 1s 38us/step - loss: 0.2419 - acc: 0.9008 - val_loss: 0.2464 - val_acc: 0.8987\n",
      "Epoch 7/30\n",
      "32584/32584 [==============================] - 1s 34us/step - loss: 0.2413 - acc: 0.9008 - val_loss: 0.2421 - val_acc: 0.9013\n",
      "Epoch 8/30\n",
      "32584/32584 [==============================] - 1s 30us/step - loss: 0.2388 - acc: 0.9026 - val_loss: 0.2414 - val_acc: 0.9004\n",
      "Epoch 9/30\n",
      "32584/32584 [==============================] - 1s 25us/step - loss: 0.2372 - acc: 0.9027 - val_loss: 0.2399 - val_acc: 0.9031\n",
      "Epoch 10/30\n",
      "32584/32584 [==============================] - 1s 28us/step - loss: 0.2362 - acc: 0.9028 - val_loss: 0.2395 - val_acc: 0.9017\n",
      "Epoch 11/30\n",
      "32584/32584 [==============================] - 1s 25us/step - loss: 0.2360 - acc: 0.9022 - val_loss: 0.2403 - val_acc: 0.8985\n",
      "Epoch 12/30\n",
      "32584/32584 [==============================] - 1s 28us/step - loss: 0.2364 - acc: 0.9009 - val_loss: 0.2382 - val_acc: 0.9007\n",
      "Epoch 13/30\n",
      "32584/32584 [==============================] - 1s 24us/step - loss: 0.2357 - acc: 0.9019 - val_loss: 0.2408 - val_acc: 0.8980\n",
      "Epoch 14/30\n",
      "32584/32584 [==============================] - 1s 27us/step - loss: 0.2372 - acc: 0.9009 - val_loss: 0.2414 - val_acc: 0.8986\n",
      "Epoch 15/30\n",
      "32584/32584 [==============================] - 1s 25us/step - loss: 0.2348 - acc: 0.9022 - val_loss: 0.2367 - val_acc: 0.9000\n",
      "Epoch 16/30\n",
      "32584/32584 [==============================] - 1s 28us/step - loss: 0.2350 - acc: 0.9013 - val_loss: 0.2373 - val_acc: 0.9003\n",
      "Epoch 17/30\n",
      "32584/32584 [==============================] - 1s 25us/step - loss: 0.2341 - acc: 0.9030 - val_loss: 0.2382 - val_acc: 0.8986\n",
      "Epoch 18/30\n",
      "32584/32584 [==============================] - 1s 28us/step - loss: 0.2339 - acc: 0.9027 - val_loss: 0.2388 - val_acc: 0.8998\n",
      "Epoch 19/30\n",
      "32584/32584 [==============================] - 1s 34us/step - loss: 0.2339 - acc: 0.9030 - val_loss: 0.2365 - val_acc: 0.9030\n",
      "Epoch 20/30\n",
      "32584/32584 [==============================] - 1s 36us/step - loss: 0.2323 - acc: 0.9032 - val_loss: 0.2351 - val_acc: 0.9019\n",
      "Epoch 21/30\n",
      "32584/32584 [==============================] - 1s 39us/step - loss: 0.2321 - acc: 0.9034 - val_loss: 0.2347 - val_acc: 0.9025\n",
      "Epoch 22/30\n",
      "32584/32584 [==============================] - 1s 32us/step - loss: 0.2324 - acc: 0.9024 - val_loss: 0.2360 - val_acc: 0.9030\n",
      "Epoch 23/30\n",
      "32584/32584 [==============================] - 1s 37us/step - loss: 0.2321 - acc: 0.9033 - val_loss: 0.2362 - val_acc: 0.9017\n",
      "Epoch 24/30\n",
      "32584/32584 [==============================] - 1s 37us/step - loss: 0.2313 - acc: 0.9031 - val_loss: 0.2351 - val_acc: 0.9023\n",
      "Epoch 25/30\n",
      "32584/32584 [==============================] - 1s 28us/step - loss: 0.2354 - acc: 0.9029 - val_loss: 0.2422 - val_acc: 0.9001\n",
      "Epoch 26/30\n",
      "32584/32584 [==============================] - 1s 28us/step - loss: 0.2360 - acc: 0.9027 - val_loss: 0.2385 - val_acc: 0.9020\n",
      "Epoch 27/30\n",
      "32584/32584 [==============================] - 1s 38us/step - loss: 0.2339 - acc: 0.9035 - val_loss: 0.2355 - val_acc: 0.9011\n",
      "Epoch 28/30\n",
      "32584/32584 [==============================] - 1s 39us/step - loss: 0.2315 - acc: 0.9042 - val_loss: 0.2355 - val_acc: 0.9022\n",
      "Epoch 29/30\n",
      "32584/32584 [==============================] - 1s 37us/step - loss: 0.2300 - acc: 0.9047 - val_loss: 0.2352 - val_acc: 0.9027\n",
      "Epoch 30/30\n",
      "32584/32584 [==============================] - 1s 27us/step - loss: 0.2297 - acc: 0.9048 - val_loss: 0.2344 - val_acc: 0.9036\n"
     ]
    }
   ],
   "source": [
    "model_history = mlp_model.fit(X_train, y_train, epochs=30, batch_size=64, validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_pred = mlp_model.predict_classes(X_train)\n",
    "\n",
    "test_pred = mlp_model.predict_classes(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Getting evaluation metrics and evaluating model performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[31335  1717]\n",
      " [ 2140  5538]]\n",
      "[[13359   806]\n",
      " [  948  2343]]\n"
     ]
    }
   ],
   "source": [
    "confusion_matrix_train = confusion_matrix(y_train, train_pred)\n",
    "confusion_matrix_test = confusion_matrix(y_test, test_pred)\n",
    "\n",
    "print(confusion_matrix_train)\n",
    "print(confusion_matrix_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate Accuracy, True Positive Rate and True Negative Rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Specificity:  0.9480515551252572\n",
      "Train Recall:  0.7212815837457671\n",
      "Train Precision:  0.7633356305995865\n",
      "Train Accuracy:  0.9053032163024798\n"
     ]
    }
   ],
   "source": [
    "# Metrics on train data for ann_model 1\n",
    "#Accuracy\n",
    "accuracy_Train_M1 = (confusion_matrix_train[0,0]+confusion_matrix_train[1,1])/(confusion_matrix_train[0,0]+confusion_matrix_train[0,1]+confusion_matrix_train[1,0]+confusion_matrix_train[1,1])\n",
    "#specificity or true negative rate (TNR)\n",
    "specificity_Train_M1 = confusion_matrix_train[0,0]/(confusion_matrix_train[0,0]+confusion_matrix_train[0,1])\n",
    "#sensitivity, recall, hit rate, or true positive rate (TPR)\n",
    "recall_Train_M1 = confusion_matrix_train[1,1]/(confusion_matrix_train[1,0]+confusion_matrix_train[1,1])\n",
    "#precision\n",
    "precision_Train_M1 = confusion_matrix_train[1,1]/(confusion_matrix_train[0,1]+confusion_matrix_train[1,1])\n",
    "\n",
    "print(\"Train Specificity: \",specificity_Train_M1)\n",
    "print(\"Train Recall: \",recall_Train_M1)\n",
    "print(\"Train Precision: \",precision_Train_M1)\n",
    "print(\"Train Accuracy: \",accuracy_Train_M1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Specificity:  0.9430991881397811\n",
      "Test Recall:  0.7119416590701915\n",
      "Test Precision:  0.7440457288027945\n",
      "Test Accuracy:  0.8995187901008249\n"
     ]
    }
   ],
   "source": [
    "# Metrics on test data\n",
    "#Accuracy\n",
    "accuracy_Test_M1 = (confusion_matrix_test[0,0]+confusion_matrix_test[1,1])/(confusion_matrix_test[0,0]+confusion_matrix_test[0,1]+confusion_matrix_test[1,0]+confusion_matrix_test[1,1])\n",
    "#specificity or true negative rate (TNR)\n",
    "specificity_Test_M1 = confusion_matrix_test[0,0]/(confusion_matrix_test[0,0]+confusion_matrix_test[0,1])\n",
    "#sensitivity, recall, hit rate, or true positive rate (TPR)\n",
    "recall_Test_M1 = confusion_matrix_test[1,1]/(confusion_matrix_test[1,0]+confusion_matrix_test[1,1])\n",
    "#precision\n",
    "precision_Test_M1 = confusion_matrix_test[1,1]/(confusion_matrix_test[0,1]+confusion_matrix_test[1,1])\n",
    "\n",
    "print(\"Test Specificity: \",specificity_Test_M1)\n",
    "print(\"Test Recall: \",recall_Test_M1)\n",
    "print(\"Test Precision: \",precision_Test_M1)\n",
    "print(\"Test Accuracy: \",accuracy_Test_M1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['val_loss', 'val_acc', 'loss', 'acc'])\n"
     ]
    }
   ],
   "source": [
    "print(model_history.history.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8XHW5+PHPM5N9b5auaZp0gW6UtJSCUJBVaEUWRaGCWgRxQ1zwunIREL3qVcQF8YIWuAj0VhCp/lgsWHZom67QlNJ0S9N0yd7s2zy/P74nyTRNO9M202zP+/Wa18w5c+bM98wk55nv812OqCrGGGPMkfj6ugDGGGP6PwsWxhhjQrJgYYwxJiQLFsYYY0KyYGGMMSYkCxbGGGNCsmBhzCAiIjtE5KK+LocZfCxYmEFFRF4RkSoRie3rskSKiKiI1ItInYjsFpF7RcR/lPs4T0RKIlVGM/hYsDCDhojkAucAClx+gt876kS+H3CqqiYBFwKfBr5wgt/fDDEWLMxg8lngHeAR4HPBT4hIvIj8SkR2ikiNiLwhIvHec3NF5C0RqRaRXSKy0Fv/iojcFLSPhSLyRtCyishXRWQLsMVb9xtvHwdEZLWInBO0vV9EfiAiW0Wk1nt+rIjcLyK/6lbef4jIN0IdsKq+D7wOTO/+nIjEish9IlLq3e7z1iUCzwOjvdpJnYiMDvVeZmizYGEGk88Cj3u3S0RkRNBzvwROA84C0oHvAAERycGdOH8HZAH5wLqjeM8rgTOAqd7yKm8f6cATwF9FJM577lvAAmA+kAJ8HmgAHgUWiIgPQEQycTWGJ0O9uYhMxdWm1vbw9A+BM73ynArMAW5X1XpgHlCqqknerfQojtkMQRYszKAgInOBccASVV0NbMWlZ/BOwp8Hvq6qu1W1XVXfUtVm4DrgJVV9UlVbVbVCVY8mWPyXqlaqaiOAqv7F20ebqv4KiAVO9ra9CXey3qzOem/blUANLkAAXAu8oqr7jvC+a0SkCvgH8Cfg4R62uQ64W1X3q2oZcBfwmaM4NmM6WbAwg8XngH+parm3/ARdqahMIA4XQLobe5j14doVvCAit4nIJi/VVQ2keu8f6r0eBa73Hl8PPBbifWep6jBVnaCqt6tqoIdtRgM7g5Z3euuMOWonulHOmF7ntT18CvCLyF5vdSyQJiKnAu8CTcAEYH23l+/CpWd6Ug8kBC2P7GGbzmmbvfaJ7+JqCBtVNeD9+peg95oAvNfDfv4CvOeVdwrw98OU6WiU4mpbG73lHG/dQeU2JhxWszCDwZVAO67dIN+7TcE1/H7W+9W9CLhXREZ7Dc0f8rrXPg5cJCKfEpEoEckQkXxvv+uAj4tIgohMBG4MUY5koA0oA6JE5A5c20SHPwE/FpFJ4swQkQwAVS3BtXc8BjzdkdY6Tk8Ct4tIltcOcgcuKAHsAzJEJLUX3scMARYszGDwOeBhVS1W1b0dN+D3wHVet9Zv42oYq4BK4OeAT1WLcQ3Ot3nr1+EagwF+DbTgTqyP4gLLkbyIayz/AJfyaeLgNNW9wBLgX8AB4M9AfNDzjwKnEDoFFa57gAJgA+7Y13jrOnpRPQls83qBWXrKHJHYxY+M6R9E5FzcL//cw7RBGNNnrGZhTD8gItHA14E/WaAw/ZEFC2P6mIhMAaqBUcB9fVwcY3pkaShjjDEhWc3CGGNMSINmnEVmZqbm5ub2dTGMMWZAWb16dbmqZoXabtAEi9zcXAoKCvq6GMYYM6CIyM7QW1kayhhjTBgsWBhjjAnJgoUxxpiQLFgYY4wJyYKFMcaYkCxYGGOMCcmChTHGmJAGzTgLY4wZ6Jpa29lT08SemkbKaptpbVcCAaUtoLQHAt69uwU/PmlEMh+dMSqiZbNgYYwxJ4CqUlrTxM6KevbWNHUGhT3VXY+rGlqPad8fO3W0BQtjjBloWtoCbNlfS2HpATbtqaVwTw2b9tRS03hwMBiWEM3I1HhGp8YxMyeN0WnxjEyJY1RaHMOTY4nx+/H7hSif4Pd13fs7l334BETkMCXpPRYsjDHHrbU9QGt7gISYoXdKqapvYdOeAxR23EoPsLWsjtZ2N6N3XLSPySNT+OiMUUwZlcL4zERGpcYxKjWe+Bj/sb9xQyUUv+1u6eNh9ud76Yh6NvS+WRNZbc2w/XUYnQ+JmX1dGtMDVaW2qYWmVqWxtZ2m1oB3305jazvN3n1Ta4DGlnbqm9uobW6jtqmVA41tHGhqpbbJW/bum1rd9ZomZCVyxvgMzshL58zxGYxIiTu0AO2tIH7wHdq/prU9wKY9B1izs4rVxdVs2VdLQoyftIQYUuOjSY2PJsW7T42PJi0+mtQE9zgxNor4aD9x0T7iovz4fL37azsQUIorGyjcc8AFh1IXHPbUNHVuMzw5lqmjUzh/8nCmjkph6ugUcjMS8fdGWap3ucCw8y13X/a+W++PgVmfO/79h2DBwoQlEFBqm9toaQvQ3Nbu3QcOuo/du5opK79PUu1WAhLFnsyz2DpyPlszzqWJuM5fn63t2vk4OS6KCVlJTMhKYnxWIslx0X19qP3Gln21PL6imJff30dslL/zBHnYW0I07QGlsr6FivoWKutaqKxvdo+9W1VdI59tfoKbfP+PYh3DG4HpvBE4hVWBk2km5rBliYv2kRwXTUpcFMlx0STHRTEmLZ7kuChS4qNJjo1CgbXFVSxfV8SalXt5Vso5JekA+Sm1TIypZriWEVu3G2r3QmIWTLmM6tz5rNQprC6pZe3Oajbsru4MPCNT4pg6OoXmtnb21zaxZX8tNQ0uQIUjJspHfLS/K4BE+4nzlhNi/CTERpEQ7Sc+xk9irJ+EmCi3Pqbr8f7a5s6g8P6eA9S3tAPg9wkTshKZk5fO1FEpTPECQ2ZS7HF/74D70VW5zQsOXu2hxruce2wKjD0DZnwKcs6C0TMhuoeg3MsGzcWPZs+erTbrbO+raWxlyapdPPLWDnZXN/a4TRzN3Bb1V270P89ehvHfrddwsq+Ey/1vMloqqddYXgiczrPtZ/OWTsfnjybG7yPKL9Q2tdEe6PobHJ4c64LH8MTOIDJheBKjUuJ6/Zdif9TSFuCFjXt5/J2drNheSYzfx4dPziLKJ9Q0th50qw3jpJkSF0VGUizpiTHkxDXwlfKfMql+NTsyzyM+UEdW1Tp82ka7L4ba4bOpHzOXlnEfRkbOIC42hrhoHwkxUcREBdUCVKGhAiqKut22uRNa84GDj4koSgMZ7NZMqmJGEDMsm6zmYibXvk08zVRqEsv0dDYNO5+o8eeRn5fFrJxhjE6L7/GY2gNKbVPQZ1HfREvlLqpJpk7jutWSXO2oqa3duw/Q2NJGY2s7Dc3tNLS0U9/SRkNL+0F/h8GSY6M6g8GUUclMHZXKpBFJxEUfYwpJFZqqoabE1RZqSqCm+ODlur1d2yeNgJwPwbiz3P2IaeA7jvRVNyKyWlVnh9zOgoXpyfbyeh55czt/XV1CQ0s7Z+Slc9GUEcRF+4iN8hMb7SPG72NE5SqmrPoh8XXFVEy+nsqzfog/PoWYKB8xPojbs4K4TU8RvfkfSFMNJA6H6Z+AGZ+E0bNoaXdV+61lde62v77zcfDJMCk2irkTM7lo6ggumDyc9MTD/woOx96aJjbvq2XKqGSGJx/fr7KaxlaWv7+fF97by+riKiZmJTFrXBqzcoYxM2dYWGXdVdnAkyuLWVKwi/K6Fsamx3PdGeP45GnZZBzm12rHSbO6oevE6fcJ6YkxZCTGMCwxhmi/d5LftQr++jmoL4eP/hJmfdatb65zv1q3Lodtr8D+jW59/DDI+zCMPw/iUqBi68GBoammqyC+aEjPc3nztBxIHQup2d7jbAIJWby/r54V2ytYub2Sldsr8fmEM7Pj+VhSIafVv056yctISx3EpcLJH4WpV8CE8yHKO3ZVqC/rFpy8MlVug/YWV+Zzvg2n33TUv7RVlZZ2F1gaWtpprD9A7MYlJPpaSZv8YWTUqeA/xkRMWzPsWgnblsO2V6FsM7TUHryNP9b7zLzPLjUHho2D7NPd5xrBBmwLFuaoqSpvb61g0Zvbefn9/UT5hMtPHcMNZ+cyfUzqwRs3HYCXfgQFi2BYLlz+e8g75/A7b2uGLf+CDf8HH7zo/rkzJsKZX3ENc93+GVSVsrpmtu6vZ1t5He/tPsDy9/ez90ATPoHTxg3joikjuGjqCCZkJYU8tl2VDazYXsmKbRWs3FHJzoqGzudOGpHEWRMyOWtCBmeMzyA1PnQqrKy2mWWF+3hh417e3lpOa7syIiWWM/Iy2F5eT+GeA52/VMdnJjIzZ1hnADlpRDJ+n9AeUF7ZvJ+/vLOTVz4oQ4ALJo/g+jNzOHdSVu/UpFRh5UPw4g8gZTRc8xiMOvXw29fug+2vesFjOdTu6XouJRsyJrjvLXOSu8+Y4E5sx3oi7dDa5N6vcCls/n8uGMUku1/T9ftdYAiusfhj3Em0owzDcmHTP2Hry66c5/8ATr326H+BN1a5z+udB6Cxsmt9dCKMPd2lfcZ9CMbMhpiEnvehCvs2esHhFdfG0Nrg2mmyT3ftealjDw4MiZkRDQhHYsHChFa6Flb+idbkMbzdNI77N6ewYr+PjMQYrjtzHNefmdPzr+4tL8E/vg4HdruT/QU/hJjE8N+3sRo2LYW1f4FdK2DaVXD57yA2+YgvU1U2lh5gWeE+Xtq0j42l7uQxPjORi6e6wDErZxg+gR0VDazcXsGKbZWs2F7ZmUJLS4hmTm46c/LSmTwyhXd31/DW1nJW7aikqTWAT+CU7DTOmpDB2RMymZ07rDPdUFLVwIsb9/Hie3tZtbMSVRiXkcCl00ZyyfSR5GendZ7gG1ra2FBSw5riKtbsrGZtcRUV9S2AqyXNyE5lZ0UDu6sbyUqOZcHpY7l2Ts5hUy/HpLnOfU/vPQUnXQpX/dH9+g6XKpRvgfZmSJ9w+JNjb2trge2vQeHf3d9HyhgvKEzsCg5pOT0Hgu2vwbIfQekayJoCF94BJ88LfSKu3Qfv3A+r/gwtde7zmvstd0IPbjfYtxFQV5sand+VHsqY6Mq67RV3qy9z+8082dXOJpwP4852tbR+pl8ECxG5FPgN4Af+pKo/6/b8OGARkAVUAteraon33OeA271N71HVR4/0XhYsury8aR/rdlUjIvgEfN69iCACse0NzNn+B6aV/B+tvjiiA434cH8H9fGjics9HX/2aTB6lvuH6DiJN1TCiz+E9U+4f4Irfg9j5xx7QVXhzd/Ay3e5f7ZPPQbDJ4f98t3Vjfx70z6Wbdrf+es+I95HdJSfvbWuP3tmUgxz8tI5Iy+DM8anc9Lw5B5/sTe3tbO2uJq3tlbwVlE563ZV0xZQYqJ8nJYzjLrmNt7d7VIvk0cmc+n0kVwybSSTRyaH1cdd1aXbOoPHriqGJcSwYE4OF08d0ZUu6i1lH8CSz0D5B3D+D92Jr4feR4OSqvsx8vLdLk019ky46E5XI+iuage8+Vv3wyXQ6n64zP0mjDyl5303VruUUvFbLoCUrnG15A6Jw7uCQ96HIXVMrx9eb+vzYCEifuAD4GKgBFgFLFDVwqBt/gr8U1UfFZELgBtU9TMikg4UALMBBVYDp6lq1eHez4KFy8Pf8ex7/Ktw32G3+YhvFXdFP8oIqnis/SJ+2XYN55w0gq+cXMc0ipDSNbB7DVR3XGlRIOtkGJXvqtX15e6f6dz/6L0eGNtfh6dugJYGuPy3cMrVR72L2ro6ipfdT857f6BNolmXfydj51zJhKzEYxqwVNfcxqodlbxVVM7b2yqI8fu4ZJoLELmZR1GL6k7VpVjqy13qIT7t2Pd1OBufgWdvcfn+T/zZnbiGovY2WPsYvPIz12B80qVw4Y9gxFTY/z688Wt4968gPshfAGd/w9VajkZrE+xe7YJS9mwYPrXP0knHqj8Eiw8Bd6rqJd7y9wFU9b+CttkIXKKqJeL+o2tUNUVEFgDnqeoXve3+B3hFVZ883PsN5WARCChPrCzm58+/T0t7gG9dfBI3zs0jyu9DVQkoBKqL8b3wXfwfPE9g+DSaLr2XttGn4RchMbaHfHN9uUtT7V7j/hlK17oq+WW/PnLO+1gd2AN/XQi73oEzvgQX/xiiwmjEDgTcP/zye6C6GHLPcSmAsvfh1AVwyU8hIb33y9uTtmYoKXB5/rr9LtdeXwZ1Ze5xXZlbbm9220fFwfSrYc5Nrvvj8WpvhWV3wDt/cLnxTz46IH7ZRlxLA6z4I7xxn2v3GHMa7C6A6AQ4bSF86JYh/TmFGywiOc5iDLAraLkEOKPbNuuBT+BSVVcBySKScZjXHvJtisjNwM0AOTk5vVbwgaRofx3f/9sGVu2o4qwJGfz0qlMO+uUrgXb8K/8H/79/Aihc/GN8Z36ZBH+IRtzETJh0sbudCCmjYOE/Xb75nftdkPrkI4f/J1aFLctcCmvfey5tcP3TMOFClxZ47b/h9Xuh6GX46K9g6uWRKXdrIxS9BIXPwuYXDu7l4oty4wkSsyBpuMuhJ2a6xwmZLjBuWALr/uIaTE+/yaVBjqbG1t7mcuUfvADv/9P1DJrzRfjIPeEF26EgJgHO+ZYLDG/8GjY/72rGZ3zJBo4ehUjWLD6JqzXc5C1/Bpijql8L2mY08HsgD3gNFzim4QJArKre4233n0CDqv7qcO831GoWLW0B/vjqVn7/7yLiY/zc/tEpXH1a9sEpl91rXAPn3g0w6SMw/5euO15/15lGiYOr/+xywMF2rXI9sXa+6XrBXPCfMO3jh+bk96yHZ78Ke9+FqVfC/P92J+rj1VznenYVPusCVms9xKfDlMtct8/0PBcg4oeFTkk0VsP6xbDqT1CxBRIyYOZnXA+xw31XjVUuCH7wgnv/pmrX4Jp7Nsy+MXKB0QxKAyIN1W37JOB9Vc22NNSRrSmu4ntPb+CDfXVcNmMUP/rYNLKSvf7ogXaXAnnzPlj5oGtwm/dz1299IOVSgxtoL7gdzv6mO5m+fLf7BZ2YBed+x/1aPNIv6PZW14j+6s9dj615v4BTPnn0n0VTjevyW/isq0m0NbnPdspl7rMdN/f4uo+qui6rKx+Czc+55ZMugdO/ABMucDnxD15wt+J3QNtdYJl0idtuwgX9sqeN6f/6Q7CIwjVwXwjsxjVwf1pVNwZtkwlUqmpARH4CtKvqHV4D92pglrfpGlwDdyWHMSCDRWsTvHSnSyPEpbhh/HEpEJfW9bhzXSoNxPK3N95l3fsfkBvXwMcmRDEutj4oH77fjazVACAurXHhf7qBTgNRcNfPEdNhf6HLM591K3zoqxAbenxFp/3vw9JboGSVa+i87Ndu3EF3bS2uh0xFkQtOFUVQXuRy3O0tkDwKplzuAkTOmb06krZTTQmsfgRWP+q+05jkrvTWiOkuOJw0D8bMisz7myGlz4OFV4j5wH24rrOLVPUnInI3UKCqS0XkauC/cD2eXgO+qqrN3ms/D/zA29VPVPXhI73XgAsWVTtgyWdhz3qqhp8JbU34WmqJaq0lurWWmEDPU2scJCoekrLcL9yk4V258cThkHNGZBqiT7SOQWWv/BfMuAbO/fax55kD7a6h8+Ufgz8azvu+uw8enVy90wu2nsQs16139CwXILJPP3FdUNtaXBfQopch+zRXi0gbe2Le2wwZ/SJYnEgDKlh88CL87Quowve5hcU10w7ZJDFaGRHTQlZMC8Ojm8iIaiIrpo0LZ53M5IkTXEA4ml/WpkvlNlh6K+x43S1HJ3aNTO64ZU50A9Ei0bXVmH6kP/SGMt0F2t0v5Nf+Gx05gx9G/wdPbY3i/k/PZNKIJBJjo0iKiSIx1k9Ubw/SMl3Sx8Nnl7peVIlZkDxyYLXnGNMHLFicKPXl8PSNbiqAmZ/hkdSv8MQL27n9o5MjfjlE0wOfD0bN6OtSGDNgWLA4EYJn/Lz8dxSkX8Y9D77DpdNGcuPcvL4unTHGhGS5jkjqaJx9eJ4boHXTMspPuoZbnljL2GHx/OKTM07ItXONMeZ4Wc0iUlrqXbfPd//qerF8/H9oj03jG4tWUtXQwt++chYpdlU4Y8wAYcEiEppq4M+XQPlmN7rYm/HzN8s+4I2icn7+iVOYNnqAjn0wxgxJFiwiYdM/oWwTXPO4G+ELvPpBGb/79xauPi2bT822vvLGmIHF2iwioegld93cyR8FoLS6kW8sXsvJI5L58RXTrZ3CGDPgWLDobYF22PpvmHgRiNDSFuCWJ9bQ2q784bpZxMfY9AzGmIHH0lC9bfcaNwvoxIsA+Nnz77OmuJr7Pz2L8WFcK9oYY/ojq1n0tqJl7spb48/juXf3sOjN7Sw8K9cG3hljBjQLFr2t6CUYM5vtDbF856kN5I9N4wfzp/R1qYwx5rhYsOhN9RWwew2t4y/gy39ZTZRfuP+6WcRE2cdsjBnYrM2iN239N6Csi53N+3tr+cN1sxiTFt/XpTLGmONmP3l7U9FLkJDBa3XZ+H3CeSdn9XWJjDGmV1iw6C2BAGx9GSZcwNpdB5g8MpmEGKu4GWMGBwsWvWXveqgvIzDhItbtqmZmjl00xxgzeEQ0WIjIpSKyWUSKROR7PTyfIyLLRWStiGzwLsOKiMSIyMMi8q6IrBeR8yJZzl5R9BIA21PPoK65jZljh/VxgYwxpvdELFiIiB+4H5gHTAUWiMjUbpvdDixR1ZnAtcAfvPVfAFDVU4CLgV+JSP+uBRW9DKPyKSh3I7TzrWZhjBlEInkCngMUqeo2VW0BFgNXdNtGgRTvcSpQ6j2eCrwMoKr7gWog5DVi+0xjNexaCZMuZt2ualLjo8nLSOzrUhljTK+JZLAYA+wKWi7x1gW7E7heREqA54CveevXA1eISJSI5AGnAYdM1SoiN4tIgYgUlJWV9Xb5w7ftFdB2mHgRa4uryR+bhs9nkwUaYwaPSAaLns6W2m15AfCIqmYD84HHvHTTIlxwKQDuA94C2g7ZmeqDqjpbVWdnZfVhN9WilyA2lbqsfDbvq7XGbWPMoBPJvp0lHFwbyKYrzdThRuBSAFV9W0TigEwv9fTNjo1E5C1gSwTLeuxUXXvFhPPYsLsOVZiZY43bxpjBJZI1i1XAJBHJE5EYXAP20m7bFAMXAojIFCAOKBORBBFJ9NZfDLSpamEEy3rs9hdCbSlMvJi1u6oByM+2moUxZnCJWM1CVdtE5BbgRcAPLFLVjSJyN1CgqkuB24CHROSbuBTVQlVVERkOvCgiAWA38JlIlfO4eV1mmXgha5/ZzYSsRFIT7NraxpjBJaJDjFX1OVzDdfC6O4IeFwJn9/C6HcDJkSxbryl6CYZPQ5NHsbZ4I+dPHt7XJTLGmF7Xv8cu9HfNtbDzbZh0EbsqG6mobyF/rKWgjDGDjwWL47H9dQi0ui6zu6oArCeUMWZQsmBxPIpeguhEGHsma4uriY/2c/KI5L4ulTHG9DoLFsdK1V1CdfyHISqGtbuqmZGdSpTfPlJjzOBjZ7ZjVVEE1cUw8SKaWtspLK2x8RXGmEHLgsWxCuoyu7H0AK3tau0VxphBy4LFsdqyDDImwbBc1hZ7jdvWE8oYM0hZsDgWrY2w802YdDEAa3dVMyYtnuEpcX1cMGOMiQwLFsdix5vQ1gQTLwRgXXG1Xb/CGDOoWbA4FkUvQVQcjDub/Qea2F3daCkoY8ygZsHiWBQtg9xzIDq+c/JA6wlljBnMLFgcrcrtrtvsxIsAWFtcTbRfmDY6JcQLjTFm4LJgcbS2vuzuO4NFFVNHpxIX7e/DQhljTGRZsDhaW16CtHGQMYG29gAbSmqsvcIYM+hZsDgabc2w/TXXZVaEzftqaWxtt8F4xphBz4LF0Sh+B1rrD2qvAJg51hq3jTGDW0SDhYhcKiKbRaRIRL7Xw/M5IrJcRNaKyAYRme+tjxaRR0XkXRHZJCLfj2Q5w1a0DPwxricUsG5XNRmJMYxNj+/jghljTGRFLFiIiB+4H5gHTAUWiMjUbpvdDixR1Zm4a3T/wVv/SSBWVU8BTgO+KCK5kSpr2EpWw+hZEJsEuMbtmTlpiEgfF8wYYyIrkjWLOUCRqm5T1RZgMXBFt20U6OhzmgqUBq1PFJEoIB5oAQ5EsKzhqdwGmRMBqGloZWtZvY2vMMYMCZEMFmOAXUHLJd66YHcC14tICe5a3V/z1j8F1AN7gGLgl6pa2f0NRORmESkQkYKysrJeLn43LfVQtxeG5QGwrqSjvcIat40xg18kg0VPuRnttrwAeERVs4H5wGMi4sPVStqB0UAecJuIjD9kZ6oPqupsVZ2dlZXVu6XvrnK7u093xVhbXIUIzLBgYYwZAiIZLEqAsUHL2XSlmTrcCCwBUNW3gTggE/g08IKqtqrqfuBNYHYEyxpaVfdgUc3JI5JJio3qw0IZY8yJEclgsQqYJCJ5IhKDa8Be2m2bYuBCABGZggsWZd76C8RJBM4E3o9gWUOr3Obu0/MIBJR1u6rJt1qFMWaIiFiwUNU24BbgRWATrtfTRhG5W0Qu9za7DfiCiKwHngQWqqrielElAe/hgs7DqrohUmUNS+U2SMiAuFS2V9RT09hqg/GMMUNGRHMoqvocruE6eN0dQY8LgbN7eF0drvts/1G5vTMFta7YZpo1xgwtNoI7XEHBYu2uKpJjo5iYldTHhTLGmBPDgkU42pqhZldnt9m1xdWcOjYNn88G4xljhgYLFuGo2gkopI+noaWN9/fWWnuFMWZIsWARjqBus++W1NAeUAsWxpghxYJFOIK6zXZcRjXfZpo1xgwhFizCUbkNYlMgIYO1xVWMy0ggPTGmr0tljDEnjAWLcFRuh/Q8FNe4bfNBGWOGGgsW4ajcBsPy2FPTxP7aZhtfYYwZcixYhNLeBtU7IX1815XxrHHbGDPEWLAIpWYXBNq8YFFFbJSPySNTQr/OGGMGkZDBQkRuEZGhm3fp7Dabx+Z9tUwemUxMlMVYY8zQEs5ZbySwSkSWeNfUHlrDlju7zY6nrLaZ4SlxfVseY4zpAyGDhareDkwC/gwsBLaIyE9FZEKEy9Y/VG6HqHhIGkl5XTOZSbF9XSJjjDnhwsqneNOG7/VubcAw4CkR+UUEy9Y/eN1m2xEq61vITLIbUWgHAAAbkklEQVTxFcaYoSfkFOUicivwOaAc+BPwH6ra6l3+dAvwncgWsY9VboP08VQ1tBBQrGZhjBmSwrmeRSbwcVXdGbxSVQMicllkitVPBAKugXvihZTXNQMWLIwxQ1M4aajngMqOBRFJFpEzAFR1U6QK1i/U7oG2JkgfT0VdCwAZloYyxgxB4QSLB4C6oOV6b11IXu+pzSJSJCLf6+H5HBFZLiJrRWSDiMz31l8nIuuCbgERyQ/nPXtVULdZq1kYY4aycIKFeA3cgEs/EV5bhx93Le15wFRggYhM7bbZ7bhrc88ErgX+4L3H46qar6r5wGeAHaq6LpwD6lVB3WbLvZpFlgULY8wQFE6w2CYit4pItHf7OrAtjNfNAYpUdZuqtgCLgSu6baNAx3DoVKC0h/0sAJ4M4/16X+U28EVDSjbldc1E+4WU+IhettwYY/qlcILFl4CzgN1ACXAGcHMYrxsD7ApaLvHWBbsTuF5ESnBtI1/rYT/XcJhgISI3i0iBiBSUlZWFUaSjVLkd0nLAH0V5bTMZibEMtTGJxhgD4Q3K26+q16rqcFUdoaqfVtX9Yey7p7OqdlteADyiqtnAfOAxr0uu24FrSG9Q1fcOU7YHVXW2qs7OysoKo0hHyes2C1BR30JmsjVuG2OGpnDaHuKAG4FpQOdcF6r6+RAvLQHGBi1nc2ia6UbgUm9/b3vvlQl0BKNr6asUlKqrWeR8CIDyOlezMMaYoSicNNRjuPmhLgFexZ30a8N43SpgkojkiUgM7sS/tNs2xcCFACIyBReMyrxlH/BJXFvHiddQAS21kJ4HQHmtTfVhjBm6wgkWE1X1P4F6VX0U+ChwSqgXqWobcAvwIrAJ1+tpo4jcLSKXe5vdBnxBRNbjahALg3penQuUqGo4jem9L6gnlKpSbmkoY8wQFk7XnlbvvlpEpuPmh8oNZ+eq+hyu4Tp43R1BjwuBsw/z2leAM8N5n4gICha1zW20tAXItDSUMWaICidYPOhdz+J2XBopCfjPiJaqP6jcBuKDtBzKq7wBeVazMMYMUUcMFl67wQFVrQJeA8afkFL1B5XbISUbomKpqK8HbPS2MWboOmKbhTda+5YTVJb+pXLbQY3bgPWGMsYMWeE0cC8TkW+LyFgRSe+4RbxkfS1ojEXnvFCWhjLGDFHhtFl0jKf4atA6ZTCnpBqrobGyq2ZR14IIpCdYsDDGDE0hg4Wq5p2IgvQrnbPNdtUshiXEEOUP68KCxhgz6IQzgvuzPa1X1f/t/eL0E0HdZgHv2ttWqzDGDF3hpKFOD3ochxtxvQYY/MFiWC4AFXUt1hPKGDOkhZOGOmgmWBFJxU0BMnhV7oCkkRCTCLiaxSnZaX1bJmOM6UPHkoRvACb1dkH6laCeUOAauC0NZYwZysJps/gHXVOL+3BXvVsSyUL1ucptMPEiAJpa26lrbrM0lDFmSAunzeKXQY/bgJ2qWhKh8vS9lnqo2wvpuUDQGAurWRhjhrBwgkUxsEdVmwBEJF5EclV1R0RL1leqdrj7zp5Q7trbVrMwxgxl4bRZ/BUIBC23e+sGp27dZis6axYWLIwxQ1c4wSJKVVs6FrzHgzcn09lttmP0tjcvlKWhjDFDWDjBoizoYkWIyBVAeeSK1Mcqt0N8OsS7rrKWhjLGmPCCxZeAH4hIsYgUA98FvhjOzkXkUhHZLCJFIvK9Hp7PEZHlIrJWRDaIyPyg52aIyNsislFE3vWuzx15h3SbbSY5Noq4aP8JeXtjjOmPwhmUtxU4U0SSAFHVcK6/jYj4gfuBi4ESYJWILPWujtfhdtzlVh8Qkam4q+rlikgU8BfgM6q6XkQy6LpiX2RVboecMzoXy+taLAVljBnyQtYsROSnIpKmqnWqWisiw0TknjD2PQcoUtVtXjvHYuCKbtsokOI9TgVKvccfATao6noAVa1Q1fZwDui4tDXDgZKDaxa1zZaCMsYMeeGkoeapanXHgnfVvPlH2L7DGGBX0HKJty7YncD1IlKCq1V0TC1yEqAi8qKIrBGR7/T0BiJys4gUiEhBWVlZGEUKoboYNHBQsKiot2BhjDHhBAu/iHSeLUUkHgjn7Ck9rNNuywuAR1Q1GxeAHvMu5RoFzAWu8+6vEpELD9mZ6oOqOltVZ2dlZYVRpBC69YQCS0MZYwyENyjvL8DLIvKwt3wD8GgYrysBxgYtZ9OVZupwI3ApgKq+7TViZ3qvfVVVywFE5DlgFvByGO977LqNsWhrD1DVYDPOGmNMyJqFqv4CuAeYgpsX6gVgXBj7XgVMEpE8EYkBrgWWdtumGDflOSIyBTcFehnwIjBDRBK8xu4PA4VEWuV2iEmGxEy32NCCKmQmW7Awxgxt4dQsAPbiRnF/CtgOPB3qBaraJiK34E78fmCRqm4UkbuBAlVdCtwGPCQi38SlqBaqqgJVInIvLuAo8Jyq/r+jPLajV7nNXUpVXAatvNYbY5FoaShjzNB22GAhIifhagMLgArg/3BdZ88Pd+eq+hyu4Tp43R1BjwuBsw/z2r/gUmAnTuU2GDm9c7FzEkGrWRhjhrgjpaHex6WIPqaqc1X1d7h5oQan9jbXG6pbTyiw0dvGGHOkYPEJXPppuYg85PVG6qmH0+BwoAQCrd3GWLg0lPWGMsYMdYcNFqr6jKpeA0wGXgG+CYwQkQdE5CMnqHwnTo/dZpuJifKRHBtu044xxgxO4fSGqlfVx1X1Mlz313XAIfM8DXjdus2CG2ORlRSLyOCtUBljTDiO6hrcqlqpqv+jqhdEqkB9pnI7RMVB8qjOVeV1zZaCMsYYjjJYDGqV22FYLvi6PpLyOpvqwxhjwIJFl25TkwNU1LWQYWMsjDHGggUAgYC79nZQsFBVN4mgjbEwxhgLFgDU7YW2Rjd621PT2Epru1oayhhjsGDhHGa2WYBMa+A2xhgLFoBr3IZDLqcKNnrbGGPAgoVTuQ18UZDaNaN6RWfNwoKFMcZYsAAXLNJywN81UrujZmHjLIwxxoKF00O32fK6ZnwCwxIsWBhjjAUL1UO6zYJr4E5PjMXvs6k+jDHGgkVDBTQfOKgnFHSM3rZahTHGQISDhYhcKiKbRaRIRA6ZfFBEckRkuYisFZENIjLfW58rIo0iss67/TFihexhAkGwqT6MMSZYxObeFhE/cD9wMVACrBKRpd7V8TrcDixR1QdEZCruqnq53nNbVTU/UuXrNGIa3LgMMicdtLqiroVxOQkRf3tjjBkIIlmzmAMUqeo2VW0BFgNXdNtGgRTvcSpQGsHy9CwmEcbOgfhhB612M85azcIYYyCywWIMsCtoucRbF+xO4HoRKcHVKr4W9Fyel556VUTOiWA5D9HQ0kZDS7uloYwxxhPJYNFTNyLttrwAeERVs4H5wGMi4gP2ADmqOhP4FvCEiKR0ey0icrOIFIhIQVlZWa8VvMKm+jDGmINEMliUAGODlrM5NM10I7AEQFXfBuKATFVtVtUKb/1qYCtwUvc3UNUHVXW2qs7OysrqtYKX2VQfxhhzkEgGi1XAJBHJE5EY4FpgabdtioELAURkCi5YlIlIltdAjoiMByYB2yJY1oOU11qwMMaYYBHrDaWqbSJyC/Ai4AcWqepGEbkbKFDVpcBtwEMi8k1cimqhqqqInAvcLSJtQDvwJVWtjFRZu6uo99JQyZaGMsYYiGCwAFDV53AN18Hr7gh6XAic3cPrngaejmTZjqSjZpFuV8kzxhjARnD3qLyumZS4KGKj/H1dFGOM6RcsWPSgvL7FLqdqjDFBLFj0oLy2mcxECxbGGNPBgkUPyuuarXHbGGOCWLDoQUV9i3WbNcaYIBYsumltD1Dd0EqGpaGMMaaTBYtuOqf6sDSUMcZ0smDRTblN9WGMMYewYNFNV7CwmoUxxnSwYNFNeeeMs1azMMaYDhYsuqmwNJQxxhzCgkU35XXNxEX7SIixqT6MMaaDBYtuyuvcGAuRnq7dZIwxQ5MFi27K65otBWWMMd1YsOjG1SysJ5QxxgSzYNGN1SyMMeZQFiyCBAJKpc0LZYwxh4hosBCRS0Vks4gUicj3eng+R0SWi8haEdkgIvN7eL5ORL4dyXJ2qG5spT2gZFgayhhjDhKxYCEifuB+YB4wFVggIlO7bXY7sERVZwLXAn/o9vyvgecjVcbubKoPY4zpWSRrFnOAIlXdpqotwGLgim7bKJDiPU4FSjueEJErgW3AxgiW8SAWLIwxpmeRDBZjgF1ByyXeumB3AteLSAnwHPA1ABFJBL4L3HWkNxCRm0WkQEQKysrKjrvAXVN9WBrKGGOCRTJY9DSqTbstLwAeUdVsYD7wmIj4cEHi16pad6Q3UNUHVXW2qs7Oyso67gKX11rNwhhjehIVwX2XAGODlrMJSjN5bgQuBVDVt0UkDsgEzgCuFpFfAGlAQESaVPX3ESwvFfXNRPmE1PjoSL6NMcYMOJEMFquASSKSB+zGNWB/uts2xcCFwCMiMgWIA8pU9ZyODUTkTqAu0oECoLy2hfTEGHw+m+rDGGOCRSwNpaptwC3Ai8AmXK+njSJyt4hc7m12G/AFEVkPPAksVNXuqaoTxgbkGWNMzyJZs0BVn8M1XAevuyPocSFwdoh93BmRwvWgvL7FxlgY0w+0trZSUlJCU1NTXxdl0IiLiyM7O5vo6GNLs0c0WAw05bXNTMhM7OtiGDPklZSUkJycTG5urs0A3QtUlYqKCkpKSsjLyzumfdh0Hx5VpaK+mcxkS0MZ09eamprIyMiwQNFLRISMjIzjqqlZsPDUt7TT1BogI9HSUMb0BxYoetfxfp4WLDw2xsIYYw7PgoWnot4LFpaGMmbIq6ioID8/n/z8fEaOHMmYMWM6l1taWsLaxw033MDmzZsjXNITxxq4PWW17g/A0lDGmIyMDNatWwfAnXfeSVJSEt/+9sGTX6sqqorP1/Nv7ocffjji5TyRLFh4OiYRzLKahTH9yl3/2Ehh6YFe3efU0Sn86GPTjvp1RUVFXHnllcydO5cVK1bwz3/+k7vuuos1a9bQ2NjINddcwx13uNEBc+fO5fe//z3Tp08nMzOTL33pSzz//PMkJCTw7LPPMnz48F49pkizNJSnwptEMN1qFsaYIygsLOTGG29k7dq1jBkzhp/97GcUFBSwfv16li1bRmFh4SGvqamp4cMf/jDr16/nQx/6EIsWLeqDkh8fq1l4yuuaSUuIJtpv8dOY/uRYagCRNGHCBE4//fTO5SeffJI///nPtLW1UVpaSmFhIVOnHnzpnvj4eObNmwfAaaedxuuvv35Cy9wbLFh4bKoPY0w4EhO7Bu5u2bKF3/zmN6xcuZK0tDSuv/76HscyxMR0ZSz8fj9tbW0npKy9yX5GeyrqWuw6FsaYo3LgwAGSk5NJSUlhz549vPjii31dpIixmoWnvK6ZKaNTQm9ojDGeWbNmMXXqVKZPn8748eM5++wjTnU3oEkfTvLaq2bPnq0FBQXH/PpT7nyRT8zK5s7L+1d+1JihaNOmTUyZMqWvizHo9PS5ishqVZ0d6rWWhgKa29qpbWqzNJQxxhyGBQu6us1mWAO3Mcb0yIIFXQPyrDeUMcb0LKLBQkQuFZHNIlIkIt/r4fkcEVkuImtFZIOIzPfWzxGRdd5tvYhcFclydtQsLA1ljDE9i1hvKBHxA/cDFwMlwCoRWepdHa/D7bjLrT4gIlNxV9XLBd4DZqtqm4iMAtaLyD+8S7X2ujKrWRhjzBFFsmYxByhS1W2q2gIsBq7oto0CHf1VU4FSAFVtCAoMcd52EWNpKGOMObJIBosxwK6g5RJvXbA7getFpARXq/haxxMicoaIbATeBb7UU61CRG4WkQIRKSgrKzvmglbUtZAY4yc+xn/M+zDGDB7nnXfeIQPs7rvvPr7yla8c9jVJSUkAlJaWcvXVVx92v6G6+N933300NDR0Ls+fP5/q6upwix4xkQwWPV2WqXsNYQHwiKpmA/OBx0TEB6CqK1R1GnA68H0RiTtkZ6oPqupsVZ2dlZV1zAUtr2u2nlDGmE4LFixg8eLFB61bvHgxCxYsCPna0aNH89RTTx3ze3cPFs899xxpaWnHvL/eEskR3CXA2KDlbLw0U5AbgUsBVPVtLyBkAvs7NlDVTSJSD0wHjn3U3RG4eaGscduYfun578Hed3t3nyNPgXk/O+zTV199NbfffjvNzc3ExsayY8cOSktLyc/P58ILL6SqqorW1lbuuecerrji4Oz6jh07uOyyy3jvvfdobGzkhhtuoLCwkClTptDY2Ni53Ze//GVWrVpFY2MjV199NXfddRe//e1vKS0t5fzzzyczM5Ply5eTm5tLQUEBmZmZ3HvvvZ0z1t5000184xvfYMeOHcybN4+5c+fy1ltvMWbMGJ599lni4+N79SOLZM1iFTBJRPJEJAa4FljabZti4EIAEZmCa58o814T5a0fB5wM7IhUQd28UFazMMY4GRkZzJkzhxdeeAFwtYprrrmG+Ph4nnnmGdasWcPy5cu57bbbONIsGA888AAJCQls2LCBH/7wh6xevbrzuZ/85CcUFBSwYcMGXn31VTZs2MCtt97K6NGjWb58OcuXLz9oX6tXr+bhhx9mxYoVvPPOOzz00EOsXbsWcBMafvWrX2Xjxo2kpaXx9NNP9/pnErGahdeT6RbgRcAPLFLVjSJyN1CgqkuB24CHROSbuBTVQlVVEZkLfE9EWoEA8BVVLY9UWcvrmpmZMyxSuzfGHI8j1AAiqSMVdcUVV7B48WIWLVqEqvKDH/yA1157DZ/Px+7du9m3bx8jR47scR+vvfYat956KwAzZsxgxowZnc8tWbKEBx98kLa2Nvbs2UNhYeFBz3f3xhtvcNVVV3XOevvxj3+c119/ncsvv5y8vDzy8/MBNwX6jh07eulT6BLRiQRV9Tlcw3XwujuCHhcCh8y8paqPAY9Fsmwd2gNKZX0LWZaGMsYEufLKK/nWt77VeRW8WbNm8cgjj1BWVsbq1auJjo4mNze3xynJg4kc2ny7fft2fvnLX7Jq1SqGDRvGwoULQ+7nSDWY2NiuzIjf7z8o3dVbhvwI7qqGFgIKmXY5VWNMkKSkJM477zw+//nPdzZs19TUMHz4cKKjo1m+fDk7d+484j7OPfdcHn/8cQDee+89NmzYALipzRMTE0lNTWXfvn08//zzna9JTk6mtra2x339/e9/p6Ghgfr6ep555hnOOeec3jrckIb8FOUdYywyEi1YGGMOtmDBAj7+8Y939oy67rrr+NjHPsbs2bPJz89n8uTJR3z9l7/8ZW644QZmzJhBfn4+c+bMAeDUU09l5syZTJs27ZCpzW+++WbmzZvHqFGjDmq3mDVrFgsXLuzcx0033cTMmTMjknLqyZCfonxrWR33/usDvnr+RKba9SyM6RdsivLIOJ4pyod8zWJCVhL3Xzerr4thjDH92pBvszDGGBOaBQtjTL80WFLk/cXxfp4WLIwx/U5cXBwVFRUWMHqJqlJRUUFc3CGzJoVtyLdZGGP6n+zsbEpKSjieCULNweLi4sjOzj7m11uwMMb0O9HR0eTl5fV1MUwQS0MZY4wJyYKFMcaYkCxYGGOMCWnQjOAWkTLgyBO1HFkmELGZbfuAHU//N9iOabAdDwy+Y+rpeMapasirxw2aYHG8RKQgnCHvA4UdT/832I5psB0PDL5jOp7jsTSUMcaYkCxYGGOMCcmCRZcH+7oAvcyOp/8bbMc02I4HBt8xHfPxWJuFMcaYkKxmYYwxJiQLFsYYY0Ia8sFCRC4Vkc0iUiQi3+vr8vQGEdkhIu+KyDoROfrLB/YxEVkkIvtF5L2gdekiskxEtnj3w/qyjEfrMMd0p4js9r6ndSIyvy/LeDREZKyILBeRTSKyUUS+7q0fkN/TEY5nIH9HcSKyUkTWe8d0l7c+T0RWeN/R/4lITFj7G8ptFiLiBz4ALgZKgFXAAlUt7NOCHScR2QHMVtUBOZhIRM4F6oD/VdXp3rpfAJWq+jMvqA9T1e/2ZTmPxmGO6U6gTlV/2ZdlOxYiMgoYpaprRCQZWA1cCSxkAH5PRzieTzFwvyMBElW1TkSigTeArwPfAv6mqotF5I/AelV9INT+hnrNYg5QpKrbVLUFWAxc0cdlGvJU9TWgstvqK4BHvceP4v6RB4zDHNOApap7VHWN97gW2ASMYYB+T0c4ngFLnTpvMdq7KXAB8JS3PuzvaKgHizHArqDlEgb4H4hHgX+JyGoRubmvC9NLRqjqHnD/2MDwPi5Pb7lFRDZ4aaoBkbLpTkRygZnACgbB99TteGAAf0ci4heRdcB+YBmwFahW1TZvk7DPeUM9WEgP6wZDXu5sVZ0FzAO+6qVATP/zADAByAf2AL/q2+IcPRFJAp4GvqGqB/q6PMerh+MZ0N+Rqraraj6QjcukTOlps3D2NdSDRQkwNmg5Gyjto7L0GlUt9e73A8/g/kgGun1eXrkjv7y/j8tz3FR1n/fPHAAeYoB9T14e/GngcVX9m7d6wH5PPR3PQP+OOqhqNfAKcCaQJiIdF74L+5w31IPFKmCS1zsgBrgWWNrHZTouIpLoNdAhIonAR4D3jvyqAWEp8Dnv8eeAZ/uwLL2i46TquYoB9D15jad/Bjap6r1BTw3I7+lwxzPAv6MsEUnzHscDF+HaYpYDV3ubhf0dDeneUABeV7j7AD+wSFV/0sdFOi4iMh5XmwB32dwnBtoxiciTwHm46ZT3AT8C/g4sAXKAYuCTqjpgGowPc0zn4dIbCuwAvtiR7+/vRGQu8DrwLhDwVv8Al+cfcN/TEY5nAQP3O5qBa8D24yoGS1T1bu8csRhIB9YC16tqc8j9DfVgYYwxJrShnoYyxhgTBgsWxhhjQrJgYYwxJiQLFsYYY0KyYGGMMSYkCxbGHAURaQ+agXRdb85ULCK5wbPSGtOfRIXexBgTpNGbPsGYIcVqFsb0Au8aIj/3rh+wUkQmeuvHicjL3kR0L4tIjrd+hIg8411rYL2InOXtyi8iD3nXH/iXN/LWmD5nwcKYoxPfLQ11TdBzB1R1DvB73KwAeI//V1VnAI8Dv/XW/xZ4VVVPBWYBG731k4D7VXUaUA18IsLHY0xYbAS3MUdBROpUNamH9TuAC1R1mzch3V5VzRCRctxFdVq99XtUNVNEyoDs4GkWvKmxl6nqJG/5u0C0qt4T+SMz5sisZmFM79HDPD7cNj0JnqOnHWtXNP2EBQtjes81Qfdve4/fws1mDHAd7tKWAC8DX4bOC9SknKhCGnMs7FeLMUcn3rvyWIcXVLWj+2ysiKzA/Qhb4K27FVgkIv8BlAE3eOu/DjwoIjfiahBfxl1cx5h+ydosjOkFXpvFbFUt7+uyGBMJloYyxhgTktUsjDHGhGQ1C2OMMSFZsDDGGBOSBQtjjDEhWbAwxhgTkgULY4wxIf1/S4JMrG/8tY8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(model_history.history['acc'])\n",
    "plt.plot(model_history.history['val_acc'])\n",
    "plt.title('Accuracy Plot')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.legend(['Train', 'Validation'])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xt8XVWd///XJyc5SXNrkjaF3lNKgV7pJRSQckcElKuM0AEVvDCgPNAvoyMoKsNPRgb9IuN3GBlQ8AZUlEGLU2TUKQoItIGWQqm19zaktGnTS5o0l5N8fn/snfTkfnI5ze39fDz24+y99t7rrJ3Tns9Za+29lrk7IiIiPZXS3wUQEZHBTYFERER6RYFERER6RYFERER6RYFERER6RYFERER6RYFEZBAws0lmdsjMIkf5fbea2QVH8z1l8FEgkUGjv77UzOwGM2sIv8ibln9P8nu2uFZ33+7u2e7ekIT3cjOrCq/rPTN7oLsBy8zOMbPSvi6bDA6p/V0AkUHiVXdf1N+FSKKT3X2jmZ0EvAj8DXi4f4skg4VqJDIkmNlnzWyjmVWY2VIzGxemm5l9z8x2m9kBM1tjZrPCfZeY2btmVhn+Ev9SD973RTP7TNz2DWb2cty2m9nNZrbBzPaZ2UNmZq3KvS4sw7tmNt/MfgZMAp4Lawn/ZGZFYV6p4XnjwuusCK/7s3F53m1mT5vZT8N815pZcSLX4+5/BV4CZrVzrelm9qCZlYXLg2FaFvA8MC6uxjauu39LGbwUSGTQM7PzgG8DHwPGAtuAJeHuC4GzgBOAPOAaYG+470fAP7h7DsEX5/8mqYgfAU4BTg7L+KGw3H8H3A18AsgFLgP2uvvHge3ApWFz1v3t5PkUUAqMA64G/sXMzo/bfxnB3yAPWAok1BRnZjOAM4FV7ez+GnAaMDe8loXAXe5eBVwMlIXlzXb3skTeT4YGBRIZCq4DHnP3N929FrgTON3MioB6IAc4CTB3X+fuO8Pz6oEZZpbr7vvc/c1O3uM0M9sft5zWjfLd5+773X07sJzgixjgM8D97r7SAxvdfVtXmZnZRGAR8BV3r3H31cAPgY/HHfayuy8L+1R+RvDF35k3zWwf8FyY1+PtHHMdcI+773b3cuCfW72nDFMKJDIUjCOohQDg7ocIah3j3f1/CX6NPwTsMrNHzCw3PPSjwCXANjP7k5md3sl7vObueXHLa90o3/tx69VAdrg+EdjUjXyajAMq3L0yLm0bML6T98xoahbrwHx3z3f3qe5+l7s3dvC+8YFuW5gmw5wCiQwFZcDkpo2wzX4U8B6Au3/f3RcAMwmauL4cpq9098uBMcCvgad78N5VQGbc9rHdOHcHMLWDfZ0Ny10GFJhZTlzaJMLrTaIWf+fwPZuasDSM+DCmQCKDTZqZZcQtqcCTwI1mNtfM0oF/AV53961mdoqZnWpmaQRf+jVAg5lFzew6Mxvp7vXAQaAnt9auBq4ys0wzOx74dDfO/SHwJTNbEN4UcLyZNX1R7wKOa+8kd98B/AX4dvg3mBO+7xM9KH93PAXcZWaFZjYa+Abw87jyjjKzkUkugwxACiQy2CwDDsctd7v7H4GvA88AOwl+5V8bHp8LPArsI2iK2Qt8N9z3cWCrmR0Ebgau70F5vgfUEXyR/oRufJm7+y+BewkCYSVBragg3P1tgi/t/R3cTbYYKCKoETwLfNPdf9+D8nfHt4ASYA3wNvBmmNZ0t9dTwOawzGryGkZME1uJiEhvqEYiIiK9okAiIiK9okAiIiK9okAiIiK9MiwGbRw9erQXFRX1dzFERAaVN954Y4+7F3Z13LAIJEVFRZSUlPR3MUREBhUz63LIHlDTloiI9JICiYiI9IoCiYiI9Mqw6CMRkaGhvr6e0tJSampq+rsoQ0pGRgYTJkwgLS2tR+crkIjIoFFaWkpOTg5FRUXETTQpveDu7N27l9LSUqZMmdKjPNS0JSKDRk1NDaNGjVIQ6UNmxqhRo3pVy1MgEZFBRUGk7/X2b6pA0olnV5Xy89cSuo1aRGTYUiDpxH+veZ8nXt/e38UQkQFi7969zJ07l7lz53Lssccyfvz45u26urqE8rjxxhtZv359kkt6dKmzvRP5mWm8896B/i6GiAwQo0aNYvXq1QDcfffdZGdn86UvtZx3zN1xd1JS2v+d/vjjjye9nEebaiSdyM+Ksq86sV8ZIjJ8bdy4kVmzZnHzzTczf/58du7cyU033URxcTEzZ87knnvuaT520aJFrF69mlgsRl5eHnfccQcnn3wyp59+Ort37+7Hq+g51Ug6kZ8ZpTbWyOG6BkZEI/1dHBGJ88/PreXdsoN9mueMcbl889KZPTr33Xff5fHHH+fhhx8G4L777qOgoIBYLMa5557L1VdfzYwZM1qcc+DAAc4++2zuu+8+br/9dh577DHuuOOOXl/H0aYaSSfyM4OHcypUKxGRLkydOpVTTjmlefupp55i/vz5zJ8/n3Xr1vHuu++2OWfEiBFcfPHFACxYsICtW7cereL2KdVIOpGXGQVgX1Ud4/NG9HNpRCReT2sOyZKVldW8vmHDBv7t3/6NFStWkJeXx/XXX9/ucxrRaLR5PRKJEIvFjkpZ+5pqJJ0oyAo+5P3V9f1cEhEZTA4ePEhOTg65ubns3LmTF154ob+LlFSqkXRCTVsi0hPz589nxowZzJo1i+OOO44zzjijv4uUVObu/V2GpCsuLvaeTGxVXlnLKff+gXsun8knTi/q+4KJSLesW7eO6dOn93cxhqT2/rZm9oa7F3d1rpq2OpEX1kj2ValpS0SkI0kNJGZ2kZmtN7ONZtbhPW1mdrWZuZkVx6XdGZ633sw+1N08+0JaJIWcjFQ9SyIi0omk9ZGYWQR4CPggUAqsNLOl7v5uq+NygNuA1+PSZgDXAjOBccAfzOyEcHeXefal/Ew9lCgi0plk1kgWAhvdfbO71wFLgMvbOe7/A+4H4u+NuxxY4u617r4F2Bjml2iefSY/K0pFlQKJiEhHkhlIxgM74rZLw7RmZjYPmOjuv03w3C7zjMv7JjMrMbOS8vLynl0BwZ1buv1XRKRjyQwk7Q1w33yLmJmlAN8D/rEb53aaZ4tE90fcvdjdiwsLCxMobvvUtCUi0rlkBpJSYGLc9gSgLG47B5gFvGhmW4HTgKVhh3tH53aVZ5/Lz4yyT01bIgKcc845bR4ufPDBB/nc5z7X4TnZ2dkAlJWVcfXVV3eYb1ePKDz44INUV1c3b19yySXs378/0aInVTIDyUpgmplNMbMoQef50qad7n7A3Ue7e5G7FwGvAZe5e0l43LVmlm5mU4BpwIqu8kyG/Mw0quoaqIs1JvNtRGQQWLx4MUuWLGmRtmTJEhYvXtzluePGjeNXv/pVj9+7dSBZtmwZeXl5Pc6vLyUtkLh7DLgVeAFYBzzt7mvN7B4zu6yLc9cCTwPvAr8DPu/uDR3lmaxrAMhrHiZFtRKR4e7qq6/mt7/9LbW1tQBs3bqVsrIy5s6dy/nnn8/8+fOZPXs2v/nNb9qcu3XrVmbNmgXA4cOHufbaa5kzZw7XXHMNhw8fbj7ulltuaR5+/pvf/CYA3//+9ykrK+Pcc8/l3HPPBaCoqIg9e/YA8MADDzBr1ixmzZrFgw8+2Px+06dP57Of/SwzZ87kwgsvbPE+fSmpQ6S4+zJgWau0b3Rw7Dmttu8F7k0kz2QqCAdurKiuY0xuxtF6WxHpyvN3wPtv922ex86Gi+/rcPeoUaNYuHAhv/vd77j88stZsmQJ11xzDSNGjODZZ58lNzeXPXv2cNppp3HZZZd1OBf6D37wAzIzM1mzZg1r1qxh/vz5zfvuvfdeCgoKaGho4Pzzz2fNmjXcdtttPPDAAyxfvpzRo0e3yOuNN97g8ccf5/XXX8fdOfXUUzn77LPJz89nw4YNPPXUUzz66KN87GMf45lnnuH666/vm79VHD3Z3oV8Pd0uInHim7eamrXcna9+9avMmTOHCy64gPfee49du3Z1mMef//zn5i/0OXPmMGfOnOZ9Tz/9NPPnz2fevHmsXbu23eHn47388stceeWVZGVlkZ2dzVVXXcVLL70EwJQpU5g7dy6Q3GHqNWhjF5qGklfTlsgA00nNIZmuuOIKbr/9dt58800OHz7M/Pnz+fGPf0x5eTlvvPEGaWlpFBUVtTtsfLz2aitbtmzhu9/9LitXriQ/P58bbrihy3w6Gy8xPT29eT0SiSStaUs1ki40DSWvEYBFBIK7sM455xw+9alPNXeyHzhwgDFjxpCWlsby5cvZtm1bp3mcddZZPPHEEwC88847rFmzBgiGn8/KymLkyJHs2rWL559/vvmcnJwcKisr283r17/+NdXV1VRVVfHss89y5pln9tXlJkQ1ki40DdyohxJFpMnixYu56qqrmpu4rrvuOi699FKKi4uZO3cuJ510Uqfn33LLLdx4443MmTOHuXPnsnDhQgBOPvlk5s2bx8yZM9sMP3/TTTdx8cUXM3bsWJYvX96cPn/+fG644YbmPD7zmc8wb968ozrbooaRT8D0r/+O606dxF0fmdH1wSKSNBpGPnk0jHySFWRF1bQlItIBBZIE5Gm8LRGRDimQJEDjbYkMHMOhOf5o6+3fVIEkAflZGm9LZCDIyMhg7969CiZ9yN3Zu3cvGRk9f+Bad20lID8zjX1q2hLpdxMmTKC0tJTeTA0hbWVkZDBhwoQen69AkoC8zCgHa+qJNTSSGlElTqS/pKWlMWXKlP4uhrSib8UEFGSm4Q4HDqtWIiLSmgJJAvLDp9vVvCUi0pYCSQI03paISMcUSBLQPJS87twSEWlDgSQBGm9LRKRjCiQJONJHohqJiEhrCiQJyIpGiEZSNN6WiEg7khpIzOwiM1tvZhvN7I529t9sZm+b2Woze9nMZoTp14VpTUujmc0N970Y5tm0b0wyryF8z2C8Lc2SKCLSRtIeSDSzCPAQ8EGgFFhpZkvdPX7eyCfd/eHw+MuAB4CL3P0J4IkwfTbwG3dfHXfede7e83Hhe0AjAIuItC+ZNZKFwEZ33+zudcAS4PL4A9z9YNxmFtDeADqLgaeSVsoEBSMAK5CIiLSWzEAyHtgRt10aprVgZp83s03A/cBt7eRzDW0DyeNhs9bXrb2Jj5MgGAFYTVsiIq0lM5C09wXfpsbh7g+5+1TgK8BdLTIwOxWodvd34pKvc/fZwJnh8vF239zsJjMrMbOSvhjgTSMAi4i0L5mBpBSYGLc9ASjr5PglwBWt0q6lVW3E3d8LXyuBJwma0Npw90fcvdjdiwsLC7tZ9LbyM9PYf7hew1eLiLSSzECyEphmZlPMLEoQFJbGH2Bm0+I2PwxsiNuXAvwdQYBpSks1s9HhehrwESC+tpI0+ZlRGhqdgzWxo/F2IiKDRtLu2nL3mJndCrwARIDH3H2tmd0DlLj7UuBWM7sAqAf2AZ+My+IsoNTdN8elpQMvhEEkAvwBeDRZ1xAvPxwmZV9VHSNHpB2NtxQRGRSSOh+Juy8DlrVK+0bc+hc6OfdF4LRWaVXAgr4tZWLys4Lgsa+6jiKy+qMIIiIDkp5sT9CREYB155aISDwFkgRpBGARkfYpkCSouY9EDyWKiLSgQJKgnIxUUkxNWyIirSmQJCglxcjP1HhbIiKtKZB0g8bbEhFpS4GkG/Izo+zTUPIiIi0okHRDflZUne0iIq0okHRDfmaaAomISCsKJN3QNJS8Bm4UETlCgaQb8rOi1MUaqa5r6O+iiIgMGAok3ZCfeWS8LRERCSiQdIPG2xIRaUuBpBsKsjTelohIawok3aCmLRGRthRIukFNWyIibSmQdENeODOimrZERI5QIOmG1EgKuRmpGm9LRCROUgOJmV1kZuvNbKOZ3dHO/pvN7G0zW21mL5vZjDC9yMwOh+mrzezhuHMWhOdsNLPvm5kl8xpaC4ZJUdOWiEiTpAUSM4sADwEXAzOAxU2BIs6T7j7b3ecC9wMPxO3b5O5zw+XmuPQfADcB08LlomRdQ3uCp9tVIxERaZLMGslCYKO7b3b3OmAJcHn8Ae5+MG4zC+h07BEzGwvkuvurHoxT8lPgir4tduc03paISEvJDCTjgR1x26VhWgtm9nkz20RQI7ktbtcUM1tlZn8yszPj8iztKs9k0lDyIiItJTOQtNd30abG4e4PuftU4CvAXWHyTmCSu88DbgeeNLPcRPMEMLObzKzEzErKy8t7dAHt0VDyIiItJTOQlAIT47YnAGWdHL+EsJnK3WvdfW+4/gawCTghzHNCInm6+yPuXuzuxYWFhT2+iNbyM9Oormugpl4DN4qIQHIDyUpgmplNMbMocC2wNP4AM5sWt/lhYEOYXhh21mNmxxF0qm92951ApZmdFt6t9QngN0m8hjb0UKKISEupycrY3WNmdivwAhABHnP3tWZ2D1Di7kuBW83sAqAe2Ad8Mjz9LOAeM4sBDcDN7l4R7rsF+DEwAng+XI6apvG29lXXcezIjKP51iIiA1LSAgmAuy8DlrVK+0bc+hc6OO8Z4JkO9pUAs/qwmN2Sp/G2RERa0JPt3ZQfNm3pzi0RkYACSTfFN22JiIgCSbc1NW1pvC0RkYACSTelp0bIikaoUNOWiAigQNIjeZlR1UhEREIKJD2Qn6XxtkREmiiQ9EB+ZpQKPZAoIgIokPRIvpq2RESaKZD0QH5mGvs03a6ICKBA0iP5WVEO1sSINTT2d1FERPqdAkkPND3dvv+w+klERBRIekAPJYqIHKFA0gNNw6TooUQREQWSHmkeuFE1EhERBZKeUNOWiMgRCiQ9oKYtEZEjFEh6YERahGhqimokIiIokPSImQUPJSqQiIgkFkjMbKqZpYfr55jZbWaWl9yiDWz5mVE1bYmIkHiN5BmgwcyOB34ETAGe7OokM7vIzNab2UYzu6Od/Teb2dtmttrMXjazGWH6B83sjXDfG2Z2Xtw5L4Z5rg6XMQleQ5/SeFsiIoFEA0mju8eAK4EH3f3/AGM7O8HMIsBDwMXADGBxU6CI86S7z3b3ucD9wANh+h7gUnefDXwS+Fmr865z97nhsjvBa+hT+VlpVCiQiIgkHEjqzWwxwZf6b8O0tC7OWQhsdPfN7l4HLAEujz/A3Q/GbWYBHqavcveyMH0tkNHUtDZQBDUSNW2JiCQaSG4ETgfudfctZjYF+HkX54wHdsRtl4ZpLZjZ581sE0GN5LZ28vkosMrda+PSHg+btb5uZtbem5vZTWZWYmYl5eXlXRS1+5qathobvc/zFhEZTBIKJO7+rrvf5u5PmVk+kOPu93VxWntf8G2+dd39IXefCnwFuKtFBmYzgX8F/iEu+bqwyevMcPl4B2V+xN2L3b24sLCwi6J2X15mGo0OB2tUKxGR4S3Ru7ZeNLNcMysA3iKoETzQxWmlwMS47QlAWQfHQtD0dUXce04AngU+4e6bmtLd/b3wtZKgw39hItfQ15oeStyn5i0RGeYSbdoaGfZnXAU87u4LgAu6OGclMM3MpphZFLgWWBp/gJlNi9v8MLAhTM8D/hu4091fiTs+1cxGh+tpwEeAdxK8hj6l8bZERAKpiR5nZmOBjwFfS+QEd4+Z2a3AC0AEeMzd15rZPUCJuy8FbjWzC4B6YB9BZz7ArcDxwNfN7Oth2oVAFfBCGEQiwB+ARxO8hj7VNN6WZkoUkeEu0UByD0FAeMXdV5rZcYS1h864+zJgWau0b8Stf6GD874FfKuDbBckWOakUtOWiEggoUDi7r8Efhm3vZngbqphK69plkQ1bYnIMJdoZ/sEM3vWzHab2S4zeybsDB+2cjNSiaQYFWraEpFhLtHO9scJOsrHETwL8lyYNmwdGbhRTVsiMrwlGkgK3f1xd4+Fy4+Bvn84Y6BpbIDKXR3uztN4WyIiCQeSPWZ2vZlFwuV6YG8yCzYg/OxKePoTHe7Oz0xT05aIDHuJBpJPEdz6+z6wE7iaYNiUoW3cPHjvDairbne3xtsSEUl8iJTt7n6Zuxe6+xh3v4Lg4cShrehMaKyH0hXt7s7PjOqBRBEZ9nozQ+LtfVaKgWrSqWAR2PpKu7vzsoJZEt01cKOIDF+9CSTtjro7pKTnwNiTYevL7e4uyIxS3+BU1TUc5YKJiAwcvQkkw+NneNEZ8F4J1B9us6t5vC11uIvIMNZpIDGzSjM72M5SSfBMydBXdCY01EFpSZtdzeNtqZ9ERIaxTodIcfeco1WQAWvSaWApQfPWlDNb7NJ4WyIivWvaGh4yRsKxs2Fb2w53jbclIqJAkpiiM2HHCqivaZHcVCPRQ4kiMpwpkCRi8hnQUBs8nBhn5Ig0zNS0JSLDmwJJIiafDlib5q1IipGbkaamLREZ1hRIEjEiH46dBVtfarOrICuqpi0RGdYUSBI1eRHsWAmxlkEjLzNN422JyLCW1EBiZheZ2Xoz22hmd7Sz/2Yze9vMVpvZy2Y2I27fneF5683sQ4nmmTRFZ0DsMJS92SJZ422JyHCXtEBiZhHgIeBiYAawOD5QhJ5099nuPhe4H3ggPHcGcC0wE7gI+I+mIewTyDM5Jp8RvLZq3srPjOrJdhEZ1pJZI1kIbHT3ze5eBywBLo8/wN0Pxm1mcWTYlcuBJe5e6+5bgI1hfl3mmTSZBTBmZpsBHDVLoogMd8kMJOOBHXHbpWFaC2b2eTPbRFAjua2LcxPKM8z3JjMrMbOS8vLyHl9EC0VnwI7XoeFI4MjPinK4voGaeg3cKCLDUzIDSXujA7cZ6NHdH3L3qcBXgLu6ODehPMN8H3H3YncvLizso1mBixZBfTWUrWpOah64Uf0kIjJMJTOQlAIT47YnAGWdHL8EuKKLc7ubZ99q7ic5Mqx8ftPAjVVq3hKR4SmZgWQlMM3MpphZlKDzfGn8AWY2LW7zw8CGcH0pcK2ZpZvZFGAasCKRPJMqazQUntTiwcQ81UhEZJjrdPTf3nD3mJndCrwARIDH3H2tmd0DlLj7UuBWM7sAqAf2AZ8Mz11rZk8D7wIx4PPu3gDQXp7JuoZ2FS2Ct5ZAQwwiqXEjACuQiMjwlLRAAuDuy4BlrdK+Ebf+hU7OvRe4N5E8j6rJZ8DKH8LOt2DCgiNNW7pzS0SGKT3Z3l1N/STbgn6SPM2SKCLDnAJJd+UcA6NPaO5wj6amkJ2eqqYtERm2FEh6YvIZsP01aAyeHdF4WyIynCmQ9ETRIqg9CO+vAYJnSTQCsIgMVwokPdHqeZL8rKjmJBGRYUuBpCdyx0LB1OZxtzTelogMZwokPVV0Bmz/CzQ2aARgERnWFEh6avIiqDkAu94hPzNKZW2M+obG/i6ViMhRp0DSU0VN/SSvkJ8VPJSoO7dEZDhSIOmpkRMgvwi2vqzxtkRkWFMg6Y3Ji2D7XygYEYw0o1uARWQ4UiDpjaJFcHgfJ9gOzOClDX00gZaIyCCiQNIbYT/JmL0ruXDGMTzx+nYO12mmRBEZXhRIeiNvEoycBNte5jNnHsf+6nqeebO0v0slInJUKZD0VtEi2PYXiieNZM6EkTz28hYaG9ud/VdEZEhSIOmtojOgei9Wvp5PL5rC5j1VLF+/u79LJSJy1CiQ9Fbz/CSvcMnssYwdmcEPX9rSv2USETmKFEh6K78IcifA1pdJi6TwyQ8U8ermvawtO9DfJRMROSqSGkjM7CIzW29mG83sjnb2325m75rZGjP7o5lNDtPPNbPVcUuNmV0R7vuxmW2J2zc3mdfQJbOgeWvbK+DO4lMmkRmN8KOXVSsRkeEhaYHEzCLAQ8DFwAxgsZnNaHXYKqDY3ecAvwLuB3D35e4+193nAucB1cD/xJ335ab97r46WdeQsMlnQFU5lP+VkZlpfKx4Is+9VcaugzX9XTIRkaRLZo1kIbDR3Te7ex2wBLg8/oAwYFSHm68BE9rJ52rg+bjjBp5pH4TUDHjx2wDceEYRsUbnp69u7ddiiYgcDckMJOOBHXHbpWFaRz4NPN9O+rXAU63S7g2bw75nZuntZWZmN5lZiZmVlJcn+Ynz3HFw1pfh3d/A+t8xeVQWH5yuBxRFZHhIZiCxdtLafcDCzK4HioHvtEofC8wGXohLvhM4CTgFKAC+0l6e7v6Iuxe7e3FhYWH3S99dH7gNCqfDsi9B7SE9oCgiw0YyA0kpMDFuewJQ1vogM7sA+BpwmbvXttr9MeBZd28en93dd3qgFnicoAmt/6VG4dIH4cAOWP4vnFKUrwcURWRYSGYgWQlMM7MpZhYlaKJaGn+Amc0D/pMgiLT3FN9iWjVrhbUUzMyAK4B3klD2npl0Giy4EV7/AbbzLT2gKCLDQtICibvHgFsJmqXWAU+7+1ozu8fMLgsP+w6QDfwyvJW3OdCYWRFBjeZPrbJ+wszeBt4GRgPfStY19MgFd0PmaHjuC1wys5CxIzN0K7CIDGmpyczc3ZcBy1qlfSNu/YJOzt1KO53z7n5eHxax743Ig4vvg199irSSH/LJD3yI+57/K2vLDjBz3Mj+Lp2ISJ/Tk+3JMPMqOP6D8L/f4u9PjOgBRREZ0hRIksEMPvx/ASd3+Z18bMEEnnurjN16QFFEhiAFkmTJnwzn3Anrl/G5Y9eFDyhu6+9SiYj0OQWSZDrtFjhmNmNe/jqXnZjFz1/fpgcURWTIUSBJpkgaXPpvUPk+X814Rg8oisiQpECSbBMWwMKbGPPXn/HRMTv1gKKIDDkKJEfDeXdhOWP5Bo+wfc8BXvybHlAUkaFDgeRoyMiFS+5n5MH1fCHrDzz4hw3UxtRXIiJDgwLJ0TL9Ujjxw3yOp6l4bwPf+PVa3NXEJSKDnwLJ0XTJ/UQiqfyy4BFeKlnFT/6ytb9LJCLSawokR9PICXDlwxxbv50/ZN7Jm88/xisb9/R3qUREekWB5Gibfil280ukHzud76d+n4qff4odO3f1d6lERHpMgaQ/FBxH5NMvsH/h7VzCn0l95EyqN73a36USEekRBZL+Ekkl75Jv8s4HlxBrbCT9Z5fQuPzb0BDr75KJiHSLAkk/O/mMi3jx3P/i1w0fIOVP98HjF0OFRgoWkcFDgWQAuP7s2bw251+4re5W6netg4fPhNVPgW4PFpFBQIFkADAzvnXlLEonXMJFNd/GC0BBAAAXBElEQVSmqmA6/Ppm+NWn4PC+/i6eiEinFEgGiPTUCA9/fAFVI8Zx0b5/onrRV2HdUviPD8DGP/Z38UREOpTUQGJmF5nZejPbaGZ3tLP/djN718zWmNkfzWxy3L6GcB731nO5TzGz181sg5n9wsyiybyGo2lMTgaPfGIBu6ti3LDpLOpv/D2k58DPr4Lf3g51Vf1dRBGRNpIWSMwsAjwEXAzMABab2YxWh60Cit19DvAr4P64fYfdfW64XBaX/q/A99x9GrAP+HSyrqE/zJmQx/1Xz2HFlgruLknDb3oRTr8VSh6DhxfBjhX9XUQRkRaSWSNZCGx0983uXgcsAS6PP8Ddl7t7dbj5GjChswzNzIDzCIIOwE+AK/q01APA5XPHc/PZU3ni9e1c9p9v8vSom6m9fmlwa/BjH4I//DPE6vq7mCIiQHIDyXhgR9x2aZjWkU8Dz8dtZ5hZiZm9ZmZNwWIUsN/dmx626DBPM7spPL+kvLy8Z1fQj778oRO598pZ1MYa+KdfreHUJ2v47vGPc2j6tfDyA/DoefD+O/1dTBGRpAYSayet3ftZzex6oBj4TlzyJHcvBv4eeNDMpnYnT3d/xN2L3b24sLCweyUfACIpxnWnTuaFL57FU589jQ9MHcUPXt3N7FUf4cHCe6jbX4Y/ei68/D1o1JD0ItJ/UpOYdykwMW57AlDW+iAzuwD4GnC2u9c2pbt7Wfi62cxeBOYBzwB5ZpYa1krazXMoMTNOnzqK06eO4v0DNTy5YjtPrEjnp5X38r2sH3P2H+4mtm4ZqefdCem5kJoOqRltXyNRsPbisIhI71iy5sQws1Tgb8D5wHvASuDv3X1t3DHzCPo7LnL3DXHp+UC1u9ea2WjgVeByd3/XzH4JPOPuS8zsYWCNu/9HZ2UpLi72kpKSvr7EflMXa+SFte/zs79sZdyO57gn7cfkWnWX55GaEdwFNuujcOrNUDAl6WUVkcHLzN4IW4Y6Py6ZkyuZ2SXAg0AEeMzd7zWze4ASd19qZn8AZgM7w1O2u/tlZvYB4D+BRoLmtwfd/UdhnscRdNwXENz1dX18TaY9Qy2QxFu38yDPvLSaDW+/jjXUUTw+kwtPHMm0gjSsoRZitRCrOfK6fwesew68AU76SHBH2KRT+/syRGQAGhCBZKAYyoGkyd5Dtfz8te389NWt7K2qY9b4XD575nFcMnssaZFWXWEHy2DFI8EtxTUHYMIpcPrn4aRLIZLM1k4RGUwUSOIMh0DSpKa+gWdXvcejL21mc3kV40Zm8KlFU7jmlInkZKS1PLj2ELz1FLz6EOzbAnmT4NRbYN71wTzzIjKsKZDEGU6BpEljo7N8/W4e+fNmXt9SQU56KtcunMiNZ0xhXN6IVgc3wPrn4dV/h+2vBp328z8BM66AzIJgSR8JKRpRR4aw3X+Fhjo4drZuTAkpkMQZjoEk3prS/Tz60haWvb2ThkZnRFqEnIzUcEk7sp6exomx9Zyx5xdM2/u/pHjcbcWWAhl5QVAZkQ8jwtemQJM/BUZPg1HHQzSr/y62rzU2BM1/tQeD15qDLdcbYzDrKsgd198llZ5qiMGfvwN/vh+8EY6ZDfM/DrP/Lvi3PYwpkMQZ7oGkSem+ap57aycVVbVU1sSorIlxsKY+XK/nUG2QVl3XwFj2Mj1lGwUpVUwfGWN6XozjsuooTK0mUrMPqivg8H44XAF1h1q+Uc44GH18EFRGTQsDzFTImwwpkd5fSGMj1B4IylC1J2iGG31C7/JubICyVbBpOWx+MWjqqznQ9traE4nC3Otg0Rchv6jnZZCj70ApPPNZ2P4XOHkxjF8Aq34OO1cHn+tJHwmCypRzhmWNXIEkjgJJ98QaGqmsifHX9yv5y6Y9vLJxD2+VHqCh0YmmplA8OZ8zjh/NB6aOYvb4kaQ21ARfvHs2wN6NwbJnA+zdEHwZN4lEIfuYoMaSlhm8Ni1pmRDNhmiYHkkPzq3eA9V7jwSN6r3B4q0ewoxmw7h5wRfB+AUwobjrWkLF5jBwLIctfz5S1mPnBM0bGXlBkErPhYyR7ayPhJr9QZPgqp8HwWjONXDm7UHwlIFt3W/hN58PapUffgBOvubIvvffhjd/Bmt+EXzGIyfBvOuCHwx5EzvOc4hRIImjQNJ7lTX1rNxawSsb9/LKxj389f1KAHLSU1k4pYCTxuYwbUwOx4/J5rjCLDKjqcHEXNV7WwaWQ+VQXxWMZFxXHfzir68Ot8OlKUhYStCEljkqWLJGHVnPHB2+FgTvUVoC770RfAE01gfn54w9EljGLwhqSKUrg8CxaTns3xYclzsBpp4Dx50Lx50DWaO7/wc6WAZ/+X9Q8nhwm/XMK+HMf4RjZ/X2T9+5xsa4prZOloZaGHsyTDo9qL0N5z6A+hr4n6/Byh/C2Llw9WNBjbmjY//6W1j1s6CmisHUc4MbUqZdGDyXNYQpkMRRIOl7ew7V8trmvbyycS8rtuxl695qGhqP/FsanzeC48dkt1wKs8nP6mLUf/cjz7yk53a/OaG+Bna9EwaWMLhUbG55TDQHppwZBI6p5wYBpq++WA+Vw2sPwYofQl0lnPhhOOsfg0CWCPcgmFaVH1kO7W61vgeqwrTD++lglKAj0nOD62uqcY0ogEmnhcvpQYBJTe/VZQ8a5euDCeN2vRM8Q3X+NyE1wZko9m2D1U/AqifgYGlQwy46E068GE64aEjWVBRI4iiQJF9drJGte6vYuPtQi2XznkPU1Dc2H5ebkcqxIzM4JjeDsSMzODY3g2NGBuvH5AbbBVlRrA9/Mdcd3EPd9pX4no1EJ80nffJCiKR1fWKcxkbnvf2H2VR+iM3lVc2vuw7WMH1sLqcU5XPKlAJOOjaXSIoFM1u+/gi89h9B08jU82HqeVBb2bYGEd95X3OgbbNdk4yRkDUGsgohuzB4HVEAI/LC5rZ2lvTcoO/IPQio218Nl9eCmiIEIx6MX9AysGQVDq1ai3tQq3j+K5A2Aq54GE64sGd5NTYEf8P1zwdLxaYg/ZjZQVA58SIYO29I9KkokMRRIOk/TV/ATYFlx75q3j9Qw/sHa3j/QA3lh2rbTE0fjaRQmJNORloKaZFgSY1YuB68pqakEE0N1mONzuG6BqpqgxsFquuC16raGIfrG6hvaPkGmdEI+ZlRRmVHKcgKl8woBdlRRmVFKchKp7ouxqbdh9i0p4pNuw+xZU8VtbGWAXHqmGwKs9N5570DlB2oASAnI5UFk/M5paiAU6cUMLswhfRVPw76UarCUaijOUf6WZq+7Fv3wWQVQvaYoJmtKXgk+ss5UYfKYcdrQVDZ/irsfCvoL2gqY8EUKDguWEZNPbKefUzfBJnGRqjcGfSv7dsKFVvg4HtBjbShLihLQ33QVNkQC1/rj6RFs4N+sNzxrV7DpamWVXMAnvsirP0vmHIWXPkI5I5tUxx3p6a+kcra4AaUQzWx5htQgtf65rT6Bmf62BzmTcrnOMpI2fA8rP9d8Pf0xuBvdMKH4MRLoPBEwIKmWgtf47eb11PCsfHSISV1QARyBZI4CiQDV31DI+WVtbx/sIZdB2rYeaCGXQdrKK+spTbWSH1DsMQanbpYy/VYo1Pf0EiKGVnpETKjqWRGI2Q1vaanMiIaISsa7IumpnDgcD0VVXVtlr1VtS1qTgApBhMLMplamM1xo7OYOubI66hWtabSfdWs3FrBii37WLm1go27g7u9oqkpzJ2Yx2mTcika2UhNShYxTyHW4DQ0OrFGp9E93A6uCaAgK8qY3AwKs9MZk5vOmJx0stNT+7Sm1kZdVdAUuHsd7N0U1GAqNgVNOvG1pLSsIKDkTw4CXzTzyM0TaZnhdtaR17QRQa2rIgwYTYFj37ag76aJRYJ+rbQMSEkLao2RtCPrKalBc1LTem1l0Dd1sCy4i6+1zNFBQKnaA4d2wXlfgzO+2ObuvoqqOn6xcgc/f20b7+0/3OWfKT01JWgpDP+95GakcvLEPOZNyueUY5z5NSvJ2vr7YIrsusoefBCEQSUcbLV58NX0I4OwpueGPzIKw/7Dwrbb0axeByMFkjgKJJKI6rpYc2DJSIsweVQm6ak9u6V476FaSrbtY+WWClZsrWBt2cEWfUgdSYtY0E3UzrEj0iIU5gRBJQguGWSnpxJJsRZLaoqRYkZqJHxNMVJSrMUcDE0BqWVa8JqfFeX4wmzG540gJcWCGsCBHbB3cxhcwgCzf3swOkJ9FdQfDvq1uhLNgYKi4Lmj/KJgKQjXR07sdpNjs9pKOLgzqNE0BZem9VgNnPu1NmPKvV16gJ+8upWlb5VRF2vkA1NHsWja6ODZqvTg2ars9FSyw2escjJSyUoPfpA0Njqbyg+xasd+Vm3fz6rt+/jbrkqaPrYpo7MonpDFhdmbmZ51iHF5GcGcHd4IePDqHrcdrsfixsdrM1Ze03I4aAat2hPUcmMdBL/UjCCgfPzZHt9FqEASR4FE+ltVbYwDh+tJbf6yTyESifvSD7/sIWhiOXg4xu7KGnZX1rK7Mqih7T5Y27y9u7KW8oO1VNXFSCA+9ciItEibGyamjclmUkEmqa3Hb4Og76C+Orgbrz68K6/pjrz0nCB4ZBb0a5NNbayB599+n5+8upVV2/eTGY1w1fzxfOL0Ik44pnd3YFXVxlhTeoBVO/axevt+3ty+nz2HghpXTnoqC4rimjwnjOzxj5Q26qrCoLInuF2+qvxIkKneCxd+q2d3IqJA0oICiQxl7kEzWUPTa+slbDpre17cenjnlzvsrqxl4+5DbNhd2dy3tfPAkdpGNJLClNFZFI3OJDOa2txvFU1NIRo50q+VlmrN240eNEfWxRqpjTVS13BkvTbW0Lwv1uikRYxoaoT01CDP5tdICulpEaKRFNLTUshOTyW/qX8r7OvKjEbabf7beeAwT76+nadWbGfPoTqOG53Fx0+fzEcXTCC39Rh0fcTdKd13mDe27WPF1gpWbqlgQ9jkmZ6awskT8zh1SgGnFBUwf3I+2ekDb8BUBZI4CiQivVNZU8+m8io27DoSXLZVVFMba6A+FvRV1TU09Wl5p814KUZz0GkODGGwSI0Y9TGPCzQNQeAJg05XoqkpLQJLflaUw3UNLF+/m0Z3zj/pGD5x+mQWHT+6uQZ4NFVU1bEyDCrxTZ6RFOOkY3M48Zgcpoa1v6mF2Uweldl29O6jSIEkjgKJyNHVEN4IUR8GhBSz5tpFu81iCXB36hu8uQZTWROjorqOffE3TjRv11NRVcu+6nrqYo18ZM5Yrj9tMhMLMvv4SnvnUG2MVdv3sWJLBau272fj7kO8f/BI7S8tYkwelcXxhdlMHZMVPo+Vw+icKNGmWmAYlJNxI4YCSRwFEhEZLCpr6tlcHj6TVX6ITeHrtlYP/bbWOrA0rf/ok8VMHtWzgVQTDSQDr1FORGQYy8lI4+SJeZw8Ma9Fel2ske0VQYDZX11PXUMjtfVBk2JT819drJG6hoa49UYy0vqoU78TCiQiIoNANDWF48fkcPyYgTe+V1J7cczsIjNbb2YbzeyOdvbfbmbvmtkaM/ujmU0O0+ea2atmtjbcd03cOT82sy1mtjpc5ibzGkREpHNJCyRmFgEeAi4GZgCLzWxGq8NWAcXuPgf4FXB/mF4NfMLdZwIXAQ+aWXw978vuPjdcVifrGkREpGvJrJEsBDa6+2Z3rwOWAJfHH+Duy929Otx8DZgQpv/N3TeE62XAbqAwiWUVEZEeSmYgGQ/siNsuDdM68mng+daJZrYQiAKb4pLvDZu8vmdm7Y5/bWY3mVmJmZWUl5d3v/QiIpKQZAaS9m5qbvfeNTO7HigGvtMqfSzwM+BGd296GulO4CTgFKAA+Ep7ebr7I+5e7O7FhYWqzIiIJEsyA0kpED/TywSgrPVBZnYB8DXgMnevjUvPBf4buMvdX2tKd/edHqgFHidoQhMRkX6SzECyEphmZlPMLApcCyyNP8DM5gH/SRBEdselR4FngZ+6+y9bnTM2fDXgCuCdJF6DiIh0IWnPkbh7zMxuBV4AIsBj7r7WzO4BStx9KUFTVjbwy/Dx/u3ufhnwMeAsYJSZ3RBmeUN4h9YTZlZI0HS2Grg5WdcgIiJdGxZDpJhZObCth6ePBvb0YXEGgqF2TbqegW+oXdNQux5o/5omu3uXnczDIpD0hpmVJDLWzGAy1K5J1zPwDbVrGmrXA727psE/O72IiPQrBRIREekVBZKuPdLfBUiCoXZNup6Bb6hd01C7HujFNamPREREekU1EhER6RUFEhER6RUFkk50NZ/KYGNmW83s7XAel0E597CZPWZmu83snbi0AjP7vZltCF/z+7OM3dHB9dxtZu/FzblzSX+WsTvMbKKZLTezdeF8Ql8I0wfzZ9TRNQ3Kz8nMMsxshZm9FV7PP4fpU8zs9fAz+kU4wkhieaqPpH3hfCp/Az5IMG7YSmCxu7/brwXrBTPbSjD/y6B9kMrMzgIOEQyfMytMux+ocPf7woCf7+7tDuY50HRwPXcDh9z9u/1Ztp4IhzAa6+5vmlkO8AbBUEY3MHg/o46u6WMMws8pHF4qy90PmVka8DLwBeB24L/cfYmZPQy85e4/SCRP1Ug61uV8KnL0ufufgYpWyZcDPwnXf0Lwn3xQ6OB6Bq1wUNU3w/VKYB3B9BGD+TPq6JoGpXDQ20PhZlq4OHAewQSD0M3PSIGkY92dT2UwcOB/zOwNM7upvwvTh45x950Q/KcHxvRzefrCreGcO48NpmageGZWBMwDXmeIfEatrgkG6edkZhEzW00waeDvCeZ72u/usfCQbn3fKZB0LOH5VAaRM9x9PsH0x58Pm1Vk4PkBMBWYC+wE/m//Fqf7zCwbeAb4orsf7O/y9IV2rmnQfk7u3uDucwmm91gITG/vsETzUyDpWELzqQwm4bTFhEP2P8vQmctlV9z0AmMJfmUNWu6+K/yP3gg8yiD7nMJ292eAJ9z9v8LkQf0ZtXdNg/1zAnD3/cCLwGlAnpk1jQjfre87BZKOdTmfymBiZllhRyFmlgVcyNCZy2Up8Mlw/ZPAb/qxLL3W9IUbupJB9DmFHbk/Ata5+wNxuwbtZ9TRNQ3Wz8nMCs0sL1wfAVxA0O+zHLg6PKxbn5Hu2upEeDvfgxyZT+Xefi5Sj5nZcQS1EAjmoXlyMF6PmT0FnEMw5PUu4JvAr4GngUnAduDv3H1QdGB3cD3nEDSXOLAV+Iem/oWBzswWAS8BbwNN02N/laBPYbB+Rh1d02IG4edkZnMIOtMjBJWJp939nvA7YgnBFOargOvjZ63tNE8FEhER6Q01bYmISK8okIiISK8okIiISK8okIiISK8okIiISK8okIj0ATNriBsFdnVfjhZtZkXxowOLDDSpXR8iIgk4HA45ITLsqEYikkThHDD/Gs7/sMLMjg/TJ5vZH8MB//5oZpPC9GPM7Nlwroi3zOwDYVYRM3s0nD/if8InkkUGBAUSkb4xolXT1jVx+w66+0Lg3wlGSiBc/6m7zwGeAL4fpn8f+JO7nwzMB9aG6dOAh9x9JrAf+GiSr0ckYXqyXaQPmNkhd89uJ30rcJ67bw4H/nvf3UeZ2R6CyZLqw/Sd7j7azMqBCfFDU4RDl//e3aeF218B0tz9W8m/MpGuqUYiknzewXpHx7QnfsyjBtS/KQOIAolI8l0T9/pquP4XghGlAa4jmO4U4I/ALdA8+VDu0SqkSE/pV41I3xgRzjjX5Hfu3nQLcLqZvU7ww21xmHYb8JiZfRkoB24M078APGJmnyaoedxCMGmSyIClPhKRJAr7SIrdfU9/l0UkWdS0JSIivaIaiYiI9IpqJCIi0isKJCIi0isKJCIi0isKJCIi0isKJCIi0iv/P28J7qsqYeBQAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(model_history.history['loss'])\n",
    "plt.plot(model_history.history['val_loss'])\n",
    "plt.title('Loss Function Plot')\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('Epoch')\n",
    "plt.legend(['Train', 'Validation'])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Visualize the above plots and analyze the Train and Validation curves"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experimenting with Hidden Layers\n",
    "\n",
    "We have to be careful about overfitting!!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 32584 samples, validate on 8146 samples\n",
      "Epoch 1/10\n",
      "32584/32584 [==============================] - 2s 73us/step - loss: 0.4830 - acc: 0.8132 - val_loss: 0.4911 - val_acc: 0.8048\n",
      "Epoch 2/10\n",
      "32584/32584 [==============================] - 2s 53us/step - loss: 0.4779 - acc: 0.8132 - val_loss: 0.4881 - val_acc: 0.8048\n",
      "Epoch 3/10\n",
      "32584/32584 [==============================] - 2s 54us/step - loss: 0.4741 - acc: 0.8132 - val_loss: 0.4844 - val_acc: 0.8048\n",
      "Epoch 4/10\n",
      "32584/32584 [==============================] - 2s 52us/step - loss: 0.4688 - acc: 0.8132 - val_loss: 0.4775 - val_acc: 0.8048\n",
      "Epoch 5/10\n",
      "32584/32584 [==============================] - 2s 53us/step - loss: 0.4607 - acc: 0.8132 - val_loss: 0.4663 - val_acc: 0.8048\n",
      "Epoch 6/10\n",
      "32584/32584 [==============================] - 2s 56us/step - loss: 0.4474 - acc: 0.8132 - val_loss: 0.4482 - val_acc: 0.8048\n",
      "Epoch 7/10\n",
      "32584/32584 [==============================] - 2s 65us/step - loss: 0.4261 - acc: 0.8132 - val_loss: 0.4217 - val_acc: 0.8048\n",
      "Epoch 8/10\n",
      "32584/32584 [==============================] - 2s 69us/step - loss: 0.3958 - acc: 0.8132 - val_loss: 0.3859 - val_acc: 0.8048\n",
      "Epoch 9/10\n",
      "32584/32584 [==============================] - 2s 66us/step - loss: 0.3605 - acc: 0.8224 - val_loss: 0.3504 - val_acc: 0.8354\n",
      "Epoch 10/10\n",
      "32584/32584 [==============================] - 2s 59us/step - loss: 0.3277 - acc: 0.8523 - val_loss: 0.3186 - val_acc: 0.8592\n",
      "Confusion matrices:\n",
      "---------------------\n",
      "Confusion matrix - Train:\n",
      "[[31984  1068]\n",
      " [ 4524  3154]]\n",
      "Confusion matrix - Test:\n",
      "[[13679   486]\n",
      " [ 1948  1343]]\n",
      "---------------------\n",
      "Evaluation metrics on train data for new model:\n",
      "------------------------------------\n",
      "Train Specificity:  0.9676872806486748\n",
      "Train Recall:  0.41078405834852827\n",
      "Train Precision:  0.7470393178588347\n",
      "Train Accuracy:  0.8627056223913577\n",
      "------------------------------------\n",
      "Evaluation metrics on train data for model 1:\n",
      "Train Specificity:  0.9480515551252572\n",
      "Train Recall:  0.7212815837457671\n",
      "Train Precision:  0.7633356305995865\n",
      "Train Accuracy:  0.9053032163024798\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "Evaluation metrics on test data for new model:\n",
      "------------------------------------\n",
      "Test Specificity:  0.9656900811860218\n",
      "Test Recall:  0.4080826496505621\n",
      "Test Precision:  0.7342810278840897\n",
      "Test Accuracy:  0.8605637030247479\n",
      "------------------------------------\n",
      "Evaluation metrics on test data for model 1:\n",
      "Test Specificity:  0.9430991881397811\n",
      "Test Recall:  0.7119416590701915\n",
      "Test Precision:  0.7440457288027945\n",
      "Test Accuracy:  0.8995187901008249\n",
      "------------------------------------\n",
      "------------------------------------\n"
     ]
    }
   ],
   "source": [
    "ann_model_hiddenExp = Sequential()\n",
    "\n",
    "# Adding more hidden layers\n",
    "ann_model_hiddenExp.add(Dense(300, input_dim=21, activation='sigmoid', kernel_initializer='normal'))\n",
    "ann_model_hiddenExp.add(Dense(200, activation='sigmoid', kernel_initializer='normal'))\n",
    "ann_model_hiddenExp.add(Dense(100, activation='sigmoid', kernel_initializer='normal'))\n",
    "ann_model_hiddenExp.add(Dense(1, activation='sigmoid', kernel_initializer='normal'))\n",
    "\n",
    "ann_model_hiddenExp.compile(loss='binary_crossentropy', optimizer='sgd', metrics=['accuracy'])\n",
    "\n",
    "ann_model_hiddenExp.fit(X_train, y_train, epochs=10, batch_size=64, validation_split=0.2)\n",
    "\n",
    "# Predictions\n",
    "train_pred = ann_model_hiddenExp.predict_classes(X_train)\n",
    "test_pred = ann_model_hiddenExp.predict_classes(X_test)\n",
    "\n",
    "#Evaluation metrics\n",
    "confusion_matrix_train = confusion_matrix(y_train, train_pred)\n",
    "confusion_matrix_test = confusion_matrix(y_test, test_pred)\n",
    "\n",
    "print(\"Confusion matrices:\")\n",
    "print(\"---------------------\")\n",
    "print(\"Confusion matrix - Train:\")\n",
    "print(confusion_matrix_train)\n",
    "print(\"Confusion matrix - Test:\")\n",
    "print(confusion_matrix_test)\n",
    "print(\"---------------------\")\n",
    "\n",
    "# Metrics on train data\n",
    "#Accuracy\n",
    "accuracy_Train_hiddenExp = (confusion_matrix_train[0,0]+confusion_matrix_train[1,1])/(confusion_matrix_train[0,0]+confusion_matrix_train[0,1]+confusion_matrix_train[1,0]+confusion_matrix_train[1,1])\n",
    "#specificity or true negative rate (TNR)\n",
    "specificity_Train_hiddenExp = confusion_matrix_train[0,0]/(confusion_matrix_train[0,0]+confusion_matrix_train[0,1])\n",
    "#sensitivity, recall, hit rate, or true positive rate (TPR)\n",
    "recall_Train_hiddenExp = confusion_matrix_train[1,1]/(confusion_matrix_train[1,0]+confusion_matrix_train[1,1])\n",
    "#precision\n",
    "precision_Train_hiddenExp = confusion_matrix_train[1,1]/(confusion_matrix_train[0,1]+confusion_matrix_train[1,1])\n",
    "\n",
    "print(\"Evaluation metrics on train data for new model:\")\n",
    "print(\"------------------------------------\")\n",
    "print(\"Train Specificity: \",specificity_Train_hiddenExp)\n",
    "print(\"Train Recall: \",recall_Train_hiddenExp)\n",
    "print(\"Train Precision: \",precision_Train_hiddenExp)\n",
    "print(\"Train Accuracy: \",accuracy_Train_hiddenExp)\n",
    "print(\"------------------------------------\")\n",
    "\n",
    "print(\"Evaluation metrics on train data for model 1:\")\n",
    "print(\"Train Specificity: \",specificity_Train_M1)\n",
    "print(\"Train Recall: \",recall_Train_M1)\n",
    "print(\"Train Precision: \",precision_Train_M1)\n",
    "print(\"Train Accuracy: \",accuracy_Train_M1)\n",
    "print(\"------------------------------------\")\n",
    "print(\"------------------------------------\")\n",
    "\n",
    "# Metrics on test data\n",
    "#Accuracy\n",
    "accuracy_Test_hiddenExp = (confusion_matrix_test[0,0]+confusion_matrix_test[1,1])/(confusion_matrix_test[0,0]+confusion_matrix_test[0,1]+confusion_matrix_test[1,0]+confusion_matrix_test[1,1])\n",
    "#specificity or true negative rate (TNR)\n",
    "specificity_Test_hiddenExp = confusion_matrix_test[0,0]/(confusion_matrix_test[0,0]+confusion_matrix_test[0,1])\n",
    "#sensitivity, recall, hit rate, or true positive rate (TPR)\n",
    "recall_Test_hiddenExp = confusion_matrix_test[1,1]/(confusion_matrix_test[1,0]+confusion_matrix_test[1,1])\n",
    "#precision\n",
    "precision_Test_hiddenExp = confusion_matrix_test[1,1]/(confusion_matrix_test[0,1]+confusion_matrix_test[1,1])\n",
    "\n",
    "print(\"Evaluation metrics on test data for new model:\")\n",
    "print(\"------------------------------------\")\n",
    "print(\"Test Specificity: \",specificity_Test_hiddenExp)\n",
    "print(\"Test Recall: \",recall_Test_hiddenExp)\n",
    "print(\"Test Precision: \",precision_Test_hiddenExp)\n",
    "print(\"Test Accuracy: \",accuracy_Test_hiddenExp)\n",
    "print(\"------------------------------------\")\n",
    "\n",
    "print(\"Evaluation metrics on test data for model 1:\")\n",
    "print(\"Test Specificity: \",specificity_Test_M1)\n",
    "print(\"Test Recall: \",recall_Test_M1)\n",
    "print(\"Test Precision: \",precision_Test_M1)\n",
    "print(\"Test Accuracy: \",accuracy_Test_M1)\n",
    "print(\"------------------------------------\")\n",
    "print(\"------------------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.callbacks import EarlyStopping # https://keras.io/callbacks/#earlystopping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Earlystopping feature  helps us define the model's learning process, we can set the model to stop training if the validation accuracy or validation loss does not change by a threshold for n epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "earlystop = EarlyStopping(monitor='val_acc', min_delta=0.001, patience=10, \\\n",
    "                          verbose=1, mode='auto')\n",
    "callbacks_list = [earlystop]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 32584 samples, validate on 8146 samples\n",
      "Epoch 1/200\n",
      "32584/32584 [==============================] - 1s 40us/step - loss: 0.3485 - acc: 0.8392 - val_loss: 0.3098 - val_acc: 0.8588\n",
      "Epoch 2/200\n",
      "32584/32584 [==============================] - 1s 27us/step - loss: 0.2907 - acc: 0.8726 - val_loss: 0.2882 - val_acc: 0.8732\n",
      "Epoch 3/200\n",
      "32584/32584 [==============================] - 1s 24us/step - loss: 0.2716 - acc: 0.8853 - val_loss: 0.2711 - val_acc: 0.8809\n",
      "Epoch 4/200\n",
      "32584/32584 [==============================] - 1s 27us/step - loss: 0.2613 - acc: 0.8906 - val_loss: 0.2652 - val_acc: 0.8867\n",
      "Epoch 5/200\n",
      "32584/32584 [==============================] - 1s 24us/step - loss: 0.2579 - acc: 0.8921 - val_loss: 0.2615 - val_acc: 0.8889\n",
      "Epoch 6/200\n",
      "32584/32584 [==============================] - 1s 32us/step - loss: 0.2557 - acc: 0.8939 - val_loss: 0.2721 - val_acc: 0.8806\n",
      "Epoch 7/200\n",
      "32584/32584 [==============================] - 1s 38us/step - loss: 0.2559 - acc: 0.8922 - val_loss: 0.2598 - val_acc: 0.8912\n",
      "Epoch 8/200\n",
      "32584/32584 [==============================] - 1s 35us/step - loss: 0.2514 - acc: 0.8957 - val_loss: 0.2528 - val_acc: 0.8925\n",
      "Epoch 9/200\n",
      "32584/32584 [==============================] - 1s 32us/step - loss: 0.2496 - acc: 0.8956 - val_loss: 0.2529 - val_acc: 0.8925\n",
      "Epoch 10/200\n",
      "32584/32584 [==============================] - 1s 23us/step - loss: 0.2474 - acc: 0.8978 - val_loss: 0.2552 - val_acc: 0.8912\n",
      "Epoch 11/200\n",
      "32584/32584 [==============================] - 1s 27us/step - loss: 0.2496 - acc: 0.8965 - val_loss: 0.2504 - val_acc: 0.8950\n",
      "Epoch 12/200\n",
      "32584/32584 [==============================] - 1s 24us/step - loss: 0.2460 - acc: 0.8977 - val_loss: 0.2558 - val_acc: 0.8933\n",
      "Epoch 13/200\n",
      "32584/32584 [==============================] - 1s 28us/step - loss: 0.2474 - acc: 0.8974 - val_loss: 0.2540 - val_acc: 0.8925\n",
      "Epoch 14/200\n",
      "32584/32584 [==============================] - 1s 23us/step - loss: 0.2473 - acc: 0.8962 - val_loss: 0.2518 - val_acc: 0.8925\n",
      "Epoch 15/200\n",
      "32584/32584 [==============================] - 1s 27us/step - loss: 0.2478 - acc: 0.8978 - val_loss: 0.2503 - val_acc: 0.8957\n",
      "Epoch 16/200\n",
      "32584/32584 [==============================] - 1s 23us/step - loss: 0.2462 - acc: 0.8962 - val_loss: 0.2488 - val_acc: 0.8969\n",
      "Epoch 17/200\n",
      "32584/32584 [==============================] - 1s 27us/step - loss: 0.2458 - acc: 0.8968 - val_loss: 0.2505 - val_acc: 0.8963\n",
      "Epoch 18/200\n",
      "32584/32584 [==============================] - 1s 23us/step - loss: 0.2455 - acc: 0.8973 - val_loss: 0.2508 - val_acc: 0.8948\n",
      "Epoch 19/200\n",
      "32584/32584 [==============================] - 1s 29us/step - loss: 0.2449 - acc: 0.8980 - val_loss: 0.2508 - val_acc: 0.8966\n",
      "Epoch 20/200\n",
      "32584/32584 [==============================] - 1s 27us/step - loss: 0.2469 - acc: 0.8973 - val_loss: 0.2493 - val_acc: 0.8947\n",
      "Epoch 21/200\n",
      "32584/32584 [==============================] - 1s 35us/step - loss: 0.2445 - acc: 0.8991 - val_loss: 0.2494 - val_acc: 0.8980\n",
      "Epoch 22/200\n",
      "32584/32584 [==============================] - 1s 24us/step - loss: 0.2453 - acc: 0.8983 - val_loss: 0.2546 - val_acc: 0.8898\n",
      "Epoch 23/200\n",
      "32584/32584 [==============================] - 1s 31us/step - loss: 0.2490 - acc: 0.8962 - val_loss: 0.2692 - val_acc: 0.8822\n",
      "Epoch 24/200\n",
      "32584/32584 [==============================] - 1s 26us/step - loss: 0.2513 - acc: 0.8972 - val_loss: 0.2561 - val_acc: 0.8938\n",
      "Epoch 25/200\n",
      "32584/32584 [==============================] - 1s 27us/step - loss: 0.2527 - acc: 0.8952 - val_loss: 0.2562 - val_acc: 0.8930\n",
      "Epoch 26/200\n",
      "32584/32584 [==============================] - 1s 26us/step - loss: 0.2486 - acc: 0.8970 - val_loss: 0.2511 - val_acc: 0.8981\n",
      "Epoch 27/200\n",
      "32584/32584 [==============================] - 1s 29us/step - loss: 0.2469 - acc: 0.8972 - val_loss: 0.2556 - val_acc: 0.8917\n",
      "Epoch 28/200\n",
      "32584/32584 [==============================] - 1s 25us/step - loss: 0.2481 - acc: 0.8968 - val_loss: 0.2547 - val_acc: 0.8927\n",
      "Epoch 29/200\n",
      "32584/32584 [==============================] - 1s 27us/step - loss: 0.2471 - acc: 0.8979 - val_loss: 0.2492 - val_acc: 0.8957\n",
      "Epoch 30/200\n",
      "32584/32584 [==============================] - 1s 26us/step - loss: 0.2446 - acc: 0.8989 - val_loss: 0.2490 - val_acc: 0.8963\n",
      "Epoch 31/200\n",
      "32584/32584 [==============================] - 1s 31us/step - loss: 0.2496 - acc: 0.8981 - val_loss: 0.2587 - val_acc: 0.8905\n",
      "Epoch 00031: early stopping\n",
      "Confusion matrices:\n",
      "---------------------\n",
      "Confusion matrix - Train:\n",
      "[[30459  2593]\n",
      " [ 1796  5882]]\n",
      "Confusion matrix - Test:\n",
      "[[13025  1140]\n",
      " [  801  2490]]\n",
      "---------------------\n",
      "Evaluation metrics on train data for new model:\n",
      "------------------------------------\n",
      "Train Specificity:  0.921547863971923\n",
      "Train Recall:  0.7660849179473821\n",
      "Train Precision:  0.6940412979351033\n",
      "Train Accuracy:  0.8922415909648908\n",
      "------------------------------------\n",
      "Evaluation metrics on test data for new model:\n",
      "------------------------------------\n",
      "Test Specificity:  0.9195199435227673\n",
      "Test Recall:  0.7566089334548769\n",
      "Test Precision:  0.6859504132231405\n",
      "Test Accuracy:  0.8888061411549038\n",
      "------------------------------------\n"
     ]
    }
   ],
   "source": [
    "ann_es_hiddenExp = Sequential()\n",
    "\n",
    "# Adding more hidden layers\n",
    "ann_es_hiddenExp.add(Dense(64, input_dim=21, activation='sigmoid'))\n",
    "ann_es_hiddenExp.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "ann_es_hiddenExp.compile(loss='binary_crossentropy', optimizer='sgd', metrics=['accuracy'])\n",
    "\n",
    "ann_history=ann_es_hiddenExp.fit(X_train, y_train, epochs=200, batch_size=64,callbacks=callbacks_list,validation_split=0.2)\n",
    "\n",
    "# Predictions\n",
    "train_pred = ann_es_hiddenExp.predict_classes(X_train)\n",
    "test_pred = ann_es_hiddenExp.predict_classes(X_test)\n",
    "\n",
    "#Evaluation metrics\n",
    "confusion_matrix_train = confusion_matrix(y_train, train_pred)\n",
    "confusion_matrix_test = confusion_matrix(y_test, test_pred)\n",
    "\n",
    "print(\"Confusion matrices:\")\n",
    "print(\"---------------------\")\n",
    "print(\"Confusion matrix - Train:\")\n",
    "print(confusion_matrix_train)\n",
    "print(\"Confusion matrix - Test:\")\n",
    "print(confusion_matrix_test)\n",
    "print(\"---------------------\")\n",
    "\n",
    "# Metrics on train data\n",
    "#Accuracy\n",
    "accuracy_Train_hiddenExp = (confusion_matrix_train[0,0]+confusion_matrix_train[1,1])/(confusion_matrix_train[0,0]+confusion_matrix_train[0,1]+confusion_matrix_train[1,0]+confusion_matrix_train[1,1])\n",
    "#specificity or true negative rate (TNR)\n",
    "specificity_Train_hiddenExp = confusion_matrix_train[0,0]/(confusion_matrix_train[0,0]+confusion_matrix_train[0,1])\n",
    "#sensitivity, recall, hit rate, or true positive rate (TPR)\n",
    "recall_Train_hiddenExp = confusion_matrix_train[1,1]/(confusion_matrix_train[1,0]+confusion_matrix_train[1,1])\n",
    "#precision\n",
    "precision_Train_hiddenExp = confusion_matrix_train[1,1]/(confusion_matrix_train[0,1]+confusion_matrix_train[1,1])\n",
    "\n",
    "print(\"Evaluation metrics on train data for new model:\")\n",
    "print(\"------------------------------------\")\n",
    "print(\"Train Specificity: \",specificity_Train_hiddenExp)\n",
    "print(\"Train Recall: \",recall_Train_hiddenExp)\n",
    "print(\"Train Precision: \",precision_Train_hiddenExp)\n",
    "print(\"Train Accuracy: \",accuracy_Train_hiddenExp)\n",
    "print(\"------------------------------------\")\n",
    "\n",
    "\n",
    "# Metrics on test data\n",
    "#Accuracy\n",
    "accuracy_Test_hiddenExp = (confusion_matrix_test[0,0]+confusion_matrix_test[1,1])/(confusion_matrix_test[0,0]+confusion_matrix_test[0,1]+confusion_matrix_test[1,0]+confusion_matrix_test[1,1])\n",
    "#specificity or true negative rate (TNR)\n",
    "specificity_Test_hiddenExp = confusion_matrix_test[0,0]/(confusion_matrix_test[0,0]+confusion_matrix_test[0,1])\n",
    "#sensitivity, recall, hit rate, or true positive rate (TPR)\n",
    "recall_Test_hiddenExp = confusion_matrix_test[1,1]/(confusion_matrix_test[1,0]+confusion_matrix_test[1,1])\n",
    "#precision\n",
    "precision_Test_hiddenExp = confusion_matrix_test[1,1]/(confusion_matrix_test[0,1]+confusion_matrix_test[1,1])\n",
    "\n",
    "print(\"Evaluation metrics on test data for new model:\")\n",
    "print(\"------------------------------------\")\n",
    "print(\"Test Specificity: \",specificity_Test_hiddenExp)\n",
    "print(\"Test Recall: \",recall_Test_hiddenExp)\n",
    "print(\"Test Precision: \",precision_Test_hiddenExp)\n",
    "print(\"Test Accuracy: \",accuracy_Test_hiddenExp)\n",
    "print(\"------------------------------------\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### * history object stores the member history as a dictionary. The details of what happened during training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Let us plot the training and validation loss "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4VOX5//H3TQCRRbGA1YIsKrWyiRgVv6JBpdRdXKogVLFahMpXvvVnKy6tFmur4opSK3WriuJWEa2WWgsitRWCgAqIoAIiCgFlX0Pu3x/PCRnCJDNJZjKT5PO6rlyZOet9ZrvPs5znmLsjIiJSnnqZDkBERLKfkoWIiCSkZCEiIgkpWYiISEJKFiIikpCShYiIJKRkIdXCzHLMbKOZtU3lsplkZoeaWcr7nptZHzNbEvN8oZmdkMyyldjXI2Z2Q2XXL2e7vzOzJ1K9Xcmc+pkOQLKTmW2MedoY2AbsjJ5f6e7jK7I9d98JNE31snWBux+Wiu2Y2RXAIHfvHbPtK1Kxban9lCwkLnff9WMdnble4e7/LGt5M6vv7oXVEZuIVD9VQ0mlRNUMz5nZs2a2ARhkZseZ2X/NbK2ZfWVmY8ysQbR8fTNzM2sfPX86mv+GmW0ws/+YWYeKLhvNP83MPjGzdWb2gJn928wGlxF3MjFeaWaLzexbMxsTs26Omd1rZmvM7FPg1HJen5vMbEKpaWPN7J7o8RVmtiA6nk+js/6ytrXczHpHjxub2VNRbPOAo+Ls97Nou/PM7OxoelfgQeCEqIpvdcxre0vM+kOjY19jZhPN7MBkXptEzKxfFM9aM/uXmR0WM+8GM1thZuvN7OOYY+1pZu9H01ea2ehk9ydp4O7601+5f8ASoE+pab8DtgNnEU469gaOBo4llFgPBj4BhkfL1wccaB89fxpYDeQCDYDngKcrsez+wAbgnGjeNcAOYHAZx5JMjK8A+wLtgW+Kjx0YDswD2gAtgGnhKxR3PwcDG4EmMdteBeRGz8+KljHgZGAL0C2a1wdYErOt5UDv6PFdwFRgP6AdML/UshcCB0bvycVRDN+N5l0BTC0V59PALdHjvlGM3YFGwB+BfyXz2sQ5/t8BT0SPD4/iODl6j26IXvcGQGdgKXBAtGwH4ODo8UxgQPS4GXBspr8LdflPJQupiunu/qq7F7n7Fnef6e7vuXuhu38GjAPyyln/RXfPd/cdwHjCj1RFlz0TmOPur0Tz7iUklriSjPEP7r7O3ZcQfpiL93UhcK+7L3f3NcDt5eznM+AjQhID+CGw1t3zo/mvuvtnHvwLeAuI24hdyoXA79z9W3dfSigtxO73eXf/KnpPniEk+twktgswEHjE3ee4+1ZgJJBnZm1ilinrtSlPf2CSu/8reo9uB/YhJO1CQmLqHFVlfh69dhCSfkcza+HuG9z9vSSPQ9JAyUKq4ovYJ2b2AzP7m5l9bWbrgVFAy3LW/zrm8WbKb9Qua9nvxcbh7k44E48ryRiT2hfhjLg8zwADoscXE5JccRxnmtl7ZvaNma0lnNWX91oVO7C8GMxssJnNjap71gI/SHK7EI5v1/bcfT3wLdA6ZpmKvGdlbbeI8B61dveFwP8jvA+romrNA6JFLwM6AQvNbIaZnZ7kcUgaKFlIVZTuNvow4Wz6UHffB/gNoZolnb4iVAsBYGbG7j9upVUlxq+Ag2KeJ+ra+xzQJzozP4eQPDCzvYEXgT8QqoiaA/9IMo6vy4rBzA4GHgKGAS2i7X4cs91E3XxXEKq2irfXjFDd9WUScVVku/UI79mXAO7+tLsfT6iCyiG8Lrj7QnfvT6hqvBt4ycwaVTEWqSQlC0mlZsA6YJOZHQ5cWQ37fA3oYWZnmVl9YATQKk0xPg/8n5m1NrMWwHXlLezuK4HpwOPAQndfFM3aC2gIFAA7zexM4JQKxHCDmTW3cB3K8Jh5TQkJoYCQN68glCyKrQTaFDfox/EscLmZdTOzvQg/2u+4e5kltQrEfLaZ9Y72/UtCO9N7Zna4mZ0U7W9L9LeTcAA/MbOWUUlkXXRsRVWMRSpJyUJS6f8BlxJ+CB4mnFmnVfSDfBFwD7AGOASYTbguJNUxPkRoW/iQ0Pj6YhLrPENosH4mJua1wC+AlwmNxBcQkl4ybiaUcJYAbwBPxmz3A2AMMCNa5gdAbD3/m8AiYKWZxVYnFa//d0J10MvR+m0J7RhV4u7zCK/5Q4REdipwdtR+sRdwJ6Gd6WtCSeamaNXTgQUWetvdBVzk7turGo9UjoUqXpHawcxyCNUeF7j7O5mOR6S2UMlCajwzO9XM9o2qMn5N6GEzI8NhidQqShZSG/QCPiNUZZwK9HP3sqqhRKQSVA0lIiIJqWQhIiIJ1ZqBBFu2bOnt27fPdBgiIjXKrFmzVrt7ed3NgVqULNq3b09+fn6mwxARqVHMLNFIBICqoUREJAlKFiIikpCShYiIJFRr2ixEpHrt2LGD5cuXs3Xr1kyHIklo1KgRbdq0oUGDsoYGK5+ShYhUyvLly2nWrBnt27cnDPYr2crdWbNmDcuXL6dDhw6JV4hD1VDjx0P79lCvXvg/fnyiNUQE2Lp1Ky1atFCiqAHMjBYtWlSpFFi3Sxbjx8OQIbB5c3i+dGl4DjCwyoNtitR6ShQ1R1Xfq7pdsrjxxpJEUWzz5jBdRER2qdvJYtmyik0XkayxZs0aunfvTvfu3TnggANo3br1rufbtyd324vLLruMhQsXlrvM2LFjGZ+i6ulevXoxZ86clGyrutXtaqi2bUPVU7zpIpJa48eHUvuyZeE7dtttVarubdGixa4f3ltuuYWmTZty7bXX7raMu+Pu1KsX/7z48ccfT7ifq666qtIx1iZ1u2Rx223QuPHu0xo3DtNFJHWK2weXLgX3kvbBNHQoWbx4MV26dGHo0KH06NGDr776iiFDhpCbm0vnzp0ZNWrUrmWLz/QLCwtp3rw5I0eO5IgjjuC4445j1apVANx0003cd999u5YfOXIkxxxzDIcddhjvvvsuAJs2beL888/niCOOYMCAAeTm5iYsQTz99NN07dqVLl26cMMNNwBQWFjIT37yk13Tx4wZA8C9995Lp06dOOKIIxg0aFDKX7Nk1O1kMXAgjBsH7dqBWfg/bpwat0VSrZrbB+fPn8/ll1/O7Nmzad26Nbfffjv5+fnMnTuXN998k/nz5++xzrp168jLy2Pu3Lkcd9xxPPbYY3G37e7MmDGD0aNH70o8DzzwAAcccABz585l5MiRzJ49u9z4li9fzk033cSUKVOYPXs2//73v3nttdeYNWsWq1ev5sMPP+Sjjz7ikksuAeDOO+9kzpw5zJ07lwcffLCKr07l1O1kASExLFkCRUXhvxKFSOpVc/vgIYccwtFHH73r+bPPPkuPHj3o0aMHCxYsiJss9t57b0477TQAjjrqKJYsWRJ32+edd94ey0yfPp3+/fsDcMQRR9C5c+dy43vvvfc4+eSTadmyJQ0aNODiiy9m2rRpHHrooSxcuJARI0YwefJk9t13XwA6d+7MoEGDGD9+fKUvqqsqJQsRSb+y2gHT1D7YpEmTXY8XLVrE/fffz7/+9S8++OADTj311LjXGzRs2HDX45ycHAoLC+Nue6+99tpjmYreRK6s5Vu0aMEHH3xAr169GDNmDFdeeSUAkydPZujQocyYMYPc3Fx27txZof2lgpKFiKRfBtsH169fT7Nmzdhnn3346quvmDx5csr30atXL55//nkAPvzww7gll1g9e/ZkypQprFmzhsLCQiZMmEBeXh4FBQW4Oz/+8Y/57W9/y/vvv8/OnTtZvnw5J598MqNHj6agoIDNpav0qkHd7g0lItWjuHo3hb2hktWjRw86depEly5dOPjggzn++ONTvo///d//5ZJLLqFbt2706NGDLl267KpCiqdNmzaMGjWK3r174+6cddZZnHHGGbz//vtcfvnluDtmxh133EFhYSEXX3wxGzZsoKioiOuuu45mzZql/BgSqTX34M7NzXXd/Eik+ixYsIDDDz8802FkhcLCQgoLC2nUqBGLFi2ib9++LFq0iPr1s+t8PN57Zmaz3D030brZdSQiIjXQxo0bOeWUUygsLMTdefjhh7MuUVRV7ToaEZEMaN68ObNmzcp0GGmlBm4REUlIyUJERBJKa7Iws1PNbKGZLTazkXHmDzWzD81sjplNN7NOpea3NbONZnZt6XVFRKT6pC1ZmFkOMBY4DegEDCidDIBn3L2ru3cH7gTuKTX/XuCNdMUoIiLJSWfJ4hhgsbt/5u7bgQnAObELuPv6mKdNgF39eM2sH/AZMC+NMYpIDdW7d+89LrC77777+PnPf17uek2bNgVgxYoVXHDBBWVuO1FX/Pvuu2+3i+NOP/101q5dm0zo5brlllu46667qrydVEtnsmgNfBHzfHk0bTdmdpWZfUooWVwdTWsCXAf8trwdmNkQM8s3s/yCgoKUBS4i2W/AgAFMmDBht2kTJkxgwIABSa3/ve99jxdffLHS+y+dLF5//XWaN29e6e1lu3Qmi3j38NvjCkB3H+vuhxCSw03R5N8C97r7xvJ24O7j3D3X3XNbtWpV5YBFpOa44IILeO2119i2bRsAS5YsYcWKFfTq1WvXdQ89evSga9euvPLKK3usv2TJErp06QLAli1b6N+/P926deOiiy5iy5Ytu5YbNmzYruHNb775ZgDGjBnDihUrOOmkkzjppJMAaN++PatXrwbgnnvuoUuXLnTp0mXX8OZLlizh8MMP52c/+xmdO3emb9++u+0nnjlz5tCzZ0+6devGueeey7fffrtr/506daJbt267BjB8++23d9386cgjj2TDhg2Vfm3jKr45SKr/gOOAyTHPrweuL2f5esC66PE7wJLoby3wDTC8vP0dddRRLiLVZ/78+bs9z8vb82/s2DBv06b48x9/PMwvKNhzXjJOP/10nzhxoru7/+EPf/Brr73W3d137Njh69ati7Zd4IcccogXFRW5u3uTJk3c3f3zzz/3zp07u7v73Xff7Zdddpm7u8+dO9dzcnJ85syZ7u6+Zs0ad3cvLCz0vLw8nzt3rru7t2vXzgsKCnbFUvw8Pz/fu3Tp4hs3bvQNGzZ4p06d/P333/fPP//cc3JyfPbs2e7u/uMf/9ifeuqpPY7p5ptv9tGjR7u7e9euXX3q1Knu7v7rX//aR4wY4e7uBx54oG/dutXd3b/99lt3dz/zzDN9+vTp7u6+YcMG37Fjxx7bLv2eubsD+Z7Eb3o6SxYzgY5m1sHMGgL9gUmxC5hZx5inZwCLANz9BHdv7+7tgfuA37t7ZgZxF5GsFVsVFVsF5e7ccMMNdOvWjT59+vDll1+ycuXKMrczbdq0XTcV6tatG926dds17/nnn6dHjx4ceeSRzJs3L+EggdOnT+fcc8+lSZMmNG3alPPOO4933nkHgA4dOtC9e3eg/GHQIdxfY+3ateTl5QFw6aWXMm3atF0xDhw4kKeffnrXleLHH38811xzDWPGjGHt2rUpv4I8bVdwu3uhmQ0HJgM5wGPuPs/MRhEy2SRguJn1AXYA3wKXpiseEUmvqVPLnte4cfnzW7Ysf35Z+vXrxzXXXMP777/Pli1b6NGjBwDjx4+noKCAWbNm0aBBA9q3bx93WPJYZnvWnH/++efcddddzJw5k/3224/Bgwcn3I6XM95e8fDmEIY4T1QNVZa//e1vTJs2jUmTJnHrrbcyb948Ro4cyRlnnMHrr79Oz549+ec//8kPfvCDSm0/nrReZ+Hur7v79939EHe/LZr2myhR4O4j3L2zu3d395PcfY+eT+5+i7tnX9cAEcm4pk2b0rt3b37605/u1rC9bt069t9/fxo0aMCUKVNYunRpuds58cQTGR/d4vWjjz7igw8+AMLw5k2aNGHfffdl5cqVvPFGSU/+Zs2axW0XOPHEE5k4cSKbN29m06ZNvPzyy5xwwgkVPrZ9992X/fbbb1ep5KmnniIvL4+ioiK++OILTjrpJO68807Wrl3Lxo0b+fTTT+natSvXXXcdubm5fPzxxxXeZ3k0NpSI1GgDBgzgvPPO261n1MCBAznrrLPIzc2le/fuCc+whw0bxmWXXUa3bt3o3r07xxxzDBDuenfkkUfSuXPnPYY3HzJkCKeddhoHHnggU6ZM2TW9R48eDB48eNc2rrjiCo488shyq5zK8pe//IWhQ4eyefNmDj74YB5//HF27tzJoEGDWLduHe7OL37xC5o3b86vf/1rpkyZQk5ODp06ddp1179U0RDlIlIpGqK85qnKEOUaG0pERBJSshARkYSULESk0mpLNXZdUNX3SslCRCqlUaNGrFmzRgmjBnB31qxZQ6NGjSq9DfWGEpFKadOmDcuXL0fjstUMjRo1ok2bNpVeX8lCRCqlQYMGdOjQIdNhSDVRNZSIiCSkZCEiIgkpWYiISEJKFiIikpCShYiIJKRkISIiCSlZiIhIQkoWIiKSkJKFiIgkpGQhIiIJKVmIiEhCShYiIpKQkoWIiCSkZCEiIgkpWYiISEJKFiIikpCShYiIJKRkISIiCSlZiIhIQkoWIiKSkJKFiIgklNZkYWanmtlCM1tsZiPjzB9qZh+a2Rwzm25mnaLpPzSzWdG8WWZ2cjrjFBGR8qUtWZhZDjAWOA3oBAwoTgYxnnH3ru7eHbgTuCeavho4y927ApcCT6UrThERSSydJYtjgMXu/pm7bwcmAOfELuDu62OeNgE8mj7b3VdE0+cBjcxsrzTGKiIi5aifxm23Br6Ieb4cOLb0QmZ2FXAN0BCIV910PjDb3bfFWXcIMASgbdu2KQhZRETiSWfJwuJM8z0muI9190OA64CbdtuAWWfgDuDKeDtw93Hunuvuua1atUpByCIiEk86k8Vy4KCY522AFWUsC6Gaql/xEzNrA7wMXOLun6YlQhERSUo6k8VMoKOZdTCzhkB/YFLsAmbWMebpGcCiaHpz4G/A9e7+7zTGKCIiSUhbsnD3QmA4MBlYADzv7vPMbJSZnR0tNtzM5pnZHEK7xaXF04FDgV9H3WrnmNn+6YpVRETKZ+57NCPUSLm5uZ6fn5/pMEREahQzm+XuuYmW0xXcIiKSkJKFiIgkpGQhIiIJKVmIiEhCShYiIpKQkoWIiCSkZCEiIgkpWYiISEJKFiIikpCShYiIJKRkISIiCSlZiIhIQkoWIiKSkJIFsHkzFBRkOgoRkexV55NFURG0aQOjRmU6EhGR7FXnk0W9enD00TB1aqYjERHJXnU+WQDk5cFHH8Hq1ZmOREQkOylZAL17h//TpmU0DBGRrKVkAeTmwt57w9tvZzoSEZHsVD/TAWSDhg3h8cehU6dMRyIikp2ULCIXXZTpCEREspeqoSLbt8Nf/wpz52Y6EhGR7KNkEXGHgQPhiScyHYmISPZRsojstRccd5yutxARiUfJIkbv3qEa6ttvMx2JiEh2UbKIkZcXqqPeeSfTkYiIZBclixjHHhuqo2bMyHQkIiLZRV1nYzRqBIsWhYEFRUSkhJJFKQcdlOkIRESyT1qroczsVDNbaGaLzWxknPlDzexDM5tjZtPNrFPMvOuj9Raa2Y/SGWeslSth8GCYMqW69igikv3SlizMLAcYC5wGdAIGxCaDyDPu3tXduwN3AvdE63YC+gOdgVOBP0bbS7t99oFnn4XXX6+OvYmI1AxJJQszO8TM9ooe9zazq82seYLVjgEWu/tn7r4dmACcE7uAu6+PedoE8OjxOcAEd9/m7p8Di6Ptpd3ee0PPnhpUUEQkVrIli5eAnWZ2KPAo0AF4JsE6rYEvYp4vj6btxsyuMrNPCSWLqyu47hAzyzez/IIU3hc1Lw9mzYL16xMvKyJSFySbLIrcvRA4F7jP3X8BHJhgHYszzfeY4D7W3Q8BrgNuquC649w9191zW7VqlSCc5OXlhdut/vvfKdukiEiNlmyy2GFmA4BLgdeiaQ0SrLMciO1b1AZYUc7yE4B+lVw3pY47Djp3hi1bqmuPIiLZLdlkcRlwHHCbu39uZh2ApxOsMxPoaGYdzKwhocF6UuwCZtYx5ukZwKLo8SSgv5ntFe2rI1Btl8o1bhxus3reedW1RxGR7JbUdRbuPp+oPcHM9gOaufvtCdYpNLPhwGQgB3jM3eeZ2Sgg390nAcPNrA+wA/iWUHIhWu55YD5QCFzl7jsrdYRVUFQUhv/IqZZ+WCIi2cvc92gK2HMhs6nA2YTkMgcoAN5292vSGl0F5Obmen5+fsq2N2cOnHIKPPMM/KjarvIQEaleZjbL3XMTLZdsNdS+UTfX84DH3f0ooE9VAsx2hx4aekOpC62ISPLJor6ZHQhcSEkDd63WtCnk5ur+FiIikHyyGEVoe/jU3Wea2cGUNEbXWr17w8yZsGlTpiMREcmspJKFu7/g7t3cfVj0/DN3Pz+9oWVeXh4UFsK772Y6EhGRzEp2uI82Zvayma0ys5Vm9pKZ1fqBvI8/Hq69ViPRiogkWw31OOHah+8Rht14NZpWqzVrBqNHww9+kOlIREQyK9lk0crdH3f3wujvCSB142tksW3bYPp0Xc0tInVbsslitZkNMrOc6G8QsCadgWWLt96CE05Qu4WI1G3JJoufErrNfg18BVxAGAKk1uvVC+rV0/UWIlK3Jdsbapm7n+3urdx9f3fvR7hAr9bbZx/o0QPefmEltG8fMkf79jB+fKZDExGpNlW5U17WDPWRbr33n89/P27OlqUrw2BRS5fCkCFKGCJSZ1QlWcS750StlDfzLrazF+9xbMnEzZvhxhszF5SISDVKatTZMiQegbCW6F3wAu/wCUczc/cZy5ZlJiARkWpWbrIwsw3ETwoG7J2WiLJQ03Yt6LU0zm3z2rat/mBERDKg3Good2/m7vvE+Wvm7lUpldQst93GvEZH8SvuYCt7hWmNG8Ntt2U2LhGRalKVNou6Y+BAPh12F6P5FTM4Ftq1g3HjYODATEcmIlItlCySdMKve2MGU3/7NixZokQhInWKkkWS9tsPjj4aXqsTd/MQEdmdkkUF9OsX7m/x5ZeZjkREpHopWVTAOeeEEsbHH2c6EhGR6lV3ejSlwOGHw6pVUF+vmojUMSpZVIBZSBTuUFSU6WhERKqPkkUFLVsGHTvCc89lOhIRkeqjZFFBrVvDxo3wyiuZjkREpPooWVRQTg6cdRa8/nq4i56ISF2gZFEJ/frBhg0wdWqmIxERqR5KFpVwyinQpImqokSk7lAn0Epo1CiMIdixY6YjERGpHkoWlTRiRKYjEBGpPmmthjKzU81soZktNrORceZfY2bzzewDM3vLzNrFzLvTzOaZ2QIzG2NmWXdnvoULYdq0TEchIpJ+aUsWZpYDjAVOAzoBA8ysU6nFZgO57t4NeBG4M1r3f4DjgW5AF+BoIC9dsVbWsGHhT0SktktnyeIYYLG7f+bu24EJwDmxC7j7FHffHD39L9CmeBbQCGgI7AU0AFamMdZK6dcP5s+HRYsyHYmISHqlM1m0Br6Ieb48mlaWy4E3ANz9P8AU4Kvob7K7L0hTnJV2TpT61CtKRGq7dCaLeG0M8e7njZkNAnKB0dHzQ4HDCSWN1sDJZnZinPWGmFm+meUXFBSkLPBktWsH3bvDxInVvmsRkWqVzmSxHDgo5nkbYEXphcysD3AjcLa7F18TfS7wX3ff6O4bCSWOnqXXdfdx7p7r7rmtWrVK+QEko18/yM+HdesysnsRkWqRzmQxE+hoZh3MrCHQH5gUu4CZHQk8TEgUq2JmLQPyzKy+mTUgNG5nXTUUwPDh8PXXsO++mY5ERCR90nadhbsXmtlwYDKQAzzm7vPMbBSQ7+6TCNVOTYEXop6xy9z9bELPqJOBDwlVV39391fTFWtVtGiR6QhERNLP3OM2I9Q4ubm5np+fn5F9T5kCt94KkyZB06YZCUFEpFLMbJa75yZaTmNDpYBZSBj/+EemIxERSQ8lixTo1Qu+8x11oRWR2kvJIgXq14czz4RXX4XCwkxHIyKSekoWKXLOOfDtt/DOO5mOREQk9ZQsUuRHP4K+fcOd9EREahsNUZ4iTZrA5MmZjkJEJD1UskixNWtg9epMRyEiklpKFim0bh0ceCA8+GCmIxERSS0lixTad184psMqXvn9PKhXD9q3h/HjMx2WiEiVKVmk0vjx9Pv8Pubs6MwSbwtLl8KQIdWaMNxh48Zq252I1BFKFql0442cs+MFACZxdpi2eTPceGPad71hQ7gRU9++MHhw2ncnInWMkkUqLVtGRxbThQ8Zz8CSm3csW5b2XT/wAHTtGq4kf/nlatmliNQhShap1LYtAH/hUp7h4pK7P0XT02X9erj7bjj9dBg9OoxVNXZsWncpInWMkkUq3XYbNG5MD2ZzCJ/hwB8bjGDzb25P624ffBC++QZuvjnkpXPPhT//GTZtSutuRaQOUbJIpYEDYdy4cL9VM2YccA7DC+/lopf7p23MqOJSxZlnQm40yPCIEWHokWefTc8+RaTuUbJItYEDYckSKCri2K8mMnas8dpr8LOfhZ5KqTZjBmzZEkoVxY4/Hl56CQYNSv3+RFJl1aowrP/OnZmORJKhZJFmw4bBLbfAE0/AyJGp336fPvDllyWlCghtFuedB40apX5/Iqny9NNhTLU//AHefz/T0UgiShbV4De/gZ//HO69FxYuTN12v/gilFb22y/+/EcfhauuSt3+RFLpySehSxe45x64445MRyOJKFlUAzMYMyZUGR12WGq2uWEDdO8O119f9jLLlsFDD8HixanZp0iqfPABzJ0LV14Zrgv661/h668zHZWUR8mimuTkhB93CA3Pr75ate0V94C64IKylxk6NNyYSWNVSbZ56ilo0AD69w+f08LCUBKW7KVkUc127oT774cLL4Tp0yu3jQ0b4K67du8BFc+BB4b9PPZY6DUlki2mTIEzzoCWLeH734dTTgkdCdXYnb2ULKpZTk4oVbRrF37sP/yw4tuIva4ikREjQnJ54omK70ckXd57LySHYkOHhg4ZX3yRuZikfEoWGdCqVbhRUtOmkJcHL76Y/LruoRdJolJFsaOPhquvhs6dKx+vSCq5h5OmVq1Kpp13Hny6AldPAAAVhklEQVT8cRioWbKTkkWGtGsHb78NHTvC558nv54ZzJwJf/pT8uvcf38o5otk2vr1odpp4sTdp9erFz7bmzfD2rWZiU3Kp2SRQYccEtotrrkmPP/73+Gdd8pefuvW0BDYuDG0bl2xfa1YEdouRDLppZdC77wDD9xz3qZN4SRK3Wizk5JFhjVoEIrk7uHivbw8uOEG2L59z2XvuQcOPzzcka+inngCLr88DGMukilPPhlK08ccs+e8Jk3C6AOPPgrbtlV/bFI+JYssYQZvvgk//Wm4ovW440IdbrHiMaAOOyzcka+ihgwJDYgPPJC6mEUqYulSmDoVLrkkfN7jGTYMCgrCdReSXZQsMmX8+NCaF3P71WbN4JFHwv0oli6FHj1g0aKweEV6QMXTsmUYturJJ8MggyLV7amnwv/yxiz74Q9D9exDD1VPTJI8JYtMGD8+nOovXRrqn0rdfrVfv9Cl9re/hUMPDaWK228P/dKPPrryu7366tCA+MgjKToOkQro3RtGjSq/x1O9euGq7nfegU8+qa7IJBnm6RgKtXjjZqcC9wM5wCPufnup+dcAVwCFQAHwU3dfGs1rCzwCHAQ4cLq7LylrX7m5uZ6fn5+Ow0i99u1DgiitXbswYm0p//pX6M00Y0bVkgWEM7dOnUIPKZFs9M03oRH86KPLrq6S1DGzWe6esCN+2pKFmeUAnwA/BJYDM4EB7j4/ZpmTgPfcfbOZDQN6u/tF0bypwG3u/qaZNQWK3H1zWfurUcmiXr3445WbQVHRHpMXL4aPPgoljqoqLAxDgIhUp5dfDl1mdb1P9kk2WaSzGuoYYLG7f+bu24EJwDmxC7j7lJgE8F+gDYCZdQLqu/ub0XIby0sUNU5Zt1ktY/qhh6YmUUBJoohXsBFJh23b4Iorwo0kk7VlS7gHTHE7h2ReOpNFayD24v3l0bSyXA68ET3+PrDWzP5qZrPNbHRUUqkdotuv7qZx44p9m6rghRdCTdi771bL7qSOe/31ULV0ySXJr9OoURgS5N5703PTMKm4dCaLeLWNcd92MxsE5AKjo0n1gROAa4GjgYOBwXHWG2Jm+WaWX1BQkIqYq0ep26/Srl14PnBgtez+Rz8Ku7z44vAlFkmnJ5+EAw4IN+pKllnoRjt7dmirk8xLZ7JYTmicLtYGWFF6ITPrA9wInO3u22LWnR1VYRUCE4Eepdd193Hunuvuua1iB5qpCWJuv8qSJWUnijhdbKtqn33g+efDVd2XXhq3mUQkJVavhr/9LXy8K9pWNmhQGD9N3WizQzqTxUygo5l1MLOGQH9gUuwCZnYk8DAhUawqte5+ZlacAU4G6t61xwm62FbFMceEK8Jfew1Gj068vEhlvP9+SBIVqYIq1qxZSBjPPacScFmqs4oubf1i3L3QzIYDkwldZx9z93lmNgrId/dJhGqnpsALFvrILXP3s919p5ldC7xlYcYs4M/pijVr3XhjuDAi1ubNYXoKqqyuuioMStiiRZU3JRJX376walUoIVTGz38ekk284W8k3GWwfftwTVa6pfU6i+pUo7rOJquCXWyTNn58SDjLloUeWLfdBgMH4q5+7ZI66qadXu+9Bz17wm9+U7VkkQ1dZ6WqKtLFNtm2jTKqtiYMn8655+pOZZI6N98cqjt37KjadtzDcP5z58afX1QUuudu3BiGsqnq/moCd7j2Wvjud+GXv6yefSpZZLNku9hWpG2jjKqtzRNe4ZVX4NZbU3sI6bZpE/znP+EsVrJHUVG4RqJlyzCyclVs3w7nnx9GpG3ZEpo3DyPUvvBCmD91auhq26wZfOc7cPDBtX905YkTw+0NRo2qfBVfRakaKtuVUWW0m4oMH1JG1ZZjXHZpEU8+Ge6r0bdvyo4gbdavD3G+916469rDD8O552Y6KoFwj+2TT4Znn4X+/au+vYkTQ6+qBg1K/gYOhCOPDB/9Z54JVV45OSUdNj75JCSQVHj77TBeVZcu4a9Dh7CvTHAvuRL+gw+qXtWXbDUU7l4r/o466iivs8zcw2do9z+zPZdt1y7+su3a+aZN7l26uLds6b58eWpD/PJL92XLwuOtW92//rpq21u/3v1//se9fn333//e/aKL3GfPDvOmTnW/8kr3f/7TfceOqu1HKmfwYPd99nHfvLn69/3JJ+7PPlu1bRQVuT/wQPgsubvfeuvuX5dGjdx79HDfsCHMX7w49d+Z8syb5/6f/6RmW4QORwl/YzP+I5+qvzqdLMpJAHt4+mn3xo13X65x4zDd3RcscG/SxP3++6seVlGR+7vvuvfvH37UL7ssTL/8cvf27d0//rjy237iCfecHPeXXtpz3p/+VHKIrVq5Dx3q/tZb7jt3Vn5/kryCAve99w7vc6b97W/u06ZVbJ3t292HDQufn8GDS6Zv2OA+Y4b7Y4+5X3ON+wUXlMzr3z8sf+utqYm7LOn4DCtZ1CUJEkDc5du1CyWPdu32WK64BFAVzz3nftRRIZR99w1frsWLw7yZM8OPeIsWVTs7WrCg7HmbNrm/+KL7hReGl+KAA9wLC8O8LVsqv09JzgMPuH/+eTXvtNTnuvDJ8d69e0hcb76Z3Ca++cb9lFPC5/ZXvyr5zCQya1ZJwvj97yt9BAldf334TCcbVzKULOqaBAmgMvLz3SdPTn75r74KpQl396uvdj/8cPc//rGkqB5r0SL3Qw4JX+RJk5Lb/qZN7uefH+KqiE2bSqqoduxw79DB/cc/DmecxfFK1f373+HMOyPKOGH6euyL3rWr+157ub/6avmbWLnS/fvfd2/QwP3xxyseQmGh+8UXh13feWeljqJcS5eG6q9Bg1K7XSULqZKiIvfjj3dv3jyczVx7bUgA774b5n/ySWgnOPdc9zPOcO/dO1Q1TZkS5m/alPiHeOVK99zcUMJYt678ZTdvdu/TJ+TCZ56p/HGtXx+OpXnz8Ok/4gj3Rx7JTN16bTJlSqi+7NkzQwm4nKrYNWvC56x+fffnny97Ezt3hrauilZbxdqxI3wvevZ037at8tuJ5yc/CUlv6dLUblfJQqrs03smetucL7w+272xbfLmjbf6X/4S5n3wgfthh7l37Roa+nr2DFVNFf0gb9gQivDu4Ucm3g/Nli3uP/pRSBRPPFG1Yyq2aZP7uHEhfnB/442SGKRi/v73cMbbuXMoXaZcMqXmBJ081q4NJz/Dh++56qOPprbKbMeOcFLinrrqolmzwqFcd11qthdLyUKqpqLtIClwxx3uP/vZ7j2Ytm51P/30sPtHH039PouK3KdPL2k4vP569379wmGuWpX6/dU2kya5N2zo3r17aNhOuWQ/h0l08ti0qeR93rAhfM5GjAiLjRiR+tA3bQrtH2PHVn1b55wTSuBr11Z9W6UpWUjVVKSHVYrceGPYxVlnhS+ae0gWZ53l/vDDadvtbv7wB/f99/ddJ6W5ue533109+66JLrjA/eij3desSdMOkv0cVuDkZvly94MOcu/WLSz2i1+ktsG42LZt7mefHfZR1c/vqlUl3XhTTclCqqYi126ksHF97Fh3syI/quFcX8Sh7u3aedFT6SvNxLNzZ+ix9bvfuffqVdLlt6jIfcgQ9z//OTU9xkr75JNwpv7MM6GK7O67Q0cA99BAP3BgOMM8+eRQpXL88SFO93BNSfG02L9588L8rVtTW8W2fXvJdhO1N1VJGj6H27aFjhL164fXOZ22bg1tehDaxiqqsDBBd9kUfPeULKRq0nBGl5Snn/aXGvYP10iw0rfSMO3VX4kU/8iuXOnepk3JYXbqFNppPvyw8tsurnKbMiX+y/3CC2H+22+7H3xwOBs+/viQME45paS9Z8qU8Lz0X3H34rvvDt2HL7nEffz4qlWxPfpo6BiwenXlt+Huyf3QpamEW1SUgviTtGWL+6mnhsOs6Mf44YdDyS1urCn67ilZSNWksK64QqLt5dPDX+GslP04pEpRkftHH4Uf3z59Qn19cQ+bRYvc7703/ECXdxY/b577LbeEBuEbbwzTtm1zHzMmdD1dsMD9iy9C/XSqqkf+8Y9wHUCLFiUv6bHH7rn9nTvD2fD69SWNtO7un30W4r7nnrBu374lVYWVkuznKx1tZ2noZp7Ili2hZPjRR8mvs2GD+3e/G04O4n6eUvTdU7KQqktBL5QKS/X20mzTppKL/B56qCTctm1DldVLL5XMv+OOUBopPpwTTghn+dWpsDAkpFtv3b1nUJ8+4Yr42Je8b9+S+W3blkw/66yQUKqkoqMOpOrHPV0dNyoQ4/r17ieeGHr4nXtuuDbjiivcX389zF+3zn30aPcBA0J4//1vGRtK0XdFyUKqR6q/9OmodkjHmWQZ2/z88zDcSL9+7s2ahXrx4jP0Cy90z8tzf/DBMFZWtigqcr/5Zvcbbgglnt//PvxYvfhiyTIvv+w+YUL4QStur6iSTJ0UpOvzVYEE9M037iedFEp2XbuGi1O/972SIXYWLSrZzMUXp/9YlCykeqS6OiENbSCpHAqlItvcvr3kynH39PS4qbIMVMm4e0Z627l7ejpupPiEaefOUAW1cmWCz4zaLJQsapxUlxhS+QOWwkEWK7XNbJaBa2kyvu90dNxINgFlafuLkoVkl3RUO6S6TSXZH5J0nJ1m4gw/00kvE8ecjhOCZJfN9OtdBiULyS6p/qKk40ufbBJI9dlpRc44U/kDm652g0xVbSUr1ScZyb5/Wdp5Q8lCskuqi+DpqE5I9TZTfcaZ6qSSBY29Wauir00GrxmpKiULyT6ZOiuuSFVQKn+Mk42xJpRokj3mLP1BrLB0tS9kogSZgJKF1G7p+lHKRON6qttK0tGZoIZXtVRKNXa53mOZaiydKVlI7VYTqjtSfYafjgb4ZNXwRtwapZpfQyULqf2yvSHVPbW9oTLZtTeT3UPrmmounSlZiNRGmarGyNR1MnVRlpYs6iEiNcfAgbBkCRQVhf8DB8ZfZtw4aNcOzML/cePiL5us226Dxo13n9a4cZhemRilbBV5rauRkoVIbZTqH+x0JCCJL0tfawulkJovNzfX8/PzMx2GiEiNYmaz3D030XJpLVmY2almttDMFpvZyDjzrzGz+Wb2gZm9ZWbtSs3fx8y+NLMH0xmniIiUL23JwsxygLHAaUAnYICZdSq12Gwg1927AS8Cd5aafyvwdrpiFBGR5KSzZHEMsNjdP3P37cAE4JzYBdx9irtvjp7+F2hTPM/MjgK+C/wjjTGKiEgS0pksWgNfxDxfHk0ry+XAGwBmVg+4G/hl2qITEZGk1U/jti3OtLit6WY2CMgF8qJJPwded/cvzOJtZtd6Q4AhAG3btq1SsCIiUrZ0JovlwEExz9sAK0ovZGZ9gBuBPHffFk0+DjjBzH4ONAUamtlGd9+tkdzdxwHjou0UmNnSOHG0BFZX9WCyQG05DtCxZKPachygY6modokXSWPXWTOrD3wCnAJ8CcwELnb3eTHLHElo2D7V3ReVsZ3BhEbw4ZWMIz+ZbmHZrrYcB+hYslFtOQ7QsaRL2tos3L0QGA5MBhYAz7v7PDMbZWZnR4uNJpQcXjCzOWY2KV3xiIhI5aWzGgp3fx14vdS038Q87pPENp4Ankh1bCIikry6MNzHuEwHkCK15ThAx5KNastxgI4lLWrNcB8iIpI+daFkISIiVaRkISIiCdXaZJFoEMOaxMyWmNmHUY+xGjW0rpk9ZmarzOyjmGnfMbM3zWxR9H+/TMaYjDKO45ZooMs50d/pmYwxWWZ2kJlNMbMFZjbPzEZE02vi+1LWsdSo98bMGpnZDDObGx3Hb6PpHczsveg9ec7MGmYsxtrYZhENYvgJ8EPCxYEzgQHuPj+jgVWSmS0hXGtS4y40MrMTgY3Ak+7eJZp2J/CNu98eJfL93P26TMaZSBnHcQuw0d3vymRsFWVmBwIHuvv7ZtYMmAX0AwZT896Xso7lQmrQe2NhqIom7r7RzBoA04ERwDXAX919gpn9CZjr7g9lIsbaWrJIOIihVA93nwZ8U2ryOcBfosd/IXy5s1oZx1EjuftX7v5+9HgD4Tqo1tTM96WsY6lRojucboyeNoj+HDiZcOEyZPg9qa3JoqKDGGY7B/5hZrOi8bBquu+6+1cQvuzA/hmOpyqGR/djeawmVNuUZmbtgSOB96jh70upY4Ea9t6YWY6ZzQFWAW8CnwJrowucIcO/Y7U1WSQ9iGENcby79yDcG+SqqEpEMu8h4BCgO/AVYaTkGsPMmgIvAf/n7uszHU9VxDmWGvfeuPtOd+9OGEfvGODweItVb1QlamuySGoQw5rC3VdE/1cBLxM+SDXZyqiuubjOeVWG46kUd18ZfcGLgD9Tg96XqF78JWC8u/81mlwj35d4x1KT3xt3XwtMBXoCzaNx9iDDv2O1NVnMBDpGPQkaAv2BGjnulJk1iRruMLMmQF/go/LXynqTgEujx5cCr2Qwlkor/mGNnEsNeV+ixtRHgQXufk/MrBr3vpR1LDXtvTGzVmbWPHq8N9CH0P4yBbggWiyj70mt7A0FEHWVuw/IAR5z99syHFKlmNnBhNIEhLG8nqlJx2JmzwK9CUMtrwRuBiYCzwNtgWXAj909qxuPyziO3oRqDgeWAFcW1/lnMzPrBbwDfAgURZNvINT117T3paxjGUANem/MrBuhATuHcBL/vLuPir7/E4DvEG5DPSjmVg7VG2NtTRYiIpI6tbUaSkREUkjJQkREElKyEBGRhJQsREQkISULERFJSMlCJAEz2xkzeumcVI5ibGbtY0eyFclWab0Ht0gtsSUahkGkzlLJQqSSovuM3BHdh2CGmR0aTW9nZm9Fg9i9ZWZto+nfNbOXo3sWzDWz/4k2lWNmf47uY/CP6ApezOxqM5sfbWdChg5TBFCyEEnG3qWqoS6Kmbfe3Y8BHiSMGED0+El37waMB8ZE08cAb7v7EUAPYF40vSMw1t07A2uB86PpI4Ejo+0MTdfBiSRDV3CLJGBmG929aZzpS4CT3f2zaDC7r929hZmtJtyQZ0c0/St3b2lmBUCb2OEaomG133T3jtHz64AG7v47M/s74YZLE4GJMfc7EKl2KlmIVI2X8bisZeKJHetnJyVtiWcAY4GjgFkxo4+KVDslC5GquSjm/3+ix+8SRjoGGEi4RSbAW8Aw2HWjm33K2qiZ1QMOcvcpwK+A5sAepRuR6qIzFZHE9o7uYFbs7+5e3H12LzN7j3DiNSCadjXwmJn9EigALoumjwDGmdnlhBLEMMKNeeLJAZ42s30JN/O6N7rPgUhGqM1CpJKiNotcd1+d6VhE0k3VUCIikpBKFiIikpBKFiIikpCShYiIJKRkISIiCSlZiIhIQkoWIiKS0P8HpSVTgAZcZi8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "history_dict = ann_history.history\n",
    "loss_values = history_dict['loss']\n",
    "val_loss_values = history_dict['val_loss']\n",
    "\n",
    "epochs = range(1,len(ann_history.epoch)+1) # creating a range object for plotting on x axis\n",
    "\n",
    "plt.plot(epochs, loss_values, 'ro', label='Training loss')           \n",
    "plt.plot(epochs, val_loss_values, 'b--', label='Validation loss')      \n",
    "plt.title('Training and validation loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### From the graph we see that training loss consistently decreases but validation loss shows ups and downs as the epochs increase!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Let us plot the training and validation accuracy "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4VNX5wPHvyxrCLosikUWLlR1CDFpRUZQCVVGrAkIVN9QKWoutWKnyw7rUXSsu2CIoUYpSFC24URQVBYIQEBBBRIwgBGTfQ97fH+cmDMMkM5mZm5mE9/M8eTJz1/fO9t57zrnniKpijDHGlKRSogMwxhiT/CxZGGOMCcuShTHGmLAsWRhjjAnLkoUxxpiwLFkYY4wJy5KFiZiIVBaRnSLSLJ7LJpKI/EJE4t5+XETOE5E1Ac9XiMiZkSwbxb7+KSJ/iXZ9YyJRJdEBGP+IyM6Ap6nAPuCg9/xGVc0qzfZU9SBQK97LHg1U9Zfx2I6IXA8MUtXuAdu+Ph7bNqYkliwqMFUt+rH2zlyvV9UPi1teRKqoan5ZxGZMOPZ5TC5WDHUUE5G/ici/ReQ1EdkBDBKR00XkCxHZKiLrReRpEanqLV9FRFREWnjPJ3rzZ4jIDhH5XERalnZZb35vEflGRLaJyD9E5DMRGVxM3JHEeKOIrBKRLSLydMC6lUXkCRHZLCLfAr1KeH1GisikoGljRORx7/H1IrLcO55vvbP+4raVKyLdvcepIvKKF9tSoEuI/a72trtURC7yprcHngHO9Ir4NgW8tqMC1r/JO/bNIvKmiDSJ5LUpzetcGI+IfCgiP4vITyLy54D9/NV7TbaLSLaIHB+qyE9EPi18n73Xc7a3n5+BkSLSSkRmeceyyXvd6gas39w7xjxv/lMikuLF3DpguSYisltEGhR3vCYMVbW/o+APWAOcFzTtb8B+4ELciUMN4FSgK+6q80TgG2Cot3wVQIEW3vOJwCYgA6gK/BuYGMWyjYEdQF9v3h+BA8DgYo4lkhjfAuoCLYCfC48dGAosBdKABsBs9zUIuZ8TgZ1AzYBtbwQyvOcXessIcC6wB+jgzTsPWBOwrVygu/f4UeAjoD7QHFgWtOwVQBPvPbnSi+FYb971wEdBcU4ERnmPe3oxdgJSgGeB/0Xy2pTyda4LbABuA6oDdYBMb95dQA7QyjuGTsAxwC+CX2vg08L32Tu2fOBmoDLu83gy0AOo5n1OPgMeDTier7zXs6a3/BnevLHA/QH7GQ5MTfT3sDz/JTwA+yujN7r4ZPG/MOvdAbzuPQ6VAJ4PWPYi4Ksolr0W+CRgngDrKSZZRBjjaQHz/wPc4T2ejSuOK5zXJ/gHLGjbXwBXeo97A9+UsOw7wC3e45KSxdrA9wL4feCyIbb7FfAb73G4ZDEBeCBgXh1cPVVauNemlK/z74DsYpb7tjDeoOmRJIvVYWK4DJjvPT4T+AmoHGK5M4DvAPGeLwIujff36mj6s2Io80PgExE5RUT+6xUrbAdGAw1LWP+ngMe7KblSu7hljw+MQ923O7e4jUQYY0T7Ar4vIV6AV4EB3uMrgaJGASJygYjM9YphtuLO6kt6rQo1KSkGERksIjleUcpW4JQItwvu+Iq2p6rbgS1A04BlInrPwrzOJwCrionhBFzCiEbw5/E4EZksIj96MYwPimGNusYUh1HVz3BXKd1EpB3QDPhvlDEZrM7CuDPNQC/gzmR/oap1gHtwZ/p+Wo878wVARITDf9yCxRLjetyPTKFwTXv/DZwnImm4YrJXvRhrAG8AD+KKiOoB70cYx0/FxSAiJwLP4YpiGnjb/Tpgu+Ga+a7DFW0Vbq82rrjrxwjiClbS6/wDcFIx6xU3b5cXU2rAtOOClgk+vr/jWvG192IYHBRDcxGpXEwcLwODcFdBk1V1XzHLmQhYsjDBagPbgF1eBeGNZbDPd4B0EblQRKrgysEb+RTjZOAPItLUq+y8s6SFVXUDrqjkJWCFqq70ZlXHlaPnAQdF5AJc2XqkMfxFROqJuw9laMC8WrgfzDxc3rwed2VRaAOQFljRHOQ14DoR6SAi1XHJ7BNVLfZKrQQlvc7TgGYiMlREqolIHRHJ9Ob9E/ibiJwkTicROQaXJH/CNaSoLCJDCEhsJcSwC9gmIifgisIKfQ5sBh4Q12ighoicETD/FVyx1ZW4xGFiYMnCBBsOXI2rcH4Bd2btK+8HuR/wOO7LfxKwEHdGGe8YnwNmAkuA+birg3BexdVBvBoQ81bgdmAqrpL4MlzSi8S9uCucNcAMAn7IVHUx8DQwz1vmFGBuwLofACuBDSISWJxUuP67uOKiqd76zYCBEcYVrNjXWVW3AecDv8VVqH8DnO3NfgR4E/c6b8dVNqd4xYs3AH/BNXb4RdCxhXIvkIlLWtOAKQEx5AMXAK1xVxlrce9D4fw1uPd5v6rOKeWxmyCFlT/GJA2vWGEdcJmqfpLoeEz5JSIv4yrNRyU6lvLObsozSUFEeuGKFfbiml7m486ujYmKV//TF2if6FgqAiuGMsmiG7AaVzzRC7jYKiRNtETkQdy9Hg+o6tpEx1MRWDGUMcaYsOzKwhhjTFgVps6iYcOG2qJFi0SHYYwx5cqCBQs2qWpJTdWBCpQsWrRoQXZ2dqLDMMaYckVEwvViAFgxlDHGmAhYsjDGGBOWr8lCRHqJG05ylYiMCDG/uYjMFJHFIvKR1/9O4byrRWSl93e1n3EaY4wpmW/JwrsLdwyuW+c2wAARaRO02KPAy6raAddFwYPeusfgbvPvirvV/14Rqe9XrMYYY0rm55VFJrBKVVer6n5gEu5uykBtcP3HAMwKmP9r4ANV/VlVt+D6wyl2RDNjjDH+8jNZNOXwvulzObLb6RxcR2QAlwC1vZ5AI1kXERniDdmYnZeXF7fAjTFBsrKgRQuoVMn9z8oKt4apYPxMFqH69Q++XfwO4GwRWYjrsfJHXJ9AkayLqo5V1QxVzWjUKGwzYWNMNLKyYMgQ+P57UHX/hwyxhOGnJEzOfiaLXA4f4CUN15NoEVVdp6qXqmpn4G5v2rZI1jXGlJG774bduw+ftnu3m27iL0mTs5/JYj7QSkRaikg1oD+uP/oiItJQRApjuAsY5z1+D+gpIvW9iu2e3jRjTFlbW0w/fMVNT8Kz4nKlNMm5DF9r35KFNzDJUNyP/HLcsIZLRWS0iFzkLdYdWCEi3wDHAvd76/4M3IdLOPOB0d40Y0wk4vkj0qyYkWdDTU/Ss+KkEcn7EmlyLuvXWlUrxF+XLl3UJLmJE1WbN1cVcf8nTkyu7ZWXfYczcaJqaqqq+wlxf6mp0cdYmu01b374coV/zZvHckQVQ6SvY6SvYZxeayBbI/iNTfiPfLz+LFkkuUT+gPmRpOJ5LKXdd7hjKc2PSKSvzcSJ+smxv9VfM0PXp2UUv5xI6H2LRHO0vtmyRfVvf1PdvLmEheL9uYn0fYn08xWn19qShUku8T7jjPcXL5HHEql4/4iU8rWZP98t8re/lRCjH6+ND8l+cZOeCqr31n2i+BOMeH9uSvPjHu+TghJYsjDJJd5nnJFuz48fLz/OnuP54xDn5datUz140D0++2zVVq1UCwpKOI5EXUFGYM+4V/XcSrP0fc7TvkzV+mzW7TUaR18UVBhjJMks3p/FOL02lixMcon3ly/S7fnxw17KL/3q1aqbNpWwvURdMUSwvbVrVZs1U731Vvd83Di3yGefhTmeeF0JxPkHdnTdRxVUP+RcnUeGgurf+VP0n5vSFofG+2olDq+1JQuTXCL9osR7OZ/K78Pte+dO98N62i82Fi0yqeEtqhMn6vr1qu+/H5BA/KjQnDjR1S3EkHDz8lRPOUW1Th3VL790q2zf7g71hhtCvzSlEsnrHcdk/913qins1sv5d9F2zuc9PZb1upsahy/sVyVzEjaMsGRhkk8iKmfjnXwi2PfPP6vWq+c20VqW6SMM17/zJ/2O5qqpqTruhjlFu2jRQvW3vK4PMELzaFDyD2KEMe7YoXrnnarVqqnOnh06/HDb275dNSNDNSVF9eOPD1/t+edVP/qohO1Gwo9kH8bFF6umyi5dS1rRdj7mTO3Nf/WHpl2ji8+nCv2DB1XXrFHdvz+mzUTEkoWJXWnOtON1tpTs9QEhFF5F3H33oWkPPKD6ybG/1YIQ29x2Qlv98EPVv/9d9YorVE+q8p2C6kYaqoIu55e6hbqlTpAFBaqTJ6umeb+FV1+t+tNPbvrFF6s+99yhuodw2+vTR7VyZdW33w57+NEp4wYKn37qVn3gioXxbUUXh2R28KDqxo2Hnvft667mQPW220p1mFGxZGFi49cZeTiJamkURZJavFj1978/9MXu0EF1374otjlxom6p0aRoflc+1zps1bsuXKIbNkR+CJde6jbRqdPhdQpbt6qee66b162b6vLl4bf1v/+pvvpq8fOXLlV94YXIYztCvFsGhVmuMJHu3Rt6udWrXUIptSg+/2vWuNd2+HDV7t3d56dTp0Pzb77Z/fXurVq1qqvz8pMlCxObMr4xqEii7mEo5XG88Yb7raleXXXQINVPPgnRQijKIrWFTXrr5ZlrVES1Rg1XufzDD6HD3rnz0H6ff171H/9QPXDgyOUKCtzVT/36rnjqvvuCEpu6M9wSi60CjBypWqmSam5uZMsfoQw/N3v3hl89M1P1F78I/dpFtO8Ir6wPHnQJG9xnJzPTJYYJE45c9scfXTHg738fRUylYMnCxCbSMz+fio32Nmule6ledpWApUxSmzap/uEPEdzUFUPi+/pr1cGDVatUUR0z5vB5gUVOr7wS4TGqK5bq18+1cNq+/fDt/fGPLsQ5c8JvZ+VKt+xDD0W+78OU0RXp2qanaZMm4YvTpkxxq2RlRbf7SBQmos2bVRcujKw+YtYs1V27/ItJ1ZKFiVWirizUVdB27Kh67LGuqCOqs71oRHCG+NlnGtGZamm2Gc6aNap79rjHY8eqXnml6nnnaVGR0+efl3qTRS2x9u5VHT1a9Z573PaGDSvhHoogZ5zhWktFuvwRyqCu6womaUqKawlVkoMHVdu0UW3bNkS9Thw8/LDqOeeo7t4d3fr5+fGNJ5Ali6OND3e5JqTOQt1ZdKVKql26uM21bevOsBJt+nRXhvznPycuhkcfVa1ZU7VuXdVnnon9R+Sddw79zg4aVLofyhdfdOt98UVsMcRFiJOWDzlXQfX//i+yTUyc6FadOjV+YRUUqI4a5bbbv390rZuWLlU96aTIiwhLy5LF0cSvcv5EtIZS9wVbsMD9f+MN90V5/fWYNhmzmTNd+XF6uutXKJF27HB1FfEyZ47qgw+W/ods61bVhg1Vx4+PXyxRC/oO7KeKtpZlemLj7UVXZeEcOKDaunUMRWtBCgpU//QnF9I110Sf2HftUm3SRPVXv4rhKq4EliyOJolqQRRnH3zgyumD7dt36Evy4IPuhrD168surk8/dWfz7dqFuRP7KFQW9wGouiuesFc9ASct/218tYLqtGml209wpX8s/vpX9zW85ZbYi7aef16jOp5IRJosxC1b/mVkZGh2dnaiw0iMSpVceggmAgUFZR9PFL7+GjIz4dRTYebM4pcbORL+/ndISYERI+CPf4QaNfyLa/9+OPlkqF4dZs+GY4/1b1/l2a5dULOmf9s//XRYuRJ69oRf/9r9HXdcyessWQLt20e3v6++gnbtolu30LJl8PrrcM897qsYiwMHoG1b9zlctAgqV45te4FEZIGqZoRdMJKMUh7+7Mqi/F5ZbNmievLJqo0bu76Iwlmxwt1kBq41UIn9FMXBokXFN101qhde6G7i88u2ba4YJj3dNXoobGxXeCPbunWHXxF8/31s+3v9dbePTz4p/br797sLHD+Ki/7t9VIyZUp8t4sVQx1FEnVvQhzk57sfmipVSl+B99FHrhK8sDuKzz93ZcQffljKFkshLF+u+sQTsW3jaHHXXa5Bwrp1/u2joMAlhIMHXbPTsWMPzbvwQtVatVQvusjd/xHrnee7dqk2aqTaq1fp1tu799BJjB+V0QcPumKoeLfWsmRRUcS7kjnJOjIbM8Z9Cp99Nrr1CwoOfXnGjHGtlQpz5QUXuJvUIq3gLLRqlerxx7uz2Ly86OI6mnz9tXvNH3kkMfufPt3d2NaypYujWbPY70144AG3rfnzI1v+669dcgH3mfNbPBNGpMnC6iySWeEYu4GDt6emwtixMHCgr9tThY8/hilToFYtePBBN/3RR139QqATToB77y19OAB798K//w1XXRV7uS7Azp0waxa89x7MmAFbt8LGja6Md9w4mDPn8OWbN4e//tU9vu8+N4zx+++7l+ijj2Ivtz5anH467Njh6gni8T4WUnXbHjwYbrop/LLffuvqTpo0iW2/27e7z8Y558B//hN6X4XH+Yc/wFNPuc/YCy/AddfFtu9wJk92n9nsbKhdO/btWZ1FRRDvuogItpeX59ryn3yym1Wzprv5qtAVV6g2bXr4X8+ebl5Bgevr5vbbw/c/tGJF2TRBDexb6Zxzjow9sKihd283rV27Q11ym8gUttaJ9Ew8UsuWue0GFjuVlb/+VfWYY1wvwqquyPSTT9znu0ULdwWq6ppVP/10ZPVt8TB3rpbq/pFwsGKoCqCMRpcrQIragN9/v5v8q1+59vOluZzfsUP18std/QOonnWWK+UKLgbatMkVGZxzTnSHYZLPli3u87JjR3y3+49/uM+S353phbJtm/vLzVW98cZDlevVqqn+5jeuI8lEufRS1dq1D++tNlqWLCoCn68s8migj/JHPbnKKp00yS2Sl6e6ZElsYf/0k7ux6aST3K4Kq0UKCtyNTz16uC9cUtz5a5Ja377uxCKR8vLcHfOXX6762msugSTasmWuUcEf/hD7tixZJLt4DtxTmn2mpupW6uhAXtFq7HVXEa026nvvxXIwoR086FomFV5ZPPyw6oknusMYNy7++zOJtWeP6uOPu2KZeDhwwP1Ix2VUvhiV1c2HpXHdde6kqzTd2IdiySKZlXbc3jj3+fRIvb8pqA6rPU6XPOjX6DZHeukl11nbnXeW2S5NGcrPd/dDXHhhfLa3ZYsr/pkxIz7bq2hyc+Nzj1GkycLX1lAi0gt4CqgM/FNVHwqa3wyYANTzlhmhqtNFpBrwApABFAC3qepHJe2rXLWGatHCNbsJ1rw5rFnj++4XLIAPP4Q77/R9V+Yoc+ed8Nhj8OOPdrd7WSoocB05RCPS1lBRbj6iACoDY4DeQBtggIi0CVpsJDBZVTsD/YFnvek3AKhqe+B84DER8S3WMrd2bemmx1mXLpYojD+uvhoOHoRXX419W6tWlZveahLq5Zfhyy/934+fP8CZwCpVXa2q+4FJQN+gZRSo4z2uC6zzHrcBZgKo6kZgK+4qo2Jo1qx00+NoyRJ3/8TBg77vyhyF2rRx/XuNHx/bdvbscfe4jBwZl7AqtKuugowy+HX0M1k0BX4IeJ7rTQs0ChgkIrnAdGCYNz0H6CsiVUSkJdAFOCF4ByIyRESyRSQ7Ly8v3vH75/773c1wgVJT3XSfPf00XHSRnbEZ/1x7LTRs6G7Si9Znn8G+fdCtW/ziMrHxM1mEuo8zuIJkADBeVdOAPsArXnHTOFxyyQaeBOYA+UdsTHWsqmaoakajRo3iGryvBg50d003b+5uA23ePPq7sktB1d3V3LMnVK3q667MUezGG13PwbHcXTxzJlSpAmedFb+4TGyq+LjtXA6/GkjjUDFToeuAXgCq+rmIpAANvaKn2wsXEpE5wEofYy17Awf6nhyCLVniKh579y7T3ZqjTGE3GOvXQ926R15ER+LDD+G001xXMyY5+HllMR9oJSItvdZN/YFpQcusBXoAiEhrIAXIE5FUEanpTT8fyFfVZT7GmrTi2Vht+nT3v1ev+G3TmFBWrHBVcNFUdG/Z4lrsnXde/OMy0fMtWahqPjAUeA9Yjmv1tFRERovIRd5iw4EbRCQHeA0Y7LX7bQx8KSLLgTuB3/kVZzJbsgR+9SvXKmTfPte5WSw+/hg6d4bjj49PfMYU5+ST4ZRTXMd6pZWaCu+8U+YX3iYM63U2CanCmDFwxx1Qrx5MmAC/+50rC77vvui3e+AArFvnqkiM8dszz8CwYa531C5dEh2NKU7C77Mw0cnLc62Vhg2DHj1g8WI3hGR6OrzySmytmKpWtURhys6gQW7I29JeXTz1lBvW1CQXSxZJ5tFH3XgKTz3lLsUbN3bTr7rK3fT9ySfRbXfUKBg9Om5hGhNWvXowYIAbr2Tv3sjW+eEHNz7E++/7G5spPUsWSWD/fvjuO/d41ChXuXfrrYcPInPxxa4p4oQJpd9+QYE7u1t2VDYRMIl0zz2u7i0lJbLlZ850/61yO/lYskiwFSvcSGC//rVLGjVqhB6dLTUVLr8cXn/98IHuIrFoEfz0E/TpE5+YjYlU8+al65hg5kxo1MhGKExGliwSaPJkVxexZg088ghUq1by8nfcAe++6xJKacyY4f7/+tdRhWlMTL77zt3bM3duycupuvsrzj03+k7xjH/8vCnPlGDjRtctQocO8MYb0DS4I5QQWreObl/Tp7u+Y6wXUJMIDRvCp5/C889D167FL/fDD7BtmxVBJSvL3wnyzDOu0m/8+MgSRaE1a+CWW1wT2EgUFMCJJ0L//tFEaUzsateGK690Fd1btxa/XLNm7oY8u78iOVmyiLesLDdeRaVK7n9WVsjFRo6E996DX/6ydJs/cACefbbYzR6hUiXX5Hb48NLtx5h4uvFG15PsK6+UvFz16qUvZjVlw5JFPGVlwZAhro2rqvs/ZMgRv+z5+a5+okeP0u+iVStXIT5hQmRdgWzaVPp9GBNv6emuKPSFF0J/bvPz4eyz4T//KfvYTGQsWcTT3Xcf2VRp92433bNkCZx0UvjKvpJcdRUsXepaOZWkoMCNL3D77SUvZ0xZuOsu1xNB/hH9R7u7vGfPDj3PJAdLFvEUwQh4d9/tKvFatYp+N1dc4a5Mwt1zsWCBuyO8LAZGMSacSy91IzSG6h6/8P6Kc88t25hM5CxZxFOYEfA++wzeftt9YY45JvrdHHOMO0ML1/Xz9Onuxj5rMmuSxd69rlR28+bDp8+cCZ06uZZTJjlZsoinEkbAU4URI6BJE7jttth39c9/wgMPlLzMjBmQmWlfQJM8Vq50fUa9/PKhabt3uxMpazKb3CxZxFMJI+DNnu3amt9zT3SDwYSi6rovDyUvD+bNs7u2TXJp39410Ais6N6yBS64wD6ryc66KC8jqu7u6/POi9+QpqNGwUMPuRHJ6tc/fN6uXTBtmquviKV+xJh4mzABBg+GWbOge/dER2Osi/IkcvCgu9Do3Tu+Y19fcIEbFOn114+cV7Om6/HTEoVJNldc4XqkLey6fOPGxMZjImPJwmf797uKu7Fj47/tLl1c09jA8l9wyenpp133CcYkmxo14Oqr4ZtvXHFpkybu82qSmyULn40d6wZyKU3Pm5EScfdcfPYZfPvtoenz5rlK9Dlz4r9PY+LhoYcO3VtRUGAj6ZUHlix8tHOnGwa1e3f/mq8OHOiSxquvHpo2Y4br5uP88/3ZpzGxSklxn9u33nJFppmZiY7IhGO9zvroiSdceexbbx0+kFE8paXB//4Hp512aNr06a7FSSz3chjjt/nzXV9RderEty7P+MOuLHyyaxc89pgb4S7wh9wP3bsfGolswwZ353bv3v7u05hYdejguiz3oz7PxJ9dWfikZk346COoVats9vfEE7Bjh2sqW7mytVk3ya96dfjii0RHYSJlycIHBQWuzqBTp7LbZ04OTJ3qhk/dtMld2htjTLxYMZQPhgyBG26IrAvxeLnqKti+3dWP1Ktnw1IaY+LL158UEeklIitEZJWIjAgxv5mIzBKRhSKyWET6eNOrisgEEVkiIstF5C4/44ynn35yd6impvpXqR1K4Z2wAwa49uvGGBNPviULEakMjAF6A22AASLSJmixkcBkVe0M9Aee9aZfDlRX1fZAF+BGEWnhV6zxNH6865P/llvKdr+VKsH117vHjRuX7b6NMRWfn1cWmcAqVV2tqvuBSUDfoGUUKCxdrwusC5heU0SqADWA/cB2H2ONi4ICePFFd5Z/8sllv//nnoPcXFcMZYwx8eRnsmgKBHY4ketNCzQKGCQiucB0YJg3/Q1gF7AeWAs8qqo/B+9ARIaISLaIZOfl5cU5/NKbNQtWr3Z1FolQpQo0DX6FjTEmDvxMFqFK7IOrfAcA41U1DegDvCIilXBXJQeB44GWwHAROfGIjamOVdUMVc1o1KhRfKOPwimnuJ5gL7kk0ZEYY0x8+ZkscoETAp6ncaiYqdB1wGQAVf0cSAEaAlcC76rqAVXdCHwGJP3goE2bwr33HrpBzhhjKgo/k8V8oJWItBSRargK7GlBy6wFegCISGtcssjzpp8rTk3gNOBrH2ON2ZtvuvEjKsjwIMYYcxjfkoWq5gNDgfeA5bhWT0tFZLSIXOQtNhy4QURygNeAwepGYxoD1AK+wiWdl1R1sV+xxkoV/vxneOSRsm0ua4wxZcXXO7hVdTqu4jpw2j0Bj5cBZ4RYbyeu+Wy58PHHbmzhkSMTHYkxxvjD7vONg7FjXXPVy8tNejPGmNKxZBGjzZthyhT43e/cCGDGGFMRWbKI0bffwvHHu76gjDGmorJeZ2OUmekShnXcZ4ypyOwnLgZ5ebB/vyUKY0zFZz9zMRg+HNq2dX1CGWNMRWbJIkpbtsDrr8P559uVhTGm4rOfuShNnAh79yau00BjjClLliyioOrurTj11LIdOtUYYxLFkkUUsrPhq6+suawx5uhhTWejkJEBs2fbVYUx5uhhySIKInDmmYmOwhhjyo4VQ5XS+PEwdCjs25foSIwxpuzYlUUpqMKTT7qmstWqJToaY4wpO3ZlUQrZ2ZCT4yq2bdwKY8zRxJJFKYwdC6mpcOWViY7EGGPKliWLCO15aRKv/WsX/XePo27HFpCVleiQjDGmzFidRSSysth5y18YpFsZwGvw/feHbt0eODCxsRljTBkQN+R1+ZeRkaHZ2dn+bLxFC5cggjVvDmvW+LMetRqpAAAZnElEQVRPY4wpAyKyQFUzwi1nxVCRWLuWHzmeg8Ev19q1iYnHGGPKmCWLSDRrxhl8xtVMOGK6McYcDSxZRGDDnx/je1rQmYWHJqamwv33Jy4oY4wpQxElCxE5SUSqe4+7i8itIlLP39CSx7wTfgtA5rFr3Q0WzZu7drRWuW2MOUpE2hpqCpAhIr8A/gVMA14F+vgVWDKZOxcqV4Yuq1+H1ERHY4wxZS/SYqgCVc0HLgGeVNXbgSbhVhKRXiKyQkRWiciIEPObicgsEVkoIotFpI83faCILAr4KxCRhPXxOncutG/vSp6MMeZoFOmVxQERGQBcDVzoTata0goiUhkYA5wP5ALzRWSaqi4LWGwkMFlVnxORNsB0oIWqZgFZ3nbaA2+p6qJIDyre7rwTdu9O1N6NMSbxIk0W1wA3Afer6nci0hKYGGadTGCVqq4GEJFJQF8gMFkoUMd7XBdYF2I7A4DXIozTF+edl8i9G2NM4kWULLyrgVsBRKQ+UFtVHwqzWlPgh4DnuUDXoGVGAe+LyDCgJhDqZ7kfLskcQUSGAEMAmvnUjPWrryAvD846y9VbGGPM0SjS1lAfiUgdETkGyAFeEpHHw60WYlrw7eIDgPGqmoarLH9FRIpiEpGuwG5V/SrUDlR1rKpmqGpGo0aNIjmUUnvuOegbMlUZY8zRI9IK7rqquh24FHhJVbsQ+iogUC5wQsDzNI4sZroOmAygqp8DKUDDgPn9SXAR1Ny5cOqpdlVhjDm6RZosqohIE+AK4J0I15kPtBKRliJSDffDPy1ombVADwARaY1LFnne80rA5cCkCPcXd3v2uPErugYXnhljzFEm0mQxGngP+FZV54vIicDKklbwmtoO9dZbjmv1tFRERovIRd5iw4EbRCQHdwUxWA/1bHgWkFtYQZ4IixZBfr4lC2OMsV5nS/Dkk3D77bBuHTQJe1eJMcaUP3HtdVZE0kRkqohsFJENIjJFRNJiDzO53XyzG0rVEoUx5mgXaTHUS7j6huNxTWLf9qZVaNWrQ5cuiY7CGGMSL9Jk0UhVX1LVfO9vPOBPW9UksWkTDB8OX3+d6EiMMSbxIk0Wm0RkkIhU9v4GAZv9DCzRvvgCHn8cNm5MdCTGGJN4kSaLa3HNZn8C1gOX4boAqbDmzYNKlSA9PdGRGGNM4kWULFR1rapepKqNVLWxql6Mu0Gvwpo7F9q1g1q1Eh2JMcYkXiwj5f0xblEkmYICd2Vh91cYY4wTS7II1fdThfDTT1C1qiULY4wpFGkX5aFUjLv5Qjj+eNiwAQ4eTHQkxhiTHEpMFiKyg9BJQYAavkSUJESgSiyp1BhjKpASfw5VtXZZBZJM+vWDzp1hxBEDwRpjzNEpljqLCmnvXpg6FbZuTXQkxhiTPCxZBFm0CA4csMptY4wJZMkiyLx57r8lC2OMOcSSRZC5c6FpU9ciyhhjjGPtfYKceCIcd1yiozDGmORiySLIffclOgJjjEk+VgwVYO9eqCADBxpjTFxZsgjw4IPQuDHs25foSIwxJrlYsggwb54bQrV69URHYowxycWShUfVepo1xpjiWLLwrFoFP/8MmZmJjsQYY5KPJQuP3YxnjDHFs2ThadcO7roL2rZNdCTGGJN8fE0WItJLRFaIyCoROaIPVxFpJiKzRGShiCwWkT4B8zqIyOcislRElohIip+xduwIDzwAlSv7uRdjjCmffEsWIlIZGAP0BtoAA0SkTdBiI4HJqtoZ6A88661bBZgI3KSqbYHuwAG/Yt2/Hz79FPbs8WsPxhhTvvl5ZZEJrFLV1aq6H5gE9A1aRoE63uO6wDrvcU9gsarmAKjqZlX1bdy6hQvhzDNh+nS/9mCMMeWbn8miKfBDwPNcb1qgUcAgEckFpgPDvOknAyoi74nIlyLy51A7EJEhIpItItl5eXlRBzp3rvtvldvGGBOan8lCQkwL7kxjADBeVdOAPsArIlIJ12dVN2Cg9/8SEelxxMZUx6pqhqpmNGrUKOpAC2/GaxqcyowxxgD+Jotc4ISA52kcKmYqdB0wGUBVPwdSgIbeuh+r6iZV3Y276kj3K9C5c91VhYRKb8YYY3xNFvOBViLSUkSq4SqwpwUtsxboASAirXHJIg94D+ggIqleZffZwDI/gty82d2QZ0VQxhhTPN+6KFfVfBEZivvhrwyMU9WlIjIayFbVacBw4EURuR1XRDVYVRXYIiKP4xKOAtNV9b9+xFmnjmsJlZbmx9aNMaZiEK0gfXJnZGRodnZ2osMwxphyRUQWqGpGuOXsDm5jjDFhWbIwxhgTliULY4wxYVmyMMYYE5YlC2OMMWFZsjDGGBOWJQtjjDFhWbIwxhgTliULY4wxYVmyMMYYE5YlC2OMMWFZsjDGGBOWJQtjjDFhWbIwxhgTliULY4wxYVmyMMYYE5YlC2OMMWFZsjDGGBOWJQtjjDFhWbIwxhgTliULY4wxYVmyMMYYE5YlC2OMMWFZsjDGGBOWr8lCRHqJyAoRWSUiI0LMbyYis0RkoYgsFpE+3vQWIrJHRBZ5f8/7GacxxpiSVfFrwyJSGRgDnA/kAvNFZJqqLgtYbCQwWVWfE5E2wHSghTfvW1Xt5Fd8xhhjIufnlUUmsEpVV6vqfmAS0DdoGQXqeI/rAut8jMcYY0yU/EwWTYEfAp7netMCjQIGiUgu7qpiWMC8ll7x1McicmaoHYjIEBHJFpHsvLy8OIZujDEmkJ/JQkJM06DnA4DxqpoG9AFeEZFKwHqgmap2Bv4IvCoidYLWRVXHqmqGqmY0atQozuEbY4wp5GeyyAVOCHiexpHFTNcBkwFU9XMgBWioqvtUdbM3fQHwLXCyj7EaY4wpgZ/JYj7QSkRaikg1oD8wLWiZtUAPABFpjUsWeSLSyKsgR0ROBFoBq32M1RhjTAl8aw2lqvkiMhR4D6gMjFPVpSIyGshW1WnAcOBFEbkdV0Q1WFVVRM4CRotIPnAQuElVf/YrVmOMMSUT1eBqhPIpIyNDs7OzEx2GMcaUKyKyQFUzwi1nd3AbY4wJy5KFMcaYsHyrszDGHD0OHDhAbm4ue/fuTXQophgpKSmkpaVRtWrVqNa3ZGGMiVlubi61a9emRYsWiIS6xcokkqqyefNmcnNzadmyZVTbsGIoY0zM9u7dS4MGDSxRJCkRoUGDBjFd+VmyMMbEhSWK5Bbr+2PJwhhjTFiWLIwxZS8rC1q0gEqV3P+srKg3tXnzZjp16kSnTp047rjjaNq0adHz/fv3R7SNa665hhUrVpS4zJgxY8iKIc7yziq4jTFlKysLhgyB3bvd8++/d88BBg4s9eYaNGjAokWLABg1ahS1atXijjvuOGwZVUVVqVQp9PnxSy+9FHY/t9xyS6ljq0jsysIYU7buvvtQoii0e7ebHkerVq2iXbt23HTTTaSnp7N+/XqGDBlCRkYGbdu2ZfTo0UXLduvWjUWLFpGfn0+9evUYMWIEHTt25PTTT2fjxo0AjBw5kieffLJo+REjRpCZmckvf/lL5syZA8CuXbv47W9/S8eOHRkwYAAZGRlFiSzQvffey6mnnloUX2FPGt988w3nnnsuHTt2JD09nTVr1gDwwAMP0L59ezp27MjdcX6dImXJwhhTttauLd30GCxbtozrrruOhQsX0rRpUx566CGys7PJycnhgw8+YNmyZUess23bNs4++2xycnI4/fTTGTduXMhtqyrz5s3jkUceKUo8//jHPzjuuOPIyclhxIgRLFy4MOS6t912G/Pnz2fJkiVs27aNd999F4ABAwZw++23k5OTw5w5c2jcuDFvv/02M2bMYN68eeTk5DB8+PA4vTqlY8nCGFO2mjUr3fQYnHTSSZx66qlFz1977TXS09NJT09n+fLlIZNFjRo16N27NwBdunQpOrsPdumllx6xzKeffkr//v0B6NixI23btg257syZM8nMzKRjx458/PHHLF26lC1btrBp0yYuvPBCwN1El5qayocffsi1115LjRo1ADjmmGNK/0LEgSULY0zZuv9+SE09fFpqqpseZzVr1ix6vHLlSp566in+97//sXjxYnr16hXyvoNq1aoVPa5cuTL5+fkht129evUjlomkY9bdu3czdOhQpk6dyuLFi7n22muL4gjVvFVVk6JZsiULY0zZGjgQxo6F5s1BxP0fOzaqyu3S2L59O7Vr16ZOnTqsX7+e9957L+776NatG5MnTwZgyZIlIa9c9uzZQ6VKlWjYsCE7duxgypQpANSvX5+GDRvy9ttvA+5Gx927d9OzZ0/+9a9/sWfPHgB+/jkxozVYayhjTNkbOND35BAsPT2dNm3a0K5dO0488UTOOOOMuO9j2LBhXHXVVXTo0IH09HTatWtH3bp1D1umQYMGXH311bRr147mzZvTtWvXonlZWVnceOON3H333VSrVo0pU6ZwwQUXkJOTQ0ZGBlWrVuXCCy/kvvvui3vs4dh4FsaYmC1fvpzWrVsnOoyEy8/PJz8/n5SUFFauXEnPnj1ZuXIlVaokx3l5qPcp0vEskuMIjDGmAti5cyc9evQgPz8fVeWFF15ImkQRq4pxFMYYkwTq1avHggULEh2GL6yC2xhjTFiWLIwxxoRlycIYY0xYliyMMcaEZcnCGFPude/e/Yib7J588kl+//vfl7herVq1AFi3bh2XXXZZsdsO1yz/ySefZHdA54h9+vRh69atkYRebliyMMaUewMGDGDSpEmHTZs0aRIDBgyIaP3jjz+eN954I+r9ByeL6dOnU69evai3l4x8TRYi0ktEVojIKhEZEWJ+MxGZJSILRWSxiPQJMX+niNwRvK4xJnl1737k37PPunm7d4eeP368m79p05Hzwrnssst455132LdvHwBr1qxh3bp1dOvWrejeh/T0dNq3b89bb711xPpr1qyhXbt2gOuOo3///nTo0IF+/foVdbMBcPPNNxd1cX7vvfcC8PTTT7Nu3TrOOecczjnnHABatGjBpk2bAHj88cdp164d7dq1K+rifM2aNbRu3ZobbriBtm3b0rNnz8P2U+jtt9+ma9eudO7cmfPOO48NGzYA7n6Oa665hvbt29OhQ4eiLkPeffdd0tPT6dixIz169Aj/wpWCb/dZiEhlYAxwPpALzBeRaaoa2FnKSGCyqj4nIm2A6UCLgPlPADP8itEYUzE0aNCAzMxM3n33Xfr27cukSZPo168fIkJKSgpTp06lTp06bNq0idNOO42LLrqo2M75nnvuOVJTU1m8eDGLFy8mPT29aN7999/PMcccw8GDB+nRoweLFy/m1ltv5fHHH2fWrFk0bNjwsG0tWLCAl156iblz56KqdO3albPPPpv69euzcuVKXnvtNV588UWuuOIKpkyZwqBBgw5bv1u3bnzxxReICP/85z95+OGHeeyxx7jvvvuoW7cuS5YsAWDLli3k5eVxww03MHv2bFq2bBn3PqT8vCkvE1ilqqsBRGQS0BcITBYK1PEe1wXWFc4QkYuB1cAuH2M0xvjgo4+Kn5eaWvL8hg1Lnl+cwqKowmRROA6FqvKXv/yF2bNnU6lSJX788Uc2bNjAcccdF3I7s2fP5tZbbwWgQ4cOdOjQoWje5MmTGTt2LPn5+axfv55ly5YdNj/Yp59+yiWXXFLU++2ll17KJ598wkUXXUTLli3p1KkTUHxX6Lm5ufTr14/169ezf/9+WrZsCcCHH354WLFb/fr1efvttznrrLOKlol3V+Z+FkM1BX4IeJ7rTQs0ChgkIrm4q4phACJSE7gT+L+SdiAiQ0QkW0Sy8/LyoosyjmMBG2MS5+KLL2bmzJl8+eWX7Nmzp+iKICsri7y8PBYsWMCiRYs49thjQ3ZNHijUVcd3333Ho48+ysyZM1m8eDG/+c1vwm6npL73Crs4h+K7Qh82bBhDhw5lyZIlvPDCC0X7C9Vtud9dmfuZLEJFHfzKDQDGq2oa0Ad4RUQq4ZLEE6q6s6QdqOpYVc1Q1YxGjRqVPsLCsYC//x5UD40FbAnDmHKnVq1adO/enWuvvfawiu1t27bRuHFjqlatyqxZs/j+++9L3M5ZZ51Flvcb8NVXX7F48WLAdXFes2ZN6taty4YNG5gx41AJee3atdmxY0fIbb355pvs3r2bXbt2MXXqVM4888yIj2nbtm00berOsSdMmFA0vWfPnjzzzDNFz7ds2cLpp5/Oxx9/zHfffQfEvytzP5NFLnBCwPM0AoqZPNcBkwFU9XMgBWgIdAUeFpE1wB+Av4jI0LhHWEZjARtjysaAAQPIyckpGq0OYODAgWRnZ5ORkUFWVhannHJKidu4+eab2blzJx06dODhhx8mMzMTcCPfde7cmbZt23Lttdce1sX5kCFD6N27d1EFd6H09HQGDx5MZmYmXbt25frrr6dz584RH8+oUaO4/PLLOfPMMw+rDxk5ciRbtmyhXbt2dOzYkVmzZtGoUSPGjh3LpZdeSseOHenXr1/E+4mEb12Ui0gV4BugB/AjMB+4UlWXBiwzA/i3qo4XkdbATKCpBgQlIqOAnar6aEn7i6qL8kqV3BXFkcFDQUHptmXMUcy6KC8fYumi3LcrC1XNB4YC7wHLca2elorIaBG5yFtsOHCDiOQArwGDtSwH2CjDsYCNMaY887WLclWdjqu4Dpx2T8DjZUCJw1Wp6ihfggM35u+QIYcXRfk0FrAxxpRnR/cd3AkaC9iYiqiijLpZUcX6/tjgRwkYC9iYiiYlJYXNmzfToEEDX5tvmuioKps3byYlJSXqbViyMMbELC0tjdzcXKK+38n4LiUlhbS0tKjXt2RhjIlZ1apVi+4cNhXT0V1nYYwxJiKWLIwxxoRlycIYY0xYvt3BXdZEJA8I7vSlIbApAeH4wY4lOVWUY6koxwF2LKXVXFXDdq5XYZJFKCKSHclt7OWBHUtyqijHUlGOA+xY/GLFUMYYY8KyZGGMMSasip4sxiY6gDiyY0lOFeVYKspxgB2LLyp0nYUxxpj4qOhXFsYYY+LAkoUxxpiwKmyyEJFeIrJCRFaJyIhExxMLEVkjIktEZJGIlHI4wMQSkXEislFEvgqYdoyIfCAiK73/9RMZYySKOY5RIvKj974sEpE+iYwxUiJygojMEpHlIrJURG7zpper96WE4yh374uIpIjIPBHJ8Y7l/7zpLUVkrvee/FtEqiUsxopYZyEilXFDup6PGwt8PjDAG2yp3PHGIs9Q1XJ3o5GInAXsBF5W1XbetIeBn1X1IS+R11fVOxMZZzjFHMcoIhjyN9mISBOgiap+KSK1gQXAxcBgytH7UsJxXEE5e1/E9eteU1V3ikhV4FPgNuCPwH9UdZKIPA/kqOpziYixol5ZZAKrVHW1qu4HJgF9ExzTUUlVZwM/B03uC0zwHk/AfcGTWjHHUS6p6npV/dJ7vAM37HFTytn7UsJxlDvq7PSeVvX+FDgXeMObntD3pKImi6bADwHPcymnHyKPAu+LyAIRGZLoYOLgWFVdD+4LDzROcDyxGCoii71iqqQutglFRFoAnYG5lOP3Jeg4oBy+LyJSWUQWARuBD4Bvga2qmu8tktDfsYqaLEIN1VWey9vOUNV0oDdwi1ckYhLvOeAkoBOwHngsseGUjojUAqYAf1DV7YmOJ1ohjqNcvi+qelBVOwFpuNKR1qEWK9uoDqmoySIXOCHgeRqwLkGxxExV13n/NwJTcR+k8myDV95cWO68McHxREVVN3hf8ALgRcrR++KVi08BslT1P97kcve+hDqO8vy+AKjqVuAj4DSgnogUDlKX0N+xipos5gOtvJYE1YD+wLQExxQVEanpVd4hIjWBnsBXJa+V9KYBV3uPrwbeSmAsUSv8YfVcQjl5X7zK1H8By1X18YBZ5ep9Ke44yuP7IiKNRKSe97gGcB6uDmYWcJm3WELfkwrZGgrAay73JFAZGKeq9yc4pKiIyIm4qwlww+C+Wp6ORUReA7rjulreANwLvAlMBpoBa4HLVTWpK4+LOY7uuKIOBdYANxaW+SczEekGfAIsAQq8yX/BlfeXm/elhOMYQDl7X0SkA64CuzLuJH6yqo72vv+TgGOAhcAgVd2XkBgrarIwxhgTPxW1GMoYY0wcWbIwxhgTliULY4wxYVmyMMYYE5YlC2OMMWFZsjAmDBE5GNCD6aJ49mIsIi0Ce7I1JllVCb+IMUe9PV43DMYctezKwpgoeeOM/N0bh2CeiPzCm95cRGZ6HdnNFJFm3vRjRWSqN2ZBjoj8yttUZRF50RvH4H3vDl5E5FYRWeZtZ1KCDtMYwJKFMZGoEVQM1S9g3nZVzQSewfUYgPf4ZVXtAGQBT3vTnwY+VtWOQDqw1JveChijqm2BrcBvvekjgM7edm7y6+CMiYTdwW1MGCKyU1VrhZi+BjhXVVd7Hdr9pKoNRGQTblCeA9709araUETygLTA7hq8rrU/UNVW3vM7gaqq+jcReRc34NKbwJsB4x0YU+bsysKY2Ggxj4tbJpTAvn4Ocqgu8TfAGKALsCCg91FjypwlC2Ni0y/g/+fe4zm4no4BBuKGyASYCdwMRQPd1CluoyJSCThBVWcBfwbqAUdc3RhTVuxMxZjwangjmBV6V1ULm89WF5G5uBOvAd60W4FxIvInIA+4xpt+GzBWRK7DXUHcjBucJ5TKwEQRqYsbzOsJb5wDYxLC6iyMiZJXZ5GhqpsSHYsxfrNiKGOMMWHZlYUxxpiw7MrCGGNMWJYsjDHGhGXJwhhjTFiWLIwxxoRlycIYY0xY/w+HhUEJo71S0wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "acc_values = history_dict['acc']\n",
    "val_acc_values = history_dict['val_acc']\n",
    "\n",
    "plt.plot(epochs, acc_values, 'ro', label='Training acc')\n",
    "plt.plot(epochs, val_acc_values, 'b--', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Analyze the above plot to understand the variation between training and validation accuracy curves with respect to increasing epochs!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Experiment with learning rates\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Reference Links:\n",
    "\n",
    "https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/\n",
    "\n",
    "https://keras.io/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}