notebooks/3-FitModel.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fit Model\n",
    "This notebook fits a 3-state classification model on a training set and calculates metrics on a test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "**Rule 9: Design Your Notebooks to Be Read, Run, and Explored.** We use ipywidgets to present the user with a pull-down menu to select a machine learning model.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import mlutils\n",
    "from sklearn import svm, metrics\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "import joblib\n",
    "from ipywidgets import widgets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# column names\n",
    "feature_col = \"features\" # feature vector\n",
    "value_col = \"foldClass\" # fold class to be predicted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read data set with fold type classifications and feature vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_json(\"../intermediate_data/features.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of data: 5370 \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pdbChainId</th>\n",
       "      <th>sequence</th>\n",
       "      <th>secondary_structure</th>\n",
       "      <th>length</th>\n",
       "      <th>Exptl.</th>\n",
       "      <th>resolution</th>\n",
       "      <th>R-factor</th>\n",
       "      <th>FreeRvalue</th>\n",
       "      <th>alpha</th>\n",
       "      <th>beta</th>\n",
       "      <th>coil</th>\n",
       "      <th>foldClass</th>\n",
       "      <th>ngram</th>\n",
       "      <th>features</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>16VP.A</td>\n",
       "      <td>SRMPSPPMPVPPAALFNRLLDDLGFSAGPALCTMLDTWNEDLFSAL...</td>\n",
       "      <td>CCSCCCCCCCCHHHHHHHHHHHHTCTTHHHHHHHHHHCCCCCSTTS...</td>\n",
       "      <td>366</td>\n",
       "      <td>XRAY</td>\n",
       "      <td>2.1</td>\n",
       "      <td>0.19</td>\n",
       "      <td>0.26</td>\n",
       "      <td>0.469945</td>\n",
       "      <td>0.046448</td>\n",
       "      <td>0.483607</td>\n",
       "      <td>alpha</td>\n",
       "      <td>[SRM, RMP, MPS, PSP, SPP, PPM, PMP, MPV, PVP, ...</td>\n",
       "      <td>[-2.6183412084, -0.37215537190000003, 0.140630...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1A12.C</td>\n",
       "      <td>RRSPPADAIPKSKKVKVSHRSHSTEPGLVLTLGQGDVGQLGLGENV...</td>\n",
       "      <td>CCCCCCCCCCCCCCCCCCCTTCCCCCBEEEEEEECTTSTTCSCTTC...</td>\n",
       "      <td>413</td>\n",
       "      <td>XRAY</td>\n",
       "      <td>1.7</td>\n",
       "      <td>0.19</td>\n",
       "      <td>0.22</td>\n",
       "      <td>0.038741</td>\n",
       "      <td>0.418886</td>\n",
       "      <td>0.542373</td>\n",
       "      <td>beta</td>\n",
       "      <td>[RRS, RSP, SPP, PPA, PAD, ADA, DAI, AIP, IPK, ...</td>\n",
       "      <td>[-2.2319736748, -0.4659580593, -0.038827395300...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  pdbChainId                                           sequence  \\\n",
       "1     16VP.A  SRMPSPPMPVPPAALFNRLLDDLGFSAGPALCTMLDTWNEDLFSAL...   \n",
       "3     1A12.C  RRSPPADAIPKSKKVKVSHRSHSTEPGLVLTLGQGDVGQLGLGENV...   \n",
       "\n",
       "                                 secondary_structure  length Exptl.  \\\n",
       "1  CCSCCCCCCCCHHHHHHHHHHHHTCTTHHHHHHHHHHCCCCCSTTS...     366   XRAY   \n",
       "3  CCCCCCCCCCCCCCCCCCCTTCCCCCBEEEEEEECTTSTTCSCTTC...     413   XRAY   \n",
       "\n",
       "   resolution  R-factor  FreeRvalue     alpha      beta      coil foldClass  \\\n",
       "1         2.1      0.19        0.26  0.469945  0.046448  0.483607     alpha   \n",
       "3         1.7      0.19        0.22  0.038741  0.418886  0.542373      beta   \n",
       "\n",
       "                                               ngram  \\\n",
       "1  [SRM, RMP, MPS, PSP, SPP, PPM, PMP, MPV, PVP, ...   \n",
       "3  [RRS, RSP, SPP, PPA, PAD, ADA, DAI, AIP, IPK, ...   \n",
       "\n",
       "                                            features  \n",
       "1  [-2.6183412084, -0.37215537190000003, 0.140630...  \n",
       "3  [-2.2319736748, -0.4659580593, -0.038827395300...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Total number of data:\", df.shape[0], \"\\n\")\n",
    "df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split dataset into a training set and a test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set size: 4027 \n",
      "\n",
      "alpha         2107\n",
      "alpha+beta    1266\n",
      "beta           654\n",
      "Name: foldClass, dtype: int64\n",
      "\n",
      "Test set size: 1343 \n",
      "\n",
      "alpha         703\n",
      "alpha+beta    422\n",
      "beta          218\n",
      "Name: foldClass, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "train, test = train_test_split(df, test_size=0.25, random_state=13, stratify=df[value_col])\n",
    "print(\"Training set size:\", train.shape[0], \"\\n\")\n",
    "print(train[value_col].value_counts())\n",
    "print()\n",
    "print(\"Test set size:\", test.shape[0], \"\\n\")\n",
    "print(test[value_col].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Select a Classification Method (default SVM)\n",
    "Using the pull-down menu, you can select one of three machine learning models and compare the performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "method = widgets.Dropdown(options=['SVM', 'LogisticRegression', \"NeuralNetwork\"],description='Method')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cc17921d33c4413da60f2f94a26b6a99",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Method', index=1, options=('SVM', 'LogisticRegression', 'NeuralNetwork'), value='Logisti…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(method)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(class_weight='balanced', max_iter=500, random_state=13)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "if (method.value == 'SVM'):\n",
    "    classifier = svm.SVC(gamma='auto', class_weight='balanced', random_state=13, probability=True)\n",
    "elif (method.value == 'LogisticRegression'):\n",
    "    classifier = LogisticRegression(class_weight='balanced', random_state=13, solver='lbfgs', multi_class='auto', max_iter=500)\n",
    "elif (method.value == 'NeuralNetwork'):\n",
    "    # Neural network with one hidden layer of 20 nodes\n",
    "    classifier = MLPClassifier(hidden_layer_sizes = (20), random_state=13, early_stopping=True)\n",
    "\n",
    "\n",
    "classifier.fit(train[feature_col].tolist(), train[value_col])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make prediction for the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted = classifier.predict(test[feature_col].tolist())\n",
    "expected = test[value_col]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate metrics for the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification metrics:\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "       alpha       0.87      0.79      0.83       703\n",
      "  alpha+beta       0.67      0.72      0.69       422\n",
      "        beta       0.69      0.80      0.74       218\n",
      "\n",
      "    accuracy                           0.77      1343\n",
      "   macro avg       0.75      0.77      0.76      1343\n",
      "weighted avg       0.78      0.77      0.77      1343\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUUAAAEYCAYAAADLZOR0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAA1rElEQVR4nO3de5yXc/7/8cdzmkrpfK7pqHSmFCJUIjpJviSHzXlpnZZld6211nGxtsU6bNJaNstKIZHKDyFEByXk0ImaTqIooppevz+ua6bPTDOfz2eO12dmXne3z83nc13v6329r880r3kfruv9lpnhnHMukBZ1AZxzLpV4UHTOuRgeFJ1zLoYHReeci+FB0TnnYnhQdM65GB4UXQ5JcyRdFL4/W9LsEs6/rSSTlF6S+SZ57hqSpkv6TtIzxcinxL+XKEh6WdK5UZcjFXlQLEOSVkvaKGn/mG0XSZoTYbHyZWb/NbMTyvq8ks6StEDSdknrw1/eo0sg69OApkBDMxtV1ExK63uRNCD8g/Fsnu09wu1zksznJklPJEpnZkPM7PEiFrdC86BY9tKBXxc3EwUq1M9P0m+Ae4G/EASw1sBDwMklkH0b4HMz210CeZWWr4G+khrGbDsX+LykTlAR/92UODPzVxm9gNXAdcC3QL1w20XAnJg0fYH5wHfh//vG7JsD3A68DewAOgAGXAp8AWwDbgXaA+8C3wOTgWrh8fWBFwl++baE71vmyf+i8P15wNzw/e+A7TGvXcBj4b66wL+A9UAmcBtQJdxXBfgbsBlYCVwWljc9n++mbpj3qDjfX3WCoLkufN0LVA/3DQDWAtcAm8LynB/uuxnYGZZ7O3AhcBPwREzebWPLFl7/yvA7XQWcnfd7SfLndWv489oGzAYaFXBt2eUfD1wW8/2tBW4k97+R+4A14c93IXBMuH1wnutcEuffTezP+p/AlJj87wJeBRT170wkv6dRF6AyvQiC4vHAs8Bt4bacoAg0IAhWYwhqlGeGnxuG++cAXwHdwv1Vw1/kF4A64fafw3/QBxAEmk+Ac8PjGwKnAjWB2sAzwPMx5Yv9Rcn1yx+TphVBQBoafn4eeBjYH2gCvA9cEu4bC3waHtMAeJ2Cg+JgYHd++2LS3ALMC8/TGHgHuDXcNyA8/pbwexkK/AjUD/ffRO4gmPdz2+yyhdfyPdAp3Ncc6Jb3e0ny57UC6AjUCD/fWcC1DSAIgH2B98JtQ4FZ7PuH8xfhzzKd4I/ABmC//K4rzr+b2J91TYLa6HnAMQR/xFoW9HOo6C+vRkfjRuAKSY3zbB8GfGFmk8xst5k9RRBUTopJ85iZfRzu3xVuu8vMvjezj4GPgNlmttLMvgNeBg4BMLNvzGyqmf1oZtsIag/9ky20pBoEQfA+M5shqSkwBLjKzH4ws03APcAZ4SGnA/ea2Roz+xa4I072DYHNFr95ezZwi5ltMrOvCWqAY2L27wr37zKzGQS1pU7JXl8ee4DukmqY2frwu80rmZ/Xv83sczPbQVBr7xnvpGb2DtBAUifgHOA/+aR5IvxZ7jazcQQ16ETXmd+/m+z8fiQItH8HngCuMLO1CfKrsDwoRsDMPiJoul6XZ1cL4Ms8274EMmI+r8kny40x73fk87kWgKSakh6W9KWk74E3gXqSqiRZ9H8Bn5nZXeHnNgS1jvWStkraSlBrbBJzPbHlzXttsb4BGiUYmc77/XwZbsvJI09Q/ZHw2gvDzH4ARhPUdNdLeklS5yTKk12m2J/XhiKUZxJwOXAs8FzenZKukbQsHEnfStAiaJQgz/z+3eQws/cJugtEELwrLQ+K0fkz8Ety/wKtIwg0sVoT9NVlK860RtcQ1Cj6mFkdoF+4XYkOlHRdeOyFMZvXEDTXG5lZvfBVx8y6hfvXEzSds7WOc4p3gZ+AkXHS5P1+WofbiuIHgmZjtmaxO81slpkNImg6fwo8kkR5ssuUmU/awphE0E88I6zF5ZB0DPB7glp4fTOrR9Cfmf0zLOjfR9x/N5IuI6hxriPoQ660PChGxMyWA08DV8ZsngF0DG9LSZc0GuhKUKssCbUJao5bJTUgCMwJSRoSlnNk2AzMvob1BIMH4yTVkZQmqb2k7Cb5ZOBKSS0l1WffmjExeX1H0K3woKSRYa22qqQhkv4aJnsKuEFSY0mNwvQJbz8pwGKgn6TWkuoCf4i53qaSRoS3Tv1M0AzPyiePUvl5mdkqgm6NP+azuzZB3+nXQLqkGwn6k7NtBNoWZoRZUkeCAbJfEHRH/E5Sz6KVvvzzoBitWwg69YGgzw8YTlCj+4bgL/ZwM9tcQue7l6DDfzPBgMXMJI8bTTCwsSy8f3C7pPHhvnOAagQDOluAKQS1KwhqV7OAJcAiggGmApnZ34HfADcQ/NKvIWhGPh8muQ1YAHwILA3zvC3Ja8h7rlcI/ih9SDCCGxvI0gh+BusI7hToT1Bzy5tHqf28zGyumeVXC55F0E/8OUFT/SdyN42zb0z/RtKiROcJuyueIOiXXmJmXwDXA5MkVS/ONZRXMvNJZp1zLpvXFJ1zLoYHReeci+FB0TlXbkkaLOkzScvDOyTy7q8bTgSyRNLHks5PmKf3KTrnyqPw/trPgUEETwPNB840s09i0lwP1DWz34cPS3wGNDOznQXlW+ZTOFUmSq9hql4nccIKomuHjMSJKphq6ZWrsfXBooWbzSzvk1hFUqVOG7PdO+KmsR1fzzKzwQXsPhxYbmYrAST9j2DykE9i0hhQW5IIbpz/luCWpgJ5UCxFql6H6p3PSJywgpg8/S9RF6HMtWlUM3GiCqRW9bR4TyUViu3+KeHvx08f3N9Z0oKYTRPMbEL4PoPctyOtBfrkyeIBgrkB1hHc4znazPbEO6cHRedcNAQo4cNUm83s0Dg55JW3P/BEghv1BxLMHvWKpLfM7PuCTli56v7OudSSViX+K7615H6MtCX7PvZ5PvCsBZYTTAOX33Pse4tUyEtwzrkSIlBa/Fd884EDJbWTVI1gdqYX8qT5CjgOgsc3CZ7fXxkvU28+O+eiIZKpDRbIzHZLupzg0ccqwKNm9rGkseH+8QST/D4maWl4xt8negzTg6JzLiJKpk8xrnDezBl5to2Peb8OKNSaOh4UnXPRKUZNsbR4UHTORUTJ9BuWOQ+KzrloJHdLTpnzoOici4ggLfVCUOqVyDlXeaR5TdE55wLFvCWntHhQdM5FxAdanHMuNx9occ65kOTNZ+ecy8Wbz845l81ris45l5v3KTrnXEh+87ZzzuWWgjXF1OvldM5VHsWbZDaZJU5/K2lx+PpIUpakBvHy9KDonItG9i05RVyOIFzi9EFgCNAVOFNS19g0Zna3mfU0s57AH4A3zOzbePl6UCxHBh3ZmSVTr+ej52/g2vOO32f/1WMGMu/J3zLvyd+y4Onr2P7+PdSvE6w2d9mZ/Vnw9HUsnHwdl5/Zv6yLXmRvvf4Kw445hMFHHcwjD4zbZ//K5Z9x1kkD6dmuAf8ef1+ufd9/t5Wrfnk2w/sdwkn9e7F4wXtlVewie2XWTA7p3pmDuxzIuLvv3Ge/mXHt1VdycJcD6dO7B4s/WJRrf1ZWFn0P78VpI08qqyIXjxT/FV/OEqfhOs7ZS5wW5EzgqUSZep9iOZGWJu69bhTDLn2IzI1bmTvpGl58YymfrtqYk+aeSa9xz6TXABh6TDeuOHsAW77/ka7tm3P+yCM55txx7NyVxQv3j+XluZ+wYs3XUV1OUrKysrj9j7/hkadeoGnzDEYP7cexJwylQ8cuOWnq1qvPH269m9dmTt/n+Dtu/B1HHzuIex/5Lzt37uSnHT+WZfELLSsri9/8+nJemDGbjJYt6df3cIYOH0GXLnsrP7NnvsyK5ctZ8snnzH//Pa664lLmzJ2Xs/+h+++jU+cubPu+wMXqUoaAtLRi1cuSWeI0OJdUExgMXJ4oU68plhOHdWvDijVfszrzG3btzuKZ2YsYPuCgAtOfPrg3k2cFtYjO7Zry/ker2fHTLrKy9vDWouWcfGzBx6aKpR8soFXbA2jVph3VqlVj6Mmn8fqsl3KladioCQf17E161aq5tm/f9j0L33ubU888F4Bq1apRp269sip6kSyY/z4HtO9AuwMOoFq1apx2+mhemj4tV5oXp0/jzF+MQRKH9zmC77ZuZcP69QBkrl3LzJdncO75F0ZR/MJTEi9oJGlBzOviPDnklXeJ02wnAW8najqDB8Vyo0WTuqzduDXnc+bGrWQ0rptv2hr7VWXQkZ15/tUlAHy8fD1HH9KeBnVrUmO/qgw+qistm9Yvi2IXy8YN62jeomXO56bNM9i4Ie8Klvlb8+Vq6jdsxB+vHsupJ/Tlxmsv48cffyitopaIdesyadlq7/VmZLRkXWZmrjTr162jZcu9q3q2yGjJunVBmt9dezW33XFXcWtfZUhI8V+E6z7HvCbEZJDMEqfZziCJpjNUsqAoabWkRsVNEwXl079iBfxNHHZMd95dsoot3wfNxc9Wb2Tc46/y4kOX8sL9Y/nw83XsztpTmsUtGflcYH7fQ36ysnazbOlizjjnIqbOfocaNWsyMZ8+yVRiSVxvQWlefulFGjduzCG9epda+UpDWlpa3FcCySxxiqS6QH9gWt59+ZapkNfgIpK5cSstm9bL+ZzRtB7rNn+Xb9pRJ/bimVm5O+AfnzaPvmf/jUG/vJ8t3//I8hTvT4SgZrh+3dqczxvXZ9KkafOkj23aPIODex0GwAnDRrJs6ZJSKWdJychoydo1e683M3MtzVu0yJWmRUYGa9fu7UZbl7mW5s1bMO/dt5nx0nS6dmzHeWPO5I05r3HheWPKrOxFlURNsUBmtpugj3AWsAyYnL3EafYyp6FTgNlmllRTocIGRUnPS1oo6eM8/RBIaivpU0mPS/pQ0pSwIzbbFZIWSVoqqXN4zOGS3pH0Qfj/TmV5PQs++YoOrRrTpkUDqqZXYdQJvXjpjY/2SVen1n4c3as90+cszbW9cf1aALRqVp+TBx7M5JkLy6TcxdG9Z2++WrWCtV+tZufOncyYNoVjTxia1LGNmzSlWYsMVi3/HIB5c+fQvmPn0ixusfU+9DBWLP+C1atWsXPnTqZMfpqhw0fkSjNs+AieemISZsb7782jTt26NGvenJtvu4PPV67hk89X8dikp+g/YCD/emxSRFeSHEkoLf4rETObYWYdzay9md0ebhufZ5nTx8zsjGTLVZFHny8ws28l1QDmS5qaZ38n4EIze1vSo8ClwN/CfZvNrJekS4FrgYuAT4F+4QLcxwN/AU4tm0uBrKw9XP3XqUx/4FdUqZLG49PmsWzlBi469SgAJk59G4ARxx7Mq/M+48efduY6/qm7L6BB3f3ZtTuLq+6cwtZtO8qq6EWWnp7OH28bx8VnjWTPnixOGT2GDp268vR/JgIw+pyL+HrTRkYPOYbt27eRlpbGpEce5IU5C6hVuw7X3zqO319xIbt27aRl63bc9vd/RnxF8aWnpzPu3vsZOXwwWVlZjDnvfLp27cbECcHv90UXj+XEIUOZNXMGB3c5kBo1azL+kUcjLnXxJNsdUpaUXx9FRSDpJoJqM0Bb4ESC+5gOBWoBb5pZ6zDtQOBKMxspaTVwlJllSuoD3G5mx0tqBfwDOJBghKuqme1T9QhrpUHNtFrt3vt1P7/UrjHVLJz+l6iLUObaNKqZOFEFUqt62kIzO7Qk8kpveIDVGXpb3DRbnji7xM6XrArZfJY0ADgeONLMegAfAPvlSZb3r0Hs55/D/2extzZ9K/C6mXUnGN7Pm1+QidmE7JEypdco8jU4V+GJYjefS0OFDIpAXWCLmf0Y9gkekU+a1pKODN+fCcxNIs/s+yPOK5FSOlfJFWegpbRU1KA4E0iX9CFBDW9ePmmWAeeGaRoAiTqc/grcIeltIPVmxnSunBEq7i05paJCDrSY2c8ED4nn1RZAUi1gj5mNzZvAzNrGvF8ADAjfvwt0jEn6pxIrsHOVVeqNs1TMoOicKwdU7GefS0WlDIpmthroHnU5nKvsUvGWnEoZFJ1z0RPRDabE40HROReN8JacVONB0TkXGa8pOudcDK8pOudcDK8pOudcKMqnVuLxoOici0wq3qeYeiVyzlUeiddoiX94gnWfwzQDwnWfP5b0RqI8vabonItGMZ9oiVn3eRDBei3zJb1gZp/EpKkHPAQMNrOvJDVJlK/XFJ1zkRDFXfY5qXWfzwKeNbOvAMxsU6JMPSg65yIi0tLiv4i/xGl+6z5n5DlJR6C+pDnh8iTnJCqVN5+dc5FJYvR5c5yZt5NZ9zkd6A0cB9QA3pU0z8w+L+iEHhSdc9FIrokcTzLrPq8lCKw/AD9IehPoARQYFL357JyLhIAqVRT3lUAy6z5PA46RlB6u2NmHYILpAnlN0TkXmeLcvB2urJm97nMV4NHsdZ/D/ePNbJmkmcCHwB5gopntuzZwDA+KzrlISGQPphSZmc0AZuTZNj7P57uBu5PN04Oicy4i/pifc87lkoIx0YOicy4iJdB8Lg0eFJ1zkQieaPGg6JxzObym6JxzMVKwouhBsTR1aZ/B09Nuj7oYZab3Jf+Jughl7vPHL4i6COVWSdySUxo8KDrnIuK35DjnXC4pGBM9KDrnIuLNZ+ec28tvyXHOuTy8puicczG8puicc9mKP8lsqfBJZp1zkVBya7TEzyPBEqfh8qbfhUucLpZ0Y6I8C6wpSrqffdc7yGFmVyYssXPOxZFWjKpiMkucht4ys+HJ5huv+byg8MV0zrnklMATLTlLnAb5KXuJ07xBsVAKDIpm9njsZ0n7h4u/OOdciUgiJjaSFFtBm2BmE8L3+S1x2iefPI6UtIRgUatrzezjeCdMONAi6UjgX0AtoLWkHsAlZnZpomOdcy6eMljidBHQxsy2SxoKPA8cGO+EyQy03AucCHwDYGZLgH5JHOeccwUSQZ9ivFcCCZc4NbPvzWx7+H4GUFVSo3iZJjX6bGZr8mzKSuY455yLJ03xXwkkXOJUUjOF1VFJhxPEvG/iZZrMfYprJPUFLDzxlSRYN9U55xJScrfdFCSZJU6B04BfSdoN7ADOMLMC76qB5ILiWOA+gk7NzLAAlxX5Spxzjr3N5+JItMSpmT0APFCYPBMGRTPbDJxdmEydcy4Zqfjsc8I+RUkHSJou6WtJmyRNk3RAWRTOOVdxSYlfUUhmoOVJYDLQHGgBPAM8VZqFcs5VDsUcfS6dMiWRRmY2ycx2h68niPP4n3POJSsVg2K8Z58bhG9fDx+0/h9BMBwNvFQGZXPOVWDBQEvUpdhXvIGWhQRBMLvYl8TsM+DW0iqUc64SKOYtOaUl3rPP7cqyIM65yicVJ5lN6okWSd0lnS7pnOxXaRfM7Wvu669wUv9DGHp0DyY+OG6f/SuXf8bZJw+kV/uGPDb+vpztq1Z8zmkn9s15HdGlBZMmPliWRS+yQb1aseSfZ/LRw2dx7WmH7LP/6lN6Mu++Ucy7bxQLHhjN9ucvoX6t6rRstD8zbx/BBw+dwcIHR3PZSQdFUPrCm/PqbAYcfhDHHNqVB++9e5/9yz//jJEn9qdD8zo8/MA9ufZde8XFHNKpFccf1ausilss2c3nYjzRUiqSmRDiz8AAoCvBTZJDgLlA5Vv5PEJZWVncfsM1THhyGs2aZ3DG8P4cO2gY7Tt2zklTt14D/nDz3bw268Vcx7Zr35Eps97Jyee4wzpy3OCTyrT8RZGWJu4dewzD/jSdzG9+YO7fT+XF91bz6ZotOWnueW4x9zy3GIChh7XhipN7sGX7z1SvWoXrHn2HxSs2U6tGVd655zReXbw217GpJisrixt+92v+O/UlmrdoyUnHH8WgwcPp2LlLTpp69etz8x3jmDXjhX2OH3XmGM696FdcfemFZVnsYolqMCWeZGqKpwHHARvM7HygB1C9VEvl9rF08QJatz2AVm3aUbVaNYaMOJXXZ+cOfg0bNaZ7z96kV61aYD7vzZ1DqzbtaNGydWkXudgOO7AJK9Z/x+qN29i1ew/PvLmc4X3aFpj+9P4HMvnNLwDYsOVHFq/YDMD2Hbv4dM0WWjTcvyyKXWSLF82nbbv2tGl7ANWqVeOkU0Yx++XpudI0atyEHr0Ozfdn3KfvMdSrX7+siltsUmqOPicTFHeY2R5gt6Q6wCbAb94uY5s2rKdZi4ycz02bZ7Bxw/pC5/PyC1MYcvKokixaqWnRcH/Wbt47hWfmNz+QUUBgq1E9nUG9WvH8Oyv32de6SW16tm/E/M82llpZS8KG9etokdEy53PzFhlsXL8uzhHlX3GXIyiVMiWRZoGkesAjBCPSi4D3i3pCSasTTd2TTJoinHd7IdOPlNS1JMtQHPk9w17YTupdO3cy55UZnDDslJIqVqnK7/IKepR/2GFteHfZBrZs/znX9v33S+epP5zIbx95m207dpVCKUtOSfyMy5tUfKIlmWefsyeTHS9pJlDHzD4s3WIVnaSbgNVm9lgxsxoJvEgxpzYvKU2bt2DDusyczxvXZ9KkabNC5fHW67Pp0r0njRo3KenilYrMzT/QstHemmFGw/1Z923+k7+P6teBZ95cnmtbepU0nvrDiTw953OmvbuqVMtaEpq3yGBd5tqcz+vXZdKkWfMIS1S6JFElBW/JKbCmKKlX3hfQAEgP3yck6XlJCyV9LOniPPvaSvpU0uOSPpQ0RVLNmCRXSFokaamkzuExh0t6R9IH4f87FeZiJY0L83xVUuNwW3tJM8NyviWpczhV2gjg7nAFsPaSfilpvqQlkqbmKWup696jN1+uXsHar1aza+dOXn5hKgMGDStUHi9Pm8KQk08rpRKWvAVfbKJDi3q0aVqbqulpjOrXgZfeX71Pujo1q3F09xZMn5c78I2/cgCfrdnKP6al7N/wXHocciirVi7nqy9XsXPnTqY/9wyDhiS93lK5JCnuKwrxaor73vOxlwEDk8j/AjP7VlINgpW2pubZ3wm40MzelvQocCnwt3DfZjPrJelS4FrgIuBToF84j9rxwF+AU5MoB8D+wCIzuyZc5vDPwOXABGCsmX0hqQ/wkJkNlPQC8KKZTQGQtNXMHgnf3wZcCNyf9yRh8L8YoHlGq7y7iyw9PZ3rb/0bY38xkqysPZwyegwdOnVh8qR/AXD6mAvZvGkjo4f144ft20hLS2PSvx5i2mvzqVW7Djt2/Mi7b73GjXfel+BMqSNrj3H1+LeYfvNwqqSJx//fpyz7agsXDQ56NSbODCrxI45sx6sfrOHHn3fnHNu3azPOHtiJpau+Yd59QR/qn//zHrMWflX2F5Kk9PR0br3rXsaMOomsrCxGn3UunTp3ZdK/HwFgzPm/ZNPGDQw/7ii2b/uetLQ0/jX+AV595wNq16nD5b8cw7tvv8WWbzZzePf2/Oa6GzjjF+dHfFXxpeIay0ow32LxMg+astkdWG0JljX4H3AowZovb5pZ6zDtQOBKMxspaTVwlJllhoHqdjM7XlIr4B8EaywYUNXMOks6CJgUnqcZsBP4Nvx8nJl9IykLqB4G1AOAZ4Gjga+Bz2KKXd3Mukh6jNxBsT9wG1AvLPssMxsb7/q7HdzLnp7xZqG+s/LssF9NSpyogvn88QuiLkKZat1wv4Vx1kwplKYdutuZ46bETXPfyC5xzydpMMF8r1WAiWZ2ZwHpDgPmAaOzf6cLkswks0UiaQBwPHCkmf0oaQ6wX55keSNy7OfsHvMs9pbzVuB1MztFUltgDoCZLQV6hue9ieT6FI3gD9VWM+uZ8ILgMWCkmS2RdB7BvZvOuWIoTpeiklz3OUx3F8EE2YnLVPQiJVQX2BIGxM7AEfmkaa1gtUCAMwluCk+UZ/Zow3mFLE8awT2XAGcBc83se2CVpFEACvQI02wDasccXxtYL6kqPumuc8UmQZU0xX0lkLPus5ntJGiFnpxPuiuAqQS3EyZUmkFxJsGgzIcENbx5+aRZBpwbpmkA/DNBnn8F7pD0NkF1uTB+ALpJWkjQH3pLuP1s4EIF68J+zN4v9X/Ab8NBnfbAn4D3gFcI+jadc8WUxC05jSQtiHnFDtjmt+5zRsxnJGUQdOGNJ0nJPOYngsBxgJndIqk10MzM4t6raGY/EzwSmFfbMN9awJ78+uXMrG3M+wWETVUzexfoGJP0T/kce1MB5amV3zFmtgoYnE/6twkebcz2TxIHbedckpJco6W46z7fC/zezLKSHc1Opk/xIWAPe2tX2wiqoocldQbnnCtAleLddZNw3WeCQd3/hQGxETBU0m4ze76gTJMJin3CW2M+ADCzLQqWOi0WM1sNdC9uPs658knFf745Z91ngrGGMwjGC3LEToEYc0fJ8/EyTSYo7gpHbyzMuDFBzdE554qlSjFGNZJc97nQkgmK/wCeA5pIup1gBPeGopzMOeeylcW6z3m2n5dMnsk8+/zfcMT2OILrGGlmy5LJ3DnnCqTi1RRLSzKjz62BH4HpsdvMLHWfl3LOlQvKdwA5Wsk0n19i7wJW+wHtCB6L61aK5XLOVXDlcTU/AMws1+IW4Qw5lxSQ3DnnkpaKU4cV+tlnM1sUPlztnHNFVm5ripJ+E/MxDehFMLOMc84VncpvTTF2UoTdBH2MeedFdM65QimXNcXwpu1aZvbbMiqPc64SScUlaAoMipLSwzvGy8fK2s65ckWIKikYFePVFN8n6D9cHE7N/wzB9FsAmNmzpVw251xFpnLYfA41AL4hmCUn+35FI5jO3znnikSUv4GWJuHI80fsDYbZSm9hF+dcpVHcZ59LQ7ygWIVggaZkJnJ0zrlCS8GYGDcorjezW+Lsd865IpNIyYGWeHNUpF5pnXMVihK8Eh4vDZb0maTlkq7LZ//Jkj6UtDhc4+XoRHnGqykel0SZnHOuSETxaopJLnH6KvCCmZmkg4HJQOd4+RZYUzSzbwva55xzJSGJ1fziSbjEqZltN7PsMZD9SWI8pNATQjjnXElI8ubtRpIWxHyeYGYTwvf5LXHaZ5/zSKcAdwBNgGGJTuhB0TkXmSSWHS3uEqeY2XPAc5L6EaxBf3y8E6bgZODOucqimAMtySxxmsPM3gTaS2oUL1OvKZai/aqm0aFZraiLUWYWjB8TdRHKXMfjrom6COVWCdySk3CJU0kdgBXhQEsvoBrBE3oF8qDonItMEs3nAiW5xOmpwDmSdgE7gNExAy/58qDonItMcR99TrTEqZndBdxVmDw9KDrnIiEgLQWfEfGg6JyLTAo+5edB0TkXFZW7WXKcc67UePPZOediCdJS8E5pD4rOucjIa4rOORco7iw5pcWDonMuMikYEz0oOuei481n55wLlcd1n51zrvQkN5FsmfOg6JyLhA+0OOdcHqkXEj0oOueilIJR0YOicy4yqfjscwo+ZOOcqyzKYN3ns8N1nz+U9I6kHony9Jqicy4Songzbye57vMqoL+ZbZE0BJhAPiv+xfKaonMuGgnWfC6hdZ/fMbMt4cd5BItbxeVB0TkXmSSCYiNJC2JeF8ccnt+6zxlxTnch8HKiMnnz2TkXESXzmF+x130GkHQsQVA8OtEJvaZYjsyeNZODu3WiW+cO3P3XO/fZb2b85qor6da5A4cdcjAfLFoEwE8//cTRRx7O4b160KtHN269+c9lXfQim/v6KwzvdwhDjurBxAfG7bN/5fLPOHvEQA45oCH/Hn9fzvZVKz7n1BP65rz6dG7BpIkPlmXRi2RQ3y4see5PfDTtz1x7/qB99teptR9T7r2E956+joVT/siYEUckfWwqKmbzOal1nyUdDEwETjazuMubgtcUy42srCyuuvIyXnr5FTJatuToIw5j+PARdOnaNSfNrJkvs2L5F3y07Avef+89rrz8V7z1zntUr16dma+8Rq1atdi1axcD+x/NCScOoc8RR8Q5Y/SysrK47YZreOTJaTRrnsHoYf059oRhtO/YOSdN3XoNuO6Wu3lt1ou5jm3XviNTZ7+Tk8/AQzty3OCTyrT8hZWWJu697nSG/eoBMjduZe5/f8uLbyzl05UbctJccno/Pl25gdOuephG9Wux5Lk/8b8Z88nasyfhsakmGGgpVhbJrPvcGngWGGNmnyeTqdcUy4n5779P+/YdaHfAAVSrVo1Ro8/gxenTcqV58YVpnPWLc5BEnyOO4LvvtrJ+/XokUatWLQB27drF7l27ijXqV1aWLl5A67YH0KpNO6pWq8aQk0/ltdm5g1/DRo05qGdv0tOrFpjPvLlzaNWmHS1ati7tIhfLYd3bsmLNZlZnfsOu3Vk8M2sRwwccnCuNAbX2rw7A/jWqs+W7H9mdtSepY1OREvwXj5ntBrLXfV4GTM5e9zl77WfgRqAh8JCkxZIWJCqTB8VyYt26TFq23NtSyMhoSWZmZsI068I0WVlZ9Ondk9YtmjDw+EEc3ifuXQkpYdP69TRrvrffvGmzDDatX1/ofF5+YQpDTx5VkkUrFS2a1GXtxi05nzM3biGjcd1cacb/7w06t2vGytm3s+CZ67n27imYWVLHpqI0xX8lYmYzzKyjmbU3s9vDbeOz1342s4vMrL6Z9QxfBfVP7i1TcS+qvJDUVtJHhUh/nqQWpVmmwjDbt/84b20vXpoqVarw3sLFLF+9lgXz3+fjj5L+KiJj+fSZF7aGu2vnTubMnsEJw08pqWKVmvxqRnm/gUF9u/DhZ2s54IQ/0ueMO7jnulHU3n+/pI5NOYnu3I6oMVNpgmIRnAekTFDMyGjJ2rV77z7IzFxLixYtEqZpnidNvXr16Nd/ALNnzyzdApeAps1bsGH93trwxg2ZNG7WrFB5vPX6bLoc1JNGjZuUdPFKXOamrbRsWj/nc0bT+qz7+rtcacaMOIJpry0BYGXYXO7UtmlSx6ai4jSfS0tlC4rpkh4PH/mZIqmmpN6S3pC0UNIsSc0lnQYcCvw37IeoIelGSfMlfSRpgsq4U+7Qww5j+fIvWL1qFTt37uSZp//HsOEjcqUZdtIInnziP5gZ782bR506dWnevDlff/01W7duBWDHjh289ur/o1OnzvmcJbV079Gbr1atYO1Xq9m1cycvT5vKsYOGFSqPGdOmMPTk00qphCVrwcdf0qF1Y9q0aEjV9CqMOrEXL835MFeaNRu2MODwTgA0aVCbjm2bsipzc1LHphpR/OZzaahso8+dgAvN7G1JjwKXAacQDNV/LWk0cLuZXSDpcuBaM1sAIOkBM7slfD8JGA5Mz3uC8ObSiwFatS65jv309HTuue8BThp2IllZWZx73gV07daNRx4eD8AvLxnL4CFDmfXyDLp17kDNGjV5eOK/Adiwfj2/vOBcsrKy2GN7OPW00xk6bHiJla20pKenc/2tf+OSs0eStWcPp4weQ4dOXXh60r8AGD3mQjZv2sjoof3Yvn0baWlpPDHxIaa9Pp9ateuwY8ePvPvma/z5zvsSnCk1ZGXt4eq7JjP9ocuokiYenzaPZSs3cNFpwa11E6fM5c5HZjLh5l8wf/L1SPDH+6bxzdYfAPI9NuWl4Hif8uuHqogktQXeNLPW4eeBwPUEjwqtDJNVAdab2QmS5pA7KJ4K/A6oCTQA7jezfW8WjNG796H29nsJB7sqjBUbt0ddhDJ36PB95iCo0H5a/ODCZAYrktG9Ry+bOmtu3DSdm+9fYudLVmWrKeb9C7AN+NjMjox3kKT9gIeAQ81sjaSbgP1Kp4jOVR4pWFGsdH2KrSVlB8AzCR4Qb5y9TVJVSd3C/duA2uH77AC4WVItoHx0UjmX6nz0OXLLgHMlfUjYBCYIcHdJWgIsBvqGaR8DxktaDPwMPAIsBZ4nuJPeOVcMUjDJbLxXFCpN89nMVgNd89m1GOiXT/qpwNSYTTeEL+dcCUnF5nOlCYrOuVSjlHzc1IOicy4yKRgTPSg656JRArPklAoPis65yET1KF88HhSdc5FJxZpiZbslxzmXKhI895zMs89JLHHaWdK7kn6WdG0yxfKaonMuQqW+xOm3wJXAyGTz9Zqicy4SJTBLTjJLnG4ys/nArmTL5UHROReZMl7iNCnefHbORaasljgtDA+KzrnIFHP0OaklTgvLm8/OuUgkajonETBzljiVVI1gidMXilsuryk65yJTnGefzWx3OEP+LIIJoh/NXuI03D9eUjNgAVAH2CPpKqCrmX1fUL4eFJ1zkSnuvdtmNgOYkWfb+Jj3Gwia1UnzoOici0h0cybG40HROReJVJ0QwgdanHMuhtcUnXOR8eazc85lS+62mzLnQdE5F4lU7VP0oOici4xPMuucczG8puicczE8KDrnXIxUbD7LrNgz7bgCSPoa+DKCUzcCNkdw3qhUtuuF6K65jZk1LomMJM0kuI54NpvZ4JI4X7I8KFZAkhbEmYOuwqls1wuV85rLij/R4pxzMTwoOudcDA+KFdOEqAtQxirb9ULlvOYy4X2KzjkXw2uKzjkXw4Oic87F8KDonHMxPCg651wMD4rOVRAqztJ4LocHxQou+xdFUpWoy5KK8gsk5SW45C2n+a0kJcJvyanAJMnMTNIwgoXCNwIvmdnrERctJcR8PwOAXsAyYIGZfZ29L9ICxhFbPklnEizj+T6w1My+jbRw5ZzXFCuw8Bd+CHAr8E+gATBR0slQfmpEpSX8foYD9wBZwOXA9ZJah/tS9vuJCYhXApcCO4G/AGMltYqybOWdB8UKTFJ1oDtwNtAQOBD4KzBO0rBUrgmVlthAJ6k5cDpwMrASaE0wS/5vJbVMxe9HUlrM+65AH+DYcFM1oAVwjqSMCIpXIXjzuYLJ2+yTVBOoDzwJXGFmH0p6FegM9CSYmqlS/COQVANoaWZfSOoMbANqAPsBjwOnAl2AccBc4Eoz+ymq8uYlqSGwv5l9JWkg8BbB1FtdgevN7DhJvwKuBB4F7jGz3dGVuHzySWYrCEn7m9kPYbPvBIJflHVmNlnSLoJ5HXdLOhpYTPAL/3WERY5Ce+CU8A/F2UA/M1suqR+wxMxWS2pGEBDvSaWAGOoIXCxpKzAcONTM1ks6HtgQptlMUP7HPSAWjTefKwBJtYA3JZ0g6UDgQYJm1AhJD5jZLoJfmj8CTwGvm9nH0ZU4Gmb2EUET8zfABDNbHe5aBvyfpEnAZGCKmS2LppQFM7N3CfoOLwH+YGbfhbteAo6V9CJwG/A3M9sUUTHLPW8+VxCSzgFuAF4DppnZy2GAvBFYa2Z/kFQHaBLWjlJ6dLUk5Rmp7QwcB/QAXgHeMLNNkloS9M+tNbP3oittbvl0hxwOHA4MJmjmv2NmP0tqBBwBfGxmq6IpbcXgzedyLux4NzP7j6TtwH3AGuBlYAVwM3CXpIfN7BLge6g897TF3HYzBOhN8G/+LwSDK8OB7WFNuxtwh5n9HF1p9xUTzC8GmoSb7wR+Bn4H3BDeUlTbzG6KoIgVjgfFciz8hd8jqYOkb83sWUlbgEclLQpriyuA64CaERc3EmFAHAzcQtBsHkcw2HKRpPrAMGAg8KdUC4jZwttuRhDcWnUvsMfM/hL2jV4FdCJoUrsS4EGxnIqpAQ0C/g6sChfK+itwAfCwpKpm9gLwRZRlTQHHEAysdCHok7sNwMwmZI/Om1lmqnQpSEoL/9hll6cVMIRgVHk9wS1VVc3sPkn1AMxsa2QFrmC8T7Eck3QkcDFBk/kngj6lUcD5QD+CYNm7Mo0y5xfYJI0D2gJ1gUvMbIWkUwluVXrUzPaUfUkTk9SOYIDsSaA6QZP5dDPbJekSYKOZPR9hESskH30uRyS1lXRN+L46QbP4ODNbbGafAi8CnwKDzWwKcERlCoiQqw/ukPDVCrgfOBqYEQbEown6FZenUkCU1FfSGeH7y4CpBOWsRvAY4hNhQDyPoNm8NKKiVmjefC5fDLhGUvWwT+kK4AVJE8zsYjPbLGkbwcgqBM86Vwrh0ylPmdkASQcDU4B5QHOCmtZA4Plw30HAb8xsTlTlLUB94A5JXQhqtqcBHYCvCPqEH1DwHPtBwKlmtiKqglZkHhTLCUnpZvZl2GR+UVI1M7spfHb3SUmvEDzDezxwO4CZZUVY5DIV3sS8XdJSglttzjCz+ZIOAiYCmQS33NQE9jOz5REWN19m9pKknQQ/xyVmtlLSWoK7CZoR3oIDVKlsLYCy5H2K5UDMoEon4FuC/qWXgf+Z2e3hPXYvEExqcI6ZLQuDaKV4okFSlew/AJLGE/Sp9jWzheG2MUA3M7suwmImTcGEHY8BY83s6XDb8wTN5ykRFq1S8JpiORAGxJMIRk1XA8uB3wL3SMoyszvDX6R/EPQ1XVJZAiIENeKwS+FnMxsrqS7wBMFoMwSTPHQNR2x3RVfS5JjZtDCQ/0PBpA/vE0xW8UG0JascPCiWA5KOIHgyZVD4mkAw2nw1MCGsFd4m6Srg75KamdmGAjOsQCQdaGZfxN5jaGZnSpoiaR1Bk/Nw4JHyEBCzmdmLktIJBlueAf4v5rFEV4q8+VwOhM3j5gQd8bcBZwEPE9yuMQX41szeCNNWpmbzUIJH9p41s7cl/Rqoa2a3hPufIri/r4+ZfZYq9yEWhqT+wGoz+zLqslQWHhTLEUm3A5vCm3bHAL8mqEF8VR5/4YsrHHEeA+wimADjKIL7+NbGpOkeTgThXFK8+Vy+LAUuCZtVJxHMj/gVVJ5nmbOFgyvrgb9Keo7guearswNidv+hB0RXWH7zdvkyg2BUsi/wVwumkqqUYkabf0XQtfBfoLWk48L95ab/0KUWbz6XQ9n9hpWpyRxzW1Ls7TetCR5x/GV44/pNBCPN95rZlgiL68oxD4rlUGUKhhB/VUJJNcxsR5iuMcEUWiujLK8r37z5XA5VpoAIcVclHGFmOySlhTXIrz0guuLyoOhSngpelfDvClYl3FOZHml0pcubzy4l5e0ikK9K6MqI35LjUop8VUIXMW8+u5QhX5XQpQBvPruUIl+V0EXMa4ouJYQjyDKz/xDMKD6MYLZp2LsqYUcFqxJ+nz0fogdEV9K8pugiF3MfYgeCyS2+lXQs8ChwaVhbFMEs1DXNbEmkBXYVmgdFF6mYgJizKiGQvSphC4LZgK61YFVC50qdB0UXOfmqhC6FeFB0ZU5SW4KFl8aFN2ZPBg4xs9bh/kbAHwjWKfmPpBZmti66ErvKxAdaXBSyVyW8Ppwx+wrgW0kTAMxsM1ApVyV00fOg6MpU9qqEwJHAmZJuCueEHA50lvRKOKP28cD/g8q1KqGLnjefXZnxVQldeeBB0ZWpfFYlzF6v+vFwVcJWBKsSbjKzSyIrqKu0vPnsykyeVQknA2OBYwhWJbxU0g1mtoZgmdZGkppFVVZXeXlN0ZUZX5XQlQdeU3RlxszWmtl8oD/w3/BRvceATsBCM3sjfHIFD4guKj51mIuCr0roUpYHRReFGQQjzyOo5KsSutTjfYouMpVxVUKX+rxP0UUpC7zJ7FKL1xSdcy6G1xSdcy6GB0XnnIvhQdE552J4UHSFJilL0mJJH0l6JlyTuah5PSbptPD9REld46QdIKlvEc6xOpyjMantedJsL+S5bpJ0bWHL6FKHB0VXFDvMrKeZdQd2EjzDnENSlaJkamYXmdkncZIMAAodFJ0rDA+KrrjeAjqEtbjXJT0JLJVURdLdkuZL+lDSJRBMHybpAUmfSHoJaJKdkaQ5kg4N3w+WtEjSEkmvhrN1jwWuDmupx0hqLGlqeI75ko4Kj20oabakDyQ9DCjRRUh6XtJCSR9LujjPvnFhWV6V1Djc1l7SzPCYtyR1LpFv00XOn2hxRRY+pjcEmBluOhzobmarwsDynZkdFi458Lak2cAhBM86HwQ0BT4hWLUvNt/GwCNAvzCvBuEKf+OB7Wb2tzDdk8A9ZjZXUmtgFtAF+DMw18xukTSMYP2XRC4Iz1EDmC9pqpl9A+wPLDKzayTdGOZ9OTABGGtmX0jqAzwEDCzC1+hSjAdFVxQ1JC0O378F/IugWfu+ma0Kt58AHJzdXwjUBQ4kWIjqqXA27XWSXssn/yOAN7PzMrNvCyjH8UDXcA4JgDqSaofn+L/w2JckbUnimq6UdEr4vlVY1m+APcDT4fYngGcl1Qqv95mYc1dP4hyuHPCg6Ipih5n1jN0QBocfYjcRTPQwK0+6oQRrtMSjJNJA0P1zpJntyKcsST+VIGkAQYA90sx+lDQH2K+A5Baed2ve78BVDN6n6ErLLOBXkqoCSOooaX/gTeCMsM+xOXBsPse+C/SX1C48tkG4fRtQOybdbIKmLGG6nuHbN4Gzw21DCOZvjKcusCUMiJ0JaqrZ0oDs2u5ZBM3y74FVkkaF55CkHrgKwYOiKy0TCfoLF0n6iGAy2XTgOeALgunD/gm8kffAcH3niwmaqkvY23ydDpySPdACXAkcGg7kfMLeUfCbgX6SFhE0479KUNaZQLqkD4FbgXkx+34AuklaSNBneEu4/WzgwrB8HwMnJ/GduHLAn312zrkYXlN0zrkYHhSdcy6GB0XnnIvhQdE552J4UHTOuRgeFJ1zLoYHReeci/H/AV9uEp5uMu5DAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"Classification metrics:\\n\")\n",
    "print(metrics.classification_report(expected, predicted))\n",
    "\n",
    "cm = metrics.confusion_matrix(expected, predicted)\n",
    "mlutils.plot_confusion_matrix(cm, classifier.classes_, normalize=True, title='Normalized Confusion Matrix')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUkAAAEYCAYAAADRWAT6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAvqUlEQVR4nO3dd5gUVdbH8e9vBhhEskSJBgTBLGJGVFRYEcyivIqrLrqmXdewZkyo65rWLO66opgQE4qAiCgLJoIYMIGCiCAZBIkznPePuoPtONMzTKru6fPZp5+prrpddXrWOdxb99a9MjOcc84VLivuAJxzLpV5knTOuSQ8STrnXBKeJJ1zLglPks45l4QnSeecS8KTpKsQkraS9JqklZJeKMN5+kl6szxji4OkUZL6xx2H23KeJDOcpNMkTZG0WtKC8Md8UDmc+kSgKbCNmZ1U2pOY2dNmdmQ5xPMbkrpJMkkvFdi/e9j/TgnPc4OkocWVM7OeZjaklOG6GHmSzGCS/gbcC9xKlNBaAw8Bfcrh9G2Ab8wstxzOVVEWAwdI2iZhX3/gm/K6gCL+d5bOzMxfGfgC6gGrgZOSlMkhSqLzw+teICcc6wbMAy4FFgELgD+GYzcCG4CN4RpnAzcAQxPO3RYwoFp4fybwHbAKmA30S9g/MeFzBwCTgZXh5wEJx94BbgYmhfO8CTQq4rvlx/8IcEHYlx32XQ+8k1D2X8APwM/AVODgsL9Hge/5SUIcg0Ica4Edw75zwvGHgeEJ5/8HMA5Q3P9d+Ov3L/8XLnPtD9QEXk5S5hpgP2APYHegC3BtwvFmRMm2BVEifFBSAzMbSFQ7fd7MapvZf5IFImlr4D6gp5nVIUqE0wsp1xAYGcpuA9wNjCxQEzwN+CPQBKgBXJbs2sCTwBlh+yhgBtE/CIkmE/0OGgLPAC9Iqmlmowt8z90TPnM6MACoA3xf4HyXArtJOlPSwUS/u/4WMqZLLZ4kM9c2wBJL3hzuB9xkZovMbDFRDfH0hOMbw/GNZvYGUW2qfSnj2QTsImkrM1tgZjMKKXM0MNPMnjKzXDN7FvgKOCahzH/N7BszWwsMI0puRTKz94CGktoTJcsnCykz1MyWhmveRVTDLu57PmFmM8JnNhY43xrg/4iS/FDgIjObV8z5XEw8SWaupUAjSdWSlNmW39aCvg/7Np+jQJJdA9Te0kDM7BfgFOA8YIGkkZI6lCCe/JhaJLz/qRTxPAVcCBxKITVrSZdK+jL01K8gqj03KuacPyQ7aGYfEd1eEFEydynKk2Tmeh9YBxybpMx8og6YfK35fVO0pH4BaiW8b5Z40MzGmNkRQHOi2uFjJYgnP6YfSxlTvqeA84E3Qi1vs9Ac/jtwMtDAzOoT3Q9VfuhFnDNp01nSBUQ10vnAFaWO3FU4T5IZysxWEnVQPCjpWEm1JFWX1FPSHaHYs8C1khpLahTKFzvcpQjTga6SWkuqB1yVf0BSU0m9w73J9UTN9rxCzvEGsFMYtlRN0ilAR+D1UsYEgJnNBg4hugdbUB0gl6gnvJqk64G6CccXAm23pAdb0k7ALURN7tOBKyTtUbroXUXzJJnBzOxu4G9EnTGLiZqIFwKvhCK3AFOAT4HPgGlhX2muNRZ4PpxrKr9NbFlEnRnzgWVECev8Qs6xFOgVyi4lqoH1MrMlpYmpwLknmllhteQxwCiiYUHfE9W+E5vS+QPll0qaVtx1wu2NocA/zOwTM5sJXA08JSmnLN/BVQx5h5pzzhXNa5LOOZeEJ0nnnEvCk6RzziXhSdI555JINpDYlZGqbWWqUSfuMCrNzju2jDuESlejWmbVM6Z/PHWJmTUuj3Nl121jlrs2aRlbu3iMmfUoj+uVlifJCqQadchpf3LcYVSaZ169Le4QKl2bRrWKL1SFNNi6WsEnnkrNcteR06Fv0jLrPr6/uCebKpwnSedcPARIxRaLmydJ51x8srLjjqBYniSdczERpMF8xJ4knXPxEF6TdM65osnvSTrnXFJek3TOuaL4PUnnnCuaDwFyzrlkBFmpn4JSP0LnXNWV5TVJ55wrnA8Bcs65ZLzjxjnnkvOOG+ecK4KUFs3t1K/rOueqLmUlf5XkFNIcSZ9Jmi5pStjXUNJYSTPDzwYJ5a+SNEvS15KOKu78niSdczEJNclkr5I71Mz2MLPO4f2VwDgzaweMC++R1BHoC3QCegAPSUp6IU+Szrn4SMlfpdcHGBK2hwDHJux/zszWm9lsYBbQJdmJPEk65+KhMJg82QsaSZqS8BpQyJkMeFPS1ITjTc1sAUD42STsbwH8kPDZeWFfkbzjxjkXn+Jri0sSmtBFOdDM5ktqAoyV9FWyKxayz5Kd3GuSzrn4lEPHjZnNDz8XAS8TNZ8XSmoOEH4uCsXnAa0SPt4SmJ/s/J4knXPxUNk7biRtLalO/jZwJPA5MALoH4r1B14N2yOAvpJyJG0HtAM+SnYNb26noa9G3siqX9aTt2kTuXmbOKjfHVxz7h846/gDWLx8NQADHxjBmIlfUK1aFg9f3489OrSiWnYWT4/8iDsffzPmb7BlBl52PhPeHk3DbRrz4tgPAXhz5Ms8cs9tzJ71NUNHjKfTbnsB8OMP33P84fvQZod2AOy25z5ce+u9cYVeZuvWrePoI7uxfv0G8vJy6X3s8Vx17Q0sX7aMs844lblzv6d16zb896nnqN+gQfEnTDVlH0zeFHhZ0XmqAc+Y2WhJk4Fhks4G5gInAZjZDEnDgC+AXOACM8tLdgFPkmmqx4B/sXTFL7/Zd//Q8dz71Ljf7Duh+17k1KjGPiffylY1q/Pxi9cybNQU5i5YVpnhlknvk/rRt/8Arv3buZv37bhTR+5+9Gluvvovvyvfss12DBs1qTJDrDA5OTm8+sZb1K5dm40bN9Kze1e6H9mD1159ma7dDuOSy/7OPXf+g3vu+gc33nJ73OFuEQFZWWVrzJrZd8DuhexfChxexGcGAYNKeg1vbldxhlGrZg2ys7PYKqcGGzbmseqXdXGHtUX23vdA6tb/bS1p+3btaRtqi1WZJGrXrg3Axo0b2bgxF0mMGvkap/Y7A4BT+53BG6+PiDPM0lEJXinAk2QaMjNee+hCJj19BWcdf+Dm/ef17cpHz1/FIwP7Ub/OVgC89NbHrFm3gdljB/HNqJu498lxLP95TVyhV4off/ieU3oexNkn92TaR+/FHU6Z5eXlcfB+e7NT2+Z0O+xwOu+zL4sWLaRZ8+YANGvenMWLFxVzllQkpOSvVJBRzW1Jc4DOZrakLGXidtgf72HB4pU0blCb1x+5kK/n/MRjL/yP2x4bhRkMPL8Xt//teM678Wn26dSWvLxNbH/kNTSoU4u3Hr+Etz/8ijk/Lo37a1SIxk2aMfr9GdRvsA1ffPYxl/zpNF4c+yG169SNO7RSy87O5n8fTGXlihX836kn8MWMz+MOqdyUtbldGVI/Qvc7CxavBGDx8tWMePtT9unUlkXLVrFpk2FmPP7SJDrv0gaAk3t25s33viA3dxOLl6/m/enfsXfH1nGGX6Fq5ORQv8E2AHTcdU9attmO72fPijmq8lGvfn0OOvgQxo0dQ5MmTflpwQIAflqwgMaNmxTz6dSUDjXJKpskJb0SRuDPKDhKX1JbSV9JGiLpU0nDJdVKKHKRpGnhofkO4TNdJL0n6ePws32lfqGgVs0a1K6Vs3m7+/4dmPHtfJo1+rWm1Oew3fni2+gPaN5Py+i2T/vN5bvs1pav5yys/MArybKlS8jLizor582dzdzZ39Kyddt4gyqDJYsXs3LFCgDWrl3LO+PH0a59e3r8oRfPPv0kAM8+/SQ9jz4mxihLRxLKSv5KBVW5uX2WmS2TtBUwWdKLBY63B842s0mSHgfOB+4Mx5aY2V6SzgcuA84BvgK6mlmupO7ArcAJlfNVftVkmzo8f/efAKiWnc3zo6Yw9r0v+c/NZ7Bb+5aYGd8vWMZFtzwLwCPPT2Dwjf/H1OHXIMFTr37A5zOTjp1NOVde9EemvD+RFcuXcuS+HfjzJVdTr34Dbh94OcuXLeGiP55E+4678vBTrzDtw0k8dPcgqlWrRlZWNtfeei/16jeM+yuU2k8/LeD8AWeRl5fHpk2bOO6EE+nRsxdduuzPH0/vy9An/0vLlq14YujzcYdaKqlSW0xGZkmfyElbkm4Ajgtv2wJHAc8BnYHawAQzax3KHgZcbGbHhnuSB5rZj5L2BQaZWXdJrYD7iAafGlDdzDoUct0BQFRzrV5775qd+hcsUmV98OptcYdQ6do0qlV8oSqkwdbVppbgMcESqbbN9lb3D7ckLbN8aL9yu15pVcnmtqRuQHdgfzPbHfgYqFmgWMF/HRLfrw8/8/i1tn0zMN7MdgGOKeR80UnMBptZZzPrrGpblfo7OFflibRoblfJJAnUA5ab2ZpwT3G/Qsq0lrR/2D4VmFiCc/4Yts8slyidy3DecROf0UA1SZ8S1QA/KKTMl0D/UKYh8HAx57wDuE3SJCD155x3LsUJkZWVlfSVCqpkx42ZrQd6FnKoLYCk2sAmMzuvkM+2TdieAnQL2+8DOyUUva7cAnYuU6VGZTGpKpkknXNpQOkxmDwjk6SZzQF2iTsO5zJdqtx3TCYjk6RzLn4idTpnkvEk6ZyLRxgClOo8STrnYuM1SeecS8Jrks45l4TXJJ1zrgip9FRNMp4knXOx8XGSzjmXTOpXJD1JOudi4k/cOOdc0UR5LLtd8TxJOudiIrLSYAhQ6td1nXNVVnnMJykpO6w99Xp431DSWEkzw88GCWWvkjRL0teSjirJ+T1JOufioai5nexVQn8hmh8235XAODNrB4wL75HUEegLdAJ6AA9JKnZuWE+SzrlYCMjOVtJXseeQWgJHA/9O2N0HGBK2hwDHJux/zszWm9lsYBbQpbhreJJ0zsWmBM3tRpKmJLwGFDjFvcAVwKaEfU3NbAFA+Jm/KHkL4IeEcvPCvqS848Y5FwuJknTcLClqtURJvYBFZjY1LP5X7CUL2VfscrGeJJ1zMSnzY4kHAr0l/YFo9dK6koYCCyU1N7MFkpoDi0L5eUCrhM+3BIpdhN6b28652JSl48bMrjKzlmFdqr7A22b2f8AIIH/B+/7Aq2F7BNBXUo6k7YB2wEfFxeg1SedcPErW3C6N24Fhks4G5gInAZjZDEnDgC+AXOACM8sr7mSeJJ1zsYieuCmfJGlm7wDvhO2lwOFFlBsEDNqSc3uSdM7FJh2euPEk6ZyLjT+7neF22akVI8beGXcYlWbgm9/EHUKlu6PXznGHkLZKOAQodp4knXMx8ZnJnXMuqTTIkZ4knXMx8ea2c84VrTyHAFUkT5LOudh4TdI555LwmqRzzhVlyybWjY0nSedcLJQma9wUmSQl3U+SudbM7OIKicg5lzGy0qAqmawmOaXSonDOZZy0f+LGzIYkvpe0tZn9UvEhOecyRRrkyOIn3ZW0v6QvCKuRSdpd0kMVHplzrsorjyVlK1pJZia/FzgKWApgZp8AXSswJudcBhDRPclkr1RQot5tM/uhQFYvdjZf55wrTjo0t0uSJH+QdABgkmoAF/PbhcCdc27LKc2HACU4D/gX0fq0PwJjgAsqMijnXNWX39xOdcUmSTNbAvSrhFiccxkmHWqSJend3l7Sa5IWS1ok6VVJ21dGcM65qqu45WRTpZJZkt7tZ4BhQHNgW+AF4NmKDMo5lxnSoXe7JElSZvaUmeWG11CSPK7onHMllQ5JMtmz2w3D5nhJVwLPESXHU4CRlRCbc64Kizpu4o6ieMk6bqYSJcX8r3FuwjEDbq6ooJxzGaAchgBJqglMAHKI8tlwMxsYKnnPA22BOcDJZrY8fOYq4Gyi8d4Xm9mYZNdI9uz2dmWK3jnnilEOjx6uBw4zs9WSqgMTJY0CjgfGmdntoSV8JfB3SR2BvkAnoj6WtyTtZGZFPiBToiduJO0CdARq5u8zsydL+61c+fh21jdcdM7pm9//8P1sLvn7dSxfvoyxo18nS1ls07gxd94/mKbNto0x0tKrniWuPmIHqmWJbInJP6zk5c8WsnWNbM4/sDWNatdgyeoNPDhxLms25tGpWW1O3qMZ2Vkib5Px3McL+HJh+s/LkpeXx1GH7EezbVswdNgrDDjzNL6dFa1zvnLlSurVq8e4iek1cVd5NLfNzIDV4W318DKgD9At7B8CvAP8Pex/zszWA7MlzQK6AO8XdY1ik6SkgeFiHYE3gJ7ARMCTZMx22HEn3njnQyD6I9pv1x048uje1KvfgEuvGgjAfwc/yH133sagO++PM9RS27jJuH3cd6zP3US24JojduTT+avYu1Vdvli4mpHjF3N0x8b06tSYYdN/YtX6XO55dw4r1ubSol4Olx+6PX99Jf0fEHvs4ftp174Dq1atAmDwE89sPjbwmiuoW7duXKGVSQk6ZxpJSsz+g81scGIBSdlEtwd3BB40sw8lNTWzBQBmtkBSk1C8BfBBwsfnhX1Fx1j81+BE4HDgJzP7I7A7UfvfpZBJE8bTpu12tGzVhjp1fv2DWbtmTcrMplJa63M3AZCdJbKzhGHs1bIeE79bDsDE75azV8t6AMxdvo4Va3MB+HHleqpni2rp0DuQxPwf5/HWmFH0O+Os3x0zM157eTjHnXhKDJGVjVSi3u0lZtY54TW44HnMLM/M9gBaAl1Cy7fIyxayL+lonZI0t9ea2SZJuZLqAosAH0yeYl5/+QWOOf7kze//OWggLw97mjp16/HMy6NjjKzsJLixRzua1q7BuJlL+W7pWurWrMbKdVEyXLkul7o1s3/3uc6t6vH98rXkbkrvEWvXXXkp1910G6tXr/rdsQ/em0ijxk3Yfod2MURWduX5xI2ZrZD0DtADWCipeahFNifKWxDVHFslfKwlMD9pjCW49hRJ9YHHiKq004CPtiz8X0maI6lRWcuU4rqriy/1m/LHhpu8KW/Dhg28NWYkf+h9/OZ9l19zI+99Mos+J/Tlyf88EmN0ZWcG14+aySWvfMn229SiRb3iGzIt6uVwyh7NeOKjHyshworz5uiRNGrchN333KvQ4y8Pfz4ta5H5yvrEjaTGIT8haSugO/AVMALoH4r1B14N2yOAvpJyJG0HtKOYfFZskjSz881shZk9AhwB9A/N7pQk6QZJZ5bDqY4lug+b8t4ZN4ZOu+1B4yZNf3es9wknM/r1Vyo/qAqwZuMmvlq4mt2a1+HndbnUqxk1hOrVrMbP637tnGywVXUuPrgtg9//gUWrN8QVbrmY/MF7vDnqdTrv2o7zzvo/Jk0YzwV/iv72c3NzeeO1V+hz/EkxR1k6kjbfQinqVQLNicZyfwpMBsaa2evA7cARkmYS5a3bAcxsBtEThF8Ao4ELkvVsQ5IkKWmvgi+gIVAtbBdL0iuSpkqaIWlAgWNtJX0laYikTyUNl1QrochFkqZJ+kxSh/CZLpLek/Rx+Nm+JHEkXPOucM5xkhqHfTtIGh3i/J+kDmFquN7APyVND2X+JGmypE8kvVgg1li99tIweh/3a1N79rezNm+/NXok2++4UxxhlYs6OdnUqh79Z1o9W3RsVof5P6/n43k/c9D2DQA4aPsGTJu3EoBa1bP4W7e2vPDJAmYuWRNb3OXlmhsG8fGXs5ny2UweeXwoB3Y9lAcfi1ZWmfDOOHbcqT3btmgZc5SlV9aZyc3sUzPb08x2M7NdzOymsH+pmR1uZu3Cz2UJnxlkZjuYWXszG1XcNZLdk7wrWWzAYcV+AzjLzJaFavBkSS8WON4eONvMJkl6HDgfuDMcW2Jme0k6H7gMOIeoGt3VzHIldQduBU4oQRwAWwPTzOxSSdcDA4ELgcHAeWY2U9K+wENmdpikEcDrZjYcQNIKM3ssbN9CNBj1d13G4R+DAQDbtmxV8HC5W7tmDRPffZtBdz2wed8dN1/Ld9/ORFlZtGjZmkF33lfhcVSU+ltV50/7tSJL0R/UR3NX8Mn8VcxasoYLDmpN1x0asvSXaAgQQPedGtG0Tg69d2lK712imvU/3/6OVeur3jzRr7w4jONOSN+mNpTsfl/cFA0zqqCTSzcAx4W3bYmWgXgO6AzUBiaYWetQ9jCi0e/HSpoDHGhmP4bENcjMuktqBdxHdB/BgOpm1kHSrsBT4TrNgA1A/r8ch5vZUkl5QE5IsNsDLwEHAYuBrxPCzjGznSU9wW+T5CHALUD9EPsYMzsv2fffbY+9bcRbk7bod5bOBr75TdwhVLo7eu0cdwiVqlm9GlPNrHN5nKvpjrvYqXcNT1rmX8fuXG7XK60SDSYvDUndiG6i7m9ma0KvU80CxQpm6MT368PPPH6N82ZgvJkdJ6kt0QBRzOwzYI9w3RuAOWb2RDEhGtE/ZCvC8IHiPAEca2afhHue3UrwGedcEukwOqsia7v1gOUhQXYA9iukTGtJ+4ftU4kGqRd3zvzuyjO3MJ4sojGfAKcBE83sZ6JR9ycBKLJ7KLMKqJPw+TrAgvDok09C7FwZSZRHx02Fq8gkOZqok+dTohrgB4WU+RLoH8o0BB4u5px3ALdJmgT8fmBccr8AnSRNJbqfelPY3w84W9InwAyix5Ygui1weegk2gG4DvgQGEt0b9Q5V0bpMOluSR5LFFEi2d7MbpLUGmhmZknHFoVnI3sWcqhtOG9tYFNh9/XMrG3C9hRC09bM3gcSu2qvK+SzNxQRT+3CPmNms4kGnxYsP4nfDgF6mOKTuHOuhNJljZuS1CQfAvYnag5D1Ax9sMIics5ljGwlf6WCknTc7BuG4nwMYGbLFS0tWyZmNgdI9oylc64KUwrNPp5MSZLkRkWzbBhEjwEBmyo0KudcRshOg4GSJUmS9wEvA00kDSLqIb62QqNyzlV56XJPsiTrbj8deoQPJ/pex5pZ+k/Q55yLl6pITTL0Zq8BXkvcZ2ZzKzIw51zVp0Knd0wtJWluj+TXBcFqAtsRPcbXqQLjcs5VcVVhtUQAzGzXxPdhBqBziyjunHMllipP1SSzxc9um9k0SftURDDOucxRZWqSkv6W8DYL2Ito5hznnCs9VZ2aZOIkD7lE9ygLzgvpnHNbpErUJMMg8tpmdnklxeOcyyBpMEyy6CQpqVqYoLZESzU459yWECI7DbJksprkR0T3H6eHpQxeIJpuDAAze6mCY3POVWWqAs3toCGwlGgOxvzxkka0/IFzzpWKSP+OmyahZ/tzfk2O+dJ7tXfnXEpI92e3s4kWvCrsW3iSdM6VWRrkyKRJckH+GrbOOVfeJNKi4ybZHBypH71zLq2pmFexn5daSRov6UtJMyT9JexvKGmspJnhZ4OEz1wlaZakryUdVdw1kiXJw0sQo3POlYqIapLJXiWQC1xqZjsTrch6gaSOwJXAODNrB4wL7wnH+hJN0NMDeCiMBy9SkUnSzJaVJELnnCutsq6WaGYLzGxa2F5FtAJrC6JVT4eEYkOAY8N2H+A5M1sfFgGcBXRJdo0tnuDCOefKQwkHkzeSNCXh/WAzG1zo+aS2wJ5ESz83NbMFECVSSU1CsRb8dnnreWFfkTxJOudio+KT5BIz61yC89QmmlPir2b2c5LzbvFonTSYPN05V1WVteMGQFJ1ogT5dMKTgAslNQ/HmwOLwv55QKuEj7cE5ic7v9ckK1D1bNGsfs24w6g0d/fpGHcIle6D2UvjDiFtlccQIEVVxv8AX5rZ3QmHRgD9gdvDz1cT9j8j6W5gW6Ad0SPYRfIk6ZyLTQma28U5EDgd+EzS9LDvaqLkOEzS2cBc4CQAM5shaRjwBVHP+AVmlpfsAp4knXOxKeuj22Y2kaJb5oUOYzSzQcCgkl7Dk6RzLhYCstLgmRVPks652KTBU4meJJ1zcVHazwLknHMVxpvbzjmXjCArDUZqe5J0zsVGXpN0zrnC5c8ClOo8STrnYpMGOdKTpHMuPt7cds65IlSFdbedc67ilHBi3bh5knTOxcI7bpxzrhipnyI9STrn4pQGWdKTpHMuNv7stnPOJZH6KdKTpHMuJqJcZiavcJ4knXPx8CFAzjmXnCdJ55wrkvyxRFd5vvn6a04/7ZTN72fP/o7rBt7ERX/5a3xBVYB169ZxzFGHsmH9enJz8zjm2OO58tqBADz28AP8e/DDVMuuxhE9enLDLbfHHG3p3HvdX5k8YSz1GjbioZffBeAflw1g3pxvAfhl1Uq2rlOP+4ePY+GPc/lzn660aLsDAO1325sLr78jtti3lNckXaXZqX17Ppw6HYC8vDx2aNOC3sceF29QFSAnJ4eXR46ldu3abNy4kaOPOITuRx7F2rXrGDXyNSZ8MI2cnBwWL1pU/MlSVPc+p9Dr1LO4+5qLNu/7+52DN2//+58D2bp23c3vm7Vqw/3Dx1VqjOUh6riJO4ripcG8wG5LjX97HNttvwNt2rSJO5RyJ4natWsDsHHjRjZu3Igknvj3o/zl0ivIyckBoHGTJnGGWSa7dN6fOvXqF3rMzJg45jW6/qFq/AOoYv6XCjxJVkEvPP8cJ59yatxhVJi8vDy67b83O2+3Ld0O687e++zLt7O+4f1JEzmy2wEcc9RhTJs6Oe4wK8SMqR9Qf5tGtGiz/eZ9C3+cy8UndefKM4/l86kfxBjdlstS8lcqyJgkKamtpM+3oPyZkratyJgqwoYNGxj5+giOP/GkuEOpMNnZ2bzz/lQ+/XoO06ZM5ssZn5Obm8fKFcsZM34SNw66nXPOOA0zizvUcvfuqJd/U4ts2Lgp/31zKve98BbnXH4jd/79fNasXhVjhFtAJXgVdwrpcUmLEv+2JTWUNFbSzPCzQcKxqyTNkvS1pKNKEmbGJMlSOBNIuyQ5ZvQo9thzL5o2bRp3KBWuXv36HHjwIYx76022bdGCo3sfhyT26tyFrKwsli5ZEneI5SovN5f333qDrkf12byveo0c6tZvCMCOnXanWas2/Pj9t3GFuMXKobn9BNCjwL4rgXFm1g4YF94jqSPQF+gUPvOQpOziLpBpSbKapCGSPpU0XFItSXtLelfSVEljJDWXdCLQGXha0nRJW0m6XtJkSZ9LGqwUfVRg2PPPVumm9pLFi1m5YgUAa9euZcL4cbTbqT09e/Xmf++OB2DWzG/YsGED2zRqFGOk5W/6BxNoud2ONGr267/dK5ctIS8vD4Cffvie+XNn06xletyLFmVvbpvZBGBZgd19gCFhewhwbML+58xsvZnNBmYBXYq7Rqb1brcHzjazSZIeBy4AjgP6mNliSacAg8zsLEkXApeZ2RQASQ+Y2U1h+ymgF/BawQtIGgAMAGjVunWlfKl8a9as4e23xvLAQ49W6nUr08KFC7hwwFnk5eWxaZPR5/gTOarn0WzYsIGL/3wOB+2zB9VrVOeBRx9Pi0feCnPHFefx2eT3+HnFMvofvif9LricI48/jQmjXvldh83nUz/g6QfvICu7GtnZ2Vxw3R3UqdegiDOnoOL/L2okaUrC+8FmNrjI0pGmZrYAwMwWSMrvxWsBJN60nRf2JZVpSfIHM5sUtocCVwO7AGPDH1Q2sKCIzx4q6QqgFtAQmEEhSTL8HzgYYO+9O1fqTbFatWrx48KllXnJStdpl90Y/96U3+2vUaMGj/znyRgiKn9X3PFIofsvGXTf7/YdeEQvDjyiV0WHVGFKMAvQEjPrXE6XK+xixf6NZlqSLPgLWQXMMLP9k31IUk3gIaCzmf0g6QagZsWE6FzmqKC6/kJJzUMtsjmQP2h2HtAqoVxLYH5xJ8u0e5KtJeUnxFOJqt6N8/dJqi6pUzi+CqgTtvMT4hJJtYETKytg56q0MvZuF2EE0D9s9wdeTdjfV1KOpO2AdsBHxZ0s02qSXwL9JT0KzATuB8YA90mqR/T7uJeoKf0E8IiktcD+wGPAZ8AcoGoOwnOuEklln3RX0rNAN6J7l/OAgcDtwDBJZwNzgZMAzGyGpGHAF0AucIGZ5RV3jYxJkmY2B+hYyKHpQNdCyr8IvJiw69rwcs6Vk7I2t82sqKEchxdRfhAwaEuukTFJ0jmXapQWIxA8STrnYpMGOdKTpHMuHukyC5AnSedcbFJlpp9kPEk652LjNUnnnCtKCk2HlownSedcjFI/S3qSdM7FIn8WoFTnSdI5Fxu/J+mcc0l477ZzziXhNUnnnCuC5EnSOeeS8me3nXMuidRPkZ4knXOxUZnnk6wMniSdc7FIlwkuMm35Buec2yJek3TOxcab2845VxQfAuScc0VLl3uSniSdc7HxxxKdcy4Jr0k651wSniSdcy6JdGhuy8zijqHKkrQY+D6GSzcClsRw3bhk2veF+L5zGzNrXB4nkjSa6Hsks8TMepTH9UrLk2QVJGmKmXWOO47KkmnfFzLzO8fFn7hxzrkkPEk651wSniSrpsFxB1DJMu37QmZ+51j4PUnnnEvCa5LOOZeEJ0nnnEvCk6RzziXhSdI555LwJOlcFaF0WHowDXmSrOLy/3AkZccdSyoqLLGkS7IpGKf5UJUK4UOAqjBJMjOTdDTQF1gIjDSz8TGHlhISfj/dgL2AL4EpZrY4/1isASaRGJ+kU4GWwEfAZ2a2LNbgqhivSVZhIQH0BG4GHgYaAv+W1AfSp8ZUUcLvpxdwD5AHXAhcLal1OJayv5+EBHkxcD6wAbgVOE9Sqzhjq2o8SVZhknKAXYB+wDZAO+AO4C5JR6dyTamiJCY+Sc2Bk4E+wHdAa6JVBS6X1DIVfz+SshK2OwL7AoeGXTWAbYEzJLWIIbwqyZvbVUzBZqKkWkAD4BngIjP7VNI4oAOwB9FUVBnxH4GkrYCWZjZTUgdgFbAVUBMYApwA7AzcBUwELjazdXHFW5CkbYCtzWyupMOA/xFNNdYRuNrMDpf0Z+Bi4HHgHjPLjS/iqsEn3a0iJG1tZr+EZuKRRH84881smKSNRPNa5ko6CJhOlAAWxxhyHHYAjgv/cPQDuprZLEldgU/MbI6kZkQJ8p5USpDBTsAASSuAXkBnM1sgqTvwUyizhCj+IZ4gy4c3t6sASbWBCZKOlNQOeJCo2dVb0gNmtpHoj+ga4FlgvJnNiC/ieJjZ50RN0r8Bg81sTjj0JXC8pKeAYcBwM/syniiLZmbvE917PBe4ysxWhkMjgUMlvQ7cAtxpZotiCrPK8eZ2FSHpDOBa4G3gVTMbFRLm9cA8M7tKUl2gSag9pXTvbXkq0BPcATgc2B0YC7xrZosktSS6vzfPzD6ML9rfKuT2SRegC9CD6LbAe2a2XlIjYD9ghpnNjifaqsmb22ku3Mg3M3tS0mrgX8APwCjgW+BG4B+SHjWzc4GfIXPG1CUM8+kJ7E303/ytRJ01vYDVoSbeCbjNzNbHF+3vJST3AUCTsPt2YD1wBXBtGMJUx8xuiCHEKs+TZBoLCWCTpB0lLTOzlyQtBx6XNC3UJr8FrgRqxRxuLEKC7AHcRNTMvouo8+YcSQ2Ao4HDgOtSLUHmC8N8ehMN5boX2GRmt4Z7q38F2hM1wV0F8CSZphJqSEcAdwOzw8JjdwBnAY9Kqm5mI4CZccaaAg4m6qjZmeie3i0AZjY4v/ffzH5MlVsQkrLCP3758bQCehL1Wi8gGsJV3cz+Jak+gJmtiC3gKs7vSaYxSfsDA4ia2OuI7kmdBPwR6EqUPPfOpF7swhKdpLuAtkA94Fwz+1bSCURDox43s02VH2nxJG1H1OH2DJBD1MQ+2cw2SjoXWGhmr8QYYkbw3u00IqmtpEvDdg5RM/pwM5tuZl8BrwNfAT3MbDiwXyYlSPjNPbw9w6sVcD9wEPBGSJAHEd2XnJVKCVLSAZL6hu0LgBeJ4qxB9Njk0JAgzyRqZn8WU6gZxZvb6cWASyXlhHtSFwEjJA02swFmtkTSKqKeW4ie1c4I4emZZ82sm6TdgOHAB0BzoprYYcAr4diuwN/M7J244i1CA+A2STsT1XxPBHYE5hLdU35A0XP4uwInmNm3cQWaSTxJpglJ1czs+9DEfl1SDTO7ITx7/IyksUTPIHcHBgGYWV6MIVeqMKh6taTPiIb29DWzyZJ2Bf4N/Eg0xKcWUNPMZsUYbqHMbKSkDUT/P35iZt9Jmkc0WqEZYcgPkJ1pLYQ4+T3JNJDQSdMeWEZ0f2oU8JyZDQpj/EYQTdJwhpl9GZJqRjxxISk7/x8ESY8Q3ZM9wMymhn2nA53M7MoYwywxRROQPAGcZ2bPh32vEDW3h8cYWkbymmQaCAnyGKJe2TnALOBy4B5JeWZ2e/jDuo/oXtW5mZIgIaoxh1sQ683sPEn1gKFEvdkQTVrRMfQIb4wv0pIxs1dDYr9P0SQWHxFNvvFxvJFlJk+SaUDSfkRPzhwRXoOJerMvAQaHWuMtkv4K3C2pmZn9VOQJqxBJ7cxsZuIYRzM7VdJwSfOJmqhdgMfSIUHmM7PXJVUj6rx5ATg+4TFKV4m8uZ0GQnO6OdGN/VuA04BHiYaHDAeWmdm7oWwmNbP/QPSI4UtmNknSX4B6ZnZTOP4s0fjCfc3s61QZB7klJB0CzDGz7+OOJVN5kkwjkgYBi8Ig4tOBvxDVMOamYwIoq9CjfTqwkWhCjwOJxhHOSyizS5jYwrlS8eZ2evkMODc0w44hmh9yLmTOs9j5QmfNAuAOSS8TPZd9SX6CzL//6AnSlZUPJk8vbxD1eh4A3GHR1FkZKaE3+89EtyKeBlpLOjwcT5v7jy61eXM7DeXfd8ykJnbCMKjE4T6tiR7J/FMYSH8DUU/2vWa2PMZwXRXiSTINZVJyhOSrPkrayszWhnKNiaYM+y7OeF3V4s3tNJRJCRKSrvrY28zWSsoKNczFniBdefMk6VKeil718W5Fqz5uyqRHMF3l8ua2S0kFbynIV310MfEhQC6lyFd9dCnGm9suZchXfXQpyJvbLqXIV310KcZrki4lhB5qmdmTRDOuH000Gzf8uurjTopWffw5fz5IT5CuonlN0sUuYRzkjkSTdSyTdCjwOHB+qE2KaJbuWmb2SawBu4ziSdLFKiFBbl71Echf9XFbotmOLrNo1UfnKp0nSRc7+aqPLoV5knSVTlJbooWs7goDxYcBe5pZ63C8EXAV0TovT0ra1szmxxexy2TecePikL/q49VhRvGLgGWSBgOY2RIgI1d9dKnHk6SrVPmrPgL7A6dKuiHMidkL6CBpbJhxvDvwFmTWqo8u9Xhz21UaX/XRpSNPkq5SFbLqY/564UPCqo+tiFZ9XGRm58YWqHOBN7ddpSmw6uMw4DzgYKJVH8+XdK2Z/UC0LG4jSc3iitW5fF6TdJXGV3106chrkq7SmNk8M5sMHAI8HR4tfAJoD0w1s3fDkzV4gnSpwqdKc3HwVR9d2vAk6eLwBlHPdm8yfNVHl/r8nqSLTSau+ujSj9+TdHHKA29iu9TmNUnnnEvCa5LOOZeEJ0nnnEvCk6RzziXhSdJtMUl5kqZL+lzSC2FN7NKe6wlJJ4btf0vqmKRsN0kHlOIac8IclSXaX6DM6i281g2SLtvSGF3q8iTpSmOtme1hZrsAG4iewd5MUnZpTmpm55jZF0mKdAO2OEk6VxaeJF1Z/Q/YMdTyxkt6BvhMUrakf0qaLOlTSedCNF2apAckfSFpJNAk/0SS3pHUOWz3kDRN0ieSxoXZzM8DLgm12IMlNZb0YrjGZEkHhs9uI+lNSR9LehRQcV9C0iuSpkqaIWlAgWN3hVjGSWoc9u0gaXT4zP8kdSiX36ZLOf7EjSu18FhhT2B02NUF2MXMZodEs9LM9glLNEyS9CawJ9Gz2rsCTYEviFZFTDxvY+AxoGs4V8OwguIjwGozuzOUewa4x8wmSmoNjAF2BgYCE83sJklHE62fU5yzwjW2AiZLetHMlgJbA9PM7FJJ14dzXwgMBs4zs5mS9gUeAg4rxa/RpThPkq40tpI0PWz/D/gPUTP4IzObHfYfCeyWf78RqAe0I1rY69kw2/h8SW8Xcv79gAn55zKzZUXE0R3oGObEAKgrqU64xvHhsyMlLS/Bd7pY0nFhu1WIdSmwCXg+7B8KvCSpdvi+LyRcO6cE13BpyJOkK421ZrZH4o6QLH5J3EU0ccWYAuX+QLTGTTIqQRmIbhftb2ZrC4mlxE9JSOpGlHD3N7M1kt4BahZR3MJ1VxT8Hbiqye9JuooyBvizpOoAknaStDUwAegb7lk2Bw4t5LPvA4dI2i58tmHYvwqok1DuTaKmL6HcHmFzAtAv7OtJNH9lMvWA5SFBdiCqyebLAvJrw6cRNeN/BmZLOilcQ5J2x1VJniRdRfk30f3GaZI+J5pctxrwMjCTaLq0h4F3C34wrK89gKhp+wm/NndfA47L77gBLgY6h46hL/i1l/1GoKukaUTN/rnFxDoaqCbpU+Bm4IOEY78AnSRNJbrneFPY3w84O8Q3A+hTgt+JS0P+7LZzziXhNUnnnEvCk6RzziXhSdI555LwJOmcc0l4knTOuSQ8STrnXBKeJJ1zLon/B+6cRLFqoU4iAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mlutils.plot_confusion_matrix(cm, classifier.classes_, normalize=False, title='Confusion Matrix')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results\n",
    "The three classification methods: SVM, Logistic Regression, and Neural Network perform about the same on the test dataset. We have not optimized any parameters. We leave this as an excercise for the reader.\n",
    "\n",
    "For all three methods, the prediction of the mixed class: alpha+beta has the lowest precision and recall. \n",
    "\n",
    "**Limitations of the Model**\n",
    "\n",
    "The feature vectors are created by summing ProtVectors for all 3-grams of a protein sequence and this process averages individual contributions. This step may contribute to the lower performance for the alpha+beta class, because alpha and beta related features are averaged together. In addition, the beta class is underrepresented.\n",
    "\n",
    "The limiting factor appears to the expressiveness of the feature vector using the ProtVec model. Furthermore, we used a cutoff of 25% alpha and/or beta content to define the fold classes. This means that the model will not perform well on protein sequences with minimal alpha or beta content.\n",
    "\n",
    "Alternative feature vectors can be easily explored by replacing the CalculateFeatures step with another method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save the Classification Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['../intermediate_data/classifier']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(classifier, \"../intermediate_data/classifier\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Step\n",
    "After you saved the classification model here, run the next step in the workflow [4-Predict.ipynb](./4-Predict.ipynb) or go back go back to [0-Workflow.ipynb](./0-Workflow.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "**Authors:** [Peter W. Rose](mailto:pwrose.ucsd@gmail.com), Shih-Cheng Huang, UC San Diego, October 1, 2018\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}