Hw2NoeBrown.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "36624c2c",
   "metadata": {},
   "source": [
    "# Homework 2: Regression and Ensemble Methods [25 pts]\n",
    "## Comp562 Summer II 2023\n",
    "\n",
    "### Due 11:59pm July 10, 2023\n",
    "\n",
    "In this assignment, you will use the regression and ensemble algorithms discussed in class and experiment with some toy data. To avoid unexpected behavior with cached variables, test your code with a fresh kernel."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdd16c9b",
   "metadata": {},
   "source": [
    "### Problem 1: Regression [9 pts]\n",
    "\n",
    "Use the dataset provided in `hw2_train_regression.csv` for this problem. Data has shape \\[N,2\\] where the last column is the sample label.\n",
    "\n",
    "**(1a)** Use the linear regression formulation derived in class to compute a best-fitting line to the data. Compute and print the mean squared error on the training data. [3 pts] (Hint: use [np.matmul()](https://numpy.org/doc/stable/reference/generated/numpy.matmul.html) and [np.linalg.inv()](https://numpy.org/doc/stable/reference/generated/numpy.linalg.inv.html) for matrix operations.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "ef16b97c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean squared error: 0.22022621525288605\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.linear_model import LinearRegression, LogisticRegression\n",
    "\n",
    "fn = 'hw2_train_regression.csv'\n",
    "train_data = np.genfromtxt(fn, delimiter=',')\n",
    "X = train_data[:,:-1]\n",
    "y = train_data[:,-1].reshape((-1,1))\n",
    "\n",
    "# TODO: find linear regression\n",
    "X = np.insert(X, 0, 1, axis=1)\n",
    "X_t = X.T\n",
    "X_t_X_inv = np.linalg.inv(X_t @ X)\n",
    "weights = X_t_X_inv @ X_t @ y\n",
    "\n",
    "# compute MSE\n",
    "y_pred = X @ weights\n",
    "mse = np.mean((y - y_pred) ** 2)\n",
    "print(f'mean squared error: {mse}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e642b2f",
   "metadata": {},
   "source": [
    "**(1b)** We will now rescale the data to model a probability distribution. Use logistic regression to compute a best-fitting sigmoid to the rescaled data. Compute and print the mean squared error on the training data. [3 pts]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "ca9e1f3e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "log_w: 0.22373022910346999\n",
      "logistic error: 0.04704\n"
     ]
    }
   ],
   "source": [
    "from scipy.optimize import minimize_scalar\n",
    "import numpy as np\n",
    "\n",
    "y_prob = y - np.amin(y)\n",
    "y_prob = y_prob / np.amax(y_prob)\n",
    "\n",
    "# TODO: write energy function\n",
    "def energy(w):\n",
    "    return np.mean((sigmoidfunc(np.dot(X, w)) - y_prob) ** 2)\n",
    "\n",
    "def sigmoidfunc(x):\n",
    "    return 1 / (1 + np.exp(-x))\n",
    "\n",
    "result = minimize_scalar(energy)\n",
    "log_w = result.x\n",
    "print(f'log_w: {log_w}')\n",
    "\n",
    "# compute MSE\n",
    "y_pred = sigmoidfunc(np.dot(X, log_w))\n",
    "log_error = np.mean((y_prob - y_pred) ** 2)\n",
    "print(f'logistic error: {log_error:>0.4}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "605bce56",
   "metadata": {},
   "source": [
    "**(1c)** Plot the results of both regressions. Explain any differences you observe in the plots and training error you computed. [3 pts]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "5ba2dddc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsEAAAF1CAYAAAAJAjeKAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABNn0lEQVR4nO3de5yWc/7H8den6TQqJopqKiVpSRHJIaTSidLIRuVMa+2P3WWxzsdF7bZYFut8CuVUCVFRCTlUQpLopJpKpXNNNYfv74/rnrnvme453/d93Yf38/GYh+vwva/7M/fkO5/5Xt/v5zLnHCIiIiIiqaSG3wGIiIiIiMSakmARERERSTlKgkVEREQk5SgJFhEREZGUoyRYRERERFKOkmARERERSTlKggUAM1tuZqcHtm8xs2f8jinRmVlLM9tuZml+xyIiiS20j67mdZ4ws9ur8LqU7M/M7H0zu9jvOCQ6lATLXpxz9zvnhvsdB4CZzTCzXYHOd4OZjTOzpn7HVRHOuRXOufrOuXy/YxERAXDOXemc+0d57Uom3VXtz8zsEjPLD/ThW83sWzPrX5XY/eCc6+ece9HvOCQ6lARL3ChjhOFq51x94FCgPvDvKLx3zUhfU0REAPg80IdnAI8DY80sI9Jvkmqj1FJ9SoJlL2Z2l5m9HNhuZWbOzC42sxWB0dhbQ9rWMLObzGyJmf1mZq+b2f4h598ws7VmtsXMZppZ+5BzL5jZ/8xskpntALqXFZdzbjMwATg65Bq/M7OpZrbRzBaZ2bkh5w4ws3cCow+zzexeM/s05Lwzs6vM7Gfg58Cx/mb2jZltNrNZZtYxpP2NZpZtZtsC79UzcLyLmc0JvM+vZvZgic+uZmC/mZlNDMS62Mz+UOIzf93MXgpcf4GZda7gj0xEUoiZ1TGz/5jZ6sDXf8ysTsj5v5vZmsC54YF+6NDAuRfM7N7AdiMzezfQ3200s08CffpooCXwTmAE9+9h+rP9zez5wHtsMrMJ5cXtnCsARgP1gLYh38u/A79ffg1M10ivxPdS7HdIoJ99y8zWm9kyM/tLyLVK66vrmtnLgd9hmwO/Lw4KnJthZsMD2zXM7DYz+8XM1gX66/0C58r8XSnxSUmwVNTJQDugJ3CHmR0eOP4XIAvoBjQDNgGPhbzufbzO7kDga+CVEtcdBtwHNAA+pQxmdgAwCFgc2K8HTAVeDVx/KPC4BRPtx4AdQBPg4sBXSVnA8cARZnYM8BzwR+AA4ElgYqCTbgdcDRznnGsA9AGWB67xMPCwc25foA3weinfwhhgFd7n9Hvg/sJEOuAsYCzeaMlE4NGyPg8RSVm3AifgDQgcBXQBbgMws77A34DT8e6edSvjOtfh9UmNgYOAWwDnnLsQWAEMCEyB+FeY144G9gHa4/W/D5UXtHkjtZcCucAvgcP/BA4LfC+HApnAHZX4XkJ/h8wC3gG+DVynJ3CNmfUJtC2tr74Y2A9ogdf3XwnkhHmvSwJf3YFD8O5MluynS/tdKXFISbBU1N3OuRzn3Ld4HcxRgeN/BG51zq1yzu0G7gJ+Xzha4Jx7zjm3LeTcUYV/OQe87Zz7zDlX4JzbVcp7P2JmW4ANQCPgz4Hj/YHlzrnnnXN5zrmvgbcC758GnAPc6Zzb6Zz7AQg3r2uEc26jcy4H+APwpHPuS+dcfmAe2G68Xzb5QB28ZLmWc265c25J4Bq5wKFm1sg5t90590XJNzGzFnid443OuV3OuW+AZ4ALQ5p96pybFJhzNzrkMxYRCXU+cI9zbp1zbj1wN8G+5FzgeefcAufczsC50uQCTYGDnXO5zrlPnHOuvDc3b11GP+BK59ymwGs/LuMlJ5jZZmAX3nS2C5xz68zM8PrdawP98DbgfmBIJb6Xot8hQAegsXPuHufcHufcUuDpkOuV1lfn4iW/hwb6/rnOua1h3ut84EHn3FLn3HbgZmCIFZ9OV9rvSolDSoKlotaGbO/E+wsY4GBgfOAW0mZgIV7CeJCZpZnZSPOmSmwlOHLaKORaKyvw3n9xzu0HdAQaAs1D3vv4wvcOvP/5eCO/jYGaJa4f7r1Cjx0MXFfiei2AZs65xcA1eIn8OjMba2bNAq+7HG8k48fAbbRwiz6aAYWdfKFf8EYrCpX8jOua5iqLyN6aERxJJbDdLORcef1eoVF4d9ammNlSM7upgu/fAq8/21TB9l845zLw+u+JwCmB443xRpPnhvS5HwSOQ8W+l5J9eLMSffgteKPcUHpfPRqYjDdXebWZ/cvMaoV5r3Cfe82Q60PpvyslDikJlupaCfRzzmWEfNV1zmXj3aYaiHcraz+gVeA1FvL6ckcdiho6Nx+4F3gsMIKwEvi4xHvXd879CVgP5BFMmMHruPe6bInv5b4S19vHOTcm8P6vOudOxutoHd5tPJxzPzvnhuLdEvwn8GZgqkao1cD+ZtYg5FhLILui37+ISMBqvH6oUMvAMYA1lN/vARC4S3edc+4QYADwt5ApWmX1zSvx+rOMygQdGD39P+BCM+uEd3cvB2gf0ufuF1hEV9HvpWQfvqxEH97AOXdG4P3D9tWBkey7nXNHACfh3WW8KMx7hfvc84BfK/M5SPxQEizV9QRwn5kdDGBmjc1sYOBcA7zpBL/h/bV/fwTe70W8Duws4F3gMDO70MxqBb6OM7PDA1MKxgF3mdk+ZvY7wndqoZ4GrjSz481Tz8zONLMGZtbOzHqYt/hkF17HnR/4ni8ws8aB23GbA9cqVkbIObcSb77aiMAijI54oxIl50iLiJRnDHBboL9thDeH9uXAudeBS83scDPbJ3AuLPMWAh8aGFTYitdvFfZdv+LNe92Lc24N3nqPx82sYaDvPbUigTvnfsObCnZHoM98GnjIzA4MxJQZMoe3wt9LwFfAVvMWMacH7kYeaWbHBa4dtq82s+5m1iEwjW4r3vSIcKXgxgDXmllrM6uP9zvtNedcXkW+d4k/SoKluh7Gu701xcy2AV/gLTQDeAnvdlE28EPgXLU45/YAjwC3B6YW9Mab77Ua7zbUP/Hm7oK3kG2/wPHReB3Y7jKuPQdvftqjeAv8FuMtgiBwzZF4Ixdr8RLxWwLn+gILzGw73ucxpJT5zUPxRsNXA+Px5itPrcz3LyKCd0dsDvAdMB9v0fG9AM659/H6yOl4fdjngdeE6/vaAh8C2wPtHnfOzQicG4GXaG82s+vDvPZCvGTxR2Ad3nSxivoPcEZgMODGQJxfBKbNfYi3sKyy3wuBwY8BeIvsluH118/g/R6A0vvqJsCbeAnwQuBjgn9UhHoO73fJzMD1dxFcoyIJyCowB14kKZjZP4Emzjk9/UdEUkKgOsH3QJ1EH7FMpu9F4oNGgiVpmVdDuGNgakMXvOkH4/2OS0QkmszsbDOrbWYN8e6OvZOoSWMyfS8Sf5QESzJrgDcveAfe3LIHgLd9jUhEJPr+iLc4eAne3NY/+RtOtSTT9yJxRtMhRERERCTlaCRYRERERFKOkmARERERSTm+PI2qUaNGrlWrVn68tYhItcydO3eDc65x+S2Th/psEUlkpfXbviTBrVq1Ys6cOX68tYhItZjZL+W3Si7qs0UkkZXWb2s6hIiIiIikHCXBIiIiIpJylASLiIiISMpREiwiIiIiKUdJsIiIiIikHCXBIiIiIpJylASLiIiISMpREiwiIiIiKUdJsIiIiIikHCXBIiJJwsyeM7N1ZvZ9KefNzB4xs8Vm9p2ZHRPrGEUkdU2Yl03XkdNofdN7dB05jQnzsn2NR0mwiEjyeAHoW8b5fkDbwNcVwP9iEJOICBPmZXPzuPlkb87BAdmbc7h53HxfE2ElwSKSelwBbP3J7ygizjk3E9hYRpOBwEvO8wWQYWZNYxOdiKSyUZMXkZObX+xYTm4+oyYv8ikiqOnbO4uIVNOEedmMmryI1ZtzaJaRzg192pHVKbPsFzkHY9K87axs2KdZ9AONH5nAypD9VYFja0o2NLMr8EaLadmyZUyCE5HktXpzTqWOx4JGgkUkIVX51tqYkG4vPeUGQS3MMReuoXPuKedcZ+dc58aNG0c5LBFJds0y0it1PBaUBItIQqrSrbXxIaO+Q/PBwuWESW0V0CJkvzmw2qdYRCSF3NCnHem10oodS6+Vxg192vkUkZJgEUlQlb61NqUr5ATu+p+3Gywlu7+JwEWBKhEnAFucc3tNhRARibSsTpmMGNSBzIx0DMjMSGfEoA7lT2GLIs0JFpGE1CwjnewwCW/YW2ufDoENs7ztwdsgrXaUo/OHmY0BTgMamdkq4E6gFoBz7glgEnAGsBjYCVzqT6QikoqyOmX6mvSWpCRYRBLSDX3acfO4+cWmRIS9tfb19bDiNW970DqoVT+GUcaWc25oOecdcFWMwhERqZIqLXquAiXBIpKQCjvEMjvKhQ/Ajw9422ctg7pa4CUiEs8KFz0XDnAULnoGIp4IKwkWkYRV5q21ZaNh3vUATGs9hdsfXcrqzQuiOqogIiLVU9aiZyXBIiLlWf0+fH4RAJ8c/BZXTconJ3cPEN1RBRERqZ5Y1hNOyeXRIpLENnwJM87wtk8Zz02fZMTdU4pERCS8WNYTVhIsIsljy48w5QRvu8tT0CIrLp9SJCIi4cWynrCSYBFJDjuz4b3Dve2O/4BD/wDE51OKREQkvFjWE9acYBFJfHs2wYTm3nbbP8GRtxWdqnApNRERiQuxqiesJFhEElteDry5v7edOQCOe7zY6QqVUhMRkbgVrbrBSoJFJHEV5MHr+3jbGR2h28SwzUJHFQo702tf+0YJsYhInItm3WDNCRaRxOQczDrf267ZAM74ttyXFHam2ZtzcAQ70wnzsqMbq4iIVElZdYOrS0mwiCSm726HFa/DoVfC4C0Vekk0O1MREYm8aFb4URIsIoln4QOw4D5oM9ybA2xWoZepXJqISGKJZoUfJcEikliWPOc9DrnlYDjuiQonwKByaSIiiSaadYOVBItI4ljxFnz1B2jaB058GWqklf+aELEswi4iItUXzbrBqg4hIolhzRSYNRQOOAFOeQvSalf6EiqXJiISG5EsaxatusERS4LNLA2YA2Q75/pH6roiIqyfBTPPhn2PgNPeg5r1qnypWBVhFxFJVdEsaxZJkZwO8VdgYQSvJyICm76DGWeyvcaB9P/+Nlrf8RldR05TWTMRkTiVKJV4IpIEm1lz4EzgmUhcT0QEgK0/w/Te5Li6DPzhLr7fmK76viIicS5RKvFEaiT4P8DfgYIIXU9EUt3OVTC9F7h8Ll9xP0tyGhU7HY+jCiIikjiVeKqdBJtZf2Cdc25uOe2uMLM5ZjZn/fr11X1bEUlmuzbAtN6weyN0/4DPNxwYtlm8jSqIiEjiVOKJxEhwV+AsM1sOjAV6mNnLJRs5555yznV2znVu3LhxBN5WRJJS7laY0Rd2LINu78D+xybMqIKIiESprFnOJiiI7ISDaleHcM7dDNwMYGanAdc75y6o7nVFJAXl5cDHZ8Gmb+HU8XBQN8AbVQhdaQzxOaogIiKeiFTicQ5mPQJT7/D2z30JjhhY/eACVCdYROJDQS58dh6smwknvQyZwUqLqu8rIhJ7kaz1Wyl5u+G962De6OLH2/aO6NtENAl2zs0AZkTymiKSPErtUF0BfHEZZL8Dxz0OrYbt9VrV9xURiR1fav3u2QFvXg4/vR88dkBbuPR9qB/5qbQaCRaRmCi1Q3WOrPyRsPxlOOo+aPsnnyMVEZGyav1GPAnO2QSvDIZVs4PH2vT0pj/UqR/Z9wqhJFhEYqK0DnXzlzfDfi/D4dfDETf7FJ2IiISKSa3fbb/Cc31g07LgsY7nwYBHoFbdyL1PKZQEi0hMhOs4L280nkv2exnaDIej/wVmPkQmIiIlNctIJztMvx2Rqjwbl8ETp8CebcFjJ1wFve6GtFrVv34FRfKxySIipSrZcQ5uOIXbmz3LRzu6wXFPKAEWEYkjUan1m7sL7m4IjxwdTIB73AZ3bIS+98c0AQYlwSISI6Edar/9PmVk80f5dPsx7DjmOaiRVs6rRUQkliJa63f3dpj1X3j4KG8hNMCZD8Kdm+HUG3z7HaDpECISE4Ud58cfv8I/G/2bBXsOZ9OxYznr2EN8jkxEJLlVtdRZtavy7N4OX/wPvnjMW/zW+lQ452lodUpc3P1TEiwilVblDrX5L2Q1vRsaHEHH02fQsXbDGEQrIpK6fCl15hwsGA+Tb4Vtq+GwvnDK9dDiuOi8XxUpCRaRSqlyh7rpO5hxJqQ3g+6TQQmwiEjUxbTUGcD6RTDpBlj2MTTpAINfgJbHR/59IkBJsIhUSpU61G2LYXpvqFkPen4I6U1iEKmIiMSk1Bl4Ux9m/gs+fwxq14Mz/g2dL4vrNR9KgkWkUirdoe7Mhmmng8uHntOh3sFRjE5EREJFtdQZ7D31odMF0POuqDzhLdJUHUJEKqW0jjPs8V0bYFov2L0Run8A+x0e5ehERCRUVEqdFVq/CF46C968FOo1gsunwsDHEiIBBiXBIlJJFe5Qc7fCjH6wYxl0ewf2PzaGUYqICES41Fmh3dtgyu3wv5Ngzbfe1IcrZkCLLpEKOyY0HUJEKqWw4yyzOkTOGhjfzNvu9g4c1M2HSEVEBCJQ6qyQc7BgXGDqwxpv6sPpd3ujwAlISbBIAqtqqbLqKrNDzd0WTICtJmT2j3o8IiKJwK8+OyLW/Qjv3wDLZkLTo+Dc0XFX8qyylASLJChfaj+WpyAX3tg3uD801584RETiTFz22RWxext8/C/44nGoXR/OfACOvTSuqz5UlOYEiySoskqV+cI5GFs7uD/M+ROHiEgcirs+uyJ+mQWPnwizHoGjhsKf58Jxw5MiAQaNBIskrJjVfqyoMSF/Uw8t8CcGEZE4FXd9dlnyc2HGSPj0QWjYyqv6kGCL3ipCSbBIgop67cfKeDXkGfBD8uLimfAiIvEkrvrssvy2BN4aDqu/9ha+9f0n1Knvd1RRoekQIgkqqrUfKyM0AT53Z9LcJhMRiaS46bNLU1AAXz4FT5wCG5fC4Be9mr9JmgCDRoJFElaFSpVF22shIxjnbICacTaiISISJ+Kizy7NL5/D83297VanwNlPwH7N/Y0pBpQEiySwiNV+rIr3j4X8Xd72WUuhzgH+xCEikiB87bNL8/5N8OX/gvsXvZ0yd/SUBItI5X16Hmz62tvuOxfqty52OqFrYYqIpIK8PXBv8PHG76X14Oodw2n2r49Tps9WEiwilfPNTbDidW+7+2TY/5hipxO2FqaISKrYsBgeDT7K/k/5f+f9XUcDqdVna2GciFTcT4/DD//0tk94AZr23qtJQtbCFBFJBc7BnOeLJcDn1H6S93OPLtYsVfpsJcEiUjGr3oY5V3nbHe+FQy4O2yyhamGKiKSKnM0w+mx495rgsVtW8/XWBmGbp0KfrSRYRMq3/nOYmeVttxkOR95aatPSal7GXS1MEZFUsXU1PPA7WDrd20+rDXduhtr1UrrPVhIsImXbugimnuRtH9Qdjn+6zOZxXwszyZlZXzNbZGaLzeymMOf3M7N3zOxbM1tgZpf6EaeIxMi6hfDg4ZAXGNk9rC/cvr7ooUap3GdrYZyIlC5nLbz7O287vSn0nFbuS+K6FmaSM7M04DGgF7AKmG1mE51zP4Q0uwr4wTk3wMwaA4vM7BXn3B4fQhaRaAqt/wvQ8w445bpiTVK5z1YSLJKgol6GLHcbjG8a3D97dYVfGpe1MFNDF2Cxc24pgJmNBQYCoUmwAxqYmQH1gY1AXqwDFUlFMS0f+cNEeP3C4P6w1+GwPmGbpmqfrSRYJAFFvQxZQS68sW9wf5ir/jUlFjKBlSH7q4DjS7R5FJgIrAYaAOc55wpiE55I6opp+civnoZJ1wf3r5oNjQ+L7HskAc0JFklAUS1D5hyMrR3cVwKcSCzMsZI/wD7AN0Az4GjgUTPbt0QbzOwKM5tjZnPWr18f6ThFUk5Mykc6Bx/cXDwBvvEXJcClUBIskoCiWoZsTEi3MFQDhAlmFdAiZL853ohvqEuBcc6zGFgG/K7khZxzTznnOjvnOjdu3LjkaRGppKiXj8zPhZfOgi8eDx67YyOkZ0Tm+klISbBIAopaSZtXQwYSh+QVrR6WhDEbaGtmrc2sNjAEb+pDqBVATwAzOwhoByyNaZQiKSiqpch2b4d/toJlM739FsfDXVugRlqZL0t1SoJFElBUStqEJsDn7lTnmYCcc3nA1cBkYCHwunNugZldaWZXBpr9AzjJzOYDHwE3Ouc2+BOxSOqIWimy7etgRCbs2e7td70GLp9SvWumCC2ME0lAES9p81q94PY5G6Bm8hdJT1bOuUnApBLHngjZXg3s/bxrEYmqqJQi+20J/PeY4P7vn4Mjz6lmpKlDSbBIggpX0qZK5Xc+6Az5O73ts5ZCnQOiFLGISGqLaCmyVXPhmR7B/T9+Ak07RubaKUJJsEiSqFL5nU+HwMa53nbfuVC/dSxCFRERqlE3+Mf3YOyw4P4NS6Beo+gFmqQ0J1gkSVS6/M43N8OK17zt096H/Y8J305ERCKucOAie3MOjuDAxYR52WW/8NOHiifAt29QAlxFSoJFkkSlyu/8/D/4YaS3fcLz0Kzv3m1ERCRqKj1w4Ry8dgF8eJe336AZ3LkZ0mpFNc5kpukQIkmiWUY62WES3r3K76x6G2b/n7fd4R445JLoByciIsVUauCiIB/+1Rp2bfH2O10AAx+LYnSpQSPBIkmiQuV31n8OM7O87UMugw63xy5AEREpUuG6wbk5cM/+wQT4rP8qAY4QJcEiSSKrUyYjBnUgMyMdAzIz0hkxqENwkcXWn2DqSd72gd3ghGd9i1VEJNVVaOBi50a4r0lw/7IpcMxFMYow+Wk6hEgSKbX8Ts6v8G6gY63bBE6fEdO4RESkuHLrBm9cBo8cHXzBtT/AfhEqryaAkmCRpFFqqZ3c7TA+ZCRh0Br/ghQREaCc8mjLPoEX+wcb37oWaukhRpGmJFgkCZRWI9hcLgN/DKn9O8z5FKGIiBQqs6577iSYdH2w8Z2bwSzMVaS6lASLJIHwpXbylACLiMSh0sqj7fPOlVAw0zvQ4gS4fLIP0aUOJcEiSSBcSZ3lHQcEd4YWxDAaEREpS7g++5s6fyCjYIe30+0m6H5zjKNKPaoOIZIESpbUWd4xZC7ZkDzdShMRiSOhfXb3GvNYXncYGRZIgIe+pgQ4RpQEiySB0FI7oQnwO+1+hhpppb1MRER8UNhnj6r5BM/XHlV0fGrPSdBOT/CMlWpPhzCzFsBLQBOgAHjKOfdwda8rIhVXuKI4a2HzomOT2n7HgGMP9SskEREpRVanTI7/8iqarp1ZdOzdM2bTv8thPkaVeiIxJzgPuM4597WZNQDmmtlU59wPEbi2iJSjsMzOZy17Bg/2msUZjTv4F5SIiJRu7Pk0XTs9uH/HJvrX0M35WKv2J+6cW+Oc+zqwvQ1YCKias0gMFJbZCU2AH1h3KRNWtfQxKhERKc38N0fAj+8W7U8Y+AMoAfZFRKtDmFkroBPwZSSvKyLhjZq8iFdaXlO0P3Hzqfx37TlkTl4U/slxIiLimw9mzaXv9yOL9lvtepX0wvrA6rNjLmJ/ephZfeAt4Brn3NYw568wszlmNmf9+vWReluRlNZ49zccU28RAG9t6s5fVvwdCF9+R0REfJSzicOmXgzAg7m/p9WuV73DufmMmrzIz8hSVkRGgs2sFl4C/Ipzbly4Ns65p4CnADp37qyq/SLVtTOb19p4ZXQeXHs+j6wbWnSqZMk0ERHxUW4OjBlGZsEahubeyucF7Yud1sCFPyJRHcKAZ4GFzrkHqx+SiJRr928wvTc1atZi8JJ/MXtbsApEeq00bujTzsfgRESkSEE+vDUcVnzOvbWv4/Pd7fdqooELf0RiOkRX4EKgh5l9E/g6IwLXFZFwcrfB9H6wbQm1ur/L+Wf8nsyMdAzIzEhnxKAOmlsmIhIPnIP3rvMWwvUdybFnXFZU072QBi78U+2RYOfcp4AeRyUSC/m7YOZA2PQ1nDIeDjqNrIO8BRWFpdKufe0bRk1exA192ikZFhHx08f/grnPw8nXwglXkhU4PGryIlZvzqFZRrr6ah9FtDqEiERRQR58NgR+nQ4njobmA4pOFZZKy8nNByB7cw43a8WxiIh/5jwPM+6Ho4ZBzzuLDmd1ylS/HCeUBIskAlcAX1wGq96Gzo9C6wuKnR41eVFRAlyocMWxOlsRkRj78T14729waC846xGw8DfMC+/gaVTYH0qCReKdczD3Glg+Gjr+Aw67aq8mpa0s1opjEZEYW/EFvHkZNOsE574IabXCNtMdPP/pESUi8W7+XfDTf+F3f4P2t4ZtUtrKYq04FhGJoXUL4dVzYb/mMOwNqF2v1KZl3cGT2FASLBLPfvwPfH8PHHIpdPp3qbfUbujTTiuORUT8tGUVvHwO1KwLF4yDegeU2Vx38PynJFgkXi15Hr6+FloMgi5PlZoAg3frbMSgDiqVJiLih50bvQR49za44C1oeHC5L9EdPP9pTrBIPFo5Hr4aDk16wUmvQo3w/6tqUYWIiM9yc2DMUNi41EuAm3So0Mtu6NOu2Jxg0B28WFMSLBJv1n7olULbvwucMg7S6oRtpkUVIiI+y8+DNy+HlV/C4Oeh9akVfmlhP62BDP8oCRaJJxu+gJlZsG876D4JatUvtanKoomI+Mg5mHQdLHoP+v0L2p9d7kvC3b377KYeMQhWwlESLBIvNs+HGWdA3SbQfQrUblhmcy2qEBHx0YyRMPcFOPlvcPwfy22uu3fxRwvjROLBtiUwrTekpUOPqZDepNyXaFGFiIhPZj8LH4+Eo8+HnndU6CUqiRZ/lASL+G3napjWCwr2eCPA9VtX6GUqiyYi4oOF78Ck66FtbxjwcJmVe0Lp7l38URIs4qfdv8H0XrB7PXT/ADLaV/ilKosmIhJjv8zyFsI1OwYGv1Dq0+DC0d27+KM5wSJ+yd0G0/t5UyG6vw8HHFfpS2R1ylTSKyISC7/+AGOGQEZLGPZ6mU+DC0cl0eKPkmCRKCmzhm/+Lq8KxKavvTJoB3X3NVYRESnD5pWBp8Glw4XlPw0uHJVEiz9KgkWioMxVwEcdBJ8NhV+nwYmjoflZfoYqIiJl+XUBvHAmFOTDpe97I8FVpLt38UVJsEgUlLYK+MHJP5C1sLl34Nj/QusLfIhORERKCnv3rtYX8OZlXoOL34UmR/obpESUkmCRKAi/2tcxs2Vvb7PewdDu6pjGJCIi4YW7e/fsuElkpV3vNThuOLQ+xccIJRpUHUIkCsKt9l3ecUBw56xlMYxGRETKUvLuXQN28k5hArz/IXDmAz5FJtGkJFgkCkrW8F3esX/w5JDcCteVFBGR6Ct+984xv+7w4O5f5sU8HokNJcEiURBaw7dYAnzuDqihWUgiIvEk9O7d8rrnF213rTPOj3AkRpQEi0RJVqdMPmszKHhg0DomzN9E15HTaH3Te3QdOY0J87L9C1BERIDg3bsFdS4tOtYx/xVu6Ps7H6OSaNOQlEi0LB8LuVu87QGLmbBwT+ll01QyR0TEN1mdMmmw9WfqTd8NwMA6z3JP307qm5OckmCRaNjwJXxxkbfddw40aMOoydPClk0bNXmROloRET/t3EjPb/7qbV89l7cbHepvPBITSoJFIm3z9zCjH+zTEnp9AulNgdLKppV+XEREYiA/F16/CLauhss/hJAEuMwnf0rC05xgkUjavhSm94a0dOgxtSgBhvBl08o6LiIiMfDBTbD8ExjwCLQ4ruhwYe3g7M05OIJT2LSWI3koCRaJlJw1MK0X5O+G7lOgfutip0uWTQNIr5XGDX3axTJKEREpNPtZmP0MnPRnOHposVOlPflz1ORFsYxQokjTIUQiYfdGmNYbdq2DHh9BRvu9mhTeQtOtNRGROLDsE3j/79C2N5x+916nNYUt+SkJFqmu3O0w4wzY9jOcNgkaddmrScl5ZQ+dd7SSX4kKM+sLPAykAc8450aGaXMa8B+gFrDBOdcthiGK+G/jMm8e8P5t4JxnoEbaXk2aZaSTHSbh1RS25KHpECLVkb8LZmbBxjlw8mvQpMdeTTSvTGLFzNKAx4B+wBHAUDM7okSbDOBx4CznXHtgcKzjFPHV7m0wdhi4Ahg6BuruF7aZprAlPyXBIlVVkAefDYVfP4Ljn4PmA8M207wyiaEuwGLn3FLn3B5gLFDyH+YwYJxzbgWAc25djGMU8U9BAYy7AtYvgnNfhAPalNo09MmfBmRmpDNiUAfdxUsimg4hUhWuAL4cDqsmwLGPwCEXldpU88okhjKBlSH7q4DjS7Q5DKhlZjOABsDDzrmXSl7IzK4ArgBo2bJlVIIVibnp98KiSdBvFBxyWqnNNIUtNWgkWKSynIOv/wbLXoQOd0O7P5fZXKXRJIYszDFXYr8mcCxwJtAHuN3MDtvrRc495Zzr7Jzr3Lhx48hHKhJr89+ETx6AYy6GLn8otZmmsKUOJcEilfX9PbDoYWh3DRx5e7nNNa9MYmgV0CJkvzmwOkybD5xzO5xzG4CZwFExik/EH9lz4e2r4OCucMa/wcL9vejRFLbUoSRYpDJ+fBjm3wWHXALHPFBmR1pI88okhmYDbc2stZnVBoYAE0u0eRs4xcxqmtk+eNMlFsY4TpHY2boGxp4P9Q+Ec1+CmrXLbK4pbKlDc4JFKmrpi/D1NdD8bOjyNFjF/obUYzclVpxzeWZ2NTAZr0Tac865BWZ2ZeD8E865hWb2AfAdUIBXRu17/6IWiaLcXfDa+bBrK1w+Beo1KvclKo2WOpQEi1TEygnw5eVwUE/o+irUqNj/OoVzywpvrRXOLQOUCEtUOOcmAZNKHHuixP4oYFQs4xKJOefgnb94UyHOewWaHFmhl93Qp12xfhs0hS1ZaTqESHnWfgSfnQf7d4ZTJ0Ba3Qq/VHPLRER88tnD8N1r0OM2OLx/hV+mKWypQyPBImXZ8CXMHAgNDvOeBlerfqVerrllIiI++GkKfHgXtB8Ep1xf6ZdndcpU0psClASLlGbz9zCjH9Q9CHpMgTr7V/oSmlsmIhJjvy2Bt4ZDkw4w8LEKLWAOpXUcqUPTIUTC2b4Upvf2pj70+BDSm1bpMiqPJiISQ4WPRK6RBkNegdr7VOrlqhGcWpQEi5SUswam9YL83dB9KtRvXeVLaW6ZiEiMOAcT/g82/ASDX4CMyj/pUOs4UoumQ4iE2r0RpvWGXeugx0eQ0b7al9TcMhGRGPj0IVg4EXrfB4d0q9IltI4jtSgJFimUux1mnAHbfoLT3odGXQDNDxOpirvfWcAPq7f6HQYAXy7bGNHrHd86/PqAWL1PLN4v0teuahwVddSuOdy06R5m1T2N/353DMz/vErvXfIZ46HHz3uyatcsyc/PtiKq8vlXRnW+/5MPbcTLw4+PWCxKgkUA8nfBzCzYOAdOfhOa9ABU51dEJN4dlLeav2weyYqarXky45pKL4ST1KUkWKQgDz4bCr9+BCe8CC2yik6VNT9MSbBI6e4cUP2pRCLl2rMDnrkO6tai/hXjGd2wVbUvqbt/qUNJsKQ2VwBfDodVE+DYh+GQi4qd1vwwEZE45Ry8fTWsXwgXvAWBBLi6SazWcaQOVYeQ1OUcfP03WPYidLgL2v1lryal1fNVnV8REZ/N+i8sGAc974Q2xaewqcSZVISSYEld3/8DFj0M7f4KR94Rtonq/IqIxKEl0+HDO+GILOj616LDKnEmlaHpEJKaFj0C8++EQy6BYx4sdSFF4S0xzQ8TEYkTm5bDm5dC49/t9UQ4TWGTyohIEmxmfYGHgTTgGefcyEhcV5JL3Cw2WPoSzP0rND8bujwNVvYNEc0PExGJE3t2wmsXeOs5znsZ6tQvdlqPqpfKqPZ0CDNLAx4D+gFHAEPN7IjqXleSS9zM01r1Nnx5GRzUE7q+CjV0M0REJCE4B+/8BdZ+D+c8Cwe02auJprBJZURiTnAXYLFzbqlzbg8wFhgYgetKEomLeVprp8Gn58L+neHUCZBWN3bvLSIi1fPF4zD/DehxG7TtFbaJHlUvlRGJYbBMYGXI/ipgr8d5mNkVwBUALVtW/nnekth8n6e14SuYeRY0OAxOmwS16pf/GhERiQ9LP4Ypt8PhA+CU68psqilsUlGRSILDrSja68mDzrmngKcAOnfuXNqTCSVJ+TpPa/P3MKMf1D0Iuk+GOtF9JKSISDKIm3Ucm1d4C+EatYWs/+mJcBIxkZgOsQpoEbLfHFgdgetKEvFtntb2pTC9N6TVgR5TYZ9m0X0/EZEkEDfrOHJzvIVw+Xkw5FWo0yC27y9JLRJJ8GygrZm1NrPawBBgYgSuK0nEl3laOWtgWi/I3wXdp0D9Q6L3XiIiSSQu1nE4B+9eC2u+g3OeDrsQTqQ6qj0dwjmXZ2ZXA5PxSqQ955xbUO3IJOnEcp7WjFnvcdry/gD8Ye2jnLmsIVmdYvLWIiIJz/d1HABfPQXfjoHut8JhfWL3vpIyIlIfyjk3CZgUiWuJVNeULz+jdyABfnp9FlPXteLTcfMBtFhCRKQCfK+3u/xT+OBmaHcmnHJ9bN5TUo4emyzJZdd6ei85GYDNefW5b81wQI/NFBGpDF/r7W5ZBa9fDPsfAmc/ATWUqkh06EkBkjzydsC4A4t2j/5hbLHTemymiEjF+PbI+Nxd8NqFkLfbWwhXd9/ovp+kNCXBkhzy98Drwdq/rb57d68memymiEjFxbzernPw3nWw+msvAW58WOzeW1KS7jFI4nMOXqtTtDvh8FV6bKaISAKZMC+bF++9HL55mefSzmVCztF+hyQpQCPBkvjGhPwtN7SArEAh9bgo8i4iImWaMC+bZePv4doabwHwjx1nUVeLmSUGlARLYns15MlB5+0qepKQHpspIpIY3np/CqNreGs4/rLnahw1ihYzqx+XaFISLHGr3Ed2hibA52zwngonIiKJI3cXo/dcA8Dn+UcwseCkolNazCzRpiRY4lLhIzsLn1hU+MhOCNweC02A+/8EdQ7wI0wREamO+w4q2hyae1uxU1rMLNGmhXESl8p8ZGdoAnz6TNi3bYyjExGRaruvadFm+/wxxU5pMbPEgpJgiUul3Qb7rGXP4M5Jr8CBp8QoIhERiZjxV0LuTm/7mu+5b9BRZGakY0BmRjojBnXQfGCJOk2HkLgU7pGdUw77v+BOh7uh1bAYRyUiIqUpdx1Hoeyv4dvAyG+rUyCjBVmdVAlCYk8jwRKXSj6y86EW/+awuiu8nZaDocMdPkUmIiIlFa7jyN6cgyO4jmPCvOziDfN2w9Pdg/uX7P1gI5FYURIscSmrUyYjBnUgMyOdqw98jbMbzvBONGgLJ7/ua2wiIlJcmes4Qt0bfLQ9d22JQWQipdN0CImpCt8uI1Drt+EnMGt08OCAn2IUqYiIVFRp6ziKHb9rv+D2LWuiHJFI+TQSLDFT4dtlhdbPgllDg/vDXEziFBGRyimtnFnR8XFXBA9e9DbU3icGUYmUTUmwxEyFb5cBbFsCU7sG95UAi4jErZLrOCCkzFn2XPjuNe/gIad5XyJxQNMhJGYqdLsMYM8meOfQ4L4SYBGRuFY4rW2v6W4dGsO9RwQbXvS2TxGK7E1JsMRMuLJnhceL5O+BN/cP7isBFhFJCFmdMvde4xE6D1gL4STOaDqExEyZt8sAnIPX6gRPKgEWEfHVhHnZdB05jdY3vUfXkdNKX8MRjhbCSZzTSLDETKm3ywpHDsaE/E02tMCHCEVEpFDhYubCtRyFi5mhAg+2eGt4cPuiiVoIJ3FJSbDEVNjbZQCvWnD7vF1gtnebMlSm9JqIiJSvrMXMZfavq+bA/De87UO6wyHdohilSNUpCRb/hSbA52yAtDqltw2jWqMVIiISVoUXM4fK2w3P9AzuXzQhskGJRJDmBIu/QhPg/j9BnQMqfYlKlV4TEZEKKbf2bzihT4S7c3NkAxKJMCXB4p/QBPj0j2HftlW6TJVGK0REpEzlLmYuqdhCuNWVntYmEmtKgsUfoQnwiaPhwFOrfKkqjVaIiEiZsjplMmJQBzIz0jEgMyOdEYM6hJ9mNvb84PbF70DtejGLU6SqNCdYYi80Ac4cAK0vqNblbujTrticYChntEJERCqk1MXMoX6ZBT++620f1g9alz+oocXMEg+UBEtsfX5xcLtWBnSbWO1Lllt6TUREoiN3FzzfL7g/bGy5L9FiZokXSoIl6gr/4j8sfybPt34peGLwpoi9R4VGK0REJLLuOyi4XcEnwlW59JpIhGlOsERV4V/8++/+judb3w3Ah9tOZMLhq3yOTCQ5mVlfM1tkZovN7KYy2h1nZvlm9vtYxidJJHQh3G3rK/wyLWaWeKEkWKJq1ORFWP523ml7LQDf7DyM4ctuVfkykSgwszTgMaAfcAQw1MyOKKXdP4HJsY1QksYL/YPbV8yAmrUr/FItZpZ4oSRYomrDlq08cfD9ADy49nyyFj8I6C9+kSjpAix2zi11zu0BxgIDw7T7M/AWsC6WwUmSWPwhLP/E2+50ITTrVKmXV7r0mkiUaE6wRE9BHk+2eZBT95nHdSuv5a1NwacI6S9+kajIBFaG7K8Cjg9tYGaZwNlAD+C40i5kZlcAVwC0bNky4oFKgtqzE14+J7g/8NFKX0KLmSVeKAmW6HAF8NUVnLbPJ9y/9o/FEmD9xS8SNeGeTuBK7P8HuNE5l29lPMzAOfcU8BRA586dS15DUtX9TYPbFVwIF44WM0s8UBIskeccfH09LH0ejryTIw7/A5n6i18kFlYBLUL2mwOrS7TpDIwNJMCNgDPMLM85NyEmEUrcK7WGb+hCuDsiV91HxC9KgiXyvr8XFj0Eh/0FOtxJlpmSXpHYmA20NbPWQDYwBBgW2sA517pw28xeAN5VAiyFSqvhm/V2yPrKq+dADS0pksSnJFgiq/BpcK0vgmMfKvXZ8XpakEjkOefyzOxqvKoPacBzzrkFZnZl4PwTvgYocS9cDd9LCsZ7/5oADuoAjdrGPjCRKFASLJET+jjk458FCz9SoKcFiUSPc24SMKnEsbDJr3PukljEJImjZOWexmzixlohT4H706cxjkgkenQ/QyJjXMhTg85eCzVK//uqrKcFiYiIf0Ir99Qkj9l1rwqerMZCOJF4pCRYqu/DbrArUG60/0+QflCZzfW0IBGR+BSs4etYXPeiouMTzlrgX1AiUaLpEFI9X1wG62Z6272/gH3LnyvWLCOd7DAJr2oHi4j4q3BKWvN3zoMC79jE/t+QdUxzH6MSiQ6NBEvVzb/HK4MG0O0daHR82e0D9LQgEZH4lVUwlc4F3joN/votZ3VuXfYLRBKURoKlapY8D/Pv9La7PAmZ/ctuH0JPCxIRiVM/TYZ3/waHng5Dx0JaLUAVfSQ5KQkWoJId3Or34cvLvO32t8KhV1T6/fS0IBGROJM9F964BJocCYNfLJYAq6KPJCNNh5CiDi57cw6OYAc3YV723o03zoUZZ3jbBw+Do+6NaawiIhIFG5fCK+dCvUYw7A2oU7/olCr6SLJSEiwV7+C2L4UPOnvbB3SBrq/EKEIREYmaHRvg5XPA5cMF46BB8Qo/qugjyUpJsFSsg9u1ASa28bZr7Qt9voxBZCIiElV7dsKr58HW1TD0tbBPgyutco8q+kiiUxIs5XdweTthXOPgicEqmC4ikvDy8+D+ppA9B06/C1qGr/Cjij6SrJQES9kdXEE+vF4veGKYi3F0IiIScc7BPw4I7h99fqlNszplMmJQBzIz0jEgMyOdEYM6aFGcJDxVh5DSS5Yd3QzGhPydpARYRCQuVLtk2eMnBLeHfwR19y2zuSr6SDKqVhJsZqOAAcAeYAlwqXNucwTikhgL28G9asHtoQXlXkN1JEVEoq/aJcvGng/rf/S2LxwPzTtHK1SRuFbd6RBTgSOdcx2Bn4Cbqx+SxIXQBHjIHjArvS2VLLMmIiJVVq2SZZNvhR/f9bbPeRba9IhChCKJoVpJsHNuinMuL7D7BaCHiyeD0AR48DaoUavcl6iOpIhIbFS5ZNnnj8Hnj3rbve+FDr+PcGQiiSWSC+MuA94v7aSZXWFmc8xszvr16yP4thJR45oGt89eC7Xql942hOpIiojERpVKln37Gky+xdvucgWc9OcoRCaSWMpNgs3sQzP7PszXwJA2twJ5QKlPT3DOPeWc6+yc69y4cePSmomfPjwNdq31tvv/BOkHldk8lOpIiojERqVLli36AMYHHm/fpgecMSrKEYokhnIXxjnnTi/rvJldDPQHejrnVD4gUf06HdZ97G33/hz23btgellu6NOu2EINUB1JEZFoKLWiT7hFcUs/hjHnedsNmnlPhBMRoPrVIfoCNwLdnHM7IxOSxNyWH+DjwMB+t/eg0Qlltw+jUp2yiIhUS4VKlv3yObx0VnD/2gXlLnIWSSXVrRP8KFAHmGre/1hfOOeurHZUUkxUS49tXw7TekHNetBvHjRoU+VLqY6kiEicWPElPN83uH/7Bqix9wxIlbaUVFatJNg5d2ikApHwql0Psiw5a2Ha6ZCfA6fPrFYCLCIicWLlV/Bc7+D+LWsgbe8qP1H9/SKSAPTY5DgXtdJjezbB9N7eQrjTJkHGkdW7noiI+G/lV/Bsr+D+DUuh9j5hm6q0paQ6PTY5zkWl9FjeDphxJmxdBKdVbQ6wiIjEmZIJ8J+/hnoHlNpcpS0l1WkkOM5FvPRY/m6YeTb89iV0HQNNyiz+ISIiiWDl7OIJ8GVT4ICyp7iptKWkOiXBca7S9SDLUpAHs86HtVPh+GehxaAIRSkiIr5ZNQeeDRnQGPwCtDy+3JdF9PeLSALSdIg4F7HSY87BV3+ElW/BMQ/BIZdEPlgREYmtVXPgmZ7B/dPvgvZnV+ilKm0pqU5JcAKodukx52De9bD0OTjyDvjdNRGLTURE9haT0mOr5sLz/YL7R18AJ19bqUuotKWkMiXBSW7CvGyyFjYH4I1tZ1Mr7w9k+RuSiEhSi0npsVVz4aWBkL/H2296NGQ9Fplri6QIJcFJLDQBBvj7skupu+p7MNNf/iIiUVJW6bGI9L3Zc2H02bBnW/DYHz+u/nVFUowWxiWx0AS4/fev46ihGpAiIlEW1dJj2XPhpbNhn4aQVts7dufm6l9XJAVpJDhZjW9WtHncD6PZURAslq4akCIi0dMsI53sMP1stUuPZX/tJcDpGXDxu7BfczCr3jVFUphGgpPRh90hZw0A3X98kvV5DYudVg1IEZHoiUrpseyvYXSWlwBf8h5ktFACLFJNGglONl8Oh3UzAPj44LdZu7A2EJybphqQIiLRFfHSY6vneQlw3f3gkne9BFhEqk1JcDL5/j5Y8qy3feoEujU/ixH7xKBMj4iIFBOx0mOr53lVIOruFxgBbln9a4oIoCQ44ZRae3Lpi/DdbV6j4x6H5gMB1YAUEUkkoX38afuu5kl3D7XrZXhzgJUAi0SUkuAEUlrtyQO3z+CklZd4jY64Cdr+yb8gRUSkSkL7+Pa2jId238969mHBac/Su+HBfocnknS0MC6BhKs9eUjNRZy08gJvp+V5cPQIHyITEZHqKuzj29syXql9P9tJ57w9t3H3J9v9Dk0kKWkkOIGULG3WvNZa3mt7jbfT8Bg4eWzsgxIRkYhYvTmH9ra8KAEesud2VrnGmMpaikSFRoITSGhps4y0rXx6+HAAdhXUgX5z/QpLREQioNu+axlT+x9k2A6a2wZWucaAylqKRIuS4ARSWHuyju3mm/bDio5/0H6Jj1GJiEi1rfmO53L/zr7mjfrmOq/OsMpaikSPpkMkkKxOmeDyyfoxuEBiwuGrVP1BRCSRrZ0PT55SNCq1gYYct/sxMlXWUiSqlATHqbCl0I5uViwBZpgjy7cIRUSk2tbOhydODu4fejqNLniLZf5FJJIylATHodJKoWUtbB5sNLTAp+hERKSiSq3tDnsnwCdeDX3u8ydQkRSkJDgOhSuFtvDwfsGdIXv0zHgRkThX2oAGQFbTTcUT4DMfhOMu9yNMkZSlJDgOlSyFtrxj/+DOoHVQo1aMIxIRkcoKN6CRk5vPW+9PIWvPNcGD578JbXvFNjgRUXWIeBRaDueLwy8q2h6w6g2o29iPkEREpJJKDmgAtLMVjA5NgK/8VAmwiE+UBMehwlJor7S+hSa1NgLQb8nTXN7rRJ8jExGRiipZ37edrWBynZuCB679AZp0iHFUIlJISXAcyuqUyRN999C1wXcA/HHtI/yxfz+VyRERSSCFAxoA7W158QT45lWwn/p0ET9pTnA82rKQbmsu9bZP+4Anm/Vhwrxsuo6cFn6FsYiIxJ3CPnrS+2/z1J5bgidu3wBpWtsh4jclwfFm+3KY1stb/DbgZ2hwaNkrjJUIi4j4rrRSaFnp35K15+Zgwzs3q7qPSJzQdIh4krPWS4DzdkD3KdDgUKD0FcajJi/yI0oRiWNm1tfMFpnZYjO7Kcz5883su8DXLDM7yo84k0nhQEX25hwcwYGK1U+eA2OHeo0atoa7tigBFokjGgmOF3s2wfQ+kLMaenwIDTsWnQq3wris4yKSmswsDXgM6AWsAmab2UTn3A8hzZYB3Zxzm8ysH/AUcHzso00e4QYq3rS/02zNL95Om55w4TgfIhORsmgkOB7k7YAZZ8LWH+HUCdC4eBWIkiuMyzsuIimrC7DYObfUObcHGAsMDG3gnJvlnNsU2P0CaI5US+iARG1yWV53GO1rBBLgdmcoARaJU0qC/Za/G2YOgt++hK5joOne9SJDVxgXSq+Vxg192sUqShFJDJnAypD9VYFjpbkceD+qEaWAwgGJNpbNT3UvLjp+Xu3HYOgYv8ISkXIoCfZTQT7MugDWToEuT0OLQWGbZXXKZMSgDmRmpGNAZkY6IwZ10KI4ESkp3IRTF7ahWXe8JPjGUs5fYWZzzGzO+vXrIxhi8rmhTzsG1JrLR3VuKDrWMf9lhvbr7mNUIlIezQn2S0E+jA18/J0egDaXldk8q1Omkl4RKc8qoEXIfnNgdclGZtYReAbo55z7LdyFnHNP4c0XpnPnzmETaQHydpO19hGy0p4AYFb+EdxQ7z7uURlLkbinJDjCSiuTU4xzwQS4bhM4/G+xD1REktFsoK2ZtQaygSHAsNAGZtYSGAdc6Jz7KfYhJpFNy+GNS2H113D8n6DXPZxUszaf+R2XiFSIkuAIqnA93zEhs1DO3muQRkSkSpxzeWZ2NTAZSAOec84tMLMrA+efAO4ADgAeN69cV55zrrNfMSeshe/AhKu87fNehsMH7NWkQoMiIuIbJcERVFY936KO79WQKXtD9qhmpIhElHNuEjCpxLEnQraHA8NjHVfSyNsDU++AL/8HzY6Bwc9Dw1Z7NdNDjkTinxbGRVC59XxDE+DBW72nwomISGLYtBye6+MlwMf/CS6bHDYBBj3kSCQRaCQ4gpplpJMdJhFulpEO40NKcZ69Gmo1iGFkIiJSLV+/BBP/7G2XMv0hlB5yJBL/NBIcQaXV853wuzsgJ9s7cOZCSG/qQ3QiIlJpeXtgZMtgAtzuzHITYNBDjkQSgZLgCApXz3fSiS/TeGdgrXCvz2C/3/kao4iIVND6n+DexrBri7d//J9g6KsVeqkeciQS/zQdIsKK1fNdMBK+fcXbPmUcND6pqJ1WDYuIxLHxV8K3IU97u3oONGpb4ZcX9ufq50Xil5LgaFk2Gr692dvu/Ci0OLvolFYNi4jEj9BBiTb7wYe7Q0orNz0K/jizStfVQ45E4pumQ0TDlh/h84u87SNuhMOuKnZaq4ZFROJD4aBE9uYczk+bWjwBHj6tygmwiMQ/jQRH2ub5MKmjt91yMBw9cq8mWjUsIhIfRk1eREFuDsvrXlLs+Ml13uLT5sf6E5SIxIRGgiNpxy/BBPikMXDy62GbadWwiEh86LbtXRaFJMDX7vkTrXa9SvaW3f4FJSIxoSQ4UnZvhLdbeds1akGrIaU21aphERGfbfsV7tqP+2s9C8DYvNNot+sFxhecAmhQQiQVaDpEJOTlwFsHBPeH7CmzuVYNi4j4JD8Pnj4N1s4vOnR9wZ95M+/Eon0NSoikhogkwWZ2PTAKaOyc2xCJayaMgnx4fZ/g/jBXZvOSpdEeOu9oJb8iItHmHMwbHXzoBcABh8JVszn52zV8rkEJkZRT7STYzFoAvYAV1Q8nwTgHY0M+wgokwCqNJiISY9lz4ekexY/99Vto2ApQKTORVBWJOcEPAX8Hys4Ak9GYkI9vaEG5zVUaTUQkhjYthzcvK54Ad7sR7tpSlABX1IR52XQdOY3WN71H15HTmDAvO6KhikjsVWsk2MzOArKdc9+aWXltrwCuAGjZsmV13jY+vBry/Z63G8r5/kGl0UREYmLnRvjkAfjqKbA0aNAUXAFc9RWkZ1T6crqLJ5Kcyk2CzexDoEmYU7cCtwC9K/JGzrmngKcAOnfunNijxqEJ8OAtkFa7Qi9rlpFOdpiEV6uQRUQiIG+3l/jO/Dfs2gKdzofut8K+zap12bLu4ikJFklc5SbBzrnTwx03sw5Aa6BwFLg58LWZdXHOrY1olPEk+73g9tmroda+FX7pDX3aFRtNAK1CFhGptoICWDAOProbNq+AQ0+HXvfAQe0jcnndxRNJTlWeDuGcmw8cWLhvZsuBzkldHeLXj+HT30Pt/aH/j1C3caVertJoIiIRtvxTmHIbrJ4HTTrAhROgTfeIvoXu4okkJ9UJrqiNc+HjAVCvNZw+E+o2qtJltApZRCQC1i+CqXfCT+/DvpmQ9QR0PA9qRP4ZULqLJ5KcIpYEO+daRepacWfLjzC9L9TZnw8OfJF//Oc7jeSKiPhh268wYwR8/RLUrgc974QT/gS1ojcqq7t4IslJI8Hl2fELTO8FlsbUJi9z7Tu/aYWwiEis7dkBsx6Fzx6G/N1w3HDo9neoV7W7cpWlu3giyUdJcFlyfoWPTofc7XD6x9z15AatEBYRiaX8PPjmZZh+P2z/FQ4/C06/Cw5oE5O3L/mUT40AiyQPJcGl2bMZpveBnNXQ40No2JHVm98L21QrhEVEIsw5+HmKN+93/UJo3gXOHQ0tj49ZCKoPLJLcIr+CIBnk7YAZZ8LWH+DUCdD4RKD0lcBaISwiEkGr58GLA+DVcyF/j5f8Xj4lpgkw6CmfIslOI8El5e+BT86B376Arq9B015Fp7RCWEQkSvL2wI/vwOzn4JdPYZ8DoN8o6HwppNXyJSTVBxZJbkqCQxXkw+cXwJrJcPwz0PL3xU5rhbCISIRtXglzX/CqPexYBxkHe3N+O18GdffzNTTVBxZJbkqCCzkHs6+EFW9Ap39Dm8vDNitrhbAWUIiIVEBBASyZBrOfgZ8ne8fa9oHjLoc2PaNS67cqdPdPJLkpCQYvAf7mRljyDLS/FQ6/rtKX0AIKEZFy7PgN5o2Guc/DpuVQrzGcfC0cewlktPQ7ur3o7p9IclMSDPDDSFg4CtpeBR3/UaVLlLWAQh2miKQs52DlVzDnWVgw3lt3cXBX6HkH/G4A1Kwd1bev7h061QcWSV5Kgn/+H3x7C7Q6Hzo/AmZVuowWUIiIhNi9Hea/DrOfhV+/h9oNvBHfzpfBgYfHJATdoRORsqR2Erx8DMy+CjIHwAnPg1V9HpoWUIiIhHhpIGTPgSYdoP9/oMNgqFM/piHoDp2IlCV1k+Ds9+Dzi+DAU71SaDWqV4JHCyhEREL0uNUb/W3eucp32KpLd+hEpCypmQT/+jF8+ntoeBR0mwg1qz9aqwUUIiIh2vTwOwLdoRORMiV1Ehx2QcTBa+HjAVCvFZz2AdTaN2LvpwUUIiLxQ3foRKQs8VGMMQoKF0Rkb87B4S2IeOrdSeye2hvq7A89pkLdRn6HKSIiUZLVKZMRgzqQmZGOAZkZ6YwY1EGDFSICJPFIcMkFEZm11vFMi1vYvsdRp99U2Ke5j9GJiEgs6A6diJQmaZPg0IUPjWpuYvQht1GvRg5Dl4xg0r5tfYxMRETihZ70KZK6knY6ROHCh31rbOel1nfQpNZvXLr8LrbUbe9zZCIiEg/CTZu7edx8JszL9js0EYmBpE2Cb+jTjoa1c3m29T0cWmclVy6/hYW5R2pBhIiIAGXXERaR5Je00yGyOjbmxBUP02j7j/x5xd9ZUvNkRpyp21wiIuJRHWGR1JYQSXCl52wV5MPnF3DQjhlwwjM8fv7lMYtVREQSg+oIi6S2uJ8OUek5W87B7CthxRvQ6d/QRgmwiIjs7YY+7UivlVbsmOoIi6SOuE+CKzVnyzn45kZY8gy0vwUOvy5GUYqISKJRHWGR1Bb30yEqNWfrh3/CwlHQ9v+g471RjkxERCIt1iXLVEdYJHXF/UhwaXOz9jr+8xPw7c1w8DDo/F8wi0F0IiISKSpZJiKxFPdJcIXmbC0fA7P/D5r1hxNfAIv7b0tEREpQyTIRiaW4nw5ReJuq1Ntj2e/B5xfBgafCya9DjVo+RisiIlWlkmUiEktxnwRDGXO21s2ET38PDY+CbhOhpsraiIgkKpUsE5FYStx5Axu/ho8HQL1WcNoHUGtfvyMSEZFqUMkyEYmlhBgJ3suWH2F6H6jdEHpMhbqN/I5IRESqqdzpbyIiEZR4SfCOFTC9N1gadJ8K+zT3OyIREYkQlSwTkVhJrOkQu9bBtF6QuxW6T4Z92/odkYhIXDGzvma2yMwWm9lNYc6bmT0SOP+dmR0T6RgmzMum68hptL7pPbqOnKYSZyISlxJnJHjPZm8KxM6V3hSIhkf5HZGISFwxszTgMaAXsAqYbWYTnXM/hDTrB7QNfB0P/C/w34gorPVbWOqssNYvoBFeEYkriTESnLfTWwS3ZQGcMh4ad/U7IhGReNQFWOycW+qc2wOMBQaWaDMQeMl5vgAyzKxppAJQrV8RSRSJkQQD1NoPTnoFmvXxOxIRkXiVCawM2V8VOFbZNpjZFWY2x8zmrF+/vsIBqNaviCSKxEiCa+4D3d6BloP9jkREJJ6Fe168q0IbnHNPOec6O+c6N27cuMIBVPhR9yIiPkuMJBjAwvXbIiISYhXQImS/ObC6Cm2qTLV+RSRRJE4SLCIi5ZkNtDWz1mZWGxgCTCzRZiJwUaBKxAnAFufcmkgFkNUpkxGDOpCZkY4BmRnpjBjUQYviRCTuJE51CBERKZNzLs/MrgYmA2nAc865BWZ2ZeD8E8Ak4AxgMbATuDTScajWr4gkAiXBIiJJxDk3CS/RDT32RMi2A66KdVwiIvFG0yFEREREJOUoCRYRERGRlKMkWERERERSjpJgEREREUk5SoJFREREJOUoCRYRERGRlKMkWERERERSjpJgEREREUk5SoJFREREJOWY9/CgGL+p2Xrglxi9XSNgQ4zeK17pM9BnAPoMIDKfwcHOucaRCCZRqM+OOX0G+gxAnwFE7jMI22/7kgTHkpnNcc519jsOP+kz0GcA+gxAn0Ei0M9InwHoMwB9BhD9z0DTIUREREQk5SgJFhEREZGUkwpJ8FN+BxAH9BnoMwB9BqDPIBHoZ6TPAPQZgD4DiPJnkPRzgkVERERESkqFkWARERERkWKSMgk2s8FmtsDMCsysc4lzN5vZYjNbZGZ9/IoxlszsLjPLNrNvAl9n+B1TrJhZ38DPerGZ3eR3PH4xs+VmNj/w85/jdzyxYGbPmdk6M/s+5Nj+ZjbVzH4O/LehnzGKR3323tRvp3a/rT676FhU++ykTIKB74FBwMzQg2Z2BDAEaA/0BR43s7TYh+eLh5xzRwe+JvkdTCwEfraPAf2AI4ChgX8Dqap74OefKiV3XsD7/zzUTcBHzrm2wEeBffGf+uzw1G+ndr+tPjvKfXZSJsHOuYXOuUVhTg0ExjrndjvnlgGLgS6xjU5iqAuw2Dm31Dm3BxiL929AUoBzbiawscThgcCLge0XgaxYxiThqc+WEOq3U5QffXZSJsFlyARWhuyvChxLBVeb2XeB2w2pcgs4lX/eJTlgipnNNbMr/A7GRwc559YABP57oM/xSNlS/f9h9dup9zMvpD7bE9U+u2YkLxZLZvYh0CTMqVudc2+X9rIwx5KiPEZZnwfwP+AfeN/rP4AHgMtiF51vkvbnXQVdnXOrzexAYKqZ/Rj4q1skJtRn7039dlhJ/TOvBPXZMZCwSbBz7vQqvGwV0CJkvzmwOjIR+auin4eZPQ28G+Vw4kXS/rwryzm3OvDfdWY2Hu+WYyp2qL+aWVPn3Bozawqs8zugVKE+e2/qt8NK6p95RanPLhLVPjvVpkNMBIaYWR0zaw20Bb7yOaaoC/zDKXQ23iKUVDAbaGtmrc2sNt4Cm4k+xxRzZlbPzBoUbgO9SZ1/AyVNBC4ObF8MlDYCKfEhJftsUL+dyv22+uxiotpnJ+xIcFnM7Gzgv0Bj4D0z+8Y518c5t8DMXgd+APKAq5xz+X7GGiP/MrOj8W4pLQf+6Gs0MeKcyzOzq4HJQBrwnHNugc9h+eEgYLyZgff//KvOuQ/8DSn6zGwMcBrQyMxWAXcCI4HXzexyYAUw2L8IpZD67LDUb6duv60+O0Z9tp4YJyIiIiIpJ9WmQ4iIiIiIKAkWERERkdSjJFhEREREUo6SYBERERFJOUqCRURERCTlKAkWERERkZSjJFhEREREUo6SYBERERFJOf8PLSmxooyflyEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12,6))\n",
    "# TODO: plot function at each X[:,0]\n",
    "\n",
    "#I plotted them at X[:,1] instead of X[:,0]\n",
    "#Not sure if you meant that literally or not \n",
    "\n",
    "#Linear Regression\n",
    "plt.subplot(121)\n",
    "plt.scatter(X[:, 1], y)\n",
    "plt.plot(X[:, 1], np.dot(X, weights), color='orange')\n",
    "plt.title('linear Regression')\n",
    "\n",
    "#Logistic Regression\n",
    "plt.subplot(122)\n",
    "plt.scatter(X[:, 1], y_prob)\n",
    "plt.plot(X[:, 1], sigmoid(np.dot(X, log_w)))\n",
    "plt.title('logistic Regression')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89ea48e9",
   "metadata": {},
   "source": [
    "Plots:\n",
    "- From the plots it is hard to tell what regression fit the data better \n",
    "- Both plots miss points in the data \n",
    "- The logistic regression doesn;t seem as great of fit from the graph\n",
    "\n",
    "Graph: \n",
    "- The logistic error I calculated is smaller than the MSE \n",
    "- I feel the linear error I calculated seems a bit high in comparison to the logisitc error\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47dc92ea",
   "metadata": {},
   "source": [
    "### Problem 2: Ensembles [16 pts]\n",
    "\n",
    "Use the dataset provided by scipy for this problem. Feel free to use the sklearn implementation to train weak learners and numpy for any vector/matrix operations. To control randomness, you can set random_state where applicable to see the random number generator.\n",
    "\n",
    "**(2a)** Combine your favorite weak learner with bagging to classify the data. [4 pts]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "id": "d1ac17df",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-17 {color: black;background-color: white;}#sk-container-id-17 pre{padding: 0;}#sk-container-id-17 div.sk-toggleable {background-color: white;}#sk-container-id-17 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-17 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-17 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-17 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-17 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-17 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-17 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-17 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-17 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-17 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-17 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-17 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-17 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-17 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-17 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-17 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-17 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-17 div.sk-item {position: relative;z-index: 1;}#sk-container-id-17 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-17 div.sk-item::before, #sk-container-id-17 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-17 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-17 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-17 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-17 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-17 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-17 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-17 div.sk-label-container {text-align: center;}#sk-container-id-17 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-17 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-17\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>BaggingClassifier(base_estimator=GaussianNB(), n_estimators=20)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-49\" type=\"checkbox\" ><label for=\"sk-estimator-id-49\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">BaggingClassifier</label><div class=\"sk-toggleable__content\"><pre>BaggingClassifier(base_estimator=GaussianNB(), n_estimators=20)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-50\" type=\"checkbox\" ><label for=\"sk-estimator-id-50\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">base_estimator: GaussianNB</label><div class=\"sk-toggleable__content\"><pre>GaussianNB()</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-51\" type=\"checkbox\" ><label for=\"sk-estimator-id-51\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GaussianNB</label><div class=\"sk-toggleable__content\"><pre>GaussianNB()</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "BaggingClassifier(base_estimator=GaussianNB(), n_estimators=20)"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import BaggingClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "\n",
    "data, target = load_breast_cancer(return_X_y=True)\n",
    "training_data, test_data, training_labels, test_labels = train_test_split(data, target,\n",
    "                                                                          test_size=0.1, random_state=1)\n",
    "\n",
    "#TODO: implement bagging with your choice of weak learner\n",
    "#Naives Bayes \n",
    "w_l = GaussianNB()\n",
    "bagging =  BaggingClassifier(base_estimator=weak_learn, n_estimators=20)\n",
    "bagging.fit(training_data, training_labels)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c30d370",
   "metadata": {},
   "source": [
    "**(2b)** Combine the same weak learner from (2a) with AdaBoost to fit the data. [6 pts]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "3e7c5ae3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-18 {color: black;background-color: white;}#sk-container-id-18 pre{padding: 0;}#sk-container-id-18 div.sk-toggleable {background-color: white;}#sk-container-id-18 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-18 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-18 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-18 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-18 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-18 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-18 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-18 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-18 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-18 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-18 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-18 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-18 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-18 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-18 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-18 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-18 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-18 div.sk-item {position: relative;z-index: 1;}#sk-container-id-18 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-18 div.sk-item::before, #sk-container-id-18 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-18 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-18 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-18 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-18 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-18 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-18 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-18 div.sk-label-container {text-align: center;}#sk-container-id-18 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-18 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-18\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>AdaBoostClassifier(base_estimator=GaussianNB(), n_estimators=20)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-52\" type=\"checkbox\" ><label for=\"sk-estimator-id-52\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">AdaBoostClassifier</label><div class=\"sk-toggleable__content\"><pre>AdaBoostClassifier(base_estimator=GaussianNB(), n_estimators=20)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-53\" type=\"checkbox\" ><label for=\"sk-estimator-id-53\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">base_estimator: GaussianNB</label><div class=\"sk-toggleable__content\"><pre>GaussianNB()</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-54\" type=\"checkbox\" ><label for=\"sk-estimator-id-54\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GaussianNB</label><div class=\"sk-toggleable__content\"><pre>GaussianNB()</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "AdaBoostClassifier(base_estimator=GaussianNB(), n_estimators=20)"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "# TODO: implement AdaBoost with weak learner\n",
    "adaboost = AdaBoostClassifier(base_estimator=w_l, n_estimators=20)\n",
    "adaboost.fit(training_data, training_labels)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7dcc9d1",
   "metadata": {},
   "source": [
    "**(2c)** Evaluate the combined models from (2a) and (2b) on the test partition and compute average accuracy. [2 pts]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "id": "4600eacf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Accuracy: 0.7456140350877193\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from scipy.stats import mode\n",
    "from sklearn.utils.extmath import weighted_mode\n",
    "\n",
    "# TODO: compute average accuracies\n",
    "#bagging accuracy\n",
    "bagging_accuracy = bagging.score(test_data, test_labels)\n",
    "\n",
    "#ada accuracy\n",
    "ada_accuracy = adaboost.score(test_data, test_labels)\n",
    "\n",
    "#average accuracy\n",
    "average_accuracy = np.mean([bagging_accuracy, ada_accuracy])\n",
    "print(\"Average Accuracy:\", average_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "081e1382",
   "metadata": {},
   "source": [
    "**(2d)** Find the per-model accuracy for both your bagging and boosting ensembles. [2 pts]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "id": "e018e6cb",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bagging Accuracy: 0.9473684210526315\n",
      "AdaBoost Accuracy: 0.543859649122807\n"
     ]
    }
   ],
   "source": [
    "# TODO: compute per-model accuracies for weak learners\n",
    "#print bagging accuracy\n",
    "print(\"Bagging Accuracy:\", bagging_accuracy)\n",
    "\n",
    "#print adaboost accuracy\n",
    "print(\"AdaBoost Accuracy:\", ada_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a58a1778",
   "metadata": {},
   "source": [
    "**(2e)** What, if any, differences do you observe in performance between using bagging and AdaBoost? Between your weak classifiers and ensembles? Is this result surprising? [2 pts]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa0b0ec3",
   "metadata": {},
   "source": [
    "- The accuracy for bagging is higher than AdaBoost\n",
    "- You can see the Adaboost accuracy is significantly lower, this result was very surprising to me\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}