06_evaluate.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we will load a model we've created earlier, and evaluate its performance against external models - essentially judging the relative performance.\n",
    "\n",
    "We will use the \"params\" dataset which contains predictions from multiple external models (\"value-added catalog\"). We will score our model against TEFF_MED column, which is a median of all other predictions. The value-added catalog is [described in SDSS](https://www.sdss.org/dr17/mastar/mastar-stellar-parameters/), with links to papers describing particular models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mlflow\n",
    "import stars\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.metrics import mean_squared_error\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "sl = stars.StarLoader()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have ran the 05 notebook, your model will be stored in the MLFlow's local storage (`mlruns` directory).\n",
    "\n",
    "You can investigate this model by running `mlflow ui` and visiting [local MLFlow UI](http://localhost:5000). Therein you can also find the link to the stored model for use below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "inputs: \n",
       "  [Tensor('float64', (-1, 4563))]\n",
       "outputs: \n",
       "  [Tensor('float64', (-1,))]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stored_model_path = './mlruns/1/be8132aa780d47d59fd94d4dacda8ce8/artifacts/model'\n",
    "model = mlflow.pyfunc.load_model(stored_model_path)\n",
    "model._model_meta._signature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stars evaluated only by external algorithm: 19715\n"
     ]
    }
   ],
   "source": [
    "badt = sl.get_badt()\n",
    "df_badt = sl.fits_to_pd(badt)\n",
    "\n",
    "print('Stars evaluated only by external algorithm: %d' % len(df_badt))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEICAYAAABfz4NwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAWLElEQVR4nO3df7RdZX3n8fenCfJDBKEJTEwiCW1Wx8CMVTJMHB3bGXSIVAe6VhnjL6LDTFYpWtvqckilLZ0pM+i0zsg40GHEIaAIKdUh7Vq00lhW2zWU9KJgCJgSBUNMTIIOGBhLDX7nj/NcPJyce3N/nPurvF9rnXX2efaz9/7uu3Pv5+xn73OSqkKSpB+Z6QIkSbODgSBJAgwESVJjIEiSAANBktQYCJIkwECQJDUGggYqye8m+bUBrevlSZ5KMq+9vivJvxnEutv67kiyblDrG8d2fyvJ40m+Nd3blkZjIGjMkjya5HtJDiZ5Isn/SfLzSZ77d1RVP19V/2GM63rDaH2qaldVHV9Vzw6g9iuSfLpn/W+qqo2TXfc461gKfABYWVV/r2feO1oAPtV+zj/oev1U6zN8DJ7qenyizXt3kmdHmHdDkr/tmffWI9T6aFtmQU/7fUkqybJR1n1/m7es9R1u35fkD5O8cUA/Ug2QgaDxektVvQQ4DbgK+HfA9YPeSJL5g17nLHEa8O2q2t87o6o+0wLweOBNwJ7h161t2Fu626vqvV3z7h5l3kd75t06hnofAd42/CLJPwCO7dOvd92v7Jn/0rYPrwTuBD6f5N1j2L6mkYGgCamqJ6tqM/BWYF2SM+G5d4u/1aYXtHeDTyT5TpI/T/IjSW4CXg78QXvX+KGud5IXJ9kFfLGrrTscfizJ1iRPJrk9ycltWz+dZHd3jcNnIUnWAL8KvLXn3etzQ1CtrsuTfCPJ/iQ3JjmxzRuuY12SXW2458Mj/WySnNiWP9DWd3lb/xvo/DF8WavjhgEciql2E3BR1+t1wI0TXVlVfauqPg5cAXyk++xSM8+DoUmpqq3AbuCf9pn9gTZvIXAqnT/KVVXvAnbxw3e6H+1a5qeAVwDnjrDJi4B/DbwMOARcPYYa/wj4j8CtI7x7BXh3e/wz4HTgeOATPX1eB/wEcA7w60leMcIm/xtwYlvPT7Wa31NVf8Lz3/m/+0i1zwJ/CZyQ5BXtWs5bgU8fYZmx+BxwCp2fp2YJA0GDsAc4uU/794FFwGlV9f2q+vM68rcpXlFVT1fV90aYf1NVPVBVTwO/Bvyr9odqst4BfKyqvl5VTwEbgLU9Zye/WVXfq6r7gfvpDH88T9cfzQ1VdbCqHgV+B3jXAGoc9r/bWdfw4992zVvdM29117wPdrU/Po7tDZ8lvBH4KvDNPn0+2LPdI12b2dOe+/270Qz5uzpOq+m1GPhOn/b/TGdo4AtJAK6rqquOsK7HxjH/G8BRwIIR+o7Hy9r6utc9n86ZzbDuu4L+H52ziF4LgBf1WdfiAdQ47IJ2ttHPX1bV60aY99tVdfkEtncT8GfAckYeLhrvuod/Hv3+3WiGeIagSUnyj+j8cv9F77z2DvkDVXU68BbgV5KcMzx7hFUe6Qxiadf0y+mchTwOPA0c11XXPDpDVWNd7x46F3y7130I2HeE5Xo93mrqXVe/d9VzQlV9g87F5fPoDPUMws8C+4EdA1qfBsBA0IQkOSHJm4FbgE9X1bY+fd6c5MfTOT34LvBse0DnD+3pE9j0O5OsTHIc8O+B29ptqX8NHJPkZ5IcBVwOHN213D5g2SgXMT8L/HKS5UmO54fXHA6Np7hWyybgyiQvSXIa8CsMZtx9Jl0M/PM2VDdhSU5N8l7gN+gMq/1gINVpIAwEjdcfJDlIZ+jmw8DHgPeM0HcF8CfAU8DdwDVVdVeb95+Ay9t48wfHsf2bgBvoDN8cA/widO56An4B+CSdd+NP07mgPez32vO3k3ypz3o/xQ+HRh4B/gZ43zjq6va+tv2v0zlzurmtf1CG784afnx+gOvuq6q+VlVDo3T5UE9NvdconkjyNLCNzpnGhVU1yJ+JBiD+j2mSJPAMQZLUeJeR9AKV5OXAgyPMXllVu6azHs08h4wkScAcPkNYsGBBLVu2bKbLkKQ55d577328qhb2mzdnA2HZsmUMDY1204MkqVeSb4w0z4vKkiTAQJAkNQaCJAkwECRJjYEgSQIMBElSYyBIkgADQZLUGAiSJGAOf1JZ43Rzxtbv7X63lfRC5RmCJAkwECRJjYEgSQIMBElSYyBIkgADQZLUGAiSJMBAkCQ1BoIkCTAQJEmNgSBJAgwESVJjIEiSAANBktQYCJIkYAyBkORTSfYneaCr7eQkdyZ5uD2f1DVvQ5KdSXYkOber/awk29q8q5OktR+d5NbWfk+SZQPeR0nSGIzlDOEGYE1P22XAlqpaAWxpr0myElgLnNGWuSbJvLbMtcB6YEV7DK/zYuD/VtWPA/8F+MhEd0aSNHFHDISq+jPgOz3N5wMb2/RG4IKu9luq6pmqegTYCZydZBFwQlXdXVUF3NizzPC6bgPOGT57kCRNn4leQzi1qvYCtOdTWvti4LGufrtb2+I23dv+vGWq6hDwJPCj/TaaZH2SoSRDBw4cmGDpkqR+Bn1Rud87+xqlfbRlDm+suq6qVlXVqoULF06wRElSPxMNhH1tGIj2vL+17waWdvVbAuxp7Uv6tD9vmSTzgRM5fIhKkjTFJhoIm4F1bXodcHtX+9p259ByOhePt7ZhpYNJVrfrAxf1LDO8rp8DvtiuM0iSptH8I3VI8lngp4EFSXYDvwFcBWxKcjGwC7gQoKq2J9kEPAgcAi6tqmfbqi6hc8fSscAd7QFwPXBTkp10zgzWDmTPJEnjcsRAqKq3jTDrnBH6Xwlc2ad9CDizT/vf0AJFkjRz/KSyJAkwECRJjYEgSQIMBElSYyBIkgADQZLUGAiSJMBAkCQ1BoIkCTAQJEmNgSBJAgwESVJjIEiSAANBktQYCJIkwECQJDUGgiQJMBAkSY2BIEkCDARJUmMgSJIAA0GS1BgIkiTAQJAkNQaCJAkwECRJjYEgSQIMBElSM6lASPLLSbYneSDJZ5Mck+TkJHcmebg9n9TVf0OSnUl2JDm3q/2sJNvavKuTZDJ1SZLGb8KBkGQx8IvAqqo6E5gHrAUuA7ZU1QpgS3tNkpVt/hnAGuCaJPPa6q4F1gMr2mPNROuSJE3MZIeM5gPHJpkPHAfsAc4HNrb5G4EL2vT5wC1V9UxVPQLsBM5Osgg4oarurqoCbuxaRpI0TSYcCFX1TeC3gV3AXuDJqvoCcGpV7W199gKntEUWA491rWJ3a1vcpnvbJUnTaDJDRifRede/HHgZ8OIk7xxtkT5tNUp7v22uTzKUZOjAgQPjLVmSNIrJDBm9AXikqg5U1feBzwH/BNjXhoFoz/tb/93A0q7ll9AZYtrdpnvbD1NV11XVqqpatXDhwkmULknqNZlA2AWsTnJcuyvoHOAhYDOwrvVZB9zepjcDa5McnWQ5nYvHW9uw0sEkq9t6LupaRpI0TeZPdMGquifJbcCXgEPAl4HrgOOBTUkuphMaF7b+25NsAh5s/S+tqmfb6i4BbgCOBe5oD0nSNErnxp65Z9WqVTU0NDTTZcwdN4/xox1vn5v/HiSNTZJ7q2pVv3l+UlmSBBgIkqTGQJAkAQaCJKkxECRJgIEgSWoMBEkSYCBIkhoDQZIEGAiSpMZAkCQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKkxECRJwCQDIclLk9yW5KtJHkrymiQnJ7kzycPt+aSu/huS7EyyI8m5Xe1nJdnW5l2dJJOpS5I0fpM9Q/g48EdV9feBVwIPAZcBW6pqBbClvSbJSmAtcAawBrgmyby2nmuB9cCK9lgzybokSeM04UBIcgLweuB6gKr626p6Ajgf2Ni6bQQuaNPnA7dU1TNV9QiwEzg7ySLghKq6u6oKuLFrGUnSNJnMGcLpwAHgfyX5cpJPJnkxcGpV7QVoz6e0/ouBx7qW393aFrfp3vbDJFmfZCjJ0IEDByZRuiSp12QCYT7wauDaqnoV8DRteGgE/a4L1CjthzdWXVdVq6pq1cKFC8dbryRpFJMJhN3A7qq6p72+jU5A7GvDQLTn/V39l3YtvwTY09qX9GmXJE2jCQdCVX0LeCzJT7Smc4AHgc3Auta2Dri9TW8G1iY5OslyOhePt7ZhpYNJVre7iy7qWkaSNE3mT3L59wGfSfIi4OvAe+iEzKYkFwO7gAsBqmp7kk10QuMQcGlVPdvWcwlwA3AscEd7SJKm0aQCoaruA1b1mXXOCP2vBK7s0z4EnDmZWiRJk+MnlSVJgIEgSWoMBEkSYCBIkhoDQZIEGAiSpMZAkCQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKkxECRJgIEgSWoMBEkSYCBIkhoDQZIEDCAQksxL8uUkf9hen5zkziQPt+eTuvpuSLIzyY4k53a1n5VkW5t3dZJMti5J0vgM4gzh/cBDXa8vA7ZU1QpgS3tNkpXAWuAMYA1wTZJ5bZlrgfXAivZYM4C6JEnjMKlASLIE+Bngk13N5wMb2/RG4IKu9luq6pmqegTYCZydZBFwQlXdXVUF3Ni1jCRpmkz2DOG/Ah8CftDVdmpV7QVoz6e09sXAY139dre2xW26t/0wSdYnGUoydODAgUmWLknqNuFASPJmYH9V3TvWRfq01SjthzdWXVdVq6pq1cKFC8e4WUnSWMyfxLKvBf5lkvOAY4ATknwa2JdkUVXtbcNB+1v/3cDSruWXAHta+5I+7ZKkaTThM4Sq2lBVS6pqGZ2LxV+sqncCm4F1rds64PY2vRlYm+ToJMvpXDze2oaVDiZZ3e4uuqhrGUnSNJnMGcJIrgI2JbkY2AVcCFBV25NsAh4EDgGXVtWzbZlLgBuAY4E72kOSNI0GEghVdRdwV5v+NnDOCP2uBK7s0z4EnDmIWiRJE+MnlSVJgIEgSWoMBEkSYCBIkhoDQZIETM1tpxqEm8f4ha9v7/uhbkkaN88QJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEmAgSJIaA0GSBPhtp+rlt6xKL1ieIUiSAANBktQYCJIkwECQJDUGgiQJMBAkSY2BIEkCDARJUjPhQEiyNMmfJnkoyfYk72/tJye5M8nD7fmkrmU2JNmZZEeSc7vaz0qyrc27OskYPx0lSRqUyZwhHAI+UFWvAFYDlyZZCVwGbKmqFcCW9po2by1wBrAGuCbJvLaua4H1wIr2WDOJuiRJEzDhQKiqvVX1pTZ9EHgIWAycD2xs3TYCF7Tp84FbquqZqnoE2AmcnWQRcEJV3V1VBdzYtYwkaZoM5BpCkmXAq4B7gFOrai90QgM4pXVbDDzWtdju1ra4Tfe299vO+iRDSYYOHDgwiNIlSc2kAyHJ8cDvA79UVd8drWufthql/fDGquuqalVVrVq4cOH4i5UkjWhSgZDkKDph8Jmq+lxr3teGgWjP+1v7bmBp1+JLgD2tfUmfdknSNJrMXUYBrgceqqqPdc3aDKxr0+uA27va1yY5OslyOhePt7ZhpYNJVrd1XtS1jCRpmkzm/0N4LfAuYFuS+1rbrwJXAZuSXAzsAi4EqKrtSTYBD9K5Q+nSqnq2LXcJcANwLHBHe0iSptGEA6Gq/oL+4/8A54ywzJXAlX3ah4AzJ1qLJGny/KSyJAkwECRJjYEgSQIMBElSYyBIkgADQZLUGAiSJMBAkCQ1BoIkCTAQJEmNgSBJAgwESVIzmW871Wxw80jfLyhJ4+MZgiQJMBAkSY2BIEkCDARJUmMgSJIAA0GS1BgIkiTAQJAkNQaCJAnwk8qaqPF8QvrtNXV1SBoYzxAkSYCBIElqDARJEmAgSJIaA0GSBHiXkabDWO9I8m4kaUbNmkBIsgb4ODAP+GRVXTXDJWm6Dfo/+zFgpHGZFYGQZB7w34E3AruBv0qyuaoenJINzuQ7Vv+Hs+njmYk0LrMiEICzgZ1V9XWAJLcA5wNTEwhSt7kQ0mMNrZnaF0N1MGb4TcxsCYTFwGNdr3cD/7i3U5L1wPr28qkkO6a0qndkAfD4lG5jas31+mHu78Ng6n/HjIbWkfdhZus7kr97/4Ym9/M+baQZsyUQ+u3dYRFYVdcB1019OR1Jhqpq1XRtb9Dmev0w9/dhrtcPc38frH/sZsttp7uBpV2vlwB7ZqgWSXpBmi2B8FfAiiTLk7wIWAtsnuGaJOkFZVYMGVXVoSTvBf6Yzm2nn6qq7TNcFkzj8NQUmev1w9zfh7leP8z9fbD+MUqVdwdIkmbPkJEkaYYZCJIkwEDoK8maJDuS7Exy2UzX0y3Jo0m2JbkvyVBrOznJnUkebs8ndfXf0PZjR5Jzu9rPauvZmeTqJFN2I3mSTyXZn+SBrraB1Zzk6CS3tvZ7kiybhvqvSPLNdhzuS3LeLK5/aZI/TfJQku1J3t/a59IxGGkf5sRxSHJMkq1J7m/1/2Zrn13HoKp8dD3oXNT+GnA68CLgfmDlTNfVVd+jwIKeto8Cl7Xpy4CPtOmVrf6jgeVtv+a1eVuB19D5DMgdwJumsObXA68GHpiKmoFfAH63Ta8Fbp2G+q8APtin72ysfxHw6jb9EuCvW51z6RiMtA9z4ji0bR3fpo8C7gFWz7ZjMCV/AObyo/2g/7jr9QZgw0zX1VXPoxweCDuARW16EbCjX+107uJ6Tevz1a72twH/Y4rrXsbz/6AOrObhPm16Pp1PdWaK6x/pD9GsrL+nxtvpfG/YnDoGI+zDnDsOwHHAl+h8G8OsOgYOGR2u39doLJ6hWvop4AtJ7k3nqzwATq2qvQDt+ZTWPtK+LG7Tve3TaZA1P7dMVR0CngR+dMoq/6H3JvlKG1IaPtWf1fW3YYRX0XmHOiePQc8+wBw5DknmJbkP2A/cWVWz7hgYCIcb09dozKDXVtWrgTcBlyZ5/Sh9R9qX2byPE6l5JvbnWuDHgJ8E9gK/c4RaZrz+JMcDvw/8UlV9d7SuI9QzG/dhzhyHqnq2qn6SzjcxnJ3kzFG6z0j9BsLhZvXXaFTVnva8H/g8nW+K3ZdkEUB73t+6j7Qvu9t0b/t0GmTNzy2TZD5wIvCdKascqKp97Rf8B8D/pHMcnldLT50zWn+So+j8If1MVX2uNc+pY9BvH+bacWg1PwHcBaxhlh0DA+Fws/ZrNJK8OMlLhqeBfwE8QKe+da3bOjrjq7T2te3ug+XACmBrOzU9mGR1u0Phoq5lpssga+5e188BX6w2kDpVhn+Jm5+lcxxmZf1te9cDD1XVx7pmzZljMNI+zJXjkGRhkpe26WOBNwBfZbYdg6m66DOXH8B5dO5i+Brw4Zmup6uu0+nceXA/sH24NjrjhFuAh9vzyV3LfLjtxw667iQCVtH55fka8Amm9gLgZ+mczn+fzruYiwdZM3AM8HvATjp3YJw+DfXfBGwDvtJ+ERfN4vpfR2fo4CvAfe1x3hw7BiPtw5w4DsA/BL7c6nwA+PVB/+4Oon6/ukKSBDhkJElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKn5/5/m7Y84t1MLAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE: 4007085.29\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEICAYAAABBBrPDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAYMklEQVR4nO3df7DldX3f8ecrrBKigvy40nUXs6hICky7ypbS+qO0RFmIEcxAXZoKMWRWCUx1knQCsTMynTKDqYaUSSWDgQGs8iOgZSdAdaNRxxbBi0F+Slx+JFxZYQ2Im6g0C+/+cT43Pbuc/ezuvefuPRuej5kz93ve3+/ne97ne8/e135/nHNSVUiStD0/tdgNSJImm0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KagyS/nOQLnflfTvJrY3ic45LMzHc90nwYFHpRSPJokh8n+Zsk30tyZZKXz3V9VfXpqnrHOHuUJpVBoReTX6yqlwMrgTcC5y9uO9KewaDQi05VfQ/4PIPAIMmxSf5Pkh8k+VaS42aXTfIrSR5OsjnJI0l+eaj+taHl3p7k20meSfIHQIbmXZDkfwzdX5Gkkixp99+X5IH2GA8nef/2ek/y20m+25Z9MMnx49ou0vYYFHrRSbIcOBHYkGQZcDPwX4ADgN8CbkwyleRlwCXAiVX1CuBfAneNWN9BwI3AfwIOAh4C3rwLLT0JvBPYF3gfcHGSN414nMOBc4F/1vo5AXh0Fx5HmhODQi8m/zPJZuAxBn+cPwL8e+CWqrqlqp6vqvXANHBSG/M8cFSSfapqY1XdN2K9JwH3V9UNVfV3wO8D39vZpqrq5qp6qAa+AnwBeOuIRZ8D9gaOSPKSqnq0qh7a2ceR5sqg0IvJKe1/4scBP8fgf/8/C5zWDjv9IMkPgLcAS6vqb4H3AB8ANia5OcnPjVjvqxmEDwA1+KTNx0YsN1KSE5N8PclT7fFPar1tpao2AB8CLgCeTHJtklfv7ONIc2VQ6EWn/a/9SuBjDP6gf6qqXjl0e1lVXdSW/XxVvR1YCnwb+OSIVW4EDpm9kyTD94G/BX5m6P4/Glp2bwaHrT4GHFxVrwRuYegcxza9f6aq3sIg4Ar46C48dWlODAq9WP0+8Hbga8AvJjkhyV5Jfrq9d2F5koOTvKudq3gW+BsGh3+2dTNwZJJfaieo/wNDYcDgvMbbkrwmyX5sfbXVSxkcTtoEbElyIjDystskhyf5Ny1cfgL8eDv9SGNlUOhFqao2AVczOJRzMvA7DP5YPwb8Rwb/Nn4K+E3gceAp4F8Bvz5iXd8HTgMuAv4aOAz430Pz1wPXAXcDdwJ/MjRvM4NguR54Gvh3wLrttL13e4zvMzgH8qrWt7Sg4hcXSZJ63KOQJHUZFJKkLoNCktRlUEiSupYsdgNzddBBB9WKFSsWuw1J2qPceeed36+qqV0Zs8cGxYoVK5ienl7sNiRpj5LkL3d1jIeeJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXTt8Z3aSK4B3Ak9W1VGtdh1weFvklcAPqmplkhXAA8CDbd7Xq+oDbczRDL5+ch8GX/X4waqq9m1dVwNHM/jSl/dU1aPjeHJaOCvOu3mnl330ol9YwE4kLbSd2aO4Elg9XKiq91TVyqpayeD7fj87NPuh2XmzIdFcCqxl8O1fhw2t8yzg6ap6PXAxfgewJE2UHQZFVX2VwddAvkD7Evl/C1zTW0eSpcC+VXVbDb5S72rglDb7ZOCqNn0DcHxbryRpAsz3HMVbgSeq6jtDtUOT/HmSryR5a6stA2aGlplptdl5jwFU1RbgGeDAUQ+WZG2S6STTmzZtmmfrkqSdMd+gOJ2t9yY2Aq+pqjcCvwF8Jsm+wKg9hNkv6+7N27pYdVlVraqqVVNTu/QpuZKkOZrzx4wnWQL8EoOT0ABU1bPAs236ziQPAW9gsAexfGj4cuDxNj0DHALMtHXux3YOdUmSdr/57FH8PPDtqvr7Q0pJppLs1aZfy+Ck9cNVtRHYnOTYdv7hDOCmNmwdcGabPhX4UjuPIUmaADsMiiTXALcBhyeZSXJWm7WGF57Efhtwd5JvMTgx/YGqmt07OBv4I2AD8BBwa6tfDhyYZAODw1XnzeP5SJLGbIeHnqrq9O3Uf2VE7UYGl8uOWn4aOGpE/SfAaTvqQ5K0OHxntiSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1LXDoEhyRZInk9w7VLsgyXeT3NVuJw3NOz/JhiQPJjlhqH50knvavEuSpNX3TnJdq9+eZMWYn6MkaR52Zo/iSmD1iPrFVbWy3W4BSHIEsAY4so35RJK92vKXAmuBw9ptdp1nAU9X1euBi4GPzvG5SJIWwA6Doqq+Cjy1k+s7Gbi2qp6tqkeADcAxSZYC+1bVbVVVwNXAKUNjrmrTNwDHz+5tSJIW33zOUZyb5O52aGr/VlsGPDa0zEyrLWvT29a3GlNVW4BngANHPWCStUmmk0xv2rRpHq1LknbWXIPiUuB1wEpgI/DxVh+1J1Cdem/MC4tVl1XVqqpaNTU1tUsNS5LmZk5BUVVPVNVzVfU88EngmDZrBjhkaNHlwOOtvnxEfasxSZYA+7Hzh7okSQtsTkHRzjnMejcwe0XUOmBNu5LpUAYnre+oqo3A5iTHtvMPZwA3DY05s02fCnypnceQJE2AJTtaIMk1wHHAQUlmgI8AxyVZyeAQ0aPA+wGq6r4k1wP3A1uAc6rqubaqsxlcQbUPcGu7AVwOfCrJBgZ7EmvG8LwkSWOyw6CoqtNHlC/vLH8hcOGI+jRw1Ij6T4DTdtSHJGlx+M5sSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUpdBIUnqMigkSV0GhSSpa4dBkeSKJE8muXeo9l+TfDvJ3Uk+l+SVrb4iyY+T3NVufzg05ugk9yTZkOSSJGn1vZNc1+q3J1kx/qcpSZqrndmjuBJYvU1tPXBUVf0T4C+A84fmPVRVK9vtA0P1S4G1wGHtNrvOs4Cnq+r1wMXAR3f5WUiSFswOg6Kqvgo8tU3tC1W1pd39OrC8t44kS4F9q+q2qirgauCUNvtk4Ko2fQNw/OzehiRp8Y3jHMWvArcO3T80yZ8n+UqSt7baMmBmaJmZVpud9xhAC59ngANHPVCStUmmk0xv2rRpDK1LknZkXkGR5MPAFuDTrbQReE1VvRH4DeAzSfYFRu0h1OxqOvO2LlZdVlWrqmrV1NTUfFqXJO2kJXMdmORM4J3A8e1wElX1LPBsm74zyUPAGxjsQQwfnloOPN6mZ4BDgJkkS4D92OZQlyRp8cxpjyLJauC3gXdV1Y+G6lNJ9mrTr2Vw0vrhqtoIbE5ybDv/cAZwUxu2DjizTZ8KfGk2eCRJi2+HexRJrgGOAw5KMgN8hMFVTnsD69t556+3K5zeBvznJFuA54APVNXs3sHZDK6g2ofBOY3Z8xqXA59KsoHBnsSasTwzSdJY7DAoqur0EeXLt7PsjcCN25k3DRw1ov4T4LQd9SFJWhy+M1uS1GVQSJK6DApJUpdBIUnqMigkSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeraYVAkuSLJk0nuHaodkGR9ku+0n/sPzTs/yYYkDyY5Yah+dJJ72rxLkqTV905yXavfnmTFmJ+jJGkedmaP4kpg9Ta184AvVtVhwBfbfZIcAawBjmxjPpFkrzbmUmAtcFi7za7zLODpqno9cDHw0bk+GUnS+O0wKKrqq8BT25RPBq5q01cBpwzVr62qZ6vqEWADcEySpcC+VXVbVRVw9TZjZtd1A3D87N6GJGnxzfUcxcFVtRGg/XxVqy8DHhtabqbVlrXpbetbjamqLcAzwIGjHjTJ2iTTSaY3bdo0x9YlSbti3CezR+0JVKfeG/PCYtVlVbWqqlZNTU3NsUVJ0q6Ya1A80Q4n0X4+2eozwCFDyy0HHm/15SPqW41JsgTYjxce6pIkLZK5BsU64Mw2fSZw01B9TbuS6VAGJ63vaIenNic5tp1/OGObMbPrOhX4UjuPIUmaAEt2tECSa4DjgIOSzAAfAS4Crk9yFvBXwGkAVXVfkuuB+4EtwDlV9Vxb1dkMrqDaB7i13QAuBz6VZAODPYk1Y3lmkqSx2GFQVNXp25l1/HaWvxC4cER9GjhqRP0ntKCRJE0e35ktSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUpdBIUnqMigkSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1zTkokhye5K6h2w+TfCjJBUm+O1Q/aWjM+Uk2JHkwyQlD9aOT3NPmXZIk831ikqTxmHNQVNWDVbWyqlYCRwM/Aj7XZl88O6+qbgFIcgSwBjgSWA18IslebflLgbXAYe22eq59SZLGa1yHno4HHqqqv+wsczJwbVU9W1WPABuAY5IsBfatqtuqqoCrgVPG1JckaZ7GFRRrgGuG7p+b5O4kVyTZv9WWAY8NLTPTasva9Lb1F0iyNsl0kulNmzaNqXVJUs+8gyLJS4F3AX/cSpcCrwNWAhuBj88uOmJ4deovLFZdVlWrqmrV1NTUfNqWJO2kcexRnAh8s6qeAKiqJ6rquap6HvgkcExbbgY4ZGjccuDxVl8+oi5JmgDjCIrTGTrs1M45zHo3cG+bXgesSbJ3kkMZnLS+o6o2ApuTHNuudjoDuGkMfUmSxmDJfAYn+Rng7cD7h8q/m2Qlg8NHj87Oq6r7klwP3A9sAc6pqufamLOBK4F9gFvbTZI0AeYVFFX1I+DAbWrv7Sx/IXDhiPo0cNR8epEkLQzfmS1J6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUpdBIUnqMigkSV0GhSSpy6CQJHXNKyiSPJrkniR3JZlutQOSrE/ynfZz/6Hlz0+yIcmDSU4Yqh/d1rMhySVJMp++JEnjM449in9dVSuralW7fx7wxao6DPhiu0+SI4A1wJHAauATSfZqYy4F1gKHtdvqMfQlSRqDhTj0dDJwVZu+CjhlqH5tVT1bVY8AG4BjkiwF9q2q26qqgKuHxkiSFtl8g6KALyS5M8naVju4qjYCtJ+vavVlwGNDY2dabVmb3rYuSZoAS+Y5/s1V9XiSVwHrk3y7s+yo8w7Vqb9wBYMwWgvwmte8Zld7lSTNwbz2KKrq8fbzSeBzwDHAE+1wEu3nk23xGeCQoeHLgcdbffmI+qjHu6yqVlXVqqmpqfm0LknaSXMOiiQvS/KK2WngHcC9wDrgzLbYmcBNbXodsCbJ3kkOZXDS+o52eGpzkmPb1U5nDI2RJC2y+Rx6Ohj4XLuSdQnwmar6X0m+AVyf5Czgr4DTAKrqviTXA/cDW4Bzquq5tq6zgSuBfYBb202SNAHmHBRV9TDwT0fU/xo4fjtjLgQuHFGfBo6aay+SpIXjO7MlSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqSuOQdFkkOS/FmSB5Lcl+SDrX5Bku8muavdThoac36SDUkeTHLCUP3oJPe0eZckyfyeliRpXJbMY+wW4Der6ptJXgHcmWR9m3dxVX1seOEkRwBrgCOBVwN/muQNVfUccCmwFvg6cAuwGrh1Hr1JksZkznsUVbWxqr7ZpjcDDwDLOkNOBq6tqmer6hFgA3BMkqXAvlV1W1UVcDVwylz7kiSN11jOUSRZAbwRuL2Vzk1yd5IrkuzfasuAx4aGzbTasja9bV2SNAHmHRRJXg7cCHyoqn7I4DDS64CVwEbg47OLjhhenfqox1qbZDrJ9KZNm+bbuiRpJ8wrKJK8hEFIfLqqPgtQVU9U1XNV9TzwSeCYtvgMcMjQ8OXA462+fET9BarqsqpaVVWrpqam5tO6JGknzeeqpwCXAw9U1e8N1ZcOLfZu4N42vQ5Yk2TvJIcChwF3VNVGYHOSY9s6zwBummtfkqTxms9VT28G3gvck+SuVvsd4PQkKxkcPnoUeD9AVd2X5HrgfgZXTJ3TrngCOBu4EtiHwdVOXvEkSRNizkFRVV9j9PmFWzpjLgQuHFGfBo6aay+SpIXjO7MlSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpK75fHqs/gFacd7Ni92CpAnjHoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXRPzhrskq4H/BuwF/FFVXbTILWlMdvZNfI9e9AsL3ImkuZiIoEiyF/DfgbcDM8A3kqyrqvsXtzPtTgaKNJkmIiiAY4ANVfUwQJJrgZOBRQ2KXfk4C/947T6T/jEju/JaGHc4GrZ7tkn9/aWqdusDjmwiORVYXVW/1u6/F/jnVXXuNsutBda2u4cDD3ZWexDw/QVodyHtiT3Dntm3Pe8+e2Lfe2LPsHN9/2xVTe3KSidljyIjai9IsKq6DLhsp1aYTFfVqvk2tjvtiT3Dntm3Pe8+e2Lfe2LPsHB9T8pVTzPAIUP3lwOPL1IvkqQhkxIU3wAOS3JokpcCa4B1i9yTJIkJOfRUVVuSnAt8nsHlsVdU1X3zXO1OHaKaMHtiz7Bn9m3Pu8+e2Pee2DMsUN8TcTJbkjS5JuXQkyRpQhkUkqSuPSYokpyW5L4kzydZtc2885NsSPJgkhOG6kcnuafNuyRJWn3vJNe1+u1JVgyNOTPJd9rtzDH2f0GS7ya5q91OWoj+d6ckq1vPG5Kctxg9bNPPo2173ZVkutUOSLK+/T7XJ9l/aPld2u5j7POKJE8muXeoNrY+F+L1sZ2eJ/o1neSQJH+W5IH2t+ODrT6x27rT8+Ju66raI27AP2bwJrsvA6uG6kcA3wL2Bg4FHgL2avPuAP4Fg/dp3Aqc2Oq/Dvxhm14DXNemDwAebj/3b9P7j6n/C4DfGlEfW/+7+fexV+v1tcBL23M4YpFfI48CB21T+13gvDZ9HvDRuW73Mfb5NuBNwL0L0edCvD620/NEv6aBpcCb2vQrgL9ovU3stu70vKjbeo/Zo6iqB6pq1DuxTwaurapnq+oRYANwTJKlwL5VdVsNtsjVwClDY65q0zcAx7e0PQFYX1VPVdXTwHpg9cI9q7H3vzv9/ceuVNX/BWY/dmXSDG+rq9h6G+7qdh+Lqvoq8NQC9jn218d2et6eSel5Y1V9s01vBh4AljHB27rT8/bslp73mKDoWAY8NnR/ptWWtelt61uNqaotwDPAgZ11jcu5Se5uu/Gzu7vj7H93WuhtNRcFfCHJnRl83AvAwVW1EQb/CIFXtfpctvtCGmefu/P1sUe8ptvhlTcCt7OHbOtteoZF3NYTFRRJ/jTJvSNuvf+pbu/jP3ofCzKXMTu0g/4vBV4HrAQ2Ah9fgP53p0noYVtvrqo3AScC5yR5W2fZBXkNLIBJfn3sEa/pJC8HbgQ+VFU/7C26nR52e98jel7UbT0Rb7ibVVU/P4dh2/v4j5k2vW19eMxMkiXAfgx2q2eA47YZ8+WdbWRn+0/ySeBPFqD/3WniPnalqh5vP59M8jkGh8eeSLK0qja23fEn2+Jz2e4LaZx97pbXR1U9MTs9qa/pJC9h8Af301X12Vae6G09qufF3tYTtUcxR+uANe1M/qHAYcAdbZdyc5Jj2/G3M4CbhsbMXtF0KvCldhzv88A7kuzfdu3e0Wrz1l6Qs94NzF49Ms7+d6eJ+tiVJC9L8orZaQa/u3vZeludydbbcFe3+0IaZ5+75fUx6a/p9hiXAw9U1e8NzZrYbb29nhd9W8/nDP3uvLWNMwM8CzwBfH5o3ocZnO1/kKErVIBVbYM+BPwB//+d6D8N/DGDEz93AK8dGvOrrb4BeN8Y+/8UcA9wd/tFLV2I/nfz7+QkBldlPAR8eJFfH69lcPXHt4D7ZvthcOz1i8B32s8D5rrdx9jrNQwOH/xde02fNc4+F+L1sZ2eJ/o1DbyFwSGVu4G72u2kSd7WnZ4XdVv7ER6SpK5/CIeeJEkLyKCQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6vp/NYXB2Ke/GR4AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "results = model.predict(df_badt.drop(['mangaid', 'teff', 'teff_ext'], axis=1))\n",
    "\n",
    "target = df_badt['teff_ext']\n",
    "plt.hist(target, bins=30, color='orange')\n",
    "plt.title('Distribution of TEFF_MED')\n",
    "plt.show()\n",
    "\n",
    "mse = mean_squared_error(target, results)\n",
    "print('MSE: %.2f' % mse)\n",
    "\n",
    "residuals = np.array(target).astype('float') - results\n",
    "plt.title('Residuals')\n",
    "plt.hist(residuals, bins=30)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This result is pretty bad, when compared to our previous test MSE. Looking at the distribution of TEFF_MED we can see it contains values that are outside of the range seen by our fitting procedure! There is no way our model can handle these properly, and I think it would be valid in this case to have it respond with an \"unknown\" result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model range: (2970.2K - 8975.9K)\n",
      "Stars evaluated only by external algorithm, within our model's range: 19715\n",
      "Stars outside of range: 938\n"
     ]
    }
   ],
   "source": [
    "goodt = sl.get_goodt()\n",
    "max_seen = goodt['INPUT_TEFF'].max()\n",
    "min_seen = goodt['INPUT_TEFF'].min()\n",
    "print('Model range: (%.1fK - %.1fK)' %(min_seen, max_seen))\n",
    "\n",
    "badt_lim = sl.get_badt_lim()\n",
    "df_badt_lim = sl.fits_to_pd(badt_lim)\n",
    "\n",
    "print('Stars evaluated only by external algorithm, within our model\\'s range: %d' % len(df_badt))\n",
    "print('Stars outside of range: %d' % (len(df_badt)-len(df_badt_lim)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's drop the 824 out-of-range stars from the evaluation dataset, and retry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAar0lEQVR4nO3df7RdZZ3f8ffHiDGIEZhcaEiiiTa6DLRGSdNMmTqM6BAZNdi1GIM/iFOmcSjM6KjLIf4YcWbSovXHDLXQRqUEFGPG0SHDAjXgsNQWiRcMhAAp0cRwTUwuWDRQS0349I/9XNlezr333HtPTnKyP6+1zjr7fPfz7P0898f37PPs5+wt20RERDM843A3ICIiuidJPyKiQZL0IyIaJEk/IqJBkvQjIhokST8iokGS9CMiGiRJPyZE0n+V9KEObev5kh6TNKW8vk3SH3Zi22V7N0ta0antjWO/fyXpYUk/6fa+I0aSpB9PI2mnpF9I2i/pUUn/U9IfSfrV34vtP7L9l21u69WjlbG9y/Zxtg92oO2XSfr8sO2/1vbayW57nO2YA7wHWGD7nwxb95byJvdY+Tk/WXv9WCkz9Dt4rPb4dFn3dkkHR1h3jaT/N2zdm8Zo685SZ8aw+GZJljR3lG3fXdbNLWWH4nsl3SjpNR36kUaHJOnHSF5v+7nAC4DLgT8DPtfpnUh6Zqe3eYR4AfCI7X3DV9j+QnmTOw54LbB76HWJDXl9PW77ktq620dZ97Fh677URnt3AOcPvZD0z4BpLcoN3/bLhq0/vvThZcBG4KuS3t7G/qNLkvRjVLZ/ZnsD8CZghaTT4FdHfX9VlmeUo7pHJf1U0rclPUPSdcDzgX8oR3/vqx0RXihpF/DNWqz+BvAiSZsk/UzSDZJOLPs6U9JAvY1DnyYkLQXeD7xp2FHor4aLSrs+KOlHkvZJulbS88q6oXaskLSrDM18YKSfjaTnlfqDZXsfLNt/NVXCO6W045oO/CoOteuAC2qvVwDXTnRjtn9i+2+Ay4CP1j8lxuGVX0S0xfYmYAD41y1Wv6es6wNOpkq8tv02YBdPHbF+rFbnt4GXAmePsMsLgH8LnAIcAK5oo41fA/4D8KURjkIB3l4evwO8EDgO+PSwMr8FvAQ4C/hzSS8dYZf/GXhe2c5vlzb/ge1b+PUj+LeP1fYjwHeB6ZJeWs6tvAn4/Bh12vEV4CSqn2ccAZL0Yzx2Aye2iP8SmAm8wPYvbX/bY1/J7zLbj9v+xQjrr7N9r+3HgQ8Bv1+S0WS9Bfik7R/afgxYBSwf9injI7Z/Yftu4G6qoYpfU0uMq2zvt70T+ATwtg60ccjfl09PQ49/V1u3ZNi6JbV1763FHx7H/oaO9l8DPAD8uEWZ9w7b71jnSnaX51Z/N3EYHK3jqXFozAJ+2iL+n6g+xn9DEsAa25ePsa2HxrH+R8AxwIwRyo7HKWV79W0/k+oTypD6bJv/Q/VpYLgZwLNabGtWB9o45NzyqaGV79r+rRHWfdz2Byewv+uAbwHzGHloZ7zbHvp5tPq7icMgR/rRFkn/guof+DvD15Uj3ffYfiHweuDdks4aWj3CJsf6JDCntvx8qk8TDwOPA8fW2jWFalip3e3upjrJWt/2AWDvGPWGe7i0afi2Wh0d9wTbP6I6oXsO1bBMJ7wR2Ads69D2YpKS9GNUkqZLeh2wDvi87S0tyrxO0j9VdZj/c+BgeUCVTF84gV2/VdICSccCfwF8uUzp/F/AsyX9nqRjgA8CU2v19gJzRzlx+EXgTyXNk3QcT50DODCexpW2rAdWS3qupBcA76Yz4+CH04XAq8qw2oRJOlnSJcCHqYbAnuxI62LSkvRjJP8gaT/VMMsHgE8CfzBC2fnALcBjwO3AlbZvK+v+I/DBMv773nHs/zrgGqqhlmcDfwLVbCLg3wOfpTqqfpzqJPKQvy3Pj0i6q8V2r+apYYwdwP8F/ngc7ar747L/H1J9Arq+bL9ThmY9DT2+2sFtt2T7B7b7RynyvmFtGn7O4FFJjwNbqD4xnGe7kz+TmCTlzlkREc2RI/2IiAbJ7J2Io5yk5wP3jbB6ge1d3WxPHF4Z3omIaJAj/kh/xowZnjt37uFuRkRET7nzzjsftt03PH7EJ/25c+fS3z/aZIKIiBhO0o9axXMiNyKiQZL0IyIaJEk/IqJBkvQjIhokST8iokGS9CMiGiRJPyKiQZL0IyIapO2kL2mKpO9LurG8PlHSRkkPlucTamVXSdouaZuks2vx0yVtKeuuKNdfj4iILhnPN3LfCdwPTC+vLwVutX25pEvL6z+TtABYDpxKdWu6WyS9uNx04ipgJdVNmG8ClgI3d6QnEd12fZvHLG/O9a3iyNHWkb6k2cDvUd24YsgyYOimyGuBc2vxdbafsL0D2A4sljQTmG779nLT7GtrdSIiogvaHd75a+B9QP2WZyfb3gNQnk8q8Vn8+k2tB0psFr9+h6Oh+NNIWimpX1L/4OBgm02MiIixjJn0y/1R99m+s81ttvrM61HiTw/aa2wvsr2or+9pF4mLiIgJamdM/wzgDZLOobpX6XRJnwf2Spppe08ZutlXyg8Ac2r1ZwO7S3x2i3hERHTJmEf6tlfZnm17LtUJ2m/afiuwAVhRiq0AbijLG4DlkqZKmkd10+xNZQhov6QlZdbOBbU6ERHRBZO5nv7lwHpJFwK7gPMAbG+VtJ7q9mwHgIvLzB2Ai4BrgGlUs3YycycioovGlfRt3wbcVpYfAc4aodxqYHWLeD9w2ngbGRERnZFv5EZENEiSfkREgyTpR0Q0SJJ+RESDJOlHRDRIkn5ERIMk6UdENEiSfkREgyTpR0Q0SJJ+RESDJOlHRDRIkn5ERIMk6UdENEiSfkREgyTpR0Q0SJJ+RESDtHNj9GdL2iTpbklbJX2kxC+T9GNJm8vjnFqdVZK2S9om6exa/HRJW8q6K8ptEyMiokvauXPWE8CrbD8m6RjgO5KGbnP4KdsfrxeWtIDqXrqnAqcAt0h6cbll4lXASuC7wE3AUnLLxIiIrmnnxui2/Vh5eUx5eJQqy4B1tp+wvQPYDiyWNBOYbvt22wauBc6dVOsjImJc2hrTlzRF0mZgH7DR9h1l1SWS7pF0taQTSmwW8FCt+kCJzSrLw+MREdElbSV92wdtLwRmUx21n0Y1VPMiYCGwB/hEKd5qnN6jxJ9G0kpJ/ZL6BwcH22liRES0YVyzd2w/CtwGLLW9t7wZPAl8Blhcig0Ac2rVZgO7S3x2i3ir/ayxvcj2or6+vvE0MSIiRtHO7J0+SceX5WnAq4EHyhj9kDcC95blDcBySVMlzQPmA5ts7wH2S1pSZu1cANzQua5ERMRY2pm9MxNYK2kK1ZvEets3SrpO0kKqIZqdwDsAbG+VtB64DzgAXFxm7gBcBFwDTKOatZOZOxERXTRm0rd9D/DyFvG3jVJnNbC6RbwfOG2cbYyIiA7JN3IjIhokST8iokGS9CMiGiRJPyKiQZL0IyIaJEk/IqJBkvQjIhokST8iokGS9CMiGiRJPyKiQZL0IyIaJEk/IqJBkvQjIhokST8iokGS9CMiGiRJPyKiQZL0IyIaZMw7Z0l6NvAtYGop/2XbH5Z0IvAlYC7V7RJ/3/b/LnVWARcCB4E/sf31Ej+dp26XeBPwTtvubJcijjDXq71yb86/Qhx67RzpPwG8yvbLgIXAUklLgEuBW23PB24tr5G0AFgOnAosBa4s99cFuApYSXWz9PllfUREdMmYSd+Vx8rLY8rDwDJgbYmvBc4ty8uAdbafsL0D2A4sljQTmG779nJ0f22tTkREdEFbY/qSpkjaDOwDNtq+AzjZ9h6A8nxSKT4LeKhWfaDEZpXl4fFW+1spqV9S/+Dg4Di6ExERo2kr6ds+aHshMJvqqP20UYq3GsD0KPFW+1tje5HtRX19fe00MSIi2jCu2Tu2HwVuoxqL31uGbCjP+0qxAWBOrdpsYHeJz24Rj4iILhkz6Uvqk3R8WZ4GvBp4ANgArCjFVgA3lOUNwHJJUyXNozphu6kMAe2XtESSgAtqdSIiogvGnLIJzATWlhk4zwDW275R0u3AekkXAruA8wBsb5W0HrgPOABcbPtg2dZFPDVl8+byiIiILhkz6du+B3h5i/gjwFkj1FkNrG4R7wdGOx8QERGHUL6RGxHRIEn6ERENkqQfEdEgSfoREQ2SpB8R0SBJ+hERDdLOPP2IZmn3UsgRPShH+hERDZKkHxHRIEn6ERENkqQfEdEgSfoREQ2SpB8R0SBJ+hERDZKkHxHRIEn6ERENkqQfEdEg7dwjd46kf5R0v6Stkt5Z4pdJ+rGkzeVxTq3OKknbJW2TdHYtfrqkLWXdFeVeuRER0SXtXHvnAPAe23dJei5wp6SNZd2nbH+8XljSAmA5cCpwCnCLpBeX++ReBawEvgvcBCwl98mNiOiaMY/0be+xfVdZ3g/cD8wapcoyYJ3tJ2zvALYDiyXNBKbbvt22gWuBcyfbgYiIaN+4xvQlzaW6SfodJXSJpHskXS3phBKbBTxUqzZQYrPK8vB4q/2slNQvqX9wcHA8TYyIiFG0nfQlHQf8HfAu2z+nGqp5EbAQ2AN8Yqhoi+oeJf70oL3G9iLbi/r6+tptYkREjKGtpC/pGKqE/wXbXwGwvdf2QdtPAp8BFpfiA8CcWvXZwO4Sn90iHhERXdLO7B0BnwPut/3JWnxmrdgbgXvL8gZguaSpkuYB84FNtvcA+yUtKdu8ALihQ/2IiIg2tDN75wzgbcAWSZtL7P3A+ZIWUg3R7ATeAWB7q6T1wH1UM38uLjN3AC4CrgGmUc3aycydiIguGjPp2/4OrcfjbxqlzmpgdYt4P3DaeBoYERGdk2/kRkQ0SJJ+RESDJOlHRDRIkn5ERIMk6UdENEiSfkREgyTpR0Q0SJJ+RESDJOlHRDRIkn5ERIMk6UdENEiSfkREgyTpR0Q0SJJ+RESDJOlHRDRIkn5ERIO0c7vEOZL+UdL9krZKemeJnyhpo6QHy/MJtTqrJG2XtE3S2bX46ZK2lHVXlNsmRkREl7RzpH8AeI/tlwJLgIslLQAuBW61PR+4tbymrFsOnAosBa6UNKVs6ypgJdV9c+eX9RER0SVjJn3be2zfVZb3A/cDs4BlwNpSbC1wblleBqyz/YTtHcB2YHG5kfp027fbNnBtrU5ERHTBuMb0Jc0FXg7cAZxsew9UbwzASaXYLOChWrWBEptVlofHW+1npaR+Sf2Dg4PjaWJERIyi7aQv6Tjg74B32f75aEVbxDxK/OlBe43tRbYX9fX1tdvEiIgYQ1tJX9IxVAn/C7a/UsJ7y5AN5XlfiQ8Ac2rVZwO7S3x2i3hERHRJO7N3BHwOuN/2J2urNgAryvIK4IZafLmkqZLmUZ2w3VSGgPZLWlK2eUGtTkREdMEz2yhzBvA2YIukzSX2fuByYL2kC4FdwHkAtrdKWg/cRzXz52LbB0u9i4BrgGnAzeURERFdMmbSt/0dWo/HA5w1Qp3VwOoW8X7gtPE0MCIiOiffyI2IaJAk/YiIBknSj4hokCT9iIgGSdKPiGiQJP2IiAZJ0o+IaJAk/YiIBknSj4hokCT9iIgGSdKPiGiQJP2IiAZJ0o+IaJAk/YiIBknSj4hokCT9iIgGSdKPiGiQdu6Re7WkfZLurcUuk/RjSZvL45zaulWStkvaJunsWvx0SVvKuivKfXIjIqKL2jnSvwZY2iL+KdsLy+MmAEkLgOXAqaXOlZKmlPJXASupbpQ+f4RtRkTEITRm0rf9LeCnbW5vGbDO9hO2dwDbgcWSZgLTbd9u28C1wLkTbHNEREzQZMb0L5F0Txn+OaHEZgEP1coMlNissjw83pKklZL6JfUPDg5OookREVE30aR/FfAiYCGwB/hEibcap/co8ZZsr7G9yPaivr6+CTYxIiKGm1DSt73X9kHbTwKfARaXVQPAnFrR2cDuEp/dIh4REV00oaRfxuiHvBEYmtmzAVguaaqkeVQnbDfZ3gPsl7SkzNq5ALhhEu2OiIgJeOZYBSR9ETgTmCFpAPgwcKakhVRDNDuBdwDY3ippPXAfcAC42PbBsqmLqGYCTQNuLo+IiOiiMZO+7fNbhD83SvnVwOoW8X7gtHG1LiIiOirfyI2IaJAk/YiIBhlzeCciuuT6cVyZ5M0jzniOGFWO9CMiGiRJPyKiQZL0IyIaJEk/IqJBkvQjIhoks3cOhczCiIgjVI70IyIaJEk/IqJBkvQjIhokST8iokFyIjeaYzwn2COOUjnSj4hokCT9iIgGGTPpS7pa0j5J99ZiJ0raKOnB8nxCbd0qSdslbZN0di1+uqQtZd0V5baJERHRRe0c6V8DLB0WuxS41fZ84NbyGkkLgOXAqaXOlZKmlDpXASup7ps7v8U2IyLiEBsz6dv+FvDTYeFlwNqyvBY4txZfZ/sJ2zuA7cDiciP16bZvt23g2lqdiIjokomO6Z9sew9AeT6pxGcBD9XKDZTYrLI8PB4REV3U6RO5rcbpPUq89UaklZL6JfUPDg52rHEREU030aS/twzZUJ73lfgAMKdWbjawu8Rnt4i3ZHuN7UW2F/X19U2wiRERMdxEk/4GYEVZXgHcUIsvlzRV0jyqE7abyhDQfklLyqydC2p1IiKiS8b8Rq6kLwJnAjMkDQAfBi4H1ku6ENgFnAdge6uk9cB9wAHgYtsHy6YuopoJNA24uTwiIqKLxkz6ts8fYdVZI5RfDaxuEe8HThtX6yIioqPyjdyIiAZJ0o+IaJBcZXM8cpXGiOhxOdKPiGiQJP2IiAZJ0o+IaJAk/YiIBknSj4hokMzeOdzanRH05hGvTxcR0bYc6UdENEiSfkREgyTpR0Q0SJJ+RESDJOlHRDRIkn5ERIMk6UdENEiSfkREg0wq6UvaKWmLpM2S+kvsREkbJT1Ynk+olV8labukbZLOnmzjIyJifDpxpP87thfaXlReXwrcans+cGt5jaQFwHLgVGApcKWkKR3Yf0REtOlQDO8sA9aW5bXAubX4OttP2N4BbAcWH4L9R0TECCab9A18Q9KdklaW2Mm29wCU55NKfBbwUK3uQIk9jaSVkvol9Q8ODk6yiRERMWSyF1w7w/ZuSScBGyU9MErZVlcWa3kVMdtrgDUAixYtypXGIiI6ZFJH+rZ3l+d9wFephmv2SpoJUJ73leIDwJxa9dnA7snsPyIixmfCR/qSngM8w/b+svy7wF8AG4AVwOXl+YZSZQNwvaRPAqcA84FNk2h7RHPlktwxQZMZ3jkZ+Kqkoe1cb/trkr4HrJd0IbALOA/A9lZJ64H7gAPAxbYPTqr1ERExLhNO+rZ/CLysRfwR4KwR6qwGVk90nxERMTn5Rm5ERIMk6UdENEiSfkREgyTpR0Q0yGS/nBXdkil6I2v3ZxMROdKPiGiSHOkfbfKJICJGkSP9iIgGyZF+U+UTQUQjJenH6PLm0Nvy+4thkvSjM5JcInpCkn5013imV+YNonvypt0YSfpx5Mr8+4iOS9KHJJeIduUTQc9L0o+IwydvIl2XpB8RndfpT8+H89P4UfaGk6QfETGao+zTSNeTvqSlwN8AU4DP2r78kO0sY/UREb+mq0lf0hTgvwCvAQaA70naYPu+brYjIqLjeuQTQbeP9BcD28v9dZG0DlhGdbP0iIij32F+c+h20p8FPFR7PQD8y+GFJK0EVpaXj0naNol9zgAenkT9I0n6cmQ6WvpytPQDjoa+vOVXbw4T7csLWgW7nfRbvcU97e3M9hpgTUd2KPXbXtSJbR1u6cuR6Wjpy9HSD0hfRtPtSysPAHNqr2cDu7vchoiIxup20v8eMF/SPEnPApYDG7rchoiIxurq8I7tA5IuAb5ONWXzattbD/FuOzJMdIRIX45MR0tfjpZ+QPoyItm98YWCiIiYvNwuMSKiQZL0IyIapOeSvqRnS9ok6W5JWyV9pMRPlLRR0oPl+YRanVWStkvaJunsWvx0SVvKuiskHZbrNkiaIun7km7s5b5I2lnasFlSf4/35XhJX5b0gKT7Jf1mr/VF0kvK72Lo8XNJ7+q1ftTa8Kflf/5eSV8suaBX+/LO0o+tkt5VYt3pi+2eelDN9T+uLB8D3AEsAT4GXFrilwIfLcsLgLuBqcA84AfAlLJuE/CbZZs3A689TH16N3A9cGN53ZN9AXYCM4bFerUva4E/LMvPAo7v1b6UdkwBfkL1hZ2e6wfVFzt3ANPK6/XA23u0L6cB9wLHUk2muQWY362+dP2Pr8M/vGOBu6i+1bsNmFniM4FtZXkVsKpW5+vlhzQTeKAWPx/4b4ehD7OBW4FX8VTS79W+7OTpSb/n+gJMLwlGvd6X2r5/F/gfvdoPnvo2/4lUifLG0qde7Mt5VBebHHr9IeB93epLzw3vwK+GQzYD+4CNtu8ATra9B6A8n1SKt7r0w6zyGGgR77a/pvqFP1mL9WpfDHxD0p2qLqUBvdmXFwKDwH8vw26flfQcerMvQ5YDXyzLPdcP2z8GPg7sAvYAP7P9DXqwL1RH+a+U9BuSjgXOofrSalf60pNJ3/ZB2wupjpIXSzptlOIjXfqhrUtCHEqSXgfss31nu1VaxI6IvhRn2H4F8FrgYkmvHKXskdyXZwKvAK6y/XLgcaqP2yM5kvuCqi9CvgH427GKtogdEf0o49vLqIY3TgGeI+mto1VpETsi+mL7fuCjwEbga1RDNwdGqdLRvvRk0h9i+1HgNmApsFfSTIDyvK8UG+nSDwNleXi8m84A3iBpJ7AOeJWkz9ObfcH27vK8D/gq1VVVe7EvA8BA+QQJ8GWqN4Fe7AtUb8J32d5bXvdiP14N7LA9aPuXwFeAf0Vv9gXbn7P9CtuvBH4KPEiX+tJzSV9Sn6Tjy/I0qj+GB6gu57CiFFsB3FCWNwDLJU2VNI/qhMmm8vFpv6Ql5Yz3BbU6XWF7le3ZtudSffz+pu239mJfJD1H0nOHlqnGW++lB/ti+yfAQ5JeUkJnUV3+u+f6UpzPU0M70Jv92AUskXRsacNZwP30Zl+QdFJ5fj7wb6h+P93pSzdPYHToJMg/B74P3EOVVP68xH+D6oTog+X5xFqdD1Cd8d5G7ew2sKhs4wfApxl24q7L/TqTp07k9lxfqMbB7y6PrcAHerUvpQ0Lgf7yd/b3wAm92BeqyQ6PAM+rxXquH6UNH6E6wLsXuI5qNkuv9uXbVAcSdwNndfP3ksswREQ0SM8N70RExMQl6UdENEiSfkREgyTpR0Q0SJJ+RESDJOlHRDRIkn5ERIP8f69tl1CfZfGZAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE: 33418.23\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEICAYAAABfz4NwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAATyklEQVR4nO3dcZBd5X3e8e8TCQtsIhvCQmRJjeSpxg4wbWxUqsSuy1R2EeBYtBMmauOgeuioJaSx27SplHTqdKaawR1P4jApdKjtImLHWMVu0BhTW5VDMm4pymJwQMgqwlC0QUbrODiyJ1Es+PWP+9IcVncXae9q7130/cycuef+znnPfd+9Yp897zn3kqpCkqQfGHYHJEmjwUCQJAEGgiSpMRAkSYCBIElqDARJEmAgSLOS5GeSfGmG7fcn+cdz8DpXJJkY9DjSyTAQdEZI8nSSP0vy3STfTHJHknNne7yq+lRV/d257KM0bAaCziQ/WVXnAj8GvBXYNtzuSKPFQNAZp6q+CXyRXjCQZF2S/5Xk+SRfS3LFS/sm+UdJvpHkaJKnkvxMp/6Vzn7vTvL1JN9J8ptAOtt+NcknO89XJakki9vz9yfZ317jG0n+yXR9T/Kvk/xR2/dAkvVz9XORDASdcZKsAK4CDiZZDtwL/HvgfOBfAp9NMpbkdcAtwFVV9YPATwCP9DneBcBngX8DXAA8Cbz9FLp0BHgPsBR4P/DrSd7W53XeDPw88Ddaf64Enj6F15FmZCDoTPI7SY4Ch+j9Ev4Q8D7gC1X1hap6sap2A+PA1a3Ni8ClSc6pqsNVta/Pca8GHq+qu6vq+8BHgW+ebKeq6t6qerJ6fg/4EvC3+uz6ArAEuDjJWVX1dFU9ebKvI70SA0FnkmvbX9ZXAG+h99f8jwDXtemi55M8D7wDWFZV3wN+GvinwOEk9yZ5S5/jvpFeyABQvW+MPNRnv76SXJXkfyf5dnv9q1vfXqaqDgIfBH4VOJLkriRvPNnXkV6JgaAzTvsr/A7gI/R+cf9WVb2hs7yuqm5u+36xqt4NLAO+DvznPoc8DKx86UmSdJ8D3wNe23n+w519l9CbbvoIcFFVvQH4Ap1rEFP6/ttV9Q56QVbAh09h6NKMDASdqT4KvBv4CvCTSa5MsijJ2e3e/xVJLkry3nYt4RjwXXrTNlPdC1yS5O+3C8W/QOeXPr3rDu9M8leSvJ6X3930GnrTQJPA8SRXAX1vZ03y5iR/p4XInwN/Nk1/pFkxEHRGqqpJ4E56UzAbgV+m90v5EPCv6P238QPALwLPAt8G/jbwc32O9S3gOuBm4I+BNcD/7GzfDXwG+EPgIeDznW1H6QXITuBPgH8I7Jqm20vaa3yL3jWKC1u/pTkR/wc5kiTwDEGS1BgIkiTAQJAkNQaCJAmAxcPuwGxdcMEFtWrVqmF3Q5IWlIceeuhbVTXWb9uCDYRVq1YxPj4+7G5I0oKS5P9Ot80pI0kSYCBIkhoDQZIEGAiSpMZAkCQBBoIkqTEQJEmAgSBJagwESRKwgD+prOFatfXek9736ZuvOY09kTRXPEOQJAEGgiSpMRAkSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKkxECRJgIEgSWoMBEkScBKBkOQTSY4keaxTOz/J7iRPtMfzOtu2JTmY5ECSKzv1y5I82rbdkiStviTJZ1r9wSSr5niMkqSTcDJnCHcAG6bUtgJ7qmoNsKc9J8nFwCbgktbm1iSLWpvbgC3Amra8dMwbgD+pqr8K/Drw4dkORpI0e68YCFX1+8C3p5Q3Ajva+g7g2k79rqo6VlVPAQeBy5MsA5ZW1QNVVcCdU9q8dKy7gfUvnT1IkubPbK8hXFRVhwHa44Wtvhw41NlvotWWt/Wp9Ze1qarjwHeAH+r3okm2JBlPMj45OTnLrkuS+pnri8r9/rKvGeoztTmxWHV7Va2tqrVjY2Oz7KIkqZ/ZBsJzbRqI9nik1SeAlZ39VgDPtvqKPvWXtUmyGHg9J05RSZJOs9kGwi5gc1vfDNzTqW9qdw6tpnfxeG+bVjqaZF27PnD9lDYvHeungC+36wySpHm0+JV2SPJp4ArggiQTwIeAm4GdSW4AngGuA6iqfUl2Ao8Dx4GbquqFdqgb6d2xdA5wX1sAPg78VpKD9M4MNs3JyCRJp+QVA6Gq/sE0m9ZPs/92YHuf+jhwaZ/6n9MCRZI0PH5SWZIEGAiSpMZAkCQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKkxECRJgIEgSWoMBEkSYCBIkhoDQZIEGAiSpMZAkCQBBoIkqTEQJEnAgIGQ5J8n2ZfksSSfTnJ2kvOT7E7yRHs8r7P/tiQHkxxIcmWnflmSR9u2W5JkkH5Jkk7drAMhyXLgF4C1VXUpsAjYBGwF9lTVGmBPe06Si9v2S4ANwK1JFrXD3QZsAda0ZcNs+yVJmp1Bp4wWA+ckWQy8FngW2AjsaNt3ANe29Y3AXVV1rKqeAg4ClydZBiytqgeqqoA7O20kSfNk1oFQVX8EfAR4BjgMfKeqvgRcVFWH2z6HgQtbk+XAoc4hJlpteVufWj9Bki1JxpOMT05OzrbrkqQ+BpkyOo/eX/2rgTcCr0vyvpma9KnVDPUTi1W3V9Xaqlo7NjZ2ql2WJM1gkCmjdwFPVdVkVX0f+BzwE8BzbRqI9nik7T8BrOy0X0FvimmirU+tS5Lm0SCB8AywLslr211B64H9wC5gc9tnM3BPW98FbEqyJMlqeheP97ZppaNJ1rXjXN9pI0maJ4tn27CqHkxyN/BV4DjwMHA7cC6wM8kN9ELjurb/viQ7gcfb/jdV1QvtcDcCdwDnAPe1RZI0j2YdCABV9SHgQ1PKx+idLfTbfzuwvU99HLh0kL5IkgbjJ5UlSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKkxECRJgIEgSWoMBEkSYCBIkhoDQZIEGAiSpMZAkCQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEmAgSJIaA0GSBAwYCEnekOTuJF9Psj/Jjyc5P8nuJE+0x/M6+29LcjDJgSRXduqXJXm0bbslSQbplyTp1A16hvAbwH+vqrcAfx3YD2wF9lTVGmBPe06Si4FNwCXABuDWJIvacW4DtgBr2rJhwH5Jkk7RrAMhyVLgncDHAarqL6rqeWAjsKPttgO4tq1vBO6qqmNV9RRwELg8yTJgaVU9UFUF3NlpI0maJ4OcIbwJmAT+S5KHk3wsyeuAi6rqMEB7vLDtvxw41Gk/0WrL2/rU+gmSbEkynmR8cnJygK5LkqYaJBAWA28DbquqtwLfo00PTaPfdYGaoX5iser2qlpbVWvHxsZOtb+SpBkMEggTwERVPdie300vIJ5r00C0xyOd/Vd22q8Anm31FX3qkqR5NOtAqKpvAoeSvLmV1gOPA7uAza22Gbinre8CNiVZkmQ1vYvHe9u00tEk69rdRdd32kiS5sniAdv/M+BTSV4DfAN4P72Q2ZnkBuAZ4DqAqtqXZCe90DgO3FRVL7Tj3AjcAZwD3NcWSdI8GigQquoRYG2fTeun2X87sL1PfRy4dJC+SJIG4yeVJUmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKkxECRJgIEgSWoMBEkSYCBIkhoDQZIEGAiSpMZAkCQBBoIkqTEQJEkALB52B/Tqt2rrvSe139M3X3OaeyJpJp4hSJIAA0GS1BgIkiRgDgIhyaIkDyf5fHt+fpLdSZ5oj+d19t2W5GCSA0mu7NQvS/Jo23ZLkgzaL0nSqZmLM4QPAPs7z7cCe6pqDbCnPSfJxcAm4BJgA3BrkkWtzW3AFmBNWzbMQb8kSadgoEBIsgK4BvhYp7wR2NHWdwDXdup3VdWxqnoKOAhcnmQZsLSqHqiqAu7stJEkzZNBzxA+CvwS8GKndlFVHQZojxe2+nLgUGe/iVZb3tan1k+QZEuS8STjk5OTA3ZdktQ160BI8h7gSFU9dLJN+tRqhvqJxarbq2ptVa0dGxs7yZeVJJ2MQT6Y9nbgvUmuBs4Glib5JPBckmVVdbhNBx1p+08AKzvtVwDPtvqKPnVJ0jya9RlCVW2rqhVVtYrexeIvV9X7gF3A5rbbZuCetr4L2JRkSZLV9C4e723TSkeTrGt3F13faSNJmien46srbgZ2JrkBeAa4DqCq9iXZCTwOHAduqqoXWpsbgTuAc4D72iJJmkdzEghVdT9wf1v/Y2D9NPttB7b3qY8Dl85FXyRJs+MnlSVJgIEgSWoMBEkSYCBIkhoDQZIEGAiSpMZAkCQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKkxECRJgIEgSWoWD7sDGi2rtt477C5IGhLPECRJgIEgSWpmHQhJVib53ST7k+xL8oFWPz/J7iRPtMfzOm22JTmY5ECSKzv1y5I82rbdkiSDDUuSdKoGOUM4DvxiVf0osA64KcnFwFZgT1WtAfa057Rtm4BLgA3ArUkWtWPdBmwB1rRlwwD9kiTNwqwDoaoOV9VX2/pRYD+wHNgI7Gi77QCubesbgbuq6lhVPQUcBC5PsgxYWlUPVFUBd3baSJLmyZxcQ0iyCngr8CBwUVUdhl5oABe23ZYDhzrNJlpteVufWu/3OluSjCcZn5ycnIuuS5KagQMhybnAZ4EPVtWfzrRrn1rNUD+xWHV7Va2tqrVjY2On3llJ0rQGCoQkZ9ELg09V1eda+bk2DUR7PNLqE8DKTvMVwLOtvqJPXZI0jwa5yyjAx4H9VfVrnU27gM1tfTNwT6e+KcmSJKvpXTze26aVjiZZ1455faeNJGmeDPJJ5bcDPws8muSRVvtl4GZgZ5IbgGeA6wCqal+SncDj9O5QuqmqXmjtbgTuAM4B7muLJGkezToQquor9J//B1g/TZvtwPY+9XHg0tn2RZI0OD+pLEkCDARJUmMgSJIAA0GS1BgIkiTAQJAkNQaCJAkwECRJjYEgSQIMBElSYyBIkgADQZLUGAiSJMBAkCQ1BoIkCTAQJEmNgSBJAgwESVJjIEiSAANBktQYCJIkwECQJDUGgiQJMBAkSc3iYXdA82PV1nuH3QVJI85A0Mg42dB6+uZrTnNPpDOTU0aSJMBAkCQ1BoIkCRihawhJNgC/ASwCPlZVNw+5SxpRXmuQTo+RCIQki4D/CLwbmAD+IMmuqnp8uD0bfd49JGmujEQgAJcDB6vqGwBJ7gI2AmdsIPiLfnBz/TP0jEOvdqMSCMuBQ53nE8DfnLpTki3Alvb0u0kOzEPfTrcLgG8NuxNz6FU7nnx4yD2ZG6/a9+dVYL7G8iPTbRiVQEifWp1QqLoduP30d2f+JBmvqrXD7sdccTyjzfGMrlEYy6jcZTQBrOw8XwE8O6S+SNIZaVQC4Q+ANUlWJ3kNsAnYNeQ+SdIZZSSmjKrqeJKfB75I77bTT1TVviF3a768qqbAcDyjzvGMrqGPJVUnTNVLks5AozJlJEkaMgNBkgQYCKdVkuuS7EvyYpK1U7ZtS3IwyYEkV3bqlyV5tG27JUlafUmSz7T6g0lWzfNwZpRkQxvLwSRbh92f6ST5RJIjSR7r1M5PsjvJE+3xvM62U3qf5luSlUl+N8n+9m/tAwt5TEnOTrI3ydfaeP7dQh5P68eiJA8n+fzIj6WqXE7TAvwo8GbgfmBtp34x8DVgCbAaeBJY1LbtBX6c3mcz7gOuavWfA/5TW98EfGbY4+uMZ1Ebw5uA17SxXTzsfk3T13cCbwMe69T+A7C1rW8FPjzb92kI41kGvK2t/yDwf1q/F+SY2muf29bPAh4E1i3U8bR+/Avgt4HPj/q/N88QTqOq2l9V/T5NvRG4q6qOVdVTwEHg8iTLgKVV9UD1/hXcCVzbabOjrd8NrB/WXzx9/P+vHqmqvwBe+uqRkVNVvw98e0q5+7Pdwct/5qf6Ps2rqjpcVV9t60eB/fQ++b8gx1Q9321Pz2pLsUDHk2QFcA3wsU55ZMdiIAxHv6/qWN6WiT71l7WpquPAd4AfOu09PTnTjWehuKiqDkPvFyxwYavP5n0amjaN+FZ6f1Uv2DG1KZZHgCPA7qpayOP5KPBLwIud2siOZSQ+h7CQJfkfwA/32fQrVXXPdM361GqG+kxtRsEo920Qs3mfhiLJucBngQ9W1Z/OcPI48mOqqheAH0vyBuC/Jbl0ht1HdjxJ3gMcqaqHklxxMk361OZ1LAbCgKrqXbNoNt1XdUy09an1bpuJJIuB13Pi1MewLPSvHnkuybKqOtxOz4+0+mzep3mX5Cx6YfCpqvpcKy/oMQFU1fNJ7gc2sDDH83bgvUmuBs4Glib5JCM8FqeMhmMXsKndObQaWAPsbaePR5Osa9cHrgfu6bTZ3NZ/Cvhym08cBQv9q0e6P9vNvPxnfqrv07xqr/9xYH9V/Vpn04IcU5KxdmZAknOAdwFfZwGOp6q2VdWKqlpF77+JL1fV+0Z6LMO46n6mLMDfo5fux4DngC92tv0KvbsIDtC5YwBYCzzWtv0mf/lp8rOB/0rvQtNe4E3DHt+UsV5N7w6XJ+lNlw29T9P089PAYeD77b25gd61mD3AE+3x/Nm+T0MYzzvoTR/8IfBIW65eqGMC/hrwcBvPY8C/bfUFOZ5OX67gL+8yGtmx+NUVkiTAKSNJUmMgSJIAA0GS1BgIkiTAQJAkNQaCJAkwECRJzf8DRcA8vNOVYrMAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "results_ext = model.predict(df_badt_lim.drop(['mangaid', 'teff', 'teff_ext'], axis=1))\n",
    "\n",
    "target_ext = df_badt_lim['teff_ext']\n",
    "plt.hist(target_ext, bins=30, color='orange')\n",
    "plt.title('Distribution of TEFF_MED')\n",
    "plt.show()\n",
    "\n",
    "mse = mean_squared_error(target_ext, results_ext)\n",
    "print('MSE: %.2f' % mse)\n",
    "\n",
    "residuals = np.array(target_ext).astype('float') - results_ext\n",
    "plt.title('Residuals')\n",
    "plt.hist(residuals, bins=30)\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This results in a lower MSE. We have to note thought this isn't an expression of improved absolute performance of our model. We are calculating MSE against external models, not true values, and these external modesl are possibly using similar modeling techniques as we are.\n",
    "\n",
    "All we can conclude is our model has a comparable performance to these external models."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "7e66e1a829a2a82f0cb6aeacf2ea5f44cd8944c8a18096b63b985eee8a9e9eb6"
  },
  "kernelspec": {
   "display_name": "Python 3.10.4 ('mastar')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}