11_training_deep_neural_networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Chapter 11 – Training Deep Neural Networks**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This notebook contains all the sample code and solutions to the exercises in chapter 11._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures. We also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20 and TensorFlow ≥2.0-preview."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python ≥3.5 is required\n",
    "import sys\n",
    "assert sys.version_info >= (3, 5)\n",
    "\n",
    "# Scikit-Learn ≥0.20 is required\n",
    "import sklearn\n",
    "assert sklearn.__version__ >= \"0.20\"\n",
    "\n",
    "# TensorFlow ≥2.0-preview is required\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "assert tf.__version__ >= \"2.0\"\n",
    "\n",
    "# Common imports\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "# to make this notebook's output stable across runs\n",
    "np.random.seed(42)\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "mpl.rc('axes', labelsize=14)\n",
    "mpl.rc('xtick', labelsize=12)\n",
    "mpl.rc('ytick', labelsize=12)\n",
    "\n",
    "# Where to save the figures\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"deep\"\n",
    "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n",
    "os.makedirs(IMAGES_PATH, exist_ok=True)\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
    "    path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
    "    print(\"Saving figure\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format=fig_extension, dpi=resolution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vanishing/Exploding Gradients Problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def logit(z):\n",
    "    return 1 / (1 + np.exp(-z))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure sigmoid_saturation_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xl4Tdf6wPHvisyDOVJEDTXGPLXILTFXUUNoqbHaKlo/LUqvoVeq1dYY91YHnaKCKkUNNZaosYSG0opWY4gIgpDIJMn6/bGPNMMJCSc5Gd7P8+wn2Xuvs9d7tiPvWXuvvZbSWiOEEEIUNDbWDkAIIYQwRxKUEEKIAkkSlBBCiAJJEpQQQogCSRKUEEKIAkkSlBBCiAJJEpR4IEqpIKXUR9aOA3IWi1LqhFJqRj6FlL7eAKXUxnyox0cppZVS5fOhrpFKqfNKqVRrnNNMsQxXSsVaMwaRd5Q8ByUyU0q5A37A00BFIBo4AXygtd5uKlMWuKO1jrFaoCY5iUUpdQJYrbWekUcx+AC7AHetdVS67aUw/p9FW7Cus8BHWuu56bbZA2WByzoP/1MrpcoAV4DxwGogRmudLwlCKaWB/lrr1em2OQFuWusr+RGDyF+21g5AFEjfA87Ai8BfQAWgHVDubgGt9XXrhJZVQYolM631zXyqJwmIzIeqqmL83diotb6UD/Xdk9Y6Hoi3dhwij2itZZElbQFKAxrodJ9yQRjf4u+uewDrMf5YnANewGh1zUhXRgOjgR+AOOA00B7wBLYCt4EQoFmmuvoCvwGJwAVgKqbWfzaxVDDVcTeWEZljMfN+HjO9JtIUx1GgR6Yy9sAs0zETgb+B/wOqmd5b+iXA9JoAjD/mACOBy0CJTMddDqzPSRym95qhLtN2H9N6+Vyct7PANOAz4BYQDrx5j3M03Mz7rAbMAE6YKRubbn2G6d9gAHAGiAHWpY/XVG5YupgvA0vSxZq+3rPm6jFtewXji1WS6efLmfZr07/FKtM5/hsYbO3/e7JkXeQelMgs1rQ8o5RyzMXrlmB8u+4A9AIGm9YzmwZ8CzQGgk2/fwl8DDQFIjD+qAOglGqO8YdkDdAQeAv4N/DaPWIJAGoCnYDewFCMP6T34gpsBjqbYvseWKOUqpvpPQ7FuLxVD6OFGY3xx9/XVKY+xmXRcWbqWAWUMtVx9/25YpyvwBzG0RcjkbxjqqeiuTeTi/P2BkZCaAZ8CMxWSrU2d0xgJfCU6ffHTXVfyKasOdWA54A+QBeMf+/30sX8Ckay/BpohHGJ+YRpd0vTz5dN9d5dz0Ap1Qf4CPAHGgALgY+VUj0zFX0b44tAY9P7+kop9Wgu3ovID9bOkLIUvAXjj+11IAE4AMwFnshUJghTqwWog/GttFW6/VWAFLK2oN5Pt97AtG18um0+pGsJAMuAnZnqngGEZxNLbdPrvdPtr5o5lhyeh4PANNPvtUzHfSqbshniTrc9AFMLyrS+Bliabn0wcBNwzEkcpvWzwMR71Z/D83YWWJGpzJ/p6zITSwtTPdUyHTcnLagEoFS6bVOBv9Kth2Pc58yubg30u089+4CvzPwb7L3H59AWo0UvragCtkgLSmShtf4eqAT0xPg23wY4qJSaks1L6gKpGC2iu8e4gNEayux4ut8vm37+ZmZbBdPPehh/dNLbC1RWSpU0c/x6plgOpYvlXDaxpFFKuSilZiulfldK3TD1DGsB3P1W3dR03F33Ok4OBAK9lVLOpvVBwPda64QcxpFTOT1vxzOVieCfc29p53TGe3JpdSmlKgCVgZ8eso7s3rdXpm1p71trnQxcJe/et3hAkqCEWVrrBK31dq31O1rrNhiX4WaYeos9jDvpq7nHtpx8Nu/VWy23PdnmAv2B6RgdQppgJLmHfb+ZbQKSgV6mP8qd+OfyXn7Fkf7c3DGzL7d/F1IBlWmbnZlylqjrQWX+PFgzFpFD8g8icup3jEsh5u5LncL4LDW/u0Ep5YnRCntYfwDembb9C+NSlblu5XdjeTxdLI/mIJZ/Ad9orb/XWh/HuNz0WLr9Iabjts/m9UmmnyXuVYnWOhHj3tAgjPsxkRiXKHMax9267lkPuT9vD+Mq4KGUSp+kmuTmANroJn4R6HiPYnd48Pf9e27iEQWDJCiRgVKqnFJqp1JqsFKqkVKqulKqPzAJ+ElrfSvza7TWoRi98D5VSrVSSjXBuNEdR+5bMpnNA9oppWYopWorpQYBE4DZ5gqbYtkCfKaUam2KJYD7d0U+DfRRSjVTSjXEaNWkJWOt9WngO+ALpZSv6bw8qZQaYipyDuO9dldKuZs6P2QnEOgKjMK4B5Sa0zhMzgJPKqUq3+PB3Fydt4cUhPEM1hSl1GNKqReBfg9wnPeA15VSb5hibqKUmpBu/1mgo1LqEdPzWObMAYYopV5VStVSSo3F+DKQF+9b5DFJUCKzWIyb8uOA3cBJjK7VyzG+8WdnOMa3/SCM7ubLMB7oTHiYYLTWRzEueflieljYtNxr5IjhQBiwE9hgiv3sfaoab4p3D8Z9t4Om39MbajrWfzFaagEYvfLQWl8E/oPxR/byfeLbg9Fa8CLj5b2cxvE2RieUMxitlywe8Lw9EK31HxiPD4zEuLfTGeMzk9vjfAK8itFT7wTGF4366YpMwGjBXgB+zeYY64CxGL0Tf8f4HI/RWm/IbTzC+mQkCZEnTN/sI4CBpk4XQgiRKzKShLAIpVQHwA2jR14FjJZEFMa3YCGEyDWLXeJTSr2mlApWSiUqpQLuUW6YUuqIUuqWUirc1KVWEmXhZwe8i5GgNmDcf2qrtb5t1aiEEIWWxS7xKaX6YnQ37Qo4aa2HZ1NuNMb15V8Ad4z7Fau01h9YJBAhhBBFgsVaLlrrNQBKqRYYY6tlV+6TdKsXlVLLyL7rrhBCiGKqIFxaa4vRU8wspdRIjN5BODk5Na9SpUp+xZUjqamp2NhIZ8j7kfOUMxcuXEBrzaOPyrBw95Pfn6nIhEicSzhT0s7cACYFV0H8v3f69OkorbX7/cpZNUEppUZgDOPyUnZltNaLgcUALVq00MHBwdkVtYqgoCB8fHysHUaBJ+cpZ3x8fIiOjiYkJMTaoRR4+fmZmrx9MrP3z2aCzwTebvd2vtRpKQXx/55S6lxOylktQSmlegPvY0zrEHW/8kIIYQ3zD8xn9v7ZjGkxhultp1s7nGLFKglKKfUU8DnQXWv92/3KCyGENSw7vowJ2ybQz6sf/+32XzKO5iTymsUSlKmruC3GWFklTHMJJZtGCk5frgPGKAN9tNaHsh5JCCEKhrDoMNpXa09gn0BK2NxvGEBhaZa8czYNY7yztzDmuIkHpimlHlVKxaabDGw6xvAwP5q2xyqlNlswDiGEeCgpqSkATGs7ja2Dt+Jg62DliIoniyUorfUMrbXKtMzQWp/XWrtqrc+byrXXWtuatt1dulkqDiGEeBihUaE0+KQBhy4aF3jsSpibOUTkh4LQzVwIIQqEiJgIugZ2Je5OHGUcsxswXeQXSVBCCAFEJ0TzVOBTXIu/RtCwIGqVq2XtkIo9SVBCiGIvITmBXt/24lTUKTY9v4nmlZrf/0UizxWsx4uFEMJKyjuX55s+39D5sc7WDkWYSAtKCFFsaa2JuxOHi70Lq/uvluecChhpQQkhii2/3X60/rI10QnRkpwKIElQQohi6dPgT/Hb7UeLSi0o5VDK2uEIMyRBCSGKne9//54xm8bQo3YPFvdcLK2nAkoSlBCiWNlzbg/Pr3meVp6tWNlvJbY2ciu+oJIEJYQoVqqXqc4zdZ5h4/MbcbZztnY44h7kq4MQoli4HHuZ8s7l8Szpyar+q6wdjsgBaUEJIYq8K7ev8K+v/8UrG1+xdigiFyRBCSGKtJjEGLov787FWxcZ0XSEtcMRuSCX+IQQRVZSShK+3/ny66VfWfvcWtpUaWPtkEQuSIISQhRZozeOZvvf2/nqma/oWaentcMRuSQJSghRZL3U7CUaeTTihaYvWDsU8QAkQQkhipzfLv9GQ4+GtK7SmtZVWls7HPGApJOEEKJIWRKyhEafNmLtH2utHYp4SJKghBBFxqbTm3hx/Yt0qtGJ7rW7Wzsc8ZAkQQkhioSD4Qfpv6o/TR5pwppn12Bfwt7aIYmHJAlKCFHoXY+/To/lPahcsjI/DvoRNwc3a4ckLEA6SQghCr2yTmVZ0HUB3o96U8GlgrXDERYiLSghRKF1Pf46v4T/AsCQxkOoUaaGlSMSlmTRBKWUek0pFayUSlRKBdyn7BtKqUil1C2l1FdKKQdLxiKEKNoSUhLouaInTy17iuiEaGuHI/KApVtQEcC7wFf3KqSU6gq8BXQEqgI1AD8LxyKEKKKSU5OZ+cdMDlw4wOc9P6e0Y2lrhyTygNJaW/6gSr0LeGqth2ezfzlwVms9xbTeEVimtX7kXsd1c3PTzZs3z7Dt2WefZcyYMcTFxfH0009nec3w4cMZPnw4UVFR9OvXL8v+0aNH89xzz3HhwgWGDBmSZf+ECRPo2bMnoaGhvPJK1pGQe/bsyYQJEwgJCeH111/Psn/WrFm0adOG/fv3M2XKlCz7/f39adKkCTt27ODdd9/Nsv+zzz6jTp06bNiwgXnz5mXZv3TpUqpUqcLKlSv55JNPsuxfvXo15cuXJyAggICAgCz7f/zxR5ydnfn444/57rvvsuwPCgoCYO7cuWzcuDHDPicnJzZv3gzAzJkz+emnnzLsL1euHN9//z0AgwYN4uLFixn2e3p6EhgYCMDrr79OSEhIhv21a9dm8eLFAIwcOZLTp09n2N+kSRP8/f0BGDx4MOHh4Rn2t27dmvfffx8AX19frl27lmF/x44dmT59OgDdunUjPj4+w/4ePXowceJEAHx8fMgsLz57ISEhJCcn06JFi/t+9qZNm0anTp2K3WdPozlT/wwXK1zk46c/JmpL1D0/e//+9785cOBAhv3F6bPXqVMnSpfOmMAf9u/ew372du/efURr3SLLjkys1UmiPvBDuvVjgIdSqpzWOsO/pFJqJDASwM7OjujojE3506dPExQUREJCQpZ9AKdOnSIoKIibN2+a3X/y5EmCgoK4cuWK2f2//fYbbm5unD9/3uz++Ph4goKC+Ouvv8zuP3r0KElJSZw4ccLs/uDgYKKjozl27JjZ/b/88guXLl3it99+M7v/wIEDnDlzhpMnT5rdv2/fPkqVKsWpU6fM7v/5559xdHTk9OnTZvff/SNx5syZLPvvvneAsLCwLPtTU1PT9iclJWXZb2dnl7Y/PDw8y/6IiIi0/REREVn2h4eHp+2/fPlylv3nz59P23/16lVu3bqVYX9YWFja/uvXr5OYmJhh/5kzZ9L2mzs3efHZS05ORmtNdHT0fT97x44dw9bWtth99m543uBihYsMqDiAerfr8U3YN/f87Jk7f8Xps5eSkpKlzIP83dPahtRUF1JSnNm69QJ//nmEv/++xPnz9UlNdUBrR1JTHUhNdeTDD6FMmbNcvOjAyZMjSU11JDXVEa0dSE11AJ7MUqc51mpBnQFe1VpvMa3bAUlAda312eyO26JFCx0cHGzxeB9GUFCQ2W84IiM5Tznj4+NDdHR0lm/04h8pqSms+n0VHlc9aN++vbXDKfCCgoJo186H27fh2jW4ft1Y0v9+4wbExBjLrVv//J5+PS7OklGpAt2CigVKplu/+3uMFWIRQhQCG09vpLFHY6qUqsKABgPSWhjFVXw8XL4MkZH//Ey/REUZSSgysg2xsXDnzsPX6eb2z+LqCs7O4OT0z5J5PbttPXrkrD5rJaiTQGPg7oXnxsDlzJf3hBACYMffO+i7si++Xr6s8F1h7XDyXFIShIfDuXNw/ryx3P39wgW4dAlu3szp0YwRNZycoGxZKFfO+Hl3KVcOSpeGkiX/ST7mfndxAZuH6FZ3+vRpzp8/T6dOnXL8GosmKKWUremYJYASSilHIFlrnZyp6DdAgFJqGUbPv2lAgCVjEUIUDUcijtBnZR/qlq/LJ92zdsYorKKj4c8/jeX0aePnmTNGEoqMhPvdfbGzAw8PeOSRf5b06+XLG8knNHQ/3bu3wckpf96XOcuXL+eFF16gWbNm1ktQGInmP+nWBwN+SqmvgN8BL631ea31FqXUbGAX4AR8n+l1QgjBX9f/4unlT1POqRxbBm8pdN3JtTYuv/32m7GcOAGhoUYyuno1+9eVKAGVK8Ojj0LVqsbPu0uVKsa+MmVAqfvHcO1aktWSU0JCAmPGjGHlypUkJSWRmpqaq9dbNEFprWcAM7LZ7Zqp7HxgviXrF0IULW9uf5OU1BS2Dt5KJbdK1g7nnpKTjQR05AgcP/5PUoqKMl/eyQlq1TKW2rWNnzVrQrVqULEi2Bbygej+/vtvnn76ac6fP5/WjT63nfIK+SkQQhRlAb0COHfzHHXK17F2KBloDWFhcOjQP8vRo0bHhcxKloQGDaBhQ2OpV89ISJUqPdw9nYJszZo1DBs2jLi4uAytJqu2oIQQ4mElJifywd4PmOQ9iVKOpWjk2MjaIZGcbCSg3bvh55/h4EHzLaPHHoOWLaFRo38S0qOP5uxSXFFw584d3njjDb766qssDx+DtKCEEIVYSmoKg9cOZvXvq2lRqYXVJh1MTTUS0o4dRlLauxdiYzOWcXeHxx//Z2nZ0uiUUFydP3+eHj168Ndff5lNTiAJSghRSGmtGbdlHKt/X83cznPzPTlduQLbtsGWLcbPzJ0YatWCdu2MxdvbuFdUXFpG97Np0yYGDhxIXFwcKSkp2ZaTS3xCiEJp1p5ZLDq8iImtJzKhzYR8qfOPP2DNGli3DjIPUlO1KnTpAh06QNu2xj0jkdWUKVPw9/fPttWUnrSghBCFzrW4a/j/4s+QRkP4sPOHeVaP1kYvuzVrYO1aOHXqn32Ojkbr6KmnjKVOHWkh5UR4eDhaa0qUKHHP1hNIghJCFELlnMtx6KVDeJb0xEZZvmvbn3/C0qUQGGj0vrurbFl45hno2xc6djSG5RG588033zB16lSmTZvGxo0bSUxMzDYRSYISQhQae87tYdfZXUxvO53qZapb9NhRUbBypZGYfvnln+0VKxoJqW9f49JdYX/eqCCoU6cO3333HY0aNeLEiRPZlpMEJYQoFH67/BvPfPsMHi4evN7qdUo6lLz/i+5DawgKgk8/NS7h3R0g1dUVfH1hyBDw8TFGahCWtXPnTsLSN08x5oy7c+cOycnGaHfSSUIIUeCdiz7HU8uewtnOma2Dtz50coqOhiVLjMR0976SjY1xL2nIEOjdWy7f5bVJkyZx+/btDNsqVKiAj48PK1eu5M6dO9KCEkIUbFFxUXQN7MrtpNvseWEPVUtXfeBjnToF8+bBsmX/jOJQsSK8/LKxeHpaKGhxT7t37yY0NDTDNldXV2bPns2zzz7LzJkz8fPzy1FPv/QkQQkh8tXB8INcjLnIj8//SEOPhg90jH37YNq0Buzb98+2Tp1g9Gjo2dMY6VvknzfffDNL66ls2bL069cPgCpVqvDFF1/k+riSoIQQ+apH7R6cHXeWcs65G3YhNRU2bIDZs2H/foDyODjA8OHwxhtGt3CR//bs2cPJkyczbHNxceGDDz7A5iEHGyyiQxUKIQqSVJ3KyA0jWfPHGoBcJSet4YcfoEkT417S/v3GVBNDhpzl3DnjvpMkJ+uZNGkScZnmgy9TpgzPPvvsQx9bEpQQIs+9teMtPj/6Ob9f/T3Hr9HaGHbo8ceNxPTbb8Y9JX9/Y1K/ESPO4uGRh0GL+9q/fz/Hjx/PsM3V1ZX333+fEhboKimX+IQQeWre/nnM2T+HV1u+ytQnp+boNbt3w7RpxiCtYMwUO3Wq0fHB0TEPgxW5Yq71VLJkSQYMGGCR40uCEkLkmcDjgUzcPpH+Xv1Z+NRC1H3GDvrrL5g40bikB8bo4JMnw6uvSjfxguaXX37h119/zbDN1dWVWbNmYWuhp58lQQkh8syxyGO0r9aepX2WUsIm+0s+N2/Cu+/CwoXGw7UuLjBpErz+ujHhnyh4Jk+enKX15OLiwqBBgyxWhyQoIYTFpepUbJQNszvPJiklCQdbB7PlUlLgq6+My3lXrhjbhg+HWbOM55lEwRQcHMyhQ4cybHN1deW9996zWOsJpJOEEMLCQqNCafpZU367/BtKqWyT06+/whNPwMiRRnLy9obDh+HrryU5FXSTJ08mISEhwzYnJyeGDh1q0XokQQkhLObirYt0CezCpZhLONk5mS0TF2dcvmvZ0pj6okoV+PZb2LMHWrTI54BFrv36668cOHAgw7BFLi4uzJw5EzsLPyEtl/iEEBYRnRDNU8ue4nr8dYKGBVGzbM0sZbZtg1GjjCkvbGyMe0wzZxqDuYrC4a233jLbenrhhRcsXpckKCHEQ4u/E88zK54hNCqUHwf9SPNKzTPsv34dxo0z5mMCaNQIvvjCaEWJwuP48ePs2bMnS+vJz88Pe3t7i9dn0Ut8SqmySqm1SqnbSqlzSqnnsynnoJT6VCl1WSl1XSm1QSlV2ZKxCCHyT3JqMo62jizts5RONTpl2LdtGzRoYCQnR0f44ANjenVJToXPW2+9RWJiYoZtDg4OvPjii3lSn6VbUIuAJMADaAJsUkod01qfzFRuHNAaaATcBBYD/wP6WjgeIUQe0lqTmJKIm4MbWwdvzfCcU1yc8QzTRx8Z697eEBAANbNe+ROFwPXr19myZUuW1tOMGTNwcDDfEeZhWawFpZRyAXyB6VrrWK31XmA9MMRM8erAVq31Za11ArASqG+pWIQQ+cNvtx8+AT7EJMZkSE5HjkDz5kZysrU1uo3v3i3JqTArW7Ys27dvp2nTpri4uABgb2/Pyy+/nGd1WrIFVRtI1lqfTrftGNDOTNkvgYVKqUpANDAI2GzuoEqpkcBIAA8PD4KCgiwY8sOLjY0tcDEVRHKeciY6OpqUlJRCca5+iPgB/z/96fZIN4L3B6OUIjUVVqx4lK+/rkZKig1Vq95mypQ/qF07lj17LFu/fKZyxpLnqUSJEsyfP5+QkBA+//xzunXrxsGDBy1ybLO01hZZgCeByEzbXgaCzJQtBXwLaCAZ+BUoe786mjdvrguaXbt2WTuEQkHOU860a9dON27c2Nph3Neqk6u0mqF0j+U99J2UO1prra9e1fqpp7Q2hnnV+v/+T+u4uLyLQT5TOVMQzxMQrHOQVyzZSSIWyDwoSUkgxkzZRYADUA5wAdaQTQtKCFGw7D67m0FrBtG6SmtW9luJrY0tBw5A06bG6OPlysGPPxrDFjmZfxRKiByxZII6DdgqpWql29YYyNxBAowOFAFa6+ta60SMDhKPK6XKWzAeIUQeqFyyMl0e68KGgRtwsnXG3x/atoXwcGjd2hghols3a0cpigKLJSit9W2MltA7SikXpZQ30AtYaqb4YWCoUqqUUsoOGANEaK2jLBWPEMKyouKi0FpTs2xNNgzcQImksvTvb8xmm5xs/AwKMkaGEMISLD3U0RjACbgCrABGa61PKqWeVErFpis3EUgA/gSuAk8DfSwcixDCQq7cvkLrL1szfut4AP78E1q1gu+/N0YbX70a5s+HPHhWUxRjFn0OSmt9HehtZvsewDXd+jWMnntCiAIuJjGGp5c9zcVbF3m2/rP89BP07w83bhgP4K5dK93HCyIfHx8aNGhAv379rB3KA5PBYoUQ2UpKSaLvd30JiQzhu36rOPJDa7p2NZJTz56wf3/RSk5Xr15lzJgxVKtWDQcHBzw8POjYsSPbt2/P0euDgoJQShEVlX93KwICAnA1M5jhmjVreP/99/MtjrwgY/EJIbL18oaX2fH3Dj5/egkb5ndn8WJj+7//bUwwaFPEvuL6+voSFxfHl19+Sc2aNbly5Qq7d+/m2rVr+R5LUlLSQ41vV7ZsWQtGYx1F7OMlhLCkwQ0HM7PVIpZNGsrixeDgAMuWGSNDFLXkFB0dzZ49e/jggw/o2LEjVatWpWXLlkycOJEBAwYAEBgYSMuWLXFzc6NChQr079+fixcvAnD27Fnat28PgLu7O0ophg8fDhiX21577bUM9Q0fPpwePXqkrfv4+DB69GgmTpyIu7s73t7eAMyfP59GjRrh4uJC5cqVeemll4iOjgaMFtsLL7zA7du3UUqhlGLGjBlm66xWrRrvvvsur7zyCiVLlsTT05M5c+ZkiOn06dO0a9cOR0dH6tSpw48//oirqysBAQGWOcm5VMQ+YkIISwiNCgWgtm1nlr8xhqAgYxLBPXvgebNDQBd+rq6uuLq6sn79+izTSdyVlJSEn58fx44dY+PGjURFRTFw4EAAqlSpwvfffw/AyZMnuXTpEgsXLsxVDIGBgWit2bNnD9988w0ANjY2+Pv7c/LkSZYvX86hQ4cYO3YsAG3atMHf3x9nZ2cuXbrEpUuXmDhxYrbHX7BgAQ0bNuTo0aNMnjyZSZMmceDAAQBSU1Pp06cPtra2HDx4kICAAPz8/LIMDpuf5BKfECKDgJAAXlz/Iv9t8jPvvuJNZKTRGWLzZvD0tHZ0ecfW1paAgABefvllFi9eTNOmTfH29qZ///488cQTAIwYMSKtfI0aNfjkk0+oV68e4eHheHp6pl1Wq1ChAuXL5/6xzurVqzNv3rwM215//fW036tVq8bs2bPp1asXS5Yswd7enlKlSqGU4pFHHrnv8bt06ZLWqho7diz//e9/+emnn2jdujXbt28nNDSUbdu2UbmyMbnEggUL0lpy1iAtKCFEmk2nN/HS+pdocvtN3hrUhshI8PExWk5FOTnd5evrS0REBBs2bKBbt27s37+fVq1aMWvWLACOHj1Kr169qFq1Km5ubrQwTQF8/vx5i9TfvHnzLNt27txJ586d8fT0xM3Njb59+5KUlERkZGSuj9+oUaMM65UqVeLKlSsAnDp1ikqVKqUlJ4CWLVtiY8VruZKghBAAHLhwgP6r+lPl7FSOz3+f2FjFgAHG8EWlS1s7uvzj6OhI586defvtt9m/fz8vvvgiM2bM4ObNm3Tt2hVnZ2eWLl3K4cOH2bJlC2Bc+rsXGxubDNNUANy5cydLubujhN917tw5unfvTr169Vi1ahVHjhxrAd4+AAAgAElEQVThq6++ylGd5mSekt0Y4Dc118fJL5KghBBcuX2FHit64PzLO5z92o/kZMWbbxodIvJoqp9Cw8vLi+TkZEJCQoiKimLWrFm0bduWunXrprU+7rrb6y4lJSXDdnd3dy5dupRh27Fjx+5bd3BwMElJSSxYsIDWrVtTu3ZtIiIistSZub4HUbduXSIiIjIcPzg42KoJTBKUEAJ35wq0OLmdaxsmopQx0Ovs2UWvp969XLt2jQ4dOhAYGMjx48cJCwtj1apVzJ49m44dO+Ll5YWDgwMfffQRf//9N5s2bWL69OkZjlG1alWUUmzatImrV68SG2sMoNOhQwc2b97M+vXrCQ0NZfz48Vy4cOG+MdWqVYvU1FT8/f0JCwtjxYoV+Pv7ZyhTrVo1EhIS2L59O1FRUcTFxT3Q++/cuTN16tRh2LBhHDt2jIMHDzJ+/HhsbW0zzPWVn4rRx08Ikdn1+OuEXDrOmDGw7Ztm2Noarab/+z9rR5b/XF1dadWqFQsXLqRdu3bUr1+fKVOm8Pzzz7Ny5Urc3d1ZsmQJ69atw8vLCz8/P+bPn5/hGJUrV8bPz4+pU6fi4eGR1iFhxIgRaYu3tzdubm706XP/0d0aNWrEwoULmT9/Pl5eXnzxxRfMnTs3Q5k2bdowatQoBg4ciLu7O7Nnz36g929jY8PatWtJTEzk8ccfZ9iwYUydOhWlFI6Ojg90zIeWkzk5Csoi80EVXnKeciY/54O6nXRbt/rsX9qh2XcatHZw0HrDhnyp2iLkM5UzD3OeQkJCNKCDg4MtF5DO+XxQ0s1ciGIoOTWZfisGcXDBeDjVB1dXWL8eTM+ZimJq7dq1uLi4UKtWLc6ePcv48eNp3LgxzZo1s0o8kqCEKGa01ryw6jU2+42Gv7tQpozxjJPpUR9RjMXExDB58mQuXLhAmTJl8PHxYcGCBVa7ByUJSohi5rMDgQROHgDnfPDwgG3bINPjMaKYGjp0KEOHDrV2GGkkQQlRjMTGwvK3BsE5GypV0uzapahd29pRCWGe9OITophY/etWOnVJYs8eGypXht27JTmJgk1aUEIUAz8c28mzvV3R5+3x9IRdu4rWPE6iaJIEJUQRtzv0V/o+44w+34rKnqkEBdnw2GPWjkqI+5NLfEIUYSHn/qJT1zuknm9FJc9kft4tyUkUHpKghCii4uOhY7dYks89TsXKd9iz25YaNawdlRA5JwlKiCIoMRH69oXrfzShvMcdfg6yk+QkCh1JUEIUMbHxiTTvEsqWLVC+POzeaScdIkShJAlKiCIk6U4KXp0Pc/LnOriWvMP27eDlZe2ohHgwFk1QSqmySqm1SqnbSqlzSqnn71G2mVLqZ6VUrFLqslJqnCVjEaK4SUnRNOnxCxf2/QsH5yR2bLOjSRNrRyXEg7N0N/NFQBLgATQBNimljmmtT6YvpJQqD2wB3gBWA/ZAMZhQWoi8oTV49w/mj21tsHVIYttmexlbTxR6FmtBKaVcAF9gutY6Vmu9F1gPDDFTfDywVWu9TGudqLWO0Vr/YalYhChOtIax42P5ZW1LbGzvsHG9LW3bWjsqIR6eJVtQtYFkrfXpdNuOAe3MlG0F/KaU2g/UBH4BXtVan89cUCk1EhgJ4OHhQVBQkAVDfnixsbEFLqaCSM5TzkRHR5OSkpKrcxUY+ChfflmDEiVS+c9/TuJgH01xONXymcqZwnyeLJmgXIFbmbbdBNzMlPUEmgGdgd+A2cAKwDtzQa31YmAxQIsWLbSPj4/lIraAoKAgClpMBZGcp5wpXbo00dHROT5Xkz74ky+/rIFSsGyZDc89V3xuOslnKmcK83myZIKKBUpm2lYSiDFTNh5Yq7U+DKCU8gOilFKltNY3LRiTEEXWgq/OMWeK8XDT/IWJPPecg5UjEsKyLNmL7zRgq5SqlW5bY+CkmbLHAZ1uXZspI4TIxooNkYwf+QjoEoz/dzSvj5XkJIoeiyUorfVtYA3wjlLKRSnlDfQClpop/jXQRynVRCllB0wH9krrSYj727n/BoOfdYUUBwaMuMbc90pbOyQh8oSlH9QdAzgBVzDuKY3WWp9USj2plIq9W0hrvROYAmwyla0JZPvMlBDC8Ndf0LenE6kJrnToeZVln5fDSrNxC5HnLPoclNb6OtDbzPY9GJ0o0m/7BPjEkvULUZRdugRdusDN64607ZDI5tXu2MhYMKIIk4+3EIXA9RupNGwTTlgYtGwJG9c5YG9v7aiEyFuSoIQo4OLjoXHbs1w760n5R6PYtAnczD28IUQRIwlKiAIsORladv2T8BM1cCl3g8O7y+Hubu2ohMgfkqCEKKC0ho79/+LknlrYucSyf1dJqlWTHhGi+JAEJUQBNWUK/LyuJjb2CWzfbE+jhiWsHZIQ+UoSlBAF0Lx5mg8+AFtbzerV0O5J6REhih9JUEIUMJfjujBxonEp7+uvFX16Olo5IiGsQxKUEAVI5M3mRP41C4Ap711l8GArBySEFUmCEqKA2LLrFqEnZoK25YWxkbw3RbrrieJNEpQQBUDwrwn07KEg2Rm3Siv5cuEj1g5JCKuTBCWElZ07B890tyc5zg3XSjuoXn62jK8nBBYei08IkTtXrmg6d4FLl2xo106TmjqbW7dSrB2WEAWCJCghrCQmBpq2jSDidGUaNkrlhx9s6NUrKUu59evXExISQsOGDalfvz6PPfYYJUrIM1Gi6JMEJYQVJCbC450uEBFaBTePK2zd4k6pUubLnjlzBj8/P1xdXUlJSSEpKQlPT08aNmzI448/ToMGDahfvz7Vq1eXxCWKFElQQuSzlBTweSacU4eq4FDqBof3lKVixexvOo0ePZp3332X69evp20LCwsjLCyMH3/8EWdn5wyJq1GjRjz++OMMGDCAGjVq5MdbEiJPSCcJIfKR1tB7yEUObvOkhFMMQTscqVPr3t8THR0deffdd3FxccmyLzk5mVu3bnH79m3u3LlDWFgYP/zwA9OmTePAgQN59TaEyBeSoITIR9OmwcYVlbGxS+SH9ZpWLZxy9LqXXnoJV1fX+xcE7O3t6dq1K88/L5NUi8JNEpQQ+eS92XHMmgUlSsC67x3o3qlkjl9rZ2fHhx9+aLYVlVnJkiVZtmwZSvqqi0JOEpQQ+eCjz28xbbIzAF99BT175v4YgwcPpmzZsvcs4+DgQK1atR4kRCEKHElQQuSx79bEMXaUkZzG/ecsQ4c+2HFKlCjB3Llz79mKSkxM5MiRI9SpU4e9e/c+WEVCFBCSoITIQz/tusPAASUg1ZbnRv+F/4xqD3W8fv36UbFixXuWSUpKIioqii5dujB9+nRSUuTBX1E4SYISIo+EhEC3HndIveOAj28oKxbVfOhj2tjYsGDBgiytKEfHrFNyxMfHM3/+fJ544gnCw8Mfum4h8ptFE5RSqqxSaq1S6rZS6pxS6p7diJRS9kqpP5RS8r9HFCl//gldu8KdOGcadwhlx8o6Fhtfr3v37lSvXj1t3dnZmVdffRVXV1dsbDL+l46LiyMkJAQvLy/WrVtnmQCEyCeWbkEtApIAD2AQ8IlSqv49yr8JXLVwDEJYVUQEdOh0hytXoHNn+OXHOlhygAelFP7+/jg7O+Pk5MTAgQOZO3cuJ06coGHDhjg7O2con5KSQkxMDIMGDeKll14iPj7ecsEIkYcslqCUUi6ALzBdax2rtd4LrAeGZFO+OjAYeN9SMQhhbVeuQAvvaMLP21Gv8S3WrAEHB8vX07FjR+rXr0/FihX53//+B0DVqlUJDg5m7NixODllfb4qLi6O5cuX06BBA37//XfLByWEhSmttWUOpFRTYJ/W2jndtolAO611lk61SqmNwJfADSBQa+2ZzXFHAiMBPDw8mn/77bcWiddSYmNjc/wAZXFWHM7TrVu2jBpXi0tnPXCq+BdLF4VTrkzujvH666+TkpKSlnTu5e7QR+a6noeEhPD2228THx9PcnJyhn1KKezt7RkzZgw9e/YstM9LFYfPlCUUxPPUvn37I1rrFvctqLW2yAI8CURm2vYyEGSmbB9gs+l3HyA8J3U0b95cFzS7du2ydgiFQlE/Tzdval2v8S0NWjt6nNVnzsc+0HHatWunGzdubJGYrl69qjt06KCdnZ01kGVxdnbW3bt31zdu3LBIffmtqH+mLKUgnicgWOfgb74l70HFApkfjS8JxKTfYLoUOBv4PwvWLYTV3L4NXbol8scxN2zLXeDgz67UqHL/ER/yWvny5dmxYwfvvfdetpf8duzYQe3atdm/f78VIhTi3iyZoE4Dtkqp9I+xNwZOZipXC6gG7FFKRQJrgIpKqUilVDULxiNEnktIgN694Zf9DpRyj+GnHdC4djlrh5VGKcXrr7/OgQMHqFKlSpbu6ImJiVy9epVOnToxY8YMeWZKFCgWS1Ba69sYyeYdpZSLUsob6AUszVT0BFAFaGJaXgIum36/YKl4hMhrSUnQu28SO3ZAhQrwyx432japYu2wzGrcuDF//PEHvr6+WXr5gfHM1Jw5c2jTpg0XL160QoRCZGXpbuZjACfgCrACGK21PqmUelIpFQugtU7WWkfeXYDrQKppXb6+iUIhORkGPp/M1s32KOcbbNycQJ061o7q3lxcXAgMDOSLL77I9pmpo0eP4uXlxfr1660UpRD/sGiC0lpf11r31lq7aK0f1VovN23fo7U2241Eax2ks+nBJ0RBlJICw19IZc33tuBwkw8CjtKyWdaRHAqqgQMHcvz4cerXr5+lNXV3fqmBAwfyyiuvkJCQYKUohZChjooUHx8fXnvtNWuHUaSlpMALL2iWBdqAXSwTF+1iUv+O1g4r16pXr86RI0cYM2ZMth0oli5dSsOGDTl16pQVIhRCEhRXr15lzJgxVKtWDQcHBzw8POjYsSPbt2/P0etDQkJQShEVFZXHkf4jICDA7HMNa9as4f335bnnvJKSAsOGwdKlCuxiGT73O+a82NvaYT0wOzs75syZw4YNGyhTpgx2dnYZ9sfHx3PmzBmaN2/OF198cfcRESHyTbFPUL6+vhw6dIgvv/yS06dPs3HjRrp168a1a9fyPZakpKSHen3ZsmVxc3OzUDQiveRkGDoUli0DV1fNm5/8xFdjX7B2WBbRsWNHQkND8fb2znLJT2tNXFwc48aNo1evXty8edNKUYpiKScPSxWUxdIP6t64cUMDevv27dmWWbp0qW7RooV2dXXV7u7uul+/fjo8PFxrrXVYWFiWhx+HDRumtTYeuHz11VczHGvYsGG6e/fuaevt2rXTo0aN0hMmTNDly5fXLVq00FprPW/ePN2wYUPt7OysK1WqpF988cW0hyl37dqVpc7//Oc/ZuusWrWqnjlzph45cqR2c3PTlStX1rNnz84QU2hoqG7btq12cHDQtWvX1ps2bdIuLi7666+/fqBzmp2C+LBgTt25o/XAgVqD1q6uqXrv3ryry5IP6uZWamqqnjt3rnZycjL7YK+Dg4P28PDQBw4csEp8mRXmz1R+KojnCSs8qFvouLq64urqyvr167O9GZyUlISfnx/Hjh1j48aNREVFMXDgQACqVKmCn58fACdPnuTSpUssXLgwVzEEBgaitWbPnj188803gDGlgr+/PydPnmT58uUcOnSIsWPHAtCmTZu0gUIvXbrEpUuXmDhxYrbHX7BgAQ0bNuTo0aNMnjyZSZMmceDAAQBSU1Pp06cPtra2HDx4kICAAPz8/EhMTMzVeyjKkpNhyBBYsQKwj6Hj9Ll4e1s7qryhlGLChAns27ePypUrm31m6vLly3To0IGZM2eSmppqpUhFsZGTLFZQlrwY6mj16tW6TJky2sHBQbdq1UpPmDBBHzx4MNvyf/zxhwb0hQsXtNZaL1iwQAP66tWrGcrltAXVsGHD+8a4efNmbW9vr1NSUrTWWn/99dfaxcUlSzlzLagBAwZkKFOzZk09c+ZMrbXWW7Zs0SVKlEhrEWqt9b59+zQgLSitdUKC1n36GC0nHG7qxyYO0dHx0XlapzVbUOnFxMToAQMG3HOYpFatWumIiAirxVgYP1PWUBDPE9KCyhlfX18iIiLYsGED3bp1Y//+/bRq1YpZs2YBcPToUXr16kXVqlVxc3OjRQtjfMPz589bpP7mzZtn2bZz5046d+6Mp6cnbm5u9O3bl6SkJCIjI3N9/EaNGmVYr1SpEleuXAHg1KlTVKpUicqVK6ftb9myZZbnY4qj27fhmWdg7VpQjjd5ZPQwfn77A0o5lrJ2aPnC1dWVFStWsHjxYlxcXMw+MxUcHEzdunXZtGmTlaIURZ38JcKYjbRz5868/fbb7N+/nxdffJEZM2Zw8+ZNunbtirOzM0uXLuXw4cNs2bIFuH+HBhsbmyy9nu7cuZOlXOaZUc+dO0f37t2pV68eq1at4siRI3z11Vc5qtOczD2zlFJyaeY+oqONyQa3bQM7txuUGtWb3dM+pJJbJWuHlu8GDRrE8ePHqVevXrbPTPXv359XX31VLg0Li5MEZYaXlxfJycmEhIQQFRXFrFmzaNu2LXXr1k1rfdxla2sLkGUMM3d3dy5dupRh27Fjx+5bd3BwMElJSSxYsIDWrVtTu3ZtIiIiMpSxt7e3yJhpdevWJSIiIsPxg4ODi3UCu3oVOnSAffvA0xO270rkp0nzqF2utrVDs5oaNWrw66+/MnLkSLPPTMXHx7N48WK2bdtmhehEUVasE9S1a9fo0KEDgYGBHD9+nLCwMFatWsXs2bPp2LEjXl5eODg48NFHH/H333+zadMmpk+fnuEYHh4eKKXYtGkTV69eJTY2FoAOHTqwefNm1q9fT2hoKOPHj+fChfsPNVirVi1SU1Px9/cnLCyMFStW4O/vn6FMtWrVSEhIYPv27URFRREXF/dA779z587UqVOHYcOGcezYMQ4ePMj48eOxtbUttHMEPYzwcGjbFn79Fcp7RrP75xTaNX+EZhWbWTs0q7Ozs2PBggWsW7eO0qVLZ2iZ29nZ0aZNG7p3727FCEVRVKwTlKurK61atWLhwoW0a9eO+vXrM2XKFJ5//nlWrlyJu7s7S5YsYd26dXh5eeHn58f8+fMzHMPd3R0/Pz+mTp2Kh4dH2kgOI0aMSFu8vb1xc3OjT58+942pUaNGLFy4kPnz5+Pl5cUXX3zB3LlzM5Rp06YNo0aNYuDAgbi7uzN79uwHev82NjasXbuWxMREHn/8cYYNG8bUqVNRSmXpwVXUhYbCk0/CqVNQ6tFzRD1Xlwtqr7XDKnC6dOlCaGgorVq1Srvk5+LiwqpVq+TepbC8nPSkKCiLTFiY90JCQjSgg4ODLXrcgnye9u7VumxZo7eeR52/NZPK6Pn751slloLSi+9+UlJS9Icffqjt7Oz0tm3brBJDQf5MFSQF8TyRw158ttZOkMK61q5di4uLC7Vq1eLs2bOMHz+exo0b06xZ8bistXYtPP+8Ma9T7danOd2+KZN8XuON1m9YO7QCzcbGhkmTJjFu3DgcHBysHY4ooqRNXszFxMTw2muv4eXlxaBBg6hXrx5bt24tFvegPvoIfH2N5DTohVjOP9WcYS3780GnD6wdWqEhyUnkJWlBFXNDhw5l6NCh1g4jX6Wmwr//DXdv3b37LkyZ4sqkK/uoV75esUjOQhQGkqBEsRIXBy+8AN99B7a28OYHoTzSfi9KvUgjj0b3P4AQIt/IJT5RbISHGz31vvsO3Nzgv0v/5uM7TzDvwDwSkmViPmupVq1alp6qQoC0oEQxcfAg9OkDkZHw2GPwSeBFhu37F672rmwZvAVH2+LVrT6/DR8+nKioKDZu3Jhl3+HDh7OMqCIEFIMWVGRkJN26dWPZsmUyFEsxtXQp+PgYyal9e9i0M4rXgjsQnxzP1sFbebTUo9YOsVhzd3fPMoySNTzsfGzC8op8gvr444/ZsWMHo0aNwt3dnTfeeCPLEESiaEpJgcmTjYkGExNhzBjYuhUOXN/IhZsX2DhwI/Ur1Ld2mMVe5kt8SikWL15M//79cXFxoUaNGgQGBmZ4zcWLF3nnnXcoU6YMZcqUoXv37vz5559p+8+cOUOvXr145JFHcHFxoVmzZllab9WqVWPGjBmMGDGC0qVLM2jQoLx9oyLXinSCSk5OZtGiRSQnJxMbG0tMTAyLFi1iyZIl1g5N5LHLl6FLF6OnXokS8PHHsGgR2NnB8CbDCX0tFO9Hi+jETkXAO++8Q69evTh27BjPPfccI0aMSJtBIC4ujvbt22Nvb8/u3bs5cOAAFStWpFOnTmnDfsXGxtKtWze2b9/OsWPH8PX1pW/fvpw6dSpDPfPnz6du3boEBwenzWAgCo4inaA2bdqUZQRxGxsbBg8ebKWIRH74+Wdo2hR27gQPD9ixA14ZlcprP77G/gv7AahSqoqVoxT3MmTIEAYPHkzNmjWZOXMmtra2/PzzzwB8++23aK2ZPHkyjRo1om7dunz22WfExsamtZIaN27MqFGjaNiwITVr1mTq1Kk0a9aM1atXZ6inXbt2TJo0iZo1a1KrVq18f5/i3op0gpo9ezYxMTEZtj355JN4enpaKSKRl1JT4YMPjPtMly79M/Crjw9M2j6JRYcX8fO5n60dpsiB9POY2dra4u7unjaTwJEjRwgLC+Ppp59OmxW7VKlS3LhxgzNnzgBw+/ZtJk2ahJeXF2XKlMHV1ZXg4OAs87jdnd9NFEwW7cWnlCoLfAl0AaKAf2utl5sp9yYwDKhqKvex1nqOJWP5+++/OXr0aIZtbm5u95weXRRe16/DsGFw9zbDW2/BzJnGs05z989l3oF5vNbyNSZ7T7ZuoCJH7jWPWWpqKk2aNOGNN97giSeeyFCubNmyAEycOJEtW7Ywd+5catWqhbOzM0OHDs3SEUJ6DxZslu5mvghIAjyAJsAmpdQxrfXJTOUUMBQ4DjwGbFNKXdBaf2upQP73v/9lmTPJ2dmZzp07W6oKUUDs3Gkkp/BwKFMGvvkGevQw9n1z7Bve3P4mz9Z/Fv+n/GWUiCKgWbNmrFixglKlSlGzZk2zZfbu3cvQoUPx9fUFICEhgTNnzlC7dvGd16swsliCUkq5AL5AA611LLBXKbUeGAK8lb6s1jr9/BChSqkfAG/AIgkqMTGRL7/8MsP9JycnJ8aNGydTAhQhCQkwZQosWGCsP/EEfPstVKtmrGut2fzXZjpU78A3vb+hhE0Jq8Uq4NatW4SEhGTYVrp06VwfZ9CgQcydO5epU6fi5ubGo48+yoULF/jhhx8YNWoUtWrVonbt2qxdu5ZevXphZ2eHn58fCQnyMHZhY8kWVG0gWWt9Ot22Y0C7e71IGV9pnwQ+y2b/SGAkGJMDBgUF3TeQHTt2kJycnGFbcnIy9erVy9HrcyM2NtbixyyKLH2e/vrLhffe8+LsWRdsbDRDh55l8ODznD2rOXvWSE5KKV4q+xJJpZM4sPeAxerOS9HR0aSkpBS5z1RkZCR79uyhadOmGba3bds2rXWT/j2fPHmS8uXLp61nLvP+++/z8ccf07t3b27fvk25cuVo0qQJv//+OxcvXqR///7MmTMHb29vXF1d6devH15eXkRGRqYdw1y9RVGh/huVkzk5crJgJJnITNteBoLu8zo/jETmcL86cjofVJMmTTSQtiildK9evXI6VUmuFMS5VgoiS52n5GStP/xQazs7Y/6mWrW0/uWXjGX+uPqHbvd1O33h5gWL1JmfCst8UAWB/N/LmYJ4nrDCfFCxQMlM20oCMWbKAqCUeg3jXtSTWmuLDPNw4sQJQkNDM2xzdnaWzhFFwPHj8PLLcOiQsT56NMyZA+nvc1+8dZGugV1JSE4gMVlGDhGiMLPkDZnTgK1SKv3DBI2BzB0kAFBKjcC4N9VRax1uqSD8/f2z9NQpX7483t7yUGZhFR9vTI/RvLmRnCpXNnrrffxxxuR0I/4GTy17ihvxN9gyaAuPlX3MekELIR6axRKU1vo2sAZ4RynlopTyBnoBSzOXVUoNAmYBnbXWf1sqhtu3b7N8+fIMvfecnZ2ZMGGC9N4qpH76CRo2NJ5vSkmBV1+F33+H7t0zlou/E88z3z7D6WunWTdgHU0rNjV/QCFEoWHpLm1jACfgCrACGK21PqmUelIpFZuu3LtAOeCwUirWtHz6sJUvX748Sy+91NTUYjchX1Fw6ZLRdbxTJzhzBurXh337jFlwS2a+kAzEJMUQmxTL0j5L6VC9Q/4HLISwOIs+B6W1vg70NrN9D+Cabr26Jes1HZM5c+Zw+/bttG02Njb4+vpSqlQpS1cn8khCgtFtfNYsiI0FBweYPh3efBPs7bOW11qTqlOp4FKBwy8fxtZGZpARoqgoMg8FBQcHExERkWGbo6Mj48ePt1JEIje0hu+/By8v49mm2Fjo1QtOnICpU80nJ4D/BP0H3+98SUpJkuQkRBFTZBLUvHnziI+Pz7CtatWqNGvWzEoRiZwKDjbGz+vXD8LCoEEDY4DXdesgm4ECAFh0aBEzf55Jeefy2NnYZV9QCFEoFYkEdePGDX744Ye0sboAXF1dpWt5AXf8uDHLbcuWsHs3lCtn9Mz79Vfo2PHer111chVjN4/lmTrP8GmPT6UTjBBFUJG4JhIQEJClc4TWmgEDBlgpInEvp07BjBmwcqWx7uQEY8caA7yWKXP/1+8M28ngtYPxftSbb32/lUt7QhRRhf5/ttaa+fPnp01UBsbw/EOGDCkQ00iLf/z+O3z4IQQGGlNj2NsbD9u+9RY88kjOj+Ni50Jrz9asfW4tTnZOeRewEMKqCn2C2r17N9HR0Rm22dnZMW7cOCtFJNLTGvbuhSlTGnDANByera0xIsTUqVAlF/MGxiTG4ObgxhOeT7Br2C65rCdEEVfo70HNmzeP2NjYDNu8vLyoW7eulSISYDxUu2YNtGljTBx44EB5HB1hzBgIDYVPP81dcroce5mmn8yIIesAAA6sSURBVDVl9j5jIHxJTkIUfYUqQcXHx7Nt27a0zhCXL19mx44dGcq4uroyadIka4QngCtXjMt4tWqBry8cPAhly8LQoWc5fx4WLYIaNXJ3zFuJt+i2rBsRMRG0rdo2bwIXQhQ4heoS37Vr1+jWrRsVKlRg3LhxREVFZSljY2ND795ZnhUWeejuZbxPPoHVq+HuNFzVqsH48TBiBBw+fBZ392q5PnZiciJ9V/bl+OXjrB+4nlaerSwauxCi4CpUCcrW1hYbGxsiIyN55513SEpKyjDunr29PSNHjsQ+u6c6hUVdvAjLl8OSJXDSNCSwUsZstqNHQ9euUOIh5gjUWjP8h+H8FPYTS3ov4elaT1smcCFEoVDoEpSDgwPJyclZHsoF477EkCFDrBBZ8REba9xbWrrUGMjVmNILPDzgpZeMzg9Vq1qmLqUUTz32FC0qtmBoYxlPUYjiptAlqBL3+EpeokQJnnjiCfr168f48eOzzN4pHkxsLGzZYgxFtH493O3Rb29vtJYGDzZGF7dkwzX8VjieJT0Z1mSY5Q4qhChUClUnCVtb23v23oqLiyMhIYHly5fTrFkzPv/883yMrmi5ft24dNerF7i7Q//+8O23RnLy9jZ64f1/e/cfXFV95nH8/eSGREJ+CGIR5IdIYV2pJUgKSyklitXQahU7Si21ZbsV1wIdpkut1nXGarvd6XRKO9aRUtktgsViS3fBiFVrg9KOsrCbqKwIZRHFEeVXIAmBEPLsH+deSWKSe0MunHNzP6+Z7+Sek++9eXLm5Dz53vO9z/fdd4OkNXNmepPTsv9exugHR/PynpfT96IiknEybgTV+p5TZ8455xwmTZrELbfcchai6h1aWqCmJhgpPf10sLRF60M9eTLceGPQujsLrzvWvrGWuU/O5aqLr9KaTiJZLuMSVPvVctsrKCjgpptu4pFHHiE3N6N+vbNuz56gBt4zz8Af/gDvvXfqe7FYsBbTzJlwww0wZMiZj+fPb/2ZWb+dxYTBE/jdzb8jL6bJLiLZLKOu4LFYjBOJOcwdKCgo4O677+aee+7RBzk7sHt3kJASbefOtt8fOhQqKoI2fTqce+5ZjK12N9etuo5hxcOo/FIlhXmFyZ8kIr1aRiUoM6OgoKDNooQJBQUFLF26lNmzZ4cQWfQcPgxbtsCmTafaO++07VNUBJ/6FFx5JcyYEazFFFZeH1YyjAUTFzCndA7n9zs/nCBEJFIyKkEBlJSUfChBFRYWsm7dOsrLy8MJKmQHD8KrrwZt8+YgGW3bdmoKeEJJCUydCtOmQXk5lJYGdfHCdODoAeqb6hlx7gi+d8X3wg1GRCIl4xJU//79P1g5Nzc3l/79+1NVVcWll14acmRn3tGjQR27RDJKtHYLCQPBrLrS0mCtpYkTgzZmDOREaN5mQ1MD1666lvcb3uf1ea/rnpOItJFxCWrgwIEA5OfnM2LECKqqqhg8eHDIUaXP8ePBqrLbt8OOHUFLPN6zp+PnFBTA2LFw2WVw+eVBMvr4xyE//+zG3h0nTp5g1m9nsemdTTxx0xNKTiLyIRmXoC644AJycnKYNGkSlZWVFBZmzs109+DtuLfeatt27z71eO/eD781l5CbC6NGBYmodRs5smclhc42d+e2dbdRuaOSJZ9bwo1/e2PYIYlIBGVcgpoyZQqFhYUsWbIkMtPIGxth//4gubRu773Xdvvdd09VYehMTk5QKmjMmKCNHh20MWOC/RH5lXvkwU0PsrxmOfdNu4/by24POxwRiaiMu9wtWLAg7a/Z0gINDXDkCNTVnWq1tcGI58CB4Gv7xwcPwr59U0ny0aw2ioqCRDN8eNBaPx4+PPi8UW9IQl2ZUzqHHMth3ifmhR2KiERYWi+FZjYAWAZcDewH7nb3X3fQz4B/Bb4e3/UIcJd7Z29uBZqb4c03gxFLoh09mtp2Q0OQdFonocTjdusddlOMvDwYODBYtjzRBg1qu53YV1LSk5+V2Z7f9TyTLpxEcX4x8yfODzscEYm4dP+v/hDQBAwCSoFKM6tx963t+s0FbgDGAQ48C+wClnT14jU1wf2WM6Ffv2B0U1wcfC0qCpLJeecFC+4lviZaYvu1116gouLToX1+KFNsPriZ7774XeZ9Yh6LKxaHHY6IZABLMmhJ/YXM+gGHgI+5+/b4vhXAO+5+V7u+fwF+5e5L49v/ANzm7l2uRpeTM97z8taTk9NELHacnJxT7dR2U3z72AePE9u5uQ3EYo3EYkeJxRrIzU08bsSs5bR+79raWs49myUXMlBdUR3V46rpe6wvpf9TSu7JXv4eZg9UV1fT3NxMWVlZ2KFEnv72UhPF47Rhw4Yt7p70JE/nlWIM0JxITnE1wLQO+o6Nf691v7EdvaiZzSUYcdGnTx8uuaSix4G2tASti6pJKTt58iS1tbU9f6Fe6ni/4/z1sr8Sa4ox4sUR1B/v0fupvV5zczPurnMqBfrbS00mH6d0JqhC4Ei7fYeBok76Hm7Xr9DMrP19qPgoaylAWVmZb968OX0Rp0FVVVXWVrBIxt2ZvGwy/Q/15ydjf8KXf/TlsEOKvPLycmpra6murg47lMjT315qonicUq2Vms4EVQ8Ut9tXDNSl0LcYqE82SUIyi5mxYuYKjhw/Qt32jk4DEZHOpbPwzXYg18xGt9o3Dmg/QYL4vnEp9JMMdKz5GL/c8kvcndHnjWbCkAlhhyQiGShtCcrdG4A1wP1m1s/MpgDXAys66P4o8C0zu9DMhgD/BPwqXbFIeE62nGT2mtnMfXIuL+15KexwRCSDpbt06DeAvsD7wCrgDnffamZTzaz13fFfAOuAV4HXgMr4Pslg7s78p+az5vU1LL5mMZOHTQ47JBHJYGmd7+vuBwk+39R+/4sEEyMS2w7cGW/SSzzwwgMs2bKE70z5Dgv/bmHY4YhIhovQ4guSyXYe3Mn3X/g+Xx33VX44/YdhhyMivYA+MSlpMWrAKDZ+bSPjLxif8hRSEZGuaAQlPbLhzQ2s3roagIkXTqRPrE/IEYlIb6ERlJy2mr01fP7xzzOseBgzL5mp5CQiaaURlJyWXYd2UfFYBUV5RTw1+yklJxFJO42gpNv2NezjmpXXcKz5GBv/fiPDS4aHHZKI9EJKUNJtv9n6G94+8jbP3focYz/SYY1fEZEeU4KSbps/cT4zPjqDUQNGhR2KiPRiugclKWnxFhY+vZDqvUGVbSUnETnTlKAkKXdn0TOL+NnLP+PZnc+GHY6IZAklKEnqx3/5MYtfWsyCiQtY9MlFYYcjIllCCUq6tLx6OXc+dyc3j72Zn1b8VFUiROSsUYKSTrk7T/zvE0wfOZ1Hb3iUHNPpIiJnj2bxSafMjDWz1tB0son83PywwxGRLKN/ieVDtu3fxozHZrCvYR95sTwK8wqTP0lEJM00gpI29hzZw9UrrqbpZBN1TXWc3+/8sEMSkSylBCUfONR4iIqVFdQeq2XDnA1c3P/isEMSkSymBCUANJ5o5LpV17Hj4A7Wz17P+MHjww5JRLKc7kEJAAcaD7D/6H5WzlzJlSOvDDscERGNoLKdu+M4Q4uH8sodr5AXyws7JBERQCOorHfvn+5lzn/MobmlWclJRCJFCSqL/XzTz/nBiz8gP5ZPzGJhhyMi0oYSVJZavXU131z/Ta7/m+t5+NqHVcJIRCInLQnKzAaY2e/NrMHMdpvZl7ro+20ze83M6sxsl5l9Ox0xSOqe3/U8t/7+VqYMn8KqL6wiN0e3IkUketJ1ZXoIaAIGAaVApZnVuPvWDvoa8BXgFWAU8IyZve3uj6cpFknCMMqGlLH2i2vp26dv2OGIiHSoxwnKzPoBXwA+5u71wEYzWwvcCtzVvr+7/6jV5htm9p/AFEAJ6gxrPNFI3z59uWLkFWy8aKPe1hORSEvHCGoM0Ozu21vtqwGmJXuiBVfIqcAvuugzF5gb36w3szd6EOuZMBDYH3YQGUDHKXUDzUzHKjmdU6mJ4nEakUqndCSoQuBIu32HgaIUnnsfwX2wf++sg7svBZaebnBnmpltdveysOOIOh2n1OlYpUbHKTWZfJySTpIwsyoz807aRqAeKG73tGKgLsnrzie4F/U5dz9+ur+AiIj0TklHUO5e3tX34/egcs1stLvviO8eB3Q0QSLxnK8R3J/6tLvvST1cERHJFj2eZu7uDcAa4H4z62dmU4DrgRUd9Tez2cC/AJ9x9//r6c+PgMi+/RgxOk6p07FKjY5TajL2OJm79/xFzAYA/wZ8BjgA3OXuv45/byqw3t0L49u7gKFA67f1Vrr7P/Y4EBER6TXSkqBERETSTaWOREQkkpSgREQkkpSg0szMRpvZMTNbGXYsUWNm+Wa2LF6vsc7Mqs1sRthxRUV3alpmK51D3ZfJ1yQlqPR7CPivsIOIqFzgbYIqIyXAPwOrzeyiEGOKktY1LWcDD5vZ2HBDihydQ92XsdckJag0MrMvArXAH8OOJYrcvcHd73P3N929xd2fBHYBE8KOLWytalre6+717r4RSNS0lDidQ92T6dckJag0MbNi4H7gW2HHkinMbBBBLcdOP9SdRTqraakRVBd0DnWuN1yTlKDS5wFgmSpjpMbM+gCPAcvdfVvY8URAT2paZiWdQ0ll/DVJCSoFyeoRmlkpcBWwOOxYw5RC3cZEvxyCSiNNwPzQAo6W06ppma10DnWtt1yTtJRqClKoR7gQuAh4K77GUiEQM7NL3f3yMx5gRCQ7TvDBEivLCCYCfNbdT5zpuDLEdrpZ0zJb6RxKSTm94JqkShJpYGYFtP3vdxHByXGHu+8LJaiIMrMlBKsuXxVf4FLizOxxwIGvExyjp4BPdrIyddbSOZRcb7kmaQSVBu5+FDia2DazeuBYJp0IZ4OZjQBuJ6jDuLfVir63u/tjoQUWHd8gqGn5PkFNyzuUnNrSOZSa3nJN0ghKREQiSZMkREQkkpSgREQkkpSgREQkkpSgREQkkpSgREQkkpSgREQkkpSgREQkkpSgREQkkv4fpnIt6Q3iZsAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "z = np.linspace(-5, 5, 200)\n",
    "\n",
    "plt.plot([-5, 5], [0, 0], 'k-')\n",
    "plt.plot([-5, 5], [1, 1], 'k--')\n",
    "plt.plot([0, 0], [-0.2, 1.2], 'k-')\n",
    "plt.plot([-5, 5], [-3/4, 7/4], 'g--')\n",
    "plt.plot(z, logit(z), \"b-\", linewidth=2)\n",
    "props = dict(facecolor='black', shrink=0.1)\n",
    "plt.annotate('Saturating', xytext=(3.5, 0.7), xy=(5, 1), arrowprops=props, fontsize=14, ha=\"center\")\n",
    "plt.annotate('Saturating', xytext=(-3.5, 0.3), xy=(-5, 0), arrowprops=props, fontsize=14, ha=\"center\")\n",
    "plt.annotate('Linear', xytext=(2, 0.2), xy=(0, 0.5), arrowprops=props, fontsize=14, ha=\"center\")\n",
    "plt.grid(True)\n",
    "plt.title(\"Sigmoid activation function\", fontsize=14)\n",
    "plt.axis([-5, 5, -0.2, 1.2])\n",
    "\n",
    "save_fig(\"sigmoid_saturation_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Xavier and He Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Constant',\n",
       " 'GlorotNormal',\n",
       " 'GlorotUniform',\n",
       " 'Identity',\n",
       " 'Initializer',\n",
       " 'Ones',\n",
       " 'Orthogonal',\n",
       " 'RandomNormal',\n",
       " 'RandomUniform',\n",
       " 'TruncatedNormal',\n",
       " 'VarianceScaling',\n",
       " 'Zeros',\n",
       " 'constant',\n",
       " 'deserialize',\n",
       " 'get',\n",
       " 'glorot_normal',\n",
       " 'glorot_uniform',\n",
       " 'he_normal',\n",
       " 'he_uniform',\n",
       " 'identity',\n",
       " 'lecun_normal',\n",
       " 'lecun_uniform',\n",
       " 'ones',\n",
       " 'orthogonal',\n",
       " 'serialize',\n",
       " 'zeros']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[name for name in dir(keras.initializers) if not name.startswith(\"_\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Dense at 0x10dca5470>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.layers.Dense(10, activation=\"relu\", kernel_initializer=\"he_normal\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Dense at 0x10dca5b70>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "init = keras.initializers.VarianceScaling(scale=2., mode='fan_avg',\n",
    "                                          distribution='uniform')\n",
    "keras.layers.Dense(10, activation=\"relu\", kernel_initializer=init)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nonsaturating Activation Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Leaky ReLU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def leaky_relu(z, alpha=0.01):\n",
    "    return np.maximum(alpha*z, z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure leaky_relu_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl8FPX9x/HXh3AlHCJyVMGCeKCgVSCetBiPUm9RUEG04sGhVeuB1oMKiGfFWjwBiyJyi1yitD9Fo+JVQVG8gFJQQUUEEgghAZLv74/voiHk2E2ymdnN+/l47IM9JjvvHTb7zsx8d8acc4iIiIRNraADiIiIlEQFJSIioaSCEhGRUFJBiYhIKKmgREQklFRQIiISSiooKZWZZZrZ40HnSAZmlmFmzsyaVcO8VpvZ4GqYz6Fm9p6Z5ZnZ6njPL4o8zsx6BZ1Dqo4KKkGZ2Xgzmxd0jlhFSs9FLtvNbKWZ3W9m9WJ8nn5mllPOfPYo1/J+riqUUhDvAvsCG6pwPsPM7LMSHjoaeLKq5lOGe4Bc4NDIPKtFGe/9fYGXqiuHxF/toANIjfQscAdQF//B9mzk/tsDSxRnzrntwA/VNK/11TEf4CBgjnNudTXNr0zOuWpZvlJ9tAaVpMxsLzMba2Y/mtkWM3vTzNKLPL6PmU0xszVmts3MPjezy8t5zlPMLMvMBplZNzPbYWa/KjbNvWb2aTnxcp1zPzjnvnHOvQi8CnQv9jytzGyqmW2KXF42s4NjXAwVYmYPmNmyyHJZbWZ/M7P6xaY5w8w+iEyzwcxeMrP6ZpYJtAEe2rWmGJn+5018ZtY48nNnF3vO7pFl2qK8HGbWDxgKdCyyRtov8thua3Bm9mszmxV5H2wxs5lm1rrI48PM7DMz6x1Zo91iZrPL2hwZeV1HAndF5j3MzNpGrqcXn3bXprci0/Q0s1fNLNfMvjCz3xf7mUPNbK6ZZZtZTmRT4hFmNgy4DDizyOvOKD6fyO0jzOy1yPLbGFnz2qvI4+PNbJ6Z/dnM1kbeZ8+aWVppr1uqlwoqCZmZAS8DrYCzgE7AW8DrZrZvZLL6wEeRxzsCo4AxZnZKKc/ZC5gFDHDOjXbOvQWsBP5YZJpakdvjYsh6JNAV2FHkvjTgDSAPOBE4HvgeeK2aPjy2AlcAhwHXAL2BO4vkOw2Yiy/WLsBJwJv436fzgTXA3fhNTvtSjHNuM35TVN9iD/UFXnXO/RhFjmnAw8CyIvOZVnxekf+TOUDLSM6TgP2A2ZH3yS5tgYuA8/B/LHQC7i1l+RCZ37JIhn2BkWVMW5J7gUfxJfchMNXMGkYy7wcsBBzwe6Az8ASQEpnPdOC1Iq/73RJedwPg30AOcEzkdZ0APFNs0t8BhwOn8svr/3OMr0XixTmnSwJegPHAvFIeOxn/i5la7P4lwK1lPOdU4J9FbmcCjwMDgGyge7HpBwNfFrl9OpAP7FPGPDKB7ZF8+fgPoQKgZ5FprgBWAFbkvhT8/psLI7f7ATnlzOfxEu4v8+dKea5BwH+L3H4HmFrG9KuBwcXuy4i81maR2+fg9980itxOBTYDF8eQYxjwWVnzx3/AFwBtizzeDigETi3yPHnAXkWmubPovErJ8xkwrMjttpHXmF5sOgf0KjbNwCKPt4rc99vI7XuBr4G6sbz3i82nf+Q926iE/4ODijzPt0BKkWmeBl6ryO+kLlV/0RpUcuoCpAHrI5tHcswPDDgcOBDAzFLM7E4z+zSyiSoH/9f/r4s9Vw/8X6+nOef+r9hjzwHtzOyEyO0rgNnOufIGAkwDjsKvGU0HnnZ+U1/R/AcAW4pkzwb23pU/nsysl5ktNLMfIvN+hN2XSydgQSVnMx9fUOdFbp8DGDA7hhzROAz4zhXZT+Sc+x/wHdChyHRfO+eyi9z+DmgR47xiUXQz8HeRf3fNrxOw0Pn9dhV1GPCpc25LkfvexRdz0df9hXOuoFiWeL5uiYEGSSSnWsA6/OaL4jZH/h0M3IzfnLEUv0ZzH3v+cn4CHAFcaWbvu8ifmeB3xpvZXOAKM1uG/5A9m/JlO+f+C2BmlwCfm1k/59z4IvmX4DdpFbcxiucH/zr3KuH+JviyK5GZHYdfkxwO3Ahk4V9XrJuwyuSc22Fm0/Gb9SZE/p3lnMutxhxFT2Wwo4THYv0DtjDy78+bDs2sTinT/jw/55yLbG2srj+Yq/p1S5yooJLTR/h9DoWRv5ZL8lvgJefc8/DzfqtD8B+ERa0CrsNvMhtrZgOKlhR+k8gM4H/4UWqvxRI08kF9H3C/mU2PfEB/BPQBfnLOFc8TrWXAGWZmxfJ2jjxWmq7AWufciF13mFmbYtN8DJyCf+0l2Y7fJFmeicBbZtYBOA2/PzCWHNHM50tgPzNru2stysza4fdDfRFFxljsGj1YdL/bURV4no+BS8ysbilrUdG+7ivMrFGRtagT8OXzZQUySQD0l0Jia2xmRxW7tMWXxDvAHDM73cwOMLPjzWy4me1aq1oOnGJmvzWzQ/H7mg4oaSaRkjsJ/yE6ptjO9Vfx+4aGAuOdc4UlPEV5JuP/cr02cnsSfg1wjpmdGMnfzcwett1H8tUq4fUfHnnsKfy+lsfM7Egza29mN+KL76EysiwHWplZXzNrZ2ZXR36mqHuBC8zsHjPrYGYdzezGIgM4VgO/Mz8SsdSRcM65d/H7WiYDP7H7ZsNocqwG2phZZ/OjA0v6Ltlr+M1pk8ws3fwIu0n4PwJeL2M5xMw5tw14H/hLZJmcQMXW+J4EGgLTzexoMzvIzPqY2a6yWw0cHvk/bVbKWtok/CbUCeZH83UDxgAzd629S/ipoBLb7/B/bRa9jIysMZyB/wB6Gr/GMB1ozy/b++8B/oPfF/IWfsTYpNJm5Jxbid/JfDpFSioyr2eBOvzyfaaYRP5Kfhy4NfIXby7QDb9W9gLwFX5/197ApiI/mlrC68+MPOf/Is9xMPB/kdfaG7jAOTe/jCwv4QvsH/gP9t8DdxWb5hX8vqPTI/N8E1/gu8r5LmB//CjH8r6TNAk/km1q0X0h0eQAXgRewRfbevYssF3/P+dGHn8jcvkB6FFszbKqXBH590N8IQyJ9Qmcc2vx/3d18Xk/xq/F74xM8jR+LWgR/nV1LeE5coE/AI3x//dzgPeK5JMEYPF5j0pNYmZP4UdG/b7ciUVEoqR9UFJh5r/02AH/3acLA44jIklGBSWVMQf/JchxzrmXgw4jIslFm/hERCSUNEhCRERCKW6b+Jo1a+batm0br6evlK1bt9KgQYOgYyQkLbvYLVu2jIKCAjp06FD+xLIbvd8qrrRlt2oVbNwI9erBYYdBSjTf2Ktiixcv/sk517y86eJWUG3btmXRokXxevpKyczMJCMjI+gYCUnLLnYZGRlkZWWF9vchzPR+q7iSlt3DD8PgwdCgAXzwAXTsGEw2M/s6mum0iU9EpAZ49VW49VZ/fcKE4MopFiooEZEk97//wUUXQWEh/PWvcP75QSeKjgpKRCSJbd0KPXrApk1w1lkwbFjQiaKnghIRSVLOweWXw9Kl0L49TJwItRLoUz+BooqISCwefBBeeAEaNYLZs2Gvkk5AE2IxFZSZHWxmeWY2MV6BRESk8j74oCl33OGvT5oEhx4abJ6KiHUN6gn8UYpFRCSkVqyAe+45DOdg+HA4O5rTiIZQ1AVlZr3xJ7Or7KmuRUQkTrZs8YMicnLq0KMHDIn5hCfhEdUXdc2sMXA3cDJwVRnTDQAGALRs2ZLMzMwqiFj1cnJyQpst7LTsYpeVlUVBQYGWWwXo/RabwkIYOrQjX3zRnP3330L//kt4662C8n8wpKI9ksQI/BGr1+x+MtXdOefGAmMB0tPTXVi/Aa5vp1ecll3smjRpQlZWlpZbBej9FpsRI2DhQj8Y4r77vuCMM35X/g+FWLkFFTnN8qlAp/jHERGRinjpJRg6FMxgyhRITd0WdKRKi2YNKgNoC3wTWXtqCKSYWQfnXOf4RRMRkWh89RVccon/3tN998Hpp0MybBmNpqDGAlOL3B6ML6yr4xFIRESil53tB0Vs3gy9esFttwWdqOqUW1DOuVwgd9dtM8sB8pxz6+MZTEREylZY6Necli2DI46AZ5/1m/iSRcyn23DODYtDDhERidHw4TBvHuy9tz9SRMOGQSeqWjrUkYhIApo1C+6+2x9bb+pUaNcu6ERVTwUlIpJgPv8c/vhHf/3BB6F792DzxIsKSkQkgWzatOtIEdCnD9x8c9CJ4kcFJSKSIAoKoG9f+O9/4aij4J//TK5BEcWpoEREEsRf/wrz58M++/h9UGlpQSeKLxWUiEgCmD4d7r8fUlL89bZtg04UfyooEZGQ+/RTf2ZcgIcfhpNPDjZPdVFBiYiE2MaNflBEbq4fuXf99UEnqj4qKBGRkNq5E3r3hlWroEsXGD06uQdFFKeCEhEJqdtvh1dfhebN/aCI1NSgE1UvFZSISAhNngwjR0Lt2jBjBuy/f9CJqp8KSkQkZD7+GK6KnLv8H/+Abt2CzRMUFZSISIisX+8HRWzbBldcAddcE3Si4KigRERCYscOuOgi+OYbOPZYeOKJmjUoojgVlIhISNxyC7zxBvzqV/Dii1C/ftCJgqWCEhEJgQkTYNQoqFPHl1OrVkEnCp4KSkQkYIsWwYAB/vrjj8MJJwSbJyxUUCIiAVq3Ds47D/LzYeDAX4pKVFAiIoHZvh0uuADWrIGuXeHRR4NOFC4qKBGRgNx0E7z9Nuy3n/8ybt26QScKFxWUiEgAxo3zw8jr1oWZM/3IPdmdCkpEpJq9//4vX8AdPdp/50n2pIISEalG338P55/v9z9de+0v53mSPamgRESqSX4+9OzpS6pbN/j734NOFG4qKBGRanL99fDee9C6Nbzwgv9SrpROBSUiUg3GjIGxY/3hi2bPhhYtgk4UfiooEZE4e+cduO46f33sWH92XCmfCkpEJI7WrvX7nXbsgBtugEsvDTpR4lBBiYjESV6eH7G3bh2cfDI89FDQiRKLCkpEJA6cg6uvhv/8B9q0gWnT/OnbJXoqKBGROHjiCRg/HlJT/aCIZs2CTpR4VFAiIlXszTfhxhv99WeegaOOCjZPolJBiYhUoW++8Uco37nTnyG3d++gEyUuFZSISBXZts2f22n9eujeHe6/P+hEiU0FJSJSBZzzJxv86CNo1w6mTIGUlKBTJTYVlIhIFRg1CiZOhAYN/KCIpk2DTpT4VFAiIpW0YAEMHuyvjx8PRxwRaJykoYISEamEVavgoougoADuuAN69Qo6UfJQQYmIVFBurh8UsWEDnHEG3H130ImSS1QFZWYTzex7M9tsZsvN7Kp4BxMRCTPn4Mor4ZNP4OCDYdIkDYqoatGuQd0PtHXONQbOAe4xMx2PV0RqrJEjYepUaNjQD4po0iToRMknqoJyzn3unMvfdTNyOTBuqUREQuzf/4bbbvPXn38eOnQINk+yivrQhWb2JNAPSAU+Bl4pYZoBwACAli1bkpmZWSUhq1pOTk5os4Wdll3ssrKyKCgo0HKrgDC+39aurc+gQV0oLKzDZZetpkmT1YQsIhDOZRcrc85FP7FZCnA8kAE86JzbUdq06enpbtGiRZUOGA+ZmZlkZGQEHSMhadnFLiMjg6ysLJYsWRJ0lIQTtvdbTg4cfzx89hmccw7MmgW1QjrULGzLrigzW+ycSy9vupgWrXOuwDm3EGgNXF3RcCIiicY56NfPl9Ohh/pNe2Etp2RR0cVbG+2DEpEa5P774cUXoXFjPyiiceOgEyW/cgvKzFqYWW8za2hmKWb2B6APsCD+8UREgvfyyzBkCJjB5MnQvn3QiWqGaAZJOPzmvNH4QvsauME5NzeewUREwmD5cujb12/iGzECzjwz6EQ1R7kF5ZxbD5xYDVlEREJl82bo0QOys+H88/2hjKT6aBefiEgJCgvhj3+EL7+Ejh39QWA1KKJ6aXGLiJRgxAiYM8cfIWL2bGjUKOhENY8KSkSkmLlzYdgwPyhiyhQ46KCgE9VMKigRkSK+/BIuucRfv/9+OO20YPPUZCooEZGIrCw491zYsgUuvBBuvTXoRDWbCkpEBD8o4pJLYMUK+M1v4Jln/CY+CY4KSkQEGDrUfyG3aVM/KKJBg6ATiQpKRGq8F1+Ee+7xw8inTYMDDgg6kYAKSkRquM8+g8su89cfeghOPTXYPPILFZSI1FibNvkjRWzd6g9ndOONQSeSolRQIlIjFRRAnz6wciV06gRjx2pQRNiooESkRrrzTn/q9mbN/IkH09KCTiTFqaBEpMaZNg0efBBSUuCFF6BNm6ATSUlUUCJSo3zyCVx+ub/+yCMQ0rOiCyooEalBNmzwgyK2bfMj9669NuhEUhYVlIjUCDt3wkUXwerVcPTRMHq0BkWEnQpKRGqEv/wFFiyAFi1g5kyoXz/oRFIeFZSIJL2JE+Hvf4fatf1RI1q3DjqRREMFJSJJ7aOPoH9/f/3RR+G3vw02j0RPBSUiSevHH/2giLw8uOoqGDQo6EQSCxWUiCSlHTv8OZ2+/RaOOw4ef1yDIhKNCkpEktLNN8Obb8K++/r9TvXqBZ1IYqWCEpGkM348PPYY1Knjy2m//YJOJBWhghKRpPKf//yyr+nJJ+H444PNIxWnghKRpPHDD3D++ZCfD1df7QdGSOJSQYlIUti+HXr1grVr/VDyf/wj6ERSWSooEUkKN9wA77wDrVrBjBlQt27QiaSyVFAikvCefhqeesqP1Js1C1q2DDqRVAUVlIgktHffhT/9yV8fPdofCFaSgwpKRBLWd99Bz57+S7nXXw/9+gWdSKqSCkpEElJ+vi+nH36AE0+EkSODTiRVTQUlIgnHOb9Z7/334de/9qdtr1Mn6FRS1VRQIpJwRo+GceP8OZ1mzYLmzYNOJPGgghKRhPL2235/E8A//wmdOwebR+JHBSUiCWPNGv9l3J074aaboG/foBNJPKmgRCQh5OXBeef5czydcgo8+GDQiSTeVFAiEnrO+QPALloEbdvCtGn+9O2S3FRQIhJ6jz0Gzz0HaWkwezbss0/QiaQ6qKBEJNQyM/3+JoBnn4Ujjww0jlSjcgvKzOqZ2Tgz+9rMtpjZEjM7vTrCiUjN9sMP9bjgAigogL/8xZ/CXWqOaNagagPfAicCewFDgOlm1jZ+sUSkpsvNhbvuOpyffoI//AHuvTfoRFLdyt3N6JzbCgwrctc8M1sFdAFWxyeWiNRkzkH//rBiRSMOPBCmTIGUlKBTSXWLeRyMmbUEDgE+L+GxAcAAgJYtW5KZmVnZfHGRk5MT2mxhp2UXu6ysLAoKCrTcYjB9emsmTz6I+vV3cuedH/PJJ1uDjpRwkuF31Zxz0U9sVgeYD6x0zg0sa9r09HS3aNGiSsaLj8zMTDIyMoKOkZC07GKXkZFBVlYWS5YsCTpKQnjtNb9Jr7AQhg//jLvuOjzoSAkpzL+rZrbYOZde3nRRj+Izs1rA88B24NpKZBMRKdH//gcXXeTLacgQ6Nbtp6AjSYCiKigzM2Ac0BLo6ZzbEddUIlLjbN0KPXrAxo1w1lkwfHjQiSRo0e6Dego4DDjVObctjnlEpAZyDq64ApYuhUMOgYkToZa+pVnjRfM9qDbAQOAo4Aczy4lcdJhGEakSf/sbTJ8OjRr5I0XstVfQiSQMohlm/jVg1ZBFRGqgf/0Lbr/dX584EQ47LNg8Eh5aiRaRwPz3v9Cnj9/EN3w4nHNO0IkkTFRQIhKILVv8oIisLP/vkCFBJ5KwUUGJSLUrLITLLoPPP/eb9J57ToMiZE96S4hItbvvPpg1yw+GmD0bGjcOOpGEkQpKRKrVvHlw111gBpMn+2HlIiXROSlFpNosWwZ9+/pBEffeC2ecEXQiCTOtQYlItcjOhnPPhc2boVevX4aWi5RGBSUicVdYCJde6tegDj/cnxnX9O1KKYcKSkTibvhweOkl2HtvPyiiYcOgE0kiUEGJSFzNng133+2HkU+dCgceGHQiSRQqKBGJmy++8Jv2AB54ALp3DzaPJBYVlIjERVaWHxSRkwO9e8PgwUEnkkSjghKRKldQABdf7I+1d9RRMG6cBkVI7FRQIlLl7roL5s+HffbxR4xISws6kSQiFZSIVKkXXvCHMkpJgWnToG3boBNJolJBiUiV+fRT6NfPXx85Ek45JdA4kuBUUCJSJTZu9KfNyM31I/f+/OegE0miU0GJSKXt3OlH6q1aBV26wJgxGhQhlaeCEpFKu+MOePVVaN4cZs6E1NSgE0kyUEGJSKVMmQIPPQS1a8OMGfDrXwedSJKFCkpEKmzJErjySn/9H/+Abt2CzSPJRQUlIhXy009+UMS2bXD55XDNNUEnkmSjghKRmO3cCRdeCF9/DcccA08+qUERUvVUUCISs1tugTfegJYt/aCI+vWDTiTJSAUlIjF5/nm/v6lOHXjxRWjVKuhEkqxUUCIStUWLoH9/f/2xx6Br12DzSHJTQYlIVNatg/POg/x8GDAABg4MOpEkOxWUiJRrxw644AJYswZOOAEefTToRFITqKBEpFw33ghvvw377ee/jFuvXtCJpCZQQYlImZ55Bp54AurW9SP29t036ERSU6igRKRUH3wAV1/trz/1FBx7bLB5pGZRQYlIib7/Hs4/H7Zvhz/9Ca64IuhEUtOooERkD9u3Q69e8N13/vh6jzwSdCKpiVRQIrKH66+Hd9+F1q39Kdzr1Ak6kdREKigR2c2YMf5Srx7MmgUtWgSdSGoqFZSI/Oydd+C66/z1sWMhPT3YPFKzqaBEBIC1a6FnT/+l3BtugD/+MehEUtOpoESEvDw/Ym/dOjjpJH+GXJGgRVVQZnatmS0ys3wzGx/nTCJSjZzzw8j/8x9o0wamTfOnbxcJWrRvw++Ae4A/AKnxiyMi1e3JJ/3RIlJT/aCI5s2DTiTiRVVQzrmZAGaWDrSOayIRqTZvveX3NwGMGwedOgWbR6Qo7YMSqaG+/dZ/GXfnThg8GPr0CTqRyO6qdEuzmQ0ABgC0bNmSzMzMqnz6KpOTkxPabGGnZRe7rKwsCgoKQrXc8vNrcf31nVi/vhHp6Rs57bSlZGa6oGPtQe+3ikuGZVelBeWcGwuMBUhPT3cZGRlV+fRVJjMzk7BmCzstu9g1adKErKys0Cw35/wQ8uXLoV07+Pe/m9K06YlBxyqR3m8VlwzLTpv4RGqYUaNg4kRIS4PZs6Fp06ATiZQsqjUoM6sdmTYFSDGz+sBO59zOeIYTkar1+ut+fxPA+PFwxBGBxhEpU7RrUEOAbcBtwCWR60PiFUpEqt7q1XDhhVBQALff7k/hLhJm0Q4zHwYMi2sSEYmb3Fzo0QM2bIDTT4cRI4JOJFI+7YMSSXLOwZVXwiefwMEHw+TJkJISdCqR8qmgRJLcww/D1KnQsKEfFNGkSdCJRKKjghJJYv/3f/CXv/jrEyZAhw7B5hGJhQpKJEmtXAm9e0NhIdx1F5x3XtCJRGKjghJJQjk5flDEpk1w9tkwdGjQiURip4ISSTLOweWXw2efQfv2/ku5tfSbLglIb1uRJPPAAzBjBjRuDHPm+H9FEpEKSiSJvPIK3HknmMGkSX4NSiRRqaCqQUZGBtdee23QMSTJrVgBF1/sN/HdfTecdVbQiUQqRwUF9OvXj7P02ywJbMsWOPdcyM72o/XuuCPoRCKVp4ISSXCFhf70GV9+6b/n9NxzGhQhyUFv43JkZ2czYMAAWrRoQaNGjTjxxBNZtGjRz49v2LCBPn360Lp1a1JTU+nYsSPPPvtsmc+5YMECmjRpwujRo+MdX2qAe+755QgRc+ZAo0ZBJxKpGiqoMjjnOPPMM1m7di3z5s3j448/plu3bpx88sl8//33AOTl5dG5c2fmzZvH559/zp///GcGDhzIggULSnzOGTNmcN555zF27FgGDRpUnS9HktDcuf47TmYwZQocdFDQiUSqTpWeUTfZvPHGGyxZsoT169eTmpoKwIgRI3jppZd4/vnnufXWW2nVqhW33HLLzz8zYMAAXn/9daZMmcIpp5yy2/ONHTuWW265hRkzZtC9e/dqfS2SfL76Ci65xF+/7z447bRg84hUNRVUGRYvXkxubi7Nmzff7f68vDxWrlwJQEFBAQ888ADTpk1j7dq15Ofns3379j1OtTx79mzGjBnDW2+9xfHHH19dL0GSVHa2HxSxZYs/r9Ou4+2JJBMVVBkKCwtp2bIlb7/99h6PNY58+3HkyJE8/PDDjBo1iiOOOIKGDRtyxx138OOPP+42/ZFHHsnSpUsZN24cxx13HGZWLa9Bkk9hIfTtC8uX+zPiPvus38QnkmxUUGXo3Lkz69ato1atWrRr167EaRYuXMjZZ5/NpZdeCvj9VsuXL6dJsXMaHHDAATz22GNkZGQwYMAAxo4dq5KSChk6FF5+GZo29YMjGjQIOpFIfGiQRMTmzZtZsmTJbpeDDjqIrl27cu655zJ//nxWrVrFe++9x9ChQ39eqzrkkENYsGABCxcu5KuvvuLaa69l1apVJc6jXbt2vPHGG/zrX/9i4MCBOOeq8yVKEpg504/aq1ULpk2DUv5uEkkKKqiIt99+m06dOu12ueWWW3jllVc4+eST6d+/P+3bt+fCCy9k2bJl7LfffgAMGTKEY445htNPP51u3brRoEED+vbtW+p8DjzwQDIzM5k/f75KSmLy2Wf++04Af/sbnHpqsHlE4k2b+IDx48czfvz4Uh8fNWoUo0aNKvGxvffem5kzZ5b5/JmZmbvdPvDAA/n2229jjSk12KZN/vQZW7f6wxnddFPQiUTiT2tQIiFXUAB9+vgTEHbqBE8/rUERUjOooERCbsgQ+Pe/oVkzmDUL0tKCTiRSPVRQIiE2fbo/v1NKir/epk3QiUSqT1IX1M6dOxkzZgwbNmwIOopIzD75xJ8ZF+Dvf4eTTgo2j0h1S9qC+vbbbznmmGO47rrruOCCCzRaThLKhg1+UERuLlx2GVx3XdCJRKpfUhbUnDlz6NixI59++ik7duzggw8+YOTIkUHHEonKzp3QuzesXg3p6TB6tAZFSM2UVAWVn5/PoEGDuPjii9myZQsFBQUA5ObmMnTo0N1OkyESVrfdBq/vsgn3AAAKYElEQVS9Bi1a+C/m1q8fdCKRYCRNQa1YsYIjjzySCRMmkJubu8fjzjlWrFgRQDKR6E2aBA8/DLVrw4wZsP/+QScSCU5SfFF34sSJDBo0iNzc3D32NdWpU4fGjRsza9Ysfve73wWUUKR8H30EV13lrz/6KOjtKjVdQhfU1q1b6d+/P3PmzClxrSktLY1jjz2WF154gX322SeAhCLRWb8ezjsP8vLgyitB57IUSeBNfEuXLqVDhw7MmjWrxHJKTU1l+PDhLFiwQOUkobZjB1x4IXzzDRx3HDzxhAZFiEACrkE553jqqacYPHgw27Zt2+PxevXq0bRpU+bOnUt6enoACUViM3gwZGbCr34FL74I9eoFnUgkHBKqoLKysrjkkkt44403SiyntLQ0fv/73zNhwoSfTygoEmbjx/v9TXXq+BF7kYPkiwgJVFAffPAB55xzDtnZ2eTn5+/xeFpaGo888gj9+/fXiQAlIXz44S/7mp54Ao4/Ptg8ImET+oIqLCzkgQce4J577ilxral+/frsu+++zJs3jw4dOgSQUCR269b5QRH5+b6k+vcPOpFI+IS6oH788Ud69erF4sWLS92k17NnT8aMGUNqamoACUVit3079OoFa9dC165QyqnGRGq8QEfxbdiwgY8++qjEx15//XUOPfRQ3n///T1G6dWqVYsGDRowbtw4JkyYoHKShHLDDbBwIbRq5b+MW7du0IlEwinQgrrmmmvo2rUrK1eu/Pm+nTt3ctttt3HWWWexadMmduzYsdvPpKWlceihh/Lpp5/Su3fv6o4sUin//Cc89ZQfqTdzph+5JyIlC6ygli9fzty5c9m+fTtnn30227dvZ82aNRx77LE89thjJW7SS01N5corr+Tjjz+mXbt2AaQWqbj33oM//clff+opOOaYYPOIhF1U+6DMrCkwDugO/ATc7pybXJkZ33bbbezYsYPCwkJWr15Njx49WLhwIbm5uT8f5PXnkLVrk5aWxuTJkznzzDMrM1uRQOzYUYuePf3+p+uu++U8TyJSumgHSTwBbAdaAkcBL5vZJ865zysy0y+++IL58+f/XETbtm3j9ddfL3X4eMeOHZk1axatWrWqyOxEApWXB6tWNWDbNjjxRH8wWBEpn5V3Ij8zawBsAg53zi2P3Pc8sNY5d1tpP9eoUSPXpUuXEh9bunQpGzduLDdcrVq1aN26NW3btq3S7zZlZWXRpEmTKnu+mkTLbnfO+fM3lXbZvh3WrFkCQL16R9Gli/9SrkRH77eKC/Oye/PNNxc758o91E80a1CHADt3lVPEJ8CJxSc0swHAAPBHEc/KytrjybZt28amTZvKnWlKSgpt27alYcOGZGdnRxEzegUFBSVmk/Il27JzDgoKrMIX56L7wyklpZCDDspm61ad2TkWyfZ+q07JsOyiKaiGwOZi92UDjYpP6JwbC4wFSE9PdyWdILB79+7lnpepTZs2LFq0iGbNmkURL3aZmZlkZGTE5bmTXdiWXX4+ZGWVfcnOLv2xEsbixCQlBfbaC5o0Kf0yY0YGZlksWfJx1bzoGiRs77dEEuZlF+0WsWgKKgcofmC7xsCWGDOxePFiFi5cuMc5m4r78ccfWbp0KSeddFKss5AEk5dXfsGUVTZ5eZWbf0oK7L23L5LyiqakS4MG5R95fMECn1VEYhNNQS0HapvZwc65Xas+RwIxD5C4+eabyYviE2Xbtm307NmTZcuW0bx581hnI9XEudgLpnjRlDAuJia1a/9SMEUv0ZZNWppObSESVuUWlHNuq5nNBO42s6vwo/jOBU6IZUbvv/8+H374YblrT7ts2bKFG2+8kYkTJ8YyG4mBc5CbG92msF2Xb7/tTEHBL7eLfY86ZnXqlFww0ZZNaqoKRiRZRTvM/BrgGeBHYANwdaxDzG+66aYSTyxYr1496tWrR35+PnXr1uWQQw7h6KOPpkuXLtrEVw7nYOvW2Pa5FL/s3BnrXHff2lu3bvkFU1bR1K+vghGRkkVVUM65jUCPis7k3Xff5b333qNRo0YUFBRQUFBAu3bt6Ny5M0cffTS/+c1v6NixIy1atKjoLBKSc5CTU/Ed/FlZUOw7zTGrXz+2fS4rVy7mlFO6/Fw29etXzbIQESmuWo5mvs8++3DfffdxxBFHcPjhh9OmTZukOGdTYWH5BVNe0RQWVi5DWlrs+12KTh/r2VszM7fQvn3lMouIRKNaCqp9+/bcfvvt1TGrmBQWwpYtFd/Bn51d+YJp0CD2/S5Fp9GRsEUkWYX6fFDlKSiAzZtj3+/yww/HkZfnfzbKMRulatiwYvtedt2vowqIiJQs0IIqKNizWGIpms3Fvz4ctV92nDRqFNsmseK3ayd0xYuIhFfcPl7XrYO77iq7YLbE/FXfPe21V+z7Xr766n3+8IfjaNxYBSMiElZx+3heswZGjCh7GrPdyyXWomnUyB8JIFbZ2Xk0bVqx1yUiItUjbgXVogVcc035BVMr0HP6iohIWMWtoPbfH4YOjdezi4hIstP6i4iIhJIKSkREQkkFJSIioaSCEhGRUFJBiYhIKKmgREQklFRQIiISSiooEREJJRWUiIiEkgpKRERCyVxlT4hU2hObrQe+jsuTV14z4KegQyQoLbuK0XKrGC23igvzsmvjnGte3kRxK6gwM7NFzrn0oHMkIi27itFyqxgtt4pLhmWnTXwiIhJKKigREQmlmlpQY4MOkMC07CpGy61itNwqLuGXXY3cByUiIuFXU9egREQk5FRQIiISSiooEREJJRUUYGYHm1memU0MOkvYmVk9MxtnZl+b2RYzW2JmpwedK6zMrKmZzTKzrZFldnHQmcJO77GqkQyfayoo7wngw6BDJIjawLfAicBewBBgupm1DTBTmD0BbAdaAn2Bp8ysY7CRQk/vsaqR8J9rNb6gzKw3kAUsCDpLInDObXXODXPOrXbOFTrn5gGrgC5BZwsbM2sA9AT+6pzLcc4tBOYClwabLNz0Hqu8ZPlcq9EFZWaNgbuBm4LOkqjMrCVwCPB50FlC6BBgp3NueZH7PgG0BhUDvcdik0yfazW6oIARwDjn3JqggyQiM6sDTAKec859FXSeEGoIbC52XzbQKIAsCUnvsQpJms+1pC0oM8s0M1fKZaGZHQWcCjwSdNYwKW+5FZmuFvA8fv/KtYEFDrccoHGx+xoDWwLIknD0Hotdsn2u1Q46QLw45zLKetzMbgDaAt+YGfi/dlPMrINzrnPcA4ZUecsNwPwCG4ff8X+Gc25HvHMlqOVAbTM72Dm3InLfkWhTVbn0HquwDJLoc63GHurIzNLY/a/bwfj/2Kudc+sDCZUgzGw0cBRwqnMuJ+g8YWZmUwEHXIVfZq8AJzjnVFJl0HusYpLtcy1p16DK45zLBXJ33TazHCAvEf8Tq5OZtQEGAvnAD5G/0gAGOucmBRYsvK4BngF+BDbgPyhUTmXQe6ziku1zrcauQYmISLgl7SAJERFJbCooEREJJRWUiIiEkgpKRERCSQUlIiKhpIISEZFQUkGJiEgoqaBERCSU/h9r5scSI6iwhAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(z, leaky_relu(z, 0.05), \"b-\", linewidth=2)\n",
    "plt.plot([-5, 5], [0, 0], 'k-')\n",
    "plt.plot([0, 0], [-0.5, 4.2], 'k-')\n",
    "plt.grid(True)\n",
    "props = dict(facecolor='black', shrink=0.1)\n",
    "plt.annotate('Leak', xytext=(-3.5, 0.5), xy=(-5, -0.2), arrowprops=props, fontsize=14, ha=\"center\")\n",
    "plt.title(\"Leaky ReLU activation function\", fontsize=14)\n",
    "plt.axis([-5, 5, -0.5, 4.2])\n",
    "\n",
    "save_fig(\"leaky_relu_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['deserialize',\n",
       " 'elu',\n",
       " 'exponential',\n",
       " 'get',\n",
       " 'hard_sigmoid',\n",
       " 'linear',\n",
       " 'relu',\n",
       " 'selu',\n",
       " 'serialize',\n",
       " 'sigmoid',\n",
       " 'softmax',\n",
       " 'softplus',\n",
       " 'softsign',\n",
       " 'tanh']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[m for m in dir(keras.activations) if not m.startswith(\"_\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['LeakyReLU', 'PReLU', 'ReLU', 'ThresholdedReLU']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[m for m in dir(keras.layers) if \"relu\" in m.lower()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "leaky_relu = keras.layers.LeakyReLU(alpha=0.2)\n",
    "layer = keras.layers.Dense(10, activation=leaky_relu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.advanced_activations.LeakyReLU at 0x10dc0b9b0>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer.activation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train a neural network on Fashion MNIST using the Leaky ReLU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train_full, y_train_full), (X_test, y_test) = keras.datasets.fashion_mnist.load_data()\n",
    "X_train_full = X_train_full / 255.0\n",
    "X_test = X_test / 255.0\n",
    "X_valid, X_train = X_train_full[:5000], X_train_full[5000:]\n",
    "y_valid, y_train = y_train_full[:5000], y_train_full[5000:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=leaky_relu),\n",
    "    keras.layers.Dense(100, activation=leaky_relu),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"sgd\",\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "55000/55000 [==============================] - 3s 64us/sample - loss: 1.3979 - accuracy: 0.5948 - val_loss: 0.9369 - val_accuracy: 0.7162\n",
      "Epoch 2/10\n",
      "55000/55000 [==============================] - 3s 55us/sample - loss: 0.8333 - accuracy: 0.7341 - val_loss: 0.7392 - val_accuracy: 0.7638\n",
      "Epoch 3/10\n",
      "55000/55000 [==============================] - 3s 58us/sample - loss: 0.7068 - accuracy: 0.7711 - val_loss: 0.6561 - val_accuracy: 0.7906\n",
      "Epoch 4/10\n",
      "55000/55000 [==============================] - 3s 55us/sample - loss: 0.6417 - accuracy: 0.7889 - val_loss: 0.6052 - val_accuracy: 0.8088\n",
      "Epoch 5/10\n",
      "55000/55000 [==============================] - 3s 56us/sample - loss: 0.5988 - accuracy: 0.8019 - val_loss: 0.5716 - val_accuracy: 0.8166\n",
      "Epoch 6/10\n",
      "55000/55000 [==============================] - 3s 58us/sample - loss: 0.5686 - accuracy: 0.8118 - val_loss: 0.5465 - val_accuracy: 0.8234\n",
      "Epoch 7/10\n",
      "55000/55000 [==============================] - 3s 56us/sample - loss: 0.5460 - accuracy: 0.8181 - val_loss: 0.5273 - val_accuracy: 0.8314\n",
      "Epoch 8/10\n",
      "55000/55000 [==============================] - 3s 56us/sample - loss: 0.5281 - accuracy: 0.8229 - val_loss: 0.5108 - val_accuracy: 0.8370\n",
      "Epoch 9/10\n",
      "55000/55000 [==============================] - 3s 60us/sample - loss: 0.5137 - accuracy: 0.8261 - val_loss: 0.4985 - val_accuracy: 0.8398\n",
      "Epoch 10/10\n",
      "55000/55000 [==============================] - 3s 53us/sample - loss: 0.5018 - accuracy: 0.8289 - val_loss: 0.4901 - val_accuracy: 0.8382\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ELU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def elu(z, alpha=1):\n",
    "    return np.where(z < 0, alpha * (np.exp(z) - 1), z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure elu_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl8FeW9x/HPLwnKKiBorCJgXVDrwhWq1bqkalUWt7q2asUNKtqWqq0b9Gql2ipWqApKixcFF1CwKgh41XvABaWgIFAFRECQfTlAgARInvvHc4CQ9SSZZOac832/XueVyTxzZn5nGM43sz1jzjlERESiJivsAkRERMqjgBIRkUhSQImISCQpoEREJJIUUCIiEkkKKBERiSQFlIiIRJICSkREIkkBJSnDzIab2bg0Wk6WmT1rZuvMzJlZXl0vs5Ja6uUzJ5bV0sxWmdnh9bG86jKzV83szrDrEDD1JJGezGw4cH05TZ86536UaG/tnOtewftjwBzn3O2lxvcAnnLONQ204OSW3Ry/zcZTaTmVLL87MBbIA74B1jvnttflMhPLjVHqc9fXZ04s6zH8tndDXS+rnGWfCdwFdAIOBm5wzg0vNc3xwGTgMOfcxvquUfbICbsAqVPvAteVGlfnX4B1pb6+LOrxS+kIYIVz7uN6Wl6F6uszm1lj4GbgwvpYXjmaAnOAFxKvMpxzs83sG+Ba4Ol6rE1K0SG+9FbonFtZ6rW+rhdqZheY2QdmtsHM1pvZJDM7pkS7mdmdZrbAzArNbJmZPZJoGw6cBdyWOOzlzKz9rjYzG2dmPROHiLJLLfclM3szmTqSWU6J+exrZgMTyywws0/M7PQS7TEzG2xmD5vZWjNbbWYDzKzC/1+J5T8BtE0se3GJeT1Vetpd9SSzrJqs3+p+5pp+bqAr4ICPylknnczsPTPbZmZfm9mZZnalmZWZtqacc2875+5zzr0GFFcy6ZvAz4NartSMAkrqQhNgIHAy/vDVRuAtM9sn0f4w0A94BPgBcAWwNNH2W2Aq8D/A9xKvXW27vAo0B366a4SZNQUuBkYmWUcyy9nlUeAq4Ebgv4DZwEQz+16Jaa4BdgKnAbcDfRLvqchvgT8ByxLL/mEl05ZW1bJqu34huc+cTC2lnQHMcKXOLZjZD4EPgP8DTgA+AR4E7k98FkpNf5+Z5VfxOqOSOqoyDTjZzBrVYh5SSzrEl94uMLP8UuOeds7dXZcLdc6NKfm7md0AbML/h58J/A7o45x7LjHJ1/gvTZxzG81sO7DVObeygvlvMLO38V+OExOjL8F/Ub5ZYroK63DOfVjVchLvaQLcCtzsnBufGPcr4GzgNqBvYtL/OOf+mBieb2a3AOcAL1fwGTaa2WagqLLlV6DCZSWCutrr18xq8pmr/bmBdsDycsY/DrzlnOufWN5LwFvAFOfc++VM/wwwuoJl7PJdFe2VWQ40wJ+nWliL+UgtKKDS2xSgZ6lx9XES/HDgIeAU4AD8nnoW0BZ/Dmxf4L1aLmYk8LyZNXbObcWH1RjnXEGSdSTrcPwX1e7DTM65IjObChxbYrovSr1vOXBgNZZTHZUt61hqv36T/cxV1VKeRsCqkiPM7CD8ntVPSozejv+3KrP3lKhnPVCXh6u3JX5qDypECqj0ttU593UN37sJfxittBb4Q2WVGYc/dNUL/1fsTuA/wD6Vvamaxifme7GZvQecC5xfz3WUPEy1o5y2mhxCLwas1LgGpX4Palk1Ufqy3+rWshZoWWrcrvOT00uM6wDMc859WN5MzOw+4L7KS6WLc+6DKqapyP6Jn2tq+H4JgAJKKjIP6GpmVup8wUmJtnKZWSvgaKC3c+7/EuNOYs+29iVQiD8MtKCC2WwHsitoA8A5V2hmr+L3nFoDK4FYNepIajn4wzvbgR8nhjF/ccapwEtVvLcm1uDPC5V0IrA4yfcHsX7r8jN/DvQoNa4FPtiKEstqhj/3VNmhz7o+xHcc8J1zblWVU0qdUUClt30Th09KKnLO7fqrcD8z61iqPe6cWwwMwZ/0ftLM/gEU4K/A+jlwUSXL3ID/K/kWM1sKHAI8ht97wTm32cwGAY+YWSH+MGQroJNzbkhiHovx56vaA/n4+4PKu+JqJP5Q1mHAy6WmqbSOZJfjnNtiZkOAv5rZWmAR/hxPLjC4kvVQU+8DA83sIvwfAr2AQ0kyoGq6fkvNoy4/86TEfFs559Ylxs3E7zXea2Yv4v+dVgBHmNmRzrkyQVvTQ3yJc3RHJH7Nwl9F2RH/b/9tiUnPSNQqIdJVfOntXPx/9JKvz0u0n5H4veRrAIBz7hvgTOBI4B38VU1XA1c45yZUtMDEF/xV+Cux5uDvI+mH/6t+l3uBvybGfwmMAdqUaB+A/wv+P/g9iorOGX2A/yv5WPa+ei/ZOpJdzt3AKPyVbzMT87zAObeigulr47kSr4+AzcDr1ZxHEOu3Tj6zc242e7alXeMW4feYbgVm4T/zufh/t6DvEevMnm29Ef5Kwc/xV1QCYGYNgUuBfwS8bKkm9SQhIvXKzC4ABgHHOueKwq6nNDO7DbjYOXde2LVkOu1BiUi9cs5NxO/Rtqlq2pDsAH4ddhGiPSgREYko7UGJiEgkKaBERCSSQr/MvHXr1q59+/Zhl1HGli1baNKkSdhlpBSts+TNmzePoqIijj22dMcMUpFU276WLIG1ayE7Gzp0gEYh9EkR1XU2Y8aMtc65A6qaLvSAat++PdOnT696wnoWi8XIy8sLu4yUonWWvLy8POLxeCS3/ahKpe3rj3+Ehx6Chg3h3Xfhxz8Op46orjMzW5LMdDrEJyISoKef9uGUnQ2jR4cXTulAASUiEpBXX4VfJy5QHzoULgzrsYxpQgElIhKA99+Ha68F5+Dhh+HGG8OuKPUFGlBmNtLMVpjZJjObb2Y3Bzl/EZEo+vxzuOQS2L4dfvMbuOeesCtKD0HvQT0CtHfO7YfvULS/mXUKeBkiIpGxcCF06QKbN8NVV8ETT4CVfmCK1EigAeWcm+uc29UZp0u8Dg9yGSIiUbFqFZx/vv957rnw/POQpRMngQn8MnMzG4x/3ksjfC/Bb5czTU8ST3rNzc0lFosFXUat5efnR7KuKNM6S148HqeoqEjrqxqitn1t3ZpNnz4dWbiwGUceuZnf/W4mU6dGq+/bqK2z6qqTvvhKPNwsD/irc670Uzd369y5s4vivSBRvX8gyrTOkrfrPqiZM2eGXUrKiNL2VVgI3brBe+/B4YfDRx9Bbm7YVZUVpXVWkpnNcM51rmq6OtkZdc4VJR7V3Ab/jBcRkbRQXAzXX+/DKTcXJk2KZjilg7o+WpqDzkGJSJpwDvr0gVGjoFkzmDDB70FJ3QgsoMzsQDO72syamlm2mZ2Pfzz4e0EtQ0QkTH/5Czz5JOyzD/zrX/Bf/xV2RektyIskHP5w3jP44FsC9HHOvRngMkREQjFsGNx3n7+EfORIOPvssCtKf4EFlHNuDXBWUPMTEYmKN9+Enj398FNPwRVXhFtPptAV+yIilfjoI38DbnEx9OsHvXuHXVHmUECJiFRg7lzo3h0KCuCWW+DBB8OuKLMooEREyrF0KVxwAcTjcPHFMHiwujCqbwooEZFS1q3zXRgtWwannw4vvww5oT/eNfMooEREStiyxR/W+/JLOO44f4FEGI9rFwWUiMhuO3b4CyI++QTatoWJE6Fly7CrylwKKBERfC8Rt9wC48dDq1a+C6NDDgm7qsymgBIRAe691z8uo3FjH1JHHx12RaKAEpGM98QT8Ne/+gshxoyBU04JuyIBBZSIZLgXX4Q77vDDzz3nLy2XaFBAiUjGeucd6NHDDw8YANddF2o5UooCSkQy0r//DT/7GezcCXfe6V8SLQooEck48+dD167+nqdrr4VHHw27IimPAkpEMsqKFb6XiLVr/fmm556DLH0TRpL+WUQkY2zc6ENp8WI4+WR49VVo0CDsqqQiCigRyQgFBb7T1y++gA4d/L1OTZuGXZVURgElImmvqAiuuQYmT4aDD/a9RLRuHXZVUhUFlIikNefgtttg7Fho3tz3r9euXdhVSTIUUCKS1v70J3j2Wdh3X3jrLTj++LArkmQpoEQkbT37LDzwgL9K75VX4Iwzwq5IqkMBJSJpaexY6N3bDw8ZApdcEm49Un0KKBFJO5Mnwy9+AcXF/hBfz55hVyQ1oYASkbQyaxZcdBEUFvo9qL59w65IakoBJSJpY9EifyPupk1w+eXw97+DWdhVSU0poEQkLaxZ47swWrkSfvITGDkSsrPDrkpqQwElIikvP993/rpgAXTsCK+/7i8rl9SmgBKRlLZ9O1x2GUyfDocdBhMm+BtyJfUpoEQkZRUXww03+AcPHnCA/3nQQWFXJUFRQIlISnIO7roLXnrJd/o6YQIccUTYVUmQFFAikpIGDIAnnvCPyxg7Fjp1CrsiCZoCSkRSzvPPwx/+4IdfeAF++tNw65G6oYASkZQyfjzcdJMfHjQIrr463Hqk7gQWUGa2r5kNM7MlZrbZzGaaWZeg5i8i8skncMUV/vlO994Lv/lN2BVJXQpyDyoHWAqcBTQH+gKjzax9gMsQkQy1ZEljunWDbdvgxhvhz38OuyKpazlBzcg5twV4oMSocWa2COgELA5qOSKSeZYtgz/84QTWr4fu3f1jNNSFUfqrs3NQZpYLHAXMratliEj627DB96+3enVDTjsNRo2CnMD+tJYoq5N/ZjNrALwIPO+c+6qc9p5AT4Dc3FxisVhdlFEr+fn5kawryrTOkhePxykqKtL6qkJhYRZ33XUic+c259BDN3P33bOYNm1n2GWljFT/Pxl4QJlZFjAC2A7cXt40zrmhwFCAzp07u7y8vKDLqLVYLEYU64oyrbPktWjRgng8rvVViZ07fRdGc+ZAmzYwYMAcLrro9LDLSimp/n8y0IAyMwOGAblAV+fcjiDnLyKZwTn41a/gzTehZUuYNAlWry4MuyypZ0GfgxoCHANc6JzbFvC8RSRD9OsHw4ZBo0b+vqdjjw27IglDkPdBtQN6AR2BlWaWn3hdE9QyRCT9Pfmkv4Q8OxtGj4ZTTw27IglLkJeZLwF04aeI1NioUfDb3/rhf/7TX1IumUtdHYlIJLz7Llx3nT//9Je/QI8eYVckYVNAiUjoPvsMLr0UduyAPn32dAQrmU0BJSKhWrgQunTxj23/+c/h8cfVS4R4CigRCc2qVXDeebB6tX9kxvDhkKVvJUnQpiAiodi0ye85ffONf9jgmDGwzz5hVyVRooASkXpXWOjPOX3+uX9M+9tvQ7NmYVclUaOAEpF6VVTkr9Z7/3046CB45x048MCwq5IoUkCJSL1xzt/n9OqrsN9+MGECHHZY2FVJVCmgRKTePPwwPP20P9f0xhvQsWPYFUmUKaBEpF7885/Qt6+/hPyllyCFO9mWeqKAEpE698Yb0KuXHx482D9GQ6QqCigRqVMffghXXw3FxfDf/+0foyGSDAWUiNSZOXPgwguhoAB69vQBJZIsBZSI1IklS+D88yEe9/c8DR6sLoykehRQIhK4tWt9OC1fDmee6S+KyM4OuypJNQooEQnUli3+OU7z5sHxx/sLJBo2DLsqSUUKKBEJzI4dcMUV8Omn0K4dTJwILVqEXZWkKgWUiASiuBhuusn3DtG6te/C6OCDw65KUpkCSkQCcc89MGIENGkC48fDUUeFXZGkOgWUiNTa3/4Gjz0GOTn+sRknnxx2RZIOFFAiUisvvgh33umHhw/3V++JBEEBJSI1NmkS9Ojhhx9/HK65JtRyJM0ooESkRqZN833q7dwJv/893HFH2BVJulFAiUi1zZsH3br5e55++Uv4y1/CrkjSkQJKRKpl+XJ/nmntWujSxT9GI0vfJFIHtFmJSNLicbjgAt/P3imn+CfjNmgQdlWSrhRQIpKUbdvgootg9mw4+mh/r1OTJmFXJelMASUiVSoq8lfoffABHHKIv3qvVauwq5J0p4ASkUo5B717w+uv+371Jk6Etm3DrkoygQJKRCr14IMwdKjvkfytt+C448KuSDKFAkpEKjRkiA+orCwYNQpOPz3siiSTKKBEpFyvvQa33eaHn33WXyAhUp8UUCJSRizmL4pwDvr3h5tvDrsiyUSBBpSZ3W5m082s0MyGBzlvEakfM2fCxRfD9u1w++1w331hVySZKifg+S0H+gPnA40CnreI1LFvvvG9Q2zaBFdeCQMHglnYVUmmCjSgnHNjAcysM9AmyHmLSN1avdp3YbRyJZx9NrzwAmRnh12VZLKg96CSYmY9gZ4Aubm5xGKxMMqoVH5+fiTrijKts+TF43GKioois762bs3mjjtO5Ouv9+PIIzdzxx0zmTq1KOyy9qLtq/pSfZ2FElDOuaHAUIDOnTu7vLy8MMqoVCwWI4p1RZnWWfJatGhBPB6PxPravh26d/c9lB9+OHzwQTNyc88Iu6wytH1VX6qvM13FJ5LBiov9Awf/93/hwAN9F0a5uWFXJeIpoEQylHP+IYMvvwxNm8KECX4PSiQqAj3EZ2Y5iXlmA9lm1hDY6ZzbGeRyRKT2Hn0UBg3yj8v417/gpJPCrkhkb0HvQfUFtgH3ANcmhvsGvAwRqaX/+R+45x5/CfnIkXDOOWFXJFJW0JeZPwA8EOQ8RSRY48bBLbf44UGD/P1OIlGkc1AiGWTqVB9IRUVw//3w61+HXZFIxRRQIhniP/+Bbt38k3FvugkeeijsikQqp4ASyQBLl/peIjZs8L2SP/OMujCS6FNAiaS59evhggtg2TL/PKdXXoGcUG7RF6keBZRIGtu6FS680B/e+8EP4M03oZG6cZYUoYASSVM7d8JVV8HHH8Ohh8LEidCyZdhViSRPASWShpyDnj39JeX77++7MGqj5wtIilFAiaSh++/3N+M2agTjx8Mxx4RdkUj1KaBE0sygQfDII/5ZTq+9Bj/6UdgVidSMAkokjbzyCvTp44efew66dg23HpHaUECJpIl334Vf/tIPP/ronmGRVKWAEkkDM2bApZfCjh3+ERp33RV2RSK1p4ASSXELFkCXLpCfD9dcA489pl4iJD0ooERS2MqVvgujNWvgvPP8eacs/a+WNKFNWSRFbdzouzBatAh++EMYMwb22SfsqkSCo4ASSUEFBXDJJTBrFhx5pL/XqWnTsKsSCZYCSiTFFBXBdddBLAbf+x688w4ccEDYVYkETwElkkKcg9/8xt+Au99+vn+99u3DrkqkbiigRFLIn/8MgwfDvvv6nslPOCHsikTqjgJKJEX84x/Qr5+/Su+ll+Css8KuSKRuKaBEUsC//gW/+pUfHjwYfvazcOsRqQ8KKJGImzIFrr4aiovhwQehV6+wKxKpHwookQibPRsuuggKC/0eVL9+YVckUn8UUCIRtXix7yVi40Z/SO+pp9SFkWQWBZRIBK1d68NpxQp/McSLL/rnO4lkEgWUSMTk50O3bjB/Ppx4IrzxBjRsGHZVIvVPASUSITt2wOWXw7Rp/gbcCROgefOwqxIJhwJKJCKKi+HGG2HSJN910Tvv+K6MRDKVAkokIu6+G0aOhCZN4O23fSewIplMASUSAQMG+FeDBvD669C5c9gViYRPASUSshEj4Pe/98PPPw8//Wm49YhEhQJKJEQTJvjzTgBPPAE//3m49YhESaABZWb7m9nrZrbFzJaY2S+CnL9IOtm6NZvLL4edO/35pz59wq5IJFpyAp7f08B2IBfoCIw3s1nOubkBL0ckpW3dCt9805SiIrj+enjkkbArEokec84FMyOzJsAG4Djn3PzEuBHAd865eyp6X7NmzVynTp0CqSFI8XicFi1ahF1GStE6S05BAUybNhPnYP/9O3LccerCKBnavqovquts8uTJM5xzVV4KFOQe1FHAzl3hlDALKPPUGjPrCfQEaNCgAfF4PMAyglFUVBTJuqJM66xqO3dmsWBBU5yDrCzHIYdsZOPGYP5ITHfavqov1ddZkAHVFNhUatxGoFnpCZ1zQ4GhAJ07d3bTp08PsIxgxGIx8vLywi4jpWidVS4e9/3qbd8OTZvm0b79Rr744vOwy0oZ2r6qL6rrzJI8ZBBkQOUD+5Uatx+wOcBliKSkjRuha1f44gvo0AFatYItW7TnJFKZIK/imw/kmFnJ+99PBHSBhGS0DRv8vU1Tp0Lbtr4LowYNwq5KJPoCCyjn3BZgLPAnM2tiZj8GLgZGBLUMkVSzdi2ccw78+99w2GEwebIPKRGpWtA36vYGGgGrgZeBW3WJuWSq1avh7LPh8899v3qTJ/seykUkOYHeB+WcWw9cEuQ8RVLRwoXQpQssWABHHw3vv6+eyUWqS10diQRs2jQ49VQfTh07QiymcBKpCQWUSIDeegvy8mDNGjjvPJgyBXJzw65KJDUpoEQC4Bz8/e9wySWwbRvccAOMGwfNytwFKCLJUkCJ1NK2bdCjB/z2t/6puH/8IwwbpkvJRWor6M5iRTLKkiXws5/BZ59B48Y+mK6+OuyqRNKDAkqkhiZNgmuv9fc6ff/7/km4J5wQdlUi6UOH+ESqqbAQ7rwTLrjAh9P55/sbcRVOIsHSHpRINcyb5596+/nnkJ0NDz0Ef/iDHxaRYCmgRJJQVARPPQX33ecfNvj978NLL8Epp4RdmUj6UkCJVOHLL+Gmm3xnrwDXXefDar/SffeLSKB0DkqkAoWF0L+/7w1i6lQ4+GB44w144QWFk0h90B6USDnGj4c+feDrr/3vN98Mjz0GEXx6tkjaUkCJlLBgAfzudz6gAI45xh/OO/vscOsSyUQ6xCcCrFwJt90Gxx7rw6lZM/jb32DWLIWTSFi0ByUZLR6HAQPgiSf81XlZWb4fvYcfhoMOCrs6kcymgJKMtHYtDBoETz4JGzf6cRdfDH/+M/zgB+HWJiKeAkoyynff+b2lIUP8HhPAT37ig+nUU8OtTUT2poCSjDBtGgwcCK++Cjt3+nFdu8L998Npp4Vbm4iUTwElaWvLFnjtNXjmGfjkEz8uOxuuvBLuvhtOOinc+kSkcgooSSvOwaef+sdejBoFmzf78S1bQs+e/kq9Qw8Nt0YRSY4CStLCt9/C6NHw3HO+a6JdTjsNbrzRP6OpSZPw6hOR6lNAScpavNgfwnv1VX+OaZcDD4Trr/fBdPTRoZUnIrWkgJKU4Zy/cXbCBBg7FqZP39PWuDF06wa/+IX/qceti6Q+BZRE2oYN8O67PpQmToQVK/a0NWkC3bvDFVdAly4+pEQkfSigJFI2boQPP4TJk2HKFL+XVFS0p/3gg/2TbLt18z8VSiLpSwEloXEOlizx54+mTvWhNGsWFBfvmSYnB846y+8hdekCxx8PZuHVLCL1RwEl9cI534vDF1/4vaJp0/xrzZq9p2vQAH70Ix9KZ54JP/6x77hVRDKPAkoCt2kTzJkDs2fv/dqwoey0rVvDD38IJ58MZ5zhuxvSYTsRAQWU1FBhIXzzjX9+0q7XtGknsnYtLF1a/nv2398fouvUyQfSySdD+/Y6ZCci5VNASbk2bfI3vy5dWvbn4sV+uOS5Iq8lAPvs45+rdPzxe14nnADf+57CSESSp4DKIEVFsG4drFq192v1av9z5UpYtsyHz6ZNlc8rKwu+/3048sg9r23bvuCyy06gXTvdhyQitaeASkGFhb6PuQ0b/Gv9+sp/btjgL0ZYs6a8vZ7yNWoEbdv6fuvK+3nYYX5PqaRYbD1HHBH85xWRzBRIQJnZ7UAP4HjgZedcjyDmm4qcgx07YNs2/yooqHp461bIz/ehs3nznuGKxu3YUfP6WraE3Nw9r4MO2vv3Qw7xAbT//jocJyLhCmoPajnQHzgfaFSdNxYWwvz5/vBT6Vdxcfnjq2qrqn3Hjj2v7dv3/rlr+LvvjmXQoKqn2zW8K3AKCpLfS6mpnBx/6XXLlj5IWrbce7i8ca1a+T7qSu/1iIhEVSAB5ZwbC2BmnYE21XnvnDnz6NAhr9TYK4HewFagaznv6pF4rQUuL6f9VuAqYClwXTntdwIXAvOAXuW09wXOBWYCfcppfxg4DfgYuK9Ma3b2QBo37khW1rsUFPQnK4vdr+xsOP74ZznggA6sW/cW8+Y9TnY2e71uvXUE7dodyowZo5g4cUiZ9rFjX6N169YMHz6c4cOHs337nvNJAG+//TaNGzdm8ODB/P3vo8vUF4vFABgwYADjxo3bq61Ro0ZMmDABgIceeoj33ntvr/ZWrVoxZswYAO69916mTp26uy0ej3PccccxcuRIAPr06cPMmTP3ev9RRx3F0KFDAejZsyfz58/fq71jx44MHDgQgGuvvZZly5bt1X7qqafyyCOPAHDZZZexbt26vdrPOecc+vXrB0CXLl3Ytm3bXu3du3fnrrvuAiAvL6/Murnyyivp3bs3W7dupWvXsttejx496NGjB2vXruXyy8tue7feeitXXXUVS5cu5brrym57d955JxdeeCFbt27l66+/LlND3759Offcc5k5cyZ9+pTd9h5++GFOO+00Pv74Y+67r+y2N3DgQDp27Mi7775L//79y7Q/++yzdOjQgbfeeovHH3+8TPuIESM49NBDGTVqFEOGDCnT/tpre297pZXc9kaPDnbbKy4uZsqUKUDZbQ+gTZs22vZKbXvxeJwWLVoAe7a9efPm0atX2e+9+tz2khXKOSgz6wn09L81YZ99ihOHkxxm0KxZAS1bbga2sGzZzsR79hxyOuCAfA48cB1FReuYP3/H7vFmDoC2beMcfPAKCgtXMWvWdsxciWngmGPW0K7dYrZsWcbUqQW723fVcPrpSzj00M/YvPkbJk3akmhzu39eeumXdOjQgEWL5jJmzKbd47Oy/M9f/3o6RxwRZ8aMWYwYES/z+W+++VPatl3Bxx/PJh4v296mzVRatVpITs5ciovjFBfvfVjvo48+onnz5nz11Vflvn/KlCk0bNiQ+fPnl9u+60ti4cKFZdq3bdu2u33RokVl2ouLi3e3f/vtt3u1FxUVsWrVqt3ty5YtK/P+5cuX725fvnx5mfZly5btbl+1alWZ9m+//XZ3+5o1a9hU6mqORYsW7W5fv349hYWFe7UvXLhwd3t562b+/PlLNR+DAAAF6UlEQVTEYjEKCgrKbf/qq6+IxWJs3Lix3Pa5c+cSi8VYvXp1ue2zZ8+mWbNmbN68GedcmWlmzZpFTk4OX3/9dbnv/+yzz9i+fTtz5swpt3369OnE43FmzZpVbvunn37KihUrmD27/G1v6tSpLFy4kLlz55bbHua217hx4wq3PYAGDRpo2yu17RUVFe0e3rXtlbfuoH63vWSZcy7piaucmVl/oE11zkF17tzZTS/ZLXVExGKxcv/KkYppnSUvLy+PeDxe5q98qZi2r+qL6jozsxnOuc5VTZeVxIxiZuYqeH0YTLkiIiJ7q/IQn3Murx7qEBER2UtQl5nnJOaVDWSbWUNgp3NuZxDzFxGRzFPlIb4k9QW2AfcA1yaG+wY0bxERyUBBXWb+APBAEPMSERGB4PagREREAqWAEhGRSFJAiYhIJCmgREQkkhRQIiISSQooERGJJAWUiIhEkgJKREQiSQElIiKRpIASEZFIUkCJiEgkKaBERCSSFFAiIhJJCigREYkkBZSIiESSAkpERCJJASUiIpGkgBIRkUhSQImISCQpoEREJJIUUCIiEkkKKBERiSQFlIiIRJICSkREIkkBJSIikaSAEhGRSFJAiYhIJCmgREQkkhRQIiISSQooERGJJAWUiIhEkgJKREQiqdYBZWb7mtkwM1tiZpvNbKaZdQmiOBERyVxB7EHlAEuBs4DmQF9gtJm1D2DeIiKSoXJqOwPn3BbggRKjxpnZIqATsLi28xcRkcxU64AqzcxygaOAuZVM0xPoCZCbm0ssFgu6jFrLz8+PZF1RpnWWvHg8TlFRkdZXNWj7qr5UX2fmnAtuZmYNgAnAQudcr2Te07lzZzd9+vTAaghKLBYjLy8v7DJSitZZ8vLy8ojH48ycOTPsUlKGtq/qi+o6M7MZzrnOVU1X5TkoM4uZmavg9WGJ6bKAEcB24PZaVS8iIhmvykN8zrm8qqYxMwOGAblAV+fcjtqXJiIimSyoc1BDgGOAc51z2wKap4iIZLAg7oNqB/QCOgIrzSw/8bqm1tWJiEjGCuIy8yWABVCLiIjIburqSEREIkkBJSIikRTofVA1KsBsDbAk1CLK1xpYG3YRKUbrrHq0vqpH66v6orrO2jnnDqhqotADKqrMbHoyN5LJHlpn1aP1VT1aX9WX6utMh/hERCSSFFAiIhJJCqiKDQ27gBSkdVY9Wl/Vo/VVfSm9znQOSkREIkl7UCIiEkkKKBERiSQFlIiIRJICKklmdqSZFZjZyLBriSoz29fMhpnZEjPbbGYzzaxL2HVFjZntb2avm9mWxLr6Rdg1RZW2qdpJ9e8tBVTyngb+HXYREZcDLAXOApoDfYHRZtY+xJqi6Gn8gz1zgWuAIWb2g3BLiixtU7WT0t9bCqgkmNnVQBx4L+xaosw5t8U594BzbrFzrtg5Nw5YBHQKu7aoMLMmwGVAP+dcvnPuQ+BN4LpwK4smbVM1lw7fWwqoKpjZfsCfgDvCriXVmFkucBQwN+xaIuQoYKdzbn6JcbMA7UElQdtUctLle0sBVbWHgGHOuWVhF5JKzKwB8CLwvHPuq7DriZCmwKZS4zYCzUKoJaVom6qWtPjeyuiAMrOYmbkKXh+aWUfgXOCJsGuNgqrWV4npsoAR+PMst4dWcDTlA/uVGrcfsDmEWlKGtqnkpdP3Vq2fqJvKnHN5lbWbWR+gPfCtmYH/6zfbzI51zp1U5wVGTFXrC8D8ihqGvwCgq3NuR13XlWLmAzlmdqRzbkFi3InokFWFtE1VWx5p8r2lro4qYWaN2fuv3bvw//C3OufWhFJUxJnZM0BH4FznXH7Y9USRmb0COOBm/Lp6GzjNOaeQKoe2qepJp++tjN6DqopzbiuwddfvZpYPFKTaP3J9MbN2QC+gEFiZ+OsNoJdz7sXQCoue3sBzwGpgHf6LQ+FUDm1T1ZdO31vagxIRkUjK6IskREQkuhRQIiISSQooERGJJAWUiIhEkgJKREQiSQElIiKRpIASEZFIUkCJiEgk/T/XSE/diHEg1QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(z, elu(z), \"b-\", linewidth=2)\n",
    "plt.plot([-5, 5], [0, 0], 'k-')\n",
    "plt.plot([-5, 5], [-1, -1], 'k--')\n",
    "plt.plot([0, 0], [-2.2, 3.2], 'k-')\n",
    "plt.grid(True)\n",
    "plt.title(r\"ELU activation function ($\\alpha=1$)\", fontsize=14)\n",
    "plt.axis([-5, 5, -2.2, 3.2])\n",
    "\n",
    "save_fig(\"elu_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implementing ELU in TensorFlow is trivial, just specify the activation function when building each layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Dense at 0x10dca50f0>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.layers.Dense(10, activation=\"elu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SELU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This activation function was proposed in this [great paper](https://arxiv.org/pdf/1706.02515.pdf) by Günter Klambauer, Thomas Unterthiner and Andreas Mayr, published in June 2017. During training, a neural network composed exclusively of a stack of dense layers using the SELU activation function and LeCun initialization will self-normalize: the output of each layer will tend to preserve the same mean and variance during training, which solves the vanishing/exploding gradients problem. As a result, this activation function outperforms the other activation functions very significantly for such neural nets, so you should really try it out. Unfortunately, the self-normalizing property of the SELU activation function is easily broken: you cannot use ℓ<sub>1</sub> or ℓ<sub>2</sub> regularization, regular dropout, max-norm, skip connections or other non-sequential topologies (so recurrent neural networks won't self-normalize). However, in practice it works quite well with sequential CNNs. If you break self-normalization, SELU will not necessarily outperform other activation functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.special import erfc\n",
    "\n",
    "# alpha and scale to self normalize with mean 0 and standard deviation 1\n",
    "# (see equation 14 in the paper):\n",
    "alpha_0_1 = -np.sqrt(2 / np.pi) / (erfc(1/np.sqrt(2)) * np.exp(1/2) - 1)\n",
    "scale_0_1 = (1 - erfc(1 / np.sqrt(2)) * np.sqrt(np.e)) * np.sqrt(2 * np.pi) * (2 * erfc(np.sqrt(2))*np.e**2 + np.pi*erfc(1/np.sqrt(2))**2*np.e - 2*(2+np.pi)*erfc(1/np.sqrt(2))*np.sqrt(np.e)+np.pi+2)**(-1/2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def selu(z, scale=scale_0_1, alpha=alpha_0_1):\n",
    "    return scale * elu(z, alpha)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure selu_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xt8FPW5x/HPEwIIiEZBchTUeMN7RYhtxfaYVqyKl9qD1VpAsVoQqxaRVkUUDnCwtVTRFrAIgoJWqeIN0VZpo7WIFSTeRcWCICoXXTHhEhJ+54/fxixLLptkNjO7+b5fr3llmZnMPDtM9rsz++yMOecQERGJmpywCxAREamJAkpERCJJASUiIpGkgBIRkUhSQImISCQpoEREJJIUUCJ1MLOVZjaiGdYzxszebIb15JjZn8xso5k5MytK9zrrqWeWmc0PswaJLgWUpMTM9jGzKfEX7G1m9pmZLTSzUxPmKY6/6CUPDybM48zsvFrWMcjMSmuZVuvvBaGOgDgBmBLgegriz6UwadJE4OSg1lOHvsAlwNnAvsCiZlgnZlYUf96dkyb9EhjQHDVI5skNuwDJGI8A7YFLgQ+ALvgX1E5J880ERiaN25L26tLEObe+mdZTCtQYzgE7FPjEOdcswVQf59yXYdcg0aUjKKmXmeUB3wWud84tdM6tcs694pyb6Jx7MGn2zc65T5OGtL8ImdnpZvZPM/vCzD43s7+a2ZFJ8+xnZvfHT29tNrMSM/uemQ0CRgNHJxz1DYr/zten+MzsATN7JGmZOWa22syGp1jHf+I/X4mvpzj+ezsdwcWXe1N82dvM7A0z+2HC9KojsX5m9mz8+bydeERbwzaaBdwOHBD/3ZXx8cVm9sfkeRNPvcXnmWJmE8xsg5mtM7OJZpaTME+b+PRV8Zo/NLOrzawA+Ed8tvXxdc+qZT1tzWxS/Ah9q5ktNrPvJEyvOhI7xcxejj/vJWbWs7bnLZlLASWpqHp3f46Z7RZ2MbXoAEwCvgkUAV8CT5pZGwAz6wA8DxQA5wLHAmPjv/sQ8HtgOf60177xccnmAGea2Z4J406Oz//nVOqIjwc4Pf57/1PL8/kl8CvgunitjwLzzKxH0nz/B9wJHAe8AjxoZrvXscyxwJr4uk+oZb7a9AcqgN7AlcAw4IKE6fcCFwHDgSPxR9sxYDXQLz7P0fF1/7KWddwaX+bPgOOBN4BnzGzfpPluAa4HegIbgfvNzBr4fCTqnHMaNNQ74F9gPge2Ai/hPzP5VtI8xUA51YFWNVyRMI8DzqtlHYOA0lqm1fp7tczfAagEvhP/98+Br4DOtcw/BnizhvErgRHxx7nAZ8ClCdOnA39rQB0F8edSWNf6gY+Bm2vYvnOSljMkYXrX+Ljv1FHPCGBlDcv9Y9K4WcD8pHleSprnWWB6/PFh8XWfXst6i+LTO9e2nvi2KgcuSpjeClgBjE9azmkJ85wUH9ct7L8TDcEOOoKSlDjnHgH2w3+4/jT+XfRiM0v+vOkhoEfScH+66zOzQ+Kn4FaY2SZ8kOQAB8RnOR543Tm3obHrcM5V4J9f//g62+KDe04D6kjlueyB39b/Spr0InBU0rjXEx6vjf/skuq6Guj1pH+vTVjX8cAOqk/lNcYhQGsSnrdzrhL/hijM5y0hUZOEpMw5txX/rvlZYKyZTQfGmNlE51x5fLYvnXMfNHIVm4B2ZtbaObe9amT8MzDwp8tqMx9/6moI/uijAngbaFPH7zTGHOAlM+sKfCu+/HnNWEfy7Qe+3k7OORc/y9XQN547gOTTY61rmG970r9dI9bVWLU+74RpesOdZfQfKk3xNv5NTlCfSy3H75PHJ43vmTB9F2bWCTgCmOCce8459w7QkZ3fgC0DvlFDm3OVcvzppDo55/6N72K8EH8k9bjzHXip1lEV5LWuyzm3CX9UcFLSpO/gt3nQ1uM/F0p0XAOXUYL/v/teLdPrfd74U3nlJDxvM2sFnEh6nrdEnI6gpF7xF96/APfgT618BRQCvwYWxl9Qq7Q3s/9KWkS5c+7zhH8X1PBh/4fOubfM7G/A9HhX3AqgO3AHMNc591EtJX4BbAB+bmar8Z/F/A5/9FLlAfyH6o+b2fX4o5tjgK+cc//Af9Z0YLwb7KP4+G21rO9+4DL850CJTQ6p1LEO33Z/WryLbqurucvxd/ij1PeBpfjvCn2X6rAO0t+BSWZ2Dv5NwBBgf/w2SYlz7j0zm4v/v/sl8CrQDShwzs0GVuGPdM40syeBLVXBnrCMMjObCvzWzDbgOx6vAfIJ8LtokkHC/hBMQ/QHoC0wAd8l9gWwGXgfuA3YO2G+YvyLUPLwYsI8NU13wFnx6Xn4QPogvp73gN8Cu9dT4/eBN/FNHG8Cp+EbNAYlzNMN/xlSLL7sZUBRwnN8OP78XNXvkdAkkbCcg+PzfAbkNqKOy/AhWAkUx8eNYecmiRzgJnwHXDm+m+3chOkF1NxsUWczCTU3SbQGJuPDdQPwv9TcJFFfI0VbfBfex8A2/BuMKxOm3wR8gj+lOKuOZUyKb9ttwGISmj6oodmitm2hIfMHi/8Hi4iIRIo+gxIRkUhSQImISCQpoEREJJIUUCIiEkmht5l37tzZFRQUhF3GLsrKyujQoUPYZWQUbbPULV++nMrKSo46KvkCCVKbqO5f5eXwzjtQUQGdOkGUXs6ius2WLl26wTm3T33zhR5QBQUFLFmyJOwydlFcXExRUVHYZWQUbbPUFRUVEYvFIrnvR1UU969Nm+Ckk3w4fe978Mwz0Cboa5c0QRS3GYCZrUplPp3iExFphIoKuOACePNNOOIIeOSRaIVTNlBAiYg0kHNw9dX+iKlzZ3jqKdhrr7Cryj4KKBGRBpo0CaZOhbZt4fHH4eCDw64oOymgREQa4LHH4Npr/eN774XevcOtJ5sFGlBmNsfMPjGzTWb2npldFuTyRUTCtHQp9O/vT/GNH+8/g5L0CfoI6hb81Yv3AM4BxptZr4DXISLS7FavhrPPhs2b4eKLYWTyrTolcIEGlHPuLVd9i4Kqq1QfEuQ6RESa26ZNcOaZ8MknUFQE06aBJd/iUQIX+PegzGwKMAhoh7+dwYIa5hkMDAbIz8+nuLg46DKarLS0NJJ1RZm2WepisRiVlZXaXg0Q1v5VWWmMHHkMb7zRif3338zw4a+yaFFF/b8YAZn+N5mW220k3AWzCPitS7h9d7LCwkIXxS8rRvULblGmbZa6qi/qlpSUhF1Kxghj/3IOrrwSpkzx7eSLF8MhGXROKKp/k2a21DlXWN98aenic85VOudexN8gbmg61iEikm533OHDqU0b372XSeGUDdLdZp6LPoMSkQz0+OMwfLh/PGuWv6SRNK/AAsrMupjZT8xsdzNrZWanARcCC4Nah4hIc1i6FH76U3+Kb9w4uPDCsCtqmYJsknD403l34YNvFTDMOfdEgOsQEUmr5HbyG28Mu6KWK7CAcs6tB04OankiIs3tq6/grLN8O/nJJ6udPGy61JGICP7q5D/5Cbz+OnTvDvPm6erkYVNAiUiL5xwMGwYLFvibDi5YAHvvHXZVooASkRbvzjth8mS1k0eNAkpEWrQnn4RrrvGPZ86E73wn3HqkmgJKRFqsV1/1nzs5B2PH+tZyiQ4FlIi0SGvWVLeTX3QRjBoVdkWSTAElIi1OVTv52rXw3/+tdvKoUkCJSItSUeGvDPHaa3DYYb6dvG3bsKuSmiigRKRFGT4cnnqqup28U6ewK5LaKKBEpMW48074wx+q28kPPTTsiqQuCigRaRHmz69uJ7/nHrWTZwIFlIhkvWXLfDv5jh0wZgz07x92RZIKBZSIZLU1a3zHXlkZDBgAN98cdkWSKgWUiGSt0lL/XaeqdvLp09VOnkkUUCKSlSorfTt5SYnayTOVAkpEstLw4b4xYu+9q9vKJbMooEQk6/zhD76lvKqd/LDDwq5IGkMBJSJZ5amn/L2dAGbMgO9+N9x6pPEUUCKSNUpK4IILfDv56NG+a08ylwJKRLLCxx9Xt5P37+8DSjKbAkpEMl5VO/nHH/srRMyYoXbybKCAEpGMVlnpbzS4bJm/tt5jj6mdPFsooEQko117rb9tu9rJs48CSkQy1uTJcMcd0Lo1PPoodO8edkUSJAWUiGSkBQvg6qv94xkz/KWMJLsooEQk47z2WnU7+c03w8CBYVck6aCAEpGMsnatbycvLfXNEWPGhF2RpIsCSkQyRlmZbydfswZOOknt5NlOASUiGaGqnfzVV+GQQ3w7+W67hV2VpJMCSkQywl13HcITT8Bee/kGic6dw65I0k0BJSKRN2UKPPzw/monb2EUUCISaU8/DVdd5R9Pnw4nnxxuPdJ8AgsoM2trZjPMbJWZfWVmJWZ2RlDLF5GW57XX4PzzfTv5wIErueiisCuS5hTkEVQusBo4GdgTGAXMNbOCANchIi1EYjv5hRfCJZesDLskaWaBBZRzrsw5N8Y5t9I5t8M5Nx/4D9ArqHWISMuQ3E5+zz1qJ2+JctO1YDPLB7oDb9UwbTAwGCA/P5/i4uJ0ldFopaWlkawryrTNUheLxaisrNT2qkFlJYwefQyvvtqZ/fbbwogRr7J48XbtX42Q6dvMnHPBL9SsNfA0sMI5N6SueQsLC92SJUsCr6GpiouLKSoqCruMjKJtlrqioiJisRglJSVhlxI5114Lt93m28lfegkOP9yP1/7VcFHdZma21DlXWN98gXfxmVkOMBsoB64Mevkikr2mTvXh1Lo1zJtXHU7SMgV6is/MDJgB5AN9nXPbg1y+iGSvZ56pbie/+26I4Bt/aWZBfwY1FTgS6OOc2xLwskUkS73xhm8nr6yEG2+Eiy8OuyKJgiC/B3UgMAToAXxqZqXxoX9Q6xCR7PPJJ3DmmfDVV/4WGmPHhl2RREVgR1DOuVWAGkFFJGVV7eSrV0Pv3jBrFuTo+jYSp11BREJRWQkDBsDSpXDwwbo6uexKASUiobjuOh9KeXnw1FOwzz5hVyRRo4ASkWZ3113w+99Dbi488ggccUTYFUkUKaBEpFn99a9wZfwbktOmwfe/H249El0KKBFpNm+8AT/+sf/8aeRIuOSSsCuSKFNAiUiz+PRTf3XyqnbycePCrkiiTgElImm3eTOccw589BF8+9swc6bayaV+2kVEJK127PDt5K+8AgcdBI8/Du3ahV2VZAIFlIik1XXXwaOPwp57+nbyLl3CrkgyhQJKRNJm2jSYONG3k8+bB0ceGXZFkkkUUCKSFn/7G1xxhX/8pz+pnVwaTgElIoF7883qdvIbboCf/SzsiiQTKaBEJFCffuqvTr5pkw+p8ePDrkgylQJKRAKT3E5+771qJ5fG064jIoHYsQMuusi3kxcUqJ1cmk4BJSKBuOEGf+FXtZNLUBRQItJkd98Nt95afXXyo44KuyLJBgooEWmSZ5+FoUP947vuglNOCbceyR4KKBFptLfegvPO8+3k110Hl14adkWSTRRQItIon31W3U5+3nkwYULYFUm2UUCJSINVtZOvWgXf+hbcd5/aySV42qVEpEGq2sn//W+1k0t6KaBEpEFGjvSdenvs4dvJ8/PDrkiylQJKRFI2fTr89rdqJ5fmoYASkZQ89xxcfrl/PHUq9OkTbj2S/RRQIlKvt9+ubif/9a/hssvCrkhaAgWUiNSpqp38yy+hXz+45ZawK5KWQgElIrXasgV++ENYuRK++U21k0vz0q4mIjWqaid/+WU48EB44glo3z7sqqQlUUCJSI1uvBEefljt5BIeBZSI7OKee+A3v4FWrXxIHX102BVJS6SAEpGdLFwIQ4b4x1OnwqmnhluPtFwKKBH52ttv+069igr41a/g5z8PuyJpyQINKDO70syWmNk2M5sV5LJFJL3WratuJ/+f//Gn+ETClBvw8tYC44HTAF0+UiRDJLaTn3ACzJ6tdnIJX6AB5ZybB2BmhUC3IJctIumxYwcMGgSLF8MBB6idXKIj6COolJjZYGAwQH5+PsXFxWGUUafS0tJI1hVl2mapi8ViVFZWRmJ73X33QcydeyAdOlQwZswy3n23jHffDbuqXWn/arhM32ahBJRzbhowDaCwsNAVFRWFUUadiouLiWJdUaZtlrq8vDxisVjo22vmTHjgAd9OPm9eLj/4wQmh1lMX7V8Nl+nbTGeZRVqov/8dBg/2jydPhh/8INx6RJIpoERaoHfeqW4nHzGi+ntPIlES6Ck+M8uNL7MV0MrMdgMqnHMVQa5HRBqvqp08FoMf/cjfgFAkioI+ghoFbAGuBwbEH48KeB0i0khbt8K558J//gOFhTBnjtrJJbqCbjMfA4wJcpkiEoyqdvKXXoL991c7uUSf3juJtBA33wwPPQQdO/qrk++7b9gVidRNASXSAsycCf/3f76d/C9/gWOPDbsikfopoESy3D/+Ud1O/sc/wmmnhVuPSKoUUCJZ7N13/YVfKypg+HC4/PKwKxJJnQJKJEutX1/dTn7uuXDrrWFXJNIwCiiRLFTVTv7hh9Crl28nb9Uq7KpEGkYBJZJlduyASy6BRYt8O/mTT0KHDmFXJdJwCiiRLDN6NDz4oG8nnz9f7eSSuRRQIlnk3nth/Hh/Om/uXPjGN8KuSKTxFFAiWaK4GH7+c//4D3+A008PtRyRJlNAiWSB5ct9O/n27XDNNTB0aNgViTSdAkokw23Y4NvJv/gCzjkHfve7sCsSCYYCSiSDVbWTr1gBPXtW3x1XJBsooEQylHPws5/Bv/4F3bqpnVyyjwJKJEONHg1//jPsvru/Ovl++4VdkUiwFFAiGei++2DcOH+zwYceUju5ZCcFlEiGef55uOwy//jOO6Fv33DrEUkXBZRIBlm+HH70I99OPmwY/OIXYVckkj4KKJEMkdxOPnFi2BWJpJcCSiQDbNvmj5xWrIDjj4f771c7uWQ/BZRIxFW1k7/4InTt6tvJd9897KpE0k8BJRJxY8b4L+BWtZN37Rp2RSLNQwElEmGzZ8PYsdXt5McdF3ZFIs1HASUSUS+8AJde6h/fcYfayaXlUUCJRND771e3k199NVx5ZdgViTQ/BZRIxGzc6I+WPv8czj4bbrst7IpEwqGAEomQbdv81ck/+MC3k+vq5NKSKaBEIsI5fwkjtZOLeAookYgYOxbmzPG3zJg/X+3kIgookQiYM8d/3yknBx58EHr0CLsikfApoERC9s9/VreTT5oEZ50Vbj0iUaGAEgnR++/7pojycrjqKj+IiBdoQJnZ3mb2qJmVmdkqM/tpkMsXySYVFcaZZ/p28jPPhNtvD7sikWjJDXh5k4FyIB/oATxlZq85594KeD0iGc05WLmyA2Vl/vOmBx9UO7lIMnPOBbMgsw7AF8Axzrn34uNmAx87566v7fc6duzoevXqFUgNQYrFYuTl5YVdRkbRNkvdyy+XsHUrtGnTg549oW3bsCuKPu1fDRfVbfb8888vdc4V1jdfkEdQ3YGKqnCKew04OXlGMxsMDAZo3bo1sVgswDKCUVlZGcm6okzbLDXbtuWwdat/3LVrGVu2bGfLlnBrygTavxou07dZkAG1O7ApadyXQMfkGZ1z04BpAIWFhW7JkiUBlhGM4uJiioqKwi4jo2ib1c856NMH3n23iLy8cj78cFHYJWUM7V8NF9VtZmYpzRdkk0QpsEfSuD2ArwJch0hGmz8f/v53yM2Frl112CRSlyAD6j0g18wOSxh3HKAGCRGgshKuj38ae+CBkJsbzOe/ItkqsIByzpUB84CxZtbBzE4CfgjMDmodIpnsvvvg7behoAD22y/sakSiL+gv6l4BtAPWAX8GhqrFXATKyuDmm/3j8eP9JY1EpG6B/pk45z53zp3rnOvgnDvAOfdAkMsXyVS33gpr1kDPnnDhhWFXI5IZ9D5OJM1WrfIBBXDnnTp6EkmV/lRE0uxXv4KtW/2R00knhV2NSOZQQImkUXEx/OUv0L599VGUiKRGASWSJpWV8Mtf+sfXXw/duoVbj0imUUCJpMndd8Prr/vvPI0YEXY1IplHASWSBp9+Cjfc4B9PnAjt2oVbj0gmUkCJpMFVV0EsBmecAf36hV2NSGZSQIkE7LHH4OGHoUMHmDoVUrwupogkUUCJBOjLL+EXv/CPJ0zwnz+JSOMooEQCdP31sHYtfPvb1UElIo2jgBIJyAsvwF13QevWMH26buEu0lQKKJEAbNoEF1/sH99wAxx9dLj1iGQDBZRIAK66Clau9BeDvfHGsKsRyQ4KKJEmeughf6+ndu3g/vuhTZuwKxLJDgookSb46CO4/HL/+Lbb4Igjwq1HJJsooEQaqbISLrrIfyH37LNhyJCwKxLJLgookUaaMAGefx7y82HGDH0hVyRoCiiRRnj6aRg92ofSfffBPvuEXZFI9skNuwCRTPPhh9C/PzgH48bBD34QdkUi2UlHUCINsHmzv/jrF1/4z51Gjgy7IpHspYASSZFzMHQolJTAoYf6U3s5+gsSSRv9eYmk6PbbfSi1bw/z5kFeXtgViWQ3BZRICh55pPquuDNnwrHHhluPSEuggBKpx+LFMGCAP8X3m9/A+eeHXZFIy6CAEqnDihVwzjmwdSsMHgy//nXYFYm0HAookVqsWwd9+8L69XD66TB5sr6MK9KcFFAiNfj8c//9pvfeg+OO8xeEzdW3BkWalQJKJMmmTXDGGfDaa9C9O/z1r7DHHmFXJdLyKKBEEpSVwVlnwb//DQcdBAsX+mvtiUjzU0CJxJWVwQ9/CP/8J3Tt6sOpW7ewqxJpuXRWXQR/y4yzzoJ//Qu6dPHhdNBBYVcl0rLpCEpavPXr4Xvf8+G0//7+COrww8OuSkR0BCUt2po1cOqp8O67/vp6CxfCAQeEXZWIQEBHUGZ2pZktMbNtZjYriGWKpNvrr0Pv3j6cjj3WHzkpnESiI6hTfGuB8cA9AS1PJK0WLICTToLVq31IFRfDf/1X2FWJSKJAAso5N8859xiwMYjliaTT5Mn+Xk6lpXDhhf603t57h12ViCQL5TMoMxsMDAbIz8+nuLg4jDLqVFpaGsm6oizq26y83Jg8+VCeeKIrABddtJJBg1ayeHHz1xKLxaisrIz09oqaqO9fUZTp2yyUgHLOTQOmARQWFrqioqIwyqhTcXExUawryqK8zVatgh//GF55Bdq0genTYeDAAqAglHry8vKIxWKR3V5RFOX9K6oyfZvVe4rPzIrNzNUyvNgcRYo0xTPPQM+ePpwOPNC3kw8cGHZVIlKfeo+gnHNFzVCHSODKy+Hmm+HWW/29nM44A2bPhk6dwq5MRFIRyCk+M8uNL6sV0MrMdgMqnHMVQSxfpKHeegv69/cXfM3Jgf/9X7jxRv9YRDJDUH+uo4AtwPXAgPjjUQEtWyRlO3bApEnQq5cPp4MP9t9vuukmhZNIpgnkCMo5NwYYE8SyRBrrzTdhyBBYtMj/+9JL4fbboWPHcOsSkcbRe0rJeFu3wqhRcPzxPpz23Rcee8x36imcRDKXrsUnGcs5mD8fhg+HDz7w4y6/HG65BfLywq1NRJpOASUZ6Y034Jpr/FUgAI46CqZN85cvEpHsoFN8klHWrIHBg6FHDx9Oe+0Fd9wBJSUKJ5FsoyMoyQiffeZP3d11F2zbBq1awVVXwejR+l6TSLZSQEmkffKJ78SbPBk2b/bjzj/ff6/piCPCrU1E0ksBJZH0/vvwu9/Bvff6K0KAvwL5uHFw3HHh1iYizUMBJZGxYwc8+yxMmQJPPum79MygXz+47jo44YSwKxSR5qSAktBt3AgzZ/rPl1as8ONat4aLL4YRI+Dww8OtT0TCoYCSUFRWwvPPw6xZMHeub3wAf8v1IUP8VSDy80MtUURCpoCSZuMcLFsG998PDz4Ia9f68Wb+SuNXXOF/tmoVbp0iEg0KKEkr5/yXah97DB54AJYvr5528MHw05/CJZf4xyIiiRRQErjycn/67okn/PDRR9XT9tkHLrjA3wrjW9/yR08iIjVRQEmTOeevhffEE/vxxz/6TrxNm6qnd+niW8T79YM+fXwDhIhIfRRQ0iirV8MLL8Bzz/lLDq1eDdD96+nHHAPnnOOD6Zvf1L2YRKThFFBSr/Jyf627RYv88NJL/pp4iTp1gmOOWceFF3bh1FP1mZKINJ0CSnaydau/8d+rr/ph2TJ/Z9qqNvAqeXlw4olwyil++MY34IUX3qaoqEs4hYtI1lFAtVDl5f5yQu+8A2+/XT288w5UVOw6/xFHQO/efjjxRP9vnbYTkXRSQGWxigr/2dCHH1YPy5f7IPrgA/9l2WQ5OXDkkdCzZ/XQo4duACgizU8BlaGc851yH3/sPw/6+GM/JAbSqlU1hxD49u5DDvFhdNRR/ueRR/rmhg4dmve5iIjURAEVMVu2wPr1sG6d/5k4rF27cyCVldW/vK5dfcNC1XDooT6QDj8c2rVL//MREWksBVTAnPMNBV9+CbFYasPGjdUhlEroVGnfHrp18yFUNXTrVh1GBQWw225pe6oiImnVYgJqxw4fHFu3+iHxcU3jli3LZ/lyf0RTWuqDI/FnbY/LympuMkhVmzb+agtVQ5cu1Y/33XfnMNpzT12JQUSyV+gB9ckncNNN/kV9+/bqn4mPGztu27bq0Km66V3qjmz0c8rN9U0FNQ177VXzuKow6thRoSMiAhEIqLVrlzN+fFHS2POBK4DNQN8afmtQfNgAnFfD9KHABcBqYODXY818l1rHjtey555nk5OznHXrhpCTw07D0UePom3bY+jY8VNefnkYrVr5K2zn5Pif/ftPoGfP3qxcuYiZM0d+Pb1quOOOSfTo0YPnnnuO8ePHs3179Sk8gD/96U8cfvjhPPnkk9x66+93qX727Nnsv//+PPTQQ0ydOnWX6Q8//DCdO3dm1qxZzJo1a5fpCxYsoH379kyZMoW5c+fuMr24uBiAiRMnMn/+/J2mtWvXjqeffhqAcePGsXDhwp2md+rUiUceeQSAG264gZdeeunrabFYjGOOOYY5c+YAMGzYMEpKSnb6/e7duzNt2jQABg8ezHvvvbfT9B49ejBp0iQABgwYwJqkbwSfeOKJ3HLLLQD069ePjRs37jT9lFNO4aabbgICTsdMAAAFTElEQVTgjDPOYMuWLTtNP+ussxgxYgQARUVFu2yb888/nyuuuILNmzfTt++u+96gQYMYNGgQGzZs4Lzzdt33hg4dygUXXMDq1asZOHDgLtOvvfZazj77bDZv3swHH3ywSw2jRo2iT58+lJSUMGzYsF1+f8KECfTu3ZtFixYxcuTIXaZPmrTzvpcscd/7/e8za9/bsWMHL7zwArDrvgfQrVs37XtJ+14sFiMv3oJbte8tX76cIUOG7PL7zbnvpSr0gGrTBvbbz4dH1dCrF3z/+/603J137jzNDE47Dfr29afTRo/eeVpODgwYAOeeCxs2wNVX+3GJRyXXXusvwbN8ub/3ULJRoyA3913y8vKo4f+JPn3894EWLYKHH07fthERacnMORdqAYWFhW7JkiWh1lCT4uLiGt/lSO20zVJXVFRELBbb5V2+1E77V8NFdZuZ2VLnXGF98+laACIiEkkKKBERiSQFlIiIRJICSkREIkkBJSIikdTkgDKztmY2w8xWmdlXZlZiZmcEUZyIiLRcQRxB5eK/EXsysCcwCphrZgUBLFtERFqoJn9R1zlXBoxJGDXfzP4D9AJWNnX5IiLSMgV+JQkzywe6A2/VMc9gYDBAfn7+15c/iZLS0tJI1hVl2mapi8ViVFZWans1gPavhsv0bRbolSTMrDXwNLDCOVfDRYR2pStJZA9ts9TpShINp/2r4aK6zQK7koSZFZuZq2V4MWG+HGA2UA5c2aTqRUSkxav3FJ9zrqi+eczMgBlAPtDXObe96aWJiEhLFtRnUFPxN1Dq45zbUt/MIiIi9Qnie1AHAkOAHsCnZlYaH/o3uToREWmxgmgzXwXoHrAiIhIoXepIREQiSQElIiKRFPoddc1sPbAq1CJq1hnYEHYRGUbbrGG0vRpG26vhorrNDnTO7VPfTKEHVFSZ2ZJUvkgm1bTNGkbbq2G0vRou07eZTvGJiEgkKaBERCSSFFC1mxZ2ARlI26xhtL0aRtur4TJ6m+kzKBERiSQdQYmISCQpoEREJJIUUCIiEkkKqBSZ2WFmttXM5oRdS1SZWVszm2Fmq8zsKzMrMbMzwq4rasxsbzN71MzK4tvqp2HXFFXap5om01+3FFCpmwy8EnYREZcLrAZOBvYERgFzzawgxJqiaDL+xp75QH9gqpkdHW5JkaV9qmky+nVLAZUCM/sJEAMWhl1LlDnnypxzY5xzK51zO5xz84H/AL3Cri0qzKwD0A+4yTlX6px7EXgCGBhuZdGkfarxsuF1SwFVDzPbAxgLDA+7lkxjZvlAd+CtsGuJkO5AhXPuvYRxrwE6gkqB9qnUZMvrlgKqfuOAGc65NWEXkknMrDVwP3Cvc+7dsOuJkN2BTUnjvgQ6hlBLRtE+1SBZ8brVogPKzIrNzNUyvGhmPYA+wO1h1xoF9W2vhPlygNn4z1muDK3gaCoF9kgatwfwVQi1ZAztU6nLptetJt9RN5M554rqmm5mw4AC4CMzA//ut5WZHeWc65n2AiOmvu0FYH5DzcA3APR1zm1Pd10Z5j0g18wOc869Hx93HDplVSvtUw1WRJa8bulSR3Uws/bs/G53BP4/fqhzbn0oRUWcmd0F9AD6OOdKw64niszsQcABl+G31QKgt3NOIVUD7VMNk02vWy36CKo+zrnNwOaqf5tZKbA10/6Tm4uZHQgMAbYBn8bfvQEMcc7dH1ph0XMFcA+wDtiIf+FQONVA+1TDZdPrlo6gREQkklp0k4SIiESXAkpERCJJASUiIpGkgBIRkUhSQImISCQpoEREJJIUUCIiEkkKKBERiaT/B9c9MCs0b4SXAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(z, selu(z), \"b-\", linewidth=2)\n",
    "plt.plot([-5, 5], [0, 0], 'k-')\n",
    "plt.plot([-5, 5], [-1.758, -1.758], 'k--')\n",
    "plt.plot([0, 0], [-2.2, 3.2], 'k-')\n",
    "plt.grid(True)\n",
    "plt.title(\"SELU activation function\", fontsize=14)\n",
    "plt.axis([-5, 5, -2.2, 3.2])\n",
    "\n",
    "save_fig(\"selu_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the SELU hyperparameters (`scale` and `alpha`) are tuned in such a way that the mean output of each neuron remains close to 0, and the standard deviation remains close to 1 (assuming the inputs are standardized with mean 0 and standard deviation 1 too). Using this activation function, even a 1,000 layer deep neural network preserves roughly mean 0 and standard deviation 1 across all layers, avoiding the exploding/vanishing gradients problem:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer 0: mean -0.00, std deviation 1.00\n",
      "Layer 100: mean 0.02, std deviation 0.96\n",
      "Layer 200: mean 0.01, std deviation 0.90\n",
      "Layer 300: mean -0.02, std deviation 0.92\n",
      "Layer 400: mean 0.05, std deviation 0.89\n",
      "Layer 500: mean 0.01, std deviation 0.93\n",
      "Layer 600: mean 0.02, std deviation 0.92\n",
      "Layer 700: mean -0.02, std deviation 0.90\n",
      "Layer 800: mean 0.05, std deviation 0.83\n",
      "Layer 900: mean 0.02, std deviation 1.00\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "Z = np.random.normal(size=(500, 100)) # standardized inputs\n",
    "for layer in range(1000):\n",
    "    W = np.random.normal(size=(100, 100), scale=np.sqrt(1 / 100)) # LeCun initialization\n",
    "    Z = selu(np.dot(Z, W))\n",
    "    means = np.mean(Z, axis=0).mean()\n",
    "    stds = np.std(Z, axis=0).mean()\n",
    "    if layer % 100 == 0:\n",
    "        print(\"Layer {}: mean {:.2f}, std deviation {:.2f}\".format(layer, means, stds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using SELU is easy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Dense at 0x158a45630>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.layers.Dense(10, activation=\"selu\",\n",
    "                   kernel_initializer=\"lecun_normal\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a neural net for Fashion MNIST with 100 hidden layers, using the SELU activation function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "model.add(keras.layers.Dense(300, activation=\"selu\",\n",
    "                             kernel_initializer=\"lecun_normal\"))\n",
    "for layer in range(99):\n",
    "    model.add(keras.layers.Dense(100, activation=\"selu\",\n",
    "                                 kernel_initializer=\"lecun_normal\"))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"sgd\",\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train it. Do not forget to scale the inputs to mean 0 and standard deviation 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "pixel_means = X_train.mean(axis=0, keepdims=True)\n",
    "pixel_stds = X_train.std(axis=0, keepdims=True)\n",
    "X_train_scaled = (X_train - pixel_means) / pixel_stds\n",
    "X_valid_scaled = (X_valid - pixel_means) / pixel_stds\n",
    "X_test_scaled = (X_test - pixel_means) / pixel_stds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/5\n",
      "55000/55000 [==============================] - 35s 644us/sample - loss: 1.0197 - accuracy: 0.6154 - val_loss: 0.7386 - val_accuracy: 0.7348\n",
      "Epoch 2/5\n",
      "55000/55000 [==============================] - 33s 607us/sample - loss: 0.7149 - accuracy: 0.7401 - val_loss: 0.6187 - val_accuracy: 0.7774\n",
      "Epoch 3/5\n",
      "55000/55000 [==============================] - 32s 583us/sample - loss: 0.6193 - accuracy: 0.7803 - val_loss: 0.5926 - val_accuracy: 0.8036\n",
      "Epoch 4/5\n",
      "55000/55000 [==============================] - 32s 586us/sample - loss: 0.5555 - accuracy: 0.8043 - val_loss: 0.5208 - val_accuracy: 0.8262\n",
      "Epoch 5/5\n",
      "55000/55000 [==============================] - 32s 573us/sample - loss: 0.5159 - accuracy: 0.8238 - val_loss: 0.4790 - val_accuracy: 0.8358\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train_scaled, y_train, epochs=5,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now look at what happens if we try to use the ReLU activation function instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "model.add(keras.layers.Dense(300, activation=\"relu\", kernel_initializer=\"he_normal\"))\n",
    "for layer in range(99):\n",
    "    model.add(keras.layers.Dense(100, activation=\"relu\", kernel_initializer=\"he_normal\"))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"sgd\",\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/5\n",
      "55000/55000 [==============================] - 18s 319us/sample - loss: 1.9174 - accuracy: 0.2242 - val_loss: 1.3856 - val_accuracy: 0.3846\n",
      "Epoch 2/5\n",
      "55000/55000 [==============================] - 15s 279us/sample - loss: 1.2147 - accuracy: 0.4750 - val_loss: 1.0691 - val_accuracy: 0.5510\n",
      "Epoch 3/5\n",
      "55000/55000 [==============================] - 15s 281us/sample - loss: 0.9576 - accuracy: 0.6025 - val_loss: 0.7688 - val_accuracy: 0.7036\n",
      "Epoch 4/5\n",
      "55000/55000 [==============================] - 15s 281us/sample - loss: 0.8116 - accuracy: 0.6762 - val_loss: 0.7276 - val_accuracy: 0.7288\n",
      "Epoch 5/5\n",
      "55000/55000 [==============================] - 15s 278us/sample - loss: 0.8167 - accuracy: 0.6862 - val_loss: 0.7697 - val_accuracy: 0.7032\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train_scaled, y_train, epochs=5,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not great at all, we suffered from the vanishing/exploding gradients problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch Normalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(300, activation=\"relu\"),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(100, activation=\"relu\"),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_3\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_3 (Flatten)          (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_v2 (Batc (None, 784)               3136      \n",
      "_________________________________________________________________\n",
      "dense_210 (Dense)            (None, 300)               235500    \n",
      "_________________________________________________________________\n",
      "batch_normalization_v2_1 (Ba (None, 300)               1200      \n",
      "_________________________________________________________________\n",
      "dense_211 (Dense)            (None, 100)               30100     \n",
      "_________________________________________________________________\n",
      "batch_normalization_v2_2 (Ba (None, 100)               400       \n",
      "_________________________________________________________________\n",
      "dense_212 (Dense)            (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 271,346\n",
      "Trainable params: 268,978\n",
      "Non-trainable params: 2,368\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('batch_normalization_v2/gamma:0', True),\n",
       " ('batch_normalization_v2/beta:0', True),\n",
       " ('batch_normalization_v2/moving_mean:0', False),\n",
       " ('batch_normalization_v2/moving_variance:0', False)]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bn1 = model.layers[1]\n",
    "[(var.name, var.trainable) for var in bn1.variables]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ListWrapper([<tf.Operation 'batch_normalization_v2/cond_2/Identity' type=Identity>, <tf.Operation 'batch_normalization_v2/cond_3/Identity' type=Identity>])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bn1.updates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"sgd\",\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "55000/55000 [==============================] - 5s 85us/sample - loss: 0.8756 - accuracy: 0.7140 - val_loss: 0.5514 - val_accuracy: 0.8212\n",
      "Epoch 2/10\n",
      "55000/55000 [==============================] - 4s 74us/sample - loss: 0.5765 - accuracy: 0.8033 - val_loss: 0.4742 - val_accuracy: 0.8436\n",
      "Epoch 3/10\n",
      "55000/55000 [==============================] - 4s 75us/sample - loss: 0.5146 - accuracy: 0.8216 - val_loss: 0.4382 - val_accuracy: 0.8530\n",
      "Epoch 4/10\n",
      "55000/55000 [==============================] - 4s 75us/sample - loss: 0.4821 - accuracy: 0.8322 - val_loss: 0.4170 - val_accuracy: 0.8604\n",
      "Epoch 5/10\n",
      "55000/55000 [==============================] - 4s 75us/sample - loss: 0.4589 - accuracy: 0.8402 - val_loss: 0.4003 - val_accuracy: 0.8658\n",
      "Epoch 6/10\n",
      "55000/55000 [==============================] - 4s 75us/sample - loss: 0.4428 - accuracy: 0.8459 - val_loss: 0.3883 - val_accuracy: 0.8698\n",
      "Epoch 7/10\n",
      "55000/55000 [==============================] - 4s 78us/sample - loss: 0.4220 - accuracy: 0.8521 - val_loss: 0.3792 - val_accuracy: 0.8720\n",
      "Epoch 8/10\n",
      "55000/55000 [==============================] - 4s 77us/sample - loss: 0.4150 - accuracy: 0.8546 - val_loss: 0.3696 - val_accuracy: 0.8754\n",
      "Epoch 9/10\n",
      "55000/55000 [==============================] - 4s 77us/sample - loss: 0.4013 - accuracy: 0.8589 - val_loss: 0.3629 - val_accuracy: 0.8746\n",
      "Epoch 10/10\n",
      "55000/55000 [==============================] - 4s 74us/sample - loss: 0.3931 - accuracy: 0.8615 - val_loss: 0.3581 - val_accuracy: 0.8766\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes applying BN before the activation function works better (there's a debate on this topic). Moreover, the layer before a `BatchNormalization` layer does not need to have bias terms, since the `BatchNormalization` layer some as well, it would be a waste of parameters, so you can set `use_bias=False` when creating those layers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(300, use_bias=False),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Activation(\"relu\"),\n",
    "    keras.layers.Dense(100, use_bias=False),\n",
    "    keras.layers.Activation(\"relu\"),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"sgd\",\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "55000/55000 [==============================] - 5s 89us/sample - loss: 0.8617 - accuracy: 0.7095 - val_loss: 0.5649 - val_accuracy: 0.8102\n",
      "Epoch 2/10\n",
      "55000/55000 [==============================] - 4s 76us/sample - loss: 0.5803 - accuracy: 0.8015 - val_loss: 0.4833 - val_accuracy: 0.8344\n",
      "Epoch 3/10\n",
      "55000/55000 [==============================] - 4s 79us/sample - loss: 0.5153 - accuracy: 0.8208 - val_loss: 0.4463 - val_accuracy: 0.8462\n",
      "Epoch 4/10\n",
      "55000/55000 [==============================] - 4s 76us/sample - loss: 0.4846 - accuracy: 0.8307 - val_loss: 0.4256 - val_accuracy: 0.8530\n",
      "Epoch 5/10\n",
      "55000/55000 [==============================] - 4s 79us/sample - loss: 0.4576 - accuracy: 0.8402 - val_loss: 0.4106 - val_accuracy: 0.8590\n",
      "Epoch 6/10\n",
      "55000/55000 [==============================] - 4s 77us/sample - loss: 0.4401 - accuracy: 0.8467 - val_loss: 0.3973 - val_accuracy: 0.8610\n",
      "Epoch 7/10\n",
      "55000/55000 [==============================] - 4s 78us/sample - loss: 0.4296 - accuracy: 0.8482 - val_loss: 0.3899 - val_accuracy: 0.8650\n",
      "Epoch 8/10\n",
      "55000/55000 [==============================] - 4s 76us/sample - loss: 0.4127 - accuracy: 0.8559 - val_loss: 0.3818 - val_accuracy: 0.8658\n",
      "Epoch 9/10\n",
      "55000/55000 [==============================] - 4s 78us/sample - loss: 0.4007 - accuracy: 0.8588 - val_loss: 0.3741 - val_accuracy: 0.8682\n",
      "Epoch 10/10\n",
      "55000/55000 [==============================] - 4s 79us/sample - loss: 0.3929 - accuracy: 0.8621 - val_loss: 0.3694 - val_accuracy: 0.8734\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Clipping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All Keras optimizers accept `clipnorm` or `clipvalue` arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(clipvalue=1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(clipnorm=1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reusing Pretrained Layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reusing a Keras model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's split the fashion MNIST training set in two:\n",
    "* `X_train_A`: all images of all items except for sandals and shirts (classes 5 and 6).\n",
    "* `X_train_B`: a much smaller training set of just the first 200 images of sandals or shirts.\n",
    "\n",
    "The validation set and the test set are also split this way, but without restricting the number of images.\n",
    "\n",
    "We will train a model on set A (classification task with 8 classes), and try to reuse it to tackle set B (binary classification). We hope to transfer a little bit of knowledge from task A to task B, since classes in set A (sneakers, ankle boots, coats, t-shirts, etc.) are somewhat similar to classes in set B (sandals and shirts). However, since we are using `Dense` layers, only patterns that occur at the same location can be reused (in contrast, convolutional layers will transfer much better, since learned patterns can be detected anywhere on the image, as we will see in the CNN chapter)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_dataset(X, y):\n",
    "    y_5_or_6 = (y == 5) | (y == 6) # sandals or shirts\n",
    "    y_A = y[~y_5_or_6]\n",
    "    y_A[y_A > 6] -= 2 # class indices 7, 8, 9 should be moved to 5, 6, 7\n",
    "    y_B = (y[y_5_or_6] == 6).astype(np.float32) # binary classification task: is it a shirt (class 6)?\n",
    "    return ((X[~y_5_or_6], y_A),\n",
    "            (X[y_5_or_6], y_B))\n",
    "\n",
    "(X_train_A, y_train_A), (X_train_B, y_train_B) = split_dataset(X_train, y_train)\n",
    "(X_valid_A, y_valid_A), (X_valid_B, y_valid_B) = split_dataset(X_valid, y_valid)\n",
    "(X_test_A, y_test_A), (X_test_B, y_test_B) = split_dataset(X_test, y_test)\n",
    "X_train_B = X_train_B[:200]\n",
    "y_train_B = y_train_B[:200]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(43986, 28, 28)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_A.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(200, 28, 28)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_B.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4, 0, 5, 7, 7, 7, 4, 4, 3, 4, 0, 1, 6, 3, 4, 3, 2, 6, 5, 3, 4, 5,\n",
       "       1, 3, 4, 2, 0, 6, 7, 1], dtype=uint8)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_A[:30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 1., 0., 0., 0., 0., 1., 1., 1., 0., 0., 1., 1., 0., 0., 0., 0.,\n",
       "       0., 0., 1., 1., 0., 0., 1., 1., 0., 1., 1., 1., 1.], dtype=float32)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_B[:30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A = keras.models.Sequential()\n",
    "model_A.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "for n_hidden in (300, 100, 50, 50, 50):\n",
    "    model_A.add(keras.layers.Dense(n_hidden, activation=\"selu\"))\n",
    "model_A.add(keras.layers.Dense(8, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"sgd\",\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 43986 samples, validate on 4014 samples\n",
      "Epoch 1/20\n",
      "43986/43986 [==============================] - 3s 78us/sample - loss: 0.5887 - accuracy: 0.8123 - val_loss: 0.3749 - val_accuracy: 0.8734\n",
      "Epoch 2/20\n",
      "43986/43986 [==============================] - 3s 69us/sample - loss: 0.3516 - accuracy: 0.8793 - val_loss: 0.3223 - val_accuracy: 0.8874\n",
      "Epoch 3/20\n",
      "43986/43986 [==============================] - 3s 68us/sample - loss: 0.3160 - accuracy: 0.8894 - val_loss: 0.3009 - val_accuracy: 0.8956\n",
      "Epoch 4/20\n",
      "43986/43986 [==============================] - 3s 70us/sample - loss: 0.2963 - accuracy: 0.8979 - val_loss: 0.2850 - val_accuracy: 0.9036\n",
      "Epoch 5/20\n",
      "43986/43986 [==============================] - 3s 68us/sample - loss: 0.2825 - accuracy: 0.9035 - val_loss: 0.2767 - val_accuracy: 0.9076\n",
      "Epoch 6/20\n",
      "43986/43986 [==============================] - 3s 69us/sample - loss: 0.2720 - accuracy: 0.9068 - val_loss: 0.2672 - val_accuracy: 0.9093\n",
      "Epoch 7/20\n",
      "43986/43986 [==============================] - 3s 72us/sample - loss: 0.2638 - accuracy: 0.9093 - val_loss: 0.2658 - val_accuracy: 0.9103\n",
      "Epoch 8/20\n",
      "43986/43986 [==============================] - 3s 70us/sample - loss: 0.2570 - accuracy: 0.9120 - val_loss: 0.2592 - val_accuracy: 0.9106\n",
      "Epoch 9/20\n",
      "43986/43986 [==============================] - 3s 71us/sample - loss: 0.2514 - accuracy: 0.9139 - val_loss: 0.2570 - val_accuracy: 0.9128\n",
      "Epoch 10/20\n",
      "43986/43986 [==============================] - 3s 72us/sample - loss: 0.2465 - accuracy: 0.9166 - val_loss: 0.2557 - val_accuracy: 0.9108\n",
      "Epoch 11/20\n",
      "43986/43986 [==============================] - 3s 69us/sample - loss: 0.2418 - accuracy: 0.9178 - val_loss: 0.2484 - val_accuracy: 0.9178\n",
      "Epoch 12/20\n",
      "43986/43986 [==============================] - 3s 70us/sample - loss: 0.2379 - accuracy: 0.9192 - val_loss: 0.2461 - val_accuracy: 0.9178\n",
      "Epoch 13/20\n",
      "43986/43986 [==============================] - 3s 71us/sample - loss: 0.2342 - accuracy: 0.9199 - val_loss: 0.2425 - val_accuracy: 0.9188\n",
      "Epoch 14/20\n",
      "43986/43986 [==============================] - 3s 68us/sample - loss: 0.2313 - accuracy: 0.9215 - val_loss: 0.2412 - val_accuracy: 0.9185\n",
      "Epoch 15/20\n",
      "43986/43986 [==============================] - 3s 68us/sample - loss: 0.2280 - accuracy: 0.9222 - val_loss: 0.2382 - val_accuracy: 0.9173\n",
      "Epoch 16/20\n",
      "43986/43986 [==============================] - 3s 71us/sample - loss: 0.2252 - accuracy: 0.9224 - val_loss: 0.2360 - val_accuracy: 0.9205\n",
      "Epoch 17/20\n",
      "43986/43986 [==============================] - 3s 71us/sample - loss: 0.2229 - accuracy: 0.9232 - val_loss: 0.2419 - val_accuracy: 0.9158\n",
      "Epoch 18/20\n",
      "43986/43986 [==============================] - 3s 71us/sample - loss: 0.2195 - accuracy: 0.9249 - val_loss: 0.2357 - val_accuracy: 0.9170\n",
      "Epoch 19/20\n",
      "43986/43986 [==============================] - 3s 68us/sample - loss: 0.2177 - accuracy: 0.9254 - val_loss: 0.2331 - val_accuracy: 0.9200\n",
      "Epoch 20/20\n",
      "43986/43986 [==============================] - 3s 70us/sample - loss: 0.2154 - accuracy: 0.9260 - val_loss: 0.2372 - val_accuracy: 0.9158\n"
     ]
    }
   ],
   "source": [
    "history = model_A.fit(X_train_A, y_train_A, epochs=20,\n",
    "                    validation_data=(X_valid_A, y_valid_A))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A.save(\"my_model_A.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_B = keras.models.Sequential()\n",
    "model_B.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "for n_hidden in (300, 100, 50, 50, 50):\n",
    "    model_B.add(keras.layers.Dense(n_hidden, activation=\"selu\"))\n",
    "model_B.add(keras.layers.Dense(1, activation=\"sigmoid\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_B.compile(loss=\"binary_crossentropy\", optimizer=\"sgd\",\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 200 samples, validate on 986 samples\n",
      "Epoch 1/20\n",
      "200/200 [==============================] - 0s 2ms/sample - loss: 0.9537 - accuracy: 0.4800 - val_loss: 0.6472 - val_accuracy: 0.5710\n",
      "Epoch 2/20\n",
      "200/200 [==============================] - 0s 318us/sample - loss: 0.5805 - accuracy: 0.6850 - val_loss: 0.4863 - val_accuracy: 0.8428\n",
      "Epoch 3/20\n",
      "200/200 [==============================] - 0s 318us/sample - loss: 0.4561 - accuracy: 0.8750 - val_loss: 0.4116 - val_accuracy: 0.8905\n",
      "Epoch 4/20\n",
      "200/200 [==============================] - 0s 308us/sample - loss: 0.3885 - accuracy: 0.9100 - val_loss: 0.3650 - val_accuracy: 0.9148\n",
      "Epoch 5/20\n",
      "200/200 [==============================] - 0s 311us/sample - loss: 0.3426 - accuracy: 0.9250 - val_loss: 0.3308 - val_accuracy: 0.9270\n",
      "Epoch 6/20\n",
      "200/200 [==============================] - 0s 317us/sample - loss: 0.3084 - accuracy: 0.9300 - val_loss: 0.3044 - val_accuracy: 0.9371\n",
      "Epoch 7/20\n",
      "200/200 [==============================] - 0s 309us/sample - loss: 0.2810 - accuracy: 0.9400 - val_loss: 0.2806 - val_accuracy: 0.9432\n",
      "Epoch 8/20\n",
      "200/200 [==============================] - 0s 313us/sample - loss: 0.2572 - accuracy: 0.9500 - val_loss: 0.2607 - val_accuracy: 0.9462\n",
      "Epoch 9/20\n",
      "200/200 [==============================] - 0s 312us/sample - loss: 0.2372 - accuracy: 0.9600 - val_loss: 0.2439 - val_accuracy: 0.9513\n",
      "Epoch 10/20\n",
      "200/200 [==============================] - 0s 319us/sample - loss: 0.2202 - accuracy: 0.9600 - val_loss: 0.2290 - val_accuracy: 0.9523\n",
      "Epoch 11/20\n",
      "200/200 [==============================] - 0s 315us/sample - loss: 0.2047 - accuracy: 0.9650 - val_loss: 0.2161 - val_accuracy: 0.9564\n",
      "Epoch 12/20\n",
      "200/200 [==============================] - 0s 325us/sample - loss: 0.1917 - accuracy: 0.9700 - val_loss: 0.2046 - val_accuracy: 0.9584\n",
      "Epoch 13/20\n",
      "200/200 [==============================] - 0s 335us/sample - loss: 0.1798 - accuracy: 0.9750 - val_loss: 0.1944 - val_accuracy: 0.9604\n",
      "Epoch 14/20\n",
      "200/200 [==============================] - 0s 319us/sample - loss: 0.1690 - accuracy: 0.9750 - val_loss: 0.1860 - val_accuracy: 0.9604\n",
      "Epoch 15/20\n",
      "200/200 [==============================] - 0s 319us/sample - loss: 0.1594 - accuracy: 0.9850 - val_loss: 0.1774 - val_accuracy: 0.9635\n",
      "Epoch 16/20\n",
      "200/200 [==============================] - 0s 343us/sample - loss: 0.1508 - accuracy: 0.9850 - val_loss: 0.1691 - val_accuracy: 0.9675\n",
      "Epoch 17/20\n",
      "200/200 [==============================] - 0s 328us/sample - loss: 0.1426 - accuracy: 0.9900 - val_loss: 0.1621 - val_accuracy: 0.9686\n",
      "Epoch 18/20\n",
      "200/200 [==============================] - 0s 340us/sample - loss: 0.1355 - accuracy: 0.9900 - val_loss: 0.1558 - val_accuracy: 0.9706\n",
      "Epoch 19/20\n",
      "200/200 [==============================] - 0s 306us/sample - loss: 0.1288 - accuracy: 0.9900 - val_loss: 0.1505 - val_accuracy: 0.9706\n",
      "Epoch 20/20\n",
      "200/200 [==============================] - 0s 312us/sample - loss: 0.1230 - accuracy: 0.9900 - val_loss: 0.1454 - val_accuracy: 0.9716\n"
     ]
    }
   ],
   "source": [
    "history = model_B.fit(X_train_B, y_train_B, epochs=20,\n",
    "                      validation_data=(X_valid_B, y_valid_B))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_4\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_4 (Flatten)          (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_v2_3 (Ba (None, 784)               3136      \n",
      "_________________________________________________________________\n",
      "dense_213 (Dense)            (None, 300)               235500    \n",
      "_________________________________________________________________\n",
      "batch_normalization_v2_4 (Ba (None, 300)               1200      \n",
      "_________________________________________________________________\n",
      "activation (Activation)      (None, 300)               0         \n",
      "_________________________________________________________________\n",
      "dense_214 (Dense)            (None, 100)               30100     \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 100)               0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_v2_5 (Ba (None, 100)               400       \n",
      "_________________________________________________________________\n",
      "dense_215 (Dense)            (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 271,346\n",
      "Trainable params: 268,978\n",
      "Non-trainable params: 2,368\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A = keras.models.load_model(\"my_model_A.h5\")\n",
    "model_B_on_A = keras.models.Sequential(model_A.layers[:-1])\n",
    "model_B_on_A.add(keras.layers.Dense(1, activation=\"sigmoid\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A_clone = keras.models.clone_model(model_A)\n",
    "model_A_clone.set_weights(model_A.get_weights())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "for layer in model_B_on_A.layers[:-1]:\n",
    "    layer.trainable = False\n",
    "\n",
    "model_B_on_A.compile(loss=\"binary_crossentropy\", optimizer=\"sgd\",\n",
    "                     metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 200 samples, validate on 986 samples\n",
      "Epoch 1/4\n",
      "200/200 [==============================] - 0s 2ms/sample - loss: 0.5851 - accuracy: 0.6600 - val_loss: 0.5855 - val_accuracy: 0.6318\n",
      "Epoch 2/4\n",
      "200/200 [==============================] - 0s 303us/sample - loss: 0.5484 - accuracy: 0.6850 - val_loss: 0.5484 - val_accuracy: 0.6775\n",
      "Epoch 3/4\n",
      "200/200 [==============================] - 0s 294us/sample - loss: 0.5116 - accuracy: 0.7250 - val_loss: 0.5141 - val_accuracy: 0.7160\n",
      "Epoch 4/4\n",
      "200/200 [==============================] - 0s 316us/sample - loss: 0.4779 - accuracy: 0.7450 - val_loss: 0.4859 - val_accuracy: 0.7363\n",
      "Train on 200 samples, validate on 986 samples\n",
      "Epoch 1/16\n",
      "200/200 [==============================] - 0s 2ms/sample - loss: 0.3989 - accuracy: 0.8050 - val_loss: 0.3419 - val_accuracy: 0.8702\n",
      "Epoch 2/16\n",
      "200/200 [==============================] - 0s 328us/sample - loss: 0.2795 - accuracy: 0.9300 - val_loss: 0.2624 - val_accuracy: 0.9280\n",
      "Epoch 3/16\n",
      "200/200 [==============================] - 0s 319us/sample - loss: 0.2128 - accuracy: 0.9650 - val_loss: 0.2150 - val_accuracy: 0.9544\n",
      "Epoch 4/16\n",
      "200/200 [==============================] - 0s 318us/sample - loss: 0.1720 - accuracy: 0.9800 - val_loss: 0.1826 - val_accuracy: 0.9635\n",
      "Epoch 5/16\n",
      "200/200 [==============================] - 0s 317us/sample - loss: 0.1436 - accuracy: 0.9800 - val_loss: 0.1586 - val_accuracy: 0.9736\n",
      "Epoch 6/16\n",
      "200/200 [==============================] - 0s 317us/sample - loss: 0.1231 - accuracy: 0.9850 - val_loss: 0.1407 - val_accuracy: 0.9807\n",
      "Epoch 7/16\n",
      "200/200 [==============================] - 0s 325us/sample - loss: 0.1074 - accuracy: 0.9900 - val_loss: 0.1270 - val_accuracy: 0.9828\n",
      "Epoch 8/16\n",
      "200/200 [==============================] - 0s 326us/sample - loss: 0.0953 - accuracy: 0.9950 - val_loss: 0.1158 - val_accuracy: 0.9848\n",
      "Epoch 9/16\n",
      "200/200 [==============================] - 0s 319us/sample - loss: 0.0854 - accuracy: 1.0000 - val_loss: 0.1076 - val_accuracy: 0.9878\n",
      "Epoch 10/16\n",
      "200/200 [==============================] - 0s 322us/sample - loss: 0.0781 - accuracy: 1.0000 - val_loss: 0.1007 - val_accuracy: 0.9888\n",
      "Epoch 11/16\n",
      "200/200 [==============================] - 0s 316us/sample - loss: 0.0718 - accuracy: 1.0000 - val_loss: 0.0944 - val_accuracy: 0.9888\n",
      "Epoch 12/16\n",
      "200/200 [==============================] - 0s 319us/sample - loss: 0.0662 - accuracy: 1.0000 - val_loss: 0.0891 - val_accuracy: 0.9899\n",
      "Epoch 13/16\n",
      "200/200 [==============================] - 0s 318us/sample - loss: 0.0613 - accuracy: 1.0000 - val_loss: 0.0846 - val_accuracy: 0.9899\n",
      "Epoch 14/16\n",
      "200/200 [==============================] - 0s 332us/sample - loss: 0.0574 - accuracy: 1.0000 - val_loss: 0.0806 - val_accuracy: 0.9899\n",
      "Epoch 15/16\n",
      "200/200 [==============================] - 0s 320us/sample - loss: 0.0538 - accuracy: 1.0000 - val_loss: 0.0770 - val_accuracy: 0.9899\n",
      "Epoch 16/16\n",
      "200/200 [==============================] - 0s 320us/sample - loss: 0.0505 - accuracy: 1.0000 - val_loss: 0.0740 - val_accuracy: 0.9899\n"
     ]
    }
   ],
   "source": [
    "history = model_B_on_A.fit(X_train_B, y_train_B, epochs=4,\n",
    "                           validation_data=(X_valid_B, y_valid_B))\n",
    "\n",
    "for layer in model_B_on_A.layers[:-1]:\n",
    "    layer.trainable = True\n",
    "\n",
    "model_B_on_A.compile(loss=\"binary_crossentropy\", optimizer=\"sgd\",\n",
    "                     metrics=[\"accuracy\"])\n",
    "history = model_B_on_A.fit(X_train_B, y_train_B, epochs=16,\n",
    "                           validation_data=(X_valid_B, y_valid_B))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, what's the final verdict?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000/2000 [==============================] - 0s 41us/sample - loss: 0.1431 - accuracy: 0.9705\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.1430660070180893, 0.9705]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_B.evaluate(X_test_B, y_test_B)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000/2000 [==============================] - 0s 38us/sample - loss: 0.0689 - accuracy: 0.9925\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.06887910133600235, 0.9925]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_B_on_A.evaluate(X_test_B, y_test_B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! We got quite a bit of transfer: the error rate dropped by a factor of almost 4!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.933333333333337"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(100 - 97.05) / (100 - 99.25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Faster Optimizers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Momentum optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(lr=0.001, momentum=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nesterov Accelerated Gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(lr=0.001, momentum=0.9, nesterov=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AdaGrad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Adagrad(lr=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RMSProp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.RMSprop(lr=0.001, rho=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adam Optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adamax Optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Adamax(lr=0.001, beta_1=0.9, beta_2=0.999)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nadam Optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Nadam(lr=0.001, beta_1=0.9, beta_2=0.999)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Rate Scheduling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Power Scheduling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```lr = lr0 / (1 + steps / s)**c```\n",
    "* Keras uses `c=1` and `s = 1 / decay`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(lr=0.01, decay=1e-4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/25\n",
      "55000/55000 [==============================] - 4s 66us/sample - loss: 0.4840 - accuracy: 0.8296 - val_loss: 0.4038 - val_accuracy: 0.8630\n",
      "Epoch 2/25\n",
      "55000/55000 [==============================] - 3s 63us/sample - loss: 0.3787 - accuracy: 0.8653 - val_loss: 0.3846 - val_accuracy: 0.8706\n",
      "Epoch 3/25\n",
      "55000/55000 [==============================] - 3s 62us/sample - loss: 0.3461 - accuracy: 0.8770 - val_loss: 0.3606 - val_accuracy: 0.8776\n",
      "Epoch 4/25\n",
      "55000/55000 [==============================] - 3s 63us/sample - loss: 0.3248 - accuracy: 0.8844 - val_loss: 0.3661 - val_accuracy: 0.8738\n",
      "Epoch 5/25\n",
      "55000/55000 [==============================] - 3s 62us/sample - loss: 0.3092 - accuracy: 0.8902 - val_loss: 0.3516 - val_accuracy: 0.8792\n",
      "Epoch 6/25\n",
      "55000/55000 [==============================] - 3s 63us/sample - loss: 0.2967 - accuracy: 0.8938 - val_loss: 0.3467 - val_accuracy: 0.8810\n",
      "Epoch 7/25\n",
      "55000/55000 [==============================] - 3s 63us/sample - loss: 0.2862 - accuracy: 0.8967 - val_loss: 0.3398 - val_accuracy: 0.8844\n",
      "Epoch 8/25\n",
      "55000/55000 [==============================] - 3s 61us/sample - loss: 0.2771 - accuracy: 0.8997 - val_loss: 0.3384 - val_accuracy: 0.8832\n",
      "Epoch 9/25\n",
      "55000/55000 [==============================] - 3s 62us/sample - loss: 0.2696 - accuracy: 0.9035 - val_loss: 0.3345 - val_accuracy: 0.8860\n",
      "Epoch 10/25\n",
      "55000/55000 [==============================] - 3s 62us/sample - loss: 0.2628 - accuracy: 0.9057 - val_loss: 0.3343 - val_accuracy: 0.8830\n",
      "Epoch 11/25\n",
      "55000/55000 [==============================] - 3s 61us/sample - loss: 0.2568 - accuracy: 0.9083 - val_loss: 0.3290 - val_accuracy: 0.8882\n",
      "Epoch 12/25\n",
      "55000/55000 [==============================] - 3s 62us/sample - loss: 0.2510 - accuracy: 0.9099 - val_loss: 0.3243 - val_accuracy: 0.8904\n",
      "Epoch 13/25\n",
      "55000/55000 [==============================] - 3s 61us/sample - loss: 0.2459 - accuracy: 0.9118 - val_loss: 0.3271 - val_accuracy: 0.8874\n",
      "Epoch 14/25\n",
      "55000/55000 [==============================] - 3s 62us/sample - loss: 0.2415 - accuracy: 0.9130 - val_loss: 0.3259 - val_accuracy: 0.8886\n",
      "Epoch 15/25\n",
      "55000/55000 [==============================] - 3s 62us/sample - loss: 0.2370 - accuracy: 0.9157 - val_loss: 0.3249 - val_accuracy: 0.8896\n",
      "Epoch 16/25\n",
      "55000/55000 [==============================] - 3s 61us/sample - loss: 0.2332 - accuracy: 0.9177 - val_loss: 0.3267 - val_accuracy: 0.8892\n",
      "Epoch 17/25\n",
      "55000/55000 [==============================] - 3s 63us/sample - loss: 0.2296 - accuracy: 0.9177 - val_loss: 0.3251 - val_accuracy: 0.8880\n",
      "Epoch 18/25\n",
      "55000/55000 [==============================] - 3s 61us/sample - loss: 0.2257 - accuracy: 0.9194 - val_loss: 0.3221 - val_accuracy: 0.8900\n",
      "Epoch 19/25\n",
      "55000/55000 [==============================] - 3s 61us/sample - loss: 0.2228 - accuracy: 0.9212 - val_loss: 0.3237 - val_accuracy: 0.8910\n",
      "Epoch 20/25\n",
      "55000/55000 [==============================] - 3s 60us/sample - loss: 0.2198 - accuracy: 0.9223 - val_loss: 0.3217 - val_accuracy: 0.8904\n",
      "Epoch 21/25\n",
      "55000/55000 [==============================] - 3s 63us/sample - loss: 0.2166 - accuracy: 0.9238 - val_loss: 0.3185 - val_accuracy: 0.8938\n",
      "Epoch 22/25\n",
      "55000/55000 [==============================] - 3s 61us/sample - loss: 0.2140 - accuracy: 0.9252 - val_loss: 0.3212 - val_accuracy: 0.8902\n",
      "Epoch 23/25\n",
      "55000/55000 [==============================] - 3s 62us/sample - loss: 0.2113 - accuracy: 0.9256 - val_loss: 0.3235 - val_accuracy: 0.8898\n",
      "Epoch 24/25\n",
      "55000/55000 [==============================] - 3s 62us/sample - loss: 0.2088 - accuracy: 0.9262 - val_loss: 0.3216 - val_accuracy: 0.8930\n",
      "Epoch 25/25\n",
      "55000/55000 [==============================] - 3s 62us/sample - loss: 0.2061 - accuracy: 0.9273 - val_loss: 0.3199 - val_accuracy: 0.8922\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 25\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEeCAYAAABc5biTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl8VNXdx/HPLxuEQNgNJLKpyCqLuIJW6m4fF9Rqn9a1Llj7+NjHtWpri7bWWrW1tq5Vi2vFVhAVlaqYKqiIimwii7JI2AQkEEhCCL/nj3uDwzBJJpiZyfJ9v17zytxzzr3zm2vMj3PvueeYuyMiIpJIaakOQEREmj4lGxERSTglGxERSTglGxERSTglGxERSTglGxERSTglG5FGxMyWmtm1CTju982sTs9BmNmFZlZS3bZIJCUbaVTMbKyZefiqMLMvzOwuM8tJdWzxMLNLzGymmZWYWbGZzTaz36Y6rnoyDtgn1UFIw5SR6gBE9sAbwHlAJnAk8AiQA1yeyqCqmFmWu2+LUX4RcC9wFfAmQfwDgcOTG2FiuHspUJrqOKRhUs9GGqNyd1/t7l+6+zPA08Coqkoz+46ZTTezMjNbY2Z/MrOssO5EM9tsZhnh9n5hL+nBiP1/a2ZvRGz3N7NJ4X5rzewfZtYlon6smb1sZj83sxXAimriPhUY7+4Puftid5/v7v9096sjG5nZ98L4S81svZm9ZGYtI5q0NLOHzGyTma0ws+ui9m9rZg+HsW42s/+Y2UFRbc43s2VmttXMXgbyourHmNncqLIaL5PFuKw2xszmmtl/m9nnYSwvmFmniDYZ4X+fr8PXn8zsATMrrO5zpHFSspGmoJSgl4CZFQCvAjOBocDFwA+B28O2U4GWQNUf35HAuvAnEWWF4fG6Am8Dc4FDgGOB1sBEM4v8/+coYBBwInBMNXGuBg4xs2ovNZnZicCLwOvAMOC7wH/Y9f/Vq4A5wIHAHcAfzOzwcH8DJgEFwMnhOXgbmBJ+F8zsUGAs8DAwBHgJuLW6mL6lnsAPgNOB48N4bouovxa4ELgEOIzge/4oQbFIKrm7Xno1mhfBH8mXI7YPIUgW48Lt24BFQFpEmwuBcqBVuP0+cGP4/ing1wQJqyvQKmx7RFh/K/BmVAztAQcOiYjpK6BFLbF3Bd4L910Ufvb5QGZEm2nAszUcYynwj6iyRcAvw/dHAyVAdlSbT4Drw/fPAK9H1T8S/DnYuT0GmBvV5kKgpA7bY4AyoG1E2S+AxRHbq4AbIrYNWAAUpvp3Ta/6falnI43RieEN9jKCP95vA/8b1vUD3nf3HRHtpwJZwH7hdiHf9GSOIugJTQ/LhgPbgQ/C+mHAd8LPKwkvE30Z1u0b8Rlz3b28pqDdfZW7Hw4cANxD8If1IeADM2sVNhtKcD+nJrOjtlcCe0XE2wr4KirmgRHx9iM4b5Git+vLMncvjhWrmbUFuvDNuQ6yXcS2NB0aICCN0dvAaKACWOnuFXHuVzW0txC4wsz6AbnAR2HZd4G1wHv+zQ3+NILLUrGGG6+JeL8l3uDdfS7BZbn7zOwI4B3gbIIeUjyiv6/zzWW2tDCuI2PstyneGIEdBMkwUmYd9q9SU6zSjCjZSGO01d0XV1M3HzjbzNIiejdHANuAz8PtqUAL4HpgqrtXhjek/0bwh/q1iON9TJAIltUhqdXFp+HP1uHPmQT3fP62h8f7mOBm/w53/6KaNvMJ7o9Eit7+CsgzMwt7GxDc36k37l5sZquBg4EpsPOe08EE97ekCdG/MKSpuR/IB+43s35m9l/A74G/uvtWAHcvIejNnAu8Fe73PrA3wR/dwojj3Qe0BcaZ2aFmto+ZHRuO9mpTl8DCUVY3m9kIM+thZocBTwBbgX+HzW4DzgpHxPU3swFmdlXEZbbavEFw32eimZ1kZr3M7HAzu8XMqno79wLHmtmNZtbbzC4luIEfqRDoANxkZvua2cXA9+vyfeP0Z+B6MzvdzPoAdxPc29JCW02Mko00Ke5eBJxEcO/jE+Ax4B/ATVFNCwl69oXhfmUE923K2fUewkpgBMFlpdeAeQQJqDx81cXrwKHAc8BCYEJYfpy7Lww/7xWCP/wnEfRy/kNweW/HbkeLIeyFfI+gp/A3gpvtzwF9CO6X4O7vE4zSu5zg/s8ZBDfzI48zP6wfHbY5DvhdHb9vPO4CngT+TpDwITgvZQn4LEkh+6aHLCKSemY2k+Dy5v/W2lgaDd2zEZGUMbMewAkEPbhM4FKC55UuTWVcUv+SehnNzDqY2QQz2xI+vRzz4S0L3BE+Pb0+fG8R9Q+b2QIz22FmF8bY/yozWx0+Yf2YmbVI4NcSkT23g+BZow8Ihl8fBpzk7h+mNCqpd8m+Z3MfwaigPOAc4AEzGxCj3WiC6UcGE/wr5xTgsoj6WcBPCUbe7MLMTgBuIBjR04NgYsBb6u8riEh98WDKoSPcva27t3H3Q93937XvKY1N0u7ZWDAr79fAwKqboWb2JFDk7jdEtX0XGOvuD4fbFwOXuvthUe2mAo+4+9iIsmeApe5+U7h9DPC0u3dBRERSIpn3bPYHtlclmtAsgie4ow0I6yLbxeoBxTIAmBi1b56ZdXT39ZENzWw0QS+KtOzcYRlt99pZ1zNXA/UAduzYQVqazkU0nZfd6ZzE1tTPy8KFC9e5e+fa2iUz2bRm9yeYi4FYzyq0Dusi27WOesCsps+J3pfwc3ZJNmHP6WGAFl17e9cL7gGgoF020244upaPaR4KCwsZOXJkqsNocHRedqdzEltTPy9mtiyedslMtyUEU4NEygU2x9E2l2CCv3iu+cXal2o+ZzctM9O47oQ+8TQVEZE4JTPZLAQyzKx3RNlggofkos0L62prF0usfddEX0KrzmmD8xk1tCDOjxIRkXgkLdm4+xZgPHCrmeWY2QjgNIKnh6M9AVxtZgVmlg9cQ8QkhWaWZcFiUgZkmlnLiLVFngAuDqf6aAf8kjgmOOyZm0b/rrnM/HIjetBVRKR+Jfuu1U+BbIKZdf8BXO7u88zsSNt1BcCHCBZ0mkMwO+6ksKzKvwnWHxlOcM+lFPgOgLu/BvyBYM6r5cAygvVKanXREb1YuKaEaYvj6gSJiEickjqDgLtvIGL53ojyd/hm1tuq+Z2uD1+xjjOyls/5I/DHusZ3yuCu/P7V+Tw2bQlH9O5U+w4iIhKXpjsebw+0yEjn3MN6MOWztXzxVbVLrYuISB0p2UQ559AeZKWnMfbdpakORUSkyVCyidK5TQtOHZLPPz9cQfHWRKyVJSLS/CjZxPDjET0prahk3IfLUx2KiEiToGQTw4D8thy2Twcef3cZ2yvjWrNKRERqoGRTjYtG9KJoYyn//nRNqkMREWn0lGyqcUy/PLp3aMVjU5ekOhQRkUZPyaYa6WnGhcN78uGyr5n15cZUhyMi0qgp2dTgrIP2pnWLDP4+Tb0bEZFvQ8mmBm1aZnL2Qd14efYq1mwqS3U4IiKNlpJNLS4c3pNKd558L64lG0REJAYlm1p079iK4/vn8fT0ZZRVVKY6HBGRRknJJg4XjejF11sreGFmUapDERFplJRs4nBIrw4MyM/lsWlLtNaNiMgeULKJg5lx0QitdSMisqeUbOJ08uCudGrdgsc0DFpEpM6UbOLUIiOd88K1bj7XWjciInWiZFMH5xzWPVjrZtrSVIciItKoKNnUQafWLThtSD7/+khr3YiI1IWSTR39eEQvSisqeXaG1roREYmXkk0d9c/P5fB9OvL4u0u11o2ISJyUbPbARUf0YmVxGZPnaa0bEZF4KNnsgaP77kXHnEyuGvcJvW6YxIjfT9HsAiIiNchIdQCN0UuzVrKpbDsVlcFsAkUbS7lx/BwARg0tSGVoIiINkno2e+DOyQt2JpoqpRWV3Dl5QYoiEhFp2JRs9sDKjaV1KhcRae6UbPZAfrvsOpWLiDR3SjZ74LoT+pCdmb5LWWa6cd0JfVIUkYhIw6YBAnugahDAnZMXsHJjKRnpRnZmGsf1z0txZCIiDZOSzR4aNbRgZ9L5aNnXnPnAu9z31mKuP7FviiMTEWl4dBmtHgzr0Z4zDizgkXeWsHTdllSHIyLS4CjZ1JMbTuxLVkYav3n501SHIiLS4CjZ1JO9clty5TH78eZna3nrs7WpDkdEpEFRsqlHFw7vxT6dc7j15U8p316Z6nBERBqMpCYbM+tgZhPMbIuZLTOzH1XTzszsDjNbH77uMDOLqB9iZh+Z2dbw55CIuhZm9qCZrTGzDWb2kpklZQ6ZrIw0fnVyf5as28JjU5cm4yNFRBqFZPds7gO2AXnAOcADZjYgRrvRwChgMDAIOAW4DMDMsoCJwFNAe+BxYGJYDvAz4PBwv3zga+AvCfo+uxnZZy+O7ZfHX6YsYs2msmR9rIhIg5a0ZGNmOcCZwM3uXuLuU4EXgfNiNL8AuNvdV7h7EXA3cGFYN5JgyPY97l7u7vcCBhwd1vcCJrv7GncvA8YBsRJawvzq5P5s3+Hc/sr8ZH6siEiDlcznbPYHtrv7woiyWcBRMdoOCOsi2w2IqJvt7pEzYc4Oy18DHgX+bGb5wEaCHtSrsQIys9EEvSg6d+5MYWFhHb9S9U7ons4Ln6xkQIsN9G6fXvsODVRJSUm9npemQudldzonsem8BJKZbFoDm6LKioE21bQtjmrXOrxvE10XfZxFwJdAEVAJzAGuiBWQuz8MPAzQp08fHzlyZJxfpXaHDN/Oh3f/hwnLs3jptCNIT7Pad2qACgsLqc/z0lTovOxO5yQ2nZdAMu/ZlAC5UWW5wOY42uYCJWFvprbj3Ae0ADoCOcB4qunZJFKrrAxu+l4/Pl21iWdnLE/2x4uINCjJTDYLgQwz6x1RNhiYF6PtvLAuVrt5wKDI0WkEgwGq6ocAY919g7uXEwwOOMTMOtXDd6iTkwd15dBeHbhr8gI2bt2W7I8XEWkwkpZs3H0LQS/jVjPLMbMRwGnAkzGaPwFcbWYF4b2Xa4CxYV0hweWxK8NhzlWXyKaEP2cA55tZWzPLBH4KrHT3dYn4XjUxM8acOoDi0gr++PrC2ncQEWmikj30+adANrAW+AdwubvPM7Mjzawkot1DwEsE91vmApPCMtx9G8Gw6PMJBgBcBIwKywGuBcoI7t18BXwPOD3B36ta/brmct5hPXjq/WV8ujL6lpWISPOQ1Fmf3X0DQaKILn+H4MZ/1bYD14evWMeZCQyrpm49wQi0BuOq4/bnxVkrGfPSPMaNPoxdrwCKiDR9mq4mCdq1yuK6E/rywZINvDR7VarDERFJOiWbJPnBwd0YWJDL7ybNZ+u27akOR0QkqZRskiQ9zRhzygBWbyrjkNvepNcNkxjx+ym8MLMo1aGJiCScVupMohVfl5JuRkl50LMp2ljKjePnAN8sNS0i0hSpZ5NEd05eQOUus+xAaUUld05ekKKIRESSQ8kmiVZuLK1TuYhIU6Fkk0T57bLrVC4i0lQo2STRdSf0ITtz9xmgLx+5TwqiERFJHiWbJBo1tIDbzziAgnbZGNC5TQvSDSbPW8OOHV7r/iIijZVGoyXZqKEFu4w8e2b6cm6aMIe/vrWYK4/pXcOeIiKNl3o2KfbDQ7px+tAC/vTGQqYtTvpcoSIiSRF3sjGzPDO71sweqJqu38xGmFmvxIXX9JkZvx01kH07t+Znz85kzaayVIckIlLv4ko2ZjYMWEAwweXFfLN42XHAbYkJrfnIaZHBA+ccyJbySv73HzPZXrkj1SGJiNSreHs2dwF/dvehQHlE+WRgRL1H1Qz1zmvD784YyAdLNnDXv7X2jYg0LfEmm2HA4zHKVwF59RdO83b60L354SHdefA/n/Pm/DWpDkdEpN7Em2xKgfYxyvsSLIQm9eTXp/RnQH4uVz83iy83bE11OCIi9SLeZDMR+LWZtQi33cx6AncAzycgrmarZWY6959zIDt2OFc88zHl2ytTHZKIyLcWb7K5FuhAsMxyK2AqsJhgWeZfJia05qtHxxzuPGsQs1YUc/srn6U6HBGRby2uhzrdfRNwhJkdDRxIkKQ+dvc3Ehlcc3biwK5cfEQvHp26hIN7duC/BnVNdUgiInssrmRjZucD49x9CjAlojwL+G93fyJB8TVrN5zUl5nLv+bnz8+mX9c27NO5dapDEhHZI/FeRvs70DZGeZuwThIgMz2Nv/7oQDLTjXP+9j7Db9cKnyLSOMWbbAyINVNkd6C4/sKRaPntsjnroL1ZtamclcVlON+s8KmEIyKNRY2X0cxsDkGSceA/ZrY9ojod6AG8krjwBGDS7NW7lVWt8KnlpEWkMajtns2/wp8DgUlASUTdNmApGvqccFrhU0QauxqTjbvfAmBmSwkGCGiWyBTIb5dNUYzEohU+RaSxiOuejbs/rkSTOtWt8Pn9YbqEJiKNQ7yzPmeZ2S1mttDMysysMvKV6CCbu+gVPrvktqRTTiaPTVvK3CKNzxCRhi/elTp/A/wAuB34E3Ad0BP4b+DmhEQmu4he4bNoYylnP/ge5z06nXGXHc7+eW1SGJ2ISM3iHfp8NvATd38IqAQmuvuVwK8J1rSRJCtol83TlxxKZnoa5z4ynaXrtqQ6JBGRasWbbPKAT8P3JUC78P1rwPH1HZTEp2enHJ6+5FAqKndwziPTYw4iEBFpCOJNNsuB/PD9YuCE8P3hBMsPSIr0zmvDkxcfyqayCs752/us1bLSItIAxZtsJgDHhO//DNxiZkuAscAjCYhL6mBgQVvG/vgQ1m4u59xHp7Nhy7ZUhyQisot4hz7f6O63he//BRwB/AU4w91/kcD4JE7DerTnkQsOYtn6rZz/2HQ2lVWkOiQRkZ3i7dnswt2nu/sf3f1lM8uJdz8z62BmE8xsi5ktM7MfVdPOzOwOM1sfvu4wM4uoH2JmH5nZ1vDnkKj9DzSzt82sxMzWmNnP9uR7NjbD9+3Eg+cOY8Hqzfz47zPYUr699p1ERJJgj5INgJm1NLPrgCV12O0+gmlu8oBzgAfMbECMdqOBUcBgYBBwCnBZ+LlZBCuHPkWwVPXjwMSwHDPrRDBw4SGgI7Af8O+6fr/G6rt99+Le/x7KzOVfc+kTH1JWocegRCT1akw24cOct5nZDDN718xGheXnA18A/0fw3E2twh7QmcDN7l7i7lOBF4HzYjS/ALjb3Ve4exFwN3BhWDeS4Pmge9y93N3vJZiV+uiw/mpgsrs/HdZvdvf58cTYVJx0QFfuOmsw732xnjPun6alCUQk5Wp7qHMM8D/A68AI4J9m9jeCwQI3As+4e7w3B/YHtrv7woiyWcBRMdoOCOsi2w2IqJvt7pFLHswOy18DDgPmmNm7BL2a6cD/uPvy6A8xs9EEvSg6d+5MYWFhnF+l4esAHNE1nXdWbt5ZVrSxlOv/+Qmfzv+U4fmZcR2npKSkSZ2X+qLzsjudk9h0XgK1JZuzgQvdfYKZDQZmEly6GuDudb0h0BrYFFVWTLAAW6y2xVHtWof3baLroo+zN8HS1ccBc4A/AP8gSJa7cPeHgYcB+vTp4yNHjoz/2zQCv3h/CtEj07ftgEnL07npRyPjOkZhYSFN7bzUB52X3emcxKbzEqgt2XQDZgC4+ywz2wbcsQeJBoKHQXOjynKBzXG0zQVK3N3NrLbjlAIT3H0GgJndAqwzs7bu3qwmEtPSBCLSUNQ2QCATKI/YrmDPV+ZcCGSYWe+IssHAvBht54V1sdrNAwZFjk4jGERQVT+bXVcVjbXCaLNQ3RIEndu0SHIkItLcxTMa7XYzu9fM7gWygDFV2xHltXL3LcB44FYzyzGzEcBpwJMxmj8BXG1mBWaWD1xD8AApQCHB/GxXmlkLM7siLJ8S/vw7cHo4PDqTYKLQqc2tVwPVL02wuayC979Yn4KIRKS5qi3ZvA3sCxwQvt4FukdsH0Cwime8fgpkA2sJ7qNc7u7zzOzI8PJYlYeAlwjuucwlWCX0IQB330YwLPp8YCNwETAqLMfdpwA3hfusJRgkEPN5nqYuemmCgnbZ3Pxf/chvl815j07n+Y9WpDpEEWkmalupc2R9fpi7byBIFNHl7xDc+K/aduD68BXrODOBYTV8zgPAA9823qYgemkCgO8f1I2fPv0R1/xzFkvXb+GqY/cnLc2qOYKIyLe3xw91SuPVNjuTsT8+hB8c1I2/TFnMlc/O1MOfIpJQ8S6eJk1MZnoavz/zAPbpnMPtr37Gyo2lPHz+QXRqrcEDIlL/1LNpxsyMy47alwfPPZBPV23i9PunsXhtrJHoIiLfjpKNcOLArowbfThlFTs4/f53mbZ4XapDEpEmRpfRBIDB3drxwv+M4OKxM7jgsQ8488ACpi5eT9HGUgren8J1J/TZbaCBiEi84ko2Zta9mioHytz9q/oLSVKloF02//zJ4Zz14HuM+/CbYdFFG0u5cfwcACUcEdkj8V5GW0qwlED0aymw2sy+NrM/mpl6So1cm5aZMRdeK62o5M7JC1IQkYg0BfEmhx8STGj5IMEsygCHEsyYPAZoB/ySYH6yX9dviJJsqzaWxSzXnGoisqfiTTaXA1e5+/iIsilmtgD4mbsfZWZrgVtQsmn08ttlUxQjsbRqkU5ZRSUtY0yBIyJSk3gvox1KMHVMtLnAweH79wim95dGLtacaulpxpbySk7961Tmr4peKUJEpGbxJptlhIuMRbkUqFqUrDOwoT6CktSKnFMNgoEDd581mMcvOoSvt1Zw2l+n8ejUJezY0Wwn1BaROor3Mto1wPNm9j3C9W2Agwgm6Twz3D4YeK5+w5NUqZpTLXrhp9d+diQ/f34Ov3n5UwoXrOWuswaTl9sydYGKSKMQV8/G3ScBvYEXCRYqyw3f93H3V8I297v71YkKVBqGjq1b8Lfzh3Hb6QOZsXQDJ97zNpPnrU51WCLSwMU9VNndvwRuTGAs0kiYGecc2oNDe3Xk/8bN5LInP+KHh3Tj5pP70ypLo99FZHdx/2Uws1bAEGAvonpEUaPUpJnYb6/WjL98BH98fSEPvf0507/YwKih+YybsYKVG0vJb5etmQdEBIh/BoFjCRY76xij2gGNhW2msjLSuOGkvhy1f2d+8tSH/PH1RTvrNPOAiFSJdzTanwlWvtzb3dOiXko0wuH7dox5CU0zD4gIxH8ZrSdwqruvTGAs0sitLtbMAyISW7w9m2lAn0QGIo1ffvhcTjQzePaD5XouR6QZizfZPAjcZWaXmNmhZnZg5CuRAUrjEWvmgRYZafTo2Iobxs/h9AfeZfaKjSmKTkRSKd7LaP8Kfz4co04DBAT4ZhDAnZMX7DIa7bQh+bzwSRG3TfqM0+6bxg8P6c51x/ehfU5WiiMWkWSJN9n0SmgU0mRUzTwQ7fShe3NMvzzueX0Rj7+3lFfmrOL6E/ryg4O7kZ5myQ9URJIqrmTj7ssSHYg0fbktM/nVKf05++C9+dXEedw0YQ7PzljOracNZOm6Lbv1iDRcWqTpqDbZmNkZwEvuXhG+r5Ye6pS66Nsll3GjD+PFWSv57aT5jLpvGulpRmU4gEDP54g0PTX1bP4FdAHW8s09m1h0z0bqzMw4bUgBR/fdi8Nvf5OS8spd6quez1GyEWkaqk027p4W671IfWrTMpMtUYmmip7PEWk6lEQk5ap7PictzXhuxpdsr9yR5IhEpL7VZSLOvYHvEHsizj/Wc1zSjFx3Qh9uHD+H0opvejhZ6Wnk5bbg+udnc3/hYn52bG9OHVygkWsijVS8E3GeAzwGbAe+IrhPU8UBJRvZYzU9n/PG/LX88fWFXDVuFve99TlXHbs/Jw3sQpqSjkijEm/P5lbgbuBmd499gV3kW6ju+Zzj+udxTN+9eG3eav70+kL+55mP6dulDVcftz/H9c9j4icrNWRapBGIN9nkAY8o0UgqpKUZ3zugKycM6MLLs1dyzxuLGP3kR3Rrn82aTeVsC+/paMi0SMMV7wCBV4BDExmISG3S04Lh0q9f9R3u/P4gVhaX7Uw0VbSkgUjDFG/P5nXgDjMbAMwBKiIr9VCnJFNGehpnHdSN6/81O2a9hkyLNDzxJpuHwp83xajTQ52SEvntsimKkVjM4K7JCzj/8B7sldsyBZGJSLS4LqPFWJ1zj1bqNLMOZjbBzLaY2TIz+1E17czM7jCz9eHrDjOziPohZvaRmW0Nfw6JcYwsM5tvZivijU8al1hLGmRlpDEgP5f7Chcz4o4pXP3cJ8xbWZyiCEWkSq09GzPLBKYC57v7t70Yfh+wjWDAwRBgkpnNcvd5Ue1GA6OAwQQ9p9eBJcCDZpYFTATuAe4HLgMmmllvd98WcYzrCIZpt/mWMUsDVd2Q6VFDC1i2fgt/n7aUf374JeM/LuKwfTpwyRH7cHTfvUhLM16YWaRRbCJJVGuyCSfi7MWuz9bUmZnlAGcCA929BJhqZi8C5wE3RDW/ALjb3VeE+94NXEqwiNvIMO573N2Be83sWuBo4LWwfS/gXOBq4G/fJm5p2KobMt2jYw5jTh3AVcftz7gZyxk7bSmXPPEhvTrlcGD3dkyas4qyCo1iE0kWC/5e19LI7E4Ad79ujz/IbCgwzd1bRZRdCxzl7qdEtS0Gjnf36eH2QcBb7t7GzK4K606KaP9yWH93xPajwNfAU+6+dzUxjSboRdG5c+dhzz333J5+vSarpKSE1q1bpzqMb61yh/PhmkomL63gi+LY0990bGncPbJVzLpoTeW81Cedk9ia+nn57ne/+5G7H1Rbu3gHCOQA55jZccBHwJbISne/Mo5jtAY2RZUVE/syV+uwLrJd6/C+TXTdLscxs9OBdHefYGYjawrI3R8mXH20T58+PnJkjc2bpcLCQprKeTkGuN6dfW58JWY3fUOZx/1dm9J5qS86J7HpvATiTTb9gI/D9/tE1cV7ea0EyI0qywU2x9E2Fyhxdzezao8TXqr7A/C9OGOSZsbMqh3FhsGYF+fxg4O70a9r9K+YiHwb8a7U+d16+KyFQEZ4I39RWDYYiB4cQFg2GPggRrt5wDVmZv7NNcBBBIMPegM9gXfCwWtZQFszWw0c5u5L6+F7SCNX3cSf/fPb8Mz05Yx9dymD9m7L2Qd149Qh+eS2zExhtCJNQ9woTBL1AAAVaklEQVSzPn9b7r7FzMYDt5rZJQSj0U4Dhsdo/gRwtZm9QtBzugb4S1hXCFQCV5rZgwQDBwCmADuAbhHHGQ78FTiQYGSaSI2j2L7eso0XPili3Iwv+eULc/ntpE/53sCunH1wNw7t1WHnXGxFG0speH+KRrGJxKkuSwx8F/gh0J2gx7CTux8d52F+SjB79FpgPXC5u88zsyOBV9296i7aQwSX6+aE24+EZbj7NjMbFZb9HpgPjIoY9rw6IuYNwA5331kmAtWPYmufk8WPR/TiwuE9mVNUzLgZX/LiJysZP7OITjmZbCzdznYtXy1SZ3E91GlmFwKvEtyEH0nQS2hP0GP4NN4Pc/cN7j7K3XPcvbu7PxOWvxORaPDA9e7eIXxdH3HJDHef6e7D3D3b3Q9095nVfF5hdSPRRGpiZgzaux23nX4AH/ziWO4+azCbyip3JpoqmotNJD7xTsR5LXCFu/+QYF60G919KPAUwc18kSYrOyudM4ftTUU1K4YWbSxlwswVbC6riFkvIvFfRtsHeCN8X04w/BiC+yGF7P5QpkiTU90otnSDq8bNIisjjaP77MXJg7tyTN88srM0ZaBIlXiTzXq+eR6mCBgIzAY6ArEXkBdpYmKNYsvOTOd3owbSvVMrXpq1iklzVvHavNW0ykrnmH55nDyoK0ft35nX5q7W9DjSrMWbbN4Bjie4Yf8cwRQxxxE8J/d6gmITaVAiR7EVbSylICppDOvRgZtP7s8HSzbw8uyVvDp3NS/NWkmLdGP7Dqh0DSyQ5iveZHMFUDVX++3AdmAEQeL5bQLiEmmQqkaxVfdUeHqacfi+HTl8347ccuoA3v18PT956iPKK3dd5La0opI7XvtMyUaajXgf6twQ8X4HcEfCIhJpIjLS0/jO/p0p3RZ7NfVVxWWc/eB7HNNvL47tn8e+nZvu/FkidXnOJo9ghuZ9gZvdfZ2ZjQBWuvuSRAUo0thVN7CgTcsMtmzbzu2vfsbtr35Gr045HNM3SDwH9WhPRnqalkKQJiOuZGNmw4A3CdaUGQDcCawDjgP2B2IugiYi1Q8s+M1pAxk1tICVG0t587O1vPHpGp54bxmPTF1C2+xM9uucw+yiYioqda9HGr94ezZ3AX9291+bWeTEmZOBH9d/WCJNR03T40DQ8znvsB6cd1gPSsq3M3XRV7z+6VomzFxB1DOklFZU8ofJutcjjU+8yWYYcHGM8lUEq26KSA2qmx4nWusWGZw4sCsnDuzK+I9jr2i+cmMZP3nyI47o3Ykje3eiR8ec+g5XpN7Fm2xKCaanidaXYJ4zEaln1d3raZWVzpyiYl6bF0z5171DqyDx7NeJ4ft2om2rTN3rkQYn3mQzEfi1mZ0VbruZ9SQYlfZ8AuISafaqfYj09AM4bUg+X6zbwtRF63hn0Tpe/GQlz0xfTppBQftsVm0s04Sh0qDEm2yuBV4hmICzFTCV4PLZu8AvExOaSPNW272efTu3Zt/OrblgeE8qKnfwyZcbeWfROh4oXBxzwtDfTvqU4wfk0SoraSuLiOwU73M2m4AjzOxogpme04CP3f2NmvcUkW8j3ns9melpHNyzAwf37MBf3lwUs826km0MGvNvBu3dlkN6deTQXh0Y1rP9LovD6fKbJEqd/onj7lMIFikDwMx6AHe6+9n1HZiI7Jnq7vV0zMni7IO78cGSDTw69Qse/M/npBn0z8/lkJ4dcZx/fLCcsopgdmtdfpP69G370+2AM+sjEBGpH9Xd67n55P47k0bptkpmLv+a6Us28MGSDTw9fRnl23dfQkFDraW+6OKtSBNT270eCNboGb5fJ4bv1wmA8u2V9P3la3iM463cWMZ5j05naPf2DO3ejqHd2tGu1TeL9VZdetNS2VITJRuRJijeez1VWmSk1zjUel3JNv46ZdHOh0z36ZTD0O7tMYOXZq3c2SvSpTepjpKNiAA1D7UeNbSALeXbmb2imJlffs3Hyzbyn4VrWVeybbfjlFZU8rtX5nPq4HzS0iyZX0EasBqTjZm9WMv+ufUYi4ikUG2X33JaZOxcPgHA3dnnxldiXnpbu7mcwbf8mwEFuRxQ0JaBBW05oKAtPTvm7ExAGvnWvNTWs1kfR71mfBZpIupy+c3Mqr301i47k5MHd2VO0SYef28Z28LLbG1aZDCgIJfszDSmLl6vSUabkRqTjbtrkk0RqVZ1l97GnDpgZ9KoqNzBojUlzCnayJyiYuYUbeL9LzbsdqzSikrGvDiP7h1b0bdLm2ofPlWPqHHSPRsR2WO1LZUNwQOn/fNz6Z+fyw8ODsp63TAp5uW3jaUVnHH/u5hBjw6t6Nc1l75dcunXtQ39uuby4dIN3DRh7s7kph5R46FkIyLfSm1LZcdS3eW3vNwW/Oa0gXy2ejPzV23is9WbeW3eajzMTAa7JanSikrunLxAyaaBU7IRkaSr7vLbjSf14/gBXTh+QJed5VvKt7NgzWY+W7WZmybMiXm8oo2lnPvIdHrntab3Xm3YP681vfPa0DZbU/E0FEo2IpJ08Tx4WiWnRQYHdm/Pgd3bc99bi2P2iLIz09lUVsGzH3y5SwLbq00L9s9rQ5rBe19oQEIqKdmISErU9cFTqL5HdPsZwbNAO3Y4RRtLWbR2M4vWlLBwTQmL1m5mzorimJffbpowh682l7NP5xz26dyabu2zyUhP26WdekT1Q8lGRBqN2npEaWlGtw6t6NahFUf3/WYR4V43TIp5vK3bKrntlfk7tzPTjR4dc9inU5B8iku3Mf7jIs2QUA+UbESkUdmTHlF1AxIK2mUz6coj+PyrLXzxVcnOn1+s28JbC9buvOwWqbSikpsnzsUMenbMoWfHHNq2ytytneaM25WSjYg0edVdfrvuhD60a5XFsB5ZDOvRfpd9tlfuoPcvXo05RHtz2XZ+9uwnO7fbtcqkR8ccenVsRY+OOazbUsa/PlSPKJKSjYg0eXUZkFAlIz2t2h5RftuWjL3oEJau28Ky9VtZsn4Ly9ZvYcbSr5k4a+XOodqRSisq+eULc9lcVrHzUt/e7bNpkZG+S7umeo9IyUZEmoX6HJBw/Yl92T+vDfvntdltn5qWaygp387NE+ft3DaDLrkt6dY+SD5byit487O1ezRqrqEnKSUbEZFq7EmPqKblGvLbtWTCT0ewfMNWvtywleXha8WGUqYtXsfqTWW77VNaUcnPn5/NtMXrKGifTUG7bAraZ7N3u1Z0aduSrIw0XphZtEtSbIiX7ZKabMysA/AocDywDrjR3Z+J0c6A3wOXhEWPADe4B51TMxsSHqcfMB+42N0/CeuuAy4AeoSfcb+735nI7yUiTVe99ohO6EtebkvycltycM8Ou+1X3TQ+5dt38Pair1i7uXyXS3RmkNemJRu2bGNb5a4rrZZWVHLHa59x2pB8gj+psSWrR5Tsns19wDYgDxgCTDKzWe4+L6rdaGAUMJhgdorXCWaXftDMsoCJwD3A/cBlwEQz6+3u2whmtDgfmA3sC/zbzL5092cT/u1ERIhvzrhYaho1N+2GoynfXsnq4jJWfF1K0delrNgY/Hz+4xUxj7equIwBv55M17YtyW+XTde2LenaNpv8dsHP+auK+dMbiyirqPtAhqokldVlv2G1nhCSmGzMLAc4Exjo7iXA1HC9nPOAG6KaXwDc7e4rwn3vBi4FHgRGhnHfE/Z07jWza4Gjgdfc/Q8Rx1lgZhOBEYCSjYgkzZ7MGVfTqDkILtH16JhDj445u+z3/hfrYyapttkZnHlgN1YVl7KyuIwFq7/iq5LymAMYqpRWVHLzC3PZsm07XcJeWJe2LenQKmuXtYii46yNeU2fWo/MbCgwzd1bRZRdCxzl7qdEtS0Gjnf36eH2QcBb7t7GzK4K606KaP9yWH931HEM+Bh4yN0fjBHTaIJeFJ07dx723HPP1dO3bTpKSkpo3bp1qsNocHRedqdzEltdz8u7Kyt4fmEF68ucji2NM/fPZHj+7s/xRO8zdu42tkVcSctKgwsHZu227/YdztdlzoYy5/YPdr9HVJ10g3YtjPYtjeWbduz8rFWP/x/lqxbVuiRrMi+jtQY2RZUVA7sP5wjaFke1ax0mj+i6mo4zBkgD/h4rIHd/GHgYoE+fPh7vvz6ak7r8q6w50XnZnc5JbHU9LyOBm+r4GSOB/ntw7+WJhVOqHcjw/OXDWV1cxppNZawuLmP1pvKd7xdvrG1dzd0lM9mUsPsy0rnA5jja5gIl7u5mFtdxzOwKgns3R7p7+bcJXESkoavvgQxd22bTtW12zP1G/D52kqpJWu1N6s1CIMPMekeUDQaiBwcQlg2upt08YJDtOrxiUORxzOwigvtAx1Td9xERkV2NGlrA7WccQEG7bIxgIELVpKY1ue6EPmRnptfYJlrSejbuvsXMxgO3mtklBKPRTgOGx2j+BHC1mb1CMBrtGuAvYV0hUAlcaWYPEgwcAJgCYGbnAL8DvuvuXyTo64iINAl70iOKHG23Ks59ktmzAfgpkA2sBf4BXO7u88zsyPDyWJWHgJeAOcBcYFJYRji8eRTBJbKNwEXAqLAc4LdAR2CGmZWEr90GB4iIyJ4bNbSAaTcczbbViz+Kp31Sn7Nx9w0EiSK6/B2CG/9V2w5cH75iHWcmEHNst7v3qpdgRUSk3iS7ZyMiIs2Qko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCScko2IiCRcUpONmXUwswlmtsXMlpnZj6ppZ2Z2h5mtD193mJlF1A8xs4/MbGv4c0i8+4qISPIlu2dzH7ANyAPOAR4wswEx2o0GRgGDgUHAKcBlAGaWBUwEngLaA48DE8PyGvcVEZHUSFqyMbMc4EzgZncvcfepwIvAeTGaXwDc7e4r3L0IuBu4MKwbCWQA97h7ubvfCxhwdBz7iohICmQk8bP2B7a7+8KIslnAUTHaDgjrItsNiKib7e4eUT87LH+tln13YWajCXpCAOVmNje+r9KsdALWpTqIBkjnZXc6J7E19fPSI55GyUw2rYFNUWXFQJtq2hZHtWsd3nuJros+TrX7RiUo3P1h4GEAM/vQ3Q+K/+s0Dzovsem87E7nJDadl0Ay79mUALlRZbnA5jja5gIlYbKo7Tg17SsiIimQzGSzEMgws94RZYOBeTHazgvrYrWbBwyKGmE2KKq+un1FRCQFkpZs3H0LMB641cxyzGwEcBrwZIzmTwBXm1mBmeUD1wBjw7pCoBK40sxamNkVYfmUOPatycN1/1bNgs5LbDovu9M5iU3nBbBkXl0ysw7AY8BxwHrgBnd/xsyOBF5199ZhOwPuAC4Jd30E+HnVpTAzGxqW9QfmAxe7+8x49hURkeRLarIREZHmSdPViIhIwinZiIhIwjX7ZBPvfG3NjZkVmlmZmZWErwWpjinZzOwKM/vQzMrNbGxU3TFm9lk4P99bZhbXg21NQXXnxcx6mplH/M6UmNnNKQw1qcIBS4+Gf0c2m9knZnZSRH2z/Z0BJRuIf7625ugKd28dvvqkOpgUWAn8lmBQy05m1olgZOXNQAfgQ2Bc0qNLnZjnJUK7iN+b3yQxrlTLAL4kmBWlLfBL4LkwCTf335mkziDQ4ETM1zbQ3UuAqWZWNV/bDSkNTlLO3ccDmNlBwN4RVWcA89z9n2H9GGCdmfV198+SHmiS1XBemrXw8Y4xEUUvm9kSYBjQkWb8OwPq2VQ3X5t6NoHbzWydmU0zs5GpDqYB2WX+vfCPzOfo96bKMjNbYWZ/D/9F3yyZWR7B35h56Hem2SebuszX1tz8HNgHKCB4KO0lM9s3tSE1GLXNz9dcrQMOJpiYcRjB+Xg6pRGliJllEnz3x8OeS7P/nWnuyaYu87U1K+4+3d03h8s4PA5MA76X6rgaCP3exBAuHfKhu2939zXAFcDxZtZs/qACmFkawcwo2wjOAeh3ptknm7rM19bcOcG6QRI1/154729f9HsTreqJ8WbzdyacweRRggFHZ7p7RVjV7H9nms0vQSx1nK+t2TCzdmZ2gpm1NLMMMzsH+A7BekHNRvjdWwLpQHrV+QAmAAPN7Myw/lcEayw1ixu91Z0XMzvUzPqYWZqZdQTuBQrdPfryUVP2ANAPOMXdSyPKm/XvDADu3qxfBMMQXwC2AMuBH6U6plS/gM7ADIIu/kbgfeC4VMeVgvMwhuBf55GvMWHdscBnQCnB5LA9Ux1vqs8L8ENgSfj/0iqCSXG7pDreJJ6XHuG5KCO4bFb1Oqe5/864u+ZGExGRxGvWl9FERCQ5lGxERCThlGxERCThlGxERCThlGxERCThlGxERCThlGxEmqhwbZnvpzoOEVCyEUkIMxsb/rGPfr2f6thEUqFZr2cjkmBvEKyNFGlbKgIRSTX1bEQSp9zdV0e9NsDOS1xXmNmkcJngZWZ2buTOZnaAmb1hZqVmtiHsLbWNanOBmc0Jl2heY2aPR8XQwcz+GS57/kX0Z4gki5KNSOrcArwIDCFYM+iJcPXLqlmBJxPMrXUIcDownIilmM3sMuAh4O/AIIIlIOZGfcavgIkEMw6PAx4zs+6J+0oisWluNJEEMLOxwLkEkzJGus/df25mDjzi7pdG7PMGsNrdzzWzS4G7gL3dfXNYPxJ4C+jt7ovNbAXwlLvHXMI8/Izfu/uN4XYGwWKBo939qXr8uiK10j0bkcR5GxgdVbYx4v17UXXvAf8Vvu9HMAV95OJa7wI7gP5mtolgFdU3a4lhdtUbd99uZl8Be8UXvkj9UbIRSZyt7r44Acety+WIiqhtR5fPJQX0SyeSOofF2J4fvp8PHBC1pPJwgv9n57v7WqAIOCbhUYrUA/VsRBKnhZl1iSqrdPevwvdnmNkMgoW0vk+QOA4N654mGEDwhJn9CmhPMBhgfERv6TbgT2a2BpgEtAKOcfe7E/WFRPaUko1I4hxLsGJlpCJg7/D9GOBMguWTvwJ+7O4zANx9q5mdANwDfEAw0GAi8LOqA7n7A2a2DbgGuAPYALySqC8j8m1oNJpICoQjxc5y93+lOhaRZNA9GxERSTglGxERSThdRhMRkYRTz0ZERBJOyUZERBJOyUZERBJOyUZERBJOyUZERBLu/wFIBub2peel9wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learning_rate = 0.01\n",
    "decay = 1e-4\n",
    "batch_size = 32\n",
    "n_steps_per_epoch = len(X_train) // batch_size\n",
    "epochs = np.arange(n_epochs)\n",
    "lrs = learning_rate / (1 + decay * epochs * n_steps_per_epoch)\n",
    "\n",
    "plt.plot(epochs, lrs,  \"o-\")\n",
    "plt.axis([0, n_epochs - 1, 0, 0.01])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.title(\"Power Scheduling\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exponential Scheduling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```lr = lr0 * 0.1**(epoch / s)```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "def exponential_decay_fn(epoch):\n",
    "    return 0.01 * 0.1**(epoch / 20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "def exponential_decay(lr0, s):\n",
    "    def exponential_decay_fn(epoch):\n",
    "        return lr0 * 0.1**(epoch / s)\n",
    "    return exponential_decay_fn\n",
    "\n",
    "exponential_decay_fn = exponential_decay(lr0=0.01, s=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 25"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/25\n",
      "55000/55000 [==============================] - 6s 107us/sample - loss: 0.8245 - accuracy: 0.7595 - val_loss: 1.0870 - val_accuracy: 0.7106\n",
      "Epoch 2/25\n",
      "55000/55000 [==============================] - 6s 101us/sample - loss: 0.6391 - accuracy: 0.8064 - val_loss: 0.6125 - val_accuracy: 0.8160\n",
      "Epoch 3/25\n",
      "55000/55000 [==============================] - 6s 101us/sample - loss: 0.5962 - accuracy: 0.8174 - val_loss: 0.6526 - val_accuracy: 0.8086\n",
      "Epoch 4/25\n",
      "55000/55000 [==============================] - 5s 99us/sample - loss: 0.5420 - accuracy: 0.8306 - val_loss: 0.7521 - val_accuracy: 0.7766\n",
      "Epoch 5/25\n",
      "55000/55000 [==============================] - 5s 100us/sample - loss: 0.4853 - accuracy: 0.8460 - val_loss: 0.5616 - val_accuracy: 0.8314\n",
      "Epoch 6/25\n",
      "55000/55000 [==============================] - 5s 98us/sample - loss: 0.4443 - accuracy: 0.8571 - val_loss: 0.5430 - val_accuracy: 0.8664\n",
      "Epoch 7/25\n",
      "55000/55000 [==============================] - 5s 99us/sample - loss: 0.4128 - accuracy: 0.8687 - val_loss: 0.4954 - val_accuracy: 0.8610\n",
      "Epoch 8/25\n",
      "55000/55000 [==============================] - 6s 100us/sample - loss: 0.3763 - accuracy: 0.8773 - val_loss: 0.5770 - val_accuracy: 0.8578\n",
      "Epoch 9/25\n",
      "55000/55000 [==============================] - 6s 102us/sample - loss: 0.3459 - accuracy: 0.8847 - val_loss: 0.5267 - val_accuracy: 0.8688\n",
      "Epoch 10/25\n",
      "55000/55000 [==============================] - 5s 99us/sample - loss: 0.3250 - accuracy: 0.8931 - val_loss: 0.4606 - val_accuracy: 0.8644\n",
      "Epoch 11/25\n",
      "55000/55000 [==============================] - 5s 97us/sample - loss: 0.2984 - accuracy: 0.9010 - val_loss: 0.5083 - val_accuracy: 0.8610\n",
      "Epoch 12/25\n",
      "55000/55000 [==============================] - 5s 99us/sample - loss: 0.2736 - accuracy: 0.9080 - val_loss: 0.4497 - val_accuracy: 0.8826\n",
      "Epoch 13/25\n",
      "55000/55000 [==============================] - 5s 99us/sample - loss: 0.2603 - accuracy: 0.9128 - val_loss: 0.4366 - val_accuracy: 0.8808\n",
      "Epoch 14/25\n",
      "55000/55000 [==============================] - 5s 100us/sample - loss: 0.2382 - accuracy: 0.9197 - val_loss: 0.4692 - val_accuracy: 0.8828\n",
      "Epoch 15/25\n",
      "55000/55000 [==============================] - 6s 102us/sample - loss: 0.2240 - accuracy: 0.9252 - val_loss: 0.4609 - val_accuracy: 0.8774\n",
      "Epoch 16/25\n",
      "55000/55000 [==============================] - 5s 99us/sample - loss: 0.2020 - accuracy: 0.9306 - val_loss: 0.4950 - val_accuracy: 0.8808\n",
      "Epoch 17/25\n",
      "55000/55000 [==============================] - 5s 100us/sample - loss: 0.1950 - accuracy: 0.9340 - val_loss: 0.4985 - val_accuracy: 0.8856\n",
      "Epoch 18/25\n",
      "55000/55000 [==============================] - 6s 102us/sample - loss: 0.1785 - accuracy: 0.9388 - val_loss: 0.5071 - val_accuracy: 0.8854\n",
      "Epoch 19/25\n",
      "55000/55000 [==============================] - 5s 100us/sample - loss: 0.1649 - accuracy: 0.9447 - val_loss: 0.4798 - val_accuracy: 0.8890\n",
      "Epoch 20/25\n",
      "55000/55000 [==============================] - 5s 100us/sample - loss: 0.1561 - accuracy: 0.9471 - val_loss: 0.5023 - val_accuracy: 0.8896\n",
      "Epoch 21/25\n",
      "55000/55000 [==============================] - 5s 98us/sample - loss: 0.1442 - accuracy: 0.9520 - val_loss: 0.5253 - val_accuracy: 0.8952\n",
      "Epoch 22/25\n",
      "55000/55000 [==============================] - 5s 99us/sample - loss: 0.1369 - accuracy: 0.9540 - val_loss: 0.5558 - val_accuracy: 0.8922\n",
      "Epoch 23/25\n",
      "55000/55000 [==============================] - 5s 98us/sample - loss: 0.1277 - accuracy: 0.9576 - val_loss: 0.5786 - val_accuracy: 0.8908\n",
      "Epoch 24/25\n",
      "55000/55000 [==============================] - 5s 99us/sample - loss: 0.1204 - accuracy: 0.9611 - val_loss: 0.5991 - val_accuracy: 0.8902\n",
      "Epoch 25/25\n",
      "55000/55000 [==============================] - 6s 102us/sample - loss: 0.1130 - accuracy: 0.9638 - val_loss: 0.5984 - val_accuracy: 0.8894\n"
     ]
    }
   ],
   "source": [
    "lr_scheduler = keras.callbacks.LearningRateScheduler(exponential_decay_fn)\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[lr_scheduler])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEeCAYAAABc5biTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl8VNX9//HXJySBQIAQQJAgmwLKjog7iluxrVYUa6vWpa1ibf3ZVsVq1VZtrSKlVatV+LovrSuIioJajIKKyiKbCKiI7DuBQCAsn98f9waHYSaZYGYmJO/n43Efmbnn3Dufexjyyb333HPM3REREUmmjHQHICIiNZ+SjYiIJJ2SjYiIJJ2SjYiIJJ2SjYiIJJ2SjYiIJJ2SjUgKmNmlZlZcyW0Kzez+ZMUUfsbXZnZdEvZ7rplV6rmK6DbalzaT6kvJRpLKzB43M4+xTE53bMkSHt+5UaufAzok4bMuM7PpZlZsZkVmNtPM/lrVn5MmSWkzSY/MdAcgtcLbwEVR60rTEUi6uHsJUFKV+zSzXwD3Ab8H/gdkAd2AY6ryc9IlGW0m6aMzG0mFbe6+ImpZB2BmJ5rZdjPrX1bZzK4ws41m1iF8X2hmD5nZvWa2PlyGmVlGxDZNzOyJsKzEzN42s64R5ZeGf/2fYmazzWyzmb1jZu0jAzWzM81sqpltNbOFZnaHmWVHlH9tZjeb2YgwxiVmNiSyPHz5QniG83Xk50fUO9jMxpjZijCWaWZ2RiXb9UfAKHcf4e5fuPtcd3/B3a+JOqYfmNlHYbusNbNXzaxeRJV68Y4n3L6xmY00s1VmtsnM3jWzI6LqXGxmi8xsi5m9BrSIKr/VzGZHrSv3MlmMNrs1/Lf7qZl9Gcbyspk1i6iTaWb/jPie/NPMHjSzwoqbU5JJyUbSyt3fBYYBT4UJ41DgH8D/c/evIqpeSPB9PQa4AhgM/C6i/HHgKOAs4EhgCzDOzHIi6tQFbgR+Ee4nD3iorNDMBgDPAPcDXcN65wJ/iwr798As4HBgKHC3mZWdTfQNf14OHBjxPlou8AZwGtATeAkYFR5/olYAR5Yl5VjM7HTgFeAtoA9wEvAue/7fj3s8ZmbAWKAAOAPoDbwHTDCzA8M6RxG0/0igF/AqcHsljqMy2gE/Ac4GvhfGc0dE+XXApcBlwNEEx3lBkmKRynB3LVqSthD8EtoBFEctQyPqZAGfAKOAacBzUfsoBOYDFrHuZmBJ+Loj4MAJEeWNgSLgsvD9pWGdzhF1LgS2le2X4JfoLVGfPTCMt6zO18B/o+osAG6OeO/AuVF1LgWKK2iryVH7KQTuL6f+gcCH4ectAJ4GLgayIuq8Dzxbzj7KPR7g5PD4c6LqfApcH77+D/BWVPnDwa+X3e9vBWaX1yYJvL8V2Ao0jlh3E/BFxPvlwA0R7w2YBxSm+/9CbV90ZiOp8B7BX7yRy7CyQnffTvDX5xnAAQRnLtEme/jbI/QhUGBmjYDDgF3hurJ9FhH8td4lYptt7j4v4v0yIBtoEr7vA9wUXm4rDi/h/AdoALSM2G5mVGzLwrgTZmYNzOxuM/ssvNxTDBwBtEl0H+6+3N2PAboD9xD8Yh0BfGxm9cNqvQnu55SnvOPpA9QHVke1Szfg4LDOYUS0fSj6fVVZFP7b7hWrmTUm+Hf6uKww/M58jKSdOghIKmxx9y8qqFN2ySMPaA5sqKLPjkxQO+KUZUT8vA14IcZ+Vke83h5jP5X9w+3vwOkEl30WEFz2e5Ig+VWKu88GZgMPmNnxwETgPIKzykSUdzwZwEqgX4ztNlYizF0EyTBSViW2L1MVbS9poH8kSbvwJv39wG8I7i08bWbRfwgdFd4/KHM0sMzdNwJz+fZ+Ttk+GxH8xf9ZJUKZBhzqwc326CU6UZVnO1CngjrHA0+6+0vuPhNYwrdnCt9F2fHmhj+nA6d8h/1NI7jZvytGm6wK68wl+PeIFP1+NdAi6t+w13eIay/hGc8KIu6ThZ8X776ZpJDObCQV6ppZy6h1O919tZnVAZ4C3nX3EWb2IsHlrz8Dt0TUbwXcY2b/JkgiQ4C/Arj7AjMbA4wws8EEZ0V3EPzl/Z9KxHk78JqZLQKeJzgT6gYc6e7XV2I/XwOnmNm7BJfu1seoMx84O4x7O8Hx1otRLy4ze5DgMtIEgmR1IMG9rC3Am2G1O4BXzewLgrYwghvrI9x9SwIf8zbBfZ8xZnY98DnBparTgbfdfSJB9+sPzOxG4EWgP8EN/EiFQD7wRzN7NqwT/SxSVbgXuN7M5hMk3isI2mV5Ej5LKkFnNpIKpxL8Z49cpodlfwQOAX4J4O5rgUuAG8JLQmWeIThb+Aj4P+AR4J8R5T8nuDb/SvizPnC6B89qJMTdxwM/JOix9XG43AB8k/ihAnBtuI/FfHuc0a4BVhFc8nqDoHPAxEp+zlsEPfCeJ0heo8P1p7n7fAB3f53gF//3w1jeDWPblcgHhPc8fkCQ0P6P4Gb780BngkSHu08m+Pe7kuD+zzkEN/Mj9zM3LB8c1jmNvXv5VYW/E/zx8hhBm0LQLluT8FlSCWU9bESqrfAZidnuflW6Y5H9j5lNBya5+/9Ldyy1mS6jiUiNYWZtgQEEZ3BZBM879Qh/Shop2YhITbKL4FmjYQS3CT4Dvu/uU9IalegymoiIJJ86CIiISNLpMlooLy/PDznkkHSHUe1s3ryZBg0apDuMakftsje1SWw1vV2mTp26xt2bV1RPySbUokULpkzRZd1ohYWF9O/fP91hVDtql72pTWKr6e0SPpdWIV1GExGRpFOyERGRpFOyERGRpFOyERGRpFOyERGRpFOyERGRpFOyERGRpFOyERGRpFOyERGRpFOyERGRpFOyERGRpFOyERGRpFOyERGRpFOyERGRpFOyERGRpEtpsjGzfDMbbWabzWyRmV0Qp56Z2VAzWxsuQ83MIspHmtk8M9tlZpfG2P73ZrbCzDaa2aNmVrei2L7euIvj7prAy9OXfqdjFBGRvaX6zOYBoBRoAVwIPGhmXWPUGwwMBHoCPYAzgSsiymcAvwamRW9oZgOAG4BTgLZAB+C2RIJbuqGEG0fNUsIREaliKUs2ZtYAGATc4u7F7j4JeAW4KEb1S4Dh7r7E3ZcCw4FLywrd/QF3/x+wNc62j7j7HHdfD/wlctuKlGzfybDx8xKtLiIiCUjltNCdgB3uPj9i3QzgxBh1u4ZlkfVinQHF0hUYE7VtCzNr6u5rIyua2WCCsyiyWx6ye/3SDSUUFhYm+HE1W3FxsdoiBrXL3tQmsaldAqlMNrnAxqh1RUDDOHWLourlmpm5uyfwOdHbEn7OHsnG3UcCIwHqHthx934L8nJq9JzhlVHT50/fV2qXvalNYlO7BFJ5z6YYaBS1rhGwKYG6jYDiBBJNvG2J8zl7qWPGkAGdE6kqIiIJSmWymQ9kmlnHiHU9gTkx6s4JyyqqF0usbVdGX0KLpWHdTHa607h+VoIfJSIiiUhZsnH3zcAo4HYza2BmxwFnAU/FqP4kcI2ZFZhZK+Ba4PGyQjPLNrN6gAFZZlbPzDIitv2lmXUxszzg5sht42nXKIMpt5zKIQfkctOoWRRv27HvBysiIntIddfnXwM5wCrgv8CV7j7HzPqZWXFEvRHAq8AsYDYwNlxX5k2gBDiW4J5LCXACgLuPA+4G3gG+ARYBf04kuLqZdRg6qAfLN27l7nGf7/NBiojInlLZQQB3X0fw/Ez0+okEN/bL3jtwfbjE2k//Cj7nH8A/9iXGPm2b8PNj2/Po+ws5o0crjmyfvy+7ERGRCBquJobrBnSidZMc/vDSTLZu35nucERE9ntKNjHUz87krnN6sHDNZu7934J0hyMist9Tsonj+I7NOO+I1ox87ytmLy2qeAMREYlLyaYcN/2gC/kNsrn+xZls37kr3eGIiOy3lGzK0bh+Fn85qxufLd/IyPe+Snc4IiL7LSWbCpzerSU/6N6Se/+3gC9WFVe8gYiI7EXJJgG3/qgrOVl1uOGlmezalciIOSIiEknJJgEHNKzHLWd0Ycqi9Tw1eVG6wxER2e8o2SRo0OEF9OvYjKHjPmfJ+i3pDkdEZL+iZJMgM+NvZ3cH4I+jZ5PYANQiIgJKNpVyUH59/nD6obw3fzWjpmnqaBGRRCnZVNJFR7fliLZNuP21z1i9aVu6wxER2S8o2VRSRoZx16AeFG/dzgl3T6D9DWM57q4JvDxdZzoiIvGkdNTnmmL20iLMjJLtwagCSzeUcOOoWQAM7F2QztBERKolndnsg2Hj57Ej6nmbku07GTZ+XpoiEhGp3pRs9sGyDSWVWi8iUtsp2eyDVnk5lVovIlLbKdnsgyEDOpOTVWev9T/te1AaohERqf6UbPbBwN4F3HlOdwrycjDgwMb1aJyTyUvTllC8bUe6wxMRqXbUG20fDexdsEfPs8lfreWC/5vMn8bM5h/n9UpjZCIi1Y/ObKrI0R2actXJHRk1bSmjpy9JdzgiItWKkk0VuvrkQ+jbrgk3j57N12s2pzscEZFqQ8mmCmXWyeCen/Yms04GVz87ndIdmkpaRASUbKpcQV4OQwd1Z+aSIoa/qYc8RURAySYpTu92IBce1YYR733Fu/NXpzscEZG0U7JJklvO6EKnFrlc+/ynGh1aRGo9JZskqZdVh3+dfzibtu7gmuc/ZdcuTbYmIrWXkk0SdW7ZkFvO6MLEBWt4eNJX6Q5HRCRtlGyS7MKj2nB615bcPW4eMxZvSHc4IiJpoWSTZGbGXYO6c0DDulz97HQ2bd2e7pBERFIupcnGzPLNbLSZbTazRWZ2QZx6ZmZDzWxtuAw1M4so72VmU81sS/izV0RZXTN7yMxWmtk6M3vVzNI6o1le/WzuPb83i9dt4U9j5qQzFBGRtEj1mc0DQCnQArgQeNDMusaoNxgYCPQEegBnAlcAmFk2MAZ4GmgCPAGMCdcD/BY4JtyuFbAe+FeSjidhfdvl89tTOjF6+lJ63/6mppMWkVolZcnGzBoAg4Bb3L3Y3ScBrwAXxah+CTDc3Ze4+1JgOHBpWNafYADRe9x9m7vfBxhwcljeHhjv7ivdfSvwHBAroaXcQU1yyDBYv2U7zrfTSSvhiEhNl8pRnzsBO9x9fsS6GcCJMep2Dcsi63WNKJvp7pF9iWeG68cBjwD3mlkrYAPBGdQbsQIys8EEZ1E0b96cwsLCSh5S5dxRuIXoHtAl23fylzEzyCtakNTP3lfFxcVJb5f9kdplb2qT2NQugVQmm1xgY9S6IqBhnLpFUfVyw/s20WXR+1kALAaWAjuBWcBVsQJy95HASIDOnTt7//79EzyUfbNu3NjY67c6yf7sfVVYWFhtY0sntcve1CaxqV0CCV9GM7MWZnadmT1oZs3CdceZWfsEd1EMNIpa1wjYlEDdRkBxeDZT0X4eAOoCTYEGwCjinNmkmqaTFpHaKqFkY2Z9gHkEl6R+ybe/7E8D7kjws+YDmWbWMWJdTyBW96w5YVmsenOAHpG90wg6A5SV9wIed/d17r6NoHPAkWUJMp3iTSf9/W4t0xCNiEjqJHpm83fgXnfvDUQO9DUeOC6RHbj7ZoKzjNvNrIGZHQecBTwVo/qTwDVmVhDee7kWeDwsKyS4PHZ12M257BLZhPDnJ8DFZtbYzLKAXwPL3H1NYoeaPNHTSbdqXI/WeTk8+8liFqyMdYInIlIzJHrPpg/BGU205QTdmBP1a+BRYBWwFrjS3eeYWT/gDXfPDeuNADoQ3G8BeDhch7uXmtnAcN1dwFxgoLuXhnWvA+4juHeTDcwGzq5EjEkVPZ308qISzvzX+1z+5BTG/OZ4GtfPSmN0IiLJkWiyKSF4piXaoQSJIyHuvo7g+Zno9RMJbvyXvXfg+nCJtZ/pBAkwVtlagst9+4UDG+cw4qLDOX/kR1z132k8dmlfMutoYAcRqVkS/a02BvizmdUN37uZtQOGAi8lIa5apU/bfP46sBsTF6zhb69/nu5wRESqXKLJ5jogH1gN1AcmAV8QPMdyc3JCq13O63sQlx7bjkffX8gLUxanOxwRkSqV0GU0d98IHG9mJwOHEySpae7+djKDq21u/uFhLFi1iZtGz6ZD81z6tI115VJEZP+TaNfni82srrtPcPe/u/vd7v62mWWb2cXJDrK2yKyTwf3nH86BefX41dNTWVG0Nd0hiYhUiUQvoz0GNI6xvmFYJlWkSYNs/u/iI9iybQeDn5rC1u070x2SiMh3lmiyMSDWvMZt2HvoGPmOOrVoyD0/7c2spUXc8NJM9hwGTkRk/1PuPRszm0WQZBx418x2RBTXAdoCrycvvNrrtC4tuPa0Tvz9zfkcdmAjrjjx4HSHJCKyzyrqIPBi+LMbMJZgXLIypcDXqOtz0vzmpEOYu2ITd437nE4tG3JS5wPSHZKIyD4pN9m4+20AZvY18Fw4P4ykiJkx7NweLFy9mV89OYXG9bNZvWkbrfJyGDKg8x4jEYiIVGcJ3bNx9yeUaNKjfnYmPz6iNdt2Oqs2bdOkayKyX0q063O2md1mZvPNbKuZ7Yxckh1kbffwxIV7rSvZvpNh4+elIRoRkcpLtDfaXwinagZ2AUMI5o1ZSzC4piTRsg0llVovIlLdJJpszgN+5e4jCIb3H+PuVwN/JpjTRpIo/qRr9VIciYjIvkk02bQAPgtfFwN54etxwPeqOijZU7xJ13q0zotRW0Sk+kk02XwDtApffwEMCF8fQzD9gCTRXpOu5dWjb9smvDF7BU9NXpTu8EREKpTofDajgVOAycC9wH/N7HKgABiWpNgkQvSka9t37uJXT03lT2Nmk18/mx/2ODCN0YmIlC/RUZ9vjHj9opktJpgOer67v5as4CS+rDoZ3H/B4Vz86Ef87rnpNM7J4viOzdIdlohITPs0JaS7f+Tu/3D318ysQVUHJYnJya7Dw5f05eDmuQx+agozFm9Id0giIjHt8/zDZlbPzIYAez8EIinTOCeLJ39xJPkNsrn0sY/5YlVxxRuJiKRYuckmfJjzDjP7xMw+MLOB4fqLga+A3wH/TEGcUo4DGtXj6V8eRZ0M4+JHPmJ5kfpsiEj1UtGZza3AVcAioD3wgpn9G7gJuBFo5+53JjVCSUi7Zg14/OdHsmnrDi565GPWby5Nd0giIrtVlGzOAy5193OB0wmmFWgCdA3HS9ue7AAlcd0KGjPy4iP4Zt0Wfv74J2wp3VHxRiIiKVBRsjkI+ATA3WcQTCsw1N31W6yaOubgpvzr/N7MXLKBXz09jdIdu9IdkohIhV2fs4BtEe+3o5k5q70BXVty5znd+cNLs/jpiA9ZsWkryzds1dQEIpI2iTxnc6eZbQlfZwO3mtkeCSccJ02qkZ/0bcPEBat5beaK3evKpiYAlHBEJKUqSjbvAZHzEX8AtImq41UakVSZ6d/s/dxN2dQESjYikkoVzdTZP0VxSBIs2xB7vjtNTSAiqbbPD3VK9aepCUSkulCyqcHiTU3QoXku7rr6KSKpk9JkY2b5ZjbazDab2SIzuyBOPTOzoWa2NlyGmplFlPcys6lmtiX82Stq+8PN7D0zKzazlWb222QfW3UUPTVBQV49TuzYjIkL1nDTy7PZtUsJR0RSI9EpBqrKAwTP6rQAegFjzWyGu8+JqjcYGAj0JOiA8BbBGGwPmVk2MAa4B/g3cAUwxsw6unupmTUjmNTt98CLBD3oWif9yKqp6KkJ3J27x8/jwcIv2b5jF3cN6kGdDCtnDyIi313KzmzC0aEHAbe4e7G7TwJeAS6KUf0SYLi7L3H3pcBw4NKwrD9BkrzH3be5+32AASeH5dcA4939mbB8k7vPTdqB7WfMjOsHdOa3p3TkhalLuPb5T9mxUw9+ikhyJXRmY2bR3Z3LOLDV3VcnsJtOwA53nx+xbgZwYoy6XcOyyHpdI8pm+p43HWaG68cBRwOzzOwD4BDgI+A37v5N9IeY2WCCsyiaN29OYWFhAodRM/TOgkEds3jp02UsXbGSK3rUJTPGGU5xcXGtapdEqV32pjaJTe0SSPQy2teU8zyNmW0EHgOuL2com1xgY9S6IqBhnLpFUfVyw/s20WXR+2kNHA6cBswC7gb+SzDZ2x7cfSQwEqBz587ev3//OKHXTP37w2ETv+KvY+eSl9+Q+y/oTd3MPTsUFBYWUtvaJRFql72pTWJTuwQSvYx2PrAEuJngl/hp4etvgF8QjA59EXBLOfsoBhpFrWsEbEqgbiOgODybqWg/JcBod//E3bcCtwHHmlnjcmKrtS7r14HbftSVtz5byRVPTWXr9p3pDklEaqBEk82VwO/d/U53nxAudwLXAr9w93uBqwmSUjzzgUwz6xixricQ3TmAcF3POPXmAD0ie6cBPSLKZ7LnWZi6XFXgkmPb8bezu/Pu/NVc9sQUSkqVcESkaiWabI4iuCQVbTbQN3z9IeX0+nL3zcAo4HYza2BmxwFnAU/FqP4kcI2ZFZhZK4Kk9nhYVgjsBK42s7pmdlW4fkL48zHg7LB7dBbB2dYkd9cAouW44Kg2DDu3Jx98uYZLH/uYzds0sLeIVJ1E79ksIriRPiRq/eUEl9IAmgPrKtjPr4FHgVXAWuBKd59jZv2AN9w9N6w3AujAtwnu4XAdYffmgeG6u4C5wEB3Lw3LJ5jZH4GxQH1gEhDzeR7Z07l9WpNVx7jm+Rn88L6JbNuxi+VFWymYPEGjRYvId5JosrkWeMnMfkA4vw1wBMEgnYPC932B58vbibuvI3h+Jnr9RIIb/2XvHbg+XGLtZzrQp5zPeRB4sLxYJLazehXw6TfreeyDRbvXabRoEfmuErqM5u5jgY4Ez8U0CpdXgM7u/npY59/ufk2yApXUefOzVXutKxstWkRkXyQ8goC7LwZuTGIsUk3EGxVao0WLyL5KONmYWX2CIWYOIOqMyN1HVXFckkat8nJYGiOx5DfITkM0IlITJHQZzcxOJegkMImgR9mLEcsLSYtO0iLWaNFmsHZzKU99+HVaYhKR/VuiXZ/vJejd1drdM6KWvcewl/1a5GjRAAV5Odx1dndOOfQAbhkzh7+9PlcjRotIpSR6Ga0d8CN3X5bEWKQaKRstOnKojXOPOIjbXp3DyPe+YvG6LfzzJ72oF2O+HBGRaIme2bwPdE5mIFL91ckwbvtRV27+4WGMm7OC8/9vMmuLt6U7LBHZDyR6ZvMQ8Pfwaf5ZwPbIQnefVtWBSfVkZlzWrwOtm+Tw22c/5ex/f8BjP+/Lwc1zK95YRGqtRM9sXgQOJRgh+UNgSsTySTnbSQ11ercDeXbw0WzetoNz/v0BH321Nt0hiUg1lmiyaV/O0iE5oUl117tNE0b/+jia5mZz0SMfM+bTpekOSUSqqYQuo7n7ooprSW3Upml9Rl15LFc8NZXfPvspb85ZwaeLN7Bsw1Za5eVoTDURAcpJNmZ2DvCqu28PX8elhzprt7z62Tz5yyO5YORkxs5asXu9xlQTkTLlndm8CLQkGKH5xXLqOaD+r7Vc3cw6rNi4da/1ZWOqKdmI1G5xk427Z8R6LRLPsg17J5tgvcZUE6ntlESkyrQKRxyI1ignk2DWCBGprRJONmbW2swuMLPfmdk1kUsyA5T9R6wx1TIMikp28Jv/TKNYs3+K1FoJ9UYzswsJZtjcAawmuE9TxoF/VH1osr8puy8zbPw8lm0ooVVeDtd9rxOrNm3j7vHz+HzFJB76WR86tWiY5khFJNUSHUHgdmA4cIu770xiPLKfKxtTLVrPg/K46j/TOev+97lrUHfO6qUOAyK1SaKX0VoADyvRyL46ukNTXr/6eLoVNOK3z37Kn8fMpnTHrnSHJSIpkmiyeR04KpmBSM13QKN6/Ofyo7m8X3ue+HAR5434UD3VRGqJRC+jvQUMNbOuxB6IUw91SkKy6mRw0w+7cHibJgx5cSY/vG8i953fm34dm6c7NBFJokSTzYjw5x9jlOmhTqm073c/kM4tG3Ll09O4+NGPOb1rS2Ys2cByDXMjUiMldBktxuycmqlTvrMOzXMZ/Ztj6dMmjzdmr2DZhq043w5z8/J0DewpUlNUmGzMLMvMPjIzTZ4mVa5+dibLi+IPcyMiNUOFycbdtxNMJaBHwCUpNMyNSM2XaG+0J4DLkxmI1F7xhrnJyDCmLlqf4mhEJBkSTTYNgMFm9qmZPWJm90UuyQxQar5Yw9xkZ2bQsG4mP37oA4aN/1zP5Ijs5xLtjXYYMC18HT0zpy6vyXcSa5ibIQM6c8phB/CX1z7jgXe+5J3PV/PPn/Sic0sNdSOyP0p0ps6Tkh2I1G7xhrm5+9yenNalJTeOmsmZ/5rEdQM68cvjO1Anw9IQpYjsK00xINXeaV1aMP53J3DSoc352+ufc/7IySxetyXdYYlIJVRmioGTzGykmY0zswmRSyX2kW9mo81ss5ktMrML4tQzMxtqZmvDZaiZWUR5LzObamZbwp+9Yuwj28zmmtmSROOT6qtpbl0e+lkfhv+4J3OXb+T0e97j2Y+/YfS0JRx31wTa3zCW4+6aoGdzRKqphJKNmV0KvAE0BPoTTDPQBDgc+KwSn/cAUEowsOeFwIPhEDjRBgMDgZ5AD+BM4IowlmxgDPB0GMMTwJhwfaQhYZxSQ5gZg/q05o3f9aNH6zxuGDWLa1+YwdINJXoYVKSaS/TM5jrgKnc/n2BctBvdvTfBL/ziRHZgZg2AQQTTFBS7+yTgFeCiGNUvAYa7+xJ3X0owvcGlYVl/gntN97j7Nne/DzDg5IjPag/8DLgzweOT/UjrJvV55rKjaJyTya6o7il6GFSkekq0N1oH4O3w9TYgN3x9P1AI3JDAPjoBO9x9fsS6GcCJMep2Dcsi63WNKJvpe84zPDNcPy58/y+CcdzKfSrQzAYTnEXRvHlzCgsLEziM2qW4uLjatktRSeyZP5duKEl6zNW5XdJFbRKb2iWQaLJZS3AJDWAp0I3gF3xTIPYTeXvLBTZGrSuK2G903aKoernhfZvosj32Y2ZnA3XcfbSZ9S8vIHcfCYwE6Ny5s/fvX271WqmwsJDq2i4FkyewNMYoAw3rZnL0cf2ol5W8Yfuqc7uki9okNrVLINHLaBOB74WvnwfuM7NnFl5oAAAW10lEQVTHgP8STD+QiGKgUdS6RsCmBOo2AorDs5m4+wkv1d0NXJ1gTLIfi/UwaB0zNm3bwYB73qNw3qo0RSYi0RJNNlcRJBYI7oMMIzireR64LMF9zAcyzaxjxLqewJwYdeeEZbHqzQF6RPZOI+hEMAfoCLQDJprZCmAUcKCZrTCzdgnGKfuJgb0LuPOc7hTk5WBAQV4Ow8/ryTOXHUUdMy597BN+88w0VsQY6FNEUivRhzrXRbzeBQyt7Ae5+2YzGwXcbmaXAb2As4BjY1R/ErjGzF4nGKHgWoL7MBDcI9oJXG1mD/HtmG0TgF3AQRH7OZbgvtLhqGdajRTvYdA3ftePke9+xf3vfEHhvFVc873OXHJMWzLr6NEykXSozHM2LczsOjN70MyaheuOC3t+JerXBPd4VhGcKV3p7nPMrJ+ZRfZqGwG8SjAr6GxgbLgOdy8l6BZ9MbAB+AUw0N1L3X2Hu68oW4B1wK7w/c5KxCn7ubqZdfh/p3Tkrd+fSN/2+fzltc848/73mfaNBvYUSYeEzmzMrA/wP2AhQa+vYcAa4DSCXmYxH86MFp4hDYyxfiLf9nAjvDdzfbjE2s90oE8Cn1cItE4kNqmZ2jStz2OX9mXc7BXc9upnDHrwA37atw3dCxrxwDtf7jEWm2YGFUmeRHuj/R24193/bGaRN/THAz+v+rBEqo6Z8f3uB9KvU3PueWs+j0xauPsGJHz7MCighCOSJIleRutD8KR+tOUEowGIVHu5dTO5+YwuNG9Yd68yPQwqklyJJpsSgqFhoh1KcP9FZL+xetO2mOs1M6hI8iSabMYAfzazsj8JPexKPBR4KQlxiSRNvJlBHfjj6Fms2qSu0iJVrTJjo+UTdB+uD0wCviB4cv/m5IQmkhyxHgatl5VBv47NeP6TxfQfVsg/35rP5m2xh8MRkcpL9DmbjcDxZnYywTMrGcA0d3+7/C1Fqp94M4MO7F3A12s2M+zNedz7vwU889E3/O7Ujvyk70Fk6fkcke8k0d5oALj7BIKHJwEws7bAMHc/r6oDE0mmeA+DtmvWgAcuOJzLjl/Pna9/zs0vz+bR9xfyh9MP5XtdWjDm02UMGz+PpRtKKJg8QV2mRRJUqWQTQx7BtAEiNUrvNk147oqj+d/cVdw17nOueGoq7ZvWZ1nRVrbt2AWoy7RIZejagEgcZsapXVow7rf9uPOc7ixat2V3oimjLtMiiVGyEalAZp0Mzj+yDe6xy9VlWqRiSjYiCYrXZbpuZgYzFm9IcTQi+5dy79mY2SsVbB89r4xIjTVkQGduHDWLku3fjumamWGYwVkPvM8JnZpz9cmHcES7/DRGKVI9VdRBYG0C5QurKBaRai2yy/TSDSUUhF2mT+3Sgqc+XMTDE7/i3Ic+5OgO+Vx9ckeOObgpe067JFJ7lZts3F2DbIpEKOsyHT3V75X9D+aSY9vy348XM+LdL7ng4Y/o07YJV518CP07Nd/dZVqjTEtt9V27PotIqH52Jr88vj0XHtWGF6Ys5sHCL/n5Y59wUJMcVm7cSunOoIeBukxLbaQOAiJVrF5WHS46ph2FQ05i6KDuLCv6NtGUUZdpqW2UbESSJDszg5/0bcOuXbH7TKvLtNQmSjYiSVbeKNPXPj+D2UuLUhuQSBoo2YgkWaxRputmZnD8IU15Y/ZyzvjXJM576EPemLWcHTt3xdmLyP5NHQREkqy8UaaLSrbzwpTFPP7B11z5zDQK8nK45Ni2/OSINjSun8XL05eqF5vUCEo2IikQb5TpxjlZXNavAz8/rj1vfbaSx95fyN9e/5x/vrWAw9s0ZsqiDRr4U2oEJRuRaqBOhnF6t5ac3q0lc5YV8fj7X/PC1CV71SvrxaZkI/sb3bMRqWa6tmrMsB/3JN7YA+rFJvsjJRuRaqq8Xmw/fugDXpiymC2lmrpa9g9KNiLVVKxebPUyMzizx4GsLS5lyIszOfKO/3HjqJlM/2Y9Hm8OBJFqQPdsRKqp8nqxuTtTFq3nuU8W8/L0Zfz348V0apHLeUccxDmHt+a9+avVi02qFSUbkWosXi82M6Nvu3z6tsvnz2d24bWZy3nuk8X8dexc/vb6XADKBi5QLzapDnQZTWQ/17BeFucf2YaXf3Mc4393AjnZdYgeIadk+07uHv95egIUQclGpEbp3LIhW7btjFm2bMNW7nx9LrOXFun+jqRcSpONmeWb2Wgz22xmi8zsgjj1zMyGmtnacBlqEbNQmVkvM5tqZlvCn70iyoaY2Wwz22RmC81sSCqOTaS6iNeLrV5mBo9MWsgZ/5rEycPf5R9vzuOLVZv2qPPy9KUcd9cE2t8wluPumsDL05emImSpBVJ9z+YBoBRoAfQCxprZDHefE1VvMDAQ6EnQ0/MtghlBHzKzbGAMcA/wb+AKYIyZdXT3UsCAi4GZwMHAm2a22N2fTfrRiVQDsaavzsmqw53ndKd/5+aMm72CV2cu4/53vuC+CV9waMuGnNmzFfUyM/j7m/N3b6d7PVKVUpZszKwBMAjo5u7FwCQzewW4CLghqvolwHB3XxJuOxy4HHgI6B/GfY8H1wLuM7PrgJOBce5+d8R+5pnZGOA4QMlGaoXyerEB/PTINvz0yDas2rSV12cu59WZy+POraMRC6SqWKqu3ZpZb+B9d68fse464ER3PzOqbhHwPXf/KHx/BPCOuzc0s9+HZd+PqP9aWD48aj8GTANGuPtDMWIaTHAWRfPmzfs8//zzVXS0NUdxcTG5ubnpDqPaqWntsnrLLoa8F39kgscG1CfiSnZMNa1NqkpNb5eTTjppqrsfUVG9VF5GywU2Rq0rAhrGqVsUVS83TB7RZeXt51aC+1KPxQrI3UcCIwE6d+7skXPKS6CwsBC1y95qYrvcM3MCS+MMhfPHybs49bAWnNqlBcd0aEp25t63e2tim1QFtUsglcmmGGgUta4RsCmBuo2AYnd3M0toP2Z2FcG9m37uvu27BC5SG8S611MvK4OzexewbnMpL05dwlOTF5FbN5MTOzfne11a0L/TAbwzbxXDxs9j6YYSCiZP0AOkElMqk818IDO8kb8gXNcTiO4cQLiuJ/BxjHpzgGvNzPzba4A9CDofAGBmvyC4D3RC2X0fESlfRfd6tm7fyQdfruGtz1by1merGDtzOQaY6QFSqVjKko27bzazUcDtZnYZQW+0s4BjY1R/ErjGzF4nnD0X+FdYVgjsBK42s4cIOg4ATAAwswuBvwEnuftXSTockRop3ogFAPWy6nDyoS04+dAW3DHQmbFkAxc98jHF2/YcDLRk+07+8tpnnNalBQ3qapASCaT6m/Br4FFgFbAWuNLd55hZP+ANdy+7izYC6ADMCt8/HK7D3UvNbGC47i5gLjAw7PYM8FegKfBJxA3Np939V0k9MpFaJCPD6N2mCZu3xR51eu3mUnrd/iZ92jahX8fmnNCxOV1bNSIj49tOBpqFtHZJabJx93UEz89Er59IcOO/7L0D14dLrP1MB/rEKWtfJcGKSIVa5eXE7FTQLDebc/sctHtA0GHj55HfIJvjD2lGv47N2Fy6g6FvzNMzPbWIznFFZJ/Fe4D05h92YWDvAm74/qGs3rSN979Yw3sLVjNxwRpembEs5r70TE/NpmQjIvssslPB0g0lFMS4HNa8Yd3d94LcnXkrN3H6PRNj7m/phhLe/mwlfdvl07h+VkqOQVJDyUZEvpOyRJLI8yRmxqEtG1EQ5/IbwGVPTsEMDm3ZiKPa53NU+3z6ts+nWW5dQPd69ldKNiKScvEuv91+Vlfa5Nfno4Xr+HjhOp77ZDGPf/A1AIcckMsBDbP55Ov1bN8Z9LXWvZ79h5KNiKRcRc/0HNWhKQClO3Yxa2kRHy9cx0cL1/LuvNVED7BVsn0nd4ydyw+6HxhzZAOpHpRsRCQtynump0x2ZgZ92jahT9smXNn/YNrfMDZmvdXF2+h263i6FzSm90F59GqTR+82TWjVuN7uMd10+S29lGxEZL8Rr6t1k/pZnNunNdO/2cBTkxfx8KSFABzQsC69DsojOzODNz9bSemOXYAuv6WDko2I7Dfi3ev585lddyeN7Tt38fnyTUxfvJ7p32xg+jfr+Xrtlr32VbJ9J38dq5EOUkUtLCL7jYru9QBk1cmge+vGdG/dmIuPCda1v2HsXvd6ANYUl9Lt1vG0b9aAbq0a07VVI7oVBD/z6mcDuvxWVZRsRGS/ksi9nmjxLr81bZDNxce0Y/ayIqYuWr/HA6cFeTnkN8hi7vJN7Nil3m/flZKNiNR48S6/3XJGlz2SxrrNpcxZVsScZRuZvbSIN2avYOeuPc+JSrbv5KbRsyjetoPOLRvS6YCGMR9ALTsj0tQLASUbEanxErn8BpDfIJt+HZvTr2NzgLi93zaX7uTml2fvft+yUT06t2wYJJ8WDVleVMID73zB1u3qkFBGyUZEaoWqvPxWkFeP5391LPNXbGLeyk3MWxEsH361dnePt2hlzwOd1PmAcofiqan3iJRsRETiiHf5bciAQynIy6EgL4eTDj1gd9mOnbtYtG4Lpwx/N+b+Vhdvo+ftb9IsN5sOzXI5+IAGe/yctmgdN708p0aOhq1kIyISR6KX38pk1sng4Oa5ccd+y2+Qza9O7MCXqzbz1Zpixs9ZybrNi8uNoWT7Tu58Yy4/6tlqj/mAolX3MyIlGxGRcuzL5bd4Z0R/iuqQALB+cylfrSnmy9Wbuf7FmTH3t3LjNg69ZRyt83No17QBbfLr07ZpsLTJb8Cn36znljHV+4xIyUZEpIolMvVCmSYNsunTIJ8+bfO59+0FMc+I8nKy+Enfg1i0dguL1m3ho6/Wsrl05171IpU9tHpEuya0bFSPzDqxx41L1RmRko2ISBJUZuqFMvHOiG79Udc9EoC7s3ZzKYvWbmbR2i1c8/yMmPtbU1zK8UPfoU6G0bJRPQqa5NC6SQ6t83IoaJLDorVbeGTSQrbtwzA+ZUkqu+UhMWdNjqZkIyJSTSR6j8jMaJZbl2a5denTNp/hb86P+9DqdQM6s3R9CUs3lLBk/RYmf7mWFRu3sivWkAp8+xzR8qKtHNi4Hi0b1+PAxvVo0age9bLqAEGiiU6KFVGyERGpRqryHlH0Q6tltu/cxYqirfS7+52Y+9tcupOh4z7fa31+g2xaNqrHV6uL2Rqni3c8SjYiIvu5yvaay6qTwUH59eP2mivIy+HN35/Aio1bWVG0leVFW1lRVBL+3MpnyzdWOkYlGxGRGqAqz4iGDOhMg7qZHNw8l4Ob5+613XF3TYg7rXc8mtZORKSWGti7gDvP6U5BXg5GcEZz5zndK0xaQwZ0Jie8f5MondmIiNRi+3JGFHnZbnmC2+jMRkREKm1g7wLev+FkSld8MTWR+ko2IiKSdEo2IiKSdEo2IiKSdEo2IiKSdEo2IiKSdClNNmaWb2ajzWyzmS0yswvi1DMzG2pma8NlqJlZRHkvM5tqZlvCn70S3VZERFIv1Wc2DwClQAvgQuBBM+sao95gYCDQE+gBnAlcAWBm2cAY4GmgCfAEMCZcX+62IiKSHilLNmbWABgE3OLuxe4+CXgFuChG9UuA4e6+xN2XAsOBS8Oy/gQPo97j7tvc/T7AgJMT2FZERNIglSMIdAJ2uPv8iHUzgBNj1O0alkXW6xpRNtPdIwfInhmuH1fBtnsws8EEZ0IA28xsdmKHUqs0A9akO4hqSO2yN7VJbDW9XdomUimVySYXiB4qtAhoGKduUVS93PDeS3RZ9H7ibhuVoHD3kcBIADOb4u5HJH44tYPaJTa1y97UJrGpXQKpvGdTDDSKWtcI2JRA3UZAcZgsKtpPeduKiEgapDLZzAcyzaxjxLqewJwYdeeEZbHqzQF6RPUw6xFVHm9bERFJg5QlG3ffDIwCbjezBmZ2HHAW8FSM6k8C15hZgZm1Aq4FHg/LCoGdwNVmVtfMrgrXT0hg2/KMrPxR1Qpql9jULntTm8SmdgEslVeXzCwfeBQ4DVgL3ODu/zGzfsAb7p4b1jNgKHBZuOnDwB/KLoWZWe9wXRdgLvBLd5+eyLYiIpJ6KU02IiJSO2m4GhERSTolGxERSbpan2wSHa+ttjGzQjPbambF4TIv3TGlmpldZWZTzGybmT0eVXaKmX0ejs/3jpkl9GBbTRCvXcysnZl5xHem2MxuSWOoKRV2WHok/D2yycw+NbPvR5TX2u8MKNlA4uO11UZXuXtuuHROdzBpsAz4K0Gnlt3MrBlBz8pbgHxgCvBcyqNLn5jtEiEv4nvzlxTGlW6ZwGKCUVEaAzcDz4dJuLZ/Z1I6gkC1EzFeWzd3LwYmmVnZeG03pDU4STt3HwVgZkcArSOKzgHmuPsLYfmtwBozO9TdP095oClWTrvUauHjHbdGrHrNzBYCfYCm1OLvDOjMJt54bTqzCdxpZmvM7H0z65/uYKqRPcbfC3/JfIm+N2UWmdkSM3ss/Iu+VjKzFgS/Y+ag70ytTzaVGa+ttvkD0AEoIHgo7VUzOzi9IVUbFY3PV1utAfoSDMzYh6A9nklrRGliZlkEx/5EeOZS678ztT3ZVGa8tlrF3T9y903hNA5PAO8DP0h3XNWEvjcxhFOHTHH3He6+ErgK+J6Z1ZpfqABmlkEwMkopQRuAvjO1PtlUZry22s4J5g2SqPH3wnt/B6PvTbSyJ8Zrze+ZcASTRwg6HA1y9+1hUa3/ztSaL0EslRyvrdYwszwzG2Bm9cws08wuBE4gmC+o1giPvR5QB6hT1h7AaKCbmQ0Ky/9EMMdSrbjRG69dzOwoM+tsZhlm1hS4Dyh09+jLRzXZg8BhwJnuXhKxvlZ/ZwBw91q9EHRDfBnYDHwDXJDumNK9AM2BTwhO8TcAk4HT0h1XGtrhVoK/ziOXW8OyU4HPgRKCwWHbpTvedLcLcD6wMPy/tJxgUNyW6Y43he3SNmyLrQSXzcqWC2v7d8bdNTaaiIgkX62+jCYiIqmhZCMiIkmnZCMiIkmnZCMiIkmnZCMiIkmnZCMiIkmnZCNSQ4Vzy5yb7jhEQMlGJCnM7PHwl330MjndsYmkQ62ez0Ykyd4mmBspUmk6AhFJN53ZiCTPNndfEbWsg92XuK4ys7HhNMGLzOxnkRubWXcze9vMSsxsXXi21DiqziVmNiuconmlmT0RFUO+mb0QTnv+VfRniKSKko1I+twGvAL0Ipgz6Mlw9suyUYHHE4ytdSRwNnAsEVMxm9kVwAjgMaAHwRQQs6M+40/AGIIRh58DHjWzNsk7JJHYNDaaSBKY2ePAzwgGZYz0gLv/wcwceNjdL4/Y5m1ghbv/zMwuB/4OtHb3TWF5f+AdoKO7f2FmS4Cn3T3mFObhZ9zl7jeG7zMJJgsc7O5PV+HhilRI92xEkuc9YHDUug0Rrz+MKvsQ+GH4+jCCIegjJ9f6ANgFdDGzjQSzqP6vghhmlr1w9x1mtho4ILHwRaqOko1I8mxx9y+SsN/KXI7YHvXe0eVzSQN96UTS5+gY7+eGr+cC3aOmVD6W4P/sXHdfBSwFTkl6lCJVQGc2IslT18xaRq3b6e6rw9fnmNknBBNpnUuQOI4Ky54h6EDwpJn9CWhC0BlgVMTZ0h3AP81sJTAWqA+c4u7Dk3VAIvtKyUYkeU4lmLEy0lKgdfj6VmAQwfTJq4Gfu/snAO6+xcwGAPcAHxN0NBgD/LZsR+7+oJmVAtcCQ4F1wOvJOhiR70K90UTSIOwp9mN3fzHdsYikgu7ZiIhI0inZiIhI0ukymoiIJJ3ObEREJOmUbEREJOmUbEREJOmUbEREJOmUbEREJOn+P1NmGawFB/rvAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.epoch, history.history[\"lr\"], \"o-\")\n",
    "plt.axis([0, n_epochs - 1, 0, 0.011])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.title(\"Exponential Scheduling\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The schedule function can take the current learning rate as a second argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "def exponential_decay_fn(epoch, lr):\n",
    "    return lr * 0.1**(1 / 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to update the learning rate at each iteration rather than at each epoch, you must write your own callback class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/25\n",
      "55000/55000 [==============================] - 7s 132us/sample - loss: 0.8067 - accuracy: 0.7678 - val_loss: 0.7942 - val_accuracy: 0.7780\n",
      "Epoch 2/25\n",
      "55000/55000 [==============================] - 7s 122us/sample - loss: 0.6784 - accuracy: 0.7937 - val_loss: 0.8375 - val_accuracy: 0.8120\n",
      "Epoch 3/25\n",
      "55000/55000 [==============================] - 6s 114us/sample - loss: 0.6060 - accuracy: 0.8148 - val_loss: 0.6303 - val_accuracy: 0.8304\n",
      "Epoch 4/25\n",
      "55000/55000 [==============================] - 6s 114us/sample - loss: 0.5279 - accuracy: 0.8341 - val_loss: 0.5724 - val_accuracy: 0.8196\n",
      "Epoch 5/25\n",
      "55000/55000 [==============================] - 6s 112us/sample - loss: 0.4803 - accuracy: 0.8486 - val_loss: 0.5488 - val_accuracy: 0.8486\n",
      "Epoch 6/25\n",
      "55000/55000 [==============================] - 6s 113us/sample - loss: 0.4305 - accuracy: 0.8611 - val_loss: 0.4778 - val_accuracy: 0.8470\n",
      "Epoch 7/25\n",
      "55000/55000 [==============================] - 6s 112us/sample - loss: 0.3969 - accuracy: 0.8699 - val_loss: 0.4922 - val_accuracy: 0.8584\n",
      "Epoch 8/25\n",
      "55000/55000 [==============================] - 6s 111us/sample - loss: 0.3799 - accuracy: 0.8777 - val_loss: 0.5417 - val_accuracy: 0.8614\n",
      "Epoch 9/25\n",
      "55000/55000 [==============================] - 6s 111us/sample - loss: 0.3475 - accuracy: 0.8851 - val_loss: 0.5032 - val_accuracy: 0.8734\n",
      "Epoch 10/25\n",
      "55000/55000 [==============================] - 6s 110us/sample - loss: 0.3256 - accuracy: 0.8937 - val_loss: 0.4433 - val_accuracy: 0.8802\n",
      "Epoch 11/25\n",
      "55000/55000 [==============================] - 6s 110us/sample - loss: 0.2944 - accuracy: 0.9017 - val_loss: 0.4888 - val_accuracy: 0.8742\n",
      "Epoch 12/25\n",
      "55000/55000 [==============================] - 6s 110us/sample - loss: 0.2767 - accuracy: 0.9077 - val_loss: 0.4626 - val_accuracy: 0.8706\n",
      "Epoch 13/25\n",
      "55000/55000 [==============================] - 6s 111us/sample - loss: 0.2572 - accuracy: 0.9134 - val_loss: 0.4750 - val_accuracy: 0.8770\n",
      "Epoch 14/25\n",
      "55000/55000 [==============================] - 6s 111us/sample - loss: 0.2391 - accuracy: 0.9185 - val_loss: 0.4633 - val_accuracy: 0.8900\n",
      "Epoch 15/25\n",
      "55000/55000 [==============================] - 6s 112us/sample - loss: 0.2180 - accuracy: 0.9251 - val_loss: 0.4573 - val_accuracy: 0.8768\n",
      "Epoch 16/25\n",
      "55000/55000 [==============================] - 6s 110us/sample - loss: 0.2029 - accuracy: 0.9311 - val_loss: 0.4748 - val_accuracy: 0.8840\n",
      "Epoch 17/25\n",
      "55000/55000 [==============================] - 6s 112us/sample - loss: 0.1884 - accuracy: 0.9357 - val_loss: 0.5171 - val_accuracy: 0.8840\n",
      "Epoch 18/25\n",
      "55000/55000 [==============================] - 6s 111us/sample - loss: 0.1813 - accuracy: 0.9382 - val_loss: 0.5293 - val_accuracy: 0.8822\n",
      "Epoch 19/25\n",
      "55000/55000 [==============================] - 6s 112us/sample - loss: 0.1618 - accuracy: 0.9445 - val_loss: 0.5328 - val_accuracy: 0.8872\n",
      "Epoch 20/25\n",
      "55000/55000 [==============================] - 6s 111us/sample - loss: 0.1570 - accuracy: 0.9483 - val_loss: 0.5453 - val_accuracy: 0.8870\n",
      "Epoch 21/25\n",
      "55000/55000 [==============================] - 6s 112us/sample - loss: 0.1422 - accuracy: 0.9523 - val_loss: 0.5596 - val_accuracy: 0.8892\n",
      "Epoch 22/25\n",
      "55000/55000 [==============================] - 6s 111us/sample - loss: 0.1329 - accuracy: 0.9563 - val_loss: 0.5717 - val_accuracy: 0.8894\n",
      "Epoch 23/25\n",
      "55000/55000 [==============================] - 6s 110us/sample - loss: 0.1248 - accuracy: 0.9592 - val_loss: 0.5959 - val_accuracy: 0.8930\n",
      "Epoch 24/25\n",
      "55000/55000 [==============================] - 6s 112us/sample - loss: 0.1178 - accuracy: 0.9606 - val_loss: 0.5875 - val_accuracy: 0.8896\n",
      "Epoch 25/25\n",
      "55000/55000 [==============================] - 6s 111us/sample - loss: 0.1103 - accuracy: 0.9646 - val_loss: 0.6103 - val_accuracy: 0.8904\n"
     ]
    }
   ],
   "source": [
    "K = keras.backend\n",
    "\n",
    "class ExponentialDecay(keras.callbacks.Callback):\n",
    "    def __init__(self, s=40000):\n",
    "        super().__init__()\n",
    "        self.s = s\n",
    "\n",
    "    def on_batch_begin(self, batch, logs=None):\n",
    "        # Note: the `batch` argument is reset at each epoch\n",
    "        lr = K.get_value(self.model.optimizer.lr)\n",
    "        K.set_value(self.model.optimizer.lr, lr * 0.1**(1 / s))\n",
    "\n",
    "    def on_epoch_end(self, epoch, logs=None):\n",
    "        logs = logs or {}\n",
    "        logs['lr'] = K.get_value(self.model.optimizer.lr)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "lr0 = 0.01\n",
    "optimizer = keras.optimizers.Nadam(lr=lr0)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "n_epochs = 25\n",
    "\n",
    "s = 20 * len(X_train) // 32 # number of steps in 20 epochs (batch size = 32)\n",
    "exp_decay = ExponentialDecay(s)\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[exp_decay])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_steps = n_epochs * len(X_train) // 32\n",
    "steps = np.arange(n_steps)\n",
    "lrs = lr0 * 0.1**(steps / s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEeCAYAAABc5biTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xd8FHX6wPHPk94TktA70ntTVEDAhnp6Yj/xVEQP6/kTRU89PbseKtZTxHJiPcWCKOqhHgQEFQGRXqSH0BNISEgIhOf3x0xwWTbJJmSzKc/79ZpXduf7nZlnJrvz7Mx85zuiqhhjjDGBFBLsAIwxxtR+lmyMMcYEnCUbY4wxAWfJxhhjTMBZsjHGGBNwlmyMMcYEnCUbU+OJyAgRyS3nNGki8q9AxeQuY4OIjAnAfC8WkXLds+C9jSqyzY6FiDwgIv+uquX5WL6KyMVBWG6Z21lEbhaRL6oqpmCxZFODichE90vkPfwU7NgCpYSdxodAmwAs6zoRWSgiuSKSLSKLReTRyl5OkARkm/kiIg2AO4Aave1E5EERWRqAWb8O9BGRgQGYd7URFuwAzDH7DrjSa1xhMAIJFlXNB/Irc54iMhJ4ARgN/A8IB7oCJ1XmcoIlENusFNcBP6vqukAvSETCVfVAoJdTmVR1v4i8D9wKfB/seALFjmxqvv2qus1ryAIQkUEickBEBhdXFpHrRSRHRNq479NE5BUReV5EdrvDUyIS4jFNPRF5yy3LF5HvRKSLR/kI99f/aSKyVETyRGSGiLT2DFREzhORBSJSICLrReQxEYnwKN8gIveJyAQ3xs0icqdnufvyI/cIZ4Pn8j3qHSciU0RkmxvLLyJybjm36x+BT1V1gqquUdUVqvqRqt7utU7niMhcd7tkisgXIhLlUSWqpPVxp08UkVdFZIeI7BWRmSLS16vOVSKyUUT2ichUoKFX+VG/uMs6feNjmz3o/u/+JCJr3Vg+E5FUjzphIvKsx+fkWREZLyJpZWzL4cARp4n8/NxFiMhYd7vtE5F5IjLUo3yw+zk4R0R+FpFCYCglayQiX7rz2igif/aK6Z8issr9X24QkSeL/5ciMgJ4AOgiv59BGOGWJbrbYav72V4hIpd5zbvU7wbwOfBHEYkpY1vWXKpqQw0dgInA1DLqPA6kA/WAjkAecLVHeRqwF3jRLb8UyAZu96gzBVgJnAJ0w/lipAPRbvkI4ADOUdYJQHdgITDNYx5DgRzgGuA4YAiwCnjao84GIBO4BWgL/BVQ4CS3vL77/jqgEVDfY/m5HvPpAdzgxtoW+DvO0V5Hr/X+Vynb7RVgNdCmlDpnAQdxTg91dtd7DBDj5/oIMBv40t1ubYFH3O3U2K3TDzjkrkN74Hp3nuoRx4PAUq/YvLdJWe8fBHKBye56nARsBCZ41Lkb2A1cBHQAnnc/K2mlbKNkN/7+XuPTKPtz9x7wE87nro27HQuBHm75YHd7LgHOdOvULyEOdbfb9e52/LsbV1+POvcD/YFWwDnAJuARtywaeBrne9DIHaLd/+EcYLn7eWgDnA1c4O93w60XAxQBpwV7vxKoIegB2HAM/zwn2Rx0dxKew1iPOuHAPOBT4BfgQ695pOHsVMVj3H3AZvd1O/eLeopHeaK7Y7jOfT/CrdPBo84VwP7i+QKzgPu9lj3Mjbe4zgbgP151fgPu83ivwMVedUbgseMsYVv95DWfNEpPNo2BH93l/Qa8C1wFhHvUmQN8UMo8Sl0f4FR3/aO96vwK3OW+fh/41qv8dQKTbAqARI9xfwfWeLzfCtzt8V5wfjCklbINerrbsHU5P3fH4SSDFl7TfQa87L4e7M77Ij++Kwq85jXuO+DdUqa5wWv9fW3nM9w4O5UwjxGU8d3wGJ8FXFvWutTUwU6j1XyzcL7QnsNTxYXqnL8eDpwLNMD5ZeftJ3U/7a4fgaYikgB0wvky/egxz2ycX5OdPabZr6qrPN5vASJwjqgA+gB/d0+35bqncN4HYnF+JRZb7BXbFjduv4lIrHsKZLl7eiYX6Au08HceqrpVVU/COTp6DmfHOgH42eNURy+c6zmlKW19+uD8ot3ptV264uxswdn+P3rNw/t9Zdno/m+PilVEEnH+Tz8XF7qfmZ8pXbT7t8BHWWmfu94423y517b5A79vm2Lzy4jBc/7e7w9/hsVp5TfbPf2aCzxL2Z+ZXsBWVV1RSp2yvhvF8vl9e9U61kCg5tunqmvKqHMizvW5JJxTUXsqadmeO4qDJZSFePx9CPjIx3x2erz2vrirlP/a4tM4pzTG4BxJ7APexvmCl4uqLgWWAi+JyACcC7iX4hxV+qO09QkBtgO+WiHllCPMQzg7Zk/h5Zi+WGVse2+73L/1cI6M/BXiLv94H3F5N2zIq1hovxORE4EPcD6jo3G+I3/E+Swdq7K+G8WSOfK7UKvYkU0t516I/BdwM/At8K6IeP/I6CcinjurE4EtqpoDrMD5nBxuheX+8uyGc57aX7/gXDNZ42Pw/jKW5gAQWkadAcDbqvqJqi4GNnP0r+GKKF7fOPfvQuC0Y5jfLzgX+w/52CY73DorcP4fnrzf7wQaev0Pex5DXEdxj3i24ez8AXCXd3yJEznW4iTOzj7KSvvcLcRJoI18bJuMCq6Gr+1YfETSH8hQ1UdUdZ6q/ga09KpfyNGfvYVAYxHpVMGYAKdRCxCF85molezIpuaLFJFGXuOKVHWniIQC7wAzVXWCiHyMc/rrAZyLocWaAM+JyMs4SeRO3HsiVPU3EZkCTBCRUTi/+B7D2YG8X444HwamishGYBLOr72uwAmqelc55rMBOE1EZuKcntjto85q4AI37gM46xvlo16JRGQ8zumO6TjJqjHONYV9wDdutceAL0RkDc62EJwL1RNUdZ8fi/kO57rPFBG5i98vPp8FfKeq3+M0v/5BRO4BPsa5TnGB13zScH4V3ysiH7h1AnED4/PAXSKyGifxXo+zXUo8YlHVQyLyHc4PgI+9ikv73K0WkfeAiSJyB85OOBln3dap6qcViP9CEZmHs70uxvmh0M8tW41zCu8KnNNrQ4HLvabfALQUkd44jQf24pxGnQt8IiKj3fm0BWJV9bNyxDbQXa/fKrBeNYId2dR8p+N82T2HhW7ZvTgf/GsBVDUTuBq42z0lVOw9nF9sc4HXgDdwzlcXuwbn3Pzn7t8Y4Cx17tXwi6pOwznfPsSdx884rZs2+b+qgHNz4BCc1nALS6hzO7AD55TX1ziNA8p7/8K3ODuiSTg7kMnu+DNUdTWAqn6Fs+M/241lphvbIX8W4F6vOAcnob2Gc7F9Ek5Lry1unZ9w/n834lz/uRDnQrXnfFa45aPcOmfgtEKsbE/j/Hh5E2ebgrNdfF2P8fQqcJn748eTP5+7N4EncRLxVJyWaRsrGP+DOC3pFuNsr2tUdR6Aqn6Bc63zOX7fhv/wmv4T4CucBLMTuFxVD+H8/+fgNCJZgZOUy3vK9nKcbVBrFbcCMnWUe4/EUlW9JdixmJpHRBYCs1X1r2XU+xGnFdk77vs07HMHgIh0xUlg7b0aaNQqdhrNGOMXEWmJc3ppJk4DhL/g3DfyFz8mvx6n5ZY5WhPgqtqcaMCSjTHGf4dw7jV6CucU/HLgbFUts+mx21DDuxm4AVT1m7Jr1Xx2Gs0YY0zAWQMBY4wxAWen0VxJSUnatm3bYIfhU15eHrGxscEOwyeLrWIstoqx2ComkLEtWLBgl6rWL7NisPvLqS5D+/bttbqaMWNGsEMokcVWMRZbxVhsFRPI2ID5an2jGWOMqQ4s2RhjjAk4SzbGGGMCzpKNMcaYgLNkY4wxJuAs2RhjjAk4SzbGGGMCzpKNMcaYgLNkY4wxJuAs2RhjjAk4SzbGGGMCzpKNMcaYgLNkY4wxJuAs2RhjjAk4SzbGGGMCrkqTjYgki8hkEckTkY0iMryEeiIiY0Uk0x3Gioh4lL8qIqtE5JCIjPAx/WgR2SYiOSLybxGJDOBqGWOMKUNVH9m8BBQCDYErgPEi0sVHvVHAMKAH0B04D7jeo3wRcBPwi/eEIjIUuBs4DWgJtAEeKiuwQ1qe1TDGGFMeVZZsRCQWuAi4X1VzVXU28DlwpY/qVwPjVHWzqmYA44ARxYWq+pKq/g8oKGHaN1R1maruBh7xnLYkm3MPsWdfYTnXyhhjjD/EeapnFSxIpBcwR1VjPMaNAQap6nledbOBM1V1rvu+LzBDVeO96s0GXlfViR7jFgGPq+qH7vtUYCeQqqqZXtOPwjmKIqJR2z7D7/sXV3epfmfccnNziYuLC3YYPllsFWOxVYzFVjGBjG3IkCELVLVvWfXCArJ03+KAHK9x2UB8CXWzverFiYho2dnR17S4yzki2ajqq8CrAJGN22na5oOMPr8f3ZsllbGIqpWWlsbgwYODHYZPFlvFWGwVY7FVTHWIrSqv2eQCCV7jEoC9ftRNAHL9SDQlTUsJy/m9UoSgCvd/tpRDdgHHGGMqVVUmm9VAmIi08xjXA1jmo+4yt6yser74mna79yk0b0mRQqOEKBZtzuaDeel+LsoYY4w/qizZqGoe8CnwsIjEikh/4HzgHR/V3wZuF5GmItIEuAOYWFwoIhEiEgUIEC4iUSIS4jHttSLSWUSSgPs8py1JiMB953YC4MlpK8nKs8YCxhhTWaq66fNNQDSwA/gPcKOqLhORgSKS61FvAvAFsARYCnzpjiv2DZAPnIxzzSUfOAVAVf8LPAnMADYBG4EH/AnuD90aM6BtKnv2HeCpaSsrvJLGGGOOVKXJRlWzVHWYqsaqagtVfd8d/72qxnnUU1W9S1WT3eEuz+s1qjpYVcVrSPMof0ZVG6pqgqpeo6r7/YlPRHjwj10IDxU+mJfOwk27K3HtjTGm7rLuary0bRDHdQPbOI0FpiylyBoLGGPMMbNk48NfT21Lk8Qolmbk8P7cjcEOxxhjajxLNj7ERITxj/M6A/Dkf1exPcdXRwXGGGP8ZcmmBEO7NOK0jg3Yu/8gD33hb6trY4wxvliyKYGI8PCwrsREhPLVkm18t3x7sEMyxpgay5JNKZomRTPmzA4A/GPKUnL3HwxyRMYYUzNZsinD1Se3onuzRLZkFzDum1XBDscYY2okSzZlCA0RnriwG6EhwsQfNvBr+p5gh2SMMTWOJRs/dGmSyHUDWqMK93y6hANFh4IdkjHG1CiWbPz0f6e3o3lyNCu25vDG7PXBDscYY2oUSzZ+iokI49Fh3QB47rvVbMzMC3JExhhTc1iyKYdB7eszrGcTCg4c4q6PF9tzb4wxxk+WbMrpH+d1ITUugrnrs3jXurIxxhi/WLIpp+TYCB4d1hWAf369kvSsfUGOyBhjqj9LNhVwVtfGnNu9MfsKi+x0mjHG+MGSTQU99McupMRG8OO6TN7/eVOwwzHGmGrNkk0FpcRF8vD5zum0J75awebddjrNGGNKYsnmGPyhe2PO6daIvMIi7v5kCR4PEzXGGOPBks0xevj8rtSLCWf2ml18MC892OEYY0y1ZMnmGKXGRfKQezrt0anLrXWaMcb4YMmmEpzncTrt9km/UmSt04wx5giWbCqBiPDYsG40iI9k3obdvDprXbBDMsaYasWSTSWpFxvBU5f0AOCZb1exbEt2kCMyxpjqw5JNJRrUvj5XndSSA0XK6A9/peBAUbBDMsaYasGSTSW75+xOtEmNZfX2XJ6aZk/2NMYYsGRT6aIjQnn2sp6EhghvzF7PnDW7gh2SMcYEnSWbAOjRPIlbT20HwJiPFpGdfyDIERljTHBZsgmQm4ccR4/mSWzNLuDeyda7gDGmbrNkEyBhoSE8f1lPYiNC+XLxVibNt94FjDF1lyWbAGqVGsujFzi9Czzw+TLW7Ngb5IiMMSY4qjTZiEiyiEwWkTwR2Sgiw0uoJyIyVkQy3WGsiIhHeU8RWSAi+9y/PT3KIkXkFRHZLiJZIvKFiDStivXz5YJezbiwV1MKDhzilvcXWnNoY0ydVNVHNi8BhUBD4ApgvIh08VFvFDAM6AF0B84DrgcQkQhgCvAuUA94C5jijgf4P+Akd7omwG7gxQCtj18eHtaVVikxrNy2l8e/WhHMUIwxJiiqLNmISCxwEXC/quaq6mzgc+BKH9WvBsap6mZVzQDGASPcssFAGPCcqu5X1RcAAU51y1sD01R1u6oWAB8CvhJalYmLDOPFy3sTHiq8/eNGpi3bFsxwjDGmyklVtZISkV7AHFWN8Rg3Bhikqud51c0GzlTVue77vsAMVY0XkdFu2dke9ae65ePcus8DlwB7gNeBHap6m4+YRuEcRVG/fv0+kyZNqtyV9jJtwwH+s7KQ2HB4+ORoUqL9y/W5ubnExcUFNLaKstgqxmKrGIutYgIZ25AhQxaoat+y6oUFZOm+xQE5XuOygfgS6mZ71Ytzr9t4l3nP5zcgHcgAioAlwC2+AlLVV4FXATp06KCDBw/2c1UqZpAq2ybOY8aqnXy4MZr3/9KPsNCyE05aWhqBjq2iLLaKsdgqxmKrmOoQm9+n0USkoYiMEZHxIpLqjusvIq39nEUukOA1LgHw1UTLu24CkKvOYVhZ83kJiARSgFjgU+BrP2MMKBHh6Ut60CA+kp83ZPHMt6uDHZIxxlQJv5KNiPQBVuFc1L+W33f2ZwCP+bms1UCYiLTzGNcDWOaj7jK3zFe9ZUB3z9ZpOI0Bist7AhNVNUtV9+M0DjihOEEGW0pcJC9c3osQgZfT1vLd8u3BDskYYwLO3yObp4HnVbUXsN9j/DSgvz8zUNU8nKOMh0UkVkT6A+cD7/io/jZwu4g0FZEmwB3ARLcsDef02K1uM+fiU2TT3b/zgKtEJFFEwoGbgC2qWm06KTuxTQp3Du0IwO2TfrWnexpjaj1/k00fnCbG3rbiNGP2101ANLAD+A9wo6ouE5GBIpLrUW8C8AXO9ZalwJfuOFS1EKdZ9FU4DQBGAsPc8QBjgAKcazc7gXOAC8oRY5W4/pQ2nN6pITkFB7nxvQV2/40xplbzt4FAPs49Ld464iQOv6hqFk6i8B7/Pc6F/+L3CtzlDr7msxAnAfoqy8Q53VethYQI4y7pwbn/+p6lGTk8PHU5j1/QLdhhGWNMQPh7ZDMFeEBEIt33KiKtgLHAJwGIq05IjAln/BV9iAgL4f25m/j0l83BDskYYwLC32QzBkjGOS0VA8wG1uCcxrovMKHVDV2bJvLQH517Tu+dvIRV26z/NGNM7eNXslHVHFUdgHMK7G84N02epaqD3Av/5hj86fjmXNjb6T/thncX2PNvjDG1jr9Nn68SkUhVna6qT6vqk6r6nYhEiMhVgQ6ythMRHhvWjY6N4lm/K4/bPlhI0SF7/o0xpvbw9zTam0Cij/Hxbpk5RtERobx2VV+SYsKZsWonz9oNn8aYWsTfZCOAr5/aLTi66xhTQc2TY3hpeG9CBP41Yw1fLdka7JCMMaZSlNr0WUSW4CQZBWaKyEGP4lCgJfBV4MKre/q3TeXeczrx6JcruGPSIlqnxgY7JGOMOWZl3Wfzsfu3K86NlZ43XhYCG7Cmz5Xu2gGtWbYlh8kLMxj1znz+1lPKnsgYY6qxUpONqj4EICIbgA/d58OYABMRnriwG2t25LIkI5vxi0I467RDfvUQbYwx1ZG/TZ/fskRTtaLCQ5lwZR9SYiNYlnmIx79aGeyQjDGmwvxt+hwhIg+JyGoRKRCRIs8h0EHWVU2Sohn/5z6ECvx7znrem7sx2CEZY0yF+Hte5hHcRzUDh4A7cZ4bk4nTuaYJkBNaJzOiSwQA/5iyjNm/VZvOq40xxm/+JptLgRtUdQJO9/5TVPVW4AGcZ9qYABrYLJwbBh1H0SHlxvcWsGaHdWljjKlZ/E02DYHl7utcIMl9/V/gzMoOyhztrqEdOKtLI/YWHGTkxPlk5RWWPZExxlQT/iabTUAT9/UaYKj7+iScxw+YAAsJEZ65rAfdmiayKWsf178zn/0H7XKZMaZm8DfZTAZOc18/DzwkIutxnp75egDiMj7ERITx+tV9aZQQxbwNu7n7kyU4j/4xxpjqza+Hp6nqPR6vPxaRdJzHQa9W1amBCs4crWFCFK9f3ZdLXvmRyQszaJkSw22ntw92WMYYU6oK3SWoqnNV9RlVnSoi1p9KFevaNJEXLu9FiMBz3/3Gh/M2BTskY4wpVYVvSReRKBG5E1hfifEYP53RuSEPn98VgHsnL2X6yu1BjsgYY0pWarJxb+Z8TETmicgPIjLMHX8VsA64DXi2CuI0Pvz5xJbcMqQtRYeUm99byK/pe4IdkjHG+FTWkc2DwC3ARqA18JGIvAz8HbgHaKWqTwQ0QlOqO85sz8V9mpF/oIiRE+exYZc9ONUYU/2UlWwuBUao6sXAWTiPFagHdHH7S7PnFwdZcaedg9rXJyuvkKv+/TM79+4PdljGGHOEspJNc2AegKouwnmswFhVPVjqVKZKhYeG8PIVvQ/fgzNy4jzy9tu/yBhTfZSVbMIBz5/JB7Anc1ZLsZFh/HvE8bRIjmFJRjaj3plPwQG76dMYUz34c5/NEyKyz30dATwoIkckHLefNBNk9eMjeXvkCVz8yo/MWZPJrf9ZyMtX9Lbn4Bhjgq6svdAs4Digmzv8ALTweN8N5ymepppolRrLu9edQGJ0ON8s385dHy/m0CHrZcAYE1xlPalzcBXFYSpRx0YJvHnN8fz59bl8ujCD+KgwHvxjF0Ts8dLGmOCw8yu1VO8W9Xjtqr5EhIbw1o8beebb1cEOyRhTh1myqcX6t03lxeG9CA0RXpy+hldnrQ12SMaYOqpKk42IJIvIZBHJE5GNIjK8hHoiImNFJNMdxorHOSAR6SkiC0Rkn/u3p9f0vUVklojkish2Efm/QK9bdTW0SyOeurg7AI9/tZJ3frJHSxtjql5VH9m8hHOvTkPgCmC8iHTxUW8UMAzoAXQHzgOuB6cLHWAK8C7ODaZvAVPc8YhIKs5D3SYAKUBb4JvArVL1d2HvZjx8vrOZ7/9sKe/PtY47jTFVq8qSjds79EXA/aqaq6qzgc+BK31UvxoYp6qbVTUDGAeMcMsG4zRseE5V96vqC4AAp7rltwPTVPU9t3yvqq4I2IrVEFed1Ir7z+0MwL2TlzBpXnqQIzLG1CXiz8O3RKRFCUUKFKjqTj/m0QuYo6oxHuPGAINU9TyvutnAmao6133fF5ihqvEiMtotO9uj/lS3fJyITAeWAMfjHNXMBW5W1aN+zovIKJyjKOrXr99n0qRJZa1GUOTm5hIXF1cp8/p6/QE+XFWIANd2i2BA0/BqE1tls9gqxmKrmLoa25AhQxaoat8yK6pqmQNwCCgqZdgNPAOElTKPgcA2r3F/AdJ81C0COnq8b4eT2AS4H/jAq/57wIPu69XAHpxkEwW8gJPkSl3H9u3ba3U1Y8aMSp3f+LQ12vJvU7XV3VP1kwXpxzSvyo6tMllsFWOxVUxdjQ2Yr37kEb+e1AlcDjwJvIJzpADQD+eo4EEgCbgP2As8UMI8coEEr3EJ7jRl1U0AclVVRaSs+eQDk1V1HoCIPATsEpFEVbWudoAbBh1H0SHlqWmrGPPRIkJDhPN7Ng12WMaYWszfZHMjMFpVP/UYN11EVgH/p6qDRGQH8BAlJ5vVQJiItFPV39xxPYBlPuouc8t+9lFvGXCHiIibVcFpRPCS+3oxzlFQMbt93oeb3efgPPPtakZ/+CuqMKyXJRxjTGD420CgH851EG9LcU5XAfwINCtpBqqaB3wKPCwisSLSHzgfeMdH9beB20WkqYg0Ae4AJrplaTin2W4VkUgRucUdP939+yZwgds8OhzntNtsO6o52q2nteO209txSGH0pF/t8dLGmIDxN9lsxL2Q7uUvQPEeqj6QVcZ8bgKigR3Af4AbVXWZiAx0T48VmwB8gZPglgJfuuNQ1UKcZtFX4VybGQkMc8ejqtOBe91pduA0EvB5P4+B205vz51DO6AKf/tkCRPn2FO+jTGVz9/TaHcAn4jIObjPtwH64nTSeZH7/nig1OZcqpqFkyi8x38PxHm8V+Aud/A1n4VAn1KWMx4YX1os5nc3D2lLdHgoD09dzoNfLKfg4CFuGHRcsMMyxtQifiUbVf1SRNrhHJl0cEd/DryibpNiVX05MCGaqjByQGuiwkP5+2dL+OfXK8kvLOK209tZ553GmErh75ENqpoO3BPAWEyQDe/XgqjwEMZ8tIjn//cbBQeKuPvsjpZwjDHHzO9kIyIxQE+gAV7XerxaqZka7MLezYgMC+X/PljIhFnryN1/kIfP70poiCUcY0zF+ZVsROR0nAv6KT6KFQitzKBMcP2he2Miw0K46f1feG/uJnbvK+TZy3oSGWb/ZmNMxfjbGu15nNZdzVQ1xGuwPVAtdHrnhrwz8gTiI8P4ask2rnlzHnsLDgQ7LGNMDeVvsmkFPKKqWwIYi6lm+rVJ4cPrT6J+fCQ/rM3k8td+Yufe/cEOyxhTA/mbbObweys0U4d0bpLAJzecTKuUGJZm5HDJKz+QnrUv2GEZY2oYf5PNK8DTInKdiPRzH052eAhkgCb4WqTE8NENJ9OlSQIbMvdx4fgfWL4lJ9hhGWNqEH+TzcdAR+BVnG5p5nsM80qZztQS9eMj+WDUiZzUJoWde/dz6YQfmbm6zCdLGGMM4H+yaV3K0CYwoZnqJj4qnDevOZ5zuzcmd/9BRk6cR1q6NRowxpTN3x4E7MH1BoCo8FBe+FMvWiTH8HLaWiYuKyTq65XcNbQDIXYvjjGmBCUmGxG5EPhCVQ+4r0tkN3XWLSEhwl1ndaRFcgz3Tl7CKzPXkp61j3GX9iAq3FrCG2OOVtqRzcdAI5yekz8upZ7d1FlH/emEFuza9BsTlhzkyyVb2Zqdz2tX9SUlLjLYoRljqpkSr9m4N2zu8Hhd0mCJpg7rmhrKRzeeRJPEKH7ZtIcLXv6B1dt9PXzVGFOX+dtAwJgSdWyUwOSb+9O1aQKbsvZxwUtz+Hb59mCHZYypRsrTEWcz4BR8d8T5TCXHZWqYhglRfHT9ydz58SKmLt7KqHfmM+bMDtw0+DjrNdoY43dHnFcA/wYOAjtxrtMUU8CSjSE6IpQXL+9Fp8YxhMz6AAAgAElEQVQJPP3NKp6atooVW3N46uIeREfY2VZj6jJ/T6M9DIwDElS1laq29hjsPhtzmIhw85C2vHZlX2IjQpm6eCuXTPiBLXvygx2aMSaI/E02DYHXVbUokMGY2uP0zg2ZfHN/Wrp9qv3xX7OZuy4z2GEZY4LE32TzFdAvkIGY2qd9w3im3Nyf/m1T2JVbyPDX5/LarHWoatkTG2NqFX8bCHwLjBWRLsAS4Ig+SuymTlOSpJgI3rrmBJ7+ZjWvzFzLY1+tYMHG3Tx1SXfio8KDHZ4xpor4m2wmuH/v9VFmN3WaUoWFhnD32R3p1SKJMZMW8d9l21i1fS+v/LkPHRrFBzs8Y0wV8Os0mt3UaSrD0C6N+PyvA+jYKJ71u/IY9tIcPluYEeywjDFVoMxkIyLhIjJXROzhaeaYtU6NZfJN/bmwd1PyDxRx24e/ct9nSyg4YG1PjKnNykw2qnoA51ECdlXXVIroiFDGXdKDxy/oRkRoCO/+tIlhL81hzQ7r5saY2srf1mhvAX8JZCCmbhERhvdrwac3nUzr1FhWbtvLeS/OYdK8dGutZkwt5G8DgVjgChE5A1gA5HkWquqtlR2YqRu6Nk3ki78O4B+fLeXThRnc9clivl+zi8cu6EqCtVYzptbw98imE/ALsBvnyZzdPIaugQnN1BVxkWE8c1lPnrm0BzERoXyxaAt/eOF7fk3fE+zQjDGVxN8ndQ4JdCDGXNi7Gb1a1OOv//mFpRk5XDz+B0af0Z4bBh1HqD0F1JgazR4xYKqV1qmxfHLjyYzs35qDh5Snpq3i0gk/sjEzr+yJjTHVlt/JRkSGiMirIvJfEZnuOZRjHskiMllE8kRko4gML6GeiMhYEcl0h7Hi0U+9iPQUkQUiss/929PHPCJEZIWIbPY3PlM9RIaF8o/zOvP2yBNomBDJgo27Ofv573l/7iZrPGBMDeVXshGREcDXQDwwGOcxA/WA3sDycizvJaAQp2PPK4Dxbhc43kYBw4AeQHfgPOB6N5YIYArwrhvDW8AUd7ynO904TQ11Svv6TLvtFM7r0YR9hUXcO3kJIyfOY0dOQbBDM8aUk79HNmOAW1T1cpx+0e5R1V44O/xcf2YgIrHARcD9qpqrqrOBz4ErfVS/GhinqptVNQPn8QYj3LLBONeanlPV/ar6AiDAqR7Lag38GXjCz/Uz1VRSTAQvXt6LFy7vRWJ0ODNW7WToc7P4asnWYIdmjCkH8ee0hIjsAzqr6gYR2QWcqqqLRaQjkKaqjfyYRy9gjqrGeIwbAwxS1fO86mYDZ6rqXPd9X2CGqsaLyGi37GyP+lPd8nEe79/AaT33rqo2KyGmUThHUdSvX7/PpEmTytwWwZCbm0tcXFyww/CpKmPbXXCIN5YUsjTT6W3g+Eah/LlTJImRvhsP2HarGIutYupqbEOGDFmgqn3LqufvfTaZOKfQADJwmjsvBlKAaD/nEQfkeI3L9pivd91sr3px7nUb77Ij5iMiFwChqjpZRAaXFpCqvgq8CtChQwcdPLjU6kGTlpaGxeYYNlR596eNPPH1SuZtK+K3nAM8cF5nhvVsetTjp227VYzFVjEWW+n8PY32PXCm+3oS8IKIvAn8B+fxA/7IBRK8xiUAvvoo8a6bAOSqcxhW4nzcU3VPAnaTaS0lIlx5Uium3XYKA9ulsmffAUZ/uIiRE+fZ00CNqcb8TTa34CQWcK6DPIVzVDMJuM7PeawGwkSknce4HsAyH3WXuWW+6i0DusuRP2O7u+PbAa2A70VkG/Ap0FhEtolIKz/jNDVA8+QY3h55Ak9e3J2EqDBmrNrJmc/O4r25Gzl0yFqsGVPd+HtTZ5bH60PA2PIuSFXzRORT4GERuQ7oCZwPnOyj+tvA7SLyFU4HoHcAL7plaUARcKuIvMLvfbZNBw4BzT3mczLwL5xWc9YyrZYRES7t25zB7etz32dL+Wb5dv4+eSmf/7qFxy7oFuzwjDEeynOfTUMRGSMi40Uk1R3X32355a+bcK7x7MA5UrpRVZeJyEAR8WzVNgH4AuepoEuBL91xqGohTrPoq4A9wEhgmKoWqupBVd1WPABZwCH3vfVhX0s1SIhiwpV9eGl4b1LjIpi7Pouzn5/FJ6sL7dEFxlQT/t5n0wdYhXNvzLX8fs3kDOAxfxemqlmqOkxVY1W1haq+747/XlXjPOqpqt6lqsnucJd6NJtT1YWq2kdVo1W1t6ouLGF5aSW1RDO1i4jwh+6N+Xb0IC4/oTkHipQv1h3gjGdnMmPljmCHZ0yd5++RzdPA8+69Nfs9xk8D+ld6VMZUUL3YCJ64sDuf3HgSzeNDSM/K55qJ87jhnQXWgMCYIPI32fTBuVPf21ac3gCMqVb6tEzmwZOiuO8PnYiJCOW/y7Zx+jMzeW3WOgoPHgp2eMbUOf4mm3ycrmG8dcS5/mJMtRMaIlw3sA3/u2MQZ3dtxL7CIh77agVnPT+LGavsY2tMVfI32UwBHhCRSPe9uk2JxwKfBCAuYypN48Roxv+5D29eczxtUmNZtzOPa96cxzVv/szanX71tmSMOUbl6RstGaf5cAwwG1iDc+f+fYEJzZjKNaRDA/572ync94dOxEc69+YMfXYWj0xdTnb+gWCHZ0yt5leyUdUcVR2A0+T4b8DzwFmqeoqq2oNGTI0RERbCdQPbMOPOwVx+QnOKVHlj9nqGPJ3Ge3M3crDIrucYEwjleniaqk5X1adV9UlV/U5EWopI9ey90phSpMZF8sSF3fnilgGc0DqZrLxC/j55KWc//z3fLd9uz80xppId65M6k3AeG2BMjdS1aSIfjjqRfw3vRfPkaH7bkct1b8/nsgk/sWDj7mCHZ0ytYY+FNnWeiHBu9yZ8d/sg/nFuZ+rFhPPzhiwuGv8DN7yzwBoRGFMJLNkY44oMC2XkgNbMvGsINw85jqjwEP67bBtnPjuLeycvsSeEGnMMLNkY4yUhKpw7h3YkbcwQ/nR8c1SV9+duYuCTM3h06nJ25e4veybGmCOU2uuziHxexvTez5UxptZolBjFPy/qznUDW/PUtFVMW7ad12ev5725m7j65FZcf0ob6sVGBDtMY2qEsh4xkOlH+fpKisWYaqltg3gmXNmXpRnZPPvtav63cgevzFzLOz9uYOSA1lw3oA2JMeHBDtOYaq3UZKOq11RVIMZUd12bJvLGiOP5NX0Pz367mpmrd/Li9DVM/GED1w5ozTUnt7akY0wJ7JqNMeXUs3kSb408gU9uPIkBbVPZW3CQ5777jf5jp/PE1yvYsdcaEhjjzZKNMRXUp2Uy717Xjw9HncjAdqnk7j/IhJnrGDB2Bvd/tpT0rH3BDtGYasOvx0IbY0rWr00K/dqksCh9Dy+nrWHasu2889NG3v95E+f3bMKNg46jXcP4YIdpTFDZkY0xlaRH8yQmXNmXb0afwoW9mgLw6S8ZnPHsLEa9PZ95G7KsGxxTZ9mRjTGVrH3DeJ65rCejz2jPq7PW8eH8dL5Zvp1vlm+nR7NErh3YhphDlnRM3WJHNsYESPPkGB4Z1pXZfxvCrae2pV5MOIs2Z3PrfxZy16x8Xp211h5tYOoMSzbGBFiD+ChuP7MDP9x9Go9f0I029WPJKlAe/2olJz/xPx76YhmbMq0xgandLNkYU0WiI0IZ3q8F340exOg+kfRvm0JeYRFvztnAoKdnMHLiPGas3EGRnWIztZBdszGmioWECD3qh/F/l5zI8i05vDF7PV8s3sL0lTuYvnIHzZOj+XO/llzStznJ1h2OqSXsyMaYIOrcJIFxl/bgp3tO4+6zO9KsXjTpWfk88fVKTnzif9w+6Vd+Td9jrdhMjWdHNsZUA8mxEdww6Dj+MrANM1fv4J0fN5K2eief/pLBp79k0LVpApcd34I/9mhCYrR1iWNqHks2xlQjoSHCqR0bcmrHhmzMzOP9uZv4cH46SzNyWJqxlEenLuecbo25tG9zTmyTjIgEO2Rj/GLJxphqqmVKLPec04nRZ7Rn2rJtfDgvnR/WZjJ5YQaTF2bQMiWGS/s256LezWiUGBXscI0plSUbY6q5qPBQzu/ZlPN7NmVT5j4+WpDOR/M3szFzH09NW8W4b1YxuEMDLurdjNM6NSAqPDTYIRtzFEs2xtQgLVJiuOPMDtx2entm/baTSfPS+W7F9sMt2eIjwzinW2OG9WpKv9bJhITYaTZTPVRpazQRSRaRySKSJyIbRWR4CfVERMaKSKY7jBWPk9Mi0lNEFojIPvdvT4+yO0VkqYjsFZH1InJnVaybMVUpNEQY0qEB4//ch5/uOY37z+1Mt6aJ7N1/kA/np3P5az/Rf+x0/vn1SlZt2xvscI2p8iObl4BCoCHQE/hSRBap6jKveqOAYUAPQIFvcZ4I+oqIRABTgOeAl4HrgSki0k5VCwEBrgIWA8cB34hIuqp+EPC1MyYIUuIiuXZAa64d0Jo1O/by2cItTF6YQcaefF6ZuZZXZq6lU+MEhvVswh+6N6ZZvZhgh2zqoCo7shGRWOAi4H5VzVXV2cDnwJU+ql8NjFPVzaqaAYwDRrhlg3GS5HOqul9VX8BJMKcCqOqTqvqLqh5U1VU4ial/AFfNmGqjbYN4xgztwPd3DeGjG05ieL8WJEaHs2JrDk98vZIBY2dw/ktzeHXWWjbvti5yTNWRqrpZTER6AXNUNcZj3BhgkKqe51U3GzhTVee67/sCM1Q1XkRGu2Vne9Sf6paP85qPAL8AE1T1FR8xjcI5iqJ+/fp9Jk2aVElrW7lyc3OJi4sLdhg+WWwVU5WxHTikLN5ZxNytB/l1ZxGFRb+XtUkM4fhGYfRtGEr9mJAqj628LLaKCWRsQ4YMWaCqfcuqV5Wn0eKAHK9x2YCvp0rFuWWe9eLc5OFdVtp8HsQ5envTV0Cq+irwKkCHDh108ODBpa5AsKSlpWGxlZ/F9rsz3L/5hUWkrdrB1CVbmb5iB+uyi1iXXciHq6BHs0TO6daYRNnIubbdys1iK11VJptcIMFrXALg6+qld90EIFdVVUT8mo+I3IJz7Wagqu4/lsCNqS2iI0I5u1tjzu7W+KjEs2hzNos2O7/jXl81kzM6N+T0Tg3p1TzJWrWZY1aVyWY1EOZeyP/NHdcD8G4cgDuuB/Czj3rLgDtERPT3c4DdcRofACAiI4G7gVNUdXPlroYxtYOvxPPfZdv4ZukW1uzIZc2OXManrSU1LpLTOzXgjM4N6d821e7jMRVSZclGVfNE5FPgYRG5Dqc12vnAyT6qvw3cLiJf4bRGuwN40S1LA4qAW0XkFeAv7vjpACJyBfA4MERV1wVodYypVTwTz3fT9xDdohvfLt/Ot8u3k7Ennw/mpfPBvHSiw0MZ2C6VUzs2YHCHBtZzgfFbVTd9vgn4N7ADyARuVNVlIjIQ+FpVi69gTQDaAEvc96+741DVQhEZ5o77J7ACGOY2ewZ4FEgB5nncmvOuqt4Q0DUzppYICxH6t02lf9tUHjivMyu27uW7FU7iWZKRffgR1wAdG8UzqEN9BrWvT9+WyUSEWUfyxrcqTTaqmoVz/4z3+O9xLvwXv1fgLnfwNZ+FQJ8SylpXSrDGGESEzk0S6NwkgVtPa8fW7Hy+W7GDmat28sPaXazctpeV2/YyYeY64iLDOPm4FAZ3aMCgDvVpmhQd7PBNNWLd1Rhj/NY4MZorT2zJlSe2ZP/BIuZv2M3M1TtJW7WD1dtzjzjqadsgjgFtUzn5uBT6tUmxRyPUcZZsjDEVEhkWevh0273ndCJjTz4zV+1k5uodzFmTebiRwcQfNhAi0K1pIie7yadvy2SiI6yhQV1iycYYUymaJkUzvF8LhvdrQeHBQ/yavoc5a3bx49pMFqbvPty0enzaWiJCQ+jdMomTj0ulf9sUujVNsus9tZwlG2NMpYsIC+GE1smc0DqZ0WfAvsKD/Lw+ix/WZvLD2l0s25LDT+uy+GldFs98C1HhIfRsnsQJrZI5vnUyvVvUIzbSdk+1if03jTEBFxMRxuAOTnNpgN15hfy0LpMf1mby4zrnlFtx8gGnV+suTRI4vlWyO9QjJS4ymKtgjpElG2NMlasXG3H4vh6AzNz9zN+4m3nrs5i3IYulW3JYvDmbxZuzeWP2egCOqx9L08j9bIvZRK8W9WjbII5Q69mgxrBkY4wJupS4SIZ2acTQLo0AyNt/kIWb9vDzhizmrc9iYfpu1u7MYy0wa7Nz+11sRCg9mifRq0USPZvXo2fzJOrH29FPdWXJxhhT7cRGhjGgXSoD2qUCUHjwEEu3ZPPR9PnsjUhh4aY9ZOzJd68BZR6ernlyND2b16NX8yR6NE+kc+NEa/VWTViyMcZUexFhIfRuUY+cVuEMHtwbgB05BSxM38Ov6XtYuGk3izdnk56VT3pWPl8s2gJAiMBx9ePo1jSRru7QpUmCNT4IAtvixpgaqUFC1BGn3g4WHeK3Hbks3OQknyUZ2fy2I/fw8OnCDABEoE1qLF2bJtKtaSJdmiTSpWkCCVF202kgWbIxxtQKYaEhdGqcQKfGCQzv1wKAggNFrNy2lyUZ2SzLyGZJRjart+91rv/szGPKr1sOT9+sXjQdG8XTsVECHRvH07FRPK1SYgkLtft/KoMlG2NMrRUVHkrP5kn0bJ50eNz+g0Ws3pbL0i3Zh5PQim172bw7n827nb7fikWEhdC+YRwdGibQqbGTiDo0ireGCBVgycYYU6dEhoXSrVki3Zolcrk77mDRITZk5jkdi27dy8ptOazYupeMPfkszchhacaRDxlOiY2gbYO4w0O7BvHsLjiEquLR27zxYMnGGFPnhYWG0LZBPG0bxHNu99/H5xQcYPW2vazYtpeVW3NY5fZynZlXSOb6LOauzzpiPv/48RvaNIijbf042jV0/rZtEEfz5Jg6f0+QJRtjjClBQlQ4fVsl07dV8uFxqkrGnvzDHY2u3en8XZGxm737D7IofQ+L0vccMZ+IsBBap8TSMiWG1qmxtEqNpVVKLK1SY2gYH1UnHrttycYYY8pBRGhWL4Zm9WIOd78DkJaWRre+J/Gbm4Q8E9HW7AJWbd/Lqu17j5pfVHgIrdxE1Co1ltYpvyejhgmRtea0nCUbY4ypJClxkaTERXJim5Qjxu8tOMCGXfvYkJnHhl15rHf/bszcR2Ze4eGH0HmLDg+lWb1omtWLpnlyDM3rxRzxOjGm5jTXtmRjjDEBFh8VfrhRgrfs/ANszMxj/a48Nuza57x2k9HufQcO3yfke75hNKsXQ3M3ATWrF03zejE0T46hcVJUtbp3yJKNMcYEUWJ0ON2bJdG9WdJRZdn5B9i8ex/pWfls3r2PzbvzSc/aR7o7bm/BQVZszWHF1hwfc4a4yDAaJ0YRWVTA17sW0zgpiiaJ0TROiqJxYjRNkqKIiaiaNGDJxhhjqqnE6HASo51eDrypKll5haTvzj+ckNLdhLQ5ax9bsvPJ3X/w8FHR0sx0n8tIiAqjSVI0jROjaJwUTZPEKBokRNEwIYqGCZE0iI+iXkz4MV87smRjjDE1kIgcvkbkedNqMVUlO/8AW/YU8M3sn0lp0Y6te/LZll3Alux8tmYXsDW7gJyCg+SUcM2oWERoCPXjI2mQEEnD+Cjnb0IUDcpxc6slG2OMqYVEhKSYCJJiItjRIIzBJ7Y8qo6qkplX6CSgPU4C2pKdz86c/WzfW8COnP1sz3ESUsaefDL25Fc4Hks2xhhTR4kIqXGRpMZF0rXp0afqihUcKHISz94CtucUHH69I2c/z/m5LEs2xhhjShUVHkqLlBhapMQcVfbcn/ybh3VnaowxJuAs2RhjjAk4SzbGGGMCzpKNMcaYgLNkY4wxJuAs2RhjjAm4Kk02IpIsIpNFJE9ENorI8BLqiYiMFZFMdxgrHn0liEhPEVkgIvvcvz39ndYYY0zVq+ojm5eAQqAhcAUwXkS6+Kg3ChgG9AC6A+cB1wOISAQwBXgXqAe8BUxxx5c6rTHGmOCosmQjIrHARcD9qpqrqrOBz4ErfVS/GhinqptVNQMYB4xwywbj3Iz6nKruV9UXAAFO9WNaY4wxQVCVPQi0Bw6q6mqPcYuAQT7qdnHLPOt18ShbrKrqUb7YHf/fMqY9goiMwjkSAtgvIkv9W5UqlwrsCnYQJbDYKsZiqxiLrWICGdvRna75UJXJJg7wfuhCNhBfQt1sr3px7rUX7zLv+ZQ4rVeCQlVfBV4FEJH5qtrX/9WpOhZbxVhsFWOxVYzFVrqqvGaTCyR4jUsAfPVr7V03Ach1k0VZ8yltWmOMMUFQlclmNRAmIu08xvUAlvmou8wt81VvGdDdq4VZd6/ykqY1xhgTBFWWbFQ1D/gUeFhEYkWkP3A+8I6P6m8Dt4tIUxFpAtwBTHTL0oAi4FYRiRSRW9zx0/2YtjSvln+tqozFVjEWW8VYbBVjsZVCqvLskogkA/8GzgAygbtV9X0RGQh8rapxbj0BxgLXuZO+Dvyt+FSYiPRyx3UGVgDXqupCf6Y1xhhT9ao02RhjjKmbrLsaY4wxAWfJxhhjTMDV+WTjb39tlbi8NBEpEJFcd1jlUTbcjSFPRD5zr3H5FWdp05YSyy0iMl9E9ovIRK+y00Rkpdv/3AwRaelRFiki/xaRHBHZJiK3V9a0ZcUmIq1ERD22X66I3F9Vsbl13nC39V4R+VVEzq4O26202IK93dx674rIVrfeahG5rjLmH8jYqsN286jfTpx9x7se4wKyzyhr2gpR1To9AP8BPsS5GXQAzk2gXQK4vDTgOh/ju+DcK3SKG8v7wAf+xFnWtKXEciFOP3LjgYke41Pd+V8CRAFPAT95lD8BfI/TN10nYBtw1rFO62dsrQAFwkpYp4DGBsQCD7pxhADnutu+VbC3WxmxBXW7eXxOI93XHd16fYK93cqILejbzaP+N279dwO9zyht2grv+wK1U60JA86XsxBo7zHuHeCfAVxmGr6TzePA+x7vj3Njiy8rztKm9TOmRzlyhz4K+MFrO+UDHd33W4AzPcofKf6gHsu0fsZW1pe/ymLzqLcYp9+/arPdfMRWrbYb0AHYClxa3babV2zVYrsBfwIm4fyYKE42AdlnlDVtRYe6fhqtpP7afPalVomeEJFdIjJHRAa7447o001V1+L+w/2Is7RpK8J7fnnAWqCLiNQDGlN633UVnbY8NorIZhF5U0RSAYIRm4g0xNnOy45x/oGOrVhQt5uIvCwi+4CVODv0r45x/oGOrVjQtpuIJAAPA96n2QK1zwjIfrGuJ5vy9NdWWf4GtAGa4txo9YWIHEfpfb6VFWdZ/cWVV1mxwNH9z/kTS1nT+mMXcDxO53993Gnf81h2lcUmIuHust9S1ZXHOP9Ax1Yttpuq3uSWDcS5yXv/Mc4/0LFVh+32CPCGqm72Gh+ofUZA9ot1PdmUp7+2SqGqc1V1rzqPR3gLmAOcU0Ys5e0Pzru8vMqKBY7uf86fWMqatkzqPJ5ivqoeVNXtwC3AmSISX5WxiUgIzqmFQjeGY51/QGOrLtvNjaVInUeMNANuPMb5BzS2YG83cR4MeTrwrI9wA7XPCMh+sa4nm/L01xYoivM8niP6dBORNkCkG2NZcZY2bUV4zy8W55zuMlXdjXOKobS+6yo6bUUU35UcUlWxiYgAb+A8BPAiVT1QCfMPdGzeqny7+RBWPJ9jmH+gY/NW1dttMM51o00isg0YA1wkIr/4mH9l7TMCs188lgs+tWEAPsBpeREL9CeArdGAJGAoTsuUMJynlebhnCPtgnPoOtCN5V2ObB1SYpxlTVtKPGFuLE/g/BIujqu+O/+L3HFjObIVzT+BmTitaDrifGmKW+BUeFo/Y+uHcwE3BEjBaTEzo4pjewX4CYjzGl8dtltJsQV1uwENcC5yxwGhON+DPOCPwd5uZcQW7O0WAzTyGJ4GPnbnHbB9RmnTVnj/F4idak0agGTgM/fDtQkYHsBl1Qfm4RyO7sHZKZzhUT7cjSEP59HXyf7GWdq0pcTzIM4vNc/hQbfsdJwLpfk4LehaeUwXidPHXQ6wHbjda74Vnras2IDLgfXuem7F6Xi1UVXFhnPuXoECnNMNxcMVwd5upcVWDbZbfZwd6x633hLgL5Ux/0DGFuztVsL34t1A7zPKmrYig/WNZowxJuDq+jUbY4wxVcCSjTHGmICzZGOMMSbgLNkYY4wJOEs2xhhjAs6SjTHGmICzZGNMLSMiI0Qkt+yaxlQdSzbGBIiITHQfvFU87BKRqSLSsRzzeFBElgYyTmOqgiUbYwLrO5yu5BsDZwLRwOSgRmRMEFiyMSaw9qvqNnf4Baf33o4iEg0gIv8UkVUiki8iG0TkSRGJcstGAA/gPP+k+OhohFuWKCLjxXmUcYGIrBCRyzwXLM4jiZe6j/adISKtq3LFjfEUFuwAjKkr3G7pLwOWqGq+OzoPGAlkAJ1xOtLcD9yP0+ljV5zHOw9262e7PTt/hdOB4zU4vfR2wOnssVgkcI877wLgLXfeQwOzdsaUzpKNMYF1lsfF+lggHef5RQCo6iMedTeIyOM43cjfr6r57rQHVXVbcSUROQM4CacX3hXu6HVeyw0DblbVVe40TwP/FhFR6xDRBIGdRjMmsGYBPd3hBOB/wDci0hxARC4Wkdkiss1NLM8CLcqYZy9gq0ei8WV/caJxbQEicI6GjKlylmyMCax9qrrGHeYB1+E89XCUiJyI89yQacB5OEnkPiC8EpZ78P/bu1uViKIoDMPvSgY12BTDFItVMFk1eAGGwavwBrwBi17IgDBZsIiYBIvVIgajzTDLsESGU0RxyYT3SefA3gdO+tg/8A3ev0q//uDb0o+5jSb9rwRmVCnWHvA8v5UWEaPB+Heq0GvePbAREdvfrG6khWHYSL2WImL983mN6rBfAabAKrAZEcfALXV4Px7MfwJGEbFDlVi9UVtxd8AkIk6oCwJbwHJmXsVYJPQAAABsSURBVPb+jvQ7LqmlXvtUw+MLFRC7wFFmXmfmFDgDzoEH4AA4HcyfUDfProBXYJyZM+AQuKHqfB+BC+pMRlpINnVKktq5spEktTNsJEntDBtJUjvDRpLUzrCRJLUzbCRJ7QwbSVI7w0aS1O4D96wB2VjPCZQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(steps, lrs, \"-\", linewidth=2)\n",
    "plt.axis([0, n_steps - 1, 0, lr0 * 1.1])\n",
    "plt.xlabel(\"Batch\")\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.title(\"Exponential Scheduling (per batch)\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Piecewise Constant Scheduling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "def piecewise_constant_fn(epoch):\n",
    "    if epoch < 5:\n",
    "        return 0.01\n",
    "    elif epoch < 15:\n",
    "        return 0.005\n",
    "    else:\n",
    "        return 0.001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "def piecewise_constant(boundaries, values):\n",
    "    boundaries = np.array([0] + boundaries)\n",
    "    values = np.array(values)\n",
    "    def piecewise_constant_fn(epoch):\n",
    "        return values[np.argmax(boundaries > epoch) - 1]\n",
    "    return piecewise_constant_fn\n",
    "\n",
    "piecewise_constant_fn = piecewise_constant([5, 15], [0.01, 0.005, 0.001])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/25\n",
      "55000/55000 [==============================] - 6s 111us/sample - loss: 0.8151 - accuracy: 0.7655 - val_loss: 0.6868 - val_accuracy: 0.7780\n",
      "Epoch 2/25\n",
      "55000/55000 [==============================] - 6s 102us/sample - loss: 0.8153 - accuracy: 0.7659 - val_loss: 1.0604 - val_accuracy: 0.7148\n",
      "Epoch 3/25\n",
      "55000/55000 [==============================] - 6s 104us/sample - loss: 0.9138 - accuracy: 0.7218 - val_loss: 1.3223 - val_accuracy: 0.6660\n",
      "Epoch 4/25\n",
      "55000/55000 [==============================] - 6s 103us/sample - loss: 0.8506 - accuracy: 0.7627 - val_loss: 0.6807 - val_accuracy: 0.8174\n",
      "Epoch 5/25\n",
      "55000/55000 [==============================] - 6s 101us/sample - loss: 0.7213 - accuracy: 0.8068 - val_loss: 1.0441 - val_accuracy: 0.8030\n",
      "Epoch 6/25\n",
      "55000/55000 [==============================] - 6s 101us/sample - loss: 0.4882 - accuracy: 0.8548 - val_loss: 0.5411 - val_accuracy: 0.8494\n",
      "Epoch 7/25\n",
      "55000/55000 [==============================] - 6s 101us/sample - loss: 0.4721 - accuracy: 0.8568 - val_loss: 0.5808 - val_accuracy: 0.8448\n",
      "Epoch 8/25\n",
      "55000/55000 [==============================] - 6s 101us/sample - loss: 0.4412 - accuracy: 0.8659 - val_loss: 0.5466 - val_accuracy: 0.8526\n",
      "Epoch 9/25\n",
      "55000/55000 [==============================] - 6s 100us/sample - loss: 0.4234 - accuracy: 0.8718 - val_loss: 0.5611 - val_accuracy: 0.8528\n",
      "Epoch 10/25\n",
      "55000/55000 [==============================] - 5s 99us/sample - loss: 0.4300 - accuracy: 0.8721 - val_loss: 0.5049 - val_accuracy: 0.8650\n",
      "Epoch 11/25\n",
      "55000/55000 [==============================] - 5s 100us/sample - loss: 0.4162 - accuracy: 0.8768 - val_loss: 0.5957 - val_accuracy: 0.8534\n",
      "Epoch 12/25\n",
      "55000/55000 [==============================] - 6s 101us/sample - loss: 0.4122 - accuracy: 0.8780 - val_loss: 0.5707 - val_accuracy: 0.8640\n",
      "Epoch 13/25\n",
      "55000/55000 [==============================] - 6s 101us/sample - loss: 0.3951 - accuracy: 0.8833 - val_loss: 0.5523 - val_accuracy: 0.8690\n",
      "Epoch 14/25\n",
      "55000/55000 [==============================] - 5s 100us/sample - loss: 0.3961 - accuracy: 0.8834 - val_loss: 0.7371 - val_accuracy: 0.8452\n",
      "Epoch 15/25\n",
      "55000/55000 [==============================] - 5s 100us/sample - loss: 0.4201 - accuracy: 0.8839 - val_loss: 0.6546 - val_accuracy: 0.8558\n",
      "Epoch 16/25\n",
      "55000/55000 [==============================] - 6s 100us/sample - loss: 0.2645 - accuracy: 0.9162 - val_loss: 0.4655 - val_accuracy: 0.8844\n",
      "Epoch 17/25\n",
      "55000/55000 [==============================] - 6s 100us/sample - loss: 0.2440 - accuracy: 0.9222 - val_loss: 0.4758 - val_accuracy: 0.8830\n",
      "Epoch 18/25\n",
      "55000/55000 [==============================] - 6s 100us/sample - loss: 0.2320 - accuracy: 0.9256 - val_loss: 0.4917 - val_accuracy: 0.8880\n",
      "Epoch 19/25\n",
      "55000/55000 [==============================] - 6s 100us/sample - loss: 0.2248 - accuracy: 0.9279 - val_loss: 0.4644 - val_accuracy: 0.8878\n",
      "Epoch 20/25\n",
      "55000/55000 [==============================] - 6s 100us/sample - loss: 0.2172 - accuracy: 0.9302 - val_loss: 0.5036 - val_accuracy: 0.8848\n",
      "Epoch 21/25\n",
      "55000/55000 [==============================] - 6s 100us/sample - loss: 0.2139 - accuracy: 0.9327 - val_loss: 0.4921 - val_accuracy: 0.8914\n",
      "Epoch 22/25\n",
      "55000/55000 [==============================] - 6s 101us/sample - loss: 0.2030 - accuracy: 0.9360 - val_loss: 0.5197 - val_accuracy: 0.8860\n",
      "Epoch 23/25\n",
      "55000/55000 [==============================] - 5s 100us/sample - loss: 0.2014 - accuracy: 0.9360 - val_loss: 0.5231 - val_accuracy: 0.8892\n",
      "Epoch 24/25\n",
      "55000/55000 [==============================] - 5s 100us/sample - loss: 0.1912 - accuracy: 0.9391 - val_loss: 0.5223 - val_accuracy: 0.8876\n",
      "Epoch 25/25\n",
      "55000/55000 [==============================] - 5s 99us/sample - loss: 0.1872 - accuracy: 0.9418 - val_loss: 0.5068 - val_accuracy: 0.8886\n"
     ]
    }
   ],
   "source": [
    "lr_scheduler = keras.callbacks.LearningRateScheduler(piecewise_constant_fn)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 25\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[lr_scheduler])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEeCAYAAABc5biTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3X2cnFV9///XO7dsNgmBTUggaBYEFoxCEG+qkRJBpNryI4JfW0GBUoWvli9abiy0ooi1GIVWqBRJrQbEWq0GAmKhxbhVFBW5DRGSyk0kCTdJMCGb+5vP749zNkwmM7vXJjszyc77+XjMIzPXOeeaM2cn85lzrjPnKCIwMzOrpUGNroCZmQ18DjZmZlZzDjZmZlZzDjZmZlZzDjZmZlZzDjZmZlZzDja2A0ntkkLSGxtdl2okdUr6SqPrYcVIOltSV43O/ZikK/pY5hlJF1d7bP3PwaYJSZqVg0lI2iTpKUlXS2rNWZ4F9gcebmA1e3MqcFktn0DJhyXdJ2m1pJclPSjpk5JG1/K5y+pRsw/CoueWdJCkWyQtlrRB0lJJd0o6uhb1aoA3Af/c6EoMZEMaXQFrmHuADwFDgWOBrwGtwEcjYgvwfAPr1quIeKkOT/NN4DTg74GPAy8Ck4Hz8/1ZdahDw0kaCvw38CTwfmAJcADwLmDfBlat30TEskbXYcCLCN+a7Eb6kPxB2bF/AZ7L99uBAN5Ykv5a4E5gNemD9tvAhLJznAXMAzYALwA3laTtDczMZVcD/1N2/ueAPyt5fG/ONyQ/PiTX6cD8uBP4Skn+U4FHgXXAS/n840vSTwYeANYDTwOfB4b10Ebvz893apX0MfnfQcDlpN7ghvz6TynJ192Wp5E+sNcCvwFOLMkzFLgOWJrP8SzwhZLXGaW3fLwt/w0W59c8H/jzsjp2kr6t/z2wPLf91cCgns5d4bVOyemH9PK+2hu4If8t1wOPA3+a084GuoATgMeANcCPgYPKztHj3wnYD5iTX/Mi4Jx8vitK8gTwvrLzPgNc3IfHAZwL/Eeu61PAB8vO+RbgwVzXh4D35HLTGv1/fHe8eRjNuq0jfejtQNL+wE9I/6nfDLwTGAnMkTQo5zkPuBH4BnAk6T/eYzlNpEA1EfgT4Oh8vrn53JCCw7ScfwRpWGMD0H3daBrwZEQsrlC/CcC/AzcBRwB/SOqVdKefBHwL+AqpZ3IO8D7Sh3A1ZwALI2J2pcSIWJnvfhy4BPhr4PXArcBsSVPKinyeFFCOAu4H/l3SyJx2AfBe4M+AQ4E/BRbktFNJAeVK0tBmd3vtRfqg+5P8mq4FbpR0QoXXsRl4G6lH9ol8/p7OXW4ZsBU4TVLF0ZD8N/4hcBzw56QvJxcCG0uyDScNfZ4DvBUYA3y15BxF/k6zSF883glMB84kBfRa+DQpsB0FfAf4uqRX57qOBH4APAEcA3wS+FKN6jEwNDra+Vb/G2U9G1IAWQ58Jz9up6RnQ/ow+lHZOfbJed6cHy8mfxuv8HzHk77VtpQdfxj4ZL7/f4EF+f47Sd+KZwGX5WO3AF8rKdtJ7tkAb8h1mVTl+X8CXF52bHquk6qU+Q0wp0BbLgE+XXasE7ilrC3PK0mfmI+9PT++DvhRD3V5hpJv3T3U5d8rtNF9ZXn+uyxP0XP/Jekbfhfpi8HngMkl6SeSAtIRVcqfnV9zR8mxM0hfKFTk7wQcls8xtSR9ErCF2vRsrip5PITUK/1gfnweqQfdUpLndNyzqXpzz6Z5/ZGkLknrgftI/9H/X5W8xwB/mPN35VlFz+a010jaj/QB+qMeyo8AlpWd43XAa3KeTuCw3NOZRhpi6cz3IX1j7qxy/kdI16Aek/R9SR+VNK7s+f+27Ln/jXSNakKVc6rK8VcypEkCBwA/K0u6l/TNvtSjJfeX5n/3y//OIg1VLZR0vaQ/7u4x9vDcgyX9raRHJa3Ir+lU4NU9PG/3c+9HH0XE9aS2Op30+k4BHpb0oZzlaNIw7OM9nGZDRCwoebwUGEb64gK9/52OIAW0X5XUaxGvtGd/29Z2EbGZ1MPrbrvDgcciYl1J/l/WqB4DgicINK+fkMakNwFLI2JTD3kHkYbBKs1aegFo6eW5BuV8x1ZIexkgIp6Q9DzwDlKAuZY03PQVSUcAB1Il2ETEFknvAv6AdNH6L4CrJB0XEY/k5/8safy9XLULwwtJH247q3w59W3tGxGRRp3Sl72IeFBSO3AS6ZrGTcAjkk6MiK1Vzn8xcBFpGG8e6dv/37NjICn/uwY7OQs1IlYDtwO3S/oUcDeph/PNHgu+YnOFulBSn6J/p96Wqg92/LJQcYi4F/3WduZg08zWRsRvC+Z9kHTBfFGVoLRa0hLSB+V/Vyk/HtgaEU/18Dz/A/wx6TpNZ0Qsk7ScNB5e8XpNt0jjGPcB90m6knTB/E9JvZ4HgcP78HohfaP+d0mnRoXrNpLGRMRKSUuBqWzfq3s7aRiusPxB/j3ge5JmAb8gXZtYSLruMbisyNuBOyLim7k+3cNMK+mbSucuUt+Q9ARpCBPSBfL9JR3RS++mJz3+nfLzDSIN+/48H3s1qXdZahkl158kjaf69aid9QRwlqSWkt7Nm/v5OQYUR2kr4nrSTKPvSHqLpIMlvVPSTEmjcp7PA5+Q9FeSDpM0RdJFOe0e0lDTHEnvzr/ZeKukz0oq7e10koLab+OVqaidwAepPoSGpD+Q9ClJb8ofPv8f8Cpe+cC/Ejhd0pWSXifpcEnvk/TFHl7zd0kXhb8l6fJ87kmS/kjSnaRrCZAuCl8s6QP5dV9J6sFd3cO5y+t/YS5/hKRDSENVL5Oug0G6nnCspImSxuZjC4ETJL1d0uGki+oHFX3OEpXOXV6/KZLm5DZ7raRDJP0F6QL+rTnbj0jDSN+XdFL+G58oaXqlc1bR498pD8HdRZoI8dY8CWMWaXJLqbnAX0p6o9LvgGaRZoz1p38jXSv6l9wm7wT+Jqd5k7AKHGysVxHR/e19K+k/+3xSANqQb0TEDaSLyB8hzUK7izSjqLvX8R7Sh8C/kGZafRfoYPvx9k5Sb7uzl2PlVuX6/QD4X+Aa4HMRcUt+/rtJPaZ3kMb7fwVcCvyuh9ccwAdIw1R/QrqGNA+4itQD+37Oeh0p4Hwxv+73Aqfl4buiVpNmtP2K9O1+CvDuiFib0z9NCp5P8spw0t/l/P9JGhJdQ5rJ1VeVzl1uMWnq76dJPa6HSUN4V5Ov8+XhvneTvlTcQprgcS3pmkwhBf9OZ5OmRM8F7iB96D9TdqqLcn07Sb3Fr5Gmffeb3BM9mfQef4j0HrgiJ/d3YBsQumeBmJnZLpB0Cqmnt19ELG90fXY3vmZjZrYTJJ1F6kE9S5pZ+WXSdTQHmgocbMzMds540uy5/UnLO91J+nGvVeBhNDMzqzlPEDAzs5rzMFo2ZsyYOOSQQxpdjd3OmjVraG1t7T1jk3G77MhtUtlAb5cHHnhgeUSM6y2fg002fvx4fv3rXze6Grudzs5Opk2b1uhq7HbcLjtym1Q20NtF0qIi+TyMZmZmNedgY2ZmNedgY2ZmNedgY2ZmNedgY2ZmNedgY2ZmNedgY2ZmNedgY2ZmNedgY2ZmNedgY2ZmNedgY2ZmNedgY2ZmNedgY2ZmNedgY2ZmNedgY2ZmNVfXYCNpX0m3SlojaZGk06vkk6QZklbk2wxJKkmfKWmBpK2Szq5Q/q8kPS/pZUlflzS8t7o98/JWpn5hLrc9tKTQa7ntoSVM/cJcDrr0zgFZzsysP9W7Z3M9sBEYD5wB3CBpcoV85wLTgaOAI4GTgfNK0h8BPgY8WF5Q0knApcAJwCTgYOCzRSq3ZOU6Lps9r9cP5NseWsJls+exZOU6YgCWMzPrb3XbqVNSK3Aa8LqI6ALulXQ78CFScCh1FnBNRCzOZa8BPgJ8FSAirs/H11d4qrOAf42I+TnP54BvVXiOitZt2sLf3DqPe3+7vGqeH857jnWbtuyx5b509wKmHz2xajkzs/5Wz22hDwM2R8TCkmOPAMdVyDs5p5Xmq9QDqmQyMKes7HhJbRGxojSjpHNJvSiGTThk2/G1G7fw4/nVv/2v3RhVju8Z5ZasXEdnZ2fVcqW6uroK520mbpcduU0qc7sk9Qw2I4GXy46tAkZVybuqLN9ISYqIyp+gPZclP892wSYiZgIzAYbvf+i2804c08LPLj2+6hNM/cJclqxct8PxPalc0T3RB/r+6TvL7bIjt0llbpekntdsuoDRZcdGA6sL5B0NdBUINNXKUuV5dtAydDCXnNTRY55LTuqgZejgAVvOzKy/1TPYLASGSDq05NhRwPwKeefntN7yVVKp7AvlQ2iVTBzTwlWnvr7X6xnTj57IVae+noljWtAeUK6tdRgAY0cOK1TOzKy/1W0YLSLWSJoNXCnpw8AU4BTgbRWy3wxcKOmHQAAXAf/UnShpGClQChgqaS9gY0RszWVnSfoWsBT4FDCrt/q1jx7U45BUuelHT9ypD+1GlDvqVWN4x9WdXPruIxxozKwh6j31+WNAC/Ai8G3goxExX9KxkrpK8t0I3AHMAx4D7szHuv0XsI4UqGbm+38IEBF3AV8Efgz8DlgEfKaGr2m3d+A+LQweJBatWNPoqphZk6rnBAEi4iXS72fKj/+UdGG/+3EAn8y3SueZ1svz/APwD7tS14Fk6OBBHLhPC08vd7Axs8bwcjVNYlJbK4tWrG10NcysSTnYNIn2thE8s2INxSb0mZn1LwebJjGprZXV6zfz+7WbGl0VM2tCDjZNor1tBADPeJKAmTWAg02TmNTWCuAZaWbWEA42TeJV+7YgwTPLPUnAzOrPwaZJDB8ymAP2bvEwmpk1hINNEzlobCvPePqzmTWAg00TmdQ2wtdszKwhHGyaSHtbKyvXbmLl2o2NroqZNRkHmyYyKU9/9koCZlZvDjZNpH1smv7sSQJmVm8ONk3k1fu6Z2NmjeFg00T2GjqY/ffeyz0bM6s7B5sm097WyjPeasDM6szBpsm0jx3hYTQzqzsHmyYzqa2VFWs28vJ6r/5sZvXjYNNkuld//p17N2ZWRw42TaZ79WdPEjCzenKwaTL+YaeZNYKDTZMZMWwI+40a7hlpZlZXDjZNqL2t1cNoZlZXDjZNqH3sCG81YGZ15WDThCa1tbJs9QbWbNjc6KqYWZNwsGlC7XlGmicJmFm9ONg0oVdmpPm6jZnVh4NNE+oONr5uY2b14mDThEbtNZSxI4e5Z2NmdeNg06QmefqzmdWRg02TSlsNeBjNzOqjrsFG0r6SbpW0RtIiSadXySdJMyStyLcZklSSPkXSA5LW5n+nlKQNl/RVSS9IeknSHZIm1uP17Una20bw/MvrWbdxS6OrYmZNoN49m+uBjcB44AzgBkmTK+Q7F5gOHAUcCZwMnAcgaRgwB7gF2Ae4CZiTjwN8HHhrLncA8Hvgn2r0evZYk8am6c+/e8m9GzOrvboFG0mtwGnA5RHRFRH3ArcDH6qQ/SzgmohYHBFLgGuAs3PaNGAI8OWI2BAR1wECjs/pBwF3R8QLEbEe+A5QKaA1tfZtM9J83cbMam9IHZ/rMGBzRCwsOfYIcFyFvJNzWmm+ySVpj0ZElKQ/mo/fBfwrcK2kA4CVpB7Uf1aqkKRzSb0oxo0bR2dnZx9f0p5rzabUfHN/NY/hy56omq+rq6up2qUot8uO3CaVuV2SegabkcDLZcdWAaOq5F1Vlm9kvm5TnlZ+nv8FngWWAFuAecD5lSoUETOBmQAdHR0xbdq0gi9lYPjUff/F4DETmDbt9VXzdHZ20mztUoTbZUduk8rcLknhYTRJ4yVdLOkGSWPzsamSDip4ii5gdNmx0cDqAnlHA125N9Pbea4HhgNtQCswmyo9m2Y3qa3Vv7Uxs7ooFGwkHQMsIA1J/QWvfNifCHy+4HMtBIZIOrTk2FHA/Ap55+e0SvnmA0eWzk4jTQboTp8CzIqIlyJiA2lywJu7A6S9or1thKc/m1ldFO3ZXA1cGxFHAxtKjt8NTC1ygohYQ+plXCmpVdJU4BTgmxWy3wxcKGlivvZyETArp3WShscuyNOcu4fI5uZ/7wfOlLS3pKHAx4ClEbG82EttHu1jW1m6ah3rN3n6s5nVVtFgcwxpinG550jTmIv6GNACvAh8G/hoRMyXdKykrpJ8NwJ3kK63PAbcmY8RERtJ06LPJE0AOAeYno8DXAysJ127WQa8B3hvH+rYNNrbWomAxb9378bMaqvoBIF1pN+0lDucFDgKiYiXSIGi/PhPSRf+ux8H8Ml8q3Seh0gBsFLaCtJwn/Vi24Kcy9dyyH6V5mmYmfWPoj2bOcBnJA3Pj0NSOzAD+H4N6mV10L2vjX9rY2a1VjTYXAzsSxqWGgHcC/yWNIz1qdpUzWptzIihjN5riDdRM7OaKzSMFhEvA2+XdDzwBlKQejAi7qll5ay2JNE+1qs/m1ntFQo2ks4EvhMRc3ll1lf3OmV/FhE316h+VmOT2lp55NmVja6GmQ1wRYfRvgHsXeH4qJxme6iD2kaw+Pdr2bh5a6OrYmYDWNFgIyAqHH81Oy4dY3uQSW2tbPX0ZzOrsR6H0STNIwWZAP5H0uaS5MHAJOCHtaue1Vr72DT9edGKtRw8bmQvuc3Mdk5v12y+l/99HemHlaU/vNwIPIOnPu/RJnn6s5nVQY/BJiI+CyDpGdIEgfX1qJTVT1vrMEYO9/RnM6utolOfKy1VYwOAJCa1jXDPxsxqquiqz8MkfVbSQknrJW0pvdW6klZb7W2t7tmYWU0VnY32OfJWzcBW4BLSvjErSItr2h5sUtsInn1pLZu3ePqzmdVG0WDzfuD/RsSNpOX950TEBcBnSHva2B6sfWwrm7cGS1aua3RVzGyAKhpsxgO/yfe7gDH5/l3Au/q7UlZfryzI6aE0M6uNosHmd8AB+f5vgZPy/beSth+wPVh7W/dvbTxJwMxqo2iwuRU4Id+/FvispKdJu2d+rQb1sjoaN2o4LUMHe4toM6uZolOfLyu5/z1Jz5K2g14YET+oVeWsPrqnP7tnY2a1UnSnzu1ExC+BXwJIao0If0rt4drbWvnfF1c3uhpmNkAVHUbbgaS9JF0CPN2P9bEGmTR2BM++tI4tWyutt2pmtmt6DDb5x5yfl3S/pJ9Lmp6Pnwk8BXwC+Mc61NNq7KC2VjZu2cpST382sxrorWdzBXA+sAg4CPgPSf8M/C1wGdAeEVfVtIZWF90LcnolATOrhd6CzfuBsyPifcAfkbYV2AeYHBE3RcSmWlfQ6qN7qwGvkWZmtdBbsHkVcD9ARDxC2lZgRkRs7rGU7XHGj9qL4UMGeUaamdVEb8FmKLCh5PEmvDPngDRoUPfqzx5GM7P+V2Tq81WSuj+BhgFXSNou4OR10mwPN6mt1T0bM6uJ3oLNT4DXlDz+OfDqsjyeKztAtLeN4CcLl7F1azBokBpdHTMbQHrbqXNanephu4FJba1s2LyV519ezwFjWhpdHTMbQHb6R5028Bw0tnv1Zw+lmVn/crCxbSZtW/3ZkwTMrH/VNdhI2lfSrZLWSFok6fQq+SRphqQV+TZDkkrSp0h6QNLa/O+UsvJvkPQTSV2SXpD08Vq/toFg/71bGDZ4kHs2Ztbv6t2zuZ70W53xwBnADZImV8h3LjAdOAo4EjgZOA/SEjrAHOAW0g9MbwLm5ONIGkva1O1GoA04BPiv2r2kgWPwIPGqfVtY5K0GzKyf1S3YSGoFTgMuj4iuiLgXuB34UIXsZwHXRMTiiFgCXAOcndOmkSY2fDkiNkTEdYCA43P6hcDdEfGtnL46Ih6v2QsbYNrbWt2zMbN+V2iLAUnl0527BbA+IpYVOM1hwOaIWFhy7BHguAp5J+e00nyTS9IejYjSKdeP5uN3AX8AzJP0c1Kv5pfAX0bE78qfRNK5pF4U48aNo7Ozs8DLGNgGr9vAU8s28+Mf/xhJdHV1uV0qcLvsyG1SmdslKbqfzTP08HsaSS8D3wA+2cNSNiOBl8uOrQJGVcm7qizfyHzdpjyt/DwHAm8ATgTmAV8Evk3a7G07ETETmAnQ0dER06ZNq1L15vG74c/wX4vmM/mYt7Lf6L3o7OzE7bIjt8uO3CaVuV2SosHmA6QP7a+SN00D3kLqFVwBjAE+BawGPlPlHF3A6LJjo3OZ3vKOBroiIiT1dp51wK0RcT+ApM8CyyXtHRFeaqcX7Xn156eXr2G/0Xs1uDZmNlAUvWbzUeCvIuKqiJibb1cBFwHnRMS1wAWkoFTNQmCIpENLjh0FzK+Qd35Oq5RvPnBk6ew00iSC7vRH2b4X5hUO+qDdWw2YWQ0UDTZvIQ1JlXsMeFO+fx9pCKuivHX0bOBKSa2SpgKnAN+skP1m4EJJEyUdQApqs3JaJ7AFuEDScEnn5+Nz87/fAN6bp0cPBS4H7nWvppgDxuzFkEHyJAEz61dFg80i8oX0Mh8Bui+8jwNe6uU8HwNagBdJ11E+GhHzJR2bh8e63QjcQQpwjwF35mNExEbStOgzgZXAOcD0fJyImAv8TS7zImmSQMXf89iOhgwexKv2HeGejZn1q6LXbC4Cvi/pPeT9bYA3khbpPC0/fhPw3Z5OEhEvkQJF+fGfki78dz8O4JP5Vuk8DwHH9PA8NwA39FQXqy5tNeCejZn1n0LBJiLuzNdaPgZ05MO3A1/tnlIcEf9cmypavbW3tfLrZ37P9rPLzcx2XtGeDRHxLHBZDetiu4lJbSPo2rCZFWs2NroqZjZAFA42kkYAU4D9KLvWExGz+7le1kDdM9KeWe6hNDPrH0VXEHgn6YJ+W4XkAAb3Z6Wssdq3bTWwlrENrouZDQxFZ6NdS5rddWBEDCq7OdAMMBPHtDB4kLxFtJn1m6LBph34XEQsrWFdbDcxbMggJo5p4RlPfzazflI02PyMV2ahWROY1DbCPRsz6zdFJwh8Fbg6/5p/HrCpNDEiHuzvilljtbe1ctvDS4gY1uiqmNkAUDTYfC//O7NCmicIDECT2kawev1m1mxysDGzXVc02BxU01rYbqd7+vMLa7c2uCZmNhAUXUFgUa0rYruXJ5elpeo+94v1fP2JuVxyUgfTj57Ya7nbHlrCl+5ewNKV6zhgTMuALbdk5Tom/sLtYlZU1WAj6VTgjojYlO9X5R91Diy3PbSEf7znlQ1Vl6xcx2Wz06LfPX3w3PbQEi6bPY91m7a4XBOVMytC1da/krQVmBARL+b71cRA+K1NR0dHLFiwoNHV2C1M/cJclqxct8PxoYPFaw/Yu2q53yxdxaYtO76fXG7PLjdxTAs/u/T4quW6eUfKygZ6u0h6ICLe2Fu+qj2biBhU6b4NfEsrBBqATVuCMS1Dq5ar9EHlcnt+uWrvB7O+KLw2mjWPA8a0VOzZTBzTwk3nvLlquWo9Ipfbs8sdMKalahmzogr3WCQdKOl0SZ+QdGHprZYVtPq75KQOWoZuPzLaMnQwl5zU8+96Xa45y5kVUXQhzjOArwObgWWk39Z0C+Af+r9q1ijdF4O3zboqOCuptFxfZjPtieUGcrt0TxIo+vrMiqg6QWC7TNKTwHeAyyNiS81r1QCeIFDZQL+4ubMGcrvc0PkkM+56gkc+/S72HlH9Gk+5gdwmu2Kgt0vRCQJFh9HGA18bqIHGzF5x+IRRACx4YXWDa2IDSdFg80PgLbWsiJntHjq6g83zLze4JjaQFJ2N9t/ADEmTqbwQp3/UaTZA7L/3XozeawhPPO+ejfWfosHmxvzv31RI80KcZgOIJA6fMJoFDjbWjwoNo1XYndM7dZoNYB0TRrHg+dUUmUBkVkSvwUbSUEm/lOTJ9mZNomPCKFZv2FzxR55mO6PXYBMRm0hbDPgrjlmT2DYjzUNp1k+Kzka7CfhILStiZruPw3Kw8SQB6y9FJwi0AmdIOhF4ANhuc/qIuKC/K2ZmjTN6r6FMHNPino31m6LB5gjgwXz/4LI0D6+ZDUCH50kCZv2h6E6d76h1Rcxs99IxYRT/s3AZGzdvZdgQ7zJiu8bvIDOrqGPCKDZvjW1bhJvtir5sMfAOSTMl3SVpbumtD+fYV9KtktZIWiTp9Cr5JGmGpBX5NkOSStKnSHpA0tr875QK5xgm6XFJi4vWz8xecfiE0YBnpFn/KBRsJJ0N/CcwCphG2mZgH+ANwG/68HzXAxtJC3ueAdyQl8Apdy4wHTgKOBI4GTgv12UYMAe4JdfhJmBOPl7qklxPM9sJB49rZehgeUaa9YuiPZuLgfMj4gOkddEui4ijSR/4hfrYklqB00jbFHRFxL3A7cCHKmQ/C7gmIhZHxBLgGuDsnDaNdK3pyxGxISKuAwRs2yRd0kHAB4GrCr4+MyszdPAgXjNupBfktH5RdDbawcA9+f4GYGS+/xWgE7i0wDkOAzZHxMKSY48Ax1XIOzmnleabXJL2aGy/jsaj+fhd+fE/kdZx6/Hnz5LOJfWiGDduHJ2dnQVeRnPp6upyu1TQLO2yj9bzyKJir7VZ2qSv3C5J0WCzgjSEBrAEeB3pA74NKLpB+Uig/CvSqpLzluddVZZvZL5uU5623XkkvRcYHBG3SprWU4UiYiYwE9LmaQN5g6OdNdA3ftpZzdIuj/Mk9931BEe/eWqvG6k1S5v0ldslKTqM9lPgXfn+d4HrJH0D+DZp+4EiuoDRZcdGA5UGhMvzjga6cm+m6nnyUN0XAf/I1KwfeCM16y9Fg835pMAC6TrIl0i9mu8CHy54joXAEEmHlhw7CphfIe/8nFYp33zgyNLZaaRJBPOBQ4F24KeSngdmA/tLel5Se8F6mlnmjdSsvxT9UedLJfe3AjP6+kQRsUbSbOBKSR8GpgCnAG+rkP1m4EJJPyStUHAR6ToMpGtEW4ALJH2VV9ZsmwtsBV5Vcp63ka4rvQHPTDPrs/333otR3kjN+kFffmczXtLFkm6QNDYfm5pnfhX1MdI1nhdJPaWPRsR8ScdKKp3VdiNwB2lX0MeAO/MxImIjaVr0mcBK4BxgekRsjIjNEfF89w14CdiaH2/pQz3NjO6N1Lxsje26Qj0bSccAPwKeJs36+hKwHDiRNMus4o8zy+Ue0vQKx3/KKzPcyNdmPplvlc7zEHDCffxwAAANq0lEQVRMgefrBA4sUjczq+zwCaO57eElRATbj16bFVe0Z3M1cG3+bc2GkuN3A1P7vVZmttvomDCK1es3s3TV+kZXxfZgRYPNMaRf6pd7jrQagJkNUN0z0p54zpMEbOcVDTbrSEvDlDucdP3FzAYob6Rm/aFosJkDfEbS8Pw48lTiGcD3a1AvM9tNeCM16w99WRttX9L04RHAvcBvSb/c/1RtqmZmu4sOz0izXVT0dzYvA2+XdDzpNyuDgAcj4p6eS5rZQHD4hFH8xBup2S4oujYaABExl/TjSQAkTQK+FBHv7++Kmdnuo3sjtaeWd23b58asL3b1K8oY0rYBZjaAdQeYJ57zUJrtHPeHzaxX3kjNdpWDjZn1yhup2a5ysDGzQjwjzXZFjxMEJN3eS3lfKTRrEodPGM2ch5eyat0m9m7peSM1s3K9zUZbUSD96X6qi5ntxrqXrVn4wmre1L5vg2tje5oeg01E/Hm9KmJmu7eOkjXSHGysr3zNxswK8UZqtiscbMysEG+kZrvCwcbMCuuYMIoFL6wm7W9oVpyDjZkVdviE0d5IzXaKg42ZFdY9I80/7rS+crAxs8K6N1J73GukWR852JhZYd5IzXaWg42Z9YmXrbGd4WBjZn3SMWEUTy7rYuPmrY2uiu1BHGzMrE8OL9lIzawoBxsz65PujdQ8lGZ94WBjZn3SvZGaZ6RZXzjYmFmfeCM12xkONmbWZ56RZn3lYGNmfdYxYRRLV61n1bpNja6K7SHqGmwk7SvpVklrJC2SdHqVfJI0Q9KKfJshSSXpUyQ9IGlt/ndKSdolkh6TtFrS05IuqcdrM2smR+RJAgtfcO/Giql3z+Z6YCMwHjgDuEHS5Ar5zgWmA0cBRwInA+cBSBoGzAFuAfYBbgLm5OMAAs7MaX8EnC/pz2r1gsya0baN1DyUZgXVLdhIagVOAy6PiK6IuBe4HfhQhexnAddExOKIWAJcA5yd06aRdhj9ckRsiIjrSAHmeICI+GJEPBgRmyNiASkwTa3hSzNrOts2UnvOkwSsmB63he5nhwGbI2JhybFHgOMq5J2c00rzTS5JezS231Dj0Xz8rtKT5KG3Y4EbK1VI0rmkXhTjxo2js7Oz6GtpGl1dXW6XCtwusH/LVn61YDGdnSsAt0k1bpeknsFmJFD+NWgVMKpK3lVl+Ubm4FGe1tN5riD13r5RqUIRMROYCdDR0RHTpk3r8QU0o87OTtwuO3K7wD0r5zHn4aUcd9xxSHKbVOF2Sep5zaYLGF12bDRQadC3PO9ooCv3ZgqdR9L5pGs3fxwRG3ah3mZWQYc3UrM+qGewWQgMkXRoybGjgPkV8s7PaZXyzQeOLJ2dRppEsO08ks4BLgVOiIjF/VB3MyvjjdSsL+oWbCJiDTAbuFJSq6SpwCnANytkvxm4UNJESQcAFwGzclonsAW4QNLw3IMBmAsg6Qzg74ETI+KpWr0es2bnGWnWF/We+vwxoAV4Efg28NGImC/pWEmlS8jeCNwBzAMeA+7Mx4iIjaRp0WcCK4FzgOn5OMDfAW3A/ZK68u2rtX9pZs2leyO1J7xGmhVQzwkCRMRLpEBRfvynpAv/3Y8D+GS+VTrPQ8AxVdIO6pfKmlmvvGyNFeXlasxsp3kjNSvKwcbMdpo3UrOiHGzMbKd1bJuR5qE065mDjZnttIPHjmToYHlGmvXKwcbMdtqwId0bqTnYWM8cbMxsl3RMGOUFOa1XDjZmtku6N1Jbsyl6z2xNy8HGzHZJ97I1S7o8/dmqc7Axs13SkXftXLzawcaqc7Axs11yQN5IzcHGelLX5WrMbOCZ8/BSNmzaytxntzL1C3O55KQOph89sddytz20hC/dvYClK9dxwJiWAVtuycp1TPzFwG2XYRMOqbh0WDkHGzPbabc9tITLZs9j45bUq1mych2XzZ4H0OMHVne5dZu2uNwAKFeEtt9duXl1dHTEggULGl2N3Y53GazM7ZJM/cJclqxct8Px4UMG8ZaD26qW++VTK9hQYT01l9vzyj130yfY8Nz/qmrmzD0bM9tpSysEGoANm7fy8rpNVctV+oBzuT2/XE8cbMxspx0wpqViz2bimBZu+8upVctV6xG53J5drieejWZmO+2SkzpoGTp4u2MtQwdzyUkdLtdk5Xrjno2Z7bTui8jbZl0VnM1UWq4vs6D2xHIDvV2e6zHnKzxBIPMEgcp8Ibwyt8uO3CaVDfR2kfRARLyxt3weRjMzs5pzsDEzs5pzsDEzs5pzsDEzs5pzsDEzs5pzsDEzs5pzsDEzs5pzsDEzs5pzsDEzs5pzsDEzs5pzsDEzs5pzsDEzs5qra7CRtK+kWyWtkbRI0ulV8knSDEkr8m2GJJWkT5H0gKS1+d8pRcuamVn91btncz2wERgPnAHcIGlyhXznAtOBo4AjgZOB8wAkDQPmALcA+wA3AXPy8R7LmplZY9Qt2EhqBU4DLo+Iroi4F7gd+FCF7GcB10TE4ohYAlwDnJ3TppH24flyRGyIiOsAAccXKGtmZg1Qz83TDgM2R8TCkmOPAMdVyDs5p5Xmm1yS9mhsvxHPo/n4Xb2U3Y6kc0k9IYANkh4r9lKaylhgeaMrsRtyu+zIbVLZQG+XSUUy1TPYjAReLju2ChhVJe+qsnwj87WX8rTy81QtWxagiIiZwEwASb8usgFQs3G7VOZ22ZHbpDK3S1LPazZdwOiyY6OB1QXyjga6crDo7Tw9lTUzswaoZ7BZCAyRdGjJsaOA+RXyzs9plfLNB44sm2F2ZFl6tbJmZtYAdQs2EbEGmA1cKalV0lTgFOCbFbLfDFwoaaKkA4CLgFk5rRPYAlwgabik8/PxuQXK9mRm319VU3C7VOZ22ZHbpDK3C6B6ji5J2hf4OnAisAK4NCL+TdKxwH9GxMicT8AM4MO56NeAv+4eCpN0dD72WuBx4C8i4qEiZc3MrP7qGmzMzKw5ebkaMzOrOQcbMzOruaYPNkXXa2s2kjolrZfUlW8LGl2nepN0vqRfS9ogaVZZ2gmSnsjr8/1YUqEftg0E1dpFUrukKHnPdEm6vIFVras8Yelf8+fIakkPS3p3SXrTvmfAwQaKr9fWjM6PiJH51tHoyjTAUuDvSJNatpE0ljSz8nJgX+DXwHfqXrvGqdguJcaUvG8+V8d6NdoQ4FnSqih7A58CvpuDcLO/Z+q6gsBup2S9ttdFRBdwr6Tu9doubWjlrOEiYjaApDcCB5YknQrMj4j/yOlXAMslHR4RT9S9onXWQ7s0tfzzjitKDv1A0tPAMUAbTfyeAfdsqq3X5p5NcpWk5ZJ+JmlaoyuzG9lu/b38IfMkft90WyRpsaRv5G/0TUnSeNJnzHz8nmn6YNOX9dqazV8DBwMTST9Ku0PSaxpbpd1Gb+vzNavlwJtICzMeQ2qPbzW0Rg0iaSjptd+Uey5N/55p9mDTl/XamkpE/DIiVudtHG4Cfga8p9H12k34fVNB3jrk1xGxOSJeAM4H3iWpaT5QASQNIq2MspHUBuD3TNMHm76s19bsgrRvkJWtv5ev/b0Gv2/Kdf9ivGk+Z/IKJv9KmnB0WkRsyklN/55pmjdBJX1cr61pSBoj6SRJe0kaIukM4A9J+wU1jfza9wIGA4O72wO4FXidpNNy+qdJeyw1xYXeau0i6S2SOiQNktQGXAd0RkT58NFAdgNwBHByRKwrOd7U7xkAIqKpb6RpiLcBa4DfAac3uk6NvgHjgPtJXfyVwC+AExtdrwa0wxWkb+eltyty2juBJ4B1pMVh2xtd30a3C/AB4On8f+k50qK4Expd3zq2y6TcFutJw2bdtzOa/T0TEV4bzczMaq+ph9HMzKw+HGzMzKzmHGzMzKzmHGzMzKzmHGzMzKzmHGzMzKzmHGzMBqi8t8z7Gl0PM3CwMasJSbPyh3357ReNrptZIzT1fjZmNXYPaW+kUhsbURGzRnPPxqx2NkTE82W3l2DbENf5ku7M2wQvkvTB0sKSXi/pHknrJL2Ue0t7l+U5S9K8vEXzC5JuKqvDvpL+I297/lT5c5jVi4ONWeN8FrgdmELaM+jmvPtl96rAd5PW1noz8F7gbZRsxSzpPOBG4BvAkaQtIB4re45PA3NIKw5/B/i6pFfX7iWZVea10cxqQNIs4IOkRRlLXR8Rfy0pgK9FxEdKytwDPB8RH5T0EeBq4MCIWJ3TpwE/Bg6NiN9KWgzcEhEVtzDPz/GFiLgsPx5C2izw3Ii4pR9frlmvfM3GrHZ+Apxbdmxlyf37ytLuA/443z+CtAR96eZaPwe2Aq+V9DJpF9Uf9VKHR7vvRMRmScuA/YpV36z/ONiY1c7aiPhtDc7bl+GITWWPAw+fWwP4TWfWOH9Q4fHj+f7jwOvLtlR+G+n/7OMR8SKwBDih5rU06wfu2ZjVznBJE8qObYmIZfn+qZLuJ22k9T5S4HhLTvsWaQLBzZI+DexDmgwwu6S39HngHyW9ANwJjABOiIhravWCzHaWg41Z7byTtGNlqSXAgfn+FcBppO2TlwF/HhH3A0TEWkknAV8GfkWaaDAH+Hj3iSLiBkkbgYuAGcBLwA9r9WLMdoVno5k1QJ4p9n8i4nuNrotZPfiajZmZ1ZyDjZmZ1ZyH0czMrObcszEzs5pzsDEzs5pzsDEzs5pzsDEzs5pzsDEzs5r7/wH9yRB6gLhoJQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.epoch, [piecewise_constant_fn(epoch) for epoch in history.epoch], \"o-\")\n",
    "plt.axis([0, n_epochs - 1, 0, 0.011])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.title(\"Piecewise Constant Scheduling\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performance Scheduling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/25\n",
      "55000/55000 [==============================] - 4s 79us/sample - loss: 0.5954 - accuracy: 0.8055 - val_loss: 0.5432 - val_accuracy: 0.8154\n",
      "Epoch 2/25\n",
      "55000/55000 [==============================] - 4s 74us/sample - loss: 0.5194 - accuracy: 0.8345 - val_loss: 0.5184 - val_accuracy: 0.8468\n",
      "Epoch 3/25\n",
      "55000/55000 [==============================] - 4s 73us/sample - loss: 0.5080 - accuracy: 0.8453 - val_loss: 0.5780 - val_accuracy: 0.8384\n",
      "Epoch 4/25\n",
      "55000/55000 [==============================] - 4s 73us/sample - loss: 0.5360 - accuracy: 0.8452 - val_loss: 0.7195 - val_accuracy: 0.8350\n",
      "Epoch 5/25\n",
      "55000/55000 [==============================] - 4s 74us/sample - loss: 0.5239 - accuracy: 0.8504 - val_loss: 0.5219 - val_accuracy: 0.8562\n",
      "Epoch 6/25\n",
      "55000/55000 [==============================] - 4s 74us/sample - loss: 0.5163 - accuracy: 0.8528 - val_loss: 0.5669 - val_accuracy: 0.8382\n",
      "Epoch 7/25\n",
      "55000/55000 [==============================] - 4s 74us/sample - loss: 0.5088 - accuracy: 0.8561 - val_loss: 0.6591 - val_accuracy: 0.8268\n",
      "Epoch 8/25\n",
      "55000/55000 [==============================] - 4s 77us/sample - loss: 0.3022 - accuracy: 0.8938 - val_loss: 0.3955 - val_accuracy: 0.8834\n",
      "Epoch 9/25\n",
      "55000/55000 [==============================] - 4s 76us/sample - loss: 0.2501 - accuracy: 0.9087 - val_loss: 0.4060 - val_accuracy: 0.8792\n",
      "Epoch 10/25\n",
      "55000/55000 [==============================] - 4s 75us/sample - loss: 0.2304 - accuracy: 0.9158 - val_loss: 0.3998 - val_accuracy: 0.8846\n",
      "Epoch 11/25\n",
      "55000/55000 [==============================] - 4s 75us/sample - loss: 0.2155 - accuracy: 0.9206 - val_loss: 0.3880 - val_accuracy: 0.8898\n",
      "Epoch 12/25\n",
      "55000/55000 [==============================] - 4s 75us/sample - loss: 0.2034 - accuracy: 0.9253 - val_loss: 0.4049 - val_accuracy: 0.8838\n",
      "Epoch 13/25\n",
      "55000/55000 [==============================] - 4s 77us/sample - loss: 0.1878 - accuracy: 0.9285 - val_loss: 0.4440 - val_accuracy: 0.8838\n",
      "Epoch 14/25\n",
      "55000/55000 [==============================] - 4s 80us/sample - loss: 0.1839 - accuracy: 0.9325 - val_loss: 0.4478 - val_accuracy: 0.8838\n",
      "Epoch 15/25\n",
      "55000/55000 [==============================] - 4s 76us/sample - loss: 0.1747 - accuracy: 0.9348 - val_loss: 0.5072 - val_accuracy: 0.8806\n",
      "Epoch 16/25\n",
      "55000/55000 [==============================] - 4s 75us/sample - loss: 0.1689 - accuracy: 0.9367 - val_loss: 0.4897 - val_accuracy: 0.8790\n",
      "Epoch 17/25\n",
      "55000/55000 [==============================] - 4s 78us/sample - loss: 0.1090 - accuracy: 0.9576 - val_loss: 0.4571 - val_accuracy: 0.8900\n",
      "Epoch 18/25\n",
      "55000/55000 [==============================] - 4s 74us/sample - loss: 0.0926 - accuracy: 0.9639 - val_loss: 0.4563 - val_accuracy: 0.8934\n",
      "Epoch 19/25\n",
      "55000/55000 [==============================] - 4s 75us/sample - loss: 0.0861 - accuracy: 0.9671 - val_loss: 0.5103 - val_accuracy: 0.8898\n",
      "Epoch 20/25\n",
      "55000/55000 [==============================] - 4s 75us/sample - loss: 0.0794 - accuracy: 0.9692 - val_loss: 0.5065 - val_accuracy: 0.8936\n",
      "Epoch 21/25\n",
      "55000/55000 [==============================] - 4s 75us/sample - loss: 0.0737 - accuracy: 0.9721 - val_loss: 0.5516 - val_accuracy: 0.8928\n",
      "Epoch 22/25\n",
      "55000/55000 [==============================] - 4s 76us/sample - loss: 0.0547 - accuracy: 0.9803 - val_loss: 0.5315 - val_accuracy: 0.8944\n",
      "Epoch 23/25\n",
      "55000/55000 [==============================] - 4s 78us/sample - loss: 0.0487 - accuracy: 0.9827 - val_loss: 0.5429 - val_accuracy: 0.8928\n",
      "Epoch 24/25\n",
      "55000/55000 [==============================] - 4s 80us/sample - loss: 0.0455 - accuracy: 0.9844 - val_loss: 0.5554 - val_accuracy: 0.8918\n",
      "Epoch 25/25\n",
      "55000/55000 [==============================] - 4s 79us/sample - loss: 0.0427 - accuracy: 0.9850 - val_loss: 0.5730 - val_accuracy: 0.8920\n"
     ]
    }
   ],
   "source": [
    "lr_scheduler = keras.callbacks.ReduceLROnPlateau(factor=0.5, patience=5)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "optimizer = keras.optimizers.SGD(lr=0.02, momentum=0.9)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "n_epochs = 25\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[lr_scheduler])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdMAAAEeCAYAAADRiP/HAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzsnXeYVdXVuN9Fb6IiMCo6g6iIYsGCDQiosX6fiS0aNVFSxJZoihqNGgsaxai/4CcWsCsSG9gbiTMiKMaCbUbFBrZRAQWdAYYy6/fHOnfmzJ1bzp25bYb1Ps9+5t7dzj4H7l137b2KqCqO4ziO47ScDoVegOM4juO0dVyYOo7jOE4rcWHqOI7jOK3EhanjOI7jtBIXpo7jOI7TSlyYOo7jOE4rcWHqrFOIyFEi4v5gWUZExoiIikjfQq/FcQqBC1On6BCRO4IvZhWRNSLyqYjcKCIbFnpt2SK4x8dTtC8IPYMVIvKeiJwtIpLPdYbWUxFaT52IzBeRv4pIx1bOeX021+k4hcKFqVOs/BvYBBgI/BY4FLihkAsqAJdiz2Bb4Grg78C4Aq7n9mA92wDXAZcBZxVwPY5TNLgwdYqVOlX9SlU/V9VngfuAA8IdRGR9EZksIt+IyA8i8ryI7BbX5wQRWSgiywNNsCSu/WIReSeubqyI1MTVHSIiLwda4hIReUxEugVtXURkgoh8HlznFRE5MAvP4IfgGSxQ1VuAt+KfQTwi0lVE/ikiX4vIShGZKyIjQ+2x7dj9gvtZLiKvisguEdazPLSe64H/AIclWcdGIjIteCYrRKRSRH4Var8DGA2cHtJ4BwZt24nIE8G/6TfBPBuHxg4XkWdFZLGIfC8is0Vkr7jrq4gcFVe3QERc+Ds5wYWpU/SIyCDgIGB1qE6AJ4ABwP8COwOzgOdEZJOgzx7AHcBkYBjwGKbtZXr9g4BHgZnArsA+wPM0fn5uxwTDccD2wJ3AYyKyU6bXSnJ9EZExmIa6Ok33q4BjgF9jz+Rt4OnYMwlxBXAusAuwBJjagi3kFUDnJG3dgNexf5uhwETgZhHZL2g/E3iJRm13E+CzYJ2zgHeA3YEfA72AR0Qk9rzXA+4GRgV93gCeFJGNMly/42QPVfXipagKJgDXADXYF7YG5Y+hPvsG7d3jxr4BnBO8vheYGdd+i/23b3h/MfBOXJ+xQE3o/RzgX0nWuiVQD5TG1T8M3JDmHh9P0b4AqAvucVVw/yuAvVOM6Rn0PSFU1xH4CLgseD8mmOvAUJ8RQd1mKeauAK4PXnfAftzUARPi5u2bYo5/AbckmjNUdynwn7i6DYO5d08yrwDVwC9CdQocleCZnlXo/99e2mdxzdQpVmZh2uTuwP8BT2LndDF2BXoAi0SkJlYwzXDLoM+2mPYTJv59FHbGtjQTsQv2ZV4Vt47/Ca2jpVyLPYPRQDlwiaq+mKL/lpimOCdWoaprsXveLq7vW6HXXwZ/+6dZz7jg3lZimvo9wCWJOopIRxE5X0TeCrbFa4AjgNI019gV+FHcs/wsdH+ISH8RuTkwgloG/BCsPd3cjpMzOhV6AY6ThOWq+mHw+gwRKQcuxDRJMO3oa2yrL57vM7hOPSYMwyTbukxEB0wLGk7zLdgVGcyTiCXBM/hQRI4EPhCRl1W1vAVzxbsDrU7Qlu7H9X2Y8KwDvgwEdTLOAv6Mbee+jWnYfye9wO6Abd8nOtv8Ovh7J3b2/UcaNfj/AF1CfZXW/bs6Tka4MHXaCpcAT4nIZFX9EjuPKwHqVfXjJGPeBfaMq4t/vwgoERFR1ZhQGRbXZx6wHzAlwTXmYV/aG7dQyEVCVb8L3Ej+n4jsHFprmI+wbd4RwWsC15W9sC3v1rIs9AMnHSOBx1T17mAdAgwGlob6rMK2ocO8DhwNLFTVZOfDI4EzVPWJYO4S7Mw1zKJwXZI+jpM1fJvXaROoagVQBVwQVP0b2858REQOFpEtRGQvEblERGLa6nXAj0XkPBHZWkROAg6Pm7oC6AP8VUS2FJHfAEfF9bkc+JmIXBZYmg4VkT+KSA9VnQ9MBe4QCwgxSER2E5GzROSINLfVW0SGxZWBKfrfgLml/CzJM6oFbgQmiFkfbxu8LyH/bkXzgf1EZKSIDAGuB7aI67MA2F1EBopI38DAaBKwPnCfiOwRPM8fi1ltrxea+xfBv8Vw7Cx2Vdzcz2GWwruJyM7YGfXKXNyo44ALU6dtcQ3wGxEpCzSzQ7AvzSnA+8D9mLD5EkBV5wK/AU7FzgiPoHGbmKDPu0H7uKDP/th2ZLjPk5gQPhjTRJ/HLHrrgy6/wqxSrwLeAx4HfgQsTHM/o4L5wuXqZJ1V9RvMivXikGVrPH/BtmNvx4yxdgQOUtXqNGvJNpcB/wWews6/a7EfHWGuxoRgFaZJlga7DiOwZ/s0UIkJ2LqggFkq9wJewwTpbZhgDvNn4GPsx9KDmOHZN1m6N8dphiTeLXIcx3EcJyqumTqO4zhOK3Fh6jiO4zitxIWp4ziO47QSF6aO4ziO00rczzQiHTp00O7duxd6GUVFfX09HTr477F4/Lkkxp9LYtrzc+nevTtLlixZrKr9Cr2WXOPCNCJdunShtra20MsoKioqKhgzZkyhl1F0+HNJjD+XxLT35yIiPQq9hnzQPn8OOY7jOE4ecWHqOI7jOK3EhanjOI7jtBIXpo7jOI7TSlyYOo7jOE4ryaswFaGPCDNEqBVhoQjHJeknIkwQYUlQJohYbkIRBovwiAiLRPhWhGdE2CZu/B9F+EqE70W4TYSuobaBIpSLsFyE90T4cZS119V1ZOBAmBofqjsJU6fCwIHQoQN5Gzd8s2qel9EM3/yrnF4vNmbffUfn5d4cx3GKHlXNWwGdBnofaC/QkaDLQIcm6Hcy6Pugm4EOAK0CPSVo2x30N6B9QDuDjgd9LzT2QNCvQYeCbghaAXplqP0l0GtBu4MeCboUtF/6tfdQUO3RQ/WeezQl99xj/aCx5GPcJE7VNXTQ6zktZ9fL9721RcrLywu9hKLEn0ti2vtzAWo1j3KmUCVvWWNE6Al8B2yvyvyg7m7gC1XOjev7InCHKpOD978BTlJtltgZEfoAS4C+qiwR4V5ggSp/Ddr3A6aqsrEIg4G3g74/BO0vBO03pV5/T7UsUrD++nDGGcn7XncdLFvWvD6X43os+4IFbEEXVrOc7gziY1auv3HWr5fteysrgwULko9ri7R3v8GW4s8lMe39uYjIclXtWeh15Jp8CtOdgTmq9AjVnQWMVuXQuL7LgANUeTl4vxtQrsp6xCHCYcCNqmwSvH8T+Lsq9wXv+2K5EvtiOSb/rsq2ofHXYwr67xPMPQ7Lcwn03DUmTEERSX6v9kgTdcjduBkcxmE8CsBKunArv+V3XJ/162X73kSU5557PvnANkhNTQ29evUq9DKKDn8uiWnvz2WfffZZJ4RpPrd4R4F+FVd3EmhFgr5rQYeE3m8dbA1KXL/NQL8APTZU9xHoQaH3nYOxA0F/CTo3bo7LQe9Iv/4eDduTZWWakrIybbKdmetxuw34Uuvo3GRQLd11t82qs369fN9bW6S9b9u1FH8uiWnvz4V1ZJs3nwZINUDvuLreYNutafr2BmpUaVCjRegHPAvcoMq0NGMJrpPJGhLSowdcfnnqPpdfbv3yNW7qkPF0YG2Tug6s5Z5txmf9evm+N8dxnLZAPoXpfKCTCFuH6nYCKhP0rQzaEvYTYUNMkD6qSvzXcaKxX6uyJGgbJNJkuzjZGppRVgaTJ8Pxx6fud/zx1q+sDERyP27wkpfoRH2Tum6sYpslL2b9ek3HaMb3Vlpq79dbL9o4x3GcNkE+1WDQfwUWvT1BR6Sw5j0F9N3AkndT0MqQNW9v0P+CXp/kGgeBfgW6HegGoM/FWfPOBb0atBvo4VGtebt27Zp8H6MY6NvX9k133jlvl2zp9tR226kefnh211JMtPdtu5bizyUx7f254Nu8OeE0oDvwDTANOFWVShFGiVAT6ncz8BhmefsO8ERQB3A4MBz4lQg1oVIKoMrTwFVAOfApsBC4KDT3z4HdMMviK4GjVFmUk7vNF4sWweLF0LkzLFxY6NWkZeDA9mfB6zjOuk1eU7Cp8i1wWIL6F4BeofcKnBOU+L53Anemuc61wLVJ2hYAYzJYdvFTGexSjxwJ5eVQUwNFbB1YVgZz5xZ6FY7jONnDwwm2B6qq7O/BB9vfzz4r3FoiMHAgfPstfP99oVfiOI6THVyYtgeqqqB3b9hrL3tf5Fu9Awfa3yJfpuM4TmRcmLYHKithu+1s/xTg008Lu540xJbpwtRxHET6IDIDkVpEFiKSMGY7Ik8hUhMqqxB5O9Q+EJFyRJYj8h4ikeKuZwsXpu2BqioTpptsAh07Fr0wjWmmboTkOA4wCVgFlADHAzciMrRZL9WDUe3VUOBF4IFQj2nAPGAj4HzgQUT65XrxMVyYtnUWL4ZvvoGhQ6FTJ9hss6JX+fr3h27din6ZjuPkGpGewJHAhajWoDobeBT4ZZpxA4FRwF3B+8HALsBFqK5A9SHMG+TIXC09HhembZ2Y8dF229nf0tKi10xjASJcM3Wc9k9f6ITIq6EyLtQ8GFiD6vxQ3ZtAc820KScAL6C6IHg/FPgY1XA0uyjzZI28usY4OSCRMJ0zp3DriYj7mjrOusFiE5a7JWnuBcTb9S+D5klN4jgBuCxunvi8VMuAAVHX2VpcM23rVFWZT+nmm9v7sjL4/HNYuzb1uAJTVubbvI7jtCBeushIYGPgwVbNk2VcmLZ1Ypa8sfxnpaWwZg1UVxd2XWkYONACN9XWpu3qOE77ZT62DRwlZnuME4HpqIaj5lUCgxBpUdz1bODCtK0Ts+SNEYskX+TnpjGL3iJfpuM4uUS1FpgOXIpIT0RGAD8F7k7YX6Q7cDRwR9w884E3gIsQ6YbI4cCOwEM5W3scLkzbMt9+C199ZZa8MdqIE2dsmX5u6jjrPM1itqNaicgoRGri+h4GLMVir8fTLO46qnmLu+4GSG2ZeOMjaDw7LXKVz31NHccBQDVhzHZUm8RsD+qmQZP81eG2BRQw7rprpm2ZRMJ0vfVgww2LXphuvDF06VL0CrTjOE4kXJi2ZaqqoGfPxnPSGGVlRS9MO3SwZbtm6jhOe8CFaVumshK23dYkU5jS0jah8rmvqeM47QUXpm2ZeEveGG0gChK4r6njOO0HF6ZtlaVL4csvm1ryxigrg2XLrBQxAweaMfLKlYVeieM4TutwYdpWSWR8FKON+Jq2kYxxjuM4acmrMBWhjwgzRKgVYaEICfPWiSAiTBBhSVAmiCCh9skivC9CvQhj48beJEJNqNSJNIaUEqFChJWh9vdzdsO5pB0IU3ePcRynvZBvzbRZ3jqRhFH9x2F+RzthUSwOBU4Otb+JOfq+Hj9QlVNU6RUrmE/SA3Hdfhfqs01rb6ogVFVB9+6NEilMG1H5XJg6jtNeyJswFaEhb50qNaqkylt3InCNKp+r8gVwDTRqoKpMUuU/QMrTttA178zOXRQRySx5AUpKoHPnorfu2XRTS8Fa5Mt0HMdJSz4jIA0G1qgSn7dudIK+Q4O2cL+W5KU7ElgEzIqrv0KEK4H3gfNVqUg0WIRxmJZMp05CRUXCbgVhz3nzWDpsGO8lWdMe/frx/Suv8G4O11xTU9PqZ9Kv3x68/PL3VFS8m51FFQHZeC7tEX8uifHn0j7IpzDNJG9dfG66ZUAvEUQVzeCaJwJ3xY35C1CFbTf/HHhMhGGqfBQ/WJXJwGSAbt1Ux4wZk8Glc8j338OiRWy8775snGxNgwfTva6OkhyuuaKigtY+k222gZUruzNmTEl2FlUEZOO5tEf8uSTGn0v7IJ9nppnkm4vv2xuoyUSQilCKxWm8K1yvysuq/KBKnSp3AnOAQ6LOWxSkMj6K0UacOD1wg+M47YF8CtP5QCcRouStqwza0vVLxS+BOap8nKafQqOlcJsgijAtLTU/1NWr87OmFjJwoC1z1apCr8RxHKfl5E2YqtKQt06EniKkylt3F/AnEQaIsCnwZ0L560ToIkI3TAh2FqGbSLN7OYG4nHcibCDCgUH/TiIcD/wIeDo7d5knqqqgWzfYYovkfUpLob7eJFURU1YGqvDZZ4VeieM4TsvJt2tMs7x1qlSKMEqEcN66m4HHgLeBd4AngroYzwIrgL2xM80VmFAEQIS9gM1o7hLTGbgMM0paDPweOCzOKKr4qayEIUOgY8fkfYo1r2l1NYwebaGPcPcYx3HaB3nNZ6pKwrx1qjTJWxecjZ4TlETzjElznZeAngnqFwHDM1p0MVJVBSNHpu5TrIEbxo+H2bPt76RJLkwdx2kXeDjBtsYPP5iATBSTN0wxJgmvrobbb7ft59tvh6++YsAAc5UtNgXacRwnE1yYtjXeDfwxUxkfAfToAf36FZeUGj++0SBqzRoYP57OnWGzzVwzdRynbePCtK0RxZI3RjGlYotppWvX2vvVqxu00zbixeM4jpMUF6Ztjaoq6NoVBg1K37eYhOn48ba9G2btWhg/3n1NHcdp87gwbWtUVlrYoE4RbMdiKp9mEjQqR7z0UnNn0lWr4MUXGTgQPv+86F1iHcfJBSJ9EJmBSC0iCxFJmE0s6LsLIrMQqUHka0TODLUtQGRF0FaDyLP5WH4MF6ZtjaqqaFu8YJppbS18911u1xSFefPg6qvttQice64J+XnzKCszpfWLLwq7RMdxCkKzbGKINLewFOmLxQS4GdgI2ApzkwxzKKq9gnJATlcdhwvTtkRtre2HprPkjVFs7jHl5TB4MGyyCXz9dUO1u8c4zjqKSEM2MVRrUE2VTexPwDOoTkW1DtUfUC2aDBkuTNsSUS15YxRT4IY1a2DWLNhnH0sRFxKmxbRMx3GyS1/ohMiroTIu1DwYWINqfDaxRBrDnsC3iLyIyDeIPIZIaVyfqYgsQuRZRHZKMEfOyGvQBqeVZGLJC8Wlmb7+uvnIjhljUvObbxqaNt/cdn5dM3Wc9sdiE5a7JWnOJJvYZsAuwP5YdLyrsEh6I4L244HXsTCzZwLPIDIE1aWtu4NouGbalqiqsqTfW24ZrX+/fhbDtxiEaXm5/R0zpplm2rWrJQp3Yeo46xyZZBNbAcxA9RVUVwKXAHsjsj4AqnNQXYHqclSvAJYCo3K39Ka4MG1LVFWZJW/nztH6ixSPe0x5OWy7LWy8MfTvb8I0ZGXsvqaOs04yH9sGjpJN7C1okoYznZtCXjOCuTBtS1RWRt/ijVFaWngptXq1xePdZx97X1JibjHfN+7uuK+p46yDqDZkE0OkJyKpsondDhyOyDBEOgMXArNRXYZIKSIjEOmCSDdEzgb6Yvmq84IL07bC8uXwySctE6aF1kxfecUskcPCFJoZIX32WWOAJMdx1hmaZRNDtRKRUYg0ZhNTfQ74K5ZF7BvMNSbmk7oecCPwHfAFcBBwMKpL8nUTboDUVnj/fdsWjeoWE6OszEL51dXZ4WQhCJ+Xgm3zggnTwYMB00zXrLH0q7EY/Y7jrAOoJswmhmqTbGJB3Y2Y0IzvWwns2Oq1iPQHDgHeRfXlTIa6ZtpWqAyOEFqimYKFGCoU5eWwww7Qt6+9T6CZuq+p4zh5R+RJRP4QvO4JvApcB8xG5PhMpnJh2laoqrIQglttldm4QrvH1NXBnDmNW7zQKExD7jHua+o4TgHYDXgueH0EUAv0A04mST7tZLgwbStUVdmWaJcumY0rtDB9+WVYubKpMO3b1yyNQ5ppbJmumTqOk0d6Y+esAAdgrjd1wL+xM9nIRBamIpSIcJYIN4rQN6gbIcIWmVzQaSEtseSFxgPIQql85eUmOEePbqzr1Ak22qiJMO3e3bxmXJg6jpNHPgX2QqQHcCAwM6jfEFieyUSRhKkIuwLvYxEmfkOjk+3+wOWZXNBpAStWwMcft0yYdu1qUqpQmml5OQwbBhtu2LQ+LnADuK+p4zh555/APZhQ/Rp4Pqj/EfBOJhNF1UyvBiaqsjNQF6p/hsZQTmkRoY8IM0SoFWGhCAlT7YggIkwQYUlQJog0Ot+KMFmE90WoF2Fs3NixIqwVoSZUxoTaB4pQLsJyEd4T4cdR118w5s+3tCqZWvLGKJR7zIoVlnotvMUbo6SkyZkpuK+p4zh5RvUGLErSqcDeqMaSLi8E/pbJVFGF6a7AnQnqq7G0OVFplmpHJGFA43GYqfROmLnzodiBcIw3Md+k15Nc5yVVeoVKRahtGjAPS+FzPvCgCP0yuIf801JL3hiFUvliOUyTCdMEmumnnzbPIe44jpMzVF9G9QFULYShSEdUHw1ccyITVZiuwPaQ4xmCOc+mRYSGVDuq1KiSKtXOicA1qnyuyhfANdCogaoySZX/ACsjrj+2hsFYoOSLVFmhykNYwOQjM5kn71RVQceOsPXW6fsmIqaZ5jtJeHk5dOgAoxKEx4yFFAwxcKDJ3q++ys/yHMdZxxE5DZEjQu9vBlYiUhkX4jAtUYM2PAJcJMLPgvcqwkBgAvBQxDkGA2tUiU+1MzpB36FBW7hfJnucO4uwGPgWC0t1hSprgjk+Vm0SRDnp3CKMw7RkOnUSKioqMlhC9hg6axY9BgzglZdeatH4AXV1bL1yJXMeeYTVG2yQtXXV1NSkfCY7P/wwMngwr8+b16yttLaWQTU1zHr6aeq7dQPg++/7ADsyffrrbL99fCKJtkO657Ku4s8lMf5cCsofgd8CIDIK2zE9ETgcU+J+EnkmVU1bQHuDzgb9HnQt6Bega0CfB+0ZcY5RoF/F1Z0EWpGg71rQIaH3W5tapRLXbzbo2Li6QaBbgHYA3QG0CvS8oO2XoHPj+l8Oeke69Xft2lULxuDBqkcc0fLxDz+sCqqvvJK9NalqeXl58saaGtXOnVXPOSdx+6232po++aShqrLSqu69N6vLzDspn8s6jD+XxLT35wLUagQZUZACKxQ2D15fpXBH8Ho7hcWZzBVpm1eV71UZiZ1j/gWYCBykymhVaiPK7UxS7cT37Q3U2D2mXevHqnyiSr0qbwOXAke1YA3FQV0dfPhhy89LoTC+pi++aAHuE52XQtOQggGxwA1uhOQ4Tp74ARpsZvbH/EvBbHu6ZTJRVNeYE0ToqspzqlytylWq/FuELiKcEPFa84FOIkRJtVMZtKXrF4VwGp5KYJBIk8SzrZk797TWkhcKI0zLy82fdOTIxO0JQgr27GnxHNw9xnGcPDETuDk4Kx0MPBXUbwcsyGSiqAZItwPrJ6hfL2hLS6DBTgcuFaGnCKlS7dwF/EmEASJsCvwZuCPWGAjxbpiQ7CxCNxG7FxEOFjELYxGGYGl6HgnWMB94Azv/7SbC4Zi1cNRz3/zTWktegD59TFLlU0qVl8Pw4dCrV+L2BCEFwd1jHMfJK6dj8Xg3A44OZZkZDtyXyURRDZCExIlYS4FlGVzvNOA2zAJ4CXCqKpUijAKeUm3IEHAzMAiztAW4JaiL8SyNhkt7A5OBfYAKYD/gDhF6YU649wB/D439OSaYv8McdY9SZVEG95BfqqrMIjbIrtIi8p0k/IcfLO3aX/6SvE+CbV4wYfr22827O47jZB3VpZiPaXz9hZlOlVKYivA2JkQVeF6ENaHmjkAZ8GTUi6mSMNWOKk1S7QRno+eQJNCwamMQhgRtZwFnpWhfAMnHFx1VVRbcvltG2/fNyacwnT3bEpMmOy8Fu5/evRP6mj7+uJmbiSQZ6ziOky1EumBK1naYrKsE7kd1VSbTpNNMHwz+bo8lZK0Jta3C9pSLd4u0PdDSmLzxlJXB68liXGSZ8nLo3Bn23jt1vyRRkFautOqSTMKBOI7jZIrIEOBpoA+NtjOnA+MROQjV96NOlVKYqnKJXY8FwH2qmQVJcFrJqlXwwQdwxBHp+6ajtBQWLbIQf927t36+VJSXw557Qo8eqfsliYIEdm7qwtRxnBwzEYvB+4tgyxdENgCmBm0HRZ0oqmvMnS5IC8AHH9h2aWsseWPELHo/+6z1c6Vi2TLTgFNt8cZIEgUJ3KLXcZy8MBI4t0GQQuwc9bygLTJRXWO6iHCJCPNFWBkEkm8omVzQyYBsWPLGiKl8uT43nTXLXHmiCNM0mqnjOE6OqaN57AEwT5WMzkyjusaMJ4iXC9QDZ2NB65dgFrpOLohZ8m6zTevnimmmuVb5ysst7duee6bvW1IC335rwR0Ceve2bG2umTrOOoJIH0RmIFKLyEJEEmYTC/rugsgsRGoQ+RqRM0NtAxEpR2Q5Iu8hEiUj2BPAZET2QIJ8ZSJ7AjcBj2VyG1GF6dHAKarcDKwFHlHlDOAiLGqEkwuqqmDQoOyccQ4YYII515ppebkZHkWxPo4dii5q6pnkvqaOs07RLJsYIs3PtkT6YsZCN2NZv7bC3CRjNMsIhki6jGBnYOnWXsISp6wE5mDGtX/I5CaiCtMSoCp4XQPEoqU/DRyQyQWdDMiWJS+Yde2mm+ZWmH77Lbz5ZrQtXkjpa+rC1HHWAUQasomhWoNqqmxifwKeQXUqqnWo/oDqu8E8DRnBUF2BarSMYKrfofo/mFvMz4MyFNVDUf0uk1uJKkw/BTYNXn8IHBi83gtLz+Zkm9WrLZRgtoQp2FZvLvdPn3/eHESjCtMkUZBi6Vc1zxnjHMfJPn2hEyKvhsq4UPNgYA2q8dnEElld7gl8i8iLiHyDyGOIBOdXlhGMWE7S1PM0R/U9VGcE5T1EtkLkxcg3SfQISDOwyEJzMXPhaSKcBAwA/pHJBZ2IfPghrFmTHUveGKWlFpkoV5SXmzvM7rtH658gPi+YZlpbC0uWWKxex3HaLotNWO6WpLkXEJ9vcRk0iZ8eYzNM+9wf0zqvwrZ2RwTzxEfjW4bJqJbQE9gjkwGRhKkq54VePyjCZ9gNzFfl8YyW6EQjm5a8McrKYPp0s7btEHVTIgPKy2HECOjSJVr/JMI0ZtG7cKELU8dp52SSyWsFMANV0whELgEWI7J+hvPkhBZ9o6rysirXqvK4CD2zvSgHMz4SgSFDsjdnaakFgogTXllh0SJ4553oW7xgQfC7dUvqa+rnpo7T7pmPbQNHySb2Fk1jxIdfVwKDEClYRrAWqydB1pWzgU+yuB4CRfhoAAAgAElEQVQnRlUVbLFF+ihCmZDLVGwVFfY3E2EqkjSkILgwdZx2j2pDNjFEeiKSKpvY7cDhiAxDpDOWEWw2qsuCM9c3gIsQ6YZI3jOCpQt03wVzfzkAWA1cpcrDQQ7TK7FfBv8v56tcF8mmJW+MsDDdI6PjgPSUl5umueuumY1LELhhgw3M39R9TR1nnaBZNjFUKxEZBTyFqiVBUX0Okb9ivqE9gNlA2Ce1WUYwVBNnBBOZR+JMaDEy1mLSnZlejAX9nYmdkT4gwhTMGOk84F5VVicf7rSINWvg/ffhkEOyO2/4MDLblJfDqFHmgpMJ/fsnDHHo7jGOs46gmjCbGKpNsokFdTcCNyaZZwHRM4Jl3dYnnTA9GhirygwRdsIcYjcEhqo2ScfmZJOPPjLXmGxrpuuvbypftrd5q6vhvffg17/OfGxJCbz6arPqmHuM4zhO1mlBvtJ0pDsz3Rx4xa7Nm1iUigkuSHNMzJI3m24xMXKR17Ql56UxSkrMeKm+vkl1TDN1X1PHcdoC6YRpZywQcIzVNPflcbJNVRBsKpuWvDFyofKVl5vWu/POmY8tKbHMON9+26R64ED4/ntYujTxMMdxnGIiip/pFSIsD153AS4WaSpQgzi9TraoqjKh16tX+r6ZUloKL72U3TnLy+FHP4KOHTMfGw4pGHIqDR/vbrhhFtboOI6TQ9JpprOALYEdgvIiUBp6vwOwfdSLidBHhBki1IqwUISE2QGC2P0TRFgSlAkiSKh9sgjvi1Avwti4sSeK8JoI34vwuQhXiTT+aBChIkgjVxOUyJnU88Ybb5ha9tVX2Z+7tNS0wJqa7Mz3+ecWraklW7yQNKSgu8c4jtOWSKmZqka2jIpKODvAMOAJEd5UbeZYOw6z7toJM1+eifmz3hS0vwncB0xIcI0eWLT/l4F+WNDkszBXnhi/U+WWbNxQ1olZ8tbXw/jxMGlSducP5zXNhoFTebn9ba0w9cANjuO0YaLG5m01QaSkI4HtVakBZos0ZAc4N677icA1qnwejL0GOIlAmKoyKahfGX8d1SZm01+IMBVo4Td9AaioaDTGuf12uPBC2Hjj7M0f9jXNljDt0wd23LFl45NkjunTB3r2dItex3FyjMimwEigP/G7tarXRZ0mb8KUIDuAKvHZAUYn6Ds0aAv3a6lp649oHlLqChGuBN4HzlelItFAEcZhWjKdOgkVFQm7ZZVdTz6ZXoAA9atXU33KKXzwh4zS6qWk6zffsBfw/syZVEfJOZqCmpoaVjz1FDVDh1I5a1bLJqmvZ3SHDnz63//ySdzz7ddvOK++upyKirxFBMsKNTU1efm/0tbw55KY9vpcuixZwnaXXkonGo/oig6RWKAHgMU0D1cYWZiiqnkpoKNAv4qrOwm0IkHftaBDQu+3NicJlbh+s0HHprjmr0E/B+0bqtsDdD3QrqAngv4AumW69Xft2lVzzscfqwY32lC6d1etrs7eNdasUe3YUfWvf231VC9Nm2ZrvO661k20ySaqv/lNs+pDDlHdeefWTV0IysvLC72EosSfS2La7XM59VRVES2F1ZonOZNxgQ8VrlLo3Nq5cpA6JCmZRPWP79sbqLF7j4YIhwFXAAersjhWrxak/wdV6lS5E8uqnuVQQy1k3LjmdWvX2tlptujYETbbLCu+phvMm2cvWnpeGiNBSEHwKEiO02ZZsAAmTwZVNsrvDmimbAzchGqrI/nlU5jOBzqJECU7QGXQlq5fQkQ4CJgCHKrK22m6K8WyDTF3bvO6VavgxYxy1KYnG4EbqqsZNHkybLRR64NL9O+fVJh+950ZNjuO00b4+GMYPtwUAYrlyzUpTwPDszFRJGEqQmmSsrkI/aLMoUpDdgAReoqQKjvAXcCfRBggwqbAn2nc10aELiJ0w/6dOgcZbDoEbfsCU4EjVflv3H1sIMKBQf9OIhyPnak+HeUecspHH5m7yvjx8Ru9ENMAs0U2Ajdceimdly61qPTSyo9LgswxkNtQwo7j5IDp02HYMFjcsBlY7ML0KeAqRC5G5EhEftKkZEBU9XsBKSLsi/A9lh7nHE0darBZdgBVKkUYBTyl2hDU+GZgEDRolbcEdTGepdFwaW9gMmaxW4Gl5VkfeDL0Hf+CKgdjEZ0uA4YAa4H3gMO0qVFUYbj1VkvY/atf5f5apaXmH7p2bcsCLVRXw+2324fk00/NH7Y1FsexbV7VJoI57B6zww4tn95xnBxTVwfnnAPXXQf9+sHKlRZfvPiZEvz9W4I2BSJ/QUYVpscCV2GuKS8HdXtglq4XAxsAF2Dnnxclm0SVhNkBVGmSHSA4Gz0nKInmGZPiGkkP8FRZRJZU+qyyZo25wRxyCAwYkPvrlZaaIK2utvPTTBk/3tYcft8af9j+/e3D98MPFog/wDVTx2kDfPIJHHMMvPIKnHmmufctSpz5rAjJMM1VcqIK01OBP6oyPVT3XBA96ExVRovwDXAJKYSpk4QnnjDt7qST8nO9cOCGTIVpdTXcdlvDeQirV7feHzYcuCEkTPv3h27d3AjJcYqWGTMad9OmT4fDD2/W5TWR5c0qiwXVtdmaKqoB0h6Q0JDnHRo1vZeAFqg5DrfcAptskv38pcmIBW5oico3fnzz7ZvWWhwnCSko4ha9jlOUrFoFf/gDHHEEbL212XUkEKRtApEDEXkOka8QqUbkP4gckOk0UYXpQoLgBXGchGU0Bwvd922CPk4qPv8cnnzSft11ypMFeTgKUqY8/3yzdGmttjhOElIQPK+p4xQdCxbAyJEwcaJt686eDVtsUehVtQyRX2GJwr/AdlUvBqqBxxEZm8lUUb+9/ww8JMIhBPlNgd2wIPhHBu+HA/dncnEH2yKtr29ZYu2W0quXxetriTDdYw+zPP7wQyo+/JAxY8a0fj1JQgqCaaavvdb6SziO0wqqq+HnP4exY+FPfzJjwYceMs20bXMecBaqE0N1NyPyatB2R9SJImmmqjwBbI0Fje8dlEeBbVR5Muhzgyp/inphBxOit94K++0HW26Z32uXlmau8r3/Ptx5J5x6assMl5LRL/CuSqKZLl4MtbXZu5zjOBly8cUwa5b96N9yS3j99fYgSAHKgCcS1D8etEUm8r6iKp9hktrJFv/+twm0K69M3zfblJaaFV4m/O1v0L07nJfl/wadO1vwhwS+pjH3mIULsxOX33GKiupqhp15JjzzTHYTWmSTZ56BKYEHSceOppGWZSRnipnPgP2AD+Pqfxy0RSZyBCQReoiwtwiHiXBEuGRyQSfELbeYECnEwX2mUZDeeAPuv9+MDmLbstkkRUhBcCMkp50yfjzrv/12dkOGZov58+Hoo+Ggg2xbF0yYXnVVdq8j0geRGYjUIrIQkYR5roPACqsRqQmVQaF2DeaItUVJs3ktcB0iNyLyy6DcBPwzaItMJM1UhB8D04CNEjRn5NjqBCxaBA8/DL/7HXTtmv/rl5XBsmVW1l8/ff8LL7RoR2edlZv1JAkp6L6mTruluhpuvRVRzU26xZbyxRdw6aV2BNWlixlGxvzKV63KxVqb5blG5E1UE4WQvQ/VX6SYaydU47XM5KjegMgizC4oJsTfBY5H9aHI8xBdM52I7StvpkqHuOKCtCXcdZe5mPz2t4W5fiYWvS+9BI8/bhFONtggN+tJElJw443t8+yaqdPuOP10E05gwqrQ2ul338G558JWW5nAPO0000w7xImJbCbfEInlub4Q1RpUZ0NDnuv8oPoAqnuiun5Q9sxUkEJ0YToQGK/Kl5lewEmAqp1B7L134Q4CMxGm559vmuMZZ+RuPUm2eTt0MO3UhanTrqiuhkceaXy/erVpgl99lf+1LF8OEybAoEG2hXvUUfDeexYa8M03GwV+jAxd4fpCJ0ReDZWwm+VgYA2q8Xmuk2XPOBSRbxGpROTUBO2zAn/R6YgMjLzILBDVAGkOsA3wUQ7XUtRssWpV62PQxpgzxyxjb7ut9XO1lKj7p//5D5SXwz//CT175m49/ftbepiVKy3sUQj3NXXaHSef3Nxfu64Ojj/ePnP5YPVq+w665BIT7v/zP/D3v8OOOzb2yUKSjcUmLHdL0twLiM8LtQxYL0Hf+7E47F9jgYQeQmQpqtOC9tHAXKAHFoP9cUSGodo0XrzIt8BgVBcj8h0p4s6j2ifVvYWJKkxvAq4OMri8DTQJgaPK61Ev2Fbpqdr6GLQxpkyB9dazLZRCUVJiVrSpNFNV00o339w+/LleD5h2GmcpOHAgPPZYbi/vOHlDNbnAfO45uOgic0VpbTamRFRXWxzdX/wCrr4aPvjAdsjuuw9Gjcr+9dITPc+1alXo3YuITASOwux5QHVW0LYKkTMxIb0tzaP3nR2a/2xSCdMMiCpMHwz+Tk7Qtu4YIGXj4H3pUnjgATjhhNxqeuno0MGEZCph+thj8PLLJvzjtMWsEw4pGCdMy8pMxq5YYZ45jtOmefJJ21qdPBlOOomKigoLfrJqlflwX3qpBUa59dbsGieqwrhx8MILVoYOhUcfhf/939wI7mjMx7aBt0b1g6Auav7qdLmoE7er3hp6HcXiNxJRz0y3SFEGpRjXvsjGwfu995pUyFdQ+1Sk2j+tr7cfDlttBSeemPu1pAgpGHOPaW0+c8cpOPX1cMEFFvhg7NimbV26mLvc3/8OU6fC/vvDkiWtv6YqPP007L67GRKC7Uo98wwcemghBSmoNuS5RqQnIsnzXIv8FJENERFEdgfOAB4J2oYiMgyRjoj0Aq7BQgS+m/L6IvMRab6VK7IBIhml5owaAWlhqpLJBds0q1bZGUNLjQRihkfDhsEuu2R3bS0hla/p/ffDW2/ZeUrnrGUpSk6akILgRkhOO+Chh8xn++KLE3+uRCwoyr/+Bf/9L+y1F3wY3dOjCaq2u7THHnDwwfDuu435i0VMaBcHpwHdsTzX04BTUa1EZBQiNaF+P8eCK/wA3AVMQPXOoK0EuA/b2v0YM5r9X1TTJVXdisQ7tF3JVgSkIBjDY6qsTheYIS41W/umrs7ONG6+OX3feF5/3T5IkyYV9tdgjNJS+PJLM0QIf7DXrLFoRzvsYPE480GSzDHgvqZOO2HtWvtcbbcdHHts6r7HHGMhO3/6U9hzT/NJHzky2nXq6y012mWX2ffNFlvAP/5hO02x1Im58RdtGaoJ81yj2iTPNarJH5rqc5iRbDREfhJ6dyAiy0LvO2JRkRZEno/UZ6YPAhtjvxYeTNFv3TkzBfu1969/wfXXZ66xTZlih37HJQ7wkXfKyuyD98UXjeofWPzdDz6wD3C8j1mu6N7djLISaKabbmp+466ZOm2aqVPN5eTBBxs1xFSMGAFz55qV7X77wR13pBbCa9eaPcZll0FlpaVGu+MO+74588zm1sOxY6tsGFW2PR4O/ipwZ1zbWiwb2h8zmTDpN2UQkOGb0OtkZZ0QpO907WqCdNIkc+EYO7b5f85U1NbaeenPfpa7wAeZksjXtK7OtnZ33x1+8pPE43JFkihIHTuarZRrpk6bZdUq29rdZZfMAsRvtZUFTdlzTxOKl11m30PV1TB6tB05rVljQWCGDjVhq2rfNe++a/YOnTvbHK30F21ndAa6AF8Cmwbvrah2RnVLVB/NaEZVzVsB7QM6A7QWdCHocUn6CegE0CVBmQAqofbJoO+D1oOOTTD+j6BfgX4Pehto11DbQNBy0OWg74H+OMrau3btqg1ccYUqqJ58smp9vUbitttszAsvROufD957z9Z0992NdRMnWt3MmWmHl5eXZ3c9e++tus8+zarvuUe1a1dbVlmZvY/CPfdYf5F8j6tvI+ss7ufSrrjpJvsP/OSTzZoifY5WrlT95S9tjrFjVceNU+3QQXXMGNVBg6x+p51UH3hAde3a7K+/FQC1mkc5U6iSiSDcDPQ40D+A/ilcMphjGuh9oL1AR4IuAx2aoN/JgbDcDHQAaBXoKaH200H3A301XpiCHgj6NehQ0A1BK0CvDLW/BHotaHfQI0GXgvZLt/YmwlRV9bzz7PGdfXY0gbr33qpDhkQXvvlg+XK7h8sus/c1Nar9+9sHNMI6sy5MDz9cdejQJlX33KPao4ctM1Z69Ej/hezj2va4dsWKFaoDBth3QILPVeTPUX296sUX20MUaXygO+2k+sgjxfXdEqLohSmsr3C0wlkKf21SMpgnaqD744HbgDXAIpo6uSoRouuLEIvBuL0qNcBskYYYjOfGdT8RuEaVz4Ox1wAnYcEjUGVSUL8ywaVOBG5VNT8lEcYDU4FzRRgM7AIcoMoKLOH5H4J13ZTuHppw+eW23fuPf1ig+PPPT963stK2U66+ujgMj2J07265RGPbvP/3f2YANGNGYdZZUmL+byHOP99c8sIsX27ueK+nCBUyZYqPK+Zx559vwX7WCW680ewS7rmndZ8rETN+nDXLgjuAGRPsvXf+j2TaCyLDgaeAeqAPUI3ZCq3EUrBFNnkW++GQ7np8hJkdX6jK2hYsGRF2Buao0iNUdxYwWpVD4/ouwwTey8H73YBy1aYhpkSYDdyi2pgNXYQ3gb+rcl/wvi/2A6Av8KOgbdtQ/+sxBf33CdY8DhgH0KlT911nznyqaYf6eoZceSUbz5zJB7/7HV8ceWTCe99y0iQGPPwwLz3wAKuL5bw0YNeTT2bVBhvw7oUXssexx/L99tvz9hVXRBpbU1NDr1690neMyMDbb6fs7ruZNXMmGhho7LvvaFQTfQEp3bsn/6+4YkVHEvtz+7hiGCeiPPfc80nHtRc6rljBHscdR82WW/LW1Vcn7JPJ56jLkiXscdxxdAydf67t2pWX772XVX0iR77LK/vss89yVS1ghJoUiMwC3gJ+j7nV7ASswFx0bkL1X5HniqK+gtaADmqNKg06CvSruLqTQCsS9F0LOiT0futgR0Pi+s1OsM37EehBofedg7EDQX8JOjeu/+Wgd6Rbf7Nt3hirV9v2JKjefnvz9pUrVTfaSPVnP0s8vtAccYTqttuqXnCB3cO8eZGHZn2bd9IkW8OXXzZUlZU17maFS1lZ6ql8XNse1264/HK74blzk3bJ6HN06qmqXbo0fZhduqiedlrr15ojKOZtXlimsE3weqnCtsHr3RXmZzJXVL+HJ7HAwq0hegzG5n17AzV2jxlfJ/b6hwzXEI1OnWDaNItW8pvfmNl7mBkzLIpJoVKtpaO0FD7+2By4Dz3UAkoUigS+ppdfDj16NO3Wo4fVp8LHte1x7YKlS+0Y6NBDLXBCNnCr3GyzisZjy6+BwMWB74HNMpopisQNNMhPQS8DPQb0iHCJOEdP0FWgW4fq7iJkHBSqfxH0pND7X8drlEF9Is30XtDLQ+/3jWnEoINBV4KuF2qfRci4KVlJqpnGqKlRHTFCtXNn1aeeaqzfbz/7GV5kFnYNXHtt4y/cY4/NaGjWNdNZs2wdzzzTpLq9W622vfvLfNzmm6tCvfbuvQ4ZH8V2e954I2W3rH+OigyKWzN9VuG44PVkhZcVjld4WqGZzElVogrT+hRlbeSLof/CLHp7go4guTXvKaDvYpa8m4JW0tSatwtoN9A5gaDvBtohaDsIc4vZDnQD0Odoas07F/TqYMzhtNSaNxHffae6886q3bubYHjxRXvEZ5+dfmyhuOWWRmHavbtqdXXkoVn/Enj/fVvHXXdld948096/HFvKDjt8p7vvXuhV5IlvvlHt1Uv1mGPSdm3v/1+KXJjurrBf8Lq/wkyF5QpvKOyUyVxRY/NmK2hDsxiMqlSKMEqEcAzGm4HHsNQ57wBPBHUxnsUOiffGMtmswIyLUOVp4CqgHItisRC4KDT258BuwHfAlcBRqizK4B6Ss8EGFjy6rMyilsTMFRMEIigaZs9ujHKUjUD+rSFFSEGn7bPTTst47TX4oeWHKm2HK680s+WLLy70SpxUqP4X1f8Er79BdX9Ue6A6DNU3M5kqrTAVobMIL4tkEPcwCap8q8phqvRUpVSVe4P6F1QbYzAGgv4cVfoE5Rz74dDQPkYViSsVofZrVSlRpbcqv1KlLtS2IBjfXZVtVPl3a++rCf36wcyZ5i7zySdW98ADLQ+On0uqqy00YiySUyxeZ6HW2ru3pZwq5h8fTovZaaelrF0Lc+YUeiU55osv4IYbLM3ikCGFXo2TJ9IKU1VWY6nWohj/OGABqn/0o0afskJrfMkYPz55vM5CIJI0pKDT9hk6dBmdOsHz7d0j5vLL7XN00UXp+zr5R+SDIPVa+pIBUa1578SCJjhRqK6G6dPtJBIKr/EloxgtA0tKXJi2U7p3r2f48HYuTD/5xCJV/Pa3TZNHOMXELcCtQZmGpW/7Akvo8iDweVA3LZNJI0VAAnoCx4uwP/AaUBtuVOWMTC7a7kml8RVThoZ58wq9guaUlFhaOKddMnq0BQKrrYWexejGX11taQfvu69lqckuvdTc5S64IPtrc7KD6oSG1yK3A1ej2nQ7TuQCYOtMpo2qmW4LvI4Z7QwCdgiV7TO54DpBMWp8bQXXTNs1o0dbkpOi/SiMH29GeS056njvPcvecvrpljfQaQscQWIN9D7g8EwmiqSZqrJPJpOu8xSjxtdW6N/frHnr6/OXS9WJTis1txEjLKXe889bnJOi4ssvYfJk+783ebKlS9t7b0uDli53cXW12Ul06wZ/+Ut+1utkg5gnyIdx9aOA5c27JyfqNq/j5IeSElNdli6FIo01uk4T1txacGSx3nomo4ry3PT00+04Buz/YCxqWadOMHgwbLcdbLut/d1uO6vr1s36nHkmLFoEu+5qFv1OW2EicAMiuwBzg7o9gV8Dl2UyUWRhKsI+wLFYuKUu4TZV9s3koo6TlJiv6ddfuzAtNqqrGzW3226DCy9skXY6ZgxMnAgrVljioqKguhoejcsF3bWrhQP88ktLtP3mm2ZYGLOH6NABBg2yMnOm1VVWmqFhS85bnfyjegUiC4EzgROC2neB36J6byZTRdpHE2EslqZmPWAMloVlQyydWVUmF3SclPTvb3/93LT4OPfcRs1t5Uo4+eQWTTN6tJkQzJ2bvm/eOOec5kaDqnYOesUV8PDDMH++WU699Zb5Z194Iey8M7zySqPlfn19cbrBOclRvRfVPVDtHZQ9MhWkEN0A6Szgd6ocC6wGzlNlZ+AeaBK5yHFaR1gzdYqH6mq4N+775dFH4c9/bhSwERk50pS6iorsLa/VPPNM87pERoPdusEOO8Axx1h0o5iKHR5TjG5wxYxIH0RmIFKLyEJEjkvS72JEViNSEyqDQu3DEHkNkeXB37xm7YgqTAdBQ6SgOmiIVnQ9MDbLa3LWZTykYHFy6aV2jhimQwe49lqzJKqujjzV+utbcqKiOTetrYXVq+FnP2ueLS6dMWGxBT5pm0zCsreUAMcDNyIyNEnf+1DtFSofAyDSBXgEU/A2xGIjPBLUN0XkW0T6Bq+/C94nLhkQ9cx0CTQk5v4Cc4d5C9gIi7XrONlho43sS9o10+IidiYYpr7eUvjNnWvbnVOnwn77RZpuzBizX1q5stGGp2DcfbcZvJ15ZuZj3Q2udYj0BI4Etke1BpiNyKPAL4FzM5hpDCbP/hlkNLkOkbOAfYGn4/qeTWPazbNasfomRNVMXwAOCF7fD1wnwu2Yf06CT5njtJAOHcwa0oVpcbHbbmYQtmJFU81t4UI7M+zTxzTUiy6KtO07ejTU1cF//5uHtadCFa67zqxw99478/Hz5iXKfe7ucSH6QidEXg2VcaHmwcAaVMOh+94EkmmmhwZaYyUip4bqhwJvBYI0xlsJ51G9FdW60OvkJQOiaqa/A2K/H68A1gAjMMGakfmw46SlpMS3eYuJxYstyf2ppyZWI4cONYF6+um2HfzCC6albrJJ0ilHjbJQzBUV5p5ZMGbONEvdu+5qjKXtZJXFJix3S9LcC0vEHWYZjTuhYe7HsoR9DewBPITIUlSnBfMsizhPTogatOHb0Ot6YEKK7o7TOjwKUnFx1122dRnzu0xEz55wxx22f3vaaWm3fTfcEHbcsQjOTSdOtP9vRx9d4IWss9QAvePqetO4DduIathz5EVEJgJHYTuk0ecR+Y6oiVtUI/vnZeJnWoLtY28JXKjKYhFGAF+q8knUeRwnLf37wwcfFHoVDtiW5ZQpsOeesH2EyKFjx8Lw4WbMs//+5j7yt7/BN98w7MwzzWo28MEcM8bcVletgi7NzURyz/z58OSTZpXbtWsBFuAA87Ft4K1RjX3odwIqI4xVILadUAn8GREJbfXuiBk3xZO1c9IwkYSpCLsC/wE+wfag/wEsBvbH9rwTmzI7TkuIaaaqvvVWaObMMV/LWzM4Pkq07bv55qz/9ttNIieNHm2K4SuvWJjBvPN//2dS/JRTCnBxBwDVWkSmA5ci8ltgGPBToPkBtshPgVnAUmA4cAbw16C1AlgLnIHITTRmOXsuwTUzOguNSlQDpKuBiYFvaV2o/hns7NRxskdJiRm61Nam7+vklilTLAbgMcdkNi627Xv77WbxetddiGoTH8xRo6xrQfxNly2z9f38543uWE6hOA3zCvkG27I9FdVKREYhEo5j8HMshu4PwF3ABFTvBEB1FXAYFsVoKRYO8LCgPi9EFaa7Yn478VRjvkGOkz08cENxsHQpPPAAHHdcy/OljR0LP/1p4/uQD2bfvrZzXJBz09tug5qalrnDONlF9VtUD0O1J6qlDdGHVF9AtVeo37GobhT4lw5B9bq4eeahuiuq3VHdBdX0JtUinRG5EJGqIAjEqiYlA6IK0xWYI2w8Q7BfE5EQoY8IM0SoFWGhSOLtYRFEhAkiLAnKBJGGvXFEGCbCayIsD/4OC7U9JUJNqKwS4e1Q+wIRVoTan426fidPeEjB4uDee22H4KST0vdNRnU1PPJI4/u4CEGjR5tL5urVrVxrJqxda+4wI0da1H1nXeZSbINbq3QAAB9OSURBVEt4EtAROB9LHr4Mi9cbmajC9BHgIhFip/QqwkDMqvehDK7XLNKFSEJ/onGYyr4Tdoh8KHAygAgJI10E9ahysCq9YgV4EXggbv5DQ30OwCkuXDMtPDHDo513Nh/MlpIoQtCqVQ3a6Zgxtpv/2mstv0TGPPYYLFjgWqkDcAxwMqqTMJfP6aieBlwCmaUezSQ2bx8swH0PYDa2d70MiJRSXoRYpIsLValRZTYQi3QRz4nANap8rsoXwDU0hi0cQxDpQpU6Va7DLLqaZa4JBP4obH/daSt4SMHC89pr8MYbqd1hopAoQtDatfBvi04a8zHN61bvxIkWuemww/J4UadI2ZhGy+EaYIPg9ZPAgZlMFNXP9HtgpAj7YpliOgCvqzbE643CYGCNKvGRLkYn6Ds0aAv3Gxpqe0uVRJEu4sNGnQC8oMqCuPqpInQA5gFnqza5VgMijMO0ZDp1EiqKKjJ34ampqcnJM5HVqxkNfDJ3Lgu32Sbr8+eaXD2XfDL4mmso6dqVF0tLWduae/l//6/hZU1NDRuosttJJ6E1Nbz65JOs7dGDsrLhTJ++kj32eDvFRNmh50cfMbyigo/GjeOz2bNzfr0otIf/L22Yz4BNgE+BjzAPldeA3YGVGc2kqi0uoGWg90fsOwr0q7i6k0ArEvRdCzok9H7rIE6XgF4I+q+4/lNBL04wz4egY+PqRoB2B+0Beh7oV6AbpFt/165d1WlKeXl57ibfcEPV007L3fw5JKfPJR/88INqr16qJ56Y1WkbnsusWaodOjTMf8opdrnVq7N6ucT8+teqPXqoLlmSh4tFo83/f0kDUKutkDM5LfAPhQuC18corFb4QKFO4cpM5oq6zZuMDbCt2yhEj1DRvG9voCbQRiPNI8JITIV/MFyvyhxVVqiyXJUrMDPqURHvwckXHlKwcNx/v1m6tsbwKBWjRsEFF8Cdd8K0aYwZY5fLeTjbRYssKtMJJ3jiecdQPRvVy4LX92HnpFOAY1DNJNB+q4VpJswHOomwdaguWaSLyqAtUb9KYMewdS9mpBQ/z4nAdNW0+VbDUTScYsFDChaOKVNg221bFvg9KhdeCHvtBaecwr6DFgB5ODedPNmi659xRo4v5BQ9Ij9OWK86G9WrUH040ynzJkxVqQWmA5eK0DMIRfhT4O4E3e8C/iTCABE2Bf4M3BG0VRBEuhChqwi/C+obIl2I0B04OjQmVl8qwggRuojQTYSzgb7AnCzdppMt+vd3YVoI3nnHUqqddFJuo0916mRaItDvD8ez7dZrchu8YfVquOEGOOAA+6HgrOs8i8jHiJyPyIBsTJhPzRQSRLpQpVKEUSJNNMibgceAt4F3gCeCOlRJGOkiqI9xWNBWHnf99YAbge+wvKwHAQersiSbN+lkAddMC8OUKRZi75eJjOyzzBZbwE03wYsvMmG9y3jhhUjZ21rGgw/Cl1+6O4wTYyim3P0eWIDIE4gcjkjHlk6Y0ppXhEfTjI8/u0yJWvaZZvboqryApdCJvVfgnKAkmmceFpUp2XWmYcI6vr4S2xJ2ip2SEgv5VlfnQcjzxcqVlij78MMtPFE+OPZYePpp/ufu8Wyv+/PmmyNyE0dh4kQYPBgOOigHkzttDtV3gbMQORf4CaaU3Q8sQeRO4DZU389kynSa6ZI05RPch9PJBe5rmn+mT4fvvsud4VEyrr+e+tKBTOV45j69NPvzz50LL78Mv/+9JZ93nBiqa1Cdjur/AmXAdcARQBUiszKZKqVmqsqvWr5Kx2kF4ZCCm29e2LWsK0yZAoMGwT4ZBX5pPeutR6f77mWzPUeww6RT4Lxp2T2vnTgReveGE0/M3pxO+0P1S0RuwDxDLv7/7d15dFXVvcDx74+EMTgUFByQ4ACIKMShaFUMVatWRBxoFxI7PBScUOvqwqFOCFLF1becy6CAVBB5dUFB9NE+n8Si0D6xKAJBxEIQSJRBgTCFJL/3xz6Bk5t7bxJu7j0n9/4+a52Ve/c5+9x9D4f8svfZAw1cxMX+TDPhZDXT1PryS7d8y623BlN7O/985p47mr6bZ1H1WiM2dm3a5J6X3nKLW/3GmGhELkfkDWAzbirBN4HzGnIKC6YmnGx+3tR69VXIynKrvARkz4gHKCQfvesuWLu2cU76xz+6Xk0jRtR9rMksIp0ReRyRdcDfcDMhDQdOQPWueq0642PB1ISTrRyTOuXlbm3Pa66B448PrBiX/DiLX/A6B6SF65gUOadvQ+3dCxMnwrXXuuZrY6qJvAf8G7eAyptAN1R/jOp0VBs2jaDHgqkJp5wct1kwTb75811zeqo7HkXIzYXsLifx4lmvwNKl8PjjiZ3wjTdg2zYbDmOi2Y3raHQSqg+hmnBTiAVTE142pWBqvPIKdOoUimEj+fkwbu2N6K3DYNw4tzh5fv7B9U/rbfNmN9NRjx5unTdj/FQHojoP1UYb2WzB1ISXTdyQfMXF8Ne/wtCh7plpwPLzXWVy1bBn3bjQX/8aFi2C0aMbdqI77oA9e9w9lMyZnIzx1GsJNmMC0aEDfPVV0KVIb1OmuJ9DhwZbDk91JbLw4xx6vvACXOktKTl+vOsk1aYNtGrlttato/8EtwA4uPGlpaVw3HEp/y4ms1gwNeHVsSMsXhx0KdJXZaULplde6R5YhkCXLm5Y8QcfwF0r/wLNm7t5dbOy4Mwz3Wri+/a5zkWRP3ftcj+//tqt2AjuO44ZAy+/HOj3MunPgqkJr44dYetW9wsxBE2QaWfBAti4EZ57LuiSHCTimno//e8SdPdU5MABt6OyElavhnffjV/LLCmp2XO3vBymTnWr1FjtNJxE2gGTgSuArcBDqL4R5/gWwGfAEah28qUrsAe3EhjAm6jemqRS12LPTE14dejgahhbtwZdkvRTUuJmBGrfHgYMCLo0NeTnwx3bxqCVVTV3VNcy4xkzBqoOI58J0stAOdARKADGI9IzzvEjgS0x9vVGta23pSyQggVTE2Y2cUPyPPig6+lz0klulZgQ6dcPfsQSmh2IGGdaXl53s/+SJbXHp9YnnwmGSA5wI/AoqmWofgjMA6IvWyRyMnAz8FTKylhPFkxNeNmUgnUrKan/0JGqKtd79403Dq4lSlFRw4edJNmpp8I1JyzjpsHqWib827I6JqVZtqx2nvrkM0lzDGQjstS3Dfft7gZUoLrGl/YZbom0aF4EfgfsjbH/74iUIjIbkS6Jlr0h7JmpCS+rmdZtzBj48MOanWwOHHC9oIuKYNUq97OoyD1z3LOnZn7V0HXQqX5uunChK56NbGnatrpgGWue27bAzoi0Hbi1p2sSuR7IQnUOIv2inCsf+AfQBngSmI9IHqoVh1v2hrBgasLLphSMr6TEda6pqoJJk2D9eli3zk1aX+H7/XHSSXDGGa4n7AknwCOPHGoKDWkHnfx8mDnTfZVu3YIujUmiMmqvi30kbuWWQ1xz8DPA1THPpFq9ZFo5IvfignQP4PNGKmtcFkxNeB19tHueZ8E0utGj3eLp4ILnRx+5B44DB7rg2aMHnH46tG17KM+dd9Y+TwiHj+Tnu58ffGDBNM2twTUDd0X1Sy+tN7Ay4riuQBdgkddU0QI4CpFS4AJU10c5twIpa9ewYGrCS8TVTu2ZaW0lJTB58qHxlOBqmRMmxK9hNpEOOt27u1b+Dz4IfMpgk0yquxGZDYxG5FYgDxgIXBhx5ArAv7DxhcBLwDnAFq/3b3NcLbQ1rpl3E1CU3C9wiHVAMuFmUwpG9/DD7tmoX32GgDSRDjrVz00LC2v+vWDS0p24APgtMBO4A9WViPRFpAwA1QpUSw9usB2o8t5X4obVzMI17f4bV4u9BtUDtT8uOVIaTEVoJ8IcEXaLUCzCkBjHiQjjRNjmbeNEDlXXRcgT4RMR9ng/83z7RolwQIQy33ZKffKaEOrQwYJpNNXT5fmFsIaZiNat3dreWVluZqTqDsh1mTHDHd+sWdPId+ml+Q3Kl3ZUt6N6Hao5qHY+OGGD6iJU28bIU1hjwgbV91Ht7p2jg3e+L6PmTZJUN/P6B+fmAe+I8Jlqrfbx4cB1uLZzBf4HWAdMEKEFMBd4Dvgjbj26uSJ0VaW6/WqWKjdHfng985ow6dgRPk9J/4GmY9ky2L4d7r4bXngh6NIkxYwZMGuWe63qRvQMGwZlZTBoUOx8b70F993nZhWEppJPKC6G4d6AkYKC2PlMiKlqSjbQHNBy0G6+tNdBn45y7GLQ4b73t4D+w3t9BegmUPHt3wB6lfd6FOj0GGWImzfe1rJlSzU1LVy4MPkf8sADqi1aqFZVJf+zGklSr0tlpeqPfqR67LGq332XvM9JgoZcl9zcaG3R6b/l5ibr6gcH2K0pijNBbqmsmXYDKlSJHJybH+XYnt4+/3E9ffuWqx6cfxFguZe+wHs/QITtQAnwkirjG5D3IBGG42rJZGcLhYWFdX3HjFJWVpb0a9Jp505OKy/nw/nzqTii9tCzMErmdTluwQJOX7KE1fffT+mnnyblM5KlIddlw4Z8onfEVO6+O/Y6zi++eFqTzrdhg1JY+EHMfCbEUhW1QfuClkakDQMtjHJsJejpvvddvb/cBPRR0Dcjjp8BOsp7fQboCaBZoBeCloDe5O2LmzfeZjXT2lJSM50+3f3JXlSU/M9qJEm7Lt99p9qhg+oFF7gaahPTGDXTumpu6Z6vKSJDaqap7IBUv8G50Y89EihTV6OMex5VVqmyWZVKVRYDzwPVTy8aUgYTBjal4CGPPw5btrjxoM3SuyP+2LFu6VK/Nm1ceibnM+GVyv+Ra4BsEbr60qINzsVL6x3juJVAL3/vXqBXjPMANQbuNjSvCZpNKegsXw4vvQS33w7nnBN0aZKuoMBN6pSb64bJ5Oa693V1zmmK+fCeOg0aZJ2PmrRUVoNB3wSdieuMdBHoDtCeUY67HbQI9ESvyXYl6O3evhagxaD3grYEHeG9b+HtHwj6A69JuA+uw9Gv6pM33mbNvLWlpJm3tNS1f734YvI/q5E0+nWpqlK9+GLV9u1Vt21r3HOnUErulybo/fcXat++qieeqLpvX9ClaXxYM29S1Bqcq8pKEfqKUOY7biLwNm42ixXAO14a6oawXAf8EvgeGApcp4eGtgwG1uKabv8EjFNlWj3zmrA55hjXpJnJNdMZM9xk9k8/De3aBV0a08hEXAv+pk0wZUrQpTGHK6XjTFXZjgtmkemLcKsHVL9X4H5vi3aeZcC5MfbdVEcZYuY1IZSV5QJqpj4z3bkTRo6EPn1g6NCgS2OS5NJL4cIL4amn3D9zy5ZBl8g0VHr3YjDpIZNnQXriCffdM6DTUSarrp1+/TVMmxZ0aczhsP+dJvwydX7elSvh+efdlDrnxVoO0qSLn/wEzj8ffv/72msRmPCzYGrCr2PHzGvmVYURI+Coo2y8RIaorp0WF8PrrwddGtNQFkxN+GVizXTWLLdkytix7pmxyQhXXeUaIcaOrb0okAk3C6Ym/Dp0gN273ZYJdu2C3/7WjSe1xTwzigg89hisW5fBq8g0URZMTfhl2sQNY8bA5s2u01FWVtClMSl2zTVw9tnw5JNQURF0aUx9WTA14ZdJUwquXg3PPuvGR1xwQdClMQGorp1+9RXMnBl0aUx9WTA14dehg/t5ODXTkhLIz4fS0sYtUzKoujVK27Z1EzSYjDVwIPTu7WqnlZVBl8bUhwVTE36JNPOOGeNmDxozpnHL1NhKSqBnT3jvPfcb9Nhjgy6RCZAIPPoorFlzaJF0E24WTE34VddMG9rMu2kTTJ4MVVXw6quu3SysHnsMioqgfXs3mb3JeNdfD2ee6f4OtNpp+FkwNeHXsiUcfXT9a6b79sErr7jfRNWj38vLoWtXuOQSGD0aPvooPGMP1q6FqVPd67Iyt8yayXjNmrna6erV8NZbQZfG1MWCqWka6jOl4JYtLlB27gzDh8OOHTX3Z2W5tFGj4OKL3aTxAwbAc8/BihXumWW1VD1r/eQTOPfcQ1UP1fA3SZuUGTQIzjjD3RJVVUGXJklE2iEyB5HdiBQjMqSO41sgUoTIxoj0PEQ+QWSP9zMvmcWOZMHUNA3xJm744gvXNNq5s5tCpk8fuPZaaN685nHNmrkgunWr+1P/5ptd3vvug7POguOPdwtKTp0K99+f3GetVVXwhz+4Hrs7dx5KLy93n98UOkyZpGvWDB55xM0sOXt20KVJmpeBcqAjUACMR6RnnONHAjWbb0RaAHOB6cAPgGnAXC89NYJeA66pbLaeaW0pXZ+yf3/V1q1VS0rc+6oq1cJC1QEDVEG1ZUvVYcNUV61y+/PyXHrklpdX+9zr16tOnqw6ZIhqhw41j2/RQnXjxgYVtc7rUlKiesUV7vwnn+w+I/Iz77yzQZ/ZFNh6ptHVdV0qKlS7d1ft1Uu1sjI1ZWpMxFvPFHIUyhW6+dJeV3g6xvEnKxQp/FRhoy/9CoVNCuJL26BwVczPbuTNaqamaSguhr17XRPtzJnwwx9Cv36wZImrjW7YAJMmQY8e7vhly6KFUpceKTfXjeucMcPVCH/+80OTJZSXu2evs2Y1Tjvbu+9Cr16waBFMmODm3o2c1by8HBYvTvyzTFrIynK10+XLYe7coEvTcMdANiJLfdtw3+5uQAWqa3xpnwGxaqYvAr8D9kak9wSW44J3teVxztPoLJia8Cspcb0wwAXMIUNcR52JE10QHTXqUI/fRJWWwrx5NbtP7tgBgwe7gX9vvXV4QXXfPvjNb6B/f9ecvHQp3HZbw4K+yViDB8Npp7kuATXCRROw1QXL83zbJN/utsDOiCw7gCNqnUjkeiAL1TlRPqatl6/u8ySJBVMTfpHPLa++Glatcp2MWrdu/M+KDJbNm8Pll7vevz/7mZvrbc6c+v9WKypyz0affx7uuQf++U/Xq8SYesrOdrXTTz+Ft98OujSNqgw4MiLtSGBXjRSRHOAZ4J6EzpNEFkxNuJWUuA451ZOUqsLChcmbWnDJkujNrlu3ul4g06e75uYbbnAT0c+bFzuoqrqa9LnnujGv8+e7gNqqVXLKbtJaQQGcckrTrJ3GsQbXDNzVl9YbWBlxXFegC7AIkVJgNnA8IqWIdPGO74WI+PL0inKepElpMBWhnQhzRNgtQrEIUbtAiyAijBNhm7eNE0F8+/NE+ESEPd7PPN++kSKsEGGXCOtEGBlx7vUi7BWhzNv+lrxvbBIWraZYWZm8Xrbxml2zstxvtFWrYNo0t7rLwIHu+e0777jjSkrIu/deVxsdNMg15V50kXvg1b9/cspsMkJ2Njz8sBtN1bGj6+nbpUv9V5eZMcMdn4p8/jxxm49Ud+MC42hEchC5CBgIRK7ougI4CcjztluBb7zXXwOFQCVwDyItERnh5Xu/Xl+yMaSqp5N7LqwzQWeBtgW9GHQHaM8ox90G+gVoJ9ATQVeB3u7tawFaDHofaEvQe7z3Lbz994OeA5oN2t3bN9h37vWglze07Nabt7aU9M5sSK/cVCsvV50yxfXIBdU+fVSvvlqrRFRzclSzs1WfeaZpdsFMAuvNG11Drsu0aaoiNf8rtGmjOn16/HzTp7vjUpGvdp42qvF+v0I7hb8o7PZ64A7x0vsqlMXI069Gb16XdrbCJwp7Ff6lcHbcz23kTTRF7QUi5ADfAWeqssZLex3YpMqDEccuBl5TZZL3/hZgmCoXiHAFMBXo5K4fiLABGK7Kgiif+wIgqtztvV8P3KrKew0pf6tWrXTfvn0N+s7prrCwkH79+gVdjOAdOOBqqqNGueZccJOrvvuuW+3ZAHa/xNKQ69Kli+vYHik7G7p1i51vzZroy7klI1/tPDmo7pboR6eP7BR+VjegojqQej4D8qMc29Pb5z+up2/f8upA6qnuAl0jmHpNw32BiRHnnyFCM2AZMFK1xmf58w8HhgNkZwuFhYUxv1wmKisrs2tS7bTT6Hr22RxfWkqzykqqsrIomTCBL+356EF2v0TXkOuyYUM+UDsuVVQoxx4bexrKVauOTVm+WHnSXqqqwKB9QUsj0oaBFkY5thL0dN/7rl6TgYA+CvpmxPEzQEdFOc8ToJ+BtvSlXQTaGrQN6EOgpaBH11V+a+atzZrtfDZvVm3VqmZ7mH+SCWP3SwwNuS65ubWfeIBLD0u+2nnqaOZNky2VHZAa0nU58tgjgTJ1tdF6nUeEEcAvgf6q7K9OV+UjVfaqskeVp4DvcbVXYw5fqjtKmYw0diy0aVMzrU0blx6WfNHyZIJUBtM1QLYIdXWBxkvrHeO4lUAvf+9eIrpAizAUeBC4TJWakyHXpmRkm4RpVLGG1NhMRqYRFRS40Va5ue6xfG6ue19QEJ58kXnSaiBPPKmsBoO+6fXozfGaW2P15r0dtMjryXsC6MoovXnv9XrzjojozVvgNd32iHLezt7ntgBtBToSdAto+7rKbs28tVmzXXR2XaKz6xJdul8X4s3Nm0ZbqidtuBNoDXwLzATuUGWlCH1FKPMdNxF4G/gcN77oHS8NVcqB63BNuN8DQ4HrvHSAJ4H2wMe+saQTvH1HAONxvYo3AVcBP1VlW7K+sDHGmPSXyt68qLIdFwgj0xfh5lasfq/A/d4W7TzLgHNj7Ds5zue7WTKMMcaYRmTTCRpjjDEJsmBqjDHGJMiCqTHGGJOglD4zbcr279+vIhK5IG2mywaiTDaW8ey6RGfXJbp0vi57gi5Aqlgwrb9/qep5QRciTERkqV2T2uy6RGfXJTq7LunBmnmNMcaYBFkwNcYYYxJkwbT+JgVdgBCyaxKdXZfo7LpEZ9clDaRsPVNjjDEmXVnN1BhjjEmQBVNjjDEmQRZMjTHGmARZMK2DiLQTkTkisltEikVkSNBlCgMRKRSRfSJS5m1fBF2mVBORESKyVET2i8hrEfsuE5HVIrJHRBaKSG5AxUy5WNdFRLqIiPrumTIReTTAoqaMiLQUkcne75BdIvKpiPzUtz9j75d0YcG0bi8D5UBHoAAYLyI9gy1SaIxQ1bbe1j3owgRgM27Jvyn+RBE5BpgNPAq0A5YCs1JeuuBEvS4+R/vumzEpLFeQsoGvgXzgKOAR4L+8PzAy/X5JCzYDUhwikgPcCJypqmXAhyIyD/gF8GCghTOBU9XZACJyHtDJt+sGYKWq/tnbPwrYKiKnq+rqlBc0xeJcl4ylqruBUb6k+SKyDreUZHsy+H5JF1Yzja8bUKGqa3xpnwFWM3WeEpGtIvKRiPQLujAh0hN3nwAHf5F+hd031YpFZKOITPVqZRlHRDrifr+sxO6XtGDBNL62wM6ItB3AEQGUJWweAE4BTsQNOn9bRE4Ntkih0RZ3n/jZfQNbgR8Cubga2RHAjEBLFAARaY773tO8mqfdL2nAgml8ZcCREWlHArsCKEuoqOo/VXWXqu5X1WnAR8DVQZcrJOy+iUJVy1R1qapWqOo3wAjgChHJmKAhIs2A13H9MEZ4yXa/pAELpvGtAbJFpKsvrTeuacbUpIAEXYiQWIm7T4CDz95Pxe6bSNXTr2XE7yEREWAyrjPjjap6wNtl90sayIib+HB5zy5mA6NFJEdELgIG4v6yzFgicrSIXCkirUQkW0QKgEuABUGXLZW8794KyAKyqq8HMAc4U0Ru9PY/BizPlM4ksa6LiJwvIt1FpJmItAdeAApVNbKJM12NB3oAA1TVvzZyRt8v6cKCad3uBFoD3wIzgTtUNdP/YmyOG/qwBfcc7G7guoiOWpngEWAvrmf3zd7rR1R1C64X+FjgO+B8YHBQhQxA1OuCe8a+ANd8uQLYD9wUUBlTyhs3ehuQB5T6xtkW2P2SHmyie2OMMSZBVjM1xhhjEmTB1BhjjEmQBVNjjDEmQRZMjTHGmARZMDXGGGMSZMHUGGOMSZAFU2MykLeu6KCgy2FMurBgakyKichrXjCL3P4RdNmMMYfH1jM1Jhjv4dbF9SsPoiDGmMRZzdSYYOxX1dKIbTscbIIdISLviMgeESkWkZv9mUXkLBF5T0T2ish2r7Z7VMQxvxKRz0Vkv4h8IyLTIsrQTkT+LCK7ReTfkZ9hjKk/C6bGhNMTwDzcXK6TgD+JyHlwcFWRv+KW7uoDXA9cCEypziwitwETgalAL9zyeCsiPuMxYC5uxZJZwBQR6Zy8r2RM+rK5eY1JMRF5DTcB/L6IXS+r6gMiosCrqjrMl+c9oFRVbxaRYcAfgE6qusvb3w9YCHRV1bUishGYrqoPxiiDAk+r6kPe+2xgJzBcVac34tc1JiPYM1NjgvF3YHhE2ve+10si9i0B+nuve+CW6PIvHr0YqALOEJGdwInA/9ZRhuXVL1S1QkS2AB3qV3xjjJ8FU2OCsUdV1ybhvA1pajoQ8V6xRz/GHBb7j2NMOF0Q5X2R97oIOEtEjvDtvxD3/7lIVb8FNgGXJb2UxhjAaqbGBKWliBwXkVbpLRQNcIOIfAwUAoNwgfF8b98MXAelP4nIY8APcJ2NZvtqu2OBZ0XkG+AdoA1wmar+Z7K+kDGZzIKpMcG4HCiJSNsEdPJejwJuBF4AtgD/oaofA6jqHhG5EngO+D9cR6a5wL3VJ1LV8SJSDvwWGAdsB95N1pcxJtNZb15jQsbrafszVX0r6LIYY+rHnpkaY4wxCbJgaowxxiTImnmNMcaYBFnN1BhjjEmQBVNjjDEmQRZMjTHGmARZMDXGGGMSZMHUGGOMSdD/A4PppocGvww6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.epoch, history.history[\"lr\"], \"bo-\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning Rate\", color='b')\n",
    "plt.tick_params('y', colors='b')\n",
    "plt.gca().set_xlim(0, n_epochs - 1)\n",
    "plt.grid(True)\n",
    "\n",
    "ax2 = plt.gca().twinx()\n",
    "ax2.plot(history.epoch, history.history[\"val_loss\"], \"r^-\")\n",
    "ax2.set_ylabel('Validation Loss', color='r')\n",
    "ax2.tick_params('y', colors='r')\n",
    "\n",
    "plt.title(\"Reduce LR on Plateau\", fontsize=14)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### tf.keras schedulers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/25\n",
      "55000/55000 [==============================] - 4s 77us/sample - loss: 0.4887 - accuracy: 0.8282 - val_loss: 0.4245 - val_accuracy: 0.8526\n",
      "Epoch 2/25\n",
      "55000/55000 [==============================] - 4s 71us/sample - loss: 0.3830 - accuracy: 0.8641 - val_loss: 0.3798 - val_accuracy: 0.8688\n",
      "Epoch 3/25\n",
      "55000/55000 [==============================] - 4s 71us/sample - loss: 0.3491 - accuracy: 0.8758 - val_loss: 0.3650 - val_accuracy: 0.8730\n",
      "Epoch 4/25\n",
      "55000/55000 [==============================] - 4s 78us/sample - loss: 0.3267 - accuracy: 0.8839 - val_loss: 0.3564 - val_accuracy: 0.8746\n",
      "Epoch 5/25\n",
      "55000/55000 [==============================] - 4s 72us/sample - loss: 0.3102 - accuracy: 0.8893 - val_loss: 0.3493 - val_accuracy: 0.8770\n",
      "Epoch 6/25\n",
      "55000/55000 [==============================] - 4s 73us/sample - loss: 0.2969 - accuracy: 0.8939 - val_loss: 0.3400 - val_accuracy: 0.8818\n",
      "Epoch 7/25\n",
      "55000/55000 [==============================] - 4s 77us/sample - loss: 0.2855 - accuracy: 0.8983 - val_loss: 0.3385 - val_accuracy: 0.8830\n",
      "Epoch 8/25\n",
      "55000/55000 [==============================] - 4s 68us/sample - loss: 0.2764 - accuracy: 0.9025 - val_loss: 0.3372 - val_accuracy: 0.8824\n",
      "Epoch 9/25\n",
      "55000/55000 [==============================] - 4s 67us/sample - loss: 0.2684 - accuracy: 0.9039 - val_loss: 0.3337 - val_accuracy: 0.8848\n",
      "Epoch 10/25\n",
      "55000/55000 [==============================] - 4s 73us/sample - loss: 0.2613 - accuracy: 0.9072 - val_loss: 0.3277 - val_accuracy: 0.8862\n",
      "Epoch 11/25\n",
      "55000/55000 [==============================] - 4s 71us/sample - loss: 0.2555 - accuracy: 0.9086 - val_loss: 0.3273 - val_accuracy: 0.8860\n",
      "Epoch 12/25\n",
      "55000/55000 [==============================] - 4s 73us/sample - loss: 0.2500 - accuracy: 0.9111 - val_loss: 0.3244 - val_accuracy: 0.8840\n",
      "Epoch 13/25\n",
      "55000/55000 [==============================] - 4s 73us/sample - loss: 0.2454 - accuracy: 0.9124 - val_loss: 0.3194 - val_accuracy: 0.8904\n",
      "Epoch 14/25\n",
      "55000/55000 [==============================] - 4s 71us/sample - loss: 0.2414 - accuracy: 0.9141 - val_loss: 0.3226 - val_accuracy: 0.8884\n",
      "Epoch 15/25\n",
      "55000/55000 [==============================] - 4s 73us/sample - loss: 0.2378 - accuracy: 0.9160 - val_loss: 0.3233 - val_accuracy: 0.8860\n",
      "Epoch 16/25\n",
      "55000/55000 [==============================] - 4s 69us/sample - loss: 0.2347 - accuracy: 0.9174 - val_loss: 0.3207 - val_accuracy: 0.8904\n",
      "Epoch 17/25\n",
      "55000/55000 [==============================] - 4s 71us/sample - loss: 0.2318 - accuracy: 0.9179 - val_loss: 0.3195 - val_accuracy: 0.8892\n",
      "Epoch 18/25\n",
      "55000/55000 [==============================] - 4s 69us/sample - loss: 0.2293 - accuracy: 0.9193 - val_loss: 0.3184 - val_accuracy: 0.8916\n",
      "Epoch 19/25\n",
      "55000/55000 [==============================] - 4s 67us/sample - loss: 0.2272 - accuracy: 0.9201 - val_loss: 0.3196 - val_accuracy: 0.8886\n",
      "Epoch 20/25\n",
      "55000/55000 [==============================] - 4s 68us/sample - loss: 0.2253 - accuracy: 0.9206 - val_loss: 0.3190 - val_accuracy: 0.8918\n",
      "Epoch 21/25\n",
      "55000/55000 [==============================] - 4s 68us/sample - loss: 0.2235 - accuracy: 0.9214 - val_loss: 0.3176 - val_accuracy: 0.8912\n",
      "Epoch 22/25\n",
      "55000/55000 [==============================] - 4s 69us/sample - loss: 0.2220 - accuracy: 0.9220 - val_loss: 0.3181 - val_accuracy: 0.8900\n",
      "Epoch 23/25\n",
      "55000/55000 [==============================] - 4s 71us/sample - loss: 0.2206 - accuracy: 0.9226 - val_loss: 0.3187 - val_accuracy: 0.8894\n",
      "Epoch 24/25\n",
      "55000/55000 [==============================] - 4s 68us/sample - loss: 0.2193 - accuracy: 0.9231 - val_loss: 0.3168 - val_accuracy: 0.8908\n",
      "Epoch 25/25\n",
      "55000/55000 [==============================] - 4s 68us/sample - loss: 0.2181 - accuracy: 0.9234 - val_loss: 0.3171 - val_accuracy: 0.8898\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "s = 20 * len(X_train) // 32 # number of steps in 20 epochs (batch size = 32)\n",
    "learning_rate = keras.optimizers.schedules.ExponentialDecay(0.01, s, 0.1)\n",
    "optimizer = keras.optimizers.SGD(learning_rate)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "n_epochs = 25\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For piecewise constant scheduling, try this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = keras.optimizers.schedules.PiecewiseConstantDecay(\n",
    "    boundaries=[5. * n_steps_per_epoch, 15. * n_steps_per_epoch], values=[0.01, 0.005, 0.001])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Avoiding Overfitting Through Regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## $\\ell_1$ and $\\ell_2$ regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer = keras.layers.Dense(100, activation=\"elu\",\n",
    "                           kernel_initializer=\"he_normal\",\n",
    "                           kernel_regularizer=keras.regularizers.l2(0.01))\n",
    "# or l1(0.1) for ℓ1 regularization with a factor or 0.1\n",
    "# or l1_l2(0.1, 0.01) for both ℓ1 and ℓ2 regularization, with factors 0.1 and 0.01 respectively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/2\n",
      "55000/55000 [==============================] - 7s 128us/sample - loss: 1.6073 - accuracy: 0.8112 - val_loss: 0.7314 - val_accuracy: 0.8242\n",
      "Epoch 2/2\n",
      "55000/55000 [==============================] - 6s 117us/sample - loss: 0.7193 - accuracy: 0.8256 - val_loss: 0.7029 - val_accuracy: 0.8304\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"elu\",\n",
    "                       kernel_initializer=\"he_normal\",\n",
    "                       kernel_regularizer=keras.regularizers.l2(0.01)),\n",
    "    keras.layers.Dense(100, activation=\"elu\",\n",
    "                       kernel_initializer=\"he_normal\",\n",
    "                       kernel_regularizer=keras.regularizers.l2(0.01)),\n",
    "    keras.layers.Dense(10, activation=\"softmax\",\n",
    "                       kernel_regularizer=keras.regularizers.l2(0.01))\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/2\n",
      "55000/55000 [==============================] - 7s 129us/sample - loss: 1.6597 - accuracy: 0.8128 - val_loss: 0.7630 - val_accuracy: 0.8080\n",
      "Epoch 2/2\n",
      "55000/55000 [==============================] - 7s 124us/sample - loss: 0.7176 - accuracy: 0.8271 - val_loss: 0.6848 - val_accuracy: 0.8360\n"
     ]
    }
   ],
   "source": [
    "from functools import partial\n",
    "\n",
    "RegularizedDense = partial(keras.layers.Dense,\n",
    "                           activation=\"elu\",\n",
    "                           kernel_initializer=\"he_normal\",\n",
    "                           kernel_regularizer=keras.regularizers.l2(0.01))\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    RegularizedDense(300),\n",
    "    RegularizedDense(100),\n",
    "    RegularizedDense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/2\n",
      "55000/55000 [==============================] - 8s 145us/sample - loss: 0.5741 - accuracy: 0.8030 - val_loss: 0.3841 - val_accuracy: 0.8572\n",
      "Epoch 2/2\n",
      "55000/55000 [==============================] - 7s 134us/sample - loss: 0.4218 - accuracy: 0.8469 - val_loss: 0.3534 - val_accuracy: 0.8728\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dropout(rate=0.2),\n",
    "    keras.layers.Dense(300, activation=\"elu\", kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.Dropout(rate=0.2),\n",
    "    keras.layers.Dense(100, activation=\"elu\", kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.Dropout(rate=0.2),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Alpha Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/20\n",
      "55000/55000 [==============================] - 6s 111us/sample - loss: 0.6639 - accuracy: 0.7582 - val_loss: 0.5840 - val_accuracy: 0.8410\n",
      "Epoch 2/20\n",
      "55000/55000 [==============================] - 5s 97us/sample - loss: 0.5517 - accuracy: 0.7968 - val_loss: 0.5747 - val_accuracy: 0.8430\n",
      "Epoch 3/20\n",
      "55000/55000 [==============================] - 5s 94us/sample - loss: 0.5260 - accuracy: 0.8062 - val_loss: 0.5233 - val_accuracy: 0.8486\n",
      "Epoch 4/20\n",
      "55000/55000 [==============================] - 5s 94us/sample - loss: 0.5055 - accuracy: 0.8136 - val_loss: 0.4687 - val_accuracy: 0.8606\n",
      "Epoch 5/20\n",
      "55000/55000 [==============================] - 5s 96us/sample - loss: 0.4897 - accuracy: 0.8187 - val_loss: 0.5188 - val_accuracy: 0.8588\n",
      "Epoch 6/20\n",
      "55000/55000 [==============================] - 5s 93us/sample - loss: 0.4812 - accuracy: 0.8217 - val_loss: 0.4929 - val_accuracy: 0.8508\n",
      "Epoch 7/20\n",
      "55000/55000 [==============================] - 5s 90us/sample - loss: 0.4687 - accuracy: 0.8251 - val_loss: 0.4840 - val_accuracy: 0.8572\n",
      "Epoch 8/20\n",
      "55000/55000 [==============================] - 5s 90us/sample - loss: 0.4709 - accuracy: 0.8249 - val_loss: 0.4227 - val_accuracy: 0.8660\n",
      "Epoch 9/20\n",
      "55000/55000 [==============================] - 5s 92us/sample - loss: 0.4515 - accuracy: 0.8313 - val_loss: 0.4796 - val_accuracy: 0.8670\n",
      "Epoch 10/20\n",
      "55000/55000 [==============================] - 5s 93us/sample - loss: 0.4508 - accuracy: 0.8329 - val_loss: 0.4901 - val_accuracy: 0.8588\n",
      "Epoch 11/20\n",
      "55000/55000 [==============================] - 5s 93us/sample - loss: 0.4484 - accuracy: 0.8338 - val_loss: 0.4678 - val_accuracy: 0.8640\n",
      "Epoch 12/20\n",
      "55000/55000 [==============================] - 5s 95us/sample - loss: 0.4417 - accuracy: 0.8366 - val_loss: 0.4684 - val_accuracy: 0.8610\n",
      "Epoch 13/20\n",
      "55000/55000 [==============================] - 5s 93us/sample - loss: 0.4421 - accuracy: 0.8370 - val_loss: 0.4347 - val_accuracy: 0.8640\n",
      "Epoch 14/20\n",
      "55000/55000 [==============================] - 5s 98us/sample - loss: 0.4377 - accuracy: 0.8369 - val_loss: 0.4204 - val_accuracy: 0.8734\n",
      "Epoch 15/20\n",
      "55000/55000 [==============================] - 5s 95us/sample - loss: 0.4329 - accuracy: 0.8384 - val_loss: 0.4820 - val_accuracy: 0.8718\n",
      "Epoch 16/20\n",
      "55000/55000 [==============================] - 6s 100us/sample - loss: 0.4328 - accuracy: 0.8388 - val_loss: 0.4447 - val_accuracy: 0.8754\n",
      "Epoch 17/20\n",
      "55000/55000 [==============================] - 5s 96us/sample - loss: 0.4243 - accuracy: 0.8413 - val_loss: 0.4502 - val_accuracy: 0.8776\n",
      "Epoch 18/20\n",
      "55000/55000 [==============================] - 5s 95us/sample - loss: 0.4242 - accuracy: 0.8432 - val_loss: 0.4070 - val_accuracy: 0.8720\n",
      "Epoch 19/20\n",
      "55000/55000 [==============================] - 5s 94us/sample - loss: 0.4195 - accuracy: 0.8437 - val_loss: 0.4738 - val_accuracy: 0.8670\n",
      "Epoch 20/20\n",
      "55000/55000 [==============================] - 5s 96us/sample - loss: 0.4191 - accuracy: 0.8439 - val_loss: 0.4163 - val_accuracy: 0.8790\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.AlphaDropout(rate=0.2),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.AlphaDropout(rate=0.2),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.AlphaDropout(rate=0.2),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "optimizer = keras.optimizers.SGD(lr=0.01, momentum=0.9, nesterov=True)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "n_epochs = 20\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s 39us/sample - loss: 0.4535 - accuracy: 0.8680\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.45350628316402436, 0.868]"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.3357 - accuracy: 0.8887\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.335701530437036, 0.88872725]"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with keras.backend.learning_phase_scope(1):  # TODO: check https://github.com/tensorflow/tensorflow/issues/25754\n",
    "#    history = model.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MC Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "with keras.backend.learning_phase_scope(1):  # TODO: check https://github.com/tensorflow/tensorflow/issues/25754\n",
    "    y_probas = np.stack([model.predict(X_test_scaled) for sample in range(100)])\n",
    "y_proba = y_probas.mean(axis=0)\n",
    "y_std = y_probas.std(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(model.predict(X_test_scaled[:1]), 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(y_probas[:, :1], 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(y_proba[:1], 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]], dtype=float32)"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_std = y_probas.std(axis=0)\n",
    "np.round(y_std[:1], 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = np.argmax(y_proba, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.868"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy = np.sum(y_pred == y_test) / len(y_test)\n",
    "accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MCDropout(keras.layers.Dropout):\n",
    "    def call(self, inputs):\n",
    "        return super().call(inputs, training=True)\n",
    "\n",
    "class MCAlphaDropout(keras.layers.AlphaDropout):\n",
    "    def call(self, inputs):\n",
    "        return super().call(inputs, training=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "mc_model = keras.models.Sequential([\n",
    "    MCAlphaDropout(layer.rate) if isinstance(layer, keras.layers.AlphaDropout) else layer\n",
    "    for layer in model.layers\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_36\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_33 (Flatten)         (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "mc_alpha_dropout_3 (MCAlphaD (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "dense_311 (Dense)            (None, 300)               235500    \n",
      "_________________________________________________________________\n",
      "mc_alpha_dropout_4 (MCAlphaD (None, 300)               0         \n",
      "_________________________________________________________________\n",
      "dense_312 (Dense)            (None, 100)               30100     \n",
      "_________________________________________________________________\n",
      "mc_alpha_dropout_5 (MCAlphaD (None, 100)               0         \n",
      "_________________________________________________________________\n",
      "dense_313 (Dense)            (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 266,610\n",
      "Trainable params: 266,610\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "mc_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(lr=0.01, momentum=0.9, nesterov=True)\n",
    "mc_model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "mc_model.set_weights(model.get_weights())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use the model with MC Dropout:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.17, 0.  , 0.19, 0.  , 0.64]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(np.mean([mc_model.predict(X_test_scaled[:1]) for sample in range(100)], axis=0), 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Max norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer = keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                           kernel_constraint=keras.constraints.max_norm(1.))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/2\n",
      "55000/55000 [==============================] - 8s 147us/sample - loss: 0.4745 - accuracy: 0.8329 - val_loss: 0.3988 - val_accuracy: 0.8584\n",
      "Epoch 2/2\n",
      "55000/55000 [==============================] - 7s 135us/sample - loss: 0.3554 - accuracy: 0.8688 - val_loss: 0.3681 - val_accuracy: 0.8726\n"
     ]
    }
   ],
   "source": [
    "MaxNormDense = partial(keras.layers.Dense,\n",
    "                       activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       kernel_constraint=keras.constraints.max_norm(1.))\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    MaxNormDense(300),\n",
    "    MaxNormDense(100),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. to 7."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See appendix A."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Deep Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8.1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Exercise: Build a DNN with five hidden layers of 100 neurons each, He initialization, and the ELU activation function._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8.2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Exercise: Using Adam optimization and early stopping, try training it on MNIST but only on digits 0 to 4, as we will use transfer learning for digits 5 to 9 in the next exercise. You will need a softmax output layer with five neurons, and as always make sure to save checkpoints at regular intervals and save the final model so you can reuse it later._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8.3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Exercise: Tune the hyperparameters using cross-validation and see what precision you can achieve._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8.4."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Exercise: Now try adding Batch Normalization and compare the learning curves: is it converging faster than before? Does it produce a better model?_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8.5."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Exercise: is the model overfitting the training set? Try adding dropout to every layer and try again. Does it help?_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 9. Transfer learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Exercise: create a new DNN that reuses all the pretrained hidden layers of the previous model, freezes them, and replaces the softmax output layer with a new one._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Exercise: train this new DNN on digits 5 to 9, using only 100 images per digit, and time how long it takes. Despite this small number of examples, can you achieve high precision?_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Exercise: try caching the frozen layers, and train the model again: how much faster is it now?_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.4."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Exercise: try again reusing just four hidden layers instead of five. Can you achieve a higher precision?_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.5."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Exercise: now unfreeze the top two hidden layers and continue training: can you get the model to perform even better?_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. Pretraining on an auxiliary task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this exercise you will build a DNN that compares two MNIST digit images and predicts whether they represent the same digit or not. Then you will reuse the lower layers of this network to train an MNIST classifier using very little training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.1.\n",
    "Exercise: _Start by building two DNNs (let's call them DNN A and B), both similar to the one you built earlier but without the output layer: each DNN should have five hidden layers of 100 neurons each, He initialization, and ELU activation. Next, add one more hidden layer with 10 units on top of both DNNs. You should use the `keras.layers.concatenate()` function to concatenate the outputs of both DNNs, then feed the result to the hidden layer. Finally, add an output layer with a single neuron using the logistic activation function._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.2.\n",
    "_Exercise: split the MNIST training set in two sets: split #1 should containing 55,000 images, and split #2 should contain contain 5,000 images. Create a function that generates a training batch where each instance is a pair of MNIST images picked from split #1. Half of the training instances should be pairs of images that belong to the same class, while the other half should be images from different classes. For each pair, the training label should be 0 if the images are from the same class, or 1 if they are from different classes._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.3.\n",
    "_Exercise: train the DNN on this training set. For each image pair, you can simultaneously feed the first image to DNN A and the second image to DNN B. The whole network will gradually learn to tell whether two images belong to the same class or not._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.4.\n",
    "_Exercise: now create a new DNN by reusing and freezing the hidden layers of DNN A and adding a softmax output layer on top with 10 neurons. Train this network on split #2 and see if you can achieve high performance despite having only 500 images per class._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "nav_menu": {
   "height": "360px",
   "width": "416px"
  },
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}