1_RNN_intro.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "import sys \n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RNN (Recurrent Neural Networks)\n",
    "Articulos recomendado: \n",
    "- http://karpathy.github.io/2015/05/21/rnn-effectiveness/\n",
    "- http://colah.github.io/posts/2015-08-Understanding-LSTMs/\n",
    "- http://blog.echen.me/2017/05/30/exploring-lstms/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Que tienen de nuevo respecto a MLP y CNN?\n",
    "- MLP y CNN solo aceptan un vector de entrada de tamaño fijo y devuelve un vector de salida de tamaño fijo\n",
    "- RNN trabjan con secuencias tanto a la entrada como a la salida"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![rnn_types.jpeg](rnn_types.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aclaraciones: \n",
    "- Cada cuadrado NO es una neurona si no una capa que puede contener N neuronas\n",
    "- Cada flecha representa la interconexión entre dos capas. Los pesos forman una matriz de la N1xN2 donde N1 y N2 son la cantidad de neuronas en cada capa respectivamente."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tipos\n",
    "- **One to One**: CNN, MLP\n",
    "- **One to many**: [Image captioning](https://www.youtube.com/watch?v=xKt21ucdBY0)\n",
    "- **Many to one**: Sentiment Analisys, Detectar voz de hombre vs voz de mujer\n",
    "- **Many to Many** [(sequence to sequence)](https://youtu.be/dkHdEAJnV_w): [Traducción](https://github.com/jganzabal/aind2-nlp-capstone/blob/master/machine_translation.ipynb)\n",
    "- **Many to Many Sincronizado**: Etiquetado de tramas de video, Brain Computer Interface (BCI): Controlar una silla de ruedas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modelos de lenguaje Generativos:\n",
    "- Predecir la proxima palabra en funcion de las anteriores\n",
    "- Predecir el proximo caracter en función de los anteriores\n",
    "\n",
    "**Resultado**: Probabilidad en función de una secuencia\n",
    "\n",
    "**Aplicaciones** (Mas allá de la posibilidad de generar texto):\n",
    "- OCR\n",
    "- Speach2Text\n",
    "- Detección de autores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## \"If training vanilla neural nets is optimization over functions, training recurrent nets is optimization over programs.\"\n",
    "- Es una secuencia de ejecución mas que una clasificación\n",
    "- Las RNN son Turing completo en principio [Turing Complete](https://en.wikipedia.org/wiki/Turing_completeness), [RNN Turing complete](http://binds.cs.umass.edu/papers/1995_Siegelmann_Science.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Detalles de la arquitectura:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unidad de Elman o RNN unit\n",
    "\n",
    "**Nota importante**: Cada unidad no es una neurona sino que una capa\n",
    "\n",
    "![RNN%20vs%20FNN.png](RNN vs FNN.png)\n",
    "¿Cual es el tamaño de $W_h$?\n",
    "\n",
    "Todo se conecta con todo -> Si hay M hidden units tenemos $M^2$ $W_h$s "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## En ecuaciones:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FFN\n",
    "\n",
    "$h_t = f(W_x^T X_t + b_h)$\n",
    "\n",
    "$y_t = softmax(W_o^T h_t + b_o)$\n",
    "\n",
    "$f$ usualmente RELU, sigmoidea, etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$h_t = f(W_h^T h_{t-1} + W_x^T X_t + b_h)$\n",
    "\n",
    "$y_t = softmax(W_o^T h_t + b_o)$\n",
    "\n",
    "$f$ es tanh usualmente pero puede ser RELU, sigmoid, etc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Xt\n",
      "[[ 1]\n",
      " [-1]\n",
      " [ 2]]\n",
      "Wx:\n",
      "[[-1  1]\n",
      " [ 1  0]\n",
      " [ 2  1]]\n",
      "h:\n",
      "[[ 1]\n",
      " [-2]]\n",
      "Wh:\n",
      "[[ 0 -1]\n",
      " [ 2  1]]\n",
      "\n",
      "bh\n",
      "[[-1]\n",
      " [ 1]]\n",
      "Cantidad de parametros de recurrente: 12\n"
     ]
    }
   ],
   "source": [
    "# Ejemplo con dos neuronas en la capa oculta\n",
    "M = 2 # Cantidad de neuronas en la capa oculta\n",
    "\n",
    "h = np.array([1, -2]).reshape(2,1) # Valores iniciales de la capa oculta, vector columna de Mx1\n",
    "Wh = np.array([[0,-1], [2,1]]) # Matriz de pesos de capa oculta MxM\n",
    "\n",
    "# Dimension de entrada 3 en este ejemplo\n",
    "Xt = np.array([1, -1, 2]).reshape(3,1) # vector columna\n",
    "print('Xt')\n",
    "print(Xt)\n",
    "# Dimension de Wx?\n",
    "Wx = np.array([[-1,1],[1,0],[2,1]]) # Dimensión 3x2 para que Wx.T sea de 2x3\n",
    "print(\"Wx:\")\n",
    "print(Wx)\n",
    "\n",
    "bh = np.array([-1, 1]).reshape(2,1) # Dimensión Mx1\n",
    "\n",
    "print('h:')\n",
    "print(h)\n",
    "print('Wh:')\n",
    "print(Wh)\n",
    "print()\n",
    "print('bh')\n",
    "print(bh)\n",
    "total_params_r_layer = Wh.shape[0]*Wh.shape[1] + Wx.shape[0]*Wx.shape[1] + bh.shape[0]\n",
    "print('Cantidad de parametros de recurrente:', total_params_r_layer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-3]\n",
      " [ 1]]\n",
      "h luego de time step:\n",
      "[[-0.99505475]\n",
      " [ 0.76159416]]\n"
     ]
    }
   ],
   "source": [
    "a = Wh.T.dot(h) + Wx.T.dot(Xt) + bh\n",
    "h_ = np.tanh(a)\n",
    "print(a)\n",
    "print('h luego de time step:')\n",
    "print(h_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.5  0.4  0.3]\n",
      " [ 0.5  0.1  0.5]]\n",
      "[[ 0.3832697 ]\n",
      " [-0.22186249]\n",
      " [ 0.28228065]]\n",
      "[[ 0.4081724 ]\n",
      " [ 0.22286305]\n",
      " [ 0.36896455]]\n",
      "Cantidad de parametros de capa densa de salida: 9\n"
     ]
    }
   ],
   "source": [
    "# Salida de 3 neuronas\n",
    "Wo = np.array([[0.5, 0.4],[0.3, 0.5],[0.1, 0.5]]).reshape(2,3)\n",
    "bo = np.array([0.5, 0.1, 0.2]).reshape(3,1)\n",
    "ao = Wo.T.dot(h_) + bo\n",
    "print(Wo)\n",
    "print(ao)\n",
    "def softmax(x):\n",
    "    return np.exp(x) / np.sum(np.exp(x), axis=0)\n",
    "\n",
    "print(softmax(ao))\n",
    "total_params_out_layer = Wo.shape[0]*Wo.shape[1] + bo.shape[0]\n",
    "print('Cantidad de parametros de capa densa de salida:', total_params_out_layer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Con keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import SimpleRNN, Dense\n",
    "from keras.models import Sequential"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "simple_rnn_2 (SimpleRNN)     (None, 2)                 12        \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 3)                 9         \n",
      "=================================================================\n",
      "Total params: 21\n",
      "Trainable params: 21\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "rnn_neurons = 2\n",
    "time_steps = 100 # T\n",
    "n_features = 3 # D\n",
    "input_shape = (time_steps, n_features)\n",
    "model = Sequential()\n",
    "model.add(SimpleRNN(rnn_neurons, input_shape = input_shape))\n",
    "model.add(Dense(3))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Que es time_steps?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Desplegando (Unfolding) la RNN\n",
    "#### BPTT (Back-Propagation Through Time)\n",
    "- Secuencia de longitud 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![unfold%20RNN.png](unfold RNN.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observaciones:\n",
    "- Es como una FNN con 5 hidden layers pero con pesos compartidos (Shared weights): $W_h, W_o, W_x$\n",
    "- Donde habiamos visto pesos compartidos?\n",
    "- Se puede pensar como si h fuera la entrada y X es una señal de control en cada paso\n",
    "\n",
    "Preguntas:\n",
    "- Son todas las Y importantes? En que casos?\n",
    "- Que diferencia hay entre estado interno (internal state) de la RNN y los pesos?\n",
    "- Cada cuanto se resetea el estado interno? Y los pesos?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Detalles de la entrada $X$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Entrada en Vanilla Networks:\n",
    "\n",
    "NxD, donde N es la cantidad de muestras y D es la cantidad de features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.5  0.3]\n",
      " [ 0.2  0.1]\n",
      " [ 0.7  0.3]]\n",
      "(3, 2)\n"
     ]
    }
   ],
   "source": [
    "# Ejemplo N = 3, D = 2\n",
    "Ex_1 = np.array([[0.5, 0.3], [0.2, 0.1], [0.7, 0.3]])\n",
    "print(Ex_1)\n",
    "print(Ex_1.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Entrada en RNNs:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Secuencias de longitud fija\n",
    "\n",
    "NxTxD, donde T es la longitud de la secuencia"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[ 0.5   0.3 ]\n",
      "  [ 0.2   0.1 ]\n",
      "  [ 0.7   0.3 ]]\n",
      "\n",
      " [[ 0.54  0.1 ]\n",
      "  [ 0.23  0.3 ]\n",
      "  [ 0.9   0.1 ]]\n",
      "\n",
      " [[ 0.5   0.3 ]\n",
      "  [ 0.2   0.1 ]\n",
      "  [ 0.7   0.3 ]]\n",
      "\n",
      " [[ 0.54  0.1 ]\n",
      "  [ 0.23  0.3 ]\n",
      "  [ 0.9   0.1 ]]]\n",
      "(4, 3, 2)\n"
     ]
    }
   ],
   "source": [
    "# Ejemplo N = 4, T = 3, D=2\n",
    "Ex_2 = np.array([[[0.5, 0.3], [0.2, 0.1], [0.7, 0.3]], [[0.54, 0.1], [0.23, 0.3], [0.9, 0.1]], [[0.5, 0.3], [0.2, 0.1], [0.7, 0.3]], [[0.54, 0.1], [0.23, 0.3], [0.9, 0.1]]])\n",
    "print(Ex_2)\n",
    "print(Ex_2.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Secuencias de longitud variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[list([[0.5, 0.3], [0.7, 0.3]])\n",
      " list([[0.54, 0.1], [0.23, 0.3], [0.9, 0.1]]) list([[0.7, 0.3]])\n",
      " list([[0.54, 0.1], [0.23, 0.3], [0.9, 0.1]])]\n",
      "(4,)\n",
      "(2, 2)\n",
      "(3, 2)\n",
      "(1, 2)\n"
     ]
    }
   ],
   "source": [
    "Ex_3 = np.array([[[0.5, 0.3], [0.7, 0.3]], [[0.54, 0.1], [0.23, 0.3], [0.9, 0.1]], [[0.7, 0.3]], [[0.54, 0.1], [0.23, 0.3], [0.9, 0.1]]])\n",
    "print(Ex_3)\n",
    "print(Ex_3.shape)\n",
    "print(np.array(Ex_3[0]).shape)\n",
    "print(np.array(Ex_3[1]).shape)\n",
    "print(np.array(Ex_3[2]).shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "En tensorflow es posible, en keras se suele hacer padding\n",
    "\n",
    "¿Que pasa cuando tenemos longitudes distintas? Analizar el unfolding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ejemplos de preparación de datos para una RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predicción del valor de la acción de una empresa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predecir el valor de las acciones de una compañia.\n",
    "- Se utilizarán las 5 observaciones anteriores \n",
    "- Tenemos los datos de 138 días\n",
    "- ¿Cuanto vale N, T, D?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzsvXl4o2d19/85knfL+26PPR7PvmUm\nibMD2ckCZNICbVIoaQtv2rfQUtrSJkBLfyy/QturQFtKSQNNoJQE0lCmkIUkk0D2xJPMvtnjsT0e\n77slL7Ks+/3jeaSRbXkbybYkn891+bL0PPcjndFY+uos9zlijEFRFEVRAjhW2gBFURQltlBhUBRF\nUaagwqAoiqJMQYVBURRFmYIKg6IoijIFFQZFURRlCioMiqIoyhRUGBRFUZQpqDAoiqIoU0haaQMu\nhMLCQlNdXb3SZiiKosQV+/fv7zHGFM23Li6Fobq6mrq6upU2Q1EUJa4QkeaFrNNQkqIoijIFFQZF\nURRlCioMiqIoyhRUGBRFUZQpqDAoiqIoU4iKMIjId0WkS0SOzHJeROSfRKRBRA6JyCUh5+4RkXr7\n555o2KMoiqJcONHyGB4Cbp3j/G3ARvvnXuBbACKSD3weuAK4HPi8iORFySZFURTlAoiKMBhjfgX0\nzbFkD/A9Y/EakCsiZcAtwDPGmD5jTD/wDHMLjKIA4PcbHnmjhVHv5EqboigJx3LlGCqAsyH3W+1j\nsx2fgYjcKyJ1IlLX3d29ZIYq8cEbTX3c9/hhHnmzZaVNUZSEY7mEQcIcM3Mcn3nQmAeMMbXGmNqi\nonl3dCsJzpFzgwDsO9G1wpYoSuKxXMLQClSG3F8DtM1xXFHmJCAMrzf24Rn3rbA1ipJYLJcw7AU+\nYlcnXQkMGmPagaeBd4tInp10frd9TFHm5EjbEIWuFLyTfl5q6FlpcxQloYhWueoPgVeBzSLSKiIf\nFZE/EJE/sJc8ATQCDcC/A38IYIzpA74IvGn/fME+piizMuL1cbrbzW9eVklWWhL7jms4SVGiSVS6\nqxpj7p7nvAE+Psu57wLfjYYdyurgePsQxsDuyjyaekfYd7ILv9/gcIRLWSmKslh057MSdxw5NwTA\njopsbtxSTPfwOEfaBlfYKkVJHFQYlLjjyLlBCl0plGancd3mYkTg+RNawqwo0UKFQYk7jrQNsb08\nBxEhPzOFbWXZvH6md6XNUpSEQYVBiSvGJiap7xxmR0V28Nhl1fm83TLAxKR/BS1TlMRBhUGJK052\nDOPzG3aU5wSP1VbnMToxybG2oRW0TFESBxUGJab5wv8e43uvNgXvH7I3tu2oOC8Ml1XnA/Bmk1Y6\nK0o0UGFQYpr/fquVh19pCt5/vbGX0uw01uSlB4+VZKdRlZ+hwqAoUUKFQYlZhscmGByd4HS3h86h\nMYwxvNbYx5U1+YhM3bNQW51HXVM/1pYZRVEiQYVBiVnODYwGb79yuofT3W563ONcWVMwY+1l1fn0\nerw09niW00RFSUiisvNZUZaC1r4QYWjoxT1uzV64an04YbDmO9U19bG+yLU8BipKgqIegxKztPaP\nANaH/iune3ntdC9lOVY+YTrri1zkZSTzZlP/cpupKAmHCoMSs5wbGCU1ycH7dpVzbmCU5050clVN\nwYz8AoCIsLsyN9iOW1GUC0eFQYlZWvtHqchL55oNhQCMTfjD5hcCrC9ycabHg9+vCWhFiQQVBiVm\nOTcwypq8DGoKMynJTgXC5xcC1BS5GPf5pyStFUVZPCoMSszS2j9KRW46IsKNW0tYX5Q5Zf/CdGqK\nMgG0MklRIiQqVUkicivwDcAJPGiM+cq0818DrrfvZgDFxphc+9wkcNg+12KMuSMaNinxzYjXR5/H\nGxSCz79vG16fP2x+IUBQGLrdXLtJ54IryoUSsTCIiBP4JnAz1gznN0VkrzHmWGCNMeZTIev/CLg4\n5CFGjTG7I7VDSSzO9VvhoIAwpCY5SU1yznlNkSuVrNQkGrvVY1CUSIhGKOlyoMEY02iM8QKPAHvm\nWH838MMoPK+SwLROE4aFICLUFGXS2ONeKrMUZVUQDWGoAM6G3G+1j81ARNYC64B9IYfTRKRORF4T\nkTujYI+SALQOBIRh5p6FuagpcqnHoCgREg1hCBf0na1e8C7gMWPMZMixKmNMLfBbwNdFZH3YJxG5\n1xaQuu5undaV6LT2j5DidFDkSl3UdTWFmbQPjjHi9S2RZYqS+ERDGFqBypD7a4C2WdbexbQwkjGm\nzf7dCLzA1PxD6LoHjDG1xpjaoiJNLCY6rf2jlOem4XDMnmwOR43dDuOMViYpygUTDWF4E9goIutE\nJAXrw3/v9EUishnIA14NOZYnIqn27ULgGuDY9GuV1ce5/tFFh5EgtDJJhUFRLpSIhcEY4wM+ATwN\nHAd+ZIw5KiJfEJHQ0tO7gUfM1L7IW4E6ETkIPA98JbSaSVm9BPYwLJZ1hZmIqDAoSiREZR+DMeYJ\n4Ilpx/562v2/CXPdK8DOaNigJA5jE5P0uMepWERFUoC0ZCflOelamaQoEaA7n5WY42yf1VU1XBfV\nhVBTlKkeg6JEgAqDEnM091rCsLbgwoRhfZGLxm63NtNTlAtEhUGJOZp6rW/7awsyL+j6XZU5eLyT\nHGsfiqZZirJqUGFQYo6WvhGy0pLIy0i+oOuvqrHadL/W2BtNsxRl1aDCoMQcTb0jVBdkztkwby5K\nc9KoKczk1dMqDIpyIagwKDFHS6+HqgvMLwS4cn0Bb5zpwzfpj5JVirJ6UGFQYoqJST+t/aNURygM\nV9UUMDzu40ib5hkUZbGoMCgxRdvAKD6/YW3+hSWeAwRGgGo4SVEWjwqDElNEWqoaoCgrlY3FLl7V\nBLSiLBoVBiWmaI6wVDWUq9YXUNfUx4TmGRRlUagwKDFFc+8IackOirMW1247HFfWFDDineTIucEo\nWKYoqwcVBiWmaOodYW1+5qLbbYdjZ0UOgG50U5RFosKgxBQtfZGXqgZYk5dOVloSx1UYFGVRqDCs\nIrw+P4+80RKztf1+v6G5dyTiUtUAIsLWsmyOacmqoiwKFYZVxHPHO7nv8cO8cDI2R6N2DY8z7vNT\nFYXEc4BtZdmc6BjWhnqKsghUGFYRp7utGQV1zf0rbEl4TnYOA7D2Attth2NbWTYj3kla7FbeiqLM\nT1SEQURuFZGTItIgIveFOf87ItItIgfsn4+FnLtHROrtn3uiYY8SnsCMgv3NfStsyUz8fsPXnjlF\noSuVS9bmRe1xt5ZlA5qAVpTFELEwiIgT+CZwG7ANuFtEtoVZ+qgxZrf986B9bT7weeAK4HLg8yIS\nvU8FZQqneyxhONg6iNcXW3mGR+vOcuDsAJ99zxZcqVEZLAjAxhIXTodoAlpRFkE0PIbLgQZjTKMx\nxgs8AuxZ4LW3AM8YY/qMMf3AM8CtUbBJmYYxhsZuN6XZaXh9fo60xU5tf5/Hy1efOsEV6/K5c3dF\nVB87LdnJ+qJMTUAryiKIhjBUAGdD7rfax6bzfhE5JCKPiUjlIq9VIqTH7WV4zMevX2K9vPubYifP\n8J+vNTM4OsEX79xxwa2252JbWbZ6DIqyCKIhDOHeydNLQP4XqDbGXAQ8Czy8iGuthSL3ikidiNR1\nd8dmVU0s02gnnq+oKaAqP4O6GMoz1He5qczLYFNJ1pI8/taybNoGxxgY8S7J4ytKohENYWgFKkPu\nrwHaQhcYY3qNMeP23X8HLl3otSGP8YAxptYYU1tUVBQFs1cXjXZ+oaYwk9q1eexv7seY2CjhbO71\nRNw0by62lWsCWlEWQzSE4U1go4isE5EU4C5gb+gCESkLuXsHcNy+/TTwbhHJs5PO77aPKVGmsdtN\napKDitx0Lq3Oo8ftDXYyXUmMMZzp8VAdxb0L09kWqEzSPIOiLIiIyz+MMT4R+QTWB7oT+K4x5qiI\nfAGoM8bsBf5YRO4AfEAf8Dv2tX0i8kUscQH4gjEmdmIcCURjt4d1hVYPotq1+QDsb+6nunDpPpAX\nwsDIBMNjviX1GApcqVTkpnOwNXYS7ooSy0SlLtAY8wTwxLRjfx1y+37g/lmu/S7w3WjYocxOY4+H\nrWVWDH99USYixMSmrya7zfZSegwAuypzOHh2YEmfQ1ESBd35vArw+vy09I1QU+gCIMnpID8jhW73\n+DxXLj2BcFZ14dJ5DAC71uTS0jdCbwz8mxUl1oneTiIlZmnpG2HSb1gXEjYqykqle3jpPyR/eaqb\nP/j+fpKcQl5GCt/68CVsL88Jnm/uHUEE1uQtsTBU5gJwqHWQ67cUL+lzKUq8ox7DKiBQqlpTtPzC\n8FJ9N5PG8P5L1tA1PMYjb5ydcr6510N5Tjppyc4ltWNnRQ4OgQMaTlKUeVFhSHC6hsZ49E3rw7im\nyBU8XuRaHmE40THM5pIs/uaO7dy4tYQnDrdPafvdtMSlqgEyU5PYWJzFwVYVBkWZDxWGBOV0t5u/\nfeI41/3DC/yqvptP3riRnPTk4PmirFS63eNLvpfhePswW0qtpPcdu8rp9Xh55XRv8Hxz70hU5jsv\nhEACOlb2byhKrKI5hgTkD3+wnycOd+B0CLftKOXTt2ye8eFblJWK1+dnaMxHTnoyTT0e2gfHuGp9\nQdTs6B4ep8c9zhZ7H8G1m4rISk3ifw+28a5NRQyNTdDr8UZtMM987KrM5Ud1rZztG43alDhFSUTU\nY0gw/H7DU0c6ePe2El69/wb+5bcuCfuNvCgrFSAYTvrHZ07xRz98O6q2nOyw5itstT2GtGQn795e\nylNHOxj3TdJiVyQtm8ewxkpAH9BwkqLMiQpDgjE85sNv4PJ1+RRnpc26rsg1VRiaej30uMej2o77\nRIe103hz6fkeSO/bVcbwmI/nT3Sf38OwxKWqATaXZpGa5OBAiwqDosyFCkOC0W83isvLSJlzXdBj\nsOv6A/sJorm34Xj7MMVZqRTYIgRwzYZCynPS+Nz/HOYXRzsBqIrixLa5SHY6qK3O44VTXQvKM3z+\np0f4m71Hl8EyRYktVBgSjD5bGPIzFygMw+MMjkwwODoBQOfQWNRsOdExFMwvBEh2Ovj+x64g2elg\n78E2irNSyUhZvlTXrdtLaez2UN/lnnOdMYafHWrnrZbYaU+urG6Ws2hChSHBCLSWzs1InnNdTnoy\nyU6he3ic5j5P8HhXlITBN+mnvtMdzC+Esr7IxY//4CrWFWayvTw7zNVLxy3bSxGBJw93zLmutX+U\nXo8X95hvmSxTlLl54VQ313xlH6fs2ehLiQpDgtHvsb75zxdKEpHgXobQnkmdQwsPJfkm/fziaEfY\nbzJnejx4J/1sKQs/Y2FNXgZPfvKdfOvDl4Y9v1QUZ6dRuzaPJ4+0z7nukN1wb3hchUGJDQ60DNA2\nOEp5bvqSP5cKQ4Kx0BwDnN/LEMgvOGRxoaR9J7q49/v7ebmhd8a543ZF0pbS2T2CtGTnku94Dset\nO8o40THMmR7PrGsO2ZVL6jEoscLB1gE2FWdFdSb6bKgwJBgDIxM4BLLS5v/jCbTFaOkdodCVSkl2\n2qI8hsDwn/3NM+Pwx9uHSHII60N2W8cKt+4oBZjTawi0zhidmJyyU1tRVgJjDAfPDrCrMmf+xVFA\nhSHB6B/xkpuRgsMx/+zkgDA093moyk+nODuNruGFewzNdrnp22enCoPX52fvgTYuqcojJSn2/sQq\nctPZVZnL03ZV1HQm/YYj5wZJcVq2e7yTy2meosygpW+E/pEJdlfmLcvzxd67VomIgZEJ8uZJPAco\ncqXS5xmnqcdqS1GanbqoUFJTjxWCertlAL//fJ7hsf2tnBsY5Q+vX78445eRS6vyONUxHDY/0tjt\nxuOd5OIqa0OcW/MMygoT8GDjymMQkVtF5KSINIjIfWHO/6mIHBORQyLynIisDTk3KSIH7J+9069V\nFkefx7ug/AJYHoPfQMfQGFX5GYsOJTX3ekhNcjA4OsEZ23vw+vx88/kGdlXmcu2m2J3NvSYvndGJ\nSfo83hnnAm/CazYUAppnUFaeA2cHSEt2sLkkfDFHtIlYGETECXwTuA3YBtwtItumLXsbqDXGXAQ8\nBvxdyLlRY8xu++eOSO1Z7QRCSQshsJcBYG2BJQyDoxOMTcwfOhmbmKRtcIybt5UAltcA8Phblrfw\nJzdtRGT+cNZKUWlvqmvtH51x7lDrIK7UJHausb6duccnltU2RZnOgbMD7KzIIcm5PEGeaDzL5UCD\nMabRGOMFHgH2hC4wxjxvjAnURL4GrInC8yphWFQoaZowFNv3uxbgNZy1S1xv3FpMVloSb7X0M+qd\n5J/3NbBrTQ7XxbC3AJbHAOGF4WCr9SbMTrNeR/e45hiUlcPr83O0bYjd9rCp5SAawlABhE5fabWP\nzcZHgSdD7qeJSJ2IvCYid852kYjca6+r6+7ujsziBKZ/xEvePLueAxS5zvdSqrRDSQCdC0hAN9kl\nrusKXeyuzOXtlgG+/avTnBsY5f7bt8a0twBQYQvD2f6pc6/9fsOJjmF2VGQHK7s0lKSsJCc6hvD6\n/MuWeIbotN0O9wkQdu+2iHwYqAWuDTlcZYxpE5EaYJ+IHDbGnJ7xgMY8ADwAUFtbqw31wzDqnWTc\n559313OAwixLQDJSnBS5UunPXnhbjEBF0rqCTC6uyuNf9tXT2O3mPReVcWVN9Fp3LxXZacnkpCfT\nOk0Y+ka8eH1+KnLTg/XiGkpSVpKDy5x4huh4DK1AZcj9NUDb9EUichPwWeAOY0wwVmGMabN/NwIv\nABdHwaZVSWBzW/4CcwwZKUm4UpOoys9ARCjJtkJJsyWgO4fGaB+0Qi9nejzkZiSTk5HMJVW5+A2I\nwGdu3xqFf8nyUJmfPiOU1DFoiWJpThqZtjAMq8egrCCnOt1kpyVRsQw7ngNEQxjeBDaKyDoRSQHu\nAqZUF4nIxcC3sUShK+R4noik2rcLgWuAY1GwaVXSH+yTtDBhACvWvr7Y2oSWk55MSpJj1n5J9/33\nIT704OsYY6ZMXru4Mo+MFCd/dMPGZf3jjZQ1uRkzhCHgLZVkp4V4DCoMysrR5/FSmJW6rOHZiENJ\nxhifiHwCeBpwAt81xhwVkS8AdcaYvcDfAy7gx/Y/rsWuQNoKfFtE/Fgi9RVjjArDBXK+T9LCQkkA\n3/rwpWSmWG0pAl7DbKGklr4RGrs9vH6mj6ZeD5eutWKeORnJvPaZG8lahq360WRNXnqwBXfgTdcx\ndN5jcDqEjBQnHhWGhOZExxCHWwf5YG3l/ItXgD6Pd8FRgGgRlXeyMeYJ4Ilpx/465PZNs1z3CrAz\nGjYoIX2SFph8BlhXOHV6WklWWvDDcTqBaqXvv9pM28Aov37J+eKyQAVPPLEmL52xCT89bm+wQqtz\ncAwRKLRnSLhSk9RjSHD+8ReneOZ4pzUrJAY93v4Rb7C8ernQnc8JxEJbbs9FSXZa2HJVz7iP4XEf\n6clOfn64Hb9h2WY1LxXn9zKcT0B3DI1R6Eol2a4Xd6UlaY4hgfH6/Lxyuhdj4GeHZqRGY4KV8BhU\nGBKI/hErlJSbfuF/RMV2KGl4bII3m/qYtFtddNkjQD98ZVVwbXXh8sxqXirW5M3c5NYxNE5p9vky\nXvUYEpv9zf24x32kJjn46YHYEwZjzKJK0KOFCkMC0T/iJSs1KaLGdSXZaVafoC88wwf/7VWePW41\nmgvkHa7bXMxF9o7g6oL4FoaKMJvcuobGgvs5wBYG9RgSiuPtQ8Hd/S+c6iLZKXzi+g0cbRuiYZ7J\nfsuNe9zHxKQhP3N5Q7UqDAnEwMgEuRH+Ab1zYyFXrMvn996xDiD4RjlfrZPKp27exJ7d5YtKcsci\nrtQk8jKSp2xy6xgaozQndcoa9RgSA9+kn7998ji3feNF/uSRAwD88mQ3tWvz+c3LKnEI7D0YW17D\nQgdvRZv4KiNR5mQxDfRmY3t5Do/+/lWA1fco0PoiIAzF2WlsKM7i+s3FkRkbI1Tmny9ZHZuYZGBk\nYmooKU2FIRHw+vz87kNv8HJDL9vLs3nqaAfff62ZEx3D3H/bFoqz07hqfQF7D5zjUzHU56t/gTPc\no416DAnEwCIa6C2EyvyM4LfpzqFx0pOdcVeSOh9r8tKDyefQPQwBstRjSAjqmvt4uaGXz9y+hf/5\n+DVsKc3ir396BLDCowDvu6icpt4RTnXGTjip7wIqDaOBCkMC0b+IBnoLoSo/IzgPunNojJLs5d1k\nsxysycvgXP8oxpgpu54DZNo5hnBzG5T4IVBpd+PWEpKdDr78a1aVfGl2GptKrA2egX05R9sGV8bI\nMPR7FtfNIFqoMCQQ/SORh5JCqcrPoG1gDN+kn66hcYpDvkknClX5GYz7/LT2j57f3DYtlOTzG8Z9\nOt4znpnuDV66No+/es82/vyWzcEvO+sKM0lJcnCsbWjF7JxOYF7IcnsMiRUXWMX4Jv0Mj/miKgyV\neRlM+g3tg2N0Do9x0Zrla/u7XFyxLh+Alxp6GB6zEn0lOVNDSWBVh6QlO5ffQCUqdA6Nk5HiDLY5\nAYIFFgGSnNYgnOMdsSMM/SNenA4hewEz3KOJegwJwsCoXb0QxbK2wAaw5t4RK5QUMr8hUdhQ7KI0\nO40X67vpGLQ+PELzKC5tvZ0QdA1PLUOejW1l2RxvDz/yFax9D5985O3g/p6lps8zQV5GyrKHcFUY\nEoSf2WV2xVH88K6ydzYfbRtkbMK/oDdWvCEivGtTIS/V99A2MEpJdtqUN6ErNTCsR4UhnukaGp8y\nmGo2tpZl0efxBjd0Tudrz5zipwfa6HUvfARuJPR7vMu+hwFUGBKC77x0hr/532PcsKWY67dEr4y0\nNDuNZKfwZlM/YO2KTkTeubGIoTEfLzX0BFuPB8hMtcJH2hYjvlmox7C1LBsgbJ7hTI+Hlxp6gPPV\nQqG09I7QNjBzImAk9EU5b7hQVBjinBdOdvHFnx3j1u2l/NuHLyU1KXpxcKdDqMhNZ39zH0BCegwA\n79hQiIjlFZRO+zdm2R6DdliNX4wxdA6NLygUurXcFob2mcLwX683B28HksKh/PEjb/OX/30oAktn\nYnkMKgzKIqlr6sfpEL5+1+6IWmHMRmV+RrAHU6IKQ15mChdVWG0+QhPPEJJjUGGIW4bHfYxOTC7I\n481OS2ZNXjrHpwnD2MQkP97fygZ7dklgR3IAYwynu9wzrouUleiTBCoMcU9jj5uq/Iwlq5ipCmn3\nOz3Mkki8a1MRwAyPIVDFMqzCELcE9jAs9IvN1rLsGR7Dk0faGRiZ4BPXbwBmhpIGRiYYHvfR4/YG\nuxxHit9v6B+ZWPY9DBAlYRCRW0XkpIg0iMh9Yc6nisij9vnXRaQ65Nz99vGTInJLNOxZTZzu8lCz\nhF1OA8KQlZZERkriVjcHdr+undZKPEurkuKewETChSSfwapMaurxMOqdDB770ZutVBdkcNvOUuD8\nxrMAzX3n+20ttBGfMYaW3hFOdgyHPT885mPSb+LTYxARJ/BN4DZgG3C3iGybtuyjQL8xZgPwNeCr\n9rXbsEaBbgduBf7VfjxlAUz6DWd6PdQULZ0wBEpWEzWMFODStXns/cQ1XLdpavI+NcmB0yG4xydm\nuVKJdTqHZ7Y6mYutZdn4DZzstD6wu4fHef1ML3fsriA1ySpn7p/mFTT3eoK36xcgDD+qO8tlX36W\nd/3989z+Ty8GZ6mH0hfskxSfVUmXAw3GmEZjjBd4BNgzbc0e4GH79mPAjWLVBO4BHjHGjBtjzgAN\n9uMpC6BtYBSvz8/6IteSPUdVUBgSN4wU4KI1uTgcU+vFRQRXahKe8clZrlJincWGkrbbCeiX7Qqk\np4524Dfwnp1lgJWTmu4xBJpNpiQ55vUYJv2Gv3/6JIWuVD59y2Ym/YanjnTMWBdIcEez/9lCiYYw\nVABnQ+632sfCrjHG+IBBoGCB1yqzcLrb+gOsWUJhCHoMWYntMcyFK1WnuMUz4XY9z0Vlfgbv3FjI\nf7x8hlHvJD8/1MaGYlewp1JeZgp9I1M9yObeEYqzUtlY7JpXGF4/00v38DifuGEDH79+A5tLsnji\ncPuMdSvVJwmiIwzhtuRN3xY425qFXGs9gMi9IlInInXd3d2LNDExaey23NelDCXlpCeztSybXZWJ\n1w5joWSlJWkoKY5Z6B6GUP74xo30uL3807563jjTx+07y4IbH/MzksPmGKryM9iwAGH434PtZKQ4\nuXFLCQC37yyjrrk/2M8pQN8KtdyG6AhDK1AZcn8NMH3aRXCNiCQBOUDfAq8FwBjzgDGm1hhTW1RU\nFAWz45/T3W6y05IoWOI/nCc/+U7uubp6SZ8jltFhPfFN19D4ojsCXFadz5U1+XzrhdNTwkhgewxh\nQklVBRlsKHJxbmB01n0vXp+fJ4+0c/O2EtJTrHTq7TtLMYYZ4aT+FWqgB9ERhjeBjSKyTkRSsJLJ\ne6et2QvcY9/+ALDPWM1I9gJ32VVL64CNwBtRsGlV0NjtYX2xK+FaYccamTreMyYxxiyoNLRzeOyC\nOgP/8Y0bAaaEkcAK7YQmn8cmJukYGqMqP4ON9rqANz+dlxt6GBiZ4I5d5cFjG0uy2FjsmhFO6h+Z\nIMXpIDNl+etxIhYGO2fwCeBp4DjwI2PMURH5gojcYS/7DlAgIg3AnwL32dceBX4EHAOeAj5ujNEs\n3wJp7HFTU7h0+QXFQqe4xSYvnOrm0i89y6HWgVnXGGPoWuCu5+lcVVPAb1+5lk9cv2HKl6+8zBRG\nvJPBudGt/aMYY5U6BzbA1XedL0EdHJ3gD3+wn0//+CD/vK+e7LQk3rlxatTjtp1lvNHUR9fw+XBS\nv8dLXmbyinzxi8o+BmPME8aYTcaY9caYL9vH/toYs9e+PWaM+aAxZoMx5nJjTGPItV+2r9tsjHky\nGvasBtzjPjqHxpc0v6BY6BS32OTV071M+g3/+vzpWdcsZtfzdESEL965gzsvnloPE4j5B7yGlj7L\nO6jKz2RtQSZJDgnmGYwx/MVjB/nF0U72nejirZYB9uyumNGlYM/ucgT4l30NwWMr1ScJdB5D3NJo\nVyStV2FYcrQqKTY50GJ5Ck8f6+B0t3tK2fbjb7XS1DvC+y6ycgPR3IcT+LDu83gpy0mnudcqVa3K\nzyDZ6aC6MDO4l+GhV5p4+mi2KTLpAAAgAElEQVQnn3vPVj72zhr6PN6wsxXWF7n4yFXVfO/VJn6j\ntpKqggxOdQ5TkZseNbsXgwpDnBKIYS7lHgbFoiIvnRHvJO2Do5TlrMwbVZmKb9LP4XODvG9XOb84\n2sG//6qRr7z/IsCamfDpxw4x6Te8Yu9FKI5iuXXQY7D7JbX0jZCR4qTQZR3fUOTildM9/PZ3Xue1\nxl5u2lrMR+2hQHNVGH3q5k387FA7n/nJYfzGcK5/lM+9Z/pe4eVBeyXFKY3dbhxyfmaCsnQEZgHv\nb+6fdc23f3maTz16YLlMWvWc7BxmdGKSm7YW88HaNfz3W60cah1gcGSCP/7h25TnpvGhK6qoa45+\ny/jAXPVAOWlLr1WqGsgF3LazlMKsVIbHfLxnZxn/8MFdC8oT5KQn85nbt3CodZCGLjf/fk8tN28r\niZrdi0E9hjjlSNsQ6wozo9pmWwnP1rJs0pOd1DX1896LynGP+/jNb7/Kp2/ZzHWbi/H7Dd99+Qy9\nbi9fef9O/T9ZBg6eHQRgd2UuF1fm8fhb57jjX14mI8XJxKSfx/7ganZW5DDu8/PzQ+2U5UQxlBT0\nGCxhaO4bmdKvbM/uCvbsvrB9ur92cQVdw+Ncvi6fS6ryIjf2AlFhiEMmJv283tg7IymmLA3JTge7\nKnOCHsO+E10cbRviwRfPcN3mYt4+O0Cn3XbhVIebnWtyVtLcVcGBs/3kZSQHv6m/8OnreO54Fy+c\n7OLGrSXBDZl//4GL+Kv3botqA8jcdNtj8HiZ9BvO9o1w3abo7K0SEf7g2vVReaxIUGGIQw6fG8Tj\nneTq9YUrbcqqoXZtPt/65Wk84z6eOmLVm798uodzA6M8ffT8xqSjbYMqDMvAgbMD7KrMDYZoirPS\nuPvyKu6+vGrKOhEhJz26TeiSnA5y0pPpH/FysmOYcZ+f7RXZUX2OlUZzDHHIq6d7AbiyJn+FLVk9\nXFqdx6Tf8PqZXp4/0c11m4swBh7f38qTR9p516YislKTOBpmJKQSXYbHJqjvcrN7Bdu05Nu7nwPT\nDWvXJtZ7UT2GOOTV071sKc2iwJX4HU9jhUuq8hCBrz1Tz+jEJP/nnTWMT/h54MVGhsd8fPy6DYx5\nJznaNrjSpiY8h88NYgwrKgx5GZbHsL+5n+KsVNbkJVa1mnoMcca4b5I3m/q4an3BSpuyqshJT2ZT\ncRaHzw2Sm5HM5evy+cClaxge8+EQuHlbCdvKsznePsykP2wfSCVKHDhr7V9YeY9hgrrmfmqr8xKu\nLY0KQ5xxoGWAcZ+fq2pUGJabS6utKpGbt5aQ7HRw285SMlOcXL4unwJXKtvLsxmdmORMT/g+OUp0\nON4+zJq89BWZUxAgLyOFMz1uWvtHuTTBwkigwhB3vHK6F4fAFSoMy87l1dYHQGC8Y0ZKEv/xu5fz\n5V/bCcD2civprOGkpaW+c5hNJVkrakN+ZgpjE37g/D6XREJzDHHGi/Xd7KjIiXqlhTI/77mojIwU\nJ9dvPj/+8/J1578tbixxkeJ0cLRt6ILr2JW58U36aez2cO3mlW29H9jLkJbsCE58SyTUY4gj9jf3\n8VbLAO+9qGz+xUrUSXY6ePf20lnjyclOB5tKXRxtG2RgxMtPD5xjYtK/zFbGHkNjExxvD1+t9e1f\nnuaFk10Lfqym3hG8k342Fa+wx2CHsXatySXZmXgfo4n3L0pgvv5sPQWZKXz4yrUrbYoyC9vLrI1w\n7/zq83zykQP88qROG/zWC6d5zz+9yGP7W6cc7/N4+epTJ/jxtONzcarTame9uXRlhSHgMdRWJ14Y\nCVQY4oa6pj5erO/h96+tieouTiW6XFGTz9iEP7jJbWBUR4J2Do7hN/Dpxw7ywzdagsf3nejCb6DX\nPb7gxzrVOYzIyjePrLJnob9jQ2JOk9RPmDjh68/WU+hSbyHW+bWLK7huczECXPzFZ3CPqTD0jXjZ\nVOKiIjed+x8/zJbSLC6uyuMX9o7x6WMy56K+001VfkZwLOZKsbk0i1fvvyFhu+1G5DGISL6IPCMi\n9fbvGX6ViOwWkVdF5KiIHBKR3ww595CInBGRA/bP7kjsSVQGRry81NDDh69cq95CjCMi5GemkJlq\n/T/pHAer2VxpTjr/8luXkJuRzL/sa2DUO8mv6q0wW6974cJwqnOYjSucXwiQqKIAkYeS7gOeM8Zs\nBJ6z709nBPiIMWY7cCvwdREJ3ZnyaWPMbvtH+xaHoWvYcrXXFepQnnghJclBapJDJ79heQz5Gclk\npibx0WvW8dyJLr79q9OMTfi5uCqX/hFvcFPgiNfH2y3h25t7fX7O9HimzF9WloZIhWEP8LB9+2Hg\nzukLjDGnjDH19u02oAtIzMDcEtFjC0ORtsCIK7LSkhhWYaDfMxFM1n7k6mqyUpP4xnP1ZKUl8Z6d\nZfiN5RUDPPrmWd7/rVfoGhqb8ThNvR58frPiexhWA5EKQ4kxph3A/l0812IRuRxIAUKHtH7ZDjF9\nTUT0ky8M3XZyrvACBporK0dWWjLuVR5K8vr8uMd9wfLOnPRkPnL1WoyB6zcXB0duBvIMZ/tG8Rur\nH9J0AhVJG9VjWHLmFQYReVZEjoT52bOYJxKRMuD7wO8aYwLF3fcDW4DLgHzgL+e4/l4RqRORuu7u\n1VUC2GPHYAvVY4grXKlJqz6UFPAEckNGWv7eNevYVpbNXZdVUmAfD/yNd9qewpFzM/c9nOoYxhED\nFUmrgXkzmcaYm2Y7JyKdIlJmjGm3P/jD7lQRkWzg58DnjDGvhTx2u31zXET+A/jzOex4AHgAoLa2\ndlV1Ketxj+N0SHBAiBIfuFKTGF7lVUmB8Zf5IX2NClypPPHJdwJwssPyAgIeQ0dAGMK0FTnSNkR1\nQSZpyTohb6mJNJS0F7jHvn0P8NPpC0QkBfgJ8D1jzI+nnSuzfwtWfuJIhPYkJD3D4xS6UnA4EquD\nY6LjSkta9VVJgQ/8vMzwX2rybY+h12OFSzsGAx7DeWHw+w1f/vkx9p3o4votc0arlSgRqTB8BbhZ\nROqBm+37iEitiDxor/kN4F3A74QpS/2BiBwGDgOFwJcitCch6XGPaxgpDsnSUBL9Hstjys8M3wk1\nLyMZEatk1e83dA2PkZnipH1wjB73OMYY/uzHB/n3F89wz1Vr+cztW5fT/FVLREXxxphe4MYwx+uA\nj9m3/xP4z1muvyGS518t9Li9KgxxSFaaCkO4UFIoSU4HuenJ9HrG6RvxMjFpuHFLEU8d7eBo2xB5\nGcn85O1z/N/r1vMXt2xOuLkHsYq2xIgD1GOIT1xpSbjHfBizqlJiU+i3Q0lzzU4ocKXS5/EGw0g3\nbLXCRUfODfLDN1pIS3bwf69br6KwjOg22hjHGEOv20th1soNJVEuDFdqMj6/YWzCv+ItHFaKPo+X\nrNQkUpJm/w6an5lCj9sbrEjaWOxibUEGrzX2sr+5n/ddVE52mhZeLCfqMcQ4Q6M+vJN+3dwWh7jS\n7LYY46u3Mql/xBvc3DYbha4U+jxe2m2PoTQnjR3lObxY38OId5K7r6haDlOVEFQYYpzg5jYVhrgj\ny+6XtJo3ufV55heG/MwUet3jdA6N4RBrh/+OCqs77eaSLC5ewdnOqxUVhhinR4UhbsmyPYbpCeiG\nrmEeCWk/ncj0232S5qIgM5WB0QnODYxS6Eolyelgpy0Md11eqbmFFUCFIcYJCoPmGOIO1ywew3de\nauK+xw8vag5BvBLaJ2k2ClwpGAMn2ocpzbFaZFy9voBv3LWbD12hbeZXAhWGGCfQQE89hvgjkGMY\nGpvpMQC82RS+i2gi0efxzlqqGqAg0/rbru8aDvZOcjiEPbsr5kxaK0uHvuoxTo/bi0Mgb543lxJ7\nZKVaIZTQUJIxhvouNwBvNvWtiF3LxdjEJKMTkwvKMQBMTBpKbWFQVhYVhhinxz1OfmYqTm2HEXcE\nPIbQKW69Hi8DI9b9N84ktjD0Bza3LSCUFCAQSlJWFhWGGMfa3KbeQjwSzDGEeAz1nZa3sLsyl6Nt\ngwm9MzrYJ2neUNL58yXqMcQEKgwxTrfbS5HOYYhLAlPcQof1BPILv3VFFX4D+5sTN88wX5+kALkZ\nKQQKjzSUFBuoMMQ4VmdVFYZ4JctuixGgvsuNKzWJ23eW4XQIby4wnOT3G35Ud5Zx3+RSmRo1njrS\nwbmB0WCfpLx5ylWdDgkmqEtz9G89FlBhiGGMMRpKinOsmQxTQ0kbil24UpPYUZ7NGwtMQL/Z1Mdf\nPHaIp450LJWpUWHE6+P//mA/n/vJ4WCfpPmSz3Deq9BQUmygwhDDuMd9jPv86jHEMa5pHVbru9xs\nLLYmkF1Wnc+BswML8gIClUwN9u9YpbHbgzHw/MluXm7oAVjQgKkCVwqZKU6ytCdSTKDCEMPoSM/4\nJyv1/NzngREvPe7x4MziqzcU4PX5efZY2MGHUwgIQiB5Has09ngAcAj84lgnOenJJDnn/5hZV5jJ\nhmId2RkrqDDEMOd3PaswxCuutKRg8jnw4b6xOAuAazcVs6HYxdefPcWkf+7W3Ke7bY+hO7aF4XSX\nGxH4yFXVwPyJ5wB/9d5tfO/3rlhCy5TFEJEwiEi+iDwjIvX277xZ1k2GTG/bG3J8nYi8bl//qD0G\nVLE5eHYAgPVFmStsiXKhWFPcrOqcQDgo8M3Y6RD+5KaN1He5+dmhtjkfJyAqTT0eJib9C35+r8+P\nfx7RiSanu91U5mXwh9etJyXJMW/iOUBGShI5C1yrLD2Regz3Ac8ZYzYCz9n3wzFqjNlt/9wRcvyr\nwNfs6/uBj0ZoT0Lx/MkuNha7WJOXsdKmKBdI6Nzn+k436clOKnLTg+dv31HGltIsvvFsPb5ZPvDd\n4z7aB8fYUOzC5zc093oW/Py/8e1X+d2H3sTrW7iYREJjt4f1RZkUZ6fx/92xnXuurl6W51WiS6TC\nsAd42L79MHDnQi8Uq2XiDcBjF3J9ouMe9/HGmT5u0OHncY0r9fwUt1Odw6wvzsQRsovd4RD+5KZN\nNPZ4+MWxzrCP0WiHj27bUQosPM9gjOF4+xC/PNXNfY8fWvJJcn6/obHHTU2R5RHdfXkVe3ZXLOlz\nKktDpMJQYoxpB7B/z/YpliYidSLymogEPvwLgAFjTKBkoxWY9a9IRO61H6Ouu7s7QrNjn5fqe5iY\nNFy3WYUhnslKs6a4jXgnOXh2gIvWzJwtcNPWYpKdwpFzg2EfIxBGevc2WxgWWJk0ODrBuM9PTVEm\nj791jn/e1xB2nWfcF5VwU9vgKGMTftYXaRI53pl3tKeIPAuUhjn12UU8T5Uxpk1EaoB9InIYGAqz\nbta/TmPMA8ADALW1tQk/RPf5E11kpSVRWx02baPECYF+SW+19DM87uPSqpn/n0lOB5V5GTT3joR9\njIYuN0kOYUtZFhW56QsuWe2wR2X+6c2b+J+32/juy2f4oxs2ICKMTUzyhZ8d47XGXhq7Pdz7rho+\nc/vWC/xXWjR2WyEuzYnFP/N6DMaYm4wxO8L8/BToFJEyAPt32Lo7Y0yb/bsReAG4GOgBckUkIE5r\ngLkzcKsEYwzPn+ziXRuLSF5AqZ8SuwSmuL1w0vJyZxP6tQUZnOkJnzto6HKztiCDZKeDjSWuBXsM\nnUNWVVtpdhrXbi5iYGQiOD7z5YYe/uv1Fspz0rlmQwEPvtg4q8eyUAKVUzXqMcQ9kX7q7AXusW/f\nA/x0+gIRyRORVPt2IXANcMxYAc/ngQ/Mdf1q5GjbEF3D41y3uWilTVEixBUUhi4KXalU5YcvJFhb\nkElzrydsHuB0tztYybShyEVjt3ve8laATlsESrLT2FaWDVh/WwCHWgdxCHz7ty/lXz90KfmZKXz2\nf44s6HFn43S3m+y0JN2pnwBEKgxfAW4WkXrgZvs+IlIrIg/aa7YCdSJyEEsIvmKMOWaf+0vgT0Wk\nASvn8J0I7UkIXqy3doxqfiH+CYSSTnd7qF2bN+uYynWFmXi8k8EZ3wEmJv00944E4/YbS1yM+/y0\n9ocPO4USCCUVZ6eypTQLETgWFIYBNhS7yExNIic9mc+9ZxsHzw7wwwhGjjZ2e6gpcukozgRg3hzD\nXBhjeoEbwxyvAz5m334F2DnL9Y3A5ZHYkIic6BiiIjddu6omAAGPAWYPI4EVSgJo7h2hOOt8v6Dm\nXg8+vznvMdi/rfDS3LH8jqEx8jNTSE1ykppkic+x9kGMMRxqHeT6kIq3PbvL+f5rzTz0ShMfvvLC\nxmme7nbzjg3q5SYCGsCOQRq63KzX9gAJQXZI759L184uDNX2h3zTtDxDw7RNcRuKrF3TpxZQsto5\nODalKd22smyOtQ/RNjhGr8fLrjU5wXMiwm07SmnoctM2MDrvY0/HPe6jc2ic9cWaeE4EVBhiDL/f\nWDFlTeAlBIFQUmqSg+3lObOuW5OXTpJDaJq2ea2uqZ9kpwSFIScjmYrcdI60zZ8o7hgaozT7vNe5\nrTybs32jvFRvJcJ3TiudvXaT9W3/V6cWXw5+ot0KUWmpamKgwhBjnBuwasG1oVhikJnqBGBXZe6c\ng+2TnA7W5KXTFFKyaozhF8c6uXp9IRkp50NSu6tyOdAyMO9zdw5N9RgCwvTIm2dJdgpby7KmrN9Q\n7KIsJ41fXoAwPHOsk2SncGVNwaKvVWIPFYYYY3roQIlvUpOclOekLajCLFCZFOBExzAtfSPcsn3q\nNqKLK3M5NzBKl51cDofX56fH7Z0RSgJ4u2WAzaVZpCY5p1wjIrxrYxEvNfTM2p4jHMYYnjjSztXr\nC8lZQIttJfZRYYgxVBgSj+f+7Dp+/13r5123rjCTpp6RYMnq00c7EIGbt5VMWXdxlRUCevvs7F5D\noLqpNOe8MBRlpQYLGsLtwAZ416Yihsd8HGyd3yMJcLRtiLN9o9y+M9w+WCUeUWGIMRq63ORnpiy4\nXbES+6SnOHE65i/hXFuQgXvcR689+ezpo51cWpU3ozpte3kOSQ7hwBzC0GHvYZg+QzngNYQmnkN5\nx4ZCHAK/PNUzr70BnjzSjtMh3LxNhSFRUGGIMRo08bxqCa1MOts3wvH2oRlhJIC0ZCfbyrN5u6V/\n1sfqHDq/uS2U7eWWMOysCO8x5GQks6syd8F5BmMMTx7p4MqafP0yk0CoMMQQxhgtVV3FVBdawnCm\nx8Nj+1sBwgoDwO7KXA61Ds66UznoMeRMFYb3X7qGj71jHZtLs8JdBsDV6ws43LrwkaON3R5u3VE2\n71olflBhiCF63F4GRyeCM4GV1UVFbjpOh/DFnx3jG8/Vc1VNAVUF4VtoXFyVy4h3klOdw2HPdw6N\nkeKcOShnfZGLz71325yhrY3FWfgNtMzS1C/ApN/wt08cJ8kh3LK9ZM61SnwR0c5nJbpo4nl1k5Lk\nYFtZNu2DY3xhz2buuqxq1rW7K63Nci/Wd/OTt88xMOLl7z6wK3i+Y2iM4uzUC2pPUWN3Rz3d7WFj\nyeyexd8+cZznT3bzpTt3TNmtrcQ/KgwrRK97nN/49qtsKsliz+5yrt9SHJznq8Kwenn096/EIUJa\nsnPOddUFGeRmJPP/P3EieOwLe3YEr+sYHJuReF4o6woDwjD77uof1Z3lwZfOcM9Vay+4hYYSu6gw\nrBAvn+7ldLeHruFxnjzSQUFmCgWuFDJTnJTl6Lev1UroRra5EBE+cMkajncMsbU0mwdfOkPH4Fgw\nT9E1PM42O9G8WLLSkinJTg3OV5jOsbYh/up/jnDNhgL+6r3bLug5lNhGcwzT+HHdWfadCD9iMZq8\n1dxPRoqTNz97Ew/97mVcujaPhi4328tztDulsiA+995t/OBjVwab4QW6qRpjIvIYAGoKXWE9huGx\nCT7+X2+Rk57MN+66mCSdF5KQrCqP4fmTXQyP+bhjV/msa/55XwNlOWncsGVpk2l1zX3srswlLdnJ\ndZuLuW5zMV1DY/pGUxZNoCQ1UIk0NOZjdGKSkuwL7867vjiTvQfaMMZM+aLy+b1Hae718MP/cyWF\nLu3+m6isqk+hH7zWwj89Vz/nml73+IxGZtHGM+7jePvwjG6bxdlpWguuLJpASWpgOltgVsOavPAV\nTQuhptDF0Nj5zXZgtdn42aF2PnTFWq7QnkgJzaoShu3l2TR2uxn1hq/PHvVO4vFO0jk0zojXt2R2\nHDg7wKTfzNmGWVEWiis1iay0pOCmtrN9ljDMNi1uIQQrk0LGiJ7sGMbr82ujvFVARMIgIvki8oyI\n1Nu/Z3zSicj1InIg5GdMRO60zz0kImdCzu2OxJ752Faejd9Yg3DC0es5Pz2rqWf+CVkXyv7mfkTg\nEhUGJUqUZqfRPmjNUWixhaEyAmEItM9uDJkPceCstdN6V+Xs7cOVxCBSj+E+4DljzEbgOfv+FIwx\nzxtjdhtjdgM3ACPAL0KWfDpw3hhzIEJ75iTQJ+ZYe3hh6HGfd5uXMpxU19zP5pKsKUNcFCUSSnPS\n6Biyvti09I2Qk54cUafTitx0UpMcNIYkoA+cHaTQlUpFbnrE9iqxTaTCsAd42L79MHDnPOs/ADxp\njFm6r+NzsCYvney0pODc2+n0hszbPdOzNMIw6Te83dyvYSQlqpRmp9ER9BhGIwojATgcwrrCTE6H\nlKwebB1gd6VWza0GIhWGEmNMO4D9e77p9XcBP5x27MsickhEviYis5Y5iMi9IlInInXd3YsfJGI/\nBtvKs2f1GHptjyHJITNGLEaLU53DDI/75pz/qyiLpSwnje7hcXyTfs72jUQsDGCFkwIew9DYBKe7\n3eyapV23kljMKwwi8qyIHAnzs2cxTyQiZcBO4OmQw/cDW4DLgHzgL2e73hjzgDGm1hhTW1R04QPH\nt5XlcKJ9OGzzsR47x7C9PHvJQkn/e7ANgNq1+Uvy+MrqpCQnDb+BzuFxWvtHIsovBKgpyuRs/yjj\nvkkOtw5ijDU9Tkl85t3HYIy5abZzItIpImXGmHb7g79rjof6DeAnxpiJkMdut2+Oi8h/AH++QLsv\nmG3l2YxOTHKmxzOj9USv20tmipMtpdk8d2Kuf8qF0dzr4cEXz/BrF1dE5Y2rKAECu+UPtAwwMWmi\n4jFsKHYx6Te80tAb9LJnG/CjJBaRhpL2AvfYt+8BfjrH2ruZFkayxQSxgpZ3AkcitGde5kpA97jH\nKXClUl2YSY97nOGxiRlrIuFLPz9OklO477YtUX1cRQlscnvjTC8QWalqgJu3lbCh2MWf/fggzx7v\npKYoU0d3rhIiFYavADeLSD1ws30fEakVkQcDi0SkGqgEfjnt+h+IyGHgMFAIfClCe+ZlQ7GLZKeE\nTUD3ur0UuFJYV2i9qZrttsOz9bxfDL861c0zxzr5oxs2zhieoiiRUpZjVQq9fqYPiI4wZKQk8e3f\nvhSvz8/bLQPsVm9h1RCRMBhjeo0xNxpjNtq/++zjdcaYj4WsazLGVBhj/NOuv8EYs9MYs8MY82Fj\nzOztHKNESpKDTSVZs3sMmakzBqZs//xTfOuF0xEJxKN1ZynOSuX33lF9wY+hKLORl5FMSpKDk53D\nOB1CWW50vnysL3LxDx+02nnXVmtebLWwqnY+B9hWls2xtsHg0PUAvR4vha4U1uZbwvB2ywBf/vkx\nUpwOvvrUCT7wb6/QH9IiYDGcaB9id2UuqUlzt1NWlAtBRCjNTsMYKM9NIzmKPbdu3VHKvj+7lg/W\nronaYyqxzeoUhvJsetxeuofP71vw+w19Hi+FrlTS7dbXD7/axODoBI/+/lV8/Td383bLAI+/fW7R\nzzdmJ7u3lF1YG2RFWQiBbqrRCCNNp6bIFVWxUWKbVfk/HUhAHw0JJw2MTjDpNxS4rCZ21QWZTPoN\nH7mqmq1l2dx5cQUVuem8NccA9tmo73TjN7Bljjm7ihIpgWZ6SyEMyupiVQrDVnuASWgCOrDrucBu\nJbyjIpuirFQ+ddOm4JpL1ubxVvPiheG43ZtJhUFZSgIlq1oKrUTKqhSG7LRkqvIzpghDoE9Sod32\n+tO3bOG5P7uWnJBh6pdW5dI+OEbbwGjYxx0Y8fKXjx3iueOdU/IXJzuGSUt2sLYgcyn+OYoCnC9Z\nVY9BiZRVKQxgJ6BDQkmBzqoBjyElyTGjyV2gG+ps4aR9J7p4tO4sH324jvf+80vBCVgnOobYXJKF\n06E9ZpSlIzCreWOxeqZKZKxeYbDbXrjHrbkLPcMBYZh9UM7WsmzSkh3snyWcdLJjmBSng797/0W0\n9I7wj8+cwhjD8fZhtpRq4llZWq7bXMSTn3wnmzVkqUTI6hWGsmyMgZN2/L/X48UhkJcxuzAkOx3s\nWpM7a57hRMcw64td/MZllXywtpJfHO3gRMcwfR4vW8r0zaosLSLCVq18U6LA6hWGaQnoHreX/MyU\necM9l67N42jbEGMTM6fAneocDiaYf+uKSiYmDV/6+TEA/RanKErcsGqFoSwnjbyMZI7awtBr73qe\nj0uq8vD5DYdaB6ccHxyZoH1wLCgAG4qzuLw6n5cbrN41GkpSFCVeWLXCMH02Q6/HO2d+IUAgAT09\nz3CycxiAzSXnPYPfuqIKgJLsVPIz539sRVGUWGDVCgNYeYYTHcP4Jv3BzqrzkZ+ZwsZiF6+c7ply\nPJCrCA0Z3bqjlLyM5OCGOkVRlHhg3nkMicy28my8Pj9/++QJuofHKVyAxwBw7aYivvdqMyNeHxkp\n1kt4snOYrLSk4CYjgLRkJ9//6BU621lRlLhiVXsMN20t4cYtxTz0ShMj3kmKsxbWkfLazUV4J/28\n3tgXPHayw0o8T5+Hu6Mih6oC3XCkKEr8sKo9hqy0ZL7zO5fR4x7n5YYert20sJGhl1Xnk5bs4Jen\nurl+SzHGGE50DHPHrvIltlhRFGXpichjEJEPishREfGLSO0c624VkZMi0iAi94UcXycir4tIvYg8\nKiIrkqEtdKWyZ3cFuXPsYQglLdnJVTUF/PJUNwAdQ2MMj/m0F5KiKAlBpKGkI8CvA7+abYGIOIFv\nArcB24C7RWSbffqrwC2kIOgAAAbiSURBVNeMMRuBfuCjEdqzbFy7qYgzPR6aez2c6LArkrQkVVGU\nBCDSCW7HjTEn51l2OdBgjGk0xniBR4A99pznG4DH7HUPY819jguu21wMwH+83MS3nj8NwKYS10qa\npCiKEhWWI8dQAZwNud8KXAEUAAPGGF/I8YplsCcqVBdmsrYgg4deaSInPZkv3bljwaEoRVGUWGZe\nYRCRZ4HSMKc+a4z56QKeI1yPCTPH8dnsuBe4F6CqqmoBT7v0fOb2rZzqGOYjV1eTk64lqYqiJAbz\nCoMx5qYIn6MVqAy5vwZoA3qAXBFJsr2GwPHZ7HgAeACgtrZ2VgFZTm7ZXsot28NppqIoSvyyHPsY\n3gQ22hVIKcBdwF5jTbJ5HviAve4eYCEeiKIoirKERFqu+msi0gpcBfxcRJ62j5eLyBMAtjfwCeBp\n4DjwI2PMUfsh/hL4UxFpwMo5fCcSexRFUZTIkdARlPFCbW2tqaurW2kzFEVR4goR2W+MmXXPWYBV\n3RJDURRFmYkKg6IoijIFFQZFURRlCioMiqIoyhRUGBRFUZQpxGVVkoh0A80XeHkh1ua6eCHe7IX4\nszne7IX4szne7IX4s3kh9q41xsw7XyAuhSESRKRuIeVasUK82QvxZ3O82QvxZ3O82QvxZ3M07dVQ\nkqIoijIFFQZFURRlCqtRGB5YaQMWSbzZC/Fnc7zZC/Fnc7zZC/Fnc9TsXXU5BkVRFGVuVqPHoCiK\noszBqhIGEblVRE6KSIOI3LfS9kxHRCpF5HkROS4iR0Xkk/bxfBF5RkTq7d95K21rKCLiFJG3ReRn\n9v11IvK6be+jdrv1mEFEckXkMRE5Yb/WV8Xyaywin7L/Ho6IyA9FJC3WXmMR+a6IdInIkZBjYV9T\nsfgn+314SEQuiRF7/97+mzgkIj8RkdyQc/fb9p4UkVuW297ZbA459+ciYkSk0L4f0Wu8aoRBRJzA\nN4HbgG3A3SKybWWtmoEP+DNjzFbgSuDjto33Ac8ZYzYCz9n3Y4lPYrVUD/BV4Gu2vf3AR1fEqtn5\nBvCUMWYLsAvL9ph8jUWkAvhjoNYYswNwYs00ibXX+CHg1mnHZntNbwM22j/3At9aJhtDeYiZ9j4D\n7DDGXAScAu4HsN+DdwHb7Wv+1f48WW4eYqbNiEglcDPQEnI4otd41QgDcDnQYIxpNMZ4gUeAPSts\n0xSMMe3GmLfs28NYH1gVWHY+bC97GLhzZSyciYisAd4DPGjfF+AG4DF7SazZmw28C3v2hzHGa4wZ\nIIZfY6xJi+kikgRkAO3E2GtsjPkV0Dft8Gyv6R7ge8biNaxJjmXLY6lFOHuNMb8ImUH/GtZUSbDs\nfcQYM26MOQM0YH2eLCuzvMYAXwP+gqmjkSN6jVeTMFQAZ0Put9rHYhIRqQYuBl4HSowx7WCJB1C8\ncpbN4OtYf5R++34BMBDyBou117kG6Ab+ww5/PSgimcToa2yMOQf8A9a3wXZgENhPbL/GAWZ7TePh\nvfh7wJP27Zi1V0TuAM4ZYw5OOxWRzatJGCTMsZgsyRIRF/DfwJ8YY4ZW2p7ZEJH3Al3GmP2hh8Ms\njaXXOQm4BPiWMeZiwEOMhI3CYcfl9wDrgHIgEytMMJ1Yeo3nI6b/RkTks1hh3R8EDoVZtuL2ikgG\n8Fngr8OdDnNswTavJmFoBSpD7q8B2lbIllkRkWQsUfiBMeZx+3BnwA20f3etlH3TuAa4Q0SasEJz\nN2B5ELl22ANi73VuBVqNMa/b9x/DEopYfY1vAs4YY7qNMRPA48DVxPZrHGC21zRm34sicg/wXuBD\n5nwtf6zaux7rC8NB+z24BnhLREqJ0ObVJAxvAhvtao4UrGTS3hW2aQp2fP47wHFjzD+GnNoL3GPf\nvgf46XLbFg5jzP3GmDXGmGqs13OfMeZDwPPAB+xlMWMvgDGmAzgrIpvtQzcCx4jR1xgrhHSliGTY\nfx8Be2P2NQ5httd0L/ARu3LmSmAwEHJaSUTkVqw59HcYY0ZCTu0F7hKRVBFZh5XQfWMlbAzFGHPY\nGFNsjKm234OtwCX233hkr7ExZtX8ALdjVRucBj670vaEse8dWO7eIeCA/XM7Vtz+OaDe/p2/0raG\nsf3/tW/HJghDQRzGvy614ARO4Ai2zuEYVg4h2FtYpLFwDDstFFK4h43Fu0AuIFgIPvD7VYGkOA7u\n/eGSLIBTXM8og9MBLdD8ur5RrXPgHH0+ApOaewxsgBtwBfZAU1uPgQPlHcgzDqjVu55S1hzbmMML\n5YurGurtKHv5fvZ2g+fXUe8dWNbS49H9BzD9Ro/981mSlPzTKkmS9AGDQZKUGAySpMRgkCQlBoMk\nKTEYJEmJwSBJSgwGSVLyAmo9bZaZ0f20AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11d67fb70>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "138\n"
     ]
    }
   ],
   "source": [
    "apple_stock = np.loadtxt('apple/normalized_apple_prices.csv')\n",
    "from matplotlib import pyplot as plt\n",
    "plt.plot(apple_stock)\n",
    "plt.show()\n",
    "print(len(apple_stock))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from RNN_utils import window_transform_series\n",
    "X_, y = window_transform_series(apple_stock, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(133, 5)\n",
      "[[-0.70062339 -0.82088484 -0.93938305 -0.9471652  -0.68785527]\n",
      " [-0.82088484 -0.93938305 -0.9471652  -0.68785527 -0.84325902]\n",
      " [-0.93938305 -0.9471652  -0.68785527 -0.84325902 -0.80532018]\n",
      " [-0.9471652  -0.68785527 -0.84325902 -0.80532018 -0.82058073]\n",
      " [-0.68785527 -0.84325902 -0.80532018 -0.82058073 -0.92023124]\n",
      " [-0.84325902 -0.80532018 -0.82058073 -0.92023124 -1.        ]\n",
      " [-0.80532018 -0.82058073 -0.92023124 -1.         -0.98814438]\n",
      " [-0.82058073 -0.92023124 -1.         -0.98814438 -0.85961411]\n",
      " [-0.92023124 -1.         -0.98814438 -0.85961411 -0.8706188 ]\n",
      " [-1.         -0.98814438 -0.85961411 -0.8706188  -0.92661512]]\n"
     ]
    }
   ],
   "source": [
    "print(X_.shape)\n",
    "print(X_[0:10,:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Parece un detalle pero las capas recurrentes en Keras exigen este formato\n",
    "X = X_.reshape(X_.shape[0], X_.shape[1], 1) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(N, T, D):\n",
      "(133, 5, 1)\n"
     ]
    }
   ],
   "source": [
    "print('(N, T, D):')\n",
    "print(X.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl4VOXZx/HvnR1CFkgChCWEfdWC\nRFQQsRUUV7StuFQrrRTrUhWXllZrrdW6VEvtW7XiilsVtQoWEBFRkIrs+xYIe0IWIIEkZJ37/SMD\njWFClpnkZGbuz3V5zcyZZ85zH4f8cvKc5RFVxRhjTHAJcboAY4wxzc/C3xhjgpCFvzHGBCELf2OM\nCUIW/sYYE4Qs/I0xJghZ+BtjTBCy8DfGmCBk4W+MMUEozOkCapOYmKipqalOl2GMMX5l5cqVeaqa\nVFc7n4S/iIwFngVCgZdV9QkPbcYDDwMKrFXV60+1ztTUVFasWOGL8owxJmiIyO76tPM6/EUkFHgO\nGAPsA5aLyCxV3VStTW/gt8AIVT0sIu297dcYY0zj+WLMfxiwXVUzVLUMeBcYV6PNL4DnVPUwgKrm\n+KBfY4wxjeSL8O8M7K32ep97WXV9gD4iskRElrqHiYwxxjjEF2P+4mFZzftEhwG9gfOBLsBiERmk\nqvnfWZHIJGASQEpKig9KM8YY44kv9vz3AV2rve4CZHpoM1NVy1V1J7CVql8G36Gq01Q1TVXTkpLq\nPFhtjDGmkXwR/suB3iLSXUQigGuBWTXafAx8H0BEEqkaBsrwQd/GGGMawevwV9UK4A5gHrAZmKGq\nG0XkERG5wt1sHnBQRDYBC4H7VfWgt30bY4xpHGmp0zimpaWpnedvjAk2H6/ej6JcObgzIp4OqZ6a\niKxU1bS62tntHYwxpoU4UlLOI//ZxHvL99bd2EsW/sYY00I8t3A7h4vLePDSAY3a628IC39jjGkB\n9h4q5rWvd3HVkM4M6hzX5P1Z+BtjTAvw1LythITA/Rf1bZb+LPyNMcZhq/cc5pO1mfxiZA+S41o1\nS58W/sYY4yBV5dHZm0lsE8kto3o2W78W/sYY46C5Gw6wcvdh7r2wD20im2+KFQt/Y4xxSGlFJU/M\n3ULfDjGMT+ta9wd8yMLfGGMc8uY3u9lzqJjfXdqf0JCmPbWzJgt/Y4xxwOGiMv6+IJ3z+iQxqk/z\n38jSwt8YYxzw9y/SKSyt4IFL+jvSv4W/McY0s515Rbz5zW6uObMrfTvGOFKDhb8xxjSzJ+ZuJjIs\nhMlj+jhWg4W/McY0o28zDjJvYza/HNWT9jFRjtVh4W+MMc3E5VIem7OZjrFRTBzZw9FaLPyNMaaZ\nzFqbybp9Bdx/UV9aRYQ6WotPwl9ExorIVhHZLiJTTtHuxyKiIlLnRAPGGBNISsoreerTLQzqHMtV\nQzo7XY734S8iocBzwMXAAOA6ERngoV0McCfwrbd9GmOMv3nl651kFpTwwCUDCGnmC7o88cWNJIYB\n21U1A0BE3gXGAZtqtPsT8BRwnw/6NMaYFqvSpWTmHyMjr4iM3EJ25hXx4cp9jO7fgXN6JjhdHuCb\n8O8MVJ9zbB9wVvUGIjIE6Kqq/xERC39jjN+rdCkHC0vZe7iYHblF7MwrYmduERl5hew6WExZhetE\n25jIMAZ2iuOhy04aFHGML8Lf098vJ2aFF5EQYCowoc4ViUwCJgGkpKT4oDRjjGm44rIKMvNLyDlS\nwgH3f9kFx5+XknOkhJyjpVS6TkQd4aFCSrvWdE9sw/l929MjMZruidH0SGpDYpuIJp+WsaF8Ef77\ngOq3o+sCZFZ7HQMMAr50b3xHYJaIXKGqK6qvSFWnAdMA0tLSFGOMaWYZuYWM/dtiyipd31keExVG\nx9goOsZF0SspkY5xkXSMjaJz21b0SGxDl7atCAv1nxMofRH+y4HeItId2A9cC1x//E1VLQASj78W\nkS+B+2oGvzHGtASfb86mrNLFkz86jZR20XSMi6JDbCStI5rvXvvNweutUdUKEbkDmAeEAq+q6kYR\neQRYoaqzvO3DGGOay+L0PPp0aMM1Zwb20LNPfpWp6hxgTo1lD9XS9nxf9GmMMb52rKySb3ce4saz\nuzldSpPznwEqY4xpYst2HaKswsV5Dtxfv7lZ+BtjjNvibblEhIUwLLWd06U0OQt/Y4xxW5Sey1nd\n2zl+353mYOFvjDHAgYIStmUXMrJ3Yt2NA4CFvzHGULXXDzCyd+CP94OFvzHGAFWneCbFRNLPoWkV\nm5uFvzEm6LlcytfpuYzsndjibsPQVCz8jTFBb0NmAYeLyzkvSIZ8wMLfGGNYnJ4HwLlBcrAXLPyN\nMYavtuUysFMsiW0inS6l2Vj4G2OCWmFpBat2Hw6Kq3qrs/A3xgS1pTsOUuHSoDm//zgLf2NMUFuU\nnkur8FCGdmvrdCnNysLfGBPUFqfncU7PBCLDAv+WDtVZ+BtjgtbeQ8XszCsKuiEfsPA3xgSxYLul\nQ3UW/saYoLV4Wx6d41vRMyna6VKanU/CX0TGishWEdkuIlM8vH+PiGwSkXUiskBEAn+aHGNMi1ZR\n6WLJjryguqVDdV6Hv4iEAs8BFwMDgOtEZECNZquBNFU9HfgAeMrbfo0xxhtr9+VztKQi6M7vP84X\ne/7DgO2qmqGqZcC7wLjqDVR1oaoWu18uBbr4oF9jjGm0r7blESIwvGeC06U4whfh3xnYW+31Pvey\n2twMzPX0hohMEpEVIrIiNzfXB6UZY4xni9NzOb1LPPGtI5wuxRG+CH9Pg2XqsaHIDUAa8BdP76vq\nNFVNU9W0pKTg/FPMGNP0CorLWbs3P2iHfADCfLCOfUDXaq+7AJk1G4nIaOABYJSqlvqgX2OMaZQl\nO/JwKZwXhOf3H+eLPf/lQG8R6S4iEcC1wKzqDURkCPAicIWq5vigT2OMabTF6bnERIbxva7xTpfi\nGK/DX1UrgDuAecBmYIaqbhSRR0TkCnezvwBtgPdFZI2IzKpldcYY06RUlUXb8hjeK4Hw0OC91MkX\nwz6o6hxgTo1lD1V7PtoX/RhjjLcy8orYn3+MW8/v6XQpjgreX3vGmKC0eFvVmYSjgvhgL1j4G2OC\nzKL0PFITWtO1XWunS3GUhb8xJmiUVlTyzY6DQXkjt5os/I0xQWPV7nyOlVcG9fn9x1n4G2OCxqL0\nXMJChLN7tHO6FMdZ+Btjgsbi9FzO6NaWmKhwp0txnIW/MSYo5BWWsmH/kaC+qrc6C39jTFBYsj0P\nCM5ZuzzxyUVexhjTEpVXulizN58l2/OYuSaTtq3DGdQ5zumyWgQLf2NMwHC5lC0HjvLfHXks2Z7H\nsp2HKCqrRAQGdorlnjGDCA0Jvlm7PLHwN8b4tX2Hi1mcXhX23+w4yMGiMgB6JEZz1RmdGdEzkXN6\nJgTtfftrY+FvjPFbK3Yd4tppS6lwKR1iIxnVJ4nhvRIZ0SuB5LhWTpfXoln4G2P8ksulPPKfTSS2\nieSticPomdQmKCdibyw728cY45dmrt3Pun0F/HpsX3q1j7HgbyALf2OM3zlWVslTn27l9C5xXDn4\nVFOGm9pY+Btj/M7LizPIKijhwUsHEGJn7zSKT8JfRMaKyFYR2S4iUzy8Hyki77nf/1ZEUn3RrzEm\n+GQfKeGFr3Zw8aCODOtu9+hpLK/DX0RCgeeAi4EBwHUiMqBGs5uBw6raC5gKPOltv8aY4PTMZ1sp\nr3Qx5eJ+Tpfi13yx5z8M2K6qGapaBrwLjKvRZhww3f38A+ACsaMzxpgG2phZwPsr9zFheCrdEqKd\nLsev+SL8OwN7q73e517msY17wvcCIMEHfRtjgoSq8tjszcS3CueOH/R2uhy/54vw97QHr41og4hM\nEpEVIrIiNzfXB6UZYwLFgs05/HfHQe4e3Ye4VnZLZm/5Ivz3AV2rve4CZNbWRkTCgDjgUM0Vqeo0\nVU1T1bSkJLvznjGmSnmliz/P2UzPpGiuPyvF6XICgi/CfznQW0S6i0gEcC0wq0abWcBN7uc/Br5Q\n1ZP2/I0xxpO3l+4mI6+IBy7tT3ionaHuC17f3kFVK0TkDmAeEAq8qqobReQRYIWqzgJeAd4Uke1U\n7fFf622/xpjgUFBczt8WpHNur0S+37e90+UEDJ/c20dV5wBzaix7qNrzEuBqX/RljAkuf/8inYJj\n5TxwaX+7hYMP2d9PxpgWa2deEW98s4tr0rrSPznW6XICioW/MabFemLuZsJDQ7jnwj5OlxJwLPyN\nMS3S0oyDzNuYzW3n96R9TJTT5QQcC39jTIvjcimPzt5Ep7goJo7s4XQ5AcnC3xjT4ny4ah8b9h/h\nNxf3Iyo81OlyApKFvzGmRVm+6xC/n7mBod3acvnpnZwuJ2BZ+BtjWowN+wv4+WvL6RTfihdvHGr3\n6m9CFv7GmBZhe85RfvrqMmJbhfPWzWeR2CbS6ZICmoW/McZxew8Vc8PLywgR4a2JZ9EpvpXTJQU8\nC39jjKNyjpRwwyvfcqy8krcmDqN7ot2nvzlY+BtjHJNfXMaNrywj92gpr//sTPp1tKt4m4tP7u1j\njDENVVhawU2vLWfnwSJe/9mZDElp63RJQcX2/I0xza6kvJKJ05ezcX8Bz19/BsN7JjpdUtCxPX9j\nTLMqr3Rx+9ur+HbnIf52zWBGD+jgdElByfb8jTHNptKl3DNjLQu25PDYlacxbnDN6b5Nc7E9f2NM\ns6iodPH7mRv4ZG0mv724n03H6DALf2NMk1JV5m3M5ql5W8jILeKO7/fillE9nS4r6HkV/iLSDngP\nSAV2AeNV9XCNNoOBF4BYoBJ4TFXf86ZfY4x/WL7rEI/P2cyqPfn0TIpm2o1DGWNj/C2Ct3v+U4AF\nqvqEiExxv/5NjTbFwE9VNV1EOgErRWSequZ72bcxpoVKzz7Kk59u4fPNOXSIjeSJH57Gj4d2Icwm\nX28xvA3/ccD57ufTgS+pEf6quq3a80wRyQGSAAt/YwJMVsExps7fxgcr9xEdEcb9F/Xl5yO60yrC\nbsvc0ngb/h1UNQtAVbNEpP2pGovIMCAC2FHL+5OASQApKXYwyBh/UXCsnBe+3MFrS3aiCj8b0Z3b\nv9+LdtERTpdmalFn+IvI50BHD2890JCORCQZeBO4SVVdntqo6jRgGkBaWpo2ZP3GGGfM23iAX3+w\njiMl5Vw5uDP3jOlD13atnS7L1KHO8FfV0bW9JyLZIpLs3utPBnJqaRcLzAYeVNWlja7WGNOiqCqP\nz9lM+5hI3vnFWQzsFOd0SaaevD36Mgu4yf38JmBmzQYiEgF8BLyhqu972Z8xpgXZlHWEXQeL+fm5\n3S34/Yy34f8EMEZE0oEx7teISJqIvOxuMx44D5ggImvc/w32sl9jTAswd/0BQgQutNM3/Y5XB3xV\n9SBwgYflK4CJ7udvAW95048xpuVRVeasz+LsHgkk2KxbfsdOujXGNMrW7KNk5BVxyWnJTpdiGsHC\n3xjTKHPcQz4XDfR0MqBp6Sz8jTGNMmd9FsO6tyMpxoZ8/JGFvzGmwdKzj7I9p9CGfPyYhb8xpsFm\nr89CBMbakI/fsvA3xjTY3PUHOLNbO9rHRjldimkkC39jTINszylka/ZRLjnN9vr9mYW/MaZB5q7P\nAmDsIBvv92cW/saYBpm9Pou0bm3pGGdDPv7Mwt8YU28ZuYVsOXCUi+0sH79n4W+Mqbe5Gw4AcPEg\nG+/3dxb+xph6m7M+iyEp8XSKb+V0KcZLFv7GmHrZfbCIjZlHuNSGfAKChb8xpl7mrK8a8hlrQz4B\nwcLfGFMvczdk8b2u8XRpa1M0BgILf2NMnfYeKmbdvgIusb3+gOFV+ItIOxGZLyLp7se2p2gbKyL7\nReQf3vRpjGl+czdUXdhlN3ILHN7u+U8BFqhqb2CB+3Vt/gR85WV/xhgHzF5/gNM6x9G1nQ35BApv\nw38cMN39fDpwpadGIjIU6AB85mV/xphmtu9wMWv35ttef4DxNvw7qGoWgPuxfc0GIhICPAPc72Vf\nxhgHfGoXdgWkOidwF5HPAU/f+gP17OM2YI6q7hWRuvqaBEwCSElJqefqjTFNac76LAYkx5KaGO10\nKcaH6gx/VR1d23siki0iyaqaJSLJQI6HZucAI0XkNqANECEihap60vEBVZ0GTANIS0vT+m6EMaZp\nZOYfY9WefO6/qK/TpRgfqzP86zALuAl4wv04s2YDVf3J8eciMgFI8xT8xpiWx4Z8Ape3Y/5PAGNE\nJB0Y436NiKSJyMveFmeMcdac9Vn06xhDj6Q2TpdifMyrPX9VPQhc4GH5CmCih+WvA69706cxpnkc\nKChhxe7D3DOmj9OlmCZgV/gaYzyat7FqyMdO8QxMFv7GGI9mr8+iT4c29GpvQz6ByMLfGHOSDfsL\nWL7rkO31BzALf2PMd6zYdYjrXlpKx9gorjmzq9PlmCZi4W+MOeGrbbnc8Mq3JLaJ5P1fnkNynM3Y\nFai8Pc/fGBMgZq/L4u73VtO7fQxv3DyMxDaRTpdkmpCFvzGGd5ft4XcfreeMlLa8MuFM4lqFO12S\naWIW/sYEuWmLdvDnOVsY1SeJf94wlFYRoU6XZJqBhb8xQUpVefqzrTy3cAeXnp7M1PGDiQizw4DB\nwsLfmCDkcikPzdrAW0v3cN2wrjx65WmEhpz6rrsmsFj4GxNkyitd3Pf+WmauyeSWUT2YMrYfdd1u\n3QQeC39jgkhJeSW3vb2KL7bk8Ouxfbnt/F5Ol2QcYuFvTBC59/21LNyaw6NXDuKGs7s5XY5xkB3d\nMSZILNmex+x1WUwe3ceC31j4GxMMyitd/GHWRlLatWbSeT2cLse0ABb+xgSB6f/dxfacQh66bABR\n4XYev/Ey/EWknYjMF5F092PbWtqliMhnIrJZRDaJSKo3/Rpj6i/3aCnPfp7O+X2TuKB/e6fLMS2E\nt3v+U4AFqtobWOB+7ckbwF9UtT8wDM8TvRtjmsCTn26hpKKShy4bYKd0mhO8Df9xwHT38+nAlTUb\niMgAIExV5wOoaqGqFnvZrzGmHlbtOcwHK/dx87k9bB5e8x3ehn8HVc0CcD96+puyD5AvIv8WkdUi\n8hcRsUFHY5qYy6U8PGsj7WMiueMHdj6/+a46z/MXkc+Bjh7eeqABfYwEhgB7gPeACcArHvqaBEwC\nSElJqefqjTGezFixl3X7CvjbNYNpE2mX9JjvqvNfhKqOru09EckWkWRVzRKRZDyP5e8DVqtqhvsz\nHwNn4yH8VXUaMA0gLS1N67cJxpiaCo6V89S8rZyZ2pZxgzs5XY5pgbwd9pkF3OR+fhMw00Ob5UBb\nEUlyv/4BsMnLfo0xpzB1/jbyi8t4+IqBdpDXeORt+D8BjBGRdGCM+zUikiYiLwOoaiVwH7BARNYD\nArzkZb/GmFpsPXCUN5fu5vqzUhjYKc7pckwL5dVAoKoeBC7wsHwFMLHa6/nA6d70ZYypm6ryh1kb\niIkK494xfZ0ux7RgdoWvMQFk9voslmYc4t4L+9I2OsLpckwLZuFvTIAoLqvgsdmbGZAcy/XD7Gw5\nc2p2/pcxzcjlUj7deIAKl5LUJpL2sZEkxUQSExnm9YHZ5xfuIKughL9fN8Rm5TJ1svA3phk9OW8L\nL36VcdLyqPAQ2sdEkRQTSfuYyBOP7WOj6BTXiuT4KJLjomgd4flHdvfBIqYtyuDKwZ04M7VdU2+G\nCQAW/sY0kxnL9/LiVxn85KwUJgxPJedoKblHS8k5WuJ+LCXnSCnpOYUs2Z7HkZKKk9YR1yqc5Lgo\nOsW3OvHYMTaKj9fsJzxU+O0l/R3YMuOPLPwdoq5Ktr5xJx3PvYn4XsOcLsc0sW92HOR3H61nZO9E\n/njFQMJCQ+jdIeaUnykpryTnSCmZBcfIKjhGVkEJWfklZBUcIzO/hNV7DnO4uPxE+ykX96NDbFRT\nb4oJEAEZ/seKjtIq+tQ/WE7bunIh/Xa9xRtl3fmphX9A25lXxK1vryQ1MZp/XH8GYaH1O88iKjyU\nlITWpCS0rrVNSXklWQUlHC4uY3CXeF+VbIJAwJ3tsz9jIwV/GczKOSfdPaJFObzyQ8o0lKd3dmNT\n5hGnyzFNpKC4nJtfX44Ar9yURlyrcJ+uPyo8lO6J0ZyR0pYQO8hrGiDgwr9D194cCWtHz2UPkZe5\n2+lyPFKXi5TsBWyIHAKRsTz92VanSzJNoLzSxa1vr2Tv4WJevDGNbgnRTpdkzAkBF/5h4RFEjX+J\nKC1l/5sTUZfL6ZJOkrFxOZ01m4o+l3Lr+b34YksOy3Yecros40OqykMzN/LfHQd5/IenM6y7nYFj\nWpaAC3+AlD6DWdNvMt87tozl//6b0+WcJGfZ+7hU6DFyPBOGp9I+JpKnPt2Cqt3INFC8umQX/1q2\nh9vO78mPh3ZxuhxjThKQ4Q8wbPwUNkQOZuD6J9mfsdnpcr6jw/75bIkYSGKHLrSKCOWu0b1Zsfsw\nX2yx2S0DwYLN2Tw6exNjB3bkvgvt/jqmZQrY8A8JDSXxJy/jQij410QqK04+Z9oJ+3ZspIdrFwWp\nY08sG5/WldSE1jz16VYqXbb372+q/8W2OesId/5rNYM6xfHXa75nB2FNixWw4Q/QMaU3W4b8ngHl\nG1j+7p+cLgeAvf+dAUC3EeNPLAsPDeHeC/uyNfsos9bud6o00wgPP/wwkydPRlXJOVrCzdOXU1la\nRP+8r2q9GteYliCgwx8g7YpbWd16BGek/4Odm5Y7XQ5td89je2hPOqV+dzjg0tOSGdgplmc+20ZZ\nRcs7SG1Opqrk5+fz7LPPcufke5n0xkqyDxexc/pvKCvItWM4pkUL+PCXkBBSbppGobSm8sNbKCst\ncayW3Mzd9KvYTG6XMSe9FxIi/HpsP/YdPsa/lu1xoDrTUCLCM8/8lasnP8oHh1NYvecQWR89zq3X\nXsbUqVNtBi3TogV8+AMkdOjC7uGP06tyByvf/K1jdWR8/R4AyWeP9/j+eb0TObtHO/7vi3SKSlvG\nMQrjWWlFJe8u28OYqYtYFjGYkKg25H3yNMfSl1rwG7/gVfiLSDsRmS8i6e7HtrW0e0pENorIZhH5\nuzjwkzHkwhtYHn8xw/a+xtYVXzR39wC03jGXvdKJbn2HeHxfpGrvP6+wjFe/3tnM1Zn6OFJSzgtf\n7uDcJxcy5d/raR0Rypll68h86RaKNy8COHEMwJiWzNs9/ynAAlXtDSxwv/4OERkOjKBqGsdBwJnA\nKC/7bZS+E54jVxJpPft2jhUdbda+Cw7l0K9kLfs6jkZCav/ffkZKWy4c0IFpizI4VFTWjBWaU8k5\nUsLjczcz4vEvePLTLfTtEMObNw+jZ8aHfDD1d9x1569wuVzcddddPPvss/YLwLR43ob/OGC6+/l0\n4EoPbRSIAiKASCAcyPay30aJjU8gb/Rf6aqZrHv97mbte9uiDwiXShLO/FGdbe+7qC9FZRW88OX2\nZqjMnMoXO9Yw5cN1nPvkQl5alMGovkn851fn8tbEsxjZO4m28fHcddddJ4Z6pk6dyl133UV8fLwN\n/ZgWTbzZOxGRfFWNr/b6sKqeNPQjIk9TNaG7AP9Q1QdqWd8kYBJASkrK0N27m+bePEuf/wVn58xg\nwwVvMGjkuCbpo6bVT11Cp+ItJP0+nZDQ0Drb3/f+WmatzeTL+86nU3yrevWRkVvIR6v3E9cqnAGd\nYhmQHEt8a5vHtbFmrP+aR1beRmXOD/lRnx/xi5E9PN6fR1W/E/Q1XxvTnERkpaqm1dWuzhORReRz\noKOHtzwGuIfP9wL6A8evcZ8vIuep6qKabVV1GjANIC0trcn+Zh48YSq7n/4vSQsmUzBoBHFtE5uq\nKwCKCwvoV7ScdUmX06EewQ9w9+jezFqTybOfp/Pkj08/ZdtVew7z4lc7+GxT1R9U1X+fJ8dF0T+5\n6hdB/+RY+ifHkJoQbRcf1aGgpJg/L3uYEOL4ZOIddG9X+7+RmkFvwW/8QZ3hr6qja3tPRLJFJFlV\ns0QkGfB0f4KrgKWqWuj+zFzgbOCk8G8uUa3bUHrZ83SeeSWrX7+VMye/36T9bV3yMUOkjOghV9X7\nM13atuYnZ6cw/b+7+MV5PejVvg3wv71Kl0v5YksOLy7awfJdh4lrFc4d3+/FT89JRaTqStNNmUfY\nnHWEzVlH+Wpb7omrh1uFh9IvOYaxAzsy6bweFlYe/PKTJ6gMy+a2/k+eMviN8VfeXoI4C7gJeML9\nONNDmz3AL0TkcaqGfUYBjt9trc8Zo/hm9c85Z+/LrJr7Gmdc/LMm66ti4yccJoZ+wy5q0OeKV3xE\niOs0nv5sK/+8YSiqyp2T7+FQm55ktzuN7TmFdI5vxR8uH8D4tK5ER/7v6xzZO4mRvZNOvC4pr2R7\nTiGb3L8UVu/N5/G5W4gIC+FnI7r7bFsDwcxN37K+aCZdws7j1mGXOF2OMU3C2/B/ApghIjdTFfJX\nA4hIGvBLVZ0IfAD8AFhP1cHfT1X1Ey/79Ym0G/9M+lNf0ePbB8n73g9I7NTN532UlZbQ98gStsSf\nz7Dw+o+/qyqlBXnkrZjBp3I9i9NzeeKfb7G2fAhhFQkMCA3h2WsHc8lpyYTXY2aoqPBQBnWOY1Dn\nOABcLuXWt1fyp/9somdSG87rk1THGoJDUWkpD3/zECJtePmylnFLEGOagldn+6jqQVW9QFV7ux8P\nuZevcAc/qlqpqreoan9VHaCq9/iicF8Ij4gkcvzLRGoZmW/c3CT3/t+ydA6xFBMx6IoGfe74mSM/\nSetIZXEBN76yjI3hfUhuDW/8/Exm33ku4wZ3rlfwexISIvx1/GD6dozl9ndWsSO3sFHrCTS3zv4L\nFWGZ3NzvfrrE2z34TeAKiit8TyWlz2DWDbiP00uWs+z9v/h8/cfWfUyxRtJvRMPCH6p+Afxj6tMc\nnPcPjq6dR9brd7HsqRs5r097n4zTR0eG8dJPhxIZFsLE6SvILw7u6wo+3baKVUfep0PIcO4e7ums\nZWMCR9CHP8Cwq+9nXdSZnL7pafZsW+Oz9VZWVNDz4FdsiTmbqFYNn8JPVZk8eTLHtn3DoU//j7Ls\nHT6/eKhL29a8eONQ9h8+xu3vrKK8smXdVO6Zrz/kwjd/yQ0f/pEH5r/KW2sWsuHAHioqK33aT0l5\nGQ98/SDias3Ll9pwjwl8ds881RPUAAALt0lEQVRZqm7+1umnr1A6bTilMyZS/uslhEdEer3e9JVf\n0I98dvW/vMGfPR78zz777ImLiI6/Bnx6/5ih3drx2FWDuP+DdTz6n038cdwgn6zXW2uydvFa+uMA\nZB4tY22hMisTWAvqCiPMlUCb0PYkRCbTuU0X+iX04OahY4mObPh3d/vsqZSF7uUn3R8ktV17326I\nMS2Qhb9bYqdurDr7Mc5YehffvPk7zrn5Ga/Xmb/q35RpGH3Prfuq3ppEhHgPV48CTXL16NVpXUnP\nKWTaogx6d4jhhrN9f/C7IVwuF7+a9yDg4rUxMxiQ1JW1B3ayPjuDrQd3s/foXnKPZXKkIpv8km1k\nlJWy+BC8vuWfPDriUS7pO7TefS3MWM+3h/9FUmgaU867puk2ypgWxKsrfJtSWlqarlixotn7XT51\nPEPy57P98g/ol3ZBo9ejLhdZf+pHblQ3vveb+Y1fTzNePVrpUiZOX87i9DzeuHkYw3s6d377Y1++\nw7u7H2dku5/z/OWTT9nW5XKxJz+Pf61fyDs7nkVDihkWfw3PX3ovUXWcYVVWUcHwN35IKdl8cPlH\n9E3q5MvNMKbZ1fcKXxvzr6Hvz14gVxJoM/t2igsLGr2ejA1L6aTZlPby7jzx5rx6NDRE+Pt1Q+ie\nGM1tb69i98GiJuvrVHYdyuHdjP8jsrIbf7v4V3W2DwkJIbVde3476ho+ufJj2oeksbzgHc5984cs\nzFh/ys/ePff/KA3dydXdf2XBb4KKhX8NsfEJHLrw73RyHWD9a3UHT21yln9IpQo9z73ah9U1vZio\ncF6+qWqn4ebpKzhaUt7sNdwy52E0pJhHz32EiLCGjUymtmvPFz99metSf0cpufzqq59y2ydTKfMw\nh/OS3ZtZlPcG8QzmwVHX+6p8Y/yChb8HA4dfwrLk6znr4EzWfvFuo9bRcf98tkYOIqFDl7obtzDd\nEqJ5/idnsCuviDv/tbpZJ5WftnwumZWLOb3NVYztc0aj1/O7Udfx/uUf0jZkEIsPvcq5b1zN0j1b\nT7xfUVnJ5AUPIBrGP8f+mZBT3GbbmEBk/+JrMWTC02SEpNJ50W84lNOwSdX3bl9Pd9dujqSObaLq\nmt7wnon8cdxAFm7N5clPtzRLnweLj/KPdU8SWtGeFy77tdfr65fUha9unM6Vne/lGPuYuOA67pn7\nAi6Xi3vnvcCx0HQu7/pLBnbo6oPqjfEvFv61iIxqjfxoGrFayO7Xf9Ggq3/3/3cGAN1G+NeQT00/\nOasbE4anMm1RBq8tafqZxSZ98hgadpB7z3iAuKjWPllnSEgIfxo9gbcveZ9Y+jA/53mGTx/PguzX\niNFBPHpB093TyZiWzML/FLoPPItVvX/FkOIlLJ/5j3p/ru2ez0gP7UVyt75NWF3zePDS/owd2JE/\nfrKJf361o8n6+ffGb9h6bA7dwi/gxiE/8Pn6T++Yytc3vcNFHW6nkJ2A8NyFNtxjgped6lkHV2Ul\nm5/8PqmlW1nXYyJhcclExicTndCZuKTOtE1MJrTaQcmc/Ttp/9Jgvkm9lXMmPOFg5b5TUeninhlV\nk8tMHt2HOy/o5dOzjorLSxnxxjgqpZBPfzyLTrFNe0+ddQd2kX+siPO6D2zSfoxxgs8mcwl2IaGh\nJNzwCvmvXco5O0/e+69UIU/iKAhtR1F4AqGuEtoDnc4e3/zFNpGw0BCmXjOYiLAQpn6+jZKKSn59\nUV+f/QL41eypVITt5+e9Hmny4IeqvwKMCXYW/vXQMaU3/GEbRUfzyc/dz9Hc/RQfzqS84ACuo9mE\nFuUQUZJHdPlB4ioOsi5qKKf3a/yZKi1RaIjw1I9OJzIshBe+3EFJeSUPXTbA618Ai3Zu5NvD75EU\neiaTR9R/shtjjHcs/BsgOiae6Jh46HHq4YJAvTNMSIjw6JWDiAwL5dUlOymtcPHouEGNnhKyorKS\n+xb+HiGcFy5+xMfVGmNOxcLfNIiI8PvL+hMVHsLzX+6grMLFkz86ndBG/AKYMv+lqtMtO91NvyT/\nux7CGH/m1akOInK1iGwUEZd79q7a2o0Vka0isl1EpnjTp3GeiHD/RX25Z0wfPli5j7vfW9PgW0Fv\nOLCHT7NeoY2rv51uaYwDvN3z3wD8EHixtgYiEgo8B4wB9gHLRWSWqm7ysm/jIBHhzgt6ExkWwuNz\nt1BWUcnfrxtCZFjoKT9XWFrBztxCbv38AaCSqRf8yU63NMYBXoW/qm6GOm82NgzYrqoZ7rbvAuMA\nC/8AcMuonkSGhfDwJ5v45ZsreeGGoVS6lF0Hi9iVV+x+LGLXwSJ25hWTV1hKSOR+onusYWTCBM5O\n8f9rIYzxR80x5t8Z2Fvt9T7grGbo1zSTCSO6ExEWygMfryft0c8pLP3uTdSSYiLpnhDN9/smkZoY\nTffEMyDye1zYq/733DfG+Fad4S8inwMdPbz1gKrOrEcfnv4s8HhlmYhMAiYBpKSk1GPVpqW4/qwU\n2kVHsHBLDikJrUlNiKZbQmtSE6NpE+npn1lys9dojPmfOsNfVUd72cc+oPqds7oAmbX0NQ2YBlVX\n+HrZr2lmYwd1ZOwgT/sJxpiWpjmOtC0HeotIdxGJAK4FZjVDv8YYY2rh7ameV4nIPuAcYLaIzHMv\n7yQicwBUtQK4A5gHbAZmqOpG78o2xhjjDW/P9vkI+MjD8kzgkmqv5wBzvOnLGGOM79gJ1sYYE4Qs\n/I0xJghZ+BtjTBCy8DfGmCBk4W+MMUGoxU7jKCK5wG4vVpEI5PmonJYg0LYHAm+bAm17IPC2KdC2\nB07epm6qmlTXh1ps+HtLRFbUZx5LfxFo2wOBt02Btj0QeNsUaNsDjd8mG/YxxpggZOFvjDFBKJDD\nf5rTBfhYoG0PBN42Bdr2QOBtU6BtDzRymwJ2zN8YY0ztAnnP3xhjTC0CLvwDcbJ4EdklIutFZI2I\nrHC6noYSkVdFJEdENlRb1k5E5otIuvuxrZM1NlQt2/SwiOx3f09rROSSU62jJRGRriKyUEQ2i8hG\nEbnLvdwvv6dTbI8/f0dRIrJMRNa6t+mP7uXdReRb93f0nvvW+XWvL5CGfdyTxW+j2mTxwHX+Plm8\niOwC0lTVL89PFpHzgELgDVUd5F72FHBIVZ9w/5Juq6q/cbLOhqhlmx4GClX1aSdrawwRSQaSVXWV\niMQAK4ErgQn44fd0iu0Zj/9+RwJEq2qhiIQDXwN3AfcA/1bVd0Xkn8BaVX2hrvUF2p7/icniVbUM\nOD5ZvHGQqi4CDtVYPA6Y7n4+naofTL9Ryzb5LVXNUtVV7udHqZp7ozN++j2dYnv8llYpdL8Md/+n\nwA+AD9zL6/0dBVr4e5os3q+/cDcFPhORle55jgNBB1XNgqofVKC9w/X4yh0iss49LOQXQyQ1iUgq\nMAT4lgD4nmpsD/jxdyQioSKyBsgB5gM7gHz3pFnQgMwLtPCv92TxfmaEqp4BXAzc7h5yMC3PC0BP\nYDCQBTzjbDkNJyJtgA+Bu1X1iNP1eMvD9vj1d6Sqlao6mKq50IcB/T01q8+6Ai386z1ZvD9xz4yG\nquZQNXPaMGcr8ols97js8fHZHIfr8ZqqZrt/OF3AS/jZ9+QeR/4QeFtV/+1e7Lffk6ft8ffv6DhV\nzQe+BM4G4kXk+KyM9c68QAv/gJssXkSi3QesEJFo4EJgw6k/5RdmATe5n98EzHSwFp84HpJuV+FH\n35P7YOIrwGZV/Wu1t/zye6pte/z8O0oSkXj381bAaKqOZSwEfuxuVu/vKKDO9gFwn7r1NyAUeFVV\nH3O4JK+ISA/+N09yGPCOv22TiPwLOJ+quw9mA38APgZmACnAHuBqVfWbA6i1bNP5VA0nKLALuOX4\neHlLJyLnAouB9YDLvfh3VI2T+933dIrtuQ7//Y5Op+qAbihVO+4zVPURd0a8C7QDVgM3qGppnesL\ntPA3xhhTt0Ab9jHGGFMPFv7GGBOELPyNMSYIWfgbY0wQsvA3xpggZOFvjDFByMLfGGOCkIW/McYE\nof8HEEmgful2KAkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11d67f780>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(apple_stock[:30])\n",
    "i1 = 0\n",
    "plt.plot(range(i1,i1+5),X_[i1])\n",
    "plt.scatter(i1+5, y[i1], marker='x', color='k')\n",
    "i1 = 10\n",
    "plt.plot(range(i1,i1+5),X_[i1])\n",
    "plt.scatter(i1+5, y[i1], marker='x', color='k')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modelos de lenguaje\n",
    "Predecir el siguiente caractar en función de los caracteres anteriores\n",
    "- Utilizar one-hot encoding para los caracteres\n",
    "- Tamaño de ventana de 100\n",
    "- Cantidad total de caracteres de la obra: 67561\n",
    "- Cantidad de caracteres diferentes: 71"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----------------------------------------------------------------------------------------------------\n",
      "-----------------------------------------------------------------------------------------------------\n",
      "Cantidad de caracteres: 67561\n",
      "Cantidad de caracteres unicos: 71\n",
      "['\\n', ' ', '!', '(', ')', ',', '.', '1', '2', '3', '9', ':', ';', '?', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'x', 'y', 'z', '¡', '¿', 'É', 'Ñ', 'á', 'é', 'í', 'ñ', 'ó', 'ú']\n",
      "-----------------------------------------------------------------------------------------------------\n",
      "-----------------------------------------------------------------------------------------------------\n",
      "\n",
      "BODAS DE SANGRE(1933)\n",
      "\n",
      "Poema trágico en tres actos y siete cuadros\n",
      "\n",
      "Personajes\n",
      " \n",
      "Madre\n",
      "Criada\n",
      "Leonardo\n",
      "Mozos\n",
      "Novia\n",
      "Vecina\n",
      "Novio\n",
      "Leñadores\n",
      "Suegra\n",
      "Muchachas\n",
      "Padre de la novia\n",
      "Mozos\n",
      "Mujer de Leonardo\n",
      "Luna\n",
      "Muerte (como mendiga)\n",
      "\n",
      "Acto primero\n",
      "\n",
      "CUADRO PRIMERO\n",
      "\n",
      "Habitación pintada de amarillo.\n",
      "\n",
      "NOVIO: (Entrando) Madre.\n",
      "\n",
      "MADRE: ¿Que?\n",
      "\n",
      "NOVIO:Me voy.\n",
      "\n",
      "MADRE: ¿Adónde?\n",
      "\n",
      "NOVIO:A la viña. (Va a salir)\n",
      "\n",
      "MADRE: Espera.\n",
      "\n",
      "NOVIO:¿Quieres algo?\n",
      "\n",
      "MADRE: Hijo, el almuerzo.\n",
      "\n",
      "NOVIO: Déjalo. Comeré uvas. Dame la navaja.\n",
      "\n",
      "MADRE: ¿Para qué?\n",
      "\n",
      "NOVIO: (Riendo) Para cortarlas.\n",
      "\n",
      "MADRE: (Entre dientes y buscándola) La navaja, la navaja... Malditas sean todas y el bribón que las inventó.\n",
      "\n",
      "NOVIO: Vamos a otro asunto.\n",
      "\n",
      "MADRE: Y las escopetas, y las pistolas, y el cuchillo más pequeño, y hasta las azadas y los bieldos de la era.\n",
      "\n",
      "NOVIO: Bueno.\n",
      "\n",
      "MADRE: Todo lo que puede cortar el cuerpo de un hombre. Un hombre hermoso, con su flor en la boca, que sale a las viñas o va a sus olivos propios, porque son de él, heredados...\n",
      "\n"
     ]
    }
   ],
   "source": [
    "text = open('federico-garcia-lorca/bodas_de_sangre.txt').read()\n",
    "number_of_chars = len(text)\n",
    "all_chars = sorted(set(text))\n",
    "print('-----------------------------------------------------------------------------------------------------')\n",
    "print('-----------------------------------------------------------------------------------------------------')\n",
    "print('Cantidad de caracteres: ' + str(number_of_chars))\n",
    "print('Cantidad de caracteres unicos: ' + str(len(all_chars)))\n",
    "print(all_chars)\n",
    "print('-----------------------------------------------------------------------------------------------------')\n",
    "print('-----------------------------------------------------------------------------------------------------')\n",
    "print()\n",
    "print(text[:997])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from RNN_utils import window_transform_text, encode_io_pairs\n",
    "\n",
    "window_size = 100\n",
    "step_size = 1\n",
    "inputs, outputs = window_transform_text(text,window_size,step_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BODAS DE SANGRE(1933)\n",
      "\n",
      "Poema trágico en tres actos y siete cuadros\n",
      "\n",
      "Personajes\n",
      " \n",
      "Madre\n",
      "Criada\n",
      "Leonar\n",
      "\n",
      "-----------------------------------------------------------------------------------------------------\n",
      "Salida: d\n",
      "-----------------------------------------------------------------------------------------------------\n",
      "\n",
      "ODAS DE SANGRE(1933)\n",
      "\n",
      "Poema trágico en tres actos y siete cuadros\n",
      "\n",
      "Personajes\n",
      " \n",
      "Madre\n",
      "Criada\n",
      "Leonard\n",
      "-----------------------------------------------------------------------------------------------------\n",
      "Salida: o\n",
      "-----------------------------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "print(inputs[0])\n",
    "print()\n",
    "print('-----------------------------------------------------------------------------------------------------')\n",
    "print('Salida:',outputs[0])\n",
    "print('-----------------------------------------------------------------------------------------------------')\n",
    "print()\n",
    "print(inputs[1])\n",
    "print('-----------------------------------------------------------------------------------------------------')\n",
    "print('Salida:',outputs[1])\n",
    "print('-----------------------------------------------------------------------------------------------------')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ¿Como ingresamos a la red con caracteres?\n",
    "### Categorical Data (One-hot encoding)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X, y = encode_io_pairs(text, all_chars, window_size,step_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Caracteres codificado\n",
      "[[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
      " [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0\n",
      "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
      " [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
      " [0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
      " [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0\n",
      "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]]\n"
     ]
    }
   ],
   "source": [
    "print('Caracteres codificado')\n",
    "print(X[0,0:5].astype(int))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(N,T,D):\n",
      "(67461, 100, 71)\n"
     ]
    }
   ],
   "source": [
    "print('(N,T,D):')\n",
    "print(X.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RNN en Keras\n",
    "Definamos una capa RNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import SimpleRNN, Dense\n",
    "from keras.models import Sequential"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SimpleRNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "simple_rnn_4 (SimpleRNN)     (None, 3)                 21        \n",
      "=================================================================\n",
      "Total params: 21\n",
      "Trainable params: 21\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "rnn_neurons = 3\n",
    "time_steps = 100# T\n",
    "n_features = 3 # D\n",
    "input_shape = (time_steps, n_features)\n",
    "model = Sequential()\n",
    "model.add(SimpleRNN(rnn_neurons, input_shape = input_shape))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observaciones:\n",
    "- Crece la cantidad de parametros con la cantidad de time_steps?\n",
    "- Como puedo utilizar un MLP para que tenga en cuenta los time_steps? Que desventajas tengo?\n",
    "- rnn_neurons = 1, n_features = 1, por que es 3? Cuales son en el diagrama?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_number_of_parameters(rnn_neurons, n_features):\n",
    "    parametros = rnn_neurons*n_features + rnn_neurons**2 + rnn_neurons\n",
    "    return parametros"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_number_of_parameters(3,3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SimpleRNN + Dense para predecir caracteres\n",
    "Armar modelo con los siguientes datos:\n",
    "N,T,D = (67461, 100, 71)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "simple_rnn_5 (SimpleRNN)     (None, 200)               54400     \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 71)                14271     \n",
      "=================================================================\n",
      "Total params: 68,671\n",
      "Trainable params: 68,671\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# (67461, 100, 71)\n",
    "rnn_neurons = 200 # Hyperparametro para jugar\n",
    "time_steps = 100 # T\n",
    "n_features =  71# D\n",
    "input_shape = (time_steps, n_features)\n",
    "model = Sequential()\n",
    "model.add(SimpleRNN(rnn_neurons, input_shape = input_shape))\n",
    "model.add(Dense(n_features, input_shape = input_shape, activation=\"softmax\"))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observaciones:\n",
    "- La cantidad de neuronas de la RNN queda para jugar (Overfitting, underfitting)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SimpleRNN + Dense para stock prices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "simple_rnn_19 (SimpleRNN)    (None, 10)                120       \n",
      "_________________________________________________________________\n",
      "dense_28 (Dense)             (None, 1)                 11        \n",
      "=================================================================\n",
      "Total params: 131\n",
      "Trainable params: 131\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "window_size = 40\n",
    "rnn_cells = 10\n",
    "model_rnn = Sequential()\n",
    "model_rnn.add(SimpleRNN(rnn_cells, input_shape = (window_size,1)))\n",
    "model_rnn.add(Dense(1))\n",
    "model_rnn.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observaciones:\n",
    "- Capa densa. Que función de activación estamos usando?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Un error común"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "simple_rnn_20 (SimpleRNN)    (None, 10)                510       \n",
      "_________________________________________________________________\n",
      "dense_29 (Dense)             (None, 1)                 11        \n",
      "=================================================================\n",
      "Total params: 521\n",
      "Trainable params: 521\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model_rnn_error = Sequential()\n",
    "model_rnn_error.add(SimpleRNN(rnn_cells, input_shape = (1,window_size)))\n",
    "model_rnn_error.add(Dense(1))\n",
    "model_rnn_error.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Preguntas:\n",
    "- Que diferencia hay con la anterior?\n",
    "- Por que la diferencia en cantidad de parámetros?\n",
    "- Se puede pensar como un FNN (MLP)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FFN (MLP)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_25 (Dense)             (None, 10)                410       \n",
      "_________________________________________________________________\n",
      "dense_26 (Dense)             (None, 1)                 11        \n",
      "=================================================================\n",
      "Total params: 421\n",
      "Trainable params: 421\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model_fnn = Sequential()\n",
    "model_fnn.add(Dense(rnn_cells, input_shape=(window_size,) ))\n",
    "model_fnn.add(Dense(1))\n",
    "model_fnn.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RNN\n",
      "(1, 10)\n",
      "(10, 10)\n",
      "(10,)\n",
      "\n",
      "RNN common error\n",
      "(40, 10)\n",
      "(10, 10)\n",
      "(10,)\n",
      "\n",
      "MLP\n",
      "(40, 10)\n",
      "(10,)\n"
     ]
    }
   ],
   "source": [
    "print('RNN')\n",
    "print(model_rnn.layers[0].get_weights()[0].shape)\n",
    "print(model_rnn.layers[0].get_weights()[1].shape)\n",
    "print(model_rnn.layers[0].get_weights()[2].shape)\n",
    "print()\n",
    "print('RNN common error')\n",
    "print(model_rnn_error.layers[0].get_weights()[0].shape)\n",
    "print(model_rnn_error.layers[0].get_weights()[1].shape)\n",
    "print(model_rnn_error.layers[0].get_weights()[2].shape)\n",
    "print()\n",
    "print('MLP')\n",
    "print(model_fnn.layers[0].get_weights()[0].shape)\n",
    "print(model_fnn.layers[0].get_weights()[1].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stacking RNNs\n",
    "### Podemos stackear varias capas?\n",
    "![stack%20units.png](stack units.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# No funciona por que la salida no tiene la dimensión correcta\n",
    "# (67461, 100, 71)\n",
    "rnn_neurons = 100 # Hyperparametro para jugar\n",
    "time_steps = 100 # T\n",
    "n_features =  71# D\n",
    "input_shape = (time_steps, n_features)\n",
    "model = Sequential()\n",
    "model.add(SimpleRNN(rnn_neurons, input_shape = input_shape))\n",
    "model.add(SimpleRNN(rnn_neurons))\n",
    "model.add(Dense(n_features, input_shape = input_shape, activation=\"softmax\"))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Por que falla?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# (67461, 100, 71)\n",
    "rnn_neurons_1 = 200 # Hyperparametro para jugar\n",
    "rnn_neurons_2 = 150 # Hyperparametro para jugar\n",
    "time_steps = 100 # T\n",
    "n_features =  71# D\n",
    "input_shape = (time_steps, n_features)\n",
    "print(input_shape)\n",
    "model = Sequential()\n",
    "model.add(SimpleRNN(rnn_neurons_1, input_shape = input_shape, return_sequences=True))\n",
    "model.add(SimpleRNN(rnn_neurons_2))\n",
    "model.add(Dense(n_features, input_shape = input_shape, activation=\"softmax\"))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BPTT (Back-Propagation Through Time)\n",
    "https://machinelearningmastery.com/truncated-backpropagation-through-time-in-keras/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![BPTT.png](BPTT.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Con esto podemos calcular la \"cross-entropy\" y derivar respecto a $W_o$,$W_h$ y W_x para hallar el gradiente."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![BPTT_graph.png](BPTT_graph.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Problema: Vanishing/Exploding Gradient\n",
    "Soluciones:\n",
    "- Gradient Cliping\n",
    "- Truncated BPTT (Limitar la cantidad de unfoldings)\n",
    "- LSTM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LSTM (Long Short Term Memory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![lstm.png](lstm.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- f -> forget Gate: \"Porcentaje\" que se queda de la LTM anterior\n",
    "- i -> Remeber Gate: \"Porcentaje\" que pasa de lo actual\n",
    "- o -> Learn Gate: \"Porcentaje\" a la salida o H (Recordar que h y la salida son lo mismo o STM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "lstm_1 (LSTM)                (None, 1)                 12        \n",
      "=================================================================\n",
      "Total params: 12\n",
      "Trainable params: 12\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.layers import LSTM\n",
    "rnn_neurons = 1\n",
    "time_steps = 100 # T\n",
    "n_features = 1 # D\n",
    "input_shape = (time_steps, n_features)\n",
    "model = Sequential()\n",
    "model.add(LSTM(rnn_neurons, input_shape = input_shape))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_number_of_LSTM_params(rnn_neurons, n_features):\n",
    "    return (rnn_neurons*n_features + rnn_neurons**2 + rnn_neurons)*4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_number_of_LSTM_params(rnn_neurons, n_features)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}