End_to_End_Transformer.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jarvisx17/Transformer/blob/main/transformer.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gvP_7iee3RyW"
      },
      "source": [
        "# Language Translation with Transformers Models"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QwLP-wiG7CVB",
        "outputId": "e26db435-7737-47e5-a56b-f18c7fbd4381"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "toc": true,
        "id": "yQ67KOC263DV"
      },
      "source": [
        "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
        "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Introduction\" data-toc-modified-id=\"Introduction-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Introduction</a></span><ul class=\"toc-item\"><li><span><a href=\"#Motivation\" data-toc-modified-id=\"Motivation-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Motivation</a></span></li><li><span><a href=\"#Approach\" data-toc-modified-id=\"Approach-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Approach</a></span></li><li><span><a href=\"#Steps\" data-toc-modified-id=\"Steps-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>Steps</a></span></li></ul></li><li><span><a href=\"#Importing-dependencies\" data-toc-modified-id=\"Importing-dependencies-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Importing dependencies</a></span><ul class=\"toc-item\"><li><span><a href=\"#Check-Tensorflow-version-and-verify-we-are-using-GPU\" data-toc-modified-id=\"Check-Tensorflow-version-and-verify-we-are-using-GPU-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>Check Tensorflow version and verify we are using GPU</a></span></li></ul></li><li><span><a href=\"#Download-the-datasets\" data-toc-modified-id=\"Download-the-datasets-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Download the datasets</a></span><ul class=\"toc-item\"><li><span><a href=\"#Helpful-functions\" data-toc-modified-id=\"Helpful-functions-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>Helpful functions</a></span></li><li><span><a href=\"#Define-the-url-paths-for-our-data-downloads-and-create-data-directory\" data-toc-modified-id=\"Define-the-url-paths-for-our-data-downloads-and-create-data-directory-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>Define the url paths for our data downloads and create data directory</a></span></li><li><span><a href=\"#Download-the-data\" data-toc-modified-id=\"Download-the-data-3.3\"><span class=\"toc-item-num\">3.3&nbsp;&nbsp;</span>Download the data</a></span><ul class=\"toc-item\"><li><span><a href=\"#Save-the-files-to-./data\" data-toc-modified-id=\"Save-the-files-to-./data-3.3.1\"><span class=\"toc-item-num\">3.3.1&nbsp;&nbsp;</span>Save the files to <code>./data</code></a></span></li><li><span><a href=\"#Extract-the-data\" data-toc-modified-id=\"Extract-the-data-3.3.2\"><span class=\"toc-item-num\">3.3.2&nbsp;&nbsp;</span>Extract the data</a></span></li><li><span><a href=\"#Remove-the-compressed-files\" data-toc-modified-id=\"Remove-the-compressed-files-3.3.3\"><span class=\"toc-item-num\">3.3.3&nbsp;&nbsp;</span>Remove the compressed files</a></span></li></ul></li></ul></li><li><span><a href=\"#Data-preprocessing\" data-toc-modified-id=\"Data-preprocessing-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Data preprocessing</a></span><ul class=\"toc-item\"><li><span><a href=\"#Loading-files\" data-toc-modified-id=\"Loading-files-4.1\"><span class=\"toc-item-num\">4.1&nbsp;&nbsp;</span>Loading files</a></span></li><li><span><a href=\"#Cleaning-data\" data-toc-modified-id=\"Cleaning-data-4.2\"><span class=\"toc-item-num\">4.2&nbsp;&nbsp;</span>Cleaning data</a></span><ul class=\"toc-item\"><li><span><a href=\"#Process-non-breaking-prefixes-in-corpus,-and-remove-consecutive-spaces\" data-toc-modified-id=\"Process-non-breaking-prefixes-in-corpus,-and-remove-consecutive-spaces-4.2.1\"><span class=\"toc-item-num\">4.2.1&nbsp;&nbsp;</span>Process non-breaking prefixes in corpus, and remove consecutive spaces</a></span></li></ul></li><li><span><a href=\"#Tokenizing-Text\" data-toc-modified-id=\"Tokenizing-Text-4.3\"><span class=\"toc-item-num\">4.3&nbsp;&nbsp;</span>Tokenizing Text</a></span></li><li><span><a href=\"#The-vocabulary-size-will-be-used-to-build-the-start-and-end-tokens\" data-toc-modified-id=\"The-vocabulary-size-will-be-used-to-build-the-start-and-end-tokens-4.4\"><span class=\"toc-item-num\">4.4&nbsp;&nbsp;</span>The vocabulary size will be used to build the <code>start</code> and <code>end</code> tokens</a></span></li><li><span><a href=\"#Remove-very-long-sentences\" data-toc-modified-id=\"Remove-very-long-sentences-4.5\"><span class=\"toc-item-num\">4.5&nbsp;&nbsp;</span>Remove very long sentences</a></span></li><li><span><a href=\"#Input-/-Output-creation\" data-toc-modified-id=\"Input-/-Output-creation-4.6\"><span class=\"toc-item-num\">4.6&nbsp;&nbsp;</span>Input / Output creation</a></span></li><li><span><a href=\"#Create-the-datasets\" data-toc-modified-id=\"Create-the-datasets-4.7\"><span class=\"toc-item-num\">4.7&nbsp;&nbsp;</span>Create the datasets</a></span></li></ul></li><li><span><a href=\"#Model-Building\" data-toc-modified-id=\"Model-Building-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Model Building</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Model-Composition\" data-toc-modified-id=\"Model-Composition-5.0.1\"><span class=\"toc-item-num\">5.0.1&nbsp;&nbsp;</span>Model Composition</a></span></li></ul></li><li><span><a href=\"#Positional-Encoding\" data-toc-modified-id=\"Positional-Encoding-5.1\"><span class=\"toc-item-num\">5.1&nbsp;&nbsp;</span>Positional Encoding</a></span></li><li><span><a href=\"#Self-Attention\" data-toc-modified-id=\"Self-Attention-5.2\"><span class=\"toc-item-num\">5.2&nbsp;&nbsp;</span>Self-Attention</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Linear-Transformation\" data-toc-modified-id=\"Linear-Transformation-5.2.0.1\"><span class=\"toc-item-num\">5.2.0.1&nbsp;&nbsp;</span>Linear Transformation</a></span></li></ul></li><li><span><a href=\"#Calculate-the-self-attention:\" data-toc-modified-id=\"Calculate-the-self-attention:-5.2.1\"><span class=\"toc-item-num\">5.2.1&nbsp;&nbsp;</span>Calculate the self-attention:</a></span></li><li><span><a href=\"#Multi-Head-Self-Attention\" data-toc-modified-id=\"Multi-Head-Self-Attention-5.2.2\"><span class=\"toc-item-num\">5.2.2&nbsp;&nbsp;</span>Multi-Head Self-Attention</a></span></li><li><span><a href=\"#Attention-Masks\" data-toc-modified-id=\"Attention-Masks-5.2.3\"><span class=\"toc-item-num\">5.2.3&nbsp;&nbsp;</span>Attention Masks</a></span><ul class=\"toc-item\"><li><span><a href=\"#Padding-Mask\" data-toc-modified-id=\"Padding-Mask-5.2.3.1\"><span class=\"toc-item-num\">5.2.3.1&nbsp;&nbsp;</span>Padding Mask</a></span></li><li><span><a href=\"#Look-Ahead-Mask\" data-toc-modified-id=\"Look-Ahead-Mask-5.2.3.2\"><span class=\"toc-item-num\">5.2.3.2&nbsp;&nbsp;</span>Look-Ahead Mask</a></span></li></ul></li><li><span><a href=\"#Attention-computation\" data-toc-modified-id=\"Attention-computation-5.2.4\"><span class=\"toc-item-num\">5.2.4&nbsp;&nbsp;</span>Attention computation</a></span></li><li><span><a href=\"#Multi-head-attention-sublayer\" data-toc-modified-id=\"Multi-head-attention-sublayer-5.2.5\"><span class=\"toc-item-num\">5.2.5&nbsp;&nbsp;</span>Multi-head attention sublayer</a></span></li></ul></li><li><span><a href=\"#Encoder\" data-toc-modified-id=\"Encoder-5.3\"><span class=\"toc-item-num\">5.3&nbsp;&nbsp;</span>Encoder</a></span><ul class=\"toc-item\"><li><span><a href=\"#EncoderLayer-Class\" data-toc-modified-id=\"EncoderLayer-Class-5.3.1\"><span class=\"toc-item-num\">5.3.1&nbsp;&nbsp;</span>EncoderLayer Class</a></span></li><li><span><a href=\"#Encoder-Class\" data-toc-modified-id=\"Encoder-Class-5.3.2\"><span class=\"toc-item-num\">5.3.2&nbsp;&nbsp;</span>Encoder Class</a></span></li></ul></li><li><span><a href=\"#Decoder\" data-toc-modified-id=\"Decoder-5.4\"><span class=\"toc-item-num\">5.4&nbsp;&nbsp;</span>Decoder</a></span><ul class=\"toc-item\"><li><span><a href=\"#DecoderLayer-Class\" data-toc-modified-id=\"DecoderLayer-Class-5.4.1\"><span class=\"toc-item-num\">5.4.1&nbsp;&nbsp;</span>DecoderLayer Class</a></span></li><li><span><a href=\"#Decoder-Class\" data-toc-modified-id=\"Decoder-Class-5.4.2\"><span class=\"toc-item-num\">5.4.2&nbsp;&nbsp;</span>Decoder Class</a></span></li></ul></li><li><span><a href=\"#Transformer\" data-toc-modified-id=\"Transformer-5.5\"><span class=\"toc-item-num\">5.5&nbsp;&nbsp;</span>Transformer</a></span><ul class=\"toc-item\"><li><span><a href=\"#Transformer-Class\" data-toc-modified-id=\"Transformer-Class-5.5.1\"><span class=\"toc-item-num\">5.5.1&nbsp;&nbsp;</span>Transformer Class</a></span></li></ul></li></ul></li><li><span><a href=\"#Training\" data-toc-modified-id=\"Training-6\"><span class=\"toc-item-num\">6&nbsp;&nbsp;</span>Training</a></span><ul class=\"toc-item\"><li><span><a href=\"#Define-Hyper-parameter-values\" data-toc-modified-id=\"Define-Hyper-parameter-values-6.1\"><span class=\"toc-item-num\">6.1&nbsp;&nbsp;</span>Define Hyper-parameter values</a></span></li><li><span><a href=\"#Define-the-loss-function\" data-toc-modified-id=\"Define-the-loss-function-6.2\"><span class=\"toc-item-num\">6.2&nbsp;&nbsp;</span>Define the loss function</a></span></li><li><span><a href=\"#Create-a-custom-learning-rate-schedule\" data-toc-modified-id=\"Create-a-custom-learning-rate-schedule-6.3\"><span class=\"toc-item-num\">6.3&nbsp;&nbsp;</span>Create a custom learning rate schedule</a></span><ul class=\"toc-item\"><li><span><a href=\"#Plot-the-learning-rate\" data-toc-modified-id=\"Plot-the-learning-rate-6.3.1\"><span class=\"toc-item-num\">6.3.1&nbsp;&nbsp;</span>Plot the learning rate</a></span></li></ul></li><li><span><a href=\"#Model-checkpoints\" data-toc-modified-id=\"Model-checkpoints-6.4\"><span class=\"toc-item-num\">6.4&nbsp;&nbsp;</span>Model checkpoints</a></span></li><li><span><a href=\"#Run-the-training\" data-toc-modified-id=\"Run-the-training-6.5\"><span class=\"toc-item-num\">6.5&nbsp;&nbsp;</span>Run the training</a></span></li></ul></li><li><span><a href=\"#Evaluating\" data-toc-modified-id=\"Evaluating-7\"><span class=\"toc-item-num\">7&nbsp;&nbsp;</span>Evaluating</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Translate-function\" data-toc-modified-id=\"Translate-function-7.0.1\"><span class=\"toc-item-num\">7.0.1&nbsp;&nbsp;</span>Translate function</a></span></li></ul></li><li><span><a href=\"#Let's-translate\" data-toc-modified-id=\"Let's-translate-7.1\"><span class=\"toc-item-num\">7.1&nbsp;&nbsp;</span>Let's translate</a></span></li></ul></li><li><span><a href=\"#Results\" data-toc-modified-id=\"Results-8\"><span class=\"toc-item-num\">8&nbsp;&nbsp;</span>Results</a></span></li><li><span><a href=\"#Future-Work\" data-toc-modified-id=\"Future-Work-9\"><span class=\"toc-item-num\">9&nbsp;&nbsp;</span>Future Work</a></span></li><li><span><a href=\"#Summary\" data-toc-modified-id=\"Summary-10\"><span class=\"toc-item-num\">10&nbsp;&nbsp;</span>Summary</a></span></li></ul></div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vl4vPqA4DcJZ"
      },
      "source": [
        "## Introduction\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "trCZZUC063Dk"
      },
      "source": [
        "### Motivation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BEBN5yVu63Dm"
      },
      "source": [
        "I undertook this project in order to better understand how `Transformer Models` work. I became interested in the technology when I set out to work on a language translation model using an `RNN`. However, in learning about `RNN`s and `LSTM`s I came across `Transformers`. As I started to learn more I realiazed that I needed shift to this model for my project.\n",
        "\n",
        "Since the paper [Attention is all you need](https://arxiv.org/abs/1706.03762) was published in 2017, introducing transformers,  they and their many variants have become the models of choice for `Natural Language Processing` - `NLP`. They are used for to solve many types sequence to sequence problems including language translation, information retrieval, text classification, document summarization, image captioning, and genome analysis. More recently they are showing great results in `image recognition` and `object detection`.\n",
        "\n",
        "The transformer continues to evolve with new variants such as BERT, GPT-2, GPT-3, and others. And research continues on how to improve them.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I3dUx5nu63Dr"
      },
      "source": [
        "### Approach\n",
        "\n",
        "The model is build from scratch following the method used in the original [Attention is all you need](https://arxiv.org/abs/1706.03762) paper. I also used a number of articles, which I will cite, to better learn and understand the details of the transformer\n",
        "\n",
        "As this model is built using the `Tensorflow` framework, I leveraged their transformer [tutorial](https://www.tensorflow.org/tutorials/text/transformer) for code inspiration.\n",
        "\n",
        "I also relied heavily on Jay Alammar's excellent blog post [The Illustrated Transformer](http://jalammar.github.io/illustrated-transformer/). It's kinda become the standard go-to for the visual representation and description of transformers.\n",
        "\n",
        "Another great blog post is [What Are Transformer Models in Machine Learning?](https://lionbridge.ai/articles/what-are-transformer-models-in-machine-learning/) by Rahul Agarwal.\n",
        "\n",
        "No doubt that you too will find many other great articles."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tz4rItSS63Dv"
      },
      "source": [
        "### Steps\n",
        "\n",
        "1. We start by loading all of the dependencies\n",
        "2. Download the and extract data\n",
        "3. Do some data preprocessing\n",
        "4. Tokenize the data\n",
        "5. Prep data for embedding\n",
        "6. Explain and code Positional Encoding\n",
        "7. Explain and code Self-attention\n",
        "8. Explain and code the Encoder\n",
        "9. Explain and code the Decoder\n",
        "10. Explain and code the Transformer\n",
        "11. Train the transformer is explaination\n",
        "12. Expalain how the model does inferencing.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m9JJ7FBw84tG"
      },
      "source": [
        "## Importing dependencies"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7bi5h91M3EC0",
        "outputId": "1f4e40a0-7d81-42b0-bccf-1696be47227c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive/; to attempt to forcibly remount, call drive.mount(\"/content/drive/\", force_remount=True).\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "import sys\n",
        "import tarfile\n",
        "import zipfile as zf\n",
        "import requests as rq\n",
        "\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "import math\n",
        "import re\n",
        "import time\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive/')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2oGNiOFjQYPW"
      },
      "outputs": [],
      "source": [
        "os.chdir('drive/My Drive')\n",
        "ROOT = os.getcwd()\n",
        "os.chdir(ROOT + '/Colab Notebooks')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Z_zxs_3d3lK2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7386f5a1-9562-40be-88ca-77d38533729a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Colab only includes TensorFlow 2.x; %tensorflow_version has no effect.\n"
          ]
        }
      ],
      "source": [
        "try:\n",
        "    %tensorflow_version 2.x\n",
        "except:\n",
        "    pass\n",
        "import tensorflow as tf\n",
        "\n",
        "from tensorflow.keras import layers\n",
        "import tensorflow_datasets as tfds"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aaOAIH2QUNR1"
      },
      "source": [
        "### Check Tensorflow version and verify we are using GPU"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IeCUyFKx05j0",
        "outputId": "c23b5e91-30c5-4ac4-8d53-595e1111f7ba"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TensorFlow Version: 2.9.2\n",
            "Using CUDA: True\n",
            "Using GPU: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]\n",
            "Num GPUs Available:  1\n"
          ]
        }
      ],
      "source": [
        "print(f'TensorFlow Version: {tf.__version__}')\n",
        "print(f'Using CUDA: {tf.test.is_built_with_cuda()}')\n",
        "print(f'Using GPU: {tf.config.list_physical_devices(\"GPU\")}')\n",
        "physical_devices = tf.config.experimental.list_physical_devices('GPU')\n",
        "print(\"Num GPUs Available: \", len(physical_devices))\n",
        "\n",
        "tf.config.experimental.set_memory_growth(physical_devices[0], True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M-SfYEYK63EH"
      },
      "source": [
        "## Download the datasets\n",
        "In this section we download the datasets and supporting files using the `Requests` module. We then extract the corpuses from the tar file to the data directory. We also download two zip files which are also extracted to the data directory.\n",
        "\n",
        "The dataset comes from the `European Parliament Proceedings Parallel Corpus 1996-2011` found at the [`Statistical Machine Translation`](http://www.statmt.org/europarl/) website. Specifically, we use the [`French-English parallel corpus`](http://www.statmt.org/europarl/v7/fr-en.tgz) . The following description is cited from their site.\n",
        "\n",
        ">\"*The Europarl parallel corpus is extracted from the proceedings of the European Parliament. It includes versions in 21 European languages: Romanic (French, Italian, Spanish, Portuguese, Romanian), Germanic (English, Dutch, German, Danish, Swedish), Slavik (Bulgarian, Czech, Polish, Slovak, Slovene), Finni-Ugric (Finnish, Hungarian, Estonian), Baltic (Latvian, Lithuanian), and Greek.*\"\n",
        "\n",
        "The two zip files contain lists of non-breaking prefixes in French and english. These will be applied in the preprocessing of the datasets. The Urls can be found in the code.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5BYVMjrkRFpg"
      },
      "source": [
        "### Helpful functions\n",
        "The first function simply extracts our datasets from the tarball. The second one removes the tar and zip files after their contents has been extracted."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "esWTmxHeTPdb"
      },
      "outputs": [],
      "source": [
        "#https://stackoverflow.com/questions/6058786/i-want-to-extract-a-tgz-file-and-extract-any-subdirectories-that-have-files-tha\n",
        "def extract(tar_url, extract_path='.'):\n",
        "    ''' Fetch tar file and extract to specified path\n",
        "    '''\n",
        "    tar = tarfile.open(tar_url, 'r')\n",
        "    for item in tar:\n",
        "        tar.extract(item, extract_path)\n",
        "        if item.name.find(\".tgz\") != -1 or item.name.find(\".tar\") != -1:\n",
        "            extract(item.name, \"./\" + item.name[:item.name.rfind('/')])\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Vpi0IWEHuGrP"
      },
      "outputs": [],
      "source": [
        "def remove_files(dir, filelist):\n",
        "    '''Remove files in filelist from dir\n",
        "    '''\n",
        "    for f in filelist:\n",
        "        print(f'Removing file: {f}')\n",
        "        os.remove(os.path.join(dir, f))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IRwgFtXq63EN"
      },
      "source": [
        "### Define the url paths for our data downloads and create data directory"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Emz3okSbRG78"
      },
      "outputs": [],
      "source": [
        "EN_FR_URL = 'http://www.statmt.org/europarl/v7/fr-en.tgz'\n",
        "PRE_EN_URL = 'https://sds-platform-private.s3-us-east-2.amazonaws.com/uploads/P85-Non-Breaking-Prefix-en.zip'\n",
        "PRE_FR_URL = 'https://sds-platform-private.s3-us-east-2.amazonaws.com/uploads/P85-Non-Breaking-Prefix-fr.zip'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sGfUdXlrXvOc"
      },
      "outputs": [],
      "source": [
        "# Create data directory if necessary\n",
        "if os.path.isdir('../data') is False:\n",
        "    os.mkdir('../data')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bs2Pr7lF63EQ"
      },
      "source": [
        "### Download the data\n",
        "* Use the requests library to download the files\n",
        "* Write the files to the data directory\n",
        "* Extract the data from the files\n",
        "* Remove the compressed files from data folder\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "29RmxrHKa03C"
      },
      "outputs": [],
      "source": [
        "r_en_fr = rq.get(EN_FR_URL, stream=True)\n",
        "r_pr_en = rq.get(PRE_EN_URL, stream=True)\n",
        "r_pr_fr = rq.get(PRE_FR_URL, stream=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p430OP3B63ES"
      },
      "source": [
        "#### Save the files to `./data`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nGNBgg9Qc9oH"
      },
      "outputs": [],
      "source": [
        "with open('../data/fr-en.tgz', 'wb') as fr_en:\n",
        "    for chunk in r_en_fr.iter_content(chunk_size=1024):\n",
        "        if chunk:\n",
        "            fr_en.write(chunk)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D1Z05p_RemyQ"
      },
      "outputs": [],
      "source": [
        "with open('../data/P85-Non-Breaking-Prefix-en.zip', 'wb') as pre_en:\n",
        "    for chunk in r_pr_en.iter_content(chunk_size=1024):\n",
        "        if chunk:\n",
        "            pre_en.write(chunk)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ic4MF-F-fDc1"
      },
      "outputs": [],
      "source": [
        "with open('../data/P85-Non-Breaking-Prefix-fr.zip', 'wb') as pre_fr:\n",
        "    for chunk in r_pr_fr.iter_content(chunk_size=1024):\n",
        "        if chunk:\n",
        "            pre_fr.write(chunk)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3RnGaW3563EV"
      },
      "source": [
        "#### Extract the data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "58ijRr1Zg9WQ"
      },
      "outputs": [],
      "source": [
        "extract('../data/fr-en.tgz', extract_path='../data')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8PT76buWphzb",
        "outputId": "4dbffecd-adde-4099-b070-b9c58486f034"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "File Name                                             Modified             Size\n",
            "P85-Non-Breaking-Prefix.en                     2019-11-29 10:20:32          110\n",
            "__MACOSX/._P85-Non-Breaking-Prefix.en          2019-11-29 10:20:32          761\n"
          ]
        }
      ],
      "source": [
        "import zipfile as zf\n",
        "with zf.ZipFile('../data/P85-Non-Breaking-Prefix-en.zip') as archive:\n",
        "    archive.printdir()\n",
        "    archive.extract('P85-Non-Breaking-Prefix.en', '../data/')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AKBuyshLphEn",
        "outputId": "6b5a5866-30a7-4a88-f381-472dce3e3a11"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "File Name                                             Modified             Size\n",
            "P85-Non-Breaking-Prefix.fr                     2019-11-29 10:20:58          121\n",
            "__MACOSX/._P85-Non-Breaking-Prefix.fr          2019-11-29 10:20:58          576\n"
          ]
        }
      ],
      "source": [
        "with zf.ZipFile('../data/P85-Non-Breaking-Prefix-fr.zip') as archive:\n",
        "    archive.printdir()\n",
        "    archive.extract('P85-Non-Breaking-Prefix.fr', '../data/')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b6EkqcG763EZ"
      },
      "source": [
        "#### Remove the compressed files"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1FucZYnxto3W",
        "outputId": "991a5326-b97f-448d-8b1c-7e7a702a1cf1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Removing file: P85-Non-Breaking-Prefix-en.zip\n",
            "Removing file: P85-Non-Breaking-Prefix-fr.zip\n",
            "Removing file: fr-en.tgz\n"
          ]
        }
      ],
      "source": [
        "filelist = ['P85-Non-Breaking-Prefix-en.zip', 'P85-Non-Breaking-Prefix-fr.zip', 'fr-en.tgz']\n",
        "remove_files('../data', filelist)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ITN6_EtO372t"
      },
      "source": [
        "## Data preprocessing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CMXOrfUu4M-8"
      },
      "source": [
        "### Loading files\n",
        "\n",
        "Now we assign the files to variables so that we can start the data preprocessing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LtR77eMg4xmZ"
      },
      "outputs": [],
      "source": [
        "with open(ROOT + \"/data/europarl-v7.fr-en.en\",\n",
        "          mode='r',\n",
        "          encoding='utf-8') as f:\n",
        "    europarl_en = f.read()\n",
        "\n",
        "with open(ROOT + \"/data/europarl-v7.fr-en.fr\",\n",
        "          mode='r',\n",
        "          encoding='utf-8') as f:\n",
        "    europarl_fr = f.read()\n",
        "\n",
        "with open(ROOT + \"/data/P85-Non-Breaking-Prefix.en\",\n",
        "          mode='r',\n",
        "          encoding='utf-8') as f:\n",
        "    non_breaking_prefix_en = f.read()\n",
        "\n",
        "with open(ROOT + \"/data/P85-Non-Breaking-Prefix.fr\",\n",
        "          mode='r',\n",
        "          encoding='utf-8') as f:\n",
        "    non_breaking_prefix_fr = f.read()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q9xgTFuQ4-JY"
      },
      "source": [
        "### Cleaning data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fq0OjX0M-hvZ"
      },
      "source": [
        "The first step in preparing the data is to create two `lists of non-breaking prefixes` from the two files that were downloaded. These non-breaking prefixes indicate words that do not mark the end of a sentence when encountered with a period.\n",
        "Getting the `lists of non-breaking prefixes` as a clean list of words with a period at the end so it is easier to use."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pzPOmGWS4_k5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "34a6ae69-c134-4d9e-d6f4-8e42c1a20126"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Before adding space and period:\n",
            "['st', 'a.m', 'p.m', 'vs', 'i.e', 'e.g']\n",
            "\n",
            "After adding space and period:\n",
            "[' st.', ' a.m.', ' p.m.', ' vs.', ' i.e.', ' e.g.']\n"
          ]
        }
      ],
      "source": [
        "# Split into a list\n",
        "non_breaking_prefix_en = non_breaking_prefix_en.split(\"\\n\")\n",
        "print('Before adding space and period:')\n",
        "print(non_breaking_prefix_en[len(non_breaking_prefix_en)-6:])\n",
        "print()\n",
        "\n",
        "# Add a space to the front and a period to the end\n",
        "non_breaking_prefix_en = [' ' + pref + '.' for pref in non_breaking_prefix_en]\n",
        "print('After adding space and period:')\n",
        "print(non_breaking_prefix_en[len(non_breaking_prefix_en)-6:])\n",
        "\n",
        "# Do the same for the French file\n",
        "non_breaking_prefix_fr = non_breaking_prefix_fr.split(\"\\n\")\n",
        "non_breaking_prefix_fr = [' ' + pref + '.' for pref in non_breaking_prefix_fr]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3q7Dds5D-xxW"
      },
      "source": [
        "#### Process non-breaking prefixes in corpus, and remove consecutive spaces\n",
        "We take the following steps to prepare the corpus.\n",
        "1. Iterate over the non-breaking prefix lists in order to find occurances in the corpus.\n",
        "2. Add `$$$` to the end of each prefix so that a\\they are easy to fine.\n",
        "3. Find all occurances of periods immediately followed by an alpha-numeric character and add `$$$` between the period and the character.\n",
        "4. Remove all occurances of `.$$$`. So that there are no longer any non-breaking periods.\n",
        "5. Replace consecutive spaces with a single space.\n",
        "6. Split the corpus into a list."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "spWxh3L9-yVw"
      },
      "outputs": [],
      "source": [
        "corpus_en = europarl_en\n",
        "\n",
        "# Step 1 & 2\n",
        "for prefix in non_breaking_prefix_en:\n",
        "    corpus_en = corpus_en.replace(prefix, prefix + '$$$')\n",
        "\n",
        "# Step 3\n",
        "corpus_en = re.sub(r\"\\.(?=[0-9]|[a-z]|[A-Z])\", \".$$$\", corpus_en)\n",
        "\n",
        "# Step 4\n",
        "corpus_en = re.sub(r\".\\$\\$\\$\", '', corpus_en)\n",
        "\n",
        "# Step 5\n",
        "corpus_en = re.sub(r\"  +\", \" \", corpus_en)\n",
        "\n",
        "# Step 6\n",
        "corpus_en = corpus_en.split('\\n')\n",
        "\n",
        "# Repeat all steps for the French corpus.\n",
        "corpus_fr = europarl_fr\n",
        "for prefix in non_breaking_prefix_fr:\n",
        "    corpus_fr = corpus_fr.replace(prefix, prefix + '$$$')\n",
        "\n",
        "corpus_fr = re.sub(r\"\\.(?=[0-9]|[a-z]|[A-Z])\", \".$$$\", corpus_fr)\n",
        "corpus_fr = re.sub(r\".\\$\\$\\$\", '', corpus_fr)\n",
        "corpus_fr = re.sub(r\"  +\", \" \", corpus_fr)\n",
        "corpus_fr = corpus_fr.split('\\n')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tRADLSal-963"
      },
      "source": [
        "### Tokenizing Text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sKJ1iqBJ63Ei"
      },
      "source": [
        "Use Tensorflow-datasets' `SubWordEncoder` to create a tokenizer from the corpus. Tokenizing is a type of preprocessing that decomposes parts of a given text into basic units. It encodes the units of text into a numerical representation which can be used for computation. The tokenizer will produce a vocabulary size of *$2^{13}$* or `8,192 tokens`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_wlMqYm4_BkN"
      },
      "outputs": [],
      "source": [
        "# Tokenizer for English\n",
        "tokenizer_en = tfds.deprecated.text.SubwordTextEncoder.build_from_corpus(\n",
        "    corpus_en, target_vocab_size=2**13)\n",
        "\n",
        "# Tokenizer for French\n",
        "tokenizer_fr = tfds.deprecated.text.SubwordTextEncoder.build_from_corpus(\n",
        "    corpus_fr, target_vocab_size=2**13)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2XBKSxrQ63Ek"
      },
      "source": [
        "### The vocabulary size will be used to build the `start` and `end` tokens"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jj8arK3I63El"
      },
      "source": [
        "`Start-of-sentence` and `end-of-sentence` tokens signal to the algorithm when a sentence starts and ends. Here the size of the vocabulary is used to created them."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "y0V816DV_RYY"
      },
      "outputs": [],
      "source": [
        "# Begin with the vocab_size + 2 as the base for the sos and eos tokens\n",
        "VOCAB_SIZE_EN = tokenizer_en.vocab_size + 2 # = 8190\n",
        "VOCAB_SIZE_FR = tokenizer_fr.vocab_size + 2 # = 8171"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rSYzhGZc_Tk9"
      },
      "outputs": [],
      "source": [
        "# Create the inputs and outputs by iterrating over the corpus and adding the tokens to each sentence\n",
        "inputs = [[VOCAB_SIZE_EN-2] + tokenizer_en.encode(sentence) + [VOCAB_SIZE_EN-1]\n",
        "          for sentence in corpus_en]\n",
        "\n",
        "outputs = [[VOCAB_SIZE_FR-2] + tokenizer_fr.encode(sentence) + [VOCAB_SIZE_FR-1]\n",
        "           for sentence in corpus_fr]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2nNbOO_6_ZHI"
      },
      "source": [
        "### Remove very long sentences"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JU0mCTNR63En"
      },
      "source": [
        "We will limit the length of the sentences to some maximum. This will help to speed up computation. To do so, iterate over the input and output lists using `Python's enumerate() function` in order to filter and capture the indexes of long sentences. We then use the indexes to delete the long sentences."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eWTnuxul_f2I"
      },
      "outputs": [],
      "source": [
        "MAX_LENGTH = 40\n",
        "# Use enumerate to capture indexes\n",
        "idx_to_remove = [count for count, sent in enumerate(inputs)\n",
        "                 if len(sent) > MAX_LENGTH]\n",
        "\n",
        "# Remove long sentences by index\n",
        "for idx in reversed(idx_to_remove):\n",
        "    del inputs[idx]\n",
        "    del outputs[idx]\n",
        "\n",
        "# Rinse and repeate!\n",
        "idx_to_remove = [count for count, sent in enumerate(outputs)\n",
        "                 if len(sent) > MAX_LENGTH]\n",
        "\n",
        "for idx in reversed(idx_to_remove):\n",
        "    del inputs[idx]\n",
        "    del outputs[idx]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BVZj1yLm_pCt"
      },
      "source": [
        "### Input / Output creation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TFLtUUmI_4cc"
      },
      "source": [
        "As we train with batches, we need each input to have the same length. We pad with the appropriate token, and we will make sure this padding token doesn't interfere with our training later. `Keras' pad_sequences() method` is perfect for this. We add zeros to the end as needed where sentences are less than `MAX_LENGTH`. Recall that the `end-of-sentence` tokens will signal the true end of the sentences."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "j2RAcIcm_siD"
      },
      "outputs": [],
      "source": [
        "inputs = tf.keras.preprocessing.sequence.pad_sequences(inputs,\n",
        "                                                       value=0,\n",
        "                                                       padding='post',\n",
        "                                                       maxlen=MAX_LENGTH)\n",
        "\n",
        "outputs = tf.keras.preprocessing.sequence.pad_sequences(outputs,\n",
        "                                                        value=0,\n",
        "                                                        padding='post',\n",
        "                                                        maxlen=MAX_LENGTH)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YA3xGXd-63Eq"
      },
      "source": [
        "### Create the datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f6gvtMV063Er"
      },
      "source": [
        "Passing our data into `Dataset.from_tensor_slices()` converts the data to tensorflow tensors. We pass both the English and French lists, which are sliced into tensors."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IJg9Ail663Es"
      },
      "source": [
        "We define the `BATCH_SIZE` and the `BUFFER_SIZE`. Care must be taken when choosing the batch size. If it's too large you will quickly exhaust GPU memory and get the dreaded `OOM` error. 64 seems to work fine on `Google Colab` and `Kaggle` but not on `Paperspace Gradient`, or my `ASUS ZenBook`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4u7fUZq_63Es"
      },
      "source": [
        "Care must also be taken when choosing the size of the shuffle buffer. According to the `Tensorflow` documentation - \"*For perfect shuffling, a buffer size greater than or equal to the full size of the dataset is required*.\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "snu8JluxABcV"
      },
      "outputs": [],
      "source": [
        "BATCH_SIZE = 64\n",
        "BUFFER_SIZE = 20000\n",
        "\n",
        "dataset = tf.data.Dataset.from_tensor_slices((inputs, outputs))\n",
        "\n",
        "dataset = dataset.cache()\n",
        "dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE)\n",
        "dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uzT2MXrdAGQm"
      },
      "source": [
        "## Model Building"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1ZXAID4p63Ev"
      },
      "source": [
        "#### Model Composition\n",
        "The model is made op of the following basic components:\n",
        "* Positional Encoders\n",
        "* Self-Attention Layers\n",
        "* Point-wise Feed Forward Networks layers\n",
        "\n",
        "With these componants we create th multi-layer encoder and decoder. Everything is expalined in the sections to follow.\n",
        "\n",
        "Below is the image illustrating the `transformer architecture`. It shows the parts that together constitute the model. There is a mulit-layer encoder. The output of the encoder is passed to the decoder. And the output of the decoder is the model output.\n",
        "\n",
        "Note the embedding and positional encoding prior to the encoder and decoder inputs. While not technically part of the model, these steps are necessary. Also note the linear transformation an softmax function that is performed on the decoder outputs.\n",
        "\n",
        "Continue to read and these will all be explained.\n",
        "\n",
        "<img src='img/transformer-arch.png' width='500'>\n",
        "<div align=\"center\" >Image from http://jalammar.github.io/illustrated-transformer/</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NpP2OQMuAVdj"
      },
      "source": [
        "### Positional Encoding"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cwVBmy0563Ex"
      },
      "source": [
        "Unlike sequential algorithms like `RNN`s and `LSTM`, transformers don't have a mechanism buit in to capture the relative positions of words in a sentence. This is important because the distance between words provides crucial contextual information. This is where positional encoding steps in.\n",
        "\n",
        "Positional encoding is not part of the architecture of the model. It is actually part of the preprocessing. The positional encoding vector is generated to be the same size as the embedding vector for the each word. After calculation, the positional encoding vector is added to the embedding vector. The pattern that is 'injected' into the embedding vector allows the algorithm to learn this spacial information.\n",
        "\n",
        "So how does it work?\n",
        "\n",
        "Notice in the formula shown below that PE is a function of the position of the word in the sequence ($pos$), and the embedding ($i$).  First, an angle is calculated using the formula $pos\\,/\\,10000^{2i\\,/\\,dmodel}$. Then, either take the sine or cosine of the angle. That gives the value for the word at position $pos$ and embedding index $i$. $pos$ holds constant for the word as the embedding index, $i$ increases, giving a unique pattern for that word.\n",
        "\n",
        "Upon going to the next word $pos$ increments. This shifts the pattern to the right slightly. The $PE$ formula applies the sine function for even embedding indexes ($i$) and the cosine function for odd embedding indexes. That is why you see an interleving pattern (chekerboard) in the figure below.\n",
        "\n",
        "So why this formula. The authors of the [paper](https://arxiv.org/abs/1706.03762) explain it this way:\n",
        "\n",
        "> We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset $k$, $PE_{pos+k}$ can be represented as a linear function of $PE_{pos}$.\n",
        "\n",
        "Mathematical explaination from the below mentioned [Reddit](https://www.reddit.com/r/MachineLearning/comments/6jdi87/r_question_about_positional_encodings_used_in/) post.\n",
        "> If k is fixed, than sin(k/a) and cos(k/a) are constant and PE_{pos+k} is some matrix which depends on k times PE_pos.\n",
        "\n",
        "> $\\begin{align}\n",
        "PE_{pos+k,2i} &= sin\\left(\\frac{pos}{a} + \\frac{k}{a}\\right)\\\\\n",
        "              &= cos\\left(\\frac{pos}{a}\\right)sin\\left(\\frac{k}{a}\\right) + sin\\left(\\frac{pos}{a}\\right)cos\\left(\\frac{k}{a}\\right)\\\\\n",
        "              &= (PE_{pos,2i+1})\\, u + (PE_{pos,2i})\\, v \\\\\n",
        "              &= (PE_{pos,2i}, PE_{pos,2i+1})\\, (v, u)\\\\\n",
        "\\end{align}$\n",
        "\n",
        "Here is another great explaination given be Steven Smit in his [blog](https://stevensmit.me/taking-a-look-at-transformer-architecture/):\n",
        "\n",
        "> One of the reasons for using sine and cosine functions is that they are periodic and so whether the model is learning on a sequence of length 5, or of length 500, the encodings will always have the same range ([-1, 1]).\n",
        "\n",
        "> Another, perhaps more important reason, is that they allow the model to “learn to attend by relative positions”. This follows because “for any fixed offset $k$, $PE_{pos+k}$ can be represented as a linear function of $PE_{pos$}$.” If you want a derivation of this property, look no further than this comment on [Reddit](https://www.reddit.com/r/MachineLearning/comments/6jdi87/r_question_about_positional_encodings_used_in/).\n",
        "\n",
        "> This means the model is provided with information to learn the relative proximity between words in the sequence, for any proximity (“fixed k”).\n",
        "\n",
        "Note that the residual connections within the transformer help to reinforced the positional encoding layer to layer.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vJFGGgJrAcOv"
      },
      "source": [
        "Positional encoding formula:\n",
        "\n",
        "$PE_{(pos,2i)} =\\sin(pos/10000^{2i/dmodel})$\n",
        "\n",
        "$PE_{(pos,2i+1)} =\\cos(pos/10000^{2i/dmodel})$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mhnFhTtJAPCz"
      },
      "outputs": [],
      "source": [
        "class PositionalEncoding(layers.Layer):\n",
        "\n",
        "    def __init__(self):\n",
        "        super(PositionalEncoding, self).__init__()\n",
        "\n",
        "    def get_angles(self, pos, i, d_model):\n",
        "        '''\n",
        "        pos     - (seq_length, 1) matrix\n",
        "        i       - (1, d_model) matrix\n",
        "        d_model - the size of the embedding vectors\n",
        "        '''\n",
        "        angles = 1 / np.power(10000., (2*(i//2)) / np.float32(d_model))\n",
        "        return pos * angles\n",
        "\n",
        "    def call(self, inputs):\n",
        "        '''\n",
        "        inputs - the word embeddings - (batch_size, seq_length, d_model)\n",
        "        '''\n",
        "        seq_length = inputs.shape.as_list()[-2]\n",
        "        d_model = inputs.shape.as_list()[-1]\n",
        "\n",
        "        angles = self.get_angles(np.arange(seq_length)[:, np.newaxis],\n",
        "                                 np.arange(d_model)[np.newaxis, :],\n",
        "                                 d_model)\n",
        "\n",
        "        # Interleve the results of sine and cosine funtions along the embedding vectors\n",
        "        angles[:, 0::2] = np.sin(angles[:, 0::2])\n",
        "        angles[:, 1::2] = np.cos(angles[:, 1::2])\n",
        "\n",
        "        # Add dimension for batch_size\n",
        "        pos_encoding = angles[np.newaxis, ...]\n",
        "        return inputs + tf.cast(pos_encoding, tf.float32)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "P5eMYpHe3IiP"
      },
      "outputs": [],
      "source": [
        "if os.path.isdir('../img') is False:\n",
        "    os.mkdir('../img')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "id": "PFy9GFgndCth",
        "outputId": "aa28c5db-e62f-434e-969b-35f6c8755cc6"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEKCAYAAAD+XoUoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd3hcxdWH36PuKvfewZiOAUNCCZgakwIkHwkkH8ShhJAEAoQePmroCTgkkIDpSaiBAKbj0EwHAwZ3bNxw713Fks73x5m72r1aeWVbK1nSeZ/nPrsz996Z2dVq9u78zv0dUVUcx3GclkFOYw/AcRzHaTh80nccx2lB+KTvOI7TgvBJ33EcpwXhk77jOE4Lwid9x3GcFkTWJ30RyRWRz0Tk+VAeKCIfishMEXlcRAqyPQbHcZzGQkTuF5GlIjKplv0iIn8Jc+IXIrJP0r6RIjIjbCPrYzwNcaV/LjA1qXwzMEpVdwRWAac3wBgcx3EaiweBEZvZfwwwOGxnAn8HEJFOwFXAN4D9gatEpOO2Diark76I9AG+C9wbygIcDjwZDnkIOD6bY3Acx2lMVHUcsHIzhxwH/EOND4AOItIT+DYwVlVXquoqYCyb//KoE3nb2kAG/gxcDLQL5c7AalWtCOX5QO90J4rImdi3HuTk7yutOrLjwJ520vINAPRbtwiATWWVALTu2AqAqp79AZg6cz4Aew7pC8CCzyYD0GOv3QCYPOPrRH9Dckvt3IoqAGZgbe092PqUinIAPp2zCoDuJWsA6DW4FwBlRR0AWPrFFACWt26faHufHboB8OVqG2dZySYAdu9i+z/7ej0AfUKb5YN2AEBnzgSg724DAVg/d0GiTa2yO6nbDbC37+spcwDovMeuAKyYaOMo6zfI6hfa/vnF9noGi72OqaWFAAztWZRo+7P5Np69d7K2P5uxEIDdBtv7OOUrK/fv1936Xmif506d7TWvXmPvZWGrfAAqNlUm2rbvfcjJscdN5bavqLUdu3G9nduhuDUAq1auA6BbN2t7yWIbd+9enQFYsGBZou1+fe19njtvCQCD+tv4Zs1ZDJD4/MyYbZ+bnQfZ327aV/a+7rKjvd7oc7Pr4D6JtqfMmJ/yHkSfnd1DedKX86y8U7+U8h5DrDxxevoywJ6h7otQt+fOoTwtfXmvUP68lnK6uqGhPGEby6l1/UN5br2UAYbuEuqmzt2qMoCWrFiuql3ZBnLa91EqSjMepyUrJgPJB45W1dFb2F1v4OukcjQv1la/TUi2bBhE5HvAd1T11yIyHLgQ+DnwQVjaQUT6Ai+p6u6bayunbXct2uNExvzjCgAuenA8AH95/ToAFn5lE8G+P7ZmSi6/B4D9jr0EgAWv3wbA5e1tQrxw6UQAdhvxu0Qf4zrZBLthiX2hjMi1tja8bH3mLJsFQNvT/wPA+RNeBuCql64C4Ktd7QfLnb32BODevY9KtL3xmd8CcPTTqwGYM2UpAFPDCl2H370FwI2TX7HxPmx9lB7/PQD+PPVfAHzwi8sSbZZvsC+Obz1o78GFQ88A4OTZnwHwr4F7AzDzb48AMPIK23/Bdy8H4NU8+7E1bJp9KSz7v10Sbbe/5F0A1r9mbbc7yl7j5JftfdznR1Y/+nZ7Xb+9+nHr++dHA/D0C9MAGLS7TborFq9LtJ2baz8uW7e3L5ul8+yLbpe9egDw6TszADj2+3sB8OQjbwBwzq+PAeC2W/8NwA1XnQzAJeFvDXDXrb8C4Mxz/wLA43fZ3/eEM24C4MWH7LUfc/K1ALz9hD0efILVf/TMDUD15+bzF25JtL3nMRekvAfRZ+fLV0cBMPhIey9mvWZ9DzrCynPf+CsA/Q87B4B5b1q53/BzEm0veOsOAHofejYAi8ZZuechVl4cyj1CeenbVu72LSsve+dOALoe/JtEm/G6Fe9aufNB6csrQ7lThnJy3er3/gZAhwN/XS9lgDXvW13xAb/eqjLApgkPfKKqw9gGclp30bwhx2Y8rq59icgA4Pl0c13QO29S1XdC+TXgEmA4UKSq14X6K4ASVf1T3V9JTbJ5pX8QcKyIfAcoAtoDt2M/XfLC1X4fYMFm2nAcx2l4RJCc3IbqbQHQN6kczYsLsIk/uf7Nbe0sa2v6qnqZqvZR1QHAScDrqvq/wBvACeGwkcCz2RqD4zjO1iHk5BVk3OqJMcDPQhTPN4E1qroIeAU4WkQ6BgH36FC3TWR7TT8dlwCPich1wGfAfY0wBsdxnNqpxyt9EXkUu2LvIiLzsYicfABVvQt4EfgOMBPYCJwa9q0UkT8AH4emrlXVzQnCdaJBJn1VfZPws0RVZ2HhR47jONslAkhu/Uz6qvqTDPsV+E0t++4H7q+XgQQa40p/i9lhQA9G3X85x51qQtumDSb+DX7zVQAOXGBfhBfueSoAV+52IwBtewwA4JSHJwBwWrc2APT52ITRwnadEn0MvczOveaYawDI26ctAO+tt8CjQY+ZINpxgGk2u88ygXH+sy8CMLbwYAB6FVkUilZVR6ys6WDROAcMtgiDL177AIDSaSZwFhVbGM/ycgtq2qN3MQBT82z1bdO8LwFo3ycKgoI5b4RIjWKLWCkP0TxLN5QBUBCiY9ZtsKijVgX2Aa4oscicwn7WVuUm25/TrgNxNN8iaKIrnrIKTSlvDFE5ueFnbkmibOOurLRIqChiB6AqjDOK3onKheEcrQxt5FSfk7acWrQ2Qz/Re58jstn9idcZHU+aRmv0kfGQzZ+/bac3K2Qb38usIkJOw63pNyhNYtJ3HMdpaBpQyG1QfNJ3HMeJ07DROw2KT/qO4zgxBCEnL7+xh5EVfNJ3HMeJ41f6juM4LQuf9BuR/Pmz6X3ZKfTd16KaBu1ithrfvOB5AI4ZsTMAe7ezW/sfuNhsDH7z5BgAbv+jRes8eYf5Hrx36QMA7HLC9Yk+lg09AICV5VcC0G23gwC4eaxFzpz7hNkbDDrdbv8fstEsMGa+ZPvH7GA3Fp/fwTxs8oraJtqeuHSjjXeARQuNWmHeNcu/MI+YNl3NvmB98P3ZuYtFGS3Otw9d6dyvAGjfr1uizeVlZgtR2c6sDiqDm8bSKFon2B2UrAvljjauyvISAAo72PiiyJXcNNE7lXl2TvThLw3ji372liXKIXon+OhEVguVYX8UzQPVPkk54ZiqEOETRedE44mieaqqKtPuz0mUqxJtp4vogeSIoNT6eDkeTVKXaJ74ObJdh6RsPfFIqGaPSL2FbG5vNIlJ33EcpyER/ErfcRyn5SA5iftPmhs+6TuO48QRv9J3HMdpMQgevdOoLF9Txr1jvmTSfBNsWWZ2Bu0eGAfAP6aZrcHtz5s/+oWHmB/6nweb5cBNq0wwnfOtiwF4cepdAPzp5EQqSq5/3cTSfYIQu+pgS1zy7ljz3v/w67UAnDLc/OcHdzPhd+zZjwIwb/pyAPoebIk3Wpf0SrT91qwVAIzcx8TfyEZi2SRL5lG8l9kwlAQ1tk97+1nZtdA+dGtmmkjcLiQtAVgZRNOygmprBoBFay2fQ5ugUpZuNN/9oiDkbgo2DEUd7DytMo9/aV1MnEi4jT7864NNRKJcZuVqITcqW98Vm4JIm6SYllbYeHITQq295oK83DCe9MJtQWTTEO1PIyzGxca4dUNttgxx4jYNyWypnNlY+mcmu4gWJstuFT7pO47jtBQ8Tt9xHKcl4ZO+4zhOi0FEyMn36B3HcZyWgS/vOI7jtCx80m9EevfvxA1XnMyonb4PVN9uf3GwWbjzjmcAuGnjXgD87+EDAHjz+F8DMPhos1Y4dfSHAByoFjFy4MYJiT5OftEifM770a4A7HvETgAcdKclrVlYapEpZ+1skTZtevwPAF+X/AOAlbOnAND/+L0B6DBhp0Tbr09aDMCl+3VMeV0rZ1jms84j2qbUd8qxRChdW5vdwZo5NrauBw1LHLM2RNasKk2NNJm/0mwWdgvRLmUlFi2TsGFYH2wYOlm0jlZZZFFVYRvilERJU3JTk6ZEP3vXh/ckKm+M2TBUJWwaqqN3osQq8WicTOV4JE5+Ts2EKLmJxCxREpXU15Mu4ieZ+PENFXlTo98M+52GIaeZvvFZS+QjIkUi8pGIfC4ik0XkmlD/oIjMFpEJYRuarTE4juNsDSKC5GTe6tjWCBGZLiIzReTSNPtHJc2HX4rI6qR9lUn7xtTHa8vmlX4ZcLiqrheRfOAdEXkp7LtIVZ/MYt+O4zjbRG7ckW8rEJFc4E7gKGA+8LGIjFHVKdExqnp+0vHnAHsnNVGiqvV6YZy1K3011odiftg0W/05juPUG0J9XenvD8xU1VmqWg48Bhy3meN/AjxaD6+gVrKap1lEckVkArAUGKuqH4Zd14vIF+FnTWE2x+A4jrOlmMtmvUz6vYGvk8rzQ13NPkX6AwOB15Oqi0RkvIh8ICLHb+XLSSGrQq6qVgJDRaQD8LSI7A5cBiwGCoDRwCXAtfFzReRM4EyANl16cG6rHzK88CkAFpSYgHjxSlshGnS1+eT/9qK/A3DZ4w8BcE63QwD481PfAODYn/0BgOsGm6/9ZxffkOhvybLBAAx+1KwaNPydIqGwVVCPO81518bQ90Cg2sd+wzI7vv3wUwDosao80fbiObZElzPXhOPcglYAfL3GBNs9gs9+JBzlrZhjbfUxq4TVc8y2Ib/ngESbkff+6iDkFoRzF6wxofbA/FQhN7JhqFpt5ZzizuH1zbD6wmo7hyhqoSyIrjmhvC4ItXH//Bp++kGEjUTb/MLqj1kk7kZCbVWFvU8FuemF28gLPz8ndX86ka2GDUMGJXZrhNqoj8Q4auzf/PnN1W+/+SF1zSHQRUTGJ5VHq+rorez0JODJMG9G9FfVBSIyCHhdRCaq6ldb2T6Q5Sv9CFVdDbwBjFDVRWHppwx4APv5k+6c0ao6TFWHFbXrmO4Qx3Gc7FD35Z3l0TwVtviEvwDom1TuE+rScRKxpR1VXRAeZwFvkrrev1VkM3qna7jCR0RaYULGNBHpGeoEOB6YlK0xOI7jbC31tLzzMTBYRAaKSAE2sdeIwhGRnYGOwPtJdR2j5W8R6QIcBEyJn7ulZHN5pyfwUFCvc4AnVPV5EXldRLpiy2YTgLOyOAbHcZwtRgRy87Z9KU5VK0TkbOAVIBe4X1Uni8i1wHhVjb4ATgIeU9XkYJddgLtFpAqbQ29KjvrZWrI26avqF6T5KaKqh2erT8dxnPqivvQXVX0ReDFWd2WsfHWa894D9qiXQSTRJO7IXbl4KY/88Q7u+tr0EvnI7sC98ugrALjqYRMSzw1i32mvLAXgiE4mmB6y9A07L9xZevDNJvzefOJfEn3InubV/0nBEAB6P/R7ADoO2B2Avea8A8DCx2zJ7aXj7bheRfYWRoLkhj7m0X/grrMSbT/4gSVVL50UvOyL7a7e6C7fffpbUvKZkQ/9bFvx6jDA7pqd9858G2OXPok2S4JIumidicGR0LxijfnpF4dxRd79RT2trcoZNs5IyI3Qwuq7gqsTodtFR0Ko3RQJtVZeF/z0cxP++VHScxtLdWL06n+eyD8/SnxenbQ8vX9+RDzpeVxQrcsxNUXY1BNqJEbfTjVXF4Ozj4g02ztym8Sk7ziO09DU9Y7bpoZP+o7jOGnwSd9xHKelILWn02zq+KTvOI4TQ5AUd9jmhE/6juM4caT5Wis3iUm/Q/euHHXuWQz+5b8BGP5ti6g5op3Z9vzl5/cAcPmLZuJ53TXmgT/6gV8B8NYZNwOw5ym3ALD0ILNlWFx6W6KPHnsdBsBlYyYDcOEDHwAw+KwfAzCUfgBMecKsFB7rPteO69IagLwii375eKFF6Bw2uEui7TuDRcOSjxYB0Lb7CABWBtuC73dvD8CKfIuaKZk5DYDiAd0BWPbqbAAqi3sm2ozsHxass2idVsHGoGSdRedE/vkVpSFiqLP1UbXJ9ue260AylXlFiedR9M6GhM2C+fqvKw/++XHbhdB3dbSOlTeVRdE81VdMVbX46UfRPFW1+OnnJKJ7Qh9p/h8T0TmJiKDU/fFyjWidGk72NYmfkymSpqleK2ZjaaOprZY01yipJjHpO47jNCRmuNbYo8gOPuk7juPE8eUdx3GcloSkLEs2J3zSdxzHiSF+pd+4DJC13Jf/Mn2XmED3xCjztH/gc/PTv3KHYwG4qvI9AK4uN0/5N3c6EYDnp5tg+48zzMX57KcmAjCyW3Uy8MJjzFbh8X9Z/oJx89cCcN4Iq99p0FEAjHnxLgBmTzJRdodvDwKg7YoB1tdkS2J+8fCBibYjK4TFn5ijaudDugFQHiwJBnQwYbRHkQmoq7404bfTLv0BWBYE0Y15qQnUAeavstfaPgifG9cHIbeLWVBsKjEht3U3s6euqrDx0TbVhqEkiLBQLeTGhdtEIvTIhqE0ePMnbBisjbwgSJdusP25SaFvkQ1Dbk7MT7+WROhROfLTj0gnNObH1N34MbWJk8lWDslszb98Y2h/dZmbmuf0lV385izHcZwWgkj1hUdzwyd9x3GcNPik7ziO00IQxCd9x3GcloJITXvv5oJP+o7jODFEIM+v9BuP+bOXc8kp9/PRYou6Of6mNwH41j8WA/Cfq83W4P4f3gDAwTfcB8Cv/mTH/aLIbAR6vXY7AO8/Zy/7gctHJPo4+HCLwvn7taOAaouE7/U3ewLp+zMAFpbeAcCqWZ8D0P88SwTWbZyd/+7nFtXTZf/CGq9j6cyVNo6TUy0Q2pcuB6BHV7N0WDndXlevEdb2qpCcZHlJzYQhc1dsBODAfLsqKd1g0TCtgz1E5UqL7snvYH1q1UIAqorapYxhw6ak6J2QbGZ9ZMOQH4veyU+1YYiidaIkKgWF9v5WBsuFVgW5ibbrGq1TkJs+iUoimic31bbBjpHYOVuWJKUukTfbeu2Xbh6JVzXTuaZJIfiavuM4TstBmu+aftYWrUSkSEQ+EpHPRWSyiFwT6geKyIciMlNEHg8Z4h3HcbYb7Eo/J+NWp7ZERojI9DDnXZpm/89FZJmITAjbGUn7RorIjLCNrI/Xlk2logw4XFX3AoYCI0Tkm8DNwChV3RFYBZyexTE4juNsFbk5knHLhIjkAncCxwC7Aj8RkV3THPq4qg4N273h3E7AVcA3gP2Bq0Sk47a+rqxN+mqsD8X8sClwOPBkqH8IOD5bY3Acx9kackQoyMvJuNWB/YGZqjpLVcuBx4Dj6jiMbwNjVXWlqq4CxgIjMpyTkayu6YdvuU+AHbFvu6+A1apaEQ6ZD/Su5dwzgTMBehQVcuphOzD3MBM2P/3IrBLaHXwuALOf/iMAX17xIgBjRu5p++8xQffE0/YG4PlzHwFgbe9vAtD6F39L9Jfz1j8AKCruCkDfVib+Vrxgx4zf/ywA2oY/dMkqE1vzDrD6IctNwP3ojal23sR5ibYL23UCYOYMsyU4KHjtLw/qpMyfAkDHgSa2rpq1GoD8fjsBsD5YJCwNIi1AQbjKmLnShNxOQSwt22Dfs226mVBbudj89nM7dgtnWl8ahNzIciHZhiE3sl0oq0gprwtCbl6BidRlkZ9+no2lKrSR0zq1nPzPERdqNe6fH7zwE974MT/9xBg346dfa5nNC7uba682q4ZMF3v14cneXH3dt3figQC10EVExieVR6vq6KRyb+DrpPJ87Mo9zv+IyCHAl8D5qvp1LeemnS+3hKxO+qpaCQwVkQ7A08DOW3DuaGA0wK7F7TU7I3Qcx6nJFtgwLFfVYdvY3XPAo6paJiK/xFZADt/GNmulQe4+UNXVwBvAAUAHEYm+bPoACxpiDI7jOFtCfazpY/Nb36RyjTlPVVeoalko3gvsW9dzt4ZsRu90DVf4iEgr4ChgKjb5nxAOGwk8m60xOI7jbA3RzVmZtjrwMTA4RC0WACcBY1L7kp5JxWOxeRLgFeBoEekYBNyjQ902kc3lnZ7AQ2FdPwd4QlWfF5EpwGMich3wGXBfFsfgOI6zxQhSLzYMqlohImdjk3UucL+qThaRa4HxqjoG+K2IHAtUACuBn4dzV4rIH7AvDoBrVXXlto4pa5O+qn4B7J2mfhamaDuO42yX1Ke1sqq+CLwYq7sy6fllwGW1nHs/cH+9DCTQJO7Ireg3iNW3P8bLu5rovWm3gwE44ByzVfjx5c8A8PZ5Vv/lGZY8pd8Bdo9D3xtNE/nTnUMB6HDQ7gD836szE32ccNu/ABh4gN078a2StwGYcKf9mrp707cBOKbYIleiRCJfVljEzfFD7argv/+y1arl7y5LtN22+14ALAnRMN/pb6G27xfY21/+5WcAdBxskUNffmyRQJUdbTkvSrYyb01pdZvhKmTtaqtr29HsIqKELa0GWh8VZWbDUB29Y1QVxqJ3kmwYcvIscmldZMMQi+aJ0siVh3Jkw7CpLNWWIbJhKExOorIp1YahKh69k7BZiKJ3rI38mE1DIrqnMsmGIXRTbe2QWs5EffyPN0+Lrq2jKQcduQ2D4zhOS8KTqDiO47Qc3E/fcRynheGTvuM4Tgshx5OoNC5fzV7EcafewKrXzS//wuEmdL/xg7YAtHrkAwBKbr0TgIf6mmB737RDADjvlbkA7NPBxM5Vx5ng++8nq++e7vCR+cz/+pbdABg65EgA/nb2owB8/OF8AC45fAAAbUvs8alJJrqO3Mfujo7sGRa8NzvRdue9zGojslPYJXjdz2tlb//yCV8C0Glna3Nx6acAlLbpmvI+zAmWCwDt80ws3bjW7ulo060NAOVByG3dzYRcrTJLh5ziLiltbawwcTgSadeUViT2RXVrS802IreglY0/lPMKIv98ez25QTEtrQjHh3+WyoQNQ7WffiSqFubVYsMQibAZLBXSXYVlsmGIigkxOG7LEGsvnRAZt0RoDLGyLhegWzqs+HvV4vE1fcdxnJaDIHX13mly+KTvOI6Thub668cnfcdxnBhCeifX5oBP+o7jOHGkpp13c8EnfcdxnBgC5NcxHWJTo0lM+vlt2tF33+Ec/l57AB6/8mgA7t33ZAAOvuYeAL575asAnBoiQfYbb/UnPGIv89orj7HjjrMInYG335XoY2GIXrlk12IAZKdfAzDnNEuusnSqeR7t9Fvru/u4HQF48UPLcfD73VM/IAsnLk087/2D1AxnXTatAKBnZ4uKWTbJIoO6H2HRRsuD/cGyjSGJSbjgmLVsQ6KNbxRYfxvXheid7ha9U7nSbBcKu1m0TlWFjaOqVXHKGDaGyBvJtciaNWXV0Tu5hTauNRstGicnP5ZEJT+K3gmRN4X2/ka2CwmLhYpUy4V0dfGkKhFx24X83FTbhvw0V2Fx4S2+JBs/pSGWbGv0WYdjnMbHl3ccx3FaEiK+vOM4jtNSEDx6x3Ecp0XhyzuO4zgtBJFqHam50SQm/d16FPHhJTvT5vt/AuDt+68G4OsbTfh8+cQ+ALR54AEATr/0CAD+ddZDAKwZcCAAlaf9FYCOr5pdQ+vOvRJ97NDGxMqN/7oJgHeHnw9Acb794SN7hdzDfgfAPuvnAPD6C2aZsOmT6QAUFZt1wvQvyxNtf3uPHgAsjATNORMA6DKkMwDLp5uwmz/QBObIruHrNSbStgofvulL1yfa/F4QT0vXmu1C254m1G5aaGJvbuddwpFTAKhqbWJy5J+/PrJQSGPDENWt3hjZLlgOgZKEkGvjqQiCc+u2BSnlVsGmIfLOb5Vf04YhIeRWxvzxw/682D9c/Korfny6uho2CxmE3Zrn1yTTMm/cpmFrqI82nG3Dl3ccx3FaGM11eSebidH7isgbIjJFRCaLyLmh/moRWSAiE8L2nWyNwXEcZ2sQhBzJvNWpLZERIjJdRGaKyKVp9v8uzJNfiMhrItI/aV9l0lw5Jn7u1pDNK/0K4AJV/VRE2gGfiMjYsG+Uqv4pi307juNsPfXksikiucCdwFHAfOBjERmjqlOSDvsMGKaqG0XkV8AtwIlhX4mqDt3mgSSRtSt9VV2kqp+G5+uAqUDvbPXnOI5TX9iafuatDuwPzFTVWapaDjwGHJd8gKq+oaqRb/oHQJ96fCk1aJA1fREZAOwNfAgcBJwtIj8DxmO/BlalOedM4EyADpLHqJ2+zzXPvQDAL883IXbxE+cC8ObwEwDY95RbAKj4pSVQ//RqS4Deez9bQTr5n5aA/MJR5pG/x1m3Jfo7su1HAHz4J0uEfmuZnXNBF7vT9a9F5t3/zjLzof/pMEta/p+7HgZg4Vjz1S/uO8LKb1cLoyMHmGD7ehBfN3xm/v9dd7e/7cT37I7cis4DgOpE6F+tss9BlAR9XbjbFqBtV/Pk37Qx+Ofvan1E4mlelx4kU1Fg44+88teFJOa5BZZjYE3ZpsSxCf/8svR34EZ++VEi9ChRelW4I7d1EHLj3vlQfUdtq9gx8UTo0XpqbYnQ0wVWxOtqCLcZXObjx9dFUG2q8R3ZECmbk+65BTYMXURkfFJ5tKqOTir3Br5OKs8HvrGZ9k4HXkoqF4X2K4CbVPWZugxqc2R90heRtsBTwHmqulZE/g78AdDweCtwWvy88MaNBuiTW6TZHqfjOE4CSX9hkYblqjqsXroUORkYBhyaVN1fVReIyCDgdRGZqKpfbUs/Wb1QEZF8bMJ/WFX/A6CqS1S1UlWrgHuwnz+O4zjbDVHIZj0IuQuAvknlPqEutT+RI4HLgWNVtSyqV9UF4XEW8Ca2YrJNZDN6R4D7gKmqeltSfc+kw34ATMrWGBzHcbYOy5yVaasDHwODRWSgiBQAJwEpUTgisjdwNzbhL02q7ygiheF5F2xpPFkA3iqyubxzEHAKMFFEJoS63wM/EZGh2PLOHOCXWRyD4zjOFlNfN2epaoWInA28AuQC96vqZBG5FhivqmOAPwJtgX8HHWmeqh4L7ALcLSJV2AX6TbGon60ia5O+qr5DeifZF7PVp+M4Tn1gNgz1o0yr6ovE5j1VvTLp+ZG1nPcesEe9DCKJJnFHbr4IXQtzGf7ydQDcVrwXAJfr4QBsnGarR2/+1vSUA29+B4BR+5vNwgEhmue3F/0dgBfnrAbgrz+tXh7b7RDzz3/4YLNZmP7+ZAD2Os0kh06zrc+735kNwAMn7gnApg0WPTP3jVkA9A/A2AEAACAASURBVDrBolKjCByAXbpYhMzsNvkALBk/DYC+R+wHwIISG+9qaZPyumcsMduFHiF6Zv3q0sS+tr0sGqc89N+2t9k/VFWE5cLibiltrQ8WCZENw8qS4JUf2TBsTIreCX76qzeGSKDQfxStE5VL1gVv/BCJU1lh0T4FecGGIY2ffg0bhoQ/fqwc98+P/QOmuwqL1+XEIoDibM2/9JZe/NXHtJHR+qEe+nBq0pyikZJpEpO+4zhOQ5MpxLep4pO+4zhODMGv9B3HcVoUzTRxlk/6juM4NZAWfqUvIj8Ebga6EX75AKqq7bM4tgSd9tyFk955m/PamN/8fxf8BYD9j7sEgLHfNPH0o6PMOmFS+a4AHPTM3QAcXG4WCb9ctxKAVkEU3HXea4k+5u5oCc9LgpXAylmfA9Drpt8AsOMz6wD45COzTCjYYxkAecGeYcoUE1QP2stuQ6hIukwoWvgFAD0HdwJgyefmzb/Dr0xIXl5uAujC9SamFoRzpy5aC8DQIhNGN6yN7DmgfR976ytmmn9+frcdANCqeQBUtkr1z19bbq8rL0p6HiwWItF2xfpq//+EDUPkn1+QasNQ1LogpZzwz69I9c/XmOUCVNtE1BRuNy/UxmOi090in+mftKaf/ub98+typRe3ash0TnO9emxuCHWOw29y1PVK/xbg+6o6NZuDcRzH2V5orl/QdZ30l/iE7zhOS6KZzvl1nvTHi8jjwDNAsi/Ef7IyKsdxnEbE0yVCe2AjcHRSnQI+6TuO0yxppnN+3SZ9VT012wNxHMfZnmiquRIyUdfonT7AXzETNYC3gXNVdX62BpbMxLkr2fEXjzHuvAMB2HjRTwHos9/pAOx/0w0AnFu8DwDFx/8PABd9al/VP7r9IgAGH3YxAN/NsWiajy/6c6KPu86ytJRHd7LIlftC/bSiHQE4/VCLpDn7OUuysuT5FdZXH7PGmDP+eQC+t1t3AN4vrH5rS8dblFCPfS3K6MNHrH/tY1FGJZVm2TBtuUXiFOfbx23JUit36mxjKluzLNFmu92tn00Tzaohr3u/sMcStFS1saQqUfROZMOQk2dWEKuCDUNeiNRZU1JtwxDZLJSFuvzCVBuGdh2D7UIsaUoUmRNZLFRuxoYhnjQlPyc1kiZRrowfn5pEJdliIX4HZc1oHbaZbEwEdUnWsj3SRIddJ6Se0iVuj9T1M/wAZgfaK2zPhTrHcZxmiUjmrSlS10m/q6o+oKoVYXsQ6JrFcTmO4zQagk2OmbamSF3HvUJEThaR3LCdDKzI5sAcx3EaExHJuDVF6jrpnwb8GFgMLAJOAFzcdRyneSKmAWXamiJ1jd6ZCxyb5bHUSlXFJjauWMAzZ/0BgC8POQKA8etGAHD4HSZeXhlsDnb63XEAXHf9wwDkvG3J6P927zcB2H/EWQBcc8w1iT7e6G+2C9ecYtYInRaaf/5tb1kO4lu/NwSA01eZhcKM5+xetd7f/SEA6580QXK/4HO/rG1+ou2Fb38GQI8Ddgdg9j2fALC2qEvK65y00MTirgX2Z1m3sgSotlwoCzYSAO36dQ/vjVlMSKfkLJTVtguRX/7y4JcfWSwsX2+3W+S1svFG3vkABUGEzuSfX1FubbYK441sGCJhN+6dn1wXt1HIz42XN2/LkC6GOqqqtlWICbu1HF9d3rwQ3FBkwz+/ucacZwsB6imHynbHZid9EblYVW8Rkb9icfkpqOpvszYyx3GcRqSpLt9kItPyTmS9MB74JM1WKyLSV0TeEJEpIjJZRM4N9Z1EZKyIzAiPHbfxNTiO49Qrdkdu/SzviMgIEZkuIjNF5NI0+wtF5PGw/0MRGZC077JQP11Evl0fr22zV/qq+lx4ulFV/x0b6I8ytF0BXKCqn4pIO+ATERkL/Bx4TVVvCm/ApcAlWzV6x3GcLFEf1/kikgvcCRwFzAc+FpExsQTnpwOrVHVHETkJczQ+UUR2BU4CdsNC5f8rIjupavr8n3WkrkLuZXWsS6Cqi1T10/B8HfaroTdwHPBQOOwh4Pg6jsFxHKeBEHIk81YH9gdmquosVS0HHsPmwGSS58QngSPE1paOAx5T1TJVnQ3MDO1tE5nW9I8BvgP0FpG/JO1qj13J14nwc2Vv4EOgu6ouCrsWA91rOedM4EyAnKLiunblOI6z7dT95qsuIjI+qTxaVUcnlXsDXyeV5wPfiLWROEZVK0RkDdA51H8QO7d3nUa1GTJF7yzE1vOPJXUNfx1wfl06EJG2wFPAeaq6NlkcUVUVkRoCcdg3GhgNsMfQffQ//zif3Y/5HQDjjhwIwKcHDgfg41yLijl8nK1AHbnS3uPLVy0BqpOS7D/nBQBm72Y/LtZsujLR37Jp9t72v8FWmnZ51iJp3nxzFgBtBs4FqpOmTJpikTSHD+sDQGnoo838TwHou0t1ZM78D2w8A35xBgDLy+1m5jmry1PG9/nXqwE4OSRNWb/abBg6DDTZY9O0tYk2C3rvYu9TlTlhVLaze+VqS5qyPETnxJOmRNE8q5OSqOQXRbYL9r3eun0hkDlpSqIcs2UoyqtOolLThiHVdqEqslnIkDQl3T9kLACoUZKmxJuosb+ZioPNDVFFquq0irJcVYdlezz1SaY1/c+Bz0XkYVWt85V9hIjkYxP+w0k2zEtEpKeqLhKRnsDSLR614zhOlhGtqo9mFgB9k8p9Ql26Y+aLSB5QjN38Wpdzt5jNrumLyBPh6Wci8kXSNlFEvshwrmC+ZVNV9bakXWOAkeH5SODZrRy74zhOllDQqsxbZj4GBovIQBEpwITZMbFjkufEE4DXVVVD/UkhumcgMBj4aFtfWablnXPD4/e2ou2DgFOAiSIyIdT9HrgJeEJETgfmYnf6Oo7jbF9o2pXnLWxCK0TkbOAVIBe4X1Uni8i1wHhVHYNdHP9TRGYCK7EvBsJxTwBTMA31N9sauQOZl3ciwXU5UKKqVSKyE7Az8FKGc9+h9qinI7Z0oI7jOA2Gal2v5OvQlL4IvBiruzLpeSmQNgReVa8Hrq+XgQTqmjlrHPCtcCPVq9hPlhOB/63PwdRG2fTpzDl0OPuccgsAPX5p4veNXUzA7f2L7wBwzJP2HXXhKNOYh51lq0o/6jkDgDd+MQqAm88ZAMAFPdol+nggCJpvbeplbRzdA4ATnjApYt4j1nbnHc364cuP7BaGkUNNTH+9ldkurHvbvgv7HLhDou3XR5tIfGD/oUC1f/7nS0yY7RSEzw8Xmzd+1x4mFpcGy4f2w8xioeLzkkSb+b0GhGfv2r5WZkER2S6sKjEJJregCKgWcvODEL1yQxCRg+VCeVm1ZBPZLsRtGCIht11RsF3YlGq7EImwrWI2DMmWCrX55ydE1jr658eFXqhpu1BTRI2XNy+qNlUXRah/24WWqD/X05r+dkddP9eiqhuBHwJ/U9UfYTcMOI7jNEMUqioyb02QOk/6InIAdmX/QqjL3czxjuM4TRelvoTc7Y66Lu+ch92B+3QQFwYBb2RvWI7jOI2JQlXTnNQzUVdr5beAt0SkrYi0VdVZgDtsOo7TbGmua/p1TYy+B/APoJMVZRnwM1WdnM3BRawtreCVmat46zC7Y3WH854E4K2QKP2ci48GYN9jLfH5jrNWAfDcr0zwbfvT2wG4t+8xAHz+8psAHHrT/yT66P2h+edf86y9pLGn7gTApg1rAJj2H/NH2vGCXwNQ/i8TY/doZ2Lm0i4mBM95xW5cHvLz6vQDX40aB8CiytYpr+vjOTbOvYMwunqZ3YHbcVAHAEpDIvTiHS1pe1XFjMS52qlPSlurSoMgmm9C7pINqXfgLlub6p+/Ivjp50dCbkn1+mRUtyGc0yqMr6I8lGP++fE7dCP//Eh0LcpN46cf6qrSiL1Q8w7ceJLqqJicGL2GUMuWURexckvvwK0L2fDPd+qBljzpA3cDv1PVNwBEZDhwD3BglsblOI7TeKhC3WwYmhx1nfTbRBM+gKq+KSJtsjQmx3GcRqdFL+8As0TkCuCfoXwyMCs7Q3Icx2ls6u/mrO2NLUmM3hX4D2ag1iXUOY7jNE9aYsimiBQBZwE7AhOxTFibGmJgjuM4jUY92jBsb2Ra3nkI2AS8DRwD7ILF7DcovYf04Yb7buSKQy8CYOX+lnhmxv/9GYAhfz4HgC47HQLAYXMtWmb5pT8H4J4f3QBA32CVsH7JHADKf3BHoo+fdp8HwJ13PANASYf/AtCup9kpfDD1LQB+ceggACZFESvBjqH/If0AmPVfa3u32w5JtL2y/GYAJi616JzifPuB9cFci945ttj86tcvN5fpjkPMdqF8nNk05Pczu26tmpZos7K92UREtgurQ/ROXrCTWLohROcE24Wl66Ky2TKsC9E9ha3sI1BWWv1d3qGryTUV5eltF2rzz48sFKJonSiyJi9N9E5hzPw+itapYcuQ8L5PjWFJF2kTj4LZUv/8+P768L5vqv75TXTY9YbQctf0d1XVPQBE5D7qwdbTcRxn+0ehsmVG7yQu/4JFaJaH4ziOsx0Q2TA0QzJN+nuJSJSjT4BWoSxYtsP2WR2d4zhOI9Eil3dU1U3VHMdpgbRcIXe7YPr6PA57uwtXDjB7gu43ng3AT869C4DTXnsbgIen3QrAN083wfaaY64B4J9r3gHgrTP3A+AvC+3x4hemJ/q49XtDALjx0i8B+OzvUwEY+N2rAVj20j12zpDOAOQG8XX+mFcA6Pdta/PpJ01sPaB4YKLtYJ/PB3MsmXqvIhvfikXmn99psHnhlwT//E5Hm+1C5X/Nwz+nR3VbEWsq7U8XCbmLgq1CXpGJsIvWlAKQ36YYgKVrrVwY+i7dYCt3keVC6cpqr/7CULepzITatuGcynI7JhJ2K2uxYSjMi/z07Z+mKK9mZHDCL7+yFhuG3PTCbW3CLtS0K8jkn98Yq5V1Sr6+xW36smtW8EnfcRynhdCMbRiylhxIRO4XkaUiMimp7moRWSAiE8L2nWz17ziOs/UoWrEp47atiEgnERkrIjPCY8c0xwwVkfdFZLKIfCEiJybte1BEZifNqUMz9ZnNjHAPAiPS1I9S1aFhezHNfsdxnMZFsSv9TNu2cynwmqoOBl4L5TgbMVfj3bA59c8i0iFp/0VJc+qETB1mbdJX1XFYZnfHcZwmhaJoZWXGrR44DrsJlvB4fI2xqH6pqjPC84XAUswWZ6tojNzPZ4efKPen+ykTISJnish4ERm/acPqhhyf4zgtHcUyZ2XaoEs0T4XtzC3sqbuqLgrPFwPdN3ewiOwPFABfJVVfH+bUUSJSmKnDhhZy/w78AXtL/wDcSi3Gbao6GhgNkNOmq47/96MMfs+sEL75rNkaXJ9jlgMDWlt0yc5PWbTOsyMuAyBXrLxkktky9H7nbgC+94K9X8/9+51Ef3/Nfw2A1p17AfDuexYRdPodFtUz50b7fiz8zGwXhnyrLwAzX7LEJgMvsF9lS8oeAODzJRsSbbcN0Svvz1gOwPltLeJmzRIrd93dbBfKPjZbhqIdLfmLVs0HoKKj9SU51RG0K4PtQn5IirIoslmIyqstWqegtUXzrAwJUQoi24USW49s38newxVl1UlU2kbROWUWrdO2MDVpStuYLUOb/ChaJ1gshNcbHZ+fFLKSSJoSs12Il3NjESkx14YaZdhy24U48eiedMdnSpriNzA2F+os5C5X1WGbO0BE/gv0SLPr8pQeVVVEdDPt9MScjkeqJkKLLsO+LAqw+fIS4NrNjadBJ31VXRI9F5F7gOcbsn/HcZw6oVovQq01pUfWtk9ElohIT1VdFCb1pbUc1x54AbhcVT9Iajv6lVAmIg8AF2YaT4Mu74QXFfEDYFJtxzqO4zQeilZVZtzqgTHAyPB8JPBs/AARKQCeBv6hqk/G9vUMj4LpARnn1Kxd6YvIo8BwbM1rPnAVMDyEFCkwB/hltvp3HMfZaqLonexzE/CEiJwOzAV+DCAiw4CzVPWMUHcI0FlEfh7O+3mI1HlYRLpiK48TMCv8zZK1SV9Vf5Km+r5s9ec4jlN/aCTUZrcX1RXAEWnqxwNnhOf/Av5Vy/mHb2mfTeKO3F59enDubZcw7ORRAJz22iMA/HvqhwAcNMfE12u+ex0A/5y4LwBv/tKsEe5bbI+/fm4mAH/8romzD974l0Qf428x+4Qdjr4SgK9fs/f47D1MTH+5g/nQz3vUfl3tcNwBVv/ywwDs12VnAMqrTId5PYi2AL2C8PnyvDUAdN2tCwAblpmHf5fDdwSgYpwtz+X22yWc+SoAa7G+c4NXPsD8tam2C/NWbQSgoJ1ZOixaYyJsZLtQsj7yz7fyumC7UBTKkeUCQIfWJjTXl+1CYVK5vmwX0smlW2q70BBrm03FdsH15xhKfYVkbnc0iUnfcRynYWm+Ngw+6TuO48Spx+id7Q2f9B3HcWrgV/qO4zgth4aL3mlwfNJ3HMeJoWgiMKG50SQm/U5rFvGjF//AnzseBMB+HS2apd+ffwPAnSfeCED7ECUS2S50GHc/AL94z6Jk7rzjGQBu2/AUAO167pDoY+zrZvFwwV27AzDpFotIKXzXIoX2GGHHTvuPJVcZcOlVAHxd8iAA789fB0Bxvo3h7SmJm4+5tLNF3axasBCA7vv0A6B0nEX4FO18KJBku9B5AFCdIGXpBrNIiCwWAOaF6JyCkCRl/qpQDrYLK0ISlaI2IWnKRou06dA17F9k4+0QLCwiywXYdtuFwuCRELdcSD5nW20X4pYLkNl2Ib4/k+1CuoCW5mK70ESH3XD4lb7jOE4LQhXdVJ75uCaIT/qO4zg1aJibsxoDn/Qdx3HS4cs7juM4LQTV+jJU2+5oEpP+oiXruOmWt5hZYn74eWuPBuCc7sMBeGyaedgvvN+s+R98b1cAjr3THEjfPG0QADfO/xKAt/7vIwD2vvyuRB/LXroHgCv6m9zXqU97AKb+7XEAdjnrBAAefcK8/Hcr6pcyxjETzUJhWBsTX5+ZU534pee+ZqUd2S50++FuAFS8atYPOQP2DEe+YGMpN6E0t9AE4NmrTWTNb9M+0easZebXH9kuzF1u5aJgoVCyztYji8J4Vi5ZD0DrYLtQXmJtFofjK0rXJ9qOxN2KYMOQEHKDUNs6P7Jh2JRarkq1WIhuY4/bMgAU5KUXbmsTduO6YzohMpPtQpxMwm1dLBQytRlna/TTbNguOJnx6B3HcZyWgipa6ZO+4zhOi0BVqdpUkfnAJohP+o7jOHEUv9J3HMdpSfik7ziO00JQVarcT7/x6NG9LZec/C2e7LM3ALf/5s8A/GlfS7n7SIgieWrwKQA8drDZFXzz+EsBmD7xawD6fsOie14Z/ToAd/44ipqBsZcXArD6fovO2euMAwF45ubXANj1YUsEtqzsBgBeCElSeoUkJf+ZuBiAk3eyaJqV82Yk2u493KKJSh8Jtgt7HRP2WPROSXEfoDpJyryQIKWgtUXrfLUyROa075poc9Yyi7Zp1c5sFdauKQtli8bZGJKmdAtRSOUl9h51bhuidUrs/M4huieK1AEoDtE7ke1Cu4IoesfaaBWzYYiic+LROlpVM2FKrbYLGSwRcnNSbRfix6c7J5PtwtbQELYLniRl+6C5Ru9kLXmQiNwvIktFZFJSXScRGSsiM8Jjx2z17ziOs9WE6J1M27ZS1zlRRCpFZELYxiTVDxSRD0Vkpog8HpKob5ZsZox7EBgRq7sUeE1VBwOvhbLjOM52RRS9k2mrB+o6J5ao6tCwHZtUfzMwSlV3BFYBp2fqMGuTvqqOA1bGqo8DHgrPHwKOz1b/juM420JVZVXGrR7Y6jlRbK3xcODJLTm/IXJDJ9NdVReF54uB7rUdKCJnish4ERm/cmNZw4zOcRwHEiGbdVje6RLNU2E7cwt7quucWBTa/0BEoom9M7BaVaOfHPOB3pk6bDQhV1VVRHQz+0cDowH67ryHPn3c1ZT/3VaLvhhj1gh7jDOR9fzgl3/+Vf8C4KvjTaRs07UvAI8/NRaAP7z/DQAmPWBCZP8J/070d/hxOwHw0W0m8o746DE79nKzRhg7byNQ7Zf/7/fmAnBpt9YA/G2mjaHf8MEAbBj3daLt4m+aX37VP6ytTb3Msz/yy5+3xgTSyBt/WmSpUGzC7bTgfV9UXL3ct3CFjadNexOg1wd//cgvf/VSa6NL2P/lBmujUxsrV5ZHNgypoi1A+5iffquE7ULMhiHy089NFW4Lc1OF3mQ//Ygatgtx//yYX34mr3w7J7W8pX75W+OVnw3bhfrAhdttpO535C5X1WGbO0BE/gv0SLPr8tQuNzsn9lfVBSIyCHhdRCYCa+oywDgNPekvEZGeqrpIRHoCSxu4f8dxnIwo9Re9o6pH1rZPROo0J6rqgvA4S0TeBPYGngI6iEheuNrvAyzINJ6GXt4ZA4wMz0cCzzZw/47jOJlRpaq8IuNWD2ScE0Wko4gUhuddgIOAKaqqwBvACZs7P042QzYfBd4HhojIfBE5HbgJOEpEZgBHhrLjOM72hUJVVVXGrR5IOyeKyDARuTccswswXkQ+xyb5m1R1Sth3CfA7EZmJrfHfl6nDrC3vqOpPatl1RLb6dBzHqQ+UhnHZVNUVpJkTVXU8cEZ4/h6wRy3nzwL235I+m8QduQu+Xsxl593EhumW2Py/z6wCYNgFLwIw/VRTrf60ypKRP3i+1Z/56NMArHnFvjBPbDUbgIEH2R2w718yOtHHIf+0O23veeQMAHqqieAFQam7++1ZAIzsaHfNPjbZkpwPOtoSpq+dZl79PU49BICKV9+tfgFDDgBAcl4G4OsS+4EVCbcTl5rIWljcBYBJC9YCUNTRtJ8vF1q5bYeiRJPrVpoQ2zoItUvnmaYzYJDdETxrg4nZXdvZOZs22v5u4fhN4Y7cjsFPv7I8XWJ0E5jbFaQKt/E7cCNhNyLulZ+XtDvTHbnV+0ndH1NM6+Kn31SEW/fL3w7R6s93c6NJTPqO4zgNizZbGwaf9B3HceK4tbLjOE7LQVWprJ/onO0On/Qdx3Fq4Ms7juM4LQdf3mlcWnfoxB7/cxJ73PoVAJN+YR7xbf/5BgD//K7ZMfzvXWad8OVJFrXzl93MNuDdYea7/8EZvwdg/1EXAXDZgecl+ujS2e6krgw3Qd/ymkXjHBeidZ4Zbze67fYdi9ZZNetzAPpffBQAZZd/DEDu3laWnA8Sbc+vagdUR+t8vjjYKnQ0m41P560GoE3XfgB88bWV23cyi4c1wXKhbXGrRJvLQ0RP3/4dAJg7eT4APTsMBGDThvTROp3apEbrFBelRuoAtA3++ZUx24V4tE4UaRNF61TbLmT2vq95TOr+TNE6dfHTzxStk4m6HL+lcTfuld9EUNDKWl1imjRNYtJ3HMdpSBStLxfN7Q6f9B3HceIoaJVf6TuO47QIVKGy3G/OchzHaRmo+pp+YzKkfQVvHbaajhe9A8Ad95nNwnnBZuHzY6389z1NIP1oeH8Axv3gl0C1xcKFQ81ioaiHWSVUavUf9ffPmX/RyC4mnl4wbiYA1/5gZwCWTzNhduD/HQdA6SVms5DzzXMBkJxPAZiLed4XtuuUaPujYKvQqnMvAN6dZQnFIuH2k9lWLg59r1piomv7zibcRhYLffoWJ9qcN9WE2z4dTbj9YN3KULZzyoOQ27292TBEwm3HVsE/Pwi3xYWpoi1U2y5Ewm0k7CYSoeen+ucX5KaKsnkxBTQ/qVxfwm06kXVLbRYy2Takw4XblkOVT/qO4zgtBA/ZdBzHaTkoUOVCruM4TgtB1YVcx3GcloL6zVmO4zgtCJ/0G5cF0+dzxaEX8eQX7wPw8d7PAXBFqUXtLDhzXwCePMSidX448QUAzulxGADLqnYBoFWuRZ389mGLtLlyUMdEH6e+NgGAv591oJ3zqkXr7HDH6QCUnfGUHXjwSQDk5JntwrTSNtZ2sFR4LUTmtO0+INH22KmW67i4l0XafDJzOQCdurcFYEWwZejQ1dqaP2MFAEMGdwZg9gRL4DKo646JNt9bswyA/iHiJ4rW6Vls0ToVpRsA6NLaonUqy0sB6FgUyiFaJ2HDsKk6eqfamsHqEklTaonWya8lWqe2SB2oGa1TI5pnCxOipD0mQ5jL9poQJd6ER+s0Bs33jtyGTowOgIjMEZGJIjJBRMY3xhgcx3FqJdyRm2nbVkSkk4iMFZEZ4bFjmmMOC3NltJWKyPFh34MiMjtp39BMfTbmlf5hqrq8Eft3HMdJi9JgcfqXAq+p6k0icmkoX5IyFtU3gKFgXxLATODVpEMuUtUn69phk1jecRzHaVBUqWqY6J3jgOHh+UPAm8Qm/RgnAC+p6sat7bBRlnewL9JXReQTETmzkcbgOI6TFlW70s+01QPdVXVReL4Y6J7h+JOAR2N114vIFyIySkQKM3XYWFf6B6vqAhHpBowVkWmqOi75gPBlcCZAt7wCjtyhI50uOhmAS567HIBrvnsdAKfNNxF23N17APDqG+ZHf2BHEzUvv/tDAJ48bicA7nz1vwB868afJvpbfp0Js91vt7YrxlwPwMKBhwKQ38bOeXWOia7tepmv/hOfLwSguN+uADz7mfnud+7fL9H2F9NNdO0WbBSWzjdbhh136Wr7P5gNwP5DzaZh+nsTARjc3dp8de2yUG6baDPyy+/dPlW47dbG/uYVwXahS+vgnx+E2k6RDUMotytMtViAmsJtYcwvvyCmgBbEhNu4DUNeGiU3kw1DTWE3tZzWhiGD+JtJDK6LXhoXarMh3DrbB3XMnNUlpkuOVtXRyQeIyH+BHmnOvTylP1UVkVq/SUSkJ7AH8EpS9WXYl0UBMBr7lXDt5gbcKJO+qi4Ij0tF5Glgf2Bc7JjR2Itgp1ZtmmfslOM42yda5yv55ao6bPNN6ZG17RORJSLSU1UXhUl96Waa+jHwtKomMh4l/UooE5EHgAszDbjBwkMghwAAC3dJREFUl3dEpI2ItIueA0cDkxp6HI7jOLUS4vQzbfXAGGBkeD4SeHYzx/6E2NJO+KJA7Gfu8dRhLm2MK/3uwNPhp3ge8IiqvtwI43Acx0mL0mCGazcBT4jI6cBc7GoeERkGnKWqZ4TyAKAv8Fbs/IdFpCu2OjkBOCtThw0+6avqLGCvhu7XcRynzqhSWZ79SV9VVwBHpKkfD5yRVJ4D9E5z3OFb2qeHbDqO48RQhSptnlJik5j0C4cMYeDYNxnV3aJzSv/Xbjo7sI1FohxztUXWPHmC2S0ceq9ZJvz1Hvui/NV1Ztuw60t3AFByjEXmLD/s94k+8kddAcBLK80KobiftTX6w68B6LLTfgDcNc4sEXoMsciaVz6y/X12MnF+9nS73yyKzIHq6Jxvj7BznvvEErbsM8Kiid5/zn6x7d3PLCD+HSwWhnSzaJ3ydasA6FvcKtFm+UaLAEpE75RZtE73NhatE0XndGodJU2JonVyU8qtYpE6AIWxUJmimO1CQW7qOfHonPyYUhS3aYCaET6ZonPikTnxaJ+057D5cs3zsx+Z45E6TYdKn/Qdx3FaBgo0U781n/Qdx3HS4Vf6juM4LYQqhXLPnOU4jtNy8OWdRmTq7CXsf8oo5tz3MwC6/vFvAIz+7HEAfnX87QD0fNOM5spHXAbAe3ueC0C7nnZX9C2TTXjssZf57J//zOREHwP2t8inG/5j9zbssN/eADw9diYAOw/rD8CU8SbcHnGkibAvPW2++6f9fDgAd9/1PABnnbB7ou13nrTbEL614yEAPL7CrBv27dsBgLI1Jv4O6WIiclkQbgd1Mq/8TSXrAegXvPIBKoNw2zUIt5XBdqFDzAu/bX6q6No6Vk4n5LbKT++fHxEvx4XaTCItpLFdyFSug59+vC6TMLs1Qm0mYdaF2uaBor684ziO01JwIddxHKeF4ZO+4zhOC0HVo3ccx3FaDIpH7zQqOXn5tO7cm7Ny9wRg4MEmhB76mCUh3+t4S1Z+xPVvAnDAT34EwC9vNbfm7/z02wD87V7b/7OTDwbg3tEvJPq47MIfAnDd9Q8DMOr6UwH47UV/t/rTzrFzH3sagJ9cYsLvo7dPtT52+TEAty6eY2Mb0CnRdsmqJQDs26s9AGXrbNy7BOE2Smo+oIMJtZEo26tdqkjbuVX1nysSajsW2R22kehaXJgqwrYtSN3fJna7bKu8mspjUUw1LcxLPSejsJu7eWEXMidCr5kYPbMIG6/bUtHVRVknwtf0HcdxWhi+vOM4jtNCsDX9xh5FdvBJ33EcJw1+pe84jtNCUKBBUqg0Aj7pO47jxFDUo3cakz36d+Lde39K+wN/A8Da9+4EqLX8cax8z6hQvtXsG646/BQAbv2/qYk+fj2sFwCXLpkDwIm7dgHgF6sWA3DMDsEyIUTeHNzHvO4rSs0iYZ/uZpkQRdrs3Kkw0XYUaTOoONXbvm87e/ujyJpebVLL3VrlprwPnYtqpjTuWJha174gtdwuPzUEpU0sWqd1XaJ3Yt3Gy7EuapTTdFGjLgfdpnK6OtH6LbfkNpvKuOsLi97xSd9xHKdl0IyF3JqXjg2AiIwQkekiMlNELm2MMTiO49RGdKWfaWuKNPiVvojkAncCRwHzgY9FZIyqTmnosTiO49RGc73Sb4zlnf2Bmao6C0BEHgOOA3zSdxxnu6CK5mvDINrAP1FE5ARghKqeEcqnAN9Q1bNjx50JnBmKuwOTGnSgW0cXYHljD6IO+Djrj6YwRmhZ4+yvql23pQEReTmMJRPLVXXEtvTV0Gy3Qq6qjgZGA4jIeFUd1shDyoiPs35pCuNsCmMEH+eW0tQm8i2hMYTcBUDfpHKfUOc4juNkmcaY9D8GBovIQBEpAE4CxjTCOBzHcVocDb68o6oVInI28AqQC9yvqpMznDY6+yOrF3yc9UtTGGdTGCP4OJ1Agwu5juM4TuPRKDdnOY7jOI2DT/qO4zgtiO160t9e7RpEpK+IvCEiU0RksoicG+o7ichYEZkRHjs29ljB7oIWkc9E5PlQHigiH/5/e3cXYlUZhXH8/6hhatCYlJRdaGmFSllkGEmIfWpiUUGGUJFURN8YoXkR3RVFX1hKaWUiFpmWFKVlgXShmTKZoJZlpKEplBYVpfJ08b6n2ZqDkTX7dc76wWH21wxr1pm9Zu/37L12zutr+QP1umNskbRA0gZJ6yWdX2I+Jd2X3/N1kuZLOrqEfEp6UdIOSesqyw6aPyXP5HjXSjqn5jgfy+/7WkmLJLVU1k3NcW6UdFlHxdmZFVv0K+0axgCDgeslDa43qr/sBSbbHgyMAO7IsU0BltkeBCzL8yW4B1hfmX8UeNL2QOBHYFItUe3vaeA922cAZ5HiLSqfkvoBdwPn2h5KuhBhAmXk82XgwGvL28vfGGBQft0KzOigGOHgcb4PDLV9JvAFMBUg71MTgCH5e57LdSEchmKLPpV2Dbb/ABrtGmpne5vtNXn6Z1KB6keKb07ebA5wVT0RtpF0MnAFMCvPCxgNLMib1B6npGOBC4HZALb/sL2LAvNJuuKth6RuQE9gGwXk0/Zy4IcDFreXvyuBV5ysAFoknVhXnLaX2t6bZ1eQ7t1pxPmq7d9tbwY2kepCOAwlF/1+wJbK/Na8rCiS+gNnAyuBvra35VXbgb41hVX1FPAAbQ8C6gPsquxkJeR1ALATeCkPQ82S1IvC8mn7O+Bx4FtSsd8NrKa8fDa0l7+S962bgXfzdMlxHrFKLvrFk3QM8AZwr+2fquucroWt9XpYSeOAHbZX1xnHP9ANOAeYYfts4BcOGMopJJ+9SUefA4CTgF78faiiSCXk71AkTSMNnc6rO5bOrOSiX3S7BklHkQr+PNsL8+LvG6fJ+euOuuLLLgDGS/qGNDw2mjR23pKHJ6CMvG4FttpemecXkP4JlJbPi4HNtnfa3gMsJOW4tHw2tJe/4vYtSTcB44CJbrt5qLg4O4OSi36x7RryuPhsYL3tJyqrFgM35ukbgbc6OrYq21Ntn2y7Pyl/H9qeCHwEXJs3KyHO7cAWSafnRReRWm0XlU/SsM4IST3z30AjzqLyWdFe/hYDN+SreEYAuyvDQB1O0uWkIcjxtn+trFoMTJDUXdIA0gfPn9QRY6diu9gXMJb0af5XwLS646nENZJ0qrwWaM2vsaTx8mXAl8AHwHF1x1qJeRTwdp4+hbTzbAJeB7oXEN8w4NOc0zeB3iXmE3gY2EBq9T0X6F5CPoH5pM8Z9pDOnCa1lz9ApCvjvgI+J12NVGecm0hj9419aWZl+2k5zo3AmLrf/87wijYMIYTQREoe3gkhhPAfi6IfQghNJIp+CCE0kSj6IYTQRKLohxBCE4miH2onaZ+k1ty98jNJkyX9679NSQ9WpvtXOzqG0Oyi6IcS/GZ7mO0hwCWkLpAPHcbPe/DQm4TQnKLoh6LY3kFq93tnvmO0a+63vir3W78NQNIoScslvZN7rc+U1EXSI6QumK2SGj1cukp6IZ9JLJXUo67fL4S6RdEPxbH9NalX/QmkOzZ32x4ODAduybfkQ2qzexfpeQunAlfbnkLbmcPEvN0g4Nl8JrELuKbjfpsQyhJFP5TuUlKfmFZS++o+pCIO8InT8xb2kW7vH9nOz9hsuzVPrwb6/4/xhlC0bofeJISOJekUYB+pK6SAu2wvOWCbUfy9VXB7PUV+r0zvA2J4JzStONIPRZF0PDATmO7UGGoJcHtuZY2k0/IDVgDOy11YuwDXAR/n5Xsa24cQ9hdH+qEEPfLwzVGkh2jMBRotq2eRhmPW5HbGO2l77N8qYDowkNTeeFFe/jywVtIaUpfGEEIWXTbDESkP79xve1zdsYRwJInhnRBCaCJxpB9CCE0kjvRDCKGJRNEPIYQmEkU/hBCaSBT9EEJoIlH0QwihifwJw5h7ig+IxPgAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "pos_encoding = PositionalEncoding().call(tf.zeros([1, 40, 128], tf.float32))\n",
        "\n",
        "plt.pcolormesh(pos_encoding[0], cmap='RdBu')\n",
        "plt.xlabel('Depth')\n",
        "plt.xlim((0, 128))\n",
        "plt.ylabel('Position')\n",
        "plt.colorbar()\n",
        "plt.savefig('../img/pos_enc.png')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ox6N8z38A1o6"
      },
      "source": [
        "### Self-Attention\n",
        "This is the special sauce in the transformer! It solves the problems of parallelization and loss of contextual information for words that are distant from each other within a sequence. These were issues with seq2seq models such as `RNN`s and `LSTM`s.\n",
        "\n",
        "The main cause of these issues within those models is that they needed to process each word in a sequence one by one. This meant that they took longer to train. What if we could process all of the words in parallel?\n",
        "\n",
        "The other issue is the way that context and structure is derived. Those models start by passing the first word (token) to an encoder and calculating a new `state` with a random starting state. Then in the next time step use the next word and the previous state to create a new state. This process is repeated until all words are exhausted at which point a fixed length context vector is generated. The context vector is then passed to the decoder for output generation.\n",
        "\n",
        "The problem is that by the time the final context vector is computed, contextual information about the earlier states can be loss because the earlier computations do not have access to the later words. In addition the fixed sized context vector may not be large enough to retain all of the information.\n",
        "\n",
        "So how does the transformer solve these problems?\n",
        "\n",
        "The transformer's encoders and decoders consume the entire sequence and process all words (embedding) in parallel. Because of this parallelization, training time is reduced. Second, this parallel processing means that the context can be computed from all the words together, resulting in a more complete context.\n",
        "\n",
        "##### Linear Transformation\n",
        "The first thing that happens when a sequence of embeddings is passed to the transformer's inputs (encoder and decoder) is that each embedding experiences three separate linear transformations resulting in three vectors - `query`, `key`, and `value`. These transformations occur when the input vectors (embedding) are multiplied by 3 weight matrices. The proper weights are learned through training. The image below represents a vector of seq_length 2 and embedding size of 4.\n",
        "\n",
        "Why three linear transformations?\n",
        "\n",
        "Since each `weight matrix` is initialized with random weights, the resultant vectors each learn some different information about the embedding (word) being processed. This is important when calculating the attention score since we don't want to just derive the dot product of the vector with itself.\n",
        "\n",
        "<img src='img/self-attention-matrix-calculation.png' width='250'>\n",
        "<div align=\"center\" style=\"font-weight:bold\">Fig 2: Linear Transformation</div>\n",
        "<div align=\"center\" >Image from http://jalammar.github.io/illustrated-transformer/</div>\n",
        "\n",
        "Now that we have these 3 vectors for each embedding in the sequence we can calculate the `attention score`. The attention score measure the strength of the relationship between a word in the sequence with all the other words.\n",
        "\n",
        "#### Calculate the self-attention:\n",
        "1. Take the query vector for a word and calculate it's `dot product` with the transpose of the `key vector` of each word in the sequence - including itself. This is the `attention score` or `attention weight`.\n",
        "2. Then divide each of the results by the square root of the dimension of the key vector. This is the `scaled attention score`.\n",
        "3. Pass the them through a `softmax` function, so that values are contained between 0 and 1.\n",
        "4. Take each of the `value vectors` and calculate the `dot product` with the output of the `sofmax` function.\n",
        "5. Add all the `wighted value vectors` together.\n",
        "\n",
        "Notice in the figure below that we are doing matrix operations on `seq_length x embedding_size` matrices. This shows a toy example of two word with embedding size of 3.\n",
        "\n",
        "<div align=\"center\" style=\"font-weight:bold\">Attention calculation </div>\n",
        "\n",
        "<img src='img/self-attention-matrix-calculation-2.png' width='500'>\n",
        "<div align=\"center\" style=\"font-weight:bold\">Fig 3: Attention</div>\n",
        "<div align=\"center\" >Image from http://jalammar.github.io/illustrated-transformer/</div>\n",
        "\n",
        "The `dot product` results in a `seq_length x seq_length` matrix. Think `correlation matrix` where the relationships between any two members can be found by their intersections. In this case the members are words and their intersections are `attention scores`.\n",
        "\n",
        "Multiplying the `value matrix` `attention matrix` results once again in a `seq_length x embedding_size` matrix. This matrix holds the `contextual information` for each embedding.\n",
        "\n",
        "#### Multi-Head Self-Attention\n",
        "\n",
        "What I described above is `single-head self-attention`. In practice we use `multi-head self-attention`. With `multi-head self-attention`, each word is processed by several attention heads. In the original paper they used eight, which is what I use here.\n",
        "\n",
        "<img src='img/attention-arch.png' width='250'>\n",
        "<div align=\"center\" style=\"font-weight:bold\">Fig 4: Mutli-Head attention</div>\n",
        "<div align=\"center\" >Image from https://arxiv.org/pdf/1706.03762.pdf</div>\n",
        "\n",
        "For `multi-head self-attention`, the `query`, `key`, and `value` vectors are divided by the number of heads, and each segment is passed through a different head. This results in eight vectors which are then concatenated together and transformed by multiplying with another weight matrix so that the resultant vector is the size of the input embedding vector.\n",
        "\n",
        "<img src='img/transformer_attention_heads_weight_matrix_o.png' width='500'>\n",
        "<div align=\"center\" style=\"font-weight:bold\">Fig 5: Concatanation of attention vectors</div>\n",
        "<div align=\"center\" >Image from http://jalammar.github.io/illustrated-transformer/</div>\n",
        "\n",
        "From the [paper](https://arxiv.org/pdf/1706.03762.pdf):\n",
        "\n",
        "> Multi-head attention allows the model to jointly attend to information from different representation\n",
        "subspaces at different positions. With a single attention head, averaging inhibits this.\n",
        "\n",
        "> $\\begin{align}\n",
        "MultiHead(Q, K, V ) &= Concat(head1, ..., headh)W^O\\\\\n",
        "where, head_i &= Attention(QW^Q_i, KW^K_i, VW^V_i)\n",
        "\\end{align}$\n",
        "\n",
        "> In this work we employ h = 8 parallel attention layers, or heads. For each of these we use\n",
        "$d_k = d_v =\\, ^{d_{model}}/_h = 64$. Due to the reduced dimension of each head, the total computational cost\n",
        "is similar to that of single-head attention with full dimensionality\n",
        "\n",
        "#### Attention Masks\n",
        "\n",
        "Prior to the `softmax` function we need to apply a mask to the `attention score`. The mask will be either a `padding mask` or a combination of the `padding mask` and a `look-ahead mask`.\n",
        "\n",
        "##### Padding Mask\n",
        "\n",
        "The sequences that are the input to the `encoder` and the `decoder` must be of the same length. Because sentences are of varying length, one must either truncate the sentence, or pad the sentence so that they are all of a specified length. As a result, the padding might be interpreted as meaningful by the `attention` mechanism l eading to\n",
        "unreliable learning.\n",
        "\n",
        "To mitigate this effect a `padding mask` is applied so that when added to the `attention score` the values in the padded positions become so small as to be ignored. This is acheived by setting the positions in the mask to a value close to `negative infinity`.\n",
        "\n",
        "##### Look-Ahead Mask\n",
        "\n",
        "Transformers are `autoregressive models`. They look at at all of previous input in a sequence in order to predict the next output (`token`). But because, unlike `RNN`s, they recieve the entire sequence at once there needs to be a way to limit attention for any position so that only the portion of the sequence that comes before it is attended to. This is because it is desired that the model infer the next position without having a peak. If it were able to look at the next position, then it would just copy it.\n",
        "\n",
        "That's what the `look-ahead mask` accomplishes. All of the `attention scores` for words that occur to the right of a `query` word in the sequence are masked. This limits the `query` word to attend to itself and all words to the left in the sequence.\n",
        "\n",
        "The `look-ahead mask` is only applied to the decoder. More on that later.\n",
        "\n",
        "**How does it works?**\n",
        "\n",
        "1. Create a mask where the values for the upper right triangle above the diagnal are all ones and the rest zeros.\n",
        "2. Multiply the mask by $-1e9$.\n",
        "3. Add the mask to the `attention matrix`.\n",
        "\n",
        "<img src='img/Look-ahead-mask.png' width='1000'>\n",
        "\n",
        "<div align=\"center\" style=\"font-weight:bold\">Fig 6: Look-Ahead Mask implementation</div>\n",
        "\n",
        "After applying `softmax` to the `attention matrix`, all of those extremely small values will become zero. So how does that matter?\n",
        "\n",
        "In the toy `attention matrix` in the figure, I've labeled both axises $T_1,...,T_n$. The columns represent the token (word) `keys` and the rows are the token `queries`. We query for attention along the rows. The attention score for any query $T_i$, are the values in it's row with respect to the `key` columns $[T_1,...,T_n]$. Notice that the top most `query`, $T_1$ can only get the score for the left most `key` $T_1$. There is no meaningful imformation for $T_2 - T_5$  (values too low). The `query` $T_2$ has access to the set of `key`s $(T_2$, $T_1)$. And so on, until $T_5$, the last word, which has access to the entire sequence.\n",
        "\n",
        "There is one more thing. The attention mask is not applied alone. Remember that there is also padding to account for. Therefore, the `look-ahead mask` is applied along with the `padding mask`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cVcKOK5OA5LA"
      },
      "source": [
        "#### Attention computation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Ag4g6KfBFBz"
      },
      "source": [
        "This is the attention formula.\n",
        "\n",
        "$Attention(Q, K, V ) = \\text{softmax}\\left(\\dfrac{QK^T}{\\sqrt{d_k}}\\right)V $\n",
        "\n",
        "**Attention with mask**\n",
        "\n",
        "And this is what it looks like when applying the mask.\n",
        "\n",
        "$Attention(Q, K, V, M ) = \\text{softmax}\\left(\\dfrac{QK^T}{\\sqrt{d_k}}+M\\right)V $\n",
        "\n",
        "<img src='img/attention-dot-product.png' width='250'>\n",
        "<div align=\"center\" style=\"font-weight:bold\">Fig 7: Attention Dot Product</div>\n",
        "<div align=\"center\" >Image from https://arxiv.org/pdf/1706.03762.pdf</div>\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pOl6_WF2A3WG"
      },
      "outputs": [],
      "source": [
        "#\n",
        "# Calculate attention\n",
        "#\n",
        "def scaled_dot_product_attention(queries, keys, values, mask):\n",
        "    '''\n",
        "    Takes three vectors and mask\n",
        "    Returns the attention and attention weights\n",
        "    '''\n",
        "    # The steps describe above under 'Calculate the Attention'\n",
        "    # Step 1\n",
        "    product = tf.matmul(queries, keys, transpose_b=True)\n",
        "\n",
        "    # Step 2\n",
        "    keys_dim = tf.cast(tf.shape(keys)[-1], tf.float32)\n",
        "    scaled_product = product / tf.math.sqrt(keys_dim)\n",
        "\n",
        "    if mask is not None:\n",
        "        scaled_product += (mask * -1e9)\n",
        "\n",
        "    # step 4\n",
        "    attention_weights =tf.nn.softmax(scaled_product, axis=-1)\n",
        "\n",
        "    # Step 5\n",
        "    attention = tf.matmul(attention_weights, values)\n",
        "\n",
        "    return attention, attention_weights"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L2SvfuWLBWM7"
      },
      "source": [
        "#### Multi-head attention sublayer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-tdVB9DN63E7"
      },
      "source": [
        "This is where everything discussed previously gets implemented. Here is a summary of what it does.\n",
        "\n",
        "1. Instantiates 4 `dense layers`\n",
        "2. Passes the input through 3 dense layer, creating 3 `linear transformations` resulting in the `query`, `key`, and `value` vectors.\n",
        "3. Split each vector by number of heads.\n",
        "4. Pass the vectors to the `scaled_dot_product_attention()` function and calculate the attentions per head\n",
        "5. Concatenate the attention resulting from each head.\n",
        "6. Pass result through a final dense layer which reshapes the `attention vector` back to (batch_size, seq_length, d_model)\n",
        "7. Return the `attention vector` and the `attention weights`\n",
        "\n",
        "\n",
        "The `MultiHeadAttention` class accepts the `number of heads` as an argument when an object is instantiated.\n",
        "The call method accepts three tensors with dimension of (batch_size, seq_length, d_model) as input. In each encoder layer this is the same tensor. In the decoder where a layer contains two attention layers, for the first attention layer it's the same tensor.  For the second `attention layer` the `key` and `value` tensors are the output of the encoder, and the `query` tensor is the output of the first `attention layer` of the decoder."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2otZB4ibBW1O"
      },
      "outputs": [],
      "source": [
        "class MultiHeadAttention(layers.Layer):\n",
        "\n",
        "    # nb_proj is the number of heads\n",
        "    def __init__(self, nb_proj):\n",
        "        super(MultiHeadAttention, self).__init__()\n",
        "        self.nb_proj = nb_proj\n",
        "\n",
        "    def build(self, input_shape):\n",
        "        self.d_model = input_shape[-1]\n",
        "        assert self.d_model % self.nb_proj == 0\n",
        "\n",
        "        # Calculate the head dimensions.\n",
        "        self.d_proj = self.d_model // self.nb_proj\n",
        "\n",
        "        # These layers contain the weights for the linear transformations\n",
        "        self.query_lin = layers.Dense(units=self.d_model)\n",
        "        self.key_lin = layers.Dense(units=self.d_model)\n",
        "        self.value_lin = layers.Dense(units=self.d_model)\n",
        "\n",
        "        # Used to transform after concatenating the weighted value vectors\n",
        "        self.final_lin = layers.Dense(units=self.d_model)\n",
        "\n",
        "    # This method splits the query, key, and value vectors by the number of heads\n",
        "    def split_proj(self, inputs, batch_size): # inputs: (batch_size, seq_length, d_model)\n",
        "        shape = (batch_size,\n",
        "                 -1,\n",
        "                 self.nb_proj,\n",
        "                 self.d_proj)\n",
        "\n",
        "        # Reshape the tensor to account for the multiple heads and reduced vector dimension\n",
        "        splited_inputs = tf.reshape(inputs, shape=shape) # (batch_size, seq_length, nb_proj, d_proj)\n",
        "\n",
        "        # Reconfigure the axeses.\n",
        "        return tf.transpose(splited_inputs, perm=[0, 2, 1, 3]) # (batch_size, nb_proj, seq_length, d_proj)\n",
        "\n",
        "    def call(self, queries, keys, values, mask):\n",
        "        '''\n",
        "        If this is the first attention layer of the encoder, then all vector inputs are the same vector.\n",
        "        For later layers, this is the output of the previous layer.\n",
        "        For the decoder, keys and values are the output of the encoder.\n",
        "            The queries is the inferred words up to this time step.\n",
        "        '''\n",
        "        batch_size = tf.shape(queries)[0]\n",
        "\n",
        "        # Initialize the weight matrices\n",
        "        queries = self.query_lin(queries)\n",
        "        keys = self.key_lin(keys)\n",
        "        values = self.value_lin(values)\n",
        "\n",
        "        # Split the vectors by number of heads\n",
        "        queries = self.split_proj(queries, batch_size)\n",
        "        keys = self.split_proj(keys, batch_size)\n",
        "        values = self.split_proj(values, batch_size)\n",
        "\n",
        "        # Get attention and weights\n",
        "        attention, attention_weights = scaled_dot_product_attention(queries,\n",
        "                                                                    keys,\n",
        "                                                                    values,\n",
        "                                                                    mask)\n",
        "        # Flip dims 1 and 2\n",
        "        attention = tf.transpose(attention, perm=[0, 2, 1, 3])\n",
        "\n",
        "        # Concat all the attention vectors\n",
        "        concat_attention = tf.reshape(attention,\n",
        "                                      shape=(batch_size, -1, self.d_model))\n",
        "\n",
        "        # Transform the concated vectors to reshape back to (batch_size, seq_length, d_model)\n",
        "        outputs = self.final_lin(concat_attention)\n",
        "\n",
        "        return outputs, attention_weights"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pvrv4FkiBkzD"
      },
      "source": [
        "### Encoder"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HvUvEOuJ63E9"
      },
      "source": [
        "Having understood all that we have covered so far, we should be in good shape to understand what follows.\n",
        "\n",
        "The `encoder` is made up of one or more `encoder layers`. Each `encoder layer` is made up of two sub-layers. The first the `attention layer`, followed by the `point-wise feed forward layer`. The ouput of an `encoder layer` is the input to the next.\n",
        "\n",
        "<img src='img/encoder-2.png' width='450'>\n",
        "<div align=\"center\" style=\"font-weight:bold\">Fig 8: Encoder Architecture</div>\n",
        "<div align=\"center\" >Image from https://arxiv.org/pdf/1706.03762.pdf</div>\n",
        "\n",
        "\n",
        "The role of the encoding layer is to process the input and encode the `contextual` and `structural` information of the sequence. It accounts for how one word relates to another within the sequence and the strength of that relationship. This contextual information is processed at each layer, with the ouput of one being the input of the next. Because of this, it is necessary to assure that the input and output matrices are of the same shape. The final ouput then is passed as the input to the `decoder`.\n",
        "\n",
        "Regarding the `attention layers` in the encoder, this is what the paper says.\n",
        "\n",
        "> The encoder contains self-attention layers. In a self-attention layer all of the keys, values\n",
        "and queries come from the same place, in this case, the output of the previous layer in the\n",
        "encoder. Each position in the encoder can attend to all positions in the previous layer of the\n",
        "encoder"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9UwbOQg463E-"
      },
      "source": [
        "#### EncoderLayer Class\n",
        "\n",
        "The three main components of `EncoderLayer Class` are the implementation of the `multi-head attention` sub-layer, followed by a `normalization` sub-layer, a `point-wise feed forward layer`, and another `normalization` sub-layer. It also includes the residual connection that sums the `attention layer's` input with its output before normalization.\n",
        "\n",
        "The same occurs in the `FNN` where the normalized output of the `attention layer` (the input to the FNN) is added to the output of the `FNN` before normalization. The normalized output of the `FNN` is the input to the next `encoder layer`, or in the case of the last layer, the input to the decoder.\n",
        "\n",
        "To be clear, the `point-wise feed forward layer` consists of two `dense layers` with a `relu activation function` in between them.\n",
        "\n",
        "The EncoderLayer take as arguments on object instantiation the number of nodes for the `dense layers`, the number of heads for the `attention layer`, and the dropout rate.\n",
        "\n",
        "Also note that to calculate attention in the encoder only a `padding mask` is used.\n",
        "\n",
        "A summary of what this layer is doing follows.\n",
        "\n",
        "1. Creates the `attention layer` by instantiating a `multi-head attention` object, a `dropout` object, and the `normalization` object.\n",
        "2. Creates the `point-wise feedforward network` layer by instantiating two `dense` objects, a `dropout` object, and the `normalization` object.\n",
        "3. Takes as arguments the input of shape (batch_size, seq_length, d_model), the mask, and a boolean for the dropout.\n",
        "4. Pass the input and mask to the `multi-head attention` object to get the attention calculation\n",
        "5. Pass attention through dropout.\n",
        "6. Add the input to the attention and normalize.\n",
        "7. Pass the `normalized attention` thwough the `FNN`\n",
        "8. Pass through dropout.\n",
        "9. Add at the `normalized attention` to the `normalized output`.\n",
        "10. Return the output with shape (batch_size, seq_length, d_model).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pN5n9QJ0Blho"
      },
      "outputs": [],
      "source": [
        "class EncoderLayer(layers.Layer):\n",
        "\n",
        "    def __init__(self, FFN_units, nb_proj, dropout_rate):\n",
        "        '''\n",
        "        FFN-units    - # of nodes for the point wise feed forward layer\n",
        "        nb_proj      - # of heads\n",
        "        dropout_rate - The dropout rate for the layers\n",
        "        '''\n",
        "        super(EncoderLayer, self).__init__()\n",
        "        self.FFN_units = FFN_units\n",
        "        self.nb_proj = nb_proj\n",
        "        self.dropout_rate = dropout_rate\n",
        "\n",
        "    # This is a tensorflow layers method. Automatically called by __call__\n",
        "    # to automatically build the layers.\n",
        "    def build(self, input_shape):\n",
        "\n",
        "        self.d_model = input_shape[-1]\n",
        "\n",
        "        # Multi-head attention\n",
        "        self.multi_head_attention = MultiHeadAttention(self.nb_proj)\n",
        "        self.dropout_1 = layers.Dropout(rate=self.dropout_rate)\n",
        "        self.norm_1 = layers.LayerNormalization(epsilon=1e-6)\n",
        "\n",
        "        # Point wise feed forward\n",
        "        self.dense_1 = layers.Dense(units=self.FFN_units, activation=\"relu\")\n",
        "        self.dense_2 = layers.Dense(units=self.d_model)\n",
        "        self.dropout_2 = layers.Dropout(rate=self.dropout_rate)\n",
        "        self.norm_2 = layers.LayerNormalization(epsilon=1e-6)\n",
        "\n",
        "    def call(self, inputs, mask, training):\n",
        "        '''\n",
        "        inputs   -> Input tensor of shape (batch_size, seq_length, d_model)\n",
        "        mask     -> Padding mask to ignore padding zeros as data\n",
        "        training -> Boolean - If 1, then use dropout, else not\n",
        "        '''\n",
        "        # Attention sub-layer\n",
        "        # Retain only the attention. Don't need to return weights in this layer.\n",
        "        attention, _ = self.multi_head_attention(inputs,\n",
        "                                                 inputs,\n",
        "                                                 inputs,\n",
        "                                                 mask)\n",
        "        attention = self.dropout_1(attention, training=training)\n",
        "        attention = self.norm_1(attention + inputs)\n",
        "\n",
        "        # Point-Wise Feed Forward sub-layer\n",
        "        outputs = self.dense_1(attention)\n",
        "        outputs = self.dense_2(outputs)\n",
        "        outputs = self.dropout_2(outputs, training=training)\n",
        "        outputs = self.norm_2(outputs + attention)\n",
        "\n",
        "        # Returns tensor of shape (batch_size, seq_length, d_model)\n",
        "        return outputs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "me9FOCiT63E_"
      },
      "source": [
        "#### Encoder Class"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tp36YVWv63E_"
      },
      "source": [
        "The `full encoder` is a `stack of encoding layers`. This class implements the `encoder stack`. In addition to that, it also implements `embedding` and `positional encoding`. The output is the output of the `encoder stack`.\n",
        "\n",
        "Here is a summary of what the encoder is doing.\n",
        "\n",
        "1. Creates an `embedding layer`\n",
        "2. Initializes the `positional encoding` object\n",
        "3. Creates a `dropout layer`\n",
        "4. Creates number of `encoding layers` specified by `nb_layers`\n",
        "5. Takes a (batch_size, seq_length) shape input, an `attention mask`, and a boolean\n",
        "6. Passes the input through the `embedding layer` and produces a tensor of shape (batch_size, seq_length, d_model)\n",
        "7. Creates the `positional encoding vectors`\n",
        "8. Adds the `positional encoding vectors` to the `embedding vectors`\n",
        "9. Passes the resulting input tensor, the mask, and the training boolean to the `encoding layer stack`\n",
        "10. Iterates over the `encoding layer stack`\n",
        "11. Returns an output tensor of shape (batch_size, seq_length, d_model)\n",
        "\n",
        "In the code you will see this sequence:\n",
        "```Python\n",
        "   outputs = self.embedding(inputs)\n",
        "   outputs *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))\n",
        "   outputs = self.pos_encoding(outputs)\n",
        "```\n",
        "The explaination for the multiplication given [here](https://towardsdatascience.com/how-to-code-the-transformer-in-pytorch-24db27c8f9ec)\n",
        "> The reason we increase the embedding values before addition is to make the positional encoding relatively smaller. This means the original meaning in the embedding vector won’t be lost when we add them together."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hPbsh_Q1BuD4"
      },
      "outputs": [],
      "source": [
        "class Encoder(layers.Layer):\n",
        "    '''\n",
        "    nb_layers    -> Number of encoder layers\n",
        "    FFN_units    -> Number of nodes in the FFN\n",
        "    nb_proj      -> Number of attention heads\n",
        "    dropout_rate -> Droput rate\n",
        "    vocab_size   -> The size of the vocabulary\n",
        "    d_model      -> Size of embedding vectors\n",
        "    name         -> Name of the layer\n",
        "    '''\n",
        "    def __init__(self,\n",
        "                 nb_layers,\n",
        "                 FFN_units,\n",
        "                 nb_proj,\n",
        "                 dropout_rate,\n",
        "                 vocab_size,\n",
        "                 d_model,\n",
        "                 name=\"encoder\"):\n",
        "        super(Encoder, self).__init__(name=name)\n",
        "\n",
        "        # Number of encoder layers\n",
        "        self.nb_layers = nb_layers\n",
        "        # Embedding dimension\n",
        "        self.d_model = d_model\n",
        "\n",
        "        # Initialize embedding, positional encoding, droppout, and encoding layers\n",
        "        self.embedding = layers.Embedding(vocab_size, d_model)\n",
        "        self.pos_encoding = PositionalEncoding()\n",
        "        self.dropout = layers.Dropout(rate=dropout_rate)\n",
        "        self.enc_layers = [EncoderLayer(FFN_units,\n",
        "                                        nb_proj,\n",
        "                                        dropout_rate)\n",
        "                           for _ in range(nb_layers)]\n",
        "\n",
        "    def call(self, inputs, mask, training):\n",
        "        '''\n",
        "        inputs   -> Tokenized input of shape (batch_size, seq_length)\n",
        "        mask     -> Attention mask\n",
        "        training -> Boolean\n",
        "        '''\n",
        "        outputs = self.embedding(inputs)\n",
        "        outputs *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))\n",
        "        outputs = self.pos_encoding(outputs)\n",
        "        outputs = self.dropout(outputs, training)\n",
        "\n",
        "        # Forward pass\n",
        "        for i in range(self.nb_layers):\n",
        "            outputs = self.enc_layers[i](outputs, mask, training)\n",
        "\n",
        "        # Returns tensor of shape (batch_size, seq_length, d_model)\n",
        "        return outputs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qq9tQ79iBz4d"
      },
      "source": [
        "### Decoder"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l81lYgKH63FC"
      },
      "source": [
        "Like the encoder, the `decoder` is a `stack of decoder layers`. Each layer consists of three sub-layers - a `masked multi-head self-attention sub-layer`, a second `multi-head self-attention sub-layer`, and a `point-wise feed forward network sub-layer`.\n",
        "\n",
        "The `masked multi-head self-attention layer` is where the `look-ahead mask` is employed. This is the part of the decoder architecture that differs most from the encoder. The other two sub-layers are almost exactly like those of the encoder.\n",
        "\n",
        "So how does this `masked multi-head self-attention` work exactly? Hold on 'till we get the training section. For now just know that when training the model, the input to this first sub-layer is the target sequence. But for now, here is something from the paper.\n",
        "\n",
        "> Similarly, self-attention layers in the decoder allow each position in the decoder to attend to\n",
        "all positions in the decoder up to and including that position. We need to prevent leftward\n",
        "information flow in the decoder to preserve the auto-regressive property. We implement this\n",
        "inside of scaled dot-product attention by masking out (setting to −∞) all values in the input\n",
        "of the softmax which correspond to illegal connections.\n",
        "\n",
        "The second `attention sub-layer` is known as the `encoder-decoder attention sub-layer`. This is because it receives it's inputs both from the encoder output and from the `masked multi-head self-attention`.\n",
        "\n",
        "From the paper.\n",
        "\n",
        "> In \"encoder-decoder attention\" layers, the queries come from the previous decoder layer,\n",
        "and the memory keys and values come from the output of the encoder. This allows every\n",
        "position in the decoder to attend over all positions in the input sequence. This mimics the\n",
        "typical encoder-decoder attention mechanisms in sequence-to-sequence models such as\n",
        "[38, 2, 9].\n",
        "\n",
        "In the figure you can see that the encoder receives as input it's output. Huh? More on that later. The input goes through embedding an positional encoding and is passed to the `masked multi-head self-attention` sub-layer. The output of this lower attention sub-layer then becomes the input to the `encoder-decoder attention` sub-layer. That then becomes the vector used for calculating the `query vector` at that level.\n",
        "\n",
        "The output from the encoder is passed to the decoder at this level and becomes the vector used to calculate the `key vector` and the `value vector`. From there its just like the encoder.\n",
        "\n",
        "Note that the output of the decoder undergoes a linear transformation which then has `softmax` applied to it. More to come.\n",
        "\n",
        "<img src='img/decoder-arch-2.png' width='450'>\n",
        "<div align=\"center\" style=\"font-weight:bold\">Fig 9: Decoder Architecture</div>\n",
        "<div align=\"center\" >Image from https://arxiv.org/pdf/1706.03762.pdf</div>\n",
        "\n",
        "\n",
        "Also, like the encoder, there exist `residual` connections that connect sub-layer inputs to sub-layer outputs at the `normalization` sub-layers."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C0OVE0Xv63FC"
      },
      "source": [
        "#### DecoderLayer Class\n",
        "As mentioned previously, a decoder layer consists of three stacked sub-layers. The first two are `multi-head self-attention layers`, and the final is an FNN. This class implements the decoder layer. This is the summary of what it does.\n",
        "\n",
        "1. On instantiation it takes the number of nodes for the FNN, the number of heads, and the dropout rate.\n",
        "2. Creates the `masked multi-head self-attention` sub-layer with a `droput layer`, and a `nomaliztion layer`\n",
        "3. Creates the `encoder-decoder multi-head self-attention` sub-layer with a `droput layer`, and a `nomaliztion layer`\n",
        "4. Creates the `point-wise FNN` sub-layer with two dense layers, a `droput layer`, and a `nomaliztion layer`\n",
        "5. It accepts as input a decoder layer output, the encoder output, the `look-ahead mask`, the `padding mask`, and a boolean\n",
        "6. When training the first decoder layer takes the full target sequence as input. For inferencing, it takes the final output of the decoder. It also takes the look-ahead mask.\n",
        "7. Apply dropout to the attention vector\n",
        "8. Add the input to the attention vector\n",
        "9. Normalize\n",
        "10. The output (attention vector) of the first attention layer is passed as the first argument to the second attention layer along with two instances of the encoder output, and the padding mask.\n",
        "11. Apply dropout to the attention vector\n",
        "12. Add the first attention vector to the new attention vector\n",
        "13. Normalize\n",
        "14. Pass the updated attention vector into the FNN\n",
        "15. Apply dropout\n",
        "16. Add the second attention vector to the output\n",
        "17. Normalize\n",
        "18. Return tensor of shape (batch_size, seq_length, d_model)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZwVMzyl9B36B"
      },
      "outputs": [],
      "source": [
        "class DecoderLayer(layers.Layer):\n",
        "    '''\n",
        "    FFN-units    -> # of nodes for the point wise feed forward layer\n",
        "    nb_proj      -> # of heads\n",
        "    dropout_rate -> The dropout rate for the layers\n",
        "    '''\n",
        "    def __init__(self, FFN_units, nb_proj, dropout_rate):\n",
        "        super(DecoderLayer, self).__init__()\n",
        "        self.FFN_units = FFN_units\n",
        "        self.nb_proj = nb_proj\n",
        "        self.dropout_rate = dropout_rate\n",
        "\n",
        "    def build(self, input_shape):\n",
        "        self.d_model = input_shape[-1]\n",
        "\n",
        "        # Masked Multi-head attention layer\n",
        "        self.multi_head_attention_1 = MultiHeadAttention(self.nb_proj)\n",
        "        self.dropout_1 = layers.Dropout(rate=self.dropout_rate)\n",
        "        self.norm_1 = layers.LayerNormalization(epsilon=1e-6)\n",
        "\n",
        "        # encoder-decoder Multi-head attention\n",
        "        self.multi_head_attention_2 = MultiHeadAttention(self.nb_proj)\n",
        "        self.dropout_2 = layers.Dropout(rate=self.dropout_rate)\n",
        "        self.norm_2 = layers.LayerNormalization(epsilon=1e-6)\n",
        "\n",
        "        # Feed foward\n",
        "        self.dense_1 = layers.Dense(units=self.FFN_units,\n",
        "                                    activation=\"relu\")\n",
        "        self.dense_2 = layers.Dense(units=self.d_model)\n",
        "        self.dropout_3 = layers.Dropout(rate=self.dropout_rate)\n",
        "        self.norm_3 = layers.LayerNormalization(epsilon=1e-6)\n",
        "\n",
        "    def call(self, inputs, enc_outputs, mask_1, mask_2, training):\n",
        "        '''\n",
        "        input       -> Decoder output\n",
        "        enc_outputs -> Encoder output\n",
        "        mask_1      -> Look-ahead mask\n",
        "        mask_2      -> Padding mask\n",
        "        training    -> Boolean for dropout\n",
        "        '''\n",
        "        # Masked Multi-head attention layer\n",
        "        # Return attention, attention weights\n",
        "        attention, attn_wt_1 = self.multi_head_attention_1(inputs,\n",
        "                                                           inputs,\n",
        "                                                           inputs,\n",
        "                                                           mask_1)\n",
        "        attention = self.dropout_1(attention, training)\n",
        "        attention = self.norm_1(attention + inputs)\n",
        "\n",
        "        # Encoder-Decoder Multi-head attention layer\n",
        "        # Return attention, attention weights\n",
        "        attention_2, attn_wt_2 = self.multi_head_attention_2(attention,\n",
        "                                                             enc_outputs,\n",
        "                                                             enc_outputs,\n",
        "                                                             mask_2)\n",
        "        attention_2 = self.dropout_2(attention_2, training)\n",
        "        attention_2 = self.norm_2(attention_2 + attention)\n",
        "\n",
        "        # FNN\n",
        "        outputs = self.dense_1(attention_2)\n",
        "        outputs = self.dense_2(outputs)\n",
        "        outputs = self.dropout_3(outputs, training)\n",
        "        outputs = self.norm_3(outputs + attention_2)\n",
        "\n",
        "        # Return attention vector, attention weights for each attention layer\n",
        "        return outputs, attn_wt_1, attn_wt_2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pbhSCmqk63FE"
      },
      "source": [
        "#### Decoder Class"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IoL-xike63FE"
      },
      "source": [
        "The `full decoder` is a `stack of decoding layers`. This class implements the `decoder stack`. In addition to that, it also implements `embedding` and `positional encoding`. The output is the output of the `encoder stack`.\n",
        "\n",
        "Here is a summary of what the encoder is doing.\n",
        "\n",
        "1. Creates an `embedding layer`\n",
        "2. Initializes the `positional encoding` object\n",
        "3. Creates a `dropout layer`\n",
        "4. Creates number of `decoding layers` specified by `nb_layers`\n",
        "5. Takes a (batch_size, target_seq_length) shape input from decoder layer output, the encoder output, the look-ahead mask, the padding mask, and a boolean\n",
        "6. Passes the input through the `embedding layer` and produces a tensor of shape (batch_size, seq_length, d_model)\n",
        "7. Creates the `positional encoding vectors`\n",
        "8. Adds the `positional encoding vectors` to the `embedding vectors`\n",
        "9. Passes the resulting input tensor, the encoder output, the masks, and the training boolean to the `decoding layer stack`\n",
        "10. Iterates over the `decoding layer stack`\n",
        "11. Returns an output tensor of shape (batch_size, target_seq_length, d_model), and the attention weights\n",
        "\n",
        "In the code you will see this sequence:\n",
        "```Python\n",
        "   outputs = self.embedding(inputs)\n",
        "   outputs *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))\n",
        "   outputs = self.pos_encoding(outputs)\n",
        "```\n",
        "The explaination for the multiplication given [here](https://towardsdatascience.com/how-to-code-the-transformer-in-pytorch-24db27c8f9ec)\n",
        "> The reason we increase the embedding values before addition is to make the positional encoding relatively smaller. This means the original meaning in the embedding vector won’t be lost when we add them together.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Izh3aEmRCFMI"
      },
      "outputs": [],
      "source": [
        "class Decoder(layers.Layer):\n",
        "    '''\n",
        "    nb_layers    -> Number of encoder layers\n",
        "    FFN_units    -> Number of nodes in the FFN\n",
        "    nb_proj      -> Number of attention heads\n",
        "    dropout_rate -> Droput rate\n",
        "    vocab_size   -> The size of the vocabulary\n",
        "    d_model      -> Size of embedding vectors\n",
        "    name         -> Name of the layer\n",
        "    '''\n",
        "    def __init__(self,\n",
        "                 nb_layers,\n",
        "                 FFN_units,\n",
        "                 nb_proj,\n",
        "                 dropout_rate,\n",
        "                 vocab_size,\n",
        "                 d_model,\n",
        "                 name=\"decoder\"):\n",
        "        super(Decoder, self).__init__(name=name)\n",
        "        self.d_model = d_model\n",
        "        self.nb_layers = nb_layers\n",
        "\n",
        "        self.embedding = layers.Embedding(vocab_size, d_model)\n",
        "        self.pos_encoding = PositionalEncoding()\n",
        "        self.dropout = layers.Dropout(rate=dropout_rate)\n",
        "\n",
        "        # Initialize the decoder layers.\n",
        "        self.dec_layers = [DecoderLayer(FFN_units,\n",
        "                                        nb_proj,\n",
        "                                        dropout_rate)\n",
        "                           for i in range(nb_layers)]\n",
        "\n",
        "    def call(self, inputs, enc_outputs, mask_1, mask_2, training):\n",
        "        '''\n",
        "        input       -> Decoder output\n",
        "        enc_outputs -> Encoder output\n",
        "        mask_1      -> Look-ahead mask\n",
        "        mask_2      -> Padding mask\n",
        "        training    -> Boolean for dropout\n",
        "        '''\n",
        "        outputs = self.embedding(inputs)\n",
        "        outputs *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))\n",
        "        outputs = self.pos_encoding(outputs)\n",
        "        outputs = self.dropout(outputs, training)\n",
        "\n",
        "        # Iterate over the decoder layers.\n",
        "        for i in range(self.nb_layers):\n",
        "\n",
        "            attention_weights = {}\n",
        "\n",
        "            # Block 1 and block2 are the attention weights from each attention head of the layer\n",
        "            outputs, block1, block2 = self.dec_layers[i](outputs, enc_outputs, mask_1, mask_2, training)\n",
        "\n",
        "            attention_weights['decoder_layer{}_block1'.format(i+1)] = block1\n",
        "            attention_weights['decoder_layer{}_block2'.format(i+1)] = block2\n",
        "\n",
        "\n",
        "        return outputs, attention_weights"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l-K9kCN563FF"
      },
      "source": [
        "### Transformer\n",
        "Finally the full transformer!\n",
        "\n",
        "The transformer is where all the pieces come together. We'll take a look at what the transformer class does."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dGbKrSvA63FF"
      },
      "source": [
        "#### Transformer Class\n",
        "\n",
        "These are the steps that occur in this class.\n",
        "\n",
        "1. Initialize the `encoder`\n",
        "2. Initialize the `decoder`\n",
        "3. Create the output linear layer with `vocab_size_dec` number of nodes\n",
        "4. Create the `padding` and `look-ahead masks`\n",
        "5. Pass arguments to the encoder and capture the output `attention matrix`\n",
        "6. Pass arguments to the decoder and capture the output `attention matrix`\n",
        "7. Pass the output of the decoder through the final dense layer and return output of shape (batch_size, tar_seq_len, target_vocab_size)\n",
        "8. Return `prediction` (batch_size, tar_seq_len, target_vocab_size), and `attention weight matrix`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_BwKMyhLCK4X"
      },
      "outputs": [],
      "source": [
        "class Transformer(tf.keras.Model):\n",
        "    '''\n",
        "    vocab_size_enc -> Vocabulary size of the encoder input\n",
        "    vocab_size_dec -> Vocabulary size of the decoder input\n",
        "    d_model        -> Embedding size\n",
        "    nb_layers      -> Number of encoder and decoder layers\n",
        "    FFN_units      -> Noumber of nodes in the FNNs\n",
        "    nb_proj        -> Number of heads\n",
        "    dropout_rate   -> Dropout rate throught model\n",
        "    name           -> Layer name\n",
        "    '''\n",
        "    def __init__(self,\n",
        "                 vocab_size_enc,\n",
        "                 vocab_size_dec,\n",
        "                 d_model,\n",
        "                 nb_layers,\n",
        "                 FFN_units,\n",
        "                 nb_proj,\n",
        "                 dropout_rate,\n",
        "                 name=\"transformer\"):\n",
        "        super(Transformer, self).__init__(name=name)\n",
        "\n",
        "        self.encoder = Encoder(nb_layers,\n",
        "                               FFN_units,\n",
        "                               nb_proj,\n",
        "                               dropout_rate,\n",
        "                               vocab_size_enc,\n",
        "                               d_model)\n",
        "\n",
        "        self.decoder = Decoder(nb_layers,\n",
        "                               FFN_units,\n",
        "                               nb_proj,\n",
        "                               dropout_rate,\n",
        "                               vocab_size_dec,\n",
        "                               d_model)\n",
        "\n",
        "        self.last_linear = layers.Dense(units=vocab_size_dec, name=\"lin_ouput\")\n",
        "\n",
        "    def create_padding_mask(self, seq):\n",
        "\n",
        "        mask = tf.cast(tf.math.equal(seq, 0), tf.float32)\n",
        "        return mask[:, tf.newaxis, tf.newaxis, :]\n",
        "\n",
        "    def create_look_ahead_mask(self, seq):\n",
        "\n",
        "        seq_len = tf.shape(seq)[1]\n",
        "        look_ahead_mask = 1 - tf.linalg.band_part(tf.ones((seq_len, seq_len)), -1, 0)\n",
        "        return look_ahead_mask\n",
        "\n",
        "    def call(self, enc_inputs, dec_inputs, training):\n",
        "\n",
        "        enc_mask = self.create_padding_mask(enc_inputs)\n",
        "        dec_mask_1 = tf.maximum(\n",
        "            self.create_padding_mask(dec_inputs),\n",
        "            self.create_look_ahead_mask(dec_inputs)\n",
        "        )\n",
        "        dec_mask_2 = self.create_padding_mask(enc_inputs)\n",
        "\n",
        "        # encoder output\n",
        "        enc_outputs = self.encoder(enc_inputs, enc_mask, training)\n",
        "\n",
        "        # dec_outputs, attention_weights\n",
        "        dec_outputs, attention_weights = self.decoder(dec_inputs,\n",
        "                                                      enc_outputs,\n",
        "                                                      dec_mask_1,\n",
        "                                                      dec_mask_2,\n",
        "                                                      training)\n",
        "\n",
        "        # This is the prediction\n",
        "        outputs = self.last_linear(dec_outputs)\n",
        "\n",
        "        # Return prediction (batch_size, tar_seq_len, target_vocab_size), and attention_weight matrix\n",
        "        return outputs, attention_weights"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u7VuxHM0CZW2"
      },
      "source": [
        "## Training\n",
        "\n",
        "Before we train our model we'll need to do the following.\n",
        "\n",
        "1. Define the hper-parameters.\n",
        "2. Define the loss function\n",
        "3. Create a custom learning rate schedule\n",
        "4. Initialize the Atom Optimizer\n",
        "5. Initialize Checkpoint ant Checkpoint Mangager\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XKG7IIOR63FG"
      },
      "source": [
        "### Define Hyper-parameter values\n",
        "The values that we use a smaller than those proposed in the paper. We do this for computaional an time reasons. The values used in the [paper](https://arxiv.org/pdf/1706.03762.pdf) are commented out for reference."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "r790wV40CYVr"
      },
      "outputs": [],
      "source": [
        "tf.keras.backend.clear_session()\n",
        "\n",
        "# Hyper-parameters\n",
        "D_MODEL = 128 # 512\n",
        "NB_LAYERS = 4 # 6\n",
        "FFN_UNITS = 512 # 2048\n",
        "NB_PROJ = 8 # 8\n",
        "DROPOUT_RATE = 0.1 # 0.1\n",
        "\n",
        "# Initialize the transformer\n",
        "transformer = Transformer(vocab_size_enc=VOCAB_SIZE_EN,\n",
        "                          vocab_size_dec=VOCAB_SIZE_FR,\n",
        "                          d_model=D_MODEL,\n",
        "                          nb_layers=NB_LAYERS,\n",
        "                          FFN_units=FFN_UNITS,\n",
        "                          nb_proj=NB_PROJ,\n",
        "                          dropout_rate=DROPOUT_RATE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CYJAaUMj63FH"
      },
      "source": [
        "### Define the loss function\n",
        "Because the sequences are padded we need to add a mask to the loss function. So, while we are using `sparse categorical cross-entropy` we wrap the loss function in order to apply the mask."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0ZeclFRxCkeQ"
      },
      "outputs": [],
      "source": [
        "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True,\n",
        "                                                            reduction=\"none\")\n",
        "\n",
        "def loss_function(target, pred):\n",
        "    mask = tf.math.logical_not(tf.math.equal(target, 0))\n",
        "    loss_ = loss_object(target, pred)\n",
        "\n",
        "    mask = tf.cast(mask, dtype=loss_.dtype)\n",
        "    loss_ *= mask\n",
        "\n",
        "    return tf.reduce_mean(loss_)\n",
        "\n",
        "train_loss = tf.keras.metrics.Mean(name=\"train_loss\")\n",
        "train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name=\"train_accuracy\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WGHaKbp463FI"
      },
      "source": [
        "### Create a custom learning rate schedule\n",
        "The custom learning rate schedule is specified in the '[Attention is all you need](https://arxiv.org/pdf/1706.03762.pdf)' paper. This is that implementation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8aiiwrnxCqWh"
      },
      "outputs": [],
      "source": [
        "class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):\n",
        "\n",
        "    def __init__(self, d_model, warmup_steps=4000):\n",
        "        super(CustomSchedule, self).__init__()\n",
        "\n",
        "        self.d_model = tf.cast(d_model, tf.float32)\n",
        "        self.warmup_steps = warmup_steps\n",
        "\n",
        "    def __call__(self, step):\n",
        "        arg1 = tf.math.rsqrt(step)\n",
        "        arg2 = step * (self.warmup_steps**-1.5)\n",
        "\n",
        "        return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)\n",
        "\n",
        "leaning_rate = CustomSchedule(D_MODEL)\n",
        "\n",
        "# Use the Atom Optimizer\n",
        "optimizer = tf.keras.optimizers.Adam(leaning_rate,\n",
        "                                     beta_1=0.9,\n",
        "                                     beta_2=0.98,\n",
        "                                     epsilon=1e-9)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AlJATSGV63FJ"
      },
      "source": [
        "#### Plot the learning rate"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "id": "5dQY-W4Fj6J_",
        "outputId": "49de6220-45da-46d5-ff16-801d441e3cb3"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEGCAYAAACtqQjWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxV9Zn48c+Tfd9DWAKEJSxBKWpEca+4oO2UaYsj6m9qW6vTVttOl7H66/wcf/7qTO2mtdV23JdRgVJbsXWjWreqQFxQQJDkghC23ASIJBBCkuf3x/kGLuEmuUnuzb3Jfd6vV14593vO+Z7n3kCenPP9nueIqmKMMcaEQ0K0AzDGGDN8WFIxxhgTNpZUjDHGhI0lFWOMMWFjScUYY0zYJEU7gGgqKirSsrKyaIdhjDFDyttvv12vqsXB1sV1UikrK6OqqiraYRhjzJAiIh93t84ufxljjAkbSyrGGGPCxpKKMcaYsLGkYowxJmwsqRhjjAmbiCYVEZknIhtEpFpEbgiyPlVEFrv1K0SkLGDdja59g4hcGND+gIjUiciabo75fRFRESmKxHsyxhjTvYglFRFJBO4CLgIqgMtEpKLLZlcBe1R1MnA7cJvbtwJYCMwA5gF3u/4AHnJtwY45FrgA2BLWN2OMMSYkkTxTmQ1Uq6pPVVuBRcD8LtvMBx52y0uBuSIirn2Rqh5U1U1AtesPVX0V2N3NMW8HrgeGZT1/VWXJqq00HWyLdijGGBNUJJPKGGBrwOta1xZ0G1VtAxqBwhD3PYqIzAe2qerqXra7RkSqRKTK7/eH8j5ixntb93L9H97nh0vfj3YoxhgT1LAYqBeRDOB/Azf1tq2q3qOqlapaWVwctMpAzNqyez8Ayz/cFeVIjDEmuEgmlW3A2IDXpa4t6DYikgTkAg0h7htoEjABWC0im93274jIyAHEH3Nq/M0AtLZ1sNUlGGOMiSWRTCqrgHIRmSAiKXgD78u6bLMMuNItLwBeUu/5xsuAhW522ASgHFjZ3YFU9QNVHaGqZapahne57ERV3RnetxRdNf4mRLzlZ9fsiG4wxhgTRMSSihsjuQ54HvgQWKKqa0XkFhH5nNvsfqBQRKqB7wE3uH3XAkuAdcBzwLWq2g4gIk8AbwJTRaRWRK6K1HuINT5/M2dPKWbG6ByeXTOs8qUxZpiIaJViVX0GeKZL200Byy3AJd3seytwa5D2y0I4bllfY411HR3KpvomTptUyMllBfzs+Q3saDzAqNz0aIdmjDGHDYuB+niwvfEALYc6mFicybzjvKGi5+xsxRgTYyypDBE+N0g/qTiLScVZTBuZzdOrt0c5KmOMOZollSGixt8EwMTiTADmzxrDO1v28nFDczTDMsaYo1hSGSJ8/may05IozkoFYP6s0YjAn961sxVjTOywpDJE1PibmFichbg5xaPz0jl1QiF/fLcWbxa2McZEnyWVIcLnb2ZSUeZRbZ8/cQybG/bz7ta9UYrKGGOOZkllCGg62MbOT1qYNCLrqPaLjhtJalICf3q3p2IDxhgzeCypDAGb3MyviV3OVLLTkjm/ooSnV2/nYFt7NEIzxpijWFIZAnz13syvrmcqAJdUjmXP/kO8sNaKTBpjos+SyhBQU9dEgsD4woxj1p05uYjS/HQeX2HPJTPGRJ8llSGgpr6Z0vwMUpMSj1mXkCBcNnscb/oa8Ll7WYwxJlosqQwBNXVNTCrO7Hb9JZWlJCUIi1Zt7XYbY4wZDJZUYlxHh7K5oZmJxceOp3QakZ3GedNLWPp2rQ3YG2OiypJKjOssJDmph6QCcPkp49jd3GpFJo0xUWVJJcZ1Pu1xYg+XvwDOmFzEhKJMHvj7ZrvD3hgTNZZUYlzn4HtvZyoJCcJXTi9j9da9vLNlz2CEZowxx7CkEuNq/E1kpyVRlJXS67YLTiolNz2Z+17bNAiRGWPMsSypxDifv/moQpI9yUhJ4rLZ43h+7U627t4/CNEZY8zRLKnEOJ+/ucfpxF1dedp4EkR46I3NkQvKGGO6EdGkIiLzRGSDiFSLyA1B1qeKyGK3foWIlAWsu9G1bxCRCwPaHxCROhFZ06Wvn4nIehF5X0T+KCJ5kXxvg+FwIclexlMCjcpN5+LjR7F41VYa9x+KYHTGGHOsiCUVEUkE7gIuAiqAy0SkostmVwF7VHUycDtwm9u3AlgIzADmAXe7/gAecm1dLQeOU9WZwEfAjWF9Q1Gw6fAjhEM/UwH4xjmTaDrYxoNv2NiKMWZwRfJMZTZQrao+VW0FFgHzu2wzH3jYLS8F5oo3eDAfWKSqB1V1E1Dt+kNVXwV2dz2Yqr6gqm3u5VtAabjf0GA78gjh0M9UAKaPyuG86SU8+PfN7GuxsxVjzOCJZFIZAwTWDal1bUG3cQmhESgMcd+efBV4NtgKEblGRKpEpMrv9/ehy8Hn83dfSLI33547mcYDh3j0rY8jEJkxxgQ37AbqReRHQBvwWLD1qnqPqlaqamVxcfHgBtdHNf5mxhYELyTZm5mleZw9pZj7XtvE/ta23ncwxpgwiGRS2QaMDXhd6tqCbiMiSUAu0BDivscQkS8DnwWu0GFwW3mNv+mYB3P1xbfOnczu5lYee8vK4htjBkckk8oqoFxEJohICt7A+7Iu2ywDrnTLC4CXXDJYBix0s8MmAOXAyp4OJiLzgOuBz6nqkL9Jo6ND2VTf3KeZX11VlhVwxuQifvtKjY2tGGMGRcSSihsjuQ54HvgQWKKqa0XkFhH5nNvsfqBQRKqB7wE3uH3XAkuAdcBzwLWq2g4gIk8AbwJTRaRWRK5yff0GyAaWi8h7IvK7SL23wbBt7wEOtnX0eZC+qx/Om8bu5lbufdUXpsiMMaZ7SZHsXFWfAZ7p0nZTwHILcEk3+94K3Bqk/bJutp88oGBjjK++f9OJuzq+NJfPzBzFfa9v4p/nlFGcnRqO8IwxJqhhN1A/XNTU9W86cTA/uGAqrW0d/PqljQPuyxhjemJJJUb56kMvJNmbCUWZXHryWB5fsYXN7gzIGGMiwZJKjPJqfoVWSDIU35lbTmpSAj/+y4dh6c8YY4KxpBKjavxNvT6Yqy9G5KTxrbnl/PXDXby8oS5s/RpjTCBLKjGo6WAbuz45OKDpxMF85fQyJhRlcsvT62ht6whr38YYA5ZUYtKRpz2G70wFIDUpkZv+oQJffTMPWbFJY0wEWFKJQb7Dz6UP75kKwKenjmDutBH86q8b2dnYEvb+jTHxzZJKDKoZQCHJUNz0DxW0q/J/nlrDMKhmY4yJIZZUYpBvAIUkQzG+MJPvnjeF5et28eyanRE5hjEmPllSiUE1/qawD9J3ddUZEzhuTA43PbXWnhBpjAkbSyoxprOQ5ECqE4ciKTGB2744kz37W7n1mXURPZYxJn5YUokxnYUkJ42I7JkKwIzRuVxz1kSWVNXyN7t3xRgTBpZUYszhRwhH+Eyl03fmljO1JJvrl75PQ9PBQTmmMWb4sqQSYyI5nTiYtORE7lg4i8b9h7jxyQ9sNpgxZkAsqcQYX30TOWEqJBmq6aNyuH7eVF5Yt4slVVsH7bjGmOHHkkqMqalrZmIYC0mG6qunT+C0SYX836fXHb6j3xhj+sqSSozx1Ud+OnEwCQnCL/7pU6QmJfDNx97hQGv7oMdgjBn6LKnEkH0th9j1ycGwVifui1G56dx+6Sw27NrHv//J7rY3xvSdJZUYsilMjxAeiHOmjuBb55bzh3dqWbzKxleMMX0T0aQiIvNEZIOIVIvIDUHWp4rIYrd+hYiUBay70bVvEJELA9ofEJE6EVnTpa8CEVkuIhvd9/xIvrdIqDlcnXjwL38F+s7ccs4sL+KmZWtZs60xqrEYY4aWiCUVEUkE7gIuAiqAy0SkostmVwF7VHUycDtwm9u3AlgIzADmAXe7/gAecm1d3QC8qKrlwIvu9ZDi8zeTIDAuQoUkQ5WYINxx6SyKMlO4+pEq6vZZNWNjTGgieaYyG6hWVZ+qtgKLgPldtpkPPOyWlwJzxZv2NB9YpKoHVXUTUO36Q1VfBXYHOV5gXw8D/xjONzMYfP5mxkWwkGRfFGalcu+Vlezdf4hrHnmblkM2cG+M6V0kk8oYIPCifK1rC7qNqrYBjUBhiPt2VaKqO9zyTqAk2EYico2IVIlIld/vD+V9DBrvEcLRvfQVaMboXO5YOIv3tu7l+qXv28C9MaZXw3KgXr3ffkF/A6rqPapaqaqVxcXFgxxZ99pdIcloDtIHc+GMkVw/byrLVm/n1y9VRzscY0yMi2RS2QaMDXhd6tqCbiMiSUAu0BDivl3tEpFRrq9RwJCqkLjdFZKMpTOVTt84exJfOHEMv1z+EYtXbYl2OMaYGBbJpLIKKBeRCSKSgjfwvqzLNsuAK93yAuAld5axDFjoZodNAMqBlb0cL7CvK4GnwvAeBs1gF5LsCxHhJ1+YyVlTirnxyQ94Ya092MsYE1zEkoobI7kOeB74EFiiqmtF5BYR+Zzb7H6gUESqge/hZmyp6lpgCbAOeA64VlXbAUTkCeBNYKqI1IrIVa6vnwDni8hG4Dz3esjoLCQ5GCXv+yMlKYHfXnEix5fm8a0n3mXlpmBzJYwx8U7iefC1srJSq6qqoh0GAD/64wc8vXo7q//jgkGv+9UXu5tbWfC7N/DvO8jia+ZQMTon2iEZYwaZiLytqpXB1g3LgfqhyOdvZtKIwS8k2VcFmSk8etUpZKUmccV9b/Hhjk+iHZIxJoZYUokRNf4mJhbF5qWvrsbkpfPE1aeSmpTIFfetYMPOfdEOyRgTIyypxIB9LYeo2xe9QpL9UVaUyRPXnEpyonD5vW/x0S5LLMYYSyox4fAgfQxOJ+7JhKJMnrj6VBITvMRil8KMMZZUYoCvvrOQ5NA5U+k0sTiLJ645laSEBC797zd5+2ObFWZMPOs1qYjIFBF5sbMqsIjMFJF/j3xo8cPnbyYxQaJeSLK/JhVnsfQbcyjMSuWK+1bw8oYhdd+pMSaMQjlTuRe4ETgEoKrv493IaMKkxt/E2Pz0mCgk2V+l+Rks+Zc5TCzK4upHqnh69fZoh2SMiYJQkkqGqna9m70tEsHEK5+/eciNpwRTnJ3Kon85lRPG5vPtRe/y36/UWBFKY+JMKEmlXkQm4Qo0isgCYEfPu5hQtXcovvrmITXzqyc5ack8ctVsLj5uFP/17Hr+9x8/4FB7R7TDMsYMkqQQtrkWuAeYJiLbgE3AFRGNKo5s33uA1hgtJNlfacmJ/PqyEygryuCuv9WwdfcB7rriRHLTk6MdmjEmwkI5U1FVPQ8oBqap6hkh7mdCECuPEA63hATh3y6cxs8WzGTFpga++Ns32FTfHO2wjDERFkpy+AOAqjaraucdbksjF1J8qXH3qAyXy19dXVI5lke+egr1TQf53G9e56/rdkU7JGNMBHWbVERkmoh8EcgVkS8EfH0ZSBu0CIc5n7+J3PRkCjNToh1KxMyZVMjT151BWWEmX3ukil+8sIH2DhvAN2Y46mlMZSrwWSAP+IeA9n3A1ZEMKp54jxDOjPlCkgM1tiCD3399Djc9tYZfv1TN6tpGfnXpLPKHcTI1Jh51m1RU9SngKRGZo6pvDmJMccXnb+bM8th5rHEkpSUnctsXZzJrbD43L1vLxXe+xh2XzuKUiYXRDs0YEyahjKm8KyLXisjdIvJA51fEI4sDnYUkJ40YnuMpwYgIl58yjqXfmENqUgKX3fsWv3xhA2027diYYSGUpPIoMBK4EHgF73nxVpI2DDoLSQ6VkvfhNLM0jz9/+0y+cGIpd75UzaX3vMXW3fujHZYxZoBCSSqTVfX/AM2q+jDwGeCUyIYVHzoLSU6OozOVQFmpSfz8kk9x52Un8NHOfVz8q9dYUrXV7sI3ZggLJakcct/3ishxQC4wInIhxY+aOldIsiA+k0qnz31qNM9850wqRudw/dL3+fKDq9jReCDaYRlj+iGUpHKPiOQD/w4sA9YBt0U0qjjhq29iXEEGKUl2L+nYggyeuPpUbpk/g5WbdnPBL19lySo7azFmqOn1t5mq3qeqe1T1VVWdqKojgGdD6VxE5onIBhGpFpEbgqxPFZHFbv0KESkLWHeja98gIhf21qeIzBWRd0TkPRF5XUQmhxJjNNXUNTOxKL7PUgIlJAhfmlPG8/96FjPG5HD9H97nSw+s5OMGuxPfmKGix6QiInNEZIGIjHCvZ4rI48Dfe+tYRBKBu4CLgArgMhGp6LLZVcAeVZ0M3I47A3LbLQRmAPOAu0UksZc+fwtcoaqzgMfxzqxiVnuHsqlh+BSSDKdxhRk8/rVT+X/zZ/Dulr1ccPur3PniRg62tUc7NGNML3q6o/5nwAPAF4G/iMiPgReAFUB5CH3PBqpV1aeqrcAiYH6XbeYDD7vlpcBc8e4CnA8sUtWDqroJqHb99dSnAjluOReI6Qd6dBaSHG41v8IlIUH45zllvPj9szm/ooRfLv+IeXe8xusb66MdmjGmBz3dUf8Z4ARVbXFjKluB41R1c4h9j3H7dKrl2Fljh7dR1TYRaQQKXftbXfYd45a76/NrwDMicgD4BDg1WFAicg1wDcC4ceNCfCvhV+0KSQ6n6sSRUJKTxm8uP5F/qvRz01Nr+F/3r+CzM0dx48XTGZOXHu3wjDFd9HT5q0VVWwBUdQ+wsQ8JJRq+C1ysqqXAg8Avg22kqveoaqWqVhYXR+9O9s57VIbic+mj4awpxTz3r2fx3fOmsHzdLs79+cv8/PkNNB2058UZE0t6OlOZKCLLAl5PCHytqp/rpe9twNiA16WuLdg2tSKShHfZqqGXfY9pF5Fi4FOqusK1Lwae6yW+qKpxhSQLrPZVyNKSE/nOeeUsqCzlZ8+t5zd/q2bRqq384IIpXFI5lsSE4V0/zZihoKek0nX84xd97HsVUC4iE/ASwkLg8i7bLAOuBN4EFgAvqaq65PW4iPwSGI03hrMSkG763INXTXmKqn4EnA982Md4B5UvTgpJRsKYvHTuWHgCXz59Aj/+8zpuePIDHnpjMzdcNI2zpxTbZ2pMFPVUUPKVgXTsxkiuA54HEoEHVHWtiNwCVKnqMuB+4FERqQZ24yUJ3HZL8O6JaQOuVdV2gGB9uvargT+ISAdekvnqQOKPtBp/M2dPiY9CkpEya2wev//6HJ5ds5P/evZDvvzgKk4uy+cHF0y1IpXGRInE881llZWVWlVVNejH3ddyiONvfoHr503lm+fE/O00Q0JrWweLq7by6xc3UrfvIGeWF/GDC6byqbF50Q7NmGFHRN5W1cpg6+xW7ig4MkhvM7/CJSUpgX8+dTyvXv9pfnTxdNZsa2T+XX/n6keqeL92b7TDMyZu9DSmYiLkyHPpbeZXuKUlJ3L1WRO57JRxPPD6Ju59zcfydbs4s7yIaz89mVMmFNiYizER1GtSEZGn8W4sDNQIVAH/3Tnt2ITO57dCkpGWlZrEt+eW85XTy/ift7Zw/+s+Ft7zFieNz+faT0/i01NHWHIxJgJCufzlA5qAe93XJ3jPU5niXps+qvFbIcnBkp2WzDfOmcTrPzyXW+bPYGdjC199qIqLfvUaf3y3ltY2eziYMeEUyuWv01T15IDXT4vIKlU9WUTWRiqw4cznt0KSgy0tOZEvzSnjstnjeOq97fz25Wq+u3g1//nMer506nguP2UchVmp0Q7TmCEvlD+Vs0TkcD0Tt9w5wtwakaiGsc5CkpNG2CB9NCQnJrDgpFKWf/dsHvrKyUwflcMvln/EnJ+8xA+Xvs/6nZ9EO0RjhrRQzlS+D7wuIjV4Nx9OAL4pIpkcKQZpQrRtj1dI0s5UoishQThn6gjOmTqCjbv28eAbm3nynVoWV23ltEmF/K9Tx3N+RQnJiXaJ0pi+6DWpqOozIlIOTHNNGwIG5++IWGTDVI17hLCdqcSO8pJs/vPzx/NvF0zliVVb+J83P+abj71DUVYq/1RZymWzxzG2ICPaYRozJIQ6pfgkoMxt/ykRQVUfiVhUw1hNnatObGcqMSc/M4VvnjOZfzlrEq98VMfjK7bwu1dq+O0rNZxZXszls8cxd/oIO3sxpgehTCl+FJgEvAd0PiVJAUsq/eCrbyYvwwpJxrLEBOHcaSWcO62E7XsPsHjVVhav2srX/+dtirNT+cdZo/nCiaVMH5XTe2fGxJlQzlQqgQqN53ouYVRT18TEIiskOVSMzkvnu+dP4VvnTuZvG/z8vmorD72xmXtf20TFqBy+cOIY5s8aQ3G2zRwzBkJLKmuAkcCOCMcSF3z1VkhyKEpKTOD8ihLOryhhd3MrT6/ezpPv1PLjv3zIfz27nrOnFPOFE8dw3vQS0pITox2uMVETSlIpAtaJyErgYGdjCM9TMV180nII/76DVvNriCvITOHK08q48rQyNu7ax5PvbuOP72zjpfV1ZKYkcl5FCZ85fhRnTy0mNckSjIkvoSSVmyMdRLzoLCQ50Wp+DRvlJdn8cN40fnDBVN7yNfDn97fz7JqdPPXedrJTkzh/RgmfnTmKMyYXWwUFExdCmVI8oOeqmCN8hwtJ2pnKcJOYIJw+uYjTJxdxy/zjeKOmgT+v3s7za3fy5DvbyElL4sIZI7no+JGcNqnILpGZYavbpCIir6vqGSKyj6MLSgqgqmpTX/qoxt/kCknaPQ/DWXJiAmdPKebsKcXc+vnjeb3az59X7+DZNTv5/du1ZKQkcvaUYs6vKOHcaSPIy7CZgGb46OnJj2e479mDF87w5vM3WyHJOJOSlHB4evLBtnberGnghXW7+Ou6XTy7ZieJCcLssoLDkwDsJksz1IX05EcRSQRKCEhCqrolgnENisF+8uMFt7/CuIIM7rvy5N43NsNaR4fy/rZGlq/byQtrd7HR3RQ7bWS2Kx9TzEnj8+1GSxOTenryYyg3P34L+A9gF9BZJ1yBmWGLMA60dyibG/ZzztQR0Q7FxICEBGHW2Dxmjc3j3y6cxub6Zpav28VfP9zFfa/5+N0rNWSlJnH65ELOmTqCs6cUMzovPdphG9OrUGZ/fQeYqqoNfe1cROYBvwISgftU9Sdd1qfi3Zl/EtAAXKqqm926G4Gr8O7i/7aqPt9Tn+LdTfhj4BK3z29V9c6+xhwpnYUk7WmPJpiyokyuPmsiV581kX0th/h7dQOvfOTnlQ11PL92FwBTSrI4Z+oIziovprIs3wb7TUwKJalsxXvSY5+4S2Z3AecDtcAqEVmmqusCNrsK2KOqk0VkIXAbcKmIVAALgRnAaOCvIjLF7dNdn18GxgLTVLVDRGLqlKDzEcITbeaX6UV2WjLzjhvJvONGoqpsrGvi5Q11vPKRnwf/vol7XvWRkpRA5fh8TptUyGmTi5g5Jpcku1RmYkAoScUHvCwif+Homx9/2ct+s4FqVfUBiMgiYD4QmFTmc+Q+mKXAb9wZx3xgkaoeBDaJSLXrjx76/AZwuap2uPjqQnhvg6bGphObfhARppRkM6Ukm2vOmkTzwTbe8jXwRo339fMXPoIXPiIrNYlTJhQwZ1Ihp08uYmpJNgkJVgrIDL5QksoW95XivkI1Bu8sp1MtcEp326hqm4g0AoWu/a0u+45xy931OQnvLOfzgB/vktnGrkGJyDXANQDjxo3rujpiavxWSNIMXGZqEnOnlzB3egkAu5tbebOmgTdq6nmjpoEX13t/SxVmpnDKxAJOLvO+po/KIdGSjBkEPSYVdwlriqpeMUjxDEQq0KKqlSLyBeAB4MyuG6nqPcA94M3+GqzgfP4mK3dvwq4gM4XPzBzFZ2aOAmD73gO8WdPA32vqWeHbzTMf7AQgKzWJE8fnM7ssn5PLCvjU2DwbkzER0WNSUdV2ERkvIimq2tdHB2/DG+PoVOragm1TKyJJQC7egH1P+3bXXgs86Zb/CDzYx3gjylffzDlWSNJE2Oi8dL54UilfPKkU8JLMqs27va9Ne7zLZUBKYgIzS3OpLCtg9oR8Zo3Nt7NoExahjqn8XUSWAc2djSGMqawCykVkAt4v/oXA5V22WQZcCbwJLABeUlV1x3pcRH6JN1BfDqzEu5u/uz7/BHwa2AScDXwUwnsbFJ2FJG2Q3gy20XnpzJ/llecH2Lu/larNe1i1eTcrN+9205e9E/aywgxmjc3jhHH5zBqbx/RROXajrumzUJJKjftKAEK+u96NkVwHPI83/fcBVV0rIrcAVaq6DLgfeNQNxO/GSxK47ZbgDcC3AdeqajtAsD7dIX8CPCYi3wWagK+FGmukdRaStOnEJtryMlI4r6KE8yq8MZkDre2srt3Le1v38t6WvbxR08Cf3tsOeNUAjh+T6xKNd0/NmLx0exaQ6VFId9QPV4N1R/0f3q7l+79fzV+/dzaT7dn0JoapKjsaW3hv617e3bKHd7fs5YNtjRxs8+57Ls5OZeaYXGaMyeV491WSk2qJJs4M9I76YuB6vHtG0jrbVfXcsEU4zPnqrZCkGRpEhNF56YzOS+fi473B/0PtHazfsY93t+7hPZdk/rahjg7392hRVgrHjcnluNG5HDcml+NLcxmdm2aJJk6FcvnrMWAx8Fng63hjIP5IBjXc1NQ1M94KSZohKjkxgeNLvWTxpTle2/7WNj7c8Qkf1DayZvsnrNnWyGsb62l3maYgM4UZo3MOJ5vpo7IZX5hp05rjQChJpVBV7xeR77hnq7wiIqsiHdhw4qtvsgdzmWElIyWJk8YXcNL4gsNtLYfa+XCHl2A+2NbImm2fcO+rPtpcoklLTmBqSTbTRuYwbZT7PjKbfJt1NqyEklQOue87ROQzwHagoIftTYD2DmVz/X4+bYUkzTCXlpzICePyOWFc/uG2lkPtbNzVxPqdn7B+5z7W7/yE5R/uYnHVkXuYR+akHU4y0933icWZVqF5iAolqfxYRHKB7wO/BnKA70Y0qmGkds9+Wts77EzFxKW05MTDl846qSr+poOs3+ElmfU79vHhzn38vdrHoXbvrCY5UZhQlEn5iGwmj8iivCSL8hHZlBVlkJpkN23GslAeJ/xnt9iIdx+I6YMj04lt1pcx4E0GGJGdxojsNM4KuCH4UHsHPn8z63d+woc79lFd18Ta7Y08s2YHnZNUExOE8QUZRyWayZk44U0AABQVSURBVCOymFScRXqKJZtYEMrsrynAb4ESVT1ORGYCn1PVH0c8umHAqhMbE5rkxASmjsxm6shs5s860t5yqB2fv5mNdV6i2biriY11+3hxfd3hiQEiUJqfzuTiLCYUZTGxOJOJRZlMKM5kZI7NRBtMoVz+uhf4N+C/AVT1fRF5HO/ZJaYXVkjSmIFJS06kYnQOFaNzjmpvbevg44ZmNrpE81HdPnz+Zt70NdByqOPwdunJiUxwCWZiUSYTizOZUJTFhKJMctOTB/vtDHuhJJUMVV3ZJdO3RSieYcfnb7JLX8ZEQEpSAuUl2ZSXZMPxR9o7OpSdn7Swqb4ZX30zm/zNbKpvYs22Rp79YMfh+2vAq+bsJZlMxhdmMr4wg3EFGYwvyCQ3wxJOf4SSVOpFZBLeI4QRkQXAjohGNYzU+Jv59FQrJGnMYElIOHID5+mTi45a19rWwZbd+9lU7yUan99LPC+t91PfVHvUtrnpyYeTzLiCDLfsJZ6ROWn2vJpuhJJUrsUrFT9NRLbhFWwcCqXwo67xwCHqmw4yyUqzGBMTUpISmDwiy5VLKjlqXfPBNrbs3s/HDfvZuns/H+9u5uOG/XywrZHn1uw8fL8NeFWeSwvSGV+QwfjCTMa6xDMmL53SgnRy0uL3LCeU2V8+4DwRyQQSVHWfiPwrcEfEoxvifJ2D9PYcFWNiXmZqEtNH5TB9VM4x69raO9jR2MLHDV6y2dLgJZ8tu/ezavMemg4ePSKQnZZEab5LMvlHvsbkZVCan05eRvKwnTwQypkKAKraHPDye1hS6VXndGKb+WXM0JaUmMDYggzGFmRwBkdfUlNVdje3UrvnALV7DrBt737v+54DbN29n7d8DccknYyURJdk0r3kczjppDMmP52izNQhe3kt5KTSxdB8t4Osxt9EUoIwvtAKSRozXIkIhVmpFGal8qmxecesV1UaDxwKSDoHqN2zn21u+Z0te2k8cOiofZIThZKcNEblpjEqN919T2NUXvrhtsLMlJhMPP1NKvFbL78PfP5mxhVkWLkJY+KYiJCXkUJehlfNOZh9LYe8ZLP7ADsaD7C9sYWdjS1s33uA1bV7eW5tC61tHUftk5KYQEluKqNy0hmVl8bI3DRG5x5JOqPy0ijIGPzE021SEZF9BE8eAqRHLKJhxCskaZe+jDE9y05LZtrIZKaNPHY8B45cYtvR2OK+DrB9bws7XQJ6d8tedja20Np+dOJJTvSqF4zMTaMkJ5WSnDRG5qRRkpPGaZMKGZGTFvR4A9FtUlHVkJ/yaI5lhSSNMeESeImtu7Odjg5l9/5Wduz1ks6OxhZ2ftLCLvd9/c59vLLBT3NrOwCPfHX24CYVMzCdhSTtxkdjzGBISBCKslIpyko9qoBnV00H29jZ2MKo3PAnFLCkEjFHan7ZdGJjTOzISk2K6GPNIzqCLCLzRGSDiFSLyA1B1qeKyGK3foWIlAWsu9G1bxCRC/vQ550i0hSp9xQqm05sjIlHEUsqIpII3AVcBFQAl4lIRZfNrgL2qOpk4HbgNrdvBbAQmAHMA+4WkcTe+hSRSiCfGFDjbybfCkkaY+JMJM9UZgPVqupT1VZgETC/yzbzgYfd8lJgrni3mc4HFqnqQVXdBFS7/rrt0yWcnwHXR/A9hazGbzO/jDHxJ5JJZQywNeB1rWsLuo2qtuE9CKywh3176vM6YJmq9ljsUkSuEZEqEany+/19ekN94fM3M8nGU4wxcWZY3JUnIqOBS/Aed9wjVb1HVStVtbK4ODLVgzsLSdqZijEm3kQyqWwDxga8LnVtQbcRkSQgF2joYd/u2k8AJgPVIrIZyBCR6nC9kb6yQpLGmHgVyaSyCigXkQkikoI38L6syzbLgCvd8gLgJVVV177QzQ6bAJQDK7vrU1X/oqojVbVMVcuA/W7wPypqOp9LbyXvjTFxJmL3qahqm4hcBzwPJAIPqOpaEbkFqFLVZcD9wKPurGI3XpLAbbcEWIf3lMlrVbUdIFifkXoP/eVzhSTHFVghSWNMfInozY+q+gzwTJe2mwKWW/DGQoLteytwayh9BtkmqqcIPn8z4wqtkKQxJv7Yb70IqPE3MbHILn0ZY+KPJZUwa2vv4OOG/UwaYYP0xpj4Y0klzGr3HPAKSdqZijEmDllSCTNfvRWSNMbEL0sqYdZZSNJK3htj4pEllTCr8TeRn5FMvhWSNMbEIUsqYVbjb7azFGNM3LKkEmY+f5ONpxhj4pYllTBq3H+I+qZWKyRpjIlbllTCqMbN/LLLX8aYeGVJJYyOPELYLn8ZY+KTJZUwskKSxph4Z0kljGr8TVZI0hgT1+y3Xxj5bDqxMSbOWVIJk7b2DjY3NNt4ijEmrllSCZPaPQc41K5WSNIYE9csqYRJZyFJK3lvjIlnllTCpKbOTSe2MxVjTByzpBImvvomCjJTrJCkMSauRTSpiMg8EdkgItUickOQ9akistitXyEiZQHrbnTtG0Tkwt76FJHHXPsaEXlARJIj+d66qqlrZmKRXfoyxsS3iCUVEUkE7gIuAiqAy0SkostmVwF7VHUycDtwm9u3AlgIzADmAXeLSGIvfT4GTAOOB9KBr0XqvQXjq7dCksYYE8kzldlAtar6VLUVWATM77LNfOBht7wUmCsi4toXqepBVd0EVLv+uu1TVZ9RB1gJlEbwvR2ls5Ck3aNijIl3kUwqY4CtAa9rXVvQbVS1DWgECnvYt9c+3WWvfwaeG/A7CFHN4UcIW1IxxsS34ThQfzfwqqq+FmyliFwjIlUiUuX3+8NywCOPELbLX8aY+BbJpLINGBvwutS1Bd1GRJKAXKChh3177FNE/gMoBr7XXVCqeo+qVqpqZXFxcR/fUnA1rpDkWCskaYyJc5FMKquAchGZICIpeAPvy7psswy40i0vAF5yYyLLgIVudtgEoBxvnKTbPkXka8CFwGWq2hHB93UMn7+J8VZI0hhjSIpUx6raJiLXAc8DicADqrpWRG4BqlR1GXA/8KiIVAO78ZIEbrslwDqgDbhWVdsBgvXpDvk74GPgTW+snydV9ZZIvb9ANf5mG08xxhgimFTAm5EFPNOl7aaA5Rbgkm72vRW4NZQ+XXtE30t32to7+LihmbnTR0Tj8MYYE1Pses0AHS4kaWcqxhhjSWWgavydz6W3mV/GGGNJZYAOP5feCkkaY4wllYGq8VshSWOM6WRJZYB8fiskaYwxnSypDFCNv8kG6Y0xxrGkMgCN+w/R0Nxq1YmNMcaxpDIAnYUk7UzFGGM8llQGoKauszqxnakYYwxYUhkQX30zyYlWSNIYYzpZUhmAmromxhVYIUljjOlkvw0HwFdvhSSNMSaQJZV+6iwkaYP0xhhzhCWVftrqCknaIL0xxhxhSaWffH6bTmyMMV1ZUuknq05sjDHHsqTSTz5/M4WZKeRlWCFJY4zpZEmln2r8TTaeYowxXVhS6SevOrGNpxhjTCBLKv2wd38rDc2tTBphZyrGGBMooklFROaJyAYRqRaRG4KsTxWRxW79ChEpC1h3o2vfICIX9taniExwfVS7PiM22FFjT3s0xpigIpZURCQRuAu4CKgALhORii6bXQXsUdXJwO3AbW7fCmAhMAOYB9wtIom99HkbcLvra4/rOyIOTyceYUnFGGMCRfJMZTZQrao+VW0FFgHzu2wzH3jYLS8F5oqIuPZFqnpQVTcB1a6/oH26fc51feD6/MdIvbEavyskmZ8eqUMYY8yQFMmkMgbYGvC61rUF3UZV24BGoLCHfbtrLwT2uj66OxYAInKNiFSJSJXf7+/H24Kywgw+f8IYkqyQpDHGHCXufiuq6j2qWqmqlcXFxf3qY+Hscfx0wafCHJkxxgx9kUwq24CxAa9LXVvQbUQkCcgFGnrYt7v2BiDP9dHdsYwxxkRYJJPKKqDczcpKwRt4X9Zlm2XAlW55AfCSqqprX+hmh00AyoGV3fXp9vmb6wPX51MRfG/GGGOCSOp9k/5R1TYRuQ54HkgEHlDVtSJyC1ClqsuA+4FHRaQa2I2XJHDbLQHWAW3AtaraDhCsT3fIHwKLROTHwLuub2OMMYNIvD/y41NlZaVWVVVFOwxjjBlSRORtVa0Mti7uBuqNMcZEjiUVY4wxYWNJxRhjTNhYUjHGGBM2cT1QLyJ+4ON+7l4E1IcxnHCxuPrG4uobi6tvYjUuGFhs41U16N3jcZ1UBkJEqrqb/RBNFlffWFx9Y3H1TazGBZGLzS5/GWOMCRtLKsYYY8LGkkr/3RPtALphcfWNxdU3FlffxGpcEKHYbEzFGGNM2NiZijHGmLCxpGKMMSZsLKn0g4jME5ENIlItIjcMwvE2i8gHIvKeiFS5tgIRWS4iG933fNcuInKni+19ETkxoJ8r3fYbReTK7o7XSywPiEidiKwJaAtbLCJyknuv1W5fGUBcN4vINve5vSciFwesu9EdY4OIXBjQHvRn6x63sMK1L3aPXugtprEi8jcRWScia0XkO7HwefUQV1Q/L7dfmoisFJHVLrb/21N/4j0eY7FrXyEiZf2NuZ9xPSQimwI+s1mufTD/7SeKyLsi8udY+KxQVfvqwxdeyf0aYCKQAqwGKiJ8zM1AUZe2nwI3uOUbgNvc8sXAs4AApwIrXHsB4HPf891yfj9iOQs4EVgTiVjwnptzqtvnWeCiAcR1M/CDINtWuJ9bKjDB/TwTe/rZAkuAhW75d8A3QohpFHCiW84GPnLHjurn1UNcUf283LYCZLnlZGCFe39B+wO+CfzOLS8EFvc35n7G9RCwIMj2g/lv/3vA48Cfe/rsB+uzsjOVvpsNVKuqT1VbgUXA/CjEMR942C0/DPxjQPsj6nkL74mYo4ALgeWqultV9wDLgXl9Paiqvor37Juwx+LW5ajqW+r9a38koK/+xNWd+cAiVT2oqpuAaryfa9CfrfuL8VxgaZD32FNMO1T1Hbe8D/gQGEOUP68e4urOoHxeLh5V1Sb3Mtl9aQ/9BX6WS4G57vh9inkAcXVnUH6WIlIKfAa4z73u6bMflM/KkkrfjQG2Bryupef/kOGgwAsi8raIXOPaSlR1h1veCZT0El8k4w5XLGPccjhjvM5dfnhA3GWmfsRVCOxV1bb+xuUuNZyA9xduzHxeXeKCGPi83OWc94A6vF+6NT30dzgGt77RHT/s/w+6xqWqnZ/Zre4zu11EUrvGFeLx+/uzvAO4Huhwr3v67Afls7KkMjScoaonAhcB14rIWYEr3V82MTE3PJZiAX4LTAJmATuAX0QjCBHJAv4A/KuqfhK4LpqfV5C4YuLzUtV2VZ0FlOL9tTwtGnF01TUuETkOuBEvvpPxLmn9cLDiEZHPAnWq+vZgHTMUllT6bhswNuB1qWuLGFXd5r7XAX/E+4+2y50y477X9RJfJOMOVyzb3HJYYlTVXe4XQQdwL97n1p+4GvAuXyR1ae+ViCTj/eJ+TFWfdM1R/7yCxRULn1cgVd0L/A2Y00N/h2Nw63Pd8SP2/yAgrnnuUqKq6kHgQfr/mfXnZ3k68DkR2Yx3aepc4FdE+7PqbdDFvo4ZFEvCG1ybwJHBqxkRPF4mkB2w/AbeWMjPOHqw96du+TMcPUC40rUXAJvwBgfz3XJBP2Mq4+gB8bDFwrGDlRcPIK5RAcvfxbtuDDCDowcmfXiDkt3+bIHfc/Tg5zdDiEfwro3f0aU9qp9XD3FF9fNy2xYDeW45HXgN+Gx3/QHXcvTg85L+xtzPuEYFfKZ3AD+J0r/9czgyUB/dz6o/v1Ti/QtvZsdHeNd6fxThY010P8zVwNrO4+FdC30R2Aj8NeAfpgB3udg+ACoD+voq3iBcNfCVfsbzBN6lkUN411ivCmcsQCWwxu3zG1zVh37G9ag77vvAMo7+pfkjd4wNBMyy6e5n634OK128vwdSQ4jpDLxLW+8D77mvi6P9efUQV1Q/L7ffTOBdF8Ma4Kae+gPS3Otqt35if2PuZ1wvuc9sDfA/HJkhNmj/9t2+53AkqUT1s7IyLcYYY8LGxlSMMcaEjSUVY4wxYWNJxRhjTNhYUjHGGBM2llSMMcaEjSUVY/pIRAoDqtLulKMr+/ZYjVdEKkXkzj4e76uueu37IrJGROa79i+LyOiBvBdjws2mFBszACJyM9Ckqj8PaEvSI7WXBtp/KfAKXlXhRldapVhVN4nIy3hVhavCcSxjwsHOVIwJA/dcjd+JyArgpyIyW0TedM+5eENEprrtzgl47sXNrnDjyyLiE5FvB+l6BLAPaAJQ1SaXUBbg3Sz3mDtDSnfP43jFFR59PqAUzMsi8iu33RoRmR3kOMaEhSUVY8KnFDhNVb8HrAfOVNUTgJuA/+xmn2l45dBnA//hanIFWg3sAjaJyIMi8g8AqroUqAKuUK/IYRvwa7xne5wEPADcGtBPhtvum26dMRGR1PsmxpgQ/V5V291yLvCwiJTjlUTpmiw6/UW9YoQHRaQOrwz+4RLoqtouIvPwquDOBW4XkZNU9eYu/UwFjgOWe4/IIBGvbE2nJ1x/r4pIjojkqVcY0ZiwsqRiTPg0Byz/P+Bvqvp598ySl7vZ52DAcjtB/k+qN/C5ElgpIsvxquHe3GUzAdaq6pxujtN18NQGU01E2OUvYyIjlyNlwr/c305EZLQEPN8c71knH7vlfXiPAwavEGCxiMxx+yWLyIyA/S517WcAjara2N+YjOmJnakYExk/xbv89e/AXwbQTzLwczd1uAXwA1936x4CficiB/CeObIAuFNEcvH+b9+BV9kaoEVE3nX9fXUA8RjTI5tSbMwwZ1OPzWCyy1/GGGPCxs5UjDHGhI2dqRhjjAkbSyrGGGPCxpKKMcaYsLGkYowxJmwsqRhjjAmb/w/8cK+Z2sjKngAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "plt.plot(leaning_rate(tf.range(40000, dtype=tf.float32)))\n",
        "plt.ylabel(\"Learning Rate\")\n",
        "plt.xlabel(\"Train Step\")\n",
        "plt.savefig(ROOT + '/data/lr.png')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WhTXx78Q63FJ"
      },
      "source": [
        "### Model checkpoints\n",
        "We prepare the to save and restore checkpoints."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V-xv-P2_56wh"
      },
      "outputs": [],
      "source": [
        "if os.path.isdir('../ckpt') is False:\n",
        "    os.mkdir('../ckpt')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wQr8BMWNCsMk"
      },
      "outputs": [],
      "source": [
        "checkpoint_path = ROOT + \"/ckpt/\"\n",
        "\n",
        "ckpt = tf.train.Checkpoint(transformer=transformer,\n",
        "                           optimizer=optimizer)\n",
        "\n",
        "ckpt_manager = tf.train.CheckpointManager(ckpt, checkpoint_path, max_to_keep=5)\n",
        "\n",
        "if ckpt_manager.latest_checkpoint:\n",
        "    ckpt.restore(ckpt_manager.latest_checkpoint)\n",
        "    print(\"Latest checkpoint restored!!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qr4K1Wq963FK"
      },
      "source": [
        "### Run the training\n",
        "\n",
        "To train the model we provide as input to the encoder a batch of the source sequences. We also provide to the decoder a batch of target sequences that begin with the `start-of-sentence` token. This has the effect of shifting the sequence to the right.\n",
        "\n",
        "The transformer will go through everything discussed above.\n",
        "1. The encoder will encode the batch of sequences, capturing the important `contextual` and `structural` information. This continues through all the layers of the encoder.\n",
        "2. The decoder will calculate attention at the `masked multi-head self-attention` layer. Because for each word the mask is applied to those words that come after, only the previous words are considered when calcualting attention.\n",
        "3. At the `encoder-decoder attention` layer, the `attention matrix` that is the ouput of the `masked attention` layer is transformed and used for the `query`, and the output (memory) of the encoder is transformed twice and used for the `key` and value.\n",
        "4. At this layer the decoder is learning to map the target words to the encoding.\n",
        "5. This process is repeated for all the layers of the decoder until the final ouput.\n",
        "6. The final output is a list of probabilities for each word in the vocabulary.\n",
        "7. Calculate the `loss function` and `perform back propagation` and adjust the weights.\n",
        "8. Rinse and repeat."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iyD7rZmzCxiJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6180ed15-115f-45c0-9e23-31ab550395b4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Start of epoch 1\n",
            "Epoch 1 Batch 0 Loss 5.4084 Accuracy 0.0000\n",
            "Epoch 1 Batch 50 Loss 5.4664 Accuracy 0.0007\n",
            "Epoch 1 Batch 100 Loss 5.4259 Accuracy 0.0109\n",
            "Epoch 1 Batch 150 Loss 5.3637 Accuracy 0.0159\n",
            "Epoch 1 Batch 200 Loss 5.2770 Accuracy 0.0185\n",
            "Epoch 1 Batch 250 Loss 5.1837 Accuracy 0.0212\n",
            "Epoch 1 Batch 300 Loss 5.0830 Accuracy 0.0246\n",
            "Epoch 1 Batch 350 Loss 4.9687 Accuracy 0.0293\n",
            "Epoch 1 Batch 400 Loss 4.8644 Accuracy 0.0346\n",
            "Epoch 1 Batch 450 Loss 4.7691 Accuracy 0.0398\n",
            "Epoch 1 Batch 500 Loss 4.6766 Accuracy 0.0447\n",
            "Epoch 1 Batch 550 Loss 4.5967 Accuracy 0.0495\n",
            "Epoch 1 Batch 600 Loss 4.5215 Accuracy 0.0542\n",
            "Epoch 1 Batch 650 Loss 4.4468 Accuracy 0.0586\n",
            "Epoch 1 Batch 700 Loss 4.3758 Accuracy 0.0627\n",
            "Epoch 1 Batch 750 Loss 4.3077 Accuracy 0.0666\n",
            "Epoch 1 Batch 800 Loss 4.2431 Accuracy 0.0702\n",
            "Epoch 1 Batch 850 Loss 4.1816 Accuracy 0.0735\n",
            "Epoch 1 Batch 900 Loss 4.1222 Accuracy 0.0767\n",
            "Epoch 1 Batch 950 Loss 4.0657 Accuracy 0.0797\n",
            "Epoch 1 Batch 1000 Loss 4.0138 Accuracy 0.0826\n",
            "Epoch 1 Batch 1050 Loss 3.9632 Accuracy 0.0854\n",
            "Epoch 1 Batch 1100 Loss 3.9150 Accuracy 0.0881\n",
            "Epoch 1 Batch 1150 Loss 3.8694 Accuracy 0.0907\n",
            "Epoch 1 Batch 1200 Loss 3.8269 Accuracy 0.0932\n",
            "Epoch 1 Batch 1250 Loss 3.7858 Accuracy 0.0957\n",
            "Epoch 1 Batch 1300 Loss 3.7471 Accuracy 0.0980\n",
            "Epoch 1 Batch 1350 Loss 3.7114 Accuracy 0.1002\n",
            "Epoch 1 Batch 1400 Loss 3.6761 Accuracy 0.1024\n",
            "Epoch 1 Batch 1450 Loss 3.6423 Accuracy 0.1045\n",
            "Epoch 1 Batch 1500 Loss 3.6089 Accuracy 0.1066\n",
            "Epoch 1 Batch 1550 Loss 3.5783 Accuracy 0.1086\n",
            "Epoch 1 Batch 1600 Loss 3.5485 Accuracy 0.1105\n",
            "Epoch 1 Batch 1650 Loss 3.5196 Accuracy 0.1123\n",
            "Epoch 1 Batch 1700 Loss 3.4932 Accuracy 0.1142\n",
            "Epoch 1 Batch 1750 Loss 3.4659 Accuracy 0.1161\n",
            "Epoch 1 Batch 1800 Loss 3.4407 Accuracy 0.1179\n",
            "Epoch 1 Batch 1850 Loss 3.4152 Accuracy 0.1197\n",
            "Epoch 1 Batch 1900 Loss 3.3912 Accuracy 0.1213\n",
            "Epoch 1 Batch 1950 Loss 3.3676 Accuracy 0.1230\n",
            "Epoch 1 Batch 2000 Loss 3.3455 Accuracy 0.1246\n",
            "Epoch 1 Batch 2050 Loss 3.3245 Accuracy 0.1262\n",
            "Epoch 1 Batch 2100 Loss 3.3036 Accuracy 0.1277\n",
            "Epoch 1 Batch 2150 Loss 3.2832 Accuracy 0.1292\n",
            "Epoch 1 Batch 2200 Loss 3.2630 Accuracy 0.1306\n",
            "Epoch 1 Batch 2250 Loss 3.2444 Accuracy 0.1321\n",
            "Epoch 1 Batch 2300 Loss 3.2257 Accuracy 0.1335\n",
            "Epoch 1 Batch 2350 Loss 3.2085 Accuracy 0.1349\n",
            "Epoch 1 Batch 2400 Loss 3.1917 Accuracy 0.1363\n",
            "Epoch 1 Batch 2450 Loss 3.1750 Accuracy 0.1377\n",
            "Epoch 1 Batch 2500 Loss 3.1587 Accuracy 0.1391\n",
            "Epoch 1 Batch 2550 Loss 3.1429 Accuracy 0.1405\n",
            "Epoch 1 Batch 2600 Loss 3.1266 Accuracy 0.1418\n",
            "Epoch 1 Batch 2650 Loss 3.1110 Accuracy 0.1432\n",
            "Epoch 1 Batch 2700 Loss 3.0958 Accuracy 0.1446\n",
            "Epoch 1 Batch 2750 Loss 3.0806 Accuracy 0.1459\n",
            "Epoch 1 Batch 2800 Loss 3.0655 Accuracy 0.1473\n",
            "Epoch 1 Batch 2850 Loss 3.0508 Accuracy 0.1487\n",
            "Epoch 1 Batch 2900 Loss 3.0364 Accuracy 0.1500\n",
            "Epoch 1 Batch 2950 Loss 3.0215 Accuracy 0.1513\n",
            "Epoch 1 Batch 3000 Loss 3.0071 Accuracy 0.1525\n",
            "Epoch 1 Batch 3050 Loss 2.9930 Accuracy 0.1538\n",
            "Epoch 1 Batch 3100 Loss 2.9796 Accuracy 0.1550\n",
            "Epoch 1 Batch 3150 Loss 2.9663 Accuracy 0.1563\n",
            "Epoch 1 Batch 3200 Loss 2.9523 Accuracy 0.1575\n",
            "Epoch 1 Batch 3250 Loss 2.9393 Accuracy 0.1588\n",
            "Epoch 1 Batch 3300 Loss 2.9264 Accuracy 0.1600\n",
            "Epoch 1 Batch 3350 Loss 2.9132 Accuracy 0.1613\n",
            "Epoch 1 Batch 3400 Loss 2.9003 Accuracy 0.1625\n",
            "Epoch 1 Batch 3450 Loss 2.8877 Accuracy 0.1637\n",
            "Epoch 1 Batch 3500 Loss 2.8752 Accuracy 0.1650\n",
            "Epoch 1 Batch 3550 Loss 2.8630 Accuracy 0.1662\n",
            "Epoch 1 Batch 3600 Loss 2.8500 Accuracy 0.1674\n",
            "Epoch 1 Batch 3650 Loss 2.8373 Accuracy 0.1686\n",
            "Epoch 1 Batch 3700 Loss 2.8248 Accuracy 0.1698\n",
            "Epoch 1 Batch 3750 Loss 2.8125 Accuracy 0.1711\n",
            "Epoch 1 Batch 3800 Loss 2.8004 Accuracy 0.1723\n",
            "Epoch 1 Batch 3850 Loss 2.7884 Accuracy 0.1735\n",
            "Epoch 1 Batch 3900 Loss 2.7766 Accuracy 0.1747\n",
            "Epoch 1 Batch 3950 Loss 2.7650 Accuracy 0.1760\n",
            "Epoch 1 Batch 4000 Loss 2.7539 Accuracy 0.1771\n",
            "Epoch 1 Batch 4050 Loss 2.7427 Accuracy 0.1783\n",
            "Epoch 1 Batch 4100 Loss 2.7318 Accuracy 0.1794\n",
            "Epoch 1 Batch 4150 Loss 2.7209 Accuracy 0.1805\n",
            "Epoch 1 Batch 4200 Loss 2.7097 Accuracy 0.1817\n",
            "Epoch 1 Batch 4250 Loss 2.6989 Accuracy 0.1828\n",
            "Epoch 1 Batch 4300 Loss 2.6882 Accuracy 0.1840\n",
            "Epoch 1 Batch 4350 Loss 2.6774 Accuracy 0.1852\n",
            "Epoch 1 Batch 4400 Loss 2.6669 Accuracy 0.1864\n",
            "Epoch 1 Batch 4450 Loss 2.6565 Accuracy 0.1877\n",
            "Epoch 1 Batch 4500 Loss 2.6459 Accuracy 0.1890\n",
            "Epoch 1 Batch 4550 Loss 2.6358 Accuracy 0.1902\n",
            "Epoch 1 Batch 4600 Loss 2.6252 Accuracy 0.1915\n",
            "Epoch 1 Batch 4650 Loss 2.6146 Accuracy 0.1928\n",
            "Epoch 1 Batch 4700 Loss 2.6041 Accuracy 0.1940\n",
            "Epoch 1 Batch 4750 Loss 2.5937 Accuracy 0.1953\n",
            "Epoch 1 Batch 4800 Loss 2.5834 Accuracy 0.1966\n",
            "Epoch 1 Batch 4850 Loss 2.5730 Accuracy 0.1978\n",
            "Epoch 1 Batch 4900 Loss 2.5629 Accuracy 0.1991\n",
            "Epoch 1 Batch 4950 Loss 2.5531 Accuracy 0.2003\n",
            "Epoch 1 Batch 5000 Loss 2.5434 Accuracy 0.2015\n",
            "Epoch 1 Batch 5050 Loss 2.5335 Accuracy 0.2028\n",
            "Epoch 1 Batch 5100 Loss 2.5238 Accuracy 0.2040\n",
            "Epoch 1 Batch 5150 Loss 2.5141 Accuracy 0.2053\n",
            "Epoch 1 Batch 5200 Loss 2.5047 Accuracy 0.2066\n",
            "Epoch 1 Batch 5250 Loss 2.4950 Accuracy 0.2078\n",
            "Epoch 1 Batch 5300 Loss 2.4860 Accuracy 0.2090\n",
            "Epoch 1 Batch 5350 Loss 2.4769 Accuracy 0.2103\n",
            "Epoch 1 Batch 5400 Loss 2.4674 Accuracy 0.2115\n",
            "Epoch 1 Batch 5450 Loss 2.4584 Accuracy 0.2127\n",
            "Epoch 1 Batch 5500 Loss 2.4494 Accuracy 0.2139\n",
            "Epoch 1 Batch 5550 Loss 2.4404 Accuracy 0.2152\n",
            "Epoch 1 Batch 5600 Loss 2.4316 Accuracy 0.2164\n",
            "Epoch 1 Batch 5650 Loss 2.4231 Accuracy 0.2176\n",
            "Epoch 1 Batch 5700 Loss 2.4143 Accuracy 0.2187\n",
            "Epoch 1 Batch 5750 Loss 2.4057 Accuracy 0.2199\n",
            "Epoch 1 Batch 5800 Loss 2.3971 Accuracy 0.2211\n",
            "Epoch 1 Batch 5850 Loss 2.3889 Accuracy 0.2223\n",
            "Epoch 1 Batch 5900 Loss 2.3807 Accuracy 0.2234\n",
            "Epoch 1 Batch 5950 Loss 2.3728 Accuracy 0.2245\n",
            "Epoch 1 Batch 6000 Loss 2.3649 Accuracy 0.2256\n",
            "Epoch 1 Batch 6050 Loss 2.3571 Accuracy 0.2267\n",
            "Epoch 1 Batch 6100 Loss 2.3493 Accuracy 0.2278\n",
            "Epoch 1 Batch 6150 Loss 2.3414 Accuracy 0.2288\n",
            "Epoch 1 Batch 6200 Loss 2.3338 Accuracy 0.2299\n",
            "Epoch 1 Batch 6250 Loss 2.3261 Accuracy 0.2310\n",
            "Epoch 1 Batch 6300 Loss 2.3185 Accuracy 0.2321\n",
            "Epoch 1 Batch 6350 Loss 2.3110 Accuracy 0.2332\n",
            "Epoch 1 Batch 6400 Loss 2.3036 Accuracy 0.2342\n",
            "Epoch 1 Batch 6450 Loss 2.2960 Accuracy 0.2352\n",
            "Epoch 1 Batch 6500 Loss 2.2888 Accuracy 0.2362\n",
            "Epoch 1 Batch 6550 Loss 2.2816 Accuracy 0.2372\n",
            "Epoch 1 Batch 6600 Loss 2.2746 Accuracy 0.2383\n",
            "Epoch 1 Batch 6650 Loss 2.2673 Accuracy 0.2393\n",
            "Epoch 1 Batch 6700 Loss 2.2601 Accuracy 0.2402\n",
            "Epoch 1 Batch 6750 Loss 2.2531 Accuracy 0.2412\n",
            "Epoch 1 Batch 6800 Loss 2.2461 Accuracy 0.2422\n",
            "Epoch 1 Batch 6850 Loss 2.2392 Accuracy 0.2432\n",
            "Epoch 1 Batch 6900 Loss 2.2322 Accuracy 0.2441\n",
            "Epoch 1 Batch 6950 Loss 2.2255 Accuracy 0.2451\n",
            "Epoch 1 Batch 7000 Loss 2.2188 Accuracy 0.2460\n",
            "Epoch 1 Batch 7050 Loss 2.2122 Accuracy 0.2470\n",
            "Epoch 1 Batch 7100 Loss 2.2057 Accuracy 0.2479\n",
            "Epoch 1 Batch 7150 Loss 2.1993 Accuracy 0.2488\n",
            "Epoch 1 Batch 7200 Loss 2.1929 Accuracy 0.2497\n",
            "Epoch 1 Batch 7250 Loss 2.1866 Accuracy 0.2506\n",
            "Epoch 1 Batch 7300 Loss 2.1805 Accuracy 0.2514\n",
            "Epoch 1 Batch 7350 Loss 2.1744 Accuracy 0.2522\n",
            "Epoch 1 Batch 7400 Loss 2.1684 Accuracy 0.2529\n",
            "Epoch 1 Batch 7450 Loss 2.1624 Accuracy 0.2537\n",
            "Epoch 1 Batch 7500 Loss 2.1563 Accuracy 0.2544\n",
            "Epoch 1 Batch 7550 Loss 2.1503 Accuracy 0.2552\n",
            "Epoch 1 Batch 7600 Loss 2.1442 Accuracy 0.2559\n",
            "Epoch 1 Batch 7650 Loss 2.1381 Accuracy 0.2566\n",
            "Epoch 1 Batch 7700 Loss 2.1322 Accuracy 0.2574\n",
            "Epoch 1 Batch 7750 Loss 2.1264 Accuracy 0.2581\n",
            "Epoch 1 Batch 7800 Loss 2.1205 Accuracy 0.2589\n",
            "Epoch 1 Batch 7850 Loss 2.1146 Accuracy 0.2596\n",
            "Epoch 1 Batch 7900 Loss 2.1087 Accuracy 0.2603\n",
            "Epoch 1 Batch 7950 Loss 2.1029 Accuracy 0.2610\n",
            "Epoch 1 Batch 8000 Loss 2.0970 Accuracy 0.2617\n",
            "Epoch 1 Batch 8050 Loss 2.0912 Accuracy 0.2624\n",
            "Epoch 1 Batch 8100 Loss 2.0854 Accuracy 0.2630\n",
            "Epoch 1 Batch 8150 Loss 2.0797 Accuracy 0.2637\n",
            "Epoch 1 Batch 8200 Loss 2.0741 Accuracy 0.2644\n",
            "Epoch 1 Batch 8250 Loss 2.0685 Accuracy 0.2651\n",
            "Epoch 1 Batch 8300 Loss 2.0631 Accuracy 0.2658\n",
            "Epoch 1 Batch 8350 Loss 2.0577 Accuracy 0.2665\n",
            "Epoch 1 Batch 8400 Loss 2.0523 Accuracy 0.2672\n",
            "Epoch 1 Batch 8450 Loss 2.0470 Accuracy 0.2680\n",
            "Epoch 1 Batch 8500 Loss 2.0414 Accuracy 0.2687\n",
            "Epoch 1 Batch 8550 Loss 2.0362 Accuracy 0.2693\n",
            "Epoch 1 Batch 8600 Loss 2.0311 Accuracy 0.2700\n",
            "Epoch 1 Batch 8650 Loss 2.0258 Accuracy 0.2707\n",
            "Epoch 1 Batch 8700 Loss 2.0207 Accuracy 0.2714\n",
            "Epoch 1 Batch 8750 Loss 2.0155 Accuracy 0.2720\n",
            "Epoch 1 Batch 8800 Loss 2.0105 Accuracy 0.2727\n",
            "Epoch 1 Batch 8850 Loss 2.0052 Accuracy 0.2734\n",
            "Epoch 1 Batch 8900 Loss 2.0000 Accuracy 0.2741\n",
            "Epoch 1 Batch 8950 Loss 1.9950 Accuracy 0.2748\n",
            "Epoch 1 Batch 9000 Loss 1.9900 Accuracy 0.2754\n",
            "Epoch 1 Batch 9050 Loss 1.9850 Accuracy 0.2761\n",
            "Epoch 1 Batch 9100 Loss 1.9799 Accuracy 0.2768\n",
            "Epoch 1 Batch 9150 Loss 1.9752 Accuracy 0.2774\n",
            "Epoch 1 Batch 9200 Loss 1.9704 Accuracy 0.2781\n",
            "Epoch 1 Batch 9250 Loss 1.9655 Accuracy 0.2787\n",
            "Epoch 1 Batch 9300 Loss 1.9609 Accuracy 0.2794\n",
            "Epoch 1 Batch 9350 Loss 1.9562 Accuracy 0.2800\n",
            "Epoch 1 Batch 9400 Loss 1.9516 Accuracy 0.2806\n",
            "Epoch 1 Batch 9450 Loss 1.9471 Accuracy 0.2813\n",
            "Epoch 1 Batch 9500 Loss 1.9424 Accuracy 0.2819\n",
            "Epoch 1 Batch 9550 Loss 1.9379 Accuracy 0.2824\n",
            "Epoch 1 Batch 9600 Loss 1.9336 Accuracy 0.2830\n",
            "Epoch 1 Batch 9650 Loss 1.9293 Accuracy 0.2836\n",
            "Epoch 1 Batch 9700 Loss 1.9251 Accuracy 0.2841\n",
            "Epoch 1 Batch 9750 Loss 1.9208 Accuracy 0.2847\n",
            "Epoch 1 Batch 9800 Loss 1.9166 Accuracy 0.2853\n",
            "Epoch 1 Batch 9850 Loss 1.9126 Accuracy 0.2858\n",
            "Epoch 1 Batch 9900 Loss 1.9084 Accuracy 0.2864\n",
            "Epoch 1 Batch 9950 Loss 1.9043 Accuracy 0.2869\n",
            "Epoch 1 Batch 10000 Loss 1.9003 Accuracy 0.2875\n",
            "Epoch 1 Batch 10050 Loss 1.8963 Accuracy 0.2880\n",
            "Epoch 1 Batch 10100 Loss 1.8923 Accuracy 0.2885\n",
            "Epoch 1 Batch 10150 Loss 1.8884 Accuracy 0.2891\n",
            "Epoch 1 Batch 10200 Loss 1.8846 Accuracy 0.2896\n",
            "Epoch 1 Batch 10250 Loss 1.8807 Accuracy 0.2901\n",
            "Epoch 1 Batch 10300 Loss 1.8767 Accuracy 0.2906\n",
            "Epoch 1 Batch 10350 Loss 1.8726 Accuracy 0.2912\n",
            "Epoch 1 Batch 10400 Loss 1.8688 Accuracy 0.2918\n",
            "Epoch 1 Batch 10450 Loss 1.8649 Accuracy 0.2923\n",
            "Epoch 1 Batch 10500 Loss 1.8612 Accuracy 0.2928\n",
            "Epoch 1 Batch 10550 Loss 1.8574 Accuracy 0.2933\n",
            "Epoch 1 Batch 10600 Loss 1.8537 Accuracy 0.2938\n",
            "Epoch 1 Batch 10650 Loss 1.8499 Accuracy 0.2943\n",
            "Epoch 1 Batch 10700 Loss 1.8462 Accuracy 0.2949\n",
            "Epoch 1 Batch 10750 Loss 1.8426 Accuracy 0.2954\n",
            "Epoch 1 Batch 10800 Loss 1.8390 Accuracy 0.2959\n",
            "Epoch 1 Batch 10850 Loss 1.8354 Accuracy 0.2964\n",
            "Epoch 1 Batch 10900 Loss 1.8317 Accuracy 0.2969\n",
            "Epoch 1 Batch 10950 Loss 1.8282 Accuracy 0.2974\n",
            "Epoch 1 Batch 11000 Loss 1.8246 Accuracy 0.2979\n",
            "Epoch 1 Batch 11050 Loss 1.8210 Accuracy 0.2983\n",
            "Epoch 1 Batch 11100 Loss 1.8176 Accuracy 0.2988\n",
            "Epoch 1 Batch 11150 Loss 1.8141 Accuracy 0.2993\n",
            "Epoch 1 Batch 11200 Loss 1.8107 Accuracy 0.2997\n",
            "Epoch 1 Batch 11250 Loss 1.8072 Accuracy 0.3002\n",
            "Epoch 1 Batch 11300 Loss 1.8037 Accuracy 0.3007\n",
            "Epoch 1 Batch 11350 Loss 1.8002 Accuracy 0.3011\n",
            "Epoch 1 Batch 11400 Loss 1.7968 Accuracy 0.3016\n",
            "Epoch 1 Batch 11450 Loss 1.7933 Accuracy 0.3020\n",
            "Epoch 1 Batch 11500 Loss 1.7899 Accuracy 0.3025\n",
            "Epoch 1 Batch 11550 Loss 1.7864 Accuracy 0.3030\n",
            "Epoch 1 Batch 11600 Loss 1.7830 Accuracy 0.3034\n",
            "Epoch 1 Batch 11650 Loss 1.7796 Accuracy 0.3039\n",
            "Epoch 1 Batch 11700 Loss 1.7763 Accuracy 0.3043\n",
            "Epoch 1 Batch 11750 Loss 1.7729 Accuracy 0.3048\n",
            "Epoch 1 Batch 11800 Loss 1.7696 Accuracy 0.3052\n",
            "Epoch 1 Batch 11850 Loss 1.7663 Accuracy 0.3057\n",
            "Epoch 1 Batch 11900 Loss 1.7630 Accuracy 0.3061\n",
            "Epoch 1 Batch 11950 Loss 1.7598 Accuracy 0.3065\n",
            "Epoch 1 Batch 12000 Loss 1.7565 Accuracy 0.3070\n",
            "Epoch 1 Batch 12050 Loss 1.7533 Accuracy 0.3074\n",
            "Epoch 1 Batch 12100 Loss 1.7501 Accuracy 0.3079\n",
            "Epoch 1 Batch 12150 Loss 1.7470 Accuracy 0.3083\n",
            "Epoch 1 Batch 12200 Loss 1.7439 Accuracy 0.3088\n",
            "Epoch 1 Batch 12250 Loss 1.7408 Accuracy 0.3092\n",
            "Epoch 1 Batch 12300 Loss 1.7377 Accuracy 0.3097\n",
            "Epoch 1 Batch 12350 Loss 1.7346 Accuracy 0.3101\n",
            "Epoch 1 Batch 12400 Loss 1.7316 Accuracy 0.3105\n",
            "Epoch 1 Batch 12450 Loss 1.7286 Accuracy 0.3110\n",
            "Epoch 1 Batch 12500 Loss 1.7256 Accuracy 0.3114\n",
            "Epoch 1 Batch 12550 Loss 1.7227 Accuracy 0.3118\n",
            "Epoch 1 Batch 12600 Loss 1.7197 Accuracy 0.3123\n",
            "Epoch 1 Batch 12650 Loss 1.7168 Accuracy 0.3127\n",
            "Epoch 1 Batch 12700 Loss 1.7138 Accuracy 0.3132\n",
            "Epoch 1 Batch 12750 Loss 1.7110 Accuracy 0.3136\n",
            "Epoch 1 Batch 12800 Loss 1.7080 Accuracy 0.3140\n",
            "Epoch 1 Batch 12850 Loss 1.7051 Accuracy 0.3144\n",
            "Epoch 1 Batch 12900 Loss 1.7023 Accuracy 0.3148\n",
            "Epoch 1 Batch 12950 Loss 1.6995 Accuracy 0.3152\n",
            "Epoch 1 Batch 13000 Loss 1.6968 Accuracy 0.3157\n",
            "Epoch 1 Batch 13050 Loss 1.6940 Accuracy 0.3161\n",
            "Epoch 1 Batch 13100 Loss 1.6913 Accuracy 0.3165\n",
            "Epoch 1 Batch 13150 Loss 1.6886 Accuracy 0.3169\n",
            "Epoch 1 Batch 13200 Loss 1.6859 Accuracy 0.3173\n",
            "Epoch 1 Batch 13250 Loss 1.6833 Accuracy 0.3177\n",
            "Epoch 1 Batch 13300 Loss 1.6806 Accuracy 0.3181\n",
            "Epoch 1 Batch 13350 Loss 1.6780 Accuracy 0.3186\n",
            "Epoch 1 Batch 13400 Loss 1.6753 Accuracy 0.3189\n",
            "Epoch 1 Batch 13450 Loss 1.6727 Accuracy 0.3193\n",
            "Epoch 1 Batch 13500 Loss 1.6700 Accuracy 0.3197\n",
            "Epoch 1 Batch 13550 Loss 1.6675 Accuracy 0.3201\n",
            "Epoch 1 Batch 13600 Loss 1.6649 Accuracy 0.3205\n",
            "Epoch 1 Batch 13650 Loss 1.6623 Accuracy 0.3209\n",
            "Epoch 1 Batch 13700 Loss 1.6598 Accuracy 0.3213\n",
            "Epoch 1 Batch 13750 Loss 1.6573 Accuracy 0.3217\n",
            "Epoch 1 Batch 13800 Loss 1.6548 Accuracy 0.3221\n",
            "Epoch 1 Batch 13850 Loss 1.6522 Accuracy 0.3225\n",
            "Epoch 1 Batch 13900 Loss 1.6498 Accuracy 0.3229\n",
            "Epoch 1 Batch 13950 Loss 1.6473 Accuracy 0.3233\n",
            "Epoch 1 Batch 14000 Loss 1.6448 Accuracy 0.3237\n",
            "Epoch 1 Batch 14050 Loss 1.6424 Accuracy 0.3240\n",
            "Epoch 1 Batch 14100 Loss 1.6402 Accuracy 0.3244\n",
            "Epoch 1 Batch 14150 Loss 1.6382 Accuracy 0.3247\n",
            "Epoch 1 Batch 14200 Loss 1.6363 Accuracy 0.3249\n",
            "Epoch 1 Batch 14250 Loss 1.6346 Accuracy 0.3252\n",
            "Epoch 1 Batch 14300 Loss 1.6331 Accuracy 0.3254\n",
            "Epoch 1 Batch 14350 Loss 1.6316 Accuracy 0.3256\n",
            "Epoch 1 Batch 14400 Loss 1.6301 Accuracy 0.3258\n",
            "Epoch 1 Batch 14450 Loss 1.6287 Accuracy 0.3259\n",
            "Epoch 1 Batch 14500 Loss 1.6275 Accuracy 0.3261\n",
            "Epoch 1 Batch 14550 Loss 1.6262 Accuracy 0.3263\n",
            "Epoch 1 Batch 14600 Loss 1.6250 Accuracy 0.3264\n",
            "Epoch 1 Batch 14650 Loss 1.6237 Accuracy 0.3266\n",
            "Epoch 1 Batch 14700 Loss 1.6225 Accuracy 0.3267\n",
            "Epoch 1 Batch 14750 Loss 1.6212 Accuracy 0.3268\n",
            "Epoch 1 Batch 14800 Loss 1.6201 Accuracy 0.3270\n",
            "Epoch 1 Batch 14850 Loss 1.6190 Accuracy 0.3271\n",
            "Epoch 1 Batch 14900 Loss 1.6178 Accuracy 0.3273\n",
            "Epoch 1 Batch 14950 Loss 1.6167 Accuracy 0.3274\n",
            "Epoch 1 Batch 15000 Loss 1.6156 Accuracy 0.3275\n",
            "Epoch 1 Batch 15050 Loss 1.6145 Accuracy 0.3276\n",
            "Epoch 1 Batch 15100 Loss 1.6133 Accuracy 0.3277\n",
            "Epoch 1 Batch 15150 Loss 1.6122 Accuracy 0.3279\n",
            "Epoch 1 Batch 15200 Loss 1.6111 Accuracy 0.3280\n",
            "Epoch 1 Batch 15250 Loss 1.6099 Accuracy 0.3281\n",
            "Epoch 1 Batch 15300 Loss 1.6087 Accuracy 0.3282\n",
            "Epoch 1 Batch 15350 Loss 1.6076 Accuracy 0.3283\n",
            "Epoch 1 Batch 15400 Loss 1.6064 Accuracy 0.3284\n",
            "Epoch 1 Batch 15450 Loss 1.6053 Accuracy 0.3285\n",
            "Epoch 1 Batch 15500 Loss 1.6041 Accuracy 0.3287\n",
            "Epoch 1 Batch 15550 Loss 1.6030 Accuracy 0.3288\n",
            "Epoch 1 Batch 15600 Loss 1.6017 Accuracy 0.3289\n",
            "Epoch 1 Batch 15650 Loss 1.6005 Accuracy 0.3291\n",
            "Epoch 1 Batch 15700 Loss 1.5992 Accuracy 0.3292\n",
            "Epoch 1 Batch 15750 Loss 1.5980 Accuracy 0.3293\n",
            "Epoch 1 Batch 15800 Loss 1.5968 Accuracy 0.3294\n",
            "Epoch 1 Batch 15850 Loss 1.5956 Accuracy 0.3296\n",
            "Epoch 1 Batch 15900 Loss 1.5944 Accuracy 0.3297\n",
            "Epoch 1 Batch 15950 Loss 1.5932 Accuracy 0.3298\n",
            "Epoch 1 Batch 16000 Loss 1.5920 Accuracy 0.3299\n",
            "Epoch 1 Batch 16050 Loss 1.5907 Accuracy 0.3301\n",
            "Epoch 1 Batch 16100 Loss 1.5895 Accuracy 0.3302\n",
            "Epoch 1 Batch 16150 Loss 1.5883 Accuracy 0.3303\n",
            "Epoch 1 Batch 16200 Loss 1.5871 Accuracy 0.3305\n",
            "Epoch 1 Batch 16250 Loss 1.5859 Accuracy 0.3306\n",
            "Epoch 1 Batch 16300 Loss 1.5846 Accuracy 0.3307\n",
            "Epoch 1 Batch 16350 Loss 1.5833 Accuracy 0.3308\n",
            "Epoch 1 Batch 16400 Loss 1.5820 Accuracy 0.3310\n",
            "Epoch 1 Batch 16450 Loss 1.5807 Accuracy 0.3311\n",
            "Epoch 1 Batch 16500 Loss 1.5795 Accuracy 0.3313\n",
            "Epoch 1 Batch 16550 Loss 1.5783 Accuracy 0.3314\n",
            "Epoch 1 Batch 16600 Loss 1.5771 Accuracy 0.3315\n",
            "Epoch 1 Batch 16650 Loss 1.5758 Accuracy 0.3316\n",
            "Epoch 1 Batch 16700 Loss 1.5745 Accuracy 0.3318\n",
            "Epoch 1 Batch 16750 Loss 1.5734 Accuracy 0.3319\n",
            "Epoch 1 Batch 16800 Loss 1.5722 Accuracy 0.3320\n",
            "Epoch 1 Batch 16850 Loss 1.5711 Accuracy 0.3321\n",
            "Epoch 1 Batch 16900 Loss 1.5699 Accuracy 0.3322\n",
            "Epoch 1 Batch 16950 Loss 1.5688 Accuracy 0.3323\n",
            "Epoch 1 Batch 17000 Loss 1.5678 Accuracy 0.3324\n",
            "Epoch 1 Batch 17050 Loss 1.5668 Accuracy 0.3325\n",
            "Epoch 1 Batch 17100 Loss 1.5657 Accuracy 0.3326\n",
            "Epoch 1 Batch 17150 Loss 1.5648 Accuracy 0.3328\n",
            "Epoch 1 Batch 17200 Loss 1.5638 Accuracy 0.3328\n",
            "Epoch 1 Batch 17250 Loss 1.5628 Accuracy 0.3329\n",
            "Epoch 1 Batch 17300 Loss 1.5617 Accuracy 0.3330\n",
            "Epoch 1 Batch 17350 Loss 1.5607 Accuracy 0.3331\n",
            "Epoch 1 Batch 17400 Loss 1.5597 Accuracy 0.3332\n",
            "Epoch 1 Batch 17450 Loss 1.5587 Accuracy 0.3333\n",
            "Epoch 1 Batch 17500 Loss 1.5577 Accuracy 0.3334\n",
            "Epoch 1 Batch 17550 Loss 1.5567 Accuracy 0.3335\n",
            "Epoch 1 Batch 17600 Loss 1.5557 Accuracy 0.3336\n",
            "Epoch 1 Batch 17650 Loss 1.5548 Accuracy 0.3336\n",
            "Epoch 1 Batch 17700 Loss 1.5539 Accuracy 0.3337\n",
            "Epoch 1 Batch 17750 Loss 1.5529 Accuracy 0.3338\n",
            "Epoch 1 Batch 17800 Loss 1.5519 Accuracy 0.3339\n",
            "Epoch 1 Batch 17850 Loss 1.5509 Accuracy 0.3340\n",
            "Epoch 1 Batch 17900 Loss 1.5499 Accuracy 0.3341\n",
            "Epoch 1 Batch 17950 Loss 1.5489 Accuracy 0.3341\n",
            "Epoch 1 Batch 18000 Loss 1.5479 Accuracy 0.3342\n",
            "Epoch 1 Batch 18050 Loss 1.5469 Accuracy 0.3343\n",
            "Epoch 1 Batch 18100 Loss 1.5459 Accuracy 0.3344\n",
            "Epoch 1 Batch 18150 Loss 1.5449 Accuracy 0.3344\n",
            "Epoch 1 Batch 18200 Loss 1.5439 Accuracy 0.3345\n",
            "Epoch 1 Batch 18250 Loss 1.5429 Accuracy 0.3346\n",
            "Epoch 1 Batch 18300 Loss 1.5419 Accuracy 0.3347\n",
            "Epoch 1 Batch 18350 Loss 1.5408 Accuracy 0.3348\n",
            "Epoch 1 Batch 18400 Loss 1.5397 Accuracy 0.3348\n",
            "Epoch 1 Batch 18450 Loss 1.5386 Accuracy 0.3349\n",
            "Epoch 1 Batch 18500 Loss 1.5376 Accuracy 0.3350\n",
            "Epoch 1 Batch 18550 Loss 1.5366 Accuracy 0.3351\n",
            "Epoch 1 Batch 18600 Loss 1.5356 Accuracy 0.3352\n",
            "Epoch 1 Batch 18650 Loss 1.5345 Accuracy 0.3352\n",
            "Saving checkpoint for epoch 1 at /content/drive/My Drive/ckpt/ckpt-1\n",
            "Time taken for 1 epoch: 6011.645282030106 secs\n",
            "\n",
            "Start of epoch 2\n",
            "Epoch 2 Batch 0 Loss 1.3172 Accuracy 0.3650\n",
            "Epoch 2 Batch 50 Loss 1.1774 Accuracy 0.3672\n",
            "Epoch 2 Batch 100 Loss 1.1647 Accuracy 0.3684\n",
            "Epoch 2 Batch 150 Loss 1.1526 Accuracy 0.3685\n",
            "Epoch 2 Batch 200 Loss 1.1483 Accuracy 0.3686\n",
            "Epoch 2 Batch 250 Loss 1.1477 Accuracy 0.3685\n",
            "Epoch 2 Batch 300 Loss 1.1466 Accuracy 0.3689\n",
            "Epoch 2 Batch 350 Loss 1.1457 Accuracy 0.3695\n",
            "Epoch 2 Batch 400 Loss 1.1476 Accuracy 0.3704\n",
            "Epoch 2 Batch 450 Loss 1.1487 Accuracy 0.3707\n",
            "Epoch 2 Batch 500 Loss 1.1442 Accuracy 0.3704\n",
            "Epoch 2 Batch 550 Loss 1.1446 Accuracy 0.3708\n",
            "Epoch 2 Batch 600 Loss 1.1458 Accuracy 0.3708\n",
            "Epoch 2 Batch 650 Loss 1.1463 Accuracy 0.3712\n",
            "Epoch 2 Batch 700 Loss 1.1463 Accuracy 0.3713\n",
            "Epoch 2 Batch 750 Loss 1.1471 Accuracy 0.3715\n",
            "Epoch 2 Batch 800 Loss 1.1481 Accuracy 0.3715\n",
            "Epoch 2 Batch 850 Loss 1.1486 Accuracy 0.3715\n",
            "Epoch 2 Batch 900 Loss 1.1487 Accuracy 0.3713\n",
            "Epoch 2 Batch 950 Loss 1.1479 Accuracy 0.3711\n",
            "Epoch 2 Batch 1000 Loss 1.1468 Accuracy 0.3710\n",
            "Epoch 2 Batch 1050 Loss 1.1463 Accuracy 0.3715\n",
            "Epoch 2 Batch 1100 Loss 1.1465 Accuracy 0.3718\n",
            "Epoch 2 Batch 1150 Loss 1.1462 Accuracy 0.3718\n",
            "Epoch 2 Batch 1200 Loss 1.1461 Accuracy 0.3718\n",
            "Epoch 2 Batch 1250 Loss 1.1457 Accuracy 0.3720\n",
            "Epoch 2 Batch 1300 Loss 1.1450 Accuracy 0.3718\n",
            "Epoch 2 Batch 1350 Loss 1.1443 Accuracy 0.3718\n",
            "Epoch 2 Batch 1400 Loss 1.1438 Accuracy 0.3717\n",
            "Epoch 2 Batch 1450 Loss 1.1436 Accuracy 0.3719\n",
            "Epoch 2 Batch 1500 Loss 1.1431 Accuracy 0.3720\n",
            "Epoch 2 Batch 1550 Loss 1.1429 Accuracy 0.3721\n",
            "Epoch 2 Batch 1600 Loss 1.1423 Accuracy 0.3722\n",
            "Epoch 2 Batch 1650 Loss 1.1418 Accuracy 0.3722\n",
            "Epoch 2 Batch 1700 Loss 1.1413 Accuracy 0.3723\n",
            "Epoch 2 Batch 1750 Loss 1.1408 Accuracy 0.3726\n",
            "Epoch 2 Batch 1800 Loss 1.1399 Accuracy 0.3727\n",
            "Epoch 2 Batch 1850 Loss 1.1390 Accuracy 0.3730\n",
            "Epoch 2 Batch 1900 Loss 1.1381 Accuracy 0.3732\n",
            "Epoch 2 Batch 1950 Loss 1.1370 Accuracy 0.3733\n",
            "Epoch 2 Batch 2000 Loss 1.1360 Accuracy 0.3736\n",
            "Epoch 2 Batch 2050 Loss 1.1355 Accuracy 0.3737\n",
            "Epoch 2 Batch 2100 Loss 1.1348 Accuracy 0.3737\n",
            "Epoch 2 Batch 2150 Loss 1.1344 Accuracy 0.3739\n",
            "Epoch 2 Batch 2200 Loss 1.1346 Accuracy 0.3741\n",
            "Epoch 2 Batch 2250 Loss 1.1346 Accuracy 0.3743\n",
            "Epoch 2 Batch 2300 Loss 1.1341 Accuracy 0.3744\n",
            "Epoch 2 Batch 2350 Loss 1.1338 Accuracy 0.3746\n",
            "Epoch 2 Batch 2400 Loss 1.1338 Accuracy 0.3749\n",
            "Epoch 2 Batch 2450 Loss 1.1332 Accuracy 0.3751\n",
            "Epoch 2 Batch 2500 Loss 1.1329 Accuracy 0.3754\n",
            "Epoch 2 Batch 2550 Loss 1.1328 Accuracy 0.3757\n",
            "Epoch 2 Batch 2600 Loss 1.1329 Accuracy 0.3761\n",
            "Epoch 2 Batch 2650 Loss 1.1329 Accuracy 0.3763\n",
            "Epoch 2 Batch 2700 Loss 1.1322 Accuracy 0.3766\n",
            "Epoch 2 Batch 2750 Loss 1.1315 Accuracy 0.3769\n",
            "Epoch 2 Batch 2800 Loss 1.1305 Accuracy 0.3773\n",
            "Epoch 2 Batch 2850 Loss 1.1300 Accuracy 0.3777\n",
            "Epoch 2 Batch 2900 Loss 1.1291 Accuracy 0.3780\n",
            "Epoch 2 Batch 2950 Loss 1.1282 Accuracy 0.3784\n",
            "Epoch 2 Batch 3000 Loss 1.1275 Accuracy 0.3787\n",
            "Epoch 2 Batch 3050 Loss 1.1268 Accuracy 0.3789\n",
            "Epoch 2 Batch 3100 Loss 1.1264 Accuracy 0.3791\n",
            "Epoch 2 Batch 3150 Loss 1.1257 Accuracy 0.3793\n",
            "Epoch 2 Batch 3200 Loss 1.1251 Accuracy 0.3795\n",
            "Epoch 2 Batch 3250 Loss 1.1244 Accuracy 0.3798\n",
            "Epoch 2 Batch 3300 Loss 1.1237 Accuracy 0.3800\n",
            "Epoch 2 Batch 3350 Loss 1.1229 Accuracy 0.3802\n",
            "Epoch 2 Batch 3400 Loss 1.1219 Accuracy 0.3803\n",
            "Epoch 2 Batch 3450 Loss 1.1209 Accuracy 0.3805\n",
            "Epoch 2 Batch 3500 Loss 1.1199 Accuracy 0.3807\n",
            "Epoch 2 Batch 3550 Loss 1.1190 Accuracy 0.3808\n",
            "Epoch 2 Batch 3600 Loss 1.1180 Accuracy 0.3811\n",
            "Epoch 2 Batch 3650 Loss 1.1171 Accuracy 0.3813\n",
            "Epoch 2 Batch 3700 Loss 1.1156 Accuracy 0.3816\n",
            "Epoch 2 Batch 3750 Loss 1.1145 Accuracy 0.3818\n",
            "Epoch 2 Batch 3800 Loss 1.1132 Accuracy 0.3821\n",
            "Epoch 2 Batch 3850 Loss 1.1120 Accuracy 0.3823\n",
            "Epoch 2 Batch 3900 Loss 1.1110 Accuracy 0.3826\n",
            "Epoch 2 Batch 3950 Loss 1.1100 Accuracy 0.3828\n",
            "Epoch 2 Batch 4000 Loss 1.1089 Accuracy 0.3830\n",
            "Epoch 2 Batch 4050 Loss 1.1078 Accuracy 0.3832\n",
            "Epoch 2 Batch 4100 Loss 1.1063 Accuracy 0.3834\n",
            "Epoch 2 Batch 4150 Loss 1.1050 Accuracy 0.3836\n",
            "Epoch 2 Batch 4200 Loss 1.1036 Accuracy 0.3839\n",
            "Epoch 2 Batch 4250 Loss 1.1023 Accuracy 0.3841\n",
            "Epoch 2 Batch 4300 Loss 1.1010 Accuracy 0.3845\n",
            "Epoch 2 Batch 4350 Loss 1.0998 Accuracy 0.3848\n",
            "Epoch 2 Batch 4400 Loss 1.0984 Accuracy 0.3852\n",
            "Epoch 2 Batch 4450 Loss 1.0970 Accuracy 0.3855\n",
            "Epoch 2 Batch 4500 Loss 1.0957 Accuracy 0.3859\n",
            "Epoch 2 Batch 4550 Loss 1.0943 Accuracy 0.3862\n",
            "Epoch 2 Batch 4600 Loss 1.0929 Accuracy 0.3867\n",
            "Epoch 2 Batch 4650 Loss 1.0913 Accuracy 0.3870\n",
            "Epoch 2 Batch 4700 Loss 1.0898 Accuracy 0.3874\n",
            "Epoch 2 Batch 4750 Loss 1.0882 Accuracy 0.3878\n",
            "Epoch 2 Batch 4800 Loss 1.0865 Accuracy 0.3883\n",
            "Epoch 2 Batch 4850 Loss 1.0849 Accuracy 0.3887\n",
            "Epoch 2 Batch 4900 Loss 1.0837 Accuracy 0.3891\n",
            "Epoch 2 Batch 4950 Loss 1.0824 Accuracy 0.3895\n",
            "Epoch 2 Batch 5000 Loss 1.0810 Accuracy 0.3898\n",
            "Epoch 2 Batch 5050 Loss 1.0794 Accuracy 0.3902\n",
            "Epoch 2 Batch 5100 Loss 1.0778 Accuracy 0.3907\n",
            "Epoch 2 Batch 5150 Loss 1.0763 Accuracy 0.3911\n",
            "Epoch 2 Batch 5200 Loss 1.0749 Accuracy 0.3915\n",
            "Epoch 2 Batch 5250 Loss 1.0735 Accuracy 0.3919\n",
            "Epoch 2 Batch 5300 Loss 1.0722 Accuracy 0.3922\n",
            "Epoch 2 Batch 5350 Loss 1.0708 Accuracy 0.3927\n",
            "Epoch 2 Batch 5400 Loss 1.0695 Accuracy 0.3931\n",
            "Epoch 2 Batch 5450 Loss 1.0681 Accuracy 0.3935\n",
            "Epoch 2 Batch 5500 Loss 1.0666 Accuracy 0.3939\n",
            "Epoch 2 Batch 5550 Loss 1.0654 Accuracy 0.3943\n",
            "Epoch 2 Batch 5600 Loss 1.0641 Accuracy 0.3948\n",
            "Epoch 2 Batch 5650 Loss 1.0628 Accuracy 0.3952\n",
            "Epoch 2 Batch 5700 Loss 1.0615 Accuracy 0.3956\n",
            "Epoch 2 Batch 5750 Loss 1.0601 Accuracy 0.3960\n",
            "Epoch 2 Batch 5800 Loss 1.0589 Accuracy 0.3964\n",
            "Epoch 2 Batch 5850 Loss 1.0578 Accuracy 0.3967\n",
            "Epoch 2 Batch 5900 Loss 1.0566 Accuracy 0.3971\n",
            "Epoch 2 Batch 5950 Loss 1.0554 Accuracy 0.3975\n",
            "Epoch 2 Batch 6000 Loss 1.0543 Accuracy 0.3978\n",
            "Epoch 2 Batch 6050 Loss 1.0532 Accuracy 0.3981\n",
            "Epoch 2 Batch 6100 Loss 1.0521 Accuracy 0.3985\n",
            "Epoch 2 Batch 6150 Loss 1.0510 Accuracy 0.3988\n"
          ]
        }
      ],
      "source": [
        "EPOCHS = 10\n",
        "for epoch in range(EPOCHS):\n",
        "    print(\"Start of epoch {}\".format(epoch+1))\n",
        "    start = time.time()\n",
        "\n",
        "    train_loss.reset_states()\n",
        "    train_accuracy.reset_states()\n",
        "\n",
        "    for (batch, (enc_inputs, targets)) in enumerate(dataset):\n",
        "\n",
        "        # Include the start token which shifts sequence to the right\n",
        "        dec_inputs = targets[:, :-1]\n",
        "\n",
        "        # Target without the start token. The end token is included to know when the\n",
        "        # model reaches the end of the sequence\n",
        "        dec_outputs_real = targets[:, 1:]\n",
        "\n",
        "        with tf.GradientTape() as tape:\n",
        "            predictions, _ = transformer(enc_inputs, dec_inputs, True)\n",
        "            loss = loss_function(dec_outputs_real, predictions)\n",
        "\n",
        "        # Calculate and apply the gradients\n",
        "        gradients = tape.gradient(loss, transformer.trainable_variables)\n",
        "        optimizer.apply_gradients(zip(gradients, transformer.trainable_variables))\n",
        "\n",
        "        train_loss(loss)\n",
        "        train_accuracy(dec_outputs_real, predictions)\n",
        "\n",
        "        if batch % 50 == 0:\n",
        "            print(\"Epoch {} Batch {} Loss {:.4f} Accuracy {:.4f}\".format(\n",
        "                epoch+1, batch, train_loss.result(), train_accuracy.result()))\n",
        "\n",
        "    ckpt_save_path = ckpt_manager.save()\n",
        "    print(\"Saving checkpoint for epoch {} at {}\".format(epoch+1,\n",
        "                                                        ckpt_save_path))\n",
        "    print(\"Time taken for 1 epoch: {} secs\\n\".format(time.time() - start))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YRsHEemuC9m1"
      },
      "source": [
        "## Evaluating"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HsjkINx863FM"
      },
      "source": [
        "Now we are going to make some predictions.\n",
        "\n",
        "For predictions, we iterate over the model. In the first pass we input the sentence to be translated to the encoder while the decoder's input only contains the `start-of-sentence` token. From this the model generates a probability distribution. Hopefylly the highest probability results in choosing the correct first word in the translation.\n",
        "\n",
        "In the second iteration, the word that was predicted is added to the decoder's input, so that now the model tries to predict the next word based on having the `start-of-sentence` token and the first predicted word.\n",
        "\n",
        "We continue predicting and appending to the input until we reach the `end-of-sentence` token. That completes the sentence."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4YYj6pPsDA4y"
      },
      "outputs": [],
      "source": [
        "def evaluate(inp_sentence):\n",
        "    '''\n",
        "    Takes the input sentence.\n",
        "    '''\n",
        "    inp_sentence = [VOCAB_SIZE_EN-2] + tokenizer_en.encode(inp_sentence) + [VOCAB_SIZE_EN-1]\n",
        "\n",
        "    # Expand dims to account for batch_size\n",
        "    enc_input = tf.expand_dims(inp_sentence, axis=0)\n",
        "\n",
        "    # Start with the s-o-s\n",
        "    output = tf.expand_dims([VOCAB_SIZE_FR-2], axis=0)\n",
        "\n",
        "    for _ in range(MAX_LENGTH):\n",
        "        predictions, attention_weights = transformer(enc_input, output, False)\n",
        "\n",
        "        prediction = predictions[:, -1:, :]\n",
        "\n",
        "        # Get highest probability\n",
        "        predicted_id = tf.cast(tf.argmax(prediction, axis=-1), tf.int32)\n",
        "\n",
        "        # If e-o-s return\n",
        "        if predicted_id == VOCAB_SIZE_FR-1:\n",
        "            return tf.squeeze(output, axis=0), attention_weights\n",
        "\n",
        "        # Concat last prediction to decoder input\n",
        "        output = tf.concat([output, predicted_id], axis=-1)\n",
        "\n",
        "    return tf.squeeze(output, axis=0), attention_weights"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BAaYVL8V63FM"
      },
      "source": [
        "**Plot the attention weights for a decoder layer**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DnUcjddyknkw"
      },
      "outputs": [],
      "source": [
        "def plot_attention_weights(attention, sentence, result, layer):\n",
        "    fig = plt.figure(figsize=(16, 8))\n",
        "    sentence = tokenizer_en.encode(sentence)\n",
        "    attention = tf.squeeze(attention[layer], axis=0)\n",
        "\n",
        "    for head in range(attention.shape[0]):\n",
        "\n",
        "        ax = fig.add_subplot(2, 4, head+1)\n",
        "\n",
        "        # plot the attention weights\n",
        "        ax.matshow(attention[head][:-1, :], cmap='viridis')\n",
        "        fontdict = {'fontsize': 10}\n",
        "\n",
        "        ax.set_xticks(range(len(sentence)+2))\n",
        "        ax.set_yticks(range(len(result)))\n",
        "        ax.set_ylim(len(result)-1.5, -0.5)\n",
        "        ax.set_xticklabels(\n",
        "            ['<start>']+[tokenizer_en.decode([i]) for i in sentence]+['<end>'],\n",
        "            fontdict=fontdict, rotation=90)\n",
        "\n",
        "        ax.set_yticklabels([tokenizer_fr.decode([i]) for i in result\n",
        "                            if i < VOCAB_SIZE_FR-2],\n",
        "                           fontdict=fontdict)\n",
        "\n",
        "        ax.set_xlabel('Head {}'.format(head+1))\n",
        "    plt.tight_layout()\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7qxiGxgV63FN"
      },
      "source": [
        "#### Translate function\n",
        "\n",
        "We use this function to test our translations. It calls the `evaluate` function to make the predictions and if given the decoder layer and block, produces a plot of the attention weights."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SrhHyv9gDE7C"
      },
      "outputs": [],
      "source": [
        "def translate(sentence, plot=''):\n",
        "    #output, attention_weights\n",
        "    output, attention_weights = evaluate(sentence) #.numpy()\n",
        "    print(f'wts shape: {attention_weights.keys()}')\n",
        "    output = output.numpy()\n",
        "\n",
        "    predicted_sentence = tokenizer_fr.decode(\n",
        "        [i for i in output if i < VOCAB_SIZE_FR-2]\n",
        "    )\n",
        "\n",
        "    print(\"Input: {}\".format(sentence))\n",
        "    print(\"Predicted translation: {}\".format(predicted_sentence))\n",
        "    if plot:\n",
        "        plot_attention_weights(attention_weights, sentence, output, plot)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CX4tnflN63FO"
      },
      "source": [
        "### Let's translate\n",
        "\n",
        "Now we try some translations and see how it turns out. We also get to see what the attention looks like by plotting it.\n",
        "\n",
        "I verify the translations that follow by running the resultant french text through `Google translate`.\n",
        "\n",
        "---\n",
        "**Source: \"This is a really powerful tool!\"**\n",
        "\n",
        "**Translation: \"C'est un instrument très puissant\"**\n",
        "\n",
        "**Google: \"It is a very powerful instrument\"**\n",
        "\n",
        "This is very close."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "j8UKmKo7DQsN"
      },
      "outputs": [],
      "source": [
        "translate(\"This is a really powerful tool!\", plot='decoder_layer4_block2')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eGF_h1An63FP"
      },
      "source": [
        "---\n",
        "**Source: \"what is your name?\"**\n",
        "\n",
        "**Translation: \"Quel est votre nom ?\"**\n",
        "\n",
        "**Google: \"What is your name ?\"**\n",
        "\n",
        "Perfect."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XEiotEKyASW9"
      },
      "outputs": [],
      "source": [
        "translate(\"what is your name?\", plot='decoder_layer4_block2')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XBoGmRuM63FQ"
      },
      "source": [
        "---\n",
        "**Source: \"You are beutiful.\"**\n",
        "\n",
        "**Translation: \"Vous êtes très sincère\"**\n",
        "\n",
        "**Google: \"You are very sincere\"**\n",
        "\n",
        "Almost. But notice that beautiful is misspelled, so it's sort of impressive that the translation is so close."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7b0P3kqH4nTn"
      },
      "outputs": [],
      "source": [
        "translate(\"You are beutiful.\", plot='decoder_layer4_block2')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NCgPzxkX63FR"
      },
      "source": [
        "---\n",
        "**Source: \"I love you.\"**\n",
        "\n",
        "**Translation: \"Je vous aime\"**\n",
        "\n",
        "**Google: \"I like You\"**\n",
        "\n",
        "Well, in this case I'd say that Google got it wrong. The translation is correct."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ruYzB_AO5uKc"
      },
      "outputs": [],
      "source": [
        "translate(\"I love you.\", plot='decoder_layer4_block2')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5ONtp_mF63FR"
      },
      "source": [
        "---\n",
        "**Source: \"How are you?\"**\n",
        "\n",
        "**Translation: \"Comment avez-vous\"**\n",
        "\n",
        "**Google: \"How did you\"**\n",
        "\n",
        "One word wrong."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 625
        },
        "id": "eu3c0drscXik",
        "outputId": "74949fb5-6649-4275-f725-d06a717bfc1f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "wts shape: dict_keys(['decoder_layer4_block1', 'decoder_layer4_block2'])\n",
            "Input: How are you?\n",
            "Predicted translation: Comment avez-vous ?\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAABHcAAAI8CAYAAAB74EeOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde5hkdX3v+/e3Z6aZGQaG2wAi4AVEEEHEQQGDAcOjiSZG1Eg80Rw0bmK2dyPuJNt9Eo3buA8mRo3Rh4NAsrO3RBREEy+gkUsIKFcZUaIR0Q0BBQIMDDP0TPf3/FE1TdfQXT0zVb1+61fzfj1PP3St6l716V5rPl31Za1VkZlIkiRJkiSpTmOlA0iSJEmSJGn7OdyRJEmSJEmqmMMdSZIkSZKkijnckSRJkiRJqpjDHUmSJEmSpIo53JEkSZIkSaqYwx1JkiRJkqSKOdyRJEmSJEmqmMMdSZIkSZKkijncGZLo+EJEHFY6i6TRZ+dIapKdI6lp9o60bRzuDM+LgGOAN5YOImmHYOdIapKdI6lp9o60DRzuDM/v0CmeX4uIxaXDSBp5do6kJtk5kppm70jbwOHOEETEXsDhmfkV4OvAywtHkjTC7BxJTbJzJDXN3pG2ncOd4Xgd8Jnu5+fioYOqXEScEhErSufQnOwcjRQ7p/XsHI0UO6cK9o5GRlOd43BnON5Ap3TIzGuBJ0TEAWUjSdsnIg4CPgu8tnQWzcnO0ciwc6pg52hk2DnVsHc0EprsHIc7A4qI3YC/ysw7Zyx+N7BXoUjSoF4P/A86f1TVMnaORpCd02J2jkaQndNy9o5GTGOd43BnQJn5APDdLZZdCiwvk0jafhGxCPgNOgX0YEQ8q3AkbcHO0Sixc9rPztEosXPqYO9oVDTdOQ53huPjW7lMaruXANdk5kPAOXTepUDtY+doVNg5dbBzNCrsnHrYOxoFjXaObyk3gIg4DjgeWBUR75px167AojKppIH8DvAX3c8vAj4QEe/OzImCmdRl52gE2TktZudoBNk5LWfvaMQ02jkeuTOYcWAFnSHZLjM+1gKvKphL2mbd85t3y8wrADJzA/A54IVFg2kmO0cjw86pgp2jkWHnVMPe0Ugo0TmRmQu17h1C9zy6z2bmK0tnkTT67BxJTbJzJDXN3pG2j6dlDSgzJyNiv9I5pEFExNH97s/MG5rKov7sHI0CO6cedo5GgZ1TF3tHtSvVOR65MwQR8UngicAFwLrNyzPzwmKhpG0QEd/sfroUWA18BwjgSOC6zDyuVDY9np2j2tk5dbFzVDs7pz72jmpWqnM8cmc4lgL30Xv+XAKWj6qQmScBRMSFwNGZuaZ7+5nAnxSMptnZOaqanVMdO0dVs3OqZO+oWqU6xyN3JE2LiFsy8/D5lknSMNg5kppk50hqUtOd45E7QxARS+m8zdnhdKbMAGTmGxrOcVBm/qjJx9TIuTkizgb+rnv7t4CbC+bRLNrSOd0s9o4GYedUwM7RCLFzKtGW3rFzNKBGO8e3Qh+O/wnsC7wYuBzYH3ioQI5zIuJHEXF+RLw5Io4okEF1ez1wC/D27sf3usvULm3pHLB3NBg7pw52jkaFnVOPtvSOnaNBNNo5npY1BBFxY2Y+OyJuzswjI2IJcGVmHlsgyzhwDHAi8LvAiszco+kckhZOmzqnm8fekUaYnSOpaW3qHTtHtfC0rOHY2P3vA92LJN0N7N10iIj4BeCE7sduwD8AVzadQ/WKiOfTucjXk5jRD5n51FKZNKtWdA7YOxqMnVMNO0cjwc6pSit6x87RIJruHIc7w3FWROwOvBf4IrAC+G8FclwGXA/8GfDlzJwokEF1+zTwTjr70WThLJpbWzoH7B0Nxs6pg52jUWHn1KMtvXMZdo62X6Od42lZQxART8nMH8+3rIEcuwHPB15A59DBKeDqzCz1BKxVIuK3Z1uemX/bdJa2iohvZebzSudQf23pnO7j2jt92Dv92Tl1sHPqYef0Z+fUoy29Y+f0Z+f013TneOTOcHweOHqLZZ8DntNkiMx8ICJuAw6gc9Gx44ElTWaYKSKCzhXBn5qZ74+IA4F9M/PbhSIdM+PzpcAvATcAls9jvhkRZwIXAo9uXpiZN5SLpFm0onPA3tkK9k5/dk4d7Jw52DnVsXPq0YresXPmZef012jnONwZQEQcSuft+VZGxCtm3LUrM96yr8E8twG3Av8MfBJ4feFDB/+aznT7hcD76Vzh/vP0lkBjMvOtM293J/Hnl8jSYpsny6tnLEs621CFta1zupnsnT7snXnZOS1m52wVO6cudk7Lta137Jz+7Jx5Ndo5DncG83TgV+lcXOvXZix/CPhPBfIcnJlTBR53Ls/LzKMj4kaAzLy/e7X5tlgHPKV0iDbJzJNKZ1BfbescsHe2lb0zg53TenbO/Oycitg5VWhb79g528bOmaHpznG4M4DMvBi4OCKOy8yrS+cB9ouIj9M5LxQ6V3J/e2beUSjPxohYRGc6SUSsojNpLiIivrQ5C7AIOAz4bKk8bRQR+wAfBPbLzF+JiGcAx2XmpwtHE63sHLB3+rJ3+rNz2s3O2Sp2TkXsnPZrYe/YOX3YOf013TljC7HSHdApEbFrRCyJiG9ExD0R8doCOc6lczX5/bofX+ouK+VjwEXA3hHx3+kczvjBgnk+DPx59+ODwAsy8w8K5mmj84Cv0dl/AH4AvKNYGs2lLZ0D9s587J3+zsPOqYGdMzc7py7nYefUoi29Y+f0Z+f0dx4Ndo7DneF4UWaupXMI4e3AwcAZBXKsysxzM3NT9+M8YFWBHETEGPBj4D103jrwLuDlmXlBiTwAmXk5nXNmdwF2B3wrw8fbKzM/S/f/AGTmJnyr0DZqS+eAvdOXvTMvO6cOds4s7Jwq2Tn1aEvv2Dl92DnzarRzPC1rODZfMf2lwAWZ+WDnQuaNu6870f5M9/ZrgPtKBMnMqYj4RGY+m84/+OIi4tXAmcBlQAAfj4gzMvNzRYO1y7qI2JPHDvU8FniwbCTNoi2dA/ZOX/bOvOycOtg5s7BzqmTn1KMtvWPn9GHnzKvRznG4MxxfiohbgfXA73XPfdxQIMcbgI8DH6GzA/0L8PoCOTb7RkS8ErgwM3Per154/xU4JjN/DtPnqH6dztsqquNddA49PSgirqLzfyZeVTaSZtGWzgF7Zz72Tn92Th3snLnZOXWxc+rRlt6xc/qzc/prtHOiHftE/SJiD+DBzJyMiJ2BXTLz7tK5SoqIh4CdgU10yjiAzMxdC+VZk5lHzLg9Bnxn5jJBRCym804FAfxrZm4sHEmzsHNmZ+/Ux86pg50zOzunPnZOPeydx7Nz6tNk53jkzoAiYjnwtMz8zozFe9Lg+bvdK7jPOaXLzLc1lWWLx92lxOP28ZWI+BqPHVZ5KvDlgnlaZYt9+ZbusgMjYjIz7yybTpu1oXO6OeydrWPvzMHOqYOd05+dUw87px5t6B07Z6vZOXMo0TkOdwa3EbgwIo7MzHXdZWcDfwQ09Yfiuhmfvw/444Yed14RsTvwNGDp5mWZeUWhOHcAVwMndG+flZkXFcrSRm3YlzW/tmwne2fr2Dtza8u+rP7asp3snK1j58ytLfuy5teGbWXnbB07Z26N78eeljUEEfFh4JbMPDciDgQu7l7oqkSWG0s99pYi4o3A24H9gZuAY4GrM/OFhfJ8APhN4AbgHOBrJc5V7R5i+iY6h1Ke3X0ngFZo076subVtO9k7ffPYO320bV/W7Nq2neycvnnsnD7ati9rbm3aVnZO3zx2Th9N78e+FfpwnM1jF9b6beDcglnaNK17O3AM8JPMPAl4NvBAqTCZ+V46U+5PA6cBP4yID0bEQQ1H+TywAngicHVEPLXhx++nTfuy5ta27WTvzMHemVfb9mXNrm3byc6Zg50zr7bty5pbm7aVnTMHO2deje7HnpY1BJl5a3QcQmdyecJ837OD2JCZGyKCiNip+3t6eslAmZkRcTdwN50Lke0OfC4iLs3M9zQUY8/M/COAiLgEuDwiHgB+H3hjZr66oRyP475cB7dTX/bO7FrZO+7LdXA79WXnzM7O0UDcVnOyc2Zn5+BwZ5g+TWcytyYz72/ygaNz1fTNE+XlEbH5MLSiV08H7oiI3YAvAJdGxP3ATwplISLeTmdiei+dbXVGZm6MzlXdfwg0VT4PRcSTM/P2zPxa9xC9/YD7gTUNZein2L6sbVJ0O9k7W8fe2Sp2Th3snNnZObOzczQMvr56PDtndnYOXnNnaKJzNey7gFdm5tdL52mbiPhFYCXw1cycKJThfcA5mfm4AoyIwzLz+w3leDqdPwo/aOLxtpX7ch3cTvOzd3oeq7W9475cB7fT/OycnseyczQwt1V/dk7PY9k5ONyRJEmSJEmqmhdUliRJkiRJqpjDHUmSJEmSpIo53BmyiDi9dIbN2pQFzDMf82h7tG07mWdubcoC5tH2adt2Mk9/bcrTpizQvjyaW5u2VZuygHnm06Y8TWRxuDN8rdmBaFcWMM98zKPt0bbtZJ65tSkLmEfbp23byTz9tSlPm7JA+/Jobm3aVm3KAuaZT5vyONyRJEmSJEnS3Hy3rK7xsaW5bNEuA69nYmoD42NLBw+0ZMngWTY9wvji5YNnAZiaGngVE5OPML5o8DxTOy0eeB0AGyfWsWR854HXs2l5DCENbHpkHYuXD55nycODbyuAiY3rGF8yWJ4NGx5gYuO64fyCRsx4LM1lYysGXs9EbmA8Bu+cjU8dvHMANj34CItXDv7vfMmdw9lthtGDuX7DULJs5FGWsNPgK1o+hL8xwMZNj7BkCH8jYsNw3n11WPvy2qn77s3MVUOINFLa1jmb9lg28DoANq1fx+Jlg//tHNaz4U0b1rF46WB5ljw4vHc0nphaz/jYYL/rqaXD+fuwceM6lgz4vAJgbNOQnucM6Xny2vV32TlzGB8bUu8M6fXV0w5/eOB13HPfJKv2XDTwegB+8KM9B17HsP5dxcbJgdcBMDH1CONjQ3i9t7Rdr/fG1g3+XHBY+/H6qYeZmNow6xPl4fzWRsCyRbtw3G6vKB1jWh6wb+kIPcbWPlI6wrRHDmnX3897njWcJz3D8oRr1peOMO3a6z5ROkJrLRtbwbHLXlo6xrSf/+X+pSP02Pu9w3niNAxTa35QOkKPOPyw0hF6xPduKx2hxyXr/vYnpTO00bKxFRy74mWlY0y799efWTpCj2xP5bDPV35aOkKPDYe06znp+H3teU4KcMlNf2rnzGHZ2AqOW/HrpWNM+/LXrigdoceLX/660hGmLb5nbekIPdYdunfpCD2WX/WvpSNMu/rhi+e8z9OyJEmSJEmSKuZwR5IkSZIkqWIOdyRJkiRJkirmcEeSJEmSJKliDnckSZIkSZIq5nBHkiRJkiSpYg53JEmSJEmSKuZwR5IkSZIkqWIOdyRJkiRJkirmcEeSJEmSJKliDnckSZIkSZIq5nBHkiRJkiSpYvMOdyJi34g4PyJ+FBHXR8SXI+KQJsJtj4g4MSKOL51D0vaxcyQ1zd6R1CQ7R9JC6DvciYgALgIuy8yDMvM5wB8C+zQRbjudCFg+UoXsHElNs3ckNcnOkbRQ5jty5yRgY2Z+avOCzPxOZl4ZHWdGxHcjYk1EnArTk93LI+LiiLgtIj4UEb8VEd/uft1B3a87LyI+GRHXdL/uxIg4JyK+HxHnbX68iHhRRFwdETdExAURsaK7/PaIeF93+ZqIODQingy8CXhnRNwUEScM9bclaaHZOZKaZu9IapKdI2lBzDfceSZw/Rz3vQI4CngWcDJwZkQ8oXvfs+iUwGHA64BDMvO5wNnAW2esY3fgOOCdwBeBjwCHA0dExFERsRfwXuDkzDwauA5414zvv7e7/JPAuzPzduBTwEcy86jMvLLfDxcRp0fEdRFx3cTUhnl+FZIasON0Tto5UkuMbO/YOVIrjWzngK+vpJIWD/C9vwB8JjMngZ9FxOXAMcBa4NrMvAsgIn4EXNL9njV0ptWbfSkzMyLWAD/LzDXd77kFeDKwP/AM4KqIABgHrp7x/Rd2/3s9nTLcJpl5FnAWwMolq3Jbv19So0arcxbtZedI7Vd179g5UnWq7hzYoncW2ztSk+Yb7twCvGo71vvojM+nZtye2uIxH53la2Z+3SRwaWa+Zp7HmWSwQZWkdrBzJDXN3pHUJDtH0oKY77SsfwJ2iojTNy+IiCO751peCZwaEYsiYhXwAuDbQ853DfD8iDi4+9g7x/xXkn8I2GXIOSQ1w86R1DR7R1KT7BxJC6LvcCczEzgFODk6b9V3C/BnwN10rvJ+M/AdOiX1nsy8e5jhMvMe4DTgMxFxM51DBg+d59u+BJziBb+k+tg5kppm70hqkp0jaaFEp1+0csmqPG63bT6tdMHkAfuWjtBjbO0jpSNMe+SQVaUj9LjnWUtKR+jxhGvWl44w7drrPsHah+6M0jnaaOWivfLYZS8tHWPaz/9+/9IReuz93kWlI0ybWvOD0hF6xNGHlY7QI753W+kIPS5Z97fXZ+bq0jnaZuWivfLYFS8rHWPavb/xzNIRemR7Kod9vvLT0hF6bDikXc9Jx+9rz3NSgEtu+lM7Zw4rF++Vx6349dIxpn351itKR+jx4pe/rnSEaYvvWVs6Qo91h+5dOkKP5Vf9a+kI065++GIe3HTvrK+v5jstS5IkSZIkSS3mcEeSJEmSJKliDnckSZIkSZIq5nBHkiRJkiSpYg53JEmSJEmSKuZwR5IkSZIkqWIOdyRJkiRJkirmcEeSJEmSJKliDnckSZIkSZIq5nBHkiRJkiSpYg53JEmSJEmSKuZwR5IkSZIkqWKLSwdolYjSCabF7XeWjtBj9eX3lo4w7dvH31M6Qo8D/23v0hF6Pfhw6QTTYsPG0hFaKzPJycnSMabt/Rs/LR2hxw//9NmlI0x72h/vVDpCj7VP2bl0hB7Llx5SOkKvK0sHaKfMKXL9+tIxpu35t9eWjtDrqENLJ5iWG9v1t/O217bn+THAPt9YWTpCr5tKB2ixqSQnJkqnmPbLT3pu6Qg9Fu12d+kI0zY+bb/SEXqc9KGrSkfo8Xdf/cXSEaZt+Oglc97nkTuSJEmSJEkVc7gjSZIkSZJUMYc7kiRJkiRJFXO4I0mSJEmSVDGHO5IkSZIkSRVzuCNJkiRJklQxhzuSJEmSJEkVc7gjSZIkSZJUMYc7kiRJkiRJFXO4I0mSJEmSVDGHO5IkSZIkSRVzuCNJkiRJklQxhzuSJEmSJEkVc7gjSZIkSZJUMYc7kiRJkiRJFXO4I0mSJEmSVLHFpQNsrYj4AnAAsBT4KJ3B1EGZeUb3/tOA1Zn5loh4LfA2YBz4FvCfM3OySHBJ1bJ3JDXJzpHUJDtHGi01Hbnzhsx8DrCaTrFcBJwy4/5TgfMj4rDu58/PzKOASeC3ZlthRJweEddFxHUTUxsWNr2kGg21d2Z2zsa0cyQ9zgJ2zqMLn15SbRb29RX2jtSkao7cAd4WEZvL5gDgKcBtEXEs8EPgUOAq4M3Ac4BrIwJgGfDz2VaYmWcBZwGsXLIqFzS9pBoNtXdmds6uY3vaOZK2tICds4edI2lLC/v6yuc6UqOqGO5ExInAycBxmflIRFxG5/DB84FXA7cCF2VmRqdx/iYz/7BUXkn1s3ckNcnOkdQkO0caPbWclrUSuL9bPIcCx3aXXwT8OvAaOkUE8A3gVRGxN0BE7BERT2o6sKTq2TuSmmTnSGqSnSONmFqGO18FFkfE94EPAdcAZOb9wPeBJ2Xmt7vLvge8F7gkIm4GLgWeUCS1pJrZO5KaZOdIapKdI42YKk7LysxHgV+Z475fnWXZ3wN/v9C5JI0ue0dSk+wcSU2yc6TRU8uRO5IkSZIkSZqFwx1JkiRJkqSKOdyRJEmSJEmqmMMdSZIkSZKkijnckSRJkiRJqpjDHUmSJEmSpIo53JEkSZIkSaqYwx1JkiRJkqSKOdyRJEmSJEmqmMMdSZIkSZKkijnckSRJkiRJqpjDHUmSJEmSpIotLh2gLXJykqkHHiwdo7WufdETS0eYdutHDywdoceu3x0vHaHH/p//aekIjxmL0gnaK5N89NHSKaZl6QBbOOiMq0tHmDZVOsAWrvjLT5aO0OOlT3pu6QjaGgm5aVPpFO113XdLJ3jM7ruXTtDjx798dukIPV7y+yeVjqCtFQFLlpROMW1sl6WlI/TabZfSCR7TsieCr1p5fekIPS699oTSEab97JG57/PIHUmSJEmSpIo53JEkSZIkSaqYwx1JkiRJkqSKOdyRJEmSJEmqmMMdSZIkSZKkijnckSRJkiRJqpjDHUmSJEmSpIo53JEkSZIkSaqYwx1JkiRJkqSKOdyRJEmSJEmqmMMdSZIkSZKkijnckSRJkiRJqpjDHUmSJEmSpIo53JEkSZIkSaqYwx1JkiRJkqSKjexwJyLeFBE3dT9+HBHfLJ1J0uiycyQ1zd6R1CQ7R2q3kR3uZOanMvMo4BjgDuAvCkeSNMLsHElNs3ckNcnOkdptcekADfgo8E+Z+aUt74iI04HTAZayvOlckkaTnSOpabP2jp0jaYFs3XOd2LnpXNIObaSHOxFxGvAk4C2z3Z+ZZwFnAew6tkc2l0zSKNqmzgk7R9Lg+vWOnSNp2Lbluc7KRXvZO1KDRna4ExHPAd4NnJCZU6XzSBptdo6kptk7kppk50jtNrLX3KEzTd4D+Gb3ol9nlw4kaaTZOZKaZu9IapKdI7XYyB65k5mvL51B0o7DzpHUNHtHUpPsHKndRvnIHUmSJEmSpJHncEeSJEmSJKliDnckSZIkSZIq5nBHkiRJkiSpYg53JEmSJEmSKuZwR5IkSZIkqWIOdyRJkiRJkirmcEeSJEmSJKliDnckSZIkSZIq5nBHkiRJkiSpYg53JEmSJEmSKuZwR5IkSZIkqWKLSwdoj4BFi0qHmJYTE6Uj9MiHHi4dYdo+l7Vrt334FQ+WjtAjL1leOsJjfu78uK+I0gkek1k6Qa82/W5a5uF8tHSEXi362wnAxtIB2imWLGHxPvuVjjFt07/fVTpCj7HDn146wmOmpkon6PHyH764dIQttOs5svpYvIix3VaWTjFt0/+5o3SEHpPPPKB0hGnrV42XjtDj1y5/c+kIPXY6vD3PdTb989z3+cpLkiRJkiSpYg53JEmSJEmSKuZwR5IkSZIkqWIOdyRJkiRJkirmcEeSJEmSJKliDnckSZIkSZIq5nBHkiRJkiSpYg53JEmSJEmSKuZwR5IkSZIkqWIOdyRJkiRJkirmcEeSJEmSJKliDnckSZIkSZIq5nBHkiRJkiSpYg53JEmSJEmSKuZwR5IkSZIkqWIOdyRJkiRJkirmcEeSJEmSJKliDnckSZIkSZIqtrh0gJIi4nTgdIClLC+cRtKos3MkNamncxbtUjiNpB2BvSOVM/JH7kTEmyPipu7HfjPvy8yzMnN1Zq5eEktLRZQ0Qra6c9ipVERJI2RrO2d8bFmpiJJGzFb3ziJ7R2rSyB+5k5mfAD5ROoekHYOdI6lJdo6kptk7UjuN/JE7kiRJkiRJo8zhjiRJkiRJUsUc7kiSJEmSJFXM4Y4kSZIkSVLFHO5IkiRJkiRVzOGOJEmSJElSxRzuSJIkSZIkVczhjiRJkiRJUsUc7kiSJEmSJFXM4Y4kSZIkSVLFHO5IkiRJkiRVzOGOJEmSJElSxRzuSJIkSZIkVczhjiRJkiRJUsUWlw7QFhFBjI+XjjEtJyZKR+gVUTrBtIcOaNdM8tEf71I6Qq+8r3QCba3M0gm0NVq2nR6amiwdocfYsqWlI/TaUDpAS01OMrX2odIpHhPt+ls+9vAjpSM8ZlO7/o0/d/fbS0focdXSp5SOoK2UGzcxeffPS8eYFovb9dJ3/Lv/p3SEaWMH71c6Qo8bf+lTpSP0eOFV7yodYdrYxj73NRdDkiRJkiRJw+ZwR5IkSZIkqWIOdyRJkiRJkirmcEeSJEmSJKliDnckSZIkSZIq5nBHkiRJkiSpYg53JEmSJEmSKuZwR5IkSZIkqWIOdyRJkiRJkirmcEeSJEmSJKliDnckSZIkSZIq5nBHkiRJkiSpYg53JEmSJEmSKuZwR5IkSZIkqWIOdyRJkiRJkirmcEeSJEmSJKlirRvuRMSHIuLNM27/SUScERFnRsR3I2JNRJzave/EiPiHGV/7VxFx2oz1fC8ibo6IDzf+g0iqgp0jqUl2jqSm2TvSjqF1wx3g74FXz7j9auDnwFHAs4CTgTMj4glzrSAi9gROAQ7PzCOBDyxcXEmVs3MkNcnOkdQ0e0faAbRuuJOZNwJ7R8R+EfEs4H46xfOZzJzMzJ8BlwPH9FnNg8AG4NMR8Qrgkdm+KCJOj4jrIuK6idww3B9EUhVKdc5GHh3uDyKpCj7PkdS0Ys917B2pUa0b7nRdALwKOJXOpHkum+j9GZYCZOYm4LnA54BfBb462zdn5lmZuTozV4/H0mHkllSnxjtnCTsNI7ekOvk8R1LTmn+uY+9IjWrrcOfvgd+kU0AXAFcCp0bEoohYBbwA+DbwE+AZEbFTROwG/BJARKwAVmbml4F30jncUJLmYudIapKdI6lp9o404haXDjCbzLwlInYB7szMuyLiIuA44DtAAu/JzLsBIuKzwHeBHwM3dlexC3BxRCwFAnhX0z+DpHrYOZKaZOdIapq9I42+Vg53ADLziBmfJ3BG92PLr3sP8J5ZVvHchUsnadTYOZKaZOdIapq9I422tp6WJUmSJEmSpK3gcEeSJEmSJKliDnckSZIkSZIq5nBHkiRJkiSpYg53JEmSJEmSKuZwR5IkSZIkqWIOdyRJkiRJkirmcEeSJEmSJKliDnckSZIkSZIq5nBHkiRJkiSpYg53JEmSJEmSKuZwR5IkSZIkqWKLSwdoi5yaYuqhh0rHaK2pdetKR5h24EdvKh2hx1f+7V9KR+jx0g//cukIj5mcLJ2g3SJKJ3hMZukEvdqWp0X2X7yidIQekw+uLR1BW2NsjFi6tHSKx7TsOdemn95ZOsK0sZ2Xl47Q4x17rCkdoccV9+xSOoK20tTK5ax74bNLx5i2/KJvlY7Q42enHFw6wrR9zr+ldOd8lfUAACAASURBVIQeJ33gXaUj9Fh74vrSEaZNXjI1530euSNJkiRJklQxhzuSJEmSJEkVc7gjSZIkSZJUMYc7kiRJkiRJFXO4I0mSJEmSVDGHO5IkSZIkSRVzuCNJkiRJklQxhzuSJEmSJEkVc7gjSZIkSZJUMYc7kiRJkiRJFXO4I0mSJEmSVDGHO5IkSZIkSRVzuCNJkiRJklQxhzuSJEmSJEkVG8nhTkTsHRFfj4g1EXFdRBxcOpOk0WbvSGqSnSOpSXaO1H4jOdwBFgPvzswjgP8P+IPCeSSNPntHUpPsHElNsnOklltcOsBCyMx/B/69e3MnYEPBOJJ2APaOpCbZOZKaZOdI7TeSw53NIuIo4B3AC+e4/3TgdIClLG8wmaRR1a937BxJw7bVnTO2ouFkkkbRtry+Gl+2W4PJJI3qaVmbnQOclpm3z3ZnZp6Vmaszc/USdmo2maRRNWfv2DmSFsBWdc742LLmk0kaRVv/+monh8pSk0Z9uHNwZl5ROoSkHYq9I6lJdo6kJtk5UkuN+nDn9aUDSNrh2DuSmmTnSGqSnSO11KgPd36/dABJOxx7R1KT7BxJTbJzpJYa6eFOZh5fOoOkHYu9I6lJdo6kJtk5UnuN9HBHkiRJkiRp1DnckSRJkiRJqpjDHUmSJEmSpIo53JEkSZIkSaqYwx1JkiRJkqSKOdyRJEmSJEmqmMMdSZIkSZKkijnckSRJkiRJqpjDHUmSJEmSpIo53JEkSZIkSaqYwx1JkiRJkqSKOdyRJEmSJEmqWGRm6QytEBH3AD8Zwqr2Au4dwnqGoU1ZwDzzGcU8T8rMVcMIM2pGtHPAPP20KQuMbh57ZxZ2TmPMM7c2ZQE7Z8GNaO+0KQuYZz5tyrPgneNwZ8gi4rrMXF06B7QrC5hnPubR9mjbdjLP3NqUBcyj7dO27WSe/tqUp01ZoH15NLc2bas2ZQHzzKdNeZrI4mlZkiRJkiRJFXO4I0mSJEmSVDGHO8N3VukAM7QpC5hnPubR9mjbdjLP3NqUBcyj7dO27WSe/tqUp01ZoH15NLc2bas2ZQHzzKdNeRY8i9fcUaMi4uHMXDHj9mnA6sx8yxDWfRnw7sy8bovlbwHeARwErMrMtlxUS1IDCvXO/wJWAxuBbwO/m5kbB308Se1XqHM+TadzAvgBcFpmPjzo40lqvxKdM+P+jwFvmPn4Kscjd7QjuAo4meFcrV+Stsb/Ag4FjgCWAW8sG0fSiHtnZj4rM48EfgoM/KJOkvqJiNXA7qVz6DEOd9QaEbEqIj4fEdd2P57fXf7ciLg6Im6MiH+JiKd3ly+LiPMj4vsRcRGdF1CPk5k3Zubtzf0kkmqxgL3z5eyic+TO/o39UJJaawE7Z23366P7NR6aL2nBOiciFgFnAu9p7IfRvBaXDqAdzrKIuGnG7T2AL3Y//yjwkcz854g4EPgacBhwK3BCZm6KiJOBDwKvBH4PeCQzD4uII4EbGvspJNWkWO9ExBLgdcDbh/oTSWqzIp0TEecCLwG+B/z+sH8oSa1VonPeAnwxM+/qzJTVBg531LT1mXnU5hubzwnt3jwZeMaMgtg1IlYAK4G/iYin0fk/UUu6978A+BhAZt4cETcvfHxJFSrZO38NXJGZVw7jB5FUhSKdk5mv7/7f9I8DpwLnDu0nktRmjXZOROwH/AZw4tB/Eg3E4Y7aZAw4NjM3zFwYEX8FfDMzT4mIJwOXNR9N0ohasN6JiD8GVgG/O3hMSSNiQZ/rZOZkRJxP51QJhzuSFqJzng0cDPxbd2i0PCL+LTMPHkpibTevuaM2uQR46+YbEbF5Ar0SuLP7+Wkzvv4K4P/qfu0zgSMXPqKkEbMgvRMRbwReDLwmM6eGG1lSxYbeOdFx8ObPgZfROeVCkobeOZn5j5m5b2Y+OTOfTOc0Lgc7LeBwR23yNmB1RNwcEd8D3tRd/v8CfxYRN9J7tNkngRUR8X3g/cD1s600It4WEXfQuaDpzRFx9oL9BJJqsyC9A3wK2Ae4OiJuioj/Z2HiS6rMQnRO0Dm9Yg2wBnhC92slaaGe56iFovNGHpIkSZIkSaqRR+5IkiRJkiRVzOGOJEmSJElSxRzuSJIkSZIkVczhjiRJkiRJUsUc7kiSJEmSJFXM4Y4kSZIkSVLFHO5IkiRJkiRVzOGOJEmSJElSxRzuSJIkSZIkVczhjiRJkiRJUsUc7kiSJEmSJFXM4c6QRMcXIuKw0lkkjT47R1KT7BxJTbN3pG3jcGd4XgQcA7yxdBBJOwQ7R1KT7BxJTbN3pG3gcGd4fodO8fxaRCwuHUbSyLNzJDXJzpHUNHtH2gYOd4YgIvYCDs/MrwBfB15eOJKkEWbnSGqSnSOpafaOtO0c7gzH64DPdD8/Fw8dVOUi4pSIWFE6h+Zk52ik2DmtZ+dopNg5VbB3NDKa6hyHO8PxBjqlQ2ZeCzwhIg4oG0naPhFxEPBZ4LWls2hOdo5Ghp1TBTtHI8POqYa9o5HQZOc43BlQROwG/FVm3jlj8buBvQpFkgb1euB/0PmjqpaxczSC7JwWs3M0guyclrN3NGIa6xyHOwPKzAeA726x7FJgeZlE0vaLiEXAb9ApoAcj4lmFI2kLdo5GiZ3TfnaORomdUwd7R6Oi6c5xuDMcH9/KZVLbvQS4JjMfAs6h8y4Fah87R6PCzqmDnaNRYefUw97RKGi0c3xLuQFExHHA8cCqiHjXjLt2BRaVSSUN5HeAv+h+fhHwgYh4d2ZOFMykLjtHI8jOaTE7RyPIzmk5e0cjptHO8cidwYwDK+gMyXaZ8bEWeFXBXNI2657fvFtmXgGQmRuAzwEvLBpMM9k5Ghl2ThXsHI0MO6ca9o5GQonOicxcqHXvELrn0X02M19ZOouk0WfnSGqSnSOpafaOtH08LWtAmTkZEfuVziENIiKO7nd/Zt7QVBb1Z+doFNg59bBzNArsnLrYO6pdqc7xyJ0hiIhPAk8ELgDWbV6emRcWCyVtg4j4ZvfTpcBq4DtAAEcC12XmcaWy6fHsHNXOzqmLnaPa2Tn1sXdUs1Kd45E7w7EUuI/e8+cSsHxUhcw8CSAiLgSOzsw13dvPBP6kYDTNzs5R1eyc6tg5qpqdUyV7R9Uq1TkeuSNpWkTckpmHz7dMkobBzpHUJDtHUpOa7hyP3BmCiFhK523ODqczZQYgM9/QcI6DMvNHTT6mRs7NEXE28Hfd278F3Fwwj2bRls7pZrF3NAg7pwJ2jkaInVOJtvSOnaMBNdo5vhX6cPxPYF/gxcDlwP7AQwVynBMRP4qI8yPizRFxRIEMqtvrgVuAt3c/vtddpnZpS+eAvaPB2Dl1sHM0KuycerSld+wcDaLRzvG0rCGIiBsz89kRcXNmHhkRS4ArM/PYAlnGgWOAE4HfBVZk5h5N55C0cNrUOd089o40wuwcSU1rU+/YOaqFp2UNx8bufx/oXiTpbmDvpkNExC8AJ3Q/dgP+Abiy6RyqV0Q8n85Fvp7EjH7IzKeWyqRZtaJzwN7RYOycatg5Ggl2TlVa0Tt2jgbRdOc43BmOsyJid+C9wBeBFcB/K5DjMuB64M+AL2fmRIEMqtungXfS2Y8mC2fR3NrSOWDvaDB2Th3sHI0KO6cebemdy7BztP0a7RxPyxqCiHhKZv54vmUN5NgNeD7wAjqHDk4BV2dmqSdgrRIRvz3b8sz826aztFVEfCszn1c6h/prS+d0H9fe6cPe6c/OqYOdUw87pz87px5t6R07pz87p7+mO8cjd4bj88DRWyz7HPCcJkNk5gMRcRtwAJ2Ljh0PLGkyw0wREXSuCP7UzHx/RBwI7JuZ3y4U6ZgZny8Ffgm4AbB8HvPNiDgTuBB4dPPCzLyhXCTNohWdA/bOVrB3+rNz6mDnzMHOqY6dU49W9I6dMy87p79GO8fhzgAi4lA6b8+3MiJeMeOuXZnxln0N5rkNuBX4Z+CTwOsLHzr413Sm2y8E3k/nCvefp7cEGpOZb515uzuJP79ElhbbPFlePWNZ0tmGKqxtndPNZO/0Ye/My85pMTtnq9g5dbFzWq5tvWPn9GfnzKvRznG4M5inA79K5+JavzZj+UPAfyqQ5+DMnCrwuHN5XmYeHRE3AmTm/d2rzbfFOuAppUO0SWaeVDqD+mpb54C9s63snRnsnNazc+Zn51TEzqlC23rHztk2ds4MTXeOw50BZObFwMURcVxmXl06D7BfRHycznmh0LmS+9sz845CeTZGxCI600kiYhWdSXMREfGlzVmARcBhwGdL5WmjiNgH+CCwX2b+SkQ8AzguMz9dOJpoZeeAvdOXvdOfndNuds5WsXMqYue0Xwt7x87pw87pr+nOGVuIle6ATomIXSNiSUR8IyLuiYjXFshxLp2rye/X/fhSd1kpHwMuAvaOiP9O53DGDxbM82Hgz7sfHwRekJl/UDBPG50HfI3O/gPwA+AdxdJoLm3pHLB35mPv9Hcedk4N7Jy52Tl1OQ87pxZt6R07pz87p7/zaLBzHO4Mx4sycy2dQwhvBw4GziiQY1VmnpuZm7of5wGrCuQgIsaAHwPvofPWgXcBL8/MC0rkAcjMy+mcM7sLsDvgWxk+3l6Z+Vm6/wcgMzfhW4W2UVs6B+ydvuydedk5dbBzZmHnVMnOqUdbesfO6cPOmVejneNpWcOx+YrpLwUuyMwHOxcyb9x93Yn2Z7q3XwPcVyJIZk5FxCcy89l0/sEXFxGvBs4ELgMC+HhEnJGZnysarF3WRcSePHao57HAg2UjaRZt6Rywd/qyd+Zl59TBzpmFnVMlO6cebekdO6cPO2dejXaOw53h+FJE3AqsB36ve+7jhgI53gB8HPgInR3oX4DXF8ix2Tci4pXAhZmZ8371wvuvwDGZ+XOYPkf163TeVlEd76Jz6OlBEXEVnf8z8aqykTSLtnQO2DvzsXf6s3PqYOfMzc6pi51Tj7b0jp3Tn53TX6OdE+3YJ+oXEXsAD2bmZETsDOySmXeXzlVSRDwE7AxsolPGAWRm7looz5rMPGLG7THgOzOXCSJiMZ13KgjgXzNzY+FImoWdMzt7pz52Th3snNnZOfWxc+ph7zyenVOfJjvHI3cGFBHLgadl5ndmLN6TBs/f7V7Bfc4pXWa+raksWzzuLiUet4+vRMTXeOywylOBLxfM0ypb7Mu3dJcdGBGTmXln2XTarA2d081h72wde2cOdk4d7Jz+7Jx62Dn1aEPv2Dlbzc6ZQ4nOcbgzuI3AhRFxZGau6y47G/gjoKk/FNfN+Px9wB839LjziojdgacBSzcvy8wrCsW5A7gaOKF7+6zMvKhQljZqw76s+bVlO9k7W8femVtb9mX115btZOdsHTtnbm3ZlzW/NmwrO2fr2Dlza3w/9rSsIYiIDwO3ZOa5EXEgcHH3QlclstxY6rG3FBFvBN4O7A/cBBwLXJ2ZLyyU5wPAbwI3AOcAXytxrmr3ENM30TmU8uzuOwG0Qpv2Zc2tbdvJ3umbx97po237smbXtu1k5/TNY+f00bZ9WXNr07ayc/rmsXP6aHo/9q3Qh+NsHruw1m8D5xbM0qZp3duBY4CfZOZJwLOBB0qFycz30plyfxo4DfhhRHwwIg5qOMrngRXAE4GrI+KpDT9+P23alzW3tm0ne2cO9s682rYva3Zt2052zhzsnHm1bV/W3Nq0reycOdg582p0P/a0rCHIzFuj4xA6k8sT5vueHcSGzNwQEUTETt3f09NLBsrMjIi7gbvpXIhsd+BzEXFpZr6noRh7ZuYfAUTEJcDlEfEA8PvAGzPz1Q3leBz35Tq4nfqyd2bXyt5xX66D26kvO2d2do4G4raak50zOzsHhzvD9Gk6k7k1mXl/kw8cnaumb54oL4+IzYehFb16OnBHROwGfAG4NCLuB35SKAsR8XY6E9N76WyrMzJzY3Su6v5DoKnyeSginpyZt2fm17qH6O0H3A+saShDP8X2ZW2TotvJ3tk69s5WsXPqYOfMzs6ZnZ2jYfD11ePZObOzc/CaO0MTnath3wW8MjO/XjpP20TELwIrga9m5kShDO8DzsnMxxVgRByWmd9vKMfT6fxR+EETj7et3Jfr4Haan73T81it7R335Tq4neZn5/Q8lp2jgbmt+rNzeh7LzsHhjiRJkiRJUtW8oLIkSZIkSVLFHO4MWUScXjrDZm3KAuaZj3m0Pdq2ncwztzZlAfNo+7RtO5mnvzblaVMWaF8eza1N26pNWcA882lTniayONwZvtbsQLQrC5hnPubR9mjbdjLP3NqUBcyj7dO27WSe/tqUp01ZoH15NLc2bas2ZQHzzKdNeRzuSJIkSZIkaW5eULlrPJbmsth54PVM8Cjj7DTwep52xLqB13HPfZOs2nPRwOsB+OGtuw28jonJRxhftHzwMFNTg68DmJhaz/jYssFXFDH4OhhinsnJwdcBTOQGxmPpQOtYP/UwE7lhOL+gETO+aFkuW7Jy4PUM7d/V2HBm/RObHmF88eB5psaHk2fjxDqWjA/W7WMPbRhKlmH8mwKY2HsIPQFMPrKORcsH/7s3/h8bh5AGJqYeYXxs8H1n7cTP7s3MVUOINFLGY6dcOoTnORvzUZbE4M9z9jl8/cDrAHjwPyZZucfgz3V+/v3BfzcAE1MbGB8b7N95Tg3vufnG3MCSAXsnFg3p78MQfjcAOTmc54HD+N0APJT/YefMYTyW5rKxFQOvZ1h/P5ccMvAq2PDABpbuNngWgIk7Bu/SiY3rGF8yhP5aN5xOHtbfiBgfH0KaIT5PHsLrq2F14PrJh5iYmv311eKB1z4ilsXOHLv0JaVjTPvK164pHaHHS49/WekI03L9cF5oDUssWVI6Qo+p/7i/dIRp16z/x9IRWmvZkpUcv//rSseYNrXrEP7wDdH6Jw7nhdYwLLvi1tIRetzxfx9ROkKPAy+4s3SEHl+97c8f93asgqWxM8cufnHpGNPecfGa0hF6fOzo55WOMG2qZc9zxnYd/MX5ME2tfbh0hB6XTvxvO2cOy8ZWcOyyl5aOMW3fvxnO//Qeljv+y8GlI0xbdFW7OnnR/vuXjtAjH1xbOsK0qx+4cM77PC1LkiRJkiSpYg53JEmSJEmSKuZwR5IkSZIkqWIOdyRJkiRJkirmcEeSJEmSJKliDnckSZIkSZIq5nBHkiRJkiSpYg53JEmSJEmSKuZwR5IkSZIkqWIOdyRJkiRJkirmcEeSJEmSJKliDnckSZIkSZIqNu9wJyL2jYjzI+JHEXF9RHw5Ig5pItz2iIgTI+L40jkkbR87R1LT7B1JTbJzJC2EvsOdiAjgIuCyzDwoM58D/CGwTxPhttOJgOUjVcjOkdQ0e0dSk+wcSQtlviN3TgI2ZuanNi/IzO9k5pXRcWZEfDci1kTEqTA92b08Ii6OiNsi4kMR8VsR8e3u1x3U/brzIuKTEXFN9+tOjIhzIuL7EXHe5seLiBdFxNURcUNEXBARK7rLb4+I93WXr4mIQyPiycCbgHdGxE0RccJQf1uSFpqdI6lp9o6kJtk5khbEfMOdZwLXz3HfK4CjgGcBJwNnRsQTuvc9i04JHAa8DjgkM58LnA28dcY6dgeOA94JfBH4CHA4cEREHBURewHvBU7OzKOB64B3zfj+e7vLPwm8OzNvBz4FfCQzj8rMK+f5+SS1i50jqWn2jqQm2TmSFsTiAb73F4DPZOYk8LOIuBw4BlgLXJuZdwFExI+AS7rfs4bOtHqzL2VmRsQa4GeZuab7PbcATwb2B54BXBURAOPA1TO+/8Luf6+nU4bbJCJOB04HWBo7b+u3S2rWaHXO4l229dslNa/q3unpHJZvy7dKKqPqzuk+jq+vpELmG+7cArxqO9b76IzPp2bcntriMR+d5Wtmft0kcGlmvmaex5lkOwZVmXkWcBbAyrE9c1u/X9LQ7Tids3RfO0dqh5HtnZmds+vYHnaO1A4j2zmwxXOdRXvZO1KD5jst65+AnboTWAAi4sjuuZZXAqdGxKKIWAW8APj2kPNdAzw/Ig7uPvbOMf+V5B8C/F/iUp3sHElNs3ckNcnOkbQg+g53MjOBU4CTo/NWfbcAfwbcTecq7zcD36FTUu/JzLuHGS4z7wFOAz4TETfTOWTw0Hm+7UvAKV7wS6qPnSOpafaOpCbZOZIWSnT6RSvH9sxjl76kdIxpX7ntmtIRerz0+JeVjjAt128oHaFHLFlSOkKPqf+4v3SEades/0cenLw3Sudoo5VL983j939d6RjTpnZt1/U41j+xPefpL7vi1tIRetzxe0eUjtDjwAvuLB2hx1dv+/PrM3N16Rxts+vYHnns4heXjjHtHbeuKR2hx8eOfl7pCNOmWvY8Z2zXFaUj9Jha+3DpCD0unfjfds4cVi7aK49d9tLSMabt+0+LSkfoccd/Obh0hGmLrmpXJy86cP/SEXrkg2tLR5h29QMX8uDGe2Z9fTXfaVmSJEmSJElqMYc7kiRJkiRJFXO4I0mSJEmSVDGHO5IkSZIkSRVzuCNJkiRJklQxhzuSJEmSJEkVc7gjSZIkSZJUMYc7kiRJkiRJFXO4I0mSJEmSVDGHO5IkSZIkSRVzuCNJkiRJklQxhzuSJEmSJEkVc7gjSZIkSZJUscWlA7RGBIy1Z9b1K089tnSEHmtftl/pCNOe8NZ/Kx2hx8Pv2K10hB6LSgeY6e5WpWmVnJhg8qd3lI4xLRa368/Boj0PLx2htTauyNIRetz/3CeUjtDrttIB2ipgUXs6+SOHPLN0hF6rn1Q6wbTFd91fOkKPf/3QqtIRehz8l5tKR+j17dIB2itzipyYKB1j2l2/UDpBr59/bkPpCNP2/8FepSP0mDhg99IRekw9Zc/SEaZNfWunOe9rzzRDkiRJkiRJ28zhjiRJkiRJUsUc7kiSJEmSJFXM4Y4kSZIkSVLFHO5IkiRJkiRVzOGOJEmSJElSxRzuSJIkSZIkVczhjiRJkiRJUsUc7kiSJEmSJFXM4Y4kSZIkSVLFHO5IkiRJkiRVzOGOJEmSJElSxRzuSJIkSZIkVczhjiRJkiRJUsUc7kiSJEmSJFVscekAWysivgAcACwFPkpnMHVQZp7Rvf80YHVmviUiXgu8DRgHvgX858ycLBJcUrXsHUlNsnOk/7+9e4+x9K7LAP58d7ftbm8rvUlJLFZRKQo06VILCHIpiRcUCg0FjUlB2aBgI9giJEYNAUvEYDCgsCAXEwMVakERK1qogCKl0NLWFiGxoqVgqVYu6XV3f/4xZ2f31J1225153/d35vNJJp1zzuSc58x0njnnmXfOMiSdA4ulpyN3XtRaOy3JtiwVyyVJztrn8nOSvK+qTpm9/8TW2qlJdiX5+aHDAgtB7wBD0jnAkHQOLJBujtxJcl5V7Smb70lycpJ/q6ozknw5ySOT/GOSlyY5LclnqypJtiS5ZX9XWFXbk2xPks11xJqGB7q0qr0z1zk5fM3DA93ROcCQ1vb5ld6BQXUx7lTVU5KcmeTxrbXbq+ryLB0++L4kz0vyxSSXtNZaLTXOe1prr76/622t7UiyI0m2bjyurVF8oENr0Tv7ds7RG47ROcCyte+cY3UOsGyI51ce68CwevmzrK1JbpsVzyOTnDE7/5Ikz0rygiwVUZJcluTsqjohSarqmKp6+NCBge7pHWBIOgcYks6BBdPLuHNpkk1VdUOS1yf55yRprd2W5IYkD2+tXTE77/okv5nko1V1TZK/S3LiKKmBnukdYEg6BxiSzoEF08WfZbXW7krykytc9sz9nHdRkovWOhewuPQOMCSdAwxJ58Di6eXIHQAAAAD2w7gDAAAA0DHjDgAAAEDHjDsAAAAAHTPuAAAAAHTMuAMAAADQMeMOAAAAQMeMOwAAAAAdM+4AAAAAdMy4AwAAANAx4w4AAABAx4w7AAAAAB0z7gAAAAB0bNPYASZl48axE+xVNXaCOVsvvX7sCMu2vmL32BHm3Hja0WNHmPPQv/nm2BH2mtb/xtPSkrZz59gplk0pS5JsuuxzY0dYNq3GST7/ojeNHWHOc07+sbEjcKB2t7ET7NUm9p115XQe5+w6ZFoPz994+sfGjjDnbbc+Y+wIHKCqDalDDx07xrLdt98+doQ5J5795bEjLNu9ZcvYEea85O0Xjx1hzlt/8bljR1hWbeWf5Y7cAQAAAOiYcQcAAACgY8YdAAAAgI4ZdwAAAAA6ZtwBAAAA6JhxBwAAAKBjxh0AAACAjhl3AAAAADpm3AEAAADomHEHAAAAoGPGHQAAAICOGXcAAAAAOmbcAQAAAOiYcQcAAACgY8YdAAAAgI4t7LhTVS+pqqtnbzdW1cfHzgQsLp0DDE3vAEPSOTBtCzvutNbe2lo7NcnjktyU5I0jRwIWmM4BhqZ3gCHpHJi2TWMHGMCbknystfZX976gqrYn2Z4km+uIoXMBi+nAOieHD50LWFz77R2dA6wRz69gghZ63Kmqc5M8PMnL9nd5a21Hkh1JsnXjcW24ZMAieiCdc3Qdo3OAg3ZfvTPXORuO1TnAQfP8CqZrYcedqjotyflJntRa2z12HmCx6RxgaHoHGJLOgWlb2NfcydKafEySj89e9OsdYwcCFprOAYamd4Ah6RyYsIU9cqe19sKxMwDrh84BhqZ3gCHpHJi2RT5yBwAAAGDhGXcAAAAAOmbcAQAAAOiYcQcAAACgY8YdAAAAgI4ZdwAAAAA6ZtwBAAAA6JhxBwAAAKBjxh0AAACAjhl3AAAAADpm3AEAAADomHEHAAAAoGObxg4wKbt2jZ1g2e7bbx87wpzaNJ3/VU7aMq3PzRVba+wIc9o3vzV2hL0m9D0Fi+KeTOv7qjb6PVEXWku75+6xU0xXm873VbtrOlmS5GePmNbjrj+6+etjR+BAbajUls1jp9jrjjvGTjBn43HHjh1h2T0nP3TsCHN++7qTxo4w54TDpvNYp9XKzz2nkxIAAACAB8y4AwAAANAxlNzsMgAAC9pJREFU4w4AAABAx4w7AAAAAB0z7gAAAAB0zLgDAAAA0DHjDgAAAEDHjDsAAAAAHTPuAAAAAHTMuAMAAADQMeMOAAAAQMeMOwAAAAAdM+4AAAAAdMy4AwAAANAx4w4AAABAx4w7AAAAAB0z7gAAAAB0bNPYAcZUVduTbE+SzXXEyGmARTfXOTl85DTAotM5wNDmemfDkSOngfVlXR+501rb0Vrb1lrbdmhtHjsOsOD27ZxDctjYcYAFp3OAoc09v9rg+RUMaeHHnap6aVVdPXt72Nh5gMWmc4Ah6RxgaHoHpmnh/yyrtfaWJG8ZOwewPugcYEg6Bxia3oFpWvgjdwAAAAAWmXEHAAAAoGPGHQAAAICOGXcAAAAAOmbcAQAAAOiYcQcAAACgY8YdAAAAgI4ZdwAAAAA6ZtwBAAAA6JhxBwAAAKBjxh0AAACAjhl3AAAAADpm3AEAAADomHEHAAAAoGObxg4wFa3tTrv77rFjLNtwxBFjR5hTR04nz1/fdPLYEebs2jx2gnn1kK1jR9jrjo1jJ5i2DRP6/OzeNXaCObVpOj+e2s6dY0eY86/3TOdzkyR11FFjR5h3x9gBOCBVYyeYVxP6fWfbPXaCOTft/M7YEeZsOPzwsSPMu3PsANPVdu7Krv/+n7FjTNau2/537AjL6pZbx44w57ozPjd2hDk/ff6zx46wbMPdKz8undBPMgAAAAAeKOMOAAAAQMeMOwAAAAAdM+4AAAAAdMy4AwAAANAx4w4AAABAx4w7AAAAAB0z7gAAAAB0zLgDAAAA0DHjDgAAAEDHjDsAAAAAHTPuAAAAAHTMuAMAAADQMeMOAAAAQMeMOwAAAAAdM+4AAAAAdGxy405Vvb6qXrrP6d+pqguq6g1VdV1VXVtV58wue0pVfXifj31zVZ27z/VcX1XXVNXvD35HgC7oHGBIOgcYmt6B9WFy406Si5I8b5/Tz0tyS5JTkzw2yZlJ3lBVJ650BVV1bJKzkvxwa+0xSV67wsdtr6orq+rKe9pdq5Uf6Ms4nROdA+uUzgGGpndgHZjcuNNauyrJCVX1sKp6bJLbslQ8722t7Wqt/VeSf0jyuPu4mm8muTPJn1TVc5LcvsJt7WitbWutbTukDlvdOwJ0YbTOic6B9UjnAEPTO7A+TG7cmXl/krOTnJOlpXklOzN/HzYnSWttZ5LTk3wgyTOTXLo2MYEFoXOAIekcYGh6BxbcVMedi5I8P0sF9P4kn0xyTlVtrKrjkzw5yRVJvpLkUVV1WFV9V5KnJ0lVHZlka2vtI0lenqXDDQFWonOAIekcYGh6BxbcprED7E9r7V+q6qgkX22tfa2qLkny+CRfSNKSvLK19vUkqao/T3JdkhuTXDW7iqOSfKiqNiepJK8Y+j4A/dA5wJB0DjA0vQOLb5LjTpK01h69z/styQWzt3t/3CuTvHI/V3H62qUDFo3OAYakc4Ch6R1YbFP9sywAAAAADoBxBwAAAKBjxh0AAACAjhl3AAAAADpm3AEAAADomHEHAAAAoGPGHQAAAICOGXcAAAAAOmbcAQAAAOiYcQcAAACgY8YdAAAAgI4ZdwAAAAA6tmnsAFNRtSG1ZcvYMZbt/va3x44w7447x06wbOuFDxs7wpxPX/TmsSPM+ZkdPzF2hL12t7ETTNvuXWMnmKy2c+fYESbr9MMOGTvCnF233jp2BA5EJbVpOg/7pvc9vnvsAHu1af3s/O6N03l8nEzwMTIruvvEI/KfL37C2DGWnXThFWNHmLPxhOPHjrDXxmkd83HK23507Ahzvu+e/xg7wl738TNiWl9FAAAAAB4Q4w4AAABAx4w7AAAAAB0z7gAAAAB0zLgDAAAA0DHjDgAAAEDHjDsAAAAAHTPuAAAAAHTMuAMAAADQMeMOAAAAQMeMOwAAAAAdM+4AAAAAdMy4AwAAANCxhRx3quqEqvr7qrq2qq6sqkeMnQlYbHoHGJLOAYakc2D6FnLcSbIpyfmttUcneXuSV42cB1h8egcYks4BhqRzYOI2jR1gLbTWbk5y8+zkYUnuHDEOsA7oHWBIOgcYks6B6VvIcWePqjo1ya8ledrYWYD1Qe8AQ9I5wJB0DkzXQo87Sd6Z5NzW2r/v78Kq2p5ke5JsriMGjAUssBV7Z65zcvjAsYAFpXOAIR3w86tNWx8yYCxgUV9zZ49HtNY+sdKFrbUdrbVtrbVth9bmIXMBi2vF3tm3cw7JYUPnAhbTgXVO6RxgVRzw86uNh/vlOQxp0cedF44dAFh39A4wJJ0DDEnnwEQt+rjz62MHANYdvQMMSecAQ9I5MFELPe601p4wdgZgfdE7wJB0DjAknQPTtdDjDgAAAMCiM+4AAAAAdMy4AwAAANAx4w4AAABAx4w7AAAAAB0z7gAAAAB0zLgDAAAA0DHjDgAAAEDHjDsAAAAAHTPuAAAAAHTMuAMAAADQMeMOAAAAQMeMOwAAAAAdq9ba2Bkmoaq+keQrq3BVxyW5dRWuZzVMKUsiz/1ZxDwPb60dvxphFs2Cdk4iz32ZUpZkcfPonf3QOYORZ2VTypLonDW3oL0zpSyJPPdnSnnWvHOMO6usqq5srW0bO0cyrSyJPPdHHh6MqX2d5FnZlLIk8vDgTO3rJM99m1KeKWVJppeHlU3pazWlLIk892dKeYbI4s+yAAAAADpm3AEAAADomHFn9e0YO8A+ppQlSXZU1Xf2PaOqzq2qN6/GlVfV5VX1/w51q6p3V9WNVXX17O3UPXlW43ZXkTw8GFP7Ok0uz0i9U1X1uqr6UlXdUFXnZYKfm7ED3MvU8rB/U/s6TS7PSJ3zyX0e59xcVR/ck2c1bneVTClLMr08rGxKX6spZUnG65ynV9XnZ53zqap6xJ48q3G7q2hKedY8i9fcYVBV9Z3W2pH7nD43ybbW2stW4bovT3J+a+3Ke53/7iQfbq194GBvA+jPSL3zwiRPTXJua213VZ3QWrvlYG8PmL4xOudeH3Nxkg+11v70YG8PmL6RHud8KcmzWms3VNWvJDm9tXbuwd4eB8eRO0xGVR1fVRdX1Wdnb0+cnX96VX26qq6qqn+qqh+anb+lqt43+634JUm2jHoHgO6sYe/8cpLXtNZ2J4lhB0jW/rFOVR2d5GlJPnhfHwesD2vYOS3J0bP3tya5ec3vDPdr09gBWHe2VNXV+5w+Jslfzt5/U5I/aK19qqpOSvK3SU5J8sUkT2qt7ayqM5P8bpLnZunJ0+2ttVOq6jFJPn8ft/u6qvqtJJcleVVr7a7VvVvAhI3RO9+f5JyqOivJN5Kc11r78qrfM2CKxnqskyTPTnJZa+1bq3h/gGkbo3N+KclHquqOJN9Kcsaq3yseMOMOQ7ujtbbnNW+WDxucnTwzyaOqas/FR1fVkVlag99TVT+QpZX4kNnlT07yh0nSWrumqq5Z4TZfneTrSQ7N0t86/kaS16zWHQImb4zeOSzJna21bVX1nCTvTPKk1btLwISN0Tl7vCDJO1bjTgDdGKNzXp7kp1prn6mqC5K8MUuDDyMy7jAlG5Kc0Vq7c98zZy8I9vHW2llV9b1JLn8gV9pa+9rs3buq6l1Jzj/4qMCCWJPeSXJTkr+YvX9JkncdXExgQaxV56SqjktyepKzDj4msCBWvXOq6vgkj22tfWZ21kVJLl2VtBwUr7nDlHw0ya/uOVF7/1WrrUm+Onv/3H0+/hNJfm72sT+S5DH7u9KqOnH238rS4crXrWZooGtr0jtZer2Lp87e//EkX1qduEDn1qpzkuTsLP0DEnfex8cA68tadM5tSbZW1Q/OTj8jyQ2rF5kHy7jDlJyXZFtVXVNV1yd5yez830tyYVVdlfmjzf44yZFVdUOW/szqcytc759V1bVJrk1yXJLXrkl6oEdr1TuvT/LcWfdcGIcqA0vWqnOS5PlJ3rsGmYF+rXrntNZ2Jnlxkour6gtJfiHJBWt4HzhA/il0AAAAgI45cgcAAACgY8YdAAAAgI4ZdwAAAAA6ZtwBAAAA6JhxBwAAAKBjxh0AAACAjhl3AAAAADpm3AEAAADo2P8BKBRDru4fM7IAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 1152x576 with 8 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light",
            "tags": []
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "translate(\"How are you?\", plot='decoder_layer4_block2')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IS_i_pxJcl2S"
      },
      "source": [
        "## Results\n",
        "\n",
        "As a proof of concept the results were encouraging. We have to keep in mind that the model used here was a scaled-down version of the model specified in the paper. They used 6 encoder layers and 6 decoder layers. We used 4 and 4. They used an embedding size of 512 - we used 128. They used 2048 nodes in the FFNs - we used 512. We did this to save on computational cost.\n",
        "\n",
        "Based on these results, I have no doubt that increasing these hyper-parameter to match or exceed those used in the paper would cause the accuracy to improve greatly.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zyPAT60p63FS"
      },
      "source": [
        "## Future Work\n",
        "\n",
        "* Expand the model to match or exceed the model referenced in the original paper \"[Attention is all you need](https://arxiv.org/pdf/1706.03762.pdf)\".\n",
        "* Experiment with other varients of the transformer such as GPT-2 or BERT.\n",
        "* Ultimately to build a full two way language translation model using voice recognition, allowing two or more people speaking different languages to comunicate using their native languages."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ESPszhY63FT"
      },
      "source": [
        "## Summary\n",
        "\n",
        "We built a `Transformer` based on the paper \"[Attention is all you need] We built it from scratch using the `Tensorflow` framework. We did this because it provides an opportunity to learn all of the hidden details of the model.\n",
        "\n",
        "So what did we learn?\n",
        "\n",
        "In Seq2Seq models\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gNG3bVHm63FT"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.8"
    },
    "toc": {
      "base_numbering": 1,
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": true,
      "title_cell": "Table of Contents",
      "title_sidebar": "Contents",
      "toc_cell": true,
      "toc_position": {},
      "toc_section_display": true,
      "toc_window_display": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}