BERT_Fine_Tuning_Sentence_Classification.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "BERT Fine-Tuning Sentence Classification.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "machine_shape": "hm"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jNKaJz5j_ylj",
        "colab_type": "text"
      },
      "source": [
        "# Tutorial on  Fine-Tuning BERT for Classification Tasks"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yKtxAbMFGnu_",
        "colab_type": "text"
      },
      "source": [
        "# Running the notebook\n",
        "\n",
        "This notebook is shared in `view only mode`. If you want to run the cells inside it, you should either:\n",
        "\n",
        "- click on ` open in playground mode` in the `file` menu (your change won't be saved though)\n",
        "- save a copy in your drive that you can open in `edit` mode to be able to save your changes."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BJR6t_gCQe_x",
        "colab_type": "text"
      },
      "source": [
        "## Introduction\n",
        "\n",
        "## BERT\n",
        "\n",
        "[BERT](https://arxiv.org/abs/1810.04805), is the model we will use in this tutorial to provide readers with a better understanding of and practical guidance for using transfer learning models in NLP. BERT is a method of pretraining language representations that was used to create models that NLP practicioners can then download and use for free. You can either use these models to extract high quality language features from your text data, or you can fine-tune these models on a specific task (classification, entity recognition, question answering, etc.) with your own data to produce state of the art predictions.\n",
        "\n",
        "### This Tutorial\n",
        "\n",
        "This post will explain how you can modify and fine-tune BERT to create a powerful NLP model that quickly gives you state of the art results. \n",
        "\n",
        "### Fine-Tuning BERT\n",
        "\n",
        "In this tutorial, we will use BERT to train a text classifier. Specifically, we will take the pre-trained BERT model, and add a classification layer on top of it. Fine-tuning is helpful for the following reasons: \n",
        "\n",
        "- Easy Training: The pre-trained BERT model, already encode a lot of information about our language. As a result, it takes much less time to train our fine-tuned model - it is as if we have already trained the bottom layers of our network extensively and only need to gently tune them while using their output as features for our classification task. \n",
        "\n",
        "- Less Data: Fine-tuning tallows us to use a much smaller dataset in comparison with a model that is built from scratch. A major drawback of NLP models built from scratch is that we often need a  large dataset in order to train our network to reasonable accuracy. By fine-tuning BERT, we are now able to get away with training a model to good performance on a much smaller amount of training data.\n",
        "\n",
        "- Good Performance: Fine-tuning procedure has shown to achieve state of the art results with minimal task-specific adjustments for a wide variety of NLP  tasks.\n",
        "\n",
        "- A Shift in NLP: This shift to transfer learning resembels the same shift that pretrained ImageNet models had on computer vision.\n",
        "\n",
        "\n",
        "\n",
        "Okay, let's get started!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RX_ZDhicpHkV",
        "colab_type": "text"
      },
      "source": [
        "## Install and Import"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nSU7yERLP_66",
        "colab_type": "text"
      },
      "source": [
        "With Google Colab you can have access to free GPUs and TPUs! In casae we have a  large neural network it's best to take advantage of this (in this case we'll attach a GPU). For this tutorial you can also use CPU, but it takes a long time!!\n",
        "\n",
        "A GPU can be added by going to the menu and selecting:\n",
        "\n",
        "Edit -> Notebook Settings -> Add accelerator (GPU)\n",
        "\n",
        "Then run the following cell to confirm that the GPU is detected."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DEfSbAA4QHas",
        "colab_type": "code",
        "outputId": "6a72f926-e844-4e55-ae82-c11e54c8d9fd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 94
        }
      },
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "device_name = tf.test.gpu_device_name()\n",
        "if device_name != '/device:GPU:0':\n",
        "  raise SystemError('GPU device not found')\n",
        "print('Found GPU at: {}'.format(device_name))"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Found GPU at: /device:GPU:0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ElsnSNUridI",
        "colab_type": "text"
      },
      "source": [
        "Next, let's install the pytorch interface for BERT by [Hugging Face](https://github.com/huggingface/pytorch-transformers)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0NmMdkZO8R6q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install pytorch-transformers"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ok002ceNB8E7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 74
        },
        "outputId": "e1bc4f7c-eed4-480b-df55-7a92696fce78"
      },
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.nn import CrossEntropyLoss\n",
        "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "from sklearn.model_selection import train_test_split\n",
        "from tqdm import tqdm, trange\n",
        "import pandas as pd\n",
        "import io\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "% matplotlib inline"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rXL_A3kBIl-m",
        "colab_type": "text"
      },
      "source": [
        "**Importing some classes from pytorch_transformers**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oPmPFyvXSAZO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from  pytorch_transformers import BertTokenizer, BertConfig\n",
        "from pytorch_transformers import AdamW, WarmupLinearSchedule\n",
        "from pytorch_transformers.modeling_bert import BertLayerNorm\n",
        "\n",
        "from pytorch_transformers import BertModel\n",
        "from pytorch_transformers import cached_path"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cqG7FzRVFEIv",
        "colab_type": "text"
      },
      "source": [
        "In the case of using GPU, we need to identify and specify the GPU as the device. Later, in our training loop, we need to load data onto the device. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oYsV4H8fCpZ-",
        "colab_type": "code",
        "outputId": "fbb4ae47-56ec-40a3-fb76-e122c372a264",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 74
        }
      },
      "source": [
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "n_gpu = torch.cuda.device_count()\n",
        "torch.cuda.get_device_name(0)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Tesla K80'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "guw6ZNtaswKc",
        "colab_type": "text"
      },
      "source": [
        "## Loading Dataset\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_9ZKxKc04Btk",
        "colab_type": "text"
      },
      "source": [
        "We'll use [CoLA](https://nyu-mll.github.io/CoLA/) dataset for single sentence classification. It's a set of sentences labeled as grammatically correct or incorrect. The data is as follows:\n",
        "\n",
        "**Column 1**:\tthe code representing the source of the sentence.\n",
        "\n",
        "**Column 2**:\tthe acceptability judgment label (0=unacceptable, 1=acceptable).\n",
        "\n",
        "**Column 3**:\tthe acceptability judgment as originally notated by the author.\n",
        "\n",
        "**Column 4**:\tthe sentence."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NTU8Awqv4Eln",
        "colab_type": "text"
      },
      "source": [
        "Download the dataset from this link, extract, and move them to your local drive. Both tokenized and raw versions of the data are available. We will use the **raw** version and use BERT tokenizer to tokenize it. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DOpikSXbsvn-",
        "colab_type": "code",
        "colab": {
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7Ci8vIE1heCBhbW91bnQgb2YgdGltZSB0byBibG9jayB3YWl0aW5nIGZvciB0aGUgdXNlci4KY29uc3QgRklMRV9DSEFOR0VfVElNRU9VVF9NUyA9IDMwICogMTAwMDsKCmZ1bmN0aW9uIF91cGxvYWRGaWxlcyhpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IHN0ZXBzID0gdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKTsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIC8vIENhY2hlIHN0ZXBzIG9uIHRoZSBvdXRwdXRFbGVtZW50IHRvIG1ha2UgaXQgYXZhaWxhYmxlIGZvciB0aGUgbmV4dCBjYWxsCiAgLy8gdG8gdXBsb2FkRmlsZXNDb250aW51ZSBmcm9tIFB5dGhvbi4KICBvdXRwdXRFbGVtZW50LnN0ZXBzID0gc3RlcHM7CgogIHJldHVybiBfdXBsb2FkRmlsZXNDb250aW51ZShvdXRwdXRJZCk7Cn0KCi8vIFRoaXMgaXMgcm91Z2hseSBhbiBhc3luYyBnZW5lcmF0b3IgKG5vdCBzdXBwb3J0ZWQgaW4gdGhlIGJyb3dzZXIgeWV0KSwKLy8gd2hlcmUgdGhlcmUgYXJlIG11bHRpcGxlIGFzeW5jaHJvbm91cyBzdGVwcyBhbmQgdGhlIFB5dGhvbiBzaWRlIGlzIGdvaW5nCi8vIHRvIHBvbGwgZm9yIGNvbXBsZXRpb24gb2YgZWFjaCBzdGVwLgovLyBUaGlzIHVzZXMgYSBQcm9taXNlIHRvIGJsb2NrIHRoZSBweXRob24gc2lkZSBvbiBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcCwKLy8gdGhlbiBwYXNzZXMgdGhlIHJlc3VsdCBvZiB0aGUgcHJldmlvdXMgc3RlcCBhcyB0aGUgaW5wdXQgdG8gdGhlIG5leHQgc3RlcC4KZnVuY3Rpb24gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpIHsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIGNvbnN0IHN0ZXBzID0gb3V0cHV0RWxlbWVudC5zdGVwczsKCiAgY29uc3QgbmV4dCA9IHN0ZXBzLm5leHQob3V0cHV0RWxlbWVudC5sYXN0UHJvbWlzZVZhbHVlKTsKICByZXR1cm4gUHJvbWlzZS5yZXNvbHZlKG5leHQudmFsdWUucHJvbWlzZSkudGhlbigodmFsdWUpID0+IHsKICAgIC8vIENhY2hlIHRoZSBsYXN0IHByb21pc2UgdmFsdWUgdG8gbWFrZSBpdCBhdmFpbGFibGUgdG8gdGhlIG5leHQKICAgIC8vIHN0ZXAgb2YgdGhlIGdlbmVyYXRvci4KICAgIG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSA9IHZhbHVlOwogICAgcmV0dXJuIG5leHQudmFsdWUucmVzcG9uc2U7CiAgfSk7Cn0KCi8qKgogKiBHZW5lcmF0b3IgZnVuY3Rpb24gd2hpY2ggaXMgY2FsbGVkIGJldHdlZW4gZWFjaCBhc3luYyBzdGVwIG9mIHRoZSB1cGxvYWQKICogcHJvY2Vzcy4KICogQHBhcmFtIHtzdHJpbmd9IGlucHV0SWQgRWxlbWVudCBJRCBvZiB0aGUgaW5wdXQgZmlsZSBwaWNrZXIgZWxlbWVudC4KICogQHBhcmFtIHtzdHJpbmd9IG91dHB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIG91dHB1dCBkaXNwbGF5LgogKiBAcmV0dXJuIHshSXRlcmFibGU8IU9iamVjdD59IEl0ZXJhYmxlIG9mIG5leHQgc3RlcHMuCiAqLwpmdW5jdGlvbiogdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKSB7CiAgY29uc3QgaW5wdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoaW5wdXRJZCk7CiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gZmFsc2U7CgogIGNvbnN0IG91dHB1dEVsZW1lbnQgPSBkb2N1bWVudC5nZXRFbGVtZW50QnlJZChvdXRwdXRJZCk7CiAgb3V0cHV0RWxlbWVudC5pbm5lckhUTUwgPSAnJzsKCiAgY29uc3QgcGlja2VkUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBpbnB1dEVsZW1lbnQuYWRkRXZlbnRMaXN0ZW5lcignY2hhbmdlJywgKGUpID0+IHsKICAgICAgcmVzb2x2ZShlLnRhcmdldC5maWxlcyk7CiAgICB9KTsKICB9KTsKCiAgY29uc3QgY2FuY2VsID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnYnV0dG9uJyk7CiAgaW5wdXRFbGVtZW50LnBhcmVudEVsZW1lbnQuYXBwZW5kQ2hpbGQoY2FuY2VsKTsKICBjYW5jZWwudGV4dENvbnRlbnQgPSAnQ2FuY2VsIHVwbG9hZCc7CiAgY29uc3QgY2FuY2VsUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBjYW5jZWwub25jbGljayA9ICgpID0+IHsKICAgICAgcmVzb2x2ZShudWxsKTsKICAgIH07CiAgfSk7CgogIC8vIENhbmNlbCB1cGxvYWQgaWYgdXNlciBoYXNuJ3QgcGlja2VkIGFueXRoaW5nIGluIHRpbWVvdXQuCiAgY29uc3QgdGltZW91dFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgc2V0VGltZW91dCgoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9LCBGSUxFX0NIQU5HRV9USU1FT1VUX01TKTsKICB9KTsKCiAgLy8gV2FpdCBmb3IgdGhlIHVzZXIgdG8gcGljayB0aGUgZmlsZXMuCiAgY29uc3QgZmlsZXMgPSB5aWVsZCB7CiAgICBwcm9taXNlOiBQcm9taXNlLnJhY2UoW3BpY2tlZFByb21pc2UsIHRpbWVvdXRQcm9taXNlLCBjYW5jZWxQcm9taXNlXSksCiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdzdGFydGluZycsCiAgICB9CiAgfTsKCiAgaWYgKCFmaWxlcykgewogICAgcmV0dXJuIHsKICAgICAgcmVzcG9uc2U6IHsKICAgICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICAgIH0KICAgIH07CiAgfQoKICBjYW5jZWwucmVtb3ZlKCk7CgogIC8vIERpc2FibGUgdGhlIGlucHV0IGVsZW1lbnQgc2luY2UgZnVydGhlciBwaWNrcyBhcmUgbm90IGFsbG93ZWQuCiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gdHJ1ZTsKCiAgZm9yIChjb25zdCBmaWxlIG9mIGZpbGVzKSB7CiAgICBjb25zdCBsaSA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2xpJyk7CiAgICBsaS5hcHBlbmQoc3BhbihmaWxlLm5hbWUsIHtmb250V2VpZ2h0OiAnYm9sZCd9KSk7CiAgICBsaS5hcHBlbmQoc3BhbigKICAgICAgICBgKCR7ZmlsZS50eXBlIHx8ICduL2EnfSkgLSAke2ZpbGUuc2l6ZX0gYnl0ZXMsIGAgKwogICAgICAgIGBsYXN0IG1vZGlmaWVkOiAkewogICAgICAgICAgICBmaWxlLmxhc3RNb2RpZmllZERhdGUgPyBmaWxlLmxhc3RNb2RpZmllZERhdGUudG9Mb2NhbGVEYXRlU3RyaW5nKCkgOgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnbi9hJ30gLSBgKSk7CiAgICBjb25zdCBwZXJjZW50ID0gc3BhbignMCUgZG9uZScpOwogICAgbGkuYXBwZW5kQ2hpbGQocGVyY2VudCk7CgogICAgb3V0cHV0RWxlbWVudC5hcHBlbmRDaGlsZChsaSk7CgogICAgY29uc3QgZmlsZURhdGFQcm9taXNlID0gbmV3IFByb21pc2UoKHJlc29sdmUpID0+IHsKICAgICAgY29uc3QgcmVhZGVyID0gbmV3IEZpbGVSZWFkZXIoKTsKICAgICAgcmVhZGVyLm9ubG9hZCA9IChlKSA9PiB7CiAgICAgICAgcmVzb2x2ZShlLnRhcmdldC5yZXN1bHQpOwogICAgICB9OwogICAgICByZWFkZXIucmVhZEFzQXJyYXlCdWZmZXIoZmlsZSk7CiAgICB9KTsKICAgIC8vIFdhaXQgZm9yIHRoZSBkYXRhIHRvIGJlIHJlYWR5LgogICAgbGV0IGZpbGVEYXRhID0geWllbGQgewogICAgICBwcm9taXNlOiBmaWxlRGF0YVByb21pc2UsCiAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgYWN0aW9uOiAnY29udGludWUnLAogICAgICB9CiAgICB9OwoKICAgIC8vIFVzZSBhIGNodW5rZWQgc2VuZGluZyB0byBhdm9pZCBtZXNzYWdlIHNpemUgbGltaXRzLiBTZWUgYi82MjExNTY2MC4KICAgIGxldCBwb3NpdGlvbiA9IDA7CiAgICB3aGlsZSAocG9zaXRpb24gPCBmaWxlRGF0YS5ieXRlTGVuZ3RoKSB7CiAgICAgIGNvbnN0IGxlbmd0aCA9IE1hdGgubWluKGZpbGVEYXRhLmJ5dGVMZW5ndGggLSBwb3NpdGlvbiwgTUFYX1BBWUxPQURfU0laRSk7CiAgICAgIGNvbnN0IGNodW5rID0gbmV3IFVpbnQ4QXJyYXkoZmlsZURhdGEsIHBvc2l0aW9uLCBsZW5ndGgpOwogICAgICBwb3NpdGlvbiArPSBsZW5ndGg7CgogICAgICBjb25zdCBiYXNlNjQgPSBidG9hKFN0cmluZy5mcm9tQ2hhckNvZGUuYXBwbHkobnVsbCwgY2h1bmspKTsKICAgICAgeWllbGQgewogICAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgICBhY3Rpb246ICdhcHBlbmQnLAogICAgICAgICAgZmlsZTogZmlsZS5uYW1lLAogICAgICAgICAgZGF0YTogYmFzZTY0LAogICAgICAgIH0sCiAgICAgIH07CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPQogICAgICAgICAgYCR7TWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCl9JSBkb25lYDsKICAgIH0KICB9CgogIC8vIEFsbCBkb25lLgogIHlpZWxkIHsKICAgIHJlc3BvbnNlOiB7CiAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgIH0KICB9Owp9CgpzY29wZS5nb29nbGUgPSBzY29wZS5nb29nbGUgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYiA9IHNjb3BlLmdvb2dsZS5jb2xhYiB8fCB7fTsKc2NvcGUuZ29vZ2xlLmNvbGFiLl9maWxlcyA9IHsKICBfdXBsb2FkRmlsZXMsCiAgX3VwbG9hZEZpbGVzQ29udGludWUsCn07Cn0pKHNlbGYpOwo=",
              "ok": true,
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "status": 200,
              "status_text": ""
            }
          },
          "base_uri": "https://localhost:8080/",
          "height": 80
        },
        "outputId": "7e522b6b-6c50-4985-a443-607d2dc074b8"
      },
      "source": [
        "# Upload the train file from your local drive; if you get out of memory error, upload by the tab file 'upload'\n",
        "from google.colab import files\n",
        "uploaded = files.upload()"
      ],
      "execution_count": 107,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-b1df99b1-e8a5-4899-b2c6-b51f79c76d7e\" name=\"files[]\" multiple disabled />\n",
              "     <output id=\"result-b1df99b1-e8a5-4899-b2c6-b51f79c76d7e\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script src=\"/nbextensions/google.colab/files.js\"></script> "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_UkeC7SG2krJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df = pd.read_csv(\"in_domain_train.tsv\", delimiter='\\t', header=None, names=['sentence_source', 'label', 'label_notes', 'sentence'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3UfxtwQy3axu",
        "colab_type": "code",
        "outputId": "9d5de975-5efc-44bd-9519-f499a51e153f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 74
        }
      },
      "source": [
        "df.shape"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(8551, 4)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AQfTaYDo42zu",
        "colab_type": "code",
        "outputId": "96276637-fb63-44ec-f297-85cd9f48e896",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        }
      },
      "source": [
        "df.sample(5)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence_source</th>\n",
              "      <th>label</th>\n",
              "      <th>label_notes</th>\n",
              "      <th>sentence</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>6596</th>\n",
              "      <td>g_81</td>\n",
              "      <td>0</td>\n",
              "      <td>*</td>\n",
              "      <td>I find it easy to believe - but Joan finds it ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6466</th>\n",
              "      <td>d_98</td>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>There's everything Mary had asked for in this ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3431</th>\n",
              "      <td>ks08</td>\n",
              "      <td>0</td>\n",
              "      <td>*</td>\n",
              "      <td>Kicked the man the ball.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2398</th>\n",
              "      <td>l-93</td>\n",
              "      <td>0</td>\n",
              "      <td>*</td>\n",
              "      <td>The pitchfork loaded the truck.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1989</th>\n",
              "      <td>r-67</td>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>That the sun was out is obvious.</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     sentence_source  ...                                           sentence\n",
              "6596            g_81  ...  I find it easy to believe - but Joan finds it ...\n",
              "6466            d_98  ...  There's everything Mary had asked for in this ...\n",
              "3431            ks08  ...                           Kicked the man the ball.\n",
              "2398            l-93  ...                    The pitchfork loaded the truck.\n",
              "1989            r-67  ...                   That the sun was out is obvious.\n",
              "\n",
              "[5 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NNpEePGBPDC7",
        "colab_type": "text"
      },
      "source": [
        "### Create sentence and label lists\n",
        "\n",
        "To use BERT, We need to add special tokens at the beginning and end of each sentence for BERT to work properly"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GuE5BqICAne2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sentences = df.sentence.values\n",
        "sentences = [\"[CLS] \" + sentence + \" [SEP]\" for sentence in sentences]\n",
        "labels = df.label.values"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ex5O1eV-Pfct",
        "colab_type": "text"
      },
      "source": [
        "## Inputs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BTREubVNFiz4",
        "colab_type": "text"
      },
      "source": [
        "Next, import the BERT tokenizer, used to convert our text into tokens that correspond to BERT's vocabulary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z474sSC6oe7A",
        "colab_type": "code",
        "outputId": "bb7bbe2c-53fd-4125-fc17-748625bbedef",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 134
        }
      },
      "source": [
        "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)\n",
        "\n",
        "tokenized_texts = [tokenizer.tokenize(sent) for sent in sentences]\n",
        "print (\"Tokenize the first sentence:\")\n",
        "print (tokenized_texts[0])"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 231508/231508 [00:00<00:00, 5684699.73B/s]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Tokenize the first sentence:\n",
            "['[CLS]', 'our', 'friends', 'won', \"'\", 't', 'buy', 'this', 'analysis', ',', 'let', 'alone', 'the', 'next', 'one', 'we', 'propose', '.', '[SEP]']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "87_kXUeT2-br",
        "colab_type": "text"
      },
      "source": [
        "BERT requires specifically formatted inputs. For each tokenized input sentence, we need to create:\n",
        "\n",
        "- **input ids**: a sequence of integers identifying each input token to its index number in the BERT tokenizer vocabulary\n",
        "- **segment mask**: (optional) a sequence of 1s and 0s used to identify whether the input is one sentence or two sentences long. For one sentence inputs, this is simply a sequence of 0s. For two sentence inputs, there is a 0 for each token of the first sentence, followed by a 1 for each token of the second sentence\n",
        "- **attention mask**: (optional) a sequence of 1s and 0s, with 1s for all input tokens and 0s for all padding tokens (we'll detail this in the next paragraph)\n",
        "- **labels**: a single value of 1 or 0. In our task 1 means \"grammatical\" and 0 means \"ungrammatical\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xytsw1oIfnX0",
        "colab_type": "text"
      },
      "source": [
        "Although we can have variable length input sentences, BERT does requires our input arrays to be the same size. We address this by first choosing a maximum sentence length, and then padding and truncating our inputs until every input sequence is of the same length. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Cp9BPRd1tMIo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Set the maximum sequence length. The longest sequence in our training set is 47, but we'll leave room on the end anyway. \n",
        "# In the original paper, the authors used a length of 512.\n",
        "MAX_LEN = 128"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mFhowDMohU4H",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Use the BERT tokenizer to convert the tokens to their index numbers in the BERT vocabulary\n",
        "input_ids = [tokenizer.convert_tokens_to_ids(x) for x in tokenized_texts]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7ajDxuX0AFGL",
        "colab_type": "code",
        "outputId": "1ce3fc63-bc14-4f30-e225-86015c12be50",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 74
        }
      },
      "source": [
        "print(input_ids[0])"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[101, 2256, 2814, 2180, 1005, 1056, 4965, 2023, 4106, 1010, 2292, 2894, 1996, 2279, 2028, 2057, 16599, 1012, 102]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kDs-MYtYH8sL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "input_ids = pad_sequences(input_ids, maxlen=MAX_LEN, dtype=\"long\", truncating=\"post\", padding=\"post\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YQrlzgLsAYpm",
        "colab_type": "code",
        "outputId": "e6722268-a5a7-484d-a69d-b4005d24e663",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 240
        }
      },
      "source": [
        "print(input_ids[0])"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[  101  2256  2814  2180  1005  1056  4965  2023  4106  1010  2292  2894\n",
            "  1996  2279  2028  2057 16599  1012   102     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KhGulL1pExCT",
        "colab_type": "text"
      },
      "source": [
        "Create the attention masks "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cDoC24LeEv3N",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Create attention masks\n",
        "attention_masks = []\n",
        "\n",
        "# Create a mask of 1s for each token followed by 0s for padding\n",
        "for seq in input_ids:\n",
        "  seq_mask = [float(i>0) for i in seq]\n",
        "  attention_masks.append(seq_mask)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jqdk_4oGBIxo",
        "colab_type": "code",
        "outputId": "80c4b24a-2439-4b00-c14d-cdc1489419e1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 74
        }
      },
      "source": [
        "print(attention_masks[0])"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xjc8M4PePh8c",
        "colab_type": "text"
      },
      "source": [
        "Use train_test_split to split our data into train and validation sets for training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aFbE-UHvsb7-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_inputs, validation_inputs, train_labels, validation_labels = train_test_split(input_ids, labels, \n",
        "                                                            random_state=2018, test_size=0.1)\n",
        "train_masks, validation_masks, _, _ = train_test_split(attention_masks, input_ids,\n",
        "                                             random_state=2018, test_size=0.1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jw5K2A5Ko1RF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Convert all of our data into torch tensors, the required type for our pytorch model\n",
        "\n",
        "train_inputs = torch.tensor(train_inputs)\n",
        "validation_inputs = torch.tensor(validation_inputs)\n",
        "train_labels = torch.tensor(train_labels)\n",
        "validation_labels = torch.tensor(validation_labels)\n",
        "train_masks = torch.tensor(train_masks)\n",
        "validation_masks = torch.tensor(validation_masks)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jjcyu-3VDXpV",
        "colab_type": "code",
        "outputId": "a0011126-4c23-42a5-dfae-a897a485d684",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 74
        }
      },
      "source": [
        "print(train_inputs.shape)"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([7695, 128])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lhfEVSZePocX",
        "colab_type": "text"
      },
      "source": [
        "Select a batch size for training. \n",
        "\n",
        "For fine-tuning BERT on a specific task, the authors recommend a batch size of 16 or 32. Create an iterator of our data with torch DataLoader. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GEgLpFVlo1Z-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "batch_size = 16\n",
        "\n",
        "# Training\n",
        "train_data = TensorDataset(train_inputs, train_masks, train_labels)\n",
        "train_sampler = RandomSampler(train_data)\n",
        "train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)\n",
        "\n",
        "# Validation\n",
        "validation_data = TensorDataset(validation_inputs, validation_masks, validation_labels)\n",
        "validation_sampler = SequentialSampler(validation_data)\n",
        "validation_dataloader = DataLoader(validation_data, sampler=validation_sampler, batch_size=batch_size)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pNl8khAhPYju",
        "colab_type": "text"
      },
      "source": [
        "## Creating Classification Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fwQ7JcuJQZ0o",
        "colab_type": "text"
      },
      "source": [
        "Now what we need to do is to add a single classification layer on top of BERT model. As we feed input data, the entire pre-trained BERT's model and the additional untrained classification layer is trained on our specific task. Alternatively, we can freez all layers and just fine tune the classification head.\n",
        "\n",
        "Also note that, as we've showed beforehand, the first token of every sequence is the special classification token (\"[CLS]\"). Unlike the hidden state vector corresponding to a normal word token, the hidden state corresponding to this special token is designated by the authors of BERT as an aggregate representation of the whole sentence used for classification tasks. As such, when we feed in an input sentence to our model during training, the output is the length 768 hidden state vector corresponding to this token. The additional layer that we've added on top consists of untrained linear neurons of size \"[hidden_state, number_of_labels]\"\", so \"[768,2]\". \n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WnQW9E-bBCRt",
        "colab_type": "text"
      },
      "source": [
        "There are a few different pre-trained BERT models available.  We will use \"bert-base-uncased\" , which means the version that has only lowercase letters (\"uncased\") and is the smaller version of the two (\"base\" vs \"large\")."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x6WhYNfjLj7K",
        "colab_type": "text"
      },
      "source": [
        "**Adding a classification head**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3tnHtjq_J0QX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class BertWithClfHead(nn.Module):\n",
        "    def __init__(self, config, adapt_args, bertmodel):\n",
        "        super().__init__()\n",
        "        self.num_labels = adapt_args.num_classes\n",
        "        self.config = config\n",
        "        self.bert = bertmodel\n",
        "        self.dropout = nn.Dropout(config['hidden_dropout_prob'])\n",
        "        self.classifier = nn.Linear(config['hidden_size'], adapt_args.num_classes)\n",
        "    \n",
        "    def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None, position_ids=None, head_mask=None):\n",
        "        outputs = self.bert(input_ids, position_ids=position_ids, token_type_ids=token_type_ids,\n",
        "                            attention_mask=attention_mask, head_mask=head_mask)\n",
        "        \n",
        "        pooled_output = outputs[1] # see note below\n",
        "\n",
        "        pooled_output = self.dropout(pooled_output)\n",
        "        logits = self.classifier(pooled_output)\n",
        "\n",
        "        outputs = (logits,) + outputs[2:]  # add hidden states and attention if they are here\n",
        "\n",
        "        if labels is not None:\n",
        "            if self.num_labels == 1:\n",
        "                #  We are doing regression\n",
        "                loss_fct = MSELoss()\n",
        "                loss = loss_fct(logits.view(-1), labels.view(-1))\n",
        "            else:\n",
        "                loss_fct = CrossEntropyLoss()\n",
        "                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))\n",
        "            outputs = (loss,) + outputs\n",
        "\n",
        "        return outputs  # (loss), logits, (hidden_states), (attentions)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hfOd76l9LSwm",
        "colab_type": "text"
      },
      "source": [
        "Note that **pooled_output**: ``torch.FloatTensor``  is of shape ``(batch_size, hidden_size)``\n",
        "            of last layer hidden-state of the first token of the sequence (clf token)\n",
        "            further processed by a Linear layer and a Tanh activation function. The Linear\n",
        "            layer weights are trained from the next sentence prediction (classification)\n",
        "            objective during Bert pretraining. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cZK8KPL3D2XA",
        "colab_type": "text"
      },
      "source": [
        "Now that we have our model loaded we need to grab the training hyperparameters from within the stored model.\n",
        "For the purposes of fine-tuning, the authors recommend the following hyperparameter ranges:\n",
        "Batch size: 16, 32\n",
        "Learning rate (Adam): 5e-5, 3e-5, 2e-5\n",
        "Number of epochs: 2, 3, 4"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oG1W4xLcNyzG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from collections import namedtuple\n",
        "fine_tuning_config = namedtuple('fine_tuning_config',\n",
        "  field_names=\"num_classes, lr, n_epochs, n_warmup\")\n",
        "adapt_args = fine_tuning_config(2, 2e-5, 5,10  )"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zkgSNdsYGGN-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 74
        },
        "outputId": "a7417077-1fe9-4db1-a711-3c6951225396"
      },
      "source": [
        "bertconfig = cached_path(\"https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json\")"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 313/313 [00:00<00:00, 174530.33B/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AUls98jLIJuH",
        "colab_type": "code",
        "outputId": "8a46c4f2-473b-489a-e7a0-07df22df40d0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 74
        }
      },
      "source": [
        "import json\n",
        "with open(bertconfig, \"r\") as read_file:\n",
        "    bertconfig = json.load(read_file)\n",
        "print (bertconfig)\n"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'attention_probs_dropout_prob': 0.1, 'hidden_act': 'gelu', 'hidden_dropout_prob': 0.1, 'hidden_size': 768, 'initializer_range': 0.02, 'intermediate_size': 3072, 'max_position_embeddings': 512, 'num_attention_heads': 12, 'num_hidden_layers': 12, 'type_vocab_size': 2, 'vocab_size': 30522}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yxE7-GdiPON2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = BertWithClfHead(bertconfig, adapt_args, BertModel.from_pretrained('bert-base-uncased'))\n",
        "model.cuda()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sx07PEjf-9k1",
        "colab_type": "text"
      },
      "source": [
        "Note that alternatively you can build your model by using the built in function in pytorch-transoformers, i.e., ```BertForSequenceClassification```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HnRUNK01CetV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# alternatively you can use this\n",
        "# from pytorch_transformers.modeling_bert import BertForSequenceClassification\n",
        "# model = BertForSequenceClassification.from_pretrained(\"bert-base-uncased\", num_labels=2)\n",
        "# model.cuda()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8o-VEBobKwHk",
        "colab_type": "text"
      },
      "source": [
        "**optimizer**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zix3NG5VKmtO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Parameters:\n",
        "max_grad_norm = 1.0\n",
        "\n",
        "### In PyTorch-Transformers, optimizer and schedules are splitted and instantiated like this:\n",
        "optimizer = AdamW(model.parameters(), lr=adapt_args.lr, correct_bias=False)  # To reproduce BertAdam specific behavior set correct_bias=False\n",
        "scheduler = WarmupLinearSchedule(optimizer, warmup_steps=adapt_args.n_warmup, t_total=len(train_dataloader)*adapt_args.n_epochs)  # PyTorch scheduler\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_QXZhFb4LnV5",
        "colab_type": "text"
      },
      "source": [
        "Below is our training loop. For each pass in our loop we have a trianing phase and a validation phase. At each pass we need to have:\n",
        "\n",
        "**Training loop**:\n",
        "-  Setting the model in train mode\n",
        "- Unpack our data\n",
        "- Load data onto the GPU (Optional)\n",
        "- Zeroing out the gradients calculated in the previous pass. In pytorch the gradients accumulate by default \n",
        "- Forward pass \n",
        "- Backward pass (backpropagation)\n",
        "- Tell the network to update parameters with optimizer.step()\n",
        "- Track variables for monitoring progress\n",
        "\n",
        "**Evalution loop**:\n",
        "- Setting the model in evaluation mode\n",
        "- Unpack our data \n",
        "- Load data onto the GPU (Optional)\n",
        "- Forward pass \n",
        "- Compute loss on our validation data and track variables for monitoring progress\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9cQNvaZ9bnyy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Function to calculate the accuracy of our predictions vs labels\n",
        "def predict_accuracy(preds, labels):\n",
        "    pred_flat = np.argmax(preds, axis=1).flatten()\n",
        "    labels_flat = labels.flatten()\n",
        "    return np.sum(pred_flat == labels_flat) / len(labels_flat)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6J-FYdx6nFE_",
        "colab_type": "code",
        "outputId": "6a4a54c1-4d05-4257-c40e-367e1367c1a9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 374
        }
      },
      "source": [
        "import sys\n",
        "# Store our loss and accuracy for plotting\n",
        "train_loss_set = []\n",
        "\n",
        "# Number of training epochs (authors recommend between 2 and 4)\n",
        "epochs = adapt_args.n_epochs\n",
        "\n",
        "# trange is a tqdm wrapper around the normal python range\n",
        "for _ in trange(epochs, desc=\"Epoch\"):\n",
        "  \n",
        "  \n",
        "  # Training\n",
        "  \n",
        "  # Set our model to training mode (as opposed to evaluation mode)\n",
        "  model.train()\n",
        "  \n",
        "  # Tracking variables\n",
        "  tr_loss = 0\n",
        "  nb_tr_examples, nb_tr_steps = 0, 0\n",
        "  \n",
        "  # Train the data for one epoch\n",
        "  for step, batch in enumerate(train_dataloader):\n",
        "    # Add batch to GPU\n",
        "    batch = tuple(t.to(device) for t in batch)\n",
        "    # Unpack the inputs from our dataloader\n",
        "    b_input_ids, b_input_mask, b_labels = batch\n",
        "    # Clear out the gradients (by default they accumulate)\n",
        "    optimizer.zero_grad()\n",
        "    # Forward pass\n",
        "#     loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)\n",
        "    outputs = model(input_ids=b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels, position_ids=None, head_mask=None)\n",
        "    loss = outputs[0]\n",
        "    train_loss_set.append(loss)    \n",
        "    # Backward pass\n",
        "    loss.backward()\n",
        "    torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)\n",
        "    # Update parameters and take a step using the computed gradient\n",
        "    scheduler.step()\n",
        "    optimizer.step()\n",
        "    \n",
        "    \n",
        "    # Update tracking variables\n",
        "    tr_loss += loss\n",
        "    nb_tr_examples += b_input_ids.size(0)\n",
        "    nb_tr_steps += 1\n",
        "\n",
        "  print(\"Train loss: {}\".format(tr_loss/nb_tr_steps))\n",
        "    \n",
        "    \n",
        "#   Validation\n",
        "\n",
        "  # Put model in evaluation mode to evaluate loss on the validation set\n",
        "  model.eval()\n",
        "\n",
        "  # Tracking variables \n",
        "  eval_loss, eval_accuracy = 0, 0\n",
        "  nb_eval_steps, nb_eval_examples = 0, 0\n",
        "\n",
        "  # Evaluate data for one epoch\n",
        "  for batch in validation_dataloader:\n",
        "#     # Add batch to GPU\n",
        "    batch = tuple(t.to(device) for t in batch)\n",
        "    # Unpack the inputs from our dataloader\n",
        "    b_input_ids, b_input_mask, b_labels = batch\n",
        "    # Telling the model not to compute or store gradients, saving memory and speeding up validation\n",
        "    with torch.no_grad():\n",
        "      # Forward pass, calculate logit predictions\n",
        "      outputs = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask)\n",
        "    \n",
        "    # Move logits and labels to CPU\n",
        "    logits = outputs[0]\n",
        "    logits = logits.detach().cpu().numpy()\n",
        "    label_ids = b_labels.to('cpu').numpy()\n",
        "\n",
        "    tmp_eval_accuracy = predict_accuracy(logits, label_ids)\n",
        "    \n",
        "    eval_accuracy += tmp_eval_accuracy\n",
        "    nb_eval_steps += 1\n",
        "\n",
        "  print(\"Validation Accuracy: {}\".format(eval_accuracy/nb_eval_steps))"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\rEpoch:   0%|          | 0/5 [00:00<?, ?it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Train loss: 0.48965132236480713\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\rEpoch:  20%|██        | 1/5 [06:48<27:14, 408.50s/it]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Validation Accuracy: 0.7974537037037037\n",
            "Train loss: 0.23865172266960144\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\rEpoch:  40%|████      | 2/5 [13:36<20:25, 408.39s/it]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Validation Accuracy: 0.8090277777777778\n",
            "Train loss: 0.1420905590057373\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\rEpoch:  60%|██████    | 3/5 [20:23<13:36, 408.05s/it]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Validation Accuracy: 0.8032407407407407\n",
            "Train loss: 0.08685380965471268\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\rEpoch:  80%|████████  | 4/5 [27:10<06:47, 407.52s/it]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Validation Accuracy: 0.8148148148148148\n",
            "Train loss: 0.0559784360229969\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\rEpoch: 100%|██████████| 5/5 [33:56<00:00, 407.03s/it]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Validation Accuracy: 0.8148148148148148\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iyRa-5CcHv_g",
        "colab_type": "text"
      },
      "source": [
        "## Training Evaluation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1-G03mmwH3aI",
        "colab_type": "text"
      },
      "source": [
        "Let's take a look at our training loss over all batches:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "68xreA9JAmG5",
        "colab_type": "code",
        "outputId": "31757f5c-6bcf-49a5-a667-bb9dc0a94ddf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 533
        }
      },
      "source": [
        "plt.figure(figsize=(15,8))\n",
        "plt.title(\"Training loss\")\n",
        "plt.xlabel(\"Batch\")\n",
        "plt.ylabel(\"Loss\")\n",
        "plt.plot(train_loss_set)\n",
        "plt.show()"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA30AAAHwCAYAAAASOwaWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xm8XVV9N/7PurkhwQEsklYFbLBq\nC06tUsRqf9qqLaDFWrWKQ+tIfVpan59Wi1oREQVFwaEIpFSQISKgliEQhpBASEjITYCQeZ6Tm+Qm\nd57vWc8f5+xz9tlnD2vtvdaezufdlyX3nH3WXnte370mIaUEERERERERlVNH1hkgIiIiIiIiexj0\nERERERERlRiDPiIiIiIiohJj0EdERERERFRiDPqIiIiIiIhKjEEfERERERFRiTHoIyKitiCEmCaE\nGBRCvNzksjHycZkQ4ibT6RIREQXpzDoDREREfoQQg64/nwdgDMBU7e9/klLeppOelHIKwAtML0tE\nRJR3DPqIiCiXpJT1oEsIsQPAZ6WUjwQtL4TolFJOppE3IiKiImHzTiIiKqRaM8lfCSF+KYQYAPBx\nIcRbhBDLhBC9Qoj9QoifCCGm15bvFEJIIcTs2t+31r5/QAgxIIR4Ughxqu6yte/PEUJsEkL0CSF+\nKoRYIoT4pOJ2vF8IsbaW50eFEH/o+u5rQoh9Qoh+IcQGIcQ7ap+fJYRYVfu8WwhxpYFdSkREJcWg\nj4iIiuz9AOYCOB7ArwBMAvgCgBMBvBXA2QD+KeT3HwXwDQAnANgF4Nu6ywohfhfAHQC+XFvvdgBn\nqmReCHEagFsA/CuAWQAeAXCPEGK6EOI1tby/UUp5HIBzausFgJ8CuLL2+SsB3KWyPiIiak8M+oiI\nqMiekFLeK6WsSClHpJQrpJTLpZSTUsptAOYAeHvI7++SUnZJKScA3Abgj2Ms+14Az0gp7659dzWA\nw4r5/wiAe6SUj9Z+ewWqAeybUQ1gZwJ4Ta3p6vbaNgHABIBXCSFeLKUckFIuV1wfERG1IQZ9RERU\nZLvdfwgh/kgIMU8IcUAI0Q/gUlRr34IccP17GOGDtwQt+zJ3PqSUEsAehbw7v93p+m2l9tuTpJQb\nAXwJ1W04WGvG+pLaop8CcDqAjUKIp4QQ5yquj4iI2hCDPiIiKjLp+ft6AGsAvLLW9PFiAMJyHvYD\nONn5QwghAJyk+Nt9AH7f9duOWlp7AUBKeauU8q0ATgUwDcDltc83Sik/AuB3AfwQwK+FEDOTbwoR\nEZURgz4iIiqTFwLoAzBU6y8X1p/PlPsAvFEI8TdCiE5U+xTOUvztHQDOE0K8ozbgzJcBDABYLoQ4\nTQjxF0KIGQBGav+rAIAQ4hNCiBNrNYN9qAa/FbObRUREZcGgj4iIyuRLAP4R1cDpelQHd7FKStkN\n4MMArgLQA+APADyN6ryCUb9di2p+rwVwCNWBZ86r9e+bAeD7qPYPPADgdwB8vfbTcwGsr41a+gMA\nH5ZSjhvcLCIiKhFR7XpAREREJgghpqHabPODUsrFWeeHiIiINX1EREQJCSHOFkK8qNYU8xuojq75\nVMbZIiIiAsCgj4iIyIS3AdiGahPNvwbwfillZPNOIiKiNLB5JxERERERUYmxpo+IiIiIiKjEGPQR\nERERERGVWGfWGdB14oknytmzZ2edDSIiIiIiokysXLnysJRSdU7Y4gV9s2fPRldXV9bZICIiIiIi\nyoQQYqfO8mzeSUREREREVGIM+oiIiIiIiEqMQR8REREREVGJMegjIiIiIiIqMQZ9REREREREJcag\nj4iIiIiIqMQY9BEREREREZUYgz4iIiIiIqISY9BHRERERERUYgz6iIiIiIiISoxBHxERERERUYkx\n6CMiIiIiIioxBn1EREREREQlxqCPiIiIiIioxBj0ERERERERlRiDPiIiIiIiohJj0EdERERERFRi\n1oI+IcTPhRAHhRBrAr7/mBBitRDiOSHEUiHEG2zlhYiIiIjUrNp1FLMvmodnd/dmnRUiMsRmTd9N\nAM4O+X47gLdLKV8H4NsA5ljMCxEREREpWLjhIADgsU2HMs4JEZnSaSthKeXjQojZId8vdf25DMDJ\ntvJCRERERETUrvLSp+8zAB4I+lIIcYEQoksI0XXoEN86ERERERERqco86BNC/AWqQd9/BC0jpZwj\npTxDSnnGrFmz0sscERERERFRwVlr3qlCCPF6ADcAOEdK2ZNlXoiIiIiIiMoos5o+IcTLAfwGwCek\nlJuyygcREREREVGZWavpE0L8EsA7AJwohNgD4JsApgOAlPI6ABcDeDGAnwkhAGBSSnmGrfwQhdlz\ndBjP7O7Fe1//sqyzQkRERERklM3RO8+P+P6zAD5ra/1EOv72mqU4PDjGoI+IiIiISifzgVyI8uDw\n4FjWWSAiIiIisoJBHxERERERUYkx6CMiIiIiIioxBn1EREQUy0NrD2D2RfOw5eBg1lkhIqIQDPqI\niIgolgfWHAAArN7Tm3FOiIgoDIM+IiIiIiKiEmPQR0REREREVGIM+oiIiIiIiEqMQR+Ri5Qy6ywQ\nERERERnFoI+IiIiIiKjEGPQRERERUR0bvRCVD4M+Ihc+6IiIiIiobBj0EREREVGdEFnngIhMY9BH\nRERERERUYgz6iFzYupOIiIiIyoZBHxERERHVsX87Ufkw6CMiIqJYOLcpEVExMOgjcmEBhoiI2h0H\nciEqHwZ9REREFItgdEBEVAgM+oiIiIiIiEqMQR+RCxt3EhEREVHZMOgjIiKiWNgPmoioGBj0ERER\nERERlRiDPiIXvrQmIlLHgVyIiIqBQR8REREREVGJMegjIiKiWNinj4ioGBj0EblIjt9JRERERCXD\noI+IiIhiYZ8+IqJiYNBHRERERERUYgz6iFzYPYWIiIiIyoZBHxEREcXCgVyIiIqBQR8REREREVGJ\nMegjIiKiWDiQCxFRMTDoIyIiIiIiKjEGfURERBQL+/QRERUDgz4iF5ZfiIj0sZUnEVG+MegjAMCS\nLYcxNDaZdTaIiKiA+MKMiCjfGPQR9vWO4GM3LMeX7ng266wQEVGBcCAXIqJiYNBHGB6v1vBtPjiQ\ncU6yJ8HX1URERERULgz6iM1yiIgoFg7kUk48rETlw6CPyIUPOiIiIiIqGwZ9REREFAv79JUTDytR\n+TDoozo+vImIiIiIyodBH3HoEhfuCyIidezTR0RUDAz6iIiIKBE2FCkXxvJE5cOgj4iIiBJhkEBE\nlG8M+qiOL2rZVImISAf7gpcTDytR+TDoozqGO0RERERE5cOgj4iIiGJh6wgiomJg0Ed1bM3B2k4i\nojjYHLBcGMsTlQ+DPiIiIkqEQQIRUb4x6CMiIqJYOJBLOfGwEpUPgz7iG1oX7gsiInXs00dEVAwM\n+qiOb/aIiCgOPj+IiPKNQR8RERElwgq/cuHxJCofBn1EbnzQEREpY58+IqJisBb0CSF+LoQ4KIRY\nE/C9EEL8RAixRQixWgjxRlt5oXCSkQ4REcXAPn3lxFieqHxs1vTdBODskO/PAfCq2v8uAHCtxbwQ\nERGRJQwSiIjyzVrQJ6V8HMCRkEXeB+BmWbUMwIuEEC+1lR8iFaz1JCIiIqKyybJP30kAdrv+3lP7\njDIiwFe1RESkj608iYjyrRADuQghLhBCdAkhug4dOpR1dkqLtVxERKSDA7kQERVDlkHfXgCnuP4+\nufZZCynlHCnlGVLKM2bNmpVK5qg98W01EZE6DuRCRFQMWQZ99wD4h9oonmcB6JNS7s8wP22PzTuJ\niCgOVvgREeVbp62EhRC/BPAOACcKIfYA+CaA6QAgpbwOwP0AzgWwBcAwgE/ZyguF44taIiJKgs8R\nIqJ8sxb0SSnPj/heAvgXW+snioPlFiIidezTR0RUDIUYyIWIiIjyh336iIiKgUFfSvpHJzAxVck6\nG6H4wpaIiOLg84OIKN8Y9KXk9Zc8hP9z66qss0ER+NaaiIiIiMqGQV+KHlnfnXUWiIiIjOP7snLh\n8SQqHwZ9xJs7ERHFwoFciIiKgUEfkQvjX4rj0Q3dWLHjSNbZIEodm8SXE2N5ovJh0EdEbWvDgX7M\nvmgelm/rSZTOp2/qwoeue9JQroiKh0ECEVG+Megjorb1xObDAIAH17K/LREREZUXgz4iF7ZUIiLS\nx3tnufB4EpUPgz4iIiIiIqISY9BHREREibBPX7nweBKVD4M+guSYlXXcF0RERERUNgz6iIiIKBH2\nASMiyjcGfURERIYs3XIYH5nzJKYq7REFcXL2cmIQT1Q+nVlngPKDD29wdnYiSuTfbn8ahwfHcWRo\nHLNeOCPr7FjHydmJiIqBNX3EN3pERJQI3xmWC48nUfkw6CMiIiIiIioxBn1ELqz0JCLSxxYjRET5\nxqCP6tiag4iIiBjEE5UPgz4iIiJKhH3AiIjyjUGfYVJKHB4cyzobFBPfbrYnFliJiBp4TyQqHwZ9\nht26fBfOuOwRbOoeyDorRKSIwT4RERGVGYM+wxZvOgQA2HZoMOOc6OObPSIiioMvToiI8o1Bn2FO\n4FTEB2AR82ya5PidRETU5lgeICofBn2GidoYmLxfEhUHa7mJkuE1RESUbwz6DCvyg6/IeSdKgm+1\niYgaWB4gKh8GfZawEFlMPG5ERPp47yQiyjcGfYZ1CKd5J5+AREXBt9pERA0M4onKh0GfaQUcyKVI\neSWygdcAmdKu5xJfnJQTDytReTDos6RNn/uFx+NGRERUxWciUXkw6DOMb8WIioe1FGQKzyUqA57H\nROXDoM8SWcA2PrzJExERERGVD4M+w0QBIycOOtNQxGCdkuNhJyJq4D2RqHwY9BnmhHy8YRIRERER\nUR4w6DPMqehz156x9ogo3wpYQU9EZA3viUTlw6DPsCLfJ0Whc28G43MiIiIiKhsGfYY5ffqKGDyw\nbx8RERERUfkw6DPMr09fEQNAStczu3tRqfBEISKi7LHcQlQ+DPosKeL9ks07s9G14wj+9poluPax\nrVlnhYiIiIhKiEGfaYybSNPe3hEAwPr9/RnnpP3wbTYRUSsO5EJUPgz6DHNqy9wjdrJcSURERERE\nWWHQZ1h9yoYCRXpFyqtt3BfthW+ziYiIqB0w6DOsyGVIFoCJiIiIL0CJyodBnyVFmpw937kjIiIi\nIqIkGPQZVsTmndSQxVyF9bkdU18zERFRK7b8ISofBn2G1QdyyTgfOvJeE0lERERERPEx6DPM7+0Y\nQyrKq6mKbOtJ4fm+g4iIiNoBgz5LilSYdGf1QN8olm45nFlespbFccuyFc1rvjkf7/jBogxzQERE\neVOkMgwRqWHQZ1iR28ELAGf/+HF89IblWWeFUjI6UcGuI8NZZyMzRb5eKdy+3hE8sq4762wQFdoT\nmw/j4MBo1tkgIgMY9Bnn9Olzj96ZVV709Q5PZJ0FotQU6dokPef91xP47M1dWWeDqJCcF2JP7TiC\nD133ZLaZISIjGPQZ5twoi9RNigXfBu6K9sQKv/I5PDiedRaISmFnT/u2BiEqEwZ9hrHwSLEx4swM\ndz0RERGVGYM+W1zVZ1nM/aYn7/krN/Yryw73PRFRK7YAIiofBn2G1SdnzzYbofb1jmD2RfPwzO7e\nrLOSO1FzFk5MVTivYYnwUBIREVE7YNBnWH1y9hwXJhdvPgQAmLt8J4B85zVPpioSr/r6A7j0vnVZ\nZ4WIiMgatoIgKh8GfYb5Ts5elKCKd/lQk5UKAOC2ZbuspJ//ZsDlw1OeiIiI2gGDPkvy0gSwf3QC\nX7nrWQyOTQYuk4+c5gP3RXvJyWVKRJQrvDcSlY/VoE8IcbYQYqMQYosQ4iKf718uhFgohHhaCLFa\nCHGuzfykoUM48/Tlw/WPbcUdXXvwi6U76p/xZk7UjBV+REREVGbWgj4hxDQA1wA4B8DpAM4XQpzu\nWew/AdwhpfwTAB8B8DNb+UlbXgKrqWqLxNBmbHnJK1FWeAlQVu5YsRsHB0azzkZsvHaIiIrBZk3f\nmQC2SCm3SSnHAdwO4H2eZSSA42r/Ph7APov5SVXUg7B/dCKVJqDOOoRPXYb3M9Z2hAfADI6JSJVK\nH93u/lF85der8blfdKWQIyJ17O9MVD42g76TAOx2/b2n9pnbJQA+LoTYA+B+AP/ql5AQ4gIhRJcQ\nouvQoUM28mqMyo1yy8EBvP6Sh3BH1+7IZQ/0jWLP0eHY+XGKHR1+A8zUvlUNPqWUmKow8jGtCCO+\nlh3LN5SFiVpTjMOD4xnnJD5eO0RExZD1QC7nA7hJSnkygHMB3CKEaMmTlHKOlPIMKeUZs2bNSj2T\ncbgDKW9hflP3IABg4YbWAHZnz1DTb8+6fAHe9r2FsfNRqQVpJt7aff7WlfiDr92fPCEiIiLKLb6E\nJCofm0HfXgCnuP4+ufaZ22cA3AEAUsonAcwEcKLFPFnn14zSy7mZegOxvb0jePuVi3DFAxtafrN+\nf3+s/DRq+oKbd6re2x9c2x0rD8XCJx0RxdduheU221wiosKyGfStAPAqIcSpQohjUB2o5R7PMrsA\nvBMAhBCnoRr05bv9ZgQntlJ58HvjsJHxKQDA9Y9va1n2nB8vjpWfSruVQAqonfpOSCmx4/BQ1tkg\nso+33syNT1awcufRrLNBRJQL1oI+KeUkgAsBPAhgPaqjdK4VQlwqhDivttiXAHxOCPEsgF8C+KTM\nywR3MTnld3cnfm+H/uAO/uY33dmb7pq+Qu/gDNXPzDYK0kz71YrdeMcPFuGp7UeyzgqRFe30EgfI\n9+3w2/etwweuXYotBwezzkrhtNt5TNQOOm0mLqW8H9UBWtyfXez69zoAb7WZhzyqN+9M4XFZH73T\nZ1VhtZJSSog2vOsX+5VD/j2zuxcAsPXQIM489YSMc0NEZbau1i2id7i4A+UQEZmS9UAupdPRoTES\noyemshFwhPXp4/QEelSGYCciAtiigoqNZQCi8mHQZ1ijeWeD9+aZXuPORp8+d8znDf/8ghne74mI\nKAqfFURExcCgz5LwWjRnwnT13/iZv2Y/5q85oJQPodCnrw1bc7YIOwS233zyzSoRERER2WC1T19b\n0gicvH3mdJsPfv7WVQCAHVe8J3CZSr3/oN/66ytuUQ1MGQW62YrJuJeJqKh4/yonvgQmKh/W9Bk0\nNjmFwdFJAN7RO5vJkEDMNKdW0a9PX+jvbGSGqIa1mlR2PMeJiChPWNNn0IeuexKr9/QBiGjeCf8R\nNa0M5FKfsiF4PX6rbdcCi0qzXGvrZqhNZFy7jkScFht3rS/c/jQmKxLXfPSNFlInFe1aBiAqM9b0\nGeQEfF7eYCGopi/uTXZv7wge3dDt+53fQC5BWCxKT6UisW5ff9bZaHsMtInMMDkF0d3P7MO81fuN\npUdERAz6rFGpFUrap89x7o8X49M3dfnnw2ddKjWMLAy3MrlHrn1sK879yeL6vHUA8ODabizdetjg\nWoqlf3QC1yzcgkqF5x6la/m2HgyPT2adjULjM6NcWDlOVD4M+iz5wUObMDY55ftdUDwYt6avb2Qi\n8LtKwEihUUw17Zh90Tz8y9xVZhJLQVoFl+dqtcL7e0eaHq4f/e/lqaw/a34FikvvXYcrH9yIRzcc\nTC8frN9ue/v7RvDhOcvw5TtXG01X515iu+m4TbyCiIiKgUGfRWtrzfdaBnKp/TeVh2W9T59ryoaW\nPn12CxxlaaZT4HJZITiDII1PVTLOCZVJ1HU7NFY97zYcYHNrIiIqLwZ9FgXX6KU3fGdYnz7vZxzs\nIILrsN39zN7AfpQUTxanH5ukUR4U+d7LK6ic+JKTqHwY9FnVete85ckd6O4fBdDatMzK6J21/6pM\n2eBuYuT8s29kAg+va5/gRvUYfOH2ZwL7UWqtL3EKRJQnLCwTEVEeccoGi5zxKNyFgG/cvTZweRu1\nDvXJ2UNivrCBXC6cuwqLNx/Gsq++07V8ew6Bzlqh8mGfvvJTvWpNX93tEvzxCiqnNnzEE5Uea/oy\nlM48fU7zzug7uPDp97ezZxgAmgalsT244oodR/DrlXvsriQnhOv/e0kpcd/qfRid8B8QqMjC50NM\nMR8M5MkwFpaJiCiPGPQZsnBj64iDUYXXlnn6zGWnJQ/udd24ZHvkesPy4m4GOjA6gc3dA7Hz5+dD\n1z2JL935rNE0VdkKRg70jeJgrVlvU5ohv3lyaw8unPs0rnhgQ/wVExm2cmf7vJShaHxtUk7tUlNN\n1E4Y9BnyqRtXtHzmDKKS5VPRqclwv33efHBQPx3XNrg35x9//hTeffXjMXNXLEkO41mXL8CZ312g\n9RtnKo4Dfa3BIpnB5p36PnBtdi9l4oieDsHOOZCnMvMty3Zif9+I1XXwWiIiyjcGfRlyArGJ2hD1\nNuZqqtRGvw+vwWr9MiwvFdd3q3b1Bi5HpIOjdxKZd7B/FN/43zW+LyZN4rVERJRvDPosim7eKTB/\nzX686usPYFP3gJVHphOg6abtXd5dIC9zs4+wgkvaEyiXeDfnDvthUVlN1jphOy0HTOOlU068JxKV\nD4M+i2Q94PIvvgsBPLi2Oh3Cc3v6rE7ZEBaw+PbpY8Sh5GD/KD58/ZM4MjRubR18+NrH8728eGiJ\niIgY9FmlUthwauI6OlR/EbI+zWaaXmGxhTuZSpuWkP22+n+WbMfy7UfwqxW7tdJyB3IM6pql2UyM\n/ZDIlrRbBmSlPbay/bTJ6UvUVhj0WRTZvFM0pj/oECLxTdZ3vj2fuQJdOah96ZdYI48q6ykLldE7\n0w7Syry/3bIIwNgPiep4KiTCFyhERPnGoM8ip0AZXGgXRt8G+6VUiWhiGpyWXnNQoiJjbStlqQy1\ngnyBQkSUbwz6LFJ5jkt3TZ+NPNT+OzA6iZ7BsYBl/JqFNv/Xza95ZxkKLVFYqCmvNjh92xaPrV18\nX1JOfBFGVD4M+iyK6vtWbd5Z69NnoHnnsm09LcGX03z04rvX4k2XPRKZHxVhzUjJvHZ7+PJcojTZ\nur50zmNR4IuclysRUTEw6LNIev7rJeCu6UteW/axG5bj/ucONOdBI02/Cdh9yyLt+pS3tN3FLe7p\ny2vZNq/5ovZQhpYS7NNXLiU4JYnIg0GfTUoDucjav80079xzdLg5CxpNTJs/U5ucvb589GrKqW03\nnKgY2Cw7HdzPRET5xqDPIpWpDSqumj6TxiansOfosNb0Ck0TsIcs5z+vn70H/paDg9jfN2ItfbfQ\n0TtTyUEbY0UBtakiN+8sbs6JiNpLZ9YZKIPblu/0/dwJ6IICIuEavdNEnz6gEbj9+52rce+z+/DG\nl78oclnd9frOB6i4XBzvuuoxAMCOK95jJL240m7uUsbmNUo1z4G/lYUuHBMFKUPzTioX3mqJyoc1\nfQb0DI77fj5VCX+QNw3k0mG2ecyj67sBAONTlVi/DyuDRGyWUhpp6Rkcw9d/+xzGJqeyzkpsfPgC\nv1qxC6d+9X50949mnRUqGNX7UA5uV4XE/UZEVAwM+gyYOd1/N0aO3olGACUgrDw9w2K++rQMft+F\nztMXPMVD83LZ+8689bht+S484BngJojudiclZbGbdqXl1yv3AgC2Hx6ykn4eXlBQ+yrDPYADuZQL\n74lkypGhcdy4ZDtbNOQAm3cacOz0ab6fOzV9gaN3ugZvEXZiPq2LrOmhrdmpzz8QzP4Cn4o5OX2e\n5GA3psI5+/Jw3hDFFef0LcM5X+R7LBHZ8///6hk8tukQ/nT2CXjtScdnnZ22xpo+A2YEBH0qg6g4\nD3thqE9fa/rB3zX69EX30XP/7de8M681fY1AInlavmlovtzWCsL54jw13NdkWgniOCW8dIgoTO/I\nBABgImZ3IzKHQZ8BM2MGfe4+fbbe9E4lTNd/mj61NPNQ6HGaTSn361EYvVO5RtTHrcv8B/3RzUu7\nYgGT8i7OC4QiN+/kbaqcCnxKElEABn0GBDfvrP43dFAU1zImmsd4+1VUFEZd8R95M/q7MHuODuPw\n4Fj0gh4LNxzU/k2RPBpj+8r08A3blrCCL5uOlcudXbuxZMvhrLMBgC8SvKSUsabIYZ8+IqJ8Y9Bn\nQOBALlGjd0I0avogrdTshNX0hT2ivYVs97K+k7N7Pnrb9xbiz654VCGHzZZvP6L9mzAmiyEmamOn\ndVTPlTIFckS6vnzXanzshuWprCurGnPvPXRobBLv/OEiPL3raDYZUnTjkh14y+WPYuOBgayzQhli\nSxOi8mHQZ0CS5p3OIpWK2WYyTq1J2LQRzje+/fE8n8mQ74pANct+y330v5fhygc3KP1+qiLRV2u/\n7mf6tNbmpoz/iMrv2T292HpoCN+br3YvycrSrT0AgJ09eiPlsjaeiCjfGPQZENi8U2HkSCcw7Bka\nwy+X7zKeN5XmnXU+0YfqiIqmHvjGCw4hg9WoWrq1B9cs3KoU7F5yz1q84VsPYXTCf17AaR3qIR4L\nUa1M930q4gsMsqMMI2gSEREFYdBnQPzmnY2apf/49XOYv1ZtLjkdagO56I7eKXFn124cdE2Undfy\nUlr9TJxY5H+frs4nNx4wStX0afqXXLv0lQltbpzT84vIhDIEnO1yn2oX7IJAXr9YugNr9vZlnQ1K\ngEGfATM6/Wv61u0fCG0i4x690xTvNAze2OPaRVsby4ak4y2EuP88ODCGL9+1Gp+7uavxfazcpke5\neWfC4+Ecz46AJ6ZOTR+1Ml0QYcGm/LKqMS9BHEchDvSN4kPXLcWRofGss0KUim/esxbv/ekTWWeD\nEmDQZ8Cxx/gHfb98ahfefuWiwIjjvxdvx9DYpL2MoTWovGHxNqXfhRVYRsarTRd7CvCwqxfqbc3T\n5+FU7gbFEk6fPp1gg808icqtyFM2OLK+T41OTGFzd3qDz8x5fBtW7DiK36zak9o608SXFkTlw6DP\ngKCBXFRs6h40mJMGpxAx6anq8ytcqN3cGws5E2x2umqt8tA8aWR8qqVJbdpFqaiaW7+avqDyXrs2\nl8rBqUSUqjzcP+PKy13qy3etxruvfjx0IC2TGq1qUlkdEVFiDPoMmNmZ393o7VYY1rrw8EDwvHru\nB9t9q/cDADpd/dOMPfdcCekUhCanKjjt4vm45N61plbv853CnIcRi3R2qJ8rWb85zxPuCYor6pos\nQy1blvJybT5ZG3V0bNJ/EC1TY+IFAAAgAElEQVTTeNYQUdHkN1opkM6IwTmyfCh6p2zwK984S2w7\n3Oh/GDZlw29rg5U01/Sp5WdobBLr9/crLavzBnViqrrwr1bs9k8rpaPQmHfRX6cn6lbJV7vW+Pnh\nnqCi0LnjlCHwzP4+JTPJR1lfzpXglCQiDwZ9JdcS9Ck+EL0PMr8ArHOa/lPhczd34ZwfL25pdtpY\nr/+/ozj59T6odJvgjIxP4fCgf42nWp++WtAXsPC0GPusXWRRyGDTLMqDIjfvzIu0d2HZgyKekkTl\n05l1BsgulSkbwiZnd95A+73NnOZuqqj4gHhq+xHlxasFIcUgtZagd9RMJ8hVfX597IblAIAdV7yn\ndR1OmiFZipoWMSh/7Sbs7bjfdywUU1x5PnPKdFpnXePlrJ0DJBMR+WNNXwriPNhNFXLDmneqvKkM\ny8d0d/NOU5Ozu9ank6JTw+bdpKzexgblfVrsgXQIKN6UDQvWd+OD1y6NnLOTGtbs7cP+vpGssxFb\nnOu5DM07s+Y8O9z7Mo2XRbx/E1FRMOgz5JefO8toemmUEZ2HlW/NSsCybnGad4al56U6h+Gzu3vR\n3V9tkmmz8BRWgLjigQ2eZf2X03kLnXVh4kDfqLU5qHRrOG3tCtv7+MK5T6Nr51GMpjS4RBm896dP\n4C2XP5r6ek2fCjoBRxlqsrNutVBviZHS+soeqJd881J3dGgcNyzeVoprnYqLQZ8hpifdNj1pu66w\nydkd092jd2pmN6hm0J2Oaprvu2YJ3nXVYwCCH/i6+atUJA72j0Yud/3javMeOrwFBalSR5rCw3d/\n3wjO/tHj6HZt81mXL8CbLnvY/spr1DbTf6m7n9mL+WsOAAD6RiZwyT1rtUbxs12A43M+O2kXslhY\nzkajS0LjszQCM17apOIrv16Ny+atx6pdvVlnhdoYgz5Dwmq94jR9TKOcEjbISaP/WlifPtGyfDW9\n5FMbxBY0kIvmMfjRgs0487sLmpqZaaUQsPCJLzhGKx/6K45n7vJd2HBgoGX006IEK1+4/Rl8/taV\nAIAfPLgRNy3dgd+s2qv8e1uBAQMAUlGGWqPM+/T5XMM2A37niBXlHqmrrNuVFWf+yImAQeyI0sCg\nz5DpGvOvqYhb0+cUHkwXIXybd3a4a/oaC5hqmhqrb4zCJyoWbTwIAPVmo7r58RaAjpsZPGZSUMGk\nBOXATEzWTsCsa8upGNK4V0b/prjnal5uU6nvwbxsOBGRIqVIRQjxB0KIGbV/v0MI8W9CiBfZzVqx\nTO80+wSIWwaIyoXf936rqvf3C8lIUItW7+AxflS2L86b46A35rr709bzPO9FuzyUPcNGk7WlDDUt\nFI+1/qIqy+TgeksqN5tQf2Y1PkqneWdu9gARUSjV6qlfA5gSQrwSwBwApwCYay1XBTQ9ZIL2OA/2\n7Gspotfvju3cSysFfZrpqwqapy+u5qBXZ2AGjXUYSCOpooQ8RY3Nsr6a21na+z7OOVqGlw55Gcgl\nLfXpgEp6cXtPycGxSazceSSbzBCREapBX0VKOQng/QB+KqX8MoCX2stW8Zhu3pn0OZL49zL87+pn\n0vf7sLkBo/Kl0jdw95FhbD885Pudex68V//nA5i7fJfSelvU+zLGExXIaRWQUixLzX1qZ3ory5Ei\nN69rV9sODWJ8Mnn/mDwc+zzkoeicfZjWnixBnK7l/9y6Eh+49kn0j05knRUiikk1UpkQQpwP4B8B\n3Ff7bLqdLBVTWPPOOA+hNGv6wgocYQO5NNeBNf4y17yz1VRF4s+/vxCfvPGpWjrNS7mPgrdA+PfX\nP4l/vm1l9Irh30k/ySGp78ecF+7cfRht8jufylDbkRUpJR5ae0Dp2iuDw4Nj+MsfPoaL716TdVYC\n5fxSj+Xw4FjgHIpZN3N01p73e2xReHfjmr19AIAJAy9aiCgbqkHfpwC8BcB3pJTbhRCnArjFXraK\np9N0TV/C+2rg1AWq6/f+7fNDd2D6wHMHsPXQYPVzpeadAZ9HBFnOyFc7e4ZreWj+PjBukBJPbT+C\n+587EJm30HQUBRU8wvpPZipHAVfYaLK22Ao409qr89ccwAW3rMQczSlEimpgdBIAsGxbT+Syubi+\nAmQdKOk647JHMplDUUXaxzk/d8x08eUcUXEFDynoIqVcB+DfAEAI8TsAXiil/J7NjBXNMSF9+uLw\nq+kbnZjCzOnTQn8XdT9u6uQe8thqad4Zscw371kLANhxxXtCm3c2fhtvGe9HrTUbAQO5RK4tMBex\n0ohuxho/ME5KSonvzFuP977hZZbWYI+1AXbyHBkoODhQraHd1+tfC0PBsjzy//DzaouFMhSkg54n\nl967DgOjE7jyQ29IJR9pH8+i3zuIVPA8LwfV0TsXCSGOE0KcAGAVgP8WQlyl8LuzhRAbhRBbhBAX\nBSzz90KIdUKItUKIwg4OEzpPX4yLxS/o+8sfLNJOpyUvPo/Eo0PjLZ9dNm9dZFpBTVDVBnKp7pdD\nA2Oez/37Cfp975eHoLKTidE7G/3xogWtz//z8MyZLg5WJHDDE9vxdz9bYjhli7QOYP4eTmk+MD9/\ny0oj9wqyz2mxUOYC1c+XbMedK/dYX0/ataZh89yWgfdZWtLNJEVlPc/bjWr11PFSyn4AfwfgZinl\nmwG8K+wHQohpAK4BcA6A0wGcL4Q43bPMqwB8FcBbpZSvAfB/NfOfG2Gjd8bhd33t6xtNnq4n4bHJ\nKVxyb2uAt3jzYc/v1K941T59d63cgz/9ziN4bk+f/zIK63p2d69yvuIwfaNzktMZyMX0vTavBcx6\nP8os1m2reWdKNTju1cxfewDbAgY6KoP7n9uP3UeG1X+geEKZvyzyeZ2l6WM3LEttXc5jJ63bW9aj\nlWalPbeaTOAdMXuqkUqnEOKlAP4ejYFcopwJYIuUcpuUchzA7QDe51nmcwCukVIeBQAp5UHFtHNn\nekhNXxy2BnLxxmOqo9/55SZJTR8ksGrXUQDAs3v8A7eo5p0X370GH57TXKgwdRT8CutRb5JnXzQv\ncNnGW+HoJqv130Tk0SQ+yO2y/bDLaRxvxT/ftqreLNKEPOy7MjTv9Ls/LtkS3efSYAYykYPTxwqV\nEbypfZTx8M9fcwCzL5qHwbHJrLOSGtWg71IADwLYKqVcIYR4BYDNEb85CcBu1997ap+5vRrAq4UQ\nS4QQy4QQZyvmJ3fCHtpxbpZxb7DXPbY19PvDg43mlBUp8eRWtYeyzoTZKgGrhMQLZlS7lLovuKaB\nXDz59br5ydbpBYKbd8bbobFvdGW8Q+ZAGQrHRF55rX0vEifoTKuZZ9mbdwbhLTiZop4vJu5ReTt1\nfvTIJgDArh6NliMFpzqQy50A7nT9vQ3ABwyt/1UA3gHgZACPCyFeJ6VsqvoRQlwA4AIAePnLX25g\ntfkXFDjt6x3BccdOrwdMXvUh9xWurtuW78JttXnsfPPQVGPXmp9Effok8MKZ1Vk/BgLm/alIic/c\n1NX8u4h0OwKeSLq3K9NTNsRR0GdDfCHnrN6+yNujpbgP+jKIDgLKfXDS2rrcNHdMrXkn+ZFSomdo\nHCe+YEbWWSGDynyXLNooykmoDuRyshDit0KIg7X//VoIcXLEz/YCOMX198m1z9z2ALhHSjkhpdwO\nYBOqQWATKeUcKeUZUsozZs2apZLlwgsqJP7ZFY/i/dcoDMBh4Bwen6r4Bj8I+QxQrelDPXA9OjyB\nLQcHfRfac7T5DUzU26b9faP4zarWQQO0B3LxaY6pk0bQot40lm87guXbj4TnRX21SvJ+eytTrUf9\n2KVVEE2xJFqm42SDzu7x1mAPjU363scoWFanY1kLjHHvJbev2I0zLnsE6/b1m81QSbCmND/aseWQ\navPOGwHcA+Bltf/dW/sszAoArxJCnCqEOAbAR2ppuP0vqrV8EEKciGpzz8JONHXmqScYS6siZVMf\nMbfNfgGSBWMTldDHWdBDdlKppq+xzNzlu/Cuqx5racopEe+i/OIdz2r/xst5ax33ce7dN41BSpq/\nuGXZTty0dEfMtSTTjje8rFUq0niwdKBv1Fof4Lwpa6Dp3a5v3L0GX7zjWazceTSjHOnLOviRnv9a\nV7t/lvSUbKF67T2xpToInDNvL5VDmc/zMm+bl2rQN0tKeaOUcrL2v5sAhFa5SSknAVyIal/A9QDu\nkFKuFUJcKoQ4r7bYgwB6hBDrACwE8GUpZYo9v8362rmnGUurknBydhPVQ2NTU/V/mxjIRUpZ/04C\nmPRspDPhcmP51s1I+9ps7mMYf+1J8n3f6v3av7ng5q7AlwZFvsEVNUx1zp1XfO1+fMtntNy49vWO\n4KzLF+BHj0R1sTYvi/PIxjrzeD1091dHah4eTz7AgKlAOe8Bd9r5K+q9SFXQXL25acZLqcr6pY4N\n7XgmqwZ9PUKIjwshptX+93EAkcGZlPJ+KeWrpZR/IKX8Tu2zi6WU99T+LaWUX5RSni6lfJ2U8vb4\nm5I9kyfQROKoL7mxiUYe1u9vbaoRdAsIyvozrukVpAQmpsJvIhUpjTWF0L5dhYy2qba+oMA3VnLa\nHlrXrbV8epV+wSsKK0zkvLwZScrGuWSyZtcJDvpG/PvFmjQ8PomxyfAXQbbFqdFMvS9ujPUVodY9\nqq921sFAvaavqR+2/YNf8FuTvvyfqmRB0Z/BVKU0kAuATwP4KYCrUb3HLQXwSUt5KqzAQURiXCwT\nU9kHfe4+fRffvbbl+6AH6lTA52Ou6SEkZMs2Ck+asv6pe51RufZ3/3P6tWVA8xQXWn36Apt36qRh\n5y6r+8aub3gC/aMTOOWE5xlZexJqZeMcPZ1SLiDZXN3pFz+I33+xiXMgPhtH1tbZkodrvZq2taRz\nxdnOtEfvJApTqUg8tvlQrh5LVNWO17Dq6J07AZzn/kwI8X8B/MhGporK5Ak0MZn9HcJd0+cn6MWv\n6jx9UYGt9Kvpi7lb3H1jjg6N43eef0zo8s5qTfeVKmIB7F1XP4ZDA2PYccV7ss5KYUkU89h77XQN\nbV0NVNJ9ahah76LJ50A2tWcBrRQsra27fxSdHQIvtjDaYyo1qAU4J+MI3HXl3Fxrblq6A5feZ65J\nP1ESqs07/XzRWC5KIqimL47xHNT03blyN7YdHgr8PrCmTyXoAzDp07zT/Ylfnz4TvnVva61lkLiF\nzLBfPb7pENbs64uVrgm6m3RoIHiuRBvKWKaQsnw9IrLYnlhzniovl90RKkLzzihx99+bv7sAb7rs\nEXP5SKl5Z9LBvrJSUXw+t9D8WdH2iy17jo5knQUjSvpuA0C5t80rSdBX/KeUYYETg8e4/Y1PJgz6\nDJzEUX2PglYRFPS5d4+EWmDr3acmCmZRfQmBRgDvN0+fSgEtqLAhIfEPP3+qqcYkD7LujxOlDCFT\n3gfCKIIiDeSi1xxcev42nBkt5roplFkR4/R7n92HV3ztfmxTGFkz6HiX4V5M+sp43It4DSeVJOgr\n3xmQkMmaviR9+q5+eJPStAlRoh7yft9vONCvNk+fbK3p8+6+ipQtwYiJgse0jujj5ORlX5+Zt3Qi\nxvDeLGQBH79hOW5cst1a+mntYmlpXVnWEGVxfhaheWe7ysuLI/cZksb1UaRT8oE11b7t6/cPaP+2\nQJtJpKWMAW2Q0KBPCDEghOj3+d8AqvP1kYtCLKHMO52Bjh8v2Jzo96r8CmBn/2gxVu/xb7rofgD7\nDeQC+NesOUyNUNhZO1A3LdmOLQfDH35f/+2aRn40bgwmCgK6NyIpJa5dtBVHh8aTrzwDfuWzJ7Yc\nxrfuXVeogpVbfQCfguY/TBYPSu89Z0fPMA4OjIb+hjWsZsS5H2UhrfUGzb1KVEZlvI3m5UVVmkKD\nPinlC6WUx/n874VSStWRP9tG0FtF1T5ubt394f2ont4VPmmviZq+KEFBWNCkrE0jc8roZpbeb78/\nf4ORx2tHLei75N51eO9Pn/BdxvQL4kbh317g2LXzKL43fwO+8uvVidPNtKAcsmrT2Urrli8hW/L+\nyLrupikQKFzf8AQ+fdOKls///c7w8z1KngvtJu5DZSys5UHZm4YFdlfh+dSWeNjLIUnzTvIIukmO\nRoyC6eerv3ku9Pv3/2xp6Pdp3JiDOigPjkZPKCzRWpspIJoKYN7ROxP3c6zpdFXJjk5U6nOdRUnS\nNDPO4dA9hk5z2YFR/RpR936WUgaOzFokeXuL5z63V+48gs/e3IXL79+QKM0stzDtwt+ty3dixY7W\nl12j4/kMnPMcTMahe7yzCg6KMC9j1lTOzZZnWBE3lIwp8/Ev8aa1YNBnUFCfvpGJfBZKbBkc8w/6\nmpp3KgQW3u+npDRy45nWIZrSefN3F7Qs4xcwmLgvaAWOmmk7sWxUy96oB35F2uw7pR6m+B1rnVwp\nFWw00ktENh/73uFqYL6zJ3h0XGoWd6CsqGPcTg98MqcMI67GwcuFyqIdL2EGfQYF9ekbyembaFuC\ngj43qRRYNDeJk9LMA6dDCOs1Wd6CaNLJ2VWCXWeAmqQBW0XKTAbM8F4+7ua/RS2Y1wfwafnc7vra\nSVHPjTAmtymrGscSHhZf3E5qB2U+/mXeNi8GfQa5a/rcAWDWNX3HHzs91fUFNS9ctq2n6e+oQMb7\ntU7fyLC0b1m207dvkJtf2VmnltE9GXxzGspJNN2INhyIHm3NKfBPae5Xv+/zUJD2H+jHbMbSDJH8\nsp6D3RxbHs4RwNw+zMv2FNmGA/24/P71pW4K5tVGmwqg3M38KFgZD3v7vSJl0GeNOwDMOuh7wYx0\nx9zxm39vX+8Irnp4U/1vKVubIQrhGb3Tk0ZFtg6GESRqkJjHNh1SSyimL97xrNH0zvnx4shlGjV9\nammKgH9nVdPnZaoPZ5i0ttK7O+sTO2e/m2NLuwbJVh/NPMzTVxYfmbMM1z++zdhIy3Gktd/LXqle\n9u0jTW14PysjBn0Gdbiq99w3zJHx6OaOpr30+Jn1f0+flu7d269PmdOHySFr/xdGyub9qBOIxBkx\nNYqRPn06qWiusNGnL3kfpywHcnH2kW9Nn4n0pcR/PboZu48M1z+zdYU452/L+V7/vLjyEtTothjI\nI2+zXOdcMXFemtp+lXSce4+AyKxGKO2XEWUbsMfROpBLNvkoG+7H/Gmn2msGfQa5m3S6H+JZ9Olz\n1y5On5b9YfbW/qkEFt6avUpF/QE7kXCeQr++UU5e4hTE6oV/rZhP70bk1C4nraWrTi+QQZ8+z451\nzpkOYTY42nN0BD94aBM+84tGE980tla2xnyJ93M7vY0PHsglmTwU2tup0FEWosBvbpLUmkdtbhvd\nktpKHu6TxrXTA7Qm+2igRNxNOt2n0kiMKRuSmnA1jetMOejzq2XzNtW7o2t35OiM1YFbGp9UNEZy\nmYpo3hnFRCHMOy8hoDuQi976nNMvqpYzatsqGdf0OZxzxv3SwmTZeGgsvZcx3lO3DAOu5OAUAZC/\nN+d5y48p2pOzW8pH5HrZvDNSnAJ8KQv9GSjqeVPW+xqQn2dZGhj0GeS+lt0B4FTCWqc4hly1i8ek\n3LzTO/8e0Br0/WzR1sjAwvuQ0WreaeUOpZdmksCpUtF/xDp9+uJsurcZbR5qHpx+mccYfmnRUd9P\njW20fYVIlK82J+3tCTpGkc2VDT7SpZT42A3L8PC6boVl1T+3+RKgXGdd/rTb/i3ZbYwUlfGwFzT+\nToRBn0EiYPTOrGtN0m7eOeYzAMfYZGutil8Q552iwd0MpaJe0WelT58u9/bV/6XwxBwam8QrvnY/\nfrZoi9b6OlRH74xIJ/M+fbV1O336pne6z1/1jAU1Yar3fXSfazoZNKDRvDPlFbuMTkzhqoc3oT9g\ntN28sRUX6RyDigSWbOnBBbd0BS7jzufEVEWpeX8RXgg035v9WmnkYxvSykXZC4xFrZEi0lWA268x\nDPoMCurTl/VIiJ0p1/T5DcDhNxJjZE2f5/upivronVYGctFM0jeoVfjdkaFxAMDuIyNa62sEM8mb\ntmZ9zgKNPn2dHWYHhXCCwYqUqRXcpGwuEpvqf5nEk9t68JMFm3HRr1fH+n32Z0hNivvQua9MUywR\nnz9nGU67eH7TZ1qF6ZwWvFV2eVandtoBdBEC9jg4kAu5lfE8b8cXGwz6DGrq0+eu6asVFH78kT9O\nO0sA0q/p85suwW8aB+9NpPqnu3asecTDrEfvdKim7PdmPI37pso8fF7NLymyf1EBNPaZ+7oykS3h\nqumzvZXuGr2mgVxiDOzjn378p9bzpk8DAGzYHz0HpJ+0T5GgbY3MhmI+VbbHua+4R2pevq0HO3uG\nfNPpCpiv08tq804LByoP9wcVNguqpq7hoslLjS6li0e9HBj0GeR+bnd4CtHV77N5rZB20OcXcPk1\n+fQWHLwPE783jaoPnDxM2RC3MJC0EGGipi+Lgkx97rp6Puyspz6oTop9+qorbPyzHgxm+Citr7ng\nbzvTPFedptPumr4Pz1mGt1+5qGVZnWNbhLfo7hyq3F6zOrfTa95Z8AtHU/7PULIp6BY1OjGFHz60\nEaMZz0edTPuc3Qz6DAru05dx886O7B9Ovs07PR+1BHnw9ulT34+TFoI+3ZuaX58+lYJQ3OkmnNVF\n7qeIr23U9O05OoyewTGt39i6apxj4Nvn0iK/efqylPgQp13Tl4N95owKPC3knmoin9k+MqJX3nTt\n+Cyel9qgNF605mNLU9R2G0xhbli8DT99dAtuWroj66xoy8EjJXWdWWegTDoiavqykot5+hRq+ry8\nb791+vTZCLSvXbQVgPpb+UMDY5jWITCz1oyu+tvo3/n1idSRdLDYap++ZGl4ve17CwEA55/58kTp\n6GQrqOBZkc3/jat3eByjExW85PiZgcs4hU6dkRzT4pzHcR98eSnYR+UjKpc6x8Cp6UvrPZqJ2iRT\nR8lvCprm75vXlV2fPve/2bzTTed8ysNLFsqPoPus04rLr4xXFEW6hpNi0GdQc58+dz+kZIWrpI7p\nbA36Tnj+MfUBQ9Lgd01JCczo7KjfNCSaL76Kpzmnt1ARZjLxPH2tnw3XRuFTTfkdP1iElxw3E29+\nxQn1G6LKb+PePJ20I4NphUJyJYM3Fd5ChnckV1Oc69G9n+Jcm2ddvgCjExXsuOI90etE8wsLb1NW\nU3QKamV5zqXavNMZyEUh6ituQSJ625quHZ/F815TSdECX1Slm40Wc5fvwmObDuL6T5yRcU7aTNYH\n3oIyzJeri0GfQe7zx/3vOYu3pZ8Zl+k+o3eGneruQMymipSYPs0V9LU8ZTw1fRolCRs1fXHuDwf6\nR3H3M/vqf6dR0+fuzyilDLyxBW1P1s2RvX3u3Pk0kbVG+nq/Gx6fxLHTp9X35+iE3nFyr66+TZnW\n9GX7e1PyGvQlkZNd26S5T1+y2tW0pNO8My9ba1lONvNrv30u6yy0pZwcfkoo+3Z/JdIR0KfP7/n4\nzj/63cB0PvbmZE3gvPxq+sKehY988e1G1w8A375vXctnFSlDC09StvbpU22uY6NPn4m51SYVArrx\nyXgraPTpa/3Mb7kgeRm90xZn03QG+5nz+FacfvGD+FmtiW+cdfoNHJO0wJikTFupB9XxEkn7DAnK\nZ9LmmzrHoNG8szhviG1cykFJNu6R2d0/0lp1u9US6N6rijA4ERHQXgEtgz6D3M+A2S9+fuj3//PJ\nPw1M509nn2AyWzhm2rTohVzSepat2tXbFPR5m3eOTlRwoH+0/ncl63n66v+Nn/YNT2yPXMZvegsd\nSQcosdGnLw7/JsHqGQse4r+1eWeY3uFxfPf+DQCAu5/Zq7z+ah6cdXpr+sL7+qUhB4dYS9BtKc3C\npcpALmXV1Nza5xbV9L397ORGWWOboHJAWbeXwpXxuLffXZxBn1Hut7/Xf+JN+O77XxcrHdOTqfvV\n9IWd7ibeYs/wXWerpqBPAqt2Nea1+uY9a5qW1QlEkgZ9WTbZid+nrzWY8SsQ+22ZtwllFm9pm/PQ\nmCBewOzxcE4Nv00cm5zCxgPN89a5a41tzv+YpqGxSfzjz59KlEY7vsm3WdN385M7sGB9t/F0bYge\nhCvDgVxSWk87FhipfZW5GXM7PcoY9Bnkfgi86HnH4K9e83ux0unsMHtYXvS86S2fhZVZTBRoVN+E\nN08nIbH54GD9r62HhpqW1Rl4xkbh3ETzThVjk8nmu2nq0xfj9xWZ/e39u/evx0f/e3nL5yby5TeQ\ni+Obd6/FX//ocXS7apibfxt/nc2DFFX/UElu4caDuG/1vugFNcxfc6D+7/ijd/obm5xqenljiq0W\nCFqjd9YnZzefj4vvXovP/KLLfMIWruakg0WVQdlbdwYP5BJ+bNut2Wu7KGNg1I6nKoM+g7zBkvdv\n1eGSTbccmvWCGS2fha3CxIUwTTERnWZSe3tH8Jc/XKS0rJWgz2mSZzzlZrFr+nxqsPz79IVvQSWF\nt/RR6d+4ZEesdNfs7Ytcpl7T5/PdUzuOAAAGRid8f6szmJBbtXmnuwbW+W90ep+6cQUunPt0rPUG\nsdln81v3rsPf/Wwpth8eil7YAJN99qLUB3JJqbSQq0JJ00sL9WXT5j9voM31Fa80HCfLBdxMIvJg\n0GeQ9wEdN3ibrtg0UtXvHtca9IUxUdDoUNx49xyCaiNbqj154hbO8yDuyKmNgVwa2+5XuI8e+MLb\nAy1dYYX0qMN65YMbldYABAQ+9Y/8z1/dlwmiuSLb9U/1mj4bzNSY+n++thZ4L1jfjXX7+g2sqSqw\nT1/CrdH5daOmL3wAqqKYnKpg95Fh7d9FztkHc8H2X139GL4/f4Py8mnVMuZgAN5MFOn8JnPKfNiL\n+OImLgZ9BnmbNcRt5vB7Lwye7DmOE/1q+nLSvNM7kIub31QTqqaSzlCeoZ7BsUS/Vw1Mgmqe06jp\n81fNz6X3rmsZfdWbn1d9/X588Y5nWlJQ2fawPn31nGgMYqDywGiZY1IhD1a51hv3cg8qXDv797J5\n63HuTxbHSzyndGr6Ej+vabkAACAASURBVB1bg+dFWD6ueGAD/vz7C3Ggz785c3OW3C+T/L5XW6eu\nTd2DsUfNtanszRhLvnmZKWp8UcbASLX1XZkw6LPIG/eo3ETv/7c/xyknHGs0HzM6O/DNvzm9OS+W\nB3JRTaPTM5CLm7sWUFfCATDDWb737e2NLoD5qdceRTbvjEgn4z59YTWdTs4mpiR+s6p1JE2VoC9s\n+6N+Hb/ZcKNPnxDukWDNUn2INU9MX4wHX+CUDZHns7k8OC0Iwl5qmSws2z4yT2w5DADoGdJ70RQ5\nOTvUR1o2Lc49z/T6dIxPVvCTBZsxOpGsL3eQO7p24y2XL9AutHsXL1+Rn3QYmSM3eRJW5DVfNjDo\nsyhO8HT6y47DC2Z0Gs1HZ0cH/uo1L1Fe3kRBozNGTZ9XsqAvWdQXHhjYvUUc6BtJ9Ht301Ynr//4\n86fwH3etDvyNu+Avka+3kUII5T2uFPQppBZ0VppoNtw0smHC9HQGN2rKQ6K1hieS9iuDpGvTKQw7\n95UizdMXxh1Ij01O4cePbA4cSMpvIKKWZRLk5fO3rMRty3cmSCFdpk6BW5ftxFUPb8Kcx7eZSdDj\nK3etxn6FmlxVOXo0FFJJbh3l0IbHgkGfYa8/+Xhc+cHXA/AbyEWN6WYj06YJrVrHuAWa/3zPaXj9\nycdX1xlj9E5vYTFJ0Kfa9y8O2wHR6IR6wDowOoHLH1iP8cmK76TjzmePbTqEX3Xtrn4W8diujt6Z\n00d7RLZUgrKw9wG+U1y4C7sViW/duxZ3rdzj+31wut7mccn79A2MTuAfYk67YPOtra3rIw+FJacF\nQdj9LU8vTAC1c0xK4OalO3H1I5uwePPh5Ov0NmdWMH/tAXz9t2uiF1RYd5qS3itHajV8I5Zq+hxJ\n90sZm/dRunJwC/fVTqc2gz7D7rnwbfjQGacASFZI+cWnzzSUo2pgpROAiphnxfv/5KR6YUh1SPPO\nkIFcjknUp8/8VZzWDUtnZMUfPrQJ1z+2Db99eo/v92EpBU0CX6kU9yaoW9MXtJ1BL16mpMSNS3bg\n3+98VjFHjRFffZt3JtjP/aOTsX8b1TxPReCQ7raCPve/PXM6honKjk52VQZyaaQbUBvm87H3fEt7\nMBIgOuiI6rPXNLhLsmwl4rfvbAQsYVP4zL5oHi65Z63xdeYBg7/2VMbDntcg1CYGfRYlCfre/upZ\n+OSfzTaSjw7R2msnrDYxbk2fEKI+wIGJKRtURwD14x0IRJd/waz2XaKU4607iNP/LWh7fR/QUX2g\nMuyPEyUqWyoBs0qfvsDmnRHn1URAIO2t+XCOS5LCfZLCl83Dm8apEzAoqnWNgVyCl4lz+8yqIF2/\npxlefVsEBhEH+qalO7SSS7rLKhWJm5ZsD+wbqJt8HmrWKT9y2/qHtDDos6ildk3zJmrqptvZIVpK\nsaHBVsz1CjQCNdWALWwglz1H4/dtS9qnL2zf2y7QmJxuQjWl5pqTbG7wJvb5pEKzXndS3nW6a+P8\nVHyCPvcn1yzcErLe1poQ22XjSkX699Ey0TcxsCbLfvvOppdWhlankm+VgVyipF2YjjNKrYl1ZhX4\npd+8Mx/ue24/Lrl3Ha5+eFPksirHPe2afKKstFNAy6DPoqSd/U2Nqlft09ec1vHHTjey3pNe1Bhp\nVIhGDV+cgVzMTqJsLKnUxW2aqjpqnWofnzyKypdKTZ97GaX+eBFD1bvt6/V/UdFybkv19QfmK6Q2\n2vHV3zyHP/zP+S3LmWj9nJdzJLL5ZlTzT43t0BnIJS/7R4XKvVcliMtD5dCavX24ccl2AHYDz7Dm\nnVkYGqs29+4bmfD9PmpuxSg52UzKSF7Oc5PasTbb7DCR1CTBy2AAZmv6Zs5sDvJe9LyQoE9jvTNc\nE8kLIRp9+gxM2ZCETr84Xfabd8YM+vxyppiUd5oH2zdDW+mrNOv1XaLedDf891G1sDOnT2tO1tV8\nrmkfp1SEcgbv8TJ5fQyPT6K7fwynnvh8AOn06esQgFN/mWaNkvMyqTSjd8YM01SmhcmqjHjRb54D\nAHzqrac28mNhPSU5BZSVsdBP6oIOfynOizJsgyLW9FnU2m9O7ymRNGhspCNwTGcHXnvScfXPXvS8\nY0KXV3VMU9DXaNapPjm7nVMwaUHQtxYlpffYJseg8R/UIPo3tm/kcdOPHHlUaZ4+/+C4d3gchwaq\n85UFTlzvk/51jzUmjna/BAleP/Dpm7qc1Srzrtt0LaF2GrX/fu7mLvzFDxa5Pi/vE7Tep8/UzTlA\no5lx8vWo1eKppKP+fV7OgXQmUDezrbazGpTLD167FNcu2hrwLbWzR9Z145F13QDK2U+3KHPUmsSg\nL8dMPbD8mloef2xwJa9OecY9rYIAMF0z6FNtBqrLwuCdeGrHEQD232zpNe8Mb7JjK7iyIexMUM2N\nSk2fexH3Jfamyx6JnC7Dr6bvygc31v89o3Nay/dAa01f43P1/ezdNp1j5F2P+y+/+8zWQ4P4o288\ngN1HhiPTXLKlx/O5cra0BN0OTfRO1F22LLU8sUdude0z38nZc1I+tNu8sxwnQdfOo/je/A0tn5fl\nHKf4PntzFz57c1fW2SCDGPTlmM5N97SXHhf4nV8ANjOgcNoh9Gr6pruGseuI0bxzmuv3eWremeWb\napNN7/xS8q398wSPeSm0eUXla6qiUkvpHyi7g+2m0zeiKVvT+j0L1Pv9IPnch5OewYl0jpF32ajC\n8B1duzE6UcF9q/drrCO9k8Zd4I6suTZZc66zz82t1prG+RlNd9vzdA/JQ14mpyqYu3wXJjPqcK67\nDziQC7nZPuxHhsbx3p8uDn3RaEs7ndIM+lKkPXqnxpvEOZ94U8h6a3OF1c7s6z/xpqb58dw6hNDK\np7d5p1Pzp1qDFzY5exI2avrSohf0ufdfK9WCuHcxm30ibVIbyMXe+kfG9SZY9svuoxu6fYddn/CM\nTKrVNNRb0+eu7dRIJyiN6jpqn8dML0pToOcO3BOuMc+nuonaltDtqz8b9HZCdJ++fOzUNJp3qm7q\nLct24mu/fQ63LNtpN0NEFti+pO95Zi/W7O3HDYu32V2RSzvWZjPoy9D3P/j60O91T8j7/vVtLZ+9\n/ITntXz2kuNmBqYthN6D8ljXwBUCAp3T9KZssNU3Ji+FjjjM9unz+SyyT1++3nzpjM6vNDl7QPNO\nE5ygb2xyqmmqBNXa09V7evHpm7pw6X3rWr6LO6or0Lrfoprn+S0XuY7aBtq69gKbd6Z4subpujBJ\nqaavKdBWSKesO6tG995xdGgcQPDomrblpZ8lFVW6DenTVODiojaO3pmhqGeGTjwkBDDrhTOaPjvv\nDS/DT87/E99lg/Ok9yT7o5ceh4UbD9XT7awNzKI6OXsRR++0TWUwEj9+he1YQ3PLHE7OrpgfU1M2\nxA0GnUDvL65chIMDYzjh+Y0Bk1Sa2vYOVwuEu3pam7h4m4X5He+gbLc27wxYMAHp+W/RFDXfUULn\n6TO6It9/5oLNgEf1WnIWy7QvoIma44h92YaVJ23F+2y8fYX/CNFFwJo+sqplLM+IM0734eDtR9fy\ndl820g1KOSxLn/vzU5v+/qe3vwJve+WJTb91+vjFGb3T5GO5jM07v3LXsz6fhm+of/89FenuwL7h\nCdy2fFfkclG1SEpTNvgtElrbpc5Zdl/faOvAKxpBedfOIy3B/0TLQC7qvOeU6vURdg8KbBJs6dRR\nDWh1aR1frX6U5mpmbdPv7xVeU1ydsqHAN2EF7v66Wr/LqKCpe4yT5vMLtz+D/+/7C5MlUnIL1ndn\n0octDr/z5+DAKA4PjiVKg9LFoC9DUfdUnZuue448R1DwUG3C6Z9O2AAs3j5FXz3ntKZ1upt3qubd\nPRCMSUWs6fv3v3o1gOAC+R1de0J/7/uzODV9SP/m/KMFm4yko9S8U2E+Q9PbHzRvWdBqRicquO7x\n5mHUW2v69PLwT7d0Yf6aA7X1Jm/e6Xznnouw+nl5pR3I2I4PtAr2Wpte5rOgSjcoKtojycRALrsK\nEtBk5TO/6MI7f/hY1tlQ4j3sm7sHcOZ3FmSSF9PK/oLKjUFfhrwPjae/8W7P98nm9fO+aVZJLmyZ\nCZ9Rx9yLu5t3Ko/e2dS809yFl/jtfwb3AOd4J+m75eWXlMp+tr353vRVm7Sq9OmLOvVMjWIXtKzf\nKKCqDxX3UpsODDR911qLqdPfDnhwbTc+f+vK+t+2JKrhCvltU99Od1PCiPWlObpnHqmde3o7wb20\nyalirAnJz/uuWYKv1iZ0j5W0bg1a7DVRGY1nNJqrrkZLsaqdPl0QiqYs067oYNCXIm8Q5y2c/s7z\nmydM1zkdBVoHTwnvy+Gfetg6/YI+tw4h6jV3qvGqu++f0eadBWzf6QTKcWspfQtfMfaqzSkb4s8L\npkZ3IJc6Q4OZhOVBd5RDT8U6Jr2jd2pkq6V5p4Hrw1sIaIuaPoWNy13AE8J77MJELeId6KUo++HZ\n3b345VPRTcu9tF8eFejKkFLizpX+fbWKsxX5VJTrIkq8QfjyufFlOSYqGPRlKOotg17zztbBU0KH\naY9o3vnhM05p+c5b6Kym427eifpUEPFq+pR+oiRpmTaLe4Czy+I3TW39nTepuct3YdWuXp/lmt7d\n2x/9VCP9oDnz/KgEfSr7N2pI+sDfIaAfpXS+jVpvY5mWPn3e5p3q2fIZvbPB7z6k8gbUSUN4XlYk\nOXXivKjK6/M6r/lyizulQdSzpJ0KUbqbWoQ+ffeu3o/ufvW+WlR+3uea6gjtecaBXChVUSecX+D0\n44/8ceDyrX364ufJr+DqHUjCvbzz7+m1PKheS53s01fXUQ/6opedt3p/01xuqgXir/32OfzbL5+O\nTD9ve081CJ1UmJzd92tDG+wd+dQ5LkE1H95Ct3sRbwAbp9lv0IuE6H2kEZTXf6P/WxOSbovOCw6V\nJc3MrWduH5qoxVNNxz24SZ7uITbzojx6Z552SISsppVoB0UNNOotO2obUIKYr65Al2ZiDPpSpHuN\n+C1/1iteHLCsaB29M6x/TNDnnonc3c597UtC0xFC1Gv6VC8i9+idJi+9xIWmDO4C9eadEYX7FTuO\n4F/mrsK3XXO5SfgHO3H2g5TAmr192r/TWkfs3yU/MEo1fZ6majqCjoNuOt58Tlb0B3IJar5nYj86\n51ZjIBcDNX1hXwbctPIVXhRL/HKb6/rwPd9jJ1wYcTcxjQnj/UTd03779B7MvmgeuvtHw9Nph4NL\nLbzN+VWn5fJLg7LDefoy5Nz8OzsErvpwaw2e3zUVdpm1DuTi+Tsqcbhr+prtuOI9IWtucGobVYbO\nBxo1gwDw8yU7lH6jImnzzixqCusDuUSsu682l9v+vlH83nEzQpeN+xb6snnr1X5oSKoFIb99Etan\nT7MmKOjcUQrKXX960/GOnqsV7HgWjbo+tJp3QgCQ9TSt9Qf1WbdJav319JoGF4VabaDGdVDAfRBX\nnEGa8ujO2ujQWw4OZpwTKoIyNO90tNOLDNb0Zci5ZP7slSfivDe8rPV7n4JwUOFYiGrA9dZXvhjn\nn/lyAPECF+3aSM8PnIFcVC8ip2YQqDZZNMXZ9q+d+0eJfp8mZ1dG1fQ1CtvAun39LZ8nlcaWO7t3\n95FhzL5oHpZt69H6XaJ125yyQfqfO3GS854GKgO5eK9Hb387vx/HHlzHu34TxyZmzXSidSb7eXC6\nAQnr9hG1ydixr30mm/7OviCVSh6y30wlpnZFQTY3t3JwWcTS0qevqO1UfRT0kMTCoC9FrQWy6n+D\nurX51vSFXGdCCNz22bPw7tN/F0DriSwC/u3WEdK8M2CtTX85Uzao9j/qtPS2yFl93CF5s7gJeOc7\n87Pn6DB29gzVlhd4dk94M0wp9UdqTPOhtHTrYQDABs/0BF4ms1RRGCG7ufCql757cfcxjaoRGBmf\nwvB4o5+m9xpK0ryzpU+fzzKxefoB2ypoB73wilpb2oWsIpaFkhwz34FcclaMsnEOaN8XnGdSAc8P\nIu/5Pi1G9JCvu4LrmZK3jFnEoC9DTkDS3K+t9Xu3oLcr3r51QHgTrqAHT71PX8yrwKnpU23eaWsg\nl6QFzyxmfOhQaN75tu8trDe9bBpEByJwyoasB7XxOxbOZyrnifs6MFPT57sSIyQkpE9QefHdayLz\nfsZlD+Nf5q6q/93Spy9G807nHPGeU2ZOCf80kyTt/u34ZAWzL5qHaxdtDVzevd7Y64z4fXf/qNEp\nYFQK/WldsTovxZpGtFVYNg/lqDSajatup3O9ZjU3mLGWIHk4sAVW9KDfyX+5avra56Rm0JcDQW9M\n/CrBVC6zRm1d8Nv9IPXrWPEa8F739QBW8fedcV4XKXAKzPGbL6V/E2iM3qm2bpVNk1ItgFVplhV3\nn4T9LIv5FFXODfe2ajXH89ToObV1a/f1R6Yz5Krlc//Wyae3pk+Hd91N51jCYfu9tYmmLp2R2v64\ndtGWqJyYWaGP3UeG8ebvLsC1j1UDT71tS54vIy85wtIw8KI7z0GAzft4OxUUibzizdOXL8XfAn1W\ngz4hxNlCiI1CiC1CiItClvuAEEIKIc6wmZ+sRc2Np7J84NsV4V6m+t+g550QwW8bOzQLAd5UGr/P\nR/POuLIoyKjU0DYv3/y3b40a9OelMy2sqaRqjbBfWnHZPrbuTTo40JjrKuyacE+/0UinunxnwOBI\n/n36mk8K5zr3NhU1GWt7W8gkKQyrDzykHpQn2dSdPcMAgCVbDidIJb+CRnf1o9PkWcp8BYM2AzTl\nwDLj/ZGHPpZZOjQwhm/duxaTU/FfngXZcKAfB/rCRz0tOu/pE1bTNzI+hS/c/jQODjTvk7yegnnN\nlw3Wgj4hxDQA1wA4B8DpAM4XQpzus9wLAXwBwHJbecnSmbNPCPzOaXIVNAqSb2AWGPOJln+HFfY/\neMbJvp8H1RIG8RYydav87QV9SZt3ZlHTpzZlg0NphEWp37wzaOnxmA/LoGAUUOv7KYTpm7LTzCpq\nCeff6iuv1qzqZ/Yvf7Co5TOnYq8+Iq7CQC4tAmqPlUccDFnM+5XtS8bWW1mVYHzm9GmRy9qQVkHd\nyBQehtMzIU/NOx1Ztoozs+p8HFtdF9+9Bjcu2YGFGw8ZT/vsHy3GWZcvMJ5unvzNfz0BQK3cce/q\nfbj7mX34/vyNtrNlBIM+M84EsEVKuU1KOQ7gdgDv81nu2wC+B6CUr0k+8ZbfD/zOKdwHzXfiX9MX\nvc6gZoLuAsRJLzrWf53OstGr8f+95lNluoXmne4gIe5D3x2LfOZtpxrIVTTnuEVN2VBf3rPr/H5V\nrenTy0fQ6v/wP+dj++HqIDL9oxPYcjB88BW/fHkPh3ZNn8bdOejQO6usSGDd/n7/hZrWqbzK0OWD\nPp+YqmCfz1viRk1fR325pvQSDKHf1LpTORX/NBoT0Cdv3hknWPD7xesueTB+JlzGJqv7fEZnbf7R\nRO0gDWQohqHxycDvwm6PR4bGcd/qffW/m2pXXRvjO5CLBHqHx/UyaoHV5p2B17n/F1mXK7Nef9ac\nKW/avcYzsQRvDvK279twHBerQd9JAHa7/t5T+6xOCPFGAKdIKeeFJSSEuEAI0SWE6Dp0yPxbGpvc\ngZf3DYlTyxHUNlp3ygbvMnGacIVNzu67vOdv3dE/bbQLF0heU+e+OdkabMZLZfTOpuU9ez9oGHWV\n2jTVwvbG2iibf3/dk3jXVY8r/cZ3CoPaR6qjvMahEnzd++y+yGV0cightR5sEsA1C/37rNVbAjgv\nAxSad3o5Z0jrb1Vr8tWXMTGQS5y8+G3LwOhk6PeqvDV9OqwNOqShUpH4yJxlsVZ00W+ew4Vzn8bh\nwbHWL90/9bvvAHj31Wr3hzRYbb6umXbQJRWWzuRUBesVXlDFyY+unJXbqcSGxibxXMQI5SZsOzSI\n/X0j1teTB5kN5CKE6ABwFYAvRS0rpZwjpTxDSnnGrFmz7GfOoLDgo1GoC+9fF/WZV9BgLCq1Xrpv\nPrxJao7jUh/t06QOIRqDdcRMw33c0hqlSndUN7Vs6QUhtV9ELhM1xUJTeqGFGTNNT3WovRCIt6aw\nUQuDVhvUF8RpCeAMdjTR0i9PrWms37JmApLma6we9CWp6Qv5rbXLMGSdTk3fzOkGavo0mFpPVDIq\n95zJeg2JK92IhPPyRj+L5p26NYAqrnxwI8758eJEE6eH3ZvIjNMvnh8reBgaC66Nz6s0LvEL567C\n3/zXExi0tH+cu8PlD2zAWy5/1Mo68sZm0LcXwCmuv0+ufeZ4IYDXAlgkhNgB4CwA95RtMJewblD1\n5p0BR8HvcRX0kHZ/Wq9ti3GL1+3T5w2IdAOkoOkqkhDC7EAuaQ1SpVs+8RZo/I63ak1f2pwcTSmO\nSBl2LmvPQ6i1tH5hLSgY89uGwdFJ7KjNu+jlfSnkHYBAJ1feXeT+M+kE3a3TvKR7vtlcm1PTN6NT\nv6bPK04+UwsyNZff3zeC2RfNw+ObGi1v4taO22I78Iybepzr7endvQCqg5HYSF9HHo5tXg2PT+Hh\ndd3av/vsL7q0f3N4cAyzL5qHu5/ZG71wTuieO855PzFpfvCddmUz6FsB4FVCiFOFEMcA+AiAe5wv\npZR9UsoTpZSzpZSzASwDcJ6UUv/sz7HmodGbv3MKdXrNO6PX2ejT1/y5ykOwPmG8YqTTUtOnGTTa\naDophKgHAnEfgO7j5u5z+YpZz0+UtzC6AbPKPpZQnLLBvYzhp3poTZ9iwNZcy9D8m6se3qSZH4X9\nFrt5p94Lh4MDY1i27Yjvd048HNTsV+l6DhjUSbX5s04TUme7bZW1m+Zr1Phd1LJh349O1oI+p6ZP\nZX05KhVHnSO6Tcodq3ZWC2O/fGqXZ77QeOnZZjM7qlMjJdknRp6ShnbCX139OP7h50+ZSUyT+yVD\nmTy5rUf7N06t723Ld8Va58KNB7Ez4IVjmEZDspxd5Ibt7BnCTxdszk2rBVOsBX1SykkAFwJ4EMB6\nAHdIKdcKIS4VQpxna7154z1hXn/y8TjtpccBaNRQ6EzZEDWpuvvfQQW7sCY9zjffeG9joNWwWCRp\nTd90GzV9QOLmnc01IY1UbDb1TLwr/PrWSP033uFLe5sJRqft36ev+pluLWR1e5o/e2S93ptV2/17\nguc51Esrat47pYBMoXln0oFcGn/X8hszvTwam6g174xR06fa7E/lt7Y0mvMHr1FrwKCkGTLI5nV+\nzcItmL9mv9ZvnOzEmZw9T/sVSDf4GnHNX5pVsGlSXmKIT924Am+/clHs3+dlO5IIa/79yRtX4IcP\nb0J3f3TtepFY7dMnpbxfSvlqKeUfSCm/U/vsYinlPT7LvqNstXxAa/POey58Gx74wp/Xvouo6fP5\nLDBAdP3b6X/yO887Riuv7vRPfMEMreUbf+utz8ZALtU+fcnSaCoUu7Jos6mndk2f0jJSaTTQuLtL\nd+J3L93RO6PSA1ofRruPDGP2RfOwdOvh2u/19ofuwy0wSNNLxmeaBe/3UfloLOBtRev+LuzBFzqB\nvZMj70AulkoD9puttebbmabkmProneqtJUzl6c6u3Xj3VY/F/H24OAFINd1GSwr/XVKCEmGIKx/c\niCVbqrUzrcF9+EsfnfPj6V1HjV1PRayZufuZvTjt4vnY3K3eh9zP/r4RDIeMYlsm963eh56IwZey\nYupZmgbnZUMW03fZlNlALu2iefTOZs4YFjqFfZUlT3/pcbjsb1+LH37oDU2f//NfvBIAcMoJ/tM1\nAAG1iyHr8vZHbPTvUWOneaerpi/2lA2NLXAfn7iFJBXaefXs5GU+TUQqFXNTNlT5j0Abnl7wMkrz\n9DWnFrm811Pbq80n7+zaA6A1APLTnGW9xoSmBnKIGnHTLz3R9L276aW3eadaHnSad9Z/o5Z07PXp\nLJd2WsHriL+SL9+1GptjDuBhYn/Wp+QIaEmQZL02Nb+4sVm9byfZh9YewPt/thS3r9ht8aljnsmX\nHos3V1/Urdp1NFE6b7n8USzdqt+E0gabwfehgTFcOPdpXHDLSmvrKNM0B2GnakeJttOtM+sMlF3Y\nw6YSUdPnd/dUnbLh42f9fssy573hZTjvDS8Lya3eNBF+3+Whpk8geaGjOehzpW3x6aubtPvhsa93\nBP/lM/S/hNQe6ESHyluwsCVUC2Nhffpa1xfUrDk6P1HrVlk2eCAXPY3DFj+9oKbeRibjrtdcNK/D\nVPl6zuKtRtIxGiDGXF915MQYGbFc4jBdK9n4d/syte07e4YBAFsTjNjp5n4JVBTHHzsdANA3MpFx\nTtRdfPdavPO03wucC9kmZy7Xfb32ph9ovASyf5WnNHB6wLprz7UcDoSXBGv6LAuryXAKSYExn89n\ntkeS1E1epU/f9z/weuXfm9A0ZUPcgVxctUFp9elLMplx0JDGyvdl6S6whYZpTX8pBX0+NWv1poAq\nefP8zvubllFMPc2oWptFqgSq8W/0gUFfwpo+1fXU1xeSlok+fS3rM9C8073fr1nYHPTZGn7f1pv3\nfNR2Kb5U0UwrqPm73/dZSWsABvV9XF3ugTUHrK+rLI6bWQ36+kfiN83MYiCONOaWy1qcvap7/to+\ndGGPlLiDXOUdgz7Lppoejp6mcdKp6Ys+DK896TjfNOppGyq66QY13iC0Pk+fa7v/+rUvUf69EYH9\nTNQFPSgsjDtTd2RI722mrZuRTrq6E7/PjTnSWFN6EasM/DpmVZ9e487g5XWPV+Rk7BpNL8OmbPBL\nJk58JT3/zcKKHUcSvTyJpLLP/YIgA6uLU3i1XWDxG1ip+nm+Sko2c9M6oFH48it3JmuqGFe+joia\n44+tNkZLUtMXp9+4VTnLThFkUeOXZOqzPGPQZ1lof6YpJ+jz/959ol/2t6/Dhm+fbTJrdcdOb4xM\np9unzxsk+vXpCwvsTNecrfnWXxuZnN2d/+aBXOzdfc5/8yk4vTayqwqVclVQoaxlOeW1Nm+/fv84\nZ33xmgJK6N+E+kETCAAAIABJREFUvdegUk2f9P93nPXVP9dLpj4xuDu5GxZvw4NrqzUFSgO5CNe/\nXZ97/279bXT+WmpU41bfKq43KpCoVCQ+dN2T+Pj/LPf+Mn6GWlKKqF2VEocH/YPO5C+ikv3eT/1+\nrXlNRC4bN0MG5SEPbmnGwX0jE/WmfkX2glpNX5LJy8cynuOtbEGDI9b5HHNXZPEOqTHydfrrtolB\nn2X/j73vjrejKPv/zrk196bddNITIISQhAQCoTdBwChFRNDPq6Ji96eIvi/YEVRiR315La+irx0U\nsYH00EMJEEJCGoRUcpObens7Z39/nLvn7M7OzM7MzpZzM9/8kXt2Z595ZnZ2Zp552vHTmrj3Spo+\nCe3dtFENqK8RhA2PIIvcMxBNFFDPDUjfY7VFZJZlWogaWlctiCgnxuwJw0p/+83fPIFcYhT6Gmur\ncf2Fs33XpKInCuAM/FOBSumokUFVefvrCztw3I0P+K4FAolwSJZ8EZRqVOOxKFCx76n6BvT05X2/\nHTj4+t1r8ZEBJ/1QAQTlvvFqDQvUQYC+Fsp/sFLK06dJj1+PWrlX3mhVo2+Q4Vd2suvW1Xx5H4sa\nRa54AOSnoe3nKlFX2gjjYeW2A0a0bgFNXwybfNVl59iv3Y9rbl/pu5Y17asM3ANxmXWGB3oeTRsl\niwjJNrlJ2O9fo28WbBLlsVh54ymI4Ie1eU8HOnr6PTlHB0M7y7BCX8xYOLUJM8cUE3rTw6uUp4+b\nnL34/xWLpqCpUZx+IYosMmNMOeF41EAsufKxf+maUFMYwwjMEVLe5Ep2zNcuOgbXeQQuX9RVbyAX\nAY0Xv3wefv3+E1RY9YGQoBBcy1MDw6ymTx5+YjLmnew8ff7/RfAK2g+u3YUOT96mFZv3BV4xvekK\nWEUqaLFU4YB/Mqh6YthFC33U8yr0vGXzBb9gymqrzGdDB3JxezqaT5/GMwmuyUFTPv+FvR4tn0+b\nKqAhVRGib7GaW7vxu6e3cOoTsMK4SWt5vdd4z6QB0Vi85NYncdlPnkqQm2Rx9yq1PIJZhHtQx1pn\nZOeZtDV9PNcb2Xlr7cBB0m+Wc77dQwBJmnee9d1HcNWvni3txTbubsfu1u7kGIgZVuhLAHUcDV24\npq+IJBdQtnmnvKaOJTSqJHc3AYLw5Ox//NBJvt/HTW1CXXX5c/CuMV4aIqG4qbFWOr8hDzT92mq5\nT5TXjQ7kNot+IUBNs6VCO3AvYj3v+Oly6Xp5icrZfHnNH8P54z3r50eNUJgvCpMetQlnme8VHMfX\nB6xaVFgtn4jy6UWB7AwRlh8t/HnJigQ4IPA9ikpehz/6mb+8sMP3W2X65Y0TJo0MyHy+7zBGfgKH\nTHG2PQJt+tG97T2Y99X7IrETN9y+jaLl7k1Z6Bus0JqPzLMRCbz577nN+0sLz0d++zwW3/xQckzF\nDCv0JQCeoHDK4WMAAItnjmbeV1mQTYlOTAHP59NG3Qr8Zph3guCac49k1hdHIBcikZz95MP9fU6I\nX/jmLTJRhNRZ44cK7xOQgNa3TiD0yWgPZAWNv76wnUmXxh+e3YaWtnLiVznzTr7WIk7TifvWNBfb\nFTC/UoOSL5PD93N89y9oXzNFPkJ+B8s7pe/T+54ch3pW1/Rw4H/a9yHKKxWNB67ZLidKq3SdEcp6\neXrjQBcOcoLI+MzFlQQth/m3zvMBRkTlPLj8p8tx8z1r+XUwHs3K5i5Oc3wX8ocK6fYKfbD31Gt7\n0RbBVy4JuDyzNX1yNNLW9NFQPRzLWroKXZNwWWRm7vD8PZgsPK3QlwBcQYFef049YgzW3XQBTpwx\nSvh8kgOOqanz3eeYc1LPe1kmBLjm3Fml3ydMb/KUj0HTR8oLrAp5r6lqv8cJ3lQgl8uOm4y3zONH\nMmXRF5l3yqDovxU+gN44KGe+8NiGFnziDy+UfsuYd4pmcfXk7Iz7Aae+4n+PbmjBtXe8FKDDTmru\nJ6JrquYguu8Vl3aIaSELrOidecrvkEVFzrzT/+Szr+/Fb3nmgxJYsXmf1gYtzvlRhfbtz21Da3d5\nEx0UENXGEQA8+MouLV7C6LqQmc227+/Czx7blLrQogqWD2OaMMJJgmZuWUKUmDQ9/dny6VPB3vZi\nsvWk8PyWfdi8p0NYpmw9kgRHSLwuF3EG7UsTVuhLAKKxIwrO4n5cMlG4TJ1oeum89+RpgfuhQp+E\n6u43H1jMfd4EckR9402IX+jzbt78gVzC6YjuTR3VKLxPd5/XNJjWFMo1UUM/EPJAq+fkUSZ6p0iu\nMxFgLiCwUfdlzK+y4oPEAo+zsPfvNe8s+AK5OIY3w8U6vvz3Nfjy31ZLRbiksWVvB97x0+X40l2r\ng+UlueBqu0OfUxDGBFpjOiqqv5xef/+fx4/HhHln1HKB53yayOj0dNEdEqxDt//78wX89uktvkPA\nIG3qN9fqQosFcWUK9WR5juPBbQvTL1ySRtY0fS5k5p29HTGmn2Hgsp8sx1nffSQ2+lI+/LHVrlbX\nIJX5rNCXBNyNl2ouvbkTRwBINs+Md6D/x0lBoS8sWqeMT9+Q2rIgE49PX9m8U7bPCUggKE3pnoKm\nL6y+MJmYFt7FgVzCxwUdqdEEqqvKPGqbdwoWczYNedD94g06sau1G1/9xxoFaqrmnU58mj6qF9QC\nuXiEvgLl06e94S8iEEhHg56bfHltMz/yJm/TWjbvNN/vKtNT3NO0nnmnGCqn9rwi7LyEya1Zd6zY\nhtlfvpd5j3UYuvNgVyk4Rhh+/8xWfPlvq/HrpzZzyyQh4KrsHUz0/bZ9nZFphGHZut1Yue2AsIzb\nEimLEg6iPGsCwcBilSd8s5CkFj2pmrbuLY97q+mz0EaV5tiZM3E4fvSuhfjKW+eEljXn0xf82zv2\nacGIUCOItYkQLVhxfFc6mj4g2DYWXH55fAs1fSCh7aV5qKn2+hmGshdAccFTezBsUarOeQPeSAh9\nEc07TYGA4AaOwCcy71RFXGuhKfPOgkNpZTSX1JIwTV/Xouan6UXYVynb3wc7+7Bf8eQ8qNkrXnh8\n4x587/71gQA5IjpRh4WJcbWf8jnU9c8JC3SUpKZPFKWS9Y2cfPPDuPCHj0vRdv2pVPyqTGzqN7W0\nY8eBLq1nuX0vydbana34+WObtOpWwft//RwuufVJqbLsCNCSB4aMYrtauzH9+rvx4tboKTt0Uemi\nH81/nCJSXAImvR/78cMbY6knS7BCXwKIYnp50bETMW54ve/aI587C9+7/NiobDHBOt3wbojpaJJ0\neVfz5134hNE7cwS/v3oxv4AGCCHMkOLiZ/hCn/f9ue3lRVwNQ9jpEc2CL7gMJSCJpkFvpErV+fKP\nz24T3q+p4vPEAquEOz5MCH3BlA188DbmJk9fTWr6+vMF7oLH6jrvt+qNrOjtZ5mUDZEQQk9bs6hJ\n171+7I33Y+FNDwTv67GDHz/8qu83/T5M96sOOXrsbNvX5fNxUopo6z8pEJeVZdAAeHOIyTEufl7W\nWkGmj4tlzvneo0ztosl+3crR5m3frydsxgG3P6KsE6x+f/LVPQCSSYNg8kAxC4gi3CnnDE64s3xR\n262mz0IX7kbe1BiaPqYRC6aO9F0zRdsrlLBoTqAE0LBonoB4kqgiBCOG1ITy9dP/OB7nHj0+tJyL\nsJQNPF5Y8Gk/B36E5Vbk3QubSGih0FteRZgoBy1R3yg84AkewUJ1Lld67zLmnSLBMA5Nn8jvSrY6\n3ZQNjmPWzK+b449STEAuX5EvZ5zjBKJ56mHgGwsI3fodwNuIAumbaQEhPn0xizo6GyDWEywfJ+MC\naoKbtX6BY3Ea0Tt1m373y29gxufvCQ2kEcqP5PXv3Lc+Uj08mOxxl+eC46CrN8+8V6modOEv0UAu\nidVTrimOyPJZgBX6EkAcCw9NUdVfUI7wgC+i59qkkUN8xQNCChjmnYL2y35YF8ydgG9dNk+qbC5n\n1ryT5dNXzSsbwaePlZzdy1NQk8Bvo0snDv+ymupcib7uRry7N4/jb3oAD60TC5gAsGlPR6R2eLW+\nPDL0e7t71U686+dPF5/XMH4zBW+Aiue3lE2R3vKjx8M1X3DgfsPB6J0eoU+Tt7J5p9pJtm5v/udf\nVimVV65AAJm0EKJ2O45cPSKYknn9fOoaeIoJJ7mf5cl8JoVwE0t42HexbV9Rwybrb6hTT6VqLwoF\n4OivsP02Q5FR4arSffsqnX9AvHZV6KcSiuq0GTgUUEknBixWvde+cek8PORJVBnUTMnRLJcP93Pz\nlpXBtn1dpQVUN2UDD24RXfPOUKGQOobxskQLPqIpt/henIApnwnUVpXfmSh65+Y9HejszWNoXXCa\n2b6/Syky2a7WHu49WZ8ybyqPMPzM49OipOmDY1TT19WbL7Xn4XW7fffC6vGZd/p8z/zvLUth7aMg\nzmbIRGSUMXWOVJ8GeVk6cn0XLBTVf8wE+iRCCMc6Njx//+3FHRg+JNq26uePB/3pdHM7+q5X4mc+\nwDNLmyvbnqw1O2v8KCPCOVFcpv2m4N1jGVOkZAxW05cA4ogC1NRQ6/sdp3mnV9iaMEJs3smC0KdP\ngW8TTSQEWPa5s5j3uJo+3/PFXzrmnQAwbngdnzcQYUoMFTOi3oEQ40v/vQ5r3jgoZkoR1blcaUIU\naeDO+u4jRY0UY3WoiZh/0AeFwR+3T5/jAI9vaDFCCxCHolfKH0hF75Qxyw2lOfC/ik+lDmTfrtsf\nwUMAOY7cUt19efzkkdfYYfoDBy9+YdrnR+Y1ER74pwPT0UlZ/tamddnmvqdwOjI+fUnhmttX4gO/\nXsG8J8vOi1vFES1DUfFSRRnuOIpyniLWwKfXWVIRczP4LtMQheLSKtJrl3ecVZKyRgVW6EsAcZja\nNTXW4tkvvMkYPRffuXx+6W+ZMc8TaOXNO4l8WgWNj5Cm/a4Tp2LGGHauPJlALiSsrIgXQnDFoim4\n6NiJ3DI0WW//qmj6XKxrbvMlKDeB6ipSauiGXW2h5VkLdrVuSFsJ0Au5L12BzumkYvkfUQE+oqBL\nIPSFavrgjd7p7QPKvDPi6avqm1TdaMmW1m4HVcP/LHsV37p3He5Ysb10jRDgHy+9gTVvqKeUMAWt\n9jGeYY2bZet246Ww8PkqGu8EN6tp+3omGrpeRlDgXjednzM5sPP06bclbtO9g129mH793Xho7a4A\nn5X6DmgYmo4SxSPrd2P1Dv4heMG/cU2Ao+Rhhb4EUI6QZpbuuOH1XN8yXUxuaij9Xc4vyEfU7yKX\nUzDvpDgZxjAbFOGOj5yMr76Nn/6CG8iFcVnGFDRAB8X2XrKQLfSF+fTRm5u0Fo+aqnIgFxk/Kxaf\ncc6ndG19JY0N4Zt+cka5KOE2u7x0USl09wmSQktUxjLDzRf8JqhRhRUTfkK7Wrsj0zCFzoGAEe09\n5TD9jgN86o8v4k/P+SPb0hEieV0RZVxE8bpjvVvvPOLS/vPz2/EQZT4sR59z3dB3IEOHq+nj/G0a\nUQ8lTM+FUdeFLAklLiusd3zUl+7FL594PZyGilmyIaxvbgcA/OzRaKkvsix3sPrwha378fSmvTFU\nZobMVb96Dm/98RP8apzg3DjYYIW+BFDOXZedyVQJwuAjnM2yJOkcUZjYqHL3fPp0pWcWTh2Juupy\nYvj/PP8oX1EZ7Z1r1qkiIAbpiTWfPHr0uhdXEvAwVOeIT4jYHbJhZ3Fp8gAkrMtdoU/k0yfyg1Fh\n1bS2p6c/L+VPxr7v1+6V//b/1tf08frMf33DrrbQue/q37BN4gAV807OdcX2ufOA6hil06Osby5r\nwR1FPvwmmGYtRVw6D63dhftDIvX6efL8zdS8IF7pioP+BDR9Zro+OhGZ9SVuQVwVu1q7I+99eBr2\nb9z9SuizabTb9V7IO86g9Q1j4e3/8xSuHAiAZhJxvcKAeadnX2PNOy204Q6eODfpcZwI6ZAs+YhI\ntlXF39Fb9E8fPglTRjXwC7Oep35/4uwjMHvCsNJvrvbOU7Gbo+7UI8ZI1hIkI/ZxFPn0UWaL4fEL\nYkF1Va7kMwgAb/vv4snZzoNdmH793XiYisjJGgomD0AC/UmR7suzhR/RMyGXE4PQHyWEu6J5J0tg\ncHwn5ywqMl+lbN+8+QeP4a4Xdyg/pwrdMUU/5gpZPo2YzGabovO1f4ZvSFWg0zzWM+5Y+OD/8QVt\nWbq8bjH1jmXo8H36kvl6pQOKGGAnCT8w0xE+F3/zIfz26S1az5p4g8z5LeYNfdVARDbWgYR7pWJ9\n+hi5mGWRdcWHL5BLltWsEWCFvgRQ8umLcZMex2mSCR+6MCj59Hn+rpH0CfNp6UIaJKPpqx6YzMcM\nq2WWlQGPj6L5J122/DctsPi1NclNprRJsRtZ09Vs/OrJzb77LN7CEsBHAb0Y9XrykvHDu7Ohmtze\n9GsQkWPVRQ8tlmk5HXBEtd6wOlnPinzhTKCtuy+8kCTcU3rVaJwtbT343gMbmPdMfJ9amyzGNZM+\ncAFKgkO/bYIcjFHAy9MXpplUgWjpiNqbMqypRe/kX09rI+ua/PX05zH9+rvx+2fkhMCMywhcyMwh\nlZ7yIMl3E1dd9N7TBnKxMAJ38MQxbuOcw90PIs6xTxTMO3UWLF8QFtHC7cjl6XOjTnLz9IlMYSXK\nKOXp45OJFYQADbVVgesjhtQAAA52+TfgcfMpbd4JvqaPtzg7admsDUBkHSCTsoFFxwmYd0bTkMl8\nlj7NomJ1YcUPdvVh3g3348G1bFPFcDNY/2/3lF4mwqm3yL1rmpXqkUWpew0NQ93DRzXz1CA++rvn\nNeoMrzSfT+/7TBoygkIWtSnuOnyws7g23PLgxsTqTqM/qhjWAiVIstPbXxBGb2YhyZYmURerjic2\n7sGBTvl0Tyqwmj4LIzDtk8FCnKdGcQ7+qpy8bpAIfkk9r6np88LVMPJ8+mTy93E1fUScsoEWTAqF\ndCYoAmDR9FGB6/U1RUEwIPQlvObS9bkLryg5O0+AUv1mTTdVFEgmbDOzeU9HOXqnz5xT3byT7cMl\n39ok/E8f37jHCB3hhk0TKpT4fqfiZ2785ytYtf1A4DqNuN4FHdSGRn9MwhnPp89kM4Vm1pIVJTUP\ncjV9KQqDunltTexr0mh16eCIMTYfGDicCnsd59/yGC6+9UnpOtt7+rGnjZ/P1hSibDV034U7Djp6\n+vEfv3wmkmm6D7RPnw3kYmECpehrGTyBE0HGB433jCx0ffrkI37KgxucxUPFFQx5efpEgmMpGirP\ndRBBkwK/po9v3pk0RGOZFvp45lemQAu8NGveTaFOnj41806z70QUSCasqvf88hnPgRNF02f7pstc\n8T/62IbFl7v52dfRi795/PtkIPsNR02O7vJdCsIgQU86B2CIQESX9ZZxx6/oe+/uK+C2J1/H5T9d\nTvEXhG6ORrUoiGa+AxkqovdUaWtuGKSOSDPYZF1TuUp9fd5ALvR384dntkrReH1Ph1Kdb/reI0pC\noi6S3M+W6yr+7x4cbZRIFaUDb5PiyK+dBajFvLfQQo6x8bIoQiV6Z5xRsAiRM+903yVX0yexuvEm\nk1yOBIRJf54+f3n/Rj7ZwcUOzlL8v7PHb5LS0x+v0NfTLzaB6S+Zd/JTNvA1gGpnzaa/8YID7iaO\nxZl39HT3F+CGKcpTgq/PvFOCD9Hwkvl+3fo+9rvn8czr+yRqVAdPkSQT8MYL9xv0Rt4zEoRD8h73\nYEKCB6l3UTHRLuXAjd7pMP/UQli//vKJ13Hh3AnCMma0VvoHEWkKUKluoFNoNysYFA3TbLm+9Ukh\nLfPOOOE370y48oRgNX0JoBwC3PwQ/q/zZwMoBxiJAzpjX7apxclRMiiLV9MnKPfJs49gPiPHSxmu\n/EV8ZQbucYQ7mbyJrAXw4gUTmff8KRuyo+lTSZbbG7PQt3qHP0gIzUefx7yT69Mn2GirdLPpd+II\nhM4wdxHvSIrFp0+hrLv5iTMfX96QRtn9Bota1hCBUbYTPOXC5iTuwYRkVb5nGA/pynw+bWVYWb0q\nhHXyINpYmzJ7F/Gx40AXbvrXK+ZMzhhQOfDUnYKeei2G/GoD0H0PJsZRGgFT3Dk1X8hmygYjh8SO\n8OdAPeJnNKsyBvrNWKHPwgwGBk8cB6wfOmMmNi9dopUsPAxl80552keMG4pzZo/Ddy6fb5wfWRwz\ncbgROm6fsjR9POGOJwwC5YmW7s6ZYxvxwysXDtD33/P2fcBs0aPa+NvKN7j1xgGRpo9G3EJfGB/e\nQA8ijZ7KdR7i0PTxffrEz3qDJAX6RFELIlrQZWaHfIQhINulrswX9RWo+PSp1OV9j1v2duDUpQ97\n7pXL8Q8gzAwuE76KYUF8ktX08aJ38v1hZSE3tot1hEWQTcynL6F6pl9/t3TZKt2dpoHGCP0xI1MX\n1zkYTY+Tjh8AlPtKR4Bv7e7DH5/dKtXf/jx9g1Pqs+adCaB8clxZHznv477/M2egrbufea+mKofb\nrjpBsR71cqKJR8f3j/VqinX4T+pcejwhW6Tpc+f/QBFP3UHzTu/zfia7PJG9aD+6uLFfED1LlDIh\nDXjNv3jfoMhCTOW7Nf+N85e5MPNOAlIar14/rgIVHCYqyzKbgCS00tq+atRzJfPOglOK1hv1vdLv\n6rfLt2DHgS7m/eUcjYsOC6wxovsuwg56fGNKYnP26u72UNNsGToyMqzu65M6EJGkndTqn6Rm69N/\nehHfumx+KYgXD7KH0oWCgw/9ZgVmjm3EF5fMCS/vAI9uaMGZs8ZK0XcRt/bNfQMmg0GZhBlFXziR\nYDoftYp5pVUEzy/89WX8a9VOHOXJycyjU+lpNGRgNX0JIMc5bc86CPW/i1njh+H4aU3G6wmD9+RF\n9Iy3n6NM7iW/PQ+J8gm3uqbPjfwZnGg8z9PmnZ6/6U1tRw9b8I4ba5vbsK456EjNG99x+/QF+KB+\nu2Z/BALhjne9oLZZMy3ciHLqhe0nuvry2LavK8BXwLxTGMSGf29Pew92tXYraUN0oBrIJXCmElL1\ntXe85Ptd7THHDzfvND+pX/Wr5zh1CfjgO34GEKsArkD63O8/iiU/eiI+VhxzWglxnj65Rqe9/sdR\n/99XvoFH1reElpN9D09v2ouH1u3G/z7+utC03Yv33fas8D7zsCLmzb37jXH9TZGKq6FRJDmeo9S1\np73o6yiT/sKr6bMpGyy0UQ7kUpmfue7Ynzm2UZK+pE+fHhvamDspaCbqThxDaqpw7zWnB+6LNH2u\n1oAW7IbXlxXuKuadnb1qOXxM4dXd7czrPDOruKN30qD7qezTR0JN586bM95/HWrJ2U03VehXpqSB\n9PwNv7mlDBmWcHPN7Sux+JsPSdWvq4UD5DdH7gZLtaZ26vDEPbjxa4gViVIovkc+6HQGqhtVFf6M\nmHea8nUMqydyv5thJCtLt9y3GrUONQIyy7es94l3nlD1p+YhzVcn2vOlOabiqNr7ir9z3/oYaogf\n3vc1WJOzW/POBOBOilnT9D/+X2cLAytEPei462OnYnebucAN0onWfX+rdfoVi6bg9hXbAJRNUsYM\nrS3dd00qh9TmMHtCUCgUmbGUhT7/9Z+9Z1Hpb1EgFxodvelo+lhN3N3ajUv/5ykADKEr5eTJMj59\n7nW6barfrHlNH1/oVOEtT5m4ypp3Sh3IZCRipKm+dzX8MjzL1hhWbm9HeLJhHd8k1nVt8062Z6cS\nP6r407Ny4e3DeYhv/Mmbd0bQdiusxfyxEE8fyLCm4x8V54wRt3mnO3XkC+HWAmkgymFI2c+OX6aF\nky8wK4cnLgSeNhkMv2MGVuhLAFnV9E0Z1YApoxq492UnxluuWIDpY4JavRENNRjRUFP6/eT15zAp\nyn5csuV8QREUNS+fPvfIktB3zbmzcPM9a3HK4WNK97t6y5o+FkRJaKsHzDvpBXDCiPrS3/TjosUy\nreHEEgS27uvklu+PEsVDC/6OkcnT516n+9uRMPHzl5cuKgVRygh2wnQ+nfLf8tqesgM9H1HNO00J\nhHmPeaeuWAOw8/RlYeYW8VAKEuV5G+ub2zBiSE2gbFyfo++wzdCHcMM/X4n0fBLmnVlDEj66ovKs\n53WEPtr3WBdpxFJw6xTObTGyFWeTS8HB4qsiALc9cb9K75w/WM07rdCXAOgIRJWC8pgXD/5LFk6S\nojdp5JCQemT5kRdIVXvcuzidNHM0/v7J03z3XU1fHU/oE2j6aiVCmGUxvDMNVbOHvpRV3F7zUr7Q\nV/yf9skU5clj04mhrQY0ff7NhxPJ3JJG1EAufYZsYvOa5p00WAntf/7YJmZZaS2PQGPLLM+hIVv+\nsQ0teO9tz+La82YFypoI5JKppYyrvTfHpFDLmmBfSB7VsK+m+M60NH2OGcEizaHan9WUDSZopDig\nTMljgUAzh4B5p/XpSwDevE+ViLgPPGQnRXnzznDNDg+8D90VWOccNtz3W8QjjWqOT58XNVUEl3qE\n6CzOO6oLeNKaPvqVu/5ShAjMO8HR9Ckuj+54O8yjvY0CUTARtmDAo+Mvw4u6yEPUuUskoNL+bDRk\nR5uppO8l805Po2m/vzQgVBpQ97YMaN5XbjsQpFNwcN+aZpOslZHRoGW6/BidfxPqk8R9+iTKiDbQ\nvOjOKnPvtXes5N5LYyy6dYqTs8fHWGguUMWq0/aNdfsq7lfpf11Z3H1Fh9X0JQA3b7rJ0/UkkebQ\n/8EVx+KMI9XCMfug2OU8oe2eT5+O1q4+TBhRj7fOn4hZ44Phf8PgRu/MCY5aCCH4wRULsL65Da/s\nbM2kWVEYS3SXh23qTSNQv4Smj+fTp3raXKZj5sWJghmoLMQi804WFRX2I5t3hrRDdfTQ5p2q026V\nJ2VDKGQ1fSqFdSqjbtVVFycZVsS6vOPgobW7lGvvkoh+V2ZHv63KUYk5A7DStUQsyHxrPJ7jaotU\nIBeB1Pfjhzfis28+CkBQmyz77f71hR34/jsXyBVOAO6cltaez3S1fnoDSgyzVSjUH58gb5OzWxgB\ny1yoEhAoeHxiAAAgAElEQVTnfOWNWCn6uMYOrcfooXVKtH0+fYY0fSOG1GDKqAbUVOUwb/IIJZou\naiU0fS5crrOYIFS0gAPFzfLzW/aXfpsy39OFP4gJuwzfp0/tOyjRMTSzqmh3AP5m2x8Vzwntk8Ai\nK1jiZYao2y/dfcGxENa9bd39Stri6Oadxf+zdkjnsrN6x8GAYES/H1foY6VL0V2HLv/pcumyUbru\nmK/ep/YA91BEnwcaQssSyYqSEkKjau7VffrCy4jWsT3t7CBGpsw70xDd3RpFXUO/g5e2HcArb7TG\nxZK/bgN9ko4G1Wyl9PvxrouD1bzTavoSQGnwZGwTEQZ3YohD7njws2dyIzzJgObJa7q3YMrI0nXV\nDY6qkHXbVYvwx2e34f2nTA8tW1PNjt7JQikwQxaFPgmeLvvJU6W/E9f0Ud+ZGz2UgJ+yoeTTR7VN\nJlcbi44ooI8KRLmqmHckNsAO/H2URHS5fMFBW3cfmhnRgmWmxW/cszYGrsTIS4xb3RxtOsPDQTGQ\n1Ft/XMxtt3jGKNz+kZOZ9N1k2SxNn8lIqjw/v6wtdQ6A5oPduOi/9fICCn369FiKDdz5wnFisdqR\ncc9w3dlVrApM+Uenad4pG2UcAC6+9UkAwOalSyLXb9y8U58VI3Qc6v+44O2XXa36+9Mswwp9CSBX\noZo+F3E4Io8bVo9xw8L9nmQ3Vc9+4Vx0DqQwmDKqAe86cSr++OzW0OfnTx6Bdc1tGD6k+CmoCn3n\nzB6Pc2aPDy+IcsoGGUFOZtFIC6xgNSI+E/fpo36XojoS/qLhbjDoWDuqn6xbl1nzTrGgKoMCpdmj\n82HRCDq4y9fFQn/BwcGuPvZNCdpPbNwjXVfAvFPxLbptLZhTNUA13yMLb/7BY/jjh04q/fb6MNKk\nRZo+E3n6wpCFpY5+7/9a9QZ2Kx40mpx+k4pEKVON0YOeiJo+HujDKV2IKMQlEHr7N4spG8wggXbR\nFieGq6T3Yt53tXrHQbOVZQTWvDMBuMMqaykbsgKd/bEriL7rxKkAgLHD6jBtdDlthGxuxBsvnou/\nfeJUTG4qpq4gMX4RU5qKwV9kmuuOlYUDWstTjxgdF1tc8OpUfV/9KZ92yKRscDcXAU1fQTHqYsm8\n08x2UZyygXGNUzZo3hn+jEmII09ma170b9hCymqyrvvcgxxfvLJlQPF3XbVA05eA8FFpkap5MNmK\npHqEbxkQD2RmOndzvVtBexLnGIr9MFVG8I7Yvu8/sCHS8yrw8lpK2ZDgJ15OHRTB+kyCX795ZwZP\n3A3ACn0JoNJ9+mKP3hkhF93Nb5/HNIeQNamtr6nymYPG8aFfc+6ReP3mt5R8E1V8+hZMGYlXbjwf\n5x8zwThfYZBJMeFC1M1JC300L15NIz8oSvF/1lhU4b5sJqrwkADF025eXfKcvXGgi6LpEYQjvh8Z\nS4AktEtR4XaJkpBvvKDLA0+7yzm0oH67w5gn9JncsP3jpTf4jCQA7qFBxsxNTfAgZyViRhtoEu58\n+LYQ81pv80xNGTI+y6bhJR9XyoZnI0Qr/vfqnUrl/e0JXpOmI9Hxe9p78VpLO7t+05o+6rfDaugg\ngxX6EoBrDldpp58ut3GPfR36YWufO9GqLhxx+Tx4F2sZoc/d3BECNNRWp3LqxKtTdRjzQnLHBZo9\nV+gkCBf6aGFNNUEwLyCMLlS1Mjxe71uzy1PGr/lLQigXucfJNDHJ4e/3TTPXNyYohY1f+ncPI3BO\nvgAc4JnayvJB/c+7Hye27evEwc5o7QiDVLTMiI1VGdtRxmNc24/bn9sWWiZHCJd3bvMdMzynYUkg\nc5CW5m7wM7e/lGLtYtz98k686XuPMu9F6jOZWAqevwdrIBcr9CWAJfMPAwCcMStC6oEUkJSQqrNB\nDnvE/WBV2xCHcMUKOhMKSvOUhtDHO1VmCSJCn76Eo3cGA7kU6yeEL0TdO5C3jPZXdBzgql89J103\nLyCMLsTRO/03G2qrJGn6o3fKaOHE0f7C63xp2wGc/u1lbNrhj0eC7BRACzIiLWuZthxxU23k1Udv\nbN3fbYz0B8vW78YDr6inbGDBq0n0cZDA0nH6t5fh7O89wtWkJGdOGf8YUAnoFTXojOqa+e/VzVjz\nhtj/qSpH5HwNvQcuMb7BuJfT8oH54JAc6EBg9DVpOhHrj3tL6t0fWPNOC20cN7UJm5cu0crtlgXE\nHUGSFRjEhe43rmtSG0dTaZIy/l5zJhaTwA+tcwPMmOYqHDzrTlVTvbRN+7zRQ8M0Z/RE36cYhMbd\nMInGtA49FljdKhfEwV8uKaGcr6UyOz7CqK1vbpOjY5AtU5EteZ9SyTy5dIFP47nNZpLYi2glpV3Z\n19ErrCvq2Kos25wiRPF+S38ZbliYNYcoiBYPBcfMOIrdlJNRgaqAm3V430NLWw9e3d2WsAbVCfBh\nBAzLHs6tQQMr9FlwkdSkVK2xQZY9QVNtQiKaPolnvvOOY3Hnx07B+OH1sfEVBlPmnX0Jp2yg0Zv3\nJmcXl6WbrJKQukjfNe9UeowLkcBMvwfZKuk8fSKZL4k5IOnRcf4tj6GtO2gWWAoW4A16E0JLhXcT\nfckPRET9jpkPF7xvOwsb2qS0RlloqxdCrbzM88Y4KUNo3slhyjHke5rG68lacCrTWLVdLrKl6W8j\ncLhlGN610Gr6LCxiQlUV/+OaN4mdCD3cvFPPjzIOjRqtKZWZTIbUVuH4aU0eGsbZCgVf6FPrU1Vt\nmWn0DAhu65rbQnmn8+t19aoKfcX/zUXvFN1jnDBL0HRA+/SFv59K2jTLCECiJPFekc8Ub+0eM0tC\n9L/nHZ6APF4EzDuFJn7xa0+ysuWNaqViUkiKO/1AuZ7w60kdCrjICTR9vANcc4Fc4h2NwkAxhP+9\nVZJgaCoYju6rKJl36j2uUI+nhsEp81mhz4KPxgHTwkXTm0JKRkNNjj0MNy9dglGNtVo03T23aiAM\nE6c7915zekgd6jRTOXXiVKm6GKcdvdPNVfb4xj3Y094rfJYW1jo1NX2mkrOLAgLQd2Q3uA4VnKbg\n8OtJYtilsfkR1um5tXzTXjEdSdbP/f6jRtr5yPoWIR/uGBCbPEZmg0nfpyHNwH62mBsxfvPORNua\ngX7VQVHTJy5zsLMPd724o/TbqSixSAMV3jgZ9qOuH3QdcQvwhcEv81mhz4KPUY21uOdTp+Nbl82P\ntR4d/6ewJ3KliKmKdA186bMnDBfS1Dl9NuUjpgKeoKkqSCednD1Qv4LQSTe5W1HTV44CauZ95QV9\nHcWXhDYb5dUj40Af2edXgmfjAREEJ9fuVrOlvRdrd7aarTcmBDdHgrImhD6+2maAn2CB9bvacOfz\n26NXLsuLYpkIHBgsxUY5TH44Fb5myfN3wlJ5jpBQ3j9zx0qf0GcqeqcKtu/vVE5hwypdir6NwRPM\nhUaSY6hkhWG4SvrdeNdBU9Y6WYMV+iwAADddfAy+eem8wPU5E4ejvkYuKqAuWD59M8Y0MkrKw6Wo\nHsglBp8+amLxNld2XklD0cezut3dJpdg122bN11AEohyPkwLa53K5p3FujnKa2WIBGaWT5/UptAJ\nfheuELh1byd+9/QWVTYjIe4TY6ZwLCxf/F9G4M+KLoJuY9xchc2rvM3ZZ/+cXKh4x0nGvFN6jfGa\nV2ruXpMQcOMa02F8NR/s9v1WXbunX383nmFo5mX7Y+veTpz2rWW45aGNSvVqH74p1ZIuzAlbaoRK\nhx0xdRY9PfiS0MdTZeqoTpsBi2zgPSdPT61u74nKU9efg7bufowfXid8JjRPn4SZU1KgBTuvYCGr\nEcpSIBc2gmVzhChrBU0gSpUBnz5F805XeDKlmRVFxQuMbckqWVo9l+93/mw5mlu7cfVpM6i6+Ija\n0jRMAdk+KrTgZJYxlQieqn0S8GNU1BCrQ0wjjle6t70Ho4cG1wXRNJWENkI+bUd0XqIIE2n69OnA\n0TDw/NeqnVg8c7RCHWXsaisKnU++ugfXnjdLqd4g3Yx0YoVDxYLBBLwWMHFHrU8LVtNnkSlMHDkE\nR00YhpENYb584g/S/V51JwmTORVFefpk55Us5eljI9jRlRj9ijbp6FYU+tzxVm1I1dcriHyqm7KB\nFRH05R3FaGz7OsU+j3EgKz59rd39aO/pL93ZsKtdhlDFwYjIF0Ykht3Zkh89oVSVw/lbBTLP6dCO\ndfOa0THJazNvmSgYMu+UnV/KWqXo5p2Jmhwn8L7ZPrzx10vX7/5vSiCjqXibNEitO63QZzE4oTuB\nA8AzX3gTfv6e480y5IF3wpK1909jAgqrM5SnlCbNKNpFuk3q0TuLdeukIWFBqOnTU/SVfBw//aYj\n8aUlRwMArvz508JnRN9RZJe+mDcPKpuyk7/5kBI/WdlfJ52ygUsjxg5pbu0OL0RBdXO4u7Ubm/d0\nKD0j6wMmq2kT0pAqwy7luy4gFNcrVD3cMaWllSVTOig2QD8r80LWoBxjgXqu/H8SJsiDU+qz5p0W\nFQn5lA3qtN3ceKYg8umTnVfScCoOM1EMM9+sxCmTfle9ikFoSkKfIA2JCm578nXuPd2FLz+QomFI\nbRXqOP66g33TwmtfW09/fD5NsVB1actLfSbaF3awkoXxo/N9nPjNhwAUI0cD6eW149Yl0abIPn0x\nNMjRoGuKD3mhr/i2TQSbdt8TSSJlA2OQmn6HbME2AdPp2Gug6vP69FXiBkYCVtNnUZEIMx0sp2xI\ngJkQ0KzmfJo+OTTUxhtMhwVZwXqgdOB+Wl3/9KZ92s/SbRYlR2fBLW3KvFOmLheyWo0DncXE5DnC\n10jSm3pRLyQQvNM4xD5vKnT06je9oQhq+sy0j0tDkZ804PehNMdQwPdTknQUflS0UFwlrE/Rl+wL\nchz1Gh0nGZ9MF6VPUtm8U8/0MQvfiCx0TViN1e9q+AzTFc3Dg1Tms0KfRWWitlo8dMundunPrKLk\n7KceMUaKxvD6GqM8ySBMiEhArkkc9HARpUwQPW9K0ycCy6yMjoDHwtf++QqA4jjk5RNMdkGPtzJ2\n0Jbi/798IqhJjc28LcZ2upRlot2Z0WSE3U9/3jWJlrYefPXvq9HH0PzrrDHa5p1SwkTUvo/2PG/m\n4/HFLW/oS5SlYjJwh7epaaRsSCS/qsbriTwy414rOH8PJsRq3kkIuQDADwFUAfiF4zhLqfvXArga\nQD+AFgAfcBwn2XjhFhWJmpBNdTmQi4P7rjkDHb39CXDF4YX+7blw67uPk6IxfEjyQl+4T5+3wOBw\nbKA3GnlBIBUR0tH0AfeuaZZ+vipHpKOMxrnWphm986Z/vcK/KUMnNlPQaP5P4iZE5zmMvwr89IW4\nfcU2AMDpR47FObPH+e5lTb6V0vSl8c0pljcWyEWSSDkXoip9xjUJKrGaexs370x3kJcDuRimS1tI\nZOxbjgOx7UwIIVUAbgVwIYA5AN5FCJlDFXsRwCLHceYD+AuAb8fFj8XgQm2VeOiWfPoAHDVhGI6b\n2pQAV2yIoncOkTTbHJGC0MfTArnwCn09fekmYDcFetJfsWW/Fh1TgVxEiLpA5QjhaiRVzVrjhunF\nOAsWACZBt0bUurgCuaQtVCSBoo+WH7ICurecfv9ICBOcItJ8xvDuHA0BTscklEkn5LcLdzkzMTe4\nJAZLYnameacOHc2upQO5JJFOYbDOYXEeR58I4FXHcTY5jtML4E8ALvYWcBxnmeM4nQM/nwYwOUZ+\nLAYRakKEvnJy9ux9uTqpDIbVJx9zKSx4jLcZ7/7FMzFzkwzo0bJy2wEtOlUJmHdu29/p+61aY06g\n6VMJDBJ1YyNlsmb4jFc0L6jUlJXpJXhiHbfJbHY0fTKaLVNgzd16Zm56zMl9K2bKmIQDh1spbwOf\nNI/uPGZi3BRKwgm/TJzfqI5MpJyqIpXJL8l5LSOTu2HEKfRNArDN83v7wDUePgjg3zHyYzGIECb0\nuQJLFjZl9PyrI/TVVOUwc2yjGYYkEabpCzMNrMQEtaYWsiQ0fQ+8sqv097A69UOBKsqnz3GcUoqI\nRH36ImgvZGugIVJkJtH2cJ84ZYqCX6KSegijkbY5WFzI5YiRQC6qUDI99FTkTTnjOHLCQGyaPg2T\nZSO8SJrw6eb2TSuypQhaBxEpz4kyiMLHmh2tQXrUb++6kJU2m0YmQjEQQv4DwCIA3+Hc/zAhZAUh\nZEVLS0uyzFlkEmE+fS6yYKVGa8x05YG3zD3MADfyCNP0hQmFlQhTE72sr5wp3PnxU5S3GTni53PV\n9oOlv5P8bkxpL9TqFGj6lHz64kFUvyLxBs6c+VoAGZoSHDjGhU/WnCdrTeLTIUQ0c5Ot5+iv3KtN\nxyR0TQFNCE9BGg4+/aeVA3WU7+madzJ5dELuI3tChfKco1WH/FNdvXl0DsRiMBG98xv3rA0t4zPB\njlBXlhGn0LcDwBTP78kD13wghJwL4IsALnIcp4dFyHGcnzuOs8hxnEVjx46NhVmLykKYTXc5xHX6\nn24wkIvezihpGStMIxnGT9YWNRmYGi9pCMTdfWqJ5HM5v0/f5r3epNS05ojfL5WYskGo6YupTpXv\nQdnUyv2DBK7wy0YA28fHs2HK2Ldvip8cw6dPWuhLqE+i1hNbcCJVDVosXPBh0v8ubbcS48sP+4OP\nFcfeeH9gnjYfoCZe+llEnELfcwCOJITMIITUArgSwD+8BQghCwH8DEWBb3eMvFhkHNeceyRueBsd\n50cfubLUlz4MzcBJixEhFrShwqtM1z/wmTPkGaogJOHT5wUB0K0YTKeKEJ9g755+A0Ahwbg8aaRs\nEPr0xSicuTC9KXPbI5WywYAaNzw5e/oTb9GcUa+jeYGMCCGBvk1UKy4VFTJamfjMO8vo7AmPpl0w\nZN4pu7E3ad7pHROVEsxFaP3AGC9xf+Ouq4G3Lvf/uHo0/VkrfsQm9DmO0w/gkwDuA7AWwB2O46wh\nhNxICLlooNh3AAwF8GdCyEpCyD845CwGOa45dxauOnWGMXo5TVONOGBsgkpYexTq0xcm9En0fdYs\nRGU3cMdNHSm8X1MBSQxzOb42l17QRa8y6iuU6XLTgqGpQC46MG3OeqCzVyl6rpH28YgMXF+3s81E\nLQFcfOuT6JAQGErsaI4bVj4+QN80f4Cb8l9xmndyyqS5FNKmth0eX0N3CqKnIlpQ1K9btby5QxHR\ncMnA1sSHONgxrUmLu8+8Y3Sw+iXHGhLQcZx7ANxDXfuK5+9z46zf4tCFu5nNgk+fKSQtH4X69IUG\ncglHEqGXVSA7z9dVi1NtJO3Tp9ONOUrT58Xg9+kzQ2dfR698nSqtUCi64MYH0EClfhE+bkJ7EkLk\nry8GPDmM4KVtB7Biy36cOSvczcPx/a3W6L58AfU1wW+8Kkdi8jMTw50nTX0HQp9PQ3XQ9anSNXVg\nGxA8Qsqpzn0pWD7GAtVALokG+3LiqVOo8TZbVWaQ/eNoCwsNnDB9FABIbQ7ihk60ziwgCZ++rPWM\n7Gastjo4dQ73pNVIWuiTwcULJvp+V+UIV3NBO84LeyXy+I5X6vvOfesD/o5i8075yi767yeV+ZGK\noKhIs7PX3764N/VZOgQXvS/dQ6X+vLx5pyzMmCpKmG7yNH2Sbz6ud6tlNhkDM2H9YyJ1gdR7qnCx\nQu5Q10xdF9/6JB5Zvzv2PsvSvBYXrNBnMShx7JSRePUbF+KMDAh9px0xxgidpGXHMMFFxI8sr5kT\niCUnfVZKhiEebUvyQl94fTRPOUIEObKytfpF4eaZ1/fhN8s3+66lYQGQhK8gAGzY1YZbl70aC+0y\nDblrceFBT7oSHooh/zXNOzlOraa+6zj7SkQ6bRNP9WeSq9dEhEgXSfpEm0ISWq8odO5etdP8+KXo\neQ8DB6sAmHzGZwuLhFAdFokkAWxeusQYraQdwsMEMpFPn+yEmTWZ74hxQ6XKsfje1VoOPpxFTR/9\nPovmneyy7vsrBwYRaFMi8pXE4uoNCgCINX1/fSEe00QVROmSy37yFNq6+X5vSW6k48LVv1mhVF51\njPE0fabqiNN/ivetei8LBcMYuHMk7DuDAXLU+WDNy0HzzpD+MWDe6RMewurLCFTTvCTNfz7hCq1P\nn4VFBvDI585Ca3df2mykguQ1ffx7b5k3AWsNBGvIUmSzmWMaMW/yCMnSYr6TSM7uhczYoIX0qhzh\n+m26C17cy15rd186KRsEqr5XdgaT+CaNKPuNnn6xmsHEXiZNX2nZzZgDffNOXiCXKAnDs2BWK9Ud\nMbxbkczHWwP2tvcaiaRJXwo171SrUhi9M2s+6yahpbmNMIALhfK3Z6pbA2PDDNlMI31ViIWFAqaP\nacT8yeLIiRZmINL0mRLWsrQmjh1WJ102TKbLpKYvYN7Jb0cgP5KAbpR3+P37N6Ry4p11805dEBL+\nZRoJRFIhuyPdTSYvZUOUZpuIDCg2wXN90nj3fcwI6EQES9smrpKJ9//6uaicKCFMU6pGq0iDgL9W\nVsgnBIDNa9JzQH/BCR3jUSGrDa9kWKHPwqJCkLQcITTvNMRLloQ+QuSF2TC+Myn0USwVhUBe9E75\nJS+KANXdl5cSQpJM2ZAFRBHMTARYCkd6/SerPfFt4AyY7Ll0+OZ64koczt8ykMm/WKYd7d3E9W3w\n+BK9TiOmplR7uO825D6fvtSlYJmMzUGq0TuTRtbn7EqBFfosLCoESZuKiAQXU5xkyfxFRXsZVjbx\nlA0SZWieqkQ+ffRvwXr70rYDErWz0ZsvpJKyIe0NxN52caqHKOyFjc24zAyT6lH5zbKjPb/wq+Cb\ndybRfpmNOZc/n5+ZXh3cZ8LuO5p0Tch8kvfc/lGdG1iCadrziw5++/RmpfJJtzBf0Det5kE4l1Te\nK5SC9emzsLBgQpSnT8aMTKqO7Mh8RU2fJD9h5ZL26ZMBO5CLpNQXE/rzcgt5dF8mfzsf3dCCR9e3\nRCOqCLcJT2/aF142zv434tM3SHdEA5AJhqJyj76fVvTOuJ4PbbtAZ5f0TMl9t6X70euQsX7I2hf0\nzXvWce8x355ER/3l+e347uXHRmGrhHwh2T7L2vsxBSv0WVhYMCGKzgmYmRSzFMhFRSkQbt6ZrBGF\njEYjIPTl+Ca89KY+rmiN/YWCnHmn4fp/9ugmo/TSAmsDG2reacKnLzIFfdBjXWSKGVaGB175giMy\nD1SoRbMDZbRWYUJNsYyAjpbUE2La6ujRjUsrLSpnJHgMHf44YQzGM5mCJwVLXIFcDgVY804LiwpB\n0j4Aouidpk7647bunNw0RLpsTkF7GVYyi5o++n1WEcLtf90NkCr6JDV9gwEq3690hEpGsdBALubd\npBJFEvMgb36LkvvPjABTpLJ1bydO+MaD2L6/03g9cZhhOjHRlaMhbw6sVp6PLKds0GqfIXPuKG3O\nF5JNFJM1n0tTsEKfhUWFgM41FjeE2iND82HcydmvWDRFKVeiNDsh5cYPr5eu0wRk2A5E78zxzTuT\nMt/jhcanMUjXXy5km8t6T2FaXxPvlkUha+8oCjtx++2pbl/dV+o+9afntqKlrQd/e9GTU7J0UsOp\n02deyq9fZ3wYsfqIaSmgeeP7PLLLq9IH0k/OLly6zch8iX/vRU1fcvVlbDozBiv0WVhUCHo1EgZH\ngTD5OhwjliuixWnMUPkUCjr0g2VVArmIMXZYHV748nnylScApk8fZwUolDZAmjshScj69B1qkO0T\nlu9Q2Ng0ka6i0k7BVfnla/rEpp+yPGh3H/Uca87KoqYPgg177NE7JaHr08dMXC4Xv1OtIgUkEZ8k\n6TmgmEbFbJ1Zj1gaB6zQZ2FRIUha0yeKQGnqJFOk6TNhIqkuyMmmbBCXq6/JYVRjrXTdUSHTTLo7\nqwSaPndxbevux++e3hLb9qQvL+nTNwgWYNkmHDFuqPRmlxZOCD8Lh1HwEsBn6TVFGTMllyyqL4V5\n8lJufJhPn/ftiFiNI+m2o0lX9RHWdEazFmZuaULQTFvTJ4KWb6WhsS3OMylGfwzRO0VI1pg0OVih\nz8IiBrzj+MnGacqawpmCKHqnKBqbUh2iVIAGNq8q5qNKWsGQ+/U1VfLEEgKtuRUlZ3cX118+8Tq+\n9LfVeH7L/lh4SnohrwSo+f4FryXhTfrs6+wIpJWmAeSB69PqGPIz035O4oBEgs7ana3852OQzXRT\nNpiAjHnn2p2t6Ojp595XoQ+UD2OEydlj7A93qv/Zo6/hwh8+7q9Xg96N/1oTuGaa/bD+KMTg0yc8\n/BgcU1kANnqnhUUM+O7lxxoLVewiaU2fSCAzNSGKNGYmNq8q2kICcykb6qoTjt4p0VsBnz5CuP1P\na5B6+vP6zAkg79M3SFdgBhzIf195DZ++OJGlt+QgStAV9nNiCzMVYV2Nr5XbDg48J6IJYRnv9TtW\nbOfS0TH/lWmOzptQ9q/TqKS3v4ALf/g4aqqIdJ0rtx3Ax373PO695gx29E4JGnF+Ky5PN/87mIZB\np4/uebnZCJ0oyPtMhLMXKK1SYDV9FhYVgt6ENX0igangmJl2xZq+6DWItJWs+mQXMhOavm9fNp97\nb/PSJZg9YZgcM5KgNX0i886k3Ed7+wuZEhRihWRDRX5jNPKMF5WWzFcUslKoNwaBgyf4RNE16PZN\n3nGwp71noH4RfWegTGSvPo0nwsw7+QK4aJ43cdhD06BJugdcfQPfkkyVP3hgA3Ye7MYLW9kWEGHB\ncFq7+9DVG89BWhjSNFuM4mtYKOgf4liUYYU+C4sKQdJCn9g00szkK6rDxOa1SoFGUT6Ua1eYQFoj\nyncxgPpasWB418dPleJFFixNH9+8M5nF1Zt7SYRDaal/fU8HWtp6pMrSmr40z7+/fe96dPWls5Fl\nwXH0D45Eydl1Q/B7n1MZzwWPBKobYVLlGa1PP6ztCgcZaWNPew/+8jxfE0qDNR7cV8Ybf/NvuB8X\n3/qkFn8yMB29k0nHDBlp5J0YzDsFnVEp41UVVuizsEgB7zpxCuYcNlzpmb7EzTtFJ7Dx12FC6FPb\n9DMS48AAACAASURBVJnT9JmgMaS2Cs9+8U1Y9rmzwmlJBXKhNX3yydnjgijZtRf2gJeNfkbECGv4\nVIbu4QVf08cfiyrfjApb0sKaBG2ZsRFHygZRv7k8Mc0kDXz3wUAu/h+sOj7355ci1pnuhJVE9fK5\nRIOHFjr08oVk14G032FcsD59FhYp4Oa38037eMibiLWuAGH0TkMTokhYkU+VzodKAFBF+TAyZILM\njBtWDxiy8qSVjzlBcvakos8lnXspTcRhVsV6T2n69GUJvnGl3PU8bZ55885QDaHnfpRIt7Ksayn6\npNpgXpjUAf0OI69ljMdLgVxIuuaULCQ935qqL1+Qi/SsArG5tNGqMgOr6bOwqBDcdMlcnDFrbGL1\nheVmrzUQrCRuTZ8KEQL5TYYJgTTpvTkrT59sIBcT7WVBdjOYtY2TDuI4s6E1fYTE9aYqE7oCcNk8\nz39dZKao8npVxnOBoSmJSyumSyfLaVfCeIsq9LGeTjtlQxrnPrxeNGo1wvku40BS1i5Jwwp9FhYV\ngokjh+Bbl82Ljf6X3zrH97ual7kbxU1RQ4hPmgyEgVwknv/IGTO16QfqI/IbEyOmp9FJKIEOzFMM\n5MIum9R6J6vpGwzrbxybCLamz3g1FYlI0Ts5wpWIporGTkVC9B4W0I+x89JF01LGoukTiV4x+58J\nzTsRz2GMl2QaxzDigClmGqyjOY5Scyx+ocJ+GpywQp+FRQVBJkCIPm1KEySoynEcNNZFtw5XyaPH\nQthJvsqCq1Y2OkxuzmVoVVFjJyeI3pmcT5/cFmQwLMC3P7fNOE2WT5/16itC1SzS/6yeYBcHfLxI\nBJ6IyqJW8m6JAnFpECWqFl+Iat3JMu9M2BVDBeYCufgJRfV1VTn8S8ZnMf460oAV+iwsKgijG2tj\no00LUGGBXBprowt9hAD/+n+nSfHDQpgMLNL0/eqqEwK8yG4ykg8yY6A+6neO8NuRmNCXsHN+mli1\n/aBxmqz3ZDV9Zah8Y6t3lN8PN5CLyLwzZCDzoneGPcd6x6J5iuvTJ6wl/HnxM2FtD59bzcyp6s9E\nN+8MPu9Nzp4GwlwzTEBa02dMyLQwASv0WVhUEAghuOvjp+CHVy4wTzvktxcOHDTUsc07RykIpoQQ\nzJ00QoofFsI0haLb1bRmU8n/z4BPX2QKHloSvAeidxKRps8IW6FwJFM2WLDRz3hRVuYrQ2Vs/Wb5\n5vJzKAfi8NFDdKEKAzQcx0FPf3iKC6F5p+dtm8rTp/N8mDmjoxmwKR7zTv+FOA643FQqWQyqFNd8\nK5PKRJwiIeTgwBEaCSujUHBw98s7jdGrFFihz8KiwrBwahPmcQSlKKDXJ9GCVSgAQznmncdMVEtF\nwWcovEi40CcIFENXkLRPX9KaPqo6FfPOuFiVzb1k5UI2WBF9ve9q7iRD32KFQ9VvlOvTJ3aWCmBP\new9+/8wWZvHv3b8BR33p3tDchpLWnaFlZL8hrSAkEfjiHaCdOGOUOhsO0NWb9+c2DA3kolxNoM4A\nzUK6mj4RjGn6JMvFYd5pAvs7e80RqyBYoc/CogIhCrKii2B0R35ZBw6uPW8W854pYcaIpk90Lyjz\nDepALnR9VYLk7Inm6cuAQGciKFEaCNugD8ZYnipmivrJ2dnXi3klORoNxrVP/uEFfPGu1djU0h7w\nMfzTc1sBAO3d/SG8OMG/YwxAoafpKz/DDi4jptvdlw8eYGg25Oiv3Ivr7lzFva8kyGvCG/01C5GH\nZXPlqdEU/y5dN1PdgIlwESbW3/TfSjqwQp+FRQWCNk00gYB5p6CKGy+ei5ENtbj+wtmBeyoRM4X8\nSJksht0XafqC9ckv0AZyCBqcfXUEZGHKBkqYiEswkzfZiXeJjhpQKC0EUjbAL+hVaLPEUBgKKht6\nb19586zJVs6qa297UZvQX/CPcr8AKIYvZUNIWZq277pkx0U1w2QNOZFZLADM/vK9WNfc5rtW0DD9\ndtv45+e3M3lzeRH9lqtH/LwrwGbFvLMQxrAiVF6L7PiVG9vm1gEVUhl5jUZghT4LiwoEHX7fBIIT\nG7+OWeOLGcNZ5mWv7+kww49EmVxIPwgna4amTxZmNH0JryRUdTXV8pq+uDR/sqZVcWsDK3VNDwvk\nkpVNp0lICy+cwClyz3KuO4KbmkTDxrbPp6+k6uDTiRyYJAbfO73InerQMdWM3l/B5911MUeyoW33\nC17RJ9OCgn+dsbnbOXS1cyZhhT4LiwpEVRxCH7U4yewXWaGpt+ztNMOPAZ8+sabPf09lQTEidCeQ\nsuGJ687GomlNger+/olT0VBbze0feqFmCfcmUIgl+ZIG0t+XCXHc1JHM6/15htDn+TuGaSJ1qPic\nqQi93jHvbpJZWiITw1WFBlMoERDgCjGyBywRA7kIBWYFyKZz8T3DmKdooYw2s9WZ2sJGVdqJvUWm\nl0aC47Dq4PJiRjh0RJVYSMMKfRYWFYg4fPoCgT4kNkz5GBc3mRNS1qbWK5CJmkDfKxTkI8zx/BlV\nkMR+fHJTA8YOqwPgf5/HThkZuOYFvWmJ6zUXClkw7syGzPfmOeO597bv72JeD/v+KtVsVQTZsRBp\nzGpo5cKEB5/mUUHzIjIF3XGgE9fesRK9/YUSnagHNPGkbOBrhkQ5EXUERVVEzanHDOTCOTRIC6om\nwqFwgv32o4c2sosaNC01Gr1TxfTbWK3pwwp9FhYViDh8+ugNokwNcSah1dX0yZq30XdUFoGRDbV4\n9+Kp0uVZMLkhFwnIbj2s6ngsbNzd7vsdl3Avu7GLO61DFswgP/+Wo3Hu0WzBb3dbD/M6O3qn59DD\nDGuZglbycIlHZHz6CoLUAyrCm5+mmC/fht0VJgZ+/+7prfjrCzuwfNPecnkOQWlhWbIc7xmmYlLw\nnfPmXZ33zGp6UCMV/4GW+12mJfMFUo34NH2ymjd+ORUNqXT0TpmUDQY7NC7rlazDCn0WFhUIkXnn\n7R8+SYtmMGVD+DNpz5ssnz7ZTS+90Rdt6FiQKXvxgonc9BpJyRluPaz6ZAXP+Hz65HxDDoXlmQD4\n0pKjlZ4JS9mQAVnWOKT9QBXpskzg2OadeqORJxiFHZyxzE5F5aIe0Oh861IHN9z6+OVV+5qdyJ76\nHSIEyiDsiQJn/MhiXXOr3oNg16uj6RPx7jjy48yfZ1J/bHpNSk1Ma2mb4KYFK/RZWFQgRD5li2eO\n1qIZTGGQrnmnDFjd4PdpEmj6Aqf4anXLnJj+8MqF3I28SQd/0ea+pOlj1CcrFMSl0U0qZUMlCD+E\nqGt/w06rsxBEwjziHzA62qfwYCbsAmHv0MtLXsKfkefzKPud6X2P4VoaHnh9rXoIB+jNU6qPrG9u\nw0FPjjemeWeJqF5C8QtueVz5GRHCNLE69GT7mm7/L594HQ+t3RUsF+HgQAdx+QZnHezsyhYWFpmG\naiCX6aMbsDkkwIpWIJc4ffqkUjaEmXcK6FO/5dMHuOXlyvVydmo0n45TFOZXfvXN0jzIwB0qrLbJ\na/pMcuSlm43k7FlY0wmIMh+0wECI1fS52NfRo20W7FD/++5pjkWGThZA+MGZt71lXzG+RivqQZyW\neaeENof3LrjmqBqMyJl30s+oVXT+LY+FlsmX3pMSaWNgmSW7kI9+K7inoOmji930r1cAAJuXLpF6\n3kvHpJl/2gfWacFq+iwsKhBxnDxpmXfG6dMnUYat6SPC+6Vy1L28QiAXQH7x7O0PF/ruu+YMAMDQ\n+moMrVM/ixP1lWsCyzrZlD07iMv/wfRCXskgJDwFCY2t++iDHDLo8/TJjsUP/HpF6W9VbYvD2bQ7\nDn8zHLZJDpqPFi+Em3eW7/PKel8zr3/kQ+ybN3d0BGX6OfwWBH3NA2sjH/ThE/9WBatfvT59aWjb\nA+PWO/cb0fQ52LirPbwgWONev06jmr5DdN2xQp+FhQWAoCAps1ixFuwPnDrDED/hZdg+fd6/hV59\nvl8FB2hqqJXkTkHTxxP6BuofM7QOY4bWSderClebx9oQyR4exOvTF45DRTBUTbHw9bvXBq55X+lg\njN6pFHXP034df92AwCAarYJb9Dj3/mJk3Qg8Wy6rb14qCx06YREadQO5mDDvDPi3UWUinWdx2iXS\nyBqHRBV6Pn38ktv3d+FBholmWN269ZXLSJGSgsqB9WCaRa3QZ2FxCEBmeqMnNpn9ImsSvvbNs/CH\nqxcrB6UI8CNRfxXLvBPATZfMxeIZo3DmrLHS9AuOg6mjG6T5k10zws07nVgXlZJ5Z4QVM06hT2Zw\nHgoiHyHs8awG/1gahDKfktZZJTWC7znO9UKB/x2J6BetCOj7rgZewbzT1SCxBCvN6KGqdHSecQS6\nPlauSUDXvDP8Ibq7o0xtfH/EAdr6pJXQ05/3/Raadxpgag8nmjALpvrAqx03Ma9Z804LC4tDGqIw\nzzwwowcCOOWIMbj69JnR+JEQhVgsEkLwnpOm4faPnIz6mioBfYrWALFxw/xaN17QnMjmnVS99N9K\nkAjkEuVE+8WtB/QfFqBowpW+1JcF2YgQEtlsO0eIT7s3ODV96s+INE2lMr462MKVw7jmrUO2fu/f\nYUKs924SG9Womj7W46L+57Wp+A7UmGGdr9EUaEEtyoEW78wq7xHOzRolMkCAnz+6KcCX77fvb0nN\nW0S2XHgPNUQ0pQK5GOxKm7LBwsJi0GHGmEbpsiIh68oTppT+Xv2180t/uwvml986p0zH0D5Thk4/\nY5XXrZ6XW+mHVy5kPxBR05eTTCIfFa4gEWWR+/fqZlPsBJCFA9csRGfLEXXzThrVOaJg3lyZUDLL\n8pp3KtRRNu+krwvMKwXXnnh1j2+zrSLI+Xz6OEXdQFBCOuLboXVEpc0rw/dBVJ8bmO8n4MPnv60i\n9NH0ixFG05/AWrv7hPdNa/rSaLFXwDbShvRfWyqwQp+FxSDGr646QbqsKJDL0svml/72BhpxF5P6\nmvJUYspxXYYKy6dQOg0B58SXXgx4G3HZNYObp89LK+ICJOrzqpxbB7uSqIJGVEid8MbPRuogIJE1\nc7kcwWstHR6a2cRtT7yu/ayKtstR2Ox6+4rrZyZdsx/funcdOnryzHthQmx7T790WRPQCuTi6+fg\n8yIBjif06Wjg2IFc+HRlNMA++or+gKYTivMQ8McXWO3IshOmuZaFfHL2ZKFyCDqYzs6s0GdhUcE4\nYtxQ3++xlGli3BOp64/h9UUyNkFKEGInp5ZjgFbAlRcnP01eREXv5uaCYyaU/v78hbN95eZPHolX\nbjwfN7xtDv75ydM8fLJqM48w885NNy/BiTNGxcxFuqiEU12dPH00tu/v8v1OW6Dn4caBsO060NFY\nFw0FI9otuoS4RRlzkedvr5m3t2yYEOs1rY6ire/s6ZeaG+P4VorCFZswt02KAhkgp6Wky6gIl/S7\nchzxqIpr2gkTzEXazKQ1k77k7BGrVn2+uy+PpzftY96zPn0WFhYVhRe/fJ5PiFj/9Quw/PpzSr9n\nTxhW+ltm70fg19jJzInuhK4aal4Vyz9/DvM6W+iTo9lP5TAoOd8HNH08n74ibrliAX76nuPx7XfM\nxwnTm/CRMw8PlG2orcZVp87AvMlerR8ZqM+R5vl/37uIeV0mObtwc5Pi+ke/hziQBROsMBAAxPCK\nPDh9+jTNO0Me82nUXK0/o26uT580V7T/oPxzvI2qjHXFx37/glQdWoFcQjVJfKoiTZ9uqg0ebyxO\nlLRWgYPCMIbkaavAy/Ozr+/Dy9sPSpc3EdBHLZ9tuazo2w01T9YYD9++d70UX4cSrNBnYVGhaGqs\nxZDacqCSuuoqVA/Y8j3zhTfhzo+dUronM70RAlx+/JTA9eOnNXGfcSdxU5q+J647G9+7/NgiHc/1\nw0YMwRPXnY3H/+tsX3mmeadEPYSwFnD2Ro8nz7pVu+1956Ip+PNHT2EXZkBHTj5vznjfb16QGX89\n0X364kRnL9vszYuoC3RY0zOhETOg6QuQpMhlop0RoWLiqCIgev1WSz59DEUfP/m4mH4fx7dXpT08\nX0NZ3zKZ8aUVyEViheHRNWlKy6IVJuRF0vRBbL4Zm6aP+r18017fb1H0TiP1K5l3lv/+6j/W6Nfp\nqVd2mtzd1s29x/kcmUgj12JcUM8CbGFhkXmMH14PACAKoZUBEljU1n/9gpJAd8+nTg+Ehi4Jfd6g\nJBEmyMlNDdg1pjhR0xP75KZgOgVWuG8ZEyaC4ALuLk70Ailj3qmC/zz/KPT0F0p8eqmo0qytzqG/\nNy9Ozl5K2cAvw9u0VecIN3myKXT1SQh9EesI3/Skv6gTEAMpG4JUvcgRUvFJicPy2nlRGrqKATe4\nFoeKXect7v2OvHSeeHWPND3ewY1JUzWd8RGupeHPMUJNnyIrMgdbUfL0sX36RNqreL41VfNOLU2f\noJzaAWK5rOg5qYMDhVoBk20YPLCaPgsLixK8CyIhfu3hnInDsXCqX+tXM3CvpiqHmipXiIk2mZZO\n8yTKskwDZbfNedq8k5MDK8y8UzVC4ifOPgLXnjdLW8w47Ygxpb9rq3M+XlhwhVYd05r7P3OGMn+q\n6JLQ9EUVhZIIgBEVhJgPGCAKzlSpUNL0cQStMLibalZydt19vF/Tp0eEJ9y1tPXgu/dvCH0+jo01\n4O9b1mbacfiEeYdKRUFRDX7/MQedvf3o6WOb8XvLSdOnHg4VvhTpi/DEq3uwcVeb1rP+5OzR+VER\nmExNvSK/UO4zgrZW+uGXLqzQZ2FhAWDA5NFR2yR9acnR+OiZh+P8Y8bjDx86CZcdNxm1VdGmFRVh\nipUOQWZjSwgJmHd0DPj00AsL1yTOCbkfAldg7vMEeJBp86/fX47I6vY1L8FxkWbxf5E2wL3zxw+d\nxOQxTnRLaPqqI/JRATIfCGIw76R+V2XAvpOVZkUFKlotn3ZNoQ6ueaeiIOLtbdE3KgvewZRsHk0p\nHjTY9D7CMmMVeWNxzTs1+PAKZQUHmPOV+/ALKlJsMGqzPP2gdUiIeafheecnj7xWpBtSThi90wBP\nKn7YJjSLqrRciNhUEvrSnzaNwZp3WlgMAiwS+N3JgkB9czyyoRbXD0SrPGH6KJwwXS8K5BPXnY1d\nrUVTVK+mb1hdNdo8ARZosE/9Jc07B1aECcPr0dzajQNdxVxHNEWeJsrdxuiasw4fUpx+Ozz0ZU4y\nvQKQq+njJYAHyv6WMmucq611EXeAHkDOp0/Gd1GESjjVJYQY97mjhcjaqhy6++IPnCPCEV/8d6Tn\nlbQMntybSpo+bjgSfXiFId3hyDWFlOwTmc26nnln+Zk+hmBZ1NJweOIIojrpDrxCGV+Y9B9sRtH0\nFQriMaEeekQMl5ay8OPT9EnWJSiocoBhKmWDA6f0DckeRop635p3WlhYVCQe+uyZ+L8PnCgsIzPv\nEkJSM4Ob3NRQChjjLsKEAA997kzc/anTmM+856RpeM9J0wPXZTfO7sJx7JRiRM2DA0IfvU7samU7\ng6s6ldMYMaSm9Leu4Pipc44EAIweWsstU4reKXi3Hzp9JgB/CpD6mlwigT+kArlErCNsWGfB7LGu\nOmdc05ejVvi6mip2wQqCyiZd19etFMmXUbeuuZ6u1pHFFw3Zdsps1rXMOz1/8wLW8DbgPKGg4Kib\nInr7h+8r6CnjOJE0faGCiuHllGd2HAba7DUqVPy8TR24OU75QEFa6BNUXQkHgXHACn0WFhWOw8cO\nRWMdW2mvsodkBTdJAyXzThCMG1aPYyayk5vfdMlcjGioCVyvqwmf1ggBzj5qHL605Gh8+x3FaKHu\nJuHT5x7pK3uKx4cOAGaOaQRQXjR0t+mNteV35m7O3QA8snjnCVOweekS7vsHykKw6N1eMHcCNi9d\ngpENZeHxsf86O5GQ/y3t4cGGOnv52l4ZVMIC31BbZd6njxqdNRkw74wKFetQn0+fUph5///e60pm\nooJ7Ou+6wNn0y45vmc261qfieYZldSDStHJ9+jTET795ZzjdvnxBSQiiheZQ805pynLQ1fT5tJvS\ndfFLxmHeKVPQrZc2Uy8UHJz1nWX4x0tv+K+LhD6l6J2DB1bos7AYxBhWXxSKTp45Wqp8FiweFkwZ\niZNnjsYNFx0TWnbEkBrc/uGT8JZ55eTo9dV8bcYjnzsLQHEznMsRXH36TIwYUoN5k0bg5rfPAwBc\nPaD1AoDNS5fg8LFl7df6r1+A+waCm0TV9HlNJ4fV1+CWKxbgd1cv1iMmgOsnqPpuxw2rT0QD1iIR\nYTaqSWKYBjsLizohRDkokE4dlQ5tnz6F8c8TrkwF5RCZO4oQ1byTp4XzYuu+TiWegGI/dfXm0dtf\nYApxImGZx3vBgbLU5BX0vvaPV5hlvF3Qn1fT9NGCZOh40HzPXHKatLyPGfHpUzDvNNl+90Bhy94O\nbGppL13v7s9j895OXPeXVVTdAvPOCjgIjAPWp8/CYhBjVGMtHv7smZjc1ICnNu3Bgc4+NNRWlfzn\nvMjlsqERqa+pwh8/fFJ4wQEsnjkax01rwsULduMjv32eqem782On4OlNezFhxIAmjdr7/vP/BU1I\nvcntn7jubOQLDuo8AuUNFx2DhtoqnHXUOGleaZwxayxOH9AkXrJwkrCs7n796MOK7Zg1fmhIySAS\n0fQppRXRQyWYd7IwdlhdtP7JaLuigI66Ky47ILwZ0tCJNFYs8Lq/v1BQaocLd6NK8yC7BzcRTIaF\nldsO4ro7X8bcScOZc8adL2zH+ceMZzwpit6pruvzkrp9xTYuXRe9+YJanj5GygazXntiQUVX07fz\nYDdmjXfXM7mHhT59KrklZeuTuO/W25d3cM73HsXmpUuKv/vVNeCVENE5Dlihz8JikGPmgKbqwWvP\nREdPHrXVObzjJ09hXbM//PPIhlrjE+HfP3EqLr71SaM0b7liAX7/zBbftZqqHEY1Fk0TWZq+46c1\n4fhpTVKRIgFgw9cv9PmzsXIEThw5BLdcuVCB8yB+E+KL6eKVG88PmOp5/e8A4MNnzMQfntmK1V87\nH9Ovv7t0/YK5h+G+a87AUROG4ZN/eDG0rt9fvbjkIxi30De0rrrsSxkjbnvydeH9DJx1BDBmaF1k\nn0r6cZOv87ipI/GCZNRIk7h12WvSZX2bPrWcDQP/sy5HHyxLfvSE1nPl6J0O83oYVMzyVPDHZ7cC\nAFbvaPUdlnlx67JXmdd52kudb1JGE+st0ZcvSL/NHz20MbB+5AuO0EzQ0QjlIhXFMoQmTeN9tz1b\nEpDMRO+UJ2JyS9HHCVhG5w92Iapaha+sHgrqwAp9FhaHCOqqq0qaqpNmjg4IfWMa64xHtDp2ykhs\nXroED63dhR89tBEvbT8IAHj5hjcr2dR7ccnCSUytmLsgi3z6XCFmweSRwjrcqJhZQUOtf6q+/zNn\nBPz/vvCWo/GFtxzNfP6ogY3YM194U6iJ16keH8a4XcAaaqvQ2h2/0Jd13HTJXOb1qJ8jbc5pcvMi\n8iPNCnSDp/ACuWzY1YYzZo0JlE8KrhBLp6rpEUTv9YIVWdM0ePPLAc7hjjg5uxq/qsnZi+adcnV8\n/4FgHsQv/W218JnYArlEoCvv08eHipZa9h2GFWtp68ELW/cz7/HGv2g4HKrmndna2VhYWCSC9548\nLXBt1NDa2Hz63nT0ePz9k2UTymH1NcwgLFHg+n2JfPpqq3O46+On4BdXLTJad9KYNX6YL/onjdOO\nGINjJwcD4IwfXs/UWvJgwgdsctMQ7r3GuurUUwgA6Z/kHjmOZXqrHymSB90osSx4N8uP/9fZxuia\nhFczpqboYxf+03PblHzeTE+nrjxFR7zlaTpoxKXp84InWPL6n7f53t/Zpzw3SAl9niKqgVxUoWoO\n7D7DvecA2/d34pWdrUIaovnMRHM3NLeHFxqAyT3FsvUtzOs8oU/0bg9V804r9FlYHIKYOXZoyTzw\nvDlFX4vG2irfJDmsPvsn+V64G5+w6J0LpzZheL1ZgTNr+N3Vi31Cti5MaPpEm4whGUkhYFIYUkG9\nYKx29ORLUeoe+08zQpVOkA4evPLDkNp03mNTyMGRV6BQMbMrafoYg/cDv14hTcc0XEE7IPRJCkdx\n+fR5oZyyQbD5vndNs1LdMtob72FFb77AtTj5wl0vo0OQI1YGOgJl2DOnfWsZLv/p8hAagnvSPn38\ncrSmOYSSZCn9scnLUys6BOiIGBG6UmGFPguLQxQfOHUGAOD77zwWm5cuASGklP/m/51zBKaMktcI\nZQHuqXCdQNNnoYZh9TX49mXzfdeOPmy4MfpJCAssjWdWMF9gZvy2Yw/D769ejE+96UhMGcXXlrqY\nwEj3Eaco692kjWrg54lME95Nn1pydvVnWDDd/w+v2418oRgp0wtZTV9fviA8aDABU5o+HUjlIfTU\n1ycI5PKHZ7aWfBV1tYE6Twk1fQZ0x0lbNSahUOON/y6BD/+BTnm3grQOBeOAFfosLA5RvHvxVGxe\nuqSU1gEAvn7pXHzsrMNxzbmzYqnzhOlNuPHi8FQMOlgwpbiBvmjBxFjoH6p45wlTSn8v+9xZ+MtH\nTzZGO+pJ+txJ4QLolSdODS2TVvq6YyYW+XeDEAHAHR85GV+76Bh889J5OGLcMFx73iwQQvDNS+fh\nnNnj8Mmzj/DRcLWlLNO9ccPqYuPdu5nL5Qhe+sqbccPb5sRWHwsdPWJhp2TeCXm/NwDY19GDg519\nkbfYcex3N7W0440DXb5rsm3LFxxUxWzLvIeTd5Mr9BnUPsr453lL9OfFYlRvvoD71zTjrhd3aPGj\nGjXWfUbnnhc65p3zJhUPx2oHDn5NvZVCwcGug92h5aIIozxNH3044sWBzl79CisYsQp9hJALCCHr\nCSGvEkKuZ9yvI4TcPnD/GULI9Dj5sbCwEGPM0Dpcd8HsQPJTU/jzR0/Be0+eHgvtI8YNxealS3B2\nhBQKFmy44dYnjqxHY101rvQIgjQOG1GPmqry+BGdktPBhFTxwdNmhJZxDwNE+PAZM32C31gDlhre\niQAAEiBJREFUwpJMMKDPX3g07vr4KZ5w6sCJM0bhfadMR3WV//l3L56K2646Ae9cVOz7D5w6Ay99\n5c2lSKtt3UEB+nPnH4WRISaQFy+YiJVfOY97f+aYRoxuDGry6Pc6oqEGV50afB/TRvstBhbPGCXk\nRwVhZmZeLdIvnxBHcPXi1mWv4dgb78eDa3dp8wbEk47k4XW7sWlPh++abFTivryTSBoWFnjzgElN\n304J4cKr/d15sBuvtfD90xwH+PBvn8e1d7xkhD8ZiMRQ2qyXS8MJCkKukMOi39HTj5d3FIOsuXO3\nqdfS2ZtHc6v4vTy0dpd021hw54HuvgK2eUzYTWn6BhNic9ohhFQBuBXAeQC2A3iOEPIPx3G8GTM/\nCGC/4zhHEEKuBPAtAFfExZOFhYWFhTp+9K6FeONAd8l0dull87H0svm44mfLUVudwyfOPgLVuWJy\n8RFDqvGvVTtxy4MbcdPFx+D3z2zFGxKbsR9euQCf/tNKHDluKDbuLm/EZk8YxhUORw6YFY4fXlfK\nPbn07fOwp70H372/GG3v6MOGY/PSJVi57QAu4aQPOXv2OFx16gy8caALpyx9GOccNQ6XHjcJn/zD\ni7jqlGn47v0b8MHTZuDco8fj8Y0t+J9HwtMG/PCKBfi/5ZsxbVQj+gsO7nxhe+neXR8/BYeNGILa\n6hwWTm0KpeXF1NENuO+aMzB9TAPqqqvw43ctxNJ/r8OlCyfh+r++jG9eOg8/efRVbNvXhfqaKqz4\n4rn49VObMWficGxobsMjG1rw2fOOwjzK7HXp2+fhz89vx9cvmYv33vYsWtp6MLqxFv++5nTUVVfh\nip8tx5lHjcW3710PADh28kh85x3HYvPeDhabGDGkBge7+nDdBbPR3t2P4UOqccassbh71U488/o+\nXLFoCu54flus5mauyffX/slO1B2GLXuj+UDGkY7k5n+vC1zbLMnnjgNdwgBQPCyZdxjufnmn8nNe\n8OYAVs7YOPH7Z7b6frvjmYX/fXxT5Po2KB5s/eslfj8/uoEdyIRGc2s3fve0P63RghsfwPUXzg6Y\n56/ecRBv/XE5hUhnXx5dvXn89NHwOW5kQ02o8PQA5+Bk9Y6DeGjtbmzc3YZ/rdIbWy9tO4ARQ2qw\ncVd5vTj928vwrhOn4puXzsXOA+wxt/y1vYGDExG6+vK4e9VOLJl/mBafWQKJK3IRIeRkADc4jnP+\nwO/PA4DjODd7ytw3UGY5IaQaQDOAsY6AqUWLFjkrVqTnSG1hYWFhIYbjOOgvOKipyuHZ1/dhxZZ9\nOPfo8diwqw35goO5k0bghS37MWFEPd7zy2dx+pFj8NsPLgYAPLxuVylYxg1vm4OGumr8119WYVh9\nNY6eMBzvPGEKvvDXl/HuxVPx1bfNwZ0v7MBb5x+Gv724AwumjsTsCcVNzeY9HWioq8K4YX5ft427\n2nDeDx4DALx1/mH4/jsX+LRyr+/pwKjGWuHmeNu+TvT057FtfxeOGj8MjXXVaKytQmt3P/IFJ6Ap\nbO/px92r3sDOg9247LjJsfvLuktolOiru9u6kSMEY4b627JtXyfWNbfhrKPGlnyAvdjU0o7GumqM\naqzFPS/vxEXHTvTx4TgOXth6AMdPKwq7G3e14St/X4Plm/aWylyyYCL+uWqnMBBDbXWOa9algmOn\njMRL24r5BukDh8GIYXXVaFM0q37iurNx2reW4aSZo/D0pn2h5b912Txcd+fLzHtXnTIdv35qM/fZ\ns48ai7mTRuDHDxfz+t35sVPwqT++iB2USSsLU0c14M1zxuPJ1/ZibUiESx6uPm0GfqGgFT6Uce7R\n4yNrwysJr9x4fiB9UtoghDzvOI50OPI4hb53ALjAcZyrB36/B8Bix3E+6SmzeqDM9oHfrw2U2UPR\n+jCADwPA1KlTj9+yxX+CYWFhYWFRuSgUHOQ89pX7O3rR5DEpbO3uw7C6aiMpJFy09/RjaAXkmjtU\nsKmlHVv3dWLhlCaMaKhBoeCgsy+P2qocOnr6kcsRtHX3YU97L44aPwz1NTl09Oax/LW9mDiyHhOG\n16OpoRZr3mjF2uZW5AsO8gUHx0wcjtU7DqKuugrrmtvQ1ZfHZ847Etv2deG4qSPhOMBL2w+g4BQT\nzgPF/H7fvGcthtVVY+LIIRjVWIvGumq0dffjuGkjsamlqCXY296LGWMaMX1MA3r7C6jKEQyrr8Hu\n1m7c+cIOXHb8JLR29eFPz27D9v1duGThRGzb14VcjuCE6U3YebAbu1q70d2Xx4fPOByv7+lATRXB\nsvUtOHLcULy07QC27+/CZcdPRlt3H/rzDvZ19OKkw0dj7NA6FJyiBnnmmKEYVl+Nnz+2CZOahuCj\nZx6OZQNBX0YPrcULWw+UvrETp4/Cqh0HMHVUA/Z39GLyqAas3dmKNW+0Yu7EEThu6khUVxF09OTR\n2t2HvnwBly6cjA272jB9dCN68wUc6OxFV28e00Y34pYHN2BYfQ3edeIUPLd5P8YPr8P8ySOxu7Ub\nPf0FbNvfic17OjGyoQbdfXlcvGASHl63G411VZjS1IB1zW2YMaYRh42o9+V+XN/chrHD6kr+rq3d\nfdjQ3IbZhw3Hjv1dmNQ0BNU5gpa2HoxqrEV1FWEG8dq+vxMNtdUYWleN57fsx/7OXlTnCBbPGI1/\nrnoDlx03GTsOdKGlrQcnHz4affkC7l+zC7tau7F80160d/fjsBH1+MiZh+NgVx+e27wPcyYOx/rm\nNlTnCNp7+lFfU4X+fAHVVTm0dffh5JljMGFEHb5412o0t3bj3SdOxdD6alw49zCsb25DT38epx85\nFtv3d+IXj7+OpoYa/MfJ0/Di1gNYtb14ADF/8kis3nEQx0wcjn0dfVjf3IrTjhyLrfs6Mby+Guub\n2zC0vhrHT2vCis37MXZYHZat243jpjWhtasPza3dmDV+GF7cuh9L5h+Gjp48Ht/YglGNdWjt7sOk\nkUMwpKYKz2/ZX8qV+tEzD8eaNw6ioyePnv4CCgUHfYUCJo4YgqocwXlzxuOxjS3IEYIrFk3Bga4+\n/H3lDiyYMhKjGmtxx4pt2N3ag0+cfQS27OvEyq0HMG54Hbbu60R7dz8mNQ3B2+ZPxN9f2oFNLR14\nx/GT0Zcv4O5VO7Fq+0EcOX4o5hw2HGcdNQ4v7ziIdTtbUV1F0NRQiymjGvDo+hYsnjkKzQe74QB4\ncO0u5AjB8VObUF+Tw/zJI7Fxdzt2t3bjQFcfHMfBOxdNwWMb9+Dco8dhVGMtHnhlF/69uhkjhtTg\nvDnj0dHTj5qqHM6YNRa3PLgBw4fUoCZHMLS+GmccORaPbmhBW3c/3n/qdGWrjCQwKIU+L6ymz8LC\nwsLCwsLCwsLiUIaq0BdnIJcdALze/pMHrjHLDJh3jgCwFxYWFhYWFhYWFhYWFhZGEKfQ9xyAIwkh\nMwghtQCuBPAPqsw/ALxv4O93AHhY5M9nYWFhYWFhYWFhYWFhoYbYHBocx+knhHwSwH0AqgDc5jjO\nGkLIjQBWOI7zDwC/BPBbQsirAPahKBhaWFhYWFhYWFhYWFhYGEKsXuyO49wD4B7q2lc8f3cDuDxO\nHiwsLCwsLCwsLCwsLA5lxJqc3cLCwsLCwsLCwsLCwiJdWKHPwsLCwsLCwsLCwsJiEMMKfRYWFhYW\nFhYWFhYWFoMYVuizsLCwsLCwsLCwsLAYxLBCn4WFhYWFhYWFhYWFxSCGFfosLCwsLCwsLCwsLCwG\nMazQZ2FhYWFhYWFhYWFhMYhhhT4LCwsLCwsLCwsLC4tBDCv0WVhYWFhYWFhYWFhYDGJYoc/CwsLC\nwsLCwsLCwmIQwwp9FhYWFhYWFhYWFhYWgxhW6LOwsLCwsLCwsLCwsBjEsEKfhYWFhYWFhYWFhYXF\nIIYV+iwsLCwsLCwsLCwsLAYxiOM4afOgBEJIC4AtafPBwBgAe9JmwuKQhB17FmnAjjuLNGDHnUUa\nsOPOIi2Ixt40x3HGyhKqOKEvqyCErHAcZ1HafFgcerBjzyIN2HFnkQbsuLNIA3bcWaQFk2PPmnda\nWFhYWFhYWFhYWFgMYlihz8LCwsLCwsLCwsLCYhDDCn3m8PO0GbA4ZGHHnkUasOPOIg3YcWeRBuy4\ns0gLxsae9emzsLCwsLCwsLCwsLAYxLCaPgsLCwsLCwsLCwsLi0EMK/QZACHkAkLIekLIq4SQ69Pm\nx2JwgRCymRDyMiFkJSFkxcC1UYSQBwghGwf+bxq4TgghPxoYi6sIIcely71FpYAQchshZDchZLXn\nmvI4I4S8b6D8RkLI+9Joi0VlgTP2biCE7BiY91YSQt7iuff5gbG3nhByvue6XYstpEAImUIIWUYI\neYUQsoYQ8umB63bOs4gVgrEX+5xnzTsjghBSBWADgPMAbAfw/9u7u1DLyjqO498fTmW+NYoxTFOk\niBIx4DGHmV5MLGTSuTl6UeiFmQgGadBNWN4oCjb0Bl51ERkOmCKkNIGWUyBBNDkvjIwvEYOOqIwO\neSSVMc3j34v1nNoNc2bObvY+e2ad7+dmrf2sZy+exfnv/9r/vZ61zjbg6qp6eqIDU28k2Qusqap/\nDLT9EJipqo3tg356Vd3cksS3gQ3AOuCuqlo3iXHr+JLkYuBNYFNVrW5tQ8VZkjOA7cAaoIAdwIVV\n9doEDknHiXli7zbgzar68UF9Pw3cB6wFPgb8ATivbfZcrAVJshJYWVU7k5xKl6uuAL6BOU9jdJjY\n+xpjznle6Tt6a4E9VfVsVb0D3A9MT3hM6r9p4J62fg9dwphr31SdrcDylmCkw6qqPwEzBzUPG2df\nAbZU1Uz70rMFuGz8o9fxbJ7Ym880cH9VvV1VzwF76M7Dnou1YFW1r6p2tvU3gGeAVZjzNGaHib35\njCznWfQdvVXACwOvX+TwfzxpWAU8mmRHkhta24qq2tfWXwZWtHXjUaM0bJwZfxqlm9pUurvnptlh\n7GnEkpwFXAD8FXOeFtFBsQdjznkWfdKx76Kq+gxwOXBjmwr1H9XN0XaetsbKONMi+xlwDjAF7AN+\nMtnhqI+SnAL8GvhOVb0+uM2cp3E6ROyNPedZ9B29l4BPDLz+eGuTRqKqXmrL/cBDdJf0X5mbttmW\n+1t341GjNGycGX8aiap6papmq+o94Od0eQ+MPY1Ikg/Qfem+t6oebM3mPI3doWJvMXKeRd/R2wac\nm+TsJB8ErgI2T3hM6okkJ7cbfUlyMrAeeJIuxuaeEnYt8Ju2vhn4envS2GeBfw5MVZGGNWyc/R5Y\nn+T0NjVlfWuThnLQvchX0uU96GLvqiQfSnI2cC7wOJ6LNYQkAX4BPFNVPx3YZM7TWM0Xe4uR85aN\n7jCWpqp6N8lNdB/yE4C7q+qpCQ9L/bECeKjLESwDflVVv0uyDXggyfXA83RPfQJ4mO7pYnuAA8B1\niz9kHY+S3AdcApyZ5EXgVmAjQ8RZVc0kuYPuZARwe1Ut9AEdWqLmib1LkkzRTa/bC3wToKqeSvIA\n8DTwLnBjVc22/Xgu1kJ9AbgG2J1kV2u7BXOexm++2Lt63DnPf9kgSZIkST3m9E5JkiRJ6jGLPkmS\nJEnqMYs+SZIkSeoxiz5JkiRJ6jGLPkmSJEnqMYs+SdKSlWQ2ya4kTyTZmeTzR+i/PMm3FrDfx5Ks\nGd1IJUn6/1n0SZKWsreqaqqqzge+D/zgCP2XA0cs+iRJOpZY9EmS1DkNeA0gySlJ/tiu/u1OMt36\nbATOaVcHf9T63tz6PJFk48D+vprk8SR/T/LFxT0USZL+a9mkByBJ0gR9OMku4ERgJfDl1v4v4Mqq\nej3JmcDWJJuB7wGrq2oKIMnlwDSwrqoOJDljYN/Lqmptkg3ArcCli3RMkiT9D4s+SdJS9tZAAfc5\nYFOS1UCAO5NcDLwHrAJWHOL9lwK/rKoDAFU1M7DtwbbcAZw1nuFLknRkFn2SJAFV9Zd2Ve+jwIa2\nvLCq/p1kL93VwGG83ZazeL6VJE2Q9/RJkgQk+RRwAvAq8BFgfyv4vgR8snV7Azh14G1bgOuSnNT2\nMTi9U5KkY4K/PEqSlrK5e/qgm9J5bVXNJrkX+G2S3cB24G8AVfVqkj8neRJ4pKq+m2QK2J7kHeBh\n4JYJHIckSfNKVU16DJIkSZKkMXF6pyRJkiT1mEWfJEmSJPWYRZ8kSZIk9ZhFnyRJkiT1mEWfJEmS\nJPWYRZ8kSZIk9ZhFnyRJkiT1mEWfJEmSJPXY+48+hqucpNPQAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 1080x576 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mkyubuJSOzg3",
        "colab_type": "text"
      },
      "source": [
        "##Predicting and Evaluating on Holdout Set"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DosV94BYIYxg",
        "colab_type": "text"
      },
      "source": [
        "First we need to load the holdout dataset and prepare inputs just as we did with the training set. Then we'll evaluate predictions using [Matthew's correlation coefficient](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.matthews_corrcoef.html). This is the metric used by the  NLP community to evaluate performance on CoLA. With this metric, +1 is the best score, and -1 is the worst score."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NyBuXVGoOLQf",
        "colab_type": "code",
        "outputId": "a27b1f73-184d-4a33-fa11-921070c9219f",
        "colab": {
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7Ci8vIE1heCBhbW91bnQgb2YgdGltZSB0byBibG9jayB3YWl0aW5nIGZvciB0aGUgdXNlci4KY29uc3QgRklMRV9DSEFOR0VfVElNRU9VVF9NUyA9IDMwICogMTAwMDsKCmZ1bmN0aW9uIF91cGxvYWRGaWxlcyhpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IHN0ZXBzID0gdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKTsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIC8vIENhY2hlIHN0ZXBzIG9uIHRoZSBvdXRwdXRFbGVtZW50IHRvIG1ha2UgaXQgYXZhaWxhYmxlIGZvciB0aGUgbmV4dCBjYWxsCiAgLy8gdG8gdXBsb2FkRmlsZXNDb250aW51ZSBmcm9tIFB5dGhvbi4KICBvdXRwdXRFbGVtZW50LnN0ZXBzID0gc3RlcHM7CgogIHJldHVybiBfdXBsb2FkRmlsZXNDb250aW51ZShvdXRwdXRJZCk7Cn0KCi8vIFRoaXMgaXMgcm91Z2hseSBhbiBhc3luYyBnZW5lcmF0b3IgKG5vdCBzdXBwb3J0ZWQgaW4gdGhlIGJyb3dzZXIgeWV0KSwKLy8gd2hlcmUgdGhlcmUgYXJlIG11bHRpcGxlIGFzeW5jaHJvbm91cyBzdGVwcyBhbmQgdGhlIFB5dGhvbiBzaWRlIGlzIGdvaW5nCi8vIHRvIHBvbGwgZm9yIGNvbXBsZXRpb24gb2YgZWFjaCBzdGVwLgovLyBUaGlzIHVzZXMgYSBQcm9taXNlIHRvIGJsb2NrIHRoZSBweXRob24gc2lkZSBvbiBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcCwKLy8gdGhlbiBwYXNzZXMgdGhlIHJlc3VsdCBvZiB0aGUgcHJldmlvdXMgc3RlcCBhcyB0aGUgaW5wdXQgdG8gdGhlIG5leHQgc3RlcC4KZnVuY3Rpb24gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpIHsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIGNvbnN0IHN0ZXBzID0gb3V0cHV0RWxlbWVudC5zdGVwczsKCiAgY29uc3QgbmV4dCA9IHN0ZXBzLm5leHQob3V0cHV0RWxlbWVudC5sYXN0UHJvbWlzZVZhbHVlKTsKICByZXR1cm4gUHJvbWlzZS5yZXNvbHZlKG5leHQudmFsdWUucHJvbWlzZSkudGhlbigodmFsdWUpID0+IHsKICAgIC8vIENhY2hlIHRoZSBsYXN0IHByb21pc2UgdmFsdWUgdG8gbWFrZSBpdCBhdmFpbGFibGUgdG8gdGhlIG5leHQKICAgIC8vIHN0ZXAgb2YgdGhlIGdlbmVyYXRvci4KICAgIG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSA9IHZhbHVlOwogICAgcmV0dXJuIG5leHQudmFsdWUucmVzcG9uc2U7CiAgfSk7Cn0KCi8qKgogKiBHZW5lcmF0b3IgZnVuY3Rpb24gd2hpY2ggaXMgY2FsbGVkIGJldHdlZW4gZWFjaCBhc3luYyBzdGVwIG9mIHRoZSB1cGxvYWQKICogcHJvY2Vzcy4KICogQHBhcmFtIHtzdHJpbmd9IGlucHV0SWQgRWxlbWVudCBJRCBvZiB0aGUgaW5wdXQgZmlsZSBwaWNrZXIgZWxlbWVudC4KICogQHBhcmFtIHtzdHJpbmd9IG91dHB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIG91dHB1dCBkaXNwbGF5LgogKiBAcmV0dXJuIHshSXRlcmFibGU8IU9iamVjdD59IEl0ZXJhYmxlIG9mIG5leHQgc3RlcHMuCiAqLwpmdW5jdGlvbiogdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKSB7CiAgY29uc3QgaW5wdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoaW5wdXRJZCk7CiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gZmFsc2U7CgogIGNvbnN0IG91dHB1dEVsZW1lbnQgPSBkb2N1bWVudC5nZXRFbGVtZW50QnlJZChvdXRwdXRJZCk7CiAgb3V0cHV0RWxlbWVudC5pbm5lckhUTUwgPSAnJzsKCiAgY29uc3QgcGlja2VkUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBpbnB1dEVsZW1lbnQuYWRkRXZlbnRMaXN0ZW5lcignY2hhbmdlJywgKGUpID0+IHsKICAgICAgcmVzb2x2ZShlLnRhcmdldC5maWxlcyk7CiAgICB9KTsKICB9KTsKCiAgY29uc3QgY2FuY2VsID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnYnV0dG9uJyk7CiAgaW5wdXRFbGVtZW50LnBhcmVudEVsZW1lbnQuYXBwZW5kQ2hpbGQoY2FuY2VsKTsKICBjYW5jZWwudGV4dENvbnRlbnQgPSAnQ2FuY2VsIHVwbG9hZCc7CiAgY29uc3QgY2FuY2VsUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBjYW5jZWwub25jbGljayA9ICgpID0+IHsKICAgICAgcmVzb2x2ZShudWxsKTsKICAgIH07CiAgfSk7CgogIC8vIENhbmNlbCB1cGxvYWQgaWYgdXNlciBoYXNuJ3QgcGlja2VkIGFueXRoaW5nIGluIHRpbWVvdXQuCiAgY29uc3QgdGltZW91dFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgc2V0VGltZW91dCgoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9LCBGSUxFX0NIQU5HRV9USU1FT1VUX01TKTsKICB9KTsKCiAgLy8gV2FpdCBmb3IgdGhlIHVzZXIgdG8gcGljayB0aGUgZmlsZXMuCiAgY29uc3QgZmlsZXMgPSB5aWVsZCB7CiAgICBwcm9taXNlOiBQcm9taXNlLnJhY2UoW3BpY2tlZFByb21pc2UsIHRpbWVvdXRQcm9taXNlLCBjYW5jZWxQcm9taXNlXSksCiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdzdGFydGluZycsCiAgICB9CiAgfTsKCiAgaWYgKCFmaWxlcykgewogICAgcmV0dXJuIHsKICAgICAgcmVzcG9uc2U6IHsKICAgICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICAgIH0KICAgIH07CiAgfQoKICBjYW5jZWwucmVtb3ZlKCk7CgogIC8vIERpc2FibGUgdGhlIGlucHV0IGVsZW1lbnQgc2luY2UgZnVydGhlciBwaWNrcyBhcmUgbm90IGFsbG93ZWQuCiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gdHJ1ZTsKCiAgZm9yIChjb25zdCBmaWxlIG9mIGZpbGVzKSB7CiAgICBjb25zdCBsaSA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2xpJyk7CiAgICBsaS5hcHBlbmQoc3BhbihmaWxlLm5hbWUsIHtmb250V2VpZ2h0OiAnYm9sZCd9KSk7CiAgICBsaS5hcHBlbmQoc3BhbigKICAgICAgICBgKCR7ZmlsZS50eXBlIHx8ICduL2EnfSkgLSAke2ZpbGUuc2l6ZX0gYnl0ZXMsIGAgKwogICAgICAgIGBsYXN0IG1vZGlmaWVkOiAkewogICAgICAgICAgICBmaWxlLmxhc3RNb2RpZmllZERhdGUgPyBmaWxlLmxhc3RNb2RpZmllZERhdGUudG9Mb2NhbGVEYXRlU3RyaW5nKCkgOgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnbi9hJ30gLSBgKSk7CiAgICBjb25zdCBwZXJjZW50ID0gc3BhbignMCUgZG9uZScpOwogICAgbGkuYXBwZW5kQ2hpbGQocGVyY2VudCk7CgogICAgb3V0cHV0RWxlbWVudC5hcHBlbmRDaGlsZChsaSk7CgogICAgY29uc3QgZmlsZURhdGFQcm9taXNlID0gbmV3IFByb21pc2UoKHJlc29sdmUpID0+IHsKICAgICAgY29uc3QgcmVhZGVyID0gbmV3IEZpbGVSZWFkZXIoKTsKICAgICAgcmVhZGVyLm9ubG9hZCA9IChlKSA9PiB7CiAgICAgICAgcmVzb2x2ZShlLnRhcmdldC5yZXN1bHQpOwogICAgICB9OwogICAgICByZWFkZXIucmVhZEFzQXJyYXlCdWZmZXIoZmlsZSk7CiAgICB9KTsKICAgIC8vIFdhaXQgZm9yIHRoZSBkYXRhIHRvIGJlIHJlYWR5LgogICAgbGV0IGZpbGVEYXRhID0geWllbGQgewogICAgICBwcm9taXNlOiBmaWxlRGF0YVByb21pc2UsCiAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgYWN0aW9uOiAnY29udGludWUnLAogICAgICB9CiAgICB9OwoKICAgIC8vIFVzZSBhIGNodW5rZWQgc2VuZGluZyB0byBhdm9pZCBtZXNzYWdlIHNpemUgbGltaXRzLiBTZWUgYi82MjExNTY2MC4KICAgIGxldCBwb3NpdGlvbiA9IDA7CiAgICB3aGlsZSAocG9zaXRpb24gPCBmaWxlRGF0YS5ieXRlTGVuZ3RoKSB7CiAgICAgIGNvbnN0IGxlbmd0aCA9IE1hdGgubWluKGZpbGVEYXRhLmJ5dGVMZW5ndGggLSBwb3NpdGlvbiwgTUFYX1BBWUxPQURfU0laRSk7CiAgICAgIGNvbnN0IGNodW5rID0gbmV3IFVpbnQ4QXJyYXkoZmlsZURhdGEsIHBvc2l0aW9uLCBsZW5ndGgpOwogICAgICBwb3NpdGlvbiArPSBsZW5ndGg7CgogICAgICBjb25zdCBiYXNlNjQgPSBidG9hKFN0cmluZy5mcm9tQ2hhckNvZGUuYXBwbHkobnVsbCwgY2h1bmspKTsKICAgICAgeWllbGQgewogICAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgICBhY3Rpb246ICdhcHBlbmQnLAogICAgICAgICAgZmlsZTogZmlsZS5uYW1lLAogICAgICAgICAgZGF0YTogYmFzZTY0LAogICAgICAgIH0sCiAgICAgIH07CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPQogICAgICAgICAgYCR7TWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCl9JSBkb25lYDsKICAgIH0KICB9CgogIC8vIEFsbCBkb25lLgogIHlpZWxkIHsKICAgIHJlc3BvbnNlOiB7CiAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgIH0KICB9Owp9CgpzY29wZS5nb29nbGUgPSBzY29wZS5nb29nbGUgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYiA9IHNjb3BlLmdvb2dsZS5jb2xhYiB8fCB7fTsKc2NvcGUuZ29vZ2xlLmNvbGFiLl9maWxlcyA9IHsKICBfdXBsb2FkRmlsZXMsCiAgX3VwbG9hZEZpbGVzQ29udGludWUsCn07Cn0pKHNlbGYpOwo=",
              "ok": true,
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "status": 200,
              "status_text": ""
            }
          },
          "base_uri": "https://localhost:8080/",
          "height": 75
        }
      },
      "source": [
        "# Upload the test file from your local drive\n",
        "from google.colab import files\n",
        "uploaded = files.upload()\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-0b589dd4-b39e-45d9-963e-99234439e2be\" name=\"files[]\" multiple disabled />\n",
              "     <output id=\"result-0b589dd4-b39e-45d9-963e-99234439e2be\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script src=\"/nbextensions/google.colab/files.js\"></script> "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Saving out_of_domain_dev.tsv to out_of_domain_dev (1).tsv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r_Bo28c6H0RW",
        "colab_type": "text"
      },
      "source": [
        "**Preparing the data**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mAN0LZBOOPVh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df = pd.read_csv(\"out_of_domain_dev.tsv\", delimiter='\\t', header=None, names=['sentence_source', 'label', 'label_notes', 'sentence'])\n",
        "\n",
        "# Create sentence and label lists\n",
        "sentences = df.sentence.values\n",
        "\n",
        "# We need to add special tokens at the beginning and end of each sentence for BERT to work properly\n",
        "sentences = [\"[CLS] \" + sentence + \" [SEP]\" for sentence in sentences]\n",
        "labels = df.label.values\n",
        "\n",
        "tokenized_texts = [tokenizer.tokenize(sent) for sent in sentences]\n",
        "\n",
        "\n",
        "MAX_LEN = 128\n",
        "# Pad our input tokens\n",
        "input_ids = pad_sequences([tokenizer.convert_tokens_to_ids(txt) for txt in tokenized_texts],\n",
        "                          maxlen=MAX_LEN, dtype=\"long\", truncating=\"post\", padding=\"post\")\n",
        "# Use the BERT tokenizer to convert the tokens to their index numbers in the BERT vocabulary\n",
        "input_ids = [tokenizer.convert_tokens_to_ids(x) for x in tokenized_texts]\n",
        "input_ids = pad_sequences(input_ids, maxlen=MAX_LEN, dtype=\"long\", truncating=\"post\", padding=\"post\")\n",
        "# Create attention masks\n",
        "attention_masks = []\n",
        "\n",
        "# Create a mask of 1s for each token followed by 0s for padding\n",
        "for seq in input_ids:\n",
        "  seq_mask = [float(i>0) for i in seq]\n",
        "  attention_masks.append(seq_mask) \n",
        "\n",
        "prediction_inputs = torch.tensor(input_ids)\n",
        "prediction_masks = torch.tensor(attention_masks)\n",
        "prediction_labels = torch.tensor(labels)\n",
        "  \n",
        "batch_size = 16  \n",
        "\n",
        "\n",
        "prediction_data = TensorDataset(prediction_inputs, prediction_masks, prediction_labels)\n",
        "prediction_sampler = SequentialSampler(prediction_data)\n",
        "prediction_dataloader = DataLoader(prediction_data, sampler=prediction_sampler, batch_size=batch_size)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "levyphNvPKQw",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 107
        },
        "outputId": "7aec7e8b-b048-4614-c2a7-298f05d6a26f"
      },
      "source": [
        "print(prediction_inputs.shape)\n",
        "print(prediction_masks.shape)\n",
        "print(prediction_labels.shape)"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([516, 128])\n",
            "torch.Size([516, 128])\n",
            "torch.Size([516])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UviBtNi6H8Hs",
        "colab_type": "text"
      },
      "source": [
        "**Evaluating**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hba10sXR7Xi6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Prediction on test set\n",
        "import sys\n",
        "\n",
        "# Put model in evaluation mode\n",
        "model.eval()\n",
        "\n",
        "# Tracking variables \n",
        "predictions , true_labels = [], []\n",
        "\n",
        "# Predict \n",
        "for batch in prediction_dataloader:\n",
        "  # Add batch to GPU\n",
        "  batch = tuple(t.to(device) for t in batch)\n",
        "  # Unpack the inputs from our dataloader\n",
        "  b_input_ids, b_input_mask, b_labels = batch\n",
        "  # Telling the model not to compute or store gradients, saving memory and speeding up prediction\n",
        "  with torch.no_grad():\n",
        "    # Forward pass, calculate logit predictions\n",
        "    outputs = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask)\n",
        "  # Move logits and labels to CPU\n",
        "  logits = outputs[0]\n",
        "  logits = logits.detach().cpu().numpy()\n",
        "  label_ids = b_labels.to('cpu').numpy()\n",
        "  \n",
        "  # Store predictions and true labels\n",
        "  predictions.append(logits)\n",
        "  true_labels.append(label_ids)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cRaZQ4XC7kLs",
        "colab_type": "code",
        "outputId": "aa6f6763-af5a-4c34-d6b1-777289b3d583",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 92
        }
      },
      "source": [
        "# Import and evaluate each test batch using Matthew's correlation coefficient\n",
        "from sklearn.metrics import matthews_corrcoef\n",
        "matthews_set = []\n",
        "\n",
        "for i in range(len(true_labels)):\n",
        "  matthews = matthews_corrcoef(true_labels[i],\n",
        "                 np.argmax(predictions[i], axis=1).flatten())\n",
        "  matthews_set.append(matthews)"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/classification.py:872: RuntimeWarning: invalid value encountered in double_scalars\n",
            "  mcc = cov_ytyp / np.sqrt(cov_ytyt * cov_ypyp)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IUM0UA1qJaVB",
        "colab_type": "text"
      },
      "source": [
        "The final score will be based on the entire test set, but let's take a look at the scores on the individual batches to get a sense of the variability in the metric between batches.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xytAr_C48wnu",
        "colab_type": "code",
        "outputId": "515a63d6-2638-4e14-cb0b-145d5cb7c594",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 603
        }
      },
      "source": [
        "matthews_set"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.2581988897471611,\n",
              " -0.17407765595569785,\n",
              " 0.1794871794871795,\n",
              " -0.2277100170213244,\n",
              " 0.09759000729485333,\n",
              " 0.7644707871564383,\n",
              " 0.7125253031944253,\n",
              " -0.2182178902359924,\n",
              " 0.3333333333333333,\n",
              " 0.7125253031944253,\n",
              " 0.6180700462007377,\n",
              " 0.8703882797784892,\n",
              " 0.4666666666666667,\n",
              " 0.6201736729460423,\n",
              " 0.4472135954999579,\n",
              " 0.0,\n",
              " 0.8320502943378436,\n",
              " 1.0,\n",
              " 0.5091750772173156,\n",
              " 0.8703882797784892,\n",
              " 0.8563488385776753,\n",
              " 0.4472135954999579,\n",
              " 0.7453559924999299,\n",
              " 0.4879500364742666,\n",
              " 0.7777777777777778,\n",
              " 1.0,\n",
              " 1.0,\n",
              " 0.7453559924999299,\n",
              " 0.30261376633440124,\n",
              " 0.28941438070815306,\n",
              " 0.5238095238095238,\n",
              " 0.30261376633440124,\n",
              " 0.0]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oCYZa1lQ8Jn8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Flatten the predictions and true values for aggregate Matthew's evaluation on the whole dataset\n",
        "flat_predictions = [item for sublist in predictions for item in sublist]\n",
        "flat_predictions = np.argmax(flat_predictions, axis=1).flatten()\n",
        "flat_true_labels = [item for sublist in true_labels for item in sublist]\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p-4EBZ8D8VsZ",
        "colab_type": "code",
        "outputId": "e4bad55d-304f-48af-df70-576103c9bb55",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 74
        }
      },
      "source": [
        "matthews_corrcoef(flat_true_labels, flat_predictions)"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.5303352052449377"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jXx0jPc4HUfZ",
        "colab_type": "text"
      },
      "source": [
        "Great! In a short amount of time without doing any hyperparameter tuning, we were able to get a relatively good score. Also note that we didn't train on the entire training dataset, but set aside a portion of it as our validation set.\n",
        "\n",
        "If you look at the [GLUE leaderboard](https://gluebenchmark.com/leaderboard/submission/zlssuBTm5XRs0aSKbFYGVIVdvbj1/-LhijX9VVmvJcvzKymxy), you'll see that our score (53) using the small version of BERT is not too far from the BERT authors' CoLA score with the same model. If you want to tune the hyperparameters, use the entire train set, or use the larger BERT model instead you're likely able to push that score up higher into the mid- 60s and above, which looks like the state of the art as of August 2019!\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GfjYoa6WmkN6",
        "colab_type": "text"
      },
      "source": [
        "## Conclusion"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xlQG7qgkmf4n",
        "colab_type": "text"
      },
      "source": [
        "This post showed that with a pre-trained BERT model you can quickly and effectively develope a new model by fine tuning BERT model for your specific NLP task.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "umGfk39V_87L",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# i.e., # if you run out of memory\n",
        "# !pkill -9 -f ipykernel_launcher"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}