RL_GridWorld_DQN_dynamique.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "RL-GridWorld-DQN-dynamique.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/IALeMans/RL-meetup-Reinforcement_Learning/blob/master/RL_GridWorld_DQN_dynamique.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "iXJ1sX3iFmdT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Jeu GRID WORLD - DQN 2\n",
        "\n",
        "Source https://cdancette.fr/2018/01/03/reinforcement-learning-part3/\n",
        "\n",
        "\n",
        "## Reinforcement learning en python sur un jeu simple grâce au Q-learning, Partie 3\n",
        "\n",
        "\n",
        " Jeu dynamique, le terrain change à chaque partie : utilisation d’un réseau de neurones.\n",
        " \n",
        " \n",
        "  le terrain est modifié à chaque partie. Nous n’allons pas pouvoir stocker et visiter tous les états pour entrainer l’agent. Le réseau de neurone apprendra alors a généraliser, pour obtenir une fonction de valeur Q convenable.\n",
        "\n",
        "\n",
        "> ![Texte alternatif…](https://cdancette.fr/assets/qlearning2/capture.gif) \n",
        "\n",
        "Principe :\n",
        "\n",
        "![Texte alternatif…](https://cdancette.fr/assets/game.png)"
      ]
    },
    {
      "metadata": {
        "id": "s2gBCanWGLlV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Le jeu \n",
        "\n",
        "Une seule différence : nous allons regénérer un terrain à chaque nouvelle partie. \n",
        "\n",
        "Ainsi Le seul changement sera la manière d’encoder l’état du jeu.\n",
        "\n",
        "Ainsi, la fonction get_state sera réécrite de la manière suivante :\n",
        "\n",
        "    def _get_state(self):\n",
        "        x, y = self.position\n",
        "        if self.alea:\n",
        "            return np.reshape([self._get_grille(x, y) for (x, y) in\n",
        "                    [self.position, self.end, self.hole, self.block]], (1, 64))\n",
        "        return flatten(self._get_grille(x, y))\n",
        "\n",
        "Ici, au lieu de renvoyer un tableau de taille 16 contenant juste la position de notre agent marqué par un 1, nous devons également indiquer la position des éléments sur le terrain.\n",
        "\n",
        "Les éléments à indiquer sont : \n",
        "* notre position **self.position**, \n",
        "* la position de l’arrivée **self.end**, \n",
        "* la position du trou **self.hole**, \n",
        "* et la position du mur **self.block**.\n",
        "\n",
        "Au lieu d’un tableau de taille 16 contenant des 0, avec un 1 pour marquer la position de l’agent, nous avons maintenant 4 tableaux similaires. L’état du jeu sera la concaténation de ces 4 tableaux (obtenu avec np.reshape). L’état est donc un tableau de taille 4x16 = 64 cases, et il contiendra quatre 1, et les autres éléments seront des 0.\n",
        "\n",
        "Ici pour encoder les états, nous avons utilisé ce qui est appelé un One Hot encoder : Pour chaque élément, il y a 16 possibilités, et on encode cela en utilisant un tableau de 16 cases, en mettant un 1 pour représenter la position de l’élément, et 0 dans les autres cases. Ce n’est pas une représentation très compacte : les 16 cases possibles peuvent être encodées sur 4 bits, il nous suffirait donc de 4 cases dans notre tableau pour encoder la position de chaque élément. Mais le réseau devrait alors apprendre à décoder cette représentation, ce qui nécessiterait certainement un modèle plus complexe, donc un entrainement plus long, ou bien une architecture plus grosse. L’encodage que nous avons choisi ici est extrèmement simple (mais il n’est pas le plus compact possible, toutes les entrées possibles ne sont pas du tout utilisés), et sera utilisé très facilement par le réseau de neurone."
      ]
    },
    {
      "metadata": {
        "id": "I1YDi7KFHWJE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import random\n",
        "\n",
        "flatten = lambda l: [item for sublist in l for item in sublist]\n",
        "\n",
        "class Game:\n",
        "    ACTION_UP = 0\n",
        "    ACTION_LEFT = 1\n",
        "    ACTION_DOWN = 2\n",
        "    ACTION_RIGHT = 3\n",
        "\n",
        "    ACTIONS = [ACTION_DOWN, ACTION_LEFT, ACTION_RIGHT, ACTION_UP]\n",
        "\n",
        "    ACTION_NAMES = [\"UP   \", \"LEFT \", \"DOWN \", \"RIGHT\"]\n",
        "\n",
        "    MOVEMENTS = {\n",
        "        ACTION_UP: (1, 0),\n",
        "        ACTION_RIGHT: (0, 1),\n",
        "        ACTION_LEFT: (0, -1),\n",
        "        ACTION_DOWN: (-1, 0)\n",
        "    }\n",
        "\n",
        "    num_actions = len(ACTIONS)\n",
        "\n",
        "    def __init__(self, n, m, wrong_action_p=0.1, alea=False):\n",
        "        self.n = n\n",
        "        self.m = m\n",
        "        self.wrong_action_p = wrong_action_p\n",
        "        self.alea = alea\n",
        "        self.generate_game()\n",
        "\n",
        "    def _position_to_id(self, x, y):\n",
        "        \"\"\"Donne l'identifiant de la position entre 0 et 15\"\"\"\n",
        "        return x + y * self.n\n",
        "\n",
        "    def _id_to_position(self, id):\n",
        "        \"\"\"Réciproque de la fonction précédente\"\"\"\n",
        "        return (id % self.n, id // self.n)\n",
        "\n",
        "    def generate_game(self):\n",
        "        cases = [(x, y) for x in range(self.n) for y in range(self.m)]\n",
        "        hole = random.choice(cases)\n",
        "        cases.remove(hole)\n",
        "        start = random.choice(cases)\n",
        "        cases.remove(start)\n",
        "        end = random.choice(cases)\n",
        "        cases.remove(end)\n",
        "        block = random.choice(cases)\n",
        "        cases.remove(block)\n",
        "\n",
        "        self.position = start\n",
        "        self.end = end\n",
        "        self.hole = hole\n",
        "        self.block = block\n",
        "        self.counter = 0\n",
        "        \n",
        "        if not self.alea:\n",
        "            self.start = start\n",
        "        return self._get_state()\n",
        "    \n",
        "    def reset(self):\n",
        "        if not self.alea:\n",
        "            self.position = self.start\n",
        "            self.counter = 0\n",
        "            return self._get_state()\n",
        "        else:\n",
        "            return self.generate_game()\n",
        "\n",
        "    def _get_grille(self, x, y):\n",
        "        grille = [\n",
        "            [0] * self.n for i in range(self.m)\n",
        "        ]\n",
        "        grille[x][y] = 1\n",
        "        return grille\n",
        "\n",
        "    def _get_state(self):\n",
        "        x, y = self.position\n",
        "        if self.alea:\n",
        "            return np.reshape([self._get_grille(x, y) for (x, y) in\n",
        "                    [self.position, self.end, self.hole, self.block]], (1, 64))\n",
        "        return flatten(self._get_grille(x, y))\n",
        "    \n",
        "    def get_random_action(self):\n",
        "        return random.choice(self.ACTIONS)\n",
        "    \n",
        "    def move(self, action):\n",
        "        \"\"\"\n",
        "        takes an action parameter\n",
        "        :param action : the id of an action\n",
        "        :return ((state_id, end, hole, block), reward, is_final, actions)\n",
        "        \"\"\"\n",
        "\n",
        "        self.counter += 1\n",
        "\n",
        "        if action not in self.ACTIONS:\n",
        "            raise Exception(\"Invalid action\")\n",
        "\n",
        "        # random actions sometimes (2 times over 10 default)\n",
        "        choice = random.random()\n",
        "        if choice < self.wrong_action_p:\n",
        "            action = (action + 1) % 4\n",
        "        elif choice < 2 * self.wrong_action_p:\n",
        "            action = (action - 1) % 4\n",
        "\n",
        "        d_x, d_y = self.MOVEMENTS[action]\n",
        "        x, y = self.position\n",
        "        new_x, new_y = x + d_x, y + d_y\n",
        "\n",
        "        if self.block == (new_x, new_y):\n",
        "            return self._get_state(), -1, False, self.ACTIONS\n",
        "        elif self.hole == (new_x, new_y):\n",
        "            self.position = new_x, new_y\n",
        "            return self._get_state(), -10, True, None\n",
        "        elif self.end == (new_x, new_y):\n",
        "            self.position = new_x, new_y\n",
        "            return self._get_state(), 10, True, self.ACTIONS\n",
        "        elif new_x >= self.n or new_y >= self.m or new_x < 0 or new_y < 0:\n",
        "            return self._get_state(), -1, False, self.ACTIONS\n",
        "        elif self.counter > 200:\n",
        "            self.position = new_x, new_y\n",
        "            return self._get_state(), -10, True, self.ACTIONS\n",
        "        else:\n",
        "            self.position = new_x, new_y\n",
        "            return self._get_state(), -1, False, self.ACTIONS\n",
        "\n",
        "    def print(self):\n",
        "        str = \"\"\n",
        "        for i in range(self.n - 1, -1, -1):\n",
        "            for j in range(self.m):\n",
        "                if (i, j) == self.position:\n",
        "                    str += \"x\"\n",
        "                elif (i, j) == self.block:\n",
        "                    str += \"¤\"\n",
        "                elif (i, j) == self.hole:\n",
        "                    str += \"o\"\n",
        "                elif (i, j) == self.end:\n",
        "                    str += \"@\"\n",
        "                else:\n",
        "                    str += \".\"\n",
        "            str += \"\\n\"\n",
        "        print(str)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "SpxB4iv0HRyn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Un problème plus complexe\n",
        "\n",
        "On peut voir que le problème est plus complexe que celui de la partie précédente : \n",
        "\n",
        "\n",
        "> au lieu de 4 états différents, encodés dans un tableau de taille 16,\n",
        "\n",
        "> on a 16x15x14x13 = 43680 états possibles. \n",
        "\n",
        "\n",
        "\n",
        "Il serait difficile d’appliquer la méthode de la première partie de ce tutoriel (stocker les Q-values dans un tableau). \n",
        "\n",
        "L’utilisation d’un réseau de neurone, comme nous l’avons vu dans la partie 2, nous sera alors très utile ici. \n",
        "\n",
        "\n",
        "Avec un réseau légèrement plus complexe, nous allons pouvoir résoudre ce problème. Néanmoins, l’entrainement est plus compliqué ici. Pour garantir la convergence de la méthode classique du Q-learning, l’agent devrait parcourir tous les états un grand nombre de fois. Or ici, notre espace d’état étant très grand, l’agent ne parcourera surement pas la totalité de ces états de nombreuses fois. C’est pour cela que nous attendons de notre réseau de neurone qu’il généralise, pour appliquer ses connaissances acquises à l’entrainement sur des états qu’il n’a jamais rencontré. Il aurait été impossible de généraliser avec la méthode de l’article 1, en utilisant un tableau.\n",
        "\n",
        "Nous allons évoquer plusieurs concepts très utilisés en machine learning et en reinforcement learning : **le principe du batch**, et celui de **l’experience replay**."
      ]
    },
    {
      "metadata": {
        "id": "r5N_2a-KH0p8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Batch\n",
        "\n",
        "En machine learning, pour entrainer nos réseaux de neurones, on utilise généralement des batch de données. \n",
        "C’est à dire qu’au lieu de ne donner qu’un seul exemple, avec son label, on lui donne à chaque fois un nombre fixe d’exemples (par exemple 10 samples). Cela permet à l’algorithme de gradient de choisir une direction qui ne dépendra pas que d’un seul exemple, qui pourrait être trop précis et ne pas améliorer le score global, mais plutôt une direction moyenne, qui sera certainement plus bénéfique au réseau de manière générale.\n",
        "\n",
        " ![Texte alternatif…](https://cdancette.fr/assets/qlearning3/batches.png)\n",
        "\n",
        "Le batching est également utilisé quand le dataset entier ne rentre pas dans la RAM / la mémoire du GPU. Il est alors nécéssaire de diviser le dataset en batches, que l’on va charger en mémoire pour entrainer le réseau, puis décharger. La contrainte est alors que la taille d’un batch ne dépasse pas la taille de la mémoire (c’est surtout un problème en traitement d’image, ou les données ont une taille importante).\n",
        "\n",
        "Le batching est utilisé avec l’algorithme de **stochastig gradient descent** ou descente de gradiant stochastique, en remplaçant un exemple par un petit nombre d’exemples (ou bien avec d’autres algorithmes dérivés tels que Adam).\n",
        "\n",
        "Le batching est très souvent utilisé en deep learning. Toutefois en reinforcement learning, cela paraît plus compliqué, puisque nous n’avons qu’un exemple à chaque action effectuée. Il est donc impossible à priori d’utiliser cette méthode. Nous allons voir que la méthode de **l’experience replay** permet de résoudre ce problème"
      ]
    },
    {
      "metadata": {
        "id": "zjeDtSrfIRcV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Experience Replay\n",
        "\n",
        "L’experience replay est une méthode spécifique au reinforcement learning (contrairement au batching qui est utilisé très souvent en deep learning).\n",
        "\n",
        "Il nous permet en fait d’utiliser des batch pendant l’entrainement de notre agent, au lieu de l’entrainer à chaque mouvement sur les données qu’il vient de recevoir.\n",
        "\n",
        "Il s’agit de stocker à chaque mouvement les paramètres d’entrainement (état de départ, action, état d’arrivée, récompense, fin du jeu) dans une mémoire, au lieu d’entrainer notre réseau de neurone dessus. Et ensuite, régulièrement, on va piocher un batch dans cette mémoire (c’est à dire un certain nombre d’exemples), au hasard, et on va entrainer notre réseau sur ce batch.\n",
        "\n",
        "Cela permet d’éviter un trop grand va-et-vient des poids du réseau. En effet, le réseau oublie ce qu’il vient d’apprendre si on lui donne des exemples successifs qui ont des indications contraires (il n’arrive pas à généraliser, et va osciller). En lui donnant un batch en effet, la backpropagation va choisir une direction moyenne pour optimiser les poids du réseau afin de faire diminuer l’erreur.\n",
        "\n",
        "Cela va également nous permettre de voir plusieurs fois des situations passées. Et les exemples trop vieux seront vidés de la mémoire (on limite la taille de la mémoire en nombre d’exemples).\n",
        "\n",
        " ![Texte alternatif…](https://cdancette.fr/assets/qlearning3/experiencereplay.png)\n",
        "\n",
        "Il existe de nombreuses améliorations possibles. Par exempe, le *Prioritized experience replay (article [1] ou blog), ou on voit les situations les plus importantes en priorité, au lieu tirer des exemples au hasard dans la mémoire."
      ]
    },
    {
      "metadata": {
        "id": "Unpxxw4KIuh9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Entrainement"
      ]
    },
    {
      "metadata": {
        "id": "0jnv2CgrFlmr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# defining the neural network\n",
        "import numpy as np\n",
        "from keras.models import Sequential, load_model\n",
        "from keras.layers.core import Dense, Dropout, Activation\n",
        "from keras.optimizers import RMSprop, Adam, sgd\n",
        "from keras.layers.advanced_activations import LeakyReLU\n",
        "import random\n",
        "import os\n",
        "\n",
        "from collections import deque\n",
        "\n",
        "class Trainer:\n",
        "    def __init__(self, name=None, learning_rate=0.001, epsilon_decay=0.9999, batch_size=30, memory_size=3000):\n",
        "        self.state_size = 64\n",
        "        self.action_size = 4\n",
        "        self.gamma = 0.9\n",
        "        self.epsilon = 1.0\n",
        "        self.epsilon_min = 0.01\n",
        "        self.epsilon_decay = epsilon_decay\n",
        "        self.learning_rate = learning_rate\n",
        "        self.memory = deque(maxlen=memory_size)\n",
        "        self.batch_size = batch_size\n",
        "        \n",
        "        self.name = name\n",
        "        if name is not None and os.path.isfile(\"model-\" + name):\n",
        "                model = load_model(\"model-\" + name)\n",
        "        else:\n",
        "            model = Sequential()\n",
        "            model.add(Dense(50, input_dim=self.state_size, activation='relu'))\n",
        "            model.add(Dense(30, activation='relu'))\n",
        "            model.add(Dense(30, activation='relu'))\n",
        "            model.add(Dense(self.action_size, activation='linear'))\n",
        "            model.compile(loss='mse', optimizer=Adam(lr=self.learning_rate))\n",
        "        \n",
        "        self.model = model\n",
        "        \n",
        "    def decay_epsilon(self):\n",
        "        self.epsilon *= self.epsilon_decay\n",
        "    \n",
        "    def get_best_action(self, state, rand=True):\n",
        "\n",
        "        if rand and np.random.rand() <= self.epsilon:\n",
        "            # The agent acts randomly\n",
        "            return random.randrange(self.action_size)\n",
        "        \n",
        "        # Predict the reward value based on the given state\n",
        "        act_values = self.model.predict(np.array(state))\n",
        "\n",
        "        # Pick the action based on the predicted reward\n",
        "        action =  np.argmax(act_values[0])  \n",
        "        return action\n",
        "\n",
        "\n",
        "    def replay(self, batch_size):\n",
        "        batch_size = min(batch_size, len(self.memory))\n",
        "\n",
        "        minibatch = random.sample(self.memory, batch_size)\n",
        "\n",
        "        inputs = np.zeros((batch_size, self.state_size))\n",
        "        outputs = np.zeros((batch_size, self.action_size))\n",
        "\n",
        "        for i, (state, action, reward, next_state, done) in enumerate(minibatch):\n",
        "            target = self.model.predict(state)[0]\n",
        "            if done:\n",
        "                target[action] = reward\n",
        "            else:\n",
        "                target[action] = reward + self.gamma * np.max(self.model.predict(next_state))\n",
        "\n",
        "            inputs[i] = state\n",
        "            outputs[i] = target\n",
        "\n",
        "        return self.model.fit(inputs, outputs, epochs=1, verbose=0, batch_size=batch_size)\n",
        "\n",
        "    def save(self, id=None, overwrite=False):\n",
        "        name = 'model'\n",
        "        if self.name:\n",
        "            name += '-' + self.name\n",
        "        else:\n",
        "            name += '-' + str(time.time())\n",
        "        if id:\n",
        "            name += '-' + id\n",
        "        self.model.save(name, overwrite=overwrite)\n",
        "        \n",
        "    def remember(self, state, action, reward, next_state, done):\n",
        "        self.memory.append([state, action, reward, next_state, done])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ovqi4vYSI7r6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Paramètres du trainer\n",
        "\n",
        "\n",
        "Les paramètres sont assez similaires à ceux de l’article p’écédent. On a juste ajouté une nouvelle couche à notre réseau (pour lui donner une meilleur force de représentation des données).\n",
        "\n",
        "La ligne qui change est celle-ci :\n",
        "\n",
        "        self.memory = deque(maxlen=memory_size)\n",
        "\n",
        "**memory** est la structure de données qui va nous servir de mémoire pour stocker nos ensembles (state, action, new_state, reward). C’est grâce à cette mémoire que l’on peut faire de l’experience replay. A chaque action, on va remplir cette mémoire au lieu d’entrainer, puis on va régulièrement piocher aléatoirement des samples dans cette mémoire, pour lancer l’entrainement sur un batch de données. Pour stocker, on utilise la structure collections.deque de python. Il s’agit d’une queue qui peut avoir une taille limitée, qui va supprimer automatiquement les éléments ajoutés les premiers lorsque la taille limite est atteinte."
      ]
    },
    {
      "metadata": {
        "id": "NlwSzOBnJI6v",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Apprentissage\n",
        "\n",
        "Nous allons remplacer la fonction train par une fonction remember Au lieu de lancer une étape de backpropagation, elle va tout simplement stocker ce que l’on vient de voir, dans une queue (une structure de données qui va supprimer les éléments entrés en premier).\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "pyjansN3JP2K",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class Trainer:\n",
        "    \n",
        "    ...\n",
        "    \n",
        "    def remember(self, state, action, reward, next_state, done):\n",
        "        self.memory.append([state, action, reward, next_state, done])\n",
        "    \n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "G5-hFjYZJSXd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Et enfin, il nous faut une fonction replay qui va piocher dans la mémoire, et donner ces données aux réseau de neurone.\n",
        "\n",
        "   "
      ]
    },
    {
      "metadata": {
        "id": "701-dbaQJZXm",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "...\n",
        "  def replay(self, batch_size):\n",
        "        batch_size = min(batch_size, len(self.memory))\n",
        "\n",
        "        minibatch = random.sample(self.memory, batch_size)\n",
        "\n",
        "        inputs = np.zeros((batch_size, self.state_size))\n",
        "        outputs = np.zeros((batch_size, self.action_size))\n",
        "\n",
        "        for i, (state, action, reward, next_state, done) in enumerate(minibatch):\n",
        "            target = self.model.predict(state)[0]\n",
        "            if done:\n",
        "                target[action] = reward\n",
        "            else:\n",
        "                target[action] = reward + self.gamma * np.max(self.model.predict(next_state))\n",
        "\n",
        "            inputs[i] = state\n",
        "            outputs[i] = target"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Ddz8-c82JXd2",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "Ainsi, ici, on va utiliser random.sample pour piocher un certain nombres d’éléments aléatoirement dans la mémoire. On crée alors nos entrées et sorties dans le bon format pour le réseau de neurone, similairement à la fonction train de l’article précédent. La différence est qu’ici, on crée un batch de plusieurs samples, au lieu de n’en donner qu’un (on voit que la dimension des input et output est (batch_size, state_size), alors qu’elle n’avait qu’une dimension précedemment."
      ]
    },
    {
      "metadata": {
        "id": "MfDIxgGHJjmg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Lancer l’entrainement\n",
        "\n",
        "La fonction d’entrainement est un peu plus complexe, puisqu’on va executer une première partie ou l’on va remplir en partie la mémoire. Cela nous permettra de pouvoir créer des batch avec assez de données plus rapidement. Cette phase se déroule entre les lignes 13 et 25 du code ci-dessous.\n",
        "\n",
        "La deuxième phase est l’entrainement du réseau. On lance un entrainement à chaque 100 mouvements. On pourrait essayer d’en lancer plus ou moins souvent, l’apprentissage en serait surement impacté au niveau rapidité de convergence et qualité du minimum local. En général, lorsqu’un algorithme converge trop vite, le minimum local sera moins bon."
      ]
    },
    {
      "metadata": {
        "id": "yumWhwTzJqU-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import time \n",
        "from IPython.core.debugger import set_trace\n",
        "\n",
        "def train(episodes, trainer, wrong_action_p, alea, collecting=False, snapshot=5000):\n",
        "    batch_size = 32\n",
        "    g = Game(4, 4, wrong_action_p, alea=alea)\n",
        "    counter = 1\n",
        "    scores = []\n",
        "    global_counter = 0\n",
        "    losses = [0]\n",
        "    epsilons = []\n",
        "\n",
        "    # we start with a sequence to collect information, without learning\n",
        "    if collecting:\n",
        "        collecting_steps = 10000\n",
        "        print(\"Collecting game without learning\")\n",
        "        steps = 0\n",
        "        while steps < collecting_steps:\n",
        "            state = g.reset()\n",
        "            done = False\n",
        "            while not done:\n",
        "                steps += 1\n",
        "                action = g.get_random_action()\n",
        "                next_state, reward, done, _ = g.move(action)\n",
        "                trainer.remember(state, action, reward, next_state, done)\n",
        "                state = next_state\n",
        "\n",
        "    print(\"Starting training\")  \n",
        "    global_counter = 0\n",
        "    for e in range(episodes+1):\n",
        "        state = g.generate_game()\n",
        "        state = np.reshape(state, [1, 64])\n",
        "        score = 0\n",
        "        done = False\n",
        "        steps = 0\n",
        "        while not done:\n",
        "            steps += 1\n",
        "            global_counter += 1\n",
        "            action = trainer.get_best_action(state)\n",
        "            trainer.decay_epsilon()\n",
        "            next_state, reward, done, _ = g.move(action)\n",
        "            next_state = np.reshape(next_state, [1, 64])\n",
        "            score += reward\n",
        "            trainer.remember(state, action, reward, next_state, done)\n",
        "            state = next_state\n",
        "            if global_counter % 100 == 0:\n",
        "                l = trainer.replay(batch_size)\n",
        "                losses.append(l.history['loss'][0])\n",
        "            if done:\n",
        "                scores.append(score)\n",
        "                epsilons.append(trainer.epsilon)\n",
        "            if steps > 200:\n",
        "                break\n",
        "        if e % 200 == 0:\n",
        "            print(\"episode: {}/{}, moves: {}, score: {}, epsilon: {}, loss: {}\"\n",
        "                  .format(e, episodes, steps, score, trainer.epsilon, losses[-1]))\n",
        "        if e > 0 and e % snapshot == 0:\n",
        "            trainer.save(id='iteration-%s' % e)\n",
        "    return scores, losses, epsilons"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FIwoUxmIJ271",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "On peut alors lancer l’entrainement"
      ]
    },
    {
      "metadata": {
        "id": "QzGbPze6J6R7",
        "colab_type": "code",
        "outputId": "5f24c3c7-9586-4fb4-b4ea-cd2284897d9f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 3085
        }
      },
      "cell_type": "code",
      "source": [
        "trainer = Trainer(learning_rate=0.001, epsilon_decay=0.999995)\n",
        "scores, losses, epsilons = train(35000, trainer, 0.1, True, snapshot=2500)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Starting training\n",
            "episode: 0/35000, moves: 16, score: -25, epsilon: 0.9999200029999294, loss: 0\n",
            "episode: 200/35000, moves: 13, score: -22, epsilon: 0.9814001063725329, loss: 1.7394689321517944\n",
            "episode: 400/35000, moves: 7, score: 4, epsilon: 0.9648862218983311, loss: 2.8518009185791016\n",
            "episode: 600/35000, moves: 11, score: -20, epsilon: 0.9473988189758515, loss: 1.1752405166625977\n",
            "episode: 800/35000, moves: 18, score: -7, epsilon: 0.9318204094956437, loss: 2.499586820602417\n",
            "episode: 1000/35000, moves: 2, score: -11, epsilon: 0.9167181477463254, loss: 3.792001485824585\n",
            "episode: 1200/35000, moves: 18, score: -27, epsilon: 0.9001982493573206, loss: 2.2735531330108643\n",
            "episode: 1400/35000, moves: 15, score: -4, epsilon: 0.8835341723717306, loss: 0.05706949904561043\n",
            "episode: 1600/35000, moves: 35, score: -44, epsilon: 0.8696361708712141, loss: 3.0634708404541016\n",
            "episode: 1800/35000, moves: 37, score: -46, epsilon: 0.8517089646646874, loss: 5.344153881072998\n",
            "episode: 2000/35000, moves: 7, score: 4, epsilon: 0.8343056533893967, loss: 0.1476757526397705\n",
            "episode: 2200/35000, moves: 92, score: -101, epsilon: 0.8208207754805851, loss: 1.8457168340682983\n",
            "episode: 2400/35000, moves: 57, score: -46, epsilon: 0.8054690174708321, loss: 2.1719095706939697\n",
            "episode: 2600/35000, moves: 8, score: 3, epsilon: 0.790495284358735, loss: 1.4876153469085693\n",
            "episode: 2800/35000, moves: 29, score: -38, epsilon: 0.775299683282988, loss: 3.89227294921875\n",
            "episode: 3000/35000, moves: 12, score: -21, epsilon: 0.7599818814722132, loss: 1.9834686517715454\n",
            "episode: 3200/35000, moves: 1, score: -10, epsilon: 0.743846383093576, loss: 7.926179885864258\n",
            "episode: 3400/35000, moves: 9, score: 2, epsilon: 0.726813187718767, loss: 2.0214126110076904\n",
            "episode: 3600/35000, moves: 11, score: -20, epsilon: 0.7118942886661673, loss: 2.396277666091919\n",
            "episode: 3800/35000, moves: 18, score: -7, epsilon: 0.6980734876605514, loss: 3.698812961578369\n",
            "episode: 4000/35000, moves: 24, score: -33, epsilon: 0.6798344510648243, loss: 1.9468047618865967\n",
            "episode: 4200/35000, moves: 10, score: 1, epsilon: 0.6650247507599166, loss: 2.4519639015197754\n",
            "episode: 4400/35000, moves: 1, score: 10, epsilon: 0.6505083954222641, loss: 3.4071452617645264\n",
            "episode: 4600/35000, moves: 10, score: 1, epsilon: 0.6362325546686318, loss: 0.04726772382855415\n",
            "episode: 4800/35000, moves: 147, score: -136, epsilon: 0.6221611187696863, loss: 0.4380035996437073\n",
            "episode: 5000/35000, moves: 22, score: -11, epsilon: 0.6086990884115006, loss: 0.13677585124969482\n",
            "episode: 5200/35000, moves: 8, score: -17, epsilon: 0.5949569088439031, loss: 0.1740928739309311\n",
            "episode: 5400/35000, moves: 19, score: -28, epsilon: 0.5835434610836164, loss: 3.3097753524780273\n",
            "episode: 5600/35000, moves: 201, score: -201, epsilon: 0.5717826184579975, loss: 0.060022443532943726\n",
            "episode: 5800/35000, moves: 9, score: 2, epsilon: 0.5605782453750852, loss: 0.2713325321674347\n",
            "episode: 6000/35000, moves: 5, score: 6, epsilon: 0.5473556491108919, loss: 1.7022207975387573\n",
            "episode: 6200/35000, moves: 2, score: -11, epsilon: 0.5368741577830463, loss: 2.1379952430725098\n",
            "episode: 6400/35000, moves: 7, score: 4, epsilon: 0.5260512666409267, loss: 0.3826833963394165\n",
            "episode: 6600/35000, moves: 9, score: -18, epsilon: 0.5172434383357347, loss: 0.22868990898132324\n",
            "episode: 6800/35000, moves: 8, score: -17, epsilon: 0.5055179633053899, loss: 0.3151611089706421\n",
            "episode: 7000/35000, moves: 1, score: -10, epsilon: 0.49378416800023583, loss: 2.0301947593688965\n",
            "episode: 7200/35000, moves: 16, score: -25, epsilon: 0.4794254659529851, loss: 0.04685003310441971\n",
            "episode: 7400/35000, moves: 8, score: 3, epsilon: 0.4682130287073306, loss: 2.0644049644470215\n",
            "episode: 7600/35000, moves: 1, score: -10, epsilon: 0.4568331932263625, loss: 0.18184669315814972\n",
            "episode: 7800/35000, moves: 33, score: -22, epsilon: 0.4501893552752338, loss: 3.100210428237915\n",
            "episode: 8000/35000, moves: 12, score: -1, epsilon: 0.44038891257806556, loss: 1.3547040224075317\n",
            "episode: 8200/35000, moves: 9, score: -18, epsilon: 0.4318046036338949, loss: 0.4160614609718323\n",
            "episode: 8400/35000, moves: 2, score: 9, epsilon: 0.423550660904482, loss: 1.9357020854949951\n",
            "episode: 8600/35000, moves: 18, score: -7, epsilon: 0.41724688367141827, loss: 1.6300020217895508\n",
            "episode: 8800/35000, moves: 17, score: -26, epsilon: 0.4090952624534873, loss: 1.5407683849334717\n",
            "episode: 9000/35000, moves: 26, score: -15, epsilon: 0.40276489534444915, loss: 3.5737457275390625\n",
            "episode: 9200/35000, moves: 8, score: 3, epsilon: 0.3955997471502964, loss: 0.2477540671825409\n",
            "episode: 9400/35000, moves: 1, score: 10, epsilon: 0.3895288461215431, loss: 1.4689751863479614\n",
            "episode: 9600/35000, moves: 7, score: 4, epsilon: 0.38364892780784143, loss: 2.475965976715088\n",
            "episode: 9800/35000, moves: 6, score: -15, epsilon: 0.3782263584681846, loss: 0.3526593744754791\n",
            "episode: 10000/35000, moves: 15, score: -24, epsilon: 0.3725803840764997, loss: 3.5720791816711426\n",
            "episode: 10200/35000, moves: 58, score: -67, epsilon: 0.3672756936401816, loss: 0.17742732167243958\n",
            "episode: 10400/35000, moves: 5, score: 6, epsilon: 0.36249393463181495, loss: 3.2910196781158447\n",
            "episode: 10600/35000, moves: 51, score: -40, epsilon: 0.3578155784340249, loss: 0.19354507327079773\n",
            "episode: 10800/35000, moves: 8, score: -17, epsilon: 0.35257651860580064, loss: 0.20473240315914154\n",
            "episode: 11000/35000, moves: 2, score: -11, epsilon: 0.3472752297556495, loss: 0.19877997040748596\n",
            "episode: 11200/35000, moves: 23, score: -12, epsilon: 0.34172714418671857, loss: 2.637934684753418\n",
            "episode: 11400/35000, moves: 6, score: -15, epsilon: 0.335230229035445, loss: 1.820784568786621\n",
            "episode: 11600/35000, moves: 1, score: -10, epsilon: 0.33024093995933756, loss: 4.074269771575928\n",
            "episode: 11800/35000, moves: 27, score: -16, epsilon: 0.3255520879892071, loss: 1.1899374723434448\n",
            "episode: 12000/35000, moves: 7, score: 4, epsilon: 0.3206042303216007, loss: 1.1551300287246704\n",
            "episode: 12200/35000, moves: 6, score: 5, epsilon: 0.3164949870811239, loss: 2.6760592460632324\n",
            "episode: 12400/35000, moves: 43, score: -52, epsilon: 0.3119732250181153, loss: 0.22417578101158142\n",
            "episode: 12600/35000, moves: 3, score: 8, epsilon: 0.30697991639534034, loss: 0.28171756863594055\n",
            "episode: 12800/35000, moves: 3, score: 8, epsilon: 0.3034653080640002, loss: 1.7495909929275513\n",
            "episode: 13000/35000, moves: 1, score: 10, epsilon: 0.2979236518029852, loss: 0.249897301197052\n",
            "episode: 13200/35000, moves: 7, score: 4, epsilon: 0.2928870995673728, loss: 5.565365791320801\n",
            "episode: 13400/35000, moves: 1, score: -10, epsilon: 0.2901613610571772, loss: 2.3399455547332764\n",
            "episode: 13600/35000, moves: 6, score: 5, epsilon: 0.2862948294785686, loss: 3.8240227699279785\n",
            "episode: 13800/35000, moves: 1, score: -10, epsilon: 0.2823823822120596, loss: 4.33489990234375\n",
            "episode: 14000/35000, moves: 24, score: -13, epsilon: 0.27906565820442836, loss: 1.941666841506958\n",
            "episode: 14200/35000, moves: 4, score: 7, epsilon: 0.2755384150389769, loss: 3.4399046897888184\n",
            "episode: 14400/35000, moves: 3, score: -12, epsilon: 0.2718096546761918, loss: 1.600379228591919\n",
            "episode: 14600/35000, moves: 1, score: -10, epsilon: 0.26813403559414606, loss: 1.5612488985061646\n",
            "episode: 14800/35000, moves: 1, score: -10, epsilon: 0.2643772216117732, loss: 1.871691346168518\n",
            "episode: 15000/35000, moves: 2, score: 9, epsilon: 0.261903713247428, loss: 1.9994609355926514\n",
            "episode: 15200/35000, moves: 3, score: 8, epsilon: 0.25919272669965693, loss: 1.8172094821929932\n",
            "episode: 15400/35000, moves: 3, score: 8, epsilon: 0.2559013123599464, loss: 1.1231383085250854\n",
            "episode: 15600/35000, moves: 9, score: 2, epsilon: 0.25354640118964206, loss: 1.4962897300720215\n",
            "episode: 15800/35000, moves: 4, score: -13, epsilon: 0.2509645823152537, loss: 2.3563218116760254\n",
            "episode: 16000/35000, moves: 2, score: 9, epsilon: 0.24780367291358935, loss: 6.542445182800293\n",
            "episode: 16200/35000, moves: 4, score: 7, epsilon: 0.24379354751344176, loss: 0.3285670280456543\n",
            "episode: 16400/35000, moves: 1, score: 10, epsilon: 0.24047994953980437, loss: 0.2683144807815552\n",
            "episode: 16600/35000, moves: 5, score: 6, epsilon: 0.23811689093523677, loss: 0.6956371068954468\n",
            "episode: 16800/35000, moves: 1, score: 10, epsilon: 0.23570043746988334, loss: 2.071943759918213\n",
            "episode: 17000/35000, moves: 3, score: 8, epsilon: 0.23302287796185994, loss: 3.0789687633514404\n",
            "episode: 17200/35000, moves: 20, score: -9, epsilon: 0.23035385084863416, loss: 1.833528757095337\n",
            "episode: 17400/35000, moves: 2, score: 9, epsilon: 0.22817242310473293, loss: 3.20135760307312\n",
            "episode: 17600/35000, moves: 28, score: -17, epsilon: 0.2258207536038493, loss: 4.974000930786133\n",
            "episode: 17800/35000, moves: 1, score: 10, epsilon: 0.22322194152462063, loss: 1.8656506538391113\n",
            "episode: 18000/35000, moves: 1, score: -10, epsilon: 0.22147650472268782, loss: 1.2691651582717896\n",
            "episode: 18200/35000, moves: 1, score: 10, epsilon: 0.2198073523208958, loss: 2.677577018737793\n",
            "episode: 18400/35000, moves: 12, score: -1, epsilon: 0.21738641055180188, loss: 2.9454009532928467\n",
            "episode: 18600/35000, moves: 2, score: 9, epsilon: 0.21530625140463103, loss: 2.6660239696502686\n",
            "episode: 18800/35000, moves: 18, score: -7, epsilon: 0.21333984630978636, loss: 0.7686796188354492\n",
            "episode: 19000/35000, moves: 5, score: 6, epsilon: 0.21064544929220808, loss: 2.00478458404541\n",
            "episode: 19200/35000, moves: 2, score: 9, epsilon: 0.20859015805492773, loss: 2.5066356658935547\n",
            "episode: 19400/35000, moves: 4, score: 7, epsilon: 0.2065631829071288, loss: 2.457782745361328\n",
            "episode: 19600/35000, moves: 2, score: 9, epsilon: 0.20450988487691144, loss: 0.8765338063240051\n",
            "episode: 19800/35000, moves: 1, score: -10, epsilon: 0.20245675048746814, loss: 1.8750965595245361\n",
            "episode: 20000/35000, moves: 3, score: 8, epsilon: 0.20103645671338674, loss: 4.777826309204102\n",
            "episode: 20200/35000, moves: 6, score: 5, epsilon: 0.19959718295646836, loss: 2.453378677368164\n",
            "episode: 20400/35000, moves: 4, score: 7, epsilon: 0.19767044525004906, loss: 0.5607945919036865\n",
            "episode: 20600/35000, moves: 1, score: 10, epsilon: 0.19652530694834688, loss: 2.3085711002349854\n",
            "episode: 20800/35000, moves: 3, score: 8, epsilon: 0.1946535258395752, loss: 1.3926372528076172\n",
            "episode: 21000/35000, moves: 2, score: -11, epsilon: 0.19309864408518873, loss: 2.7315707206726074\n",
            "episode: 21200/35000, moves: 5, score: 6, epsilon: 0.19186771526883714, loss: 3.656651735305786\n",
            "episode: 21400/35000, moves: 2, score: 9, epsilon: 0.19031795698767806, loss: 2.1385812759399414\n",
            "episode: 21600/35000, moves: 4, score: 7, epsilon: 0.1891312305560406, loss: 1.6711130142211914\n",
            "episode: 21800/35000, moves: 3, score: 8, epsilon: 0.18772368048962162, loss: 2.9670674800872803\n",
            "episode: 22000/35000, moves: 7, score: 4, epsilon: 0.1864197925051836, loss: 3.9108972549438477\n",
            "episode: 22200/35000, moves: 2, score: -11, epsilon: 0.18483731471343223, loss: 0.5465146899223328\n",
            "episode: 22400/35000, moves: 7, score: -16, epsilon: 0.18369762120803249, loss: 1.963381290435791\n",
            "episode: 22600/35000, moves: 9, score: 2, epsilon: 0.18266082697554414, loss: 0.8726239800453186\n",
            "episode: 22800/35000, moves: 14, score: -3, epsilon: 0.1815381842724734, loss: 2.4956603050231934\n",
            "episode: 23000/35000, moves: 1, score: -10, epsilon: 0.17995035211870125, loss: 2.7572922706604004\n",
            "episode: 23200/35000, moves: 3, score: -12, epsilon: 0.17873083772364434, loss: 2.4263503551483154\n",
            "episode: 23400/35000, moves: 4, score: 7, epsilon: 0.17735900496191695, loss: 1.6549367904663086\n",
            "episode: 23600/35000, moves: 3, score: 8, epsilon: 0.17611213756119276, loss: 1.8833613395690918\n",
            "episode: 23800/35000, moves: 7, score: 4, epsilon: 0.17469226096207138, loss: 2.3437411785125732\n",
            "episode: 24000/35000, moves: 1, score: 10, epsilon: 0.17371497877449488, loss: 1.9282853603363037\n",
            "episode: 24200/35000, moves: 4, score: 7, epsilon: 0.17265249721817472, loss: 1.6636219024658203\n",
            "episode: 24400/35000, moves: 2, score: -11, epsilon: 0.171716673959562, loss: 2.341569423675537\n",
            "episode: 24600/35000, moves: 1, score: -10, epsilon: 0.17074408558581528, loss: 1.8605129718780518\n",
            "episode: 24800/35000, moves: 4, score: 7, epsilon: 0.16998001116944572, loss: 2.2680792808532715\n",
            "episode: 25000/35000, moves: 6, score: 5, epsilon: 0.1690012031862921, loss: 1.7882192134857178\n",
            "episode: 25200/35000, moves: 2, score: 9, epsilon: 0.16795999379844004, loss: 2.364377498626709\n",
            "episode: 25400/35000, moves: 3, score: 8, epsilon: 0.16700951798345504, loss: 3.46917986869812\n",
            "episode: 25600/35000, moves: 18, score: -7, epsilon: 0.16619234002212335, loss: 2.1712050437927246\n",
            "episode: 25800/35000, moves: 3, score: 8, epsilon: 0.16528740031678715, loss: 7.171788215637207\n",
            "episode: 26000/35000, moves: 3, score: -12, epsilon: 0.1640400753487471, loss: 2.156291961669922\n",
            "episode: 26200/35000, moves: 7, score: 4, epsilon: 0.1630498112668617, loss: 2.5319695472717285\n",
            "episode: 26400/35000, moves: 4, score: 7, epsilon: 0.162289330506825, loss: 5.127933979034424\n",
            "episode: 26600/35000, moves: 2, score: 9, epsilon: 0.16104691399595947, loss: 2.3650784492492676\n",
            "episode: 26800/35000, moves: 2, score: 9, epsilon: 0.15987315115718148, loss: 1.7120075225830078\n",
            "episode: 27000/35000, moves: 8, score: 3, epsilon: 0.15879128684204266, loss: 2.1890783309936523\n",
            "episode: 27200/35000, moves: 11, score: 0, epsilon: 0.1574559375349427, loss: 1.4106974601745605\n",
            "episode: 27400/35000, moves: 1, score: 10, epsilon: 0.1565312456420458, loss: 1.7815542221069336\n",
            "episode: 27600/35000, moves: 2, score: -11, epsilon: 0.15554819619407173, loss: 1.1535643339157104\n",
            "episode: 27800/35000, moves: 4, score: 7, epsilon: 0.15473061138436242, loss: 2.2803425788879395\n",
            "episode: 28000/35000, moves: 78, score: -67, epsilon: 0.15367663110146723, loss: 3.0671842098236084\n",
            "episode: 28200/35000, moves: 3, score: 8, epsilon: 0.1525718417662191, loss: 2.4043290615081787\n",
            "episode: 28400/35000, moves: 30, score: -19, epsilon: 0.1516834167154099, loss: 1.6761083602905273\n",
            "episode: 28600/35000, moves: 3, score: -12, epsilon: 0.15092915455781816, loss: 2.12532114982605\n",
            "episode: 28800/35000, moves: 3, score: 8, epsilon: 0.15003529054941503, loss: 1.9132308959960938\n",
            "episode: 29000/35000, moves: 1, score: 10, epsilon: 0.14886509857909497, loss: 1.85899817943573\n",
            "episode: 29200/35000, moves: 2, score: 9, epsilon: 0.14818559460487712, loss: 1.597426414489746\n",
            "episode: 29400/35000, moves: 2, score: 9, epsilon: 0.14738312549535587, loss: 0.7207599878311157\n",
            "episode: 29600/35000, moves: 5, score: 6, epsilon: 0.146634116319602, loss: 1.5690181255340576\n",
            "episode: 29800/35000, moves: 3, score: 8, epsilon: 0.14585244588888435, loss: 2.83492374420166\n",
            "episode: 30000/35000, moves: 1, score: -10, epsilon: 0.14505245742144446, loss: 1.800123929977417\n",
            "episode: 30200/35000, moves: 3, score: 8, epsilon: 0.14431240654522906, loss: 1.8869935274124146\n",
            "episode: 30400/35000, moves: 1, score: -10, epsilon: 0.1435617744429163, loss: 3.128735303878784\n",
            "episode: 30600/35000, moves: 1, score: 10, epsilon: 0.14264020495041085, loss: 2.9791834354400635\n",
            "episode: 30800/35000, moves: 2, score: -11, epsilon: 0.1419557512841619, loss: 2.3890867233276367\n",
            "episode: 31000/35000, moves: 2, score: 9, epsilon: 0.14127670108304421, loss: 1.9425110816955566\n",
            "episode: 31200/35000, moves: 6, score: 5, epsilon: 0.14066699726151907, loss: 3.1680028438568115\n",
            "episode: 31400/35000, moves: 6, score: 5, epsilon: 0.13999621179211238, loss: 3.616201877593994\n",
            "episode: 31600/35000, moves: 3, score: 8, epsilon: 0.13925967425919034, loss: 1.8668732643127441\n",
            "episode: 31800/35000, moves: 2, score: 9, epsilon: 0.13860114370724755, loss: 1.2295362949371338\n",
            "episode: 32000/35000, moves: 13, score: -2, epsilon: 0.13786160576750714, loss: 3.377840995788574\n",
            "episode: 32200/35000, moves: 1, score: -10, epsilon: 0.13716029967544055, loss: 2.4194159507751465\n",
            "episode: 32400/35000, moves: 8, score: 3, epsilon: 0.13647347861721765, loss: 2.3826584815979004\n",
            "episode: 32600/35000, moves: 15, score: -4, epsilon: 0.13558249625749258, loss: 1.8829500675201416\n",
            "episode: 32800/35000, moves: 2, score: 9, epsilon: 0.1347195577245124, loss: 2.01269268989563\n",
            "episode: 33000/35000, moves: 1, score: -10, epsilon: 0.13385541855991195, loss: 1.577794075012207\n",
            "episode: 33200/35000, moves: 19, score: -8, epsilon: 0.13312988596699868, loss: 0.7775616645812988\n",
            "episode: 33400/35000, moves: 2, score: 9, epsilon: 0.13226932987414786, loss: 1.7639992237091064\n",
            "episode: 33600/35000, moves: 8, score: 3, epsilon: 0.13145705315225972, loss: 3.466488838195801\n",
            "episode: 33800/35000, moves: 3, score: 8, epsilon: 0.13070726337683625, loss: 1.2127180099487305\n",
            "episode: 34000/35000, moves: 1, score: -10, epsilon: 0.12977279284787985, loss: 1.2779690027236938\n",
            "episode: 34200/35000, moves: 2, score: -11, epsilon: 0.1288823737282867, loss: 3.1674792766571045\n",
            "episode: 34400/35000, moves: 1, score: 10, epsilon: 0.12823123328271918, loss: 2.0437698364257812\n",
            "episode: 34600/35000, moves: 2, score: 9, epsilon: 0.12765293460212282, loss: 3.0531821250915527\n",
            "episode: 34800/35000, moves: 11, score: 0, epsilon: 0.12700419534511814, loss: 1.8241941928863525\n",
            "episode: 35000/35000, moves: 3, score: 8, epsilon: 0.12626970126773496, loss: 1.945786952972412\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "EkRbzxfmKGaK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Puis affichons les courbes de loss"
      ]
    },
    {
      "metadata": {
        "id": "YbVngKkTKKQ2",
        "colab_type": "code",
        "outputId": "4681d704-5673-477e-9007-2ff904fe49df",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 328
        }
      },
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "\n",
        "def smooth(vector, width=30):\n",
        "    return np.convolve(vector, [1/width]*width, mode='valid')\n",
        "  \n",
        "  \n",
        "sc = smooth(scores, width=500)\n",
        "\n",
        "\n",
        "fig, ax1 = plt.subplots()\n",
        "ax1.plot(sc)\n",
        "ax2 = ax1.twinx()\n",
        "ax2.plot(epsilons, color='r')\n",
        "ax1.set_ylabel('Score')\n",
        "ax2.set_ylabel('Epsilon', color='r')\n",
        "ax2.tick_params('y', colors='r')\n",
        "plt.title(\"Score, and Epsilon over training\")\n",
        "ax1.set_xlabel(\"Episodes\")\n",
        "plt.figure()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 0 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaYAAAEVCAYAAACrL0HYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsnXeYU8XXgN+bzfZdll1YOkgRR4oi\noAhIx469/6wIWFFBFMHCZ1cUBUURpKnYu6hYUBGlKUWpwtB7W2B32V6S+/1xk2yyqVuTkHmfJw/3\nzp2Ze5Is92TOnKLpuo5CoVAoFKGCKdgCKBQKhULhjFJMCoVCoQgplGJSKBQKRUihFJNCoVAoQgql\nmBQKhUIRUijFpFAoFIqQwhxsARTBRwjRFXgZaIrxY+UoMFpKuTioglUDQoitwDAp5cJy7e8CgzDe\nqzNfSykfrcR9ugHPSikvsM29VUr5XKWEDhJCiFjgeinlnAqOc7x3P/1+w/i7+qcKYioiAKWYIhwh\nhAZ8B9whpZxna7sKmCuEaC6lzA+qgDXL69WlPKSUywGfD+YwoDNwK1AhxRToe5dSDqykXIoIQykm\nRX2gMfCXvUFK+ZUQYrldKQkhxgB3AaXA98BDUkpdCPEAcDfGKktirEwybCuGY8C5wLPAt8AE4EIg\nBpgupXzBn2BCiHHAzRh/pxuBm6WUWUKIp2xyNwU6AUeAy6WUB2yrvzlANDCvsh+KEGIn8BZwPdAC\nmCalHCeEMAPTgN5AFLAWGAx0AWZKKU8uN8/pwFSgHlAIjJFS/iyE6Ae8CCwErgDigMFSyj88yHIt\n8KTtc9gP3AHEAouBBlLKUlu/b4CfgHfw8nnb3tds4CbgPCnlblt7Q+BroI4QYpGUsrcQQgces72/\n9kA34E0gEbACD0gpf7W9l5lSypP9fDc7Mb7PvcAy2/u/A0gDRkkpPxVCxGF8f+cAG4B/gEZSysFe\nvirFCYjaY1IcAVYAvwshhgohWgFIKfcCCCF6AcMwHjIdgV7ANUKI7sBooJ+U8lRgN8aDxs5AoJuU\n8nPgEYwH22lAB9v4S3wJZVMw9wFnAW0xHsT3OXW5FhgJtAEOA0Ns7VMxVkKnAEuBVhX9QJzogfEw\n7gAMF0J0wlgZtAJOtcm1wdbP03swAZ8Ab9o+o2HAx0KIZFuXzsBfUsp2GErwCQ9ztABmAFfY5pgH\nvC2l/A84iKEgEUIkAAOAL/H/eTeTUgq7UgKQUh4CHgWWSSl7O/XVbH0twHRggk2O8RgK2hPevhtn\n6gNWKeVptr72leswoAlwEobSut3LPRQnMEoxRThSSh04D+PX8ghguxBig82cB3AxME9KmSOlLAb6\nAV9h7M98IaU8bOs3EzjfaerfpJSFtuNLgbeklEVSyjyMX8RX4QMp5SqguZTyuJTSiqFkWjt1+VNK\nucsm/79AC9uv7bOAT219vgDyfNxmhBBiU7lXL6frc6SUFtt7XAT0BDIwHvpXAglSynFSyp+9zN8K\naIShnJBSrgR22WQEyJFSzrUd/4OxMivPecDvUsqttvOZQH/byu0L4DJb+4XAcillBv4/7+99fCbl\nce57BvCZ7XgRrt+HM27fjYc+ZoyVHbi+994Yf1elUspdVGHVqwhflClPgZQyG8NU9KTNpDMY+MS2\nQqiPYT6y97Wb99Kd24FMoIHT+TGn47rAJCGE3XwXCyz3JZNtBTDJZiYCw9zj/JDKdjq2YJjV0mzn\nx22y6kKILB+38bfH5PweMoFUKeVyIcT9wP3Ae0KI74B7vYxPB7JsD2jneRpgrHY8vQdPc2TaT6SU\n2bZ9wfoYiulr4EEMc6BdIfv7vJ3flz+c+94EPGBb8UUBmpcxgbwvi01plu+TWu6e+4DmFZBXcQKg\nFFOEI4RoBrS0e+DZTDovCSGuwzADHcF4CNr717MdHsLYN7FTz9bmif3AK1LKivxSH4lhKusqpcwV\nQjyPsW/hC/sDvA6QbTOlpfno74/6Tsdp2B6YUsovgC+EEGkY+zWjgV88jD8EpAkhNCfl5Otz8sQh\nnEyFQohUjP2dI1LKg0IIi5OJ8UFbt8p83j4RQjTFMCmeLaVcLYRoC2yurvmdOA4kOZ03roF7KEIc\nZcpTNAe+se3pACCEOAvDtLICw3HhMiFEqs189A3GQ3AecJWToroL72aXucAwIUSUEEITQjwhhLjQ\nj1wNgE02pXQShkkxydcAKWUBsAbDzAZwA4ZTQWW5Xghhsq0iewGLhBC325wykFIeAzYB3lL078TY\n6L8eQAjRE8O053O1WI5fgD5CCLvZ7G5gvt3hAWPV9BSwWkppd32vzOcNUILh/OBpJZSOYRbdZPs7\nuNP2nnx+J5VgOXC17XNvDlxUzfMrwgClmCIcKeUyjIfMVCGEtMX9TMKIZ9klpfwLw8NrNfAfxn7A\nxzYX4fEYD+tNGOajx73cZgrG3soGjAd5OwyPMoQQLwoh7vYwZhrQVwghgVeBUcBAIcRIP2/pHmCM\nEGIzhuPCfz76etpj+s3p+gaMB+UGYLKUcgPGQ7+rEGKLEGIjxn7TRE+T21ZJNwD32fpOBq51MmH5\nxeaEMgzDfX8T0AfjR4CdLzDMeJ85tXn9vP2wGMPxYL8Qorz5bQ3wA8YqaRlGiMFfgJsXYRWZhuG9\nuA3jfXyCd8WvOEHRVD0mRTCxrSK6SinfCLYszthdm0+EIONww9n0KYSYAJillA/6GaY4gVArJkWw\niQc+D7YQitBACHEZsEIIEWszEw7CWKEpIgjl/KAIKlLK3/z3UkQQ8zD2EzdiOHl8j2GuVEQQypSn\nUCgUipBCmfIUCoVCEVKEpSkvIyOnSsu81NQEMjPDIzdpOMkK4SVvOMkK4SVvOMkKkSNvenqyt6Do\nkCIiV0xms6dA9NAknGSF8JI3nGSF8JI3nGQFJW+oEZGKSaFQKBShi1JMCoVCoQgplGJSKBQKRUih\nFJNCoVAoQgqlmBQKhUIRUijFpFAoFIoyNK0jmrYNTbvPw7Vz0bTlaNoyNG1cTYmgFJNCoVAoDDQt\nEXgD8JYqbDJwNXAOcD6a1r4mxIgoxaQdzybx6XGwf7//zgqFImJZvvEQ63cc9d+xhljwz152HDge\njFsXYeQqdH9Ialpr4Bi6vgddt2KUQRlYE0KEZeaH1NSEygWYbV0PU16H7KOkv/9+9QtWQ6SnJwdb\nhAoRTvKGk6wQXvKGk6zgKu+gIMt+/QXt/Papkc9X10uBUjSPCSIaARlO54eBNtUvRJgqpkqnDmnT\ngdRTBObPPuPIY8+g16/vf0yQSU9PJiMjJ9hiBEw4yRtOskJ4yRtOsoKrvK99voa128pWSxPvO4e6\nSbEVnvNwVgHL1h/k0p4tMZkCywQ0ZPwCl/PZYwf4lbciVLMyq7H0RhFlykPTKBg8FIqLifsofFZM\nCoWiejiSVcCx44U++7RuXMflfNSbSyp1r7HTljF38Q4mfba6UuMB8gtLKz22BtiPsWqy0xRPJr9q\nILIUE1B03f8gIYH4ObPBYgm2OAqFopZ4+9sNPDJtGQ+/tdRnv28W73BrKyiqvILYf7TyyWE37sqs\n9NhqR9d3AnXQtJZomhm4BJhfE7eKOMWk10mBG28kavcuYn6rkc9UoYgYSkqtWEO8ptuOA8cZMn4B\nf/93yNFmtVZM5v92Vl5BZOYU8dEvmwPqm5IY43IeF1vLyVo1rSuathAYDIxA0xaiaaPQtCttPe4B\nPgYWAZ+i64G9sQoScYoJgOHDAYif+maQBVEowpfcghLuemUho95YHGxRvLJx5zGefW+lW/uwl39n\nyPgFfLNou0v7lK/WOY5nPtK/rP3rdVh1nbzCEg4ey6fUYvV4v6ISC5k5Rew86OpR9+uqvQEp8Oy8\nYpfzVz9ZXbuKX9dXoev90PWW6Hpb2/FEdP1r2/U/0fUettcrNSVGZCqmM86guHc/YpYsImrd2mBL\no1CEJfsycgE4nl/i8XpGVgFDxi/g4LHg1DkqLrEw4RPf+zvfLtnJ0eyyPadVm8uczkwmjTsvLQvT\nWbLuAPe/tojHpv/FnRMWelQY97z6Bw9NWcK3i3e6Xdu+z7f7d35h2eeYVqfM2WLr3myf405EIlMx\nAQX3GKumhDcmBlkShSI8yXFSSO/9tMnlmsWqM2baMgAem/5XjcuSW1Di8mAH2H04N6Cxo6cu5a4X\nf2XbfncFEBtdZkp75wfX9/jfjmOAYRbMyS/meH7Zamf11iNuc73wwSrGTFvq5nlnR+7JchyPuKaT\n4zhYij2YRKxiKh5wHqXtOhA792tMO903OxUKhXeefncFv63a6zj/Y7Wrc9asb9fXmizPzVnJA68v\n4r7XFrmY2F54f5Xj+KnbzwLgyj6tPc6x/0gez88p6//myD4AdD4l3et9J362ht2HcnjqneWMmLyY\nkZP9mzQzsozV2ZDxCxgyfgFFJWUOWLO+3+g4bpqe6DiOiY68x3TkvWM7JhP5941A03Xi35kZbGkU\nirDhv53H2HUwx+UXvjNHswv5zmnvplFagsv1VTKDIeMXsGhNYJ7G+zJyOXA0z+O1RWv2s31/mYnM\n+dhOUnw0LRomM3vsAC7t2dLR3u+MJl7vmRBXFuLZua33eMen3lnB3gzPsgXCk7OXu7VpgEnTGH7l\naQDEnODVaj0RuYoJKLrsSqz1042YprzK/3EpFJFEQZHvMIsv/9jmcl4+icCUrw0Hg3d+dDWNeWPc\nrOU8PuNvCovdXbbLz7HrYFnQafuWqQC8Ovwclz6zxw5g9tgB3HKB8Hi/YZe4Zl2476rT6HV644Bk\nLc/bD/f1ef1wZoHjuEcHI0Ro7M1dAOgq0nl1+Dk+FeOJSkQrJmJjKbh9GKbsLOI/eDfY0igUYYG3\nHG5b9horqL+c3LIBDjjF8VTUTVt3cjC4d+KffvvvtTlkWHXd4eIdbfb8mNM0jVlj+ru19+zY2K3f\nkIvbMWtMf2aN6c/ssQOY9pBvhQPQ6/TGRJujeOR/nf32BVi4eh8AaclxjrbU5Fg0z+mBTmgiWzEB\nBbffgTW5DgmvvgT5kbfJqFBUFLnHc0zPix/843fssJd/dzn/8e9dXvtarFaGvlTW/7r+J/ud/6//\nDjFk/AKGvfS7375gKB1nU+MLd3b32deuJGKio3hjZG+P/c442VjhpNeNB+DUk1K9phayk1tQgsWm\ntFOSYnz2jQQiXjHp9etTOHgopqwsYn/5KdjiKBQhyZa9WYx6czE5+cU0qJvgsU9sjOteyOQRnh/c\nzsjdnvepwH2/6LPft5KZU+RybmfENacDRsBvRXly8Fm8Pqofb47s7bYf5ouE2LJ9qDNFmZPEsEva\ncfP5p3Bx9xYu/d9+uB8dWqW57HOBYX584PVFjnNzVMQ/lsMziWt1U3jDTcRPeZ2ECS9SNOgyMKuP\nRaFwxr4aGuHkeTb+7h40qBvP6q1HmPzFWoqKLS6muqT4aMfxJ79t4YaBbWmWnuQwtwE0rV/mfVae\nI9nuOe0emrKEV4efQ2pyLD/9vdvR3rJcfjs7cTH+HQdiY6JoVomkqJqm8fI9PbBYdRqmJlBqsaJp\nEGUyMaBLM7f+0WYTD11/BmB4B9rdxhetVWV4yqNUM2BpewqFN92GebMk7tOPgi2OQhE0SkqtAe8D\nJdo819qdlOpoK2+qszN/xR6OZBc4lNLVfQ23bV856GZ895/H9oemLGH3IVclUich2mNfb+a26qJ+\nSjwNU41VljnKRJSp4o/UBf/sq26xwh6lmGzkj3wIPSaGhFfGQ1GR/wEKRZiyeO0BXvxgFU/NXs62\nfWVBpUXFFu56ZSGTvwwsG0pinKEMnINQy+O8Inrpw38dx3YPtF2HDEV1OKuAt75ex8RPV/PZAsNE\nd0G35l7nfeqdFS7n5R0ENGDWmP6VUhS1xelt6gHQ14fbeqQSut9aLWNt3oKCIXcStW8vcR9/EGxx\nFIoaY/YPG9myN5vdh3N53ikIdcInhuJYu+1oteVns7s+Axx1KjdRx5as1O7hN3baMlbKDNbvOMZP\ny3djsVpZuv4gAI/f0pXXH+jlGFvefXrsTV1czju0SmPW2AEh781mN/c5Byf7c5KIFJRicqLgnvvQ\nY2NJmPiy8tBTRBzOzgYzncxonkx7Y250dYEubzJ7e6xRcdu+qiqPvw3+O15e6Eh5FG02kZwQw5u2\ne/y7pSzdz+O3dOWU5nUBuLj7SQB0b9/Q59yhQvlM4ooylGJywtq4CQV33kvUwQNq1aSIaFZsOgwY\nbsye9o1iypnvyiug9NR4x3F8rG9nooKiUqJ8VHi1P8A9meXaNE1xHF/VtzXjbjuTnh0bufULRZo3\nSAq2CCGLUkzlyL9rOHpsLIkTX1bZIBQnHEXFgRXHtFh1LFarixuzM54eqs8OO5txt53JzDH9iXZK\no9Ojg+sKZvrofi7new7neg2CBUixlTX3V57cpGm0alwn5E14dkwmzcV1fOigdt47RxjKL7oceoMG\n5A9/gMSJE4j79CMKh9wRbJEUEYp9n8dUTQ/ao9mFjJ7quXqrpxLeEz9d49Y25cE+FJdaPZrivLl+\nd2iZ5uJ5Zh97UsNkdh3KYfyH/gNzAZ+rqnDlyj6tubxXK3ILS6iToEx7dkJmxSSEmCSEWCaEWCqE\nOCuYshQMvRs9Npb4t6eAteIBewpFVSm1WBk5eTHTv90Q8JhlGw7y+ULDo+3TBVscOesyc4pYsu6A\nR6WUXtdIf3Pfa+7pfjyV9Y6PNVd4b6Rj6zTHcYdWZce7DlU0bqjsuF6dWB69uYv3zmGEyaQppVSO\nkFgxCSH6Am2llD2EEO2A2UCPYMmjp6dTePV1xH/0PtGL/qCkr3s+LYWiJsktKCG3oITlGw9z9+X+\n++8/kueI++nfuSk/L98DGN5vH/+6xeu4jKxC7pwQWPqel++p3H9JZ7Peg9eW1Rlqd1Kqm/Jr0SCJ\nMTd1ITYmis27s2jVpCxwVtM0xt12JkezCznz1AaVkkURHoTKimkg8A2AlHIjkCqE8BzKXUsUXX8j\nAAmvvQK1WdpYocBzoTlPWKxWvvpzO0/M/NvR9sjUZY5jb0rpycFlRolSi+vf97lnumYtmHjfOcwc\n05/6KfFUFntGb+d9oit6t3Lpc/vFp/LUkG7Ex5oxaRqnnpTqFiPVqnEdpZQigJBYMQGNgFVO5xm2\nNt+1iGuQkh7nUHTu+cT+Op+YX3+m+LwLgyWKIgKZ85N0HG/alcmpTtkV7HirhBoI5ijv+zXrtx9z\nOa+bFOulZ9Vo26wuL9/Tg5JSK43reU9NpIg8QkUxlcfnLmdqagLmKhbPSk9P9t/pqf+DX+eT8sn7\ncOO1VbpfVQhI1hAinOQNVVmjzSZHQtI/1x2g95lGQtDqkrdTO+8u1fdc3YknZ5Stuip7z0DGhdLn\nH0qyBEK4yVsRQkUx7cdYIdlpAhzw1jkzs2rBr+mBJmxsexopPc4h5rvvyPr0K0oGnFel+1aGgGUN\nEcJJ3lCWtXG9BHbb0vUUFpZy4GA2jRrW4ciRXD8jA+PIkVxiok0Ul7g69yTGmdGsZS7lrw4/p1Kf\nUSh/tp6IFHnDRZmFyh7TfOAaACFEF2C/lDL4fyWaRu4LE9BNJpLGPao89BS1wtptR9l/pCyGbvXW\nI9w5YSGXPfxtwHME4lg9oHPZXtIk2z7S5BG9iXZyBU9NrhkznkLhi5BYMUkplwohVgkhlgJWYHiw\nZbJj6dCRomtvIO7Tj4j5bb7aa1LUKAeO5vHa5+7xQ3asuu4xrim9bhw3DGzLG1+uo3Pb+tx+cTse\neH0Rd1/egW7tGpJbUMKmXZkUlVgc2cDPO6s5Py03SkekOO0jNbBlbbAnGVUoahtND0OPs4yMnCoJ\nXdFlcNSG9aT170lJ1zPJ+uE314CKGiZSTAzBIBRl9efQcPtFp9K7UxOXvs3SE3lm6Nk1LltFCMXP\n1heRIm96enJYRCmHiikvpLF06EjRBRcRvWol0YvdAxEVitoirU6c47h+ShxpdWJDTikpFFVFKaYA\nyR8+EoD4d2cFWRJFJJNXWEKpxYpV1zmSXcix46p2mOLEIyT2mMKB0rO7U3pyW2Lm/4iWnYWeUjfY\nIilOQJqlJ7I3w3B8aN8ylf92umZGmDY38BRFCkW4olZMgaJpFN5wE1pREXGffBhsaRQnLBpxMVG8\ndHcPHr6hs9/e9lpECsWJhFJMFaDwptvQ4+OJnz4VSt2zMSsUVSUrt4jU5FjS6waW/mfznqwalkih\nqH2UYqoAer16FF53I1F7dhPz84/BFkdxglFSaiW3oIQDR8sCyB+7uSuAw8VboYgElGKqIAVD7kDX\nNBJffgEsgRVdUyjAKFG+53Au3kI0CoqNVXiMU9G8k5ulMHvsANo0DWpOY4WiVlGKqYJY2rWn6Orr\nMG/cQJzy0FN44NjxQmZ8t4Fjxwtd2j/8ZTNPzl7O4rWes22V2nLjdTkl3e1aH1vsUnnuu+q0Kkqr\nUIQeSjFVgrxxT2NNSyPpmXFox44GWxxFiPHhL5tZtuEQH8zf7NL++79GFdd3ftzkNuZ4XjHfL90J\nwF//HXK7nhDr6kB703mnMHRQO49KTKEId5RiqgTWxk3If+AhtIICEqa+GWxxFCHGv1uMWkrONZXe\n+8ldGdnZcziXkW8sZuHq/V77RJUrZT6wazPOOa1xFSVVKEITpZgqScGtt6NHRxPz7ddQXBxscRQh\nwiEvme//8KF0npy93O+8zvtOCsWJjvprryxJSRTedCvmHduJnz092NIoQoSVmw777dO0vmtRvCiT\na/qy4Vd2dBujOeVnfPGu7pWUTqEIAE2bhKYtQ9OWomlnlbs23HZtMZr2Wk2JoBRTFcgb+wTWunVJ\nePlFTAe8/yJWRA6tm6S4nG/bn+3Wp265UhIWq6uXnmjh2TX821cuY9aY/jRMTaiilAqFFzStL9AW\nXe8BDAUmO12rA4wGeqPrvYD2aFqN/EpSiqkK6Gn1yHv8KUy5OSS++GywxVEEGV3XWb3liEvb83NW\nkZXrms+usLgsOLvU4l7jKyk+2uP8mqa5rJwUihpgIPANALq+EUi1KSSAYtsrCU0zAwnAsZoQIizL\nXpSWWvSqllavNkpLoWtXWLsWVq2CLl2CLZFCoVB4w/cvG02bDsxD1+fazhcBQ9H1zbbzm4A3gALg\nE3T9oZoQMiyTuNZaafUAiX78aepefyWFTz9Hzsz3qm1eiJw6McGgumX1V0vJmdljBwBwNLuQ0VOX\n0qR+IkePFzL+rh6kJMZ4HBPJn21NEynyVqK0epkiM1ZOjwGnAMeBBWhaJ3Tde2XLSqJMedVASb8B\nlJx+BrHffYNpx/Zgi6MIAcbc6D8BK5SZ9VKTY5k6qq9XpaRQ1BL7gUZO500Ae0R4O2A7un4EXS8G\nFgFda0IIpZiqA02j4N770XSdhCmT/fdXhB1b92WzSma4tR87XsiWve6JVD1l/TY7xSJZbQ4P/9j2\npDbsqBFTvUJRUeYD1wCgaV2A/ei6fWm2E2iHptkzDJ8JbKkJIZRiqiaKLruS0tZtiPtoDlFbNvsf\noAgbNu7K5IX3VzHl63Vuee4efmspL37wj4sTw/N3nO3mpPD4rV2ZcE8Px/lKabiV17dVpL2gW/Oa\nEl+hCBxdXwqsQtOWYnjkDUfTBqNpV6Lrh4AJwO9o2mLgX3R9UU2IoRRTdWE2k/fE02ilpSRMeCHY\n0iiqkV9W7HEcFxaXJe7NzisLrF6yriz/XeN6RpzSGyN7A/Dk4LNo0ySFlKQyN/FoW8DsB78YP2Jy\n80tqQHKFohLo+lh0vSe63gtdX4Ouv4uuf2279ja63t127ZGaEkEppmqkeNCllHTqTNw3X6lVU4iy\nessR1m+vWH7Do07JWAuKyly99x7OdRy/95N0G5cYF83ssQM4qZH7hvMbX67j2PFCx3wx0SHiZapQ\nhABKMVUnmkbB8AcAVJXbEGXyl2uZ+FnFnIgappYV7csrLFNMOu6hFp3b1g943offWuo4Lh9kq1BE\nMkoxVTNFFw7CWrcucbNnELWtRvYFFdWA817R7B82Mt/JXFeejbsyHcf2DOEAJg/BriWl7gGzzrxy\nb0+P7W2aqHpLCoUdpZiqm7g4cp9+AVNeLgmvTwy2NAon1u8oM+GVWnQyMgt4/fM1LF57gE9+2+K1\ngJ/zKmmhk2LKzCly62uO8v1fKs3m7FCes9s39DlOoYgklGKqAYquvxFLk6bE/PwDWH3/glbUHhM/\nLTPhZWQVMOS5+azZVqasHnjd3cHoaHahW5s99uhwZoHbtbsu71Ap2aJV9nCFwoH631ATmEwUDzwf\nU2am2msKUb5ZvMOtLa+w1BFfZGf01KVu/ez7Qb+ucjf/xQbgxNCgbrxbm8qBp1CUoRRTDZH/0CNY\nk5JJ/L/HMB30XEpbETya1POcoftvD9Vjy5OTX8Kfa/ZTUGTx29cTI6/rVKlxCkWkoBRTDWFt0pS8\nx8ZhOp5N3Jx3gi1OxPNLOeeGeime93r+2+U/A8Nj0//iXQ/l0T1le/BE3SSVdkih8EXQk7gKIQYD\nzwLbbE2/SCmfD55E1UfR9TeS9OTjxH75GfkPjoZoz+UMFDXPx7+5eki+84PnUuenNAtMuTjzyr09\nMZtN1EkITOEEYu5TKCKZUFkxfSql7Gd7nRBKCUBPrkPh/24xqtzOejvY4igC4Nsl7ntP/kirExew\nUgJjP+npId0Ye5MqkaJQeCJUFNMJS96j47Am1yH+rTeguNj/AEW1k+/k7u2Po8fdXcBrguYNkmhp\nywghAjQBKhSRQtBNeTb6CiF+AqKBh6WU//rqnJqaQFULBVaiLkklb5QMtw+GyZNJn/sp3Htvxaeo\nLVmriVCTd+qXvjM9XNSzJT8u3ek49yb/me0asnKju3NEVd7vp89fTGyMmShTYF55ofbZ+iKcZAUl\nbyhRq4pJCDEMGFau+WPgKSnlPCFED2AOcJqveUKtUKA/tDsfoN7MmVifeZZjl1wD8e7uwt6IlAJm\nNUmWU667x2/tyvNzVjnOr+t/Mhee3YKU+Gg+se1DOcvfpkkdtu0/zqjrO9GxVT2GeFBMVX2/eQH2\nC8XP1hvhJCtEjrzhosxq1ZQnpZwppexe7vW6lHKe7foyIF0IcULtDusNGlBwxz1EHTpI/Dszgy1O\nxHHGyUb+uit7t6JNkxSmj+6QIdXAAAAgAElEQVRHo7QEHr3tLC48uwUA55/luezEtv3HAejYqh4A\n0x7qy1V9Wjty4p11aoOaFl+hiDiCvsckhHhECPE/23FHIENKWbkAkRAm/977sSYlk/DmJCiqnX0M\nhcEh2wo7Md7wijRHmXjhzu70PL2JS7/6Nhdye2qiLxZuozwx0VFc0rMlQwa1Y1CPk7jpvFNqUnSF\nIiIJumICPgLuFEL8AbwNDA2yPDWCnppG0VXXYjpyhLjPPwm2OBHFl38Y5e7Xb/cdo3TEln7ozzX7\nAfjhr11e+ybGRXN13zbUUaXQFYpqJ+jOD1LKvUD/YMtRG+QPf4DYr78g6YkxFPcfiLVps2CLdEKR\nk1/MmGnL6N6+If87ty3R5RxknGsp+eK9nyR9z2hKpzb1XHLpKRSK2iEUVkwRg7VVa/KefREtP5/E\nF54JtjgnHF8v2kFhsYWFq/dz1yt/uF2Pjq7Yn7u9wN8dl7SvFvkUCkVgKMVUyxT+72YsLVsR++3X\nkJvrf4DCIwVFpeQXupYjdy5J4YnGaYkBzd2nU2OX87Q6sV56KhSKmkApptpG0yi87n9oRUXEfflZ\nsKUJW4ZP+pP7XisrU+GrQF9qsqFYLj2npc85bxjYFoBkWxYHL+WZFApFDaMUUxAovGUwenQ08W9N\nhtLAsxIoIDu3iJFvLHZrX7Lecwb39duPOgr6Jcb53lJt09SoImuxGBrpO1vQbU5+ibchCoWiBlCK\nKQhYGzai6OrrMO/YTsyCX4ItTtig6zoPvrmE43nFLm0Auw+5m0X3ZeQy8bOyrA/+ah6ZTcZ/h1Kr\n1aWard3NXKFQ1A5KMQWJ/LuGA5D4wrNQ4F4JVeGOJ3OdvWhfmyZ13K6Nm7WcVo0Dj3S3pwX6deVe\n/tmc4WgvKjnhwuoUipBGKaYgYenQkYKbb8P833riPpoTbHHCllKLle37jzNr3kaP1+0FaR+92X8m\n76ioshXVlK/XO451q9psUihqE6WYgkj+mMeNvaYZ01Q2iEpSatF5bs5Kx3nbZiku13cdNPKJxcdU\nPmSvfau0So9VKBQVRymmIGJt2IjCWwZj3r6NuC8+DbY4IY/Vg5vc905ZwQHOOa2xWx9wXQ1545iX\nkheqsJ9CUbsoxRRk8h8YhR4dTcKkCVBY6H9ABGPxYFKbX65k+qbdmR7HJgXgwFB+taVQKIKDUkxB\nxtqkKQVD7yJq9y7iZ0ZulVurrnP3KwuZNne91z52N25fXHz2SR7bE+P8K6YYtTJSKEICpZhCgPxR\no7GmppL4yotEbfK8iX+ik51bTHGpleUbD3vt42nFVB5vWRpMARbiK48nbz+FQlGzKMUUAuh1U8l9\n+gW0/HwSXn812OIEhb82HHQcZ+d5LkFvsXrP7hBjNvHYzV1JiIt2ZHCoDM/fcbbL+ajrz6j0XAqF\nonIoxRQiFF1zPZZmzYn95ku0w95XDScqnzvVPsrJ96aYvK+YikutnGzbIxrQpSlDB7VzXBtzY+eA\n5WhczzWfXnxs0BPwKxQRh1JMoYLZTP7wEWgWCwmTI2vVtP+Ia3Hxg0fzPfab8d1/Ac1njjJxzmmN\nSU2O5axTGyBapFZKLnuOPYVCUbsoxRRCFN54C5YmTYmfMY2odWuDLU6FsOo63y7Zwd/rD3DwmGfF\n4o0nZv7tcv7WN+tZtHa/W7/ttjLnALPG+C/h9erwc7jnio4VksWZZ4Z2q/RYhUJReZRiCiXi48mZ\n+AaarpP84H1hld562fqDfLNoB8+9s5zHpv9FSWnV0vi888Mmflu11yV26dQWdQFo1bgOmqYxfXS/\nKt3DG/06N6VBanxAnnwKhcILmhaHpl2Opt2Opg1xvAJAGdBDjJIB51J08aXE/vAdMd9/C0NuDrZI\nAbFknWt27w9/2cLgi06t0pwf/rKZuJgoR9CsPZfdJT0Nl3BzVM38rrr1AlEj8yoUEcZPgBXY5dSm\nA7P9DVQrphAkb/Sj6HFxJI95MGyKCW7aneVy/ucad1OcN05rXc/rtVnzNlJcYkHXdTbsNIJn4zzE\nG7VokBTw/RQKRa0Qg64PQNdvd3oFtGIKWDEJIToKIa6wHdetrKQK/1g6dCT/nvswHTkCU6YEW5wa\np6jYqEk17rYzPV6/+9U/uO+1Px3nzs55Y27sTIeWqYwb7HmsQqEIGhvQNO+/On0QkGISQjyIsfx6\n2tY0TgjxRGVuqAiMgjvuRY+Lg0mTIL9izgTB4JKeLV3OL+zWIuCxm/dmA8bekTcKiiyOtEL2gn4A\nokUqD93QmSiTWvwrFNWCpk1C05ahaUvRtLPKXWuOpi1G05ajadP8zNQM2Gqb50/HKwAC/d/8P6A7\ncMx2Phq4JMCxikqg169Pwc23waFDxM+aHmxxXCgusfD+fOni5r25XI66oio6P3git8CoJFtTe0sK\nRcSjaX2Btuh6D2AoMLlcj1eBV9H1boAFTfP1C3Q8cAXwKDDO6eWXQP+H50gpHWH3tmPvYfiKaiH/\nobGQnEzC669iOnTQ/4BaYsE/+/j9n30ON2+rVXeseuz8/s++Grt/ZdMLKRQKvwwEvgFA1zcCqWia\nYaLQNBPQG/jWdn04ur7b60y6/geGnugKdAGKbW1+0fQAXJKFENOBfcCVwDPA9RjKalggN6luSkst\nutkcIQk333gDHngAhg2DGTOCLY1CoQhvfP+q07TpwDx0fa7tfBEwFF3fjKY1BBZheNt1ARah64/6\nmOsZ4HzbGA3oC3yFrr/oT8hA3cWHAyMwlNPNwGIgaLvymZlV23NJT08mIyOnmqSpWdLvuovSadMx\nz5xJ1nmDKOk/MKjyWKxW7nh5oeO8Y6s08gpL2HHA+Dw/ff5irn/8BwDq1Yljwr09fc6XX1jqcGyY\nPXaAy7Uh4xd4HFO+X2UJp78DCC95w0lWiBx509OTKzpEK3fcFHgd2AnMQ9MGoevzvIztD/RE1w3r\nmqaZgT+BalNMN0spXwFeCbC/orqIiSHnjanUvaA/yaPuJ/OPZeh1glc3aEE5E936HcdczhOcglKP\nHvdfX+pQFX9kKBSKamU/0MjpvAlgD1I8AuxC143Elpr2G9AB8KaYTA6lBKDrpWhaQFtAge4xXSWE\nUFXUgkTpaZ0oGD6CqH17SZg4IaiyrNzknmC2npdSE4GQlWtUjT2luXsEQgdV0lyhqG3mA9cAoGld\ngP3ourE00/VSYDuaZk/f3xWQPuZahaZ9i6Y9YHt9B6wIRIhAV0zxwE4hhAQcqZ+llH0CHO9ACNEX\n+BwYIqX83tbWCZiKERW8Vkp5T0XnPdHJe3gscR++R/ystykecC4lffoFRY4t5ZwcAI7aSpL3O6NJ\nhedbtMb4MbZ5T5bbtbqJMRWeT6FQVAFdX4qmrULTlmI4LgxH0wYD2ej618BI4F2bI8Q64Dsfs40E\nrgPOxni2v4/x7PdLoIrp2QD7+UQI0QYYBSwpd+k1YISUcoUQ4iMhxEVSyh+r454nDLGxHH9zOik3\nXkPyiHs5tuwfiIsLtlQuxNlKRJzdviF//3cIAF3X0TTv+62d29Zn9dYjHtMAXT+wLcdyiti4y3O5\ndIVCUQPo+thyLWucrm0Fevkcr2mtnc6W2152WgHb/YkQkClPSunm9mdrqygHgKsAx89uIUQM0EpK\naV/ifQecW4m5T3hKBpxLwR33ELVvL3Efzqn1+x84mufz+m+r9gJwy/llSmbznixGv7WU3Yc8b9QW\nlxom58R494SpSfHRjP5fZ849s5mjTXgw+SkUipDiN+BX27/lX78GMkFAKyYhRHm3v8lCiK+klH69\nK5yRUubb5nNurg84/yQ+DDT2NU9qagJVdRevhHdK0HCRddyj8OF7JI8bS3KfHtDTt9dbdVFSanXx\nkju7QyP+3uAaWzV+uPFD6qTmZfWPXvroXwCeemcF3716udu8Jtv32KB+ktfv5NpzBb+uNJSe3JNV\nrd9dOP0dQHjJG06ygpK32tD1VlWdIlBTXn+gpz3IVgjh1+1PCDEMKB/n9KSU8mc/9/IbPRlR7uLl\nZY2pQ/R7H5Ny7eVYr76GzAVL0OvXr3E5Xv10tcv5HZe049bzT2H4pLIMIymxhpLx9tl6an//x40A\nHDma53Wc2SnW7rFbulbbdxdOfwcQXvKGk6wQOfLWijLTtEfR9RfRNM9mHV2/1d8UgSomU7nMD6VC\nCJ9uf1LKmcDMAObOAJwT/TXFcFlUeKGkd1/yHh1H0vNPk/j8U+ROerPG77nByS38mn5tMGmaS9nx\n6aP7+c3IIHdnsnT9QW65QLilFWqQGu91nKZp1Ra7pFAoapx/bP/+VtkJAlVMq4QQ31JmHzyPAN3+\n/CGlLBFCbBJC9JJSLsbYg3qjOuY+kSkYPoK4Tz8i7tOPyH/kMayNK+4RV1ly8h2OmUx50HDMLK9o\nTmqUzK6Drr/o7Ga9Nk1T6NPJVd7kBFWUT6E4IdB1u1XsI6Ahur4XTTsd6AR8GcgUgcYxjbTdpBXQ\nEpiD4V1XIYQQg4QQC4ELgReFEPOd5n9RCLEE2CalDGiDLKIxmym453600lKS778HLNWfNNWZ+ill\nHoAFRaWO4/hYs8vKyc7YG7t4nWvpemNvapnTHlWMWSVmVShOMN4DuqNpTTEU0mnAu4EMrEgck1VK\n+SCAEOJuIBGoUBU7KeU8PEQJSyn/w0gOqKgAhTfdSsxP84j95WfiPvmQwpv8mm4rzXlnNefjX7cA\n8OeaAwy+qJ3P/jHR3hWN3eD3419l+R9jYyIk96FCETk0Rde/QNNGAVPR9YloWkCLjkB/ps7BNU1F\nAkawlCKYmEzkTngNPS6OxGf/D/OqarGuYtV1/li9z5GVoTL4il3SNNh1MAdzVFkfVU9JoTjhiLU9\nCK4Evre1BVRqOtCnQZqU0lGXQ0o5EVABJSGAtUlTcp9/GS0zk5Rrr0DLPOZ/kB/+3ZzBez9JJnz8\nr6Nt/vI9juPHbulapfk37c7i6XdXsPNg+HhBKRSKCrMQI2b1gC07+Uh8pzByEKhiihVCOGw3Qoiu\ngMoXEyIU3jKY/IfGYMrNIfGZ/4MASpn4IivXcG44cNRwy9+6L9uRkLVPpyac3LRiaRNPb1Op6soK\nhSKcMTJItEDXr7O1fIN7CJFHAt1jehCYa0vkasLIMntLReVU1Bz5Ix4i9tuvif9wDmgauRMr59i4\nNyOXUktZJEBhcSkvvL/Kcd6zYyNPwzxSPyUOXYcoP27k7U5K9XldoVCEIZrWHnjG9q8OrAWeBDb7\nG+pTMQkh6gBDpZSTgFOEEE9gJOXbDOzxNVZRy8TGkv3ld6TccDXxH7xHcf9zKb7UPdOCN3YcOM6z\n7610a/9ztWtI2cZdmR4zgXti/N090HWdWd9v9Nnv7ss7BCynQqEIG94FpgGPY/g89cbwV+jub6A/\nU97bQAMAIcQpGC7iD2KkRn+90uIqagRrw0Ycn/EuemwsyQ/dj2lv4L8dPCklgE8WbHU579auQcBz\nmjSNKJOJvn6yjicnKKuwQnECkoeuz0bXJbq+CV2fAbiXEfCAP8XUWkppL517DfC5lPI3KeV0XL30\nFCGC5eS25D8wClNWFkmPPVLt8yd5SLbqD9FCmeoUighkAZp2BZqWgKYloWmXAcvQNM1WNsMr/hST\nc5xSP8C51nVAlQgVtU/+w2Mp6dad2J/mEfPd3Gqdu3yGB4VCofDC/wFfATnAcQznh/8DLECJr4H+\nnjJmIUQDWx2lHhgmPIQQSRgBtopQRNPIeX0KemwsSY+NRjvuXtyvskRXc4aGCffUTnZ0hUJRS2ja\nQwDoejS6bgK6o+sm2/F7tmOfEfX+njLjgf8wKhU+K6XMFELEA4sxNrEUIYqlTVvyHxxN1KGDJD73\nVLXN68/DriJ0bJ1GvZTQKnaoUCiqzKBy5y85HbcMZAKfislWRbYx0EhK+bKtrQB4REo5JXA5FcEg\n/76RlJ4iiH93FtFLFnntt2z9Qa/XyuMro0NFSUuOrba5FApFyFD+IaH5uOYRv3YZKWWJlPJ4ubb5\n3vorQoiYGHJfmABA4vjnvHZbKQ/XlkQuiObKKUKhOAGpWoQ/gWd+UIQpJX36UdxvANF/LyN60R8e\n+/hLoFodmRvefrgfr9zbk4u6t3C0nd2+YZXnVSgUIY/u5dgrSjFFAHmjxgCQ+NyTUFrqdl04Bcx2\n8qCECopKmXBPT8bf5TcuzivRZhNpdeK4oldZ1WV/hQUVCkVY0hNN2+14lZ3vwXCi80ugKYkUYUxp\n9x4UXnEVcd98RcKkCeSPftRjvzsuaY9mgjXbjrq09+nUpNqcFKLNUbRvmUrzBgElGVYoFOGHqOoE\nSjFFCLkTXiNm8SISXn2JoiuvwXJyW8e1DTszAcOk17ltfU5qmMy0uRvYc9gIY8vJ9xlyUGEevqFz\ntc6nUChCCF3fVdUplCkvQtBT6pL7/EtoVivJdw+F4rLy6Cs3Gc4PRcUWNE2jcb1EsvPKri+tgNee\nQqFQVBWlmCKIoiuvofCqa4heu5rYb792u+6cJOTUFmX7To/cqFY4CoWi9lCKKcI4PvIRdE0j8akn\nMP/9l8u1aKd0Q86phyqTH0+hUCgqi1JMEcaTS44zq89gyMig7rWXEfPjPMe1JvXLskwp851CoQgW\nSjFFGHszcpnb9XKev/xRMJmoc/tNXH1gBQCN0hKCLJ1CoVAoxRSxrGh9FlmffA1mMzd9+hKtD2+v\n1nRDCoVCUVmUYopgSrv34Pis94m2lvLgj69BSZlb+ANXnw5AmyZ1giWeQqGIUJRiinCy+53H/A4D\naXl0N8kj7gWLBYAz2tbnzZG9GXNTlyBLqFAoIg2lmCKc4hILM/sNZUf9lsR98SlJT4xxXEuIi1aF\nARUKRa2jnjoRzOHMfH77Zx8FsQk8cc3TlLZuQ/ys6SSOGwt6lRMEKxQKRaVQKYkimLFvl8UxHU9I\nIfuLb0m54SoS3n4LrbjYKJkR5TvzuEKhUFQ3ta6YhBB9gc+BIVLK721tCzFKtefZuj0kpVxV27JF\nOtZmzcn6ah51r76E+HdmYl63luMz3sXatFmwRVMoFBFErSomIUQbYBSwxMPl26WU62tTnkjjUGa+\n3z56gwZkzfuF5PvvIfbH70m58Royf/gNEhP9jlUoFIrqoLb3mA4AVwHZtXzfgNiw45gjo/aJyO5D\ngb03vU4Kx9/9kIIbb8G88T9SBt+Iduyo/4EKhSL80bRJaNoyNG0pmnaWlz4vomkLa0qEWl0xSSnz\nAYTwWK7jGSFEfWAjMFJKWeBtntTUBMzmqu19pKcnO45z8ot5ec5KVm/JAOC7Vy+v0tzVjbOsnsjJ\nLybzeCEtGvmOOUo9mFOx+8yaDtnHiJk3j/o3XAkffgjt21dZ3lAinGSF8JI3nGQFJS8AmtYXaIuu\n90DT2gGzKV/cT9PaA32A6q2H40SNKSYhxDBgWLnmJ6WUP3vo/jqwVkq5TQgxFRgOvOJt7swATFK+\nSE9PJiOj7CH9wvur2LqvbBHnfC3YlJfVE0PGLwBgyoN9iI/1/pUeOZbn9Vr9lDjP95n1IUmPjCJ+\nzmysvXpxbNEK9AYNqiRvqBBOskJ4yRtOskLkyBuAMhsIfAOArm9E01LRtDro+nGnPq8CjwNPVViA\nANH0ILgFCyHeBb6wOz+Uu3YxcL2U8jZv40tLLXpVV0yKCjJuHDz3HKSnw48/QteuwZZIoVBUHN95\nxzRtOjAPXZ9rO18EDEXXN9vOBwONgE+Ad9H1fjUhZNDdxYUQGvALcI2UMgvoB/h0gqjOFZOu6wx9\n6XeX6zMe6UeUyfv226ZdmTRIjSetTvWUG/dFRVZMrz/Qi+SEGK/93vtpE3+s3g9Al1PS+WdzhuPa\n7LEDfAty7ygS8gpJnPQKes+e5I17moI774Vy+fXC6ZdnOMkK4SVvOMkKkSNvJcx/Zf/BNS0NuB04\nF2ha4ZtXgFp1fhBCDLK5hl8IvCiEmC+l1IHpwG9CiD+B5sCU2pIpr7DUre3PNQewellJ5uQX8/LH\n//LwW0trWrQK42/xa1dKXUU6XU9Jr9jkZjP5j/4f2e98iJ6URNK4R0m+ZygUeN0KVCgU4cd+jBWR\nnSYYTmsAA4B0YBHwNdAFTZtUE0LUtvPDPGCeh/bPgM9qUxY707/d4Nb2/s+S7fuzGTrIfaN/697Q\ncSjcceA4z7630nFusQZmlq1XJ46zOzRkxvf/VfiexYMuJbPrmaTccDVxX32Bef06ct6aQenpZ1R4\nLoVCEXLMB54G3kbTugD70XW7eekL4AsANK0lhinvwZoQIuJTEq3fccxj+5J1ngvlvfHVOsdxIHFB\nNYmzUgIoKbV47Vt+L9HkZIKLjanYfp21UWMyf/iVgluHYN4sqXt+PxKfe8olO7lCoQhDdH0psApN\nWwpMBoajaYPRtCtrU4yIV0y+yM4r9nn9o1+21JIkgbFKZni95ryayshyNb+lJsVW/GYJCeS+8hpZ\nn8/F2rQZCZMnUvfic2HjxorPpVAoQgddH4uu90TXe6Hra9D1d9H1r8v12VlTjg8QoYqpqMTCik2H\nKSm1+uz33ZIdPq9v2p1ZnWJVmc8XbvN6LTu3TMn+u+WIy7XrB5xc6XuW9O1P5sKlFN5wE9Fr/oUu\nXYifMRWsvj9bhUKh8EZEKSZd19mbkcu0L9cy9Zv1zHTaY2lcz72s+IJ/9vmcr9fpjatdxppC7ilT\novb3OuXBPjw5+Cw6nVy/SnPryXXImTyV7Hc+hKQkkh4fQ93LLyLqP/f9O4VCofBHRCmmZRsO8n+z\nlvPrit0ArNh02HHtsVu68vbD/So039L1nvehaotWjQOvLltqKTPlPXV7NwDiY82c1Kj6oseLB10K\n69ZRNOgyov9eRurAXiQ+9QTkB3cvTqFQhBcRpZiOHi9ya4uNjqJFwyQS46IxR/mOPStPhV2uq5mG\nqfEB9/37v0OO42hzDX7tjRpx/J0PyP7oc6xNm5Pw1mTS+pxN9J8La+6eCoXihCKiFNPhY+6/3ItK\nLI7kpprmrph2H/IexLbvSB4bd9XcPpOu6xzJ8h4nVGoJfB+nXi0EAztTfO4FHPvzL/LvG4lp315S\nrr2cpIdHQpH7jwOFQqFwJqIUU78u/oOV3xzZh9fu7+U49+WZt+tgDhM+/pfNe7KqRb7y/LpyL7c/\nO58Z33mON7J72t15WVm81WEviqxlY8Nkd9dlHapZSh8kJJD3f8+Q9e1PWMSpxM+ZTcr1V6IdD51Y\nMIVCEXpElGJK8JHg1NEnzkydxLK0Pl84ebrlFniO05m72Lf3XmX5+DfDHX3ZhoOs3XaU31bt5aDT\nqs++b9T55DKT4thpyzzOZffEswTBW670rLPJ/HkhRYMuI2bpYlJ7dSPmR7c4a4VCoQAiTDElxUdX\neIxzfaafl+/22KcmzXl2Xvt8DR/+spnHphvl0P/dksH2/cbKIyqAvbENtkDimd8HKc4oPp7jM94l\nb8zjmI4dJeW2/5Ew/jnlVq5QKNyIKMWUnBDDFb1bubWbPOwteWLesl3VLVKl+HPNft74cp0jz1+U\nKXCnjYp48lU7ZjP5D40h8+eFWFqcROLEl6l7yfmY160JnkwKhSLkiCjFBHDZOa149DbXoowX92jh\n1u/a/m3c2hLjgp6MHYCDR12dOMo7bXhLQAvQqU29GpGpIlg6dCTr+/kUXXoF0SuXU/eS84mfMlk5\nRigUCiACFRNAz9ObuJR56N6+kVuf/p0NR4nTWpc9yK/qayiroYPa1bCEBp6CfgF+8mJStGP1kcy1\nbnIl0g/VANZGjTk+a46RrTw+nqSnnyD13N5EbQ2tNE8KhaL2iUjFZGf22AHMGtOfJvUT3a7F2AoR\n2hOj6rrOMltArTmqdj62iriD337xqQGNS67EPltNUjzoUo4tXUXBLYMxy02k9u5G0sMj0TI9J9dV\nKBQnPhGtmMBz7BKAyaRhjtIotuXTe+3ztY7y6z/97XvFUl1kZBUG3Lf36U3oYgv4dc7yUJ7aKG5Y\nUfS0euS+OpnsWe9jad2G+DmzqXe6IPH5p1XWCIUiAol4xeSLaHMUxSWGYlq3/aijfc/hXHp2dDf/\nVSe7Dla8OuUhmyt5cYn38hcNKpAtorYpvvRyMn9dRO4TT2Otn07C66+S1rsbMb/+HGzRFApFLaIU\nkw8KikrZm5Hr1t4gNZ52J6W6tFXE7OaPnPxinn53RYXH7TuSB8AWH8UM4wOI5Qoq8fEUPPAgxxav\nIP+BUZgOHiDlxmtJvmMw5hV/B1s6hUJRCyjFFADli+z169yUs9s3dGmzr6yqg8rGGp1u87hLrxu6\nq6KASUwk74mnyPzlT0o6dyFu7lekDjqP5LuHouUcD7Z0CoWiBlGKKQDsKxE7vU5rhDnKxOyxA+jW\nrgHgXnyvKuw57G7GE83rEh/ru9JsS1umcE+VbGNjojipYfVlEq8tLO07kPXjArI++YqS0zoR99Xn\n1B10nkoKq1CcwCjFFADjP/jH5TwmukxBLN9olM54+t0VVTbn6brOV39uIyvXPT9fl1PSubxXa8f5\noB4n0bNjI54d2s3RZs8aXuyhAKLVqmOqQCBuSGEyUTLgXLJ+/p2CW27HvGkjda+5jOS7bsf891/g\nI25LoVCEH0oxBUB+UanLubdMC1/4qCAbCHsO5/L9UvfsEg9e14kBXZtyWus0R1uHlmkMu6Q9TdOT\nHG12F3dPZkWrVa9QhoiQxGwm99XXyfz+F0pPbkvc11+Seun5pPbsSuzHH4DFu9OHQqEIH5RiqgTO\nLuYN08qCYOev2OPW95PftrDSqSChL7wliT2tdT2iTCaXOkqeVj/2tqxc1wwKuq5jCecVUzlKu51N\n5h9/kfXp1xRedQ1Re/dQZ8S9pPbuRsz336r8ewpFmKMUUxVp4iU7A0BhcSnzV+zhrW/WBzRXUbHv\nX/zR5jITYnmHDIDFaw8A8OEvm13a7SmKThC9ZBAdTUn/geRMm82xv1dT8L+bidq2lZQhN5PW7Qzi\nZ0yFXHePSoVCEfooxTzWkr0AABedSURBVFRF7OUkPOGsO3zlr7OTke07oNafKS4lKcZj++FMwzFj\n0+6aqRsVbKxNmpL7+ltkLlhC4Q03YTq4n6THx1Cv2+kkvPgM2qFD/idRKBQhg1JM1cyz761wKCHn\nVc20uRv8jj2c6TvLgXM9KU/58C7o1sLj9W37IsO92tKhIzmTp3L0343kPTQGSktJnPQKad07k/jE\nGLQj3n9EKBSK0EEpJh8Ekt1h5LWdXM53HMghJ9/YK3LWHYHsMy34Z5/P6857RJ7WXzHRZV/nTqfM\nEcs3RtaKQU9PJ3/M4xxdI8l5cQIkJJAwfSr1zjiVpIdGYNq1M9giKhQKHyjF5INhl7R3OT+5aQpj\nb+ri0nZ6m3p0bJXm0qZphunui4Vb/d6j1GLlswVbOeIhDqp9y1RGXH+GS9v1A04mJTGG1k3c6yqZ\nTWVfp3MG8hPVhOeX+HgKh97F0ZXryH3mBSzNmhP//jukdetE3UHnwcSJmPbtDbaUCoWiHLWan0YI\nYQZmAW1s935YSrlYCNEJmIqxEFgrpbynNuUKlNH/6+ziGWcnPTUenKqra8CarUf4c80Bv3MuXX+Q\nn5bvZsl6974P39CZ9PRkMjLKVj8XdGvhYrJzxuxUyTYloWy/yR5fFWhBxBOO+HgK7r6PgiF3Ejv3\nK+I+nEP0X0thxd+kmUZTfP5FFNxxNyU9zgFziKdsUigigNpeMd0C5EkpewFDgYm29teAEVLKc4AU\nIcRFtSyXV+rVKatfZPZSwjzNQ42jN75c53PegqJSVm89Ql6hYfazm//s9O/StKKiumQV95Q9IhAH\njBOamBiKrr2B7G9+4Oi6LTB1KqWndSL2p3nUvfpS6nU8meThdxLzw/dQ7B7krFAoaofaVkwfAKNs\nxxlAPSFEDNBKSmnPWvodcG4ty+WVXqc3cRx7K5GRWk4xlVcydpydIV54fxWTv1jLXxvc938evuEM\nbjlfVFjWnIKyh2lCnHvdpYdvOMOtLVLR09Ph7rvJmr+QrLk/UnDbUPTYOOI+/4SUwTdSr+PJJI4b\nq0x9CkUQqFW7hZSyBLA/tUcCHwH1gUynboeBxr7mSU1NwGz2nTfOH+npgeWNu/GidsxdvMPnmJbN\nXN28M70opuwiC22bG1nJ7fn39hx2j7U5uWU90p0yOgQqa9eEWGANAKu3HnGM69u5GX/8u5cObRtQ\nvxYSvAYqbyiQ3qAOXHah8dJ1WLkSPv4Y08cfk/D2WyTMngF33gnjxkHDhv4nrGl5w+mzDSNZQckb\nStSYYhJCDAOGlWt+Ukr5sxBiONAFuBRIL9fH70ZIph+3an+U37fxx31XnUZqcqzXMdZi15RFr370\nj8d+O/dkUTfO/0duKSpx3KuisjpjH5dnW0llZeWjl5T6GlJlqiJvbeNR1panwqNPw0OPE//eLBIm\nvYJpyhT0GTMoPv8iigZdSnG/gej16oWGvCFKOMkKkSNvuCizGlNMUsqZwMzy7UKIoRgK6QopZYkQ\nIgNw/l/eFNhfU3JVBntlWG8kBKBsAMd+kj/iYiq/GmzVOJkdB1z/YC0254ewz5VXm8TEUHDHPRTc\ndBvxc2YT9+4sYr+fS+z3cwEoPqc3xedeQNEVV2Ft2izIwioUJxa1usckhGgN3A1cJaUsBId5b5MQ\nopet21XAT7UpV1UJVDF5y4VXHm97WYFwac9WAMQ6ZUC32AKqlGKqBAkJFNx9H5nL/uHYwmXkPv4k\nJd26E7NkEUlPP0Falw6kXHs5sZ98iJaREWxpFYoTgtr2jR2GsTr6QQjH5v75GPtNbwshTMDfUspf\na1muKuGsBHwRqGKqCo1tufuKnMqr2xWTN69CRQBoGpb2HSho34GCEQ9hOnSQmJ9/JO7Tj4j543di\n/vgdgJJOnSkeMJDiAedT2vVM5X6uUFSC2nZ+eAx4zMOl/4DetSlLdaJpGhd3P4kf/nIvWeFMXmHN\n7u8AHuOsykx5Kp66urA2bEThrbdTeOvtRG3bQswP84j5/Vei/15G9Jp/SZz0CtY6KZT07U9x/4EU\n9+qD9aSWRvS1QqHwiXpSVRPX9Gvjt489LVFNxhPFeFi9Waw6Gp5LZSiqjqVNWwruH0n2V99zVO4k\ne84nFAweil63LrHffUPyqPup160TaWd1Imn0g8T88L0qD69Q+EDZGaqRU5qlsHlvttfrdlNeRqb3\nMuxVVR3OK6aiYguxMVFYrDpRyoxXK+hJyRRfeDHFF14Muk7Utq1E//E7MYv+IHrRH8S/N4v492ah\nR0VRemY3YzXVfyClp58BUVULgVAoThSUYqpG0uvG+1RMYKyWMnOKvHeoov5wVkw5+cXExsRjsejK\njBcMNA3LyW2xnNyWwqF3Qmkp5lUrifn9V2L+WIB5xd9E/72MxPHPYU1Npbhvf0r6DaS43wCsTSqe\n+UOhqBY0bRLQHSNF3Ah0fYXTtf7Ai4AFkMAwdL3aK3Oqp1U1csO5benUxnd8y7CXfvdobjulWUq1\nyOCcD+/ocSPwt9RqVR55oYDZTOnZ3ckf+wRZPy7g6MbtZM98j4KbbkWPTyDum69IHjmceme0I7XP\n2SSOe5ToBb9CftXi9hSKgNG0vkBbdL0HRtq4yeV6TAeuQdfPAZKBC/+/vXuPjqO+Djj+nd2VLPmB\nLSNZli35IUu6fmA74AABGzC4oYTTGEoo5ZSACZCeEgiPkKZOc5oEmqQcnk2BUtzwaEiBpienPOoU\nE6BusM3DGPltX8vGL0mWZMBgbIwt7U7/+I2slbRybGt3NfLezzl7dva3szN3f5Z8NTO/ub9MhGGJ\nKY0GFeRx0ZndC6xed/GkTq+3N3W/vlAzZlja42kJKpZ/9nnb4ZF5Jjz8ouEcmvun7HvwYT6qXc9H\nS5az7yd3c3DOl4lu38bAxx5h2JWXUTxhNEUzvwjz5lHwb08QXbcW2jI/kMbkpDnA8wD4/gagCM9L\nnspgBr7fXqer6z2oaeOlmqI77Nra4n5vSxIZE2qffw5Ll8KiRbBsGaxZA3uT/qApKIBp02DGDPc4\n/XQQgQHdCwobk+TIp048bwGwEN9/IXj9BnA9vr+py3plwBvAmfj+h+kOsl9eY8p2SaJj9a8vrefN\ndU0UDy3gnhvPBuC6u1/vcf38vAhXXyg8vnADUytP5vYrOiYfPJ5Yk/f1xPwLDr9+Yv4Fx7Sd49Gf\nSruEPtZpZ7gHQCJByQf1fPrya8RqVxBbvYpYbS3eO+8cXt2PRolXTiA+cTKt079AfPIU93pcJWT5\nGmPo+7aLXIn3OEoSdU9knjcCV2z7W5lIStBPE1PYDQhKChXkH133HmpNcPYpI8mLRZg+oTiToZn+\nKhKBKVP4fMQYuOYbru3gQWIb1xOrfY/Yqlpim5SobiRWt4kBLz1/+KOJwUOIT5pMW1U1bVOnEa+q\nIV5V7QZY2KAY01kjkDx19yigY7I4d1rvf4Af4PuvZCoIS0wZcMms8ezdf4jLzq086s94nscZk9Jb\nvXpwYfepL8wJZMAA2qafStv0UzvafJ9I/U5iq1YS27SR6OY6l7Tee5e85W/Ds0mrFhYSHz+Btqpq\n4lVVxCdNoXX6qXYjcG57BbgTeAzPOw1oxPeTD83uBx7E9zNaNs4SUwYMHZTPzZdNPer1/9BIvuO1\n/0Arvu/jeW5aeJMDPI9ExRgOVYzhEHM72ltbiW7ZTGzdGqKb64huqSO6eTOx9zcTW7+20yYSQ4fR\nNnkK8ZqJtIkQr5lIXCaSGFFqCetE5/vL8LwVeN4yIAHchOddC3wCLAKuAarxvPaZI57B9xekOwxL\nTCGQ7ooMp1YXU1v3AT5w33Mr8f3UpYpMDsnLIz5xEvGJnUeI4vtEdjUSrdtEbO0aYqveI7ZmNXlv\nLSP/zaWdVk0UFRGvrCI+diyJirHEyyuIj68kPnYcibJRkJ+fxS9kMsb353dpWZW0nJXRNZaYQqC2\n7oOMbW/DdjcHY17UEpNJwfNIjBpNYtRoWs87v6P9wAF3GlA3EK1TYhs3Et20kdiqWvJWLO+2Gd/z\nSIwoJVFeTmJUOUyspqC4jERFBfEgiTFoUBa/mOnPLDGFwMABmf9nOHDQ7nsxx6CwkPjUacSnTuvc\n3tZGpGkX0Z07iOzYTnTrFqI7dxJpqCfaUE9s9Sq8Fe/CS+7uy2SJ4mJ3lDVuPPFxlSTKK0iMHk18\ndIVLXIMHYwxYYsqauTPH8eLSbZ3aqsqHsrn+E+bOGp/WfV15QRXPvb65U9sfKpVkzFGJxVxCKa+A\ns2Z2fz+RILK7hZM/aWHvmo0uge3ccfg5tmE9eStrU246XjbKjRisqaGtWlwCq65x+7JrWznFElOW\nXHpOJZeeU9npHqNbL5/G1l17mTS2KK37mvPF8m6J6QtVNgzdZEEkQqJ0JJxSzcGaad3fTySINDYQ\n3bGdSP1Ooo0NROrriW7fSnRzHflvLIY3Fnf6iD9wEPGKChKjRhMfXe6eyytIjBtPfNx4tz9LXCcU\nS0x9qCA/yinj0z8iL1XB1rLigWnfjzHHLBLpOOJKZd8+YlvqiG5Soju2E1u3lujW990QeN2Y8iN+\nYSHxMWNdwhpZRmLkSBKjK1xbRQXx0RWuUobpNywx9aFIFv/KO5CFSQqN6bXBg7vfm9Vu3z53hNXY\nQLR+J9FtW4m+v8Vd69q2tcfEBZAoLulIWqXBY2QZibJRJMrKYEo1UGBTj4SEJaY+5GUxMS1e2cg1\nF03M2v6MSbvBg4nXCPEaoTXV+/v3E2luItq0yyWrHduJ1ruBGZGGemLvb8Fbu7rHzRdHoySKS4LE\nVeqeR5S6R8mI4LV7prAwY1/TWGIyxpwoBg0iUTmBROUEOHtWylW8fZ8SaWoi0rQreDQRaaxn4J4P\naNtRT6SlmVid4q1eecRdJYaclJS8RpAYESSx0iCRBUnNHz7crn8dB0tMJ6jvXDGdB37dcV9cfp7d\nx2SMP3gI8aohxKuqO7UPLBnCx+1FUX0f79O9RJqbiTQ3EdndQqSlmUhLi3vd3OSWW5qIba478v7y\n8oKjruCIqyRIXiPLiFeMcacTR47EH1ZkCSyJJaYT1JTxwyk7eSC7PnSV2GfUlPRxRMb0E56Hf9JQ\n4icNJV5dc+R1W1td4mpPVs1NLok1NwfJrIlIczOxdWvxat/rcTP+wEHEy8uD5FXiEtiIUrecdDrR\nLy7OietglpiyLD8W4VBb2mci7sbzPH76zS+x59ODLHxzG3NnpvdeKWMMkJd3uHLGEfk+3sd7OpJX\nYwPRhvrgtGIjkYYGog07iW3SI28mEuHAzbfBz+9P45cIH0tMWXbn9Wfw/cfe4pbLU9zjkQFFQwbw\n9QslK/syxvTA8/CLhhMvGk5cjjAI6dAhIh/sDo62mons7lj2du8msruFeEX3WbJPNJaYsqy0aGBW\nJuwzxvRD+flHdQR2zNP99TN2RdwYY0yoWGIyxhgTKpaYjDHGhEpWrzGJSAx4HJgQ7Pu7qrpERBYD\ng4D9wap3qOqKbMZmjDEmHLI9+OFqYL+qzhKRKcCTwBnBe99Q1bU9f9QYY0wuyHZi+hXwbLC8G0h/\naW1jjDH9muf7fp/sWER+BsRV9e+CU3kfAcXABuA2VT3Q02fb2uJ+LHbi3/1sjDFp1i/qHmXsiElE\nbgBu6NL8I1VdJCI3AacBXw3afw6sVtUtIvIocBNwX0/b3rPns17FVlIyhN3tdbFCrj/FCv0r3v4U\nK/SvePtTrJA78ZaU9I87oLJ+xCQi1wN/Blyqqp+neP9i4M9VdV5WAzPGGBMK2R6VVwn8FXBee1IS\nEQ/4HXC5qn4MzAZsEIQxxuSobA9+uAE34OG3Iofrt10ILABeE5H9QAPw4yzHZYwxJiT6bPCDMcYY\nk4pVfjDGGBMqlpiMMcaEiiUmY4wxoWKJyRhjTKjk1ESBIvIg8CXAB25V1eV9FMds4D+BdUHTGuAe\n4GkgCuwCrlbVgyJyFXAbkAAWqOrjIpIHPAWMBeK4OoPvZyDOU4AXgAdV9WERqehtjCIyHXgU92+w\nWlVvzGC8TwEzgA+DVe5V1YVhiFdE7gHOwf0O/gOwnHD3bdd45xLCvhWRgcG+SoEC4O+BVYS0b3uI\n93JC2LfZlDNHTCJyHlCtqmcB1wP/1Mch/Z+qzg4e3wbuAh5R1XOAzcB1IjII+CHwR7j7u24XkeHA\nXwAfq+os4Ke4/yjSKtj3Q8BrSc3piPEfcX8UzASGishXMhgvwPeT+nlhGOIVkfOBU4KfxYuCfYS5\nb1PFCyHsW1w1mXdV9TzgCuABQty3PcQL4ezbrMmZxATMAZ4HUNUNQJGInNS3IXUyG3gxWH4J9wN4\nJrBcVT8JagcuBWbivst/Beu+GrSl20HgYqAxXTGKSD4wPulItX0bmYo3lTDE+3tc9ROAj3FTvswm\nvH2bKt5UxSr7PF5V/Q9VvSd4WQHUE+K+7SHeVEIRb7bkUmIaiato3m530NZXJovIiyKyRES+DAxS\n1YPBey1AGd1j7tauqgnAD34Y00ZV21IU0u1VjEHbnhTrZipegJtF5HUReU5EisMQr6rGVbV97rHr\ngd8S7r5NFW+cEPZtOxFZBjyDO/UV2r7tIV4Icd9mQy4lpq76sspuHXAncAkwDzd5YvL1vp5iO9b2\nTEpHjJmO+2lgvqpeAKwkdUWRPotXRC7B/Ud/cy9i6qk97X3bJd5Q962qno27DvarLtsOZd92iTfU\nfZsNuZSYGul8hDQKdyE061S1ITiE91V1C9CEO7VYGKwyGhdv15i7tQcXPz1VPZSF0Pf1JkZcf5+c\nYt2MUNXXVHVl8PJFYGpY4hWRPwZ+AHxFVT8h5H3bNd6w9q2IzAgG6RDEFwM+DWvf9hDvmjD2bTbl\nUmJ6BTfaBRE5DWhU1T6pcy8iV4nId4PlkbgROU8CXwtW+RrwMvA2cLqIDBORwbhzym/gvkv7Of+v\nAv+bpdBf7U2MqtoKbBSRWUH7ZcE2MkJEfhMUDoaO4sB9Hq+IDAXuBf5EVT8KmkPbt6niDWvfAucC\ndwQxlgKDCXHf9hDvYyHt26zJqVp5InI37gchAdykqqv6KI4huPPJw4B83Gm9WuCXuCGj23HDPltF\n5HLgr3Hnjh9S1X8XkSjwC6Aad9H/WlXdmeYYZwD3A+OAVlxx3atwQ1OPO0YRmQw8hvuj6G1V/U4G\n430ImA98BuwL4m3p63hF5C9xp2c2JTXPC/Yfxr5NFe+TuFN6YevbQtyp8QqgEPe79S69/N3KYN+m\nincf7vaRUPVtNuVUYjLGGBN+uXQqzxhjTD9gickYY0yoWGIyxhgTKpaYjDHGhIolJmOMMaGSU9XF\njUkmIuMABd7s8tZCVb23h888B9yhqg292G8V8KqqjjvebRhzIrPEZHLdblWdfbQrq+qVGYzFGIMl\nJmNSEpE23Nw45+Puxr9WVdeKyDZcpeYCYAHupsaBwF3B9ARn4m70bcXdCHmzqq4XkbOBf8EV3FyR\ntJ+ioL0EGArcr6rPBFNN3I27ybIAuEX7aP4wY7LNrjEZk1oUWBscTT2Km9Mn2TeBF1T1fFwpmPba\nZL8Ebg/aHwAeCdrvA/5GVefgaiO2+wnwclCw81zgLhEpwVWZfiDYzrX0s+rQxvSGHTGZXFciIou7\ntH0veF4UPC/FlYJJ9hvgKREZC/w38LSIDANKk45sFgPPBctTgSXB8uvALcHy+bgaaPOC163AeFzJ\nqp+JyBm4BNg+n5AxJzxLTCbXpbzGJCLQcUbBw52WO0xVfy9uKvc5uCOarwNdp69O/pyHq9EInSfZ\nOwh8S1Xf7fLZd0RkEXAh8EMReUdV//bov5Yx/ZedyjOmZxcEz7OA1clviMi3gXJVfQk3R9GZwfQV\nu4LrTOCuRb0VLK8Hzkpqb7cEN6U2IlIoIv8sIjERuROIquqvgVuTPmvMCc+OmEyuS3Uqb2vwfKqI\n3AgUAdd0WWcj8KyI7MUdAc0P2q8BHhCROG6W1/ajqO8BD4vIDlwl+XY/Bn4hIkuAAcACVW0TkTrg\ndyKyJ9j+j3r3NY3pP6y6uDEpiIgP5KlqW1/HYkyusVN5xhhjQsWOmIwxxoSKHTEZY4wJFUtMxhhj\nQsUSkzHGmFCxxGSMMSZULDEZY4wJlf8HlVecfoJB9aUAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 0 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "osmPih6jKalb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "On voit très clairement que notre réseau a appris à jouer de manière satisfaisante. Le score moyen en fin d’apprentissage est légèrement au dessus de 0, ce qui veut dire que l’agent arrive à la fin en moyenne en moins de 10 coups."
      ]
    },
    {
      "metadata": {
        "id": "dIApcYOKKeQZ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "\n",
        "\n",
        "from IPython import display\n",
        "import time\n",
        "g = Game(4, 4, 0.1, alea=True)\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "EqKvuUzvKe7Y",
        "colab_type": "code",
        "outputId": "bd25d568-d10f-45c5-f983-7d73d3a5cf0e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 155
        }
      },
      "cell_type": "code",
      "source": [
        "state = g.reset()\n",
        "state = g._get_state()\n",
        "print(\"state\")\n",
        "print(\"  \")\n",
        "g.print()\n",
        "done = False\n",
        "time.sleep(5)\n",
        "while not done:\n",
        "    time.sleep(1)\n",
        "    display.clear_output(wait=True)\n",
        "    print(trainer.model.predict(np.array(g._get_state())))\n",
        "    action = trainer.get_best_action(g._get_state(), rand=False)\n",
        "    print(Game.ACTION_NAMES[action])\n",
        "    next_state, reward, done, _ = g.move(action)\n",
        "    g.print()\n",
        "print(reward)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[3.7476544 5.4571967 6.7407804 6.145734 ]]\n",
            "DOWN \n",
            "....\n",
            "....\n",
            "o.¤.\n",
            "...x\n",
            "\n",
            "10\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}