7 Actor Critic models - A3C.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Asynchronous Advantage Actor Critic\n",
    "  \n",
    "## Overview\n",
    "\n",
    "In this tutorial we will see a few useful concepts:\n",
    "\n",
    "- A3C, one of the simplest and most effective Deep RL techniques, which basically made DQN obsolete.\n",
    "- Eager execution: operations are executed immediately as they are called from Python. This to make research and development more intuitive.\n",
    "- Model subclassing: we will subclass tf.keras.Model and custumize our forward pass. Because of eager executing the forward pass can be written imperatively.\n",
    "\n",
    "We will need to:\n",
    "\n",
    "- Create a master agent supervisor\n",
    "- Create worker agents\n",
    "- Implement A3C\n",
    "- Train \n",
    "\n",
    "Paper: Asynchronous Methods for Deep Reinforcement Learning by Volodymyr Mnih\n",
    "\n",
    "\n",
    "# Task: CartPole\n",
    "\n",
    "\n",
    "\n",
    "Cartpole is a simple game in which the player needs to balance pole connected to a cart with un-actuated joint. The cart can move left or right. At the beginning cart position, velocity, pole angle, and velocity are randomly initialized between +/-0.05.\n",
    "\n",
    "The agent can apply a force of +1 or -1 to the cart which will move it left or right). The goals is to balance the pole acting on the cart.\n",
    "\n",
    "At every timestep in which the pole is upright the agent receive a positive reward of +1. If the pole is more than 15 degrees from vertical or the cart moves more than 2.4 units from the center the episode ends.\n",
    "\n",
    "\n",
    "\n",
    "## Actor critic model\n",
    "\n",
    "### Policy gradients\n",
    "\n",
    "\n",
    "The idea behind policy gradients is to parametrise the action probability distribution given some input state.in DL a network do the job: takes the state of the game (using raw pixels, RAM or others) output probability distributions. Then it will sample from the distributions and decides what we should do.\n",
    "\n",
    "More formally, policy gradients are a special case of the more general score function gradient estimator. The general case is expressed in the form of Expectation(x | ) [f(x)]:the expectation of the reward function f, under the policy p.Using the log derivative trick, we demonstrate that we can search for our network’s parameters to find the action samples that maximises rewards, just like for normal Deep Learning problems: ∇ Ex[f(x)] = Ex[f(x) ∇ log p(x)]. This last equation is saying that bu moving θ in the direction of the gradient we can maximize the value of the expected reward function.\n",
    "\n",
    "As we said the Value functions is telling us how good a certain state is, basically corresponds to the loss function of a supervised learning problem. The value function, the Critic part of the algorithm, is used by the agent uses This is where the “Critic” part of the name becomes relevant. The agent uses the value estimate (the critic) to update the policy (the actor).\n",
    "\n",
    "Implementation\n",
    "Let’s first define what kind of model we’ll be using. The master agent will have the global network and each local worker agent will have a copy of this network in their own process. We will instantiate the model using Kera's model class, which can be find in tensorflow.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-01T10:54:41.702766Z",
     "start_time": "2019-05-01T10:54:40.360852Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"\"\n",
    "\n",
    "import threading\n",
    "import gym\n",
    "import multiprocessing\n",
    "import numpy as np\n",
    "from queue import Queue\n",
    "import argparse\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.python import keras\n",
    "from tensorflow.python.keras import layers\n",
    "\n",
    "tf.enable_eager_execution()\n",
    "\n",
    "verbose = False\n",
    "\n",
    "class ActorCriticModel(keras.Model):\n",
    "    def __init__(self, state_size, action_size):\n",
    "        super(ActorCriticModel, self).__init__()\n",
    "        self.state_size = state_size\n",
    "        self.action_size = action_size\n",
    "        self.dense1 = layers.Dense(100, activation='relu') \n",
    "        self.policy_logits = layers.Dense(action_size)\n",
    "        self.dense2 = layers.Dense(100, activation='relu')\n",
    "        self.values = layers.Dense(1)\n",
    "\n",
    "    def call(self, inputs):\n",
    "        # Forward pass\n",
    "        x = 0 # Complete here\n",
    "        logits = 0 # Complete here\n",
    "        v1 = 0 # Complete here\n",
    "        values = 0 # Complete here\n",
    "        return logits, values\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Agent \n",
    "\n",
    "\n",
    "To have a baseline we will create a random agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-01T10:54:41.843342Z",
     "start_time": "2019-05-01T10:54:41.704251Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class RandomAgent(object):\n",
    "    \"\"\"\n",
    "      env_name: OpenAi-s gym enviroment name.\n",
    "      max_eps: Maximum number of episodes the agent will run.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, env_name, max_eps):\n",
    "        self.env = gym.make(env_name)\n",
    "        self.max_episodes = max_eps\n",
    "        self.global_moving_average_reward = 0\n",
    "        self.res_queue = Queue()\n",
    "\n",
    "    def run(self):\n",
    "        reward_avg = 0\n",
    "        for episode in range(self.max_episodes):\n",
    "\n",
    "            done = False\n",
    "            self.env.reset()\n",
    "            reward_sum = 0.0\n",
    "            steps = 0\n",
    "            while not done:\n",
    "                # Sample randomly from the action space and step\n",
    "                _, reward, done, _ = self.env.step()  # Complete here\n",
    "                steps += 1\n",
    "                reward_sum += reward\n",
    "                # Record statistics\n",
    "                self.global_moving_average_reward = record(\n",
    "                    episode, reward_sum, 0, self.global_moving_average_reward,\n",
    "                    self.res_queue, 0, steps)\n",
    "\n",
    "                reward_avg += reward_sum\n",
    "\n",
    "        final_avg = reward_avg / float(self.max_episodes)\n",
    "        print(\"Average score across {} episodes: {}\".format(\n",
    "            self.max_episodes, final_avg))\n",
    "        return final_avg\n",
    "\n",
    "\n",
    "def record(episode, episode_reward, worker_idx, global_ep_reward, result_queue,\n",
    "           total_loss, num_steps):\n",
    "    \"\"\"Store score  statistics.\n",
    "    \"\"\"\n",
    "    if global_ep_reward == 0:\n",
    "        global_ep_reward = episode_reward\n",
    "    else:\n",
    "        global_ep_reward = global_ep_reward * 0.99 + episode_reward * 0.01\n",
    "        if verbose:\n",
    "            print(f\"Episode: {episode}:  \"\n",
    "                  f\"Episode Reward: {int(episode_reward)}:  \"\n",
    "                  f\"Steps: {num_steps}:  \"\n",
    "                  f\"Worker: {worker_idx}: \")\n",
    "    result_queue.put(global_ep_reward)\n",
    "    return global_ep_reward\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Master Agent \n",
    "\n",
    "The master agent is responsible to keep a shared optimizer that updates its global network. Each worker agent will update both the global network and hte optimiser. Ee’ll use the AdamOptimizer with a very small learning rate for the CartPole task.\n",
    "\n",
    "One of the disadvantages of A3C is that some agents might end up working with an outdated version of the network, but that is not a huge problem.\n",
    "\n",
    "The master agent will run the train function to instantiate and start each of the agents. The master agent handles the coordinating and supervision of each agent. Each of these agents will run asynchronously. The workers will be implemented with a threads to make it more easy to understan.. Let's pretend GIL is not a think in Python (Global Interpreter Lock: a single python process can't run threads in parallel and can't use multiple cores). But it can run them concurrently during I/O bound operations. \n",
    "\n",
    "We will also create a \"play\" method to allow us to use a trained instance,\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-01T11:24:27.832697Z",
     "start_time": "2019-05-01T11:24:27.646641Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class MasterAgent(object):\n",
    "    def __init__(self,\n",
    "                 env_name='CartPole-v0',\n",
    "                 save_dir='.',\n",
    "                 algorithm='A3C',\n",
    "                 lr=0.0001,\n",
    "                 max_eps=100):\n",
    "        self.game_name = env_name\n",
    "        self.save_dir = save_dir\n",
    "        self.algorithm = algorithm\n",
    "        self.max_eps = max_eps\n",
    "\n",
    "        env = gym.make(self.game_name)\n",
    "        self.state_size = 0 # Complete here\n",
    "        self.action_size = 0 # Complete here\n",
    "        self.opt = 0 # Complete here\n",
    "        print(self.state_size, self.action_size)\n",
    "\n",
    "        self.global_model = ActorCriticModel(\n",
    "            self.state_size, self.action_size)  # global network\n",
    "        self.global_model(\n",
    "            tf.convert_to_tensor(\n",
    "                np.random.random((1, self.state_size)), dtype=tf.float32))\n",
    "\n",
    "    def train(self):\n",
    "        if self.algorithm == 'random':\n",
    "            random_agent = RandomAgent(self.game_name, self.max_eps)\n",
    "            random_agent.run()\n",
    "            return\n",
    "\n",
    "        res_queue = Queue()\n",
    "\n",
    "        workers = [\n",
    "            Worker(\n",
    "                self.state_size,\n",
    "                self.action_size,\n",
    "                self.global_model,\n",
    "                self.opt,\n",
    "                res_queue,\n",
    "                i,\n",
    "                game_name=self.game_name,\n",
    "                save_dir=self.save_dir,\n",
    "                max_eps=self.max_eps)\n",
    "            for i in range(multiprocessing.cpu_count())\n",
    "        ]\n",
    "\n",
    "        for i, worker in enumerate(workers):\n",
    "            print(\"Starting worker {}\".format(i))\n",
    "            worker.start()\n",
    "\n",
    "        moving_average_rewards = []  # record episode reward to plot\n",
    "        while True:\n",
    "            reward = res_queue.get()\n",
    "            if reward is not None:\n",
    "                moving_average_rewards.append(reward)\n",
    "            else:\n",
    "                break\n",
    "        [w.join() for w in workers]\n",
    "\n",
    "        plt.plot(moving_average_rewards)\n",
    "        plt.ylabel('Moving avg episode reward')\n",
    "        plt.xlabel('Step')\n",
    "        plt.show()\n",
    "        \n",
    "        \n",
    "    def play(self):\n",
    "        env = gym.make(self.game_name).unwrapped\n",
    "        state = env.reset()\n",
    "        model = self.global_model\n",
    "        model_path = os.path.join(self.save_dir,\n",
    "                                  'model_{}.h5'.format(self.game_name))\n",
    "        print('Loading model {}'.format(model_path))\n",
    "        model.load_weights(model_path)\n",
    "        done = False\n",
    "        step_counter = 0\n",
    "        reward_sum = 0\n",
    "\n",
    "        while not done:\n",
    "            env.render(mode='rgb_array')\n",
    "            policy, value = model(\n",
    "                tf.convert_to_tensor(state[None, :], dtype=tf.float32))\n",
    "            policy = tf.nn.softmax(policy)\n",
    "            action = np.argmax(policy)\n",
    "            state, reward, done, _ = env.step(action)\n",
    "            reward_sum += reward\n",
    "            print(\"{}. Reward: {}, action: {}\".format(\n",
    "                step_counter, reward_sum, action))\n",
    "            step_counter += 1\n",
    "\n",
    "        env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Memory\n",
    "To keep track of the hsitory, we’ll implement a Memory class. We want tokeep track of our actions, rewards, states of each step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-01T10:54:42.121875Z",
     "start_time": "2019-05-01T10:54:42.110872Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Memory:\n",
    "    def __init__(self):\n",
    "        self.states = []\n",
    "        self.actions = []\n",
    "        self.rewards = []\n",
    "\n",
    "    def store(self, state, action, reward):\n",
    "        self.states.append(state)\n",
    "        self.actions.append(action)\n",
    "        self.rewards.append(reward)\n",
    "\n",
    "    def clear(self):\n",
    "        self.states = []\n",
    "        self.actions = []\n",
    "        self.rewards = []\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Worker Agent\n",
    "\n",
    "The most important part of the algorithm is the worker agent. To be asynchronous the worker agent inherits from the threading class, and we override the run method from Thread.\n",
    "\n",
    "\n",
    "\n",
    "### Running the algorithm\n",
    "\n",
    "We’ll run all the threads for a set maximum number of episodes. This is where the third Actor part happens. Our agent will “act” according to our policy function, becoming the actor while the action is judged by the “critic,” which is our value function.\n",
    "\n",
    "this might seem complicated but all it only does:\n",
    "\n",
    "- Get the action probability distribution based on the current frame\n",
    "- Act according to policy\n",
    "\n",
    "The the agent will update the global model with the local gradient when the agent has taken the set number of steps (args.update_freq) or the agent has died then. \n",
    "\n",
    "After this it will all start again. The worker agent will repeat the process of resetting the network parameters to all of those in the global network, and repeating the process of interacting with its environment, computing the loss, and then applying the gradients to the global network. \n",
    "\n",
    "\n",
    "## Computing the losses\n",
    "\n",
    "The worker agent calculates the losses to obtain gradients with respect to all of its own network parameters. This is where the last A of A3C come into play, the advantage. These are then applied to the global network. The losses are calculated as:\n",
    "\n",
    "- Value Loss: L=∑(R — V(s))²\n",
    "- Policy Loss: L = -log(𝝅(s)) * A(s)\n",
    "\n",
    "Where R is the discounted rewards, V our value function (of an input state), 𝛑 our policy function (of an input state as well), and A our advantage function. We use the discounted rewards to estimate our Q value since we don’t directly determine the Q value with A3C.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-01T10:54:43.394864Z",
     "start_time": "2019-05-01T10:54:43.041106Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Worker(threading.Thread):\n",
    "    # Set up global variables across different threads\n",
    "    global_episode = 0\n",
    "    # Moving average reward\n",
    "    global_moving_average_reward = 0\n",
    "    best_score = 0\n",
    "    save_lock = threading.Lock()\n",
    "\n",
    "    def __init__(self,\n",
    "                 state_size,\n",
    "                 action_size,\n",
    "                 global_model,\n",
    "                 opt,\n",
    "                 result_queue,\n",
    "                 idx,\n",
    "                 max_eps=500,\n",
    "                 game_name='CartPole-v0',\n",
    "                 save_dir='/tmp',\n",
    "                 update_freq=20,\n",
    "                 gamma=0.99):\n",
    "        super(Worker, self).__init__()\n",
    "        self.state_size = state_size\n",
    "        self.action_size = action_size\n",
    "        self.result_queue = result_queue\n",
    "        self.global_model = global_model\n",
    "        self.opt = opt\n",
    "        self.local_model = ActorCriticModel(self.state_size, self.action_size)\n",
    "        self.worker_idx = idx\n",
    "        self.max_eps = max_eps\n",
    "        self.game_name = game_name\n",
    "        self.env = gym.make(self.game_name).unwrapped\n",
    "        self.save_dir = save_dir\n",
    "        self.update_freq = update_freq\n",
    "        self.gamma = gamma\n",
    "        self.ep_loss = 0.0\n",
    "\n",
    "    def run(self):\n",
    "        total_step = 1\n",
    "        mem = Memory()\n",
    "        while Worker.global_episode < self.max_eps:\n",
    "            current_state = self.env.reset()\n",
    "            mem.clear()\n",
    "            ep_reward = 0.\n",
    "            ep_steps = 0\n",
    "            self.ep_loss = 0\n",
    "\n",
    "            time_count = 0\n",
    "            done = False\n",
    "            while not done:\n",
    "                logits, _ = self.local_model(\n",
    "                    tf.convert_to_tensor(\n",
    "                        current_state[None, :], dtype=tf.float32))\n",
    "                probs = tf.nn.softmax(logits)\n",
    "\n",
    "                action = np.random.choice(self.action_size, p=probs.numpy()[0])\n",
    "                new_state, reward, done, _ = self.env.step(action)\n",
    "                if done:\n",
    "                    reward = -1\n",
    "                ep_reward += reward\n",
    "                mem.store(current_state, action, reward)\n",
    "\n",
    "                if time_count == self.update_freq or done:\n",
    "                    # Calculate gradient wrt to local model. We do so by tracking the\n",
    "                    # variables involved in computing the loss by using tf.GradientTape\n",
    "                    with tf.GradientTape() as tape:\n",
    "                        total_loss = self.compute_loss(done, new_state, mem,\n",
    "                                                       self.gamma)\n",
    "                    self.ep_loss += total_loss\n",
    "                    # Calculate local gradients\n",
    "                    grads = tape.gradient(total_loss,\n",
    "                                          self.local_model.trainable_weights)\n",
    "                    # Push local gradients to global model\n",
    "                    self.opt.apply_gradients(\n",
    "                        zip(grads, self.global_model.trainable_weights))\n",
    "                    # Update local model with new weights\n",
    "                    self.local_model.set_weights(\n",
    "                        self.global_model.get_weights())\n",
    "\n",
    "                    mem.clear()\n",
    "                    time_count = 0\n",
    "\n",
    "                    if done:  # done and print information\n",
    "                        Worker.global_moving_average_reward = \\\n",
    "                          record(Worker.global_episode, ep_reward, self.worker_idx,\n",
    "                                 Worker.global_moving_average_reward, self.result_queue,\n",
    "                                 self.ep_loss, ep_steps)\n",
    "                        # We must use a lock to save our model and to print to prevent data races.\n",
    "                        if ep_reward > Worker.best_score:\n",
    "                            with Worker.save_lock:\n",
    "                                print(\"Saving best model to {}, \"\n",
    "                                      \"episode score: {}\".format(\n",
    "                                          self.save_dir, ep_reward))\n",
    "                                self.global_model.save_weights(\n",
    "                                    os.path.join(\n",
    "                                        self.save_dir,\n",
    "                                        'model_{}.h5'.format(self.game_name)))\n",
    "                                Worker.best_score = ep_reward\n",
    "                        Worker.global_episode += 1\n",
    "                ep_steps += 1\n",
    "\n",
    "                time_count += 1\n",
    "                current_state = new_state\n",
    "                total_step += 1\n",
    "        self.result_queue.put(None)\n",
    "\n",
    "    def compute_loss(self, done, new_state, memory, gamma=0.99):\n",
    "        if done:\n",
    "            reward_sum = 0.  # terminal\n",
    "        else:\n",
    "            reward_sum = self.local_model(\n",
    "                tf.convert_to_tensor(new_state[None, :],\n",
    "                                     dtype=tf.float32))[-1].numpy()[0]\n",
    "\n",
    "        # Get discounted rewards\n",
    "        discounted_rewards = []\n",
    "        for reward in memory.rewards[::-1]:  # reverse buffer r\n",
    "            reward_sum = 0 # Complete here\n",
    "            discounted_rewards.append(reward_sum)\n",
    "        discounted_rewards.reverse()\n",
    "\n",
    "        logits, values = self.local_model(\n",
    "            tf.convert_to_tensor(np.vstack(memory.states), dtype=tf.float32))\n",
    "        # Get our advantages\n",
    "        advantage = tf.convert_to_tensor(\n",
    "            np.array(discounted_rewards)[:, None], dtype=tf.float32) - values\n",
    "        # Value loss\n",
    "        value_loss = advantage**2\n",
    "\n",
    "        # Calculate our policy loss\n",
    "        policy = 0 # Complete here\n",
    "        entropy = tf.nn.softmax_cross_entropy_with_logits_v2(\n",
    "            labels=policy, logits=logits)\n",
    "\n",
    "        policy_loss = tf.nn.sparse_softmax_cross_entropy_with_logits(\n",
    "            labels=memory.actions, logits=logits)\n",
    "        policy_loss *= tf.stop_gradient(advantage)\n",
    "        policy_loss -= 0.01 * entropy\n",
    "        total_loss = tf.reduce_mean((0.5 * value_loss + policy_loss))\n",
    "        return total_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training A3C"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-01T10:55:23.521468Z",
     "start_time": "2019-05-01T10:55:08.418761Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4 2\n",
      "Starting worker 0\n",
      "Starting worker 1\n",
      "Starting worker 2\n",
      "Starting worker 3\n",
      "Starting worker 4\n",
      "Starting worker 5\n",
      "Starting worker 6\n",
      "Starting worker 7\n",
      "WARNING:tensorflow:From /Users/leonardodemarchi/venv3/lib/python3.6/site-packages/tensorflow/python/distribute/distribute_lib.py:1595: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "Saving best model to ., episode score: 15.0\n",
      "Saving best model to ., episode score: 16.0\n",
      "Saving best model to ., episode score: 10.0\n",
      "Saving best model to ., episode score: 11.0\n",
      "Saving best model to ., episode score: 14.0\n",
      "Saving best model to ., episode score: 13.0\n",
      "Saving best model to ., episode score: 12.0\n",
      "Saving best model to ., episode score: 34.0\n",
      "Saving best model to ., episode score: 48.0\n",
      "Saving best model to ., episode score: 51.0\n",
      "Saving best model to ., episode score: 86.0\n",
      "Saving best model to ., episode score: 118.0\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4W+X1wPHvkffedmI7jrP33mEmIUBZbYEOKKv8Wjpo\ngZZOCrSFUlpoKbSMAoXSlhJG2RsSsgMJ2dPZiTM94r1kyX5/f9xrxY6X7FiWbZ3P8/ixdIf03kS+\nR+86rxhjUEopFbgc/i6AUkop/9JAoJRSAU4DgVJKBTgNBEopFeA0ECilVIDTQKCUUgFOA4FSSgU4\nDQRKKRXgNBAopVSAC/Z3AbyRnJxssrOz/V0MpZTqVdatW1dojElp77heEQiys7NZu3atv4uhlFK9\niogc9OY4bRpSSqkA57NAICIDRGSxiGwXkW0icusp+28XESMiyb4qg1JKqfb5smnIDdxujFkvIjHA\nOhH52BizXUQGAOcDuT58f6WUUl7wWY3AGHPMGLPeflwO7AAy7N1/AX4GaA5spZTys27pIxCRbGAS\nsFpEvggcMcZs6o73Vkop1TafjxoSkWjgVeA2rOaiO7Cahdo77ybgJoCsrCxfFlEppQKaT2sEIhKC\nFQT+a4x5DRgCDAI2icgBIBNYLyL9Tj3XGPOUMWaqMWZqSkq7w2CVUkp1ks9qBCIiwDPADmPMQwDG\nmC1AaqNjDgBTjTGFviqHUsr39uRXcLSkmrOH65e23siXNYIzgGuBuSKy0f65yIfvp5Tykyv/vorr\nnl1DjavO30VRneCzGoExZgUg7RyT7av3V0p1HWMMd725lf5xEXz/3CFYFf6TSqpcAHx+oIizhmmt\noLfRmcVK9SIr9xRy37vbcdfVd+v7LtyRz/Of5fLghzvZfLi02f6hqdEALN+trby9kQYCpfyspKqW\nNzcewZi2p9XUuOr4xj9W8/Ty/azZX9RNpbMs21Xgebz9WFmz/dW1VpPQ6n0nuq1MqutoIFDKz55e\nvo9bX9zIW5uONtvnqqvH6bZusicqaz3bP9qe123lA8gvr2FYajSRoUHsPF7ebH+RXbatR8sor3F1\na9nU6dNAoJSfhQRZf4Yvrz3UbN9VT33GjN8vAqCo4mQg+Ow0vnkXlDv5zVvb2FtQ4fU5+eVO0mLD\nGZ4Www67RvDC6lzueXs71bV1VLvqOHNoMnX1hqufXs1fF+1m65HmTUiqZ9JAoJSfOd1We//O481v\nzGsPFlNS5aKkqpaiKisQnDk0mV155VQ43Z16v8U783lu1QGufvozr8/JL3OSGhPG+Mw4Nh8upaiy\nljte38KzK/dz4EQlAPNGWSPDtxwp5aGPd3HJ31Z0qnyq+2kgUMrPKu0bemGFk5tfWM+La3J5YXWu\n55s3wKq9JyiqdALWDbfewIpOdsyW11jvl1fmpLSq/WYcYwwFFU5SYsKYlBVPtauOyfd+7Nn/xoYj\nAGTER3Dm0KbJhA8UVnaqjKp7aSBQys8af7N/d/Mx7nlnO3/8IIcvPLLcs/31DUcoqrRu2heP78/g\n5Cge+nhnp96vcRv+6v3tNzGVVbupddeTEhPG1IGJzfY/uWwfAFlJkfz7xulkJkQAEBIkXPzX5Ww6\nVNKpcvYF7Q0A6Ck0ECjlZ5VON8NSo/n1paMBqKqto7TaulkPTIrkprMH8/H2PO59ZzsAyVFhXD0j\ni115FRSUOzv8fuU1bkKChLBgB5/ta3/00fGyGgDSYsMZkBjJ69+fTXpcOADxkSGe4wYnR+NwCK9+\nbzavfHcWT103lcraOhbvzO9wGXubwgpnsz6RCqebafct5KXPe362fQ0ESvlZpbOOmPBgvnnGIK6c\nktlk34NXTuAbM5omXXQ4hElZ8QCszy3u8PuVVbtIigpjysAEPt13gryyGg6eaL0J53BxFYDnm/6k\nrARG9o8F4NqZAz3HhQZbt5O02HCmZScyZ0Qqg5Oj2HGsrNd8M+6s7z2/jkv+tqJJbWvHsTIKK2q5\n//0cr5rg/EkDgVJ+Vu50ExVmTfL/3rlD+NLEdGYPSQJgRFoMA5OiuGXeMAYmRfLVqVagGJMeR2iw\no1Ojh8pr3MSEBzN9UCI5x8u49G8rOOfBJVTVttz5fLi4GoDMhEjPtm+dNQiASyekM7JfTJOA0FhG\nQgQfbsvjscV7OlzO3uRQkfVv9EmOVfupqze8/Lk1CqykysWTy/b6rWze0ECglB898EEOmw6VEG0H\ngiEp0Tz89Un85IIR/Hj+cOLsppcfzx/O0p/O4YErJwAQHhLE2cNS+OfKA3y07TgA9fWG3BNV1Ne3\n/e273OkiNiKEcRlxGGMNDQV4bf2RFo8/VFRFeIiD5OhQz7bZQ5LZf/9FDE+L4YPbzubeL41t8dwL\nxliJhf+xYr+3/yS9UnayFSRfXGPd/JfuyueVdYcBmDEokZV7evaMaw0ESp2itNrF9c+u4alTvsXV\n1xuOl9Z06Xs9vsR6jyBH09w9k7MSuGXesDbP/dq0AQDc/vImqmvr+Mn/NnH2g4t5c1PLN/QGDTWC\nsRlxTbav2tvyzepwcTWZCZHN8gud+rwl18wcyB0XjaSkykXuiap2j++tGnItfbrvBJ/uPeG51gvG\npDFhQDw7jpdT6247LcjinHyue3YNde0Ecl/QQKDUKV76PJeluwp44ANrVM6+ggp+89Y2vvjYSmbe\nv6jDN7StR0r51r/WejqAW3KsEwFm/ug0Fnx7JuVONx9sO8a2I9Zw09VtdAC/uCaXzYdLiQ4LJjUm\nrMm+NfuLmrXlG2PYerSUoSnRHS5fg4vHpxMe4uD+93d0+jV6uqLKWi6bkE5WYiS/eG0zOcfLCQkS\nnvjGFMZlxFHrrufG5z5nSwt5mhqs2FPIsl0FbfbX+IoGAqVOcbTEuim76w0/eGE9c/+8lOdWHWCL\nPSpk7cGO5fm56unPWLgjr8UUErHhVpNQblHnvi3PGJRIYlQoy3YVcqzUaqfe2Gi45u0vb+KxxXs4\nVFSFq66eX7y2BbA6MkWEadkJAJw3Ko3Citpms433FVZyuLiaM4c1nR/QERnxEdwwexAfbc/jREXH\nRzn1BFsOl/K7d7Z70n2AFSQf/DCH97Yco7iqloyECP54xXhyi6p48fNDpESH4XAI4+ya14o9hVz6\n6IpWm+4K7X+bbUeb53LyNQ0ESp2ioMJJTHgwQ1KieGfzsWb7Nx0qIed4GcWNcv+0ptLp9kzgemPD\nEfYWVFBSdfK8xCir3f2hr07oVFkdDuGMocm8u+UYZTVuIkKC2JVXzpGSasprXLy6/jAPfriTsx5Y\nzGOL9xBsN0GdNzoNgIe/PokvTkznR/OtZqjzHlrG0ZJqz+uvP2iNSppld1531mUT0qmrNyzc0b05\nkrrK858d5B8r9vPIwt2ebcfLanhs8V6+/9/1uOoMSVGhzBqSxJ0XW8OA8+y+l4FJkU1eK+eUXE11\n9YZVewo9Q4FbSurnaxoIlDpFQbmTUf1jWXT7uYy2h0lOGZhAQmQI0wclsnxPIRc+vJyvP9V+ioYj\n9k11dP9Y1h0sZt6flzLxno+Z++clfLbvBJW1dVw1fcBp5fC/8YxsT/vzLfOGISI8sWQPi3cWNDnu\n4YW7cdcbfvelsfz8gpGA9W39ka9P8lwnwJsbT9ZcGtq+T21G6qhR/WNIiAxh3cGOD3f1J6e7jg+2\nHsNVb/37Ltpxck7EjlNu2AmRVlC/YnIGgKet/9S+lFNHej2xZA9X/2M1q/Za27VGoJSffZKTx5r9\nRaTYN76/XzOFW+YO5eXvzGLD3eczaUA8+wqsNtydeeUcaqdJp2EM/vfnDGmyfV9BJW9tOkp5jYuY\n8JCWTvXapKwEMuIbxvjHc+3MgTz/WS63LNhAsEM4a1gy3zrTGu45OSueKyZn4nA07/hdc8c8shIj\nm0wAK6muJcghnlFNnSUiTBgQz6ZDvSsR3ftbjvPd59ez0M72ujOvnBMVTl5ck8uNz61tcmzD3I74\nyFDuvHgUL90007PvhW/N4I9XjCMrMZKHF+7igQ9yqHS62ZBb7AkADbb7IRD4bIUypXqjhj/uhrb7\nrKRIfnz+CM/+CQOsP/aYsGDKnW42HS5hQGJk8xeyNYzBnz7oZGqGB64cz9ubjvLC6lzPa52uBd+e\nycMLdzEhM56sxEieW3UAgCevncK8UWm46+q5bGI6o/vHEhzU8ve/1NhwLhnfnyeX7aO02kVcRAil\n1S7iI0K8GiHUnokD4lm6azelVS7PsNiebne+1YxTVuMmNjyYsho3U3630LN/fGYcv//yOBwiDG7U\nof6tswY3eZ3Zdg6mdQeLeXntYR5fspdtR8tYuqtprS06LJjCCif5ZTWkxob76rKa0RqBUi1obZjo\ntOxEkqJC+fNXJxDskHa/vR0pqSY02EFKdBjnjbLa5S+bkM7sISc7X6PDTz8QZCVF8tDXJhIRGkS6\nXTuAkwEoOMjB+Mz4VoNAg7kjU6mrN56FaEqqrIDQFc4ZnoIx9KqUEw21P4Czhzdtvnv3ljN54/tn\nMDYjjtHpsaee2qJxmfGex42DQMP/U0Oz0qkBwte0RqBUI4OTo9hXWMntjWoBjaXEhLHurvmAtTzj\nlnZy7lc564gOC0ZEePTqSeSV1RAeEsQ3z8jmr4t2U+2qO+2moZbMH53G9qNlHX7tiQPiyYiP4PEl\ne7l4XH+rZtBF394nZMaTEhPGJzn5fGlSRpe8pq81DgRZjWp+T107hTHpcS2d0qZLx/dnSU4+8ZGh\nvLr+sGf7by4dQ2JUKCkxYSzfU8jDC3dT6XRz/ezsLqmNtUdrBEo14nTXc8XkzGaTrVpy3qg0lu8u\nZFUbs0Zr3fWE2t/Cw0OCGJgU5Xl8yfj+AK2mdjgdT107heU/m9Ph84KDHNx63jB2HCvjscV7OFFR\n22U1AodDmDk4qcX5Cqdad7CIM/7wiV9TUxhjOFhUydgM69t+cnQY184cyKSseM63Z0x3VHxkKM/c\nMI2rT8kfNTwtmn5x4QQ5hMsnZXCkpJrfvL2dXXneLx50OjQQKNVIWY2LGC+ban44bygx4cEtzg9o\n4HTXERbS8p/Z5ZMb8gZ516zQESLSrEPYWw1pIf788S62HysjvosCAcDUgQkcL6vx9J20ZtGOfI6U\nVPOhnT7DH0qrXdS46rlkfDrfO3cI549J494vjeX1759x2q89acDJJqI1d8xr0mQ3f/TJIPNpK7O9\nu5oGAqVsxhgqnG6vA0FYcBBnDk1myc6CVtMC1NadrBGcataQJHb+7kKmtJDj35/iIkL4zjmDmzzv\nKmcNS8YhcPebW/n9eztYuquAx5c0/9ZfbA9b3Z1X4ZeUC3Ay/XZGfAQ/v3Bkk6R7p8vhEP5x3VT+\n+c1pzTqFh6dF86evTCAs2NFsCLCvaCBQylZVW4cxdGio5GUT0jleVsMrLaw3DOB01XvSM7ckLDio\nw+XsDr/8wiiy7YlQQ1M7n17iVINTovnixAwW7yzgqWX7uP7ZNTzwwU7WHSxmT/7JZpCGSXfVrjou\narRAT3f5w/s53GHPwu4X55vRO+eNTmPOiNRm20WEK6dk8p2zB7Nsd0G7Q5S7ggYCpWwNM4A70sF6\n4dh+jM2I5YU1LS8+UltXT1gbgaAn+/WlY/jWmYO4ekbLKaY764yhzdNVXPHEKs57aKmn76C4qpbB\nKVZ/ys68coq8mMXdVerqDf9dfZD1uVaqjn7dOIyzsatmZJGdFOWZlOhLvfMTqpQP3PQfaw5BR4Zz\nigiXjE9n8+HSFv9gne62awQ92ZyRqdx5yehmmVFP18RG7eOpMWE0fvlrnlnNC6tzKalyMSQlmte+\nPxuAT/d2fN2Fzso5Xub5UgCQGnt6s6o7q39cBJ/cfg4zB59eeg9v9M5PqFJdrLq2js12Zkhv+wga\nnDfKqt4va2Hst9Nd32Obf/xlcHIU35iRxX+/NYNlP5tD/7iT8x425pZwx+tbyDleTnxECOMz4oiL\nCGFRTvflKGpI8XDLvGFcPL6/X///umPoKGggUIo739jCqLs/6PT5Q1KiSY0Ja3HxkdpeXCPwFYdD\nuO/L4zhjaDLhIUHccdEowBo9s+6u+Z5AnBAVSnCQg/NGpbFwex6uurbz+XeVhtrAjWdk89jVk7vl\nPf1NP6Eq4C1pNDJjWnYCMwd1rCouIpw5NJkVewqb3ayc7joNBO24eHx/9t9/Eamx4YSHBHHR2Kbz\nK+aNSqWsxs1r6w93y9rHVU7rfSNDA2e+rX5CVUCrddc3WRTmhW/PJCK0400BF4ztR0mVq1lmyVp3\n7+0s7k6Nm0B+MHcoAPPslBzTsq3htT9/dQsfbW+/iaiwwtlkEfmOqqh1ExrkCKgA3mrIE5FyoNXw\na4zp+lkwSnWzQ8VV1NUb/nD5OM4clkxIO7l4WnPO8BTCgh18kpPfJKW0BoKOG5AYyf77L/IEh5SY\nMMJDHNS46ll3sNgz4a01U3+3kKSoUE8qkLYcKanmqqc+Izk6lJe+M4uQIAeVTjdRYYHVr9PqJ9QY\nE2Pf7B8BfgFkAJnAz4GHu6d4SvnWfjuXzPB+Mac1YSg8JIjpLSxSrp3FnXNqJ+mKn89lSEoUG3NL\nWjmjqROVtdS46to9bldeOblFVazPLeH5zw4CVn6oqC7ICNubePNV5TJjzOPGmHJjTJkx5gngi74u\nmFLdoWG9gKw2Ukl765zhKezKq+A/nx7wbNPO4q6RHB3GOcNT2XS4pM1O48Y3/9X7219StMppHR8b\nHswzK/ZTV2/NLo8KoP4B8C4QVIrIN0QkSEQcIvINoPtXV1bKB46V1hAa7CDJXjLydFw3K5upAxN4\nevl+T6em013XaooJ1TGTB8bjdNe3mfr7RKOJZyt2t5+eodLukP7mGYM4XFzN3oIKKmu1aaglVwNf\nBfLsn6/Y25Tq1d7adJQnl+0jPS68S8ZrhwY7+OKkDHKLqthbUIG7rp56g/YRdJEpAxMAePDDnbyy\n9lCLCekK7XV/AZbvbj6c11VX32St6YYRQg1rMm87WkplADYNtXm1IhIEfNkY0+GmIBEZAPwbSMPq\ndH7KGPOIiDwIXArUAnuBbxpjvGv4U6oL3bJgAwDuLkxqdvYwK33Cmv3FngVitGmoa/SPi2BkvxhW\n7ClkRaO+mBU/n0OF082lf1tBcrQ1C/iCMWl8uC2P9bnF3P7yJh69ehJj0uO4/70cnl25n413zyc+\nMpQquylpXEYcYcEOth4po9Lp9ltaCX9p8xNqjKkDrurka7uB240xo4GZwM0iMhr4GBhrjBkP7AJ+\n2cnXV6rTGrclt5cSuSOyEiNJiAxh06ESnC6rLVtrBF3nT1+Z0Gzbi2sOsfN4Oa464xkK/OVJVorv\nyx9fxf7CShbYuaAaUoYvWGMlCaxy1uEQiAwNYkJmPCv3FFJVG3g1Am8+oStF5FEROUtEJjf8tHeS\nMeaYMWa9/bgc2AFkGGM+MsY0JPL4DGskklLdZtWeQkbeZc0kTowK5e/XdN3sUc8i7YdLqLU7NUN1\n1FCXGZsRxx+vGAfAwKRIJmXF887mo5RWn5w3kBwdyrkjUpicdTKn0cETVRhjcNdb/ydvbjwCWH0E\nUaHWCnJfGNePnOPlHCmpDrg+Am/C3kT79z2NthlgrrdvIiLZwCRg9Sm7bgRe8vZ1lOoKaw8WA3DV\n9Czuv3xcl7/+hMx4lu3aTbGdSlmbhrrWiH7WFKZrZw4kyCH89u3tbDpk5Ynacc+FBDmE0GAHC26a\nyb6CSv796UHe3HiEjYdKKKlyMTglipzj5fzopY3kl9cQad/0vzgxg3+uPEBuUVWXrsHQG7QbCIwx\nHV/vrhERiQZeBW4zxpQ12v4rrOaj/7Zy3k3ATQBZWVktHaJUpxSUO4mPDPFJEAAru2a9gbUHrICj\ngaBrTRwQzwe3ncXw1Bj2FVoDGN/bcoyYsOAms8LDgoMY1T+W+aNTWbAmly8/vorY8GD+dtUkLv7r\nCl7fYNUKBidb6a4To0J579az+CQnn1ndkPGzJ/GqIUxELgbGAJ4eFGPMPa2f4TkvBCsI/NcY81qj\n7TcAlwDzTCvJQ4wxTwFPAUydOtU/SxSpPimvrIa0GN91Bo7PtNY7fmuj1R6d7qOFTQLZSLtWMCQl\nivjIEEqqXGQmRLR47OwhyQQ5hLp6ww/mDm226Hxko2ag6LBgLpuQ7ruC91DtflURkb8DXwN+CAjW\n8NF2V6oQazzeM8AOY8xDjbZfCPwMa6Ka75feUeoUeeVOn+aYT4oOY1xGHGsOWBOaunKFL9WUiDAh\n0+oLSIhseS5IeEgQb/3gDG6dN4zrZmUDsOj2czjLHuHl6KZUzz2ZN3XW2caY64BiY8xvgVnAcC/O\nOwO4FpgrIhvtn4uAR4EY4GN72987W3ilOiO/rIY0Hw8P/OLEk98q41u5Qamu0TC/oK0Zx2PS4/jR\n/OGEh1jf/oekRHPjGYMA2Hm83PeF7OG8CQQNY+uqRCQdcAH92zvJGLPCGCPGmPHGmIn2z3vGmKHG\nmAGNtn33dC5AqY6orzcUlDtJjfHtqlNfn271a6X5aXWrQHLROCsJXU4Hb+hTsq0A4nR3zzoHPZk3\nfQTviEg88CCwHmvE0NM+LZVSPlJUVYu73vi8RhAdFszSn55LsKaX8LmhqTF8ZUoms4d2rIM3NjyE\ni8f17/B5fZE3o4butR++KiLvAOHGmFLfFksp38grsyYcdcc39YFJUT5/D2V5sIWJZt547BuBsQJZ\ne9oNBCKyAlgKLAdWahBQvVl+mZWLJjXAUggo1RZv6q3XAjuBK4BVIrJWRP7i22Ip5RsnawQaCJRq\n4E3T0H4RqcFKElcLzAFG+bpgSnWlj7Yd5+evbuarUwcAkBKtnbhKNfBmHsFe4A2sLKLPYCWMu9DX\nBVOqK/31k90UV7l4ctk+kqNDdbavUo1489fwVyAXKwvpLcD1IjLEp6VSqotlxJ+cdfrby8b6sSRK\n9TzeNA09Ajxi5wz6JvAbrIyhgZWeT/VqhRVWArh/3jCNOSNT/VwapXoWb0YN/Rk4E4gGVgF3Y40g\nUqrXOFpSzRWTMzUIKNUCbyaUfQo8YIzJ83VhVN+zfHcBH2/P45qZAxmeFuOXMrjq6skrqyEjXkcK\nKdUSb/oIXgPmi8hdACKSJSLTfVss1Vf859OD/PvTg1z0yHI2HfLPiqS5RVXUGxiQGOmX91eqp/Mm\nEDyGlWiuYcH6cnubUu3aW1DBzMGJJEWH8uCHO1s9rq4L1w1ucKLCyfLdBezJrwDwW41EqZ7Om0Aw\nwxhzM1ADYIwpBjSdomqXq66egyeqmJyVwKXj01lzoKjJWsENXvo8lyF3vEdRZW2Xvv8ji3Zz3bNr\nWLPfSgc9RNNBK9UibwKBS0SCsJLNISIpgKbrU+26642tuOsNg1OimT00iVp3PevsZSIBth4pJfsX\n7/KH93MAeGXtoS57b2MMi3bkYwy8tv4w6XHhRAfYguRKecvbeQSvA6kich+wAvi9T0ul+oSVewsB\nOHt4MtMHJRHkEFbZ2wA+2m6NPyiushYe/+MHOU0CxenYnV/BkZJqz+sP1WYhpVrlzTyC/4rIOmAe\n1gplXzLG7PB5yVSvV1Hj5pqZWaTay0JOyIxj5Z4T/PQCKKtxsSH35E1/1uAkth8r4843tjJjUCIj\n+8UwMSuelOgwkjqRDmLRjnwABiVHsb+wkmHaLKRUq9oMBHaT0DZjzEggp3uKpPoCYwzlNW5iw0M8\n284dkcpfFu5iX0EF9727g+W7T9YO4iND+Nq0ATy1bB87jpV5tgc7hFW/mNvhbKGr9hYysl8Mc0em\n8viSvRoIlGpDm01Dxpg6YKeIZHVTeVQfUe2qw11viGkUCK6ankVIkIN/f3qwSRAQgcsmpHPNjOZL\nYbvrDf/+9CD1HRxVtDuvgtHpsZwzPAURGG+va6uUas6bPoIEYJuILBKRtxp+fF0w1buVVbsBiI04\nWelMiQnjvFGpvLruMK56a7zBFZMz2ff7i/jCuP5kJUXy5s1neI7/k73YyKOL9/DXT3a3+D55ZTVM\nv28hWw6XUlZj9TWU1bg4XlbD0NRoZgxO4vNfncfo9FifXKdSfYE3wyju8nkpVJ9Tbt+UGzcNAXxp\nYgbvbTkOWHl/rG/s4tk/YUA8D311AlFhwVwwph8Lt+fxwbbjPLfqADfPGUrIKUs/rthdSH65k0sf\nXUGQQ1j+szmeDuehKVZzULKmnFaqTd50Fi/tjoKovqXh23lMeNOP2LxRaZ7H5wxPweEQTnX55EzP\n48e/MZmPtufx3efXsWJ3YbNcQaXVLs/junrDx9vz+NOHO4kJD2ZSVkKXXItSfZ0OrFY+cbJpqGmN\nIMghLP3pubjqTItB4FQOhzB3ZCqx4cF85/l1/PT8EXz77MGUVru48bnPyS+3VhybNTiJT/ed4Ndv\nbQPg1e/NJiVGawJKeUMDgfKJMk/TUPOPWEcXdQ8NdjBrSBIfbsvj9+/vICk6lEU5+Z4moJH9Ylhw\n00zufWc7Gw+V8IWx/ZgyUGsDSnnLq0AgIhFAljGm9WQxKuDtzivn3S3H+N65Q3h38zGAJqOGTsfQ\n1Gg+3JaHMfDjlzc12deQOuKuS0Z3yXspFWi8WY/gUuBPWPmFBonIROAeY8xlvi6c6l3+85mVaTTn\nWDkfbc/jnOEppHZR88x3zhlCWHAQocEOT0qKH84dyszBSYzNiOuS91AqUHlTI/gNMB1YAmCM2Sgi\ng3xYJtUDvbP5KNuOlvHzC0e2ekxhhROAD7ZZo4J+NH94kxFBpyM2PIRb5g0D4NIJ6fzq9S1cO3Ng\nhyeaKaWa8yYQuIwxpaf8QXd9zmDVo/3ghQ0AXD09q9W8/oeKqps8H9XfN/l9MuIjeO6buiSGUl3F\nm0CwTUSuBoJEZBjWAvarfFss1dPEhAVT7nTz1qaj3DxnaIvHHCqu4qrpAwh2OHDXG8KCdVlrpXoD\nb2YW/xAYAziBBUAZcJsvC6V6ILtCuDgnv8XdZTUuSqpcDEyK4t4vjeX+y8d1Y+GUUqfDmwllVcCv\n7B8VgMpqXJTXuAkNdrA+t5jSahdxp8wP2HyoFLCGciqlepdWA4GIvE0bfQE6aihwHCuxJm1dMTmT\nBWtyeXqGoGPEAAAau0lEQVTZPq6fnc3Law/xnbMHExzkYM2BIhyCjt9Xqhdqq0bwJ/v35UA/4Hn7\n+VVAni8LpXqWhtFA80ensmBNLo8u3sOji/cA1vj+C8b0Y/PhEoanxXTZvAGlVPdpNRA05BgSkT8b\nY6Y22vW2iKz1eclUj1FeY6WLSIsNZ2BSJAdPVHn2bTxUwgVj+nGoqIphqdospFRv5E1ncZSIDG54\nYs8h6FiOANWrVTqtQBAdFszbPzyTm+cM8ex7cU0uTyzZy96CSjISIvxVRKXUafAmEPwIWCIiS0Rk\nKbAYuNW3xVI9SUWjQBAbHsKlE9I9+4qrXPzxA2umb0a8BgKleiNvRg19YM8faJhSmmOMcbZ3nogM\nAP4NpGF1Oj9ljHlERBKBl4Bs4ADwVWNM16xYrnyiIRBEhVkflxFpMXxlSiYXje+P01XPd59fB0C/\nOJ3lq1Rv1G6NQERCgO9gLVBzF/Bte1t73MDtxpjRwEzgZhEZDfwCWGSMGQYssp+rHqzC6SYkSAgL\ntj4uIsKDX5nAnBGpXDi2H/d9eSwA2R3MKqqU6hm8mVn8BBACPG4/v9be9q22TjLGHAOO2Y/LRWQH\nkAF8ETjXPuxfWDmMft7Bcqtu8s7moxwpriY6LLjVvEHfmDGQOSNSSdemIaV6JW8CwTRjzIRGzz8R\nkU2tHt0CEckGJgGrgTQ7SAAcx2o6Uj1QYYXTk2Mos52OYA0CSvVe3nQW14mIZ5iIPYKozts3EJFo\n4FXgNmNMWeN9xhhDK5PWROQmEVkrImsLCgq8fTvVhUqqaj2Po8N0DSOl+ipv/rp/CiwWkX1YGWcG\nAt/05sXtvoRXgf8aY16zN+eJSH9jzDER6Q+0mLzGGPMU8BTA1KlTNdupH5RUnVwPuKvSSSuleh5v\nRg0tskcNjbA37fRy1JAAzwA7jDEPNdr1FnA98Af795sdLrXqFo0DwY5jZW0cqZTqzbwZNfQVINQY\nsxm4DFggIpO9eO0zsDqW54rIRvvnIqwAMF9EdgPn2c9VD1RSfTIQ9NMFYJTqs7xpGrrLGPOKiJwJ\nzMPKQfQEMKOtk4wxK/AkL25mXodKqbpMSVUt8ZGhXh8L8NJNM1tdjEYp1ft51Vls/74YeNoY8y7W\n+sWql1l7oIiJ93zM+1uOtXrMgx/mMPV3H3Pjc5+z8VAJDoFp2Yk6KkipPsybGsEREXkSmA/8UUTC\n8C6AqB5m6xFrzYA/fpDDF8b1b7a/rt7w2OK9AHxiL0CTEBmCw6EdxUr1Zd7c0L8KfAhcYIwpARKx\nRhKpXuZYqbWuQG5RFdW1zUcAbz9qdQgPS43mSxOtfEIRIbrcpFJ9XVsL08Ta4/7DsWb/YucJcgKa\nhrqXcdXV89n+IgDqDWw4VMzsIcme/Ut25nPTf6ycQf/+v+kYY6WfnjMy1S/lVUp1n7aahl4ALgHW\nYU36atw+YIDBLZ2keqZ/rTrApkMlZCVGcqSkmhW7C5sEghv++bnncb/YcESEZ26Y5o+iKqW6WVsL\n01xi/x7UfcVRvrK/sBKAW+YN46XPc3l8yV6qauv4zWVjyC+vaXKsTh5TKrB4lTdARC4HzsSqCSw3\nxrzh01KpLueuM6TFhnHllEzCgh18fqCY51YdYFJWPK66kxO3Gy86o5QKDO0GAhF5HBgKLLA3fVdE\n5htjbvZpyVSXKqtxedYTvnRCOjMGJzL9vkXc+uJGABwC2++5kHDtHFYq4Hgzamgu1oihfxpj/glc\nZG9TPVhVrZtfvb6Fw8XW+sLlNW5iw0/G/dSYcEaknVxjOCY8RIOAUgHKm6ahPUAWcNB+PsDepnoo\nV109d7+5jf+tO0y9Mdx/+XjKalwknDKj+L/fnsGJilq2HyslNUZTSCgVqLwJBDHADhFZg9VHMB1Y\nKyJvARhjLvNh+VQHlde4eHr5fv637jAAb248yrJdhRRWODl/TL8mxyZHh5EcHcaIfjEtvZRSKkB4\nEwju9nkpVJe59cWNfJKTT7BD+MvXJvKPFfvZdKgEgJhwXVNAKdWcN2mol4rIQGCYMWahiEQAwcaY\nct8XT3XURvumHxEaxKUT0rl0QjpfeGQ5O46VERvuzVLTSqlA400a6m8D/wOetDdlAjp8tIcamhoN\nwN+umuTZdu6IFACcbq8XllNKBRBvRg3djLW2QBmAMWY3oHkHeqjSKhcXjEnj3BEn/4vmj7aWhY4K\n1aYhpVRz3twZnMaY2obZpiISTCvrDCv/WbO/iDHpsRRV1TIpMr7JvslZCbzy3VmMTY/zU+mUUj2Z\nN4FgqYjcAUSIyHzg+8Dbvi2W6ohPcvK48bm1zB6SRFFlLQlRzZeLmJad6IeSKaV6A2+ahn4BFABb\ngO8A7wF3+rJQqmNe33AUgFV7T1BXb0j0cgUypZQC70YN1QNP2z+qByqvcTEwKZL8MifVrjqCdCEZ\npVQH6EpjfUBFjZuM+AieuX4qcHLkkFJKeUOHkfQBFU43WVGRzB6azKa7zycuUucLKKW8pzWCPqC8\nxk20PWtYg4BSqqO8SUP9Ns2Hi5ZiLVf5pDGmpvlZqjtVON3EhGnlTinVOd7UCPYBFZzsMC4DyoHh\naAey3xljqHCerBEopVRHeXP3mG2Mabx47dsi8rkxZpqIbPNVwVTLjDH87t0dJESG8N1zhuCqM9TV\nG6LDtElIKdU53gSCaBHJMsbkAohIFtAwLKXWZyVTLXp78zGeWbEfgNyiKn5ywQgArREopTrNm7vH\n7cAKEdkLCDAI+L6IRAH/8mXhVFNbDpdyz9vbyYiPYM7IFF5cc4jLJmQAaB+BUqrTvJlQ9p6IDANG\n2pt2NuogfthnJVNNGGO49NEVAFw8rj9fn5bF85/lcs0zqwGIDNVlJpVSneNNGurNwI+BCmPMJh0l\n5B/rc4s9j782bQBj0mOb7B+YFNXdRVJK9RHejBq6FKgDXhaRz0XkJ3Y/gepGG3KtBWfW3zWfs4en\nICLcMDsbgNV3zNPlJpVSndZuIDDGHDTGPGCMmQJcDYwH9vu8ZKqJvQUVJEaFktgos+idF49i9R3z\nSIvVheeVUp3nVQ+jvVTl1+yfOuBnviyUam5vfiVDUpo2/wQHOTQIKKVOmzczi1cDIcArwFeMMft8\nXqoAtHx3AenxEQxJaTlh3N6CCs4bldbNpVJKBQJvagTXGWN2+rwkAcwYw7XPrAHgwB8ubra/rt5w\norKWfnH67V8p1fW8GT66U0QuBsYA4Y223+PLggWSwoqT8/KMMTQsC9qgrNoFQFyEzh5WSnU9b4aP\n/h2rb+CHWBPKvgIM9OK8Z0UkX0S2Nto2UUQ+E5GNIrJWRKafRtl7hUqnm4JyZ5vH7Cuo8Dw+cKLK\n89hVV48xhlINBEopH/Jm+OhsY8x1QLEx5rfALKyEc+15DrjwlG0PAL81xkwE7raf92l3vrGVafct\nZE9+RZPtOcfLeHfzMQD2FlR6ti/dmQ9AjauOWfd/wn8+O6iBQCnlU970EVTbv6tEJB04AfRv7yRj\nzDIRyT51M9AwEyoOOOpdMXuvVXsLAbj1xQ289YMzWbGnkOiwIK544lMAYsKn88aGI0SGBpESE8Zv\n3t7OxzvyuHBsfwornLy/5bhnsli8rjWglPIBbwLBOyISDzwIrMe6mXc2/fRtwIci8ies2sjsTr5O\nryEIoUEOth0t458r9/O7d3c02X/ds1Yn8aSseL5z9mDe33qcD7YeZ+WeEwCsO1hMXpk1mVtrBEop\nX/BmQtm9xpgSY8yrWH0DI40xd3fy/b4H/MgYMwD4EfBMaweKyE12P8LagoKCTr6df9W668krr+Hb\nZw8iPS7cEwQcAnNGpJCVGOk5NjI0iAvH9ueRr0/iiimZAMSEB1NbV8+iHXmABgKllG90aKlKY4zT\nGFN6Gu93PfCa/fgVoNXOYmPMU8aYqcaYqSkpKafxlv7z7Mr9GAMDE6O4arqVlePcESns+t0X+Pu1\nU3jy2ilkxEcAMGtwkue80f2t1rMrp2QS7BA+3GYFglgNBEopH+ju3MVHgXOAJcBcYHc3v3+32VtQ\nwR/ezwFgUEoUF4ztx6HiKm47bzjBQQ6CgVH9Y1n5i7kcKKxkQKPawZVTMimscPKtswazJ7+C5bsL\nCXII4SGaYVQp1fV8FghEZAFwLpAsIoeBXwPfBh4RkWCgBrjJV+/vb29uOALA36+ZzNSBCYgID1w5\nocVjs5Obpo4IDwnitvOsgVm/vWwMN7+wgQvG6KxipZRveJNiYnILm0uBg8YYd2vnGWOuamXXFC/L\n1iuU1biIDQ+hxlXHPe9s5+rpWQxPi2FfYSVZiZFcOLbdAVZtGpwSzfu3ntVFpVVKqea8qRE8DkwG\nNmNNKBsLbAPiROR7xpiPfFi+Hu1QURVnPbCYS8b35x17TsALq3M9+88aluyvoimllNe86Sw+Ckyy\nO26nAJOAfcB8AmBCWFv2F1oTwRqCwKliw7VzVynV83lTIxhujNnW8MQYs11ERhpj9p2aEyfQFFfV\nNnl+9vAUIkOC2HS4hGOlNWQnR7ZyplJK9RzeBIJtIvIE8KL9/GvAdhEJA1w+K1kvcKJRsrhb5w3j\nR/NPZt7YcriUYWktp5RWSqmexJtAcAPwfaxZwQArgZ9gBYE5vilW73Ci0kmwQ8i590KCHE1rR+My\n4/xUKqWU6hhv0lBXA3+2f05V0cK2Ps8Yw+/e3cHKPYUkRoUSHNSheXlKKdWjeDN89AzgN1jpJTzH\nG2MG+65YPdv+wkqeWWEt2zxSF41XSvVy3jQNPYOVF2gd1nrFAe/zA0Wex5GhOttXKdW7eRMISo0x\n7/u8JL3IhtwS4iJCuPdLYxl8yqxgpZTqbbwJBItF5EGsZHGepbaMMet9Vqoe7khJNdlJkVw2Id3f\nRVFKqdPmTSCYYf+e2mibwUoaF5AKyp1NksQppVRv5s2ooYAeItrY5sMlDE+LIa+shikDE/xdHKWU\n6hKtBgIRucYY87yI/Lil/caYh3xXrJ7nSEk1lz26kumDEimucpEWG+7vIimlVJdoq0bQ0Auq4yOB\npTutVdLW7LdGDKXGhPmzOEop1WVaDQTGmCfth48bY3rnWpGnqa7ecN+7O7hhdjaf7rPWEA4PcVDj\nqic+MtTPpVNKqa7hTWfxShE5ALwEvGaMKfZtkXqOQ0VVPLtyP29sPEJmQgTnDE/h6eum8vamo8wb\nlerv4imlVJfwZvH64cCdwBhgnYi8IyLX+LxkPUCN25o/V1RZy978CjITIggNdnDFlExCNK2EUqqP\n8OpuZoxZY4z5MdZi80XAv3xaqh6iuvbkROrK2joyE3TIqFKq72k3EIhIrIhcLyLvA6uAY1gBoc+r\ndjXNqJGREOGnkiillO94UyPYBEwE7jHGDDfG/NwYs87H5eoRauxAcPE4a93hoSm6voBSqu/xprN4\nsDHGiEi0iEQbYwIm9XR1bT0At8wbxq8uHkV6vNYIlFJ9jzc1gjEisgFrwfrtIrJORMb6uFw9QlWt\nG7AyjGoQUEr1Vd4EgqeAHxtjBhpjsoDb7W19XkPTUHiIpppWSvVd3gSCKGPM4oYnxpglnJx13Kc1\ndBZH6JoDSqk+zJs+gn0ichfwH/v5NcA+3xWp52joIwgP1jkDSqm+y5s73I1ACtZ6BK/Zj2/0ZaF6\niiqXm9Agh65JrJTq07xJQ10M3NINZelxamrrtFlIKdXntZWG+q22TjTGXNb1xelZql11RGhHsVKq\nj2urRjALOAQsAFYD0i0l6kGqXfVaI1BK9XltBYJ+wHzgKuBq4F1ggTFmW3cUrCeorq3ToaNKqT6v\n1V5QY0ydMeYDY8z1wExgD7BERH7QbaXzs9LqWmLCvBlYpZRSvVebdzkRCQMuxqoVZAN/BV73fbF6\nhv2FlcwbmebvYiillE+11Vn8b2As8B7wW2PM1m4rlR8UV9ZSVFXLEDuxXGmVi8KKWoakBsTcOaVU\nAGtrgPw1wDDgVmCViJTZP+UiUtY9xes+t760kXl/XkphhROAvYVWbr0hmnFUKdXHtdVH4DDGxNg/\nsY1+Yowxsd1ZyO5wuKgKgF+9voX6esOWw6UADE+L8WexlFLK53w2ZVZEnhWRfBHZesr2H4pIjohs\nE5EHfPX+HRUbEQLAh9vy+OMHOXySk092UiQDEnVVMqVU3+bLITHPAY8C/27YICJzgC8CE4wxThHp\nMSvAF1XWcsn4/hw4UcmTy6xUStfPGujnUimllO/5rEZgjFmGtb5xY98D/mCMcdrH5Pvq/TuqqLKW\n5Ogw/vfd2Z5t18zUQKCU6vu6e5D8cOAsEbkPqAF+Yoz5vJvL0IzTXUeF001SVCjhIUEs/PHZbDta\nxjDtH1BKBYDuDgTBQCLWBLVpwMsiMtgYY049UERuAm4CyMrK8mmhiiprAUiMDgVgaGoMQ1M1CCil\nAkN351c+DLxmLGuAeiC5pQONMU8ZY6YaY6ampKT4tFAnKqxAkBQV6tP3UUqpnqi7A8EbwBwAERkO\nhAKF3VyGZjw1gqgwP5dEKaW6n8+ahkRkAXAukCwih4FfA88Cz9pDSmuB61tqFupuJwOB1giUUoHH\nZ4HAGHNVK7uu8dV7dtaJSm0aUkoFLl2DESiqdBLkEOLsSWVKKRVINBBgNQ0lRIbgcATc2jtKKaWB\nAKxRQ9o/oJQKVBoIsGoEGgiUUoFKAwFwtKSatNhwfxdDKaX8IuADQXFlLUdLaxjVv89l1lZKKa8E\n9IK8Ty7dy/3v5wAwJl0DgVIqMAVsjaDWXc/fl+71PB+bHufH0iillP8EbI1gzf4iiqtc/P2aKUzK\niidBO4uVUgEqYAPBpsMlAMwakqQTyZRSAS1gm4a2HC4lOylSg4BSKuAFbiA4UsrYDO0XUEqpgAwE\nRZW1HCmpZpwGAqWUCsxAsOVIKYAGAqWUIgADgTGG9QeLARijgUAppfr+qCF3XT1BDkFEWJ9bzOWP\nrwJg+qBE7ShWSin6eI3g4YW7GPPrD/nLwt0ALNqR59l3+aQMfxVLKaV6lD4dCAYkROK0ZxDvLaig\nrNoNwNyRqVw5JdPPpVNKqZ5BesCSwe2aOnWqWbt2bafOzSur4fy/LKPWXU+1q44JmXG8+YMzu7iE\nSinV84jIOmPM1PaO69M1AoC02HBe+PYMpg9KBCAqrM93iyilVIcExF1xTHocz94wjUc/2cOsIUn+\nLo5SSvUoAREIAIIcwq3nDfN3MZRSqsfp801DSiml2qaBQCmlApwGAqWUCnAaCJRSKsBpIFBKqQCn\ngUAppQKcBgKllApwGgiUUirA9YpcQyJSABzs5OnJQGEXFqc30GsODHrNgeF0rnmgMSalvYN6RSA4\nHSKy1pukS32JXnNg0GsODN1xzdo0pJRSAU4DgVJKBbhACARP+bsAfqDXHBj0mgODz6+5z/cRKKWU\nalsg1AiUUkq1oU8HAhG5UER2isgeEfmFv8vTVUTkWRHJF5GtjbYlisjHIrLb/p1gbxcR+av9b7BZ\nRCb7r+SdIyIDRGSxiGwXkW0icqu9vc9eM4CIhIvIGhHZZF/3b+3tg0RktX19L4lIqL09zH6+x96f\n7c/yd5aIBInIBhF5x37ep68XQEQOiMgWEdkoImvtbd32+e6zgUBEgoDHgC8Ao4GrRGS0f0vVZZ4D\nLjxl2y+ARcaYYcAi+zlY1z/M/rkJeKKbytiV3MDtxpjRwEzgZvv/si9fM4ATmGuMmQBMBC4UkZnA\nH4G/GGOGAsXA/9nH/x9QbG//i31cb3QrsKPR875+vQ3mGGMmNhoq2n2fb2NMn/wBZgEfNnr+S+CX\n/i5XF15fNrC10fOdQH/7cX9gp/34SeCqlo7rrT/Am8D8ALvmSGA9MANrclGwvd3zOQc+BGbZj4Pt\n48TfZe/gdWbaN725wDuA9OXrbXTdB4DkU7Z12+e7z9YIgAzgUKPnh+1tfVWaMeaY/fg4kGY/7lP/\nDnb1fxKwmgC4ZruZZCOQD3wM7AVKjDFu+5DG1+a5bnt/KdDbFul+GPgZUG8/T6JvX28DA3wkIutE\n5CZ7W7d9vgNmzeJAYowxItLnhoOJSDTwKnCbMaZMRDz7+uo1G2PqgIkiEg+8Doz0c5F8RkQuAfKN\nMetE5Fx/l6ebnWmMOSIiqcDHIpLTeKevP999uUZwBBjQ6Hmmva2vyhOR/gD273x7e5/4dxCREKwg\n8F9jzGv25j59zY0ZY0qAxVhNI/Ei0vAlrvG1ea7b3h8HnOjmop6OM4DLROQA8CJW89Aj9N3r9TDG\nHLF/52MF/Ol04+e7LweCz4Fh9oiDUODrwFt+LpMvvQVcbz++HqsdvWH7dfZIg5lAaaPqZq8g1lf/\nZ4AdxpiHGu3qs9cMICIpdk0AEYnA6hfZgRUQrrQPO/W6G/49rgQ+MXYjcm9gjPmlMSbTGJON9ff6\niTHmG/TR620gIlEiEtPwGDgf2Ep3fr793Uni4w6Yi4BdWO2qv/J3ebrwuhYAxwAXVvvg/2G1jS4C\ndgMLgUT7WMEaPbUX2AJM9Xf5O3G9Z2K1oW4GNto/F/Xla7avYzywwb7urcDd9vbBwBpgD/AKEGZv\nD7ef77H3D/b3NZzGtZ8LvBMI12tf3yb7Z1vDvao7P986s1gppQJcX24aUkop5QUNBEopFeA0ECil\nVIDTQKCUUgFOA4FSSgU4DQRKtUJEfmVn/dxsZ4WcISK3iUikv8umVFfS4aNKtUBEZgEPAecaY5wi\nkgyEAquwxm0X+rWASnUhrREo1bL+QKExxglg3/ivBNKBxSKyGEBEzheRT0VkvYi8YudDasgv/4Cd\nY36NiAz114Uo1R4NBEq17CNggIjsEpHHReQcY8xfgaNYeePn2LWEO4HzjDGTgbXAjxu9RqkxZhzw\nKFZWTaV6JM0+qlQLjDEVIjIFOAuYA7wkzVe5m4m16NFKOxNqKPBpo/0LGv3+i29LrFTnaSBQqhXG\nSgG9BFgiIls4mQCsgQAfG2Ouau0lWnmsVI+iTUNKtUBERojIsEabJgIHgXIgxt72GXBGQ/u/nUVy\neKNzvtbod+OaglI9itYIlGpZNPA3Ow20GyvD5U3AVcAHInLU7ie4AVggImH2eXdiZbwFSBCRzVhr\nD7dWa1DK73T4qFI+YC+uosNMVa+gTUNKKRXgtEaglFIBTmsESikV4DQQKKVUgNNAoJRSAU4DgVJK\nBTgNBEopFeA0ECilVID7f76wnh8W9HvAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x122fc2eb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "master = MasterAgent(max_eps=500)\n",
    "master.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-01T10:45:13.330419Z",
     "start_time": "2019-05-01T10:45:01.627595Z"
    }
   },
   "source": [
    "# Playing with the trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-01T10:56:49.125429Z",
     "start_time": "2019-05-01T10:56:33.104845Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading model ./model_CartPole-v0.h5\n",
      "0. Reward: 1.0, action: 0\n",
      "1. Reward: 2.0, action: 1\n",
      "2. Reward: 3.0, action: 0\n",
      "3. Reward: 4.0, action: 1\n",
      "4. Reward: 5.0, action: 0\n",
      "5. Reward: 6.0, action: 1\n",
      "6. Reward: 7.0, action: 0\n",
      "7. Reward: 8.0, action: 1\n",
      "8. Reward: 9.0, action: 0\n",
      "9. Reward: 10.0, action: 1\n",
      "10. Reward: 11.0, action: 0\n",
      "11. Reward: 12.0, action: 1\n",
      "12. Reward: 13.0, action: 0\n",
      "13. Reward: 14.0, action: 1\n",
      "14. Reward: 15.0, action: 0\n",
      "15. Reward: 16.0, action: 1\n",
      "16. Reward: 17.0, action: 0\n",
      "17. Reward: 18.0, action: 1\n",
      "18. Reward: 19.0, action: 0\n",
      "19. Reward: 20.0, action: 1\n",
      "20. Reward: 21.0, action: 0\n",
      "21. Reward: 22.0, action: 1\n",
      "22. Reward: 23.0, action: 0\n",
      "23. Reward: 24.0, action: 1\n",
      "24. Reward: 25.0, action: 0\n",
      "25. Reward: 26.0, action: 1\n",
      "26. Reward: 27.0, action: 0\n",
      "27. Reward: 28.0, action: 1\n",
      "28. Reward: 29.0, action: 0\n",
      "29. Reward: 30.0, action: 0\n",
      "30. Reward: 31.0, action: 1\n",
      "31. Reward: 32.0, action: 0\n",
      "32. Reward: 33.0, action: 1\n",
      "33. Reward: 34.0, action: 0\n",
      "34. Reward: 35.0, action: 1\n",
      "35. Reward: 36.0, action: 0\n",
      "36. Reward: 37.0, action: 1\n",
      "37. Reward: 38.0, action: 0\n",
      "38. Reward: 39.0, action: 1\n",
      "39. Reward: 40.0, action: 0\n",
      "40. Reward: 41.0, action: 1\n",
      "41. Reward: 42.0, action: 0\n",
      "42. Reward: 43.0, action: 1\n",
      "43. Reward: 44.0, action: 0\n",
      "44. Reward: 45.0, action: 1\n",
      "45. Reward: 46.0, action: 0\n",
      "46. Reward: 47.0, action: 1\n",
      "47. Reward: 48.0, action: 0\n",
      "48. Reward: 49.0, action: 1\n",
      "49. Reward: 50.0, action: 0\n",
      "50. Reward: 51.0, action: 1\n",
      "51. Reward: 52.0, action: 1\n",
      "52. Reward: 53.0, action: 0\n",
      "53. Reward: 54.0, action: 1\n",
      "54. Reward: 55.0, action: 0\n",
      "55. Reward: 56.0, action: 1\n",
      "56. Reward: 57.0, action: 0\n",
      "57. Reward: 58.0, action: 1\n",
      "58. Reward: 59.0, action: 0\n",
      "59. Reward: 60.0, action: 1\n",
      "60. Reward: 61.0, action: 0\n",
      "61. Reward: 62.0, action: 1\n",
      "62. Reward: 63.0, action: 1\n",
      "63. Reward: 64.0, action: 0\n",
      "64. Reward: 65.0, action: 1\n",
      "65. Reward: 66.0, action: 0\n",
      "66. Reward: 67.0, action: 1\n",
      "67. Reward: 68.0, action: 1\n",
      "68. Reward: 69.0, action: 0\n",
      "69. Reward: 70.0, action: 1\n",
      "70. Reward: 71.0, action: 0\n",
      "71. Reward: 72.0, action: 1\n",
      "72. Reward: 73.0, action: 0\n",
      "73. Reward: 74.0, action: 1\n",
      "74. Reward: 75.0, action: 0\n",
      "75. Reward: 76.0, action: 1\n",
      "76. Reward: 77.0, action: 0\n",
      "77. Reward: 78.0, action: 1\n",
      "78. Reward: 79.0, action: 0\n",
      "79. Reward: 80.0, action: 1\n",
      "80. Reward: 81.0, action: 0\n",
      "81. Reward: 82.0, action: 1\n",
      "82. Reward: 83.0, action: 0\n",
      "83. Reward: 84.0, action: 1\n",
      "84. Reward: 85.0, action: 0\n",
      "85. Reward: 86.0, action: 1\n",
      "86. Reward: 87.0, action: 0\n",
      "87. Reward: 88.0, action: 1\n",
      "88. Reward: 89.0, action: 0\n",
      "89. Reward: 90.0, action: 0\n",
      "90. Reward: 91.0, action: 1\n",
      "91. Reward: 92.0, action: 0\n",
      "92. Reward: 93.0, action: 1\n",
      "93. Reward: 94.0, action: 0\n",
      "94. Reward: 95.0, action: 1\n",
      "95. Reward: 96.0, action: 0\n",
      "96. Reward: 97.0, action: 1\n",
      "97. Reward: 98.0, action: 0\n",
      "98. Reward: 99.0, action: 0\n",
      "99. Reward: 100.0, action: 1\n",
      "100. Reward: 101.0, action: 0\n",
      "101. Reward: 102.0, action: 1\n",
      "102. Reward: 103.0, action: 0\n",
      "103. Reward: 104.0, action: 1\n",
      "104. Reward: 105.0, action: 0\n",
      "105. Reward: 106.0, action: 1\n",
      "106. Reward: 107.0, action: 0\n",
      "107. Reward: 108.0, action: 1\n",
      "108. Reward: 109.0, action: 0\n",
      "109. Reward: 110.0, action: 1\n",
      "110. Reward: 111.0, action: 0\n",
      "111. Reward: 112.0, action: 1\n",
      "112. Reward: 113.0, action: 0\n",
      "113. Reward: 114.0, action: 1\n",
      "114. Reward: 115.0, action: 0\n",
      "115. Reward: 116.0, action: 1\n",
      "116. Reward: 117.0, action: 0\n",
      "117. Reward: 118.0, action: 1\n",
      "118. Reward: 119.0, action: 0\n",
      "119. Reward: 120.0, action: 1\n",
      "120. Reward: 121.0, action: 0\n",
      "121. Reward: 122.0, action: 0\n",
      "122. Reward: 123.0, action: 1\n",
      "123. Reward: 124.0, action: 0\n",
      "124. Reward: 125.0, action: 1\n",
      "125. Reward: 126.0, action: 0\n",
      "126. Reward: 127.0, action: 1\n",
      "127. Reward: 128.0, action: 1\n",
      "128. Reward: 129.0, action: 0\n",
      "129. Reward: 130.0, action: 1\n",
      "130. Reward: 131.0, action: 0\n",
      "131. Reward: 132.0, action: 1\n",
      "132. Reward: 133.0, action: 0\n",
      "133. Reward: 134.0, action: 1\n",
      "134. Reward: 135.0, action: 0\n",
      "135. Reward: 136.0, action: 1\n",
      "136. Reward: 137.0, action: 0\n",
      "137. Reward: 138.0, action: 1\n",
      "138. Reward: 139.0, action: 0\n",
      "139. Reward: 140.0, action: 1\n",
      "140. Reward: 141.0, action: 1\n",
      "141. Reward: 142.0, action: 0\n",
      "142. Reward: 143.0, action: 1\n",
      "143. Reward: 144.0, action: 0\n",
      "144. Reward: 145.0, action: 1\n",
      "145. Reward: 146.0, action: 0\n",
      "146. Reward: 147.0, action: 1\n",
      "147. Reward: 148.0, action: 1\n",
      "148. Reward: 149.0, action: 0\n",
      "149. Reward: 150.0, action: 1\n",
      "150. Reward: 151.0, action: 0\n",
      "151. Reward: 152.0, action: 1\n",
      "152. Reward: 153.0, action: 0\n",
      "153. Reward: 154.0, action: 1\n",
      "154. Reward: 155.0, action: 0\n",
      "155. Reward: 156.0, action: 1\n",
      "156. Reward: 157.0, action: 0\n",
      "157. Reward: 158.0, action: 1\n",
      "158. Reward: 159.0, action: 0\n",
      "159. Reward: 160.0, action: 1\n",
      "160. Reward: 161.0, action: 0\n",
      "161. Reward: 162.0, action: 0\n",
      "162. Reward: 163.0, action: 1\n",
      "163. Reward: 164.0, action: 0\n",
      "164. Reward: 165.0, action: 1\n",
      "165. Reward: 166.0, action: 0\n",
      "166. Reward: 167.0, action: 1\n",
      "167. Reward: 168.0, action: 0\n",
      "168. Reward: 169.0, action: 1\n",
      "169. Reward: 170.0, action: 0\n",
      "170. Reward: 171.0, action: 0\n",
      "171. Reward: 172.0, action: 1\n",
      "172. Reward: 173.0, action: 0\n",
      "173. Reward: 174.0, action: 1\n",
      "174. Reward: 175.0, action: 0\n",
      "175. Reward: 176.0, action: 1\n",
      "176. Reward: 177.0, action: 0\n",
      "177. Reward: 178.0, action: 1\n",
      "178. Reward: 179.0, action: 0\n",
      "179. Reward: 180.0, action: 1\n",
      "180. Reward: 181.0, action: 0\n",
      "181. Reward: 182.0, action: 0\n",
      "182. Reward: 183.0, action: 1\n",
      "183. Reward: 184.0, action: 0\n",
      "184. Reward: 185.0, action: 1\n",
      "185. Reward: 186.0, action: 0\n",
      "186. Reward: 187.0, action: 1\n",
      "187. Reward: 188.0, action: 0\n",
      "188. Reward: 189.0, action: 1\n",
      "189. Reward: 190.0, action: 1\n",
      "190. Reward: 191.0, action: 0\n",
      "191. Reward: 192.0, action: 1\n",
      "192. Reward: 193.0, action: 0\n",
      "193. Reward: 194.0, action: 1\n",
      "194. Reward: 195.0, action: 0\n",
      "195. Reward: 196.0, action: 1\n",
      "196. Reward: 197.0, action: 0\n",
      "197. Reward: 198.0, action: 1\n",
      "198. Reward: 199.0, action: 0\n",
      "199. Reward: 200.0, action: 1\n",
      "200. Reward: 201.0, action: 1\n",
      "201. Reward: 202.0, action: 0\n",
      "202. Reward: 203.0, action: 1\n",
      "203. Reward: 204.0, action: 0\n",
      "204. Reward: 205.0, action: 1\n",
      "205. Reward: 206.0, action: 0\n",
      "206. Reward: 207.0, action: 1\n",
      "207. Reward: 208.0, action: 1\n",
      "208. Reward: 209.0, action: 0\n",
      "209. Reward: 210.0, action: 1\n",
      "210. Reward: 211.0, action: 0\n",
      "211. Reward: 212.0, action: 1\n",
      "212. Reward: 213.0, action: 0\n",
      "213. Reward: 214.0, action: 1\n",
      "214. Reward: 215.0, action: 0\n",
      "215. Reward: 216.0, action: 1\n",
      "216. Reward: 217.0, action: 0\n",
      "217. Reward: 218.0, action: 0\n",
      "218. Reward: 219.0, action: 1\n",
      "219. Reward: 220.0, action: 0\n",
      "220. Reward: 221.0, action: 1\n",
      "221. Reward: 222.0, action: 0\n",
      "222. Reward: 223.0, action: 1\n",
      "223. Reward: 224.0, action: 0\n",
      "224. Reward: 225.0, action: 1\n",
      "225. Reward: 226.0, action: 0\n",
      "226. Reward: 227.0, action: 0\n",
      "227. Reward: 228.0, action: 1\n",
      "228. Reward: 229.0, action: 0\n",
      "229. Reward: 230.0, action: 1\n",
      "230. Reward: 231.0, action: 0\n",
      "231. Reward: 232.0, action: 1\n",
      "232. Reward: 233.0, action: 0\n",
      "233. Reward: 234.0, action: 0\n",
      "234. Reward: 235.0, action: 1\n",
      "235. Reward: 236.0, action: 0\n",
      "236. Reward: 237.0, action: 1\n",
      "237. Reward: 238.0, action: 0\n",
      "238. Reward: 239.0, action: 1\n",
      "239. Reward: 240.0, action: 1\n",
      "240. Reward: 241.0, action: 0\n",
      "241. Reward: 242.0, action: 1\n",
      "242. Reward: 243.0, action: 0\n",
      "243. Reward: 244.0, action: 1\n",
      "244. Reward: 245.0, action: 0\n",
      "245. Reward: 246.0, action: 1\n",
      "246. Reward: 247.0, action: 1\n",
      "247. Reward: 248.0, action: 0\n",
      "248. Reward: 249.0, action: 1\n",
      "249. Reward: 250.0, action: 0\n",
      "250. Reward: 251.0, action: 1\n",
      "251. Reward: 252.0, action: 0\n",
      "252. Reward: 253.0, action: 1\n",
      "253. Reward: 254.0, action: 0\n",
      "254. Reward: 255.0, action: 1\n",
      "255. Reward: 256.0, action: 1\n",
      "256. Reward: 257.0, action: 0\n",
      "257. Reward: 258.0, action: 1\n",
      "258. Reward: 259.0, action: 0\n",
      "259. Reward: 260.0, action: 1\n",
      "260. Reward: 261.0, action: 0\n",
      "261. Reward: 262.0, action: 1\n",
      "262. Reward: 263.0, action: 0\n",
      "263. Reward: 264.0, action: 0\n",
      "264. Reward: 265.0, action: 1\n",
      "265. Reward: 266.0, action: 0\n",
      "266. Reward: 267.0, action: 1\n",
      "267. Reward: 268.0, action: 0\n",
      "268. Reward: 269.0, action: 1\n",
      "269. Reward: 270.0, action: 0\n",
      "270. Reward: 271.0, action: 1\n",
      "271. Reward: 272.0, action: 0\n",
      "272. Reward: 273.0, action: 1\n",
      "273. Reward: 274.0, action: 0\n",
      "274. Reward: 275.0, action: 0\n",
      "275. Reward: 276.0, action: 1\n",
      "276. Reward: 277.0, action: 0\n",
      "277. Reward: 278.0, action: 1\n",
      "278. Reward: 279.0, action: 0\n",
      "279. Reward: 280.0, action: 0\n",
      "280. Reward: 281.0, action: 1\n",
      "281. Reward: 282.0, action: 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "282. Reward: 283.0, action: 1\n",
      "283. Reward: 284.0, action: 0\n",
      "284. Reward: 285.0, action: 1\n",
      "285. Reward: 286.0, action: 1\n",
      "286. Reward: 287.0, action: 0\n",
      "287. Reward: 288.0, action: 1\n",
      "288. Reward: 289.0, action: 0\n",
      "289. Reward: 290.0, action: 1\n",
      "290. Reward: 291.0, action: 0\n",
      "291. Reward: 292.0, action: 1\n",
      "292. Reward: 293.0, action: 1\n",
      "293. Reward: 294.0, action: 0\n",
      "294. Reward: 295.0, action: 1\n",
      "295. Reward: 296.0, action: 0\n",
      "296. Reward: 297.0, action: 1\n",
      "297. Reward: 298.0, action: 0\n",
      "298. Reward: 299.0, action: 1\n",
      "299. Reward: 300.0, action: 0\n",
      "300. Reward: 301.0, action: 1\n",
      "301. Reward: 302.0, action: 1\n",
      "302. Reward: 303.0, action: 0\n",
      "303. Reward: 304.0, action: 1\n",
      "304. Reward: 305.0, action: 0\n",
      "305. Reward: 306.0, action: 1\n",
      "306. Reward: 307.0, action: 0\n",
      "307. Reward: 308.0, action: 1\n",
      "308. Reward: 309.0, action: 0\n",
      "309. Reward: 310.0, action: 0\n",
      "310. Reward: 311.0, action: 1\n",
      "311. Reward: 312.0, action: 0\n",
      "312. Reward: 313.0, action: 1\n",
      "313. Reward: 314.0, action: 0\n",
      "314. Reward: 315.0, action: 1\n",
      "315. Reward: 316.0, action: 0\n",
      "316. Reward: 317.0, action: 0\n",
      "317. Reward: 318.0, action: 1\n",
      "318. Reward: 319.0, action: 0\n",
      "319. Reward: 320.0, action: 1\n",
      "320. Reward: 321.0, action: 0\n",
      "321. Reward: 322.0, action: 0\n",
      "322. Reward: 323.0, action: 1\n",
      "323. Reward: 324.0, action: 0\n",
      "324. Reward: 325.0, action: 1\n",
      "325. Reward: 326.0, action: 0\n",
      "326. Reward: 327.0, action: 1\n",
      "327. Reward: 328.0, action: 1\n",
      "328. Reward: 329.0, action: 0\n",
      "329. Reward: 330.0, action: 1\n",
      "330. Reward: 331.0, action: 0\n",
      "331. Reward: 332.0, action: 1\n",
      "332. Reward: 333.0, action: 0\n",
      "333. Reward: 334.0, action: 1\n",
      "334. Reward: 335.0, action: 1\n",
      "335. Reward: 336.0, action: 0\n",
      "336. Reward: 337.0, action: 1\n",
      "337. Reward: 338.0, action: 1\n",
      "338. Reward: 339.0, action: 0\n",
      "339. Reward: 340.0, action: 1\n",
      "340. Reward: 341.0, action: 0\n",
      "341. Reward: 342.0, action: 1\n",
      "342. Reward: 343.0, action: 0\n",
      "343. Reward: 344.0, action: 1\n",
      "344. Reward: 345.0, action: 0\n",
      "345. Reward: 346.0, action: 1\n",
      "346. Reward: 347.0, action: 0\n",
      "347. Reward: 348.0, action: 0\n",
      "348. Reward: 349.0, action: 1\n",
      "349. Reward: 350.0, action: 0\n",
      "350. Reward: 351.0, action: 1\n",
      "351. Reward: 352.0, action: 0\n",
      "352. Reward: 353.0, action: 1\n",
      "353. Reward: 354.0, action: 0\n",
      "354. Reward: 355.0, action: 1\n",
      "355. Reward: 356.0, action: 0\n",
      "356. Reward: 357.0, action: 0\n",
      "357. Reward: 358.0, action: 0\n",
      "358. Reward: 359.0, action: 1\n",
      "359. Reward: 360.0, action: 0\n",
      "360. Reward: 361.0, action: 1\n",
      "361. Reward: 362.0, action: 0\n",
      "362. Reward: 363.0, action: 1\n",
      "363. Reward: 364.0, action: 0\n",
      "364. Reward: 365.0, action: 1\n",
      "365. Reward: 366.0, action: 1\n",
      "366. Reward: 367.0, action: 0\n",
      "367. Reward: 368.0, action: 1\n",
      "368. Reward: 369.0, action: 0\n",
      "369. Reward: 370.0, action: 1\n",
      "370. Reward: 371.0, action: 0\n",
      "371. Reward: 372.0, action: 1\n",
      "372. Reward: 373.0, action: 1\n",
      "373. Reward: 374.0, action: 0\n",
      "374. Reward: 375.0, action: 1\n",
      "375. Reward: 376.0, action: 1\n",
      "376. Reward: 377.0, action: 0\n",
      "377. Reward: 378.0, action: 1\n",
      "378. Reward: 379.0, action: 0\n",
      "379. Reward: 380.0, action: 1\n",
      "380. Reward: 381.0, action: 0\n",
      "381. Reward: 382.0, action: 1\n",
      "382. Reward: 383.0, action: 0\n",
      "383. Reward: 384.0, action: 1\n",
      "384. Reward: 385.0, action: 0\n",
      "385. Reward: 386.0, action: 0\n",
      "386. Reward: 387.0, action: 1\n",
      "387. Reward: 388.0, action: 0\n",
      "388. Reward: 389.0, action: 1\n",
      "389. Reward: 390.0, action: 0\n",
      "390. Reward: 391.0, action: 1\n",
      "391. Reward: 392.0, action: 0\n",
      "392. Reward: 393.0, action: 0\n",
      "393. Reward: 394.0, action: 0\n",
      "394. Reward: 395.0, action: 1\n",
      "395. Reward: 396.0, action: 0\n",
      "396. Reward: 397.0, action: 1\n",
      "397. Reward: 398.0, action: 0\n",
      "398. Reward: 399.0, action: 1\n",
      "399. Reward: 400.0, action: 0\n",
      "400. Reward: 401.0, action: 1\n",
      "401. Reward: 402.0, action: 1\n",
      "402. Reward: 403.0, action: 0\n",
      "403. Reward: 404.0, action: 1\n",
      "404. Reward: 405.0, action: 1\n",
      "405. Reward: 406.0, action: 0\n",
      "406. Reward: 407.0, action: 1\n",
      "407. Reward: 408.0, action: 0\n",
      "408. Reward: 409.0, action: 1\n",
      "409. Reward: 410.0, action: 1\n",
      "410. Reward: 411.0, action: 0\n",
      "411. Reward: 412.0, action: 1\n",
      "412. Reward: 413.0, action: 0\n",
      "413. Reward: 414.0, action: 1\n",
      "414. Reward: 415.0, action: 0\n",
      "415. Reward: 416.0, action: 1\n",
      "416. Reward: 417.0, action: 0\n",
      "417. Reward: 418.0, action: 1\n",
      "418. Reward: 419.0, action: 0\n",
      "419. Reward: 420.0, action: 0\n",
      "420. Reward: 421.0, action: 1\n",
      "421. Reward: 422.0, action: 0\n",
      "422. Reward: 423.0, action: 1\n",
      "423. Reward: 424.0, action: 0\n",
      "424. Reward: 425.0, action: 0\n",
      "425. Reward: 426.0, action: 0\n",
      "426. Reward: 427.0, action: 1\n",
      "427. Reward: 428.0, action: 0\n",
      "428. Reward: 429.0, action: 1\n",
      "429. Reward: 430.0, action: 0\n",
      "430. Reward: 431.0, action: 1\n",
      "431. Reward: 432.0, action: 0\n",
      "432. Reward: 433.0, action: 1\n",
      "433. Reward: 434.0, action: 0\n",
      "434. Reward: 435.0, action: 1\n",
      "435. Reward: 436.0, action: 1\n",
      "436. Reward: 437.0, action: 0\n",
      "437. Reward: 438.0, action: 1\n",
      "438. Reward: 439.0, action: 1\n",
      "439. Reward: 440.0, action: 0\n",
      "440. Reward: 441.0, action: 1\n",
      "441. Reward: 442.0, action: 1\n",
      "442. Reward: 443.0, action: 0\n",
      "443. Reward: 444.0, action: 1\n",
      "444. Reward: 445.0, action: 0\n",
      "445. Reward: 446.0, action: 1\n",
      "446. Reward: 447.0, action: 0\n",
      "447. Reward: 448.0, action: 1\n",
      "448. Reward: 449.0, action: 0\n",
      "449. Reward: 450.0, action: 1\n",
      "450. Reward: 451.0, action: 0\n",
      "451. Reward: 452.0, action: 1\n",
      "452. Reward: 453.0, action: 0\n",
      "453. Reward: 454.0, action: 0\n",
      "454. Reward: 455.0, action: 1\n",
      "455. Reward: 456.0, action: 0\n",
      "456. Reward: 457.0, action: 1\n",
      "457. Reward: 458.0, action: 0\n",
      "458. Reward: 459.0, action: 1\n",
      "459. Reward: 460.0, action: 0\n",
      "460. Reward: 461.0, action: 0\n",
      "461. Reward: 462.0, action: 0\n",
      "462. Reward: 463.0, action: 1\n",
      "463. Reward: 464.0, action: 0\n",
      "464. Reward: 465.0, action: 1\n",
      "465. Reward: 466.0, action: 0\n",
      "466. Reward: 467.0, action: 1\n",
      "467. Reward: 468.0, action: 0\n",
      "468. Reward: 469.0, action: 1\n",
      "469. Reward: 470.0, action: 0\n",
      "470. Reward: 471.0, action: 1\n",
      "471. Reward: 472.0, action: 1\n",
      "472. Reward: 473.0, action: 0\n",
      "473. Reward: 474.0, action: 1\n",
      "474. Reward: 475.0, action: 1\n",
      "475. Reward: 476.0, action: 0\n",
      "476. Reward: 477.0, action: 1\n",
      "477. Reward: 478.0, action: 1\n",
      "478. Reward: 479.0, action: 0\n",
      "479. Reward: 480.0, action: 1\n",
      "480. Reward: 481.0, action: 0\n",
      "481. Reward: 482.0, action: 1\n",
      "482. Reward: 483.0, action: 0\n",
      "483. Reward: 484.0, action: 1\n",
      "484. Reward: 485.0, action: 0\n",
      "485. Reward: 486.0, action: 1\n",
      "486. Reward: 487.0, action: 0\n",
      "487. Reward: 488.0, action: 1\n",
      "488. Reward: 489.0, action: 0\n",
      "489. Reward: 490.0, action: 0\n",
      "490. Reward: 491.0, action: 1\n",
      "491. Reward: 492.0, action: 0\n",
      "492. Reward: 493.0, action: 0\n",
      "493. Reward: 494.0, action: 0\n",
      "494. Reward: 495.0, action: 1\n",
      "495. Reward: 496.0, action: 0\n",
      "496. Reward: 497.0, action: 1\n",
      "497. Reward: 498.0, action: 0\n",
      "498. Reward: 499.0, action: 1\n",
      "499. Reward: 500.0, action: 0\n",
      "500. Reward: 501.0, action: 1\n",
      "501. Reward: 502.0, action: 0\n",
      "502. Reward: 503.0, action: 1\n",
      "503. Reward: 504.0, action: 1\n",
      "504. Reward: 505.0, action: 1\n",
      "505. Reward: 506.0, action: 0\n",
      "506. Reward: 507.0, action: 1\n",
      "507. Reward: 508.0, action: 1\n",
      "508. Reward: 509.0, action: 0\n",
      "509. Reward: 510.0, action: 1\n",
      "510. Reward: 511.0, action: 0\n",
      "511. Reward: 512.0, action: 1\n",
      "512. Reward: 513.0, action: 0\n",
      "513. Reward: 514.0, action: 1\n",
      "514. Reward: 515.0, action: 0\n",
      "515. Reward: 516.0, action: 1\n",
      "516. Reward: 517.0, action: 0\n",
      "517. Reward: 518.0, action: 1\n",
      "518. Reward: 519.0, action: 0\n",
      "519. Reward: 520.0, action: 0\n",
      "520. Reward: 521.0, action: 1\n",
      "521. Reward: 522.0, action: 0\n",
      "522. Reward: 523.0, action: 1\n",
      "523. Reward: 524.0, action: 0\n",
      "524. Reward: 525.0, action: 1\n",
      "525. Reward: 526.0, action: 0\n",
      "526. Reward: 527.0, action: 0\n",
      "527. Reward: 528.0, action: 0\n",
      "528. Reward: 529.0, action: 1\n",
      "529. Reward: 530.0, action: 0\n",
      "530. Reward: 531.0, action: 1\n",
      "531. Reward: 532.0, action: 0\n",
      "532. Reward: 533.0, action: 1\n",
      "533. Reward: 534.0, action: 0\n",
      "534. Reward: 535.0, action: 1\n",
      "535. Reward: 536.0, action: 0\n",
      "536. Reward: 537.0, action: 1\n",
      "537. Reward: 538.0, action: 0\n",
      "538. Reward: 539.0, action: 1\n",
      "539. Reward: 540.0, action: 1\n",
      "540. Reward: 541.0, action: 0\n",
      "541. Reward: 542.0, action: 1\n",
      "542. Reward: 543.0, action: 1\n",
      "543. Reward: 544.0, action: 1\n",
      "544. Reward: 545.0, action: 0\n",
      "545. Reward: 546.0, action: 1\n",
      "546. Reward: 547.0, action: 0\n",
      "547. Reward: 548.0, action: 1\n",
      "548. Reward: 549.0, action: 0\n",
      "549. Reward: 550.0, action: 1\n",
      "550. Reward: 551.0, action: 0\n",
      "551. Reward: 552.0, action: 1\n",
      "552. Reward: 553.0, action: 0\n",
      "553. Reward: 554.0, action: 1\n",
      "554. Reward: 555.0, action: 0\n",
      "555. Reward: 556.0, action: 1\n",
      "556. Reward: 557.0, action: 0\n",
      "557. Reward: 558.0, action: 1\n",
      "558. Reward: 559.0, action: 0\n",
      "559. Reward: 560.0, action: 1\n",
      "560. Reward: 561.0, action: 0\n",
      "561. Reward: 562.0, action: 0\n",
      "562. Reward: 563.0, action: 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "563. Reward: 564.0, action: 0\n",
      "564. Reward: 565.0, action: 0\n",
      "565. Reward: 566.0, action: 0\n",
      "566. Reward: 567.0, action: 1\n",
      "567. Reward: 568.0, action: 0\n",
      "568. Reward: 569.0, action: 1\n",
      "569. Reward: 570.0, action: 0\n",
      "570. Reward: 571.0, action: 1\n",
      "571. Reward: 572.0, action: 0\n",
      "572. Reward: 573.0, action: 1\n",
      "573. Reward: 574.0, action: 0\n",
      "574. Reward: 575.0, action: 1\n",
      "575. Reward: 576.0, action: 1\n",
      "576. Reward: 577.0, action: 1\n",
      "577. Reward: 578.0, action: 0\n",
      "578. Reward: 579.0, action: 1\n",
      "579. Reward: 580.0, action: 1\n",
      "580. Reward: 581.0, action: 0\n",
      "581. Reward: 582.0, action: 1\n",
      "582. Reward: 583.0, action: 0\n",
      "583. Reward: 584.0, action: 1\n",
      "584. Reward: 585.0, action: 0\n",
      "585. Reward: 586.0, action: 1\n",
      "586. Reward: 587.0, action: 0\n",
      "587. Reward: 588.0, action: 1\n",
      "588. Reward: 589.0, action: 0\n",
      "589. Reward: 590.0, action: 1\n",
      "590. Reward: 591.0, action: 0\n",
      "591. Reward: 592.0, action: 0\n",
      "592. Reward: 593.0, action: 1\n",
      "593. Reward: 594.0, action: 0\n",
      "594. Reward: 595.0, action: 1\n",
      "595. Reward: 596.0, action: 0\n",
      "596. Reward: 597.0, action: 0\n",
      "597. Reward: 598.0, action: 0\n",
      "598. Reward: 599.0, action: 1\n",
      "599. Reward: 600.0, action: 0\n",
      "600. Reward: 601.0, action: 1\n",
      "601. Reward: 602.0, action: 0\n",
      "602. Reward: 603.0, action: 1\n",
      "603. Reward: 604.0, action: 0\n",
      "604. Reward: 605.0, action: 1\n",
      "605. Reward: 606.0, action: 0\n",
      "606. Reward: 607.0, action: 1\n",
      "607. Reward: 608.0, action: 0\n",
      "608. Reward: 609.0, action: 1\n",
      "609. Reward: 610.0, action: 1\n",
      "610. Reward: 611.0, action: 0\n",
      "611. Reward: 612.0, action: 1\n",
      "612. Reward: 613.0, action: 1\n",
      "613. Reward: 614.0, action: 1\n",
      "614. Reward: 615.0, action: 0\n",
      "615. Reward: 616.0, action: 1\n",
      "616. Reward: 617.0, action: 0\n",
      "617. Reward: 618.0, action: 1\n",
      "618. Reward: 619.0, action: 0\n",
      "619. Reward: 620.0, action: 1\n",
      "620. Reward: 621.0, action: 0\n",
      "621. Reward: 622.0, action: 1\n",
      "622. Reward: 623.0, action: 0\n",
      "623. Reward: 624.0, action: 1\n",
      "624. Reward: 625.0, action: 0\n",
      "625. Reward: 626.0, action: 1\n",
      "626. Reward: 627.0, action: 0\n",
      "627. Reward: 628.0, action: 1\n",
      "628. Reward: 629.0, action: 0\n",
      "629. Reward: 630.0, action: 0\n",
      "630. Reward: 631.0, action: 0\n",
      "631. Reward: 632.0, action: 0\n",
      "632. Reward: 633.0, action: 1\n",
      "633. Reward: 634.0, action: 0\n",
      "634. Reward: 635.0, action: 1\n",
      "635. Reward: 636.0, action: 0\n",
      "636. Reward: 637.0, action: 1\n",
      "637. Reward: 638.0, action: 0\n",
      "638. Reward: 639.0, action: 1\n",
      "639. Reward: 640.0, action: 0\n",
      "640. Reward: 641.0, action: 1\n",
      "641. Reward: 642.0, action: 1\n",
      "642. Reward: 643.0, action: 1\n",
      "643. Reward: 644.0, action: 1\n",
      "644. Reward: 645.0, action: 0\n",
      "645. Reward: 646.0, action: 1\n",
      "646. Reward: 647.0, action: 0\n",
      "647. Reward: 648.0, action: 1\n",
      "648. Reward: 649.0, action: 0\n",
      "649. Reward: 650.0, action: 1\n",
      "650. Reward: 651.0, action: 0\n",
      "651. Reward: 652.0, action: 1\n",
      "652. Reward: 653.0, action: 0\n",
      "653. Reward: 654.0, action: 0\n",
      "654. Reward: 655.0, action: 1\n",
      "655. Reward: 656.0, action: 0\n",
      "656. Reward: 657.0, action: 1\n",
      "657. Reward: 658.0, action: 0\n",
      "658. Reward: 659.0, action: 1\n",
      "659. Reward: 660.0, action: 0\n",
      "660. Reward: 661.0, action: 0\n",
      "661. Reward: 662.0, action: 0\n",
      "662. Reward: 663.0, action: 1\n",
      "663. Reward: 664.0, action: 0\n",
      "664. Reward: 665.0, action: 1\n",
      "665. Reward: 666.0, action: 0\n",
      "666. Reward: 667.0, action: 1\n",
      "667. Reward: 668.0, action: 0\n",
      "668. Reward: 669.0, action: 1\n",
      "669. Reward: 670.0, action: 0\n",
      "670. Reward: 671.0, action: 1\n",
      "671. Reward: 672.0, action: 0\n",
      "672. Reward: 673.0, action: 1\n",
      "673. Reward: 674.0, action: 1\n",
      "674. Reward: 675.0, action: 1\n",
      "675. Reward: 676.0, action: 1\n",
      "676. Reward: 677.0, action: 0\n",
      "677. Reward: 678.0, action: 1\n",
      "678. Reward: 679.0, action: 0\n",
      "679. Reward: 680.0, action: 1\n",
      "680. Reward: 681.0, action: 0\n",
      "681. Reward: 682.0, action: 1\n",
      "682. Reward: 683.0, action: 0\n",
      "683. Reward: 684.0, action: 1\n",
      "684. Reward: 685.0, action: 0\n",
      "685. Reward: 686.0, action: 1\n",
      "686. Reward: 687.0, action: 0\n",
      "687. Reward: 688.0, action: 1\n",
      "688. Reward: 689.0, action: 0\n",
      "689. Reward: 690.0, action: 1\n",
      "690. Reward: 691.0, action: 0\n",
      "691. Reward: 692.0, action: 0\n",
      "692. Reward: 693.0, action: 1\n",
      "693. Reward: 694.0, action: 0\n",
      "694. Reward: 695.0, action: 0\n",
      "695. Reward: 696.0, action: 0\n",
      "696. Reward: 697.0, action: 1\n",
      "697. Reward: 698.0, action: 0\n",
      "698. Reward: 699.0, action: 1\n",
      "699. Reward: 700.0, action: 0\n",
      "700. Reward: 701.0, action: 1\n",
      "701. Reward: 702.0, action: 0\n",
      "702. Reward: 703.0, action: 1\n",
      "703. Reward: 704.0, action: 0\n",
      "704. Reward: 705.0, action: 1\n",
      "705. Reward: 706.0, action: 0\n",
      "706. Reward: 707.0, action: 1\n",
      "707. Reward: 708.0, action: 0\n",
      "708. Reward: 709.0, action: 1\n",
      "709. Reward: 710.0, action: 1\n",
      "710. Reward: 711.0, action: 1\n",
      "711. Reward: 712.0, action: 1\n",
      "712. Reward: 713.0, action: 0\n",
      "713. Reward: 714.0, action: 1\n",
      "714. Reward: 715.0, action: 0\n",
      "715. Reward: 716.0, action: 1\n",
      "716. Reward: 717.0, action: 0\n",
      "717. Reward: 718.0, action: 1\n",
      "718. Reward: 719.0, action: 0\n",
      "719. Reward: 720.0, action: 1\n",
      "720. Reward: 721.0, action: 0\n",
      "721. Reward: 722.0, action: 1\n",
      "722. Reward: 723.0, action: 0\n",
      "723. Reward: 724.0, action: 1\n",
      "724. Reward: 725.0, action: 0\n",
      "725. Reward: 726.0, action: 1\n",
      "726. Reward: 727.0, action: 0\n",
      "727. Reward: 728.0, action: 1\n",
      "728. Reward: 729.0, action: 0\n",
      "729. Reward: 730.0, action: 0\n",
      "730. Reward: 731.0, action: 0\n",
      "731. Reward: 732.0, action: 0\n",
      "732. Reward: 733.0, action: 1\n",
      "733. Reward: 734.0, action: 0\n",
      "734. Reward: 735.0, action: 1\n",
      "735. Reward: 736.0, action: 0\n",
      "736. Reward: 737.0, action: 1\n",
      "737. Reward: 738.0, action: 0\n",
      "738. Reward: 739.0, action: 1\n",
      "739. Reward: 740.0, action: 0\n",
      "740. Reward: 741.0, action: 1\n",
      "741. Reward: 742.0, action: 0\n",
      "742. Reward: 743.0, action: 1\n",
      "743. Reward: 744.0, action: 1\n",
      "744. Reward: 745.0, action: 1\n",
      "745. Reward: 746.0, action: 1\n",
      "746. Reward: 747.0, action: 0\n",
      "747. Reward: 748.0, action: 1\n",
      "748. Reward: 749.0, action: 0\n",
      "749. Reward: 750.0, action: 1\n",
      "750. Reward: 751.0, action: 0\n",
      "751. Reward: 752.0, action: 1\n",
      "752. Reward: 753.0, action: 0\n",
      "753. Reward: 754.0, action: 1\n",
      "754. Reward: 755.0, action: 0\n",
      "755. Reward: 756.0, action: 1\n",
      "756. Reward: 757.0, action: 0\n",
      "757. Reward: 758.0, action: 1\n",
      "758. Reward: 759.0, action: 0\n",
      "759. Reward: 760.0, action: 1\n",
      "760. Reward: 761.0, action: 0\n",
      "761. Reward: 762.0, action: 0\n",
      "762. Reward: 763.0, action: 1\n",
      "763. Reward: 764.0, action: 0\n",
      "764. Reward: 765.0, action: 0\n",
      "765. Reward: 766.0, action: 0\n",
      "766. Reward: 767.0, action: 1\n",
      "767. Reward: 768.0, action: 0\n",
      "768. Reward: 769.0, action: 1\n",
      "769. Reward: 770.0, action: 0\n",
      "770. Reward: 771.0, action: 1\n",
      "771. Reward: 772.0, action: 0\n",
      "772. Reward: 773.0, action: 1\n",
      "773. Reward: 774.0, action: 0\n",
      "774. Reward: 775.0, action: 1\n",
      "775. Reward: 776.0, action: 0\n",
      "776. Reward: 777.0, action: 1\n",
      "777. Reward: 778.0, action: 1\n",
      "778. Reward: 779.0, action: 0\n",
      "779. Reward: 780.0, action: 1\n",
      "780. Reward: 781.0, action: 1\n",
      "781. Reward: 782.0, action: 1\n",
      "782. Reward: 783.0, action: 0\n",
      "783. Reward: 784.0, action: 1\n",
      "784. Reward: 785.0, action: 0\n",
      "785. Reward: 786.0, action: 1\n",
      "786. Reward: 787.0, action: 0\n",
      "787. Reward: 788.0, action: 1\n",
      "788. Reward: 789.0, action: 0\n",
      "789. Reward: 790.0, action: 1\n",
      "790. Reward: 791.0, action: 0\n",
      "791. Reward: 792.0, action: 1\n",
      "792. Reward: 793.0, action: 0\n",
      "793. Reward: 794.0, action: 1\n",
      "794. Reward: 795.0, action: 0\n",
      "795. Reward: 796.0, action: 0\n",
      "796. Reward: 797.0, action: 1\n",
      "797. Reward: 798.0, action: 0\n",
      "798. Reward: 799.0, action: 0\n",
      "799. Reward: 800.0, action: 0\n",
      "800. Reward: 801.0, action: 1\n",
      "801. Reward: 802.0, action: 0\n",
      "802. Reward: 803.0, action: 1\n",
      "803. Reward: 804.0, action: 0\n",
      "804. Reward: 805.0, action: 1\n",
      "805. Reward: 806.0, action: 0\n",
      "806. Reward: 807.0, action: 1\n",
      "807. Reward: 808.0, action: 0\n",
      "808. Reward: 809.0, action: 1\n",
      "809. Reward: 810.0, action: 0\n",
      "810. Reward: 811.0, action: 1\n",
      "811. Reward: 812.0, action: 1\n",
      "812. Reward: 813.0, action: 0\n",
      "813. Reward: 814.0, action: 1\n",
      "814. Reward: 815.0, action: 1\n",
      "815. Reward: 816.0, action: 1\n",
      "816. Reward: 817.0, action: 0\n",
      "817. Reward: 818.0, action: 1\n",
      "818. Reward: 819.0, action: 0\n",
      "819. Reward: 820.0, action: 1\n",
      "820. Reward: 821.0, action: 0\n",
      "821. Reward: 822.0, action: 1\n",
      "822. Reward: 823.0, action: 0\n",
      "823. Reward: 824.0, action: 1\n",
      "824. Reward: 825.0, action: 0\n",
      "825. Reward: 826.0, action: 1\n",
      "826. Reward: 827.0, action: 0\n",
      "827. Reward: 828.0, action: 1\n",
      "828. Reward: 829.0, action: 0\n",
      "829. Reward: 830.0, action: 0\n",
      "830. Reward: 831.0, action: 1\n",
      "831. Reward: 832.0, action: 0\n",
      "832. Reward: 833.0, action: 1\n",
      "833. Reward: 834.0, action: 0\n",
      "834. Reward: 835.0, action: 0\n",
      "835. Reward: 836.0, action: 0\n",
      "836. Reward: 837.0, action: 1\n",
      "837. Reward: 838.0, action: 0\n",
      "838. Reward: 839.0, action: 1\n",
      "839. Reward: 840.0, action: 0\n",
      "840. Reward: 841.0, action: 1\n",
      "841. Reward: 842.0, action: 0\n",
      "842. Reward: 843.0, action: 1\n",
      "843. Reward: 844.0, action: 0\n",
      "844. Reward: 845.0, action: 1\n",
      "845. Reward: 846.0, action: 0\n",
      "846. Reward: 847.0, action: 1\n",
      "847. Reward: 848.0, action: 1\n",
      "848. Reward: 849.0, action: 0\n",
      "849. Reward: 850.0, action: 1\n",
      "850. Reward: 851.0, action: 1\n",
      "851. Reward: 852.0, action: 1\n",
      "852. Reward: 853.0, action: 0\n",
      "853. Reward: 854.0, action: 1\n",
      "854. Reward: 855.0, action: 0\n",
      "855. Reward: 856.0, action: 1\n",
      "856. Reward: 857.0, action: 0\n",
      "857. Reward: 858.0, action: 1\n",
      "858. Reward: 859.0, action: 0\n",
      "859. Reward: 860.0, action: 1\n",
      "860. Reward: 861.0, action: 0\n",
      "861. Reward: 862.0, action: 1\n",
      "862. Reward: 863.0, action: 0\n",
      "863. Reward: 864.0, action: 1\n",
      "864. Reward: 865.0, action: 0\n",
      "865. Reward: 866.0, action: 1\n",
      "866. Reward: 867.0, action: 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "867. Reward: 868.0, action: 0\n",
      "868. Reward: 869.0, action: 1\n",
      "869. Reward: 870.0, action: 0\n",
      "870. Reward: 871.0, action: 0\n",
      "871. Reward: 872.0, action: 0\n",
      "872. Reward: 873.0, action: 1\n",
      "873. Reward: 874.0, action: 0\n",
      "874. Reward: 875.0, action: 1\n",
      "875. Reward: 876.0, action: 0\n",
      "876. Reward: 877.0, action: 0\n",
      "877. Reward: 878.0, action: 1\n",
      "878. Reward: 879.0, action: 0\n",
      "879. Reward: 880.0, action: 1\n",
      "880. Reward: 881.0, action: 0\n",
      "881. Reward: 882.0, action: 1\n",
      "882. Reward: 883.0, action: 0\n",
      "883. Reward: 884.0, action: 1\n",
      "884. Reward: 885.0, action: 1\n",
      "885. Reward: 886.0, action: 1\n",
      "886. Reward: 887.0, action: 1\n",
      "887. Reward: 888.0, action: 1\n",
      "888. Reward: 889.0, action: 0\n",
      "889. Reward: 890.0, action: 1\n",
      "890. Reward: 891.0, action: 1\n",
      "891. Reward: 892.0, action: 0\n",
      "892. Reward: 893.0, action: 1\n",
      "893. Reward: 894.0, action: 0\n",
      "894. Reward: 895.0, action: 1\n",
      "895. Reward: 896.0, action: 0\n",
      "896. Reward: 897.0, action: 1\n",
      "897. Reward: 898.0, action: 0\n",
      "898. Reward: 899.0, action: 1\n",
      "899. Reward: 900.0, action: 0\n",
      "900. Reward: 901.0, action: 1\n",
      "901. Reward: 902.0, action: 0\n",
      "902. Reward: 903.0, action: 0\n",
      "903. Reward: 904.0, action: 1\n",
      "904. Reward: 905.0, action: 0\n",
      "905. Reward: 906.0, action: 1\n",
      "906. Reward: 907.0, action: 0\n",
      "907. Reward: 908.0, action: 1\n",
      "908. Reward: 909.0, action: 0\n",
      "909. Reward: 910.0, action: 0\n",
      "910. Reward: 911.0, action: 0\n",
      "911. Reward: 912.0, action: 0\n",
      "912. Reward: 913.0, action: 1\n",
      "913. Reward: 914.0, action: 0\n",
      "914. Reward: 915.0, action: 1\n",
      "915. Reward: 916.0, action: 0\n",
      "916. Reward: 917.0, action: 1\n",
      "917. Reward: 918.0, action: 0\n",
      "918. Reward: 919.0, action: 1\n",
      "919. Reward: 920.0, action: 0\n",
      "920. Reward: 921.0, action: 1\n",
      "921. Reward: 922.0, action: 0\n",
      "922. Reward: 923.0, action: 1\n",
      "923. Reward: 924.0, action: 0\n",
      "924. Reward: 925.0, action: 1\n",
      "925. Reward: 926.0, action: 1\n",
      "926. Reward: 927.0, action: 1\n",
      "927. Reward: 928.0, action: 1\n",
      "928. Reward: 929.0, action: 0\n",
      "929. Reward: 930.0, action: 1\n",
      "930. Reward: 931.0, action: 0\n",
      "931. Reward: 932.0, action: 1\n",
      "932. Reward: 933.0, action: 0\n",
      "933. Reward: 934.0, action: 1\n",
      "934. Reward: 935.0, action: 0\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-7-be4a4a4a254b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmaster\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mplay\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-3-ea075ba1afb6>\u001b[0m in \u001b[0;36mplay\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     77\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     78\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 79\u001b[0;31m             \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'rgb_array'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     80\u001b[0m             policy, value = model(\n\u001b[1;32m     81\u001b[0m                 tf.convert_to_tensor(state[None, :], dtype=tf.float32))\n",
      "\u001b[0;32m~/venv3/lib/python3.6/site-packages/gym/envs/classic_control/cartpole.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, mode)\u001b[0m\n\u001b[1;32m    186\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpoletrans\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_rotation\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    187\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 188\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mviewer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreturn_rgb_array\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m==\u001b[0m\u001b[0;34m'rgb_array'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    189\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    190\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/venv3/lib/python3.6/site-packages/gym/envs/classic_control/rendering.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, return_rgb_array)\u001b[0m\n\u001b[1;32m    103\u001b[0m             \u001b[0mbuffer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpyglet\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mimage\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_buffer_manager\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_color_buffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    104\u001b[0m             \u001b[0mimage_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbuffer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_image_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 105\u001b[0;31m             \u001b[0marr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfrombuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mimage_data\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0muint8\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    106\u001b[0m             \u001b[0;31m# In https://github.com/openai/gym-http-api/issues/2, we\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    107\u001b[0m             \u001b[0;31m# discovered that someone using Xmonad on Arch was having\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/venv3/lib/python3.6/site-packages/pyglet/image/__init__.py\u001b[0m in \u001b[0;36mdata\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    738\u001b[0m         \u001b[0;34m:\u001b[0m\u001b[0mtype\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0msequence\u001b[0m \u001b[0mof\u001b[0m \u001b[0mbytes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    739\u001b[0m         \"\"\"\n\u001b[0;32m--> 740\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    741\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    742\u001b[0m     \u001b[0;34m@\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetter\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/venv3/lib/python3.6/site-packages/pyglet/image/__init__.py\u001b[0m in \u001b[0;36m_get_data\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    720\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_current_pitch\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpitch\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    721\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 722\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_ensure_string_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    723\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_current_data\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    724\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "master.play()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}