cartPole/cartPole3DoubleDQN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **CartPole-v1 (maximum 500 steps)**\n",
    "#### Double Neural Network With Experience Replay (uses a Target Network to improve stability as Deep Mind did with Atari)\n",
    "#### The updating of the Target Network is carried out less frequently. The online model gives us the action the agent is going to take next and the target network gives us the score of that action. This method helps avoid overestimates.\n",
    "#### Two types of target-network-weights updating were compared: full updating and soft updating (tau = 0.5)\n",
    "#### The input to the network consists of 4 float values (Cart Position, Cart Velocity, Pole Angle and Pole Velocity at the Tip).\n",
    "#### The outputs of the network don't go through any activation function; instead of probabilities, the outputs are directly state-action values.\n",
    "##### (The optimal policy involves choosing the Argmax of the output vector of the Neural Network as the action for the next step).\n",
    "##### **This environment is considered to be solved when getting an average of 475 over 100 consecutive episodes.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oqXU0xmddNLk"
   },
   "outputs": [],
   "source": [
    "# Imports the libraries and creates the model\n",
    "# Hyperparameter tuning was carried out for the model (learning_rate, optimizers, batch_size, etc.) and for the learning loop (number of replay steps, discount factor, epsilon_decay, etc.)\n",
    "\n",
    "\n",
    "import random\n",
    "import numpy as np\n",
    "from tensorflow import keras\n",
    "import gym\n",
    "env = gym.make('CartPole-v1')\n",
    "model = keras.Sequential()\n",
    "model.add (keras.layers.Dense(units = 32, input_shape=(4,), activation = \"relu\"))\n",
    "model.add (keras.layers.Dense(units = 32, activation = \"relu\"))\n",
    "model.add (keras.layers.Dense(units = 8, activation = \"relu\"))\n",
    "model.add (keras.layers.Dense(units = 8, activation = \"relu\"))\n",
    "model.add (keras.layers.Dense(units = 2))\n",
    "model.compile(optimizer=keras.optimizers.Adam(lr=0.001),loss =\"mse\")\n",
    "#print (model.summary())\n",
    "\n",
    "# At first, the two models have the same parameters; however, the static model (target network) will be updated less frequently\n",
    "\n",
    "staticModel = keras.models.clone_model( model, input_tensors=None, clone_function=None)\n",
    "#print (staticModel.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_9ZHkM86M-NI"
   },
   "outputs": [],
   "source": [
    "# Memory Replay is carried out in this function.\n",
    "# 512 random previous experiences are \"remembered\" and processed in each step with a batch size of 128\n",
    "# This way we can prevent overfitting most recent states\n",
    "# This function has been vectorised with numpy \"Put_along_axis\" and \"Take_along_axis\" functions (see comment down below) to avoid a slow for loop.\n",
    "\n",
    "def executeMemoryReplayStep(memoryPreviousStates, memoryStates, memoryARD, model, staticModel, itemsInMemory):\n",
    "    randomList = np.random.permutation(itemsInMemory) # After that number everything is 0\n",
    "    \n",
    "    memoryPreviousStates = memoryPreviousStates[randomList[:numberOfLearningFromReplaySteps]]\n",
    "    memoryStates = memoryStates[randomList[:numberOfLearningFromReplaySteps]]\n",
    "    memoryARD = memoryARD[randomList[:numberOfLearningFromReplaySteps]]\n",
    "\n",
    "    \n",
    "    \n",
    "    previousState = memoryPreviousStates\n",
    "    \n",
    "    state = memoryStates\n",
    "    \n",
    "    \n",
    "    action, reward, done = np.split(memoryARD,3,axis=1)\n",
    "    \n",
    "    action = action.astype(int)\n",
    "    \n",
    "    target = model.predict(previousState)\n",
    "    \n",
    "    # Np.put_along_axis changes the value for the action selected in \"action\" into the \"target\" array.\n",
    "    # I.e., for each row, one of the two columns are modified, these indexes (0 or 1) for each row appear in the \"action\" array\n",
    "    # The individual updates are those of the Q-learning algorithm: only the reward obtained if done=False=0 and the reward plus the discounted value of the next state if done=True=1\n",
    "    # However, given than now we have two networks, the online network will still use Argmax to choose the action to do in each future state but the value of such action is given by the staticModel instead.\n",
    "    # This way, with Np.Take_long_axis, we take from the staticModelPredictions those indexes that correspond to the argmax of the predictions of the onlineModel for each future state. I.e, online model chooses an action and we take that index from the predictions of the static model for each future state:\n",
    "    np.put_along_axis(target, action, (reward + (1-done)*discount*np.take_along_axis(staticModel.predict(state),np.expand_dims(np.argmax(model.predict(state), axis=1),axis=1),axis=1)), axis=1)\n",
    "    \n",
    "    \n",
    "    targetArray = np.array(target)\n",
    "    \n",
    "    model.fit(memoryPreviousStates, targetArray, epochs=1, verbose = 0, batch_size =128)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Trial 1: The static model is fully updated every 500 steps\n",
    "#### (The static weights are replaced with those of the online model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "y48ruy6h9tcg",
    "outputId": "bb854b91-1f6e-41e1-bbf7-d97418252b75"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WEIGHTS SAVED\n",
      "0  Episode Reward:  32.0 .Epsilon: 0.995\n",
      "1  Episode Reward:  14.0 .Epsilon: 0.990025\n",
      "2  Episode Reward:  17.0 .Epsilon: 0.985074875\n",
      "3  Episode Reward:  16.0 .Epsilon: 0.9801495006250001\n",
      "4  Episode Reward:  12.0 .Epsilon: 0.9752487531218751\n",
      "5  Episode Reward:  40.0 .Epsilon: 0.9703725093562657\n",
      "6  Episode Reward:  16.0 .Epsilon: 0.9655206468094844\n",
      "7  Episode Reward:  11.0 .Epsilon: 0.960693043575437\n",
      "8  Episode Reward:  48.0 .Epsilon: 0.9558895783575597\n",
      "9  Episode Reward:  21.0 .Epsilon: 0.9511101304657719\n",
      "10  Episode Reward:  15.0 .Epsilon: 0.946354579813443\n",
      "11  Episode Reward:  14.0 .Epsilon: 0.9416228069143757\n",
      "12  Episode Reward:  21.0 .Epsilon: 0.9369146928798039\n",
      "13  Episode Reward:  12.0 .Epsilon: 0.9322301194154049\n",
      "14  Episode Reward:  20.0 .Epsilon: 0.9275689688183278\n",
      "15  Episode Reward:  26.0 .Epsilon: 0.9229311239742362\n",
      "16  Episode Reward:  15.0 .Epsilon: 0.918316468354365\n",
      "17  Episode Reward:  22.0 .Epsilon: 0.9137248860125932\n",
      "18  Episode Reward:  9.0 .Epsilon: 0.9091562615825302\n",
      "19  Episode Reward:  21.0 .Epsilon: 0.9046104802746175\n",
      "20  Episode Reward:  32.0 .Epsilon: 0.9000874278732445\n",
      "21  Episode Reward:  52.0 .Epsilon: 0.8955869907338783\n",
      "22  Episode Reward:  12.0 .Epsilon: 0.8911090557802088\n",
      "WEIGHTS SAVED\n",
      "23  Episode Reward:  45.0 .Epsilon: 0.8866535105013078\n",
      "24  Episode Reward:  36.0 .Epsilon: 0.8822202429488013\n",
      "25  Episode Reward:  13.0 .Epsilon: 0.8778091417340573\n",
      "26  Episode Reward:  22.0 .Epsilon: 0.8734200960253871\n",
      "27  Episode Reward:  13.0 .Epsilon: 0.8690529955452602\n",
      "28  Episode Reward:  14.0 .Epsilon: 0.8647077305675338\n",
      "29  Episode Reward:  14.0 .Epsilon: 0.8603841919146962\n",
      "30  Episode Reward:  17.0 .Epsilon: 0.8560822709551227\n",
      "31  Episode Reward:  15.0 .Epsilon: 0.851801859600347\n",
      "32  Episode Reward:  12.0 .Epsilon: 0.8475428503023453\n",
      "33  Episode Reward:  24.0 .Epsilon: 0.8433051360508336\n",
      "34  Episode Reward:  29.0 .Epsilon: 0.8390886103705794\n",
      "35  Episode Reward:  66.0 .Epsilon: 0.8348931673187264\n",
      "36  Episode Reward:  14.0 .Epsilon: 0.8307187014821328\n",
      "37  Episode Reward:  18.0 .Epsilon: 0.8265651079747222\n",
      "38  Episode Reward:  9.0 .Epsilon: 0.8224322824348486\n",
      "39  Episode Reward:  14.0 .Epsilon: 0.8183201210226743\n",
      "40  Episode Reward:  46.0 .Epsilon: 0.8142285204175609\n",
      "41  Episode Reward:  11.0 .Epsilon: 0.810157377815473\n",
      "42  Episode Reward:  28.0 .Epsilon: 0.8061065909263957\n",
      "43  Episode Reward:  12.0 .Epsilon: 0.8020760579717637\n",
      "44  Episode Reward:  12.0 .Epsilon: 0.798065677681905\n",
      "45  Episode Reward:  12.0 .Epsilon: 0.7940753492934954\n",
      "WEIGHTS SAVED\n",
      "46  Episode Reward:  16.0 .Epsilon: 0.7901049725470279\n",
      "47  Episode Reward:  11.0 .Epsilon: 0.7861544476842928\n",
      "48  Episode Reward:  52.0 .Epsilon: 0.7822236754458713\n",
      "49  Episode Reward:  27.0 .Epsilon: 0.778312557068642\n",
      "50  Episode Reward:  16.0 .Epsilon: 0.7744209942832988\n",
      "51  Episode Reward:  11.0 .Epsilon: 0.7705488893118823\n",
      "52  Episode Reward:  14.0 .Epsilon: 0.7666961448653229\n",
      "53  Episode Reward:  9.0 .Epsilon: 0.7628626641409962\n",
      "54  Episode Reward:  11.0 .Epsilon: 0.7590483508202912\n",
      "55  Episode Reward:  15.0 .Epsilon: 0.7552531090661897\n",
      "56  Episode Reward:  24.0 .Epsilon: 0.7514768435208588\n",
      "57  Episode Reward:  10.0 .Epsilon: 0.7477194593032545\n",
      "58  Episode Reward:  13.0 .Epsilon: 0.7439808620067382\n",
      "59  Episode Reward:  13.0 .Epsilon: 0.7402609576967045\n",
      "60  Episode Reward:  13.0 .Epsilon: 0.736559652908221\n",
      "61  Episode Reward:  17.0 .Epsilon: 0.7328768546436799\n",
      "62  Episode Reward:  17.0 .Epsilon: 0.7292124703704616\n",
      "63  Episode Reward:  14.0 .Epsilon: 0.7255664080186093\n",
      "64  Episode Reward:  16.0 .Epsilon: 0.7219385759785162\n",
      "65  Episode Reward:  25.0 .Epsilon: 0.7183288830986236\n",
      "66  Episode Reward:  10.0 .Epsilon: 0.7147372386831305\n",
      "67  Episode Reward:  17.0 .Epsilon: 0.7111635524897149\n",
      "68  Episode Reward:  10.0 .Epsilon: 0.7076077347272662\n",
      "69  Episode Reward:  21.0 .Epsilon: 0.7040696960536299\n",
      "70  Episode Reward:  17.0 .Epsilon: 0.7005493475733617\n",
      "71  Episode Reward:  25.0 .Epsilon: 0.697046600835495\n",
      "72  Episode Reward:  28.0 .Epsilon: 0.6935613678313175\n",
      "73  Episode Reward:  16.0 .Epsilon: 0.6900935609921609\n",
      "WEIGHTS SAVED\n",
      "74  Episode Reward:  35.0 .Epsilon: 0.6866430931872001\n",
      "75  Episode Reward:  10.0 .Epsilon: 0.6832098777212641\n",
      "76  Episode Reward:  9.0 .Epsilon: 0.6797938283326578\n",
      "77  Episode Reward:  21.0 .Epsilon: 0.6763948591909945\n",
      "78  Episode Reward:  10.0 .Epsilon: 0.6730128848950395\n",
      "79  Episode Reward:  15.0 .Epsilon: 0.6696478204705644\n",
      "80  Episode Reward:  19.0 .Epsilon: 0.6662995813682115\n",
      "81  Episode Reward:  12.0 .Epsilon: 0.6629680834613705\n",
      "82  Episode Reward:  16.0 .Epsilon: 0.6596532430440636\n",
      "83  Episode Reward:  14.0 .Epsilon: 0.6563549768288433\n",
      "84  Episode Reward:  15.0 .Epsilon: 0.653073201944699\n",
      "85  Episode Reward:  11.0 .Epsilon: 0.6498078359349755\n",
      "86  Episode Reward:  14.0 .Epsilon: 0.6465587967553006\n",
      "87  Episode Reward:  20.0 .Epsilon: 0.6433260027715241\n",
      "88  Episode Reward:  14.0 .Epsilon: 0.6401093727576664\n",
      "89  Episode Reward:  9.0 .Epsilon: 0.6369088258938781\n",
      "90  Episode Reward:  12.0 .Epsilon: 0.6337242817644086\n",
      "91  Episode Reward:  12.0 .Epsilon: 0.6305556603555866\n",
      "92  Episode Reward:  11.0 .Epsilon: 0.6274028820538087\n",
      "93  Episode Reward:  10.0 .Epsilon: 0.6242658676435396\n",
      "94  Episode Reward:  12.0 .Epsilon: 0.6211445383053219\n",
      "95  Episode Reward:  13.0 .Epsilon: 0.6180388156137953\n",
      "96  Episode Reward:  13.0 .Epsilon: 0.6149486215357263\n",
      "97  Episode Reward:  9.0 .Epsilon: 0.6118738784280476\n",
      "98  Episode Reward:  13.0 .Epsilon: 0.6088145090359074\n",
      "99  Episode Reward:  32.0 .Epsilon: 0.6057704364907278\n",
      "100  Episode Reward:  33.0 .Epsilon: 0.6027415843082742\n",
      "101  Episode Reward:  16.0 .Epsilon: 0.5997278763867329\n",
      "102  Episode Reward:  10.0 .Epsilon: 0.5967292370047992\n",
      "103  Episode Reward:  19.0 .Epsilon: 0.5937455908197752\n",
      "104  Episode Reward:  9.0 .Epsilon: 0.5907768628656763\n",
      "105  Episode Reward:  10.0 .Epsilon: 0.5878229785513479\n",
      "106  Episode Reward:  14.0 .Epsilon: 0.5848838636585911\n",
      "107  Episode Reward:  9.0 .Epsilon: 0.5819594443402982\n",
      "108  Episode Reward:  10.0 .Epsilon: 0.5790496471185967\n",
      "WEIGHTS SAVED\n",
      "109  Episode Reward:  14.0 .Epsilon: 0.5761543988830038\n",
      "110  Episode Reward:  11.0 .Epsilon: 0.5732736268885887\n",
      "111  Episode Reward:  14.0 .Epsilon: 0.5704072587541458\n",
      "112  Episode Reward:  14.0 .Epsilon: 0.567555222460375\n",
      "113  Episode Reward:  20.0 .Epsilon: 0.5647174463480732\n",
      "114  Episode Reward:  9.0 .Epsilon: 0.5618938591163328\n",
      "115  Episode Reward:  10.0 .Epsilon: 0.5590843898207511\n",
      "116  Episode Reward:  10.0 .Epsilon: 0.5562889678716474\n",
      "117  Episode Reward:  11.0 .Epsilon: 0.5535075230322891\n",
      "118  Episode Reward:  13.0 .Epsilon: 0.5507399854171277\n",
      "119  Episode Reward:  11.0 .Epsilon: 0.547986285490042\n",
      "120  Episode Reward:  17.0 .Epsilon: 0.5452463540625918\n",
      "121  Episode Reward:  10.0 .Epsilon: 0.5425201222922789\n",
      "122  Episode Reward:  10.0 .Epsilon: 0.5398075216808175\n",
      "123  Episode Reward:  12.0 .Epsilon: 0.5371084840724134\n",
      "124  Episode Reward:  10.0 .Epsilon: 0.5344229416520513\n",
      "125  Episode Reward:  24.0 .Epsilon: 0.531750826943791\n",
      "126  Episode Reward:  16.0 .Epsilon: 0.5290920728090721\n",
      "127  Episode Reward:  12.0 .Epsilon: 0.5264466124450268\n",
      "128  Episode Reward:  15.0 .Epsilon: 0.5238143793828016\n",
      "129  Episode Reward:  13.0 .Epsilon: 0.5211953074858876\n",
      "130  Episode Reward:  31.0 .Epsilon: 0.5185893309484582\n",
      "131  Episode Reward:  12.0 .Epsilon: 0.5159963842937159\n",
      "132  Episode Reward:  11.0 .Epsilon: 0.5134164023722473\n",
      "133  Episode Reward:  17.0 .Epsilon: 0.510849320360386\n",
      "134  Episode Reward:  15.0 .Epsilon: 0.5082950737585841\n",
      "135  Episode Reward:  14.0 .Epsilon: 0.5057535983897912\n",
      "136  Episode Reward:  12.0 .Epsilon: 0.5032248303978422\n",
      "137  Episode Reward:  9.0 .Epsilon: 0.500708706245853\n",
      "138  Episode Reward:  10.0 .Epsilon: 0.4982051627146237\n",
      "139  Episode Reward:  19.0 .Epsilon: 0.49571413690105054\n",
      "140  Episode Reward:  11.0 .Epsilon: 0.4932355662165453\n",
      "141  Episode Reward:  13.0 .Epsilon: 0.4907693883854626\n",
      "142  Episode Reward:  16.0 .Epsilon: 0.4883155414435353\n",
      "143  Episode Reward:  13.0 .Epsilon: 0.4858739637363176\n",
      "144  Episode Reward:  22.0 .Epsilon: 0.483444593917636\n",
      "WEIGHTS SAVED\n",
      "145  Episode Reward:  10.0 .Epsilon: 0.4810273709480478\n",
      "146  Episode Reward:  12.0 .Epsilon: 0.47862223409330756\n",
      "147  Episode Reward:  13.0 .Epsilon: 0.47622912292284103\n",
      "148  Episode Reward:  152.0 .Epsilon: 0.4738479773082268\n",
      "149  Episode Reward:  78.0 .Epsilon: 0.47147873742168567\n",
      "150  Episode Reward:  103.0 .Epsilon: 0.46912134373457726\n",
      "151  Episode Reward:  57.0 .Epsilon: 0.46677573701590436\n",
      "152  Episode Reward:  76.0 .Epsilon: 0.46444185833082485\n",
      "WEIGHTS SAVED\n",
      "153  Episode Reward:  58.0 .Epsilon: 0.46211964903917074\n",
      "154  Episode Reward:  46.0 .Epsilon: 0.4598090507939749\n",
      "155  Episode Reward:  18.0 .Epsilon: 0.457510005540005\n",
      "156  Episode Reward:  11.0 .Epsilon: 0.45522245551230495\n",
      "157  Episode Reward:  66.0 .Epsilon: 0.4529463432347434\n",
      "158  Episode Reward:  51.0 .Epsilon: 0.4506816115185697\n",
      "159  Episode Reward:  14.0 .Epsilon: 0.4484282034609769\n",
      "160  Episode Reward:  97.0 .Epsilon: 0.446186062443672\n",
      "161  Episode Reward:  21.0 .Epsilon: 0.4439551321314536\n",
      "162  Episode Reward:  39.0 .Epsilon: 0.4417353564707963\n",
      "163  Episode Reward:  35.0 .Epsilon: 0.43952667968844233\n",
      "164  Episode Reward:  22.0 .Epsilon: 0.43732904629000013\n",
      "WEIGHTS SAVED\n",
      "165  Episode Reward:  92.0 .Epsilon: 0.4351424010585501\n",
      "166  Episode Reward:  23.0 .Epsilon: 0.43296668905325736\n",
      "167  Episode Reward:  43.0 .Epsilon: 0.43080185560799106\n",
      "168  Episode Reward:  20.0 .Epsilon: 0.4286478463299511\n",
      "169  Episode Reward:  10.0 .Epsilon: 0.42650460709830135\n",
      "170  Episode Reward:  23.0 .Epsilon: 0.42437208406280985\n",
      "171  Episode Reward:  37.0 .Epsilon: 0.4222502236424958\n",
      "172  Episode Reward:  45.0 .Epsilon: 0.42013897252428334\n",
      "173  Episode Reward:  19.0 .Epsilon: 0.4180382776616619\n",
      "174  Episode Reward:  13.0 .Epsilon: 0.4159480862733536\n",
      "175  Episode Reward:  78.0 .Epsilon: 0.41386834584198684\n",
      "176  Episode Reward:  35.0 .Epsilon: 0.4117990041127769\n",
      "177  Episode Reward:  13.0 .Epsilon: 0.40974000909221303\n",
      "178  Episode Reward:  13.0 .Epsilon: 0.40769130904675194\n",
      "179  Episode Reward:  13.0 .Epsilon: 0.40565285250151817\n",
      "180  Episode Reward:  37.0 .Epsilon: 0.4036245882390106\n",
      "WEIGHTS SAVED\n",
      "181  Episode Reward:  91.0 .Epsilon: 0.4016064652978155\n",
      "182  Episode Reward:  130.0 .Epsilon: 0.3995984329713264\n",
      "183  Episode Reward:  63.0 .Epsilon: 0.3976004408064698\n",
      "184  Episode Reward:  30.0 .Epsilon: 0.39561243860243744\n",
      "185  Episode Reward:  16.0 .Epsilon: 0.3936343764094253\n",
      "186  Episode Reward:  112.0 .Epsilon: 0.39166620452737816\n",
      "187  Episode Reward:  12.0 .Epsilon: 0.3897078735047413\n",
      "188  Episode Reward:  13.0 .Epsilon: 0.3877593341372176\n",
      "189  Episode Reward:  25.0 .Epsilon: 0.3858205374665315\n",
      "WEIGHTS SAVED\n",
      "190  Episode Reward:  38.0 .Epsilon: 0.38389143477919885\n",
      "191  Episode Reward:  38.0 .Epsilon: 0.3819719776053028\n",
      "192  Episode Reward:  33.0 .Epsilon: 0.3800621177172763\n",
      "193  Episode Reward:  84.0 .Epsilon: 0.37816180712868996\n",
      "194  Episode Reward:  51.0 .Epsilon: 0.37627099809304654\n",
      "195  Episode Reward:  18.0 .Epsilon: 0.3743896431025813\n",
      "196  Episode Reward:  57.0 .Epsilon: 0.37251769488706843\n",
      "197  Episode Reward:  124.0 .Epsilon: 0.3706551064126331\n",
      "WEIGHTS SAVED\n",
      "198  Episode Reward:  160.0 .Epsilon: 0.36880183088056995\n",
      "199  Episode Reward:  116.0 .Epsilon: 0.3669578217261671\n",
      "200  Episode Reward:  18.0 .Epsilon: 0.36512303261753626\n",
      "201  Episode Reward:  43.0 .Epsilon: 0.3632974174544486\n",
      "202  Episode Reward:  43.0 .Epsilon: 0.3614809303671764\n",
      "203  Episode Reward:  18.0 .Epsilon: 0.3596735257153405\n",
      "204  Episode Reward:  15.0 .Epsilon: 0.3578751580867638\n",
      "205  Episode Reward:  31.0 .Epsilon: 0.35608578229633\n",
      "206  Episode Reward:  20.0 .Epsilon: 0.3543053533848483\n",
      "207  Episode Reward:  49.0 .Epsilon: 0.35253382661792404\n",
      "208  Episode Reward:  40.0 .Epsilon: 0.3507711574848344\n",
      "WEIGHTS SAVED\n",
      "209  Episode Reward:  30.0 .Epsilon: 0.34901730169741024\n",
      "210  Episode Reward:  92.0 .Epsilon: 0.3472722151889232\n",
      "211  Episode Reward:  92.0 .Epsilon: 0.3455358541129786\n",
      "212  Episode Reward:  75.0 .Epsilon: 0.3438081748424137\n",
      "213  Episode Reward:  99.0 .Epsilon: 0.3420891339682016\n",
      "214  Episode Reward:  53.0 .Epsilon: 0.3403786882983606\n",
      "215  Episode Reward:  28.0 .Epsilon: 0.3386767948568688\n",
      "216  Episode Reward:  11.0 .Epsilon: 0.33698341088258443\n",
      "WEIGHTS SAVED\n",
      "217  Episode Reward:  64.0 .Epsilon: 0.3352984938281715\n",
      "218  Episode Reward:  31.0 .Epsilon: 0.33362200135903064\n",
      "219  Episode Reward:  49.0 .Epsilon: 0.33195389135223546\n",
      "220  Episode Reward:  42.0 .Epsilon: 0.3302941218954743\n",
      "221  Episode Reward:  16.0 .Epsilon: 0.32864265128599696\n",
      "222  Episode Reward:  44.0 .Epsilon: 0.326999438029567\n",
      "223  Episode Reward:  120.0 .Epsilon: 0.3253644408394192\n",
      "224  Episode Reward:  104.0 .Epsilon: 0.3237376186352221\n",
      "225  Episode Reward:  68.0 .Epsilon: 0.322118930542046\n",
      "WEIGHTS SAVED\n",
      "226  Episode Reward:  65.0 .Epsilon: 0.32050833588933575\n",
      "227  Episode Reward:  65.0 .Epsilon: 0.31890579420988907\n",
      "228  Episode Reward:  115.0 .Epsilon: 0.3173112652388396\n",
      "229  Episode Reward:  91.0 .Epsilon: 0.3157247089126454\n",
      "230  Episode Reward:  109.0 .Epsilon: 0.3141460853680822\n",
      "WEIGHTS SAVED\n",
      "231  Episode Reward:  106.0 .Epsilon: 0.3125753549412418\n",
      "232  Episode Reward:  81.0 .Epsilon: 0.31101247816653554\n",
      "233  Episode Reward:  28.0 .Epsilon: 0.30945741577570285\n",
      "234  Episode Reward:  69.0 .Epsilon: 0.3079101286968243\n",
      "235  Episode Reward:  59.0 .Epsilon: 0.3063705780533402\n",
      "236  Episode Reward:  59.0 .Epsilon: 0.30483872516307353\n",
      "237  Episode Reward:  106.0 .Epsilon: 0.3033145315372582\n",
      "WEIGHTS SAVED\n",
      "238  Episode Reward:  74.0 .Epsilon: 0.3017979588795719\n",
      "239  Episode Reward:  16.0 .Epsilon: 0.30028896908517405\n",
      "240  Episode Reward:  63.0 .Epsilon: 0.2987875242397482\n",
      "241  Episode Reward:  72.0 .Epsilon: 0.29729358661854943\n",
      "242  Episode Reward:  96.0 .Epsilon: 0.29580711868545667\n",
      "243  Episode Reward:  69.0 .Epsilon: 0.2943280830920294\n",
      "244  Episode Reward:  44.0 .Epsilon: 0.29285644267656924\n",
      "245  Episode Reward:  66.0 .Epsilon: 0.2913921604631864\n",
      "WEIGHTS SAVED\n",
      "246  Episode Reward:  74.0 .Epsilon: 0.28993519966087045\n",
      "247  Episode Reward:  30.0 .Epsilon: 0.2884855236625661\n",
      "248  Episode Reward:  115.0 .Epsilon: 0.28704309604425327\n",
      "249  Episode Reward:  37.0 .Epsilon: 0.285607880564032\n",
      "250  Episode Reward:  91.0 .Epsilon: 0.28417984116121187\n",
      "251  Episode Reward:  36.0 .Epsilon: 0.2827589419554058\n",
      "252  Episode Reward:  64.0 .Epsilon: 0.28134514724562876\n",
      "253  Episode Reward:  12.0 .Epsilon: 0.2799384215094006\n",
      "254  Episode Reward:  61.0 .Epsilon: 0.27853872940185365\n",
      "WEIGHTS SAVED\n",
      "255  Episode Reward:  56.0 .Epsilon: 0.27714603575484437\n",
      "256  Episode Reward:  81.0 .Epsilon: 0.2757603055760701\n",
      "257  Episode Reward:  62.0 .Epsilon: 0.2743815040481898\n",
      "258  Episode Reward:  16.0 .Epsilon: 0.2730095965279488\n",
      "259  Episode Reward:  78.0 .Epsilon: 0.27164454854530906\n",
      "260  Episode Reward:  32.0 .Epsilon: 0.2702863258025825\n",
      "261  Episode Reward:  16.0 .Epsilon: 0.2689348941735696\n",
      "262  Episode Reward:  31.0 .Epsilon: 0.26759021970270175\n",
      "263  Episode Reward:  15.0 .Epsilon: 0.2662522686041882\n",
      "264  Episode Reward:  11.0 .Epsilon: 0.2649210072611673\n",
      "WEIGHTS SAVED\n",
      "265  Episode Reward:  139.0 .Epsilon: 0.26359640222486147\n",
      "266  Episode Reward:  24.0 .Epsilon: 0.26227842021373715\n",
      "267  Episode Reward:  98.0 .Epsilon: 0.2609670281126685\n",
      "268  Episode Reward:  16.0 .Epsilon: 0.25966219297210513\n",
      "269  Episode Reward:  86.0 .Epsilon: 0.2583638820072446\n",
      "270  Episode Reward:  67.0 .Epsilon: 0.2570720625972084\n",
      "271  Episode Reward:  129.0 .Epsilon: 0.25578670228422234\n",
      "272  Episode Reward:  41.0 .Epsilon: 0.25450776877280124\n",
      "273  Episode Reward:  33.0 .Epsilon: 0.2532352299289372\n",
      "WEIGHTS SAVED\n",
      "274  Episode Reward:  22.0 .Epsilon: 0.2519690537792925\n",
      "275  Episode Reward:  113.0 .Epsilon: 0.2507092085103961\n",
      "276  Episode Reward:  56.0 .Epsilon: 0.2494556624678441\n",
      "277  Episode Reward:  42.0 .Epsilon: 0.24820838415550486\n",
      "278  Episode Reward:  138.0 .Epsilon: 0.24696734223472733\n",
      "279  Episode Reward:  95.0 .Epsilon: 0.2457325055235537\n",
      "WEIGHTS SAVED\n",
      "280  Episode Reward:  148.0 .Epsilon: 0.24450384299593592\n",
      "281  Episode Reward:  121.0 .Epsilon: 0.24328132378095624\n",
      "282  Episode Reward:  90.0 .Epsilon: 0.24206491716205145\n",
      "283  Episode Reward:  89.0 .Epsilon: 0.2408545925762412\n",
      "WEIGHTS SAVED\n",
      "284  Episode Reward:  116.0 .Epsilon: 0.23965031961336\n",
      "285  Episode Reward:  141.0 .Epsilon: 0.2384520680152932\n",
      "286  Episode Reward:  155.0 .Epsilon: 0.23725980767521673\n",
      "287  Episode Reward:  147.0 .Epsilon: 0.23607350863684065\n",
      "WEIGHTS SAVED\n",
      "288  Episode Reward:  140.0 .Epsilon: 0.23489314109365644\n",
      "289  Episode Reward:  17.0 .Epsilon: 0.23371867538818816\n",
      "290  Episode Reward:  143.0 .Epsilon: 0.23255008201124722\n",
      "291  Episode Reward:  111.0 .Epsilon: 0.231387331601191\n",
      "292  Episode Reward:  20.0 .Epsilon: 0.23023039494318503\n",
      "WEIGHTS SAVED\n",
      "293  Episode Reward:  115.0 .Epsilon: 0.2290792429684691\n",
      "294  Episode Reward:  177.0 .Epsilon: 0.22793384675362674\n",
      "295  Episode Reward:  146.0 .Epsilon: 0.22679417751985861\n",
      "WEIGHTS SAVED\n",
      "296  Episode Reward:  184.0 .Epsilon: 0.22566020663225933\n",
      "297  Episode Reward:  168.0 .Epsilon: 0.22453190559909803\n",
      "298  Episode Reward:  116.0 .Epsilon: 0.22340924607110255\n",
      "299  Episode Reward:  123.0 .Epsilon: 0.22229219984074702\n",
      "WEIGHTS SAVED\n",
      "300  Episode Reward:  148.0 .Epsilon: 0.2211807388415433\n",
      "WEIGHTS SAVED\n",
      "301  Episode Reward:  500.0 .Epsilon: 0.22007483514733558\n",
      "302  Episode Reward:  158.0 .Epsilon: 0.2189744609715989\n",
      "303  Episode Reward:  122.0 .Epsilon: 0.2178795886667409\n",
      "304  Episode Reward:  98.0 .Epsilon: 0.2167901907234072\n",
      "WEIGHTS SAVED\n",
      "305  Episode Reward:  146.0 .Epsilon: 0.21570623976979014\n",
      "306  Episode Reward:  134.0 .Epsilon: 0.21462770857094118\n",
      "307  Episode Reward:  17.0 .Epsilon: 0.21355457002808648\n",
      "308  Episode Reward:  35.0 .Epsilon: 0.21248679717794605\n",
      "309  Episode Reward:  32.0 .Epsilon: 0.21142436319205632\n",
      "310  Episode Reward:  15.0 .Epsilon: 0.21036724137609603\n",
      "311  Episode Reward:  11.0 .Epsilon: 0.20931540516921554\n",
      "312  Episode Reward:  24.0 .Epsilon: 0.20826882814336947\n",
      "313  Episode Reward:  46.0 .Epsilon: 0.20722748400265262\n",
      "314  Episode Reward:  12.0 .Epsilon: 0.20619134658263935\n",
      "315  Episode Reward:  15.0 .Epsilon: 0.20516038984972615\n",
      "316  Episode Reward:  40.0 .Epsilon: 0.2041345879004775\n",
      "317  Episode Reward:  10.0 .Epsilon: 0.2031139149609751\n",
      "WEIGHTS SAVED\n",
      "318  Episode Reward:  36.0 .Epsilon: 0.20209834538617025\n",
      "319  Episode Reward:  99.0 .Epsilon: 0.2010878536592394\n",
      "320  Episode Reward:  32.0 .Epsilon: 0.2000824143909432\n",
      "321  Episode Reward:  10.0 .Epsilon: 0.19908200231898848\n",
      "322  Episode Reward:  90.0 .Epsilon: 0.19808659230739353\n",
      "323  Episode Reward:  15.0 .Epsilon: 0.19709615934585656\n",
      "324  Episode Reward:  57.0 .Epsilon: 0.19611067854912728\n",
      "325  Episode Reward:  94.0 .Epsilon: 0.19513012515638165\n",
      "WEIGHTS SAVED\n",
      "326  Episode Reward:  106.0 .Epsilon: 0.19415447453059972\n",
      "327  Episode Reward:  88.0 .Epsilon: 0.19318370215794672\n",
      "328  Episode Reward:  13.0 .Epsilon: 0.192217783647157\n",
      "329  Episode Reward:  90.0 .Epsilon: 0.1912566947289212\n",
      "330  Episode Reward:  12.0 .Epsilon: 0.1903004112552766\n",
      "331  Episode Reward:  56.0 .Epsilon: 0.18934890919900021\n",
      "332  Episode Reward:  14.0 .Epsilon: 0.18840216465300522\n",
      "333  Episode Reward:  60.0 .Epsilon: 0.18746015382974018\n",
      "334  Episode Reward:  14.0 .Epsilon: 0.1865228530605915\n",
      "335  Episode Reward:  93.0 .Epsilon: 0.18559023879528855\n",
      "WEIGHTS SAVED\n",
      "336  Episode Reward:  98.0 .Epsilon: 0.1846622876013121\n",
      "337  Episode Reward:  200.0 .Epsilon: 0.18373897616330553\n",
      "338  Episode Reward:  99.0 .Epsilon: 0.182820281282489\n",
      "339  Episode Reward:  18.0 .Epsilon: 0.18190617987607657\n",
      "340  Episode Reward:  99.0 .Epsilon: 0.18099664897669618\n",
      "WEIGHTS SAVED\n",
      "341  Episode Reward:  142.0 .Epsilon: 0.1800916657318127\n",
      "342  Episode Reward:  139.0 .Epsilon: 0.17919120740315364\n",
      "343  Episode Reward:  56.0 .Epsilon: 0.17829525136613786\n",
      "344  Episode Reward:  70.0 .Epsilon: 0.17740377510930716\n",
      "345  Episode Reward:  34.0 .Epsilon: 0.17651675623376062\n",
      "WEIGHTS SAVED\n",
      "346  Episode Reward:  95.0 .Epsilon: 0.1756341724525918\n",
      "347  Episode Reward:  61.0 .Epsilon: 0.17475600159032884\n",
      "348  Episode Reward:  39.0 .Epsilon: 0.17388222158237718\n",
      "349  Episode Reward:  50.0 .Epsilon: 0.1730128104744653\n",
      "350  Episode Reward:  16.0 .Epsilon: 0.17214774642209296\n",
      "351  Episode Reward:  17.0 .Epsilon: 0.1712870076899825\n",
      "352  Episode Reward:  63.0 .Epsilon: 0.17043057265153258\n",
      "353  Episode Reward:  12.0 .Epsilon: 0.16957841978827493\n",
      "354  Episode Reward:  82.0 .Epsilon: 0.16873052768933355\n",
      "355  Episode Reward:  54.0 .Epsilon: 0.1678868750508869\n",
      "WEIGHTS SAVED\n",
      "356  Episode Reward:  217.0 .Epsilon: 0.16704744067563246\n",
      "357  Episode Reward:  169.0 .Epsilon: 0.1662122034722543\n",
      "WEIGHTS SAVED\n",
      "358  Episode Reward:  243.0 .Epsilon: 0.16538114245489302\n",
      "359  Episode Reward:  16.0 .Epsilon: 0.16455423674261854\n",
      "360  Episode Reward:  252.0 .Epsilon: 0.16373146555890544\n",
      "WEIGHTS SAVED\n",
      "361  Episode Reward:  270.0 .Epsilon: 0.16291280823111093\n",
      "362  Episode Reward:  11.0 .Epsilon: 0.16209824418995536\n",
      "WEIGHTS SAVED\n",
      "363  Episode Reward:  500.0 .Epsilon: 0.16128775296900558\n",
      "364  Episode Reward:  18.0 .Epsilon: 0.16048131420416054\n",
      "365  Episode Reward:  224.0 .Epsilon: 0.15967890763313974\n",
      "366  Episode Reward:  87.0 .Epsilon: 0.15888051309497406\n",
      "WEIGHTS SAVED\n",
      "367  Episode Reward:  500.0 .Epsilon: 0.1580861105294992\n",
      "368  Episode Reward:  12.0 .Epsilon: 0.1572956799768517\n",
      "WEIGHTS SAVED\n",
      "369  Episode Reward:  500.0 .Epsilon: 0.15650920157696743\n",
      "370  Episode Reward:  37.0 .Epsilon: 0.1557266555690826\n",
      "WEIGHTS SAVED\n",
      "371  Episode Reward:  500.0 .Epsilon: 0.1549480222912372\n",
      "WEIGHTS SAVED\n",
      "372  Episode Reward:  39.0 .Epsilon: 0.15417328217978102\n",
      "WEIGHTS SAVED\n",
      "373  Episode Reward:  500.0 .Epsilon: 0.1534024157688821\n",
      "WEIGHTS SAVED\n",
      "374  Episode Reward:  500.0 .Epsilon: 0.1526354036900377\n",
      "375  Episode Reward:  12.0 .Epsilon: 0.1518722266715875\n",
      "WEIGHTS SAVED\n",
      "376  Episode Reward:  500.0 .Epsilon: 0.15111286553822956\n",
      "WEIGHTS SAVED\n",
      "377  Episode Reward:  500.0 .Epsilon: 0.15035730121053842\n",
      "WEIGHTS SAVED\n",
      "378  Episode Reward:  500.0 .Epsilon: 0.14960551470448571\n",
      "379  Episode Reward:  256.0 .Epsilon: 0.14885748713096328\n",
      "WEIGHTS SAVED\n",
      "380  Episode Reward:  500.0 .Epsilon: 0.14811319969530845\n",
      "WEIGHTS SAVED\n",
      "381  Episode Reward:  292.0 .Epsilon: 0.1473726336968319\n",
      "382  Episode Reward:  11.0 .Epsilon: 0.14663577052834775\n",
      "WEIGHTS SAVED\n",
      "383  Episode Reward:  500.0 .Epsilon: 0.14590259167570602\n",
      "384  Episode Reward:  13.0 .Epsilon: 0.1451730787173275\n",
      "385  Episode Reward:  200.0 .Epsilon: 0.14444721332374086\n",
      "WEIGHTS SAVED\n",
      "386  Episode Reward:  441.0 .Epsilon: 0.14372497725712216\n",
      "387  Episode Reward:  83.0 .Epsilon: 0.14300635237083656\n",
      "388  Episode Reward:  74.0 .Epsilon: 0.14229132060898236\n",
      "WEIGHTS SAVED\n",
      "389  Episode Reward:  500.0 .Epsilon: 0.14157986400593744\n",
      "WEIGHTS SAVED\n",
      "390  Episode Reward:  125.0 .Epsilon: 0.14087196468590776\n",
      "391  Episode Reward:  12.0 .Epsilon: 0.14016760486247823\n",
      "392  Episode Reward:  11.0 .Epsilon: 0.13946676683816583\n",
      "393  Episode Reward:  12.0 .Epsilon: 0.138769433003975\n",
      "394  Episode Reward:  98.0 .Epsilon: 0.13807558583895513\n",
      "395  Episode Reward:  87.0 .Epsilon: 0.13738520790976036\n",
      "396  Episode Reward:  85.0 .Epsilon: 0.13669828187021155\n",
      "397  Episode Reward:  13.0 .Epsilon: 0.13601479046086049\n",
      "398  Episode Reward:  108.0 .Epsilon: 0.1353347165085562\n",
      "399  Episode Reward:  11.0 .Epsilon: 0.1346580429260134\n",
      "WEIGHTS SAVED\n",
      "400  Episode Reward:  94.0 .Epsilon: 0.13398475271138335\n",
      "401  Episode Reward:  161.0 .Epsilon: 0.13331482894782642\n",
      "402  Episode Reward:  187.0 .Epsilon: 0.13264825480308728\n",
      "WEIGHTS SAVED\n",
      "403  Episode Reward:  153.0 .Epsilon: 0.13198501352907185\n",
      "404  Episode Reward:  11.0 .Epsilon: 0.1313250884614265\n",
      "405  Episode Reward:  108.0 .Epsilon: 0.13066846301911936\n",
      "406  Episode Reward:  140.0 .Epsilon: 0.13001512070402377\n",
      "407  Episode Reward:  124.0 .Epsilon: 0.12936504510050365\n",
      "WEIGHTS SAVED\n",
      "408  Episode Reward:  114.0 .Epsilon: 0.12871821987500112\n",
      "409  Episode Reward:  129.0 .Epsilon: 0.12807462877562611\n",
      "410  Episode Reward:  16.0 .Epsilon: 0.12743425563174798\n",
      "411  Episode Reward:  123.0 .Epsilon: 0.12679708435358925\n",
      "412  Episode Reward:  12.0 .Epsilon: 0.1261630989318213\n",
      "413  Episode Reward:  129.0 .Epsilon: 0.1255322834371622\n",
      "WEIGHTS SAVED\n",
      "414  Episode Reward:  151.0 .Epsilon: 0.12490462201997637\n",
      "415  Episode Reward:  151.0 .Epsilon: 0.1242800989098765\n",
      "416  Episode Reward:  140.0 .Epsilon: 0.12365869841532712\n",
      "417  Episode Reward:  12.0 .Epsilon: 0.12304040492325048\n",
      "WEIGHTS SAVED\n",
      "418  Episode Reward:  142.0 .Epsilon: 0.12242520289863423\n",
      "419  Episode Reward:  15.0 .Epsilon: 0.12181307688414106\n",
      "420  Episode Reward:  15.0 .Epsilon: 0.12120401149972035\n",
      "421  Episode Reward:  136.0 .Epsilon: 0.12059799144222175\n",
      "422  Episode Reward:  166.0 .Epsilon: 0.11999500148501063\n",
      "423  Episode Reward:  33.0 .Epsilon: 0.11939502647758558\n",
      "WEIGHTS SAVED\n",
      "424  Episode Reward:  151.0 .Epsilon: 0.11879805134519765\n",
      "425  Episode Reward:  206.0 .Epsilon: 0.11820406108847166\n",
      "426  Episode Reward:  36.0 .Epsilon: 0.1176130407830293\n",
      "427  Episode Reward:  63.0 .Epsilon: 0.11702497557911415\n",
      "428  Episode Reward:  11.0 .Epsilon: 0.11643985070121858\n",
      "429  Episode Reward:  38.0 .Epsilon: 0.11585765144771248\n",
      "WEIGHTS SAVED\n",
      "430  Episode Reward:  374.0 .Epsilon: 0.11527836319047392\n",
      "WEIGHTS SAVED\n",
      "431  Episode Reward:  493.0 .Epsilon: 0.11470197137452155\n",
      "432  Episode Reward:  33.0 .Epsilon: 0.11412846151764894\n",
      "433  Episode Reward:  45.0 .Epsilon: 0.1135578192100607\n",
      "WEIGHTS SAVED\n",
      "434  Episode Reward:  500.0 .Epsilon: 0.11299003011401039\n",
      "WEIGHTS SAVED\n",
      "435  Episode Reward:  500.0 .Epsilon: 0.11242507996344034\n",
      "WEIGHTS SAVED\n",
      "436  Episode Reward:  500.0 .Epsilon: 0.11186295456362313\n",
      "WEIGHTS SAVED\n",
      "437  Episode Reward:  500.0 .Epsilon: 0.11130363979080501\n",
      "WEIGHTS SAVED\n",
      "438  Episode Reward:  500.0 .Epsilon: 0.11074712159185099\n",
      "WEIGHTS SAVED\n",
      "439  Episode Reward:  500.0 .Epsilon: 0.11019338598389174\n",
      "WEIGHTS SAVED\n",
      "440  Episode Reward:  500.0 .Epsilon: 0.10964241905397228\n",
      "WEIGHTS SAVED\n",
      "441  Episode Reward:  500.0 .Epsilon: 0.10909420695870241\n",
      "WEIGHTS SAVED\n",
      "442  Episode Reward:  440.0 .Epsilon: 0.1085487359239089\n",
      "WEIGHTS SAVED\n",
      "443  Episode Reward:  500.0 .Epsilon: 0.10800599224428936\n",
      "WEIGHTS SAVED\n",
      "444  Episode Reward:  294.0 .Epsilon: 0.10746596228306791\n",
      "445  Episode Reward:  267.0 .Epsilon: 0.10692863247165257\n",
      "WEIGHTS SAVED\n",
      "446  Episode Reward:  500.0 .Epsilon: 0.1063939893092943\n",
      "WEIGHTS SAVED\n",
      "447  Episode Reward:  500.0 .Epsilon: 0.10586201936274783\n",
      "WEIGHTS SAVED\n",
      "448  Episode Reward:  500.0 .Epsilon: 0.10533270926593409\n",
      "449  Episode Reward:  19.0 .Epsilon: 0.10480604571960442\n",
      "WEIGHTS SAVED\n",
      "450  Episode Reward:  500.0 .Epsilon: 0.1042820154910064\n",
      "WEIGHTS SAVED\n",
      "451  Episode Reward:  500.0 .Epsilon: 0.10376060541355137\n",
      "WEIGHTS SAVED\n",
      "452  Episode Reward:  500.0 .Epsilon: 0.1032418023864836\n",
      "453  Episode Reward:  15.0 .Epsilon: 0.10272559337455119\n",
      "WEIGHTS SAVED\n",
      "454  Episode Reward:  405.0 .Epsilon: 0.10221196540767843\n",
      "455  Episode Reward:  13.0 .Epsilon: 0.10170090558064004\n",
      "456  Episode Reward:  130.0 .Epsilon: 0.10119240105273684\n",
      "457  Episode Reward:  12.0 .Epsilon: 0.10068643904747315\n",
      "458  Episode Reward:  23.0 .Epsilon: 0.10018300685223579\n",
      "WEIGHTS SAVED\n",
      "459  Episode Reward:  106.0 .Epsilon: 0.1\n",
      "460  Episode Reward:  51.0 .Epsilon: 0.1\n",
      "461  Episode Reward:  60.0 .Epsilon: 0.1\n",
      "462  Episode Reward:  16.0 .Epsilon: 0.1\n",
      "463  Episode Reward:  54.0 .Epsilon: 0.1\n",
      "464  Episode Reward:  56.0 .Epsilon: 0.1\n",
      "465  Episode Reward:  81.0 .Epsilon: 0.1\n",
      "466  Episode Reward:  58.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "467  Episode Reward:  78.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "468  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "469  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "470  Episode Reward:  199.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "471  Episode Reward:  302.0 .Epsilon: 0.1\n",
      "472  Episode Reward:  72.0 .Epsilon: 0.1\n",
      "473  Episode Reward:  199.0 .Epsilon: 0.1\n",
      "474  Episode Reward:  173.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "475  Episode Reward:  200.0 .Epsilon: 0.1\n",
      "476  Episode Reward:  237.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "477  Episode Reward:  211.0 .Epsilon: 0.1\n",
      "478  Episode Reward:  211.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "479  Episode Reward:  189.0 .Epsilon: 0.1\n",
      "480  Episode Reward:  194.0 .Epsilon: 0.1\n",
      "481  Episode Reward:  181.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "482  Episode Reward:  184.0 .Epsilon: 0.1\n",
      "483  Episode Reward:  176.0 .Epsilon: 0.1\n",
      "484  Episode Reward:  177.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "485  Episode Reward:  281.0 .Epsilon: 0.1\n",
      "486  Episode Reward:  255.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "487  Episode Reward:  150.0 .Epsilon: 0.1\n",
      "488  Episode Reward:  181.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "489  Episode Reward:  206.0 .Epsilon: 0.1\n",
      "490  Episode Reward:  211.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "491  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "492  Episode Reward:  262.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "493  Episode Reward:  281.0 .Epsilon: 0.1\n",
      "494  Episode Reward:  176.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "495  Episode Reward:  271.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "496  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "497  Episode Reward:  208.0 .Epsilon: 0.1\n",
      "498  Episode Reward:  53.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "499  Episode Reward:  72.0 .Epsilon: 0.1\n",
      "500  Episode Reward:  65.0 .Epsilon: 0.1\n",
      "501  Episode Reward:  84.0 .Epsilon: 0.1\n",
      "502  Episode Reward:  99.0 .Epsilon: 0.1\n",
      "503  Episode Reward:  112.0 .Epsilon: 0.1\n",
      "504  Episode Reward:  61.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "505  Episode Reward:  170.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "506  Episode Reward:  357.0 .Epsilon: 0.1\n",
      "507  Episode Reward:  178.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "508  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "509  Episode Reward:  210.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "510  Episode Reward:  237.0 .Epsilon: 0.1\n",
      "511  Episode Reward:  190.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "512  Episode Reward:  219.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "513  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "514  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "515  Episode Reward:  39.0 .Epsilon: 0.1\n",
      "516  Episode Reward:  29.0 .Epsilon: 0.1\n",
      "517  Episode Reward:  31.0 .Epsilon: 0.1\n",
      "518  Episode Reward:  43.0 .Epsilon: 0.1\n",
      "519  Episode Reward:  41.0 .Epsilon: 0.1\n",
      "520  Episode Reward:  55.0 .Epsilon: 0.1\n",
      "521  Episode Reward:  18.0 .Epsilon: 0.1\n",
      "522  Episode Reward:  36.0 .Epsilon: 0.1\n",
      "523  Episode Reward:  47.0 .Epsilon: 0.1\n",
      "524  Episode Reward:  27.0 .Epsilon: 0.1\n",
      "525  Episode Reward:  55.0 .Epsilon: 0.1\n",
      "526  Episode Reward:  23.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "527  Episode Reward:  31.0 .Epsilon: 0.1\n",
      "528  Episode Reward:  27.0 .Epsilon: 0.1\n",
      "529  Episode Reward:  62.0 .Epsilon: 0.1\n",
      "530  Episode Reward:  44.0 .Epsilon: 0.1\n",
      "531  Episode Reward:  47.0 .Epsilon: 0.1\n",
      "532  Episode Reward:  19.0 .Epsilon: 0.1\n",
      "533  Episode Reward:  50.0 .Epsilon: 0.1\n",
      "534  Episode Reward:  190.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "535  Episode Reward:  202.0 .Epsilon: 0.1\n",
      "536  Episode Reward:  14.0 .Epsilon: 0.1\n",
      "537  Episode Reward:  19.0 .Epsilon: 0.1\n",
      "538  Episode Reward:  28.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "539  Episode Reward:  452.0 .Epsilon: 0.1\n",
      "540  Episode Reward:  181.0 .Epsilon: 0.1\n",
      "541  Episode Reward:  73.0 .Epsilon: 0.1\n",
      "542  Episode Reward:  67.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "543  Episode Reward:  96.0 .Epsilon: 0.1\n",
      "544  Episode Reward:  256.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "545  Episode Reward:  187.0 .Epsilon: 0.1\n",
      "546  Episode Reward:  134.0 .Epsilon: 0.1\n",
      "547  Episode Reward:  72.0 .Epsilon: 0.1\n",
      "548  Episode Reward:  100.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "549  Episode Reward:  171.0 .Epsilon: 0.1\n",
      "550  Episode Reward:  311.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "551  Episode Reward:  236.0 .Epsilon: 0.1\n",
      "552  Episode Reward:  138.0 .Epsilon: 0.1\n",
      "553  Episode Reward:  49.0 .Epsilon: 0.1\n",
      "554  Episode Reward:  162.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "555  Episode Reward:  311.0 .Epsilon: 0.1\n",
      "556  Episode Reward:  79.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "557  Episode Reward:  238.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "558  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "559  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "560  Episode Reward:  278.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "561  Episode Reward:  218.0 .Epsilon: 0.1\n",
      "562  Episode Reward:  264.0 .Epsilon: 0.1\n",
      "563  Episode Reward:  210.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "564  Episode Reward:  14.0 .Epsilon: 0.1\n",
      "565  Episode Reward:  440.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "566  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "567  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "568  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "569  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "570  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "571  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "572  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "573  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "574  Episode Reward:  141.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "575  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "576  Episode Reward:  66.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "577  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "578  Episode Reward:  245.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "579  Episode Reward:  239.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "580  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "581  Episode Reward:  131.0 .Epsilon: 0.1\n",
      "582  Episode Reward:  125.0 .Epsilon: 0.1\n",
      "583  Episode Reward:  90.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "584  Episode Reward:  116.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "585  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "586  Episode Reward:  180.0 .Epsilon: 0.1\n",
      "587  Episode Reward:  147.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "588  Episode Reward:  315.0 .Epsilon: 0.1\n",
      "589  Episode Reward:  154.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "590  Episode Reward:  135.0 .Epsilon: 0.1\n",
      "591  Episode Reward:  65.0 .Epsilon: 0.1\n",
      "592  Episode Reward:  57.0 .Epsilon: 0.1\n",
      "593  Episode Reward:  123.0 .Epsilon: 0.1\n",
      "594  Episode Reward:  150.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "595  Episode Reward:  132.0 .Epsilon: 0.1\n",
      "596  Episode Reward:  107.0 .Epsilon: 0.1\n",
      "597  Episode Reward:  31.0 .Epsilon: 0.1\n",
      "598  Episode Reward:  117.0 .Epsilon: 0.1\n",
      "599  Episode Reward:  145.0 .Epsilon: 0.1\n",
      "600  Episode Reward:  21.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "601  Episode Reward:  81.0 .Epsilon: 0.1\n",
      "602  Episode Reward:  63.0 .Epsilon: 0.1\n",
      "603  Episode Reward:  145.0 .Epsilon: 0.1\n",
      "604  Episode Reward:  150.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "605  Episode Reward:  135.0 .Epsilon: 0.1\n",
      "606  Episode Reward:  15.0 .Epsilon: 0.1\n",
      "607  Episode Reward:  52.0 .Epsilon: 0.1\n",
      "608  Episode Reward:  130.0 .Epsilon: 0.1\n",
      "609  Episode Reward:  128.0 .Epsilon: 0.1\n",
      "610  Episode Reward:  55.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "611  Episode Reward:  134.0 .Epsilon: 0.1\n",
      "612  Episode Reward:  22.0 .Epsilon: 0.1\n",
      "613  Episode Reward:  21.0 .Epsilon: 0.1\n",
      "614  Episode Reward:  11.0 .Epsilon: 0.1\n",
      "615  Episode Reward:  135.0 .Epsilon: 0.1\n",
      "616  Episode Reward:  148.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "617  Episode Reward:  158.0 .Epsilon: 0.1\n",
      "618  Episode Reward:  323.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "619  Episode Reward:  387.0 .Epsilon: 0.1\n",
      "620  Episode Reward:  28.0 .Epsilon: 0.1\n",
      "621  Episode Reward:  194.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "622  Episode Reward:  320.0 .Epsilon: 0.1\n",
      "623  Episode Reward:  29.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "624  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "625  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "626  Episode Reward:  122.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "627  Episode Reward:  131.0 .Epsilon: 0.1\n",
      "628  Episode Reward:  113.0 .Epsilon: 0.1\n",
      "629  Episode Reward:  124.0 .Epsilon: 0.1\n",
      "630  Episode Reward:  109.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "631  Episode Reward:  141.0 .Epsilon: 0.1\n",
      "632  Episode Reward:  135.0 .Epsilon: 0.1\n",
      "633  Episode Reward:  171.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "634  Episode Reward:  209.0 .Epsilon: 0.1\n",
      "635  Episode Reward:  157.0 .Epsilon: 0.1\n",
      "636  Episode Reward:  164.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "637  Episode Reward:  131.0 .Epsilon: 0.1\n",
      "638  Episode Reward:  129.0 .Epsilon: 0.1\n",
      "639  Episode Reward:  137.0 .Epsilon: 0.1\n",
      "640  Episode Reward:  132.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "641  Episode Reward:  151.0 .Epsilon: 0.1\n",
      "642  Episode Reward:  125.0 .Epsilon: 0.1\n",
      "643  Episode Reward:  123.0 .Epsilon: 0.1\n",
      "644  Episode Reward:  132.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "645  Episode Reward:  129.0 .Epsilon: 0.1\n",
      "646  Episode Reward:  125.0 .Epsilon: 0.1\n",
      "647  Episode Reward:  117.0 .Epsilon: 0.1\n",
      "648  Episode Reward:  126.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "649  Episode Reward:  122.0 .Epsilon: 0.1\n",
      "650  Episode Reward:  116.0 .Epsilon: 0.1\n",
      "651  Episode Reward:  123.0 .Epsilon: 0.1\n",
      "652  Episode Reward:  31.0 .Epsilon: 0.1\n",
      "653  Episode Reward:  126.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "654  Episode Reward:  113.0 .Epsilon: 0.1\n",
      "655  Episode Reward:  26.0 .Epsilon: 0.1\n",
      "656  Episode Reward:  22.0 .Epsilon: 0.1\n",
      "657  Episode Reward:  114.0 .Epsilon: 0.1\n",
      "658  Episode Reward:  110.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "659  Episode Reward:  117.0 .Epsilon: 0.1\n",
      "660  Episode Reward:  18.0 .Epsilon: 0.1\n",
      "661  Episode Reward:  116.0 .Epsilon: 0.1\n",
      "662  Episode Reward:  115.0 .Epsilon: 0.1\n",
      "663  Episode Reward:  115.0 .Epsilon: 0.1\n",
      "664  Episode Reward:  110.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "665  Episode Reward:  128.0 .Epsilon: 0.1\n",
      "666  Episode Reward:  136.0 .Epsilon: 0.1\n",
      "667  Episode Reward:  119.0 .Epsilon: 0.1\n",
      "668  Episode Reward:  98.0 .Epsilon: 0.1\n",
      "669  Episode Reward:  12.0 .Epsilon: 0.1\n",
      "670  Episode Reward:  30.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "671  Episode Reward:  93.0 .Epsilon: 0.1\n",
      "672  Episode Reward:  108.0 .Epsilon: 0.1\n",
      "673  Episode Reward:  104.0 .Epsilon: 0.1\n",
      "674  Episode Reward:  130.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "675  Episode Reward:  96.0 .Epsilon: 0.1\n",
      "676  Episode Reward:  14.0 .Epsilon: 0.1\n",
      "677  Episode Reward:  100.0 .Epsilon: 0.1\n",
      "678  Episode Reward:  13.0 .Epsilon: 0.1\n",
      "679  Episode Reward:  40.0 .Epsilon: 0.1\n",
      "680  Episode Reward:  14.0 .Epsilon: 0.1\n",
      "681  Episode Reward:  88.0 .Epsilon: 0.1\n",
      "682  Episode Reward:  36.0 .Epsilon: 0.1\n",
      "683  Episode Reward:  12.0 .Epsilon: 0.1\n",
      "684  Episode Reward:  91.0 .Epsilon: 0.1\n",
      "685  Episode Reward:  16.0 .Epsilon: 0.1\n",
      "686  Episode Reward:  35.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "687  Episode Reward:  87.0 .Epsilon: 0.1\n",
      "688  Episode Reward:  103.0 .Epsilon: 0.1\n",
      "689  Episode Reward:  108.0 .Epsilon: 0.1\n",
      "690  Episode Reward:  15.0 .Epsilon: 0.1\n",
      "691  Episode Reward:  101.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "692  Episode Reward:  115.0 .Epsilon: 0.1\n",
      "693  Episode Reward:  113.0 .Epsilon: 0.1\n",
      "694  Episode Reward:  110.0 .Epsilon: 0.1\n",
      "695  Episode Reward:  58.0 .Epsilon: 0.1\n",
      "696  Episode Reward:  27.0 .Epsilon: 0.1\n",
      "697  Episode Reward:  55.0 .Epsilon: 0.1\n",
      "698  Episode Reward:  48.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "699  Episode Reward:  116.0 .Epsilon: 0.1\n",
      "700  Episode Reward:  109.0 .Epsilon: 0.1\n",
      "701  Episode Reward:  112.0 .Epsilon: 0.1\n",
      "702  Episode Reward:  119.0 .Epsilon: 0.1\n",
      "703  Episode Reward:  13.0 .Epsilon: 0.1\n",
      "704  Episode Reward:  13.0 .Epsilon: 0.1\n",
      "705  Episode Reward:  59.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "706  Episode Reward:  102.0 .Epsilon: 0.1\n",
      "707  Episode Reward:  107.0 .Epsilon: 0.1\n",
      "708  Episode Reward:  102.0 .Epsilon: 0.1\n",
      "709  Episode Reward:  109.0 .Epsilon: 0.1\n",
      "710  Episode Reward:  105.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "711  Episode Reward:  10.0 .Epsilon: 0.1\n",
      "712  Episode Reward:  96.0 .Epsilon: 0.1\n",
      "713  Episode Reward:  97.0 .Epsilon: 0.1\n",
      "714  Episode Reward:  22.0 .Epsilon: 0.1\n",
      "715  Episode Reward:  100.0 .Epsilon: 0.1\n",
      "716  Episode Reward:  98.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "717  Episode Reward:  98.0 .Epsilon: 0.1\n",
      "718  Episode Reward:  94.0 .Epsilon: 0.1\n",
      "719  Episode Reward:  99.0 .Epsilon: 0.1\n",
      "720  Episode Reward:  110.0 .Epsilon: 0.1\n",
      "721  Episode Reward:  23.0 .Epsilon: 0.1\n",
      "722  Episode Reward:  100.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "723  Episode Reward:  99.0 .Epsilon: 0.1\n",
      "724  Episode Reward:  18.0 .Epsilon: 0.1\n",
      "725  Episode Reward:  98.0 .Epsilon: 0.1\n",
      "726  Episode Reward:  98.0 .Epsilon: 0.1\n",
      "727  Episode Reward:  107.0 .Epsilon: 0.1\n",
      "728  Episode Reward:  20.0 .Epsilon: 0.1\n",
      "729  Episode Reward:  100.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "730  Episode Reward:  102.0 .Epsilon: 0.1\n",
      "731  Episode Reward:  104.0 .Epsilon: 0.1\n",
      "732  Episode Reward:  99.0 .Epsilon: 0.1\n",
      "733  Episode Reward:  103.0 .Epsilon: 0.1\n",
      "734  Episode Reward:  19.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "735  Episode Reward:  106.0 .Epsilon: 0.1\n",
      "736  Episode Reward:  122.0 .Epsilon: 0.1\n",
      "737  Episode Reward:  102.0 .Epsilon: 0.1\n",
      "738  Episode Reward:  109.0 .Epsilon: 0.1\n",
      "739  Episode Reward:  30.0 .Epsilon: 0.1\n",
      "740  Episode Reward:  37.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "741  Episode Reward:  98.0 .Epsilon: 0.1\n",
      "742  Episode Reward:  114.0 .Epsilon: 0.1\n",
      "743  Episode Reward:  102.0 .Epsilon: 0.1\n",
      "744  Episode Reward:  114.0 .Epsilon: 0.1\n",
      "745  Episode Reward:  101.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "746  Episode Reward:  107.0 .Epsilon: 0.1\n",
      "747  Episode Reward:  94.0 .Epsilon: 0.1\n",
      "748  Episode Reward:  105.0 .Epsilon: 0.1\n",
      "749  Episode Reward:  105.0 .Epsilon: 0.1\n",
      "750  Episode Reward:  103.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "751  Episode Reward:  104.0 .Epsilon: 0.1\n",
      "752  Episode Reward:  109.0 .Epsilon: 0.1\n",
      "753  Episode Reward:  19.0 .Epsilon: 0.1\n",
      "754  Episode Reward:  14.0 .Epsilon: 0.1\n",
      "755  Episode Reward:  71.0 .Epsilon: 0.1\n",
      "756  Episode Reward:  122.0 .Epsilon: 0.1\n",
      "757  Episode Reward:  16.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "758  Episode Reward:  111.0 .Epsilon: 0.1\n",
      "759  Episode Reward:  125.0 .Epsilon: 0.1\n",
      "760  Episode Reward:  117.0 .Epsilon: 0.1\n",
      "761  Episode Reward:  123.0 .Epsilon: 0.1\n",
      "762  Episode Reward:  19.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "763  Episode Reward:  118.0 .Epsilon: 0.1\n",
      "764  Episode Reward:  111.0 .Epsilon: 0.1\n",
      "765  Episode Reward:  16.0 .Epsilon: 0.1\n",
      "766  Episode Reward:  116.0 .Epsilon: 0.1\n",
      "767  Episode Reward:  115.0 .Epsilon: 0.1\n",
      "768  Episode Reward:  13.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "769  Episode Reward:  125.0 .Epsilon: 0.1\n",
      "770  Episode Reward:  116.0 .Epsilon: 0.1\n",
      "771  Episode Reward:  110.0 .Epsilon: 0.1\n",
      "772  Episode Reward:  12.0 .Epsilon: 0.1\n",
      "773  Episode Reward:  122.0 .Epsilon: 0.1\n",
      "774  Episode Reward:  114.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "775  Episode Reward:  113.0 .Epsilon: 0.1\n",
      "776  Episode Reward:  11.0 .Epsilon: 0.1\n",
      "777  Episode Reward:  109.0 .Epsilon: 0.1\n",
      "778  Episode Reward:  108.0 .Epsilon: 0.1\n",
      "779  Episode Reward:  109.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "780  Episode Reward:  113.0 .Epsilon: 0.1\n",
      "781  Episode Reward:  94.0 .Epsilon: 0.1\n",
      "782  Episode Reward:  98.0 .Epsilon: 0.1\n",
      "783  Episode Reward:  18.0 .Epsilon: 0.1\n",
      "784  Episode Reward:  23.0 .Epsilon: 0.1\n",
      "785  Episode Reward:  107.0 .Epsilon: 0.1\n",
      "786  Episode Reward:  91.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "787  Episode Reward:  92.0 .Epsilon: 0.1\n",
      "788  Episode Reward:  42.0 .Epsilon: 0.1\n",
      "789  Episode Reward:  92.0 .Epsilon: 0.1\n",
      "790  Episode Reward:  17.0 .Epsilon: 0.1\n",
      "791  Episode Reward:  100.0 .Epsilon: 0.1\n",
      "792  Episode Reward:  107.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "793  Episode Reward:  104.0 .Epsilon: 0.1\n",
      "794  Episode Reward:  121.0 .Epsilon: 0.1\n",
      "795  Episode Reward:  100.0 .Epsilon: 0.1\n",
      "796  Episode Reward:  14.0 .Epsilon: 0.1\n",
      "797  Episode Reward:  148.0 .Epsilon: 0.1\n",
      "798  Episode Reward:  18.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "799  Episode Reward:  120.0 .Epsilon: 0.1\n",
      "800  Episode Reward:  110.0 .Epsilon: 0.1\n",
      "801  Episode Reward:  106.0 .Epsilon: 0.1\n",
      "802  Episode Reward:  146.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "803  Episode Reward:  160.0 .Epsilon: 0.1\n",
      "804  Episode Reward:  94.0 .Epsilon: 0.1\n",
      "805  Episode Reward:  164.0 .Epsilon: 0.1\n",
      "806  Episode Reward:  16.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "807  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "808  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "809  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "810  Episode Reward:  305.0 .Epsilon: 0.1\n",
      "811  Episode Reward:  198.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "812  Episode Reward:  204.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "813  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "814  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "815  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "816  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "817  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "818  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "819  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "820  Episode Reward:  279.0 .Epsilon: 0.1\n",
      "821  Episode Reward:  10.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "822  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "823  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "824  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "825  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "826  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "827  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "828  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "829  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "830  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "831  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "832  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "833  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "834  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "835  Episode Reward:  53.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "836  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "837  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "838  Episode Reward:  10.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "839  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "840  Episode Reward:  160.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "841  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "842  Episode Reward:  486.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "843  Episode Reward:  477.0 .Epsilon: 0.1\n",
      "844  Episode Reward:  173.0 .Epsilon: 0.1\n",
      "845  Episode Reward:  135.0 .Epsilon: 0.1\n",
      "846  Episode Reward:  145.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "847  Episode Reward:  182.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "848  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "849  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "850  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "851  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "852  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "853  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "854  Episode Reward:  308.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "855  Episode Reward:  278.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "856  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "857  Episode Reward:  285.0 .Epsilon: 0.1\n",
      "858  Episode Reward:  346.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "859  Episode Reward:  288.0 .Epsilon: 0.1\n",
      "860  Episode Reward:  166.0 .Epsilon: 0.1\n",
      "861  Episode Reward:  150.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "862  Episode Reward:  171.0 .Epsilon: 0.1\n",
      "863  Episode Reward:  136.0 .Epsilon: 0.1\n",
      "864  Episode Reward:  177.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "865  Episode Reward:  148.0 .Epsilon: 0.1\n",
      "866  Episode Reward:  150.0 .Epsilon: 0.1\n",
      "867  Episode Reward:  127.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "868  Episode Reward:  179.0 .Epsilon: 0.1\n",
      "869  Episode Reward:  149.0 .Epsilon: 0.1\n",
      "870  Episode Reward:  138.0 .Epsilon: 0.1\n",
      "871  Episode Reward:  57.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "872  Episode Reward:  166.0 .Epsilon: 0.1\n",
      "873  Episode Reward:  150.0 .Epsilon: 0.1\n",
      "874  Episode Reward:  123.0 .Epsilon: 0.1\n",
      "875  Episode Reward:  34.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "876  Episode Reward:  155.0 .Epsilon: 0.1\n",
      "877  Episode Reward:  142.0 .Epsilon: 0.1\n",
      "878  Episode Reward:  132.0 .Epsilon: 0.1\n",
      "879  Episode Reward:  134.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "880  Episode Reward:  130.0 .Epsilon: 0.1\n",
      "881  Episode Reward:  125.0 .Epsilon: 0.1\n",
      "882  Episode Reward:  131.0 .Epsilon: 0.1\n",
      "883  Episode Reward:  121.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "884  Episode Reward:  118.0 .Epsilon: 0.1\n",
      "885  Episode Reward:  117.0 .Epsilon: 0.1\n",
      "886  Episode Reward:  114.0 .Epsilon: 0.1\n",
      "887  Episode Reward:  124.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "888  Episode Reward:  141.0 .Epsilon: 0.1\n",
      "889  Episode Reward:  122.0 .Epsilon: 0.1\n",
      "890  Episode Reward:  123.0 .Epsilon: 0.1\n",
      "891  Episode Reward:  131.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "892  Episode Reward:  151.0 .Epsilon: 0.1\n",
      "893  Episode Reward:  134.0 .Epsilon: 0.1\n",
      "894  Episode Reward:  126.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "895  Episode Reward:  142.0 .Epsilon: 0.1\n",
      "896  Episode Reward:  11.0 .Epsilon: 0.1\n",
      "897  Episode Reward:  131.0 .Epsilon: 0.1\n",
      "898  Episode Reward:  124.0 .Epsilon: 0.1\n",
      "899  Episode Reward:  129.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "900  Episode Reward:  126.0 .Epsilon: 0.1\n",
      "901  Episode Reward:  126.0 .Epsilon: 0.1\n",
      "902  Episode Reward:  297.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "903  Episode Reward:  85.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "904  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "905  Episode Reward:  126.0 .Epsilon: 0.1\n",
      "906  Episode Reward:  134.0 .Epsilon: 0.1\n",
      "907  Episode Reward:  125.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "908  Episode Reward:  145.0 .Epsilon: 0.1\n",
      "909  Episode Reward:  129.0 .Epsilon: 0.1\n",
      "910  Episode Reward:  129.0 .Epsilon: 0.1\n",
      "911  Episode Reward:  134.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "912  Episode Reward:  120.0 .Epsilon: 0.1\n",
      "913  Episode Reward:  124.0 .Epsilon: 0.1\n",
      "914  Episode Reward:  132.0 .Epsilon: 0.1\n",
      "915  Episode Reward:  123.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "916  Episode Reward:  138.0 .Epsilon: 0.1\n",
      "917  Episode Reward:  127.0 .Epsilon: 0.1\n",
      "918  Episode Reward:  131.0 .Epsilon: 0.1\n",
      "919  Episode Reward:  127.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "920  Episode Reward:  28.0 .Epsilon: 0.1\n",
      "921  Episode Reward:  129.0 .Epsilon: 0.1\n",
      "922  Episode Reward:  148.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "923  Episode Reward:  381.0 .Epsilon: 0.1\n",
      "924  Episode Reward:  61.0 .Epsilon: 0.1\n",
      "925  Episode Reward:  87.0 .Epsilon: 0.1\n",
      "926  Episode Reward:  60.0 .Epsilon: 0.1\n",
      "927  Episode Reward:  79.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "928  Episode Reward:  127.0 .Epsilon: 0.1\n",
      "929  Episode Reward:  158.0 .Epsilon: 0.1\n",
      "930  Episode Reward:  77.0 .Epsilon: 0.1\n",
      "931  Episode Reward:  93.0 .Epsilon: 0.1\n",
      "932  Episode Reward:  79.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "933  Episode Reward:  83.0 .Epsilon: 0.1\n",
      "934  Episode Reward:  134.0 .Epsilon: 0.1\n",
      "935  Episode Reward:  134.0 .Epsilon: 0.1\n",
      "936  Episode Reward:  135.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "937  Episode Reward:  76.0 .Epsilon: 0.1\n",
      "938  Episode Reward:  65.0 .Epsilon: 0.1\n",
      "939  Episode Reward:  78.0 .Epsilon: 0.1\n",
      "940  Episode Reward:  138.0 .Epsilon: 0.1\n",
      "941  Episode Reward:  82.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "942  Episode Reward:  193.0 .Epsilon: 0.1\n",
      "943  Episode Reward:  213.0 .Epsilon: 0.1\n",
      "944  Episode Reward:  30.0 .Epsilon: 0.1\n",
      "945  Episode Reward:  135.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "946  Episode Reward:  138.0 .Epsilon: 0.1\n",
      "947  Episode Reward:  146.0 .Epsilon: 0.1\n",
      "948  Episode Reward:  135.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "949  Episode Reward:  351.0 .Epsilon: 0.1\n",
      "950  Episode Reward:  47.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "951  Episode Reward:  309.0 .Epsilon: 0.1\n",
      "952  Episode Reward:  91.0 .Epsilon: 0.1\n",
      "953  Episode Reward:  213.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "954  Episode Reward:  412.0 .Epsilon: 0.1\n",
      "955  Episode Reward:  153.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "956  Episode Reward:  325.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "957  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "958  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "959  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "960  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "961  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "962  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "963  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "964  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "965  Episode Reward:  181.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "966  Episode Reward:  383.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "967  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "968  Episode Reward:  425.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "969  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "970  Episode Reward:  38.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "971  Episode Reward:  175.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "972  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "973  Episode Reward:  337.0 .Epsilon: 0.1\n",
      "974  Episode Reward:  119.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "975  Episode Reward:  258.0 .Epsilon: 0.1\n",
      "976  Episode Reward:  194.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "977  Episode Reward:  190.0 .Epsilon: 0.1\n",
      "978  Episode Reward:  165.0 .Epsilon: 0.1\n",
      "979  Episode Reward:  178.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "980  Episode Reward:  155.0 .Epsilon: 0.1\n",
      "981  Episode Reward:  91.0 .Epsilon: 0.1\n",
      "982  Episode Reward:  94.0 .Epsilon: 0.1\n",
      "983  Episode Reward:  102.0 .Epsilon: 0.1\n",
      "984  Episode Reward:  84.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "985  Episode Reward:  56.0 .Epsilon: 0.1\n",
      "986  Episode Reward:  179.0 .Epsilon: 0.1\n",
      "987  Episode Reward:  52.0 .Epsilon: 0.1\n",
      "988  Episode Reward:  39.0 .Epsilon: 0.1\n",
      "989  Episode Reward:  110.0 .Epsilon: 0.1\n",
      "990  Episode Reward:  42.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "991  Episode Reward:  234.0 .Epsilon: 0.1\n",
      "992  Episode Reward:  173.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "993  Episode Reward:  136.0 .Epsilon: 0.1\n",
      "994  Episode Reward:  282.0 .Epsilon: 0.1\n",
      "995  Episode Reward:  78.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "996  Episode Reward:  251.0 .Epsilon: 0.1\n",
      "997  Episode Reward:  279.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "998  Episode Reward:  303.0 .Epsilon: 0.1\n",
      "WEIGHTS SAVED\n",
      "999  Episode Reward:  308.0 .Epsilon: 0.1\n"
     ]
    }
   ],
   "source": [
    "numberEpisodes = 1000\n",
    "\n",
    "discount = 0.99\n",
    "counter = 0\n",
    "epsilon = 1\n",
    "epsilonDecay = 0.995\n",
    "rewards = []\n",
    "\n",
    "memorySize = 5000\n",
    "memoryPreviousStates = np.zeros((memorySize,4))\n",
    "memoryStates = np.zeros((memorySize,4))\n",
    "memoryARD = np.zeros((memorySize,3))\n",
    "numberOfLearningFromReplaySteps = 512\n",
    "itemsInMemory = 0\n",
    "initialiseReplayAfter = 1000 \n",
    "\n",
    "for episode in range (numberEpisodes):\n",
    "    epsilon = max (epsilon*epsilonDecay, 0.1)\n",
    "    \n",
    "    \n",
    "    done = False\n",
    "    prevObs = env.reset()\n",
    "    \n",
    "    episodeReward = 0\n",
    "    for step in range (600):\n",
    "        if done:\n",
    "            break\n",
    "        #env.render()\n",
    "        \n",
    "\n",
    "        target = model.predict(np.expand_dims(prevObs, axis = 0))[0]\n",
    "        if (random.random()<epsilon):\n",
    "            action = random.choice([0,1])\n",
    "        else:\n",
    "            action = np.argmax(target)\n",
    "        \n",
    "        obs, reward, done, _ = env.step (action)\n",
    "\n",
    "\n",
    "        memoryPreviousStates[counter] = prevObs\n",
    "        memoryStates[counter] = obs\n",
    "        if (episodeReward <= 498):\n",
    "            memoryARD[counter] = [float(action), reward, float (done)]\n",
    "        else: # Is not the fault of the agent's action that the environment restarts after 500 epochs. So we set done=False for the memory replay\n",
    "            memoryARD[counter] = [float(action), reward, float (0)]\n",
    "        \n",
    "        if (counter%500 == 0):\n",
    "            model.save_weights(\"weights.h5\")\n",
    "            staticModel.load_weights(\"weights.h5\")\n",
    "            print (\"WEIGHTS SAVED\")\n",
    "        \n",
    "        if (itemsInMemory < (counter+1)):\n",
    "            itemsInMemory = min ((itemsInMemory+1), memorySize)\n",
    "        if (itemsInMemory >= initialiseReplayAfter):\n",
    "            executeMemoryReplayStep(memoryPreviousStates, memoryStates, memoryARD, model, staticModel, itemsInMemory)\n",
    "\n",
    "\n",
    "        prevObs = obs\n",
    "        episodeReward += reward\n",
    "        counter = (counter + 1) % memorySize\n",
    "    print (episode, \" Episode Reward: \", episodeReward, \".Epsilon:\", epsilon)\n",
    "    rewards.append(episodeReward)\n",
    "\n",
    "env.close()\n",
    "model.save_weights(\"weights.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 265
    },
    "colab_type": "code",
    "id": "sSRzCzHLHEMF",
    "outputId": "16f81d29-237b-4082-96d0-78d8b1e73588"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO2dd5wW1dXHf+fZBksvKyBtARHEQhFRAjH2iBrNa0yiyRuN+sYUNb5pRpOYnmh8E43GxMSuKWqixoJoRMSGCqxKk7rA0mEXdlm2sPW57x/PzPNMuXfmTnvK7P1+PrDPzNy5c6fcM2fOPedcYoxBoVAoFPEikesGKBQKhSJ8lHBXKBSKGKKEu0KhUMQQJdwVCoUihijhrlAoFDGkONcNAIChQ4eyysrKXDdDoVAoCor3339/P2OsgrctL4R7ZWUlqqqqct0MhUKhKCiIaJtomzLLKBQKRQxRwl2hUChiiBLuCoVCEUOUcFcoFIoYooS7QqFQxBAp4U5ENUS0mohWEFGVtm4wES0kok3a30HaeiKiu4momohWEdGMKE9AoVAoFHa8aO6nM8amMcZmass3AVjEGJsIYJG2DADzAEzU/l0D4N6wGqtQKBQKOYL4uV8E4DTt96MAXgfwfW39YyyVS/g9IhpIRCMYY3uCNFSRe1btPAgC4fhRAwLX9e7mAzjU1on/fLQXLe1dKC0uQllxAkcP64uPTRiK/c3tmFDRF6MHl0vX2dGVxCPvbEVzWxcOHu5EeWkxOruTaG7rwmdnjsLMysHpsq+t34f2ziTmHT+C27Z+vYqxYW8TLp4xEkSE9XsPobmtC6t2NmL2hCE4ZkR/6XbVHmrDMx/uwmWzxmDRun2o2d+CriTD2j2HMPeooTipcjA27GvCxdNHorjIXd861NaJfyzdjv+aPhLD+veSbkehsLmuGfNX7sGYIb2xta4FIAIsqcm37G9BaXECQ/qU4vvnTsYTy3eg9lAbSooSOHn8EMwaN1hQu5iX1+zBzMrBGNq3zFe7X127D6t2HgQAlJUUoa6pHceM6IfPzRwNIvJVZxBkhTsD8AoRMQB/YYzdB2CYQWDvBTBM+z0SwA7Dvju1dSbhTkTXIKXZY8yYMf5ar8gqF96zBABQc9v5geu67P73XMuUFSew4ZfzpOtcvasRv16wnrutals9Fn3ntPTyVY+kguZ452Js27D+vTB34lCc+/u30usmHtEXC7/9Cel23f/WFtz/1lYM7F2Cm/+92iSnXt9Ql/59oLkDXz9tgmt9r67dh9teWo+6pnbccsEU6XYUCo8sqcFf3+PH5nDkPDq7GR55p8a0zuszeqitE1/72wc4fuQAvHD9XE/76vz4uTXY3dhmWz+0bxnOPGYYZ49okTXLzGWMzUDK5HItEZ1q3Khp6Z5m/WCM3ccYm8kYm1lRwY2eVfRw2ruSnsontV5/16XTbNua27t8tYG339b9LZ7qaGpL1XG4sxuMATeeO4lbrr6lXaq+zu6kVm+np3YUCt0G6T3nqCEoLy0CAHzvk5Ow9dbzccfnpprKb65rBpBSBnwfszt1zB0Nrb7r6EoyXHrSaNtXg37/s43U1WCM7dL+1gL4N4BZAPYR0QgA0P7WasV3ARht2H2Utk6hiIzuJMOvXlwHAEjk4BNYhu5kSoAU5Wn78gWvk8MdOpx6yeX6vjOkvizs63Mz252rcCeiPkTUT/8N4BwAawA8D+AKrdgVAJ7Tfj8P4HLNa+YUAI3K3q6Imjc31WHFjpS9szgRXicPU16khXuI7Ys7BPdrdUjTjHP9zkwZMGRanB1kbO7DAPxbGxAoBvAPxtjLRLQcwD+J6GoA2wB8Tiu/AMB5AKoBtAK4MvRWKxQW9M9qIHrh6VWI6OW7QhbuPWX6Y+vVsl7/xnzR3JlAc8/RfXIV7oyxLQCmctYfAHAmZz0DcG0orVMoJDH2n+KifNGdzCSV5i5J5m7KyGtduOdcc0fqRZTrduioCFVFLGAG9SjXGpwIfaAwX9tXKFgNH7q5K9dXlTEGovwxzCjhrogFSYPqHrlZxmfnVZq7HFYzhu4j7vZOTIRwXYOYUEQW91yZZZRwV8QEg809TzXjLuUt4xle8I/o8uX6quo293y5vUq4K2KBSTsKsXOF2U91s4zS3OUhZO6B2xdTrs1djDFuG3I17q2EuyIWFILjiDLLyOHXjJGLEH8jerOV5q5QhIhRIEQ+oOW5+tQOullGZBuWFWr6+RXCCy0oXgRlroVq2iyTcwNRCiXcFbEgWQBO30llc5fCGNFJ6f8kBlQDXNYwbgljTDCgmqcRqgpFIWAyueep7Aw7iKmnIjK/5Fpj1tMP5Mvzp4S7IhZkUzvy23eTakDVM0RkGFB1JteXlTF+G9WAqkKRh4Q5SJfJLRNalbHEPH5iR3RHcj+gyvhau/JzVyj8Y7S556tenB5QDSqE8vUEc0yuzSGpAVWyvWTyNiukQlEIFMB4alpzL06obueEdfxEOkI155p77l8wRtRTpogFpk/5iHuYU/WNreIJNDKukGG3KM54iFDNtWBlqUHdXDdDRz1miliQD4r7kur9mPrzV/D6hlrTel3o6GmJlSukN/TLFWWEahhffiKbu8oto1AEwOgtkyvZ+f62BgBAVU0Dd3tXMjU9nvKWccb8FWbfLhLyuX5nJplK+atQhE5U2lEY/VRvm5ufu9dTKIRxhqB4uf5B7lUYlzKT8jf8uv2ghLsiFuTKI8ELoU2zl/+nGgjrvUz7ubtEqubFgGreWNyVcFfEBDff6DARmgVE5XWbe1iukD0Io7eMTFm/hBEEl0n5mx/3Vwl3RSzIpjLrt++GlX6gEL5SwoL3IhVdvXx4aRLsLwo1oKpQBMA6CPezC4/NXVsEwlfv9Hkgg/KbHKT8DSp/M/c2f26uEu6KWGAVqKXF2X+03fp1WBpcTxhI1fGU8je6Zrii3xMi+4tCRagqFAEwC7zwhrX8KGK2OUC1v3qKhHwadMtHbBGq6d/OkaphBIf5tb2nJ+oA5c3LVwl3RSywdkpZoZzNbJJ50ucLDtl7GeSlGfQxMJrc8uU+K+GuiAXWDiXb0bMh2/VDGD/dw6ivJ2C8j7xfRnIZG5bU7y3UgKpCESpuUY1+4UdIiso6H7gnCeUgMJ9+rcEGVIPdHX3/PBpPVcJdEQ9s0+zJmmXctocgkfWmhGUCyhebbq7Ix8Rhma8yu81dRagqFAGwKnuy/TyrNnfDp7tCjGlA1fh/lCl/Q3wM8mU+XyXcFbHAZnMPSY3jmmV8ml/CdonrCcFMqckvJMtG2xRHdHme4GjuuUIJd0UsMGeFlPebyEY/1IWTlhRSKKxkhUJPEOo83OZSDZTy1/ee+v5GbxmrXUb5uSsUvrGZ3KVdIcNvi/BY2TtUQeM3T1Aube7TfrYw1Qbkz5iIEu6KWGDLJBhSR/fjOy3q3Jmvi4C5ZfJEeGQDL2axYInD/O8LAB3dyXQb7BGquUFauBNRERF9SETzteVxRLSUiKqJ6EkiKtXWl2nL1dr2ymiarlBksA+omnv6vkPtSCbt3czNxMHb7pb9UaaNCjH2mAXtr8sFzo/EYfkTxeRFc78BwDrD8m8A3MkYOwpAA4CrtfVXA2jQ1t+plVMoIsU+oGov86fXq7PSFhH56Aud7/Auld+XqxNhjWNwbe45Qkq4E9EoAOcDeEBbJgBnAHhKK/IogE9rvy/SlqFtP5PyKVWaIpbIBDEt3VrvuB+PMPPAhJY4LJxqCgKj6IjUFTJEbH7ueR6h+nsANwLQxvsxBMBBxliXtrwTwEjt90gAOwBA296olVcoIsPqWxypPuFmfhGI30ziMIUTfvMEBYpQDUkAE1HevHxdhTsRXQCgljH2fpgHJqJriKiKiKrq6urCrFrRw6Ec5V10O2ponb4HGe+5ZhlRhGqkLZEjQbzcMvnrCjkHwIVEVAPgCaTMMXcBGEhExVqZUQB2ab93ARgNANr2AQAOWCtljN3HGJvJGJtZUVER6CQUCv9ZIV0KhCAxdKFvDFFXiLFkb05fP7erFkbisKBimEKoIyxchTtj7GbG2CjGWCWASwG8xhj7IoDFAC7Ril0B4Dnt9/PaMrTtr7FcvboUPQZ7ahnJrJA+uqJfGRJ6N+ihvSqKCbLDupS83DK5Ioif+/cBfJuIqpGyqT+orX8QwBBt/bcB3BSsiQqFO7YJHnIai25dZLzVnN3kpEKeyI6sYHpJu9zUfPggIp5ZJkdtKXYvkoEx9jqA17XfWwDM4pRpA/DZENqmUEjjMylkdiNUQ0gcNutXr6K2qT2U9uQtLtHGoq+yYAOqIblChlJLOKgIVUUs8Buhmg3ZrsuNMPyfYy/YLfidQzVnlmCOt0y+u0IqFHkNJ/g06+iCSBR+7pY4TJHC+BI0pm8mWH+YMdrcvQpUUfmOriT+uLgaHV1JfgELKreMQhE2pqyQgOwHcpganuiI+dLZ404YE2RbeeSdrfi//2zAQ0u2SpUvuAhVhSLfkUk/ILOfTAG/tt1MEJNS3WUh+fFU03UNS7y2dnSb/rrB89jJ+8RhCkU+Y4tQzVE7+GjeMsoTUgpzEjiyvUxlcsvkakrDfHrulHBXxA6eQBDh1nn9fGLbIxRTf9Oaez5JgBhhvOde75p7jiHZNvByy+RvhKpCUXCEFWjE65dehXPGW0YhgygJXDpSVXADEibNPaS2eCyfTyY3JdwVscNTEFMWM/ipgVXveHmRmrxlPIrl0AZBOZp7rlDCXRELfE+z5zZZh4eOKpwbNX2M8I7VE9Gvr9DmbvgdWpbHiMtHiRLuithB5P/zOAyZIMrnrYS3HOYXLuU05a9ns0weDago4a6IJz6zQsrkBfHbfYMOqPbE/HueIlQjsLl7bQcvK6SKUFUoQsSnyd2+nTGs3X0oUFvkjDLu5EMUbjYQmdjIsmzFNKAalg3do2SOIpDKL3nUFIUiHLy4QlrhdeW/L90mfVxunborZFLX3P21rbunSHcDXq5UlCl/Zc18BMqbLyxPWSEVinzFFqEqu5+L6yPXLONRiMim/HXDGqjVE+BFqIqzQmZ+hx8wxq+wrbMbz6/YzW2D275Ro4S7InZ4cYV0N8t4P75ol6Apf62ae75oiGFjf1FLas2Bgpj8XcvbXlqPR96p8bVv1CizjCKW+PeW8S8whS+UtLdMMGHcHVNh7gTvPsrMoVrf3BF5OwCgrtmcgpn3VacGVBWKkCD4n0PV3hHtPdOtapFpR//r1zSc7CE2d2uEqps5Rsdoc//KY1XejulxvahA/jhCKuGuiCny3jLRBxbpGnvQunrigKoXjN4y2+tb/VUiuMTyvvb+DhsFSrgrFFkiaMrfnmmWsS+Is0IGSD8QWkSrSvmrUEQGGafvccPFLBNKxGpIdSXlJgOKAcaJV+RfhUatOVsfOX6nd8wGSrgrYoF9guxwelmYWSGDSveeqLlzEQ6oBrnn4VzbBDflbyhVe29Lbg6rUESJfD4SW6h4BB/Reo1B0w/0xAFVIGNucbtsRpt77vIr54/qroS7IpbIBzG5LIN519QFkiWovOmJA6q8ay/S0BOJ8G3unmdiyqMgJiXcFbEj5T4XnVlG9OoQHTPjLaMPqPpDmWWciSLlL69ut3JqgmyFIkL85nOPoluGN6CaH0IjaiwJfw0Jw/SZmPj7BYpQFa735iqrUv4qFBHj3ywTXNjbtEZrPnef/b8nau5ePJ/M3jK5uVYENaCqUESGW4SqlzQAXsoG1dncjhWFm2Y+IroOngZUQ8KrB04eKe5KuCvigf3zWa6XCZTsULH5QvvNexNXae4A70qJrp5pDlWP10o8E5NXs4y340aJEu6K2EHk7App7JBuKX/DQLbOR9/dhm0HWsT1xFZXF+NpJqbomuFpqj+bwpCjt7IS7opYElZHDxLEtGZXI7YfaMVbm/ZL77+8psFTW+KI6BssnUBMcAEDzaEqcl/16grpuwXho/K5K2JHyuYu7mbG/ioa/PR1XMshL/jD2/4r49BThLsRLwI7FyYRe/oB+0xMYh96hmVb6zF5eH8MKC8JvW1Kc1fEEr/93O4amT8SNZ/aEiWmlL+G9RnNnb9foGn2AmaDTJfn1S0oW9fUjs/f9x6eXbHL20EkcRXuRNSLiJYR0Uoi+oiIfqatH0dES4momoieJKJSbX2Ztlytba+MpOUKhREPA1vevGXs64IqiE77O7WtJ2rugLz2Xlocvq4aSoSqoI7NdanxlfEVfTy2Sg6Zq9EO4AzG2FQA0wCcS0SnAPgNgDsZY0cBaABwtVb+agAN2vo7tXIKRdYgkvdIeX1jnWk5zPztYdMjZbuHN2lpkX/h7nbLZF8wBM6AquDOba9PCffKITkS7ixFs7ZYov1jAM4A8JS2/lEAn9Z+X6QtQ9t+JuVT2Jaix2Psarc8u0a4DYhGW3bqDo5jBT1EdRdGqMI5gVgkmrvH8l4099aObgBA37Johj6lrgYRFRHRCgC1ABYC2AzgIGOsSyuyE8BI7fdIADsAQNveCGAIp85riKiKiKrq6uqsmxUK35CHrJBu8PplrlSVHpJ9wDdBhPv8Vbt97WdPNQ37HAGCfTu6Ugn6o3gpAZLCnTHWzRibBmAUgFkAJgc9MGPsPsbYTMbYzIqKiqDVKRShEEQ71mW+Ww1BHPZ6AsZ7YHyRug2olgUQkn96fbPvfY1wv7wEz1S7JtyDtNsJT7Uyxg4CWAxgNoCBRKR/T4wCoA/57gIwGgC07QMAHAiltQqFBERAkUMsupP8DhKAEob1UQ2omnFLJWEkiM09LPgpf/l0dCWRIKA4onbLeMtUENFA7XdvAGcDWIeUkL9EK3YFgOe0389ry9C2v8Z6irFQkTeE9akbxYPr9x2QjfGAwoB/AaMybzjBC7iSvU/tXd0oKy6KoFUpZCz5IwA8SkRFSL0M/skYm09EawE8QUS/BPAhgAe18g8C+CsRVQOoB3BpBO1WKExY+4/Tp66Tv7iMwAxrCj+v9ERh7sXzKT9cIXkTZPMr6ehKRvpCchXujLFVAKZz1m9Byv5uXd8G4LOhtE6h8InfTmPriD4EqqtbnYOwcjLt5CqNbaEQpVlGPreMfL6i9q5kZPZ2QEWoKmJKWZH4c1fU2Wqb2rCz4bC5bB4NYvYU2W6OUM1IVbfJOowv9JljB/k7tq+9MshGqL6xsQ5PLN+RW81doQibqIdgiPxp7rN+tUi6/qhwHFDNoxdNtkhNmShXtsSguefqSvGmeOTd0iseWgYgOk8ZQGnuipgS2oCqBykhLfT9J75xWuwxiC5fLmZish+GkzjM4U6VRjigqoS7InYQkaMrpBd8TbMXkdjtKcLceP1MicMk9l3wzY9j6uiBoQV8eb2X3MfOoQqluStiRa5tx7k+vm9XyJ4i3Y1wLpYwnzsIU47sj0HlJaFfLFmPHe5kHQ7lo7S5K+GuyDov+Az1diKyZF0RZIX0S0/xlvGbfjcdwYpspmqwTqHIKeFw35TmrogV33pyRaT1uwlfL5/aDExea5OsMySTe4+AN/GK+0TZFJ5pLMKUv4AS7oqYEYVWxetAX/vEhPAPlEN6YqC3pzlUDblnkslo2uPaBpDtWVRmGYUiJPROftO8wPntBHOoOkucsHKD2+r1tVfhwc20iGB276iwtdWz5q68ZRQKR7x0Zi8KsCchYejZkWjZPUW6G+AJdNG7US+b4ESJBm6Hh/exl6kao4yqVcJdEQtEUY254GBrJw60dAi3O06z57jNGtYeT2lvcoXkpPwVkRlQpawNPlvbxGvjw0tqcLCV/zyUlagIVYUiNDx1ex9C4sXVe/Di6j2e93MjV3bkXGMToC4v70QiPE9It2qsxxFN0r2z4TAGlpfa1ivNXaFwoScEDnnxn44LXr7BjLb5sDV3aU8o4r9YRM2JUnNXwl2RN/QuCWdwye3zPSpzhhcB4KeOuJphrJhMbAST/7q+zgmi7L347Pnc+Y0TKR8lSnNXKJyJSu7lkzy1NSWP2pYP6EI/QXZ3xGy2gXdskfuvyIwTBkq4K/KGsJ5z9yAmeaLQlp1sxo4Dqj1EmBtP0+g26v58ZFIC+zXLBL3foiaK6o1y6F8Jd0VWefbDXcJtufVxyX+8ZBuME169n8LU3N2EvXW76AUk0tyjTB+thLsiqzzyTo1wW5AJppnZF9KlrId6OeuEftYeZuvxQ88Q5f4x55bJlZ+7wOYu0tyVWUYRF5w6Xa409wfe2iLcxli02pUXbGHtcZX2hvNijBmuv/NMTOlSOba58xA1R2nuitgQVXCJN9918+IvX1yHHfWtYTbHN84BTnGV5mI8TZai/40gQlWWlEnIfuykwS5zoLk9/TvKgDsl3BVZxbHPhTag6r2iLoFR1ItZJgy8DKjGVXM3vsTMg6vaX8H91U0cCcpmyl9LGwTrje255M/vZsorzV0RF5w6XZDnPKigE31RMMZCF6LK5i4PY96fizBT/nq992KzTKairftbMuX9NEoSJdwVWcXpcznQgKogH4lbWZ3uEFS9qDXpnugtwztHceKwzPawNXdxcJK9HO/QficgCYIS7oqskq+mhI6u/E/ckq/XLmyM52kU0vIRwHk4oCoS7srmrogLjt4yAZ5zD56Q3I7WLhDu/Gn25LQ4EX47tD0rpK9qCgsP7kpGV8iwBlS9Jg4TrRM990pzV8SGfHSFBJTmnk8wwW/dbOeezz17k3XY2iAMYlJ+7oqYs7muRbgtmM09WNn2rm5BWXmbr6z09T2gavWW8VdNQeFvQDV3k4kTEX82JlH5CNuihLvCM7nOquhajw/pKdLceacaZSY/J3qCMLfCfbkKXSH1v2TyKw8D2eCklEmIUy4HZhk1WYcia0QZWOIppQCncEtHl/T+QsXdYZ/y0iK0dnQ77u+GVRuNq5nGeH+8jKWky2Uh5W9tUxtm/WoRelnysYuyPHYLrH5Kc1fkHFGH84KbMhVMizG4QvrY+8nlO1xqdV4XGo5RTPZVySTD+9saImtOrklyxlNd0w8gzMRh/PVLt9QDANo6zVI79WJxdrVNmKYOVDZ3RQxw19z9Pehtnd2epqDjtUI0UUhYuWWMVTh1aCffdfs2hkfeqcFn7n0Hb2ysC9bAPMI8oCofv5DJ5+7/K1F2L5F3lThC1SjcvaQx9o+rcCei0US0mIjWEtFHRHSDtn4wES0kok3a30HaeiKiu4momohWEdGM6JqvyBbGvuJXKXIV7T4f9Mm3vIwnqzKat596rBqYjpdAoTC0RaevG179G/c1AQB2NRwOfvB8xM+AaoIiTz/Q1skfgBc11qy5G4R7mI2yIKO5dwH4DmNsCoBTAFxLRFMA3ARgEWNsIoBF2jIAzAMwUft3DYB7Q2+1oiBxE365dIVsE3nLSA6O1Ta14fmVu4X1k2SHdrpGNr2dZTTCRC4vXoQkOCcmOtW0qySi95YRCXeRScikuRulbi7NMoyxPYyxD7TfTQDWARgJ4CIAj2rFHgXwae33RQAeYyneAzCQiEaE3nJFVjF9Kkt0nH9V7cCC1XssdYTf4fx8fvN2EWnusnzx/qWh2L4dzTIOATP5kpY4DIznOWJAr/Rv+Xz50fu5C80yBaa5pyGiSgDTASwFMIwxpvfevQCGab9HAjCOTu3U1lnruoaIqoioqq4uPvbCuOJViH7vqVX4xt8/sNThvI8fAcXLCeMnAlTk585zqeOdxqbaZs/H5OF0jWzeMoa2RDkwl0vGDuljOzeZ3DLhRajy6xFr7vznQyTco0RauBNRXwBPA/hfxtgh4zaWupKeriZj7D7G2EzG2MyKigovuypyjG+bu6tZxvtD3+1Hc+ecwRZBcFVYGqB5QFVczqOzjMEsEx/hzgBMHTUAP/nUFJw6cWh6vdvzYR5QDbdN1heMWHPnt9H4YjYWyXn6ASIqQUqw/50x9oy2ep9ubtH+1mrrdwEYbdh9lLZOUcCYzTJ+63DeMSzNPcxvXe65+jl/yTY5apw2P/dMOuL4iPYUA8tLceWccamIT9tWFyEPCs3mLqpGpLmLxj6a27sNZfJEc6fUq+hBAOsYY3cYNj0P4Art9xUAnjOsv1zzmjkFQKPBfKPowUQxoCqaZMO5IV6Khm+9dXSF9DCgmirPtDoDNioGZHLL+E/5K/tOED13ogHVDXszxo5sDX7LaO5zAHwJwBlEtEL7dx6A2wCcTUSbAJylLQPAAgBbAFQDuB/AN8JvtiLbmF0ho/Uh5rF4fS32NrbZ1vNs4mEKumzP6OOkufNyy2Rs7pE1KfuI8+M6Ykw/kKomuMnO83UVlN97KDO1XlGWpLtr+gHG2NsQX9YzOeUZgGsDtkuRRzDGsKm2KXA9bp/KIo2WMYYrH1mOkQN7Y8lNZ5i2+dHcPQ4OmRY/NfVIrNnV6PmY0odz2Ma7fvrpx8nmDjjbpWWDmfwEoMlOZeg8+YZ9Y4dhwD5bg98qQlXhyvMrd+P8u99OL/u2ufvcTxfguw7aA3XCThBlxVp71N3Sg8nd5OceJ0Rn5Hbt9e36i87PlRHtY/8KEJll+BgHYPPJLKPo4WzYG1xrB+Da20QKjcgzAeBr7m59x1uSMZ5Q9S42ZPuzk7DmbtFWhjFNYD5h8i6yXD3htSTz9iRj2NnQKo4m5SF5GcWaO+GYEf1t69u1OIod9a2me6VmYlLklNUhmSH8ess4TaQRllC7eLotFANASkDo7e5dUuTbth3Gp7h9DtXMy8DXwHLMKNVSMetRrc+t2I25v1mM6/7xYdbaQAD++MUZeOyqWab1a/ccwofbG/Dx2xdjf3NHVtqihLvClbc27TctZ9ssIwowAgRBTC6ClPeSOWvKME5JbdBSK/7C9XMd6w0Dr6mLdeEetXkqmzhpxca/Voo14a5v/u6/VgIAXl23T/7YAgVE9r4QAf17leCU8UNs2+5YuJFbPiqUcFdkDdcBVcEnqqPmHoLNuW9ZMc485gh895yj0+ue/vpsAJpZBsDQvqU46oi+qXWBjyjGb/qBuGnuJgEuKQCLE3puGf8SU/ZxEpVzGtguzfIkL0q4KzwTlSukm1mGt52ffsAbf716FsqKi3De8ZkUSCeOHYzS4gQY9EChTFIqP8hqaF7SDwCZaxonm7ur+U6wXp8hK8iApUgBsU1O7kma7uYAACAASURBVMPEmO0ZvJRwV2QNv0FM+oAqb7sfoWZtx9ghfQBktC7dDzkTxs5MnTVKBxXHlL+8dTG1uRPnt5vM1u9bEFOHzSfG4812OnZpsRLuijwnqvQDInThzvvknXfXW/4aY8DqQmf8vGeMIZkEilzsvWHhxSyTcoVM/a5vabfvUKDsOdgWyPTFe04+9+d3pfaV9WsXx1mJv/CU5q6IL66ukHzBqQ+oygbquBWzNiNBZo2vxDAwp/uSGz/1/bykZF8JzukHOGYZbYc/Lt7suU35SFVNPQ60dOC19bW2bZkIVOc6enFm1VpWUx+oXXJe7s5ty3bOfSXcC4hkkuGe1zahviU7rlQi/GpVrjZ3wfr0gGpUnUPPJqj1vuKijPaVcjc0TwSRK+zpB6LIfJNb7lq0ybbO68dSnzL+lInZQNTU0qIEOkSzZEeEEu4FxNKt9fjtKxtx8zOrctoOv7myXaMpBT0jbVeXdUdzEcHW9usalf63OKEPzFHazz2oNUbWnOP12sbM1G5zu+Xhdn97FYcv3FfuOIgrH16GTk1Ae3XXLCtOeAumCgEl3AsI/c3f2hH9Q3KorRP3vLYJXRxtw8tk1Eb8DqjqAiwMt0fucbWOqB+npCgT6qhHqBpNQrkaUG3t6DItMxYv/3YRGTu2/QkZ1r/Mto43NR8APP3+Tt9teGnNXizeUIcd9a0AxKY50TNcVlLEjbSO8ktQCfcs097VjXeq3bUTHmHNLiPDnQs34revbMSCNXtt29q7/b1c/LZe19ytnjF+3f+sl1GXBfqLzGiWASw294jtMk5fN42HO23r9Bf+0L52IRdXjEpxccIuwooEX0nf0YKavGC9G+mMjkLNXftrWV9aROkUBNlCCfcs8+sX1+ELDywNlFkwG1nl9M/P+ma7F0Znt1+h6uYbzD8vq8BbueMgjv7hS3hi+XZBPd7apWuE+jmXJPQBVc1bJoua+6baZvxjaeq89jQexsNLtqa3NR62a+76C2l/czsqb3oRS3wqDvmA21cI33fcvtIppa4xIO7av3+AC+95W1iWh9Og/umTKoTPcCJBWY9FcE35qwiX6rrUXJsNrd4HRfVHI0rR3tbZjW8+/mG6I7VwTEBOEaNO+DfLmHfcur8FHd1JvLByt6922I6rHXjUoHKMHVKOn154LIDMpA9JxgxJqfxdfdm9Xly1By+u2oOLZ4zElx5churaZlxwwpGo6FfG1dyt/u3/WLodc44aaiuX7zy3YhdW7DjI3WZM4WulmONeeGLlIOFxGg93oqJf6ivnxdXucwhZj0kOivu1px9lK6dTlCB0+bVn+kQJ9yyjCwc/2l82Zt1ZUr0fr6zN5OJobu+ylTn9t6+j5rbzPdftd4JsXePRt+sDU01t9rb5Qa+3V0kR3vje6Yb1lLKtMnk3TLdjyPL4su2o1ibdPqy9YA9ZhDsDs31FiezN+c4NT6zwtV8x53z79yoRlj/UlhHuIt7f1oATx6ZeECKB7NVEWkSU9UAzZZbJMk5vfjeyMV+mVTjc+zrff5o30Mrjo90Z85N7WDn/zKznrQ9MfbT7ELe8G3abO/+4BL6fe1SMHVKe/v2zF9amf+svWKtwBzKmJB2OlaLgcXoxeg0M4n397GxoNS1/5t530r/bLHZy/dnhPcmOPu4JQpdPc6ZflHDPMoGmANOFXIiqe3tXt8lFSzQYZeWQpNZ8/t1vp+v3a6vWNfckA5ZuOeDqUubd5i6uJ+XnzmyBTlFQXsr/kG7RvGSsgmnXwcNp7V6nUDV3L5gGVD2+zfRreOrti9Pr5v5msah4+qtJR3ZyFGsfTWnuakA11ui33I+g010Bw+y+J/96ESbf8nJ6+V+S7mI8LVLEem2yD7+Jw4wukJ+/7z2bNhUUoeZuGFA1EobX0sJvnWpb17uE3x1b2rvAGLMJ9x319pmpZF/OhQjvy6+E4y3jhP7cbq9vdSxXs78F3/3XSjS1m695d5IhmWR4bgVvvEd87XOhuSube5bJmGW832j94Qiz/x5sNT+8soOUh9rkhXtGcw/Hy6bNIb+7VH2SkyDrZhmjn7v/S2/es3JoH1uJ3qX84JuW9m7UNbdL2WxFL6o/v7EZjy/bbhpTKBScxqlEmvuA3iVcE8zexjap5/DGp1ZxUxbsbDiM+97c4rq/laJE9pO7Kc09y6Tnd/RxnzP21dxrZ4cOyw9mpieU8JlbxmredzXLeLw+ouPquWUYYzAqiFF10TJBZOUzH+zErF8tkqrjg+0NXOF120vrse2As7ZaKBhPr6QogZEDe+P2S04wlZl//VzuS/vWl9Y7TtuoI8pFc/lDy/DE8h3cbU4WsSIi7jiVmqwjRngxyzy/cjd2GyaF1oV7FA9Ee1e3Jz9cL9GimaJuA6p8rHbO1nZ/mrse2i49qw5S3jJh2Nxl9hNp3Ys4SbSMfOHkMenfm2qbhcIHSLmxPr5sOxpb5b+88pkEAUtuOgOfmznatH704HJTfn4j//NoVSRtcRoLSyT43jJqDtUYIest09mdxDcf/xCfvy+TqlR/OKJ4HCb96GV86cGl0uVlB5aAzICoX1O19VhPVomFFxDeyy/j5x5+4BivNr9jodNHDzQtv+2Qn6VqWz1ufmY1vveU92jNXGG99LK3okRwQd/OQaCXcoXsEeh5TJxvtC4Q9za2pddFqbkDwDubD0iXdbJbWrfp5+r30Q47f4psbakB1VT7jZc8qghVp8hKJ2aMHYRnr52TXn5x9R586cGlmH3rItu90DOKHog4s2h3kuGv721De1c3GGP4+9JtNs8Tr3i97Gcew58XNyqc7l5qQFV5y8Qap2g7I+nAHcMj4zfsPwqcp4OzLstp7mJvGQ8NQ7hfNvrgqy53w4pQ5X0J+HVjLC1KYJpFe39r037saWyz2Zd1k1ZY12jVzoPcF/3tL6/HLc+uwfSfL8S7Ww7gh/9eg8sfWoomDwPxVrwOyH9q6pG49eLjfR8vTJTm3gPI9F/nG51+EAy9sNvFT3b3wcP4z0f2RF9R4PScWr9K9Gb7zT4euuYu7asMgJn93AH/5+GGXzdGPZDn4StPsm2zejXpPvNOEbfJJJPSMl9duw8X3rME/6oyu89u3NeEv2geJa0d3SjTppdbXtOAi//0jq2eR9+pcT2WX/r1MjsEfv/cyaZl62V46MszM/uWeXMmdLp9RSqIKf7Iph/QO5dRmdNlnEjWffbP7+Krf30/K2lgnQSkTbjr3jIu8kLUObzY98NEz+eeTNpnawoKrxq/Zhk9edbpk47AWRZTxPYDraYITD1dtNN5fO1v7+OoH77ketzNWp4kPV+Szr5DbaZlY6DVJkvQFQD85PmPXI/lF+tLbPLwfunf86+fiwG97akKvnvO0ZhQ0Qc3nDXR07GcvuoSCUKnCmKKN7IDqrpZxvhwZgYm+XvvaUx51rywarfJVq/z9qb9qNnf4rHFfBwnchaZZXxqvF7zuDsNfjIm3wpjhKqx3/p518i8FPy+OEoMEy+fM8Us3C/587umCEw9lYFV6G3a14Q3NtZhweo9ptxCThg/Lru6k+nn0vpsfP/p1VL1iQgymG09z5PHD07/Hj2onHsvrztjIhZ957TQBtFf/+5pKKJos4nyUMI9y+jPi5s2qptlWju6cUBLu+umketzR97wxApTfgyd/35wKU777eseW5ziiyePwf2XZz5ZnQdUzcvpiZR8PtxhdgovHzV6ENP6vU1p2e7bFdKi1fHq8WuWKTXkV/nszFH429UnC8vqgT3WwM4vPLAUVzy0DNf94wPp4+rP8JNVO3DUD1/CSb961bRexCNLtuJVyReIET+PgfVjqLy0GOO0ADKjyYY3Fub1Q0p0+yqH9kGRKJJW+bnHB1mzjNHn/MRfpjpNRoPlPxG9DRMD7zpoD00PwoljB+Fsg1bo1HyRWcYN0Wet1zzYTv3Fi4mHiPDGxjo0Hu7E0q3BJliWwb9ZJtONiQjHjOgnLKv7t1uvtR4YZhsMN6yYv2q3yYavb9OjnPc3d9j24fHTF9bifx6T9zUPIv94Ywv//sbH8Mw3PmYawOaVC5oJ1IjH/GahoIR7lpE1y/BG1mU1d50dLvkzvGB90J2EpHVbED/39q5u3LFwo/cdBejpBGQg4mcRjOrr2qu3zKRhKSFufSk4CSU9h7mxyIodB7npk295dg3G/2ABmtu7sGFvE677x4f4wTOrsaO+Fa0dXVxz2TMf7MTVkkFCm/Y1SZULAu+FObC8FDPGmHO+JyLU3EXtiBpX4U5EDxFRLRGtMawbTEQLiWiT9neQtp6I6G4iqiaiVUQ0I8rGFyKyWSF5njF6ZxLt28uSeOqxd2vwsVsX2VKa+sEqeKzvmaa2Tpz5u9exaudBTqItfR+XCFXO8//sh7u8NtWxk1396HL5ejys9QrXFdJj1U9ccwqe+cbHOPXYK/ruOUcLy3z6j0u49f/1vW0AgA17D6G2KTWG09DagY/fvhhfeayK6/3x7X/KB0edfeebeOAtD3laIhrrSJWzF/Rqc3ccUM1BQjcZzf0RAOda1t0EYBFjbCKARdoyAMwDMFH7dw2Ae8NpZnzQb7GbyyLPp133ThMJSavm/vCSGuxubLO5qvnBKnisL5iqmgZsrmvB717ZGGoQU9i+wakUBLKukPwOGdWAqleb+6A+dg0UAIjTq63phL1oktvrW9OavS7AllQfwD2Lqz20ls9rLqkVgGAeSrLnyZsaNVyzTB4Kd8bYmwCsBseLADyq/X4UwKcN6x9jKd4DMJCI+Akeeij6PV6w2lm48+zMbgm4rFOO6dp2GK6EVsFjrVKfqLmkiGztc/PycSJHXpAA+Dq6/wFVd8LKxc4TSuWWjJNvbqyzTfQhYmtdS3oANOzQfS/JzPx4W8kKaL7N3duxHCfryFPNnccwxpg+AeFeAPpI20gAxsQfO7V1NojoGiKqIqKquro6n80oPKzaYOPhznRIuBGexpqZtCKz7eZnVqPyphcB2Gfh4ZX3i7Xdep3La+pRedOLWKnNf1lSlLAdz2n2GtMxOOvuWrQpcFutyF6ObHfIsHKx84RSUYLSwURA6vm65zU5zfvu16rxjA/zmAwyA/9BrorsJU1wzKVh3v+wpoT0QuABVZa6Gp6lB2PsPsbYTMbYzIqKiqDNKBisj8vUn72CGb9YaCvHs7nzwvgfX7Y9/dv6MOrC/Y+LN+PWl9b5bHEK/bNSD5LR27BQ0+gWrUt9XpcUJXDQMvn36xtr0dja6UsLr2tq99ni4Ij7djSfE1Fq7gkilBabu7s12CgIx48cEFpdYSL7wuSZZTzP6OVQvr4l+8+xX+G+Tze3aH91w9kuAMbcm6O0dQodwQOwo77VpDXwBquSFk28w5I3xEk4/OUN7xMMmOrWqv7Jp6aY2qCj5wwpThA215kDpRas3ouvPFYVygxGYRCkFVHq8mFpilzhnuC33aubqYhs2JT9PD6yL0z+AHd451QrUFIifZ587vc8gCu031cAeM6w/nLNa+YUAI0G840C4hH1j9++2JSHm9fpuplZuFsnr5bVUnhmIDf0TiJKfLZbi4h9ac1e7oO8YV+Tu1DNsyniwh1QdT+3sHyhefIsQWR7prqSDJ/6w9uhHLMoQXj22jn4+UXHBqrn2tMn2NaFGaFqRVc4uNfM4/1w6n9epqUMCxlXyMcBvAtgEhHtJKKrAdwG4Gwi2gTgLG0ZABYA2AKgGsD9AL4RSasLGCdF4v1tDenfnVybe+qvvmmHwcXROluQE/NXyU2lZ0TvJGnbpEBUH+7sxkNvb7WtTyZZTgdHjcjb3O3ronj/DCovwWmTKkK0ufPd+qyn/dT7O7F2z6FQjllEhGmjB2L04PJA9QztWybcFpWXEhBOENNRR/QVbnvMIWo4KlzTnjHGLhNsOpNTlgG4Nmij4ozT82J8eLk2d4vXiXGArDvJQvuMvPSk0Zg9YQjuWrQJWzQTS5FFuDt9ze9qsA+SdSWZq1kmv/R28b0K+x314Y/PAQDcGVKwFq/dUVtNdMUi6DPIM+8EGlCVLZceULWvk6F3SZFjeWtK5mygIlSzjGw+cKvN/dq/f5CegUgXrEa/9q4kC83u2aukCBdNG2nKWZLOZ6799RKhCqRePm0Sc1dmgyDpEKKcFi2s+yeyH0f55aS3PejXR+gzXrmZZbS/l5w4CgBw7JH909vCfiFaB7SjRgn3LOMkWIzPodU+qoeNG+swzvzenWTSHWt5TQO+5pAamKv5udjcjfDC0ruSSVzx0DKp9kXN71+V05Bl5UxYA8VRDkoSok2dnDHbBavHaXc/rZdtzrzjhqPmtvNNZqWwX+SlnEGVsF9mRrxlo1cEpsMlcIQxhnvf2Ix+vex5pnV4WRa7upm0Z8ALK1M29/s9hH7bbO4eskICctkYszWe+p+P5DISigdU+X78TowdUu7q0x2lXz3P5h4mRemXf1DNXW5dkPp48Mcp/B+XR4k1ECVilOaeZYwDRlstudUZA97dfAC3v7wBtzy7xrqruSDM2n1XMun5s/vWl9Y7VW9CVzr0xzPLM4blBNkIVRmN+E9fdE+zFKVdPHKbe0iau9MLTn+pRvEBIivI9WRtfva3RpAP798Lp0+KLsZHCfcsU2x4+n85f61tO89LxopexCjcuyUGLINAHjR338cIvcZgiM7Qul7mSgwsLw3YmmAk9JlHIkK/d0EDsfi7R/9kyAjnq+aMw4IbPu77GFazzHs/OBNDHLyDgqKEe5ZxzYzooQ5jXV1JZrJ18+x7svDaaDXLRK2554PLu55SwQivWfni4ulEIhHd3K+m4wS1uUvceC/PhqzdnGuWsSxPHt4v0LhIsTLLxBsnQUAkZ3cVae5Ggetm23eCa5bR2yXhLRMGn5852r1QQP529cl49+YzPO8nmkbQyN2XTcdbN57ut2mhQxF7y+iPR9BxA0ezTKCanZFpdtCvkuIsRPEa6THC/cG3t+JfVTvcC0aM8QFdxEl3KvOQMY7m3tmdDG1ibJ6wynReflvCwKi1hZVnxYnhA8owYkBvT/vIDhiOH9oHQ/p6M8VEKbz8CF0/cjqocJcd55CuL8CAqpWgEcQlWZ6Oqcd4y/xCs29/NgsaoRNOGi9jcg9jkjOgesbv3nCMkPMCr4X652jGLBPd4FbqONHUa8Sa/94v1tma5l8/F8eNHJCeuk7Ed885Gh+fmBlQi1KzS/gwuZckEh6+AM3Ph19yZY6TOax+bsePHIB1ew55nmsg28K9x2ju+YKbMJSxEWbMMub11bXNmFU5GCt+fDZGDOjls4X8NuoPdiaIyXf1QoxnHlYovhO9fQp349fKKx/txcm/XmTafpyWIdFN0F13xkRMNUQuRjnomgpi8nbTvNiI01a7wDZ3zjrtb7RmJfuBrYfT7+ez187B+l9Y5y9yJ9uukD1Gc88X3DqYjPKW0dztWlVRgjCwvDRQR+C1scgSXh71IGKUwR06ZZLC/QfnTcZxR/JT2r675YBwP6+K+JA+0Ql3P84yXgYP094ykgOi2RqEdm2O1g6ZU52iRa+mrov359PqChk1SrhnGadnWj4iMvWXkxU43SGDeEbwbe7kWiZMsjFRhqzmfs2p5kyFDMBHuxtR0bfMlt7YiNdzGDe0j6fyXvATbeml/ekxGQn5lSDiRjG7430f2a8hJ2Xi3GOH4/eXTgtsxgviweaH2Jpl2jq7HbXkjq4kugJ4lPiFMbFQYYwfum8l7QrJsY3IpAdwgzd/qzVxWCR+7ob+FbXN/W9Xn+zLrY0IONzRjfPvfhuzfr0Ib24UzyLm9f1UObQP5l8/13ObZPF6y/zcA5kXQpjT17kxcmBvqWvq1CYi9/EZmSbefskJEqXCI5bCvamtE5NveRm/f1U8RdvRP3oJF4SUy9oLScZQXlqE6WP4WeI4lhZuHQA/53tGc/dPS7t9SjBrBGLUn9VRT/4waTg/0tDIxdNH4pwpw2zrZQfS/JiWjsujGY28mWXkB1Sdysh8YVT08xb4I3NNo0wIp3PkQG+eWUGJjXB/YeXu9LRhB5pTk1H822Xex/V7myJvlxWGVKcXae9SmnsSWFK9Hy9/ZJ9kOwyb+GGOl4f+ua0LrGgGVDMdLGqbu8zg1h2fn4b7Lp8Z+Fj9e+Xe+unncnq5ByJXWR5+TW76M11WXISHvhz8vhjJh6C5sImFcG/r7Mb1j3+IL9z/HoBUnhUA2F5vn1n9pqdXcTXTbMEYA5HYNLNOYvIExhjuFkwcnek4/qWvjObezRh++vxH2Ma5xmEQtVnG7+CWVw3v7sumY/71/kPWc4kXjyUvQUxOXwR8b5noJW/QuIrpYwaF1JLwyL1KEQL6Z/JObZIIo824oaUDgwxeCE8s34HxFdENXLnBWMo+16vULtyfXbFLak7L3Y1twtzQYZhNWjvsmrs1699Huxq5QVhhEfWAaraiBS+cemRWjhMmF5wwAvNX7fFlGpP1lvFDpBGqTsd1OfD86+eiMsLBcL/EQnPfUtcMIGOLbjdMCrG/2T6fZy5zgTCW6gA8zd3LZMU1B/gas37u1pq82CmP5mS+M3ZIIrkEZ57J4oCq34CSOH6+W9EHD/3kcJEK4/d4EbNxzWVyy4g4buQA9C2T05N7lWRP5MZCuF94zxIAGTuwMTKw8XBnaDO8h0FSM8uUczT3MNDP1erN8j9zx0ntf9zI/rjtM8fb1hs/0RNEkXsaRW1zj3rAtpDRBZCfr6dspI0IE72XZKvZY7TJQBZ8M3pTXSzMMjq65m4cEDzQ0oFOiyCSeWZ31LdiQHkJ+jtMmuEHBrHmHvZxjMjamI8e1g/lpfbHwtjRu5MMNfvF/t1+MUWo5qmQ6BPRSzmfKCtOnaMnbxkPA6rO9YgriPSLO4LHbUDvEnzx5DGmdY9ddTJeXbcvHRAVJQUt3A93dOPg4Y70sn7z2ww246a2Lrtwl7iTH799McZX9MFr3zmNu/2Flbtx4thBnt2b9BdQWHlNrOjBS9aOIBv5rHsaWbFqcbsb2zy3TcTIgb1tsxTlqWw3jd/EFT3tgBfFfbB2XaJIG5Ers0xQVv7kHNu64QN64b9PGRv6sXgUtFnm4Xe2Yvatr9nWGzX39q5u22TTsvdxiyD68P1tDbj+8Q9dZ6t/cvl2vL1pv3mllhysd8QaoDWCVPZzuaFVINx9PimnTarATfMmO5apHFpuW+fXLHPRtGgHMAfleNKNbKALaC+C+ghtTCdKc1qQqOt7vzgDL1wnDmbKU10iEAUt3MsF2u8hQ5a+ts6kTXP/YHuDY717XbRSffvuRuc5Mb//9Gr894NLTev8mmWG9ecPiA7obTYbpWW6pR8cK/kZ2NTGdxP1q9n071WCr31igntBWCNU5Y/37LVzAKTMCHddOt1T+7ySjQkX/vzfJwau45tnTvS9r26O8WKWmT1hCIBovrjCcIWcd/wIHD9KHMzk9/keM9iumOQLhS3cObbhv723DQvXZSZAbu/qxh6LsF6w2h78A6QGX2ub2nDq/y12PK7+ZeDHBph08XMXMaQPX7iLJpv41tlHm5ZPHDsYy354Jk4eN9jxOE1tndz12baBe3FmKfYhjPxy1jHDcEMAwSnDuccNdzw+kNJEnfi25f57QR9nktXCl/3gTJw4NvVcRZHPPUp0xwNeu6dpUeSXzuKnCV/103PwyrdOja5xASlomzvPV/xHloml2zqTtomojbxTvR+/eXk9nvzqbJz2f4vR0MoXbkbchPuaXY24+ZnV3G1Ofu5OiCZ+sL7g9DZdNXccrpo7DpU3vZjedkQ/9zTAhw7zNfds2D2NGpoXIaG7NWbDd71XSRG+dfbRuEsQRBY1Pzr/GDxwRbjRmTpf/lglzp4yLB3EJns5j+ifea6ijE/I9oDqiAG9UXPb+cJdwna2CJvC1twltN+7F23i+rrrXPf4h1i5sxHVtc1cwb5BS1Gwp/EwDmsDtfqArdGuvbOhFe1dqfW3vbQeq3c12uq68amVeH7lbl9mGav5JSpEAV7ZyK9uxIvtVjeVZLuNuSDKU+xdWoQ5Rw1NX3vjl1C/smJMlJgMhgJKFO75ZWVANfpjZJvCFu6S2m9tk1i417ekBhBFKWw/+fs38eH2Bsy+9TV85bEqAJkIzqVb61FVU4+W9i7M/c1i/OS5jwDYPWHaOrvxuT+/i39W7Uyt4Jhlrjl1vO3YRoEelnB3Un7+9bXZ+Pv/nMzdlo0UvObjyZfVU6nmwse65rbzccbkI0Kv9xefPg5f/lilbb2X+/DuzWcII5md0C+j8VivfucTWPjtT7juG+ULNn+iVQqDghbush4n9725xbWMHgjF47/+9A4A4O3qlOeL0RvnxqdXYbfmxvfGxjocc8vLeNVg8weAS/78DpbV1KeXO7uT6FOWaXt5aREuOGGE7biDDW53o7WBG6Of9Z84dtcgHgUnVQ7GkL4Z2/6EipSm9tTXZvsWnBfPGCld1u+Aqm6W4dnc//nV2dL1eGHWuMGYpEXy3n7JCen75zVjoYgvnTIWP73w2PSyHyE9YkBv/PazUwHA0xSM1jxCADBMM730LinC2VOGYcWPz8Y9X5iOJ685xbSvrMI1sLwEn5kxyrZ+6ih7ttQYKtVZoaBt7tnWJgGg9lAbHl6yNb28pa4FZ9/5JoCUpwkvo+KaXeZkYE1tXRhumAavvLSI62I3ffTA9HjBkD6lafvfQ29vxc/nr8XYIeaR+hljBuJH508xrfv5Rcdyk5FNqOhjmmji++fa3RVvv+QEXDJzFGZWmgdh//nV2fjcX961lbditFd+/bQJuPf1za776IheJpfNGoPHl23nbuMJ91njBuMLJ4/BP5Zm9pGN1nXC+NIY2rcMV86pxPxVezBlRPjBKf17FaNXSRFqm9o9m2UunHqkLb9Nn9IitHDyBx2vpcbVj8G7nusM08tdcILd7dRoTvvhecdg1rjBuOiPdsVpxY9TPuDXnDoeRQngUi3u4wAACS9JREFUrDtSfWj04HL81/SRrgP/YZONmb+yTUFr7sce2d+XRiND5RC+i9OsXy8y5a4x0iyZbbKsOIEKg4b85Fdnc4NjJg3vhyvnVAIwu1xdOacSb3zvNBxrmfrtmW/MseWuvnx2JW69ODNJwFitnrsunY6XbkiFQI8f2gdfP83urtinrBinT7KbHGYJOt5VczJC8+7LzC6J+sujpIjwB8u28UNTWqXxhScyQ33K8oXz2FWz0mmSRQOqPzNowN8/dzJ+dMEUbrkgTBs9CNeePgF3fG6qsMzTX5+N331WvJ3HB7ecjSU3nYGB5anr0dLuPOm2DM9dN8e0PP/6uXjrxtNx3vGpa6vfh3OmiL12nHjg8pm470sn4iunjsfU0QPxvU9OMm03ytFJw/vhqCP64ZefPi49qcadn5+GS2eZIzuBaCaIiTMFrbkXFyWw/ufn4vevbsQbm/bj7kunYev+Fnz54eUAgP/876mYeERfVNc1o70ziU/dk5qcY/LwfthS12Ka2f2SE0fhmBH98Yv5awEA/Qwj4TecOdGXd8Rdl07D397bhuU1Denjrt/bhJvnHYPiogTmHTcc844fgQkVfdMP7ieOrsCjV81Ca0cXykuLUdfUjtnjh+Dk8UPS9RIRxg7JDHz+6PxjUF3bLNWmn190HM6eMgzHjRyAru4kPj5xKL5x2lGez+3Gcyfhg20NeHVdJjPkjz81BfOOH46RA3tzI3ef+tpsjBjYG321wbk7Pz8NOxta8cljh+OU8UNw5jFHoLq2GTfPOwbHjeyP0ydV4PMnjcb8VXvQt6wYK3YcxLQxA/HkNadgSN9SbDvQilOPrkBXdxIzxgzEd86ZZDsmkDLbbL31PLy4eg/mHWc3f4VBUYLwvU86B2udOHZw2mVQFt00d9tnTsDPX7B/rXnhze+djgMt7TjqiH6oue18tHV2442NdTaFYPLw/tjwy3NRVlyEXy1Y5zkf/VmWCU6uPf0ofP0TEzB/9R48smQrrue4kjpFbc4cOwjzV+1BpeGZHzUodR1OCqjh/+EL03Hv65ulnDMKDYpmujQ6F8BdAIoAPMAYu82p/MyZM1lVVVVox3/grS2YPWGITbN9Ytl2VA7tg1M0QVnf0oE/La7GZ04clbafrtrViLGDy3G4sxt3LNyIb519NEYO7I26pnbc/9YWXHrSaLy1aT8mDe+HZVvr8ebGOnz1ExNS+VYOtOALJ4/Byh0HMXl4/7T9dcHqPWg83IlLTxqNLftb0rZsK3saD2NQeWlkqQmioK6pHd1JhuIiwtC+4dibFfnD/uZ2lBUnTMpOtmGMYduBVlta3W0HWjB6UHnBJSsLEyJ6nzHG9Y0NXbgTURGAjQDOBrATwHIAlzHG1or2CVu4KxQKRU/ASbhHYbCeBaCaMbaFMdYB4AkAF0VwHIVCoVAIiEK4jwSww7C8U1unUCgUiiyRM28ZIrqGiKqIqKquri5XzVAoFIpYEoVw3wXAmGlnlLbOBGPsPsbYTMbYzIqKigiaoVAoFD2XKIT7cgATiWgcEZUCuBTA8xEcR6FQKBQCQvdzZ4x1EdF1AP6DlCvkQ4yxj8I+jkKhUCjERBLExBhbAGBBFHUrFAqFwp2CTj+gUCgUCj6RRKh6bgRRHYBtPncfCmC/a6l4oc65Z6DOuWcQ5JzHMsa4Hil5IdyDQERVogituKLOuWegzrlnENU5K7OMQqFQxBAl3BUKhSKGxEG435frBuQAdc49A3XOPYNIzrngbe4KhUKhsBMHzV2hUCgUFpRwVygUihhS0MKdiM4log1EVE1EN+W6PWFBRKOJaDERrSWij4joBm39YCJaSESbtL+DtPVERHdr12EVEc3I7Rn4g4iKiOhDIpqvLY8joqXaeT2p5SoCEZVpy9Xa9spcttsvRDSQiJ4iovVEtI6IZveAe/wt7ZleQ0SPE1GvON5nInqIiGqJaI1hned7S0RXaOU3EdEVXtpQsMJdm/HpjwDmAZgC4DIiCn/m49zQBeA7jLEpAE4BcK12bjcBWMQYmwhgkbYMpK7BRO3fNQDuzX6TQ+EGAOsMy78BcCdj7CgADQCu1tZfDaBBW3+nVq4QuQvAy4yxyQCmInXusb3HRDQSwDcBzGSMHYdU7qlLEc/7/AiAcy3rPN1bIhoM4CcATkZqEqSf6C8EKRhjBfkPwGwA/zEs3wzg5ly3K6JzfQ6paQs3ABihrRsBYIP2+y9ITWWol0+XK5R/SKWGXgTgDADzARBSUXvF1vuNVFK62drvYq0c5focPJ7vAABbre2O+T3WJ/IZrN23+QA+Gdf7DKASwBq/9xbAZQD+YlhvKuf2r2A1d/SQGZ+0T9HpAJYCGMYY26Nt2gtAn2Y+Dtfi9wBuBJDUlocAOMgY69KWjeeUPl9te6NWvpAYB6AOwMOaKeoBIuqDGN9jxtguAL8FsB3AHqTu2/uI93024vXeBrrnhSzcYw8R9QXwNID/ZYwdMm5jqVd5LPxYiegCALWMsfdz3ZYsUgxgBoB7GWPTAbQg85kOIF73GAA0k8JFSL3YjgTQB3bTRY8gG/e2kIW71IxPhQoRlSAl2P/OGHtGW72PiEZo20cAqNXWF/q1mAPgQiKqQWpC9TOQskcPJCI9LbXxnNLnq20fAOBANhscAjsB7GSMLdWWn0JK2Mf1HgPAWQC2MsbqGGOdAJ5B6t7H+T4b8XpvA93zQhbusZ3xiYgIwIMA1jHG7jBseh6APmJ+BVK2eH395dqo+ykAGg2ff3kPY+xmxtgoxlglUvfxNcbYFwEsBnCJVsx6vvp1uEQrX1AaLmNsL4AdRDRJW3UmgLWI6T3W2A7gFCIq155x/Zxje58teL23/wFwDhEN0r56ztHWyZHrQYeAAxbnAdgIYDOAH+a6PSGe11ykPtlWAVih/TsPKXvjIgCbALwKYLBWnpDyHNoMYDVS3gg5Pw+f534agPna7/EAlgGoBvAvAGXa+l7acrW2fXyu2+3zXKcBqNLu87MABsX9HgP4GYD1ANYA+CuAsjjeZwCPIzWu0InUV9rVfu4tgKu0868GcKWXNqj0AwqFQhFDCtkso1AoFAoBSrgrFApFDFHCXaFQKGKIEu4KhUIRQ5RwVygUihiihLtCoVDEECXcFQqFIob8P0IrU7ThkIhGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.plot (list(range(len(rewards))), rewards)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ty0NUj0ZK1AN"
   },
   "source": [
    "### This algorithm seems more stable than the Single Network with and without Memory Replay.\n",
    "# ------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Trial 2: Soft Updating with Tau = 0.5\n",
    "### (Every 500 steps, the staticModel weights become the average of those of the static model and those of the online network)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function used to soft update the static Model weights for any given Tau\n",
    "\n",
    "def updateStaticWeights(model, staticModel,tau):\n",
    "    modelWeights = model.get_weights()\n",
    "    staticWeights = staticModel.get_weights()\n",
    "    \n",
    "    #Python broadcasting:\n",
    "    \n",
    "    for i in range (len(staticWeights)):\n",
    "     \n",
    "        staticWeights[i] = (1-tau) * staticWeights[i] + tau * modelWeights[i]\n",
    "\n",
    "        \n",
    "    staticModel.set_weights(staticWeights)\n",
    "\n",
    "\n",
    "    return model, staticModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "STATIC WEIGHTS UPDATED\n",
      "0  Episode Reward:  20.0 .Epsilon: 0.995\n",
      "1  Episode Reward:  26.0 .Epsilon: 0.990025\n",
      "2  Episode Reward:  32.0 .Epsilon: 0.985074875\n",
      "3  Episode Reward:  25.0 .Epsilon: 0.9801495006250001\n",
      "4  Episode Reward:  14.0 .Epsilon: 0.9752487531218751\n",
      "5  Episode Reward:  33.0 .Epsilon: 0.9703725093562657\n",
      "6  Episode Reward:  17.0 .Epsilon: 0.9655206468094844\n",
      "7  Episode Reward:  16.0 .Epsilon: 0.960693043575437\n",
      "8  Episode Reward:  23.0 .Epsilon: 0.9558895783575597\n",
      "9  Episode Reward:  13.0 .Epsilon: 0.9511101304657719\n",
      "10  Episode Reward:  10.0 .Epsilon: 0.946354579813443\n",
      "11  Episode Reward:  10.0 .Epsilon: 0.9416228069143757\n",
      "12  Episode Reward:  27.0 .Epsilon: 0.9369146928798039\n",
      "13  Episode Reward:  11.0 .Epsilon: 0.9322301194154049\n",
      "14  Episode Reward:  22.0 .Epsilon: 0.9275689688183278\n",
      "15  Episode Reward:  21.0 .Epsilon: 0.9229311239742362\n",
      "16  Episode Reward:  16.0 .Epsilon: 0.918316468354365\n",
      "17  Episode Reward:  37.0 .Epsilon: 0.9137248860125932\n",
      "18  Episode Reward:  19.0 .Epsilon: 0.9091562615825302\n",
      "19  Episode Reward:  21.0 .Epsilon: 0.9046104802746175\n",
      "20  Episode Reward:  24.0 .Epsilon: 0.9000874278732445\n",
      "21  Episode Reward:  17.0 .Epsilon: 0.8955869907338783\n",
      "22  Episode Reward:  13.0 .Epsilon: 0.8911090557802088\n",
      "STATIC WEIGHTS UPDATED\n",
      "23  Episode Reward:  34.0 .Epsilon: 0.8866535105013078\n",
      "24  Episode Reward:  23.0 .Epsilon: 0.8822202429488013\n",
      "25  Episode Reward:  74.0 .Epsilon: 0.8778091417340573\n",
      "26  Episode Reward:  28.0 .Epsilon: 0.8734200960253871\n",
      "27  Episode Reward:  14.0 .Epsilon: 0.8690529955452602\n",
      "28  Episode Reward:  12.0 .Epsilon: 0.8647077305675338\n",
      "29  Episode Reward:  34.0 .Epsilon: 0.8603841919146962\n",
      "30  Episode Reward:  16.0 .Epsilon: 0.8560822709551227\n",
      "31  Episode Reward:  26.0 .Epsilon: 0.851801859600347\n",
      "32  Episode Reward:  10.0 .Epsilon: 0.8475428503023453\n",
      "33  Episode Reward:  10.0 .Epsilon: 0.8433051360508336\n",
      "34  Episode Reward:  9.0 .Epsilon: 0.8390886103705794\n",
      "35  Episode Reward:  26.0 .Epsilon: 0.8348931673187264\n",
      "36  Episode Reward:  15.0 .Epsilon: 0.8307187014821328\n",
      "37  Episode Reward:  16.0 .Epsilon: 0.8265651079747222\n",
      "38  Episode Reward:  31.0 .Epsilon: 0.8224322824348486\n",
      "39  Episode Reward:  26.0 .Epsilon: 0.8183201210226743\n",
      "40  Episode Reward:  19.0 .Epsilon: 0.8142285204175609\n",
      "41  Episode Reward:  28.0 .Epsilon: 0.810157377815473\n",
      "42  Episode Reward:  33.0 .Epsilon: 0.8061065909263957\n",
      "43  Episode Reward:  16.0 .Epsilon: 0.8020760579717637\n",
      "44  Episode Reward:  24.0 .Epsilon: 0.798065677681905\n",
      "STATIC WEIGHTS UPDATED\n",
      "45  Episode Reward:  15.0 .Epsilon: 0.7940753492934954\n",
      "46  Episode Reward:  25.0 .Epsilon: 0.7901049725470279\n",
      "47  Episode Reward:  43.0 .Epsilon: 0.7861544476842928\n",
      "48  Episode Reward:  12.0 .Epsilon: 0.7822236754458713\n",
      "49  Episode Reward:  15.0 .Epsilon: 0.778312557068642\n",
      "50  Episode Reward:  23.0 .Epsilon: 0.7744209942832988\n",
      "51  Episode Reward:  30.0 .Epsilon: 0.7705488893118823\n",
      "52  Episode Reward:  15.0 .Epsilon: 0.7666961448653229\n",
      "53  Episode Reward:  10.0 .Epsilon: 0.7628626641409962\n",
      "54  Episode Reward:  39.0 .Epsilon: 0.7590483508202912\n",
      "55  Episode Reward:  34.0 .Epsilon: 0.7552531090661897\n",
      "56  Episode Reward:  32.0 .Epsilon: 0.7514768435208588\n",
      "57  Episode Reward:  12.0 .Epsilon: 0.7477194593032545\n",
      "58  Episode Reward:  16.0 .Epsilon: 0.7439808620067382\n",
      "59  Episode Reward:  12.0 .Epsilon: 0.7402609576967045\n",
      "60  Episode Reward:  21.0 .Epsilon: 0.736559652908221\n",
      "61  Episode Reward:  19.0 .Epsilon: 0.7328768546436799\n",
      "62  Episode Reward:  35.0 .Epsilon: 0.7292124703704616\n",
      "63  Episode Reward:  13.0 .Epsilon: 0.7255664080186093\n",
      "64  Episode Reward:  16.0 .Epsilon: 0.7219385759785162\n",
      "65  Episode Reward:  28.0 .Epsilon: 0.7183288830986236\n",
      "66  Episode Reward:  27.0 .Epsilon: 0.7147372386831305\n",
      "67  Episode Reward:  17.0 .Epsilon: 0.7111635524897149\n",
      "STATIC WEIGHTS UPDATED\n",
      "68  Episode Reward:  33.0 .Epsilon: 0.7076077347272662\n",
      "69  Episode Reward:  58.0 .Epsilon: 0.7040696960536299\n",
      "70  Episode Reward:  17.0 .Epsilon: 0.7005493475733617\n",
      "71  Episode Reward:  18.0 .Epsilon: 0.697046600835495\n",
      "72  Episode Reward:  24.0 .Epsilon: 0.6935613678313175\n",
      "73  Episode Reward:  15.0 .Epsilon: 0.6900935609921609\n",
      "74  Episode Reward:  26.0 .Epsilon: 0.6866430931872001\n",
      "75  Episode Reward:  20.0 .Epsilon: 0.6832098777212641\n",
      "76  Episode Reward:  16.0 .Epsilon: 0.6797938283326578\n",
      "77  Episode Reward:  20.0 .Epsilon: 0.6763948591909945\n",
      "78  Episode Reward:  27.0 .Epsilon: 0.6730128848950395\n",
      "79  Episode Reward:  36.0 .Epsilon: 0.6696478204705644\n",
      "80  Episode Reward:  26.0 .Epsilon: 0.6662995813682115\n",
      "81  Episode Reward:  13.0 .Epsilon: 0.6629680834613705\n",
      "82  Episode Reward:  30.0 .Epsilon: 0.6596532430440636\n",
      "83  Episode Reward:  50.0 .Epsilon: 0.6563549768288433\n",
      "84  Episode Reward:  38.0 .Epsilon: 0.653073201944699\n",
      "85  Episode Reward:  26.0 .Epsilon: 0.6498078359349755\n",
      "STATIC WEIGHTS UPDATED\n",
      "86  Episode Reward:  24.0 .Epsilon: 0.6465587967553006\n",
      "87  Episode Reward:  56.0 .Epsilon: 0.6433260027715241\n",
      "88  Episode Reward:  80.0 .Epsilon: 0.6401093727576664\n",
      "89  Episode Reward:  35.0 .Epsilon: 0.6369088258938781\n",
      "90  Episode Reward:  16.0 .Epsilon: 0.6337242817644086\n",
      "91  Episode Reward:  24.0 .Epsilon: 0.6305556603555866\n",
      "92  Episode Reward:  59.0 .Epsilon: 0.6274028820538087\n",
      "93  Episode Reward:  29.0 .Epsilon: 0.6242658676435396\n",
      "94  Episode Reward:  62.0 .Epsilon: 0.6211445383053219\n",
      "95  Episode Reward:  121.0 .Epsilon: 0.6180388156137953\n",
      "STATIC WEIGHTS UPDATED\n",
      "96  Episode Reward:  100.0 .Epsilon: 0.6149486215357263\n",
      "97  Episode Reward:  20.0 .Epsilon: 0.6118738784280476\n",
      "98  Episode Reward:  36.0 .Epsilon: 0.6088145090359074\n",
      "99  Episode Reward:  60.0 .Epsilon: 0.6057704364907278\n",
      "100  Episode Reward:  84.0 .Epsilon: 0.6027415843082742\n",
      "101  Episode Reward:  42.0 .Epsilon: 0.5997278763867329\n",
      "102  Episode Reward:  21.0 .Epsilon: 0.5967292370047992\n",
      "103  Episode Reward:  68.0 .Epsilon: 0.5937455908197752\n",
      "STATIC WEIGHTS UPDATED\n",
      "104  Episode Reward:  79.0 .Epsilon: 0.5907768628656763\n",
      "105  Episode Reward:  26.0 .Epsilon: 0.5878229785513479\n",
      "106  Episode Reward:  50.0 .Epsilon: 0.5848838636585911\n",
      "107  Episode Reward:  20.0 .Epsilon: 0.5819594443402982\n",
      "108  Episode Reward:  16.0 .Epsilon: 0.5790496471185967\n",
      "109  Episode Reward:  58.0 .Epsilon: 0.5761543988830038\n",
      "110  Episode Reward:  84.0 .Epsilon: 0.5732736268885887\n",
      "111  Episode Reward:  16.0 .Epsilon: 0.5704072587541458\n",
      "112  Episode Reward:  53.0 .Epsilon: 0.567555222460375\n",
      "113  Episode Reward:  88.0 .Epsilon: 0.5647174463480732\n",
      "114  Episode Reward:  22.0 .Epsilon: 0.5618938591163328\n",
      "115  Episode Reward:  18.0 .Epsilon: 0.5590843898207511\n",
      "116  Episode Reward:  21.0 .Epsilon: 0.5562889678716474\n",
      "STATIC WEIGHTS UPDATED\n",
      "117  Episode Reward:  140.0 .Epsilon: 0.5535075230322891\n",
      "118  Episode Reward:  19.0 .Epsilon: 0.5507399854171277\n",
      "119  Episode Reward:  16.0 .Epsilon: 0.547986285490042\n",
      "120  Episode Reward:  28.0 .Epsilon: 0.5452463540625918\n",
      "121  Episode Reward:  53.0 .Epsilon: 0.5425201222922789\n",
      "122  Episode Reward:  21.0 .Epsilon: 0.5398075216808175\n",
      "123  Episode Reward:  27.0 .Epsilon: 0.5371084840724134\n",
      "124  Episode Reward:  37.0 .Epsilon: 0.5344229416520513\n",
      "125  Episode Reward:  109.0 .Epsilon: 0.531750826943791\n",
      "126  Episode Reward:  32.0 .Epsilon: 0.5290920728090721\n",
      "127  Episode Reward:  13.0 .Epsilon: 0.5264466124450268\n",
      "128  Episode Reward:  22.0 .Epsilon: 0.5238143793828016\n",
      "STATIC WEIGHTS UPDATED\n",
      "129  Episode Reward:  34.0 .Epsilon: 0.5211953074858876\n",
      "130  Episode Reward:  20.0 .Epsilon: 0.5185893309484582\n",
      "131  Episode Reward:  21.0 .Epsilon: 0.5159963842937159\n",
      "132  Episode Reward:  58.0 .Epsilon: 0.5134164023722473\n",
      "133  Episode Reward:  28.0 .Epsilon: 0.510849320360386\n",
      "134  Episode Reward:  47.0 .Epsilon: 0.5082950737585841\n",
      "135  Episode Reward:  14.0 .Epsilon: 0.5057535983897912\n",
      "136  Episode Reward:  35.0 .Epsilon: 0.5032248303978422\n",
      "137  Episode Reward:  15.0 .Epsilon: 0.500708706245853\n",
      "138  Episode Reward:  91.0 .Epsilon: 0.4982051627146237\n",
      "STATIC WEIGHTS UPDATED\n",
      "139  Episode Reward:  206.0 .Epsilon: 0.49571413690105054\n",
      "140  Episode Reward:  151.0 .Epsilon: 0.4932355662165453\n",
      "141  Episode Reward:  36.0 .Epsilon: 0.4907693883854626\n",
      "142  Episode Reward:  44.0 .Epsilon: 0.4883155414435353\n",
      "143  Episode Reward:  121.0 .Epsilon: 0.4858739637363176\n",
      "144  Episode Reward:  11.0 .Epsilon: 0.483444593917636\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "145  Episode Reward:  29.0 .Epsilon: 0.4810273709480478\n",
      "STATIC WEIGHTS UPDATED\n",
      "146  Episode Reward:  55.0 .Epsilon: 0.47862223409330756\n",
      "147  Episode Reward:  125.0 .Epsilon: 0.47622912292284103\n",
      "148  Episode Reward:  165.0 .Epsilon: 0.4738479773082268\n",
      "149  Episode Reward:  22.0 .Epsilon: 0.47147873742168567\n",
      "150  Episode Reward:  29.0 .Epsilon: 0.46912134373457726\n",
      "151  Episode Reward:  13.0 .Epsilon: 0.46677573701590436\n",
      "152  Episode Reward:  24.0 .Epsilon: 0.46444185833082485\n",
      "153  Episode Reward:  10.0 .Epsilon: 0.46211964903917074\n",
      "154  Episode Reward:  29.0 .Epsilon: 0.4598090507939749\n",
      "STATIC WEIGHTS UPDATED\n",
      "155  Episode Reward:  72.0 .Epsilon: 0.457510005540005\n",
      "156  Episode Reward:  43.0 .Epsilon: 0.45522245551230495\n",
      "157  Episode Reward:  32.0 .Epsilon: 0.4529463432347434\n",
      "158  Episode Reward:  26.0 .Epsilon: 0.4506816115185697\n",
      "159  Episode Reward:  34.0 .Epsilon: 0.4484282034609769\n",
      "160  Episode Reward:  44.0 .Epsilon: 0.446186062443672\n",
      "161  Episode Reward:  32.0 .Epsilon: 0.4439551321314536\n",
      "162  Episode Reward:  22.0 .Epsilon: 0.4417353564707963\n",
      "163  Episode Reward:  19.0 .Epsilon: 0.43952667968844233\n",
      "164  Episode Reward:  23.0 .Epsilon: 0.43732904629000013\n",
      "165  Episode Reward:  23.0 .Epsilon: 0.4351424010585501\n",
      "166  Episode Reward:  33.0 .Epsilon: 0.43296668905325736\n",
      "167  Episode Reward:  16.0 .Epsilon: 0.43080185560799106\n",
      "168  Episode Reward:  25.0 .Epsilon: 0.4286478463299511\n",
      "169  Episode Reward:  24.0 .Epsilon: 0.42650460709830135\n",
      "170  Episode Reward:  15.0 .Epsilon: 0.42437208406280985\n",
      "171  Episode Reward:  17.0 .Epsilon: 0.4222502236424958\n",
      "172  Episode Reward:  31.0 .Epsilon: 0.42013897252428334\n",
      "173  Episode Reward:  14.0 .Epsilon: 0.4180382776616619\n",
      "STATIC WEIGHTS UPDATED\n",
      "174  Episode Reward:  36.0 .Epsilon: 0.4159480862733536\n",
      "175  Episode Reward:  147.0 .Epsilon: 0.41386834584198684\n",
      "176  Episode Reward:  16.0 .Epsilon: 0.4117990041127769\n",
      "177  Episode Reward:  43.0 .Epsilon: 0.40974000909221303\n",
      "178  Episode Reward:  16.0 .Epsilon: 0.40769130904675194\n",
      "179  Episode Reward:  27.0 .Epsilon: 0.40565285250151817\n",
      "180  Episode Reward:  10.0 .Epsilon: 0.4036245882390106\n",
      "181  Episode Reward:  10.0 .Epsilon: 0.4016064652978155\n",
      "182  Episode Reward:  20.0 .Epsilon: 0.3995984329713264\n",
      "183  Episode Reward:  32.0 .Epsilon: 0.3976004408064698\n",
      "184  Episode Reward:  34.0 .Epsilon: 0.39561243860243744\n",
      "185  Episode Reward:  18.0 .Epsilon: 0.3936343764094253\n",
      "186  Episode Reward:  15.0 .Epsilon: 0.39166620452737816\n",
      "187  Episode Reward:  34.0 .Epsilon: 0.3897078735047413\n",
      "STATIC WEIGHTS UPDATED\n",
      "188  Episode Reward:  70.0 .Epsilon: 0.3877593341372176\n",
      "189  Episode Reward:  17.0 .Epsilon: 0.3858205374665315\n",
      "190  Episode Reward:  96.0 .Epsilon: 0.38389143477919885\n",
      "191  Episode Reward:  22.0 .Epsilon: 0.3819719776053028\n",
      "192  Episode Reward:  50.0 .Epsilon: 0.3800621177172763\n",
      "193  Episode Reward:  30.0 .Epsilon: 0.37816180712868996\n",
      "194  Episode Reward:  18.0 .Epsilon: 0.37627099809304654\n",
      "195  Episode Reward:  28.0 .Epsilon: 0.3743896431025813\n",
      "196  Episode Reward:  14.0 .Epsilon: 0.37251769488706843\n",
      "197  Episode Reward:  16.0 .Epsilon: 0.3706551064126331\n",
      "198  Episode Reward:  13.0 .Epsilon: 0.36880183088056995\n",
      "199  Episode Reward:  105.0 .Epsilon: 0.3669578217261671\n",
      "200  Episode Reward:  11.0 .Epsilon: 0.36512303261753626\n",
      "201  Episode Reward:  35.0 .Epsilon: 0.3632974174544486\n",
      "202  Episode Reward:  28.0 .Epsilon: 0.3614809303671764\n",
      "STATIC WEIGHTS UPDATED\n",
      "203  Episode Reward:  31.0 .Epsilon: 0.3596735257153405\n",
      "204  Episode Reward:  32.0 .Epsilon: 0.3578751580867638\n",
      "205  Episode Reward:  16.0 .Epsilon: 0.35608578229633\n",
      "206  Episode Reward:  15.0 .Epsilon: 0.3543053533848483\n",
      "207  Episode Reward:  31.0 .Epsilon: 0.35253382661792404\n",
      "208  Episode Reward:  45.0 .Epsilon: 0.3507711574848344\n",
      "209  Episode Reward:  13.0 .Epsilon: 0.34901730169741024\n",
      "210  Episode Reward:  79.0 .Epsilon: 0.3472722151889232\n",
      "211  Episode Reward:  49.0 .Epsilon: 0.3455358541129786\n",
      "212  Episode Reward:  84.0 .Epsilon: 0.3438081748424137\n",
      "213  Episode Reward:  43.0 .Epsilon: 0.3420891339682016\n",
      "214  Episode Reward:  13.0 .Epsilon: 0.3403786882983606\n",
      "215  Episode Reward:  18.0 .Epsilon: 0.3386767948568688\n",
      "STATIC WEIGHTS UPDATED\n",
      "216  Episode Reward:  48.0 .Epsilon: 0.33698341088258443\n",
      "217  Episode Reward:  35.0 .Epsilon: 0.3352984938281715\n",
      "218  Episode Reward:  87.0 .Epsilon: 0.33362200135903064\n",
      "219  Episode Reward:  31.0 .Epsilon: 0.33195389135223546\n",
      "220  Episode Reward:  15.0 .Epsilon: 0.3302941218954743\n",
      "221  Episode Reward:  13.0 .Epsilon: 0.32864265128599696\n",
      "222  Episode Reward:  36.0 .Epsilon: 0.326999438029567\n",
      "223  Episode Reward:  17.0 .Epsilon: 0.3253644408394192\n",
      "224  Episode Reward:  52.0 .Epsilon: 0.3237376186352221\n",
      "225  Episode Reward:  51.0 .Epsilon: 0.322118930542046\n",
      "226  Episode Reward:  11.0 .Epsilon: 0.32050833588933575\n",
      "227  Episode Reward:  13.0 .Epsilon: 0.31890579420988907\n",
      "228  Episode Reward:  46.0 .Epsilon: 0.3173112652388396\n",
      "229  Episode Reward:  16.0 .Epsilon: 0.3157247089126454\n",
      "STATIC WEIGHTS UPDATED\n",
      "230  Episode Reward:  87.0 .Epsilon: 0.3141460853680822\n",
      "231  Episode Reward:  166.0 .Epsilon: 0.3125753549412418\n",
      "232  Episode Reward:  103.0 .Epsilon: 0.31101247816653554\n",
      "233  Episode Reward:  14.0 .Epsilon: 0.30945741577570285\n",
      "234  Episode Reward:  75.0 .Epsilon: 0.3079101286968243\n",
      "235  Episode Reward:  16.0 .Epsilon: 0.3063705780533402\n",
      "236  Episode Reward:  45.0 .Epsilon: 0.30483872516307353\n",
      "237  Episode Reward:  20.0 .Epsilon: 0.3033145315372582\n",
      "STATIC WEIGHTS UPDATED\n",
      "238  Episode Reward:  191.0 .Epsilon: 0.3017979588795719\n",
      "239  Episode Reward:  32.0 .Epsilon: 0.30028896908517405\n",
      "240  Episode Reward:  98.0 .Epsilon: 0.2987875242397482\n",
      "241  Episode Reward:  14.0 .Epsilon: 0.29729358661854943\n",
      "242  Episode Reward:  40.0 .Epsilon: 0.29580711868545667\n",
      "243  Episode Reward:  49.0 .Epsilon: 0.2943280830920294\n",
      "244  Episode Reward:  89.0 .Epsilon: 0.29285644267656924\n",
      "STATIC WEIGHTS UPDATED\n",
      "245  Episode Reward:  99.0 .Epsilon: 0.2913921604631864\n",
      "246  Episode Reward:  52.0 .Epsilon: 0.28993519966087045\n",
      "247  Episode Reward:  14.0 .Epsilon: 0.2884855236625661\n",
      "248  Episode Reward:  34.0 .Epsilon: 0.28704309604425327\n",
      "249  Episode Reward:  15.0 .Epsilon: 0.285607880564032\n",
      "250  Episode Reward:  33.0 .Epsilon: 0.28417984116121187\n",
      "251  Episode Reward:  16.0 .Epsilon: 0.2827589419554058\n",
      "252  Episode Reward:  48.0 .Epsilon: 0.28134514724562876\n",
      "253  Episode Reward:  33.0 .Epsilon: 0.2799384215094006\n",
      "254  Episode Reward:  36.0 .Epsilon: 0.27853872940185365\n",
      "255  Episode Reward:  36.0 .Epsilon: 0.27714603575484437\n",
      "256  Episode Reward:  24.0 .Epsilon: 0.2757603055760701\n",
      "257  Episode Reward:  39.0 .Epsilon: 0.2743815040481898\n",
      "258  Episode Reward:  48.0 .Epsilon: 0.2730095965279488\n",
      "STATIC WEIGHTS UPDATED\n",
      "259  Episode Reward:  20.0 .Epsilon: 0.27164454854530906\n",
      "260  Episode Reward:  12.0 .Epsilon: 0.2702863258025825\n",
      "261  Episode Reward:  117.0 .Epsilon: 0.2689348941735696\n",
      "262  Episode Reward:  12.0 .Epsilon: 0.26759021970270175\n",
      "263  Episode Reward:  112.0 .Epsilon: 0.2662522686041882\n",
      "264  Episode Reward:  18.0 .Epsilon: 0.2649210072611673\n",
      "265  Episode Reward:  75.0 .Epsilon: 0.26359640222486147\n",
      "266  Episode Reward:  16.0 .Epsilon: 0.26227842021373715\n",
      "267  Episode Reward:  56.0 .Epsilon: 0.2609670281126685\n",
      "268  Episode Reward:  12.0 .Epsilon: 0.25966219297210513\n",
      "269  Episode Reward:  12.0 .Epsilon: 0.2583638820072446\n",
      "270  Episode Reward:  37.0 .Epsilon: 0.2570720625972084\n",
      "STATIC WEIGHTS UPDATED\n",
      "271  Episode Reward:  13.0 .Epsilon: 0.25578670228422234\n",
      "272  Episode Reward:  62.0 .Epsilon: 0.25450776877280124\n",
      "273  Episode Reward:  19.0 .Epsilon: 0.2532352299289372\n",
      "274  Episode Reward:  72.0 .Epsilon: 0.2519690537792925\n",
      "275  Episode Reward:  10.0 .Epsilon: 0.2507092085103961\n",
      "276  Episode Reward:  59.0 .Epsilon: 0.2494556624678441\n",
      "277  Episode Reward:  213.0 .Epsilon: 0.24820838415550486\n",
      "278  Episode Reward:  21.0 .Epsilon: 0.24696734223472733\n",
      "279  Episode Reward:  25.0 .Epsilon: 0.2457325055235537\n",
      "STATIC WEIGHTS UPDATED\n",
      "280  Episode Reward:  78.0 .Epsilon: 0.24450384299593592\n",
      "281  Episode Reward:  39.0 .Epsilon: 0.24328132378095624\n",
      "282  Episode Reward:  22.0 .Epsilon: 0.24206491716205145\n",
      "283  Episode Reward:  75.0 .Epsilon: 0.2408545925762412\n",
      "284  Episode Reward:  26.0 .Epsilon: 0.23965031961336\n",
      "285  Episode Reward:  24.0 .Epsilon: 0.2384520680152932\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "286  Episode Reward:  16.0 .Epsilon: 0.23725980767521673\n",
      "287  Episode Reward:  69.0 .Epsilon: 0.23607350863684065\n",
      "STATIC WEIGHTS UPDATED\n",
      "288  Episode Reward:  172.0 .Epsilon: 0.23489314109365644\n",
      "289  Episode Reward:  37.0 .Epsilon: 0.23371867538818816\n",
      "290  Episode Reward:  40.0 .Epsilon: 0.23255008201124722\n",
      "291  Episode Reward:  254.0 .Epsilon: 0.231387331601191\n",
      "292  Episode Reward:  44.0 .Epsilon: 0.23023039494318503\n",
      "293  Episode Reward:  55.0 .Epsilon: 0.2290792429684691\n",
      "294  Episode Reward:  43.0 .Epsilon: 0.22793384675362674\n",
      "STATIC WEIGHTS UPDATED\n",
      "295  Episode Reward:  23.0 .Epsilon: 0.22679417751985861\n",
      "296  Episode Reward:  15.0 .Epsilon: 0.22566020663225933\n",
      "297  Episode Reward:  10.0 .Epsilon: 0.22453190559909803\n",
      "298  Episode Reward:  22.0 .Epsilon: 0.22340924607110255\n",
      "299  Episode Reward:  72.0 .Epsilon: 0.22229219984074702\n",
      "300  Episode Reward:  230.0 .Epsilon: 0.2211807388415433\n",
      "301  Episode Reward:  37.0 .Epsilon: 0.22007483514733558\n",
      "302  Episode Reward:  76.0 .Epsilon: 0.2189744609715989\n",
      "STATIC WEIGHTS UPDATED\n",
      "303  Episode Reward:  132.0 .Epsilon: 0.2178795886667409\n",
      "304  Episode Reward:  103.0 .Epsilon: 0.2167901907234072\n",
      "305  Episode Reward:  111.0 .Epsilon: 0.21570623976979014\n",
      "306  Episode Reward:  101.0 .Epsilon: 0.21462770857094118\n",
      "STATIC WEIGHTS UPDATED\n",
      "307  Episode Reward:  100.0 .Epsilon: 0.21355457002808648\n",
      "308  Episode Reward:  117.0 .Epsilon: 0.21248679717794605\n",
      "309  Episode Reward:  151.0 .Epsilon: 0.21142436319205632\n",
      "310  Episode Reward:  119.0 .Epsilon: 0.21036724137609603\n",
      "311  Episode Reward:  98.0 .Epsilon: 0.20931540516921554\n",
      "STATIC WEIGHTS UPDATED\n",
      "312  Episode Reward:  139.0 .Epsilon: 0.20826882814336947\n",
      "313  Episode Reward:  121.0 .Epsilon: 0.20722748400265262\n",
      "314  Episode Reward:  52.0 .Epsilon: 0.20619134658263935\n",
      "315  Episode Reward:  88.0 .Epsilon: 0.20516038984972615\n",
      "STATIC WEIGHTS UPDATED\n",
      "316  Episode Reward:  120.0 .Epsilon: 0.2041345879004775\n",
      "317  Episode Reward:  108.0 .Epsilon: 0.2031139149609751\n",
      "318  Episode Reward:  36.0 .Epsilon: 0.20209834538617025\n",
      "319  Episode Reward:  65.0 .Epsilon: 0.2010878536592394\n",
      "320  Episode Reward:  101.0 .Epsilon: 0.2000824143909432\n",
      "321  Episode Reward:  89.0 .Epsilon: 0.19908200231898848\n",
      "STATIC WEIGHTS UPDATED\n",
      "322  Episode Reward:  103.0 .Epsilon: 0.19808659230739353\n",
      "323  Episode Reward:  19.0 .Epsilon: 0.19709615934585656\n",
      "324  Episode Reward:  109.0 .Epsilon: 0.19611067854912728\n",
      "325  Episode Reward:  104.0 .Epsilon: 0.19513012515638165\n",
      "326  Episode Reward:  102.0 .Epsilon: 0.19415447453059972\n",
      "327  Episode Reward:  108.0 .Epsilon: 0.19318370215794672\n",
      "STATIC WEIGHTS UPDATED\n",
      "328  Episode Reward:  106.0 .Epsilon: 0.192217783647157\n",
      "329  Episode Reward:  28.0 .Epsilon: 0.1912566947289212\n",
      "330  Episode Reward:  38.0 .Epsilon: 0.1903004112552766\n",
      "331  Episode Reward:  86.0 .Epsilon: 0.18934890919900021\n",
      "332  Episode Reward:  22.0 .Epsilon: 0.18840216465300522\n",
      "333  Episode Reward:  92.0 .Epsilon: 0.18746015382974018\n",
      "334  Episode Reward:  20.0 .Epsilon: 0.1865228530605915\n",
      "335  Episode Reward:  69.0 .Epsilon: 0.18559023879528855\n",
      "336  Episode Reward:  12.0 .Epsilon: 0.1846622876013121\n",
      "STATIC WEIGHTS UPDATED\n",
      "337  Episode Reward:  85.0 .Epsilon: 0.18373897616330553\n",
      "338  Episode Reward:  115.0 .Epsilon: 0.182820281282489\n",
      "339  Episode Reward:  52.0 .Epsilon: 0.18190617987607657\n",
      "340  Episode Reward:  101.0 .Epsilon: 0.18099664897669618\n",
      "341  Episode Reward:  68.0 .Epsilon: 0.1800916657318127\n",
      "342  Episode Reward:  25.0 .Epsilon: 0.17919120740315364\n",
      "343  Episode Reward:  81.0 .Epsilon: 0.17829525136613786\n",
      "STATIC WEIGHTS UPDATED\n",
      "344  Episode Reward:  94.0 .Epsilon: 0.17740377510930716\n",
      "345  Episode Reward:  113.0 .Epsilon: 0.17651675623376062\n",
      "346  Episode Reward:  112.0 .Epsilon: 0.1756341724525918\n",
      "347  Episode Reward:  134.0 .Epsilon: 0.17475600159032884\n",
      "STATIC WEIGHTS UPDATED\n",
      "348  Episode Reward:  127.0 .Epsilon: 0.17388222158237718\n",
      "349  Episode Reward:  112.0 .Epsilon: 0.1730128104744653\n",
      "350  Episode Reward:  116.0 .Epsilon: 0.17214774642209296\n",
      "351  Episode Reward:  84.0 .Epsilon: 0.1712870076899825\n",
      "352  Episode Reward:  137.0 .Epsilon: 0.17043057265153258\n",
      "STATIC WEIGHTS UPDATED\n",
      "353  Episode Reward:  86.0 .Epsilon: 0.16957841978827493\n",
      "354  Episode Reward:  16.0 .Epsilon: 0.16873052768933355\n",
      "355  Episode Reward:  115.0 .Epsilon: 0.1678868750508869\n",
      "356  Episode Reward:  126.0 .Epsilon: 0.16704744067563246\n",
      "357  Episode Reward:  118.0 .Epsilon: 0.1662122034722543\n",
      "358  Episode Reward:  11.0 .Epsilon: 0.16538114245489302\n",
      "STATIC WEIGHTS UPDATED\n",
      "359  Episode Reward:  96.0 .Epsilon: 0.16455423674261854\n",
      "360  Episode Reward:  154.0 .Epsilon: 0.16373146555890544\n",
      "361  Episode Reward:  25.0 .Epsilon: 0.16291280823111093\n",
      "362  Episode Reward:  41.0 .Epsilon: 0.16209824418995536\n",
      "363  Episode Reward:  65.0 .Epsilon: 0.16128775296900558\n",
      "364  Episode Reward:  14.0 .Epsilon: 0.16048131420416054\n",
      "365  Episode Reward:  108.0 .Epsilon: 0.15967890763313974\n",
      "STATIC WEIGHTS UPDATED\n",
      "366  Episode Reward:  112.0 .Epsilon: 0.15888051309497406\n",
      "367  Episode Reward:  16.0 .Epsilon: 0.1580861105294992\n",
      "368  Episode Reward:  60.0 .Epsilon: 0.1572956799768517\n",
      "369  Episode Reward:  101.0 .Epsilon: 0.15650920157696743\n",
      "370  Episode Reward:  13.0 .Epsilon: 0.1557266555690826\n",
      "371  Episode Reward:  77.0 .Epsilon: 0.1549480222912372\n",
      "372  Episode Reward:  110.0 .Epsilon: 0.15417328217978102\n",
      "STATIC WEIGHTS UPDATED\n",
      "373  Episode Reward:  49.0 .Epsilon: 0.1534024157688821\n",
      "374  Episode Reward:  166.0 .Epsilon: 0.1526354036900377\n",
      "375  Episode Reward:  62.0 .Epsilon: 0.1518722266715875\n",
      "376  Episode Reward:  131.0 .Epsilon: 0.15111286553822956\n",
      "377  Episode Reward:  52.0 .Epsilon: 0.15035730121053842\n",
      "378  Episode Reward:  27.0 .Epsilon: 0.14960551470448571\n",
      "379  Episode Reward:  18.0 .Epsilon: 0.14885748713096328\n",
      "STATIC WEIGHTS UPDATED\n",
      "380  Episode Reward:  61.0 .Epsilon: 0.14811319969530845\n",
      "381  Episode Reward:  29.0 .Epsilon: 0.1473726336968319\n",
      "382  Episode Reward:  25.0 .Epsilon: 0.14663577052834775\n",
      "383  Episode Reward:  46.0 .Epsilon: 0.14590259167570602\n",
      "384  Episode Reward:  102.0 .Epsilon: 0.1451730787173275\n",
      "385  Episode Reward:  72.0 .Epsilon: 0.14444721332374086\n",
      "386  Episode Reward:  51.0 .Epsilon: 0.14372497725712216\n",
      "387  Episode Reward:  130.0 .Epsilon: 0.14300635237083656\n",
      "388  Episode Reward:  14.0 .Epsilon: 0.14229132060898236\n",
      "STATIC WEIGHTS UPDATED\n",
      "389  Episode Reward:  101.0 .Epsilon: 0.14157986400593744\n",
      "390  Episode Reward:  121.0 .Epsilon: 0.14087196468590776\n",
      "391  Episode Reward:  81.0 .Epsilon: 0.14016760486247823\n",
      "392  Episode Reward:  73.0 .Epsilon: 0.13946676683816583\n",
      "393  Episode Reward:  59.0 .Epsilon: 0.138769433003975\n",
      "STATIC WEIGHTS UPDATED\n",
      "394  Episode Reward:  221.0 .Epsilon: 0.13807558583895513\n",
      "395  Episode Reward:  168.0 .Epsilon: 0.13738520790976036\n",
      "396  Episode Reward:  78.0 .Epsilon: 0.13669828187021155\n",
      "397  Episode Reward:  12.0 .Epsilon: 0.13601479046086049\n",
      "398  Episode Reward:  35.0 .Epsilon: 0.1353347165085562\n",
      "STATIC WEIGHTS UPDATED\n",
      "399  Episode Reward:  128.0 .Epsilon: 0.1346580429260134\n",
      "400  Episode Reward:  94.0 .Epsilon: 0.13398475271138335\n",
      "401  Episode Reward:  72.0 .Epsilon: 0.13331482894782642\n",
      "402  Episode Reward:  113.0 .Epsilon: 0.13264825480308728\n",
      "403  Episode Reward:  37.0 .Epsilon: 0.13198501352907185\n",
      "404  Episode Reward:  91.0 .Epsilon: 0.1313250884614265\n",
      "STATIC WEIGHTS UPDATED\n",
      "405  Episode Reward:  41.0 .Epsilon: 0.13066846301911936\n",
      "406  Episode Reward:  95.0 .Epsilon: 0.13001512070402377\n",
      "407  Episode Reward:  87.0 .Epsilon: 0.12936504510050365\n",
      "408  Episode Reward:  95.0 .Epsilon: 0.12871821987500112\n",
      "409  Episode Reward:  56.0 .Epsilon: 0.12807462877562611\n",
      "410  Episode Reward:  115.0 .Epsilon: 0.12743425563174798\n",
      "STATIC WEIGHTS UPDATED\n",
      "411  Episode Reward:  85.0 .Epsilon: 0.12679708435358925\n",
      "412  Episode Reward:  95.0 .Epsilon: 0.1261630989318213\n",
      "413  Episode Reward:  49.0 .Epsilon: 0.1255322834371622\n",
      "414  Episode Reward:  17.0 .Epsilon: 0.12490462201997637\n",
      "415  Episode Reward:  110.0 .Epsilon: 0.1242800989098765\n",
      "416  Episode Reward:  14.0 .Epsilon: 0.12365869841532712\n",
      "417  Episode Reward:  12.0 .Epsilon: 0.12304040492325048\n",
      "418  Episode Reward:  128.0 .Epsilon: 0.12242520289863423\n",
      "STATIC WEIGHTS UPDATED\n",
      "419  Episode Reward:  125.0 .Epsilon: 0.12181307688414106\n",
      "420  Episode Reward:  133.0 .Epsilon: 0.12120401149972035\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "421  Episode Reward:  11.0 .Epsilon: 0.12059799144222175\n",
      "422  Episode Reward:  114.0 .Epsilon: 0.11999500148501063\n",
      "STATIC WEIGHTS UPDATED\n",
      "423  Episode Reward:  175.0 .Epsilon: 0.11939502647758558\n",
      "424  Episode Reward:  110.0 .Epsilon: 0.11879805134519765\n",
      "425  Episode Reward:  103.0 .Epsilon: 0.11820406108847166\n",
      "426  Episode Reward:  46.0 .Epsilon: 0.1176130407830293\n",
      "427  Episode Reward:  12.0 .Epsilon: 0.11702497557911415\n",
      "428  Episode Reward:  161.0 .Epsilon: 0.11643985070121858\n",
      "429  Episode Reward:  23.0 .Epsilon: 0.11585765144771248\n",
      "STATIC WEIGHTS UPDATED\n",
      "430  Episode Reward:  59.0 .Epsilon: 0.11527836319047392\n",
      "431  Episode Reward:  24.0 .Epsilon: 0.11470197137452155\n",
      "432  Episode Reward:  13.0 .Epsilon: 0.11412846151764894\n",
      "433  Episode Reward:  34.0 .Epsilon: 0.1135578192100607\n",
      "434  Episode Reward:  13.0 .Epsilon: 0.11299003011401039\n",
      "435  Episode Reward:  117.0 .Epsilon: 0.11242507996344034\n",
      "436  Episode Reward:  12.0 .Epsilon: 0.11186295456362313\n",
      "437  Episode Reward:  82.0 .Epsilon: 0.11130363979080501\n",
      "438  Episode Reward:  27.0 .Epsilon: 0.11074712159185099\n",
      "439  Episode Reward:  11.0 .Epsilon: 0.11019338598389174\n",
      "440  Episode Reward:  81.0 .Epsilon: 0.10964241905397228\n",
      "STATIC WEIGHTS UPDATED\n",
      "441  Episode Reward:  139.0 .Epsilon: 0.10909420695870241\n",
      "442  Episode Reward:  135.0 .Epsilon: 0.1085487359239089\n",
      "443  Episode Reward:  126.0 .Epsilon: 0.10800599224428936\n",
      "444  Episode Reward:  48.0 .Epsilon: 0.10746596228306791\n",
      "445  Episode Reward:  86.0 .Epsilon: 0.10692863247165257\n",
      "STATIC WEIGHTS UPDATED\n",
      "446  Episode Reward:  14.0 .Epsilon: 0.1063939893092943\n",
      "447  Episode Reward:  32.0 .Epsilon: 0.10586201936274783\n",
      "448  Episode Reward:  105.0 .Epsilon: 0.10533270926593409\n",
      "449  Episode Reward:  151.0 .Epsilon: 0.10480604571960442\n",
      "450  Episode Reward:  52.0 .Epsilon: 0.1042820154910064\n",
      "451  Episode Reward:  122.0 .Epsilon: 0.10376060541355137\n",
      "452  Episode Reward:  21.0 .Epsilon: 0.1032418023864836\n",
      "STATIC WEIGHTS UPDATED\n",
      "453  Episode Reward:  156.0 .Epsilon: 0.10272559337455119\n",
      "454  Episode Reward:  87.0 .Epsilon: 0.10221196540767843\n",
      "455  Episode Reward:  139.0 .Epsilon: 0.10170090558064004\n",
      "STATIC WEIGHTS UPDATED\n",
      "456  Episode Reward:  132.0 .Epsilon: 0.10119240105273684\n",
      "457  Episode Reward:  120.0 .Epsilon: 0.10068643904747315\n",
      "458  Episode Reward:  130.0 .Epsilon: 0.10018300685223579\n",
      "459  Episode Reward:  151.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "460  Episode Reward:  153.0 .Epsilon: 0.1\n",
      "461  Episode Reward:  136.0 .Epsilon: 0.1\n",
      "462  Episode Reward:  131.0 .Epsilon: 0.1\n",
      "463  Episode Reward:  149.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "464  Episode Reward:  138.0 .Epsilon: 0.1\n",
      "465  Episode Reward:  170.0 .Epsilon: 0.1\n",
      "466  Episode Reward:  200.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "467  Episode Reward:  150.0 .Epsilon: 0.1\n",
      "468  Episode Reward:  135.0 .Epsilon: 0.1\n",
      "469  Episode Reward:  143.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "470  Episode Reward:  176.0 .Epsilon: 0.1\n",
      "471  Episode Reward:  158.0 .Epsilon: 0.1\n",
      "472  Episode Reward:  166.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "473  Episode Reward:  148.0 .Epsilon: 0.1\n",
      "474  Episode Reward:  209.0 .Epsilon: 0.1\n",
      "475  Episode Reward:  159.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "476  Episode Reward:  153.0 .Epsilon: 0.1\n",
      "477  Episode Reward:  141.0 .Epsilon: 0.1\n",
      "478  Episode Reward:  64.0 .Epsilon: 0.1\n",
      "479  Episode Reward:  153.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "480  Episode Reward:  166.0 .Epsilon: 0.1\n",
      "481  Episode Reward:  151.0 .Epsilon: 0.1\n",
      "482  Episode Reward:  165.0 .Epsilon: 0.1\n",
      "483  Episode Reward:  23.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "484  Episode Reward:  163.0 .Epsilon: 0.1\n",
      "485  Episode Reward:  160.0 .Epsilon: 0.1\n",
      "486  Episode Reward:  61.0 .Epsilon: 0.1\n",
      "487  Episode Reward:  144.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "488  Episode Reward:  156.0 .Epsilon: 0.1\n",
      "489  Episode Reward:  154.0 .Epsilon: 0.1\n",
      "490  Episode Reward:  39.0 .Epsilon: 0.1\n",
      "491  Episode Reward:  162.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "492  Episode Reward:  174.0 .Epsilon: 0.1\n",
      "493  Episode Reward:  126.0 .Epsilon: 0.1\n",
      "494  Episode Reward:  47.0 .Epsilon: 0.1\n",
      "495  Episode Reward:  120.0 .Epsilon: 0.1\n",
      "496  Episode Reward:  42.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "497  Episode Reward:  362.0 .Epsilon: 0.1\n",
      "498  Episode Reward:  133.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "499  Episode Reward:  160.0 .Epsilon: 0.1\n",
      "500  Episode Reward:  158.0 .Epsilon: 0.1\n",
      "501  Episode Reward:  173.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "502  Episode Reward:  131.0 .Epsilon: 0.1\n",
      "503  Episode Reward:  139.0 .Epsilon: 0.1\n",
      "504  Episode Reward:  140.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "505  Episode Reward:  215.0 .Epsilon: 0.1\n",
      "506  Episode Reward:  190.0 .Epsilon: 0.1\n",
      "507  Episode Reward:  125.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "508  Episode Reward:  133.0 .Epsilon: 0.1\n",
      "509  Episode Reward:  250.0 .Epsilon: 0.1\n",
      "510  Episode Reward:  181.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "511  Episode Reward:  392.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "512  Episode Reward:  178.0 .Epsilon: 0.1\n",
      "513  Episode Reward:  177.0 .Epsilon: 0.1\n",
      "514  Episode Reward:  124.0 .Epsilon: 0.1\n",
      "515  Episode Reward:  137.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "516  Episode Reward:  167.0 .Epsilon: 0.1\n",
      "517  Episode Reward:  217.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "518  Episode Reward:  265.0 .Epsilon: 0.1\n",
      "519  Episode Reward:  152.0 .Epsilon: 0.1\n",
      "520  Episode Reward:  196.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "521  Episode Reward:  261.0 .Epsilon: 0.1\n",
      "522  Episode Reward:  235.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "523  Episode Reward:  181.0 .Epsilon: 0.1\n",
      "524  Episode Reward:  124.0 .Epsilon: 0.1\n",
      "525  Episode Reward:  126.0 .Epsilon: 0.1\n",
      "526  Episode Reward:  35.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "527  Episode Reward:  68.0 .Epsilon: 0.1\n",
      "528  Episode Reward:  134.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "529  Episode Reward:  423.0 .Epsilon: 0.1\n",
      "530  Episode Reward:  214.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "531  Episode Reward:  360.0 .Epsilon: 0.1\n",
      "532  Episode Reward:  148.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "533  Episode Reward:  212.0 .Epsilon: 0.1\n",
      "534  Episode Reward:  278.0 .Epsilon: 0.1\n",
      "535  Episode Reward:  138.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "536  Episode Reward:  166.0 .Epsilon: 0.1\n",
      "537  Episode Reward:  221.0 .Epsilon: 0.1\n",
      "538  Episode Reward:  131.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "539  Episode Reward:  170.0 .Epsilon: 0.1\n",
      "540  Episode Reward:  182.0 .Epsilon: 0.1\n",
      "541  Episode Reward:  139.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "542  Episode Reward:  103.0 .Epsilon: 0.1\n",
      "543  Episode Reward:  320.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "544  Episode Reward:  212.0 .Epsilon: 0.1\n",
      "545  Episode Reward:  152.0 .Epsilon: 0.1\n",
      "546  Episode Reward:  145.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "547  Episode Reward:  221.0 .Epsilon: 0.1\n",
      "548  Episode Reward:  272.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "549  Episode Reward:  177.0 .Epsilon: 0.1\n",
      "550  Episode Reward:  139.0 .Epsilon: 0.1\n",
      "551  Episode Reward:  198.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "552  Episode Reward:  136.0 .Epsilon: 0.1\n",
      "553  Episode Reward:  167.0 .Epsilon: 0.1\n",
      "554  Episode Reward:  149.0 .Epsilon: 0.1\n",
      "555  Episode Reward:  140.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "556  Episode Reward:  144.0 .Epsilon: 0.1\n",
      "557  Episode Reward:  166.0 .Epsilon: 0.1\n",
      "558  Episode Reward:  151.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "559  Episode Reward:  155.0 .Epsilon: 0.1\n",
      "560  Episode Reward:  172.0 .Epsilon: 0.1\n",
      "561  Episode Reward:  176.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "562  Episode Reward:  132.0 .Epsilon: 0.1\n",
      "563  Episode Reward:  101.0 .Epsilon: 0.1\n",
      "564  Episode Reward:  182.0 .Epsilon: 0.1\n",
      "565  Episode Reward:  121.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "566  Episode Reward:  101.0 .Epsilon: 0.1\n",
      "567  Episode Reward:  15.0 .Epsilon: 0.1\n",
      "568  Episode Reward:  20.0 .Epsilon: 0.1\n",
      "569  Episode Reward:  39.0 .Epsilon: 0.1\n",
      "570  Episode Reward:  46.0 .Epsilon: 0.1\n",
      "571  Episode Reward:  122.0 .Epsilon: 0.1\n",
      "572  Episode Reward:  173.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "573  Episode Reward:  30.0 .Epsilon: 0.1\n",
      "574  Episode Reward:  128.0 .Epsilon: 0.1\n",
      "575  Episode Reward:  111.0 .Epsilon: 0.1\n",
      "576  Episode Reward:  21.0 .Epsilon: 0.1\n",
      "577  Episode Reward:  125.0 .Epsilon: 0.1\n",
      "578  Episode Reward:  40.0 .Epsilon: 0.1\n",
      "579  Episode Reward:  41.0 .Epsilon: 0.1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "STATIC WEIGHTS UPDATED\n",
      "580  Episode Reward:  55.0 .Epsilon: 0.1\n",
      "581  Episode Reward:  112.0 .Epsilon: 0.1\n",
      "582  Episode Reward:  146.0 .Epsilon: 0.1\n",
      "583  Episode Reward:  39.0 .Epsilon: 0.1\n",
      "584  Episode Reward:  122.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "585  Episode Reward:  108.0 .Epsilon: 0.1\n",
      "586  Episode Reward:  140.0 .Epsilon: 0.1\n",
      "587  Episode Reward:  98.0 .Epsilon: 0.1\n",
      "588  Episode Reward:  86.0 .Epsilon: 0.1\n",
      "589  Episode Reward:  88.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "590  Episode Reward:  172.0 .Epsilon: 0.1\n",
      "591  Episode Reward:  88.0 .Epsilon: 0.1\n",
      "592  Episode Reward:  187.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "593  Episode Reward:  100.0 .Epsilon: 0.1\n",
      "594  Episode Reward:  133.0 .Epsilon: 0.1\n",
      "595  Episode Reward:  112.0 .Epsilon: 0.1\n",
      "596  Episode Reward:  24.0 .Epsilon: 0.1\n",
      "597  Episode Reward:  83.0 .Epsilon: 0.1\n",
      "598  Episode Reward:  107.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "599  Episode Reward:  151.0 .Epsilon: 0.1\n",
      "600  Episode Reward:  102.0 .Epsilon: 0.1\n",
      "601  Episode Reward:  49.0 .Epsilon: 0.1\n",
      "602  Episode Reward:  100.0 .Epsilon: 0.1\n",
      "603  Episode Reward:  32.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "604  Episode Reward:  121.0 .Epsilon: 0.1\n",
      "605  Episode Reward:  120.0 .Epsilon: 0.1\n",
      "606  Episode Reward:  38.0 .Epsilon: 0.1\n",
      "607  Episode Reward:  141.0 .Epsilon: 0.1\n",
      "608  Episode Reward:  46.0 .Epsilon: 0.1\n",
      "609  Episode Reward:  29.0 .Epsilon: 0.1\n",
      "610  Episode Reward:  87.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "611  Episode Reward:  154.0 .Epsilon: 0.1\n",
      "612  Episode Reward:  201.0 .Epsilon: 0.1\n",
      "613  Episode Reward:  107.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "614  Episode Reward:  96.0 .Epsilon: 0.1\n",
      "615  Episode Reward:  208.0 .Epsilon: 0.1\n",
      "616  Episode Reward:  184.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "617  Episode Reward:  143.0 .Epsilon: 0.1\n",
      "618  Episode Reward:  170.0 .Epsilon: 0.1\n",
      "619  Episode Reward:  145.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "620  Episode Reward:  141.0 .Epsilon: 0.1\n",
      "621  Episode Reward:  109.0 .Epsilon: 0.1\n",
      "622  Episode Reward:  144.0 .Epsilon: 0.1\n",
      "623  Episode Reward:  114.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "624  Episode Reward:  119.0 .Epsilon: 0.1\n",
      "625  Episode Reward:  167.0 .Epsilon: 0.1\n",
      "626  Episode Reward:  126.0 .Epsilon: 0.1\n",
      "627  Episode Reward:  132.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "628  Episode Reward:  127.0 .Epsilon: 0.1\n",
      "629  Episode Reward:  107.0 .Epsilon: 0.1\n",
      "630  Episode Reward:  32.0 .Epsilon: 0.1\n",
      "631  Episode Reward:  122.0 .Epsilon: 0.1\n",
      "632  Episode Reward:  122.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "633  Episode Reward:  120.0 .Epsilon: 0.1\n",
      "634  Episode Reward:  108.0 .Epsilon: 0.1\n",
      "635  Episode Reward:  105.0 .Epsilon: 0.1\n",
      "636  Episode Reward:  172.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "637  Episode Reward:  103.0 .Epsilon: 0.1\n",
      "638  Episode Reward:  42.0 .Epsilon: 0.1\n",
      "639  Episode Reward:  11.0 .Epsilon: 0.1\n",
      "640  Episode Reward:  102.0 .Epsilon: 0.1\n",
      "641  Episode Reward:  130.0 .Epsilon: 0.1\n",
      "642  Episode Reward:  102.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "643  Episode Reward:  106.0 .Epsilon: 0.1\n",
      "644  Episode Reward:  103.0 .Epsilon: 0.1\n",
      "645  Episode Reward:  159.0 .Epsilon: 0.1\n",
      "646  Episode Reward:  51.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "647  Episode Reward:  157.0 .Epsilon: 0.1\n",
      "648  Episode Reward:  106.0 .Epsilon: 0.1\n",
      "649  Episode Reward:  83.0 .Epsilon: 0.1\n",
      "650  Episode Reward:  112.0 .Epsilon: 0.1\n",
      "651  Episode Reward:  121.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "652  Episode Reward:  480.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "653  Episode Reward:  125.0 .Epsilon: 0.1\n",
      "654  Episode Reward:  145.0 .Epsilon: 0.1\n",
      "655  Episode Reward:  150.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "656  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "657  Episode Reward:  139.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "658  Episode Reward:  151.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "659  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "660  Episode Reward:  122.0 .Epsilon: 0.1\n",
      "661  Episode Reward:  16.0 .Epsilon: 0.1\n",
      "662  Episode Reward:  176.0 .Epsilon: 0.1\n",
      "663  Episode Reward:  15.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "664  Episode Reward:  132.0 .Epsilon: 0.1\n",
      "665  Episode Reward:  146.0 .Epsilon: 0.1\n",
      "666  Episode Reward:  29.0 .Epsilon: 0.1\n",
      "667  Episode Reward:  153.0 .Epsilon: 0.1\n",
      "668  Episode Reward:  33.0 .Epsilon: 0.1\n",
      "669  Episode Reward:  11.0 .Epsilon: 0.1\n",
      "670  Episode Reward:  17.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "671  Episode Reward:  209.0 .Epsilon: 0.1\n",
      "672  Episode Reward:  161.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "673  Episode Reward:  208.0 .Epsilon: 0.1\n",
      "674  Episode Reward:  157.0 .Epsilon: 0.1\n",
      "675  Episode Reward:  148.0 .Epsilon: 0.1\n",
      "676  Episode Reward:  12.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "677  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "678  Episode Reward:  140.0 .Epsilon: 0.1\n",
      "679  Episode Reward:  135.0 .Epsilon: 0.1\n",
      "680  Episode Reward:  142.0 .Epsilon: 0.1\n",
      "681  Episode Reward:  129.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "682  Episode Reward:  137.0 .Epsilon: 0.1\n",
      "683  Episode Reward:  137.0 .Epsilon: 0.1\n",
      "684  Episode Reward:  134.0 .Epsilon: 0.1\n",
      "685  Episode Reward:  134.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "686  Episode Reward:  175.0 .Epsilon: 0.1\n",
      "687  Episode Reward:  138.0 .Epsilon: 0.1\n",
      "688  Episode Reward:  142.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "689  Episode Reward:  165.0 .Epsilon: 0.1\n",
      "690  Episode Reward:  202.0 .Epsilon: 0.1\n",
      "691  Episode Reward:  142.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "692  Episode Reward:  142.0 .Epsilon: 0.1\n",
      "693  Episode Reward:  200.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "694  Episode Reward:  226.0 .Epsilon: 0.1\n",
      "695  Episode Reward:  149.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "696  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "697  Episode Reward:  356.0 .Epsilon: 0.1\n",
      "698  Episode Reward:  110.0 .Epsilon: 0.1\n",
      "699  Episode Reward:  190.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "700  Episode Reward:  321.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "701  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "702  Episode Reward:  198.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "703  Episode Reward:  359.0 .Epsilon: 0.1\n",
      "704  Episode Reward:  64.0 .Epsilon: 0.1\n",
      "705  Episode Reward:  94.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "706  Episode Reward:  333.0 .Epsilon: 0.1\n",
      "707  Episode Reward:  272.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "708  Episode Reward:  179.0 .Epsilon: 0.1\n",
      "709  Episode Reward:  173.0 .Epsilon: 0.1\n",
      "710  Episode Reward:  104.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "711  Episode Reward:  287.0 .Epsilon: 0.1\n",
      "712  Episode Reward:  125.0 .Epsilon: 0.1\n",
      "713  Episode Reward:  117.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "714  Episode Reward:  108.0 .Epsilon: 0.1\n",
      "715  Episode Reward:  233.0 .Epsilon: 0.1\n",
      "716  Episode Reward:  115.0 .Epsilon: 0.1\n",
      "717  Episode Reward:  87.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "718  Episode Reward:  218.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "719  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "720  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "721  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "722  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "723  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "724  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "725  Episode Reward:  443.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "726  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "727  Episode Reward:  282.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "728  Episode Reward:  110.0 .Epsilon: 0.1\n",
      "729  Episode Reward:  193.0 .Epsilon: 0.1\n",
      "730  Episode Reward:  106.0 .Epsilon: 0.1\n",
      "731  Episode Reward:  61.0 .Epsilon: 0.1\n",
      "732  Episode Reward:  76.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "733  Episode Reward:  64.0 .Epsilon: 0.1\n",
      "734  Episode Reward:  116.0 .Epsilon: 0.1\n",
      "735  Episode Reward:  328.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "736  Episode Reward:  82.0 .Epsilon: 0.1\n",
      "737  Episode Reward:  151.0 .Epsilon: 0.1\n",
      "738  Episode Reward:  135.0 .Epsilon: 0.1\n",
      "739  Episode Reward:  138.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "740  Episode Reward:  178.0 .Epsilon: 0.1\n",
      "741  Episode Reward:  204.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "742  Episode Reward:  189.0 .Epsilon: 0.1\n",
      "743  Episode Reward:  294.0 .Epsilon: 0.1\n",
      "744  Episode Reward:  137.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "745  Episode Reward:  163.0 .Epsilon: 0.1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "746  Episode Reward:  165.0 .Epsilon: 0.1\n",
      "747  Episode Reward:  163.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "748  Episode Reward:  174.0 .Epsilon: 0.1\n",
      "749  Episode Reward:  167.0 .Epsilon: 0.1\n",
      "750  Episode Reward:  154.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "751  Episode Reward:  151.0 .Epsilon: 0.1\n",
      "752  Episode Reward:  138.0 .Epsilon: 0.1\n",
      "753  Episode Reward:  147.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "754  Episode Reward:  397.0 .Epsilon: 0.1\n",
      "755  Episode Reward:  150.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "756  Episode Reward:  144.0 .Epsilon: 0.1\n",
      "757  Episode Reward:  171.0 .Epsilon: 0.1\n",
      "758  Episode Reward:  152.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "759  Episode Reward:  135.0 .Epsilon: 0.1\n",
      "760  Episode Reward:  161.0 .Epsilon: 0.1\n",
      "761  Episode Reward:  166.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "762  Episode Reward:  145.0 .Epsilon: 0.1\n",
      "763  Episode Reward:  139.0 .Epsilon: 0.1\n",
      "764  Episode Reward:  159.0 .Epsilon: 0.1\n",
      "765  Episode Reward:  156.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "766  Episode Reward:  145.0 .Epsilon: 0.1\n",
      "767  Episode Reward:  243.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "768  Episode Reward:  187.0 .Epsilon: 0.1\n",
      "769  Episode Reward:  164.0 .Epsilon: 0.1\n",
      "770  Episode Reward:  188.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "771  Episode Reward:  148.0 .Epsilon: 0.1\n",
      "772  Episode Reward:  154.0 .Epsilon: 0.1\n",
      "773  Episode Reward:  163.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "774  Episode Reward:  163.0 .Epsilon: 0.1\n",
      "775  Episode Reward:  191.0 .Epsilon: 0.1\n",
      "776  Episode Reward:  241.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "777  Episode Reward:  161.0 .Epsilon: 0.1\n",
      "778  Episode Reward:  158.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "779  Episode Reward:  319.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "780  Episode Reward:  444.0 .Epsilon: 0.1\n",
      "781  Episode Reward:  270.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "782  Episode Reward:  195.0 .Epsilon: 0.1\n",
      "783  Episode Reward:  247.0 .Epsilon: 0.1\n",
      "784  Episode Reward:  187.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "785  Episode Reward:  288.0 .Epsilon: 0.1\n",
      "786  Episode Reward:  171.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "787  Episode Reward:  211.0 .Epsilon: 0.1\n",
      "788  Episode Reward:  130.0 .Epsilon: 0.1\n",
      "789  Episode Reward:  28.0 .Epsilon: 0.1\n",
      "790  Episode Reward:  65.0 .Epsilon: 0.1\n",
      "791  Episode Reward:  38.0 .Epsilon: 0.1\n",
      "792  Episode Reward:  87.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "793  Episode Reward:  33.0 .Epsilon: 0.1\n",
      "794  Episode Reward:  149.0 .Epsilon: 0.1\n",
      "795  Episode Reward:  156.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "796  Episode Reward:  277.0 .Epsilon: 0.1\n",
      "797  Episode Reward:  113.0 .Epsilon: 0.1\n",
      "798  Episode Reward:  19.0 .Epsilon: 0.1\n",
      "799  Episode Reward:  135.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "800  Episode Reward:  151.0 .Epsilon: 0.1\n",
      "801  Episode Reward:  139.0 .Epsilon: 0.1\n",
      "802  Episode Reward:  199.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "803  Episode Reward:  184.0 .Epsilon: 0.1\n",
      "804  Episode Reward:  210.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "805  Episode Reward:  278.0 .Epsilon: 0.1\n",
      "806  Episode Reward:  91.0 .Epsilon: 0.1\n",
      "807  Episode Reward:  112.0 .Epsilon: 0.1\n",
      "808  Episode Reward:  62.0 .Epsilon: 0.1\n",
      "809  Episode Reward:  161.0 .Epsilon: 0.1\n",
      "810  Episode Reward:  11.0 .Epsilon: 0.1\n",
      "811  Episode Reward:  10.0 .Epsilon: 0.1\n",
      "812  Episode Reward:  11.0 .Epsilon: 0.1\n",
      "813  Episode Reward:  11.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "814  Episode Reward:  22.0 .Epsilon: 0.1\n",
      "815  Episode Reward:  13.0 .Epsilon: 0.1\n",
      "816  Episode Reward:  31.0 .Epsilon: 0.1\n",
      "817  Episode Reward:  17.0 .Epsilon: 0.1\n",
      "818  Episode Reward:  58.0 .Epsilon: 0.1\n",
      "819  Episode Reward:  200.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "820  Episode Reward:  242.0 .Epsilon: 0.1\n",
      "821  Episode Reward:  292.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "822  Episode Reward:  196.0 .Epsilon: 0.1\n",
      "823  Episode Reward:  337.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "824  Episode Reward:  293.0 .Epsilon: 0.1\n",
      "825  Episode Reward:  241.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "826  Episode Reward:  281.0 .Epsilon: 0.1\n",
      "827  Episode Reward:  228.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "828  Episode Reward:  175.0 .Epsilon: 0.1\n",
      "829  Episode Reward:  195.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "830  Episode Reward:  199.0 .Epsilon: 0.1\n",
      "831  Episode Reward:  157.0 .Epsilon: 0.1\n",
      "832  Episode Reward:  169.0 .Epsilon: 0.1\n",
      "833  Episode Reward:  167.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "834  Episode Reward:  168.0 .Epsilon: 0.1\n",
      "835  Episode Reward:  105.0 .Epsilon: 0.1\n",
      "836  Episode Reward:  189.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "837  Episode Reward:  66.0 .Epsilon: 0.1\n",
      "838  Episode Reward:  152.0 .Epsilon: 0.1\n",
      "839  Episode Reward:  125.0 .Epsilon: 0.1\n",
      "840  Episode Reward:  162.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "841  Episode Reward:  155.0 .Epsilon: 0.1\n",
      "842  Episode Reward:  138.0 .Epsilon: 0.1\n",
      "843  Episode Reward:  155.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "844  Episode Reward:  131.0 .Epsilon: 0.1\n",
      "845  Episode Reward:  152.0 .Epsilon: 0.1\n",
      "846  Episode Reward:  137.0 .Epsilon: 0.1\n",
      "847  Episode Reward:  146.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "848  Episode Reward:  157.0 .Epsilon: 0.1\n",
      "849  Episode Reward:  192.0 .Epsilon: 0.1\n",
      "850  Episode Reward:  160.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "851  Episode Reward:  186.0 .Epsilon: 0.1\n",
      "852  Episode Reward:  176.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "853  Episode Reward:  209.0 .Epsilon: 0.1\n",
      "854  Episode Reward:  180.0 .Epsilon: 0.1\n",
      "855  Episode Reward:  220.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "856  Episode Reward:  362.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "857  Episode Reward:  312.0 .Epsilon: 0.1\n",
      "858  Episode Reward:  11.0 .Epsilon: 0.1\n",
      "859  Episode Reward:  310.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "860  Episode Reward:  329.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "861  Episode Reward:  301.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "862  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "863  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "864  Episode Reward:  16.0 .Epsilon: 0.1\n",
      "865  Episode Reward:  15.0 .Epsilon: 0.1\n",
      "866  Episode Reward:  12.0 .Epsilon: 0.1\n",
      "867  Episode Reward:  16.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "868  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "869  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "870  Episode Reward:  206.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "871  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "872  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "873  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "874  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "875  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "876  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "877  Episode Reward:  224.0 .Epsilon: 0.1\n",
      "878  Episode Reward:  178.0 .Epsilon: 0.1\n",
      "879  Episode Reward:  35.0 .Epsilon: 0.1\n",
      "880  Episode Reward:  98.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "881  Episode Reward:  281.0 .Epsilon: 0.1\n",
      "882  Episode Reward:  306.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "883  Episode Reward:  299.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "884  Episode Reward:  319.0 .Epsilon: 0.1\n",
      "885  Episode Reward:  269.0 .Epsilon: 0.1\n",
      "886  Episode Reward:  136.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "887  Episode Reward:  100.0 .Epsilon: 0.1\n",
      "888  Episode Reward:  119.0 .Epsilon: 0.1\n",
      "889  Episode Reward:  148.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "890  Episode Reward:  146.0 .Epsilon: 0.1\n",
      "891  Episode Reward:  140.0 .Epsilon: 0.1\n",
      "892  Episode Reward:  131.0 .Epsilon: 0.1\n",
      "893  Episode Reward:  157.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "894  Episode Reward:  142.0 .Epsilon: 0.1\n",
      "895  Episode Reward:  169.0 .Epsilon: 0.1\n",
      "896  Episode Reward:  213.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "897  Episode Reward:  178.0 .Epsilon: 0.1\n",
      "898  Episode Reward:  210.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "899  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "900  Episode Reward:  287.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "901  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "902  Episode Reward:  55.0 .Epsilon: 0.1\n",
      "903  Episode Reward:  63.0 .Epsilon: 0.1\n",
      "904  Episode Reward:  62.0 .Epsilon: 0.1\n",
      "905  Episode Reward:  27.0 .Epsilon: 0.1\n",
      "906  Episode Reward:  79.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "907  Episode Reward:  157.0 .Epsilon: 0.1\n",
      "908  Episode Reward:  385.0 .Epsilon: 0.1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "STATIC WEIGHTS UPDATED\n",
      "909  Episode Reward:  313.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "910  Episode Reward:  239.0 .Epsilon: 0.1\n",
      "911  Episode Reward:  261.0 .Epsilon: 0.1\n",
      "912  Episode Reward:  58.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "913  Episode Reward:  219.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "914  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "915  Episode Reward:  46.0 .Epsilon: 0.1\n",
      "916  Episode Reward:  177.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "917  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "918  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "919  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "920  Episode Reward:  340.0 .Epsilon: 0.1\n",
      "921  Episode Reward:  240.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "922  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "923  Episode Reward:  250.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "924  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "925  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "926  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "927  Episode Reward:  12.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "928  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "929  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "930  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "931  Episode Reward:  50.0 .Epsilon: 0.1\n",
      "932  Episode Reward:  18.0 .Epsilon: 0.1\n",
      "933  Episode Reward:  49.0 .Epsilon: 0.1\n",
      "934  Episode Reward:  13.0 .Epsilon: 0.1\n",
      "935  Episode Reward:  16.0 .Epsilon: 0.1\n",
      "936  Episode Reward:  29.0 .Epsilon: 0.1\n",
      "937  Episode Reward:  10.0 .Epsilon: 0.1\n",
      "938  Episode Reward:  16.0 .Epsilon: 0.1\n",
      "939  Episode Reward:  48.0 .Epsilon: 0.1\n",
      "940  Episode Reward:  47.0 .Epsilon: 0.1\n",
      "941  Episode Reward:  21.0 .Epsilon: 0.1\n",
      "942  Episode Reward:  11.0 .Epsilon: 0.1\n",
      "943  Episode Reward:  16.0 .Epsilon: 0.1\n",
      "944  Episode Reward:  15.0 .Epsilon: 0.1\n",
      "945  Episode Reward:  12.0 .Epsilon: 0.1\n",
      "946  Episode Reward:  14.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "947  Episode Reward:  49.0 .Epsilon: 0.1\n",
      "948  Episode Reward:  54.0 .Epsilon: 0.1\n",
      "949  Episode Reward:  20.0 .Epsilon: 0.1\n",
      "950  Episode Reward:  40.0 .Epsilon: 0.1\n",
      "951  Episode Reward:  46.0 .Epsilon: 0.1\n",
      "952  Episode Reward:  45.0 .Epsilon: 0.1\n",
      "953  Episode Reward:  44.0 .Epsilon: 0.1\n",
      "954  Episode Reward:  48.0 .Epsilon: 0.1\n",
      "955  Episode Reward:  50.0 .Epsilon: 0.1\n",
      "956  Episode Reward:  48.0 .Epsilon: 0.1\n",
      "957  Episode Reward:  18.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "958  Episode Reward:  52.0 .Epsilon: 0.1\n",
      "959  Episode Reward:  27.0 .Epsilon: 0.1\n",
      "960  Episode Reward:  13.0 .Epsilon: 0.1\n",
      "961  Episode Reward:  53.0 .Epsilon: 0.1\n",
      "962  Episode Reward:  15.0 .Epsilon: 0.1\n",
      "963  Episode Reward:  12.0 .Epsilon: 0.1\n",
      "964  Episode Reward:  21.0 .Epsilon: 0.1\n",
      "965  Episode Reward:  20.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "966  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "967  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "968  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "969  Episode Reward:  257.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "970  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "971  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "972  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "973  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "974  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "975  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "976  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "977  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "978  Episode Reward:  116.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "979  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "980  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "981  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "982  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "983  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "984  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "985  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "986  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "987  Episode Reward:  51.0 .Epsilon: 0.1\n",
      "988  Episode Reward:  42.0 .Epsilon: 0.1\n",
      "989  Episode Reward:  24.0 .Epsilon: 0.1\n",
      "990  Episode Reward:  305.0 .Epsilon: 0.1\n",
      "991  Episode Reward:  14.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "992  Episode Reward:  317.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "993  Episode Reward:  358.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "994  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "995  Episode Reward:  353.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "996  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "997  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "998  Episode Reward:  500.0 .Epsilon: 0.1\n",
      "STATIC WEIGHTS UPDATED\n",
      "999  Episode Reward:  500.0 .Epsilon: 0.1\n"
     ]
    }
   ],
   "source": [
    "numberEpisodes = 1000\n",
    "\n",
    "discount = 0.99\n",
    "counter = 0\n",
    "epsilon = 1\n",
    "epsilonDecay = 0.995\n",
    "rewards = []\n",
    "tau = 0.5\n",
    "memorySize = 5000\n",
    "memoryPreviousStates = np.zeros((memorySize,4))\n",
    "memoryStates = np.zeros((memorySize,4))\n",
    "memoryARD = np.zeros((memorySize,3))\n",
    "numberOfLearningFromReplaySteps = 512\n",
    "itemsInMemory = 0\n",
    "initialiseReplayAfter = 1000 \n",
    "\n",
    "for episode in range (numberEpisodes):\n",
    "    epsilon = max (epsilon*epsilonDecay, 0.1)\n",
    "    \n",
    "    \n",
    "    done = False\n",
    "    prevObs = env.reset()\n",
    "    \n",
    "    episodeReward = 0\n",
    "    for step in range (600):\n",
    "        if done:\n",
    "            break\n",
    "        #env.render()\n",
    "        \n",
    "\n",
    "        target = model.predict(np.expand_dims(prevObs, axis = 0))[0]\n",
    "        if (random.random()<epsilon):\n",
    "            action = random.choice([0,1])\n",
    "        else:\n",
    "            action = np.argmax(target)\n",
    "        \n",
    "        obs, reward, done, _ = env.step (action)\n",
    "\n",
    "\n",
    "        memoryPreviousStates[counter] = prevObs\n",
    "        memoryStates[counter] = obs\n",
    "        if (episodeReward <= 498):\n",
    "            memoryARD[counter] = [float(action), reward, float (done)]\n",
    "        else: # Is not the fault of the agent's action that the environment restarts after 500 epochs. So we set done=False for the memory replay\n",
    "            memoryARD[counter] = [float(action), reward, float (0)]\n",
    "        \n",
    "        if (counter % 500 == 0):\n",
    "            \n",
    "            model, staticModel = updateStaticWeights(model, staticModel,tau)\n",
    "            print (\"STATIC WEIGHTS UPDATED\")\n",
    "        \n",
    "        if (itemsInMemory < (counter+1)):\n",
    "            itemsInMemory = min ((itemsInMemory+1), memorySize)\n",
    "        if (itemsInMemory >= initialiseReplayAfter):\n",
    "            executeMemoryReplayStep(memoryPreviousStates, memoryStates, memoryARD, model, staticModel, itemsInMemory)\n",
    "\n",
    "\n",
    "        prevObs = obs\n",
    "        episodeReward += reward\n",
    "        counter = (counter + 1) % memorySize\n",
    "    print (episode, \" Episode Reward: \", episodeReward, \".Epsilon:\", epsilon)\n",
    "    rewards.append(episodeReward)\n",
    "\n",
    "env.close()\n",
    "model.save_weights(\"doubleweights.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO2dd7weRfX/P+e5Jb0XEtJuQgKElkBCCCAIBJCmQUEFC6hIfioqlq8QRL8o4lcsFFFBKUoREUURSBBIpRNyQwnphdz05N70m3JzyzO/P57dfWZnZ2dn21Nu5v16JffZ3dmZ2Xb27DlnzhBjDAaDwWBoX2SK3QGDwWAwJI8R7gaDwdAOMcLdYDAY2iFGuBsMBkM7xAh3g8FgaIdUFrsDANC3b19WU1NT7G4YDAZDWbFgwYJtjLF+sm0lIdxrampQW1tb7G4YDAZDWUFEa/22GbOMwWAwtEOMcDcYDIZ2iBHuBoPB0A4xwt1gMBjaIUa4GwwGQztES7gTUR0RfUBE7xFRrbWuNxHNIKKV1t9e1noionuIaBURLSSik9I8AIPBYDB4CaO5n80YG8sYG28tTwUwizE2CsAsaxkALgQwyvo3BcB9SXXWYDAYDHrEiXOfDOAs6/cjAOYCuNFa/yjL5RJ+i4h6EtFAxtjmOB01GAzhqNu2D7OW1WPM4B4YX9PbWc8Yw1MLNmDy2EGorszrd7sPtOCVFQ3o06UaA3p0xIh+XZ1tO/Y14/G31qKptQ3NrVncdOFoZDJU0OMJwysrGrD7QAtas1nMWlqPEX27YPTA7rjw+IGesut37MdfXq/DVz5Sg8G9OgMAZi7Zig827gYRsHlXEw60tGFk/6646tRh6Nm5Wtk2YwwPv1GHBWt3YkTfLmhjDOt3HEBNn86oyGRw5SlDsLepFX98eTUGdO+ISaMPw5ghPRM/B7rCnQF4iYgYgD8xxu4HcJgtsBljm4mov1V2EID13L4brHUu4U5EU5DT7DF06NDoR2AwGKSc9Zu5zu+62y92fv930Rb84KmFqNu+Dz/42NHO+u89+R5mLauX7vPS4i24Y8YKZ3lU/274zMlDUup5fK7689uedb06V0mF+7l3voyDrVk8OX8dFt96AQDgq4/KB1Ue3rMTLh83WNn22u378dPnlvhu79axErdOy2/v371jKsJd1yxzOmPsJORMLtcR0ZmKsrLXuWdGEMbY/Yyx8Yyx8f36SUfPGgyGFNh9oAUAsH1vs2v9pt1Nvvu0Zt2P8J6mluQ7liLVlRnPMdgcbM0CAPY1twXWk/Wpg6eNmwDpS6fVoML6wrnrs2NydXDbv3/ekfjCxGGBdUZBS7gzxjZZf+sBPA1gAoCtRDQQAKy/9it/AwD+lT4YwKakOmwwGNIhjJGFqHRNMjKS6i3z6qneMgFF+O1pnsZA4U5EXYiom/0bwPkAFgF4FsDVVrGrATxj/X4WwFVW1MxEALuNvd1gKD3CzLApFq0oL9meE6JFmFGUF94kecWk+ZLUsbkfBuBpqxOVAP7GGHuBiOYD+AcRXQNgHYBPW+WfB3ARgFUA9gP4cuK9NhgMkfETJ0o5I7wJStmZKoNA2rJdNa902CmneYFun19e+09Tcw8U7oyxDwGMkazfDmCSZD0DcF0ivTMYDAUjjKDJlJlZJkNqoZ0s+sJbps0nhRmhajAYAKgFjSgWy024hzF/JPkO4FuV9aGoNneDwVB+LNq4O/Q+KkEjCrwys8ogjMldVU6nDr+XA0m2p3kejXA3GNohl/zutUTrE00aZWdzJ32NPEnzjcuhKjllxixjMBgSRwzrCyNm2rNZRkVohyrXri3ImWt7Ap3ywQh3g8GQQyFpPKGQZSY5MqQXow7Ej5h0CW/+t0xzT1G6l9klMhgMaREiErIsNXd9s4xiW1jRT96frkFM4WoLhRHuBoMhNIfqCFUdfB2qUs09vX4Y4W4wGAAERMsIyxXlJtwpxCCmBIeyuh2m3nOW5heQEe4GwyGGnzxRm2WEaJnyku2h0g8ozTKhHaqSOgo0QtUId4PBACCcqaXsQiGRrEauwq8deShkehjhbjAcoohaaHsOhUyqv2FfDzLnqTsrpDHLGAyGIlLuI1QzoQYxxWvL36Fq0g8YDIYiEEbQFCwHV0KEcagqiXHgstNrRqgaDIbE8BMo6sRhTFhuvyQaLROQfsDkljEYDOkTInFY4dLnJkMmo9/n9MwyeuuSwgh3g8EQmvIS7SEn64i4TVre5VC1cstwK41ZxmAwpI4yzl1cjiHdP/OnN3HjUwujVxCBQgb3+Jp15Eb31DDC3WAwANDL537+MYfZayK38/aaHXiydn3k/aOQCZVbJrlp9nhMbhmDwVAQRDml41D9xtkjc8tlZpcJI0STPDS3Q7WwE2Qb4W4wGADomS7s6I4yk+2OdI/rCNbZP3Amplg90McId4PBEIgtsPJOwSJ2JgJhRqgmeWzu0ajJ1auDEe4GgwGAnvAhR3MvL+kus3f7kvCh2dp+mpExMoxwNxgMAPSEj60Bl5vmTgmZk5I47EKdOyPcDYZDDcf+LKxWRsswV5kyk+2htOa4XyV+phhjljEYyoiWtiz+Ubse2Wy5ibtw2AIrr7mX1/FSCIdqkvncXX2w6yjQq7GyIK0YDO2Ue+esxl0zV6BDZQaTxw4qdndioROWV27ZIG3sYyv0K8n1MjCau8FQPmzbexAAsPtAS5F7ki62jCIfk06pk1Scu54/1meyjgJHGhnhbjAkQLkJOxnqafasMo4GXF4HnLEknc51SsvkZGzuBkMZUWYTEgHwF+J6g5jKM1omU8CXUlBsuxnEZDAYCoo6cRhzlSk34R6m30qzTMKTdaSJEe4GQwKUW/SIDJVD1RMtU4gOJUkJjFAtNNrCnYgqiOhdIppmLQ8nonlEtJKIniSiamt9B2t5lbW9Jp2uGwzFpwytMg5RTBRhQgpLiTBRPrHj3H3WE1Hu/BXo3IXR3K8HsJRb/iWAuxhjowDsBHCNtf4aADsZYyMB3GWVMxgMJU6ofO5pdiQFytWcFAct4U5EgwFcDOBBa5kAnAPgKavIIwAutX5PtpZhbZ9Eaea1NBhKgPYgM5RPqTBCtdwOOJQIijmIyTXTkjBCldTVJ4qu5n43gBsAZK3lPgB2McZareUNAOwRHIMArAcAa/tuq7zB0O5oX3qLKp+7JZzKNRQyRMKz8joyfwKFOxFdAqCeMbaAXy0pyjS28fVOIaJaIqptaGjQ6qzBUGqUm+05DgQ988alf3gdF/721UJ0SZtipSoWZ12iEDNCxUUn/cDpAD5BRBcB6AigO3KafE8iqrS088EANlnlNwAYAmADEVUC6AFgh1gpY+x+APcDwPjx4w+dJ8RgKDLkk/1LZ5o9ncRh763fFblvqREi4Zkyt0wZaf6Bmjtj7CbG2GDGWA2AKwDMZox9HsAcAJdbxa4G8Iz1+1lrGdb22exQUm8M7ZaL73kVn/nTm6517cksExTnTkRlPFmHftm0TE6FvlXiJA67EcDfieg2AO8CeMha/xCAx4hoFXIa+xXxumgwlAaLN+0pdhdSJUj4EHjNvXSlu0yXzL+Ukkvnq1NGPKc5h2oJZoVkjM0FMNf6/SGACZIyTQA+nUDfDIayodw0WRnKCbJZPtrDXi5VZH3TzUPf2NSCA81tiffJ6kVK9coxKX8NBkMgjlAsg8k6ZH3TnUP1+J+8FLpuZXmJX8NkhTQYDKkQNXEYIW9zL2XVXWqWKWi3fVL+mqyQBkP5UbqiTp/AaBkqj2n2lH0rcscJVHKDmAwGg4R2FCyjtrmXUVZIWd90zTJR6laVcY1QTaQH+hjhbjAYtLAH4QClPXhLFo1SDlE+SWOEu8GQAKUs7LQJyBzmipYpRH8iotLc416msC8HtxafO4HGoWowlAEqU0apI8oYnayQ5T6HaiG6XSqnxgh3g8EAIHi0LR8tUyoCTIYqzj2NunWxc/MUyjRkhLvBYAAQNEE2ywnIMpisQ25zL5yvIGgO1UJhhLvBIKG1LYvjbnkR/35ng1b5lrbSFXYifgInTOKwUuUnzy725P8BSsNXQHYoqbG5GwzFo7GpFXsPtuKnzy3RKv/LF5al3KP0EeX2/uZWz/ZSD4V8+I06LNrozQFUrJdSWcyhajAY2hdBJorH31qXLwt7DtB4k3UUy5yTVLSMDn7HWGjnuxHuBkMMSt1MEQbRocovOmYZYTksxdJkk4pzj/tyMiNUDYYi056EdhIwMCHlbzSyRZLuhdSa+SMU51AVOXpAt9T6YbJCGgwGABrD4yn+dHXFMkEnNbF33HdTLiskQ2WGcOWEoThuUI94FSowmrvBoCDoM7w9KfiqI/VOsxfV5h5pt9jkfQXFhzGgNcvQq0t1qu0Y4W4wSCjnkadB6JqcPDZ47ndUIV08s0yO+OkHNMow+W+7H63Z3MqqMHP/RcAId4PBAMD7FeJZJipbX0TKclQL+9y1tGUBAJUV6YpfI9wNhhiUq7ALiz1CNe5cpMU3yxRghCrXhuz+aLM19wqjuRsMhkKgGqFqF4mZOCyqWeaXLyzDTf/+IFqjSNAsEyu3TG6cgD2aucKYZQyG4lEKDrhisW7Hfteya4RqxDqj7nff3NV44u11wQV9CEqKVkiMWcZgKANKSWiEJUjQPvxGHVZsbcyVZcII1ciDmIrkUE1oekAts46PQ9XOh9+azQl341A1GIpJwLNcytkR/QgTCbRx1wEAkmn2wNDU0oYtu5tCtZ0t0unKOOak4l8vY5YxGIrIoTQdm41M6Gc5aewaocqAax+txcRfzArXSLEcqgmFtmrNocq3K45QpVzGUQCoMmYZg6Hw6Cp45WyW0cGO7GD2NHvcYKBXV24LXV+xXpqlNIOUrblXmmgZg8FQLGzF3SMTI0rJYpllSuElbH892Db3yozR3A2GghNVBrW2ZVEzdTrunbsq0f6kgZaJwVUoJ5yIYkTLFNmhmmUM98xaie17D0aqJ84IVcehamvuxuZuMBQeXSEkPp5NrTmt7PezS1+46+Bo7iwvIAlx4twT6VZo7Ov01ofbceeMFfjh09Fj5uOSD4U0wt1gKBphZVGpz1SkQma5yA864kddUvTEYUWyuduTdexvbgMANFsv4dBoXFi/Eao5hzShjZloGYOhaEQVQUlNClEq2MKdsfyLK47mXqzTYstRW6inPYBIRaFe/Ea4GwwJEjffeanRlvVqoXFs7sV2qNomkeqIwj1O9+0JsvOzWhnN3WAoONrCWXg+bY29lGV7mMAR3ubu7A+KMVlHcc/Myvq9ANJN2hV0bkpmmj0i6khEbxPR+0S0mIh+aq0fTkTziGglET1JRNXW+g7W8ipre026h2AwJE/syShKWbr7IBN3jlkGLK9pUvlN1mHb3J95bxOA9AcQycn1YenmPbmllKMzdY7wIIBzGGNjAIwFcAERTQTwSwB3McZGAdgJ4Bqr/DUAdjLGRgK4yypnMJQlYUP38rK9DKW7BP74+WgZd/4U/WONO1lHUqGUVZURzTIhR6iKkM/vNAg8QpZjr7VYZf1jAM4B8JS1/hEAl1q/J1vLsLZPolIYQWAwhEF3hKrwiJZC7pIksUzUnnht/ijDHHLc09OWkNG+GJN3FFoKar2+iKiCiN4DUA9gBoDVAHYxxlqtIhsADLJ+DwKwHgCs7bsB9JHUOYWIaomotqGhId5RGAwpEVaUOJp7Gch4nS62OWYZPlqGivYSa40o3EVh3qNTVaR64n6RkRgbmSJawp0x1sYYGwtgMIAJAEbLill/ZV2WjF5m9zPGxjPGxvfr10+3vwZDQYg+AjPe/qWG2yzDjVDlzTIA/vrWWq1Rn3HNMslp7mk6VOV99E5jWELRMoyxXQDmApgIoCcRVVqbBgPYZP3eAGAIAFjbewDYkURnDYZCETeOuxzNM9JBTFziMKcc3C+vVfV78aP/LMK3nng3sI0op4VPKxxVcxeJWk/cy1pIy4xOtEw/Iupp/e4E4FwASwHMAXC5VexqAM9Yv5+1lmFtn83K8U43GGLQXm74fOIwYYQqd4AHW3OjPnfsaw6sjz8v+w62+pbjuW36Eud3VM1dfHEl9QUgw69m0fVYCtEyAwHMIaKFAOYDmMEYmwbgRgDfI6JVyNnUH7LKPwSgj7X+ewCmJt9tgyFd4g6vby/qzF/eWIMXFm0BIOSW4c6PLSd14iZ4s8yxt7yI99fvCtyHP5d2RsWwiH2zk3cVk7S1+MqgAoyxhQBOlKz/EDn7u7i+CcCnE+mdwVBkgoS0KM/ai1C3Wb/jAL721wX41ImD8scq2Nxtga0jrMTzs3DjbowZ0lO5D/9CiKy5K+oc0a8LPmzYh1OG98a8NWoLslbrPoX4yU4KgRmhajBIiD4Cs3wQraUqBx9zlRPrsdZrCK4oFlpeEG8OOa2fH/wXQLcOOR23a4dAXTdR0o4QN8LdYJAQRUi3ZVlZOlJ1sYV/zubuinTXrkMsKRNvtg3fpo2zxHzq3je021I1xH8BhLlicS5vTpYXTnU3wt1giAH/qJ76i1llpbmLqBRJxphv4rAkBjH9Yc4q1NbtwKKNu3HUj17AjCVbXW3HRfwqkdnck7p2uv6aUnCoGgyHHFEESn3jwbK2uav6Lppl3DZ3a72GsPKLc//1i8tx+R/fxLuWg3XO8vrAfeLg0tzDvJxivAII5MnvniZGuBsMCsI+zHz5u2euSLo7kYmr/bryuQuTdTDHoRosrsRueF4IVgF+RGkSUYtiO60us0yyL49SecEb4W4wSNB9QL3CKf/z7pkrS9IGH9WR54xQRXTNPUiQ2nXxI0iT0NzFrsmibtK+VuL5MWYZg6GMEMVDsSanEIkrt1xmGdHmHsahGlA062juCQt3j+ae99KGqj7REaomWsZgKBqyB//3s1fi2kdrtfZPw16cFGLPVEKaMcaJImGyjhgOVdGUI/sKiDhuSbsfSecDKpVLXtjAToOhTFA9oL95KWdL37b3oCTlr7tsqQh3d3RLxD5x0TJ8jY5A1uoHUy+npbmL1yliPXF74nKoGrOMwVCajL9tpoawKmSP4hE1WsY5Zq30A0HbvaNdkziH3pHE3jj3pK6Vf24ZYTmZ5nwxwt1gkJDUNHLF0NxlmnlsZ6ErWkZu1ojSNz+zTIYLl2lL4Ry6+x/GZxCvL2mn+eUxwt2gZNHG3VhV31jsbhQc7WiZgM/9UnGo6hDU1Xy0jDsUMlRumaDtMpt7CtEyhbgsHasy+NJpNfk+eLJCpivojc3doOSS370GAKi7/eIi96Q4hBUComZXHM1dsi7ifvn98xv9NPeouWX4dfJomeB6AyHxCyHFEapW3U997TQc1r2jf5cSas8Po7kbDBLEB10nNS3gFZAshUiPwD4EbY8gxVyDmIQ2bMEfJSuk90XhHcSURvx5dLNS9DZNVkiDoZSwHuafT18q3Rz0sJZMtIxGN4L8DPncMhRdOErWZV2ae+5v2oOYpJp7QtdKtxYTLWMwFAHPg675IJaqQ1VeMEyd4q5egaxDVlKYX+XY7znJ15bA148yKVqMrJZh2iQqsWn2DIZDEechjvk0lopD1ZULxk9EBdjc8yl/3WXtofx+DsLz7nwZl9+XS9W7tdE9ibbvrE582ynEufMvXWcQU4GvVdqRM8ahajCocMwEPpsVGm1uexE095Tq9U/5q7a5r6zfCwA40NyGb0sm0Zbb3JM1y6jaDNOOlnnLp0wuKyQ3zteYZQyGwhM0TN6mWbAZeM0ySfYqOno2d739Ce7JOtocU4q6/gMtbdL1Mi3aLytk5+oKdSM+iH1ztSm0zXPtGcNx6+Rjo7VZUCOMF6O5GwwKnEgQn+f0oCCwvHHupREKGWa7dB/ut6i5677AZOdCjJbJSl4UMuEfFr+pAXO/mW//Lhs3OIKQlneykJEygNHcDQYf3A+o34PZKki2Uohz1yWMCWnGkq3+KX81pbs8QkWuRfPmC77+pEar6o4F8A5S028/SJgbs4yhJGjLMnztsQV4Z93OYnelIOjKEFFgibsVQ7YHCSC/PgV11T1ZRx7HoRqg4eq0GzSISZaHXQdPbhl4Xyh+JyApIVxoI40R7gYttu5pwguLt+C6x98pdleKgp/gEoVNKYRCytDpxu4DLVp15TR3PsJFEuIiQSaYidwDveQ2d05zjzgJuRjJI0tZLHspehOOBbelKuOeZs/kczcYCo6u+BDNMuKexXCoRnmftLRlMXd5g7KMI5g8Nne9Bv20btcgJiesMr89DSe1rkMViK5xezR+MxOTwZAjm2XYvV9Pm0wa2TB5GaLAOtjqjp4pGc1dZobgaPKJZOFxQiGFSuyAoSBZ5X0Revtj/+bNMuI5bo0we4fXLONF6vAN3ZJaMTDpBwwG5CaYHnPrS9ixr7lofQga4CIKnovveU3YvzSEexAtbWEche6skLqhkG0SoUwgIf2AXZd/nHsUu3ulMFCBr8K+RlKHagxhLJpdPMtGczeUAsUQUf9dtAUA0CCMaiwEulERQYKmVMwyQfnLm1uDtWFnhKpGfTJkaQQYmDRPjV+ce66e8Ce1IiOIOolZxq/aQocwJoUR7oZQFPI+r7Ce8GKaNhiAucvr8dqqbdLtQRpvqZhlgtAS7vwI1QihkDJzSpbJnbO8WUZ8efgJd9VLpqrCX3PnKpCsFB2xwcfqa7snt/ZuHKqGgtHU0obhN03HM+9tLHZXAOQ/zYs9EOhLf5nvW05mauBJY3LnIGRfHUGy7GCrjs1dPlmH/X4LElYyoXywpU2aOMwvWgbwt92rNPoKwSxj9/8f89dj7fb91rpkMXHuhpKhofEgGAN+/eLyYncFQP4BT1NALtq4GweavYJNM7rPV9DYlJrmvmTTHuyXHK/oCJaRj3OXa+5Bwkp2rn7y3BJ5hkmFQ9XvS2HbXn/fjMfmbh3urdOWOOtklyrXDe4rwreFPH4mPZPPvZ3S1NJWNs41GcXoe9pmmcamFlzyu9fw7b97k1nZBLUcZP9Nquv7m1tx5I/+ixcXbwksK7e5MzS1tOGie17FLc8u9mzXEe6uCTS49foOVb9QSK6M/RmgSDng90K98Lev+LYt2tydd4hrXTLRMrr7Fn0mJiIaQkRziGgpES0mouut9b2JaAYRrbT+9rLWExHdQ0SriGghEZ2U8jGUPPV7mnD0j1/AQ6+tKXZXlOgIorTnfZS1lcYEyUBeoL2z1jvqVtehGqS5J9X3TbsOoLk1i1+9sCxyHaqu6Nnc5dd+y+6m3PYAcdXq45/gFQc7EZvs5eEs+5zznYqw2T5dq+Vt8uYfn1OgirkPA5H7DJWCWaYVwPcZY6MBTARwHREdA2AqgFmMsVEAZlnLAHAhgFHWvykA7ku812XGhl0HAADPLdxc5J7oUSrRARWOWSbdrwZV7UFfLMHRMkn1PXcyxCyUMqT2dahj9f/8erDiYWvu4kxMD79RF7iv3Y4Mvi5HuLvi6OOFQn7z7JEY2a+rtE235u4lijJTKh/ogcKdMbaZMfaO9bsRwFIAgwBMBvCIVewRAJdavycDeJTleAtATyIamHjPywjH818qV71MyDgO1WTrXbJpj0sDlk/arFdXa4CwTcKcxRjDuXe+DEBPw/avR778t7fXYcaSrYH786GQOtZn8dj9Bh/xL0B7rlppegMLOz+8LicM7uHrUCVFVE5cPDMxeUsk2p5IKJs7EdUAOBHAPACHMcY2A7kXAID+VrFBANZzu22w1ol1TSGiWiKqbWhQD3sudxzHYBnL9mK8lzIpRctcdt8buHfuaqkjNSxBZpnL7nszdhu8pqoz2MjvZeVnatqlOUjMLxRS3M63yeN3HfnVG3bmvnL5kuIpvvbRWo3e8v0il0O1IkP5GZ8CTC6EYLu8iKpESU7WQURdAfwLwHcYY3tURSXrPMfLGLufMTaeMTa+X79+ut0oS9ISUkkTJp1pIbB9YEmbZVosbbtFYt8NS9QshaHa4G3SCWruNrqH4DcTky5+Nne/VMA2FQlIQV5zz1D+BRjoUI3VdGFHpIpoCXciqkJOsD/OGPu3tXqrbW6x/tZb6zcAGMLtPhjApmS6W94UZ7Ri+Cx6xZ5BxiYfLZNsvbYwkdl3bbTNMoUQ7lwbUW3uYP4CWVfpsJWURRv3YPayes92j9lH2O5rcw9YV1VBmFDTG3d/dqxv3xZInOI2BKCSi5bJcD4D/vr5Dz6iwDK6kM/vNNCJliEADwFYyhi7k9v0LICrrd9XA3iGW3+VFTUzEcBu23xzqGI/FMUIJxx+0/P4/j/fj11PMc0ySUfL2M/zwZaIgpKjIJq7yywTR3MPjlZREaR57m9uVdbbEmJkqdvmDowZ0gM9OlU56xoaD2Lb3nxaisusCbj9qKzgNfd8Pnr+a0LqUI0ggv3Opze3TPFt7qcD+CKAc4joPevfRQBuB3AeEa0EcJ61DADPA/gQwCoADwD4RvLdLi+KPYz+3+/EH3FaDJNN2uasvOYuES7aoZDJjrCavnAz9jS5Q/r4JqLmE2eKI9J9P2UChFF3Tvjm2nRz14wVkdpnYCAil2nl5J/PxPjbZqp35Kjw2NxzjfLXT2uCbO0WfV6GBfwoDpxDlTH2Gvy7NElSngG4Lma/2hX2RS51h6rq3i6O5m63nZJwb/W3uetHyyTXt9UNe3Hd397BeccchgeuGp9vI+wLxKdL/jb3ZI6hX9cOyu1rtu1zfp8+sg9eX7Xdt19iMrGodn4gty/vUOUdwi4HtY+TmJK6xIHRM8liRqgWAPtGKnWHqo1M45BNXJw2jlkmpfQDjnMyxkstjM29bts+zK/b4bvdjt7ZaEWM2CRhlmIMvseZlNIh1qPq9q2Tj+P2U385MZYzacRxJvtp7jxamnuIc1Vsz1Wg5m6Ij32jlolsl1KMrmcytnBP2Swj2cav8wv9A8L17azfzAUA1N1+sbKcWGN4xd3PcRnP5q7TLl+XyrRVEeCkdGnuYCCK7m8gctu3eYeqq03F/kmQywqZfL1+GOFeAOyHs7xzyxS+zYqUHdFxNEGboEFMQC6UU+dB9iuThF2fOf95SeyLknnNKX7wmrRcc3fXk4kh3EVywl0dfmlDhNCaje7pNCl/2wH2zVvyNvcYW9PAjl5TmSXeW78Lj4T8OW0AACAASURBVM9bG6n+bz0hTxh2oLnNyZcCqD+vdTT3Nsakybp0Ca25S7p0wd2veK4gA0Pdtn144NU1kfvGs6+5FZv3NAUXRP6rDAjW3LOMxTbL8FRk8nfzaUf0AZAbxarnRNd/DsRoGJKsSxMj3AtI2djcrb+ruGHeshF9qfeDgs0yl/7hddz89KJY7Yha3JUPvIWv/XWBs6yS3zo297Ysw6Nv5l9AO/c147w7X8bqBvcw+tUN+8RdrTbUQu3Dhr1YtHG3skx940GptnrVn99W7heGFxdvxem3z9Yq684wGWBzR+6+O//YAYH1fvuckZ51ooacIeIULoYJNb3RvWOV9CWTIQqtYetGWpXMCFVDdOwbqdRlu/jw2/lMctsK3Zu8WSbJiBQZYu3vWflNdNAR7uJLfcaSrVhZvxd/nLvatf7b1peEeB3E/cVJw8+542Vc8rv83K1+PZKtF2PTk+Sqh972NanxYZWyU+iNliH06FSFX152vLLN4f26BPaLt7m3ZRkqK8jXryKGf8ZxqIpZIdPGCPcCYN+8pa65q3pXnDj33N+0BwrFuSw6favf454DVjaVnLoN9/Jt05fgtZXbPEI+CPlx6oubsOfp7bod/qM+ARw/qIdVr3/FYpoAz1yowv4dKivkjXFkOLNMSxtz7P+yfuScsb7dK2mMcC8AjPsE3L73IO6ZtTJV52pjUwtqpk7HE2+vC7VfseLc73xpOWqmTvesL1ROngMtbaiZOh0/fPqDVOq3o2Rswpq4RLNMfeNBfOGheZjymDyBlu9I1MRzpwRz4s9mSNcTEU47og86VmV8NHf3127G+Yrzmqj4r6cOlcEirYJzqDa3Zp19ZP0QT0/UQWSyuoxZph1g3zSMATc8tRB3zliBt9f4xzvHZavl1Hrw1Q9D7edoSZK7Lk35es/sVfINtuZeoC+ev81bh9nLglPfxkWWblaFaHK3o0aWb20M27B7sQCndfcB+deFHZ6YzcpfRuLXrn2qxCkCa+t2uMx2Us1dgDfLNLdlUW0Jd/nLL7oElu3KryuF9AOGmOQ1d6CxKT0bZx77MzPcXqrizkOWotVQfMjttnSsMjVTp+Oxt6JFzfB85eFw6WSjkHW0Ub3youauuq479jX7TplXSkZBQi5qJcvkBr87Z6zAnTNWeKbDE30El//xTdfk3tUSzV08zRluENPB1jZUV2Q8E5A4+0a43f0TkIWvKw5GuBcA+1ozxpwHVZw8IEny6Q7CPc7FdgmI7ecnyNbrmOigLFWYoI0GIV7HNsFkwXPSz2bgNJ+IFanw0utCKthRK37XN2e+tMpaN8M+SQ7+LVz4pY5ZJmOlMmhqacP6HQccbV/WC49DNUwopHR6Di4NgnZN0TDCvQDYN2+WMbQJN2uahJXVqhs3ScG/92CrNEpDFGJhX1J+tua/zVuHmqnT0dQSf3KOJLDvBz+HqngYopnZduKG9dsUw+Yu8oOPHYVvnHUEenepzpllmNrsJl77s470zv3we8usd/unjvcxhUhCIbMMP30uN/Zg7Y5cCKrUoao+HCl+R1PoVNpGuBcAx+aO/INdWQjhHlIg21//sp4lGS1z3C0vYuytXkebqMDlc8vEa/v3s1cCyE2E/dVH0je7BNEaINy95d3SPepIzc8/OC/SfmEIuq1PGtoLN1xwNIjIKbt9b/BMUPapOmVEHzzIJVUDgGnW3MQ9O1drCVA75a89rmDvwVYQ5PdZhijyBNlBl9c4VNsBtkbQlmXOg52qWcb6G9osoxDgfITHvA+3o2bqdKzfsT9iD+VD/8X2k5h69jt/fxebrNGmX320FjOXpu8wDUInNn7kD5/H3TNzKXJFh6rtQAx7Wj70GSSVJCcM7qncPrhXJ+e3/XLbtOuAX3FPtAzg/+zkvgaC+1iRydnXq6wc70olIsHH1DunqnGolj38LdOmYXN/c/X2eNOp2X/DOlSVoZD5jU/W5qbITTrix9u+9dDFkO7/eS8/CZgYaRGH9//3fHSuDo7MkGGH8/maZZBTAu6emfviEI/f0eQ90S/hz1MYAaPz9Takd2fl9gE9Ojq/7UdgqzAOgCfvyM/jZ9Ls1blKut7jUKWctm6nHCZLO5fJ9oyQdEznDJdKDikj3AsAr0Hb2oHfQ7Vwwy5c+cBb+NULyyK3l48RTu4m42tKK/48rs09CiM0RjTKqKqkUF9fJw7Na7S25k6Um1GoZup0zF3unbbOpk1Q3Xf5DF6KYr5K2jTQUeHQvPGCo1FVkd9uC83mNv+XrhMtw/XTb07VXl2qtfoovhz4L2uRpMMV+RekMcu0A/h7xnGG+egAtv1xZf1e6XYd2uSKXSCOEJXcdLx8zU+iEbprAe27l8NGy0Thjk+PibRfVUXGEe69u1RjUM9OyvJdO+QTsNpmlQzlXuYA8Mgbdc52UciIJvb6RrmmG/YL583V2xO/hn4mp0e/MgFfP+sI17r8wKRgh6qOWaZnp6rA2HKxLiA3SpUgP39RrKe6p9REy5QRr6xowLvrvBP18hq0fQMlPDubCz4pUhjUxTmNw7otxeng4uLR3EPEuYvsPtCi9eUiTg2nS2WGHKc4wev0/NfXT3UtD+ieN0e0cGYZWxjOWd7gbOeF+/2vrMa1j/qMRBWWw2rujQdbXWGEw/tG+4rhkZkTe3epxpmSKBdbiVf5IGSX0E+4V1boiTNx98tOGgRArkSIacPiOFRNnHsZc9Wf38Yn7/VO1OtKX+po1eq7JM6NYD/kYYWiqrjjUEU+Fe9t05eivlEvxSuQs9Er84gIcsE+B2GF1q79zRjz05ccm7WKak2BIJKL9rCEO/n7C2wmjT7M+W1P7UZE0mPjp377v+fd5jnevCOeyzhRRUcP6Ib/Xn9G5P0drMMeN6yXs+rEIXInq2PeU/Vb6lBVNa8XLcNzzUeG566FT26ZpPD0zZhlyh9eI7U1PD8Zl0TIoV136FBIifNKrBNw2yHFKeFkNLdmMW3hJnzmT28qR5H6fWmE9R3YA13+MMcnrQFHnKglft/HrjnFtY0IuHzcYADAry47Af265ecXte+BDMm1VlWK3wqFc08m3E+u6eVZJ+PPXzoZHauiOYhtBvXshB9ffAyunDAU5xzdHwBwxqi+uOfKE6Xl7fuIPwfnH3OYq4xsekdVCKnULCPc0eI1t/vhN0LVFQqp83wqAxP8+5U0ZiamAsBfUMcenqJDPT+CMTmzjF1X3fb9mDgiX7BFIx3vGb+a7UREqMLxxJrsB9vPluyXmtfWBFWf+x8Z2RcnDe2pHWsuIy8kCEcN6ObaRgD+75PHY8yQnrhs3GAs3pTPt97i2NzJ4ywF1NEjvDNQjP6RCfcJw3tjfp3XVMhzwuAeODzAZwCo749VP78QQM408otPHY/1O/bj7pkrcPPFo9Glg1zMyLJ+jhnSEy8tyYerOg5Vbj/ZC/n2T6lTAfNUhfhai3N/eCbrKLBZxgj3AuCOlrHn7UxPujv540PvqTCZcL//Pn+981tnQI1KWPHc8dJy/PyTx6Nu2z4s29LITSwuL3/pH16X9lHH1/DXr+Y07TBmJRFbyNiyokNlxsnrQkSorszgixOHAQAquVS19j1AFH7yb1GubeVs5rxT1qaThjb+h8+dFK4TAmcf1c9j7x7SuzNW/vwi5X6y+HJv/nTrRcgduEzg9rTCIHUcqjLh7id3Peu1FHe9J89Ey7QDeOFka5NB8ifOdefTHYRBVdyvruaE5rUEgMfn5VIUn3vny/jaXxc4j8h9c1fjlRUNrrIq+7Jq29EDuuFvX82bUII0szs+PQa/vWKsdJsj3K06Xvrumc42sVZ7wAyQjw65d+5qqeauQuzvmm35LyFZds1O1Wr9bcyQnoGx6UFEDRe05XWrS7i7y/C+HhtRc7918rE4/5gBVrngvlRX5sp061DpuhfkfQw/E5ONuBfB/W4w0TLtgvwlDRK8SZhrHIdqSAdbfhSqLBZSvk/YwVYPS7RLEdkLUJwOTjWqUfVSG1/TC6eN7Oss+8VM2wzp3RmfGHO4dJttYrK1ymF9umD0wO4AvFoZr9naL8S2LMMrK7cp2xcRBVvQKGGV5v7EtRPx2DUTQrUvI6rbIj+NYv4e8k3UpQiFvOrUmlC5mmwn+tihPZ17we82KNeJOgAj3AuCK849sskkfHth21BGsvisTzcZl39/VA+z6mNCFOY6NlUiQr9uHXDLx4+R94Wrw9bQxfcqr7nzXxbTrbwouogv3qB0CpUV/sc3emA3dO8YLRRU1SddHLOMy1kvFHKiZbz7AcARwiA0uUPVTRibuz16VeiOEv+Uv3JHbloYm3tI/jF/Pe6duwpzf3C29j62Jsnnck/ToRp1zlZVcb+6/vJ6HT527IDYkRYynnh7ve821VeJyiwjvhQks7a5sJ+/+TefCwA4Y1Q/dO+Ue2wmjz0cz7y3ySV4bK1S7AMvUHSc0H6I77QXF/sL919ddoLy0z+paI2grx8/HLMM9zYWBV7eLCPX3H/+SX1Hqk2VNYrW3VbygrbYWr/R3ENyw78Wom77/lAxxTLBGBTJEuetrjuIae32fWjkBiJFsbm/t34XfvXC8vCdjIlfX9dt3+87+w+QD0+0CRsNMbJ/V/TvlhuQdES/rgDc16rSR7jzWUD5ySXCEqa/XTtWpq4dAsEvSP/9vKGQ4ssrP2tVfh1fRDwf0qMVVtpmmShnRicCTVcyGJt7iWHfS2HMETLB6HcDJGpzD6jso7+ei8vve5NrWxHnrqhn/U6v3be+sQkvLt4S2NewbNi5H3OW1ftGJJz56zm48oG3pNvGD+uFYw/v4VoXFOeu2mrvKnP2ibHqvM1938Hos3GFEe69u1Sr7eEhpYuvuSGimJINYuKPj0/mxR+HO+Zd6ItGV+zZmqoUJitXnVqlZPvl9uzbNT/GgX85pP3eNWaZkFRVZNDcmkVTS5tv/K6IXHNPuGMcYcwy/DycarOM/1bZi+5zD8zDqvq9mPX9j/rut277fvTuqpfsCQAWbdyNzz84D7sPtGDO/5ylvZ+NTphcmO2OVqxllskXipOdMozzsm/XamzZHT3UU8RPWYgqpGTRMqKGnlc45I1E+TKxrwX/YtetJspj+6+vn4Y3P9yeappvGUa4h6SDLdxDRInIHoo0Mx1mQw6UenfdTpw4tJd6EJNi/wPNbWCMgbH8p7YdojdnmX+2wzN/PUeac8SPS373mvN7xpJkvgqi2osBudZfYdkoxAFUfJz7gRhOaFuz/cSYw/Hs+5uUZXt21stvrovfoLCoA31kce7k0tzz85q6hb5/5IxMz/Z7Mejmoon7hTOkd2dpuKnJ515i2NEHBxTa1z9q16Nm6nRngEkUzT3OZc9H5OhJ9/VWCgFlecWmAy1tuOie1zDih887phhd05AYv66LmHNFB9nDFCyY/LfLzDK2bV10+CaluQ/tkxMSH9V4KXasqgg9VP+4Qd1x1lHyum3Hp51awCaqRirLHeSxuUvCc+OaZeyXVBWvuQfvFpoYt1YiBAp3IvozEdUT0SJuXW8imkFEK62/vaz1RET3ENEqIlpIRPGGvpUgdtSDyub+1IINAPLaq9Tm7hfnHtD+O+t2YlV9o7IMC2GW4clKtKT8Nv/KWtsYlm7eAwD4f48tcG3be7Bw85YGHq/kuOLMZZtPHObVJEUtly+zP4bN/XMThuLhL5+MT1mZDFVUV2RCa9XTvnWGZxo7GzvKRxzUFd0sI9HchQmkHYdqQB0qxCJt1nFoa+4cOs+Uci7i0C1GR+foHgZwgbBuKoBZjLFRAGZZywBwIYBR1r8pAO5Lppulgy3clREPwhWUau4R2//UvW/g3DtfUZaxI8vCj1BV3JSKqnbu958DM6zzME4kSRBRZJCOzV2muatylO+PaZY566j+WrbmqgpS91+zzfFWhkc71YSoqcc1y7S6BjFx/SNumj1OUvFJ2LSiZQQczZ37mlKfp4hfJkHbi625M8ZeASDOpzYZwCPW70cAXMqtf5TleAtATyIamFRnS4EOlqf9QLO+zV32Ji+EQzXJlL+qbX6TRwDAQ6+t0Y5KAICp//pAu2xYojxMWtEyXKEbLzgapwzvjY/6mDaAeNdedgx2/hpvWQrlgOX3s3nkKxNwy8ePBQDf+X+jyihZ4jBRWGclDtWOVRUY0juX6EwMw9R56dl1RjEnJfnYpu1ejWpzP4wxthkArL+2EW4QAH7kyQZrnQcimkJEtURU29AQze5aDOzPeNXUYCIyIRukVcd5q8vqPuEnL+LzD8pDBB1UJvcYEkkneZXNywobfNxgg6QdWI5Zhqu3pm8XPPn/TnXNvGTzy8vCD7gRkV1b1SjUKNEk/B4fPbIfunTIXT871USVIFEja+4yE5bgOLW3+DWhZZYRlk+3Ug5cOjbYtKVq2w/lo1JAu0zSDlXZaZAeDmPsfsbYeMbY+H799CMmSgWdfFn2yQgX5+7d8tSCDZjiMxuPvG/eOvY05ScE9kOmJeW35X+/9N0zcfNFo/3rEdrvFGIiaZU2FXe2vUiau2Inmeau4jPjh4TvgIAs0kY1nF7tUNXruB0X7uShz7jNPVH9FoFZIUnuUAU4c43oUNVod1T/rqi7/WKMr+nN7ad3DKGUnIAq0x5gFlW4b7XNLdZfO95tAwD+Dh4MQB2vVWYwx+SR7ghVnv/55/uuHNdB6AhBWfu60+xliHDtmSN8S27e446tDpOaoDLFWOCkn6WwD2fch/l75x2Jow7r5lmvOmfRzDLuZUe4c34Evs2ol0xulslvz2aZ78C6vHAXTESSvojnPe1480I6TVVEFe7PArja+n01gGe49VdZUTMTAey2zTftBfvChcm4GF6QxkOnb7Ii6kFM+d9BAvj022e7lmUmCj/iPHiBUxdGMMuo9ojS1zhy5duTRklfECrNPYpDVWyjQ0Xu5czn7ufj9pONc89vb82yRMwyOvsk4Xj27qfeM22be+BTR0RPADgLQF8i2gDgFgC3A/gHEV0DYB2AT1vFnwdwEYBVAPYD+HIKfS4u1t0mCscHX/2QK+LeKBekQaGQ4S59/Z4m9OpSjaqKjNZXRZjwTHe/wgu1bh31hXsczT1w7EACWixP3iyjX3FFhpCNkThMhsphncSnf5WV/7zFR3OP2oS93/Z9+WgrP4eqnxD3RssEd0blowgi7pXj9y96+gHG2JU+myZJyjIA18XtVCnjaO6cJNnT1ILbpi/13UdmA09Sc9/f3IoJ/zcLV5w8BLdfdoLvtHQ88lGzub+ym44vHl6466eVjaO5B32wnDK8t7pASGShkHr7JCvcVfHacaaJs7ETbfGDl9wDieJp7jxEhI+PORzPWaNvHZu7UM4x12g0LZaRjUpOVNCm+VkeAjNCNSLfeuJd3DNrJQDghJ+8pCyrEqQiYe+LcT+bga/99R0AcOzyOmYZeTv++/HHEEVzH6QxRycQLte2SJAf4xtnjQxdp0oTjGYS8N927RnD8eoNZ4euM6rNXbf7lRUZvDH1HNz52THcvrzNPUHhDnef/RyqTh2eya412g15/7raDuNPDXKomvQDpcHGXQewZNMelwC5c8YKrX3DmUD0NRIg90krDuHXcQfI++RfPo5ZpnvHKjx93WlaZeNo7kFfLFGiOlR2/LDRMoA6l83NFx+jnYyOR/ZC/PLpNQCS0dwB4PCendChMu8Y56s9fnD3SHXKLkeGyNXnrI+GHkYHEpuRau4JClq170ruX0gDkzhME9tJOKyPOwGQOP2bDFnYpN8NECbc76HX1vi0F01zt1ct29KI3ftbhPKc5h7yruxQlXFyoAcRx+YedlpBHVTvi3z6Af36goRtlMOX2ZDtgUdpCRC72itOHqIdL+6pQ9K5DPlEvAT0IwyhNXfud5iJ7dN2mAZhNPeQiA+7KvGVLWvCaO4qZ6govO7y+XII61D9sGGvZ91HfzPHd9+wD0cYgR0vWiZ5VOfSFkJhtL7AT/UI0lgcVOSqT9G3ONrqTuvlf0S/rpGdtrJrTeR+AfqFPMqyRcqWddttj7Rr4T7pjrn47pPvJVpnmDe3kwbAx6G6fa932L5K+RQFjd8ADj3hnv9998yc74CfwWiXoLnzdYZJJwCE0/Qro07rg3T8WKrrEUlzDxAsUQSPHc0ibS9lOVbfGD1fvKxvYsoEf7OMfMCd7EWj41BNUs3WvQ+LnlumnFndsA9Pv7sx0TrDCBBVjpfXV2/DuNtmYqYwOEk1G5JoU/YTBFEHMd389CJJSbt8/ndYARxG048XLZOGWca/zih9DXrRRTLLKK6H6twnIVzizAcrN8u4be5+ce5xLnVY/cHlTw3lUC3uF0K7Fu5pEObi2rZvmaPvvfW7AADz69w52cSir6/a5vwWZm7zJnAid7sqwpqn+TrDau5hzDJxYpDTEO5qzT33N3wopKrOCJq74pylrbnHmzJQvp5cZhnmWecuKyzLS7mW5A7V5NAdL2KiZcoQ+9rm/3ovtn1hRYEkLm/mpknTFV56I1SFdgP28ZstR4cwmnssh2oKZhnVg+rEuYc4H0GHF0XZU2nuaWuP8aYM9AlvdJllcn/Fkn73SRo2d14IJ5haJnVMtEyKOJq7QuqIm8SbhxfC4heAX7V6Zhn38sGAaQP9pljTIZxDNb049yjo2NzDHF9wtEx4oaX62okTCtldY2Rx907RRYjeqFO55v7oNRPw1IKN6M/ldtdF9sLj/U1pwt+ixuZeYoQRICqbu31h/TR3ezvfHhPkr1eTz+2kM0JVPI4vPDRPWV7H1ONHGAEzc6l+gjSRqF38+5SJGDOkp3Sb6nrbMj2MJhhU1s8mr9pLmVtGp1MS7v/iODx//RmB5W6++JiILfiNhGY4emA+OZp9TcXTNrJ/N0y98GiPoJaZOnRuv1dXbgsuBL2ACt3bMG3N3gh3gYbGg/jH/PW+2zeFmE2+uS2Lh15bI51dyL6wouzwau753x7N3Uea6byAxF0XrN2pLF8ozT0OUV9AE0f0wegB3myLgKbmHsJPEBwK6bfef8foKX/9+3H+sQMwuJd3Umebuz87Fnd+ZkyopHC6ffvchKGozBDOGNWXSz9QbCNHeIrsTzVmGZEpj9Xi3XW7cOaR/TCgh97AGz8efWMt3q4TJ7GykGTEA9Q2eLGsnwzXc6iGE4QtOgnsfYgaAXPK8N6Yt8bn/KXEaUf0wRur83nvlefJOqwwXyZBZX2H2Uc0y6QlYC49MdrAJR6/+4KIcMLgHmCMTxwWvZ2wu37n3FFOeHAU9EMhjUO1oGzeldPM40Re1Fpa8J4mfzueX0y62Cq/WdTIZX3ce7BVaZaxtf2wx8dnvQxL1Mkcxg7pidEDow1tTwrVebKvRzibe7R+qDRX1SCmpNIPpIHsXORDHwkMzMkhH2Uy6zCcXNPL+f2dc4/0LSfeDr94fim+aJk027IMC9buxBurcyaekk/5e6ghe5jtsMWwKJU+x+Yub9++MXiBvuuAOLDIve+2vQdx3C0vKrW11ixDdYZCxwlv2+s/CXaaiIcyrE9ndKqqwLItjQVpX3Wa7I+ZME7gqC+6qA5VdZ7y4gr+oFmuGANarPjfOCGyOjxx7USMvPm/off70ys5pWfvwVZMumMutu7xn08YCDcIMi5GuAvYwtU2bSxYuxOX3fdGrLpUeLVxsY7878uFfnSsykinXVM1u3DDLjS3ZbFm277AvgXx6g1n44xfzQksF9lc7/M8F3L4uMp/0WYJnjBKZVRNWrUbH+f+wFXjsXZ7/tqWsOKuzlgJQpblNXfV10kQOuYP3S+Dl5ZsxRurtmFVw15cOWGos/5bf3snULAD7vmETeKwiEQNi7Mdh/bfhhjDq1XC3S9MUuw3v7ynyT1gZGCPTk6OD10u/+ObocqrGNLb3+HmQuNayOzrBPI8AIylK9zF9sSBYzz2PRJmxG4aZhm+/fOOOUy/ziILftmLzrlVLM09/wItnbfU5x7MmWGaOMVqznJJjilJl3t3qc5vNjb3aKiciowx/KN2vXR0nb1fq/XNHSu/uGKbPZP8PxdscOW/sW9uOyTQ7zA27TqAJZv3RO5bIdHR3CeEmEhDV/vt2Vl/khAb1TgDEfteCSN4omruqvs5bZNFWqjORYZyz4+d3iDsqGieKHv+9oqx+NHF/pPAA0C9hqYu0qdL+Lj8qLRb4a4K3ZtftxM3PLUQP3l2sWeb/SzbN1WscTGKffnoEz7/jS1MWgMcnw2N4W+sYqFjnqr2eYnKnv9CanF9uvo/jI6zL0R/omprzYpoJb9zBxTfrq4iyB/AGEOrY3MvrKiaPHYQvnqG/yTwQHA8u+z4enUJr3BEpV0K92yWKTUdezTatr0HMfn3r+Hpdzc420RzicymrYvq4vslXOK7rUoJoHrYSw0dzb1DlfdWlD0c933hpND55MNg20Q/M34wnpwyEeOG9fIta0clhXnZNCoiqKLCT6IhohagxUXuXM4P4mMs2gu0UPAK2hUnD9HaJ8pkLFFpdzb311dtw+cfnIfHv3qKbxnbjscAvL9hN7775Ps475gB2Lmv2RHqLdksNuzcj5uf/iByX1R2fzFunDGWC//i9mluy/p+gTQHpAsoJXT8HyoBxXPs4T20s/pFEQfnjO6PU0b0xudOGRY4QCeKWUZmCnzo6vHKL4QgqiuTH6FaCIKmAGTg/BpxzDIpnQT+GezeSU8j5x2qaVP2mns2y/DKigas37EfADB7WT0AuCas/uoj853fK7Y24k1rkAovcy6/7w2c8as5jjbW2sbwvSff9zgxw1C3fb/vNlG4z15WjzXb9rk0+tnL6nH7f5dJ938gRtx5GL5+1hGuGOCXf3CWa/tDV4/HRccPUNahY9rqIBFQ/DP5s0uPw8zvnQlAX5hGsahliDDlzCO0Rl4ebY1qPW1kX+36ZTl8Jo0+DGN90h/ooBLuKoqdkjZo9C9jzPF9xcnzb1PTRzMAQBP+WsqEtuzs+44OuwAADexJREFUFlK4l73mfswtL6CpJYuOVRks+9mFjjawlHM2zlyaE/iPvVmHHz+Tt7Ov35kXvnbcNO9QTTqZ0GHdOzjhUqLmfc0jtZ7y33j8Hd+65sq88ylw6og+WLRxN4Bc/4f16eLaPmn0YTjm8O54/oMtvnVo2dxlwp2AltbcvsP7dMHI/jlhWioDc04c2gsLfnRuKK07yS+uowd0w7ItjcqXXYmcKiltEtMkf6tkGfD4vHUA4jpUc/u+8J0zY6XREOG/wjpX6wntTprlkqCshTtjDE0tuYfF/qvSRnjBDgAfNvjHerdmmTQnTBxauZs5zhdBIanMEDpa2kZU7UnncfIzy9hfOLxNXle4txTAdBXWnBJGuMz9n7OQZQzn3PGydPsT107Emu3xxysUC/sreVifzhg9oDteWJxXEHIjVPNKVxIO1Y4Ja837mvPPsExoy2RR0n1QUdbCffs+96jJ8bfN8B1JecdLy0PV/dBra5RmlSgEpdUtRTIZcj4lfXOBBFh2dTR3uVmGHMdxR07465plmkr4fD/ylQm4OmBy9Zq+ua+kuz47Bkce5k1s1qtLNXpxcdNyFFp9YC/TxU7Xe91ZIx1zqn2nZAguNb4UHap7D+aVP1XEEo8xy2hi29ltVEPkfzd7Vai6X1ZMfB2VJknkzaCenbBx14HE20qKnOaecX6rOKx7B3xkZD/8650NrvXHHd4jsB0+Jr0yQ2jNspxZJobmHidNcdoM6K6flO6TJw5OsSfFo0uHStTdfjEAYM5yS7g7WSDdNvk4wl3XNDX7+x8NNZqaN8vofll008iRnxRl7VDdsLNwQvGSEwbGrkP2Sf7Pr50au940qciQE77lpzHz+TKGcqNWjz28O966aRLOPLJfYDu8eeO6s0c6v21Tlltz1+x8CCaPzWU5PCXEYKo4qDS4314xFk9/47RE2lGGQpaQMuyZLo8IH1i+HqAwYxtG9OuKkf27apfnhbvsnpT1+MxRwc9CUhjhrsmPL/FOSjBxRHxBEEaDS5LxivjtL51W4/xuzTL0sz6f/R4wO+68a4dKXHf2Ec56xqCdNrlP15x5oboi49LMmyWae9gH/Ztnj8R3FZn+AODUI/qg7vaLMaKf/sMdh84dKnDR8QPwpy+O82ybPHYQThzqf33CUELyW0mnqpwCYYc8iv0udmSPjJ3785aCUf29ZjNZ+GbUxHFRKGvhftm4QbFCyMJg24Q7VmWw4EfnYumtF+DvU+Ra9/fOywuS8yW5PvpyWmomQ/jdlScm3NtgVLban3ziWEw6uj8AYMe+ZvSz+uvnM+jfvSN+eNHRePjLE1BZkcGTUyYCCBeK2M36OvjmOXmtncCZZTibvBixE8T3zz8S1587KtQ+adOnSzXu/fw4fOxYdRjpocL/XnIMvj1pFM4dnXteSlGYi9hBHD//5HE4bpDb9DisT2ftsRtpUdbCvX+3jtKH4+yj/D997vv8Sco6u3WoxMfHHO5Z371jFSYd3R8PXX0y+nTt4HjH77EE85UThmL+zefiIyP74lMn5Scy6NXZK0TFT7iPjzncqcePV284W7ndj6+fdYRn3QNXjXe9YGTcdNFojBvWC6cd0Qf9ra+L/c3+ET5TzjzCSSTW1ceuePunjvfdn4hQd/vF+PakUaiqzD3Y1ZUZJ+6fjzLgX546lJKgsH0LheqTKrlbKZ2XHp2r8L3zjnS+yviuRZknlVeQe2gOMArCb05Z2xS54rYLnXVhFZA0KGuHKgB8+fQa7D7QgnlrtuPddbm867//3ElOml4+7/cXJg7FhccPxPv/ez7G3PqStL6HvzIB44b1wg/OPwpD+3TGpDvmok+XDshkCA996WRP+U+MORyf4F4Gf7VGxo4e2B1VFYSbLjoaHaoyuH7SKIy7bSYA4GsfPQI/fW6Jp56zj+qH5z/YjD5dOmDqvxdi295m/P5zJ+KMUf3Qo1MVfnTxaGdw1vu3nI/Tb5+NlrYsDrZmcddnx+C7T77v1PeXL5+MlVsbMeXMI9CrcxXOPqo/Hp+3Dl86rQY1fbtgU4ATd2T/rvjX13N2X3vwx459ejndu1RXuvazuWLCUDxZux7vrtuFWycfiwwRnpzvTeD2ldOHY8+BVnz1jBF47v3NWL610RWNUFWRweiB3V1jGWQ8+83TsWhjaSVXe+WGs6Xx3WlRVZHBitsuxPZ9B3HqL2YDAEb174qV9XsL1oco8M7w//ukv1LgR79u+TElg3t1itUX27xyRP+uqNu2z5OJ9Sgrkokfq9FH8WX83+vPwNuFmGGMMVb0f+PGjWNxuW/uKjbsxmls2I3TWDabdda/vqqBffRXs9mwG6ex389e6ax/d91O9uP/fMCG3TiN3fz0QjZ94SbXfmnwj/nr2Nf/WssYY2zYjdPYuJ/N8C37tcdq2bAbp7FV9Y2u9dMXbmJfePAt6T7ZbJb9590N7Pon3gnsS1tblt07ZxWrrdvBaut2sBcWbWbH/Pi/bNiN0zxlW9uybNiN09i1j8wPrNdm5pItrLGpxbP+nbU72EW/fYXtP9iqVU9DYxN7bWWDZ/2eA83siXlr2aQ75rLFG3ezFxZtZowxtm77PvalP89jP3tusXZf2wM/e24xu/Ol5coyf397Lbvhn++zhsYm9uoK7zktJX79wjI27MZp7DcvLgu1X0trG5v8+9fYKyvq2dpt+9i8D7cn0p+Xl9ezHXsPsmw29ywcefPz7GN3vcxmL9vqKTtzyRa2a19zIu0GAaCW+chVYrHSHsohogsA/BZABYAHGWO3q8qPHz+e1dZ6R2iGoaUti4UbdmPM4B6esKT6PU24d+5qXPOR4a7PVMYY6rbvx/C+hf+Eamg8iA5VGXTvKP9kbGxqQd22/Th+cHAYYVLsaWpBS2tWOjBn064D6N2luqCDMAyHLtksw/qd+0vCvCFSSs8CES1gjI2XbktauBNRBYAVAM4DsAHAfABXMsaW+O2ThHA3GAyGQw2VcE/DoToBwCrG2IeMsWYAfwcwOYV2DAaDweBDGsJ9EID13PIGa53BYDAYCkQawt0/Az9fiGgKEdUSUW1DQ2EyHBoMBsOhQhrCfQMAflqSwQA2iYUYY/czxsYzxsb361e4IbkGg8FwKJCGcJ8PYBQRDSeiagBXAHg2hXYMBoPB4EPig5gYY61E9E0ALyIXCvlnxph3JmqDwWAwpEYqI1QZY88DeD6Nug0Gg8EQTFnnljEYDAaDnFRGqIbuBFEDgLURd+8LYFuC3SkHzDEfGphjPjSIc8zDGGPSiJSSEO5xIKJavxFa7RVzzIcG5pgPDdI6ZmOWMRgMhnaIEe4Gg8HQDmkPwv3+YnegCJhjPjQwx3xokMoxl73N3WAwGAxe2oPmbjAYDAYBI9wNBoOhHVLWwp2ILiCi5US0ioimFrs/SUFEQ4hoDhEtJaLFRHS9tb43Ec0gopXW317WeiKie6zzsJCI1LOAlyhEVEFE7xLRNGt5OBHNs473SStXEYiog7W8ytpeU8x+R4WIehLRU0S0zLrWpx4C1/i71j29iIieIKKO7fE6E9GfiaieiBZx60JfWyK62iq/koiuDtOHshXu1oxPfwBwIYBjAFxJRMcUt1eJ0Qrg+4yx0QAmArjOOrapAGYxxkYBmGUtA7lzMMr6NwXAfYXvciJcD2Apt/xLAHdZx7sTwDXW+msA7GSMjQRwl1WuHPktgBcYY0cDGIPcsbfba0xEgwB8G8B4xthxyOWeugLt8zo/DOACYV2oa0tEvQHcAuAU5CZBusV+IWjhN7lqqf8DcCqAF7nlmwDcVOx+pXSszyA3beFyAAOtdQMBLLd+/wm5qQzt8k65cvmHXGroWQDOATANuXkBtgGoFK83cknpTrV+V1rlqNjHEPJ4uwNYI/a7nV9jeyKf3tZ1mwbgY+31OgOoAbAo6rUFcCWAP3HrXeWC/pWt5o5DZMYn61P0RADzABzGGNsMANbf/lax9nAu7gZwA4CstdwHwC7GWKu1zB+Tc7zW9t1W+XJiBIAGAH+xTFEPElEXtONrzBjbCOA3ANYB2IzcdVuA9n2decJe21jXvJyFu9aMT+UMEXUF8C8A32GM7VEVlawrm3NBRJcAqGeMLeBXS4oyjW3lQiWAkwDcxxg7EcA+5D/TZZT9MVsmhckAhgM4HEAX5EwSIu3pOuvgd5yxjr+chbvWjE/lChFVISfYH2eM/dtavZWIBlrbBwKot9aX+7k4HcAniKgOuQnVz0FOk+9JRHZaav6YnOO1tvcAsKOQHU6ADQA2MMbmWctPISfs2+s1BoBzAaxhjDUwxloA/BvAaWjf15kn7LWNdc3LWbi32xmfiIgAPARgKWPsTm7TswBsj/nVyNni7fVXWV73iQB2259/5QBj7CbG2GDGWA1y13E2Y+zzAOYAuNwqJh6vfR4ut8qXlUbHGNsCYD0RHWWtmgRgCdrpNbZYB2AiEXW27nH7mNvtdRYIe21fBHA+EfWyvnrOt9bpUWynQ0yHxUUAVgBYDeDmYvcnweP6CHKfXwsBvGf9uwg5e+MsACutv72t8oRc5NBqAB8gF41Q9OOIeOxnAZhm/R4B4G0AqwD8E0AHa31Ha3mVtX1Esfsd8VjHAqi1rvN/APRq79cYwE8BLAOwCMBjADq0x+sM4Ank/AotyGng10S5tgC+Yh3/KgBfDtMHk37AYDAY2iHlbJYxGAwGgw9GuBsMBkM7xAh3g8FgaIcY4W4wGAztECPcDQaDoR1ihLvBYDC0Q4xwNxgMhnbI/wcPABl8IXSS+gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.plot (list(range(len(rewards))), rewards)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Even though there is still some \"catastrophic forgetting\", this is the most stable algorithm we have achieved for the CartPole. The trend slowly but steadily goes up as the agent is trained on more episodes\n",
    "\n",
    "## The slower updating of the Target Network Weights with Tau altogether with bigger batch sizes seem to help prevent overestimates of certain actions in certain states, thus not forgetting was has already been learnt (catastrophic forgetting is quite frequent in Reinforcement Learning because the target keeps changing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "DoubleDQN.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}