1_5_contextual_bandits.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "In this part, We will see how to solve problems in which there are states, but they aren't determined by the previous states or actions. Additionally, we won't be considering delayed rewards. All of that comes in Part 2. This simplified way of posing the RL problem is referred to as the Contextual Bandit.\n",
        "\n",
        "This tutorial contains a simple example of how to build a policy-gradient based agent that can solve the contextual bandit problem.\n"
      ],
      "metadata": {
        "id": "lgdbNdgX3sdj"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NLlo3BBe3Ogw",
        "outputId": "29216a66-245b-4fe2-924a-4529cc14d663"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.10/dist-packages/tensorflow/python/compat/v2_compat.py:108: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "non-resource variables are not supported in the long term\n"
          ]
        }
      ],
      "source": [
        "import tensorflow as tf\n",
        "import tf_slim as slim\n",
        "import numpy as np\n",
        "import tensorflow.compat.v1 as tf\n",
        "tf.disable_v2_behavior()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here we define our contextual bandits. In this example, we are using three four-armed bandit. What this means is that each bandit has four arms that can be pulled. Each bandit has different success probabilities for each arm, and as such requires different actions to obtain the best result. The pullBandit function generates a random number from a normal distribution with a mean of 0. The lower the bandit number, the more likely a positive reward will be returned. We want our agent to learn to always choose the bandit-arm that will most often give a positive reward, depending on the Bandit presented."
      ],
      "metadata": {
        "id": "rw6jVmR_3q4w"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class contextual_bandit():\n",
        "    def __init__(self):\n",
        "        self.state = 0\n",
        "        #List out our bandits. Currently arms 4, 2, and 1 (respectively) are the most optimal.\n",
        "        self.bandits = np.array([[0.2,0,-0.0,-5],[0.1,-5,1,0.25],[-5,5,5,5]])\n",
        "        self.num_bandits = self.bandits.shape[0]\n",
        "        self.num_actions = self.bandits.shape[1]\n",
        "\n",
        "    def getBandit(self):\n",
        "        self.state = np.random.randint(0,len(self.bandits)) #Returns a random state for each episode.\n",
        "        return self.state\n",
        "\n",
        "    def pullArm(self,action):\n",
        "        #Get a random number.\n",
        "        bandit = self.bandits[self.state,action]\n",
        "        result = np.random.randn(1)\n",
        "        if result > bandit:\n",
        "            #return a positive reward.\n",
        "            return 1\n",
        "        else:\n",
        "            #return a negative reward.\n",
        "            return -1"
      ],
      "metadata": {
        "id": "7qwxes_24gCX"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**The Policy-Based Agent**\n",
        "\n",
        "\n",
        "The code below established our simple neural agent. It takes as input the current state, and returns an action. This allows the agent to take actions which are conditioned on the state of the environment, a critical step toward being able to solve full RL problems. The agent uses a single set of weights, within which each value is an estimate of the value of the return from choosing a particular arm given a bandit. We use a policy gradient method to update the agent by moving the value for the selected action toward the recieved reward."
      ],
      "metadata": {
        "id": "QYewEpCj4zZM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class agent():\n",
        "    def __init__(self, lr, s_size,a_size):\n",
        "        #These lines established the feed-forward part of the network. The agent takes a state and produces an action.\n",
        "        self.state_in= tf.placeholder(shape=[1],dtype=tf.int32)\n",
        "        state_in_OH = slim.one_hot_encoding(self.state_in,s_size)\n",
        "        output = slim.fully_connected(state_in_OH,a_size,\\\n",
        "            biases_initializer=None,activation_fn=tf.nn.sigmoid,weights_initializer=tf.ones_initializer())\n",
        "        self.output = tf.reshape(output,[-1])\n",
        "        self.chosen_action = tf.argmax(self.output,0)\n",
        "\n",
        "        #The next six lines establish the training proceedure. We feed the reward and chosen action into the network\n",
        "        #to compute the loss, and use it to update the network.\n",
        "        self.reward_holder = tf.placeholder(shape=[1],dtype=tf.float32)\n",
        "        self.action_holder = tf.placeholder(shape=[1],dtype=tf.int32)\n",
        "        self.responsible_weight = tf.slice(self.output,self.action_holder,[1])\n",
        "        self.loss = -(tf.log(self.responsible_weight)*self.reward_holder)\n",
        "        optimizer = tf.train.GradientDescentOptimizer(learning_rate=lr)\n",
        "        self.update = optimizer.minimize(self.loss)"
      ],
      "metadata": {
        "id": "H6Q_aKya4_f8"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Training the Agent**\n",
        "\n",
        "\n",
        "We will train our agent by getting a state from the environment, take an action, and recieve a reward. Using these three things, we can know how to properly update our network in order to more often choose actions given states that will yield the highest rewards over time."
      ],
      "metadata": {
        "id": "t5e80A9o5icE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tf.reset_default_graph() #Clear the Tensorflow graph.\n",
        "\n",
        "cBandit = contextual_bandit() #Load the bandits.\n",
        "myAgent = agent(lr=0.001,s_size=cBandit.num_bandits,a_size=cBandit.num_actions) #Load the agent.\n",
        "weights = tf.trainable_variables()[0] #The weights we will evaluate to look into the network.\n",
        "\n",
        "total_episodes = 10000 #Set total number of episodes to train agent on.\n",
        "total_reward = np.zeros([cBandit.num_bandits,cBandit.num_actions]) #Set scoreboard for bandits to 0.\n",
        "e = 0.1 #Set the chance of taking a random action.\n",
        "\n",
        "init = tf.initialize_all_variables()\n",
        "\n",
        "# Launch the tensorflow graph\n",
        "with tf.Session() as sess:\n",
        "    sess.run(init)\n",
        "    i = 0\n",
        "    while i < total_episodes:\n",
        "        s = cBandit.getBandit() #Get a state from the environment.\n",
        "\n",
        "        #Choose either a random action or one from our network.\n",
        "        if np.random.rand(1) < e:\n",
        "            action = np.random.randint(cBandit.num_actions)\n",
        "        else:\n",
        "            action = sess.run(myAgent.chosen_action,feed_dict={myAgent.state_in:[s]})\n",
        "\n",
        "        reward = cBandit.pullArm(action) #Get our reward for taking an action given a bandit.\n",
        "\n",
        "        #Update the network.\n",
        "        feed_dict={myAgent.reward_holder:[reward],myAgent.action_holder:[action],myAgent.state_in:[s]}\n",
        "        _,ww = sess.run([myAgent.update,weights], feed_dict=feed_dict)\n",
        "\n",
        "        #Update our running tally of scores.\n",
        "        total_reward[s,action] += reward\n",
        "        if i % 500 == 0:\n",
        "            print(\"Mean reward for each of the \" + str(cBandit.num_bandits) + \" bandits: \" + str(np.mean(total_reward,axis=1)))\n",
        "        i+=1\n",
        "for a in range(cBandit.num_bandits):\n",
        "    print(\"The agent thinks action \" + str(np.argmax(ww[a])+1) + \" for bandit \" + str(a+1) + \" is the most promising....\")\n",
        "    if np.argmax(ww[a]) == np.argmin(cBandit.bandits[a]):\n",
        "        print(\"...and it was right!\")\n",
        "    else:\n",
        "        print(\"...and it was wrong!\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xsw-YkxA5nH9",
        "outputId": "260da8a0-7b94-4247-bf26-9f50bb93bec8"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.10/dist-packages/tensorflow/python/util/dispatch.py:1260: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.cast` instead.\n",
            "/usr/local/lib/python3.10/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py:1697: UserWarning: `layer.apply` is deprecated and will be removed in a future version. Please use `layer.__call__` method instead.\n",
            "  warnings.warn('`layer.apply` is deprecated and '\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.10/dist-packages/tensorflow/python/util/tf_should_use.py:288: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.\n",
            "Instructions for updating:\n",
            "Use `tf.global_variables_initializer` instead.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mean reward for each of the 3 bandits: [ 0.    0.   -0.25]\n",
            "Mean reward for each of the 3 bandits: [ 3.75 38.   33.5 ]\n",
            "Mean reward for each of the 3 bandits: [ 6.5  76.5  63.75]\n",
            "Mean reward for each of the 3 bandits: [ 12.75 112.25  96.75]\n",
            "Mean reward for each of the 3 bandits: [ 14.5  148.25 134.5 ]\n",
            "Mean reward for each of the 3 bandits: [ 15.   181.25 173.5 ]\n",
            "Mean reward for each of the 3 bandits: [ 15.25 221.   209.5 ]\n",
            "Mean reward for each of the 3 bandits: [ 17.75 259.5  239.  ]\n",
            "Mean reward for each of the 3 bandits: [ 14.   298.   273.25]\n",
            "Mean reward for each of the 3 bandits: [ 16.75 338.25 307.75]\n",
            "Mean reward for each of the 3 bandits: [ 19.75 374.   345.  ]\n",
            "Mean reward for each of the 3 bandits: [ 15.75 413.5  376.  ]\n",
            "Mean reward for each of the 3 bandits: [ 50.75 450.25 412.75]\n",
            "Mean reward for each of the 3 bandits: [ 91.   488.75 445.5 ]\n",
            "Mean reward for each of the 3 bandits: [125.5  525.75 481.5 ]\n",
            "Mean reward for each of the 3 bandits: [161.75 563.75 516.75]\n",
            "Mean reward for each of the 3 bandits: [196.25 603.   554.5 ]\n",
            "Mean reward for each of the 3 bandits: [231.25 642.   588.  ]\n",
            "Mean reward for each of the 3 bandits: [269.5  679.75 627.  ]\n",
            "Mean reward for each of the 3 bandits: [308.   715.25 661.5 ]\n",
            "The agent thinks action 4 for bandit 1 is the most promising....\n",
            "...and it was right!\n",
            "The agent thinks action 2 for bandit 2 is the most promising....\n",
            "...and it was right!\n",
            "The agent thinks action 1 for bandit 3 is the most promising....\n",
            "...and it was right!\n"
          ]
        }
      ]
    }
  ]
}