CompleteCodeBREAKOUT.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# **BREAKOUT Player Implementation**\n",
        "\n",
        "You can run this code in Google Colab and watch your NN learn to play BreakOut.\n",
        "I tried to solve and recreate the succes of the famus DQN paper \"Playing Atari with Deep Reinforcement Learning\" (https://arxiv.org/abs/1312.5602).\n",
        "However, while doing so I insisted on doing it by myself. So, I was not aware of the Atari env wrappers used by DeepMind and how convenient they are. Also, I was not aware of the Categorical 51 DQN algorithem and how this approach leads to improved stability and performance in reinforcement learning tasks.\n",
        "\n",
        "The result was me banging my head against the wall, asking myself, \"How could this be so hard if somebody already solved this ??\".\n",
        "Well, it could be challenging when you know someone has solved it, but you refuse to learn from their experience and insist on starting from scratch, relying solely on your own logic.\n",
        "\n",
        "I gained perspective and tried various methods to converge the system to a solution. Experimenting with different reward systems and momentum approaches daily was fun in a frustrating way. I learned a lot and even exposed myself to 'reward models' and 'curiosity models.' But actually opening another code and seeing the solution - that I would not do. At some point I realised I should get off my high horse and learn from other solutions.\n",
        "\n",
        "When I saw this solution 2 things sturct me -\n",
        "*   The environment wrappers make the rest of the code so elegant and smooth; I can't believe they were accessible, and I did not use them.\n",
        "*   The logic behind C51 Categorical DQN is so elegant, simple, and clean.\n",
        "\n",
        "I know there are more approaches I can learn about and implement, like Proximal Policy Optimization (PPO), and I am excited to keep digging into this field. Maybe now that I have seen this solution in code, I will try to mix it with one of the ideas I had while I was banging my head against the wall. To be continued..\n",
        "\n",
        "If you are reading this, I hope I am making this field sound clearer, simplified, and approachable while sharing my review. I am open for questions.\n",
        "\n",
        "The explanation about the Neural network and training process is shared in the file 'Build Neural Network.ipynb.' The explanation about the Atari wrappers is in the file 'Atari Env Wrappers Explained.ipynb.' More files about my attempts will be shared later on.\n",
        "\n"
      ],
      "metadata": {
        "id": "4uKR3z--bLkH"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Credit & Source - the code is from https://wandb.ai/cleanrl/cleanrl.benchmark/runs/2r5fagq8/overview"
      ],
      "metadata": {
        "id": "81TUQZHQa8xQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "IMPORTS"
      ],
      "metadata": {
        "id": "NxaV938STOUv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# @title\n",
        "########################## IMPORTS #########################################\n",
        "%pip install -U ale_py==0.8.0\n",
        "%pip install gym[atari,accept-rom-license]\n",
        "%pip install gymnasium\n",
        "import numpy as np\n",
        "import gymnasium\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "import torch.nn.functional as F\n",
        "from torch.utils.tensorboard import SummaryWriter\n",
        "\n",
        "import argparse\n",
        "from distutils.util import strtobool\n",
        "import collections\n",
        "import numpy as np\n",
        "import gym\n",
        "from gym.wrappers import TimeLimit #, Monitor\n",
        "from gym.spaces import Discrete, Box, MultiBinary, MultiDiscrete, Space\n",
        "import time\n",
        "import random\n",
        "import os\n",
        "\n",
        "import matplotlib\n",
        "matplotlib.use('Agg')\n",
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "import pandas as pd\n",
        "from PIL import Image\n",
        "from google.colab.patches import cv2_imshow\n",
        "import cv2\n",
        "import copy\n",
        "!apt-get install -y python-opengl ffmpeg\n",
        "%pip install pyvirtualdisplay\n",
        "from IPython import display as ipythondisplay\n",
        "from pyvirtualdisplay import Display\n",
        "from gym.wrappers.record_video import RecordVideo\n",
        "from IPython.display import display, HTML\n",
        "from IPython.display import clear_output\n",
        "import io\n",
        "import base64\n",
        "%pip install 'shimmy>=0.2.1'\n",
        "from gym.spaces import Discrete\n",
        "from torch.distributions.categorical import Categorical\n",
        "from collections import deque\n",
        "from gym import spaces\n",
        "cv2.ocl.setUseOpenCL(False)\n"
      ],
      "metadata": {
        "cellView": "form",
        "id": "s3A-6GRCTLuO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Use CPU or GPU as device"
      ],
      "metadata": {
        "id": "44h0LjAzTTWk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# @title\n",
        "if torch.cuda.is_available():\n",
        "    device = torch.device(\"cuda\")\n",
        "    print(\"Using the GPU\")\n",
        "\n",
        "else:\n",
        "    device = torch.device(\"cpu\")"
      ],
      "metadata": {
        "id": "YwpNoYD0W23t",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Basic Atari env wrappers for DQN implementation"
      ],
      "metadata": {
        "id": "kYvpyv0NzCyn"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "BzHHmPQSQ_OZ",
        "cellView": "form"
      },
      "outputs": [],
      "source": [
        "# @title\n",
        "\n",
        "class NoopResetEnv(gym.Wrapper):\n",
        "    def __init__(self, env, noop_max=30):\n",
        "        \"\"\"Sample initial states by taking random number of no-ops on reset.\n",
        "        No-op is assumed to be action 0.\n",
        "        \"\"\"\n",
        "        gym.Wrapper.__init__(self, env)\n",
        "        self.noop_max = noop_max\n",
        "        self.override_num_noops = None\n",
        "        self.noop_action = 0\n",
        "        assert env.unwrapped.get_action_meanings()[0] == 'NOOP'\n",
        "\n",
        "    def reset(self, **kwargs):\n",
        "        \"\"\" Do no-op action for a number of steps in [1, noop_max].\"\"\"\n",
        "        self.env.reset(**kwargs)\n",
        "        if self.override_num_noops is not None:\n",
        "            noops = self.override_num_noops\n",
        "        else:\n",
        "            noops = self.unwrapped.np_random.integers(1, self.noop_max + 1) #pylint: disable=E1101 ####randit --> integers\n",
        "        assert noops > 0\n",
        "        obs = None\n",
        "        for _ in range(noops):\n",
        "            obs, _, done, _ = self.env.step(self.noop_action)\n",
        "            if done:\n",
        "                obs = self.env.reset(**kwargs)\n",
        "        return obs\n",
        "\n",
        "    def step(self, ac):\n",
        "        return self.env.step(ac)\n",
        "\n",
        "class FireResetEnv(gym.Wrapper):\n",
        "    def __init__(self, env):\n",
        "        \"\"\"Take action on reset for environments that are fixed until firing.\"\"\"\n",
        "        gym.Wrapper.__init__(self, env)\n",
        "        assert env.unwrapped.get_action_meanings()[1] == 'FIRE'\n",
        "        assert len(env.unwrapped.get_action_meanings()) >= 3\n",
        "\n",
        "    def reset(self, **kwargs):\n",
        "        self.env.reset(**kwargs)\n",
        "        obs, _, done, _ = self.env.step(1)\n",
        "        if done:\n",
        "            self.env.reset(**kwargs)\n",
        "        obs, _, done, _ = self.env.step(2)\n",
        "        if done:\n",
        "            self.env.reset(**kwargs)\n",
        "        return obs\n",
        "\n",
        "    def step(self, ac):\n",
        "        return self.env.step(ac)\n",
        "\n",
        "class EpisodicLifeEnv(gym.Wrapper):\n",
        "    def __init__(self, env):\n",
        "        \"\"\"Make end-of-life == end-of-episode, but only reset on true game over.\n",
        "        Done by DeepMind for the DQN and co. since it helps value estimation.\n",
        "        \"\"\"\n",
        "        gym.Wrapper.__init__(self, env)\n",
        "        self.lives = 0\n",
        "        self.was_real_done  = True\n",
        "\n",
        "    def step(self, action):\n",
        "        obs, reward, done, info = self.env.step(action)\n",
        "        self.was_real_done = done\n",
        "        # check current lives, make loss of life terminal,\n",
        "        # then update lives to handle bonus lives\n",
        "        lives = self.env.unwrapped.ale.lives()\n",
        "        if lives < self.lives and lives > 0:\n",
        "            # for Qbert sometimes we stay in lives == 0 condition for a few frames\n",
        "            # so it's important to keep lives > 0, so that we only reset once\n",
        "            # the environment advertises done.\n",
        "            done = True\n",
        "        self.lives = lives\n",
        "        return obs, reward, done, info\n",
        "\n",
        "    def reset(self, **kwargs):\n",
        "        \"\"\"Reset only when lives are exhausted.\n",
        "        This way all states are still reachable even though lives are episodic,\n",
        "        and the learner need not know about any of this behind-the-scenes.\n",
        "        \"\"\"\n",
        "        if self.was_real_done:\n",
        "            obs = self.env.reset(**kwargs)\n",
        "        else:\n",
        "            # no-op step to advance from terminal/lost life state\n",
        "            obs, _, _, _ = self.env.step(0)\n",
        "        self.lives = self.env.unwrapped.ale.lives()\n",
        "        return obs\n",
        "\n",
        "class MaxAndSkipEnv(gym.Wrapper):\n",
        "    def __init__(self, env, skip=4):\n",
        "        \"\"\"Return only every `skip`-th frame\"\"\"\n",
        "        gym.Wrapper.__init__(self, env)\n",
        "        # most recent raw observations (for max pooling across time steps)\n",
        "        self._obs_buffer = np.zeros((2,)+env.observation_space.shape, dtype=np.uint8)\n",
        "        self._skip       = skip\n",
        "\n",
        "    def step(self, action):\n",
        "        \"\"\"Repeat action, sum reward, and max over last observations.\"\"\"\n",
        "        total_reward = 0.0\n",
        "        done = None\n",
        "        for i in range(self._skip):\n",
        "            obs, reward, done, info = self.env.step(action)\n",
        "            if i == self._skip - 2: self._obs_buffer[0] = obs\n",
        "            if i == self._skip - 1: self._obs_buffer[1] = obs\n",
        "            total_reward += reward\n",
        "            if done:\n",
        "                break\n",
        "        # Note that the observation on the done=True frame\n",
        "        # doesn't matter\n",
        "        max_frame = self._obs_buffer.max(axis=0)\n",
        "\n",
        "        return max_frame, total_reward, done, info\n",
        "\n",
        "    def reset(self, **kwargs):\n",
        "        return self.env.reset(**kwargs)\n",
        "\n",
        "class ClipRewardEnv(gym.RewardWrapper):\n",
        "    def __init__(self, env):\n",
        "        gym.RewardWrapper.__init__(self, env)\n",
        "\n",
        "    def reward(self, reward):\n",
        "        \"\"\"Bin reward to {+1, 0, -1} by its sign.\"\"\"\n",
        "        return np.sign(reward)\n",
        "\n",
        "\n",
        "class WarpFrame(gym.ObservationWrapper):\n",
        "    def __init__(self, env, width=84, height=84, grayscale=True, dict_space_key=None):\n",
        "        \"\"\"\n",
        "        Warp frames to 84x84 as done in the Nature paper and later work.\n",
        "        If the environment uses dictionary observations, `dict_space_key` can be specified which indicates which\n",
        "        observation should be warped.\n",
        "        \"\"\"\n",
        "        super().__init__(env)\n",
        "        self._width = width\n",
        "        self._height = height\n",
        "        self._grayscale = grayscale\n",
        "        self._key = dict_space_key\n",
        "        if self._grayscale:\n",
        "            num_colors = 1\n",
        "        else:\n",
        "            num_colors = 3\n",
        "\n",
        "        new_space = gym.spaces.Box(\n",
        "            low=0,\n",
        "            high=255,\n",
        "            shape=(self._height, self._width, num_colors),\n",
        "            dtype=np.uint8,\n",
        "        )\n",
        "        if self._key is None:\n",
        "            original_space = self.observation_space\n",
        "            self.observation_space = new_space\n",
        "        else:\n",
        "            original_space = self.observation_space.spaces[self._key]\n",
        "            self.observation_space.spaces[self._key] = new_space\n",
        "        assert original_space.dtype == np.uint8 and len(original_space.shape) == 3\n",
        "\n",
        "    def observation(self, obs):\n",
        "        if self._key is None:\n",
        "            frame = obs\n",
        "        else:\n",
        "            frame = obs[self._key]\n",
        "\n",
        "        if self._grayscale:\n",
        "            frame = cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY)\n",
        "        frame = cv2.resize(\n",
        "            frame, (self._width, self._height), interpolation=cv2.INTER_AREA\n",
        "        )\n",
        "        if self._grayscale:\n",
        "            frame = np.expand_dims(frame, -1)\n",
        "\n",
        "        if self._key is None:\n",
        "            obs = frame\n",
        "        else:\n",
        "            obs = obs.copy()\n",
        "            obs[self._key] = frame\n",
        "        return obs\n",
        "\n",
        "\n",
        "class FrameStack(gym.Wrapper):\n",
        "    def __init__(self, env, k):\n",
        "        \"\"\"Stack k last frames.\n",
        "        Returns lazy array, which is much more memory efficient.\n",
        "        See Also\n",
        "        --------\n",
        "        baselines.common.atari_wrappers.LazyFrames\n",
        "        \"\"\"\n",
        "        gym.Wrapper.__init__(self, env)\n",
        "        self.k = k\n",
        "        self.frames = deque([], maxlen=k)\n",
        "        shp = env.observation_space.shape\n",
        "        self.observation_space = spaces.Box(low=0, high=255, shape=(shp[:-1] + (shp[-1] * k,)), dtype=env.observation_space.dtype)\n",
        "\n",
        "    def reset(self):\n",
        "        ob = self.env.reset()\n",
        "        for _ in range(self.k):\n",
        "            self.frames.append(ob)\n",
        "        return self._get_ob()\n",
        "\n",
        "    def step(self, action):\n",
        "        ob, reward, done, info = self.env.step(action)\n",
        "        self.just_rgb_render()\n",
        "        self.frames.append(ob)\n",
        "        #print(ob.shape)\n",
        "        return self._get_ob(), reward, done, info\n",
        "\n",
        "    def _get_ob(self):\n",
        "        assert len(self.frames) == self.k\n",
        "        return LazyFrames(list(self.frames))\n",
        "\n",
        "class ScaledFloatFrame(gym.ObservationWrapper):\n",
        "    def __init__(self, env):\n",
        "        gym.ObservationWrapper.__init__(self, env)\n",
        "        self.observation_space = gym.spaces.Box(low=0, high=1, shape=env.observation_space.shape, dtype=np.float32)\n",
        "\n",
        "    def observation(self, observation):\n",
        "        # careful! This undoes the memory optimization, use\n",
        "        # with smaller replay buffers only.\n",
        "        return np.array(observation).astype(np.float32) / 255.0\n",
        "\n",
        "class LazyFrames(object):\n",
        "    def __init__(self, frames):\n",
        "        \"\"\"This object ensures that common frames between the observations are only stored once.\n",
        "        It exists purely to optimize memory usage which can be huge for DQN's 1M frames replay\n",
        "        buffers.\n",
        "        This object should only be converted to numpy array before being passed to the model.\n",
        "        You'd not believe how complex the previous solution was.\"\"\"\n",
        "        self._frames = frames\n",
        "        self._out = None\n",
        "\n",
        "    def _force(self):\n",
        "        if self._out is None:\n",
        "            self._out = np.concatenate(self._frames, axis=-1)\n",
        "            self._frames = None\n",
        "        return self._out\n",
        "\n",
        "    def __array__(self, dtype=None):\n",
        "        out = self._force()\n",
        "        if dtype is not None:\n",
        "            out = out.astype(dtype)\n",
        "        return out\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self._force())\n",
        "\n",
        "    def __getitem__(self, i):\n",
        "        return self._force()[i]\n",
        "\n",
        "    def count(self):\n",
        "        frames = self._force()\n",
        "        return frames.shape[frames.ndim - 1]\n",
        "\n",
        "    def frame(self, i):\n",
        "        return self._force()[..., i]\n",
        "\n",
        "def wrap_atari(env, max_episode_steps=None):\n",
        "    assert 'NoFrameskip' in env.spec.id\n",
        "    env = NoopResetEnv(env, noop_max=30)\n",
        "    env = MaxAndSkipEnv(env, skip=4)\n",
        "\n",
        "    assert max_episode_steps is None\n",
        "\n",
        "    return env\n",
        "\n",
        "def wrap_deepmind(env, episode_life=True, clip_rewards=True, frame_stack=False, scale=False):\n",
        "    \"\"\"Configure environment for DeepMind-style Atari.\n",
        "    \"\"\"\n",
        "    if episode_life:\n",
        "        env = EpisodicLifeEnv(env)\n",
        "    if 'FIRE' in env.unwrapped.get_action_meanings():\n",
        "        env = FireResetEnv(env)\n",
        "    env = WarpFrame(env)\n",
        "    if scale:\n",
        "        env = ScaledFloatFrame(env)\n",
        "    if clip_rewards:\n",
        "        env = ClipRewardEnv(env)\n",
        "    if frame_stack:\n",
        "        env = FrameStack(env, 4)\n",
        "    return env\n",
        "\n",
        "\n",
        "class ImageToPyTorch(gym.ObservationWrapper):\n",
        "    \"\"\"\n",
        "    Image shape to channels x weight x height\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, env):\n",
        "        super(ImageToPyTorch, self).__init__(env)\n",
        "        old_shape = self.observation_space.shape\n",
        "        self.observation_space = gym.spaces.Box(\n",
        "            low=0,\n",
        "            high=255,\n",
        "            shape=(old_shape[-1], old_shape[0], old_shape[1]),\n",
        "            dtype=np.uint8,\n",
        "        )\n",
        "\n",
        "    def observation(self, observation):\n",
        "        return np.transpose(observation, axes=(2, 0, 1))\n",
        "\n",
        "def wrap_pytorch(env):\n",
        "    return ImageToPyTorch(env)\n",
        "\n",
        "# Reference: https://arxiv.org/pdf/1707.06887.pdf\n",
        "# https://github.com/ShangtongZhang/DeepRL/blob/master/deep_rl/agent/CategoricalDQN_agent.py\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# C51 additional wrappers code"
      ],
      "metadata": {
        "id": "J9qYGo1zz45Z"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# @title\n",
        "\n",
        "def image_return_distributions(pmfs, x_length, y_length, dpi=100):\n",
        "    fig, ax = plt.subplots(figsize=(x_length/dpi,y_length/dpi), constrained_layout=True, dpi=100)\n",
        "    current_palette = sns.color_palette(n_colors=env.action_space.n)\n",
        "    df = pd.DataFrame(pmfs.T)\n",
        "    for idx, y in enumerate(df.columns):\n",
        "        ax.bar(np.linspace(args.v_min, args.v_max, num=args.n_atoms).astype(int), df[y], color=current_palette[idx]) ######## np.int\n",
        "    ax.set(xlabel='return distribution', ylabel='probs')\n",
        "    fig.canvas.draw()\n",
        "    X = np.array(fig.canvas.renderer.buffer_rgba())\n",
        "    return_distribution_rgb_array = np.array(Image.fromarray(X).convert('RGB'))\n",
        "    plt.close(fig)\n",
        "    return return_distribution_rgb_array\n",
        "\n",
        "def image_q_values(q_values, x_length, y_length, dpi=100):\n",
        "    fig, ax = plt.subplots(figsize=(x_length/dpi,y_length/dpi), constrained_layout=True, dpi=100)\n",
        "    df = pd.DataFrame(q_values.T)\n",
        "    sns.barplot(x=df.index, y=0, data=df, ax=ax)\n",
        "    ax.set(xlabel='actions', ylabel='q-values')\n",
        "    fig.canvas.draw()\n",
        "    X = np.array(fig.canvas.renderer.buffer_rgba())\n",
        "    # Image.fromarray(X)\n",
        "    q_value_rgb_array = np.array(Image.fromarray(X).convert('RGB'))\n",
        "    plt.close(fig)\n",
        "    return q_value_rgb_array\n",
        "\n",
        "\n",
        "class QValueAndReturnDistributionVisualizationWrapper(gym.Wrapper):\n",
        "\n",
        "    def __init__(self, env):\n",
        "        super().__init__(env)\n",
        "        self.env.reset()\n",
        "        self.image_shape = self.env.render(mode=\"rgb_array\").shape\n",
        "        self.q_values = np.array([[0.,0.,0.,0.]])\n",
        "        self.return_pmfs = np.zeros((4, 51))\n",
        "        # self.metadata['video.frames_per_second'] = 60\n",
        "        self.states = collections.deque(maxlen=3000) ## Personal addition\n",
        "\n",
        "    def set_q_values(self, q_values,renderme= False):\n",
        "        self.q_values = q_values\n",
        "        if renderme :\n",
        "            self.render()\n",
        "\n",
        "    def set_return_pmfs(self, return_pmfs):\n",
        "        self.return_pmfs = return_pmfs\n",
        "\n",
        "    def just_rgb_render(self , mode=\"rgb_array\"): ## Personal addition\n",
        "        env_rgb_array = super().render(mode)\n",
        "        self.states.append(env_rgb_array)\n",
        "\n",
        "    def render(self, mode=\"human\"):\n",
        "        if mode==\"rgb_array\":\n",
        "            env_rgb_array = super().render(mode)\n",
        "            self.q_value_rgb_array = image_q_values(\n",
        "                self.q_values, self.image_shape[1], self.image_shape[0], dpi=100)\n",
        "            self.return_rgb_array = image_return_distributions(\n",
        "                self.return_pmfs, self.image_shape[1]*2, self.image_shape[0], dpi=100)\n",
        "            return np.append(np.append(\n",
        "                env_rgb_array,\n",
        "                self.q_value_rgb_array, axis=1), self.return_rgb_array,\n",
        "                axis=0)\n",
        "        else:\n",
        "            super().render(mode)\n"
      ],
      "metadata": {
        "id": "TdfvnMLCT_Gq",
        "cellView": "form"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Create Args object"
      ],
      "metadata": {
        "id": "nC-ulTdUzlxg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# @title\n",
        "class Args:\n",
        "    def __init__(self, gym_id=\"BreakoutNoFrameskip-v4\", learning_rate=25e-5, seed=1,\n",
        "                 total_timesteps=10000000, torch_deterministic=True, cuda=True,\n",
        "                 prod_mode=False, capture_video=True, wandb_project_name=\"cleanRL\",\n",
        "                 wandb_entity=None, n_atoms=51, v_min=-10, v_max=10,\n",
        "                 buffer_size=1000000, gamma=0.99, target_network_frequency=10000,\n",
        "                 max_grad_norm=0.5, batch_size=32, start_e=1.0, end_e=0.01,\n",
        "                 exploration_fraction=0.10, learning_starts=80000, train_frequency=4):\n",
        "\n",
        "        ####\n",
        "        self.gym_id = \"BreakoutNoFrameskip-v4\"\n",
        "        self.learning_rate = 25e-5\n",
        "        self.seed = 2\n",
        "        self.total_timesteps = 10000000\n",
        "        self.torch_deterministic = True\n",
        "        self.cuda = True\n",
        "        self.prod_mode = False\n",
        "        self.capture_video = True\n",
        "        self.wandb_project_name = \"cleanRL\"\n",
        "        self.wandb_entity = None\n",
        "        self.n_minibatch = 4\n",
        "        self.num_envs = 8\n",
        "        self.num_steps = 128\n",
        "        self.gamma = 0.99\n",
        "        self.gae_lambda = 0.95\n",
        "        self.ent_coef = 0.01\n",
        "        self.vf_coef = 0.5\n",
        "        self.max_grad_norm = 0.5\n",
        "        self.clip_coef = 0.1\n",
        "        self.update_epochs = 4\n",
        "        self.kle_stop = False\n",
        "        self.kle_rollback = False\n",
        "        self.target_kl = 0.03\n",
        "        self.gae = True\n",
        "        self.norm_adv = True\n",
        "        self.anneal_lr = True\n",
        "        self.clip_vloss = True\n",
        "        self.batch_size = 32  # Adjusted based on the formula in the first code\n",
        "        self.n_atoms = 51\n",
        "        self.v_min = -10  # Added based on the first code\n",
        "        self.v_max = 10   # Added based on the first code\n",
        "        self.buffer_size = 1000000  # Added based on the first code\n",
        "        self.target_network_frequency = 10000  # Added based on the first code\n",
        "        self.start_e = 1.0  # Added based on the first code\n",
        "        self.end_e = 0.01  # Added based on the first code\n",
        "        self.exploration_fraction = 0.10  # Added based on the first code\n",
        "        self.learning_starts = 80000  # Added based on the first code\n",
        "        self.train_frequency = 4  # Added based on the first code\n",
        "\n",
        "\n",
        "\n",
        "# Example usage\n",
        "args = Args()\n",
        "print(args.gym_id)\n",
        "if not args.seed:\n",
        "    args.seed = int(time.time())\n",
        "args.batch_size = int(args.num_envs * args.num_steps)\n",
        "args.minibatch_size = int(args.batch_size // args.n_minibatch)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uoMSFXPvzrJ5",
        "outputId": "dfc0f567-20fc-4bfa-91bf-2917d8aee74b",
        "cellView": "form"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "BreakoutNoFrameskip-v4\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Environment Set Up & Seeding"
      ],
      "metadata": {
        "id": "tsTk4v4a0CN3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# @title\n",
        "# TRY NOT TO MODIFY: setup the environment\n",
        "experiment_name = f\"{args.gym_id}__{args.seed}__{int(time.time())}\"\n",
        "writer = SummaryWriter(f\"runs/{experiment_name}\")\n",
        "writer.add_text('hyperparameters', \"|param|value|\\n|-|-|\\n%s\" % (\n",
        "        '\\n'.join([f\"|{key}|{value}|\" for key, value in vars(args).items()])))\n",
        "\n",
        "# TRY NOT TO MODIFY: seeding\n",
        "device = torch.device('cuda' if torch.cuda.is_available() and args.cuda else 'cpu')\n",
        "env = gym.make(args.gym_id)\n",
        "env = wrap_atari(env)\n",
        "env = gym.wrappers.RecordEpisodeStatistics(env) # records episode reward in `info['episode']['r']`\n",
        "if 1 : #args.capture_video:\n",
        "    #env = QValueAndReturnDistributionVisualizationWrapper(env)\n",
        "    #env = ProbsVisualizationWrapper(env)\n",
        "    env = QValueAndReturnDistributionVisualizationWrapper(env)\n",
        "    #env = Monitor(env, f'videos/{experiment_name}')\n",
        "env = wrap_pytorch(\n",
        "    wrap_deepmind(\n",
        "        env,\n",
        "        clip_rewards=True,\n",
        "        frame_stack=True,\n",
        "        scale=False,\n",
        "    )\n",
        ")\n",
        "random.seed(args.seed)\n",
        "np.random.seed(args.seed)\n",
        "torch.manual_seed(args.seed)\n",
        "torch.backends.cudnn.deterministic = args.torch_deterministic\n",
        "env.seed(args.seed)\n",
        "env.action_space.seed(args.seed)\n",
        "env.observation_space.seed(args.seed)\n",
        "# respect the default timelimit\n",
        "print(env.action_space)\n",
        "assert isinstance(env.action_space, Discrete), \"only discrete action space is supported\"\n"
      ],
      "metadata": {
        "id": "T9dBXof30DJx",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#class ReplayBuffer"
      ],
      "metadata": {
        "id": "TCbr0Mne567N"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# @title\n",
        "# modified from https://github.com/seungeunrho/minimalRL/blob/master/dqn.py#\n",
        "class ReplayBuffer():\n",
        "    def __init__(self, buffer_limit):\n",
        "        self.buffer = collections.deque(maxlen=buffer_limit)\n",
        "\n",
        "    def put(self, transition):\n",
        "        self.buffer.append(transition)\n",
        "\n",
        "    def sample(self, n):\n",
        "        mini_batch = random.sample(self.buffer, n)\n",
        "        s_lst, a_lst, r_lst, s_prime_lst, done_mask_lst = [], [], [], [], []\n",
        "\n",
        "        for transition in mini_batch:\n",
        "            s, a, r, s_prime, done_mask = transition\n",
        "            s_lst.append(s)\n",
        "            a_lst.append(a)\n",
        "            r_lst.append(r)\n",
        "            s_prime_lst.append(s_prime)\n",
        "            done_mask_lst.append(done_mask)\n",
        "\n",
        "        return np.array(s_lst), np.array(a_lst), \\\n",
        "               np.array(r_lst), np.array(s_prime_lst), \\\n",
        "               np.array(done_mask_lst)"
      ],
      "metadata": {
        "id": "NXa3kUwZ05HA",
        "cellView": "form"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Build ***Neural Network***"
      ],
      "metadata": {
        "id": "dQSZe-wxXQOt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# @title\n",
        "# tricks taken from https://github.com/cpnota/autonomous-learning-library/blob/6d1111afce0d1582de463326f7d078a86e850551/all/presets/atari/models/__init__.py#L16\n",
        "# apparently matters\n",
        "class Linear0(nn.Linear):\n",
        "    def reset_parameters(self):\n",
        "        nn.init.constant_(self.weight, 0.0)\n",
        "        if self.bias is not None:\n",
        "            nn.init.constant_(self.bias, 0.0)\n",
        "\n",
        "class Scale(nn.Module):\n",
        "    def __init__(self, scale):\n",
        "        super().__init__()\n",
        "        self.scale = scale\n",
        "\n",
        "    def forward(self, x):\n",
        "        return x * self.scale\n",
        "\n",
        "class QNetwork(nn.Module):\n",
        "    def __init__(self, frames=4, n_atoms=51, v_min=-10, v_max=10):\n",
        "        super(QNetwork, self).__init__()\n",
        "        self.n_atoms = n_atoms\n",
        "        self.atoms = torch.linspace(v_min, v_max, steps=n_atoms).to(device)\n",
        "        self.network = nn.Sequential(\n",
        "            Scale(1/255),\n",
        "            nn.Conv2d(frames, 32, 8, stride=4),\n",
        "            nn.ReLU(),\n",
        "            nn.Conv2d(32, 64, 4, stride=2),\n",
        "            nn.ReLU(),\n",
        "            nn.Conv2d(64, 64, 3, stride=1),\n",
        "            nn.ReLU(),\n",
        "            nn.Flatten(),\n",
        "            nn.Linear(3136, 512),\n",
        "            nn.ReLU(),\n",
        "            Linear0(512, env.action_space.n * n_atoms)\n",
        "        )\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = torch.Tensor(x).to(device)\n",
        "        return self.network(x)\n",
        "\n",
        "    def get_action(self, x, action=None):\n",
        "        logits = self.forward(x)\n",
        "        # probability mass function for each action\n",
        "        pmfs = torch.softmax(logits.view(len(x), env.action_space.n, self.n_atoms), dim=2)\n",
        "        q_values = (pmfs*self.atoms).sum(2)\n",
        "        if action is None:\n",
        "            action = torch.argmax(q_values, 1)\n",
        "        return action, pmfs[torch.arange(len(x)), action], q_values, pmfs\n"
      ],
      "metadata": {
        "id": "ioLPMvpo1H1d",
        "cellView": "form"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def linear_schedule(start_e: float, end_e: float, duration: int, t: int):\n",
        "    slope =  (end_e - start_e) / duration\n",
        "    return max(slope * t + start_e, end_e)\n",
        "\n",
        "rb = ReplayBuffer(args.buffer_size)\n",
        "q_network = QNetwork(n_atoms=args.n_atoms, v_min=args.v_min, v_max=args.v_max).to(device)\n",
        "target_network = QNetwork(n_atoms=args.n_atoms, v_min=args.v_min, v_max=args.v_max).to(device)\n",
        "target_network.load_state_dict(q_network.state_dict())\n",
        "optimizer = optim.Adam(q_network.parameters(), lr=args.learning_rate, eps=0.01/args.batch_size)\n",
        "loss_fn = nn.MSELoss()\n",
        "print(device.__repr__())\n",
        "print(q_network)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EHLcvum11X8d",
        "outputId": "cad98e6c-7000-418a-a77c-6c2024a1e06f"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "device(type='cpu')\n",
            "QNetwork(\n",
            "  (network): Sequential(\n",
            "    (0): Scale()\n",
            "    (1): Conv2d(4, 32, kernel_size=(8, 8), stride=(4, 4))\n",
            "    (2): ReLU()\n",
            "    (3): Conv2d(32, 64, kernel_size=(4, 4), stride=(2, 2))\n",
            "    (4): ReLU()\n",
            "    (5): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1))\n",
            "    (6): ReLU()\n",
            "    (7): Flatten(start_dim=1, end_dim=-1)\n",
            "    (8): Linear(in_features=3136, out_features=512, bias=True)\n",
            "    (9): ReLU()\n",
            "    (10): Linear0(in_features=512, out_features=204, bias=True)\n",
            "  )\n",
            ")\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Mount Google Drive"
      ],
      "metadata": {
        "id": "a_y5pWkrn6JS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')\n",
        "checkpoint_path='/content/gdrive/My Drive/DeepLearning/BreakOut/New/ckpt-{}.pk'"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wtwvjIyp5St_",
        "outputId": "c8aa11df-f1be-4dbd-de99-27b4eb489d2a"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/gdrive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **RUN THE GAME**"
      ],
      "metadata": {
        "id": "uVY0ZHm78zZ2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "# TRY NOT TO MODIFY: start the game\n",
        "obs = env.reset()\n",
        "episode_reward = 0\n",
        "for global_step in range(args.total_timesteps): ### One step at a loop\n",
        "    # ALGO LOGIC: put action logic here\n",
        "    epsilon = linear_schedule(args.start_e, args.end_e, args.exploration_fraction*args.total_timesteps, global_step) ## Use epsilon greedy policy\n",
        "    action, pmf, q_values, pmfs = q_network.get_action(obs.reshape((1,)+obs.shape)) ### Ask our Agent what he thinks the next action should be\n",
        "    action = action.tolist()[0] ## converting a PyTorch tensor to a Python list and then extracting the first and only element from that list\n",
        "    if args.capture_video:\n",
        "        env.set_q_values(np.array(q_values.tolist()))\n",
        "        env.set_return_pmfs(np.array(pmfs.tolist())[0])\n",
        "    if random.random() < epsilon: ## Use epsilon greedy policy\n",
        "        action = env.action_space.sample()\n",
        "\n",
        "    # TRY NOT TO MODIFY: execute the game and log data.\n",
        "    next_obs, reward, done, info = env.step(action) ## Take action in env ## done = True if life is lost\n",
        "    episode_reward += reward\n",
        "\n",
        "    # TRY NOT TO MODIFY: record rewards for plotting purposes\n",
        "    if 'episode' in info.keys(): ## condition is true when the environment signals the end of an episode (the current step resulted in the end of an episode)\n",
        "        print(f\"global_step={global_step}, episode_reward={info['episode']['r']}\")\n",
        "        writer.add_scalar(\"charts/episode_reward\", info['episode']['r'], global_step)\n",
        "        writer.add_scalar(\"charts/epsilon\", epsilon, global_step)\n",
        "\n",
        "    # ALGO LOGIC: training.\n",
        "    rb.put((obs, action, reward, next_obs, done)) ## Add to replay buffer\n",
        "    if global_step > args.learning_starts and global_step % args.train_frequency == 0: ## Train every train_frequency\n",
        "        s_obs, s_actions, s_rewards, s_next_obses, s_dones = rb.sample(args.batch_size)\n",
        "        with torch.no_grad():\n",
        "            _, next_pmfs, _, _ = target_network.get_action(s_next_obses) ## Get pmf value for best next possible action\n",
        "            next_atoms = torch.Tensor(s_rewards).to(device).unsqueeze(-1) + args.gamma * q_network.atoms  * (1 - torch.Tensor(s_dones).to(device).unsqueeze(-1))\n",
        "            ### if done == 1 then next_atoms = rewards\n",
        "            # projection\n",
        "            delta_z = q_network.atoms[1]-q_network.atoms[0] ##Delta of pmfs linspace #self.atoms = torch.linspace(v_min, v_max, steps=n_atoms)\n",
        "            tz = next_atoms.clamp(args.v_min, args.v_max) # Don't overshoot with next_atoms\n",
        "\n",
        "            b = (tz - args.v_min)/ delta_z\n",
        "            l = b.floor().clamp(0, args.n_atoms-1)\n",
        "            u = b.ceil().clamp(0, args.n_atoms-1)\n",
        "            # (l == u).float() handles the case where bj is exactly an integer\n",
        "            # example bj = 1, then the upper ceiling should be uj= 2, and lj= 1\n",
        "            d_m_l = (u + (l == u).float() - b) * next_pmfs\n",
        "            d_m_u = (b - l) * next_pmfs\n",
        "            target_pmfs = torch.zeros_like(next_pmfs)\n",
        "            for i in range(target_pmfs.size(0)):\n",
        "                target_pmfs[i].index_add_(0, l[i].long(), d_m_l[i])\n",
        "                target_pmfs[i].index_add_(0, u[i].long(), d_m_u[i])\n",
        "\n",
        "        _, old_pmfs, _, _ = q_network.get_action(s_obs, s_actions)\n",
        "        loss = (-(target_pmfs * old_pmfs.clamp(min=1e-5).log()).sum(-1)).mean()\n",
        "        # loss = (target_pmfs * (target_pmfs.clamp(min=1e-5).log() - old_pmfs.clamp(min=1e-5).log())).sum(-1).mean()\n",
        "        writer.add_scalar(\"losses/td_loss\", loss, global_step)\n",
        "\n",
        "        # optimize the midel\n",
        "        optimizer.zero_grad()\n",
        "        loss.backward()\n",
        "        nn.utils.clip_grad_norm_(list(q_network.parameters()), args.max_grad_norm)\n",
        "        optimizer.step()\n",
        "\n",
        "        # update the target network\n",
        "        if global_step % args.target_network_frequency == 0:\n",
        "            target_network.load_state_dict(q_network.state_dict())\n",
        "\n",
        "    # TRY NOT TO MODIFY: CRUCIAL step easy to overlook\n",
        "    obs = next_obs\n",
        "    if done:\n",
        "        # important to note that because `EpisodicLifeEnv` wrapper is applied,\n",
        "        # the real episode reward is actually the sum of episode reward of 5 lives\n",
        "        # which we record through `info['episode']['r']` provided by gym.wrappers.RecordEpisodeStatistics\n",
        "        obs, episode_reward = env.reset(), 0\n",
        "\n",
        "    ### Save NN weight to google drive\n",
        "    '''\n",
        "    if global_step%20000 == 0 :\n",
        "        torch.save(q_network.state_dict(), checkpoint_path.format(global_step//20000))\n",
        "        print(\"saved ckpt\",global_step//20000)\n",
        "    '''\n",
        "\n",
        "torch.save(q_network.state_dict(), checkpoint_path.format(1))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 748
        },
        "id": "K2IzbNCd1brv",
        "outputId": "1dec1b81-d9e2-4172-a907-fac163214db2"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/gym/core.py:43: DeprecationWarning: \u001b[33mWARN: The argument mode in render method is deprecated; use render_mode during environment initialization instead.\n",
            "See here for more information: https://www.gymlibrary.ml/content/api/\u001b[0m\n",
            "  deprecation(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "global_step=138, episode_reward=1.0\n",
            "global_step=325, episode_reward=2.0\n",
            "global_step=484, episode_reward=1.0\n",
            "global_step=787, episode_reward=4.0\n",
            "global_step=1026, episode_reward=3.0\n",
            "global_step=1214, episode_reward=2.0\n",
            "global_step=1424, episode_reward=2.0\n",
            "global_step=1537, episode_reward=0.0\n",
            "global_step=1753, episode_reward=3.0\n",
            "global_step=1926, episode_reward=2.0\n",
            "global_step=2185, episode_reward=3.0\n",
            "global_step=2298, episode_reward=0.0\n",
            "global_step=2439, episode_reward=1.0\n",
            "global_step=2602, episode_reward=1.0\n",
            "global_step=2717, episode_reward=0.0\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-11-2c94ac8b1a16>\u001b[0m in \u001b[0;36m<cell line: 4>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      5\u001b[0m     \u001b[0;31m# ALGO LOGIC: put action logic here\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m     \u001b[0mepsilon\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlinear_schedule\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart_e\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mend_e\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexploration_fraction\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtotal_timesteps\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mglobal_step\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m## Use epsilon greedy policy\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m     \u001b[0maction\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpmf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mq_values\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpmfs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mq_network\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_action\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreshape\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0mobs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m### Ask our Agent what he thinks the next action should be\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      8\u001b[0m     \u001b[0maction\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0maction\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtolist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;31m## converting a PyTorch tensor to a Python list and then extracting the first and only element from that list\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      9\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcapture_video\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-8-cf6c57d8b9f0>\u001b[0m in \u001b[0;36mget_action\u001b[0;34m(self, x, action)\u001b[0m\n\u001b[1;32m     40\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     41\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mget_action\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maction\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 42\u001b[0;31m         \u001b[0mlogits\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     43\u001b[0m         \u001b[0;31m# probability mass function for each action\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     44\u001b[0m         \u001b[0mpmfs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msoftmax\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlogits\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mview\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maction_space\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mn_atoms\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdim\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-8-cf6c57d8b9f0>\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, x)\u001b[0m\n\u001b[1;32m     37\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     38\u001b[0m         \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 39\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnetwork\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     40\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     41\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mget_action\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maction\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_wrapped_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1516\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_compiled_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# type: ignore[misc]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1517\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1518\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1519\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1520\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1525\u001b[0m                 \u001b[0;32mor\u001b[0m \u001b[0m_global_backward_pre_hooks\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0m_global_backward_hooks\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1526\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1527\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1528\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1529\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/nn/modules/container.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m    213\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    214\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 215\u001b[0;31m             \u001b[0minput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    216\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    217\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_wrapped_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1516\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_compiled_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# type: ignore[misc]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1517\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1518\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1519\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1520\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1525\u001b[0m                 \u001b[0;32mor\u001b[0m \u001b[0m_global_backward_pre_hooks\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0m_global_backward_hooks\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1526\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1527\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1528\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1529\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/nn/modules/linear.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m    112\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    113\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 114\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mF\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinear\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    115\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    116\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mextra_repr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Play one round and display the last 3000 states from it"
      ],
      "metadata": {
        "id": "zKk_pdIwYd-N"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "env.reset()\n",
        "env.states.clear()\n",
        "while True:\n",
        "\n",
        "    action, _, _, _ = q_network.get_action(obs.reshape((1,) + obs.shape))\n",
        "    action = action.tolist()[0]\n",
        "\n",
        "    next_obs, _, done, _ = env.step(action)\n",
        "\n",
        "    env.just_rgb_render()\n",
        "    obs = next_obs\n",
        "\n",
        "    was_real_done = env.was_real_done\n",
        "    if  was_real_done :\n",
        "        break\n",
        "    if done :\n",
        "      env.reset()\n",
        "      print(\"lost ball\")\n"
      ],
      "metadata": {
        "id": "b_c6opyUOJZ1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e1cb9977-3327-43bf-986a-74fd54dec349"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "lost ball\n",
            "lost ball\n",
            "lost ball\n",
            "lost ball\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(len(env.states))"
      ],
      "metadata": {
        "id": "WmNHXwuYHwCB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ff66cd25-971a-496d-aafd-2fb0872e98fe"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "220\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def play_states(states,sec):\n",
        "    states = np.array(states)\n",
        "    for i in range(states.shape[0]):\n",
        "        cv2_imshow(states[i])\n",
        "        time.sleep(sec)\n",
        "        clear_output(wait=True)\n",
        "\n",
        "play_states(env.states,0.1)"
      ],
      "metadata": {
        "id": "UhmiUroXMfHB",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 227
        },
        "outputId": "8f96a3a0-76bb-45ee-cc9e-74f863fb7fa9"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<PIL.Image.Image image mode=RGB size=160x210>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAKAAAADSCAIAAABCR1ywAAACpElEQVR4nO3bMW4TURRA0QS5RqyAioIlpKaIpmA1XgEr8DIiFkBhpaCMvBhEgRBFioiCdr6C7bGdXJ1Tfo3sL109zZPiXF0BALC463N+2WazefaZ9Xp98POj7xo9c4zX8vlvlrgML9fqUl98zKT+z/P8Y4LjBI4TOE7gOIHjLrZF77sJ25wPY4IBXqxr77Y27+A4geMEjhM4TuA4geMEjhM4TuA4geMEjhM4TuA4geOGP9k5xb9jcDqjP/ua4DiB4wSOEzhO4DiB4wSOEzhO4DiB4wSOEzhO4DiB4wSOEzhO4DiB4wSOEzhO4DiB4wSOEzhu+MP3adqd8x4c7WH21ATHCRwncJzAcQLHDbfoXx+eznkPTsQExwkcJ3CcwHECxw236D9vf57zHpyICY4TOE7gOIHjBI4bbtGPH23Rr8qP+WMTHCdwnMBxAscJHDfcot8/3Z3zHhztdvbUBMcJHCdwnMBxAscNt+gvXx9nz2+232fPd9OnZW7EQR7ml2gTXCdwnMBxAscJHDfcokfb8lLPs6zN7efZcxMcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHECxwkcJ3CcwHGr3+++XfoO7GGadrPn9/fzz5vgOIHjBI5bXfoC7Ge7vZk9H72bTXCcCY4YTTYAAAAAAACM/QW36j2PAeL0LQAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Create a video"
      ],
      "metadata": {
        "id": "5lKSYVw0oV7E"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "def creat_video(model,name,env):\n",
        "    states_tryout = list(env.states)\n",
        "    image_arrays = np.array(states_tryout)\n",
        "\n",
        "    # Define the video writer\n",
        "    fourcc = cv2.VideoWriter_fourcc(*'mp4v')\n",
        "\n",
        "    video_writer = cv2.VideoWriter('output_video_{}.mp4'.format(name), fourcc, 60.0, (image_arrays[0].shape[1], image_arrays[0].shape[0]))\n",
        "\n",
        "    # Write each frame to the video file\n",
        "    for image_array in image_arrays:\n",
        "        video_writer.write(image_array)\n",
        "\n",
        "    # Release the video writer\n",
        "    video_writer.release()\n",
        "\n",
        "    # Display the video in Colab\n",
        "    video_path = 'output_video_{}.mp4'.format(name)\n",
        "    video_file = io.open(video_path, 'r+b').read()\n",
        "    encoded = base64.b64encode(video_file)\n",
        "    HTML(data='''<video alt=\"test\" controls>\n",
        "                    <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n",
        "                </video>'''.format(encoded.decode('ascii')))\n",
        "    return states_tryout\n",
        "\n",
        "import os\n",
        "import glob\n",
        "\n",
        "def delete_videos_by_name(name_pattern):\n",
        "    video_files = glob.glob(f'output_video_{name_pattern}.mp4')\n",
        "\n",
        "    for video_file in video_files:\n",
        "        try:\n",
        "            os.remove(video_file)\n",
        "            print(f\"Deleted: {video_file}\")\n",
        "        except OSError as e:\n",
        "            print(f\"Error deleting {video_file}: {e}\")\n",
        "\n",
        "# Replace 'your_name_pattern' with the actual name pattern you want to delete\n",
        "delete_videos_by_name('your_name_pattern')"
      ],
      "metadata": {
        "id": "KDGcz6PeC0GW"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "states_try = creat_video(q_network,1,env)"
      ],
      "metadata": {
        "id": "nS9qFAboGj3B"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Show import and install requirments of the code"
      ],
      "metadata": {
        "id": "v1zkNb7IYSMy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%pip list\n",
        "%pip freeze > requirements.txt"
      ],
      "metadata": {
        "id": "3EjiHfd2rEk5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d841c798-53fe-4203-8d20-0756a78738ff"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
            "  and should_run_async(code)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Package                          Version\n",
            "-------------------------------- ---------------------\n",
            "absl-py                          1.4.0\n",
            "aiohttp                          3.9.1\n",
            "aiosignal                        1.3.1\n",
            "alabaster                        0.7.13\n",
            "albumentations                   1.3.1\n",
            "ale-py                           0.7.5\n",
            "altair                           4.2.2\n",
            "anyio                            3.7.1\n",
            "appdirs                          1.4.4\n",
            "argon2-cffi                      23.1.0\n",
            "argon2-cffi-bindings             21.2.0\n",
            "array-record                     0.5.0\n",
            "arviz                            0.15.1\n",
            "astropy                          5.3.4\n",
            "astunparse                       1.6.3\n",
            "async-timeout                    4.0.3\n",
            "atpublic                         4.0\n",
            "attrs                            23.1.0\n",
            "audioread                        3.0.1\n",
            "autograd                         1.6.2\n",
            "AutoROM                          0.4.2\n",
            "AutoROM.accept-rom-license       0.6.1\n",
            "Babel                            2.14.0\n",
            "backcall                         0.2.0\n",
            "beautifulsoup4                   4.11.2\n",
            "bidict                           0.22.1\n",
            "bigframes                        0.17.0\n",
            "bleach                           6.1.0\n",
            "blinker                          1.4\n",
            "blis                             0.7.11\n",
            "blosc2                           2.0.0\n",
            "bokeh                            3.3.2\n",
            "bqplot                           0.12.42\n",
            "branca                           0.7.0\n",
            "build                            1.0.3\n",
            "CacheControl                     0.13.1\n",
            "cachetools                       5.3.2\n",
            "catalogue                        2.0.10\n",
            "certifi                          2023.11.17\n",
            "cffi                             1.16.0\n",
            "chardet                          5.2.0\n",
            "charset-normalizer               3.3.2\n",
            "chex                             0.1.7\n",
            "click                            8.1.7\n",
            "click-plugins                    1.1.1\n",
            "cligj                            0.7.2\n",
            "cloudpickle                      2.2.1\n",
            "cmake                            3.27.9\n",
            "cmdstanpy                        1.2.0\n",
            "colorcet                         3.0.1\n",
            "colorlover                       0.3.0\n",
            "colour                           0.1.5\n",
            "community                        1.0.0b1\n",
            "confection                       0.1.4\n",
            "cons                             0.4.6\n",
            "contextlib2                      21.6.0\n",
            "contourpy                        1.2.0\n",
            "cryptography                     41.0.7\n",
            "cufflinks                        0.17.3\n",
            "cupy-cuda12x                     12.2.0\n",
            "cvxopt                           1.3.2\n",
            "cvxpy                            1.3.2\n",
            "cycler                           0.12.1\n",
            "cymem                            2.0.8\n",
            "Cython                           3.0.6\n",
            "dask                             2023.8.1\n",
            "datascience                      0.17.6\n",
            "db-dtypes                        1.2.0\n",
            "dbus-python                      1.2.18\n",
            "debugpy                          1.6.6\n",
            "decorator                        4.4.2\n",
            "defusedxml                       0.7.1\n",
            "diskcache                        5.6.3\n",
            "distributed                      2023.8.1\n",
            "distro                           1.7.0\n",
            "dlib                             19.24.2\n",
            "dm-tree                          0.1.8\n",
            "docutils                         0.18.1\n",
            "dopamine-rl                      4.0.6\n",
            "duckdb                           0.9.2\n",
            "earthengine-api                  0.1.384\n",
            "easydict                         1.11\n",
            "ecos                             2.0.12\n",
            "editdistance                     0.6.2\n",
            "eerepr                           0.0.4\n",
            "en-core-web-sm                   3.6.0\n",
            "entrypoints                      0.4\n",
            "et-xmlfile                       1.1.0\n",
            "etils                            1.6.0\n",
            "etuples                          0.3.9\n",
            "exceptiongroup                   1.2.0\n",
            "Farama-Notifications             0.0.4\n",
            "fastai                           2.7.13\n",
            "fastcore                         1.5.29\n",
            "fastdownload                     0.0.7\n",
            "fastjsonschema                   2.19.0\n",
            "fastprogress                     1.0.3\n",
            "fastrlock                        0.8.2\n",
            "filelock                         3.13.1\n",
            "fiona                            1.9.5\n",
            "firebase-admin                   5.3.0\n",
            "Flask                            2.2.5\n",
            "flatbuffers                      23.5.26\n",
            "flax                             0.7.5\n",
            "folium                           0.14.0\n",
            "fonttools                        4.46.0\n",
            "frozendict                       2.3.10\n",
            "frozenlist                       1.4.1\n",
            "fsspec                           2023.6.0\n",
            "future                           0.18.3\n",
            "gast                             0.5.4\n",
            "gcsfs                            2023.6.0\n",
            "GDAL                             3.4.3\n",
            "gdown                            4.6.6\n",
            "geemap                           0.29.6\n",
            "gensim                           4.3.2\n",
            "geocoder                         1.38.1\n",
            "geographiclib                    2.0\n",
            "geopandas                        0.13.2\n",
            "geopy                            2.3.0\n",
            "gin-config                       0.5.0\n",
            "glob2                            0.7\n",
            "google                           2.0.3\n",
            "google-ai-generativelanguage     0.4.0\n",
            "google-api-core                  2.11.1\n",
            "google-api-python-client         2.84.0\n",
            "google-auth                      2.17.3\n",
            "google-auth-httplib2             0.1.1\n",
            "google-auth-oauthlib             1.2.0\n",
            "google-cloud-aiplatform          1.38.1\n",
            "google-cloud-bigquery            3.12.0\n",
            "google-cloud-bigquery-connection 1.12.1\n",
            "google-cloud-bigquery-storage    2.24.0\n",
            "google-cloud-core                2.3.3\n",
            "google-cloud-datastore           2.15.2\n",
            "google-cloud-firestore           2.11.1\n",
            "google-cloud-functions           1.13.3\n",
            "google-cloud-iam                 2.13.0\n",
            "google-cloud-language            2.9.1\n",
            "google-cloud-resource-manager    1.11.0\n",
            "google-cloud-storage             2.8.0\n",
            "google-cloud-translate           3.11.3\n",
            "google-colab                     1.0.0\n",
            "google-crc32c                    1.5.0\n",
            "google-generativeai              0.3.1\n",
            "google-pasta                     0.2.0\n",
            "google-resumable-media           2.7.0\n",
            "googleapis-common-protos         1.62.0\n",
            "googledrivedownloader            0.4\n",
            "graphviz                         0.20.1\n",
            "greenlet                         3.0.2\n",
            "grpc-google-iam-v1               0.13.0\n",
            "grpcio                           1.60.0\n",
            "grpcio-status                    1.48.2\n",
            "gspread                          3.4.2\n",
            "gspread-dataframe                3.3.1\n",
            "gym                              0.25.2\n",
            "gym-notices                      0.0.8\n",
            "gymnasium                        0.29.1\n",
            "h5netcdf                         1.3.0\n",
            "h5py                             3.9.0\n",
            "holidays                         0.38\n",
            "holoviews                        1.17.1\n",
            "html5lib                         1.1\n",
            "httpimport                       1.3.1\n",
            "httplib2                         0.22.0\n",
            "huggingface-hub                  0.19.4\n",
            "humanize                         4.7.0\n",
            "hyperopt                         0.2.7\n",
            "ibis-framework                   6.2.0\n",
            "idna                             3.6\n",
            "imageio                          2.31.6\n",
            "imageio-ffmpeg                   0.4.9\n",
            "imagesize                        1.4.1\n",
            "imbalanced-learn                 0.10.1\n",
            "imgaug                           0.4.0\n",
            "importlib-metadata               7.0.0\n",
            "importlib-resources              6.1.1\n",
            "imutils                          0.5.4\n",
            "inflect                          7.0.0\n",
            "iniconfig                        2.0.0\n",
            "install                          1.3.5\n",
            "intel-openmp                     2023.2.3\n",
            "ipyevents                        2.0.2\n",
            "ipyfilechooser                   0.6.0\n",
            "ipykernel                        5.5.6\n",
            "ipyleaflet                       0.18.1\n",
            "ipython                          7.34.0\n",
            "ipython-genutils                 0.2.0\n",
            "ipython-sql                      0.5.0\n",
            "ipytree                          0.2.2\n",
            "ipywidgets                       7.7.1\n",
            "itsdangerous                     2.1.2\n",
            "jax                              0.4.23\n",
            "jaxlib                           0.4.23+cuda12.cudnn89\n",
            "jeepney                          0.7.1\n",
            "jieba                            0.42.1\n",
            "Jinja2                           3.1.2\n",
            "joblib                           1.3.2\n",
            "jsonpickle                       3.0.2\n",
            "jsonschema                       4.19.2\n",
            "jsonschema-specifications        2023.11.2\n",
            "jupyter-client                   6.1.12\n",
            "jupyter-console                  6.1.0\n",
            "jupyter_core                     5.5.1\n",
            "jupyter-server                   1.24.0\n",
            "jupyterlab_pygments              0.3.0\n",
            "jupyterlab-widgets               3.0.9\n",
            "kaggle                           1.5.16\n",
            "kagglehub                        0.1.4\n",
            "keras                            2.15.0\n",
            "keyring                          23.5.0\n",
            "kiwisolver                       1.4.5\n",
            "langcodes                        3.3.0\n",
            "launchpadlib                     1.10.16\n",
            "lazr.restfulclient               0.14.4\n",
            "lazr.uri                         1.0.6\n",
            "lazy_loader                      0.3\n",
            "libclang                         16.0.6\n",
            "librosa                          0.10.1\n",
            "lida                             0.0.10\n",
            "lightgbm                         4.1.0\n",
            "linkify-it-py                    2.0.2\n",
            "llmx                             0.0.15a0\n",
            "llvmlite                         0.41.1\n",
            "locket                           1.0.0\n",
            "logical-unification              0.4.6\n",
            "lxml                             4.9.3\n",
            "malloy                           2023.1067\n",
            "Markdown                         3.5.1\n",
            "markdown-it-py                   3.0.0\n",
            "MarkupSafe                       2.1.3\n",
            "matplotlib                       3.7.1\n",
            "matplotlib-inline                0.1.6\n",
            "matplotlib-venn                  0.11.9\n",
            "mdit-py-plugins                  0.4.0\n",
            "mdurl                            0.1.2\n",
            "miniKanren                       1.0.3\n",
            "missingno                        0.5.2\n",
            "mistune                          0.8.4\n",
            "mizani                           0.9.3\n",
            "mkl                              2023.2.0\n",
            "ml-dtypes                        0.2.0\n",
            "mlxtend                          0.22.0\n",
            "more-itertools                   10.1.0\n",
            "moviepy                          1.0.3\n",
            "mpmath                           1.3.0\n",
            "msgpack                          1.0.7\n",
            "multidict                        6.0.4\n",
            "multipledispatch                 1.0.0\n",
            "multitasking                     0.0.11\n",
            "murmurhash                       1.0.10\n",
            "music21                          9.1.0\n",
            "natsort                          8.4.0\n",
            "nbclassic                        1.0.0\n",
            "nbclient                         0.9.0\n",
            "nbconvert                        6.5.4\n",
            "nbformat                         5.9.2\n",
            "nest-asyncio                     1.5.8\n",
            "networkx                         3.2.1\n",
            "nibabel                          4.0.2\n",
            "nltk                             3.8.1\n",
            "notebook                         6.5.5\n",
            "notebook_shim                    0.2.3\n",
            "numba                            0.58.1\n",
            "numexpr                          2.8.8\n",
            "numpy                            1.23.5\n",
            "oauth2client                     4.1.3\n",
            "oauthlib                         3.2.2\n",
            "opencv-contrib-python            4.8.0.76\n",
            "opencv-python                    4.8.0.76\n",
            "opencv-python-headless           4.8.1.78\n",
            "openpyxl                         3.1.2\n",
            "opt-einsum                       3.3.0\n",
            "optax                            0.1.7\n",
            "orbax-checkpoint                 0.4.4\n",
            "osqp                             0.6.2.post8\n",
            "packaging                        23.2\n",
            "pandas                           1.5.3\n",
            "pandas-datareader                0.10.0\n",
            "pandas-gbq                       0.19.2\n",
            "pandas-stubs                     1.5.3.230304\n",
            "pandocfilters                    1.5.0\n",
            "panel                            1.3.4\n",
            "param                            2.0.1\n",
            "parso                            0.8.3\n",
            "parsy                            2.1\n",
            "partd                            1.4.1\n",
            "pathlib                          1.0.1\n",
            "pathy                            0.10.3\n",
            "patsy                            0.5.4\n",
            "peewee                           3.17.0\n",
            "pexpect                          4.9.0\n",
            "pickleshare                      0.7.5\n",
            "Pillow                           9.4.0\n",
            "pip                              23.1.2\n",
            "pip-tools                        6.13.0\n",
            "platformdirs                     4.1.0\n",
            "plotly                           5.15.0\n",
            "plotnine                         0.12.4\n",
            "pluggy                           1.3.0\n",
            "polars                           0.17.3\n",
            "pooch                            1.8.0\n",
            "portpicker                       1.5.2\n",
            "prefetch-generator               1.0.3\n",
            "preshed                          3.0.9\n",
            "prettytable                      3.9.0\n",
            "proglog                          0.1.10\n",
            "progressbar2                     4.2.0\n",
            "prometheus-client                0.19.0\n",
            "promise                          2.3\n",
            "prompt-toolkit                   3.0.43\n",
            "prophet                          1.1.5\n",
            "proto-plus                       1.23.0\n",
            "protobuf                         3.20.3\n",
            "psutil                           5.9.5\n",
            "psycopg2                         2.9.9\n",
            "ptyprocess                       0.7.0\n",
            "py-cpuinfo                       9.0.0\n",
            "py4j                             0.10.9.7\n",
            "pyarrow                          10.0.1\n",
            "pyasn1                           0.5.1\n",
            "pyasn1-modules                   0.3.0\n",
            "pycocotools                      2.0.7\n",
            "pycparser                        2.21\n",
            "pyct                             0.5.0\n",
            "pydantic                         1.10.13\n",
            "pydata-google-auth               1.8.2\n",
            "pydot                            1.4.2\n",
            "pydot-ng                         2.0.0\n",
            "pydotplus                        2.0.2\n",
            "PyDrive                          1.3.1\n",
            "PyDrive2                         1.6.3\n",
            "pyerfa                           2.0.1.1\n",
            "pygame                           2.5.2\n",
            "Pygments                         2.16.1\n",
            "PyGObject                        3.42.1\n",
            "PyJWT                            2.3.0\n",
            "pymc                             5.7.2\n",
            "pymystem3                        0.2.0\n",
            "PyOpenGL                         3.1.7\n",
            "pyOpenSSL                        23.3.0\n",
            "pyparsing                        3.1.1\n",
            "pyperclip                        1.8.2\n",
            "pyproj                           3.6.1\n",
            "pyproject_hooks                  1.0.0\n",
            "pyshp                            2.3.1\n",
            "PySocks                          1.7.1\n",
            "pytensor                         2.14.2\n",
            "pytest                           7.4.3\n",
            "python-apt                       0.0.0\n",
            "python-box                       7.1.1\n",
            "python-dateutil                  2.8.2\n",
            "python-louvain                   0.16\n",
            "python-slugify                   8.0.1\n",
            "python-utils                     3.8.1\n",
            "pytz                             2023.3.post1\n",
            "PyVirtualDisplay                 3.0\n",
            "pyviz_comms                      3.0.0\n",
            "PyWavelets                       1.5.0\n",
            "PyYAML                           6.0.1\n",
            "pyzmq                            23.2.1\n",
            "qdldl                            0.1.7.post0\n",
            "qudida                           0.0.4\n",
            "ratelim                          0.1.6\n",
            "referencing                      0.32.0\n",
            "regex                            2023.6.3\n",
            "requests                         2.31.0\n",
            "requests-oauthlib                1.3.1\n",
            "requirements-parser              0.5.0\n",
            "rich                             13.7.0\n",
            "rpds-py                          0.15.2\n",
            "rpy2                             3.4.2\n",
            "rsa                              4.9\n",
            "safetensors                      0.4.1\n",
            "scikit-image                     0.19.3\n",
            "scikit-learn                     1.2.2\n",
            "scipy                            1.11.4\n",
            "scooby                           0.9.2\n",
            "scs                              3.2.4.post1\n",
            "seaborn                          0.12.2\n",
            "SecretStorage                    3.3.1\n",
            "Send2Trash                       1.8.2\n",
            "setuptools                       67.7.2\n",
            "shapely                          2.0.2\n",
            "Shimmy                           1.3.0\n",
            "six                              1.16.0\n",
            "sklearn-pandas                   2.2.0\n",
            "smart-open                       6.4.0\n",
            "sniffio                          1.3.0\n",
            "snowballstemmer                  2.2.0\n",
            "sortedcontainers                 2.4.0\n",
            "soundfile                        0.12.1\n",
            "soupsieve                        2.5\n",
            "soxr                             0.3.7\n",
            "spacy                            3.6.1\n",
            "spacy-legacy                     3.0.12\n",
            "spacy-loggers                    1.0.5\n",
            "Sphinx                           5.0.2\n",
            "sphinxcontrib-applehelp          1.0.7\n",
            "sphinxcontrib-devhelp            1.0.5\n",
            "sphinxcontrib-htmlhelp           2.0.4\n",
            "sphinxcontrib-jsmath             1.0.1\n",
            "sphinxcontrib-qthelp             1.0.6\n",
            "sphinxcontrib-serializinghtml    1.1.9\n",
            "SQLAlchemy                       2.0.23\n",
            "sqlglot                          17.16.2\n",
            "sqlparse                         0.4.4\n",
            "srsly                            2.4.8\n",
            "stanio                           0.3.0\n",
            "statsmodels                      0.14.1\n",
            "sympy                            1.12\n",
            "tables                           3.8.0\n",
            "tabulate                         0.9.0\n",
            "tbb                              2021.11.0\n",
            "tblib                            3.0.0\n",
            "tenacity                         8.2.3\n",
            "tensorboard                      2.15.1\n",
            "tensorboard-data-server          0.7.2\n",
            "tensorflow                       2.15.0\n",
            "tensorflow-datasets              4.9.4\n",
            "tensorflow-estimator             2.15.0\n",
            "tensorflow-gcs-config            2.15.0\n",
            "tensorflow-hub                   0.15.0\n",
            "tensorflow-io-gcs-filesystem     0.34.0\n",
            "tensorflow-metadata              1.14.0\n",
            "tensorflow-probability           0.22.0\n",
            "tensorstore                      0.1.45\n",
            "termcolor                        2.4.0\n",
            "terminado                        0.18.0\n",
            "text-unidecode                   1.3\n",
            "textblob                         0.17.1\n",
            "tf-slim                          1.1.0\n",
            "thinc                            8.1.12\n",
            "threadpoolctl                    3.2.0\n",
            "tifffile                         2023.12.9\n",
            "tinycss2                         1.2.1\n",
            "tokenizers                       0.15.0\n",
            "toml                             0.10.2\n",
            "tomli                            2.0.1\n",
            "toolz                            0.12.0\n",
            "torch                            2.1.0+cu121\n",
            "torchaudio                       2.1.0+cu121\n",
            "torchdata                        0.7.0\n",
            "torchsummary                     1.5.1\n",
            "torchtext                        0.16.0\n",
            "torchvision                      0.16.0+cu121\n",
            "tornado                          6.3.2\n",
            "tqdm                             4.66.1\n",
            "traitlets                        5.7.1\n",
            "traittypes                       0.2.1\n",
            "transformers                     4.35.2\n",
            "triton                           2.1.0\n",
            "tweepy                           4.14.0\n",
            "typer                            0.9.0\n",
            "types-pytz                       2023.3.1.1\n",
            "types-setuptools                 69.0.0.0\n",
            "typing_extensions                4.5.0\n",
            "tzlocal                          5.2\n",
            "uc-micro-py                      1.0.2\n",
            "uritemplate                      4.1.1\n",
            "urllib3                          2.0.7\n",
            "vega-datasets                    0.9.0\n",
            "wadllib                          1.3.6\n",
            "wasabi                           1.1.2\n",
            "wcwidth                          0.2.12\n",
            "webcolors                        1.13\n",
            "webencodings                     0.5.1\n",
            "websocket-client                 1.7.0\n",
            "Werkzeug                         3.0.1\n",
            "wheel                            0.42.0\n",
            "widgetsnbextension               3.6.6\n",
            "wordcloud                        1.9.3\n",
            "wrapt                            1.14.1\n",
            "xarray                           2023.7.0\n",
            "xarray-einstats                  0.6.0\n",
            "xgboost                          2.0.2\n",
            "xlrd                             2.0.1\n",
            "xxhash                           3.4.1\n",
            "xyzservices                      2023.10.1\n",
            "yarl                             1.9.4\n",
            "yellowbrick                      1.5\n",
            "yfinance                         0.2.33\n",
            "zict                             3.0.0\n",
            "zipp                             3.17.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Closing and finishing"
      ],
      "metadata": {
        "id": "w9AWiBHHocV1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "env.close()\n",
        "writer.close()"
      ],
      "metadata": {
        "id": "J6gEumOqYFY5"
      },
      "execution_count": 20,
      "outputs": []
    }
  ]
}