CS224W_Colab2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XuXWJLEm2UWS"
   },
   "source": [
    "# **CS224W - Colab 2**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8gzsP50bF6Gb"
   },
   "source": [
    "In Colab 2, we will work to construct our own graph neural network using PyTorch Geometric (PyG) and then apply that model on two Open Graph Benchmark (OGB) datasets. These two datasets will be used to benchmark your model's performance on two different graph-based tasks: 1) node property prediction, predicting properties of single nodes and 2) graph property prediction, predicting properties of entire graphs or subgraphs.\n",
    "\n",
    "First, we will learn how PyTorch Geometric stores graphs as PyTorch tensors.\n",
    "\n",
    "Then, we will load and inspect one of the Open Graph Benchmark (OGB) datasets by using the `ogb` package. OGB is a collection of realistic, large-scale, and diverse benchmark datasets for machine learning on graphs. The `ogb` package not only provides data loaders for each dataset but also model evaluators.\n",
    "\n",
    "Lastly, we will build our own graph neural network using PyTorch Geometric. We will then train and evaluate our model on the OGB node property prediction and graph property prediction tasks.\n",
    "\n",
    "**Note**: Make sure to **sequentially run all the cells in each section**, so that the intermediate variables / packages will carry over to the next cell\n",
    "\n",
    "We recommend you save a copy of this colab in your drive so you don't lose progress!\n",
    "\n",
    "Have fun and good luck on Colab 2 :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZGKqVEbbMEzf"
   },
   "source": [
    "# Device\n",
    "You might need to use a GPU for this Colab to run quickly.\n",
    "\n",
    "Please click `Runtime` and then `Change runtime type`. Then set the `hardware accelerator` to **GPU**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OCK7krJdp4o8"
   },
   "source": [
    "# Setup\n",
    "As discussed in Colab 0, the installation of PyG on Colab can be a little bit tricky. First let us check which version of PyTorch you are running"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "2vkP8pA1qBE5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PyTorch has version 1.10.2\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import os\n",
    "print(\"PyTorch has version {}\".format(torch.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L6d22O6DqGSZ"
   },
   "source": [
    "Download the necessary packages for PyG. Make sure that your version of torch matches the output from the cell above. In case of any issues, more information can be found on the [PyG's installation page](https://pytorch-geometric.readthedocs.io/en/latest/notes/installation.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Nwwq0nSdmsOL"
   },
   "source": [
    "# 1) PyTorch Geometric (Datasets and Data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Sf7vUmdNKCjA"
   },
   "source": [
    "PyTorch Geometric has two classes for storing and/or transforming graphs into tensor format. One is `torch_geometric.datasets`, which contains a variety of common graph datasets. Another is `torch_geometric.data`, which provides the data handling of graphs in PyTorch tensors.\n",
    "\n",
    "In this section, we will learn how to use `torch_geometric.datasets` and `torch_geometric.data` together."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ic-o1P3r6hr2"
   },
   "source": [
    "## PyG Datasets\n",
    "\n",
    "The `torch_geometric.datasets` class has many common graph datasets. Here we will explore its usage through one example dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "zT5qca3x6XpG"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ENZYMES(600)\n"
     ]
    }
   ],
   "source": [
    "from torch_geometric.datasets import TUDataset\n",
    "\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    root = './enzymes'\n",
    "    name = 'ENZYMES'\n",
    "\n",
    "    # The ENZYMES dataset\n",
    "    pyg_dataset= TUDataset(root, name)\n",
    "\n",
    "    # You will find that there are 600 graphs in this dataset\n",
    "    print(pyg_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NLm5vVYMAP2x"
   },
   "source": [
    "## Question 1: What is the number of classes and number of features in the ENZYMES dataset? (5 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "8iF_Kyqr_JbY"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ENZYMES dataset has 6 classes\n",
      "ENZYMES dataset has 3 features\n"
     ]
    }
   ],
   "source": [
    "def get_num_classes(pyg_dataset):\n",
    "    # TODO: Implement a function that takes a PyG dataset object\n",
    "    # and returns the number of classes for that dataset.\n",
    "\n",
    "    num_classes = 0\n",
    "\n",
    "    ############# Your code here ############\n",
    "    ## (~1 line of code)\n",
    "    ## Note\n",
    "    ## 1. Colab autocomplete functionality might be useful.\n",
    "    num_classes = pyg_dataset.num_classes\n",
    "    #########################################\n",
    "\n",
    "    return num_classes\n",
    "\n",
    "def get_num_features(pyg_dataset):\n",
    "    # TODO: Implement a function that takes a PyG dataset object\n",
    "    # and returns the number of features for that dataset.\n",
    "\n",
    "    num_features = 0\n",
    "\n",
    "    ############# Your code here ############\n",
    "    ## (~1 line of code)\n",
    "    ## Note\n",
    "    ## 1. Colab autocomplete functionality might be useful.\n",
    "    num_features = pyg_dataset.num_features\n",
    "    #########################################\n",
    "\n",
    "    return num_features\n",
    "\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    num_classes = get_num_classes(pyg_dataset)\n",
    "    num_features = get_num_features(pyg_dataset)\n",
    "    print(\"{} dataset has {} classes\".format(name, num_classes))\n",
    "    print(\"{} dataset has {} features\".format(name, num_features))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rwKbzhHUAckZ"
   },
   "source": [
    "## PyG Data\n",
    "\n",
    "Each PyG dataset stores a list of `torch_geometric.data.Data` objects, where each `torch_geometric.data.Data` object represents a graph. We can easily get the `Data` object by indexing into the dataset.\n",
    "\n",
    "For more information such as what is stored in the `Data` object, please refer to the [documentation](https://pytorch-geometric.readthedocs.io/en/latest/modules/data.html#torch_geometric.data.Data)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7sCV3xJWCddX"
   },
   "source": [
    "## Question 2: What is the label of the graph with index 100 in the ENZYMES dataset? (5 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "LIis9oTZAfs3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data(edge_index=[2, 168], x=[37, 3], y=[1])\n",
      "Graph with index 100 has label 4\n"
     ]
    }
   ],
   "source": [
    "def get_graph_class(pyg_dataset, idx):\n",
    "    # TODO: Implement a function that takes a PyG dataset object,\n",
    "    # an index of a graph within the dataset, and returns the class/label \n",
    "    # of the graph (as an integer).\n",
    "\n",
    "    label = -1\n",
    "\n",
    "    ############# Your code here ############\n",
    "    ## (~1 line of code)\n",
    "    label = pyg_dataset[idx].y.item()\n",
    "    #########################################\n",
    "\n",
    "    return label\n",
    "\n",
    "# Here pyg_dataset is a dataset for graph classification\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    graph_0 = pyg_dataset[0]\n",
    "    print(graph_0)\n",
    "    idx = 100\n",
    "    label = get_graph_class(pyg_dataset, idx)\n",
    "    print('Graph with index {} has label {}'.format(idx, label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fKhcVeAhCwoY"
   },
   "source": [
    "## Question 3: How many edges does the graph with index 200 have? (5 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "f5m2DOfhBtWv"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Graph with index 200 has 53 edges\n"
     ]
    }
   ],
   "source": [
    "def get_graph_num_edges(pyg_dataset, idx):\n",
    "    # TODO: Implement a function that takes a PyG dataset object,\n",
    "    # the index of a graph in the dataset, and returns the number of \n",
    "    # edges in the graph (as an integer). You should not count an edge \n",
    "    # twice if the graph is undirected. For example, in an undirected \n",
    "    # graph G, if two nodes v and u are connected by an edge, this edge\n",
    "    # should only be counted once.\n",
    "\n",
    "    num_edges = 0\n",
    "\n",
    "    ############# Your code here ############\n",
    "    ## Note:\n",
    "    ## 1. You can't return the data.num_edges directly\n",
    "    ## 2. We assume the graph is undirected\n",
    "    ## 3. Look at the PyG dataset built in functions\n",
    "    ## (~4 lines of code)\n",
    "    num_edges = pyg_dataset[idx].num_edges\n",
    "    num_edges //= 2\n",
    "    #########################################\n",
    "\n",
    "    return num_edges\n",
    "\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    idx = 200\n",
    "    num_edges = get_graph_num_edges(pyg_dataset, idx)\n",
    "    print('Graph with index {} has {} edges'.format(idx, num_edges))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AXa7yIG4E0Fp"
   },
   "source": [
    "# 2) Open Graph Benchmark (OGB)\n",
    "\n",
    "The Open Graph Benchmark (OGB) is a collection of realistic, large-scale, and diverse benchmark datasets for machine learning on graphs. Its datasets are automatically downloaded, processed, and split using the OGB Data Loader. The model performance can then be evaluated by using the OGB Evaluator in a unified manner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HnazPGGAJAZN"
   },
   "source": [
    "## Dataset and Data\n",
    "\n",
    "OGB also supports PyG dataset and data classes. Here we take a look on the `ogbn-arxiv` dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "Gpc6bTm3GF02"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The ogbn-arxiv dataset has 1 graph\n",
      "Data(num_nodes=169343, x=[169343, 128], node_year=[169343, 1], y=[169343, 1], adj_t=[169343, 169343, nnz=1166243])\n"
     ]
    }
   ],
   "source": [
    "import torch_geometric.transforms as T\n",
    "from ogb.nodeproppred import PygNodePropPredDataset\n",
    "\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    dataset_name = 'ogbn-arxiv'\n",
    "    # Load the dataset and transform it to sparse tensor\n",
    "    dataset = PygNodePropPredDataset(name=dataset_name,\n",
    "                                  transform=T.ToSparseTensor())\n",
    "    print('The {} dataset has {} graph'.format(dataset_name, len(dataset)))\n",
    "\n",
    "    # Extract the graph\n",
    "    data = dataset[0]\n",
    "    print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Cw0xZJKZI-n3"
   },
   "source": [
    "## Question 4: How many features are in the ogbn-arxiv graph? (5 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "ZP844_nT2ZJl"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The graph has 128 features\n"
     ]
    }
   ],
   "source": [
    "def graph_num_features(data):\n",
    "    # TODO: Implement a function that takes a PyG data object,\n",
    "    # and returns the number of features in the graph (as an integer).\n",
    "\n",
    "    num_features = 0\n",
    "\n",
    "    ############# Your code here ############\n",
    "    ## (~1 line of code)\n",
    "    num_features = data.num_features\n",
    "    #########################################\n",
    "\n",
    "    return num_features\n",
    "\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    num_features = graph_num_features(data)\n",
    "    print('The graph has {} features'.format(num_features))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9DP_yEQZ0NVW"
   },
   "source": [
    "# 3) GNN: Node Property Prediction\n",
    "\n",
    "In this section we will build our first graph neural network using PyTorch Geometric. Then we will apply it to the task of node property prediction (node classification).\n",
    "\n",
    "Specifically, we will use GCN as the foundation for your graph neural network ([Kipf et al. (2017)](https://arxiv.org/pdf/1609.02907.pdf)). To do so, we will work with PyG's built-in `GCNConv` layer. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "O4CcOUEoInjD"
   },
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "-DCtgcHpGIpd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.10.2\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pandas as pd\n",
    "import torch.nn.functional as F\n",
    "print(torch.__version__)\n",
    "\n",
    "# The PyG built-in GCNConv\n",
    "from torch_geometric.nn import GCNConv\n",
    "\n",
    "import torch_geometric.transforms as T\n",
    "from ogb.nodeproppred import PygNodePropPredDataset, Evaluator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0IK9z0wQIwzQ"
   },
   "source": [
    "## Load and Preprocess the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "0ibJ0ieoIwQM"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device: cuda\n"
     ]
    }
   ],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    dataset_name = 'ogbn-arxiv'\n",
    "    dataset = PygNodePropPredDataset(name=dataset_name,\n",
    "                                  transform=T.ToSparseTensor())\n",
    "    data = dataset[0]\n",
    "\n",
    "    # Make the adjacency matrix to symmetric\n",
    "    data.adj_t = data.adj_t.to_symmetric()\n",
    "\n",
    "    device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "\n",
    "    # If you use GPU, the device should be cuda\n",
    "    print('Device: {}'.format(device))\n",
    "\n",
    "    data = data.to(device)\n",
    "    split_idx = dataset.get_idx_split()\n",
    "    train_idx = split_idx['train'].to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OgUA815bNJ8w"
   },
   "source": [
    "## GCN Model\n",
    "\n",
    "Now we will implement our GCN model!\n",
    "\n",
    "Please follow the figure below to implement the `forward` function.\n",
    "\n",
    "\n",
    "![test](https://drive.google.com/uc?id=128AuYAXNXGg7PIhJJ7e420DoPWKb-RtL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch_geometric.nn import Sequential, GCNConv\n",
    "from torch.nn import BatchNorm1d, Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "IgspXTYpNJLA"
   },
   "outputs": [],
   "source": [
    "class GCN(torch.nn.Module):\n",
    "    def __init__(self, input_dim, hidden_dim, output_dim, num_layers,\n",
    "                 dropout, return_embeds=False):\n",
    "        # TODO: Implement a function that initializes self.convs, \n",
    "        # self.bns, and self.softmax.\n",
    "\n",
    "        super(GCN, self).__init__()\n",
    "\n",
    "        # A list of GCNConv layers\n",
    "        self.convs = None\n",
    "\n",
    "        # A list of 1D batch normalization layers\n",
    "        self.bns = None\n",
    "\n",
    "        # The log softmax layer\n",
    "        self.softmax = None\n",
    "\n",
    "        ############# Your code here ############\n",
    "        ## Note:\n",
    "        ## 1. You should use torch.nn.ModuleList for self.convs and self.bns\n",
    "        ## 2. self.convs has num_layers GCNConv layers\n",
    "        ## 3. self.bns has num_layers - 1 BatchNorm1d layers\n",
    "        ## 4. You should use torch.nn.LogSoftmax for self.softmax\n",
    "        ## 5. The parameters you can set for GCNConv include 'in_channels' and \n",
    "        ## 'out_channels'. For more information please refer to the documentation:\n",
    "        ## https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.conv.GCNConv\n",
    "        ## 6. The only parameter you need to set for BatchNorm1d is 'num_features'\n",
    "        ## For more information please refer to the documentation: \n",
    "        ## https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html\n",
    "        ## (~10 lines of code)\n",
    "        self.num_layers = num_layers\n",
    "        \n",
    "        self.convs = torch.nn.ModuleList([GCNConv(input_dim, hidden_dim)])\n",
    "        self.convs.extend([\n",
    "            GCNConv(hidden_dim, hidden_dim) for i in range(1, self.num_layers-1)\n",
    "        ])\n",
    "        self.convs.append(GCNConv(hidden_dim, output_dim))\n",
    "        \n",
    "        self.bns = torch.nn.ModuleList([\n",
    "            BatchNorm1d(num_features=hidden_dim) for i in range(self.num_layers-1)\n",
    "        ])\n",
    "\n",
    "        self.softmax = torch.nn.LogSoftmax(dim=-1)\n",
    "        #########################################\n",
    "\n",
    "        # Probability of an element getting zeroed\n",
    "        self.dropout = dropout\n",
    "\n",
    "        # Skip classification layer and return node embeddings\n",
    "        self.return_embeds = return_embeds\n",
    "\n",
    "    def reset_parameters(self):\n",
    "        for conv in self.convs:\n",
    "            conv.reset_parameters()\n",
    "        for bn in self.bns:\n",
    "            bn.reset_parameters()\n",
    "\n",
    "    def forward(self, x, adj_t):\n",
    "        # TODO: Implement a function that takes the feature tensor x and\n",
    "        # edge_index tensor adj_t and returns the output tensor as\n",
    "        # shown in the figure.\n",
    "\n",
    "        out = None\n",
    "\n",
    "        ############# Your code here ############\n",
    "        ## Note:\n",
    "        ## 1. Construct the network as shown in the figure\n",
    "        ## 2. torch.nn.functional.relu and torch.nn.functional.dropout are useful\n",
    "        ## For more information please refer to the documentation:\n",
    "        ## https://pytorch.org/docs/stable/nn.functional.html\n",
    "        ## 3. Don't forget to set F.dropout training to self.training\n",
    "        ## 4. If return_embeds is True, then skip the last softmax layer\n",
    "        ## (~7 lines of code)\n",
    "        out = x\n",
    "        for i in range(self.num_layers-1):\n",
    "            out = self.convs[i](out, adj_t)\n",
    "            out = self.bns[i](out)\n",
    "            out = F.relu(out)\n",
    "            out = F.dropout(out, p=self.dropout, training=self.training)\n",
    "        out = self.convs[-1](out, adj_t)\n",
    "        if not self.return_embeds:\n",
    "            out = self.softmax(out)\n",
    "\n",
    "        #########################################\n",
    "\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "FF1hnHUhO81e"
   },
   "outputs": [],
   "source": [
    "def train(model, data, train_idx, optimizer, loss_fn):\n",
    "    # TODO: Implement a function that trains the model by \n",
    "    # using the given optimizer and loss_fn.\n",
    "    model.train()\n",
    "    loss = 0\n",
    "\n",
    "    ############# Your code here ############\n",
    "    ## Note:\n",
    "    ## 1. Zero grad the optimizer\n",
    "    ## 2. Feed the data into the model\n",
    "    ## 3. Slice the model output and label by train_idx\n",
    "    ## 4. Feed the sliced output and label to loss_fn\n",
    "    ## (~4 lines of code)\n",
    "    optimizer.zero_grad()\n",
    "    pred = model(data.x, data.adj_t)\n",
    "    loss = loss_fn(pred[train_idx], data.y[train_idx].reshape(-1))\n",
    "    \n",
    "#     if i%10 == 9:\n",
    "#         print(f'Epoch {i+1}/{epochs}: loss: {loss.item():.3f} - accuracy: {acc:.3f}')\n",
    "    #########################################\n",
    "\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    return loss.item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "aJdlrJQhPBsK"
   },
   "outputs": [],
   "source": [
    "# Test function here\n",
    "@torch.no_grad()\n",
    "def test(model, data, split_idx, evaluator, save_model_results=False):\n",
    "    # TODO: Implement a function that tests the model by \n",
    "    # using the given split_idx and evaluator.\n",
    "    model.eval()\n",
    "\n",
    "    # The output of model on all data\n",
    "    out = None\n",
    "\n",
    "    ############# Your code here ############\n",
    "    ## (~1 line of code)\n",
    "    ## Note:\n",
    "    ## 1. No index slicing here\n",
    "    out = model(data.x, data.adj_t)\n",
    "\n",
    "    #########################################\n",
    "\n",
    "    y_pred = out.argmax(dim=-1, keepdim=True)\n",
    "\n",
    "    train_acc = evaluator.eval({\n",
    "        'y_true': data.y[split_idx['train']],\n",
    "        'y_pred': y_pred[split_idx['train']],\n",
    "    })['acc']\n",
    "    valid_acc = evaluator.eval({\n",
    "        'y_true': data.y[split_idx['valid']],\n",
    "        'y_pred': y_pred[split_idx['valid']],\n",
    "    })['acc']\n",
    "    test_acc = evaluator.eval({\n",
    "        'y_true': data.y[split_idx['test']],\n",
    "        'y_pred': y_pred[split_idx['test']],\n",
    "    })['acc']\n",
    "\n",
    "    if save_model_results:\n",
    "        print (\"Saving Model Predictions\")\n",
    "\n",
    "        data = {}\n",
    "        data['y_pred'] = y_pred.view(-1).cpu().detach().numpy()\n",
    "\n",
    "        df = pd.DataFrame(data=data)\n",
    "        # Save locally as csv\n",
    "        df.to_csv('ogbn-arxiv_node.csv', sep=',', index=False)\n",
    "\n",
    "\n",
    "    return train_acc, valid_acc, test_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "o7F46xkuLiOL"
   },
   "outputs": [],
   "source": [
    "# Please do not change the args\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    args = {\n",
    "        'device': device,\n",
    "        'num_layers': 3,\n",
    "        'hidden_dim': 256,\n",
    "        'dropout': 0.5,\n",
    "        'lr': 0.01,\n",
    "        'epochs': 100,\n",
    "    }\n",
    "    args"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "dT8RyM2cPGxM"
   },
   "outputs": [],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    model = GCN(data.num_features, args['hidden_dim'],\n",
    "                dataset.num_classes, args['num_layers'],\n",
    "                args['dropout']).to(device)\n",
    "    evaluator = Evaluator(name='ogbn-arxiv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.nn import Linear, ReLU, Dropout\n",
    "model = Sequential('x, edge_index', [\n",
    "    (GCNConv(data.num_features, 256), 'x, edge_index -> x'),\n",
    "    (BatchNorm1d(num_features=256), 'x -> x'),\n",
    "    ReLU(inplace=True),\n",
    "    (Dropout(p=0.5), 'x -> x'),\n",
    "    (GCNConv(256, 256), 'x, edge_index -> x'),\n",
    "    (BatchNorm1d(num_features=256), 'x -> x'),\n",
    "    ReLU(inplace=True),\n",
    "    (Dropout(p=0.5), 'x -> x'),\n",
    "    (GCNConv(256, 256), 'x, edge_index -> x'),\n",
    "    (BatchNorm1d(num_features=256), 'x -> x'),\n",
    "    ReLU(inplace=True),\n",
    "    (Dropout(p=0.5), 'x -> x'),\n",
    "    (GCNConv(256, dataset.num_classes), 'x, edge_index -> x'),\n",
    "    (torch.nn.LogSoftmax(dim=-1), 'x -> x')\n",
    "]).to(device)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {
    "id": "qd5O5cnPPdVF"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01, Loss: 4.0472, Train: 27.12%, Valid: 29.82% Test: 26.68%\n",
      "Epoch: 02, Loss: 2.4399, Train: 26.88%, Valid: 31.05% Test: 29.70%\n",
      "Epoch: 03, Loss: 2.2892, Train: 38.63%, Valid: 41.24% Test: 38.52%\n",
      "Epoch: 04, Loss: 1.9023, Train: 43.01%, Valid: 44.94% Test: 42.75%\n",
      "Epoch: 05, Loss: 1.8165, Train: 46.31%, Valid: 51.13% Test: 51.93%\n",
      "Epoch: 06, Loss: 1.6822, Train: 42.24%, Valid: 40.35% Test: 43.88%\n",
      "Epoch: 07, Loss: 1.5802, Train: 39.23%, Valid: 28.90% Test: 26.97%\n",
      "Epoch: 08, Loss: 1.5184, Train: 34.51%, Valid: 24.03% Test: 21.81%\n",
      "Epoch: 09, Loss: 1.4840, Train: 34.21%, Valid: 25.72% Test: 24.54%\n",
      "Epoch: 10, Loss: 1.4438, Train: 39.78%, Valid: 35.80% Test: 36.30%\n",
      "Epoch: 11, Loss: 1.4070, Train: 44.72%, Valid: 40.95% Test: 42.94%\n",
      "Epoch: 12, Loss: 1.3757, Train: 45.53%, Valid: 40.68% Test: 43.75%\n",
      "Epoch: 13, Loss: 1.3462, Train: 44.50%, Valid: 38.48% Test: 41.54%\n",
      "Epoch: 14, Loss: 1.3302, Train: 44.37%, Valid: 38.22% Test: 40.77%\n",
      "Epoch: 15, Loss: 1.3071, Train: 46.14%, Valid: 41.95% Test: 44.02%\n",
      "Epoch: 16, Loss: 1.2852, Train: 48.49%, Valid: 46.33% Test: 48.16%\n",
      "Epoch: 17, Loss: 1.2652, Train: 50.26%, Valid: 48.83% Test: 50.68%\n",
      "Epoch: 18, Loss: 1.2509, Train: 50.50%, Valid: 47.63% Test: 50.20%\n",
      "Epoch: 19, Loss: 1.2301, Train: 50.72%, Valid: 46.36% Test: 49.12%\n",
      "Epoch: 20, Loss: 1.2222, Train: 51.19%, Valid: 45.80% Test: 48.44%\n",
      "Epoch: 21, Loss: 1.2110, Train: 52.19%, Valid: 46.99% Test: 49.05%\n",
      "Epoch: 22, Loss: 1.1943, Train: 52.82%, Valid: 47.66% Test: 49.35%\n",
      "Epoch: 23, Loss: 1.1831, Train: 53.03%, Valid: 47.32% Test: 48.95%\n",
      "Epoch: 24, Loss: 1.1792, Train: 53.09%, Valid: 46.23% Test: 48.04%\n",
      "Epoch: 25, Loss: 1.1640, Train: 53.89%, Valid: 46.39% Test: 48.30%\n",
      "Epoch: 26, Loss: 1.1555, Train: 55.37%, Valid: 47.85% Test: 49.68%\n",
      "Epoch: 27, Loss: 1.1471, Train: 57.67%, Valid: 51.50% Test: 53.01%\n",
      "Epoch: 28, Loss: 1.1405, Train: 60.28%, Valid: 55.73% Test: 56.65%\n",
      "Epoch: 29, Loss: 1.1318, Train: 62.16%, Valid: 58.94% Test: 59.55%\n",
      "Epoch: 30, Loss: 1.1229, Train: 63.64%, Valid: 60.91% Test: 61.26%\n",
      "Epoch: 31, Loss: 1.1164, Train: 64.65%, Valid: 62.14% Test: 61.95%\n",
      "Epoch: 32, Loss: 1.1093, Train: 65.47%, Valid: 63.25% Test: 62.70%\n",
      "Epoch: 33, Loss: 1.1075, Train: 65.94%, Valid: 64.05% Test: 63.49%\n",
      "Epoch: 34, Loss: 1.0980, Train: 66.41%, Valid: 65.49% Test: 64.91%\n",
      "Epoch: 35, Loss: 1.0913, Train: 67.01%, Valid: 66.40% Test: 66.25%\n",
      "Epoch: 36, Loss: 1.0869, Train: 67.25%, Valid: 66.64% Test: 66.47%\n",
      "Epoch: 37, Loss: 1.0804, Train: 67.70%, Valid: 67.04% Test: 66.71%\n",
      "Epoch: 38, Loss: 1.0755, Train: 68.16%, Valid: 67.32% Test: 66.94%\n",
      "Epoch: 39, Loss: 1.0698, Train: 68.44%, Valid: 67.69% Test: 67.01%\n",
      "Epoch: 40, Loss: 1.0669, Train: 68.78%, Valid: 68.12% Test: 67.48%\n",
      "Epoch: 41, Loss: 1.0623, Train: 69.06%, Valid: 68.32% Test: 67.79%\n",
      "Epoch: 42, Loss: 1.0558, Train: 69.46%, Valid: 68.99% Test: 68.33%\n",
      "Epoch: 43, Loss: 1.0512, Train: 69.98%, Valid: 69.49% Test: 68.14%\n",
      "Epoch: 44, Loss: 1.0478, Train: 70.23%, Valid: 69.51% Test: 67.69%\n",
      "Epoch: 45, Loss: 1.0456, Train: 70.28%, Valid: 69.40% Test: 67.52%\n",
      "Epoch: 46, Loss: 1.0415, Train: 70.22%, Valid: 68.98% Test: 67.28%\n",
      "Epoch: 47, Loss: 1.0396, Train: 70.19%, Valid: 68.55% Test: 67.18%\n",
      "Epoch: 48, Loss: 1.0352, Train: 70.45%, Valid: 68.77% Test: 67.46%\n",
      "Epoch: 49, Loss: 1.0336, Train: 70.67%, Valid: 69.06% Test: 67.33%\n",
      "Epoch: 50, Loss: 1.0295, Train: 70.85%, Valid: 69.51% Test: 67.77%\n",
      "Epoch: 51, Loss: 1.0268, Train: 70.98%, Valid: 70.06% Test: 68.82%\n",
      "Epoch: 52, Loss: 1.0202, Train: 70.82%, Valid: 69.85% Test: 69.03%\n",
      "Epoch: 53, Loss: 1.0215, Train: 70.92%, Valid: 69.96% Test: 69.29%\n",
      "Epoch: 54, Loss: 1.0161, Train: 71.30%, Valid: 70.54% Test: 69.43%\n",
      "Epoch: 55, Loss: 1.0108, Train: 71.41%, Valid: 69.85% Test: 67.78%\n",
      "Epoch: 56, Loss: 1.0107, Train: 71.20%, Valid: 69.20% Test: 66.64%\n",
      "Epoch: 57, Loss: 1.0084, Train: 71.44%, Valid: 69.97% Test: 67.95%\n",
      "Epoch: 58, Loss: 1.0031, Train: 71.47%, Valid: 70.34% Test: 69.07%\n",
      "Epoch: 59, Loss: 1.0041, Train: 71.46%, Valid: 70.00% Test: 68.93%\n",
      "Epoch: 60, Loss: 0.9999, Train: 71.74%, Valid: 69.91% Test: 68.52%\n",
      "Epoch: 61, Loss: 0.9973, Train: 71.93%, Valid: 70.09% Test: 68.60%\n",
      "Epoch: 62, Loss: 0.9958, Train: 71.99%, Valid: 70.32% Test: 69.09%\n",
      "Epoch: 63, Loss: 0.9916, Train: 71.86%, Valid: 70.47% Test: 69.63%\n",
      "Epoch: 64, Loss: 0.9915, Train: 71.79%, Valid: 70.33% Test: 69.76%\n",
      "Epoch: 65, Loss: 0.9861, Train: 72.14%, Valid: 70.95% Test: 70.08%\n",
      "Epoch: 66, Loss: 0.9838, Train: 72.23%, Valid: 70.39% Test: 68.85%\n",
      "Epoch: 67, Loss: 0.9808, Train: 72.04%, Valid: 69.79% Test: 67.91%\n",
      "Epoch: 68, Loss: 0.9785, Train: 72.03%, Valid: 70.02% Test: 68.58%\n",
      "Epoch: 69, Loss: 0.9756, Train: 72.13%, Valid: 70.42% Test: 69.12%\n",
      "Epoch: 70, Loss: 0.9730, Train: 72.32%, Valid: 70.59% Test: 68.73%\n",
      "Epoch: 71, Loss: 0.9734, Train: 72.50%, Valid: 70.91% Test: 68.88%\n",
      "Epoch: 72, Loss: 0.9719, Train: 72.57%, Valid: 71.38% Test: 70.13%\n",
      "Epoch: 73, Loss: 0.9712, Train: 72.58%, Valid: 71.37% Test: 70.15%\n",
      "Epoch: 74, Loss: 0.9680, Train: 72.69%, Valid: 71.50% Test: 70.02%\n",
      "Epoch: 75, Loss: 0.9645, Train: 72.78%, Valid: 71.47% Test: 69.87%\n",
      "Epoch: 76, Loss: 0.9639, Train: 72.80%, Valid: 71.74% Test: 70.72%\n",
      "Epoch: 77, Loss: 0.9620, Train: 72.70%, Valid: 71.53% Test: 70.78%\n",
      "Epoch: 78, Loss: 0.9600, Train: 72.75%, Valid: 71.35% Test: 69.92%\n",
      "Epoch: 79, Loss: 0.9611, Train: 72.58%, Valid: 70.04% Test: 67.42%\n",
      "Epoch: 80, Loss: 0.9591, Train: 72.60%, Valid: 69.76% Test: 67.00%\n",
      "Epoch: 81, Loss: 0.9557, Train: 72.84%, Valid: 70.68% Test: 68.91%\n",
      "Epoch: 82, Loss: 0.9511, Train: 72.98%, Valid: 71.29% Test: 70.05%\n",
      "Epoch: 83, Loss: 0.9481, Train: 73.02%, Valid: 71.52% Test: 70.00%\n",
      "Epoch: 84, Loss: 0.9473, Train: 73.00%, Valid: 71.63% Test: 70.46%\n",
      "Epoch: 85, Loss: 0.9472, Train: 72.74%, Valid: 71.25% Test: 70.49%\n",
      "Epoch: 86, Loss: 0.9467, Train: 72.86%, Valid: 71.13% Test: 69.85%\n",
      "Epoch: 87, Loss: 0.9472, Train: 72.76%, Valid: 70.74% Test: 69.31%\n",
      "Epoch: 88, Loss: 0.9402, Train: 72.78%, Valid: 70.64% Test: 69.43%\n",
      "Epoch: 89, Loss: 0.9377, Train: 73.18%, Valid: 71.72% Test: 70.76%\n",
      "Epoch: 90, Loss: 0.9394, Train: 73.15%, Valid: 71.54% Test: 70.11%\n",
      "Epoch: 91, Loss: 0.9355, Train: 72.97%, Valid: 70.91% Test: 69.23%\n",
      "Epoch: 92, Loss: 0.9360, Train: 73.19%, Valid: 71.64% Test: 70.37%\n",
      "Epoch: 93, Loss: 0.9332, Train: 73.34%, Valid: 71.40% Test: 70.23%\n",
      "Epoch: 94, Loss: 0.9309, Train: 73.49%, Valid: 71.31% Test: 69.92%\n",
      "Epoch: 95, Loss: 0.9296, Train: 73.56%, Valid: 71.62% Test: 70.08%\n",
      "Epoch: 96, Loss: 0.9309, Train: 73.57%, Valid: 71.70% Test: 70.34%\n",
      "Epoch: 97, Loss: 0.9260, Train: 73.58%, Valid: 71.60% Test: 70.03%\n",
      "Epoch: 98, Loss: 0.9241, Train: 73.58%, Valid: 71.44% Test: 69.63%\n",
      "Epoch: 99, Loss: 0.9261, Train: 73.54%, Valid: 71.53% Test: 69.73%\n",
      "Epoch: 100, Loss: 0.9230, Train: 73.49%, Valid: 71.34% Test: 69.87%\n"
     ]
    }
   ],
   "source": [
    "import copy\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    # reset the parameters to initial random value\n",
    "    model.reset_parameters()\n",
    "\n",
    "    optimizer = torch.optim.Adam(model.parameters(), lr=args['lr'])\n",
    "    loss_fn = F.nll_loss\n",
    "\n",
    "    best_model = None\n",
    "    best_valid_acc = 0\n",
    "\n",
    "    for epoch in range(1, 1 + args[\"epochs\"]):\n",
    "        loss = train(model, data, train_idx, optimizer, loss_fn)\n",
    "        result = test(model, data, split_idx, evaluator)\n",
    "        train_acc, valid_acc, test_acc = result\n",
    "        if valid_acc > best_valid_acc:\n",
    "            best_valid_acc = valid_acc\n",
    "            best_model = copy.deepcopy(model)\n",
    "        print(f'Epoch: {epoch:02d}, '\n",
    "              f'Loss: {loss:.4f}, '\n",
    "              f'Train: {100 * train_acc:.2f}%, '\n",
    "              f'Valid: {100 * valid_acc:.2f}% '\n",
    "              f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dQtt-EKA8P4r"
   },
   "source": [
    "## Question 5: What are your `best_model` validation and test accuracies?(20 points)\n",
    "\n",
    "Run the cell below to see the results of your best of model and save your model's predictions to a file named *ogbn-arxiv_node.csv*. \n",
    "\n",
    "You can view this file by clicking on the *Folder* icon on the left side pannel. As in Colab 1, when you sumbit your assignment, you will have to download this file and attatch it to your submission."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {
    "id": "EqcextqOL2FX"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best model: Train: 72.80%, Valid: 71.74% Test: 70.72%\n"
     ]
    }
   ],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    best_result = test(best_model, data, split_idx, evaluator, save_model_results=True)\n",
    "    train_acc, valid_acc, test_acc = best_result\n",
    "    print(f'Best model: '\n",
    "          f'Train: {100 * train_acc:.2f}%, '\n",
    "          f'Valid: {100 * valid_acc:.2f}% '\n",
    "          f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "R8pOD6y80TyI"
   },
   "source": [
    "# 4) GNN: Graph Property Prediction\n",
    "\n",
    "In this section we will create a graph neural network for graph property prediction (graph classification).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vRg5VOEdQTa4"
   },
   "source": [
    "## Load and preprocess the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "LXb-O5QUIgTH"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device: cuda\n",
      "Task type: binary classification\n"
     ]
    }
   ],
   "source": [
    "from ogb.graphproppred import PygGraphPropPredDataset, Evaluator\n",
    "from torch_geometric.data import DataLoader\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    # Load the dataset \n",
    "    dataset = PygGraphPropPredDataset(name='ogbg-molhiv')\n",
    "\n",
    "    device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "    print('Device: {}'.format(device))\n",
    "\n",
    "    split_idx = dataset.get_idx_split()\n",
    "\n",
    "    # Check task type\n",
    "    print('Task type: {}'.format(dataset.task_type))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "7cHHbgW1c5hi"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/envs/pyg/lib/python3.8/site-packages/torch_geometric/deprecation.py:12: UserWarning: 'data.DataLoader' is deprecated, use 'loader.DataLoader' instead\n",
      "  warnings.warn(out)\n"
     ]
    }
   ],
   "source": [
    "# Load the dataset splits into corresponding dataloaders\n",
    "# We will train the graph classification task on a batch of 32 graphs\n",
    "# Shuffle the order of graphs for training set\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    train_loader = DataLoader(dataset[split_idx[\"train\"]], batch_size=32, shuffle=True, num_workers=0)\n",
    "    valid_loader = DataLoader(dataset[split_idx[\"valid\"]], batch_size=32, shuffle=False, num_workers=0)\n",
    "    test_loader = DataLoader(dataset[split_idx[\"test\"]], batch_size=32, shuffle=False, num_workers=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "AYrSnOj0Y4DK"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'device': 'cuda',\n",
       " 'num_layers': 5,\n",
       " 'hidden_dim': 256,\n",
       " 'dropout': 0.5,\n",
       " 'lr': 0.001,\n",
       " 'epochs': 30}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "  # Please do not change the args\n",
    "    args = {\n",
    "        'device': device,\n",
    "        'num_layers': 5,\n",
    "        'hidden_dim': 256,\n",
    "        'dropout': 0.5,\n",
    "        'lr': 0.001,\n",
    "        'epochs': 30,\n",
    "    }\n",
    "args"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7WLhguSTeazy"
   },
   "source": [
    "## Graph Prediction Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u05Z14TRYPGn"
   },
   "source": [
    "### Graph Mini-Batching\n",
    "Before diving into the actual model, we introduce the concept of mini-batching with graphs. In order to parallelize the processing of a mini-batch of graphs, PyG combines the graphs into a single disconnected graph data object (*torch_geometric.data.Batch*). *torch_geometric.data.Batch* inherits from *torch_geometric.data.Data* (introduced earlier) and contains an additional attribute called `batch`. \n",
    "\n",
    "The `batch` attribute is a vector mapping each node to the index of its corresponding graph within the mini-batch:\n",
    "\n",
    "    batch = [0, ..., 0, 1, ..., n - 2, n - 1, ..., n - 1]\n",
    "\n",
    "This attribute is crucial for associating which graph each node belongs to and can be used to e.g. average the node embeddings for each graph individually to compute graph level embeddings. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Pcic9NNU3nGK"
   },
   "source": [
    "### Implemention\n",
    "Now, we have all of the tools to implement a GCN Graph Prediction model!  \n",
    "\n",
    "We will reuse the existing GCN model to generate `node_embeddings` and then use  `Global Pooling` over the nodes to create graph level embeddings that can be used to predict properties for the each graph. Remeber that the `batch` attribute will be essential for performining Global Pooling over our mini-batch of graphs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "id": "3_Kq3zyjeZ22"
   },
   "outputs": [],
   "source": [
    "from ogb.graphproppred.mol_encoder import AtomEncoder\n",
    "from torch_geometric.nn import global_add_pool, global_mean_pool\n",
    "\n",
    "### GCN to predict graph property\n",
    "class GCN_Graph(torch.nn.Module):\n",
    "    def __init__(self, hidden_dim, output_dim, num_layers,\n",
    "                 dropout, pooling=global_mean_pool):\n",
    "        super(GCN_Graph, self).__init__()\n",
    "\n",
    "        # Load encoders for Atoms in molecule graphs\n",
    "        self.node_encoder = AtomEncoder(hidden_dim)\n",
    "\n",
    "        # Node embedding model\n",
    "        # Note that the input_dim and output_dim are set to hidden_dim\n",
    "        self.gnn_node = GCN(hidden_dim, hidden_dim,\n",
    "            hidden_dim, num_layers, dropout, return_embeds=True)\n",
    "\n",
    "        self.pool = None\n",
    "\n",
    "        ############# Your code here ############\n",
    "        ## Note:\n",
    "        ## 1. Initialize self.pool as a global mean pooling layer\n",
    "        ## For more information please refer to the documentation:\n",
    "        ## https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#global-pooling-layers\n",
    "        self.pool = pooling\n",
    "\n",
    "        #########################################\n",
    "\n",
    "        # Output layer\n",
    "        self.linear = torch.nn.Linear(hidden_dim, output_dim)\n",
    "\n",
    "\n",
    "    def reset_parameters(self):\n",
    "        self.gnn_node.reset_parameters()\n",
    "        self.linear.reset_parameters()\n",
    "\n",
    "    def forward(self, batched_data):\n",
    "        # TODO: Implement a function that takes as input a \n",
    "        # mini-batch of graphs (torch_geometric.data.Batch) and \n",
    "        # returns the predicted graph property for each graph. \n",
    "        #\n",
    "        # NOTE: Since we are predicting graph level properties,\n",
    "        # your output will be a tensor with dimension equaling\n",
    "        # the number of graphs in the mini-batch\n",
    "\n",
    "    \n",
    "        # Extract important attributes of our mini-batch\n",
    "        x, edge_index, batch = batched_data.x, batched_data.edge_index, batched_data.batch\n",
    "        embed = self.node_encoder(x)\n",
    "\n",
    "        out = None\n",
    "\n",
    "        ############# Your code here ############\n",
    "        ## Note:\n",
    "        ## 1. Construct node embeddings using existing GCN model\n",
    "        ## 2. Use the global pooling layer to aggregate features for each individual graph\n",
    "        ## For more information please refer to the documentation:\n",
    "        ## https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#global-pooling-layers\n",
    "        ## 3. Use a linear layer to predict each graph's property\n",
    "        ## (~3 lines of code)\n",
    "        x = self.gnn_node(embed, edge_index)\n",
    "        x = self.pool(x, batch)\n",
    "        out = self.linear(x)\n",
    "        \n",
    "        #########################################\n",
    "\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "id": "FJjnGuMSbjX0"
   },
   "outputs": [],
   "source": [
    "def train(model, device, data_loader, optimizer, loss_fn):\n",
    "    # TODO: Implement a function that trains your model by \n",
    "    # using the given optimizer and loss_fn.\n",
    "    model.train()\n",
    "    loss = 0\n",
    "\n",
    "    for step, batch in enumerate(tqdm(data_loader, desc=\"Iteration\")):\n",
    "        batch = batch.to(device)\n",
    "\n",
    "        if batch.x.shape[0] == 1 or batch.batch[-1] == 0:\n",
    "            pass\n",
    "        else:\n",
    "            ## ignore nan targets (unlabeled) when computing training loss.\n",
    "            is_labeled = batch.y == batch.y\n",
    "\n",
    "            ############# Your code here ############\n",
    "            ## Note:\n",
    "            ## 1. Zero grad the optimizer\n",
    "            ## 2. Feed the data into the model\n",
    "            ## 3. Use `is_labeled` mask to filter output and labels\n",
    "            ## 4. You may need to change the type of label to torch.float32\n",
    "            ## 5. Feed the output and label to the loss_fn\n",
    "            ## (~3 lines of code)\n",
    "            optimizer.zero_grad()\n",
    "            pred = model(batch)\n",
    "            labels = batch.y[is_labeled].type(torch.DoubleTensor).to(device)\n",
    "            loss = loss_fn(pred[is_labeled], labels)\n",
    "\n",
    "            #########################################\n",
    "\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "    return loss.item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "id": "ztPHXq_Gzn7U"
   },
   "outputs": [],
   "source": [
    "# The evaluation function\n",
    "def eval(model, device, loader, evaluator, save_model_results=False, save_file=None):\n",
    "    model.eval()\n",
    "    y_true = []\n",
    "    y_pred = []\n",
    "\n",
    "    for step, batch in enumerate(tqdm(loader, desc=\"Iteration\")):\n",
    "        batch = batch.to(device)\n",
    "\n",
    "        if batch.x.shape[0] == 1:\n",
    "            pass\n",
    "        else:\n",
    "            with torch.no_grad():\n",
    "                pred = model(batch)\n",
    "\n",
    "            y_true.append(batch.y.view(pred.shape).detach().cpu())\n",
    "            y_pred.append(pred.detach().cpu())\n",
    "\n",
    "    y_true = torch.cat(y_true, dim = 0).numpy()\n",
    "    y_pred = torch.cat(y_pred, dim = 0).numpy()\n",
    "\n",
    "    input_dict = {\"y_true\": y_true, \"y_pred\": y_pred}\n",
    "\n",
    "    if save_model_results:\n",
    "        print (\"Saving Model Predictions\")\n",
    "        \n",
    "        # Create a pandas dataframe with a two columns\n",
    "        # y_pred | y_true\n",
    "        data = {}\n",
    "        data['y_pred'] = y_pred.reshape(-1)\n",
    "        data['y_true'] = y_true.reshape(-1)\n",
    "\n",
    "        df = pd.DataFrame(data=data)\n",
    "        # Save to csv\n",
    "        df.to_csv('ogbg-molhiv_graph_' + save_file + '.csv', sep=',', index=False)\n",
    "\n",
    "    return evaluator.eval(input_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "id": "MR1wQ4hMZeMw"
   },
   "outputs": [],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    model = GCN_Graph(args['hidden_dim'],\n",
    "              dataset.num_tasks, args['num_layers'],\n",
    "              args['dropout']).to(device)\n",
    "    evaluator = Evaluator(name='ogbg-molhiv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qJGTNZiuZy0A"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8b84032391b24f67807b23202303e735",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "266a892d0b0f4276b3f0fbc1ac560cbb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c358bbb6b02b4c26b41abc055a84b56e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1d6a13c990034f1e9076ac96b8ee5133",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01, Loss: 1.1715, Train: 72.74%, Valid: 74.14% Test: 71.64%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3267d426c7774de6b6e3d2514db77b2f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8c2f29032d7e4cf695d5152ace0ed650",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4ec0566992df4368a9c02cca663332aa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5973184cd8a549549d7fbb19ea7d13b7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 02, Loss: 0.0117, Train: 76.10%, Valid: 76.88% Test: 70.24%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e40c515c56bd4d699cf41f9cb9135003",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "77461e26b13f4e26aac25015a308240d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e84d2632429848029a1a226d172cc58c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d4997f2005334c19a0dc307ffdb786a4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 03, Loss: 0.0291, Train: 76.60%, Valid: 72.63% Test: 69.90%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6bf3531c65a34ee49253db9a7951d96f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fe76e9ae6a88417992991d6d0c82c969",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6b448fb32b3747fbaf0abc1d5ffa2b60",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ab9a53aec62b4c58852085d41bdd6594",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 04, Loss: 0.4337, Train: 77.72%, Valid: 75.44% Test: 70.82%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "18c217ab60004e39b4947f69d0e99080",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9d11930e41624fc78bf3b3f073d78936",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ec29839980b5475db6a1f44cb2128238",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a5ba3f3ecd0941138fb54beb66724078",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 05, Loss: 0.0511, Train: 76.97%, Valid: 76.74% Test: 70.14%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "900b7caef5a14e20aa21010281da51a3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8dfd00f4e0df4df5886c8386198563a9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "567b5ba4512e4a02b0cbc84f65accec8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6a205859056b4cbfa5e31211cad33316",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 06, Loss: 0.0448, Train: 78.55%, Valid: 76.05% Test: 73.62%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b7b8225690924470a9d9f08c32bfdb31",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5d5a474349754765aad3f0f982d1b87c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5addd6f3dea9447fa46c4ec0607378cf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5d88eec4ad6942dab4a62f87c1e8819a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 07, Loss: 0.0332, Train: 78.11%, Valid: 76.70% Test: 71.88%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e4c71eed38044b729ec3c7ea8609b822",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e32212dbe3904d1c9c168a0b79e5d235",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "75b54d70285d424c808a99f2bc5d76f4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7949f5900851420d93e7751dff5005d0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 08, Loss: 0.0276, Train: 78.28%, Valid: 73.76% Test: 72.15%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1f34ca90e5054efa94366293511acc84",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8373e27fc019405db8be896d2f761eab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e12ea625e0344d369c6301b3c675abd2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "069e7d8f92bd4d76bb683d44bec3b7d1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 09, Loss: 0.0255, Train: 78.87%, Valid: 73.79% Test: 70.70%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f27096bddd33465d9f2550ea34a203c4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "783b930ab11748c889d246af682e6464",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a9115a1c901e4263944e1b1d22c5b36c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3456b10699a444129f5df8b8fffa855c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 10, Loss: 0.0294, Train: 79.30%, Valid: 74.21% Test: 73.54%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ee698596592c4c40abcb46de3f55a72d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7fb6349a0d204612a2f0637d89bc9666",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "27acb2f81e654f4ab9f3194f3858a485",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "17623142695840ae82c79a14644389ca",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 11, Loss: 0.0348, Train: 79.65%, Valid: 77.50% Test: 71.82%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "657353e455db4ceabe2a0016e092e929",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "009d7dc4ade94a36a57962540a3463a5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c2156c9fc52e466e81c38a9a47be7bfc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "52c84e46c3be448eafa617335566ba16",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 12, Loss: 0.0321, Train: 80.19%, Valid: 72.43% Test: 71.57%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e8ebb0f7df664898ac870d5c86d726bb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8c466e6b2b984e0c96f99678bbca8e32",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e7fdeb2055b04cbab2002c79274a701e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "725fcdc6b1974d84b6de42f5b0eae578",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 13, Loss: 0.0318, Train: 80.19%, Valid: 77.49% Test: 69.63%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3751d3d471f840b8bff9036e1ca2da54",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2829cb15d8ef48b593ffe3ce74113722",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "062ad26afac442329d11b63486bb8ea6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f819bb2cf43446eba8b47bf2f552e599",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 14, Loss: 0.0393, Train: 79.99%, Valid: 77.43% Test: 73.01%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ac67b08962fd404083cf243983fbcc8f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dfbd46d1e1ac400785e62ef9379fd733",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0bdff5f14e8247a1ad1debdd2c78dadc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "46c0c4a172e9485ab79d1bc50ae865b3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 15, Loss: 0.0511, Train: 80.82%, Valid: 76.43% Test: 70.04%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a3d6eb74beb84c65a108aa8e91d5986f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "236febbc28bf4db1a85ce1710fb032c2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f7af9d3f4c5544b58ffd337cb2d4f9de",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d3283f1cb88a43b9a79bebb76ce0c24f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 16, Loss: 0.0249, Train: 81.53%, Valid: 76.15% Test: 73.28%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2cc2362916774d6b9ccd1edcbf41520d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "712fe6e2b5fe4072b35c6ecaa52c532d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1aa32d083c1945128d406b6e434408e6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f63deac626594660845b9aab22ebbdef",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 17, Loss: 0.0244, Train: 81.05%, Valid: 75.30% Test: 72.60%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ff304b0db26344bcbdc3fcb032964a72",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4d7824b0f877466caaa3854d82cad100",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a6413062d6fc4917a02483f53ce7526b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2c423dc0a8b64a25bb907ec9bf51a79b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 18, Loss: 0.4053, Train: 80.44%, Valid: 76.83% Test: 72.90%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a15e8bda1c194ed9ae505a393687ab83",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f397af9235f444c3b5a88ba729e022f9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8a949966af2745bda75552d9e5cb04c5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7419da16ce4e48fb978f26c07a7d2203",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 19, Loss: 0.0507, Train: 80.76%, Valid: 74.64% Test: 70.28%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "919e311e4c004ca7955b5c1c3506c428",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f392c3bc608c4225bd1485520a4dbb6f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e1ae35af6f2e4581a1399ceffb518496",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bb02154482c9426798e66c2b973fb1b0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 20, Loss: 0.0336, Train: 82.24%, Valid: 77.83% Test: 71.89%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6e6cdaf6d1614316b693ddc54bc18527",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a9662ee81ee94ec293089de542e628cc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d1ad5d162a324c41b8ef83102c9eb7f1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4ea697cdac864426a5c143c937c7bb15",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 21, Loss: 0.9077, Train: 82.40%, Valid: 76.99% Test: 73.09%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0802a17e04994ef7b6fe77f71e1068dc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bbd6d097196d4ea5bfaa410b37621ca7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "109d841abf2f4530be27fdaebcb3e16d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1e0d162437234235a1f23fa9323a2fcf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 22, Loss: 0.0382, Train: 82.48%, Valid: 76.54% Test: 73.44%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f771f64571374eb5b4ffe067c2e8e511",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1d06b3c1925948d2bdeb4b0da4bf8281",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "087e757316db4d919dd416572dca55dc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9fd4c061371544ec9c88e06f4d6df973",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 23, Loss: 0.0314, Train: 82.43%, Valid: 77.44% Test: 71.50%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bd4e68c47c5c4e25b983c09c46816cd4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4b9cc017d48446f48fb3887f42b26f8d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1772b7306a91413683b55333ced35152",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f27355ead194481ab833c2430fc7cb4c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 24, Loss: 0.0246, Train: 82.75%, Valid: 75.62% Test: 74.74%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a6a570fee4d645fcb6410ca7456d5efc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b550be55723c4453b7627ea868fbfc16",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fa7e4d97d86b41aeacabc5fff38855aa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1940753e77834defb632ee7741e1b7f2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 25, Loss: 0.7731, Train: 81.98%, Valid: 74.00% Test: 72.65%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b57438ecd5144b3eadb2c2ac110c9162",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f0a4d1c543214b12bb6a5600d2e7ff57",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4bc02d18bb3f453bba11703bda544aab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f76ceb08dd8b4e14877e3348214ae867",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 26, Loss: 1.4363, Train: 83.14%, Valid: 76.12% Test: 73.21%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ac6bdbcf4d764afc993fc27054e683c5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "86a69f0f9bdf43b9b7b6da11a0103007",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "101556f08251496a85d600a9b358d885",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "39e34ce7bb164258af026012ee49b862",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 27, Loss: 0.0199, Train: 83.54%, Valid: 77.95% Test: 72.95%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e60039f2f0104a9da274e8614b6aa101",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3875aba80706473880741d42c43faf55",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "feb60be3162d42a3909c3e88069e072e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7399f2a397f94bb991c55184831f469e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 28, Loss: 0.0277, Train: 83.40%, Valid: 78.33% Test: 74.03%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6bf5351c39b7481c99bad4ea08260dd0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "91094660702248708bab1f9349a69020",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fceefab0ac33401d9a45415add75e24c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b128242f12d9414bbaa5284ac31b5e30",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 29, Loss: 0.6173, Train: 84.03%, Valid: 77.73% Test: 72.96%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "33d69d638ac34595a44efa627db54f84",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import copy\n",
    "\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    model.reset_parameters()\n",
    "\n",
    "    optimizer = torch.optim.Adam(model.parameters(), lr=args['lr'])\n",
    "    loss_fn = torch.nn.BCEWithLogitsLoss()\n",
    "\n",
    "    best_model = None\n",
    "    best_valid_acc = 0\n",
    "\n",
    "    for epoch in range(1, 1 + args[\"epochs\"]):\n",
    "        print('Training...')\n",
    "        loss = train(model, device, train_loader, optimizer, loss_fn)\n",
    "\n",
    "        print('Evaluating...')\n",
    "        train_result = eval(model, device, train_loader, evaluator)\n",
    "        val_result = eval(model, device, valid_loader, evaluator)\n",
    "        test_result = eval(model, device, test_loader, evaluator)\n",
    "\n",
    "        train_acc, valid_acc, test_acc = train_result[dataset.eval_metric], val_result[dataset.eval_metric], test_result[dataset.eval_metric]\n",
    "        if valid_acc > best_valid_acc:\n",
    "            best_valid_acc = valid_acc\n",
    "            best_model = copy.deepcopy(model)\n",
    "        print(f'Epoch: {epoch:02d}, '\n",
    "              f'Loss: {loss:.4f}, '\n",
    "              f'Train: {100 * train_acc:.2f}%, '\n",
    "              f'Valid: {100 * valid_acc:.2f}% '\n",
    "              f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6I17-Qso_n88"
   },
   "source": [
    "## Question 6: What are your `best_model` validation and test ROC-AUC scores? (20 points)\n",
    "\n",
    "Run the cell below to see the results of your best of model and save your model's predictions over the validation and test datasets. The resulting files are named *ogbn-arxiv_graph_valid.csv* and *ogbn-arxiv_graph_test.csv*. \n",
    "\n",
    "Again, you can view these files by clicking on the *Folder* icon on the left side pannel. As in Colab 1, when you sumbit your assignment, you will have to download these files and attatch them to your submission."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "id": "Oq5QaG21dOOO"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ca6f0a79157e47c28edb52dc16240da3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a024e6ecdc9a4e7bb9c409fa64fb00ef",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving Model Predictions\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f042b7c3809044bb9c0633cc9a4ac86e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving Model Predictions\n",
      "Best model: Train: 83.40%, Valid: 78.33% Test: 74.03%\n"
     ]
    }
   ],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    train_acc = eval(best_model, device, train_loader, evaluator)[dataset.eval_metric]\n",
    "    valid_acc = eval(best_model, device, valid_loader, evaluator, save_model_results=True, save_file=\"valid\")[dataset.eval_metric]\n",
    "    test_acc  = eval(best_model, device, test_loader, evaluator, save_model_results=True, save_file=\"test\")[dataset.eval_metric]\n",
    "\n",
    "    print(f'Best model: '\n",
    "          f'Train: {100 * train_acc:.2f}%, '\n",
    "          f'Valid: {100 * valid_acc:.2f}% '\n",
    "          f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gBi_t8n0iZ4P"
   },
   "source": [
    "## Question 7 (Optional): Experiment with the two other global pooling layers in Pytorch Geometric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch_geometric.nn import (global_add_pool, global_mean_pool, \n",
    "                                global_add_pool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "id": "MR1wQ4hMZeMw"
   },
   "outputs": [],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    model = GCN_Graph(args['hidden_dim'],\n",
    "                      dataset.num_tasks,\n",
    "                      args['num_layers'],\n",
    "                      args['dropout'],\n",
    "                      pooling=global_max_pool).to(device)\n",
    "    evaluator = Evaluator(name='ogbg-molhiv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "id": "qJGTNZiuZy0A"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7b1e1c43816f483a80b267ea6f1cd04c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f09ccaf353404a518cbcdb02cb66480b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "360775a3287d44dcb8def627087b7f4b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7f702711403f4ef79d78866abab7a611",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01, Loss: 0.4254, Train: 73.89%, Valid: 62.46% Test: 66.66%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7f98c6055c7e44deb4f3128f27f382cb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "685f7f6146dd4ffd975fb579e5c0a1b7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "95985ffa54f64797a4711057b7bc4ab2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ac1bd8470ea44a3cae76f7f746d84e12",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 02, Loss: 0.0224, Train: 76.04%, Valid: 69.15% Test: 71.59%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "78289b4b9859474b9f537e46b3733412",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0c2062b92bc14d2da884f15395904f1c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6bc8a8acc989435a9766b246990f5313",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ea2888d07c9e4c41b2aa718e47458e82",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 03, Loss: 0.0120, Train: 78.02%, Valid: 73.70% Test: 72.12%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b620a749219747019b608b72e238be33",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "15ac91ebee7f4894a5bcf4fe58ed188e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2b023067284a4d2786886d4cc4ac6923",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "741ee31890a44e6a97dfa09fcfa9863a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 04, Loss: 0.1211, Train: 78.12%, Valid: 74.37% Test: 70.23%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e2c4d557881a49c4985c1199575840b6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cadf98bca3b94b5ca857ea8d8d86e6b5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e1ca659c34064126bd6ff3d6c877d250",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9b7ed5f632ba41d0be56b2d70c99645d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 05, Loss: 0.0383, Train: 79.33%, Valid: 69.01% Test: 70.96%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "80970a27518a44c690151817cc8d8e00",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "27e5002a78a342b396928f41b52ae7d1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "882f8dd9a22f4f2a8b3be53cda5599b9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9a7e4844bb674c5d8027297beb44e788",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 06, Loss: 0.0417, Train: 79.98%, Valid: 72.68% Test: 73.88%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a86e0ae43f4349f3ba11e46b2a54bac8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d33909e919d401899f01b4f71d443d4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9c011a3ed46f4511accd0f415db56601",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b00c8f3939754242a6ed9ea2d3974ca9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 07, Loss: 0.0199, Train: 80.33%, Valid: 73.31% Test: 73.08%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9677a2941696435086222de957732718",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9242950ab1de41af929669baa8dd0df5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3e67099d42234dc0bc09b913a0c2b418",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ec231d79cef54d97af2a61fb10ed6e4e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 08, Loss: 0.5160, Train: 80.90%, Valid: 71.05% Test: 73.62%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a723cb2c947643fa81020bab05993b01",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b38a222647104bedaaa1500683e4a1b8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5e3ec1f0f60d4b09955eea53aedb3ead",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3aa1599644654ba79f09fece37771da1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 09, Loss: 0.0240, Train: 81.40%, Valid: 75.53% Test: 77.15%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0986531134394700a1477ed87e36d285",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f55f4adcf9314bbfb0e753c01c71f476",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dda4672dff6a4df1a99704502d6c236a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6ef0df4434624116a2685e86a1de677c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 10, Loss: 0.0242, Train: 81.72%, Valid: 75.84% Test: 71.65%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bbefe30ac9544fe187023e5d03e818f2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5939d66d74a241f99a0c410f300dab94",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "12d07140f2f64455aeb1fc81ec6fa8c0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4e8f79bc39b5475c91d0fc9102d60c07",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 11, Loss: 0.0677, Train: 82.15%, Valid: 75.11% Test: 76.50%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "935190486d524e93865537e06b3f0f91",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1910656b47234b42b71b13839b3de6b5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b6ac18f66145403c820372ec9e4db5a2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a220113ddac64a40a3bd2205b6a14c80",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 12, Loss: 0.0488, Train: 82.12%, Valid: 77.65% Test: 76.03%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "09adc35573eb48769c38e7d5137c8183",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1b7d10d85765444d98d051d8118db289",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "102fdb6421dc414386bb7c4b281262c1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a571e96ddbf541f6990f1d0ac9c3344e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 13, Loss: 0.0566, Train: 83.57%, Valid: 77.71% Test: 75.65%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0eadf463cdbd4a1da93b4600b440290f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b5f2a6a9c3ac4f508936fe9ce8d67360",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "df9640919567432b9627a12bcb5de6ce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "54f18b7d502842aca99ab245aa14f1fb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 14, Loss: 0.0233, Train: 83.23%, Valid: 75.58% Test: 76.52%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d21393551134a219692a4b1996f9dcb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a7e3e2bef495416097df957372d3282b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c8eab848b97347eda86b62796d9c17de",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "03cca24fd6b247bf90fd9fbda3274fc0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 15, Loss: 0.0260, Train: 83.64%, Valid: 76.06% Test: 76.25%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0554d2fa108d43878352c64fea70ea4b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5eb2bdb62911420fb5c6ac13e8b50151",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6d1c61eeecc541ffb13b7124b184f52b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b0501c38b92c41cebdbe6436e43e2638",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 16, Loss: 0.2135, Train: 84.14%, Valid: 76.21% Test: 76.74%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cc37718049134d4f82def9661a3a0a06",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0b370115ba7c423fb3e46ef6d3cff0aa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d157403bb81540aa828e724387d3ab0b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ecd53081e14949ebb015c28b66e0ce3d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 17, Loss: 0.0459, Train: 83.86%, Valid: 74.98% Test: 76.00%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fb144d6ca189495098fc00e3cef04acd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "49a3f6f4fe7a4120b2cae6fa54c5729c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d7f500d9a0a541168a6e04960787e265",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "76b86f3bb2c64e2391b73fe50002cc2b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 18, Loss: 0.0864, Train: 84.02%, Valid: 76.03% Test: 74.20%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c8fca24e09a242cfab0d141e56789a07",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eeece435a2d74e5c81b9ae767e44a57f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "823ff0e4b6ed47809f63630dcb166af9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6a1e3694975941c3aa52f63946e0d181",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 19, Loss: 0.0322, Train: 84.49%, Valid: 75.76% Test: 75.60%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "da95699035574dc0a1817ff325c003bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3e53ad7772134d6bbc3dc93f305768eb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9521ea57ae274722888ab54fb6e337e9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "482f54ac8778462084393e42c5d6fa52",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 20, Loss: 0.0132, Train: 85.55%, Valid: 78.69% Test: 76.81%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "effc42081398433cb81fd8520e9409ad",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "878b1714814c4cc6857a770266b974a2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "829992df667f4aac80413eb2ebb108ed",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "380ef305e3004922af29cde4aa4e589f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 21, Loss: 0.0094, Train: 84.97%, Valid: 74.97% Test: 77.94%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b619aa794da3454e952d98955da370a8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9942215a7f0047c5a603021adbc957ae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "121986b178ba41dd9f5a5b40c23f7682",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f6c4384c71aa44fa9cf547ae6e3c91c0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 22, Loss: 1.1919, Train: 85.85%, Valid: 78.52% Test: 76.28%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "451621896175410ba72495a9b91f8962",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2ae0dae449b54e2bb252f356e556920a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d0c4119c2dd447e08e94bb7e6a98e35c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b41d70d7d0364e4fa7afd97c59e096b9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 23, Loss: 0.0103, Train: 85.64%, Valid: 75.91% Test: 75.88%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ecf84402f80a4334aa1f30bb4ed3812c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "39235b6f325c4b02ae1229c57b6c6829",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "647fd5c3c46b47799dcdd80554f732a8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a2e47ce5c8a94359af37377a3239dc57",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 24, Loss: 0.0131, Train: 85.92%, Valid: 76.81% Test: 76.78%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "03b27e6af88f40e9b8cb7ed9643a6ed8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2b920e4a94134475ac1142e9b2ed3878",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5102343996004b3c8a37059d0c630a6d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c8672a0414a140c5a5db6655aa730330",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 25, Loss: 0.0194, Train: 86.28%, Valid: 77.26% Test: 74.75%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eca4d67099a64db7a00499bd81e1484a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "932ba9af74024514931b034874657a63",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d21a22ae95884205b1b8617dbacb81e3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "abc676a561e244d0a11f628abf22cd92",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 26, Loss: 0.0185, Train: 86.12%, Valid: 77.80% Test: 77.13%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "88dec5a06b1942a093e496e0dac7f17e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4487f5b360a34447843dd1f95672da25",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "58099534cecb4d168aea1073e1b2290f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c08fbe751bbb47738ccaff26de1d5bae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 27, Loss: 0.8386, Train: 85.75%, Valid: 75.76% Test: 75.78%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d570efec15674860be152d876fcdcae0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e496050092bc43989c33797dd376832d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "51accbb5c0a34bd28e168093cd51447b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b34608d2557a4f7a863a362bbc833cd3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 28, Loss: 0.6317, Train: 86.14%, Valid: 79.46% Test: 75.45%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1bcee0e3af4c47688699f5861f5842de",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8bde43dc53c54df0a6a7f122ff40b2c4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "299ca23a5076443fb313d6f35d143603",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1440f66c5ede4949928c2357e50c88d2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 29, Loss: 0.8540, Train: 86.85%, Valid: 81.11% Test: 77.69%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fdff800e7f3647268b61f05e7732656e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "95486740cd074694bd305d377ae455e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bc947dd10d2f4604bb2f6e9ccda7fa04",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "adaff9fb2f404475a548b87351dff4aa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 30, Loss: 0.0078, Train: 86.56%, Valid: 76.63% Test: 76.40%\n"
     ]
    }
   ],
   "source": [
    "import copy\n",
    "\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    model.reset_parameters()\n",
    "\n",
    "    optimizer = torch.optim.Adam(model.parameters(), lr=args['lr'])\n",
    "    loss_fn = torch.nn.BCEWithLogitsLoss()\n",
    "\n",
    "    best_model = None\n",
    "    best_valid_acc = 0\n",
    "\n",
    "    for epoch in range(1, 1 + args[\"epochs\"]):\n",
    "        print('Training...')\n",
    "        loss = train(model, device, train_loader, optimizer, loss_fn)\n",
    "\n",
    "        print('Evaluating...')\n",
    "        train_result = eval(model, device, train_loader, evaluator)\n",
    "        val_result = eval(model, device, valid_loader, evaluator)\n",
    "        test_result = eval(model, device, test_loader, evaluator)\n",
    "\n",
    "        train_acc, valid_acc, test_acc = train_result[dataset.eval_metric], val_result[dataset.eval_metric], test_result[dataset.eval_metric]\n",
    "        if valid_acc > best_valid_acc:\n",
    "            best_valid_acc = valid_acc\n",
    "            best_model = copy.deepcopy(model)\n",
    "        print(f'Epoch: {epoch:02d}, '\n",
    "              f'Loss: {loss:.4f}, '\n",
    "              f'Train: {100 * train_acc:.2f}%, '\n",
    "              f'Valid: {100 * valid_acc:.2f}% '\n",
    "              f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "id": "Oq5QaG21dOOO"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c84d024aed654619b5d123417554a9b4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9c288368c2914130bcbc9b4347920826",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving Model Predictions\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d7165bb6750f45ffa9395ddbb3d6c835",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving Model Predictions\n",
      "Best model: Train: 86.85%, Valid: 81.11% Test: 77.69%\n"
     ]
    }
   ],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    train_acc = eval(best_model, device, train_loader, evaluator)[dataset.eval_metric]\n",
    "    valid_acc = eval(best_model, device, valid_loader, evaluator, save_model_results=True, save_file=\"valid\")[dataset.eval_metric]\n",
    "    test_acc  = eval(best_model, device, test_loader, evaluator, save_model_results=True, save_file=\"test\")[dataset.eval_metric]\n",
    "\n",
    "    print(f'Best model: '\n",
    "          f'Train: {100 * train_acc:.2f}%, '\n",
    "          f'Valid: {100 * valid_acc:.2f}% '\n",
    "          f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "id": "MR1wQ4hMZeMw"
   },
   "outputs": [],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    model = GCN_Graph(args['hidden_dim'],\n",
    "                      dataset.num_tasks,\n",
    "                      args['num_layers'],\n",
    "                      args['dropout'],\n",
    "                      pooling=global_add_pool).to(device)\n",
    "    evaluator = Evaluator(name='ogbg-molhiv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "id": "qJGTNZiuZy0A"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a7db8bb907a044168b47bcdecda05cb2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0ff11fd0dff34c459471487c69577dc4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b97e476ee9824409b6d03e6a39329c91",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e9e2e658a1f74ae5afdb6672667439cf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01, Loss: 0.0428, Train: 68.09%, Valid: 70.10% Test: 58.40%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "27212f26206b49a7b4204e1a0b1b60cf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a41a827ad507497d80bc6143da5b4bc1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5e730260f3d14482b589a46edaaa31ff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e71f807027a6429994c7932924807bca",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 02, Loss: 0.0199, Train: 67.27%, Valid: 60.66% Test: 58.33%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0d2eaab1ef294807990401e38a69840d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b5987be34d9645bd928787fa821eb613",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d7205319737b4181aef316375dd16c7f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d52c4e07cf4a43d69f7185a9509c329a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 03, Loss: 0.0919, Train: 69.32%, Valid: 66.44% Test: 59.85%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ee36ddf57f2c4a7b8f5c97badc89559b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0cd9f78176fa48b0b48dbadbaffc042e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5cf871dd51724b93bc8ca55475b180b0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "10aa888be00341c581eb863f94286c9b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 04, Loss: 0.0200, Train: 71.80%, Valid: 65.10% Test: 63.20%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "99de3a70ac90410781837301f38d18e1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "71493463e3c04da2892b2216db0bdc5f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "675c717e535b49d8807c4b627daaf80c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6095d83bef1c47c7a0541f68ecaed10f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 05, Loss: 1.5892, Train: 74.02%, Valid: 71.35% Test: 71.24%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8832c895cfdf43c1ac5952c32b832959",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bb9c611ba5d24591906b7c381ef78683",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b32afc9800dd4cbdb8a1613525558510",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6057d0e6a7c240f19d6b3f6854d42f1b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 06, Loss: 0.7315, Train: 75.34%, Valid: 72.72% Test: 73.06%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e13e07c68a01420fb212e2295f101bdb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6b1ac5c148f842f09e1d593e91281b02",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e0771b3e2d2744fb95c137217b00276e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d7c058b6bda5468082afdb1cb0e37f73",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 07, Loss: 0.5575, Train: 74.55%, Valid: 72.83% Test: 74.99%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "56975c70eba94560ba52c184d4ad03ee",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "77bba019b1cc4f3f932196d7b73113f2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "81312afd428349a681bde461c9e6f2f7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "60a427b0312c4fc695e415d8f3c0ca3c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 08, Loss: 0.0282, Train: 76.09%, Valid: 70.40% Test: 72.36%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f891f22b445a42a0a586e2f7889bd0b7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1324615c42464570960f2b7c84929fb4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4346dcd489c24b7fbc7e3082d6e6d807",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "49dd200b94ee4f9a817306b102319de4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 09, Loss: 0.1383, Train: 76.51%, Valid: 64.04% Test: 69.90%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c09af356a9654349b5a67c3f26b5c694",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "71fdaced73bd4adea866d54207c7985e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7dfc0752c5724863991d4ae8b2716834",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0d04942580204319976683564fd8b100",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 10, Loss: 0.0515, Train: 77.54%, Valid: 71.25% Test: 70.83%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b4695949f30a4d01b97a6cab5cb4b3ae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1cb97eebc35a47f6b33918c32e91d16f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6425bb5c5a22497ba545bc3892d674f9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "75bf1ca2f3a3481e860d8d7e542c92c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 11, Loss: 0.0099, Train: 76.20%, Valid: 75.07% Test: 74.72%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "46c21084835949988777ab548b33f6d4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fa36490d3c9d420cbfd32289cfa50bda",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e219c4ac1b1f4cbfa224c501215d6d35",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aaa18b94a7c94b17881396bf04b2b45d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 12, Loss: 0.0273, Train: 77.20%, Valid: 63.63% Test: 66.50%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6e7e39a60c894c0fb2befe1fe2a2518a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "162e77eb7f5a4aaf82a94844e80bcca4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fe49611ae9404839a5103e37fbe9736a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4864826c476c4ebfb3482a939ddf119d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 13, Loss: 0.0168, Train: 77.99%, Valid: 69.49% Test: 71.06%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b7ebd23ce40f47c2a3e809d36e8fabf4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "66fc381992514255b0b6f387611a7853",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "477472cdef744da9b028d567992c4aee",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "97d39385ba644fc6ac53f55d74df2601",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 14, Loss: 0.0315, Train: 78.61%, Valid: 76.53% Test: 73.64%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "16762434544f4178a4fb867922f2fd60",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f38adfd97b6948a582e49db14e9cd4d7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d24e35dc43d3488a9ef7a37cddc08bc5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "edb5539aa7dd41ee86ad682d83b9795e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 15, Loss: 0.0324, Train: 78.92%, Valid: 74.35% Test: 74.39%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e45e860b24b94ae99de1e327a8a3cf01",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "892c1b03d51a4cbd9c01e816fefa4acd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4db02a4f2cec4a628cda341be773d1d4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ff25c5ce68bd450cafc13dd121aa2a5d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 16, Loss: 0.0226, Train: 78.37%, Valid: 74.90% Test: 71.94%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9fec0db7c56a4ab89fec05ea3a2b3dce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7d5483966a034a58b475e5e811061381",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e0df555f3f6f4494b626f958d8fa7b87",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6ef612f61afe405994e2986d59c3a72a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 17, Loss: 0.0239, Train: 79.52%, Valid: 75.85% Test: 74.19%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ab38e5a13a80441d911fa70bbd9a9192",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3ce42602b3cb41d2bdc4871759e0e8eb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8bed4644fb3b410e98dedc5136c359cf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "32aec28ce571463e9f4a7e1143955eda",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 18, Loss: 0.0096, Train: 79.61%, Valid: 72.05% Test: 72.48%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e9e3fd71567c49a0944677294303d96e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "007d983998dc448f9aaa845e0357acdc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "295057ade40f47a0b9ba6b8f589d9a65",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9654196894fc40b2a4a32b186034a2aa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 19, Loss: 0.0417, Train: 81.00%, Valid: 79.51% Test: 75.62%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "652e3cd2c14d4b8589e11b813dfea955",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "33535b7cb04543d5b7b02aa4f29d048c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "792ab842e6384b9cadf13016ac5ed9f3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "81f7bb6e2bb0486dae3245c07058aed6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 20, Loss: 0.0361, Train: 80.15%, Valid: 81.21% Test: 73.42%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0a9518a02cad40debd0cede522ed2756",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4b488105db2e4a30a14f36c5610793ab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d0cd8241a90540b581b44f3ddc340ef7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b12a31115faf4b3589dd8495344d76b5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 21, Loss: 0.1549, Train: 81.21%, Valid: 75.35% Test: 73.29%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5ffcc10eede6428f8f25769c925b5470",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "549fba1d96de4cc4b6dd228c7f5d4696",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "18195ab534964b45a0f6bd54147790f3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0040567f019549b7a95cb0a110a14703",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 22, Loss: 0.0293, Train: 80.77%, Valid: 80.58% Test: 74.66%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aac27078b52348b18dd012bbac6ca582",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "338abc016f4d4afe97b1d6af51077846",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b47c215b2c774be0933e91424739991e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "66a2e674009a4460b5526c0e285b07e3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 23, Loss: 0.0281, Train: 81.11%, Valid: 80.35% Test: 75.92%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b71e895352a042db95a046f57ce71659",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9123845ad4a5483e81f288f55dd882d7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9666d46c18104d95857fc708dd9b9ad0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0f7787f210fa44449571f831544a351e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 24, Loss: 0.0158, Train: 81.30%, Valid: 79.70% Test: 74.82%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "68f9b60fda6f43c1b4a29649b61028b2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ed7ad5d88be7496e8dae545ce00377e1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6fc2ee363219417486cd8c1fa024445e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "948e4edee27c465e85787852e651837a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 25, Loss: 0.0206, Train: 81.19%, Valid: 78.59% Test: 75.01%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "152401ed6a0c41aebe97a32a6ecd9e2b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a06ee5a902ce479399b9f6c3d42ae4ed",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3469e7fdeedb4d5b8b573c74f74c6cb5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "64d4d8f26e9246b7a651db90f720b9ab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 26, Loss: 0.0152, Train: 80.95%, Valid: 79.00% Test: 76.44%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "37690ffe2ed64a17aed254391b897f8c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "10587a4714304244977c301c68acc244",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "112db334cade409b99ff156eb49cb158",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0218519eb0fb468fa0a583748bda2bc6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 27, Loss: 0.0301, Train: 82.24%, Valid: 80.94% Test: 73.44%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8eb7875253954952b75ede3b41ca3e75",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0e78df823d7447a8b8ceaf8d4c1ce0fe",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "218ba8198da2463098a8e3f891758dd4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "72adf59d0ff1425689d9b4ec96e4e87f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 28, Loss: 0.0607, Train: 82.05%, Valid: 79.83% Test: 74.61%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ed7a3ecd306a46d1bcf582ad5d7ac1e5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "826b5d94eaa047f49da631f587b5f311",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e0d85d966db24328af1ff6b2d93e4bc1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "97fcc0e63967436c9b19e4bfcd659ab7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 29, Loss: 0.0112, Train: 80.95%, Valid: 78.24% Test: 74.15%\n",
      "Training...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1b7a76ef27fa48868da781ac5d4f315f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4f07ae80a5184ba199426b57a52c556b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a2cc4788060942259233efdacd4ece1e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "33769dc7497d4facad12aeb1abb86125",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 30, Loss: 0.0541, Train: 81.84%, Valid: 81.23% Test: 74.48%\n"
     ]
    }
   ],
   "source": [
    "import copy\n",
    "\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    model.reset_parameters()\n",
    "\n",
    "    optimizer = torch.optim.Adam(model.parameters(), lr=args['lr'])\n",
    "    loss_fn = torch.nn.BCEWithLogitsLoss()\n",
    "\n",
    "    best_model = None\n",
    "    best_valid_acc = 0\n",
    "\n",
    "    for epoch in range(1, 1 + args[\"epochs\"]):\n",
    "        print('Training...')\n",
    "        loss = train(model, device, train_loader, optimizer, loss_fn)\n",
    "\n",
    "        print('Evaluating...')\n",
    "        train_result = eval(model, device, train_loader, evaluator)\n",
    "        val_result = eval(model, device, valid_loader, evaluator)\n",
    "        test_result = eval(model, device, test_loader, evaluator)\n",
    "\n",
    "        train_acc, valid_acc, test_acc = train_result[dataset.eval_metric], val_result[dataset.eval_metric], test_result[dataset.eval_metric]\n",
    "        if valid_acc > best_valid_acc:\n",
    "            best_valid_acc = valid_acc\n",
    "            best_model = copy.deepcopy(model)\n",
    "        print(f'Epoch: {epoch:02d}, '\n",
    "              f'Loss: {loss:.4f}, '\n",
    "              f'Train: {100 * train_acc:.2f}%, '\n",
    "              f'Valid: {100 * valid_acc:.2f}% '\n",
    "              f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "id": "Oq5QaG21dOOO"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "482a8443f4bc40d58130618fe4731dc2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8ad9359571f040f5ba1e8eed6ac147c3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving Model Predictions\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a325f476f78948258e0f92594721f16c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/129 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving Model Predictions\n",
      "Best model: Train: 81.84%, Valid: 81.23% Test: 74.48%\n"
     ]
    }
   ],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    train_acc = eval(best_model, device, train_loader, evaluator)[dataset.eval_metric]\n",
    "    valid_acc = eval(best_model, device, valid_loader, evaluator, save_model_results=True, save_file=\"valid\")[dataset.eval_metric]\n",
    "    test_acc  = eval(best_model, device, test_loader, evaluator, save_model_results=True, save_file=\"test\")[dataset.eval_metric]\n",
    "\n",
    "    print(f'Best model: '\n",
    "          f'Train: {100 * train_acc:.2f}%, '\n",
    "          f'Valid: {100 * valid_acc:.2f}% '\n",
    "          f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "e7JXsMTBgeOI"
   },
   "source": [
    "# Submission\n",
    "\n",
    "You will need to submit four files on Gradescope to complete this notebook. \n",
    "\n",
    "1.   Your completed *CS224W_Colab2.ipynb*. From the \"File\" menu select \"Download .ipynb\" to save a local copy of your completed Colab. **PLEASE DO NOT CHANGE THE NAME!** The autograder depends on the .ipynb file being called \"CS224W_Colab2.ipynb\".\n",
    "2.  *ogbn-arxiv_node.csv* \n",
    "3.  *ogbg-molhiv_graph_valid.csv*\n",
    "4.  *ogbg-molhiv_graph_test.csv*\n",
    "\n",
    "Download the csv files by selecting the *Folder* icon on the left panel. \n",
    "\n",
    "To submit your work, zip the files downloaded in steps 1-4 above and submit to gradescope. **NOTE:** DO NOT rename any of the downloaded files. "
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "machine_shape": "hm",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "PyGeom (pyg)",
   "language": "python",
   "name": "pygeom"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}