notebooks/cis6930_week11_more_topics_part1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "cis6930-week11-more-topics-part1.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tydRDo6t2bjd"
      },
      "source": [
        "# CIS6930 Week 11 More Topics\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "Preparation: Go to `Runtime > Change runtime type` and choose `GPU` for the hardware accelerator."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5MMZ5mxJ2iUy"
      },
      "source": [
        "gpu_info = !nvidia-smi -L\n",
        "gpu_info = \"\\n\".join(gpu_info)\n",
        "if gpu_info.find(\"failed\") >= 0:\n",
        "    print(\"Not connected to a GPU\")\n",
        "else:\n",
        "    print(gpu_info)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YbAny3QK2j6B"
      },
      "source": [
        "## Preparation\n",
        "\n",
        "For this notebookt, we use Hugging Face's `transformers` library."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1qK5hsVkyVF_"
      },
      "source": [
        "!pip install transformers"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bsI5DgYCH_Vb"
      },
      "source": [
        "import transformers"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mi4fAAIfIBKe"
      },
      "source": [
        "from transformers import TrainingArguments"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e5jB4eGHyfEl"
      },
      "source": [
        "## Twitter Classification Dataset (Again!)\n",
        "\n",
        "In Week 6, we created a custom dataset for the Twitter dataset (Please see [the Google Colab notebook](https://colab.research.google.com/drive/1DZN-Bo2HBnPQPm4jrQzEIchhHdN682qP?usp=sharing))\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O4WGTpfRez-s"
      },
      "source": [
        "# https://www.kaggle.com/crowdflower/twitter-airline-sentiment\n",
        "# License CC BY-NC-SA 4.0\n",
        "!gdown --id 1BS_TIqm7crkBRr8p6REZrMv4Uk9_-e6W"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jnxo0_CBe0Aw"
      },
      "source": [
        "import pandas as pd\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "\n",
        "from torch.utils.data import Dataset, TensorDataset, DataLoader\n",
        "\n",
        "# Loading dataset\n",
        "df = pd.read_csv(\"Tweets.csv\")\n",
        "\n",
        "# Label encoder\n",
        "le = LabelEncoder()\n",
        "y = le.fit_transform(df[\"airline_sentiment\"].values)\n",
        "df[\"label\"] = y\n",
        "\n",
        "# Splint into 60% train, 20% valid, 20% test\n",
        "train_df, test_df = train_test_split(\n",
        "    df, test_size=0.2, random_state=1)\n",
        "\n",
        "train_df, valid_df = train_test_split(\n",
        "    train_df, test_size=0.25, random_state=1)  # 0.25 x 0.8 = 0.2\n",
        "\n",
        "print(len(train_df), len(valid_df), len(test_df))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Yc77syuIXcoe"
      },
      "source": [
        "df.columns"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7aJuP3eXe0FS"
      },
      "source": [
        "class TweetDataset(Dataset):\n",
        "    def __init__(self,\n",
        "                 df,\n",
        "                 tokenizer,\n",
        "                 max_length=256):\n",
        "        self.df = df\n",
        "        input_ids = []\n",
        "        for text in self.df[\"text\"].tolist():\n",
        "            d = tokenizer(text,\n",
        "                          max_length=max_length,\n",
        "                          padding=\"max_length\",\n",
        "                          return_tensors=\"pt\")\n",
        "            for k, v in d.items():\n",
        "                # To remove unnecessary list\n",
        "                d[k] = v.squeeze(0)\n",
        "            input_ids.append(d)\n",
        "\n",
        "        self.df[\"input_ids\"] = input_ids\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.df)\n",
        "\n",
        "    def __getitem__(self, idx):\n",
        "        return {**self.df.iloc[idx][\"input_ids\"],\n",
        "                \"labels\": self.df.iloc[idx][\"label\"]}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7N_h3PnH6MSM"
      },
      "source": [
        "### Trainer!\n",
        "\n",
        "So far, we have used a hand-made training function. `transformers` has the `Trainer` class that takes care of customizable training procedures.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CwsoZN7NGSn2"
      },
      "source": [
        "from transformers import (\n",
        "    AutoModelForSequenceClassification,\n",
        "    AutoTokenizer,\n",
        "    Trainer,\n",
        "    TrainingArguments\n",
        ")\n",
        "\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n",
        "model = AutoModelForSequenceClassification.from_pretrained(\"bert-base-uncased\", num_labels=3)\n",
        "\n",
        "\n",
        "train_dataset = TweetDataset(train_df, tokenizer, max_length=256)\n",
        "valid_dataset = TweetDataset(valid_df, tokenizer, max_length=256)\n",
        "test_dataset = TweetDataset(test_df, tokenizer, max_length=256)\n",
        "\n",
        "\n",
        "# https://huggingface.co/transformers/main_classes/trainer.html#trainingarguments\n",
        "training_args = TrainingArguments(\n",
        "    output_dir='./results',          # output directory\n",
        "    num_train_epochs=3,              # total number of training epochs\n",
        "    per_device_train_batch_size=16,  # batch size per device during training\n",
        "    per_device_eval_batch_size=64,   # batch size for evaluation\n",
        "    warmup_steps=500,                # number of warmup steps for learning rate scheduler\n",
        "    weight_decay=0.01,               # strength of weight decay\n",
        "    logging_dir='./logs',            # directory for storing logs\n",
        "    logging_steps=100,\n",
        ")\n",
        "\n",
        "# \n",
        "trainer = Trainer(\n",
        "    model=model,                         # the instantiated 🤗 Transformers model to be trained\n",
        "    args=training_args,                  # training arguments, defined above\n",
        "    train_dataset=train_dataset,         # training dataset\n",
        "    eval_dataset=valid_dataset             # evaluation dataset\n",
        ")\n",
        "\n",
        "trainer.train()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TvRnct-zQOxM"
      },
      "source": [
        "%ls"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lCHUndBNdvre"
      },
      "source": [
        "%ls results/checkpoint-500"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8tnW5r6vd8zW"
      },
      "source": [
        "### Visualizing the experiment with TensorBoard\n",
        "\n",
        "As shown in the name, it's originally developed as part of the Tensorflow framework, but it is now compatible with PyTorch and other frameowkrks.\n",
        "\n",
        "https://www.tensorflow.org/tensorboard\n",
        "\n",
        "You can use Tensorboard in Google Coalb by just running the following two lines. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gu_04LpdPfPh"
      },
      "source": [
        "%load_ext tensorboard\n",
        "%tensorboard --logdir logs"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}