code_practices/TextFooler_Tutorial.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "TextFooler_Tutorial",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyPe97dQLaP2cHBHJsDWbIRg",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ndb796/Deep-Learning-Paper-Review-and-Practice/blob/master/code_practices/TextFooler_Tutorial.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u0g8gb8fXAwa"
      },
      "source": [
        "#### <b>TextFooler: 텍스트 분류 모델 공격 실습</b>\n",
        "\n",
        "* 특정 클래스로 분류되는 하나의 문장이 <b>다른 클래스로 인식</b>되도록 만들어 봅시다.\n",
        "* <b>Black-box attack</b>: 공격자가 공격 대상 모델의 파라미터를 모르지만 <b>쿼리(query)</b>를 날려 confidence score를 구할 수 있습니다.\n",
        "* <b>Wore-wise perturbing attack</b>: 문장 내 단어(word)를 하나씩 변경하는 방식으로 공격합니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-ma7OHZpJePk"
      },
      "source": [
        "#### <b>기본 라이브러리 설치</b>\n",
        "\n",
        "* <b>TextFooler</b> 소스코드를 다운로드합니다.\n",
        "* <b>필수적인 라이브러리</b>: Python, TensorFlow, Torch, NLTK 등을 설치</b>합니다.\n",
        "* <b>[런타임]</b> → <b>[런타임 유형 변경]</b> → <b>[GPU]</b>\n",
        "* Colab Notebook 재부팅이 필요하다면 <b>[Restart Runtime]</b> 버튼을 누릅니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kbZA-iUZTN04",
        "outputId": "0ffc30a7-a646-4209-8599-81f0f9773e9d"
      },
      "source": [
        "# TextFooler 소스코드 다운로드 및 필수 라이브러리 설치 (+ 오류가 발생하지 않도록 Boto3 최신 라이브러리 설치)\n",
        "!git clone https://github.com/ndb796/TextFooler\n",
        "%cd TextFooler\n",
        "!pip install -r requirements.txt"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "fatal: destination path 'TextFooler' already exists and is not an empty directory.\n",
            "/content/TextFooler\n",
            "Requirement already satisfied: absl-py==0.9.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 1)) (0.9.0)\n",
            "Requirement already satisfied: astor==0.8.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 2)) (0.8.1)\n",
            "Requirement already satisfied: beautifulsoup4==4.9.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 3)) (4.9.1)\n",
            "Requirement already satisfied: boto3==1.16.23 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 4)) (1.16.23)\n",
            "Requirement already satisfied: botocore==1.19.23 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 5)) (1.19.23)\n",
            "Requirement already satisfied: certifi==2020.4.5.2 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 6)) (2020.4.5.2)\n",
            "Requirement already satisfied: chardet==3.0.4 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 7)) (3.0.4)\n",
            "Requirement already satisfied: click==7.1.2 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 8)) (7.1.2)\n",
            "Requirement already satisfied: docutils==0.15.2 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 9)) (0.15.2)\n",
            "Requirement already satisfied: feedparser==5.2.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 10)) (5.2.1)\n",
            "Requirement already satisfied: gast==0.3.3 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 11)) (0.3.3)\n",
            "Requirement already satisfied: grpcio==1.29.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 12)) (1.29.0)\n",
            "Requirement already satisfied: h5py==2.10.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 13)) (2.10.0)\n",
            "Requirement already satisfied: idna==2.9 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 14)) (2.9)\n",
            "Requirement already satisfied: importlib-metadata==1.6.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 15)) (1.6.1)\n",
            "Requirement already satisfied: jmespath==0.10.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 16)) (0.10.0)\n",
            "Requirement already satisfied: joblib==0.15.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 17)) (0.15.1)\n",
            "Requirement already satisfied: Keras-Applications==1.0.8 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 18)) (1.0.8)\n",
            "Requirement already satisfied: Keras-Preprocessing==1.1.2 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 19)) (1.1.2)\n",
            "Requirement already satisfied: lxml==4.5.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 20)) (4.5.1)\n",
            "Requirement already satisfied: Markdown==3.2.2 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 21)) (3.2.2)\n",
            "Requirement already satisfied: nltk==3.5 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 22)) (3.5)\n",
            "Requirement already satisfied: numpy==1.18.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 23)) (1.18.0)\n",
            "Requirement already satisfied: protobuf==3.12.2 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 24)) (3.12.2)\n",
            "Requirement already satisfied: python-dateutil==2.8.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 25)) (2.8.1)\n",
            "Requirement already satisfied: python-docx==0.8.10 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 26)) (0.8.10)\n",
            "Requirement already satisfied: regex==2020.6.8 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 27)) (2020.6.8)\n",
            "Requirement already satisfied: requests==2.23.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 28)) (2.23.0)\n",
            "Requirement already satisfied: s3transfer==0.3.3 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 29)) (0.3.3)\n",
            "Requirement already satisfied: six==1.15.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 30)) (1.15.0)\n",
            "Requirement already satisfied: soupsieve==2.0.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 31)) (2.0.1)\n",
            "Requirement already satisfied: tensorboard==2.3.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 32)) (2.3.0)\n",
            "Requirement already satisfied: termcolor==1.1.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 33)) (1.1.0)\n",
            "Requirement already satisfied: torch==1.7.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 34)) (1.7.0+cu101)\n",
            "Requirement already satisfied: tqdm==4.46.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 35)) (4.46.1)\n",
            "Requirement already satisfied: urllib3==1.25.9 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 36)) (1.25.9)\n",
            "Requirement already satisfied: Werkzeug==1.0.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 37)) (1.0.1)\n",
            "Requirement already satisfied: zipp==3.1.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 38)) (3.1.0)\n",
            "Requirement already satisfied: imgaug==0.2.5 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 39)) (0.2.5)\n",
            "Requirement already satisfied: folium==0.2.1 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 40)) (0.2.1)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf==3.12.2->-r requirements.txt (line 24)) (50.3.2)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard==2.3.0->-r requirements.txt (line 32)) (1.7.0)\n",
            "Requirement already satisfied: wheel>=0.26; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from tensorboard==2.3.0->-r requirements.txt (line 32)) (0.35.1)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard==2.3.0->-r requirements.txt (line 32)) (0.4.2)\n",
            "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard==2.3.0->-r requirements.txt (line 32)) (1.17.2)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch==1.7.0->-r requirements.txt (line 34)) (0.16.0)\n",
            "Requirement already satisfied: dataclasses in /usr/local/lib/python3.6/dist-packages (from torch==1.7.0->-r requirements.txt (line 34)) (0.7)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.6/dist-packages (from torch==1.7.0->-r requirements.txt (line 34)) (3.7.4.3)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from imgaug==0.2.5->-r requirements.txt (line 39)) (1.4.1)\n",
            "Requirement already satisfied: scikit-image>=0.11.0 in /usr/local/lib/python3.6/dist-packages (from imgaug==0.2.5->-r requirements.txt (line 39)) (0.16.2)\n",
            "Requirement already satisfied: Jinja2 in /usr/local/lib/python3.6/dist-packages (from folium==0.2.1->-r requirements.txt (line 40)) (2.11.2)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard==2.3.0->-r requirements.txt (line 32)) (1.3.0)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard==2.3.0->-r requirements.txt (line 32)) (0.2.8)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard==2.3.0->-r requirements.txt (line 32)) (4.1.1)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard==2.3.0->-r requirements.txt (line 32)) (4.6)\n",
            "Requirement already satisfied: matplotlib!=3.0.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image>=0.11.0->imgaug==0.2.5->-r requirements.txt (line 39)) (3.2.2)\n",
            "Requirement already satisfied: imageio>=2.3.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image>=0.11.0->imgaug==0.2.5->-r requirements.txt (line 39)) (2.4.1)\n",
            "Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image>=0.11.0->imgaug==0.2.5->-r requirements.txt (line 39)) (2.5)\n",
            "Requirement already satisfied: PyWavelets>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image>=0.11.0->imgaug==0.2.5->-r requirements.txt (line 39)) (1.1.1)\n",
            "Requirement already satisfied: pillow>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image>=0.11.0->imgaug==0.2.5->-r requirements.txt (line 39)) (7.0.0)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.6/dist-packages (from Jinja2->folium==0.2.1->-r requirements.txt (line 40)) (1.1.1)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard==2.3.0->-r requirements.txt (line 32)) (3.1.0)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.6/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard==2.3.0->-r requirements.txt (line 32)) (0.4.8)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image>=0.11.0->imgaug==0.2.5->-r requirements.txt (line 39)) (2.4.7)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image>=0.11.0->imgaug==0.2.5->-r requirements.txt (line 39)) (0.10.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image>=0.11.0->imgaug==0.2.5->-r requirements.txt (line 39)) (1.3.1)\n",
            "Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx>=2.0->scikit-image>=0.11.0->imgaug==0.2.5->-r requirements.txt (line 39)) (4.4.2)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ljts3E-jirEX"
      },
      "source": [
        "%%capture\n",
        "# Google Colab에서 실행하기 위해 추가적으로 Pattern 라이브러리 설치\n",
        "!wget https://files.pythonhosted.org/packages/1e/07/b0e61b6c818ed4b6145fe01d1c341223aa6cfbc3928538ad1f2b890924a3/Pattern-3.6.0.tar.gz\n",
        "!sudo tar -xvf Pattern-3.6.0.tar.gz\n",
        "%cd Pattern-3.6\n",
        "!python setup.py install\n",
        "%cd .."
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1rQusdGVTVzr"
      },
      "source": [
        "#### <b>워드 임베딩(Word Embedding) 불러오기</b>\n",
        "\n",
        "* <b>워드 임베딩(Word Embedding)</b>을 위한 Counter Fitted Vector 텍스트 파일을 다운로드합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BgaX_gKqUPLN",
        "outputId": "15cac5ab-db56-4a31-fc36-5c4c4dc32d07"
      },
      "source": [
        "!wget https://postechackr-my.sharepoint.com/:u:/g/personal/dongbinna_postech_ac_kr/ESV6s3ZdmTxLiDpzmEc2LQgBmEQF2C_0kqCsu013uq80gg?download=1 -O counter-fitted-vector.txt.zip\n",
        "!unzip counter-fitted-vector.txt.zip"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-11-20 22:05:26--  https://postechackr-my.sharepoint.com/:u:/g/personal/dongbinna_postech_ac_kr/ESV6s3ZdmTxLiDpzmEc2LQgBmEQF2C_0kqCsu013uq80gg?download=1\n",
            "Resolving postechackr-my.sharepoint.com (postechackr-my.sharepoint.com)... 40.108.156.33\n",
            "Connecting to postechackr-my.sharepoint.com (postechackr-my.sharepoint.com)|40.108.156.33|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: /personal/dongbinna_postech_ac_kr/Documents/Research/datasets/counter-fitted-vectors.txt.zip?originalPath=aHR0cHM6Ly9wb3N0ZWNoYWNrci1teS5zaGFyZXBvaW50LmNvbS86dTovZy9wZXJzb25hbC9kb25nYmlubmFfcG9zdGVjaF9hY19rci9FU1Y2czNaZG1UeExpRHB6bUVjMkxRZ0JtRVFGMkNfMGtxQ3N1MDEzdXE4MGdnP3J0aW1lPWZmT0RZNkNOMkVn [following]\n",
            "--2020-11-20 22:05:27--  https://postechackr-my.sharepoint.com/personal/dongbinna_postech_ac_kr/Documents/Research/datasets/counter-fitted-vectors.txt.zip?originalPath=aHR0cHM6Ly9wb3N0ZWNoYWNrci1teS5zaGFyZXBvaW50LmNvbS86dTovZy9wZXJzb25hbC9kb25nYmlubmFfcG9zdGVjaF9hY19rci9FU1Y2czNaZG1UeExpRHB6bUVjMkxRZ0JtRVFGMkNfMGtxQ3N1MDEzdXE4MGdnP3J0aW1lPWZmT0RZNkNOMkVn\n",
            "Reusing existing connection to postechackr-my.sharepoint.com:443.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 65009363 (62M) [application/x-zip-compressed]\n",
            "Saving to: ‘counter-fitted-vector.txt.zip’\n",
            "\n",
            "counter-fitted-vect 100%[===================>]  62.00M  12.0MB/s    in 6.6s    \n",
            "\n",
            "2020-11-20 22:05:34 (9.44 MB/s) - ‘counter-fitted-vector.txt.zip’ saved [65009363/65009363]\n",
            "\n",
            "Archive:  counter-fitted-vector.txt.zip\n",
            "  inflating: counter-fitted-vectors.txt  \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3fzZocPJnQI4"
      },
      "source": [
        "* 오리지널 저장소에서 제안한 사용 가능한 단어의 개수가 너무 많으므로 <b>단어의 개수를 줄여 실습</b>합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "urbElcOQJoT_",
        "outputId": "dc283da5-ab65-4031-fc62-97c862a483b2"
      },
      "source": [
        "with open(\"counter-fitted-vectors.txt\", \"r\") as f:\n",
        "    words = f.readlines()\n",
        "\n",
        "# 사용할 단어의 총 개수(약 60,000개)\n",
        "print(\"단어의 개수:\", len(words))\n",
        "\n",
        "# 단어의 개수 줄이기: Google Colab 메모리 오류가 나지 않도록 하기(18,000개의 단어만 사용)\n",
        "target_size = 18000\n",
        "print(\"필터링 된 단어의 개수:\", target_size)\n",
        "\n",
        "with open(\"counter-fitted-vectors.txt\", \"w\") as f:\n",
        "    f.write(''.join(words[:target_size]))"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "단어의 개수: 65713\n",
            "필터링 된 단어의 개수: 18000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3-GXFLXkT0ZT"
      },
      "source": [
        "#### <b>공격 모델(Targeted Model) 불러오기: BERT</b>\n",
        "\n",
        "* <b>워드 임베딩(Word Embedding)</b>을 위한 Counter Fitted Vector 텍스트 파일을 다운로드합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8rTUw6IkUQ5i",
        "outputId": "1d902ea5-5dc7-4b78-c2fc-fee6415b3b18"
      },
      "source": [
        "!rm -rf ./bert_mr\n",
        "!mkdir -p ./bert_mr\n",
        "!wget https://postechackr-my.sharepoint.com/:u:/g/personal/dongbinna_postech_ac_kr/EYuRnUyF9DRKtGxsO--wAqoBG0mIo56jaoJGY4TQwW-8-g?download=1 -O ./bert_mr/pytorch_model.bin\n",
        "!wget https://postechackr-my.sharepoint.com/:u:/g/personal/dongbinna_postech_ac_kr/ERtJ3QAZl-RDk_iPG8kzf5EBBAPEClPfXknXswTpKA0dvQ?download=1 -O ./bert_mr/bert_config.json\n",
        "!wget https://postechackr-my.sharepoint.com/:t:/g/personal/dongbinna_postech_ac_kr/EQSBPRwI6AJLheb_KtCOWEsBud5YTH6KtZ5X1srV1q5DDA?download=1 -O ./bert_mr/vocab.txt"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-11-20 22:05:37--  https://postechackr-my.sharepoint.com/:u:/g/personal/dongbinna_postech_ac_kr/EYuRnUyF9DRKtGxsO--wAqoBG0mIo56jaoJGY4TQwW-8-g?download=1\n",
            "Resolving postechackr-my.sharepoint.com (postechackr-my.sharepoint.com)... 40.108.156.33\n",
            "Connecting to postechackr-my.sharepoint.com (postechackr-my.sharepoint.com)|40.108.156.33|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: /personal/dongbinna_postech_ac_kr/Documents/Research/models/BERT%20for%20Movie%20Review/pytorch_model.bin?originalPath=aHR0cHM6Ly9wb3N0ZWNoYWNrci1teS5zaGFyZXBvaW50LmNvbS86dTovZy9wZXJzb25hbC9kb25nYmlubmFfcG9zdGVjaF9hY19rci9FWXVSblV5RjlEUkt0R3hzTy0td0Fxb0JHMG1JbzU2amFvSkdZNFRRd1ctOC1nP3J0aW1lPTY0YnNhYUNOMkVn [following]\n",
            "--2020-11-20 22:05:38--  https://postechackr-my.sharepoint.com/personal/dongbinna_postech_ac_kr/Documents/Research/models/BERT%20for%20Movie%20Review/pytorch_model.bin?originalPath=aHR0cHM6Ly9wb3N0ZWNoYWNrci1teS5zaGFyZXBvaW50LmNvbS86dTovZy9wZXJzb25hbC9kb25nYmlubmFfcG9zdGVjaF9hY19rci9FWXVSblV5RjlEUkt0R3hzTy0td0Fxb0JHMG1JbzU2amFvSkdZNFRRd1ctOC1nP3J0aW1lPTY0YnNhYUNOMkVn\n",
            "Reusing existing connection to postechackr-my.sharepoint.com:443.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 437983779 (418M) [application/octet-stream]\n",
            "Saving to: ‘./bert_mr/pytorch_model.bin’\n",
            "\n",
            "./bert_mr/pytorch_m 100%[===================>] 417.69M  18.4MB/s    in 23s     \n",
            "\n",
            "2020-11-20 22:06:02 (18.0 MB/s) - ‘./bert_mr/pytorch_model.bin’ saved [437983779/437983779]\n",
            "\n",
            "--2020-11-20 22:06:02--  https://postechackr-my.sharepoint.com/:u:/g/personal/dongbinna_postech_ac_kr/ERtJ3QAZl-RDk_iPG8kzf5EBBAPEClPfXknXswTpKA0dvQ?download=1\n",
            "Resolving postechackr-my.sharepoint.com (postechackr-my.sharepoint.com)... 40.108.156.33\n",
            "Connecting to postechackr-my.sharepoint.com (postechackr-my.sharepoint.com)|40.108.156.33|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://postechackr-my.sharepoint.com/personal/dongbinna_postech_ac_kr/_layouts/15/download.aspx?UniqueId=00dd491b971943e493f88f1bc9337f91&originalPath=aHR0cHM6Ly9wb3N0ZWNoYWNrci1teS5zaGFyZXBvaW50LmNvbS86dTovZy9wZXJzb25hbC9kb25nYmlubmFfcG9zdGVjaF9hY19rci9FUnRKM1FBWmwtUkRrX2lQRzhremY1RUJCQVBFQ2xQZlhrblhzd1RwS0EwZHZRP3J0aW1lPVd5ekhlS0NOMkVn [following]\n",
            "--2020-11-20 22:06:03--  https://postechackr-my.sharepoint.com/personal/dongbinna_postech_ac_kr/_layouts/15/download.aspx?UniqueId=00dd491b971943e493f88f1bc9337f91&originalPath=aHR0cHM6Ly9wb3N0ZWNoYWNrci1teS5zaGFyZXBvaW50LmNvbS86dTovZy9wZXJzb25hbC9kb25nYmlubmFfcG9zdGVjaF9hY19rci9FUnRKM1FBWmwtUkRrX2lQRzhremY1RUJCQVBFQ2xQZlhrblhzd1RwS0EwZHZRP3J0aW1lPVd5ekhlS0NOMkVn\n",
            "Reusing existing connection to postechackr-my.sharepoint.com:443.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 313 [application/json]\n",
            "Saving to: ‘./bert_mr/bert_config.json’\n",
            "\n",
            "./bert_mr/bert_conf 100%[===================>]     313  --.-KB/s    in 0s      \n",
            "\n",
            "2020-11-20 22:06:03 (72.5 MB/s) - ‘./bert_mr/bert_config.json’ saved [313/313]\n",
            "\n",
            "--2020-11-20 22:06:03--  https://postechackr-my.sharepoint.com/:t:/g/personal/dongbinna_postech_ac_kr/EQSBPRwI6AJLheb_KtCOWEsBud5YTH6KtZ5X1srV1q5DDA?download=1\n",
            "Resolving postechackr-my.sharepoint.com (postechackr-my.sharepoint.com)... 40.108.156.33\n",
            "Connecting to postechackr-my.sharepoint.com (postechackr-my.sharepoint.com)|40.108.156.33|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: /personal/dongbinna_postech_ac_kr/Documents/Research/models/BERT%20for%20Movie%20Review/vocab.txt?originalPath=aHR0cHM6Ly9wb3N0ZWNoYWNrci1teS5zaGFyZXBvaW50LmNvbS86dDovZy9wZXJzb25hbC9kb25nYmlubmFfcG9zdGVjaF9hY19rci9FUVNCUFJ3STZBSkxoZWJfS3RDT1dFc0J1ZDVZVEg2S3RaNVgxc3JWMXE1RERBP3J0aW1lPTBiTmRlYUNOMkVn [following]\n",
            "--2020-11-20 22:06:04--  https://postechackr-my.sharepoint.com/personal/dongbinna_postech_ac_kr/Documents/Research/models/BERT%20for%20Movie%20Review/vocab.txt?originalPath=aHR0cHM6Ly9wb3N0ZWNoYWNrci1teS5zaGFyZXBvaW50LmNvbS86dDovZy9wZXJzb25hbC9kb25nYmlubmFfcG9zdGVjaF9hY19rci9FUVNCUFJ3STZBSkxoZWJfS3RDT1dFc0J1ZDVZVEg2S3RaNVgxc3JWMXE1RERBP3J0aW1lPTBiTmRlYUNOMkVn\n",
            "Reusing existing connection to postechackr-my.sharepoint.com:443.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 231508 (226K) [text/plain]\n",
            "Saving to: ‘./bert_mr/vocab.txt’\n",
            "\n",
            "./bert_mr/vocab.txt 100%[===================>] 226.08K   468KB/s    in 0.5s    \n",
            "\n",
            "2020-11-20 22:06:05 (468 KB/s) - ‘./bert_mr/vocab.txt’ saved [231508/231508]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "csRzq8lgVdX1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "11fdf686-6f63-45d5-fce5-b23eb2cd007a"
      },
      "source": [
        "# NLTK 라이브러리 설치\n",
        "import nltk\n",
        "\n",
        "nltk.download('averaged_perceptron_tagger')\n",
        "nltk.download('universal_tagset')"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
            "[nltk_data]       date!\n",
            "[nltk_data] Downloading package universal_tagset to /root/nltk_data...\n",
            "[nltk_data]   Package universal_tagset is already up-to-date!\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XKgaatoVUJgS"
      },
      "source": [
        "#### <b>TextFooler 공격 수행</b>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "C7QeVkcQLV7X",
        "outputId": "4b384c13-1403-4d1c-cfa7-a8af576029d1"
      },
      "source": [
        "!python attack_classification.py \\\n",
        "    --dataset_path=./data/mr \\\n",
        "    --target_model=bert \\\n",
        "    --target_model_path=./bert_mr/ \\\n",
        "    --counter_fitting_embeddings_path=counter-fitted-vectors.txt \\\n",
        "    --USE_cache_path=./"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-11-20 22:06:39.069142: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
            "Data import finished!\n",
            "Building Model...\n",
            "Model built!\n",
            "Building vocab...\n",
            "Building cos sim matrix...\n",
            "Start computing the cosine similarity matrix!\n",
            "tcmalloc: large alloc 2592006144 bytes == 0x11486a000 @  0x7f000548b1e7 0x7f0002ef15e1 0x7f0002f55e88 0x7f0002f56034 0x7f0002f4c96c 0x7f000318947c 0x7f0002fe8e00 0x7f0002fed74a 0x566f73 0x59fd0e 0x7f0002f4170d 0x50a12f 0x50beb4 0x507be4 0x509900 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x50ad03 0x634e72 0x634f27 0x6386df 0x639281 0x4b0dc0 0x7f0005088bf7 0x5b259a\n",
            "tcmalloc: large alloc 2592006144 bytes == 0x7efe33f52000 @  0x7f000548b1e7 0x7f0002ef15e1 0x7f0002f55e88 0x7f0002f56034 0x7f0002f4c96c 0x7f000318947c 0x7f0002fe8e00 0x7f0002fed74a 0x566f73 0x59fd0e 0x7f0002f4170d 0x50a12f 0x50beb4 0x507be4 0x509900 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x50ad03 0x634e72 0x634f27 0x6386df 0x639281 0x4b0dc0 0x7f0005088bf7 0x5b259a\n",
            "tcmalloc: large alloc 2592006144 bytes == 0x7efd99764000 @  0x7f000548b1e7 0x7f0002ef15e1 0x7f0002f55e88 0x7f0002f55fa3 0x7f000314e227 0x7f0003150682 0x7f00031514fe 0x5a48ec 0x5a4fb8 0x7f00030149e7 0x59c922 0x50ea2d 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x50ad03 0x634e72 0x634f27 0x6386df 0x639281 0x4b0dc0 0x7f0005088bf7 0x5b259a\n",
            "Cos sim import finished!\n",
            "2020-11-20 22:07:21.760431: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2200000000 Hz\n",
            "2020-11-20 22:07:21.760746: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xfc82d800 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
            "2020-11-20 22:07:21.760784: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n",
            "2020-11-20 22:07:21.769660: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1\n",
            "2020-11-20 22:07:21.798180: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-11-20 22:07:21.799056: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xfc82d9c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:\n",
            "2020-11-20 22:07:21.799088: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla T4, Compute Capability 7.5\n",
            "2020-11-20 22:07:21.803974: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-11-20 22:07:21.804549: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: \n",
            "pciBusID: 0000:00:04.0 name: Tesla T4 computeCapability: 7.5\n",
            "coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.73GiB deviceMemoryBandwidth: 298.08GiB/s\n",
            "2020-11-20 22:07:21.804601: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
            "2020-11-20 22:07:22.050804: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10\n",
            "2020-11-20 22:07:22.197327: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10\n",
            "2020-11-20 22:07:22.229906: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10\n",
            "2020-11-20 22:07:22.494565: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10\n",
            "2020-11-20 22:07:22.510740: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10\n",
            "2020-11-20 22:07:23.011188: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7\n",
            "2020-11-20 22:07:23.011414: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-11-20 22:07:23.012142: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-11-20 22:07:23.012775: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0\n",
            "2020-11-20 22:07:23.016084: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
            "2020-11-20 22:07:23.022367: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
            "2020-11-20 22:07:23.022411: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 \n",
            "2020-11-20 22:07:23.022429: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N \n",
            "2020-11-20 22:07:23.022599: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-11-20 22:07:23.023276: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-11-20 22:07:23.023982: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 12733 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)\n",
            "Start attacking!\n",
            "0 samples out of 1000 have been finished!\n",
            "2020-11-20 22:07:28.624081: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10\n",
            "20 samples out of 1000 have been finished!\n",
            "40 samples out of 1000 have been finished!\n",
            "60 samples out of 1000 have been finished!\n",
            "80 samples out of 1000 have been finished!\n",
            "100 samples out of 1000 have been finished!\n",
            "120 samples out of 1000 have been finished!\n",
            "140 samples out of 1000 have been finished!\n",
            "160 samples out of 1000 have been finished!\n",
            "180 samples out of 1000 have been finished!\n",
            "200 samples out of 1000 have been finished!\n",
            "220 samples out of 1000 have been finished!\n",
            "240 samples out of 1000 have been finished!\n",
            "260 samples out of 1000 have been finished!\n",
            "280 samples out of 1000 have been finished!\n",
            "300 samples out of 1000 have been finished!\n",
            "320 samples out of 1000 have been finished!\n",
            "340 samples out of 1000 have been finished!\n",
            "360 samples out of 1000 have been finished!\n",
            "380 samples out of 1000 have been finished!\n",
            "400 samples out of 1000 have been finished!\n",
            "420 samples out of 1000 have been finished!\n",
            "440 samples out of 1000 have been finished!\n",
            "460 samples out of 1000 have been finished!\n",
            "480 samples out of 1000 have been finished!\n",
            "500 samples out of 1000 have been finished!\n",
            "520 samples out of 1000 have been finished!\n",
            "540 samples out of 1000 have been finished!\n",
            "560 samples out of 1000 have been finished!\n",
            "580 samples out of 1000 have been finished!\n",
            "600 samples out of 1000 have been finished!\n",
            "620 samples out of 1000 have been finished!\n",
            "640 samples out of 1000 have been finished!\n",
            "660 samples out of 1000 have been finished!\n",
            "680 samples out of 1000 have been finished!\n",
            "700 samples out of 1000 have been finished!\n",
            "720 samples out of 1000 have been finished!\n",
            "740 samples out of 1000 have been finished!\n",
            "760 samples out of 1000 have been finished!\n",
            "780 samples out of 1000 have been finished!\n",
            "800 samples out of 1000 have been finished!\n",
            "820 samples out of 1000 have been finished!\n",
            "840 samples out of 1000 have been finished!\n",
            "860 samples out of 1000 have been finished!\n",
            "880 samples out of 1000 have been finished!\n",
            "900 samples out of 1000 have been finished!\n",
            "920 samples out of 1000 have been finished!\n",
            "940 samples out of 1000 have been finished!\n",
            "960 samples out of 1000 have been finished!\n",
            "980 samples out of 1000 have been finished!\n",
            "For target model bert: original accuracy: 90.400%, adv accuracy: 79.700%, avg changed rate: 15.323%, num of queries: 53.2\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "plfDK5oJmXww"
      },
      "source": [
        "* <b>공격 결과물(Adversarial Texts)</b>을 출력합니다.\n",
        "* 정상적으로 분류되는 텍스트 904개 중에서 <b>107개에 대하여 분류 결과를 변경</b>할 수 있습니다.\n",
        "* 선택 가능한 단어의 수를 줄였기 때문에, 본 결과는 <b>원본 논문보다는 낮은 공격 성공률</b>을 보입니다.\n",
        "    * 10,000개만 쓸 때의 정확도 변화: 90.400% → 86.500%\n",
        "    * <b>18,000개</b>만 쓸 때의 정확도 변화: 90.400% → 79.700%"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-eCPHfChmbo1",
        "outputId": "d1cfa6c7-986a-4fad-8c19-9f3d91511296"
      },
      "source": [
        "!cat ./adv_results/adversaries.txt"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "orig sent (0):\tdavis is so enamored of her own creation that she ca n't see how insufferable the character is\n",
            "adv sent (1):\tdavis is so elated of her own creation that she ca n't see how insufferable the character is\n",
            "\n",
            "orig sent (1):\ta muckraking job , the cinematic equivalent of a legal indictment , and a fairly effective one at that\n",
            "adv sent (0):\ta muckraking job , the cinematic equivalent of a legal indictment , and a reasonably reasonable one at that\n",
            "\n",
            "orig sent (1):\twriter director 's mehta 's effort has tons of charm and the whimsy is in the mixture , the intoxicating masala , of cultures and film genres\n",
            "adv sent (0):\twriter director 's mehta 's effort possessed shitloads of charm and the triviality is in the mixture , the bitters masala , of cropping and film genres\n",
            "\n",
            "orig sent (1):\tmuch of what we see is horrible but it 's also undeniably exceedingly clever\n",
            "adv sent (0):\tkeenly of what we see is horrible but it 's also undeniably ridiculously clever\n",
            "\n",
            "orig sent (1):\trife with the rueful , wry humor springing out of yiddish culture and language\n",
            "adv sent (0):\trampant with the rueful , jokey humor springing out of rabbinical culture and language\n",
            "\n",
            "orig sent (1):\tthose who do n't entirely 'get' godard 's distinctive discourse will still come away with a sense of his reserved but existential poignancy\n",
            "adv sent (0):\tthose who do n't entirely 'get' godard 's distinctive discourse will increasingly come away with a foreboding of his reserved but existential poignancy\n",
            "\n",
            "orig sent (0):\ttwenty three movies into a mostly magnificent directorial career , clint eastwood 's efficiently minimalist style finally has failed him big time\n",
            "adv sent (1):\ttwenty three movies into a mostly magnificent directorial career , clint eastwood 's efficiently minimalist style finally has beached him big time\n",
            "\n",
            "orig sent (1):\tparker holds true to wilde 's own vision of a pure comedy with absolutely no meaning , and no desire to be anything but a polished , sophisticated entertainment that is in love with its own cleverness\n",
            "adv sent (0):\tparker obligated true to wilde 's own designing of a pure comedy with drastically no sentiment , and no desire to be anything but a graceful , sophisticated entertainment that is in love with its own whimsy\n",
            "\n",
            "orig sent (0):\tthe plot is romantic comedy boilerplate from start to finish\n",
            "adv sent (1):\tthe plot is romantic wry boilerplate from start to finish\n",
            "\n",
            "orig sent (0):\tthe movie is concocted and carried out by folks worthy of scorn , and the nicest thing i can say is that i ca n't remember a single name responsible for it\n",
            "adv sent (1):\tthe movie is concocted and accomplished out by folks worthy of scorn , and the fairest thing i can say is that i ca n't remember a single name responsible for it\n",
            "\n",
            "orig sent (1):\tbrian tufano 's handsome widescreen photography and paul grabowsky 's excellent music turn this fairly parochial melodrama into something really rather special\n",
            "adv sent (0):\tbrian tufano 's handsome widescreen photography and paul grabowsky 's excellent music swivel this fairly parochial melodrama into something really moderately sole\n",
            "\n",
            "orig sent (1):\tas comedic spotlights go , notorious c h o hits all the verbal marks it should\n",
            "adv sent (0):\tas comedic spotlights disappear , notorious c h o smashes all the verbal marks it should\n",
            "\n",
            "orig sent (0):\tjust like the deli sandwich lots of ham , lots of cheese , with a sickly sweet coating to disguise its excrescence until just after \\( or during \\) consumption of its second half\n",
            "adv sent (1):\tjust like the café sandwich cargoes of prosciutto , cargoes of cheese , with a sickly tasty coating to hides its excrescence until just after \\( or during \\) consumption of its second half\n",
            "\n",
            "orig sent (0):\tequal parts bodice ripper and plodding costume drama\n",
            "adv sent (1):\tequitable parts bodice ripper and thundering costume drama\n",
            "\n",
            "orig sent (1):\thow good this film might be , depends if you believe that the shocking conclusion is too much of a plunge or not\n",
            "adv sent (0):\thow good this film might be , depends if you believe that the shocking conclude is too utterly of a plunge or not\n",
            "\n",
            "orig sent (1):\ti liked this film a lot\n",
            "adv sent (0):\ti rained this film a lot\n",
            "\n",
            "orig sent (0):\twhile we want macdowell 's character to retrieve her husband , we have to ask whether her personal odyssey trumps the carnage that claims so many lives around her\n",
            "adv sent (1):\twhile we desiring macdowell 's character to retrieve her husband , we have to ask whether her personal odyssey trumps the carnage that claims so many lives around her\n",
            "\n",
            "orig sent (0):\tanimated drivel meant to enhance the self image of drooling idiots\n",
            "adv sent (1):\tvivid drivel meant to enhance the self image of drooling idiots\n",
            "\n",
            "orig sent (1):\tit 's excessively quirky and a little underconfident in its delivery , but otherwise this is the best 'old neighborhood' project since christopher walken kinda romanced cyndi lauper in the opportunists\n",
            "adv sent (0):\tit 's excessively quirky and a little underconfident in its delivery , but otherwise this is the higher 'old neighborhood' project since christopher walken kinda romanced cyndi lauper in the opportunists\n",
            "\n",
            "orig sent (1):\tlike all of egoyan 's work , ararat is fiercely intelligent and uncommonly ambitious\n",
            "adv sent (0):\tlike all of egoyan 's work , ararat is fiercely wily and ridiculously ambitious\n",
            "\n",
            "orig sent (0):\tthis time mr burns is trying something in the martin scorsese street realist mode , but his self regarding sentimentality trips him up again\n",
            "adv sent (1):\tthis time mr burns is trying something in the martin scorsese street realist mode , but his independence regarding sentimentality journey him up again\n",
            "\n",
            "orig sent (1):\tthe script is smart , not cloying\n",
            "adv sent (0):\tthe script is canny , not cloying\n",
            "\n",
            "orig sent (1):\tthe movie sticks much closer to hornby 's drop dead confessional tone than the film version of high fidelity did\n",
            "adv sent (0):\tthe movie sticks much closer to hornby 's lowering dead confessional tone than the film version of high fidelity did\n",
            "\n",
            "orig sent (0):\tit 's a documentary that says that the alternate sexuality meant to set you free may require so much relationship maintenance that celibacy can start looking good\n",
            "adv sent (1):\tit 's a documentary that says that the alternate sexuality meant to set you free may require so uncommonly relationship preserved that celibate can opens looking good\n",
            "\n",
            "orig sent (1):\tan intoxicating experience\n",
            "adv sent (0):\tan beverage experimental\n",
            "\n",
            "orig sent (1):\tan hour and a half of joyful solo performance\n",
            "adv sent (0):\tan hour and a half of happier solo performance\n",
            "\n",
            "orig sent (1):\tintriguing and stylish\n",
            "adv sent (0):\tconfusing and stylish\n",
            "\n",
            "orig sent (1):\tan old fashioned scary movie , one that relies on lingering terror punctuated by sudden shocks and not constant bloodshed punctuated by flying guts\n",
            "adv sent (0):\tan old fashioned scary movie , one that relies on being terror punctuated by sudden shocks and not constant bloodshed punctuated by flying guts\n",
            "\n",
            "orig sent (1):\ttime is a beautiful film to watch , an interesting and at times captivating take on loss and loneliness\n",
            "adv sent (0):\ttime is a lustrous film to watch , an creditable and at days captivating selected on squander and loner\n",
            "\n",
            "orig sent (1):\ta solid , well formed satire\n",
            "adv sent (0):\ta solid , well formed ridicule\n",
            "\n",
            "orig sent (1):\ta riveting profile of law enforcement , and a visceral , nasty journey into an urban hades\n",
            "adv sent (0):\ta puzzling characterization of law enforcement , and a visceral , nasty trips into an municipal hades\n",
            "\n",
            "orig sent (1):\tif a horror movie 's primary goal is to frighten and disturb , then they works spectacularly well a shiver inducing , nerve rattling ride\n",
            "adv sent (0):\tif a horror movie 's primary goal is to frighten and disturb , then they works ridiculously well a shiver inducing , nerve creaking ride\n",
            "\n",
            "orig sent (1):\ta delirious celebration of the female orgasm\n",
            "adv sent (0):\ta wishful celebration of the female masturbate\n",
            "\n",
            "orig sent (0):\tproof that a thriller can be sleekly shot , expertly cast , paced with crisp professionalism and still be a letdown if its twists and turns hold no more surprise than yesterday 's weather report\n",
            "adv sent (1):\tproof that a thriller can be sleekly shot , expertly cast , paced with crisp professionalism and ever be a letdown if its twists and turns hold no more surprise than yesterday 's weather report\n",
            "\n",
            "orig sent (1):\ta startling and fresh examination of how the bike still remains an ambiguous icon in chinese society\n",
            "adv sent (0):\ta disquieting and fresh checkups of how the bike subsequently remains an ambiguous arrow in chinese society\n",
            "\n",
            "orig sent (1):\tsmart science fiction for grown ups , with only a few false steps along the way\n",
            "adv sent (0):\tlustrous science fiction for grown ups , with only a few false steps along the way\n",
            "\n",
            "orig sent (0):\ta puzzling experience\n",
            "adv sent (1):\ta shocking experience\n",
            "\n",
            "orig sent (1):\ta riveting documentary\n",
            "adv sent (0):\ta delectable writings\n",
            "\n",
            "orig sent (0):\tmost of the dialogue made me want to pack raw dough in my ears\n",
            "adv sent (1):\tmost of the dialogue made me liked to posse raw dough in my anklets\n",
            "\n",
            "orig sent (1):\ta burst of color , music , and dance that only the most practiced curmudgeon could fail to crack a smile at\n",
            "adv sent (0):\ta implosion of color , music , and dance that only the most practiced curmudgeon could fail to crack a smile at\n",
            "\n",
            "orig sent (0):\tthe sweetest thing leaves a bitter taste\n",
            "adv sent (1):\tthe sweetest thing cards a bitter tasting\n",
            "\n",
            "orig sent (1):\thashiguchi vividly captures the way young japanese live now , chafing against their culture 's manic mix of millennial brusqueness and undying , traditional politesse\n",
            "adv sent (0):\thashiguchi ridiculously captures the way young japanese live now , chafing against their culture 's manic mix of millennial brusqueness and undying , traditional politesse\n",
            "\n",
            "orig sent (1):\ta portrait of hell so shattering it 's impossible to shake\n",
            "adv sent (0):\ta spitting of hell so implosion it 's impossible to shake\n",
            "\n",
            "orig sent (1):\tpainful to watch , but viewers willing to take a chance will be rewarded with two of the year 's most accomplished and riveting film performances\n",
            "adv sent (0):\tpainful to watch , but viewers willing to making a chance will be rewarded with two of the year 's most carried and puzzling film performances\n",
            "\n",
            "orig sent (0):\tthe film has a few cute ideas and several modest chuckles but it is n't exactly kiddie friendly alas , santa is more ho hum than ho ho ho and the snowman \\( who never gets to play that flute \\) has all the charm of a meltdown\n",
            "adv sent (1):\tthe film enjoys a few cute ideas and several restricted chuckles but it is n't exactly kiddie friendly alas , santa is more ho hum than ho ho ho and the snowman \\( who never gets to play that flute \\) holds all the charmer of a meltdown\n",
            "\n",
            "orig sent (0):\ta farce of a parody of a comedy of a premise , it is n't a comparison to reality so much as it is a commentary about our knowledge of films\n",
            "adv sent (1):\ta farce of a parody of a comedy of a scenario , it is n't a comparison to reality so importantly as it is a commentary about our expertise of films\n",
            "\n",
            "orig sent (1):\tbranagh , in his most forceful non shakespeare screen performance , grounds even the softest moments in the angry revolt of his wit\n",
            "adv sent (0):\tbranagh , in his most forceful frm macbeth screen materialization , basis even the smartest moments in the angry revolt of his wit\n",
            "\n",
            "orig sent (0):\tyes they can swim , the title is merely anne sophie birot 's off handed way of saying girls find adolescence difficult to wade through\n",
            "adv sent (1):\tyes they can swim , the title is scarcely anne sophie birot 's off handed way of saying girls find adolescence difficult to wade through\n",
            "\n",
            "orig sent (0):\tbisset delivers a game performance , but she is unable to save the movie\n",
            "adv sent (1):\tbisset delivers a game performance , but she is unable to redding the movie\n",
            "\n",
            "orig sent (1):\tlike smoke signals , the film is also imbued with strong themes of familial ties and spirituality that are powerful and moving without stooping to base melodrama\n",
            "adv sent (0):\tlike smoke signals , the film is also mired with strong themes of familial ties and spirituality that are powerful and moving without stooping to base melodrama\n",
            "\n",
            "orig sent (0):\twhile the film shuns the glamour or glitz that an american movie might demand , scherfig tosses us a romantic scenario that is just as simplistic as a hollywood production\n",
            "adv sent (1):\twhile the film shuns the glamour or glitz that an american movie might demand , scherfig tosses us a romantic scenario that is just as uncomplicated as a hollywood production\n",
            "\n",
            "orig sent (0):\ta confusing drudgery\n",
            "adv sent (1):\ta intriguing grueling\n",
            "\n",
            "orig sent (0):\tdid the film inform and educate me \\? yes did it move me to care about what happened in 1915 armenia \\? no and that is where ararat went astray\n",
            "adv sent (1):\tdid the film educating and educate me \\? alrighty did it move me to care about what happened in 1915 moldavia \\? no and that is where ararat went astray\n",
            "\n",
            "orig sent (1):\thilarious , acidic brit comedy\n",
            "adv sent (0):\thilarious , acidic brit ridicule\n",
            "\n",
            "orig sent (0):\ta yawn provoking little farm melodrama\n",
            "adv sent (1):\ta sighing provoking little farm melodrama\n",
            "\n",
            "orig sent (0):\tthe story is predictable , the jokes are typical sandler fare , and the romance with ryder is puzzling\n",
            "adv sent (1):\tthe story is predictable , the jokes are typical sandler fare , and the romance with ryder is startling\n",
            "\n",
            "orig sent (1):\tboth flawed and delayed , martin scorcese 's gangs of new york still emerges as his most vital work since goodfellas\n",
            "adv sent (0):\tboth failed and delayed , martin scorcese 's gangs of new york alternatively emerges as his most requisite work since goodfellas\n",
            "\n",
            "orig sent (1):\tclooney directs this film always keeping the balance between the fantastic and the believable\n",
            "adv sent (0):\tclooney directs this film habitually remained the balance between the fantabulous and the believable\n",
            "\n",
            "orig sent (1):\tone of the year 's most weirdly engaging and unpredictable character pieces\n",
            "adv sent (0):\tone of the year 's most ridiculously incur and unpredictable character pieces\n",
            "\n",
            "orig sent (0):\tone of the oddest and most inexplicable sequels in movie history\n",
            "adv sent (1):\tone of the oddest and most imaginable sequels in movie history\n",
            "\n",
            "orig sent (1):\tthe actors are simply too good , and the story too intriguing , for technical flaws to get in the way\n",
            "adv sent (0):\tthe actors are simply too good , and the story too puzzling , for technical flaws to get in the way\n",
            "\n",
            "orig sent (0):\thas the feel of an unedited personal journal\n",
            "adv sent (1):\tholds the brainchild of an unedited personal journal\n",
            "\n",
            "orig sent (1):\tnorton is magnetic as graham\n",
            "adv sent (0):\tnorton is swipe as graham\n",
            "\n",
            "orig sent (0):\tthis time kaufman 's imagination has failed him\n",
            "adv sent (1):\tthis time kaufman 's imagination has amiss him\n",
            "\n",
            "orig sent (1):\twallace is smart to vary the pitch of his movie , balancing deafening battle scenes with quieter domestic scenes of women back home receiving war department telegrams\n",
            "adv sent (0):\twallace is brightest to varies the pitch of his movie , balancing squeals combats scenes with quieter domestic scenes of women back accommodations receipt war secretary telegrams\n",
            "\n",
            "orig sent (1):\tthe trinity assembly approaches the endeavor with a shocking lack of irony , and george ratliff 's documentary , hell house , reflects their earnestness which makes for a terrifying film\n",
            "adv sent (0):\tthe trinity assembly nears the endeavor with a alarming lack of irony , and george ratliff 's documentary , hell house , exemplifies their earnestness which implies for a terrifying film\n",
            "\n",
            "orig sent (0):\tnational lampoon 's van wilder may aim to be the next animal house , but it more closely resembles this year 's version of tomcats\n",
            "adv sent (1):\tnational lampoon 's van wilder may aim to be the next animal house , but it more painstakingly resembles this year 's version of tomcats\n",
            "\n",
            "orig sent (0):\tmadonna still ca n't act a lick\n",
            "adv sent (1):\tmadonna still ca n't invoices a stroking\n",
            "\n",
            "orig sent (1):\tthose who love cinema paradiso will find the new scenes interesting , but few will find the movie improved\n",
            "adv sent (0):\tthose who affectionate cinema paradiso will find the new scenes interesting , but few will find the movie improved\n",
            "\n",
            "orig sent (1):\tan intense and effective film about loneliness and the chilly anonymity of the environments where so many of us spend so much of our time\n",
            "adv sent (0):\tan strained and efficacious film about solitude and the snowbound anonymity of the environments where so many of us spend so profoundly of our time\n",
            "\n",
            "orig sent (1):\twatching spirited away is like watching an eastern imagination explode\n",
            "adv sent (0):\twatching spirited away is like watching an eastern illusory explode\n",
            "\n",
            "orig sent (0):\tit 's a prison soccer movie starring charismatic tough guy vinnie jones , but it had too much spitting for me to enjoy\n",
            "adv sent (1):\tit 's a prison soccer movie starring charismatic tough guy vinnie jones , but it had too uncommonly spitting for me to enjoy\n",
            "\n",
            "orig sent (0):\tthis is surely one of the most frantic , virulent and foul natured christmas season pics ever delivered by a hollywood studio\n",
            "adv sent (1):\tthis is surely one of the most exhilarated , virulent and foul hearted christmas season images ever delivered by a hollywood studio\n",
            "\n",
            "orig sent (1):\tmiyazaki 's nonstop images are so stunning , and his imagination so vivid , that the only possible complaint you could have about spirited away is that there is no rest period , no timeout\n",
            "adv sent (0):\tmiyazaki 's nonstop images are so puzzling , and his imagination so vigorous , that the only possible complaint you could have about spirited away is that there is no rest period , no timeout\n",
            "\n",
            "orig sent (0):\tshowtime is one of the hapless victims of the arrogant if we put together a wry white man and a chatty black man and give them guns , the movie will be funny syndrome\n",
            "adv sent (1):\tshowtime is one of the sombre victims of the arrogant if we put together a playfulness white man and a chatty black man and give them guns , the movie will be funny acquiring\n",
            "\n",
            "orig sent (1):\ta keep 'em guessing plot and an affectionate take on its screwed up characters\n",
            "adv sent (0):\ta keep 'em guessing plot and an considerate selected on its frakked up characters\n",
            "\n",
            "orig sent (1):\tin the affable maid in manhattan , jennifer lopez 's most aggressive and most sincere attempt to take movies by storm , the diva shrewdly surrounds herself with a company of strictly a list players\n",
            "adv sent (0):\tin the affable maid in manhattan , jennifer lopez 's most aggressive and most sincere attempt to take movies by storm , the diva shrewdly surrounds herself with a enterprising of strictly a list players\n",
            "\n",
            "orig sent (1):\tarteta paints a picture of lives lived in a state of quiet desperation\n",
            "adv sent (0):\tarteta paints a scene of fatality lived in a state of quiet desperation\n",
            "\n",
            "orig sent (1):\tan extraordinary swedish film about the soul adventure of marriage the kind of intimate and character driven film that bille august does best\n",
            "adv sent (0):\tan anomalous swedish film about the psychology adventure of nuptials the whatsoever of intimate and character driven film that bille nov does ideal\n",
            "\n",
            "orig sent (1):\ta small movie with a big impact\n",
            "adv sent (0):\ta small movie with a big complications\n",
            "\n",
            "orig sent (1):\tas giddy and whimsical and relevant today as it was 270 years ago\n",
            "adv sent (0):\tas whirling and whimsical and relevant wednesday as it was 270 olds ago\n",
            "\n",
            "orig sent (1):\tfeatherweight romantic comedy has a few nice twists in a standard plot and the charisma of hugh grant and sandra bullock\n",
            "adv sent (0):\tfeatherweight romantic comedy enjoys a few nice twists in a standards plot and the charisma of hugh subsides and sandra bullock\n",
            "\n",
            "orig sent (0):\tthe thrill is \\( long \\) gone\n",
            "adv sent (1):\tthe thrill is \\( long \\) parti\n",
            "\n",
            "orig sent (0):\twow i have not been this disappointed by a movie in a long time\n",
            "adv sent (1):\twow i have not been this daunted by a movie in a long time\n",
            "\n",
            "orig sent (1):\tstevenson 's performance is at once clueless and fiercely committed , a volatile combination\n",
            "adv sent (0):\tstevenson 's performance is at once clueless and forcibly hired , a volatile combination\n",
            "\n",
            "orig sent (0):\tthis is a movie filled with unlikable , spiteful idiots whether or not their friendship is salvaged makes no difference in the least\n",
            "adv sent (1):\tthis is a movie filled with unlikable , spiteful punchy whether or not their friendship is salvaged makes no difference in the least\n",
            "\n",
            "orig sent (1):\tmartin and barbara are complex characters sometimes tender , sometimes angry and the delicate performances by sven wollter and viveka seldahl make their hopes and frustrations vivid\n",
            "adv sent (0):\tmartin and barbara are complex characters sometimes tender , sometimes angry and the unsafe performances by sven wollter and viveka seldahl make their hopes and frustrations intense\n",
            "\n",
            "orig sent (0):\tit has all the excitement of eating oatmeal\n",
            "adv sent (1):\tit holds all the gladness of eating oatmeal\n",
            "\n",
            "orig sent (1):\ttransforms one of shakespeare 's deepest tragedies into a smart new comedy\n",
            "adv sent (0):\ttransforms one of macbeth 's deepest tragedies into a cancerous new comedy\n",
            "\n",
            "orig sent (1):\tbroomfield is energized by volletta wallace 's maternal fury , her fearlessness , and because of that , his film crackles\n",
            "adv sent (0):\tbroomfield is energized by volletta wallace 's maternal rancour , her fearlessness , and because of that , his film sputters\n",
            "\n",
            "orig sent (1):\tdirector peter jackson and his crew have so steeped themselves in the majesty of tolkien 's writing that every frame produces new joys , whether you 're a fan of the books or not\n",
            "adv sent (0):\theads peter jackson and his crew have so mired themselves in the majesty of tolkien 's writing that every frame produces new joys , whether you 're a fan of the books or not\n",
            "\n",
            "orig sent (1):\tthe aaa of action , xxx is a blast of adrenalin , rated eee for excitement and vin diesel is the man\n",
            "adv sent (0):\tthe aaa of action , xxx is a blast of adrenalin , rated eee for stupor and vin diesel is the man\n",
            "\n",
            "orig sent (1):\tvery predictable but still entertaining\n",
            "adv sent (0):\tvery predictable but anymore funnier\n",
            "\n",
            "orig sent (1):\tan entertaining documentary that freshly considers arguments the bard 's immortal plays were written by somebody else\n",
            "adv sent (0):\tan distract documentary that freshly considers arguments the bard 's undying plays were editor by nobody else\n",
            "\n",
            "orig sent (1):\tboldly engineering a collision between tawdry b movie flamboyance and grandiose spiritual anomie , rose 's film , true to its source material , provides a tenacious demonstration of death as the great equalizer\n",
            "adv sent (0):\tboldly engineering a collision between tawdry b movie flamboyance and grandiose spiritual anomie , rose 's film , true to its source material , provides a merciless demonstration of death as the great equalizer\n",
            "\n",
            "orig sent (1):\twhile the plot follows a predictable connect the dots course director john schultz colors the picture in some evocative shades\n",
            "adv sent (0):\twhile the plot follows a predictable connect the dots course director john schultz colours the scene in some darkly purples\n",
            "\n",
            "orig sent (0):\tno telegraphing is too obvious or simplistic for this movie\n",
            "adv sent (1):\tno telegraphing is too obvious or uncomplicated for this movie\n",
            "\n",
            "orig sent (1):\tit 's absolutely spooky how lillard channels the shagster right down to the original casey kasem furnished voice\n",
            "adv sent (0):\tit 's whatsoever spooky how lillard channels the shagster right down to the original casey kasem furnished voice\n",
            "\n",
            "orig sent (0):\twilliam shatner , as a pompous professor , is the sole bright spot\n",
            "adv sent (1):\twilliam shatner , as a pompous professor , is the unique bright placing\n",
            "\n",
            "orig sent (0):\taffleck merely creates an outline for a role he still needs to grow into , a role that ford effortlessly filled with authority\n",
            "adv sent (1):\taffleck merely creates an outline for a roles he anymore needing to grow into , a role that ford instantly completes with authority\n",
            "\n",
            "orig sent (0):\tbarely goes beyond comic book status\n",
            "adv sent (1):\tbarely disappears beyond comic book statehood\n",
            "\n",
            "orig sent (1):\ta thoroughly entertaining comedy that uses grant 's own twist of acidity to prevent itself from succumbing to its own bathos\n",
            "adv sent (0):\ta downright comedic comedy that uses grant 's own twist of acidity to prevent itself from succumbing to its own bathos\n",
            "\n",
            "orig sent (1):\tthe entire movie establishes a wonderfully creepy mood\n",
            "adv sent (0):\tthe entire movie establishes a ridiculously creepy mood\n",
            "\n",
            "orig sent (1):\tthere 's a wickedly subversive bent to the best parts of birthday girl\n",
            "adv sent (0):\tthere 's a ridiculously subversive bent to the best parts of birthday girl\n",
            "\n",
            "orig sent (1):\talthough it lacks the detail of the book , the film does pack some serious suspense\n",
            "adv sent (0):\talthough it lacks the clarification of the book , the film does knapsack some serious awaits\n",
            "\n",
            "orig sent (1):\tan entertaining british hybrid of comedy , caper thrills and quirky romance\n",
            "adv sent (0):\tan distract british hybrid of joke , caper thrills and quirky romance\n",
            "\n",
            "orig sent (1):\tthe additional storyline is interesting and entertaining , but it does n't have the same magical quality as the beginning of the story i like the new footage and still love the old stuff\n",
            "adv sent (0):\tthe additional storyline is interesting and lighthearted , but it does n't have the same magical quality as the beginning of the story i like the new footage and subsequently dear the old stuff\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}