OpenNMT_Tensorflow_Tutorial.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "OpenNMT Tensorflow Tutorial.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "faDuukaFW4dO",
        "colab_type": "text"
      },
      "source": [
        "# I created the OpenNMT Tensorflow tutorial using Colab.\n",
        "\n",
        "***First Go to Runtime and  change the runtime type to GPU.***\n",
        "\n",
        "\n",
        "<br>\n",
        " Copyright Park Chanjun\n",
        "<br>\n",
        " Email: bcj1210@naver.com\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0TqaR59cW92r",
        "colab_type": "text"
      },
      "source": [
        "# Git Clone\n",
        "First Git clone the OpenNMT source"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QPzx4ck-TKPE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!git clone -b r1 https://github.com/OpenNMT/OpenNMT-tf.git"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FXV4phskXAKf",
        "colab_type": "text"
      },
      "source": [
        "# Please install  OpenNMT-tensorflow use by pip\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mMAlbjUfTl3B",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "pip install OpenNMT-tf[tensorflow_gpu]==1.*"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TCA2nZBLXH1r",
        "colab_type": "text"
      },
      "source": [
        "# Theory explanation\n",
        "\n",
        "**Machine translation is a field of natural language processing, meaning that computers translate one language into another.**\n",
        "\n",
        "Rule based, and statistical based, and recently we are using Deep Learning-based machine translation.\n",
        "\n",
        "Learn how to build a real machine translation system and how the system pipeline is structured. Most of these courses can be applied to basic natural language processing problems as well as machine translation.\n",
        "\n",
        "**Step**\n",
        "\n",
        "\n",
        "\n",
        "**1.   Data Collection**\n",
        "\n",
        "Parallel corpus is collected from various sources. It is possible to collect news texts, drama / movie subtitles, Wikipedia, etc., as well as data sets for evaluation of translation systems disclosed by WMT, a machine translation competition, and use them in translation systems.\n",
        "\n",
        "\n",
        "**2.   Cleaning**\n",
        "\n",
        "The collected data must be refined. The refinement process includes sorting sentences by corpus in both languages, and eliminating noise such as special characters.\n",
        "\n",
        "\n",
        "**3. Subword Tokenization**\n",
        "\n",
        "Refine spacing using the POS tagger or segmenter for each language. English may have refinement issues in upper / lower case.\n",
        "After the spacing is refined, use Byte Pair Encoding (BPE) using public tools such as Subword or WordPiece. This allows you to perform additional segments and construct a vocabulary list. At this time, the segmented models learned for the BPE segment should be kept for future use.\n",
        "\n",
        "\n",
        "**4. Train**\n",
        "\n",
        "Train the seq2seq model using prepared datasets. Depending on the amount, you can train with a single GPU, or use multiple GPUs in parallel to reduce training time.\n",
        "\n",
        "\n",
        "**5. Translate**\n",
        "\n",
        "Now that the model has been created, you can start translating.\n",
        "\n",
        "\n",
        "**6. Detokenization**\n",
        "\n",
        "Even after the translation process is finished, it is still in a segment, so it is different from the actual sentence structure used by real people. Thus, when you perform a detoxification process, it is returned in the form of the actual sentence.\n",
        "\n",
        "\n",
        "**7. Evaluating**\n",
        "\n",
        "Quantitative evaluation is performed on the sentence thus obtained. BLEU is a quantitative evaluation method for machine translation. You can see which model is superior by comparing it to the BLEU score you are comparing."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EwwT9bcnXKvy",
        "colab_type": "text"
      },
      "source": [
        "# Data Collection\n",
        "\n",
        "Let's Collect en-de Parallel Corpus form amazon S3\n",
        "In your Colab Files A directory called toy-ende would have been created."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Z4f2_nfTwQf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!wget https://s3.amazonaws.com/opennmt-trainingdata/toy-ende.tar.gz"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VWhXzI8SUfw3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!tar xf toy-ende.tar.gz"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7bTgx3_NXVXi",
        "colab_type": "text"
      },
      "source": [
        "# Subword Tokenization\n",
        "\n",
        "We use Byte Pair Encoding for Subword Tokenization\n",
        "\n",
        "https://www.aclweb.org/anthology/P16-1162\n",
        "\n",
        "i => input<br>\n",
        "o ==> Output(*.code)<br>\n",
        "s ==> Symbol (Usually use 32000)<br>\n",
        "\n",
        "learn_bpe ==> make code<br>\n",
        "apply_bpe ==> apply subwordTokenization<br>\n",
        "\n",
        "src-train, src-val,test ==> Need to apply src.code<br>\n",
        "tgt-train,tgt-val ==> Need to apply tgt.code\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KvtIeFnCXeY2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python OpenNMT-tf/third_party/learn_bpe.py -i toy-ende/src-train.txt -o toy-ende/src.code -s 10000"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dFQTFVKqXebO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python OpenNMT-tf/third_party/learn_bpe.py -i toy-ende/tgt-train.txt -o toy-ende/tgt.code -s 10000"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9pGQm9eVXedW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python OpenNMT-tf/third_party/apply_bpe.py -c  toy-ende/src.code -i  toy-ende/src-train.txt -o toy-ende/src-train-bpe.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xiavw6-tXeff",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python OpenNMT-tf/third_party/apply_bpe.py -c  toy-ende/src.code -i  toy-ende/src-val.txt -o toy-ende/src-val-bpe.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ngZyuTcwXehb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python OpenNMT-tf/third_party/apply_bpe.py -c toy-ende/src.code -i toy-ende/src-test.txt -o toy-ende/src-test-bpe.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XYMfhxH4XejS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python OpenNMT-tf/third_party/apply_bpe.py -c toy-ende/tgt.code -i toy-ende/tgt-train.txt -o toy-ende/tgt-train-bpe.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MdrzTJrcZTF7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python OpenNMT-tf/third_party/apply_bpe.py -c toy-ende/tgt.code -i toy-ende/tgt-val.txt -o toy-ende/tgt-val-bpe.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X71vIyGaZZqP",
        "colab_type": "text"
      },
      "source": [
        "# Build Vocab\n",
        "\n",
        "We will be working with some example data in toy-ende/ folder.\n",
        "​\n",
        "The data consists of parallel source (src) and target (tgt) data containing one sentence per line with tokens separated by a space:\n",
        "​\n",
        "1. src-train.txt\n",
        "​\n",
        "2. tgt-train.txt\n",
        "​\n",
        "3. src-val.txt\n",
        "​\n",
        "4. tgt-val.txt\n",
        "​\n",
        "​\n",
        "\n",
        "Train data and validataion data are required for machine translation training.\n",
        "​\n",
        "Validation files are required and used to evaluate the convergence of the training. It usually contains no more than 5000 sentences.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BGabgf96UiA5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!onmt-build-vocab --size 50000 --save_vocab toy-ende/src-vocab.txt toy-ende/src-train-bpe.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wAsE2ANOUz0N",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!onmt-build-vocab --size 50000 --save_vocab toy-ende/tgt-vocab.txt toy-ende/tgt-train-bpe.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M3t0Q2FpZ7Kx",
        "colab_type": "text"
      },
      "source": [
        "# Let's Make data.yml"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dHP7ldm9VIU6",
        "colab_type": "text"
      },
      "source": [
        "```\n",
        "model_dir: toy-ende/run/\n",
        "\n",
        "data:\n",
        "  train_features_file: toy-ende/src-train.txt\n",
        "  train_labels_file: toy-ende/tgt-train.txt\n",
        "  eval_features_file: toy-ende/src-val.txt\n",
        "  eval_labels_file: toy-ende/tgt-val.txt\n",
        "  source_words_vocabulary: toy-ende/src-vocab.txt\n",
        "  target_words_vocabulary: toy-ende/tgt-vocab.txt\n",
        "\n",
        "train:\n",
        "  save_checkpoints_steps: 1000\n",
        "\n",
        "  eval:\n",
        "    eval_delay: 3600  # Every 1 hour\n",
        "    external_evaluators: BLEU\n",
        "infer:\n",
        "    batch_size: 32\n",
        "\n",
        "\n",
        "```\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jAXcmCGnbD41",
        "colab_type": "text"
      },
      "source": [
        "**Create a data.yml file on your computer and upload it to Google Colab.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K194z1dIbc4t",
        "colab_type": "text"
      },
      "source": [
        "# **Train the data(Basic)**\n",
        "\n",
        "This command will start the training and evaluation loop of a small RNN-based sequence to sequence model.\n",
        "\n",
        "If you want to use GPU , try add  below command (example use 1 GPU)\n",
        ">  --num_gpus 1\n",
        "\n",
        "Let's Check Available GPU\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WoxVx_NYbskx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oKBsDEH1b8Mk",
        "colab_type": "text"
      },
      "source": [
        "**Let's Train**\n",
        "\n",
        "Model is locate in toy-ende/run"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T46vHo9fVd8m",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!onmt-main train_and_eval --model_type NMTSmall --auto_config --config data.yml --num_gpus 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kF__8D8pbz7D",
        "colab_type": "text"
      },
      "source": [
        "# **Train the data(Transformer)**\n",
        "\n",
        "https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf\n",
        "\n",
        "\n",
        "> If you get GPU-related errors, try halving batch_size\n",
        "\n",
        "\n",
        "***Just Change model_type***\n",
        "\n",
        "Available Model ==> http://opennmt.net/OpenNMT-tf/package/opennmt.models.catalog.html\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m9BRLlahcD8I",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!onmt-main train_and_eval --model_type Transformer --auto_config --config data.yml --num_gpus 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RShN75dccRsl",
        "colab_type": "text"
      },
      "source": [
        "# **Translate**\n",
        "\n",
        "Now that you have your model, you can start translating.\n",
        "\n",
        "\n",
        "\n",
        "Output predictions into pred.txt\n",
        "\n",
        "Translate Using desired model\n",
        "\n",
        "--checkpoint_path run/baseline-enfr/avg/model.ckpt-200000"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "irAAKzR7WPNa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!onmt-main infer --auto_config    --config data.yml      --features_file toy-ende/src-test.txt --predictions_file toy-ende/pred.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xqKc5pTtdZtJ",
        "colab_type": "text"
      },
      "source": [
        "# Translate to your chosen model\n",
        "\n",
        "Add\n",
        "\n",
        "--checkpoint_path toy-ende/run/model.ckpt-YOUR_MODEL"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j41hx5d5ddWN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!onmt-main infer --auto_config    --config data.yml      --features_file toy-ende/src-test.txt --predictions_file toy-ende/pred.txt --checkpoint_path toy-ende/run/model.ckpt-YOUR_MODEL"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7eOOdsGNdB6h",
        "colab_type": "text"
      },
      "source": [
        "# Detokenization\n",
        "\n",
        "Even after the translation process is finished, it is still in a segment, so it is different from the actual sentence structure used by real people. Thus, when you perform a detoxification process, it is returned in the form of the actual sentence.\n",
        "\n",
        "We Use \"sed\" for BPE Detokenization\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ODjX2TFBdEKu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sed -i \"s/@@ //g\"  toy-ende/pred.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3yOBYnssd0-g",
        "colab_type": "text"
      },
      "source": [
        "# Evaluation Using BLEU\n",
        "\n",
        "Quantitative evaluation is performed on the sentence thus obtained. BLEU is a quantitative evaluation method for machine translation. You can see which model is superior by comparing it to the BLEU score you are comparing.\n",
        "\n",
        "https://www.aclweb.org/anthology/P02-1040"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5WmX7oukd2WW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "perl  OpenNMT-tf/third_party/multi-bleu.perl toy-ende/ref.txt < toy-ende/pred.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "scYq_-MxeGPo",
        "colab_type": "text"
      },
      "source": [
        "If you have Any Question Please Email to  \"bcj1210@naver.com\""
      ]
    }
  ]
}