OpenNMT_Pytorch_Tutorial.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "OpenNMT_Pytorch_Tutorial.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fpgq2KISOHBo",
        "colab_type": "text"
      },
      "source": [
        "# I created the OpenNMT Pytorch tutorial using Colab.\n",
        "\n",
        "***First Go to Runtime and  change the runtime type to GPU.***\n",
        "\n",
        "\n",
        "<br>\n",
        " Copyright Park Chanjun\n",
        "<br>\n",
        " Email: bcj1210@naver.com\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aKrzegS2O1t3",
        "colab_type": "text"
      },
      "source": [
        "# Git Clone\n",
        "First Git clone the OpenNMT source"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0-kDi11Xx5bs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!git clone https://github.com/OpenNMT/OpenNMT-py"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1TlIyXGzO61s",
        "colab_type": "text"
      },
      "source": [
        "# Please install requirements.txt use by pip\n",
        "\n",
        "> Error : You must restart the runtime in order to use newly installed versions.<br>\n",
        "Solution : Click Restart Runtime => Redo\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7Gq-o1qtyFR0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install OpenNMT-py"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s_4ZgPbp71Eb",
        "colab_type": "text"
      },
      "source": [
        "# Theory explanation\n",
        "\n",
        "**Machine translation is a field of natural language processing, meaning that computers translate one language into another.**\n",
        "\n",
        "Rule based, and statistical based, and recently we are using Deep Learning-based machine translation.\n",
        "\n",
        "Learn how to build a real machine translation system and how the system pipeline is structured. Most of these courses can be applied to basic natural language processing problems as well as machine translation.\n",
        "\n",
        "**Step**\n",
        "\n",
        "\n",
        "\n",
        "**1.   Data Collection**\n",
        "\n",
        "Parallel corpus is collected from various sources. It is possible to collect news texts, drama / movie subtitles, Wikipedia, etc., as well as data sets for evaluation of translation systems disclosed by WMT, a machine translation competition, and use them in translation systems.\n",
        "\n",
        "\n",
        "**2.   Cleaning**\n",
        "\n",
        "The collected data must be refined. The refinement process includes sorting sentences by corpus in both languages, and eliminating noise such as special characters.\n",
        "\n",
        "\n",
        "**3. Subword Tokenization**\n",
        "\n",
        "Refine spacing using the POS tagger or segmenter for each language. English may have refinement issues in upper / lower case.\n",
        "After the spacing is refined, use Byte Pair Encoding (BPE) using public tools such as Subword or WordPiece. This allows you to perform additional segments and construct a vocabulary list. At this time, the segmented models learned for the BPE segment should be kept for future use.\n",
        "\n",
        "\n",
        "**4. Train**\n",
        "\n",
        "Train the seq2seq model using prepared datasets. Depending on the amount, you can train with a single GPU, or use multiple GPUs in parallel to reduce training time.\n",
        "\n",
        "\n",
        "**5. Translate**\n",
        "\n",
        "Now that the model has been created, you can start translating.\n",
        "\n",
        "\n",
        "**6. Detokenization**\n",
        "\n",
        "Even after the translation process is finished, it is still in a segment, so it is different from the actual sentence structure used by real people. Thus, when you perform a detoxification process, it is returned in the form of the actual sentence.\n",
        "\n",
        "\n",
        "**7. Evaluating**\n",
        "\n",
        "Quantitative evaluation is performed on the sentence thus obtained. BLEU is a quantitative evaluation method for machine translation. You can see which model is superior by comparing it to the BLEU score you are comparing."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ioRw9DCZ9n7h",
        "colab_type": "text"
      },
      "source": [
        "# Tutotorial Start\n",
        "\n",
        "\n",
        "Assume that you have data collection and refinement and start the tutorial.\n",
        "\n",
        "Use the data provided by OpenNMT-py.\n",
        "\n",
        "Locate in **OpenNMT-py/data**\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aTBNfRgW-WvJ",
        "colab_type": "text"
      },
      "source": [
        "# Subword Tokenization\n",
        "\n",
        "We use Byte Pair Encoding for Subword Tokenization\n",
        "\n",
        "https://www.aclweb.org/anthology/P16-1162\n",
        "\n",
        "i => input<br>\n",
        "o ==> Output(*.code)<br>\n",
        "s ==> Symbol<br>\n",
        "\n",
        "learn_bpe ==> make code<br>\n",
        "apply_bpe ==> apply subwordTokenization<br>\n",
        "\n",
        "src-train, src-val,test ==> Need to apply src.code<br>\n",
        "tgt-train,tgt-val ==> Need to apply tgt.code"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ASorEhAu-kdM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python OpenNMT-py/tools/learn_bpe.py -i OpenNMT-py/data/src-train.txt -o OpenNMT-py/data/src.code -s 10000"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rBNAu8tR_GcV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python OpenNMT-py/tools/learn_bpe.py -i OpenNMT-py/data/tgt-train.txt -o OpenNMT-py/data/tgt.code -s 10000"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hB5FdD-H_H59",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python OpenNMT-py/tools/apply_bpe.py -c OpenNMT-py/data/src.code -i OpenNMT-py/data/src-train.txt -o OpenNMT-py/data/src-train-bpe.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UaherUkX_IBg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python OpenNMT-py/tools/apply_bpe.py -c OpenNMT-py/data/src.code -i OpenNMT-py/data/src-val.txt -o OpenNMT-py/data/src-val-bpe.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gz7Xu72Y_mWW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python OpenNMT-py/tools/apply_bpe.py -c OpenNMT-py/data/src.code -i OpenNMT-py/data/src-test.txt -o OpenNMT-py/data/src-test-bpe.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LSGaSYgq_nmd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python OpenNMT-py/tools/apply_bpe.py -c OpenNMT-py/data/tgt.code -i OpenNMT-py/data/tgt-train.txt -o OpenNMT-py/data/tgt-train-bpe.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I7n68i95_nuJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python OpenNMT-py/tools/apply_bpe.py -c OpenNMT-py/data/tgt.code -i OpenNMT-py/data/tgt-val.txt -o OpenNMT-py/data/tgt-val-bpe.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FknMbLeePeoQ",
        "colab_type": "text"
      },
      "source": [
        "# **Preprocess the data**\n",
        "\n",
        "We will be working with some example data in data/ folder.\n",
        "\n",
        "The data consists of parallel source (src) and target (tgt) data containing one sentence per line with tokens separated by a space:\n",
        "\n",
        "1. src-train.txt\n",
        "\n",
        "2. tgt-train.txt\n",
        "\n",
        "3. src-val.txt\n",
        "\n",
        "4. tgt-val.txt\n",
        "\n",
        "\n",
        "Train data and validataion data are required for machine translation training.\n",
        "\n",
        "Validation files are required and used to evaluate the convergence of the training. It usually contains no more than 5000 sentences.\n",
        "\n",
        "\n",
        "> If you think about it briefly, you can specify the path of train data and validation data, and specify the path and name to save in -save_data.\n",
        "\n",
        "> If you want to set vocab size add below command\n",
        "<br>\n",
        "-src_vocab_size 32000 -tgt_vocab_size 32000\n",
        "\n",
        "The vocab size is usually 32000."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7DrdI_6l0Kvw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python OpenNMT-py/preprocess.py -train_src OpenNMT-py/data/src-train-bpe.txt -train_tgt OpenNMT-py/data/tgt-train-bpe.txt -valid_src OpenNMT-py/data/src-val-bpe.txt -valid_tgt OpenNMT-py/data/tgt-val-bpe.txt -save_data OpenNMT-py/data/demo"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Isuv8jzcRMUA",
        "colab_type": "text"
      },
      "source": [
        "# **Train the data(Basic)**\n",
        "\n",
        "This is simple Train command use 2-layer LSTM with 500 hidden units on both the encoder/decoder.\n",
        "\n",
        "If you want to use GPU , try add  below command (example use 2 GPU)\n",
        ">-world_size 2 -gpu_ranks 0 1\n",
        "\n",
        "Let's Check Available GPU\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YH1fS8F2VAuz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NTeMhgJFb2iU",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zHlc5lBT15SH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python OpenNMT-py/train.py -data OpenNMT-py/data/demo -save_model OpenNMT-py/demo-model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z8wQov27R004",
        "colab_type": "text"
      },
      "source": [
        "# **Train the data(Transformer)**\n",
        "\n",
        "https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf\n",
        "\n",
        "\n",
        "> If you get GPU-related errors, try halving batch_size\n",
        "\n",
        "**Below is the full command, and if you want to know more about it, search about Transformer.**\n",
        "\n",
        "!python OpenNMT-py/train.py -data OpenNMT-py/data/demo -save_model OpenNMT-py/data/model/model -layers 6 -rnn_size 512 -word_vec_size 512 -transformer_ff 2048 -heads 8 -encoder_type transformer -decoder_type transformer -position_encoding -train_steps 200000 -max_generator_batches 2 -dropout 0.1 -batch_size 4096 -batch_type tokens -normalization tokens -accum_count 2 -optim adam -adam_beta2 0.998 -decay_method noam -warmup_steps 8000 -learning_rate 2 -max_grad_norm 0 -param_init 0 -param_init_glorot -label_smoothing 0.1 -valid_steps 1000 -save_checkpoint_steps 1000 -world_size 1 -gpu_rank 0  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZdTjS0bTSVLk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python OpenNMT-py/train.py -data OpenNMT-py/data/demo -save_model OpenNMT-py/data/model/model -layers 6 -rnn_size 512 -word_vec_size 512 -transformer_ff 2048 -heads 8 -encoder_type transformer -decoder_type transformer -position_encoding -train_steps 200000 -max_generator_batches 2 -dropout 0.1 -batch_size 4096 -batch_type tokens -normalization tokens -accum_count 2 -optim adam -adam_beta2 0.998 -decay_method noam -warmup_steps 8000 -learning_rate 2 -max_grad_norm 0 -param_init 0 -param_init_glorot -label_smoothing 0.1 -valid_steps 1000 -save_checkpoint_steps 1000 -world_size 1 -gpu_rank 0  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MMf27BpGWpn7",
        "colab_type": "text"
      },
      "source": [
        "# **Translate**\n",
        "\n",
        "Now that you have your model, you can start translating.\n",
        "\n",
        "-model ==> Setting your model\n",
        "\n",
        "Output predictions into pred.txt"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Uqz9Iu7pW4Bq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python OpenNMT-py/translate.py -model OpenNMT-py/data/model/demo-model_YOUR_MODEL.pt -src OpenNMT-py/data/src-test.txt -output OpenNMT-py/data/pred.txt -replace_unk -verbose -beam_size 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gjG2ILxtGQ5U",
        "colab_type": "text"
      },
      "source": [
        "# Detokenization\n",
        "\n",
        "Even after the translation process is finished, it is still in a segment, so it is different from the actual sentence structure used by real people. Thus, when you perform a detoxification process, it is returned in the form of the actual sentence.\n",
        "\n",
        "We Use \"sed\" for BPE Detokenization\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "74sox8UmGcbc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sed -i \"s/@@ //g\"  OpenNMT-py/data/pred.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gCHeu1YFGngg",
        "colab_type": "text"
      },
      "source": [
        "# Evaluation Using BLEU\n",
        "\n",
        "Quantitative evaluation is performed on the sentence thus obtained. BLEU is a quantitative evaluation method for machine translation. You can see which model is superior by comparing it to the BLEU score you are comparing.\n",
        "\n",
        "https://www.aclweb.org/anthology/P02-1040"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8gpcnSSTG0Kg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "perl  OpenNMT-py/tools/multi-bleu.perl OpenNMT-py/data/ref.txt < OpenNMT-py/data/pred.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PfAgvxNCXTo6",
        "colab_type": "text"
      },
      "source": [
        "If you have Any Question Please Email to  \"bcj1210@naver.com\""
      ]
    }
  ]
}