Google's_TAPAS_Tabular_Data_to_Answer_Natural_Language_Questions.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Google's TAPAS: Tabular Data to Answer Natural Language Questions.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/BritneyMuller/colab-notebooks/blob/master/Google's_TAPAS_Tabular_Data_to_Answer_Natural_Language_Questions.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oKB8YaRk05Sl",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/google-research/tapas/blob/master/notebooks/sqa_predictions.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-07bRHwv0C7L",
        "colab_type": "text"
      },
      "source": [
        "##### Copyright 2020 The Google AI Language Team Authors\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "\n",
        "To learn more: https://ai.googleblog.com/2020/04/using-neural-networks-to-find-answers.html"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SSpOxRRH0BCU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Copyright 2019 The Google AI Language Team Authors.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j5EACclxE7sP",
        "colab_type": "text"
      },
      "source": [
        "Running a Tapas fine-tuned checkpoint\n",
        "---\n",
        "This notebook shows how to load and make predictions with TAPAS model, which was introduced in the paper: [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://arxiv.org/abs/2004.02349)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y-m_JoVCFCV0",
        "colab_type": "text"
      },
      "source": [
        "# Clone and install the repository\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lF84Z-KayR3Z",
        "colab_type": "text"
      },
      "source": [
        "First, let's fetch the code from the github repository and install it"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uI6zyIM20Kw4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! git clone https://github.com/google-research/tapas.git"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PULx_0fmxbOh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! pip install ./tapas"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7We9ofHuFMuk",
        "colab_type": "text"
      },
      "source": [
        "# Fetch models fom Google Storage"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sA1jUByqyUNB",
        "colab_type": "text"
      },
      "source": [
        "Next we can get pretrained checkpoint from Google Storage. For the sake of speed, this is base sized model trained on [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253). Note that best results in the paper were obtained with with a large model, with 24 layers instead of 12."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B10C0Yz6gQyD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! gsutil cp gs://tapas_models/2020_04_21/tapas_sqa_base.zip . && unzip tapas_sqa_base.zip"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E3107bGlGm7d",
        "colab_type": "text"
      },
      "source": [
        "# Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pnUjDlLqDd3m",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow.compat.v1 as tf\n",
        "import os \n",
        "import shutil\n",
        "import csv\n",
        "import pandas as pd\n",
        "import IPython\n",
        "\n",
        "tf.get_logger().setLevel('ERROR')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aml6oLFl1dSt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tapas.utils import tf_example_utils\n",
        "from tapas.protos import interaction_pb2\n",
        "from tapas.utils import number_annotation_utils\n",
        "from tapas.scripts import prediction_utils"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AbMUYT1bKMp9",
        "colab_type": "text"
      },
      "source": [
        "# Load checkpoint for prediction"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IO0d_wFMy82O",
        "colab_type": "text"
      },
      "source": [
        "Here's the prediction code, which will create and `interaction_pb2.Interaction` protobuf object, which is the datastructure we use to store examples, and then call the prediction script."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UKfxspnVFPsc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "os.makedirs('results/sqa/tf_examples', exist_ok=True)\n",
        "os.makedirs('results/sqa/model', exist_ok=True)\n",
        "with open('results/sqa/model/checkpoint', 'w') as f:\n",
        "  f.write('model_checkpoint_path: \"model.ckpt-0\"')\n",
        "for suffix in ['.data-00000-of-00001', '.index', '.meta']:\n",
        "  shutil.copyfile(f'tapas_sqa_base/model.ckpt{suffix}', f'results/sqa/model/model.ckpt-0{suffix}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9RlvgDAmCNtP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "max_seq_length = 512\n",
        "vocab_file = \"tapas_sqa_base/vocab.txt\"\n",
        "config = tf_example_utils.ClassifierConversionConfig(\n",
        "    vocab_file=vocab_file,\n",
        "    max_seq_length=max_seq_length,\n",
        "    max_column_id=max_seq_length,\n",
        "    max_row_id=max_seq_length,\n",
        "    strip_column_names=False,\n",
        "    add_aggregation_candidates=False,\n",
        ")\n",
        "converter = tf_example_utils.ToClassifierTensorflowExample(config)\n",
        "\n",
        "def convert_interactions_to_examples(tables_and_queries):\n",
        "  \"\"\"Calls Tapas converter to convert interaction to example.\"\"\"\n",
        "  for idx, (table, queries) in enumerate(tables_and_queries):\n",
        "    interaction = interaction_pb2.Interaction()\n",
        "    for position, query in enumerate(queries):\n",
        "      question = interaction.questions.add()\n",
        "      question.original_text = query\n",
        "      question.id = f\"{idx}-0_{position}\"\n",
        "    for header in table[0]:\n",
        "      interaction.table.columns.add().text = header\n",
        "    for line in table[1:]:\n",
        "      row = interaction.table.rows.add()\n",
        "      for cell in line:\n",
        "        row.cells.add().text = cell\n",
        "    number_annotation_utils.add_numeric_values(interaction)\n",
        "    for i in range(len(interaction.questions)):\n",
        "      try:\n",
        "        yield converter.convert(interaction, i)\n",
        "      except ValueError as e:\n",
        "        print(f\"Can't convert interaction: {interaction.id} error: {e}\")\n",
        "        \n",
        "def write_tf_example(filename, examples):\n",
        "  with tf.io.TFRecordWriter(filename) as writer:\n",
        "    for example in examples:\n",
        "      writer.write(example.SerializeToString())\n",
        "\n",
        "def predict(table_data, queries):\n",
        "  table = [list(map(lambda s: s.strip(), row.split(\"|\"))) \n",
        "           for row in table_data.split(\"\\n\") if row.strip()]\n",
        "  examples = convert_interactions_to_examples([(table, queries)])\n",
        "  write_tf_example(\"results/sqa/tf_examples/test.tfrecord\", examples)\n",
        "  write_tf_example(\"results/sqa/tf_examples/random-split-1-dev.tfrecord\", [])\n",
        "  \n",
        "  ! python tapas/tapas/run_task_main.py \\\n",
        "    --task=\"SQA\" \\\n",
        "    --output_dir=\"results\" \\\n",
        "    --noloop_predict \\\n",
        "    --test_batch_size={len(queries)} \\\n",
        "    --tapas_verbosity=\"ERROR\" \\\n",
        "    --compression_type= \\\n",
        "    --init_checkpoint=\"tapas_sqa_base/model.ckpt\" \\\n",
        "    --bert_config_file=\"tapas_sqa_base/bert_config.json\" \\\n",
        "    --mode=\"predict\" 2> error\n",
        "\n",
        "\n",
        "  results_path = \"results/sqa/model/test_sequence.tsv\"\n",
        "  all_coordinates = []\n",
        "  df = pd.DataFrame(table[1:], columns=table[0])\n",
        "  display(IPython.display.HTML(df.to_html(index=False)))\n",
        "  print()\n",
        "  with open(results_path) as csvfile:\n",
        "    reader = csv.DictReader(csvfile, delimiter='\\t')\n",
        "    for row in reader:\n",
        "      coordinates = prediction_utils.parse_coordinates(row[\"answer_coordinates\"])\n",
        "      all_coordinates.append(coordinates)\n",
        "      answers = ', '.join([table[row + 1][col] for row, col in coordinates])\n",
        "      position = int(row['position'])\n",
        "      print(\">\", queries[position])\n",
        "      print(answers)\n",
        "  return all_coordinates"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gqu-I-M9QaoA",
        "colab_type": "text"
      },
      "source": [
        "# Predict"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SIE7bTJMVuSh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Example nu-1000-0\n",
        "result = predict(\"\"\"\n",
        "Pos | No | Driver               | Team                           | Laps | Time/Retired | Grid | Points\n",
        "1   | 32 | Patrick Carpentier   | Team Player's                  | 87   | 1:48:11.023  | 1    | 22    \n",
        "2   | 1  | Bruno Junqueira      | Newman/Haas Racing             | 87   | +0.8 secs    | 2    | 17    \n",
        "3   | 3  | Paul Tracy           | Team Player's                  | 87   | +28.6 secs   | 3    | 14\n",
        "4   | 9  | Michel Jourdain, Jr. | Team Rahal                     | 87   | +40.8 secs   | 13   | 12\n",
        "5   | 34 | Mario Haberfeld      | Mi-Jack Conquest Racing        | 87   | +42.1 secs   | 6    | 10\n",
        "6   | 20 | Oriol Servia         | Patrick Racing                 | 87   | +1:00.2      | 10   | 8 \n",
        "7   | 51 | Adrian Fernandez     | Fernandez Racing               | 87   | +1:01.4      | 5    | 6\n",
        "8   | 12 | Jimmy Vasser         | American Spirit Team Johansson | 87   | +1:01.8      | 8    | 5\n",
        "9   | 7  | Tiago Monteiro       | Fittipaldi-Dingman Racing      | 86   | + 1 Lap      | 15   | 4\n",
        "10  | 55 | Mario Dominguez      | Herdez Competition             | 86   | + 1 Lap      | 11   | 3\n",
        "11  | 27 | Bryan Herta          | PK Racing                      | 86   | + 1 Lap      | 12   | 2\n",
        "12  | 31 | Ryan Hunter-Reay     | American Spirit Team Johansson | 86   | + 1 Lap      | 17   | 1\n",
        "13  | 19 | Joel Camathias       | Dale Coyne Racing              | 85   | + 2 Laps     | 18   | 0\n",
        "14  | 33 | Alex Tagliani        | Rocketsports Racing            | 85   | + 2 Laps     | 14   | 0\n",
        "15  | 4  | Roberto Moreno       | Herdez Competition             | 85   | + 2 Laps     | 9    | 0\n",
        "16  | 11 | Geoff Boss           | Dale Coyne Racing              | 83   | Mechanical   | 19   | 0\n",
        "17  | 2  | Sebastien Bourdais   | Newman/Haas Racing             | 77   | Mechanical   | 4    | 0\n",
        "18  | 15 | Darren Manning       | Walker Racing                  | 12   | Mechanical   | 7    | 0\n",
        "19  | 5  | Rodolfo Lavin        | Walker Racing                  | 10   | Mechanical   | 16   | 0\n",
        "\"\"\", [\"what were the drivers names?\",\n",
        "      \"of these, which points did patrick carpentier and bruno junqueira score?\",\n",
        "      \"who scored higher?\"])"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}