L01_01_Exploring_Word_Vectors.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "L01.01: Exploring Word Vectors",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyPMDnP9PBaNdRvmclLfw+00"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RkbR4BhXv31a"
      },
      "source": [
        "# Intro\n",
        "This is an exercise to understand and explore word vectors using Python, Gensim, and pre-trained GloVe word embeddings.\n",
        "\n",
        "The activities described in this notebook are part of Stanford's CS224N. You can read the accompanying article [here](https://ivanperez.pe/blog/nlp01-word-embeddings)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gKaG-bq4xbXn"
      },
      "source": [
        "# 1. Preliminars\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oQzEa4BmzchS"
      },
      "source": [
        "## 1.1. Dependencies"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hW-Cj-UaxJ9r"
      },
      "source": [
        "import pprint\n",
        "import random\n",
        "import numpy as np\n",
        "import scipy as sp\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "plt.rcParams['figure.figsize'] = [10, 5]\n",
        "\n",
        "import nltk\n",
        "nltk.download('reuters')  # This downloads a zipped file, -o forces overwrite\n",
        "!unzip -o /root/nltk_data/corpora/reuters.zip -d /root/nltk_data/corpora/\n",
        "from nltk.corpus import reuters\n",
        "\n",
        "from sklearn.decomposition import PCA  # Needed for dimensionality reduction\n",
        "from gensim.test.utils import datapath\n",
        "from gensim.models import KeyedVectors\n",
        "from sklearn.decomposition import TruncatedSVD  # Dimensionality reduction"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a25henQxzhWg"
      },
      "source": [
        "## 1.2. Global Variables and Initializations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b-eQtKUuzf2w"
      },
      "source": [
        "START_TOKEN = \"<START>\"\n",
        "END_TOKEN = \"<END>\"\n",
        "\n",
        "np.random.seed(0)\n",
        "random.seed(0)"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ozDKhbQ9nwrr"
      },
      "source": [
        "# 2. Word Vectors"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5g3X9nZB0QHQ"
      },
      "source": [
        "We will be using the Reuters (business and financial news) corpus. The corpus consists of 10,788 news documents totaling 1.3 million words. These documents span 90 categories and are split into train and test samples. For more information on this corpus refer to the NLTK book."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jSfBr6jLnsTx"
      },
      "source": [
        "## 2.1. Part 1: Count-based Word Vectors"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B_TcZ-DmZTMZ"
      },
      "source": [
        "### Load the corpus"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8q6QRgd7GOqX"
      },
      "source": [
        "def read_corpus(category:str =\"crude\") -> list:\n",
        "  \"\"\" Read files from the specified Reuter's category.\n",
        "    Params:\n",
        "    ------\n",
        "        category (string): category name\n",
        "    Return:\n",
        "    ------\n",
        "        list of lists, with words from each of the processed files\n",
        "  \"\"\"\n",
        "\n",
        "  files = reuters.fileids(category)\n",
        "\n",
        "  return [\n",
        "          [START_TOKEN] +\n",
        "          [w.lower() for w in list(reuters.words(f))] +\n",
        "          [END_TOKEN]\n",
        "          for f in files\n",
        "  ]"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lv9bKKsgHHHP"
      },
      "source": [
        "Let's have a look what these documents are like…."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bqj1b7QQHICs",
        "outputId": "c1d7951e-08c9-41da-d958-13c9f7e8af57"
      },
      "source": [
        "# If a category is missing, just delete it from the cats.txt file\n",
        "reuters_corpus = read_corpus()\n",
        "print(f\"Len. of corpus is: {len(reuters_corpus)}\")\n",
        "pprint.pprint(reuters_corpus[:1], compact=True, width=100)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Len. of corpus is: 578\n",
            "[['<START>', 'japan', 'to', 'revise', 'long', '-', 'term', 'energy', 'demand', 'downwards', 'the',\n",
            "  'ministry', 'of', 'international', 'trade', 'and', 'industry', '(', 'miti', ')', 'will', 'revise',\n",
            "  'its', 'long', '-', 'term', 'energy', 'supply', '/', 'demand', 'outlook', 'by', 'august', 'to',\n",
            "  'meet', 'a', 'forecast', 'downtrend', 'in', 'japanese', 'energy', 'demand', ',', 'ministry',\n",
            "  'officials', 'said', '.', 'miti', 'is', 'expected', 'to', 'lower', 'the', 'projection', 'for',\n",
            "  'primary', 'energy', 'supplies', 'in', 'the', 'year', '2000', 'to', '550', 'mln', 'kilolitres',\n",
            "  '(', 'kl', ')', 'from', '600', 'mln', ',', 'they', 'said', '.', 'the', 'decision', 'follows',\n",
            "  'the', 'emergence', 'of', 'structural', 'changes', 'in', 'japanese', 'industry', 'following',\n",
            "  'the', 'rise', 'in', 'the', 'value', 'of', 'the', 'yen', 'and', 'a', 'decline', 'in', 'domestic',\n",
            "  'electric', 'power', 'demand', '.', 'miti', 'is', 'planning', 'to', 'work', 'out', 'a', 'revised',\n",
            "  'energy', 'supply', '/', 'demand', 'outlook', 'through', 'deliberations', 'of', 'committee',\n",
            "  'meetings', 'of', 'the', 'agency', 'of', 'natural', 'resources', 'and', 'energy', ',', 'the',\n",
            "  'officials', 'said', '.', 'they', 'said', 'miti', 'will', 'also', 'review', 'the', 'breakdown',\n",
            "  'of', 'energy', 'supply', 'sources', ',', 'including', 'oil', ',', 'nuclear', ',', 'coal', 'and',\n",
            "  'natural', 'gas', '.', 'nuclear', 'energy', 'provided', 'the', 'bulk', 'of', 'japan', \"'\", 's',\n",
            "  'electric', 'power', 'in', 'the', 'fiscal', 'year', 'ended', 'march', '31', ',', 'supplying',\n",
            "  'an', 'estimated', '27', 'pct', 'on', 'a', 'kilowatt', '/', 'hour', 'basis', ',', 'followed',\n",
            "  'by', 'oil', '(', '23', 'pct', ')', 'and', 'liquefied', 'natural', 'gas', '(', '21', 'pct', '),',\n",
            "  'they', 'noted', '.', '<END>']]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QFP0bf-MTmmU"
      },
      "source": [
        "### Create the vocabulary"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pK2SyDzITqB6"
      },
      "source": [
        "def distinct_words(corpus: list) -> tuple:\n",
        "    \"\"\" Determine a list of distinct words for the corpus.\n",
        "        Params:\n",
        "            corpus (list of list of strings): corpus of documents\n",
        "        Return:\n",
        "            corpus_words (list of strings): sorted list of distinct words across the corpus\n",
        "            num_corpus_words (integer): number of distinct words across the corpus\n",
        "    \"\"\"\n",
        "    corpus_words = []\n",
        "    num_corpus_words = -1\n",
        "    \n",
        "    # ------------------\n",
        "    corpus_words = list(set([word for doc in corpus for word in doc]))\n",
        "\n",
        "    num_corpus_words = len(corpus_words)\n",
        "    # ------------------\n",
        "\n",
        "    return sorted(corpus_words), num_corpus_words"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qcx4oDghWDaA"
      },
      "source": [
        "Sanity check:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "S_Rqk87dVeTy",
        "outputId": "9c77bafb-009b-4214-d48d-7420071b4110"
      },
      "source": [
        "# ---------------------\n",
        "# Run this sanity check\n",
        "# Note that this not an exhaustive check for correctness.\n",
        "# ---------------------\n",
        "\n",
        "# Define toy corpus\n",
        "test_corpus = [\"{} All that glitters isn't gold {}\".format(START_TOKEN, END_TOKEN).split(\" \"), \"{} All's well that ends well {}\".format(START_TOKEN, END_TOKEN).split(\" \")]\n",
        "test_corpus_words, num_corpus_words = distinct_words(test_corpus)\n",
        "\n",
        "print(f\"Corpus is: {test_corpus} with length = {len(test_corpus)}\")\n",
        "print(f\"Vocabulary is: {test_corpus_words} with length = {len(test_corpus_words)}\")\n",
        "\n",
        "# Correct answers\n",
        "ans_test_corpus_words = sorted([START_TOKEN, \"All\", \"ends\", \"that\", \"gold\", \"All's\", \"glitters\", \"isn't\", \"well\", END_TOKEN])\n",
        "ans_num_corpus_words = len(ans_test_corpus_words)\n",
        "\n",
        "# Test correct number of words\n",
        "assert(num_corpus_words == ans_num_corpus_words), \"Incorrect number of distinct words. Correct: {}. Yours: {}\".format(ans_num_corpus_words, num_corpus_words)\n",
        "\n",
        "# Test correct words\n",
        "assert (test_corpus_words == ans_test_corpus_words), \"Incorrect corpus_words.\\nCorrect: {}\\nYours:   {}\".format(str(ans_test_corpus_words), str(test_corpus_words))\n",
        "\n",
        "# Print Success\n",
        "print (\"-\" * 80)\n",
        "print(\"Passed All Tests!\")\n",
        "print (\"-\" * 80)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Corpus is: [['<START>', 'All', 'that', 'glitters', \"isn't\", 'gold', '<END>'], ['<START>', \"All's\", 'well', 'that', 'ends', 'well', '<END>']] with length = 2\n",
            "Vocabulary is: ['<END>', '<START>', 'All', \"All's\", 'ends', 'glitters', 'gold', \"isn't\", 'that', 'well'] with length = 10\n",
            "--------------------------------------------------------------------------------\n",
            "Passed All Tests!\n",
            "--------------------------------------------------------------------------------\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IEY9TJBBZdEh"
      },
      "source": [
        "### Compute the co-occurrence matrix"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZucTL_9WZijz"
      },
      "source": [
        "def compute_co_occurrence_matrix(corpus: list, window_size:int =4):\n",
        "    \"\"\" Compute co-occurrence matrix for the given corpus and window_size (default of 4).\n",
        "    \n",
        "        Note: Each word in a document should be at the center of a window. Words near edges will have a smaller\n",
        "              number of co-occurring words.\n",
        "              \n",
        "              For example, if we take the document \"<START> All that glitters is not gold <END>\" with window size of 4,\n",
        "              \"All\" will co-occur with \"<START>\", \"that\", \"glitters\", \"is\", and \"not\".\n",
        "    \n",
        "        Params:\n",
        "            corpus (list of list of strings): corpus of documents like: [\n",
        "                ['<START>', 'All', 'that', 'glitters', \"isn't\", 'gold', '<END>'],\n",
        "                ['<START>', \"All's\", 'well', 'that', 'ends', 'well', '<END>']\n",
        "              ]\n",
        "            window_size (int): size of context window\n",
        "        Return:\n",
        "            M (a symmetric numpy matrix of shape (number of unique words in the corpus , number of unique words in the corpus)): \n",
        "                Co-occurence matrix of word counts. \n",
        "                The ordering of the words in the rows/columns should be the same as the ordering of the words given by the distinct_words function.\n",
        "            word2ind (dict): dictionary that maps word to index (i.e. row/column number) for matrix M.\n",
        "    \"\"\"\n",
        "    \n",
        "    # Get the vocabulary which is like ['<END>','<START>','All',\"All's\",\n",
        "    # 'ends','glitters','gold',\"isn't\",'that','well']\n",
        "    words, num_words = distinct_words(corpus)\n",
        "\n",
        "    M = None\n",
        "    word2ind = {}\n",
        "    \n",
        "    # -------------------------------------------------------------------------\n",
        "    \n",
        "    def _get_vicinity(w_i: str, corpus: list, window_size: int=4) -> list:\n",
        "      \"\"\"\n",
        "      Returns a list with words located after or before of current word w_ij\n",
        "      given a window_size. For example, if the corpus is\n",
        "      [\n",
        "        ['<START>', 'All', 'that', 'glitters', \"isn't\", 'gold', '<END>'],\n",
        "        ['<START>', \"All's\", 'well', 'that', 'ends', 'well', '<END>']\n",
        "      ]\n",
        "      the vicinity/CONTEXT of \"well\" is: [\"All's\", 'that', 'ends', '<END>']\n",
        "\n",
        "      Parameters\n",
        "      ----------\n",
        "      - w_i (str): The target word for which we'll determine its vicinity\n",
        "      - corpus (list): A list of lists, each item represent a document\n",
        "      - window_size (int): The range where we'll find neighbor words for w_i\n",
        "\n",
        "      Returns\n",
        "      -------\n",
        "      A list of words which are neighbors of w_i given a window_size\n",
        "      \"\"\"\n",
        "      \n",
        "      neighbors = []\n",
        "\n",
        "      for doc in corpus:\n",
        "        if w_i in doc:  # The word can appear several time in the document!\n",
        "          \n",
        "          # Indices where w_i appears in the current document\n",
        "          indices = [idx for idx, value in enumerate(doc) if value == w_i]\n",
        "\n",
        "          for i in indices:\n",
        "            lower_bound = i - window_size if i > window_size else 0\n",
        "            upper_bound = i + 1 + window_size  # No error if x[i: out-of-bounds]\n",
        "\n",
        "            neighbors.append(doc[lower_bound: i])\n",
        "            neighbors.append(doc[i + 1: upper_bound])\n",
        "\n",
        "      return [neighbor[0] for neighbor in neighbors if len(neighbor) > 0]\n",
        "\n",
        "    # .........................................................................\n",
        "\n",
        "    # Initialize the Matrix\n",
        "    M = np.zeros((num_words, num_words))\n",
        "\n",
        "    # word2ind is like: {0: '<END>', 1: '<START>', 2: 'All', ... }\n",
        "    word2ind = {word:idx for idx, word in enumerate(words)}\n",
        "\n",
        "    # For each word w_i in the vocabulary:\n",
        "    #   Given w_i, how often the word w_ij co-occurs in the vicinity of w_i\n",
        "    #   with current window_size? j = 1 ... len(vocabulary)\n",
        "    for w_i in words:\n",
        "      i = word2ind[w_i]  # Get the index of the current word w_i\n",
        "      w_ij = _get_vicinity(w_i, corpus, window_size)\n",
        "\n",
        "      for j, word_j in enumerate(words):\n",
        "        if word_j in w_ij:\n",
        "          M[i, j] += 1 \n",
        "    # -------------------------------------------------------------------------\n",
        "\n",
        "    return M, word2ind"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RZkb3yFA56PS"
      },
      "source": [
        "Sanity check:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tG4Rzamw58Fp",
        "outputId": "8ef84743-2952-4acf-d6a1-60b3b5e60e5a"
      },
      "source": [
        "# ---------------------\n",
        "# Run this sanity check\n",
        "# Note that this is not an exhaustive check for correctness.\n",
        "# ---------------------\n",
        "\n",
        "# Define toy corpus and get student's co-occurrence matrix\n",
        "test_corpus = [\"{} All that glitters isn't gold {}\".format(START_TOKEN, END_TOKEN).split(\" \"), \"{} All's well that ends well {}\".format(START_TOKEN, END_TOKEN).split(\" \")]\n",
        "M_test, word2ind_test = compute_co_occurrence_matrix(test_corpus, window_size=1)\n",
        "\n",
        "# Correct M and word2ind\n",
        "M_test_ans = np.array( \n",
        "    [[0., 0., 0., 0., 0., 0., 1., 0., 0., 1.,],\n",
        "     [0., 0., 1., 1., 0., 0., 0., 0., 0., 0.,],\n",
        "     [0., 1., 0., 0., 0., 0., 0., 0., 1., 0.,],\n",
        "     [0., 1., 0., 0., 0., 0., 0., 0., 0., 1.,],\n",
        "     [0., 0., 0., 0., 0., 0., 0., 0., 1., 1.,],\n",
        "     [0., 0., 0., 0., 0., 0., 0., 1., 1., 0.,],\n",
        "     [1., 0., 0., 0., 0., 0., 0., 1., 0., 0.,],\n",
        "     [0., 0., 0., 0., 0., 1., 1., 0., 0., 0.,],\n",
        "     [0., 0., 1., 0., 1., 1., 0., 0., 0., 1.,],\n",
        "     [1., 0., 0., 1., 1., 0., 0., 0., 1., 0.,]]\n",
        ")\n",
        "ans_test_corpus_words = sorted([START_TOKEN, \"All\", \"ends\", \"that\", \"gold\", \"All's\", \"glitters\", \"isn't\", \"well\", END_TOKEN])\n",
        "word2ind_ans = dict(zip(ans_test_corpus_words, range(len(ans_test_corpus_words))))\n",
        "\n",
        "# Test correct word2ind\n",
        "assert (word2ind_ans == word2ind_test), \"Your word2ind is incorrect:\\nCorrect: {}\\nYours: {}\".format(word2ind_ans, word2ind_test)\n",
        "\n",
        "# Test correct M shape\n",
        "assert (M_test.shape == M_test_ans.shape), \"M matrix has incorrect shape.\\nCorrect: {}\\nYours: {}\".format(M_test.shape, M_test_ans.shape)\n",
        "\n",
        "# Test correct M values\n",
        "for w1 in word2ind_ans.keys():\n",
        "    idx1 = word2ind_ans[w1]\n",
        "    for w2 in word2ind_ans.keys():\n",
        "        idx2 = word2ind_ans[w2]\n",
        "        student = M_test[idx1, idx2]\n",
        "        correct = M_test_ans[idx1, idx2]\n",
        "        if student != correct:\n",
        "            print(\"Correct M:\")\n",
        "            print(M_test_ans)\n",
        "            print(\"Your M: \")\n",
        "            print(M_test)\n",
        "            raise AssertionError(\"Incorrect count at index ({}, {})=({}, {}) in matrix M. Yours has {} but should have {}.\".format(idx1, idx2, w1, w2, student, correct))\n",
        "\n",
        "# Print Success\n",
        "print (\"-\" * 80)\n",
        "print(\"Passed All Tests!\")\n",
        "print (\"-\" * 80)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--------------------------------------------------------------------------------\n",
            "Passed All Tests!\n",
            "--------------------------------------------------------------------------------\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QcW87vnzggGy"
      },
      "source": [
        "### Reduce the co-occurrence matrix dimensionality with SVD"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8SvVUT2UgoBY"
      },
      "source": [
        "def reduce_to_k_dim(M, k=2):\n",
        "    \"\"\" Reduce a co-occurence count matrix of dimensionality (num_corpus_words, num_corpus_words)\n",
        "        to a matrix of dimensionality (num_corpus_words, k) using the following SVD function from Scikit-Learn:\n",
        "            - http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html\n",
        "    \n",
        "        Params:\n",
        "            M (numpy matrix of shape (number of unique words in the corpus , number of unique words in the corpus)): co-occurence matrix of word counts\n",
        "            k (int): embedding size of each word after dimension reduction\n",
        "        Return:\n",
        "            M_reduced (numpy matrix of shape (number of corpus words, k)): matrix of k-dimensioal word embeddings.\n",
        "                    In terms of the SVD from math class, this actually returns U * S\n",
        "    \"\"\"    \n",
        "    n_iters = 10     # Use this parameter in your call to `TruncatedSVD`\n",
        "    M_reduced = None\n",
        "    print(\"Running Truncated SVD over %i words...\" % (M.shape[0]))\n",
        "    \n",
        "    # ------------------\n",
        "    svd = TruncatedSVD(n_components=k, n_iter=n_iters)\n",
        "    M_reduced = svd.fit_transform(M)\n",
        "    # ------------------\n",
        "\n",
        "    print(\"Done.\")\n",
        "    return M_reduced"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TyQCoa8lgXDV"
      },
      "source": [
        "**Note**: TruncatedSVD returns U*S, then we need to normalize (rescale) the returned vectors, so that all the vectors will appear around the unit circle (therefore closeness is directional closeness). We achieve this normalization through the NumPy concept of broadcasting."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TX8D6uuslxos"
      },
      "source": [
        "Sanity check:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D0Yw48ASlzs9",
        "outputId": "31c3ad8d-7ee1-4e26-d742-7bbe27951280"
      },
      "source": [
        "# ---------------------\n",
        "# Run this sanity check\n",
        "# Note that this is not an exhaustive check for correctness \n",
        "# In fact we only check that your M_reduced has the right dimensions.\n",
        "# ---------------------\n",
        "\n",
        "# Define toy corpus and run student code\n",
        "test_corpus = [\"{} All that glitters isn't gold {}\".format(START_TOKEN, END_TOKEN).split(\" \"), \"{} All's well that ends well {}\".format(START_TOKEN, END_TOKEN).split(\" \")]\n",
        "M_test, word2ind_test = compute_co_occurrence_matrix(test_corpus, window_size=1)\n",
        "M_test_reduced = reduce_to_k_dim(M_test, k=2)\n",
        "\n",
        "# Test proper dimensions\n",
        "assert (M_test_reduced.shape[0] == 10), \"M_reduced has {} rows; should have {}\".format(M_test_reduced.shape[0], 10)\n",
        "assert (M_test_reduced.shape[1] == 2), \"M_reduced has {} columns; should have {}\".format(M_test_reduced.shape[1], 2)\n",
        "\n",
        "# Print Success\n",
        "print (\"-\" * 80)\n",
        "print(\"Passed All Tests!\")\n",
        "print (\"-\" * 80)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Running Truncated SVD over 10 words...\n",
            "Done.\n",
            "--------------------------------------------------------------------------------\n",
            "Passed All Tests!\n",
            "--------------------------------------------------------------------------------\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QmOYgg6lxyMl"
      },
      "source": [
        "### Enable the word embeddings plotter"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4NS93712x7GX"
      },
      "source": [
        "def plot_embeddings(M_reduced, word2ind, words):\n",
        "    \"\"\" Plot in a scatterplot the embeddings of the words specified in the list \"words\".\n",
        "        NOTE: do not plot all the words listed in M_reduced / word2ind.\n",
        "        Include a label next to each point.\n",
        "        \n",
        "        Params:\n",
        "            M_reduced (numpy matrix of shape (number of unique words in the corpus , 2)): matrix of 2-dimensioal word embeddings\n",
        "            word2ind (dict): dictionary that maps word to indices for matrix M\n",
        "            words (list of strings): words whose embeddings we want to visualize\n",
        "    \"\"\"\n",
        "\n",
        "    # ------------------\n",
        "    # M_reduced contains the values to plot\n",
        "    labels = words\n",
        "    x_values = M_reduced[:, 0]  # 1st column of words embeddings (1st dimension)\n",
        "    y_values = M_reduced[:, 1]  # 2nd column of word vectors (2nd dimension)\n",
        "\n",
        "    for idx, label in enumerate(labels):\n",
        "      x = x_values[idx]\n",
        "      y = y_values[idx]\n",
        "\n",
        "      plt.scatter(x, y, marker=\"x\", color=\"red\")\n",
        "      plt.text(x + 0.01, y, label, fontsize=9)\n",
        "    # ------------------\n",
        "\n"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-GG7mmhNNXmR"
      },
      "source": [
        "Sanity check:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 317
        },
        "id": "swFOzO9cNXAX",
        "outputId": "b59a3916-5c34-42d6-e88b-b8fb542cc29b"
      },
      "source": [
        "# ---------------------\n",
        "# Run this sanity check\n",
        "# Note that this is not an exhaustive check for correctness.\n",
        "# The plot produced should look like the \"test solution plot\" depicted below. \n",
        "# ---------------------\n",
        "\n",
        "print (\"-\" * 80)\n",
        "print (\"Outputted Plot:\")\n",
        "\n",
        "M_reduced_plot_test = np.array([[1, 1], [-1, -1], [1, -1], [-1, 1], [0, 0]])\n",
        "word2ind_plot_test = {'test1': 0, 'test2': 1, 'test3': 2, 'test4': 3, 'test5': 4}\n",
        "words = ['test1', 'test2', 'test3', 'test4', 'test5']\n",
        "plot_embeddings(M_reduced_plot_test, word2ind_plot_test, words)\n",
        "\n",
        "print (\"-\" * 80)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--------------------------------------------------------------------------------\n",
            "Outputted Plot:\n",
            "--------------------------------------------------------------------------------\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAD4CAYAAAD//dEpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAdo0lEQVR4nO3de5BW9Z3n8fdHGG+ZUi5NEaIouDAbMdlA+hnjThLpKN7AAhMxQIpNi1hMsjoSJVNBrS28Zkh2geDGmUg5YmNmgxndaGfUYkVhvQQyPO0YUVMI3jYwRttLJ0VBME1/949zmhz6ep5+nr7J51V16jnnd37nPN8+/XA+z7k0RxGBmZlZd47q7wLMzGxwcGCYmVkuDgwzM8vFgWFmZrk4MMzMLJeh/V1AT1RVVcW4ceP6uwwzs0GloaHh3YgY1dPlB2VgjBs3jmKx2N9lmJkNKpLeLGd5n5IyM7NcHBhmZpbLERsYTU1NrFu3rqRl3njjDerr69u119bWMm3atEqVZmbWpZ7uv4ATW6clfVnSryX9Ie86HBgl6CgwXnjhBZqamipZmplZl8oIjGGZpqeAKcDuvOs4cgKjzf+ZtXLFChoaGqipqaGuro4ZM2ZwzjnnMGPGDBobG9m3bx8XXXQRU6dOpaamhldeeYWVK1fyyCOPUFNTQ0NDAwC33norN9xwQ3/8RGZ2pKjQ/gs4UdJmSdUR8V5E5D66SOuIsgfgHuAd4MVO5gu4A9gFvAB8NjOvFtiZDrV53q+6ujpKsmxZxOLFES0tyXRLS7y+YEGcO358RETMmTMntmzZEhERDz30UCxZsiQaGhpi3rx5h1Zx8ODB2LRpUyxcuPBQ26ZNm+KGG26I119/Pc4999zSajIzy6OC+y+gMdrvn3e1betsqNRttfcCPwQ6O0a6CJiYDp8D/gH4nKQRwDKgAATQIKk+Ij6oUF1JMjc1werVyfSqVXDttbB2LYwdCxFs376dpUuXAtDc3MyECROYMmUK1dXVzJ8/n5EjR3LzzTe3W/Xy5ctZv369T0mZWe/oxf1XD+sp/wgjTalxdH6EcRcwLzO9AxgDzAPu6qxfZ0PJRxgtLUlCJ5s/AmLPwoUxderUiIi47LLL4rnnnjvU/cCBA7F///5oSRP91ltvjTvuuCOeffbZqK2tjYiI3//+9/GZz3wmLrjggjj77LNj5MiRcdttt5VWl5lZdyq4/wLejTKOMBQVeh6GpHHAv0TEpzqY9y/A8oh4Jp1+AvgOUAMcGxG3pe3/DdgfEf+jg3UsAhYBnHLKKdVvvlni359EwFF/umTT0tzMjIsv5vjjj2fmzJk8+OCD7N27F4ArrriCSZMmcc011zB06FBaWlqoq6ujqqqK6dOnM3r0aJYtW8anP/1pILmYdOWVV7Jx48bSajIzy6NC+68TTjhhL7ABuJnkAvgy4K+AXwB/HxH/u6syBk1gZBUKhSjpL70jksO41sM6gMWLk8M7Kf96zMz6WgX3X5IaIqLQ01L66i6pPcDYzPTJaVtn7ZWT3diLF0NLS/K6enXS7icOmtlANcD2X331f0nVA1dLWk9y0ft3EfGWpA3AdyUNT/udD1xf0XeWYNiwwxN51apk3rBhPsIws4FrgO2/KnJKStJPSE4vVQFvk5wX+zOAiPiRJJHcRXUhsA9YEBHFdNkrgNY/ZLg9ItZ2934ln5JKCjl847adNjMbqCq0/yr3lFRFjjAiYl438wO4qpN595D8HUfvartxHRZmNlgMkP3XkfOX3mZmVhYHhpmZ5eLAMDOzXBwYZmaWiwPDzMxycWCYmVkuDgwzM8vFgWFmZrk4MMzMLBcHhpmZ5eLAMDOzXBwYZmaWiwPDzMxycWCYmVkuDgwzM8vFgWFmZrlUJDAkXShph6RdkpZ2MH+VpOfT4RVJTZl5BzPz6itRj5mZVV7ZT9yTNAS4EzgP2A1sk1QfES+39omIazP9/waYklnF/oiYXG4dZmbWuypxhHEmsCsiXouID4H1wKwu+s8DflKB9zUzsz5UicA4CfhNZnp32taOpFOB8cCTmeZjJRUlbZV0SWdvImlR2q/Y2NhYgbLNzKwUfX3Rey7wQEQczLSdGhEF4GvADyT9h44WjIg1EVGIiMKoUaP6olYzM8uoRGDsAcZmpk9O2zoylzanoyJiT/r6GrCZw69vmJnZAFGJwNgGTJQ0XtLRJKHQ7m4nSZ8EhgNbMm3DJR2TjlcBnwdebrusmZn1v7LvkoqIZklXAxuAIcA9EfGSpFuAYkS0hsdcYH1ERGbx04G7JLWQhNfy7N1VZmY2cOjw/ffgUCgUolgs9ncZZmaDiqSG9Jpxj/gvvc3MLBcHhpmZ5eLAMDOzXBwYZmaWiwPDzMxycWCYmVkuDgwzM8vFgWFmZrk4MMzMLBcHhpmZ5eLAMDOzXBwYZmaWiwPDzMxycWCYmVkuDgwzM8vFgWFmZrlUJDAkXShph6RdkpZ2MP9ySY2Snk+HKzPzaiXtTIfaStRjZmaVV/YjWiUNAe4EzgN2A9sk1XfwqNX7I+LqNsuOAJYBBSCAhnTZD8qty8zMKqsSRxhnArsi4rWI+BBYD8zKuewFwOMR8X4aEo8DF1agJjMzq7BKBMZJwG8y07vTtrYulfSCpAckjS1xWSQtklSUVGxsbKxA2WZmVoq+uuj9c2BcRPwnkqOIulJXEBFrIqIQEYVRo0ZVvEAzM+taJQJjDzA2M31y2nZIRLwXEQfSybuB6rzLmpnZwFCJwNgGTJQ0XtLRwFygPttB0pjM5Ezg1+n4BuB8ScMlDQfOT9vMzGyAKfsuqYholnQ1yY5+CHBPRLwk6RagGBH1wDWSZgLNwPvA5emy70u6lSR0AG6JiPfLrcnMzCpPEdHfNZSsUChEsVjs7zLMzAYVSQ0RUejp8v5LbzMzy8WBYWZmuTgwzMwsFweGmZnl4sAwM7NcHBhmZpaLA8PMzHJxYJiZWS4ODDMzy8WBYWZmuTgwzMwsFweGmZnl4sAwM7NcHBhmZpaLA8PMzHKpSGBIulDSDkm7JC3tYP51kl6W9IKkJySdmpl3UNLz6VDfdlkzMxsYyn7inqQhwJ3AecBuYJuk+oh4OdPt34BCROyT9E3g+8CcdN7+iJhcbh1mZta7KnGEcSawKyJei4gPgfXArGyHiNgUEfvSya3AyRV4XzMz60OVCIyTgN9kpnenbZ1ZCDyWmT5WUlHSVkmXdLaQpEVpv2JjY2N5FZuZWcnKPiVVCknzgQIwNdN8akTskXQa8KSk7RHxattlI2INsAaSZ3r3ScFmZnZIJY4w9gBjM9Mnp22HkTQNuBGYGREHWtsjYk/6+hqwGZhSgZrMzKzCKhEY24CJksZLOhqYCxx2t5OkKcBdJGHxTqZ9uKRj0vEq4PNA9mK5mZkNEGWfkoqIZklXAxuAIcA9EfGSpFuAYkTUA/8d+HPgnyUB/L+ImAmcDtwlqYUkvJa3ubvKzMwGCEUMvssBhUIhisVif5dhZjaoSGqIiEJPl/dfepuZWS4ODDMzy8WBYWZmuTgwzMwsFweGmZnl4sAwM7NcHBhmZpaLA8PMzHJxYJiZWS4ODDMzy8WBYWZmuTgwzMwsFweGmZnl4sAw6wVNTU2sW7eupGXeeOMN6uv/9CiZm266idNPP52amhpqamo4ePBgpcs0K4kDw6wXVCIwAG688UY2b97M5s2bGTJkSCVLNCuZA8OsEto8V2blihU0NDRQU1NDXV0dM2bM4JxzzmHGjBk0Njayb98+LrroIqZOnUpNTQ2vvPIKK1eu5JFHHqGmpoaGhgYAvv/97/OFL3yBO+64oz9+KrPDRUTZA3AhsAPYBSztYP4xwP3p/F8C4zLzrk/bdwAX5Hm/6urqMBswli2LWLw4oqUlmW5pidcXLIhzx4+PiIg5c+bEli1bIiLioYceiiVLlkRDQ0PMmzfv0CoOHjwYmzZtioULFx5qe/fdd6OlpSX27dsX5557bjz11FN99iPZRxPJU1B7vK8v+xGtkoYAdwLnAbuBbZLq4/BHrS4EPoiICZLmAt8D5kiaRPIM8DOATwAbJf1FRPhkrQ0OEdDUBKtXJ9OrVsG118LatTB2LESwfft2li5dCkBzczMTJkxgypQpVFdXM3/+fEaOHMnNN9/cbtUjR44E4LjjjuMrX/kKxWKRL37xi332o5m1VXZgAGcCuyLiNQBJ64FZQDYwZgE3peMPAD9U8nDvWcD6iDgAvC5pV7q+LRWoy6z3SUlIQBIaaXAcvXAhzbt2gcQZZ5zB9ddfz5QpUwD48MMPOXDgANdddx2SuO2227jvvvuorq6mubn50KqbmpoYNmwYEcHmzZu5/PLL+/qnMztMJQLjJOA3mendwOc66xMRzZJ+B4xM27e2Wfakjt5E0iJgEcApp5xSgbLNKqQ1NFqPMoCP33UXx118MZdeeikzZ85k2bJl7N27F4ArrriCSZMmcc011zB06FBaWlqoq6ujqqqKV199ldmzZ7Ns2TJWrFjBjh07iAhqamqYPn16f/2EZkBlAqNPRMQaYA1AoVCIbrqb9Z2I5DRUxlFLlvDYo48mYQLU1ta2W+yZZ55p1/b0008fGr/33nsrW6dZmSpxl9QeYGxm+uS0rcM+koYCJwLv5VzWbOBqDYvVq2HxYmhpSV5Xr07aw99t7KOjEkcY24CJksaT7OznAl9r06ceqCW5NjEbeDIiQlI98L8krSS56D0R+NcK1GTWNyQYNiwJiVWrDr+mMWzYoSMMs4+CsgMjvSZxNbABGALcExEvSbqF5BaueuAfgfvSi9rvk4QKab+fklwgbwau8h1SNujcdFNyJNEaDq2h4bCwjxjFIDxkLhQKUSwW+7sMM7NBRVJDRBR6urz/0tvMzHJxYJiZWS4ODDMzy8WBYWZmuTgwzMwsFweGmZnl4sAwM7NcHBhmZpaLA8PMzHJxYJiZWS4ODDMzy8WBYWZmuTgwzMwsFweGmZnl4sAwM7NcHBhmZpZLWYEhaYSkxyXtTF+Hd9BnsqQtkl6S9IKkOZl590p6XdLz6TC5nHrMzKz3lHuEsRR4IiImAk+k023tA74eEWcAFwI/kDQsM/9vI2JyOjxfZj1mZtZLyg2MWUBdOl4HXNK2Q0S8EhE70/F/B94BRpX5vmZm1sfKDYzREfFWOv5bYHRXnSWdCRwNvJppvj09VbVK0jFdLLtIUlFSsbGxscyyzcysVN0GhqSNkl7sYJiV7RcRAUQX6xkD3AcsiIiWtPl64JPAXwIjgO90tnxErImIQkQURo3yAYqZWV8b2l2HiJjW2TxJb0saExFvpYHwTif9TgAeAW6MiK2ZdbcenRyQtBb4dknVm5lZnyn3lFQ9UJuO1wIPt+0g6WjgZ8C6iHigzbwx6atIrn+8WGY9ZmbWS8oNjOXAeZJ2AtPSaSQVJN2d9vkqcDZweQe3z/6TpO3AdqAKuK3MeszMrJcoufQwuBQKhSgWi/1dhpnZoCKpISIKPV3ef+ltZma5ODDMzCwXB4aZmeXiwDAzs1wcGGZmlosDw8zMcnFgmJlZLg4MMzPLxYFhZma5ODDMzCwXB4aZmeXiwDAzs1wcGGZmlosDw8zMcnFgmJlZLmUFhqQRkh6XtDN9Hd5Jv4OZhyfVZ9rHS/qlpF2S7k+fzmdmZgNQuUcYS4EnImIi8EQ63ZH9ETE5HWZm2r8HrIqICcAHwMIy6zEzs15SbmDMAurS8TqS53Lnkj7H+xyg9TnfJS1vZmZ9q9zAGB0Rb6XjvwVGd9LvWElFSVsltYbCSKApIprT6d3ASZ29kaRF6TqKjY2NZZZtZmalGtpdB0kbgY93MOvG7EREhKTOHhB+akTskXQa8KSk7cDvSik0ItYAayB5pncpy5qZWfm6DYyImNbZPElvSxoTEW9JGgO808k69qSvr0naDEwBHgSGSRqaHmWcDOzpwc9gZmZ9oNxTUvVAbTpeCzzctoOk4ZKOScergM8DL0dEAJuA2V0tb2ZmA0O5gbEcOE/STmBaOo2kgqS70z6nA0VJvyIJiOUR8XI67zvAdZJ2kVzT+Mcy6zEzs16i5Iv+4FIoFKJYLPZ3GWZmg4qkhogo9HR5/6W3mZnl4sAwM7NcHBhmZpaLA8PMzHJxYJiZWS4ODDMzy8WBYWZmuTgwzMwsFweGmZnl4sAwM7NcHBhmZpaLA8PMzHJxYJiZWS4ODDMzy8WBYWZmuTgwzMwsl7ICQ9IISY9L2pm+Du+gz5ckPZ8Z/iDpknTevZJez8ybXE49ZmbWe8o9wlgKPBERE4En0unDRMSmiJgcEZOBc4B9wP/JdPnb1vkR8XyZ9ZiZWS8pNzBmAXXpeB1wSTf9ZwOPRcS+Mt/XzMz6WLmBMToi3krHfwuM7qb/XOAnbdpul/SCpFWSjulsQUmLJBUlFRsbG8so2czMeqLbwJC0UdKLHQyzsv0iIoDoYj1jgE8DGzLN1wOfBP4SGAF8p7PlI2JNRBQiojBq1KjuyjYzswob2l2HiJjW2TxJb0saExFvpYHwTher+irws4j4Y2bdrUcnByStBb6ds24zM+tj5Z6Sqgdq0/Fa4OEu+s6jzemoNGSQJJLrHy+WWY+ZmfWScgNjOXCepJ3AtHQaSQVJd7d2kjQOGAv83zbL/5Ok7cB2oAq4rcx6zMysl3R7SqorEfEecG4H7UXgysz0G8BJHfQ7p5z3NzOzvuO/9DYzs1wcGGZmlosDw8zMcnFgmJlZLg4MMzPLxYFhZma5ODDMzCwXB4aZmeXiwDAzs1wcGGZmlosDw8zMcnFgmJlZLg4MMzPL5YgNjKamJtatW1fSMm+88Qb19fWHpr/1rW9x1llncdZZZ7F8+fJKl2hm1qGe7r+AE1unJV0n6SlJz0paJ+nPuluHA6MEbQPjqquuYuvWrfziF7/g4Ycf5tVXX610mWZm7ZQRGMMyTT+MiLMj4vPp9PndraOs52EMKhEgHZpcuWIFDQ0N1NTUsGDBAn7605+yf/9+jjvuOO69914+9rGPcemll7Jv3z4ksWbNGlauXMm2bduoqalhxYoVVFdXA3DUUUcxdOhQhgwZ0l8/nZl9lFVo/wWcKGkzsCQiGuDQE0+PAnblqCN6PACXAS8BLUChi34XAjvSgpZm2scDv0zb7weOzvO+1dXVUZJlyyIWL45oaUmmW1ri9QUL4tzx4yMiYs6cObFly5aIiHjooYdiyZIl0dDQEPPmzTu0ioMHD8amTZti4cKF7Vb/4x//OL7+9a+XVpOZWR4V3H8BjXH4vvlGYCfwKHB8dLPvLfcI40XgK8BdnXWQNAS4EzgP2A1sk1QfES8D3wNWRcR6ST8CFgL/UGZNh4uApiZYvTqZXrUKrr0W1q6FsWMhgu3bt7N06VIAmpubmTBhAlOmTKG6upr58+czcuRIbr755g5Xv3HjRtauXcvPf/7zipZtZtbb+6+IuF3Sd4EfApcDf99NPT0/wsik1GY6OcIA/jOwITN9fToIeBcY2lG/roaSjzBaWpKETjZ/BMSehQtj6tSpERFx2WWXxXPPPXeo+4EDB2L//v3Rkib6rbfeGnfccUc8++yzUVtbe6jf1q1b46yzzooPPvigtHrMzPKq4P4LeDf+tC8+NjP+XeDy6Gbfq7RzWdJzYt+O5FnebefNBi6MiCvT6f8CfA64CdgaERPS9rHAYxHxqU7eYxGwCOCUU06pfvPNN0srMgKO+tM1/pbmZmZcfDHHH388M2fO5MEHH2Tv3r0AXHHFFUyaNIlrrrmGoUOH0tLSQl1dHVVVVUyfPp3Ro0ezbNky5s2bB0BVVRXAYdc1zMwqpkL7rxNOOGEvsAG4GfgGcAZ/un7x1xHxx67K6PaUlKSNwMc7mHVjRDyc88ctW0SsAdYAFAqF0lIuIjmMyzhqyRIee/TRQxeSamtr2y32zDPPtGt7+umnD42/+OKLJZVhZlayCu6/gB0RMTsdv6rUUrq9rTYipkXEpzoY8obFHmBsZvrktO09YJikoW3aK6t1Y69eDYsXQ0tL8rp6ddJegSMsM7NeMcD2X31xW+02YKKk8SSBMBf4WkSEpE3AbGA9UAtU/ohFgmHDko28alUyvWpVMm/YsMNuVTMzG1AG2P6rrGsYkr4M/E9gFNAEPB8RF0j6BHB3RExP+00HfgAMAe6JiNvT9tNIwmIE8G/A/Ig40N37FgqFKBbbXS7pWpv7mNtNm5kNVBXaf0lqiIhCT8uoyEXvvtajwDAzO8KVGxhH7H8NYmZmpXFgmJlZLg4MMzPLxYFhZma5DMqL3pIagRL/1PuQKpL/kmSgcV2lcV2lcV2l+ajWdWpEjOrpwoMyMMohqVjOXQK9xXWVxnWVxnWVxnV1zKekzMwsFweGmZnlciQGxpr+LqATrqs0rqs0rqs0rqsDR9w1DDMz65kj8QjDzMx6wIFhZma5fCQDQ9Jlkl6S1CKp01vQJF0oaYekXZKWZtrHS/pl2n6/pKMrVNcISY9L2pm+Du+gz5ckPZ8Z/iDpknTevZJez8yb3Fd1pf0OZt67PtPen9trsqQt6e/7BUlzMvMqur06+7xk5h+T/vy70u0xLjPv+rR9h6QLyqmjB3VdJ+nldPs8IenUzLwOf6d9VNflkhoz739lZl5t+nvfKan904F6t65VmZpekdSUmdcr20vSPZLekdThU9mUuCOt+QVJn83M67Vt1U53z3AdjANwOvAf6fpZ40OAV4HTgKOBXwGT0nk/Beam4z8Cvlmhur4PLE3HlwLf66b/COB94Ph0+l5gdi9sr1x1AXs7ae+37QX8BTAxHf8E8BYwrNLbq6vPS6bPfwV+lI7PBe5Pxyel/Y8BxqfrGdKHdX0p8xn6ZmtdXf1O+6iuy4EfdrDsCOC19HV4Oj68r+pq0/9vSB7J0Nvb62zgs8CLncyfDjwGCDgL+GVvb6uOho/kEUZE/DoidnTT7UxgV0S8FhEfkjyXY5YkAecAD6T96oBLKlTarHR9edc7m+Q55/sq9P6dKbWuQ/p7e0XEKxGxMx3/d+AdkuezVFqHn5cu6n0AODfdPrOA9RFxICJeJ3l+8pl9VVdEbMp8hraSPN2yt+XZXp25AHg8It6PiA+Ax4EL+6muecBPKvTenYqIp0i+HHZmFrAuEltJnlY6ht7dVu18JAMjp5OA32Smd6dtI4GmiGhu014JoyPirXT8t8DobvrPpf2H9fb0kHSVpGP6uK5jJRUlbW09TcYA2l6SziT51vhqprlS26uzz0uHfdLt8TuS7ZNn2d6sK2shyTfVVh39TvuyrkvT388Dklof5Twgtld66m488GSmube2V3c6q7s3t1U7ffGI1l4haSPw8Q5m3Rj5nzdecV3VlZ2IiJDU6T3N6beHTwMbMs3Xk+w4jya5H/s7wC19WNepEbFHyZMSn5S0nWSn2GMV3l73AbUR0ZI293h7fRRJmg8UgKmZ5na/04h4teM1VNzPgZ9ExAFJf01ydHZOH713HnOBByLiYKatP7dXvxu0gRER08pcxR5gbGb65LTtPZLDvaHpt8TW9rLrkvS2pDER8Va6g3uni1V9FfhZRPwxs+7Wb9sHJK0Fvt2XdUXEnvT1NUmbgSnAg/Tz9pJ0AvAIyZeFrZl193h7daCzz0tHfXZLGgqcSPJ5yrNsb9aFpGkkITw1Mo9B7uR3WokdYLd1RcR7mcm7Sa5ZtS5b02bZzRWoKVddGXOBq7INvbi9utNZ3b25rdo5kk9JbQMmKrnD52iSD0d9JFeSNpFcPwCoBSp1xFKfri/PetudO013mq3XDS4BOryjojfqkjS89ZSOpCrg88DL/b290t/dz0jO7z7QZl4lt1eHn5cu6p0NPJlun3pgrpK7qMYDE4F/LaOWkuqSNAW4C5gZEe9k2jv8nfZhXWMykzOBX6fjG4Dz0/qGA+dz+JF2r9aV1vZJkovIWzJtvbm9ulMPfD29W+os4HfpF6Le3Fbt9dbV9P4cgC+TnMs7ALwNbEjbPwE8muk3HXiF5BvCjZn200j+Qe8C/hk4pkJ1jQSeAHYCG4ERaXsBuDvTbxzJN4ej2iz/JLCdZMf3Y+DP+6ou4K/S9/5V+rpwIGwvYD7wR+D5zDC5N7ZXR58XklNcM9PxY9Off1e6PU7LLHtjutwO4KIKf967q2tj+u+gdfvUd/c77aO6/g54KX3/TcAnM8tekW7HXcCCvqwrnb4JWN5muV7bXiRfDt9KP8u7Sa41fQP4RjpfwJ1pzdvJ3P3Zm9uq7eD/GsTMzHI5kk9JmZlZCRwYZmaWiwPDzMxycWCYmVkuDgwzM8vFgWFmZrk4MMzMLJf/D7FU1JshgkPiAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bO5UPZeZQuQo"
      },
      "source": [
        "### Co-occurence plot analysis for Reuter's \"crude\" corpus"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-xzGSFhBQ0vk",
        "outputId": "c02a3807-4c46-4b08-f202-d0b3f67407e0"
      },
      "source": [
        "# -----------------------------\n",
        "# Run This Cell to Produce Your Plot\n",
        "# ------------------------------\n",
        "reuters_corpus = read_corpus()\n",
        "M_co_occurrence, word2ind_co_occurrence = compute_co_occurrence_matrix(reuters_corpus)\n",
        "M_reduced_co_occurrence = reduce_to_k_dim(M_co_occurrence, k=2)\n",
        "\n",
        "# Rescale (normalize) the rows to make them each of unit-length\n",
        "M_lengths = np.linalg.norm(M_reduced_co_occurrence, axis=1)\n",
        "M_normalized = M_reduced_co_occurrence / M_lengths[:, np.newaxis] # broadcasting\n"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Running Truncated SVD over 8185 words...\n",
            "Done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "id": "n5YqU_IkhKds",
        "outputId": "b5ad2d6a-380a-4b0a-9e1b-4ce5b858059b"
      },
      "source": [
        "# Plot a set of words using the trained embedding model\n",
        "words = ['barrels', 'bpd', 'ecuador', 'energy', 'industry', 'kuwait', 'oil', 'output', 'petroleum', 'iraq']\n",
        "\n",
        "plot_embeddings(M_normalized, word2ind_co_occurrence, words)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaIAAAD4CAYAAACuaeJKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de3RV5bnv8e+TYKS1QFAQFIyRoihqyyVSpFEiF+9AZaOg2ANUpG66a0pt95GDw5NDa0uPQzFqa6tYtKC14rEl9cIuF4NSFAkVBQQE2RHDVS6iDBGBPOePOReshFxZK5m5/D5jZKw555przmetseTnO993vdPcHRERkaikRF2AiIg0bwoiERGJlIJIREQipSASEZFIKYhERCRSLaIuoDLt2rXzzMzMqMsQEWlUVqxYscvd20ddR2002CDKzMykqKgo6jJERBoVM/so6hpqS5fmREQkUgoiERGJlIJIRJJi+/bt3HXXXTXad/z48RQWFtbq+H/729/YvHnzCVQmDZ2CSESSomPHjjzwwAN1dvzKgujIkSN1dk6pHwoiEalY+Xkoq5mXsri4mEGDBpGXl8fo0aMZOnQoPXr0YN26dQDMmTOHHj16cMMNN/Dhhx+WeU1M165dASgsLKRPnz5cccUVjBs3jvfff5958+bx4x//mBtvvBGAs88+m4kTJzJs2DBGjhzJO++8A8BHH33E4MGDk/IRSP1osKPmRCRCeXnw6acwfTqYBSE0aRKkpwfPVaN9+/Y888wzPPvss8yYMYPf/OY3TJkyhRUrVtCyZUu+/e1vV/n6F198kV/+8pdceeWVlJaWkpKSwtVXX8348ePJzs4GYNu2bdx9991kZGSwcOFCnnzySR599FFmzpzJbbfdloQPQeqLWkQiUpZ7EEL5+UH4xEIoPz/YXoMZ+3v37g1ARkYGu3fvZteuXXTo0IFWrVpx0kkn0atXLwDMrNypg2P//Oc/p6CggNGjRzNz5swKz9GpUycyMjIAGDBgAMuWLeOLL77g73//OzfccMMJv32pf2oRiUhZZkFLCILwyc8PlnNzj7WQqj3EsX3cnXbt2rFjxw72799Py5YtWblyJQBt27Zl69atuDs7duxgy5YtAJx22mk8+uijuDvnnXceN954I2lpaRw+fPjocVNTU8ucb8SIEUycOJHLL7+ck08+OdFPQeqRgkhEjhcLo1gIQY1DqCKpqalMnTqV7OxszjnnHDp16gRA69atufrqq7n00kvp06cPHTp0AODBBx/kH//4B6WlpQwePJjWrVtz/fXXc++993LBBRfwhz/84bhzjBs3js6dOx/tK5LGwxrqjfGysrJcMyuIRCT+clxMLVpEUdixYwc333wzixYtirqUSJnZCnfPirqO2khKH5GZXW1m681so5ndXcV+/2ZmbmaN6kMSaVbiQyg3F0pLg8f4PqMGZv78+QwdOpR77rkn6lLkBCR8ac7MUoHfAoOBEmC5mRW4+/vl9msF5ALLEj2niNQhs2B0XHwLKNZnlJ7eIFtEgwcP1pDtRiwZfUR9gI3uvgnAzJ4DhgHvl9vvF8BvgJ8n4ZwiUpfy8oKWTyx0YmHUAENIGr9kXJrrBHwct14SbjvKzHoBZ7n7y0k4n4jUh/KhoxCSOlLnvyMysxTgQaDaSajMbIKZFZlZ0SeffFLXpYmISAOQjCDaApwVt9453BbTCrgIKDSzYqAvUFDRgAV3f9zds9w9q337RnVfJxEROUHJCKLlwLlmdo6ZpQGjgILYk+6+z93buXumu2cCbwFD3V1js0VEJPEgcvfDwH8A/wWsBZ539zVmNtXMhiZ6fBERadqSMrOCu78CvFJu272V7JuTjHOKiEjToElPRUQkUgoiERGJlIJIREQipSASEZFIKYhERCRSCiIREYmUgkhERCKlIBIRkUgpiEREJFIKIhERiZSCSEREIqUgEhGRSCmIRESSZNq0aaxatQqArl27RlxN45GU2bdFRATuvvvuqEtolNQiEhFxr3q9wpc4P/zhD8nOzqZfv368/fbbjB07liVLltRRkU2XWkQi0rzl5cGnn8L06WAWhNCkSZCeHjxXiblz53Lo0CGWLFnCpk2bGDVqFN27d6+3spsStYhEpPlyD0IoPz8In1gI5ecH26toGa1fv55+/foB0KVLF/bu3VtfVTc5CiIRab7MgpZQbm4QPikpwWNu7rEWUiW6devG0qVLAdi0aRPp6en1VXWToyASkeYtFkbxqgkhgKFDh5Kamkp2djajR4/mkUceOW6f4uJiBg0alMxqyxw7OzubnJycOjl+fVIfkYg0b7HLcfEmTao2jFJSUnjiiSfKbOvbt+/R5Y0bN1JcXFzrckpLS0lJOdZGOHLkCKmpqbU+TnlmluruRxI+UB1QEIlI8xXfJxS7HBdbh+PDqLQ0uHxX2XoFVq9ezemnn84XX3zBLbfcwoABA5g4cSJHjhwhLS2Njz76iIkTJzJ//nzGjBnDX//6V3r16sWuXbsoLi7myy+/5OSTTwbgzDPP5Jvf/Cbbt29n3759FBcX07VrV6ZPn87MmTPZvHkzwEVm9hdgrLsfMLOPgJeBDOD65H14yaNLcyLSfJkFo+Pi+4RifUbp6WVDKCcHevcOwgeCx969g+2VWLx4Mfv27ePDDz9ky5YtzJo1i1mzZvHnP/+Zffv2MW7cOF544QUgaPkMGTKEKVOmcNJJJ7F161aWLVtGcXExX/va1ygtLWX+/Pls3LiRdu3aMXv27KP9Us888wwZGRkUFBQArAbWALeHZZwBTHP3BhlCoBaRiDR3eXlByygWOrEwKt8S2rcPVq4MwmfFiuBx5Uro0aPSltG6deswM4YMGQJAWloaa9eu5f777+dXv/oVO3bsoE2bNpgZqamp9O3bl40bNwKQlZXFjBkzmDNnDmvWrAHg8ssvZ+PGjfTs2ROA1q1bA/DQQw9x/fXXx9a/ASwFhodlbHH3zcn90JJLLSIRkfJ9QeXXU1KC8OnRIwif1NRjIbRiRaWX58477zzcnZdeeokFCxZw9tlnc+DAAYYMGcLixYsZMmQI7s6pp57K4cOHMTNWrFgBwOHDh5k5cyb//Oc/ufjiiznzzDOZM2cOkydP5vTTTwfgs88+A6BXr17k5OQwfPhwgLOAfsD6sIwG2S8Uz7wGvyCOQlZWlhcVFUVdhojIMaWlQQjFHDlSZR9RcXEx/fr14/Dhwxw4cIBu3boxZswYJk+eTMuWLUlLS+OOO+7gxhtvpEePHlx11VW0a9eO0tJSUlNT+eyzzygpKaFt27a88cYbXHTRRaSlpXHkSJAtmzdvpkuXLpx22mls3ryZ9evX8/nnnx8EXgK+H/YRbXT3Bj3xXVKCyMyuBvKBVGCGu08r9/xPgfHAYeAT4Afu/lFVx1QQiUiDEusTWrny2LZqWkRRMLMV7p4VdR21kfCnZ2apwG+Ba4DuwM1mVn6ei3eALHf/FvAC8H8TPa+ISL2JD6EePYKWUOwyXfwABjkhyYjxPsBGd9/k7l8BzwHD4ndw99fc/Ytw9S2gcxLOKyJSP1JSoE2bsi2gWJ9RmzYNqkXUGCVj1Fwn4OO49RLgO1XsfxvwahLOKyJSfwoLy46Oi4WRQihh9Tp828xuBbKA/pU8PwGYAJCRkVGPlYmI1ED50FEIJUUyPsUtBMMFYzqH28ows0HAFGCoux+s6EDu/ri7Z7l7Vvv27ZNQmoiINHTJCKLlwLlmdo6ZpQGjgIL4HcysJ/AHghDamYRziohIE5FwELn7YeA/gP8C1gLPu/saM5tqZkPD3e4n+LXvHDNbaWYFlRxORESamaT0Ebn7K8Ar5bbdG7dcN/Ogi4hIo6eeNhERiZSCSEREIqUgEhGRSCmIREQkUgoiERGJlIJIREQipSASEZFIKYhERCRSCiIREYmUgkhERCKlIBIRkUgpiEREJFIKIhERiZSCSEREIqUgEhGRSCmIREQkUgoiERGJlIJIREQipSASEZFIKYhERCRSCiIREYmUgkhERCKlIBIRaQbM7M4EXjvWzFons554CiIRkebhhIMIGAvUWRC1qKsDi4hILbmDWeXrx+3u3HHHHaxZs4bS0lIeeughgG5m1tndS8zsHqAE+AroZGaFwHzgn8D/Bj4FzgHuc/c5ZvYUMMPdl5jZrUBX4HWgBzDHzIrc/cfJfttJCSIzuxrIB1IJ3sS0cs+fDPwJ6A3sBka6e3Eyzi0i0iTk5cGnn8L06UH4uMOkSZCeHjxXgblz53Lo0CGWLFnCpk2bGDVqVIX7ufuzZjbV3XMAzCwHaA8MBr4OFJnZ/6vktYvMbCVwq7uXJPguK5TwpTkzSwV+C1wDdAduNrPu5Xa7Ddjr7l2B6cBvEj2viEiT4R6EUH5+ED6xEMrPD7a7V/iy9evX069fPwC6dOnC3r17AeJ3rrw5Be+4+2F3/wzYSRBMNX1tUiWjj6gPsNHdN7n7V8BzwLBy+wwDng6XXwAGmlXR3hQRaU7MgpZQbm4QPikpwWNu7rEWUgW6devG0qVLAdi0aRPp6ekAR4DO4S6943Y/bGbx/+b3MLMWZtYK6AB8Auyp5LVfUYddOckIok7Ax3HrJeG2Cvdx98PAPuC0JJxbRKRpiIVRvCpCCGDo0KGkpqaSnZ3N6NGjeeSRRwB2ADPM7EXgYNzuLwAvx42e2wrMAd4A7nH3UmAGcLeZFVB2cMKLwJNm9otE3mJlGtRgBTObAEwAyMjIiLgaEZF6FLscF2/SpCrDKCUlhSeeeKL85v3unnX84f2e2HLYR/Sxu48vt89agoEJ5V/7e+D3NXofJyAZLaItwFlx653DbRXuY2YtgDYEgxbKcPfH3T3L3bPat2+fhNJERBqB+D6h3FwoLT12mS7WZ9SEJaNFtBw418zOIQicUcAt5fYpAMYAbwIjgEXuTfyTFRGpKbNgdFx8n1DsMl16epWX506EuxcChUk9aAIsGXlgZtcCDxEM3/6ju99nZlOBIncvMLOWwCygJ0Fn2Ch331TVMbOysryoqCjh2kREGo1a/o6oIma2oqJLcw1ZUvqI3P0V4JVy2+6NW/4SuDEZ5xIRabLKh04zGVysKX5ERCRSCiIRkSasthOWmlmmmS2oy5rKUxCJiDRtY6lgwtJwVpwGQUEkIhKl8gPGajCArLi4mF69ejFy5EiysrLIz89n37593HTTTQDnmdkiM+tqZgM4NmHpI2FrZ7mZzQKeMLO+ZrbUzJaY2WPlZ7wxs7PM7OXweC+bWftw+8a4fRaEx800sxVmNtvMVpvZ7Wb2tJn9y8wmV/V+GtQPWkVEmpUTmOg05uOPP2bx4sW0bNmSSy65hHfffZfhw4czZ86cD4BJwDR3HxE/YamZZQKZwEB3/8zMioCb3H2Tmf0RGAK8F3ea+4FfuPtbZjYM+J/Az6ooqyPwXSAd+Ag4G9gFrAd+XdmL1CISEYnCCU50GnP++efTqlUrTjrpJC666CK2bdtGfn4+QDeCuyGkV/LS1eFEpwBt4n5KsxQ4v9y+FwPTwttH/BxoV8Hx4ltR69z9S3ffDpS4+/ZwWrcDVV0KVItIRCQK8T9azc8P/qDaiU5j1q1bx/79+2nZsiWrV6+mV69eTJgwgeHDh6939xwzSwt3LT9h6ZG45X1m1iUMo37A3HKnWQP82t3fCUo+esyU8PY+qcAFcft7JctQxWzeahGJiETlBCY6jcnMzOT222+nb9++jBkzhunTp/P8889D0Ef0GsfuyFrVhKV3As+Y2RLgEMEsOPHuAv5P2Ee0CLgp3P4o8BbBLYASvkdRUmZWqAuaWUFEmrz4y3ExNWgRFRcXM378eBYsOH6UdWOcWUEtIhGRKDTziU7jqY9IRCQKCUx0mpmZWWFrqLHSpTkRkSglYaLTeLo0JyIitVNPE502pJkUytOlORGR+nICrZ/JkyezdOlSvvrqK6ZMmUJRUREbNmzg888/Z/PmzTz33HOcf/75LF68mHvvvRegm5n9Hvh3gh+UzgHWAYfM7AFgJvAJwc1JNwF/Ifjx6zAAM3sSeMrd30jyu6+UWkQiIvUhL6/sIITYYIUqZlCYN28ee/fuZfHixSxcuJApU6bg7rRv356CggL+8z//kxkzZuDu/OQnP6GgoACCWQwOANeFh8kEfuTuPyCY3eBOd78OOBiU4WuBVmbW0cy+AXyrPkMIFEQiInXvBGdRWLVqFYsXLyYnJ4drr72WgwcPsnv3bnr37g1ARkYGu3fvZteuXRQXFzNs2DAIZla4DOgcHiZ+JoWuBHfVBlgWd6qZBJOj3kTQQqpXCiIRkboWGxEXG56dknJs2HY4Yq64uJhBgwaVedmFF17IlVdeSWFhIYWFhbz33nu0a9eO+LlJ3333XV566SW6dOnCSy+9BLA+HKzwZLhL/EwKHwKxgQyXxG2fA9wAfB94OqnvvQYURCIi9eEEZlG49tpradWqFTk5OVxxxRXcdtttlRzaePDBBxk6dCgEMysspOzUOzH/C3jUzF4F2hBM/xO7i/ZbwC53/6TW7y1BGqwgIlIfYpfj4k2aVCaM9uzZw8iRI/nwww/5/ve/T5s2bXj33Xdp1aoVJSUlTJ06lcsuu4zXX3+dnj170rlzZzp16gRA//79WbRoEWb2gbsPjDtLfDNrnbtfAmBmTwAfxFcIPJ70910DCiIRkboQPyLOHX7yE3j44WOX4+Kn9glbSuVv7XDLLbdw6NAh5s2bR3FxMSNGjKCoqIif/vSnzJ07l7POOourrrqqNlVdbGb5BP/2FwN/AzCzp4FW7j4/OW++dhREIiLJVv4+QwDLlsF3vlPlLAqxWzsAXHTRRbg7l1wSdOVkZmayb98+AD777DMyMjIA6NOnT43Lcvd/EQxkKL99zIm8zWRRH5GISDJVNkIuFkQxsTCKG74du7XD4cOHWb16NWbGihUrANi8eTOtWwd3/I5dqgNYvnw5jZ1aRCIiyVSb+wyVG6gQu7XDhg0bGDNmDG3btuXrX/861113HVu3bmV6eNwHHniAIUOGcOaZZx5tQTVmmmtORKQuuAfDtGNKS2s9fc9TTz1FSUkJ99xzT41fo7nmRESk8hFyDfR//KOW0KU5MzuV4Fe4mQQjMG5y973l9ukBPAa0Jvhh1X3uXu+/3BURqRfl7zNU0Qi5GraMxo4dW3d1NiCJtojuBha6+7nAwnC9vC+A/+HuFwJXAw+ZWXqC5xURaZgqu89Qbm619xlqrhINomEcmw7iaeB75Xdw9w/cfUO4vBXYCbRP8LwiIg1XXh7FubkMGjw4WK9ghFwinnrqKebPD37y8/DDDyflmFFKNIg6uPu2cHk70KGqnc2sD5BGMN+RiEjTVYf3GRo7diyDw5BrFkFkZgvMbHUFf8Pi9/Ng+F2lPXFmdgYwCxjn7qWV7DPBzIrMrOiTT+p9uiMRkcqVH2hQi4EHjz76KBdccAHjxo07uq1r164ADBw4kD179rBq1SrS0tL4/PPPWb58ObfffjsAV111FTk5OfTp04c333wTgLy8PGbPns2zzz7Lli1byMnJ4b777kvwDUan2sEK7j6osufMbIeZneHu28Kg2VnJfq2Bl4Ep7v5WFed6nHCuo6ysLA0vEZGGofxMCbEBCenp1V5umzx5Mi1btuSxxx5j9uzZxz2fk5PDa6+9RklJCddccw2vv/46q1evZsCAAQC8+OKLnHLKKaxdu5Yf/ehHLFq06Ohrb7nlFu69914KCwuT+GbrX6I/aC0AxgDTwse55XcwszTgr8Cf3P2FBM8nIlK/4mdKgLKj4HJzq7zL6po1a9i7dy9vvvnm0dZMeQMHDmT27Nns2rWLvLw8Zs2axdq1a5k5cyYHDhwgNzeX9evXk5qaypYtW+rqXUYq0T6iacBgM9tAMMPrNAAzyzKzGeE+NwGXA2PNbGX41yPB84qI1I8a3EuoMhdeeCGTJ0/mpptuIj09/ei0PCtXruTw4cNAMFfcsmXL+PLLL+nZsydr1qxh9+7ddOzYkXnz5pGamsobb7zB7373OyqagKBFixaUllbY29FoJNQicvfdwMAKthcB48Pl2cDx7VERkcYiFkaxVhHU+PdAI0aMIC0tjXvuuYdWrVrRv39/+vfvT4sWwT+/LVq0oGPHjvTs2ROAjh07cu655wJw6aWX8utf/5pBgwbx3e9+t9LjX3fddVxzzTXceeedCb7RaGiKHxGR6sT/SDWmBi2iKGiKHxGRpqb8TAmlpccu02nanqTQ7NsiIlWpbKYE0EwJSaJLcyIiNVF+dFwVo+WipEtzIiJNVR3OlNDcKYhERCRSCiIREYmUgkhERCKlIBIRkUgpiEREJFIKIhERiZSCSEREIqUgEhGRSCmIREQkUgoiERGJlIJIREQipSASEZFIKYhERCRSCiIREYmUgkhERCKlIBIRkUgpiEREJFIKIhERiZSCSEREIqUgEhGRSCUURGZ2qpnNN7MN4WPbKvZtbWYlZvZoIucUEZGmJdEW0d3AQnc/F1gYrlfmF8DrCZ5PRESamESDaBjwdLj8NPC9inYys95AB+AfCZ5PRESamESDqIO7bwuXtxOETRlmlgI8APwswXOJiEgT1KK6HcxsAdCxgqemxK+4u5uZV7DfROAVdy8xs+rONQGYAJCRkVFdaSIi0gRUG0TuPqiy58xsh5md4e7bzOwMYGcFu10KXGZmE4FvAGlmtt/dj+tPcvfHgccBsrKyKgo1ERFpYqoNomoUAGOAaeHj3PI7uPvo2LKZjQWyKgohERFpnhLtI5oGDDazDcCgcB0zyzKzGYkWJyIiTZ+5N8wrYFlZWV5UVBR1GSIijYqZrXD3rKjrqA3NrCAiIpFSEImISKQURCIiEikFkYiIREpBJCIikVIQiYhIpBREIiISKQWRiIhESkEkIiKRUhCJiEikFEQiIhIpBZGIiERKQSQiIpFSEImISKQURCIiEikFkYiIREpBJCIikVIQiYhIpBREIiISKQWRiIhESkEkIiKRUhCJiEikFEQiIhIpBZGIiERKQSQiIpFKKIjM7FQzm29mG8LHtpXsl2Fm/zCztWb2vpllJnJeERFpOhJtEd0NLHT3c4GF4XpF/gTc7+4XAH2AnQmeV0REmohEg2gY8HS4/DTwvfI7mFl3oIW7zwdw9/3u/kWC5xURkSYi0SDq4O7bwuXtQIcK9jkP+NTMXjSzd8zsfjNLTfC8IiLSRLSobgczWwB0rOCpKfEr7u5m5pWc4zKgJ7AZ+AswFniygnNNACYAZGRkVFeaiIg0AdUGkbsPquw5M9thZme4+zYzO4OK+35KgJXuvil8zd+AvlQQRO7+OPA4QFZWVkWhJiIiTUyil+YKgDHh8hhgbgX7LAfSzax9uD4AeD/B84qISBORaBBNAwab2QZgULiOmWWZ2QwAdz8C/AxYaGarAAOeSPC8IiLSRFR7aa4q7r4bGFjB9iJgfNz6fOBbiZxLRESaJs2sICIikVIQiYhIpBREIiISKQWRiIhESkEkIiKRUhCJiEikFEQiIhIpBZGIiERKQSQiIpFSEImISKQURCIiEqlmH0Tbt2/nrrvuiroMEZFmq9kHUceOHXnggQfKbDty5EhE1YiIND8Jzb7dILmDWeXr5RQXFzN+/Hiys7MpLi5mz5493HzzzaxcuZK3336bffv2cccddzBhwgT279/PyJEjOXjwIOeddx7vv/8+hYWFdf+eRESasKYVRHl58OmnMH16ED7uMGkSpKcHz1Xj5JNPpqCgAIChQ4dyyimncPDgQS6++GLGjRvHE088QXZ2NpMnT+aZZ57h/fd1fz8RkUQ1nUtz7kEI5ecH4RMLofz8YLtXf+fxfv36HV1+7LHHyM7O5sorr2Tnzp3s3LmTDz74gD59+gDwne98p87eiohIc9J0WkRmQUsIgvDJzw+Wc3OPtZCqkZqaCsDevXuZOXMm7733HocOHaJbt264O+eeey5FRUUMHDiQ5cuX19U7ERFpVppOiwjKhlFMDUMoXnp6Ot27dyc7O5uJEydy2mmnAXD77bdTWFjIoEGDWLZsWbKqFhFp1ppOiwiOXY6LN2lSlWGUmZnJggULymwzM+bMmVPh/q+++ioAJSUl3HrrrYnXLCLSzDWdFlF8n1BuLpSWBo/xfUYiItLgNJ0WkVkwOi6+Tyh2mS49vdaX56rTuXNnDd0WEUkC8wbaUsjKyvKioqLav7CWvyMSEWlKzGyFu2dFXUdtNJ1LczHlQ0chJCLSoDW9IBIRkUZFQSQiIpFSEImISKQURCIiEqkGO2rOzD4BPoq4jHbArohrSERjrr8x1w6qP2rNuf6z3b19Moupaw02iBoCMytqbMMg4zXm+htz7aD6o6b6GxddmhMRkUgpiEREJFIKoqo9HnUBCWrM9Tfm2kH1R031NyLqIxIRkUipRSQiIpFSEImISKSaZRCZ2dVmtt7MNprZ3RU8P93MVoZ/H5jZp3HPjTGzDeHfmPqt/GgNidR/JO65gvqt/GgN1dWfYWavmdk7ZvaemV0b99zk8HXrzeyq+q38aA0nVL+ZZZrZgbjP//f1X32N6j/bzBaGtReaWee45xrD97+q+iP9/pvZH81sp5mtruR5M7OHw/f2npn1insu8s++zrh7s/oDUoEPgS5AGvAu0L2K/X8M/DFcPhXYFD62DZfbNpb6w/X9Df3zJ+io/fdwuTtQHLf8LnAycE54nNRGVH8msLoRfP5zgDHh8gBgVrjcKL7/ldUfrkf9/b8c6FXZ9wC4FngVMKAvsKyhfPZ1+dccW0R9gI3uvsndvwKeA4ZVsf/NwJ/D5auA+e6+x933AvOBq+u02uMlUn9DUJP6HWgdLrcBtobLw4Dn3P2gu/83sDE8Xn1KpP6GoCb1dwcWhcuvxT3fWL7/ldUfOXd/HdhTxS7DgD954C0g3czOoGF89nWmOQZRJ+DjuPWScNtxzOxsgv/zjn2pa/zaOpRI/QAtzazIzN4ys+/VXZmVqkn9ecCtZlYCvELQqqvpa+taIvUDnBNesltsZpfVaaUVq0n97wLDw+UbgFZmdloNX1vXEqkfov/+V6ey99cQPvs60xyDqDZGAS+4+5GoCzlBFdV/tgdTh9wCPGRm34ymtCrdDDzl7u0Nv8QAAAHeSURBVJ0JLlXMMrPG9F2trP5tQIa79wR+CjxrZq2rOE5Ufgb0N7N3gP7AFqAx/TdQVf2N4fvf7DSm/7iTZQtwVtx653BbRUZR9rJWbV5bVxKpH3ffEj5uAgqBnskvsUo1qf824HkAd38TaEkwCWRj+fwrrD+8pLg73L6CoK/jvDqvuKxq63f3re4+PAzMKeG2T2vy2nqQSP0N4ftfncreX0P47OtO1J1U9f0HtCDo6DuHY52dF1aw3/lAMeGPfv1Yh+F/E3QWtg2XT21E9bcFTg6X2wEbqGKgQ1T1E3TWjg2XLyDoYzHgQsoOVthE/Q9WSKT+9rF6CTrbtzTE70/43UgJl+8DpobLjeL7X0X9kX//w3NnUvlghesoO1jh7Yby2dfpZxJ1AZG86eByyQcE/0c6Jdw2FRgat08eMK2C1/6AoJN8IzCuMdUP9ANWhf/xrgJua4j1E3Q2/zOscyVwZdxrp4SvWw9c05jqB/4NWBNu+xcwpIHWPyL8R/oDYEbsH+/wuQb//a+s/obw/Se4QrENOETQz3MbcAdwR/i8Ab8N39sqIKshffZ19acpfkREJFLNsY9IREQaEAWRiIhESkEkIiKRUhCJiEikFEQiIhIpBZGIiERKQSQiIpH6/+8HPxekVjHfAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sUDf2lqof_e-"
      },
      "source": [
        "We can see a clear cluster of words that includes `output`, `petroleum`, `energy`, `bdp`, and `kuwait`, if we expand this cluster a little bit we can even include `oil`, `ecuador`, and `barrels`. Oddly, `iraq` appears way too far from the main cluster. This makes me think this count-based method is not accurate enough."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oxQfQ88uVixw"
      },
      "source": [
        "## 2.2. Part 2: Prediction-Based Word Vectors"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "anLAAsrpWBBD"
      },
      "source": [
        "### Downloading and loading pre-trained GloVe word vectors\n",
        "Loaded vocab size 400K 🤩 stored in `wv_from_bin` variable"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CiUwLDETVmKu"
      },
      "source": [
        "def load_embedding_model():\n",
        "    \"\"\" Load GloVe Vectors\n",
        "        Return:\n",
        "            wv_from_bin: All 400000 embeddings, each lengh 200\n",
        "    \"\"\"\n",
        "    import gensim.downloader as api\n",
        "    wv_from_bin = api.load(\"glove-wiki-gigaword-200\")\n",
        "    print(\"Loaded vocab size %i\" % len(wv_from_bin.vocab.keys()))\n",
        "    return wv_from_bin"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6jzmmhOUVqap",
        "outputId": "af574642-e45d-4b67-b7b5-fee0bfc3095e"
      },
      "source": [
        "# -----------------------------------\n",
        "# Run Cell to Load Word Vectors\n",
        "# Note: This will take a couple minutes\n",
        "# -----------------------------------\n",
        "wv_from_bin = load_embedding_model()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[==================================================] 100.0% 252.1/252.1MB downloaded\n",
            "Loaded vocab size 400000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gaOg6H8LWMX8"
      },
      "source": [
        "### Reducing dimensionality of GloVe Word Embeddings"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "05RCFRQ-WQZW"
      },
      "source": [
        "def get_matrix_of_vectors(wv_from_bin, required_words=['barrels', 'bpd', 'ecuador', 'energy', 'industry', 'kuwait', 'oil', 'output', 'petroleum', 'iraq']):\n",
        "    \"\"\" Put the GloVe vectors into a matrix M.\n",
        "        Param:\n",
        "            wv_from_bin: KeyedVectors object; the 400000 GloVe vectors loaded from file\n",
        "        Return:\n",
        "            M: numpy matrix shape (num words, 200) containing the vectors\n",
        "            word2ind: dictionary mapping each word to its row number in M\n",
        "    \"\"\"\n",
        "    import random\n",
        "    words = list(wv_from_bin.vocab.keys())\n",
        "    print(\"Shuffling words ...\")\n",
        "    random.seed(224)\n",
        "    random.shuffle(words)\n",
        "    words = words[:10000]\n",
        "    print(\"Putting %i words into word2ind and matrix M...\" % len(words))\n",
        "    word2ind = {}\n",
        "    M = []\n",
        "    curInd = 0\n",
        "    for w in words:\n",
        "        try:\n",
        "            M.append(wv_from_bin.word_vec(w))\n",
        "            word2ind[w] = curInd\n",
        "            curInd += 1\n",
        "        except KeyError:\n",
        "            continue\n",
        "    for w in required_words:\n",
        "        if w in words:\n",
        "            continue\n",
        "        try:\n",
        "            M.append(wv_from_bin.word_vec(w))\n",
        "            word2ind[w] = curInd\n",
        "            curInd += 1\n",
        "        except KeyError:\n",
        "            continue\n",
        "    M = np.stack(M)\n",
        "    print(\"Done.\")\n",
        "    return M, word2ind"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tnv0ktWHrZSX"
      },
      "source": [
        "**Note**: To avoid running out of memory, we will work with a sample of 10K:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oo6qReH3Wvdx",
        "outputId": "2de6f278-c40b-484c-8886-a25da4eaa5bd"
      },
      "source": [
        "# -----------------------------------------------------------------\n",
        "# Run Cell to Reduce 200-Dimensional Word Embeddings to k Dimensions\n",
        "# Note: This should be quick to run\n",
        "# -----------------------------------------------------------------\n",
        "M, word2ind = get_matrix_of_vectors(wv_from_bin)\n",
        "M_reduced = reduce_to_k_dim(M, k=2)\n",
        "\n",
        "# Rescale (normalize) the rows to make them each of unit-length\n",
        "M_lengths = np.linalg.norm(M_reduced, axis=1)\n",
        "M_reduced_normalized = M_reduced / M_lengths[:, np.newaxis] # broadcasting"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Shuffling words ...\n",
            "Putting 10000 words into word2ind and matrix M...\n",
            "Done.\n",
            "Running Truncated SVD over 10010 words...\n",
            "Done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q9rKgB25W5mh"
      },
      "source": [
        "### GloVe plot analysis for Reuter's \"crude\" corpus"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "id": "lTq1BIsMW7sh",
        "outputId": "b29cb788-c535-450a-a96c-8fa45dc8f1b6"
      },
      "source": [
        "words = ['barrels', 'bpd', 'ecuador', 'energy', 'industry', 'kuwait', 'oil', 'output', 'petroleum', 'iraq']\n",
        "plot_embeddings(M_reduced_normalized, word2ind, words)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAD4CAYAAAD//dEpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXiU5b3/8feXsLkAYVPUGBCRRalFjGwnSgS0WFsiKiB1gZRFa6s51FOFH4opLrWnl6ahemiBFmzRKtoqEW0VgdB6RCVc5ogiOxHDrixKrYGQ7++PmcRJmJCBmUxC8nldV648y/0893fuTOY793M/i7k7IiIi1WlU2wGIiMjJQQlDREQiooQhIiIRUcIQEZGIKGGIiEhEGtd2AFVp166dd+rUqbbDEBE5qaxateozd29fE/uuswmjU6dO5Ofn13YYIiInFTP7pKb2rUNSIiISESUMERGJiBKGiIhERAkjTo4cOVLbIYiIRKXODnrXCncwqzA/5f/9P95++20OHTrE1KlTyc/PZ8OGDXz55Zds3bqV5557ju7du7N8+XKmTZuGmdG9e3dmzpzJJ598wogRI+jevTtNmjThnnvuISMjg/bt29O2bVs6d+7MqFGjmDx5MgsXLgRg3LhxjB07lssvv7yWGkFEJDz1MMpkZcGkSYGkAeDO34cPZ9/ixSxfvpwlS5YwdepU3J327duTm5vLvffey5w5c3B3/vM//5Pc3Fzy8vI45ZRTePXVVwEoLCzkqaee4g9/+ANTpkxhxowZvPrqqzRr1gyAHj168OWXX7Jz504OHjzIBx98oGQhInWSehgQSBL790NOTmA+OxsmTWL1woUsb92atLQ0AIqLi/n888/p27cvAMnJySxevJjPPvuMwsJC0tPTATh48CDdunWjZ8+e9OzZk5YtWwKwceNGLrvsMgD69u1LUVERABkZGcybN48zzjiDUaNGxfGFi4hETgkDAoehsrMD0zk55YnjomHDuLpjR3JmzADg0KFDPProo1jIYSt3p127dnTu3JlFixZx+umnA3D48GG2bdtGQkJCednzzz+f/Px8+vbty8qVKznrrLMAGDFiBAMHDuTUU09lwYIF8XjFIiLHTQmjTFnSKOtlAN99+WXefuAB0tLSMDOSkpI4//zzw2xqPPHEEwwbNgx3p1GjRmRnZ5f3LMo8+uij/PCHP6Rdu3a0atWKjh07AtC8eXP69evH9u3bad++Ri7QFBGJmhJGGffAGEaoSZN4ODu74kB4iNTUVFJTUwEYOHAgS5cuParMm2++WT7dvXt3Vq5cCcCECRPo2rVr+TozY+LEidG+ChGRGqNBb/gmWeTkQGYmlJYGfufkVBwIj9Lq1au5/PLL6d+/PwcPHuS6664DYMyYMWzdupWrrroqJvWIiNQE9TAg0INITAwkibIeRdmYRmJilT2M49W7d2/++c9/HrX86aefjsn+RURqktXVZ3qnpKR43G8+GOY6jFglCxGReDCzVe6eUhP7jskhKTMbambrzGyjmU0+RrkbzMzNrEZeTNQqJwclCxGRclEnDDNLAJ4CrgEuBEab2YVhyrUAMoF3o61TRETiLxY9jD7ARnff7O6HgOeA9DDlHgJ+CXwdgzpFRCTOYpEwzgE+DZkvCi4rZ2a9gXPd/dVj7cjMJppZvpnl79mzJwahiYhIrNT4abVm1gh4ArinurLuPsvdU9w9RRewiYjULbFIGNuAc0Pmk4LLyrQAegJ5ZlYI9ANy6+zAt4iIhBWLhLESuMDMzjOzpsBNQG7ZSnc/4O7t3L2Tu3cC3gGGubse2C0ichKJOmG4ewnwE+B14GNggbt/ZGbTzWxYtPsXEZG6ISZXerv7a8BrlZZNq6JsWizqFBGR+NK9pCIwb948vvjii4jLFxYWMmTIkBqMSEQk/pQwIlBVwtBzukWkIWk4CaPSPbMKt2yhd+/ejBo1ipSUFHJycjhw4AAjR45k8ODBDBo0iI0bN7J06VIKCgoYMWIEd911F4WFhVx22WXceuutTJgwgXfeeYcBAwaQmprKj370Iyrfm+vTTz/l2muvZdCgQVx77bWUXV/SpUuX8jJDhgyhsLCQwsJCLr30Um655RZ69uzJ7NmzGTNmDL179+YXv/hFzbeRiMixuHud/Ln00ks9Zh580D0z0720NDBfWupbMjK83amn+hdffOGHDh3yb3/7256RkeF//vOf3d29oKDAb7jhBnd3HzhwoH/66afu7r5lyxZv166dHzhwwN3dL730Ut+0aZO7u2dkZPjChQt9y5YtPnjwYHd3HzVqlK9YscLd3V9++WW/55573N39/PPPLw9v8ODBvmXLFt+yZYufffbZ/u9//9t37NjhTZs29R07dvjhw4e9c+fOsWsPEam3gHyvoc/l+n978yqe183cuXQ/+2xanH46mNGzZ0927NhBTk4Ov/3tbwFo3Dh884Q+p/vAgQN07twZgAEDBrB27Vouvvji8rKrV69m8uTA/RhLSkoq9Cy+CfGbXkn37t1p3rw5HTp0ICkpiQ4dOgBwyimncOTIkQqPfBURiaf6nzCqeF43GRmsfeUVDv7rXzRv3pwPP/yQ3r17M3HiRIYPHw4EnuEN0LRpU0pKSsp3Gfqh3apVKzZv3kznzp15++23SU+veButiy66iClTpnDJJZdU2GdpaSnFxcUcOXKEjz/+OCRcCzsNFROLiEi8NYwxjNCkUeaBB+jUqRMTJkygX79+jBkzhuzsbBYsWMCgQYO48sormTFjBgDXX38948aN44EHHjhq1zNmzODmm28mNTWVJk2aMGxYxUtPHn/8cR588EEGDRrEoEGDWLBgAQA/+clP6NevHz/+8Y9JSkqqmdctIhJDDeMBSqGPYA0qzMhg/NatFZ65LSJysqvzD1Cq06p6XvfcubB+fcye1y0iUt81jDGMMM/r7gS8GcPndYuI1HcN45AU6HndItIg6JBULOh53SIiUWk4CUNERKKihCEiIhFpsAnjscceY/Xq1QBhr74WEZGK6v9ZUlUou12HiIhEpn72MCqd+eWlpdx+++2kpqYyYMAA3nvvPcaOHctbb71VSwGKiJx86l8PIysrcLPBsmsu3FmYns7hoiLeev99Nm/ezE033cSFF15Y25GKiJxUYtLDMLOhZrbOzDaa2VHHeszsp2a2xsw+MLMlZtYxFvUeJfTOtJMmlV/lvW7RIga0bQvudO7cmX379tVI9SIi9VnUPQwzSwCeAq4CioCVZpbr7mtCir0PpLj7V2b2I+C/gVHR1h0mmLB3pu32ve+R264d483YvHkziYmJMa9aRKS+i0UPow+w0d03u/sh4Dmgwj2+3X2Zu38VnH0HqLnbs4a5M+2wl18moXFjUlNTufnmm/nNb35TY9WLiNRXsRjDOAf4NGS+COh7jPLjgL+FW2FmE4GJAMnJyScWTdnNBkM0uuceZs+aVeHq7n79+pVPb9y48cTqEhFpQOJ6lpSZ3QKkAL8Kt97dZ7l7iruntG/f/vgrqOrOtKFjGiIickJi0cPYBpwbMp8UXFaBmQ0BpgID3b04BvUerYo70wKB5bp/lIjICYv6brVm1hhYDwwmkChWAj9w949CylwCvAgMdfcNkew3qrvV6s60ItJA1em71bp7CfAT4HXgY2CBu39kZtPNrOx5pb8CTgdeMLMCM8uNtt5j0p1pRURiLiYX7rn7a8BrlZZNC5keEot6RESk9tTPW4OIiEjMKWGIiEhElDBERCQiShgiIhIRJQwREYmIEoaIiERECUNERCKihCEiIhFRwhARkYgoYYiISESUMEREJCJKGCIiEhElDBERiYgShoiIREQJQ0REIqKEISIiEVHCEBGRiChhiIhIRGKSMMxsqJmtM7ONZjY5zPpmZvZ8cP27ZtYpFvWKiEj8RJ0wzCwBeAq4BrgQGG1mF1YqNg7Y5+5dgGzgl9HWKyIi8RWLHkYfYKO7b3b3Q8BzQHqlMunA08HpF4HBZmYxqFtEROIkFgnjHODTkPmi4LKwZdy9BDgAtI1B3SIiEid1atDbzCaaWb6Z5e/Zs6e2wxERkRCxSBjbgHND5pOCy8KWMbPGQCvg88o7cvdZ7p7i7int27ePQWgiIhIrsUgYK4ELzOw8M2sK3ATkViqTC4wJTt8ILHV3j0HdIiISJ42j3YG7l5jZT4DXgQTgD+7+kZlNB/LdPRf4PfAnM9sI7CWQVERE5CQSdcIAcPfXgNcqLZsWMv01MCIWdYmISO2oU4PeIiLyjcLCQoYMGVIj+zazsWZ2VXD67ki2UcIQEWmA3H2euy8OziphiIjUF08++SQ9evQgIyOjfFmXLl0AGDx4MHv37mX16tUAvc2shZldZmazAczsdTPLM7P3zKx/cFmWmd1iZj8Azgmun3qsGJQwRETqisonjwbnp0yZwueff87MmTNJSEg4arO0tDSWLVvG0qVLIXBh9BXAIGBpsMj17p5G4GzVRypW4c8C29w9zd0rrKssJoPeIiISpaws2L8fsrPBLJAsHnqIj959l3379rFixQpWrFgRdtPBgwczf/58PvvsM4DtwGCgB5BhZqcAOWbWDTjC0XfiiJh6GCIitc09kCxycmDSpMD8pEkwdy4XtW7NlMmTGTlyJImJiRQVFQFQUFBASUkJAH369OHdd9/l66+/Bvg3cBHQ1t13AkOBI+5+OXAnEO4+fiVmVm0+UA9DRKS2mQV6FhBIGjk5gemMDNi6lRtHjKBps2bcf//9tGjRgoEDBzJw4EAaNw58hDdu3JgOHTpwySWX8MorrwDsBDYE974CmGJmbwL/W0UELwKvmtnf3H1GlWHW1QuuU1JSPD8/v7bDEBGJH3doFPJFv7Q0kEyOg5mtcveUGEcGNIBDUjt37uSee+6p7TBERI6t7DBUqLLDU3VEvU8YHTp04PHHH6+w7MiRI7UUjYhIGGXJIicHMjMDPYvMzIpjGnVAvR/DKCwsZPz48aSmplJYWMjevXsZPXo0BQUFvPfeexw4cIA77riDiRMncvDgQUaNGkVxcTFdu3ZlzZo15OXl1fZLEJH6zgwSEwNJouwsqbIxjcTE4z4sVVPqX8Jwr9i4IZm5WbNm5OYGbqQ7bNgwTjvtNIqLi/nWt75FRkYGs2fPJjU1lSlTpvDMM8+wZs2aeEcvIg1VVlbFz6+ypFFHkgXUt0NSWVkVu2/B85jZvBmAAQMGlBedOXMmqampXH311ezevZvdu3ezfv16+vTpA0Dfvn3jHb2INHSVk0MdShZQnxLGMc5jpqQE3MuvkNy3bx9z585l+fLlvP7667Rq1Qp354ILLqDszKyVK1fW5qsREalz6s8hqWrOYw7N1ImJiVx44YWkpqbSo0cP2rYNPF58woQJjBw5ksWLF9OzZ894vwIRkTqt/l2HEYPzmAGKioq45ZZbNOgtIicVXYcRqZPgPGYRkZNV/UkYMT6POSkpSb0LEZEQ9WsM4yQ4j1lE5GQV1RiGmbUBngc6AYXASHffV6lML2Am0JLArXUfcffnq9t3VGMYla/DULIQkQaiLo9hTAaWuPsFwJLgfGVfAbe5+0UEbrP7azNLjLLeqtXx85hFRE5W0SaMdODp4PTTwHWVC7j7enffEJzeDuwG2kdZr4iIxFm0CeNMd98RnN4JnHmswmbWB2gKbKpi/UQzyzez/D179kQZmoiIxFK1g97Bh250CLOqwsPC3d3NrMoBETM7C/gTMMbdS8OVcfdZwCwIjGFUF5uIiMRPtQnD3YdUtc7MdpnZWe6+I5gQdldRriXwKjDV3d854WhFRKTWRHtIKhcYE5weAyysXMDMmgIvAX909xejrE9ERGpJtAnjMeAqM9sADAnOY2YpZjYnWGYkcAUw1swKgj+9oqxXRETirP7dS0pEpAGry9dhiIhIA6GEISIiEVHCEBGRiChhiIhIRJQwREQkIkoYIiISESUMERGJiBKGiIhERAlDREQiooQhIiIRUcIACgsLGTKkypvy1tl9i4jEkxLGCSotrfhIjyNHjtRSJCIi8VHt8zDqHfeKz/kO3nxx7969jBo1ik2bNnHrrbdy8cUXM336dEpKSmjTpg3PP/88zZs3p0uXLowcOZIVK1bws5/9jJycHFq2bMn555/PNddcw7Rp0zAzunfvzsyZMytUnZ2dzXPPPcepp57KddddR2ZmZjxfuYhIVBpWwsjKgv37ITs7kDTcYdIkAD799FOWL19O8+bNueyyy1i4cCHLli0D4L777mPBggXcdtttlJSU8P3vf59HH32UvLw8tm/fzqJFi2jcuDG9e/cmLy+PVq1aMWnSJF599VV69uxZXv0zzzzDsmXLaNGixVE9FBGRuq7hJAz3QLLIyQnMZ2cHkkVODmRk0L17d1q0aAFAz5492blzJxMmTKC4uJhdu3bRsmVLABISEujXr1/5blNSUmjSpAl79uyhsLCQ9PR0AA4ePEi3bt0qJIxf//rX3H333Rw+fJg77riD1NTUOL14EZHoNZyEYRZIEhBIEmWJIzMTMjNZ26cPBw8epHnz5nz44YdkZWXx85//nP79+3PvvfdS9twQM8NCDmklJCQA0K5dOzp37syiRYs4/fTTATh8+DDbtm0rL9u7d29SU1MpKioiPT2dVatWxeGFi4jERsNJGPBN0ihLFhCY/+QTOnXqxIQJE1izZg2tW7fmtttuY9y4cXTr1o1WrVqV9zBC/epXvwrZtfHEE08wbNgw3J1GjRqRnZ1dYbtbb72VtWvX0qRJE+6+++4afakiIrHWsJ64VzZmEZowMjO/GdM4TuPHj+eWW24hLS0t4m3Gjh3L+PHjjzocdeTIkfLeiojIiaqzT9wzszZmttjMNgR/tz5G2ZZmVmRmT0ZT5wkLTRaZmVBaCnffHZifNCmw3r38uomsrCxuvvlmhg0bRq9evVi7di0AL7zwAr169WL48OFs2rQJOPpaiy5dugCQl5dHnz59uPLKK8nIyGDNmjX8/e9/56677mLEiBEAdOzYkTvvvJP09HRGjRrF+++/D8Ann3zCVVddFc8WEhE5pmgPSU0Glrj7Y2Y2OTh/XxVlHwL+EWV9J84MEhO/6VH8/OeB5XffHVgO5WdMlWnfvj3PPPMMzz77LHPmzOGXv/wlU6dOZdWqVTRv3pxvf/vbx6zyr3/9Kw8//DBXX301paWlNGrUiKFDh1boYezYsYPJkyeTnJzMkiVL+P3vf8+TTz7J3LlzGTduXMybQUTkREV74V468HRw+mngunCFzOxS4EzgjSjri05W1jcD3/v3w4wZgekHH/ym9/HFF+XFL730UgCSk5P5/PPP+eyzzzjzzDNp0aIFTZo0oXfv3gAVBsGB8gHyn/3sZ+Tm5nLzzTczd+7csCGdc845JCcnAzBo0CDeffddvvrqK1555RWGDx8eq1cuIhK1aHsYZ7r7juD0TgJJoQIzawQ8DtwCHPMeGWY2EZgIlH+IxlzZh3voGVNliSN4xhQTJpTFU76Zu9OuXTt27dpVfjZVQUEBAK1bt2b79u24O7t27So/M6pt27Y8+eSTuDtdu3ZlxIgRNG3alJKSkvL9ho5bmBk33ngjd955J1dccQXNmjWrmTYQETkB1SYMM3sT6BBm1dTQGXd3Mws3gn4n8Jq7F1X+Jl6Zu88CZkFg0Lu62KJyjDOmqpKQkMD06dNJTU3lvPPO45xzzgGgZcuWDB06lP79+9OnTx/OPDOQN5944gneeOMNSktLueqqq2jZsiXf+973mDZtGj169OB3v/vdUXVkZGSQlJRUPpYhIlJXRHWWlJmtA9LcfYeZnQXkuXu3SmWeAS4HSoHTgabA/7j75GPtu0bOkgoV4zOmYmXXrl2MHj2apUuX1loMInLyqrNnSQG5wJjg9BhgYeUC7n6zuye7eyfgv4A/Vpcsaly4M6YyMyueMVULFi9ezLBhw7j//vtrpX4RkWOJNmE8BlxlZhsIjE88BmBmKWY2J9rgakzlM6bKDk9lZgaWn2APY0bZWMgJmDdvHn379uXdd99l0KBBJ7wfEZGa0rAu3Kss3J1rozgc1aVLFzZu3HhC26alpTF//nySkpJOuH4Rkbp8SOrkVjk5lN3BNoSXlnL77beTmprKgAEDeO+990hLS6OoqAiAhx9+mHnz5vHss8+ybds20tLSeOSRR8jLy+PKK69k+PDh9OrVixdeeAEIXOn91ltvATB//nyysrJYunQpBQUFjBgxgrvuuqvmX7eIyAloWPeSqk6Y258vTE/ncFERb73/Pps3b+amm27i1FNPPWrTH/zgB0ybNo28vDwgcJX3nj17WLx4MV999RUpKSnccMMNYasdNGgQvXr1Ug9DROq0ht3DCBV6+/Oyge9Jk1i3aBED2rYFdzp37sy+ffuOuj6jKpdccgmNGzemZcuWnHHGGezZsyfibUVE6holjDKhA985OdCoEeTk0O173+Ptc88FMzZv3kxiYiJt2rQpPyQVeovyxo0bV3gwUkFBASUlJXz55Zfs2rWL9u3bV7lt5Qv6RETqGh2SChXmYr5hL7/Mq8GHHR05coTf/OY3FBcXM378eLp27Vrhauwbb7yRa6+9lmuuuYaLL76Ys88+mxEjRrBlyxYefvhhGjVqxPjx4xk9ejTPPvss7dq1IzF4H6vrr7+ecePGMWDAAB566KG4v3QRkeo07LOkKovhxXx5eXnMnz+fOXPq7tnFIlL/6CypeKijF/OJiNQVOiRVpqqL+eCELuZLS0s7rgcriYjUdTokVVmML+YTEYknHZKKp3AX84mIiBKGiIhERglDREQiooQhIiIRUcIQEZGIKGGIiEhElDBERCQiShgiIhIRJQwREYlIVAnDzNqY2WIz2xD83bqKcslm9oaZfWxma8ysUzT1iohI/EXbw5gMLHH3C4Alwflw/gj8yt17AH2A3VHWKyIicRZtwkgHng5OPw1cV7mAmV0INHb3xQDuftDdv4qyXhERibNoE8aZ7r4jOL0TODNMma7AfjP7q5m9b2a/MrOEcDszs4lmlm9m+Xv27IkyNBERiaVqb29uZm8CHcKsmho64+5uZuFufdsYuBy4BNgKPA+MBX5fuaC7zwJmQeButdXFJiIi8VNtwnD3IVWtM7NdZnaWu+8ws7MIPzZRBBS4++bgNi8D/QiTMEREpO6K9pBULjAmOD0GWBimzEog0czaB+cHAWuirFdEROIs2oTxGHCVmW0AhgTnMbMUM5sD4O5HgP8ClpjZasCA2VHWKyIicRbVI1rd/XNgcJjl+cD4kPnFwMXR1CUiIrVLV3qLiEhElDBERCQiShgiIhIRJQwREYmIEoaIiERECUNERCKihCEiIhFRwhARkYgoYYiISESUMEREJCJKGCIiEhElDBERiYgShoiIREQJQ0REIqKEISIiEVHCEBGRiChhiIhIRJQwREQkIlElDDNrY2aLzWxD8HfrKsr9t5l9ZGYfm9kMM7No6hURkfiLtocxGVji7hcAS4LzFZjZAOA/CDzTuydwGTAwynqljikqKiItLa22wxCRGhRtwkgHng5OPw1cF6aMA82BpkAzoAmwK8p65SR25MiR2g5BRE5AtAnjTHffEZzeCZxZuYC7rwCWATuCP6+7+8fhdmZmE80s38zy9+zZE2VoUtmUKVMYOHAg/fv3Z9GiRWzdupWhQ4cycOBAhgwZQmlpKWPHjuWtt94CYP78+WRlZQFw3333ceWVV9K7d29mzZoFwMGDB7n22msZMmQIjz76aHk969evJy0tjYEDBzJq1Cj+/e9/A9CxY0fuvPNO0tPT4/vCRSQmGldXwMzeBDqEWTU1dMbd3cw8zPZdgB5AUnDRYjO73N3/Wbmsu88CZgGkpKQctS85Du4QMlT097/9jX379rF8+XK++uor+vfvT9euXZk0aRLf+c53KC0tpVGjqr8/TJs2jdNOO43i4mK+9a1vkZGRwezZs0lNTWXKlCk888wzrFmzBoB7772X6dOnc8UVVzB9+nRmz57N3XffzY4dO5g8eTLJyck1/vJFJPaqTRjuPqSqdWa2y8zOcvcdZnYWsDtMseHAO+5+MLjN34D+wFEJQ2IkKwv274fs7EDScGf1o4+yfO1a0tauBaC4uJg1a9YwaNAggPJkEXo+gvs3OXvmzJm8/PLLJCQksHv3bnbv3s369eu58cYbAejbty+zZ88GAj2MAQMGADBgwAD++te/AnDOOecoWYicxKI9JJULjAlOjwEWhimzFRhoZo3NrAmBAe+wh6QkBtwDySInByZNCsxPmsRFb73F1UlJ5C1bRl5eHh988AEXXXQReXl5AJSWlgLQpk0bioqKAFi1ahUA+/btY+7cuSxfvpzXX3+dVq1a4e5ccMEF5OfnA7By5cryELp27crbb78NwNtvv023bt0ASEhIiEsTiEjNsNBvkce9sVlbYAGQDHwCjHT3vWaWAtzh7uPNLAH4H+AKAgPgf3f3n1a375SUFC/7MJLjFEwS5OR8sywzk/tPO423/vd/MTOSkpJ45JFHmDBhAl9//TVNmjThjTfeYN26dYwePZrk5GTatWtHcnIyDz74ICNHjqSoqIgePXpQUFBAbm4urVq1YuTIkRw+fJiePXtSUFBAXl4ea9eu5fbbb8fdOeOMM/jTn/7EKaecQpcuXdi4cWPttYtIA2Bmq9w9pUb2HU3CqElKGFFyh9AxidLSCmMaIlI/1WTC0JXe9VFZDyNU2eEpEZETpIRR34QejsrMDPQsMjMrjmmIiJyAas+SkpOMGSQmBpJE2VlS2dmBdYmJOiwlIidMYxj1VaXrMI6aF5F6SWMYcvwqJwclCxGJkhKGiIhERAlDREQiooQhIiIRUcIQEZGIKGGIiEhElDBERCQiShgiIhIRJQwREYmIEoaIiERECUNERCKihCFSSwoLCxkypMonIFfp4YcfZt68ebEPSKQaShgiIhIR3d5cJF7C3EF47969jBo1ik2bNnHrrbfSqlUr/vKXvwBQVFTEjBkzuPzyy/nHP/5BZmYmSUlJAOW/ReIpqh6GmY0ws4/MrDT4HO+qyg01s3VmttHMJkdTp8hJKSur4gOs3OGhh/h03TrmzJnDihUrmDt3Lrt37+bw4cO88sorvNiRcn8AAAhqSURBVPTSS0wKPjnxpz/9KQsXLiQ3N5fi4uLaex3SoEV7SOpD4HrgH1UVMLME4CngGuBCYLSZXRhlvSInD3fYv7/iUw8nTYK5c+memEiL00+nSZMm9OzZE3fnsssuA6BTp04cOHAAgC+++ILk5GTMjD59+tTmq5EGLKqE4e4fu/u6aor1ATa6+2Z3PwQ8B6RHU6/ISaXsqYdlj8pt1CjwOyODtYcOcfBf/6KkpIQPP/wQM2PVqlUAbN26lZYtWwLQokULioqKAFi5cmWtvRRp2OIxhnEO8GnIfBHQNw71itQdZUkjJ+ebZQ88QKfVq5kwYQIbNmxgzJgxtG7dmlNPPZVrr72W7du3kx18vO7jjz/O97//fc4++2xatGhRSy9CGrpqE4aZvQl0CLNqqrsvjGUwZjYRmAiQnJwcy12L1K6yw1AhOuXksPK99yoMhM+bN49evXpx//33VyiblpbG+++/H5dQRapSbcJw9+M/UbyibcC5IfNJwWXh6poFzILAM72jrFekbihLFjk5gcNS2dnfzENgXo/QlZNAPA5JrQQuMLPzCCSKm4AfxKFekbrBDBITv0kWZYenILA8JFmMHTu2dmIUiYC5n/gXeTMbDvwGaA/sBwrc/TtmdjYwx92/Gyz3XeDXQALwB3d/pLp9p6SkeH5+/gnHJlLnhLkOQz0LiTUzW+XuVV7mEI2oehju/hLwUpjl24Hvhsy/BrwWTV0iJ73KyUHJQk4yujWIiIhERAlDREQiooQhIiIRUcIQEZGIRHWWVE0ysz3AJyewaTvgsxiHEwt1NS6ou7EpruNXV2NTXMfvRGPr6O7tYx0M1OGEcaLMLL+mTimLRl2NC+pubIrr+NXV2BTX8auLsemQlIiIREQJQ0REIlIfE8as2g6gCnU1Lqi7sSmu41dXY1Ncx6/OxVbvxjBERKRm1McehoiI1AAlDBERichJmTDMbISZfWRmpWZW5WlnZjbUzNaZ2UYzmxyy/Dwzeze4/HkzaxqjuNqY2WIz2xD83TpMmSvNrCDk52szuy64bp6ZbQlZ1ysWcUUaW7DckZD6c0OW12ab9TKzFcG/+QdmNipkXUzbrKr3TMj6ZsHXvzHYHp1C1k0JLl9nZt+JJo4TiOunZrYm2D5LzKxjyLqwf9M4xjbWzPaExDA+ZN2Y4N9+g5mNiXNc2SExrTez/SHraqzNzOwPZrbbzD6sYr2Z2Yxg3B+YWe+QdTXWXhFx95PuB+gBdAPygJQqyiQAm4DOQFPg/4ALg+sWADcFp38L/ChGcf03MDk4PRn4ZTXl2wB7gVOD8/OAG2uozSKKDThYxfJaazOgK3BBcPpsYAeQGOs2O9Z7JqTMncBvg9M3Ac8Hpy8Mlm8GnBfcT0Ic47oy5H30o7K4jvU3jWNsY4Enw2zbBtgc/N06ON06XnFVKn8XgUcvxKPNrgB6Ax9Wsf67wN8AA/oB79Z0e0X6c1L2MNz9Y3dfV02xPsBGd9/s7oeA54B0MzNgEPBisNzTwHUxCi09uL9I93sj8Dd3/ypG9R/L8cZWrrbbzN3Xu/uG4PR2YDeBZ7DEWtj3zDHifREYHGyfdOA5dy929y3AxuD+4hKXuy8LeR+9Q+DJlvEQSZtV5TvAYnff6+77gMXA0FqKazTw5xjVfUzu/g8CXxSrkg780QPeARLN7Cxqtr0iclImjAidA3waMl8UXNYW2O/uJZWWx8KZ7r4jOL0TOLOa8jdx9Jv0kWA3NNvMmsUoruOJrbmZ5ZvZO2WHyqhDbWZmfQh8Y9wUsjhWbVbVeyZsmWB7HCDQPpFsW5NxhRpH4BtqmXB/01iJNLYbgn+jF82s7JHNdaLNgofvzgOWhiyuyTarTlWx12R7RSQej2g9IWb2JtAhzKqp7r4w3vGUOVZcoTPu7mZW5TnLwW8M3wJeD1k8hcCHZlMC52DfB0yPc2wd3X2bmXUGlprZagIfiicsxm32J2CMu5cGF0fVZvWNmd0CpAADQxYf9Td1903h91AjXgH+7O7FZnY7gR7aoDjWX52bgBfd/UjIstpuszqpziYMdx8S5S62AeeGzCcFl31OoIvXOPgNsWx51HGZ2S4zO8vddwQ/3HYfY1cjgZfc/XDIvsu+aReb2VzgvyKNK1axufu24O/NZpYHXAL8hVpuMzNrCbxK4AvDOyH7jqrNKqnqPROuTJGZNQZaEXhPRbJtTcaFmQ0hkIQHuntx2fIq/qax+vCrNjZ3/zxkdg6BcauybdMqbZsXr7hC3AT8OHRBDbdZdaqKvSbbKyL1+ZDUSuACC5zd05TAmyLXA6NHywiMHwCMAWLVY8kN7i+S/R51zDT4gVk2ZnAdEPYsipqKzcxalx3SMbN2wH8Aa2q7zYJ/v5cIHNd9sdK6WLZZ2PfMMeK9EVgabJ9c4CYLnEV1HnAB8F4UsRxXXGZ2CfA7YJi77w5ZHvZvGqO4Io3trJDZYcDHwenXgauDMbYGrqZij7tG4wrG1p3AAPKKkGU13WbVyQVuC54t1Q84EPxiVJPtFZl4jrDH6gcYTuD4XTGwC3g9uPxs4LWQct8F1hP4ZjA1ZHlnAv/MG4EXgGYxiqstsATYALwJtAkuTwHmhJTrRODbQqNK2y8FVhP40JsPnB7DNqs2NmBAsP7/C/4eVxfaDLgFOAwUhPz0qok2C/eeIXCIa1hwunnw9W8MtkfnkG2nBrdbB1wT4/d8dXG9GfxfKGuf3Or+pnGM7RfAR8EYlgHdQ7b9YbAtNwIZ8YwrOJ8FPFZpuxptMwJfFHcE39NFBMac7gDuCK434Klg3KsJORO0Jtsrkh/dGkRERCJSnw9JiYhIDClhiIhIRJQwREQkIkoYIiISESUMERGJiBKGiIhERAlDREQi8v8BU/drHTgJAWQAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W1p24BMtqWao"
      },
      "source": [
        "GloVe seems to produce better results, now it seems we have the main cluster containing most of the words in the sample set. Nevertheless, it is still odd `kuwait` is far apart from words like `oil`. Why is this happening? Perhaps it is because we are using a small sample of 10K GloVe vectors, or it may also be related to truncation when doing the dimensionality reduction. Another possibility is that in the selected 10K GloVe vectors `kuwait` doesn't appear with enough frequency in the context of the other words."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gS61-5bfuY6G"
      },
      "source": [
        "### Quantifying the similarity between individual words vectors\n",
        "We can use the pre-trained GloVe word vectors loaded through gensim before (`glove-wiki-gigaword-200`) to find polysemes. The `most_similar` method is available from the model/dataset returned by `api.load(\"glove-wiki-gigaword-200\")` and uses the concept of cosine similarity to return the top-10 most similar words for a given parameter."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tPn9hInNud-6",
        "outputId": "b5a2eabf-bd3a-41d7-c8dc-158ddda43a9f"
      },
      "source": [
        "# Next is based on the `glove-wiki-gigaword-200` GloVe vectors that contains 400K word embeddings\n",
        "wv_from_bin.most_similar(\"branch\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('branches', 0.7101372480392456),\n",
              " ('central', 0.5476117730140686),\n",
              " ('railway', 0.5329204797744751),\n",
              " ('established', 0.5197478532791138),\n",
              " ('line', 0.5076225399971008),\n",
              " ('authority', 0.491929292678833),\n",
              " ('offices', 0.48285460472106934),\n",
              " ('railroad', 0.4816432297229767),\n",
              " ('headquarters', 0.4756273925304413),\n",
              " ('department', 0.4709719121456146)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Itx1W8FkGjvA"
      },
      "source": [
        "We can find polysemes like `plant` (`factory` / `flowering`) or `break` (`broke` / `time`). Nevertheless, there were no good results for words like `branch` which didn't return tree-related words. Why is that? Probably because of the dataset used during the training process that returned the loaded word vectors. It might happen that the dataset didn't have a varied enough context for these words."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ix1SDwj__Cs4"
      },
      "source": [
        "### What about Synonyms and Antonyms?\n",
        "We can use the concept of cosine distance to confirm the similarity of two words (synonyms), but the cosine distance reveals something odd. There are cases where the opposite word is closer than a synonym word, for example: "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XFLpcgU2CT6q",
        "outputId": "8ffb787e-1f03-4049-c02f-3ba9c17aea57"
      },
      "source": [
        "# wv_from_bin.most_similar(\"top\")\n",
        "\n",
        "# w1 = top w2 = pinnacle/summit/peak/apex, w3 = bottom\n",
        "# w1 and w2 can be considered synonyms\n",
        "# w1 and w3 can be considered antonyms\n",
        "\n",
        "# Distance among synonyms are 0.76/0.67/0.67/0.79\n",
        "# wv_from_bin.distance(\"top\", \"apex\")\n",
        "\n",
        "# Distance among antonyms is 0.49\n",
        "wv_from_bin.distance(\"top\", \"bottom\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.4903566241264343"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 79
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IaqOXczMOAQH"
      },
      "source": [
        "Why is this happening? I guess it is because of the data used to train the model. Probably the training examples contain contexts for `top` related to success, leadership, and businesses which probably are not so rich in vocabulary."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "07lEijj3TqSH"
      },
      "source": [
        "### Finding analogies\n",
        "The `most_similar` method from gensim allow extra parameters that can be used to find analogies. For example, `man:king :: woman:?` can be solved like this:\n",
        "```python\n",
        "result = wv_from_bin.most_similar(positive=['woman', 'king'], negative=['man'])\n",
        "```\n",
        "This will output a list of possible answers with a given similarity value. `Queen` is the top word. \n",
        "As professor Manning showed in class, it is useful to turn the above instruction to a function like this:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        },
        "id": "sef9R3qJVSIc",
        "outputId": "4f6349e7-88d5-4159-c8f4-6031fa918149"
      },
      "source": [
        "def analogy(x1, x2, y1):\n",
        "    result = wv_from_bin.most_similar(positive=[y1, x2], negative=[x1])\n",
        "    return result[0][0]\n",
        "\n",
        "analogy(\"man\", \"king\", \"woman\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'queen'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 84
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hdQBLXSKYY_b"
      },
      "source": [
        "Gensim documentation states that the `positive` parameter contains keys that contribute positively towards the similarity and the opposite for the `negative` parameter.\n",
        "This is interesting because it kind of defines what an analogy is, in this case, we ask the model: give me words that are similar to `king` and `woman` but different than `man`. This is why we get a top-10 list of possible values when running `most_similar`:\n",
        "\n",
        "```python\n",
        "[('queen', 0.6978678703308105),\n",
        " ('princess', 0.6081745028495789),\n",
        " ('monarch', 0.5889754891395569),\n",
        " ('throne', 0.5775108933448792),\n",
        " ('prince', 0.5750998854637146),\n",
        " ('elizabeth', 0.546359658241272),\n",
        " ('daughter', 0.5399125814437866),\n",
        " ('kingdom', 0.5318052768707275),\n",
        " ('mother', 0.5168544054031372),\n",
        " ('crown', 0.5164472460746765)]\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x3PC3OgLZ-Tm"
      },
      "source": [
        "Playing with the `analogy` function we can have results like:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        },
        "id": "3uzuon7haCj_",
        "outputId": "d13759fb-5be3-4eef-e580-50a492031f27"
      },
      "source": [
        "analogy(\"legs\", \"pants\", \"chest\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'jacket'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 86
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        },
        "id": "Ks12g-YPaXk2",
        "outputId": "fe4d87f0-aac6-4002-c3e8-ab08e1cf5757"
      },
      "source": [
        "analogy(\"peru\", \"peruvian\", \"chile\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'chilean'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 90
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fY-urfYca_Hm"
      },
      "source": [
        "Sadly, not all analogies hold according to the pre-trained GloVe vectors. This is the case of `hand:glove :: foot:?` where we get:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        },
        "id": "H9B2ACbrboVv",
        "outputId": "af0c8fc3-1b21-4533-9e0a-c5c9a8f31311"
      },
      "source": [
        "analogy(\"hand\", \"glove\", \"foot\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'45,000-square'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 95
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Sud3mLtcAoC"
      },
      "source": [
        "\n",
        "I wonder if we'll get better results with a Word2Vec pre-trained model instead."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W_OVHLCXn8E6"
      },
      "source": [
        "## 2.3. Bias in Word Vectors"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O-FUyPTJn_Up"
      },
      "source": [
        "Word embeddings are susceptible to bias. This is dangerous because it can reinforce stereotypes through applications that employ these models. For example, in the code snippet below we are asking the model for words similar to \"woman\" and \"worker\" but different to \"man\"."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DglrFII6oV8W",
        "outputId": "5216968d-8e72-4e0f-f482-5e569413b07b"
      },
      "source": [
        "# Run this cell\n",
        "# Here `positive` indicates the list of words to be similar to and `negative` indicates the list of words to be\n",
        "# most dissimilar from.\n",
        "pprint.pprint(wv_from_bin.most_similar(positive=['woman', 'worker'], negative=['man']))\n",
        "print()\n",
        "pprint.pprint(wv_from_bin.most_similar(positive=['man', 'worker'], negative=['woman']))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[('employee', 0.6375863552093506),\n",
            " ('workers', 0.6068919897079468),\n",
            " ('nurse', 0.5837947726249695),\n",
            " ('pregnant', 0.5363885164260864),\n",
            " ('mother', 0.5321309566497803),\n",
            " ('employer', 0.5127025842666626),\n",
            " ('teacher', 0.5099576711654663),\n",
            " ('child', 0.5096741914749146),\n",
            " ('homemaker', 0.5019454956054688),\n",
            " ('nurses', 0.4970572590827942)]\n",
            "\n",
            "[('workers', 0.6113258004188538),\n",
            " ('employee', 0.5983108282089233),\n",
            " ('working', 0.5615328550338745),\n",
            " ('laborer', 0.5442320108413696),\n",
            " ('unemployed', 0.5368517637252808),\n",
            " ('job', 0.5278826951980591),\n",
            " ('work', 0.5223963260650635),\n",
            " ('mechanic', 0.5088937282562256),\n",
            " ('worked', 0.505452036857605),\n",
            " ('factory', 0.4940453767776489)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5XYKLkKJqgRb"
      },
      "source": [
        "We can clearly see gender bias when `nurse` and `teacher` are close to `woman` but so far from `man` that they do not appear in the top 10 list of similar words. Also, we can see that `mechanic` is close to `man` but far from `woman`. Why is this happening? again because of the data used to train the models. Sadly the trained data is a reflection of our society.\n",
        "\n",
        "This is sad, and is present in other examples that reflect how our societies generalize over different topics:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zXy3xGfGsYBg",
        "outputId": "e0578caf-9e4d-4bf9-d18b-d4223eadaac9"
      },
      "source": [
        "pprint.pprint(wv_from_bin.most_similar(positive=['latin', 'criminal'], negative=['white']))\n",
        "print()\n",
        "pprint.pprint(wv_from_bin.most_similar(positive=['white', 'criminal'], negative=['latin']))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[('trafficking', 0.4996181130409241),\n",
            " ('transnational', 0.44992437958717346),\n",
            " ('crimes', 0.43998926877975464),\n",
            " ('laundering', 0.4213477373123169),\n",
            " ('crime', 0.42046865820884705),\n",
            " ('cartels', 0.417102575302124),\n",
            " ('dealing', 0.4154001474380493),\n",
            " ('traffickers', 0.40704718232154846),\n",
            " ('judicial', 0.39766523241996765),\n",
            " ('extradition', 0.3974517583847046)]\n",
            "\n",
            "[('prosecution', 0.5594319105148315),\n",
            " ('crimes', 0.5117124915122986),\n",
            " ('fbi', 0.5068686008453369),\n",
            " ('attorney', 0.5007576942443848),\n",
            " ('investigation', 0.49686378240585327),\n",
            " ('charges', 0.49135079979896545),\n",
            " ('charged', 0.48554402589797974),\n",
            " ('prosecutors', 0.4846910238265991),\n",
            " ('attorneys', 0.47757965326309204),\n",
            " ('suit', 0.476983904838562)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vyt5jgHI1K4l"
      },
      "source": [
        "How can we avoid this bias? Not sure, the only thing I can think of is taking random samples of the data before training. Then examining these samples trying to find common biases in topics like gender, ethnicity, sexual orientation, etc."
      ]
    }
  ]
}