examples/01_intro_to_floret.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0e8be927-cc4c-44b5-807e-9e5f39b70f07",
   "metadata": {},
   "source": [
    "# Introduction to floret vectors\n",
    "\n",
    "In this document we'll explore some properties of floret embeddings. You can also learn more about them in our [blog post](https://explosion.ai/blog/floret-vectors) or by checking out the [floret repository](https://github.com/explosion/floret).\n",
    "\n",
    "## Getting started\n",
    "\n",
    "In order to demonstrate floret, we will need to install it first. So let's install it along with spaCy and pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75650836-3bd2-4d04-bd74-cb173e9a95a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "! python -m pip install floret 'spacy~=3.4.0' pandas --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82d23b16-977e-49f9-a071-3ef28d64780e",
   "metadata": {},
   "source": [
    "Let's start by downloading some floret vectors from a [special release of English vector models](https://github.com/explosion/spacy-vectors-builder/releases/tag/en-3.4.0) for spaCy v3.4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7a25a53-7fee-49f9-b4db-6b1d2d9f1bd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "! wget -nc https://github.com/explosion/spacy-vectors-builder/releases/download/en-3.4.0/en_vectors_floret_md.floret.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fa2beec-99fc-4335-868a-3a11335ef4a6",
   "metadata": {},
   "source": [
    "Now that these vectors are downloaded, let's create a spaCy pipeline that has these vectors loaded. Note that we can simply pass the `.gz` file to `spacy init vectors`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee00a593-c328-4a15-84dc-2f52b1db9d44",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! spacy init vectors en en_vectors_floret_md.floret.gz en_vectors_floret_md --mode floret"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6c2f90e-bd2c-4f4f-b6a0-b528dee10a12",
   "metadata": {},
   "source": [
    "We now have a pipeline on disk called `en_vectors_floret_md`. Let's make some comparisons with the vectors in the `en_core_web_md` pipeline. In order to do that we will first need to download it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a32ffb5-87c0-4757-95d4-9f192381610e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! spacy download en_core_web_md --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "065fdb39-ab33-4893-931d-44f582fe907b",
   "metadata": {},
   "source": [
    "## Default vs. floret vectors\n",
    "\n",
    "Great, let's now load both pipelines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1214ba7a-d8cc-4b2d-b607-8ee44a4d6e77",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "# This is the standard spaCy pipeline\n",
    "nlp_md = spacy.load(\"en_core_web_md\")\n",
    "\n",
    "# This is the spaCy pipeline with floret vectors\n",
    "nlp_fl = spacy.load(\"en_vectors_floret_md\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "185f6e06-f324-44d8-9b1a-1299368fe35a",
   "metadata": {},
   "source": [
    "As the [floret blog post](https://explosion.ai/blog/floret-vectors) explains, the floret embeddings are a bit different. floret comes with subword embeddings instead of just word embeddings. It uses a hashing trick under the hood to keep things lightweight, but having these subwords at your disposal means that you can expect different behavior. \n",
    "\n",
    "To highlight this, let's consider what might happen when there's a spelling mistake. We could type \"univercities\" (wrong spelling) instead of \"universities\" (correct spelling). Can we expect the associated vectors to be similar in the normal pipeline? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6616a8de-d00d-4358-94b7-953ebc25dd40",
   "metadata": {},
   "outputs": [],
   "source": [
    "word_1 = nlp_md.vocab[\"univercities\"]\n",
    "word_2 = nlp_md.vocab[\"universities\"]\n",
    "\n",
    "word_1.similarity(word_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25ae08e2-629e-408a-8ac1-bc59da529410",
   "metadata": {},
   "source": [
    "It turns out we can't! The reason is that \"univercities\" doesn't appear in the word embedding table of the `nlp_md` pipeline! That means that we have a vector with zeros."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1086269-763d-41b0-97c9-2ef8dce04cc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "word_1.vector"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a437e11e-3e26-4126-823d-5c6f520accd5",
   "metadata": {},
   "source": [
    "As a consequence, this token would also be considered \"out of vocabulary\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5c62743-41e5-4ff4-b4d1-fd39b5fbf2a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "word_1.is_oov"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a451673f-2d3e-4d9b-940a-2470df5f9b0b",
   "metadata": {},
   "source": [
    "The pipeline with the floret embeddings, on the other hand, gives different behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "088f7d06-144e-42ba-a395-53e94cef4486",
   "metadata": {},
   "outputs": [],
   "source": [
    "word_1 = nlp_fl.vocab[\"univercities\"]\n",
    "word_2 = nlp_fl.vocab[\"universities\"]\n",
    "\n",
    "word_1.similarity(word_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4cb2b1e-a6da-4888-aebc-71e964da24fe",
   "metadata": {},
   "source": [
    "These two tokens _are_ similar. It's because even though there's no entry for \"univercities\", there are entries for the subwords. Specifically, floret uses the following subtokens: \n",
    "\n",
    "```\n",
    "'<univ', 'unive', 'niver', 'iverc', 'verci', 'ercit', 'rciti', 'citie', 'ities', 'ties>'\n",
    "```\n",
    "\n",
    "Note: the `<` and `>` characters indicate the start and end of a token. \n",
    "\n",
    "### Consequences of subword embeddings \n",
    "\n",
    "Floret is more robust against spelling errors, but it also introduces some caveats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9e0a91c-0402-4b3c-8012-952509bf0371",
   "metadata": {},
   "outputs": [],
   "source": [
    "word_1 = nlp_fl.vocab[\"univercities\"]\n",
    "word_2 = nlp_fl.vocab[\"cities\"]\n",
    "\n",
    "word_1.similarity(word_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e47ad13f-729f-4d08-bb3c-3820f4da180b",
   "metadata": {},
   "source": [
    "Notice how the \"ities\" and \"ties>\" subtokens appear in both \"univercities\" and \"cities\". A direct consequence of this is that these words will be similar. Even if the meanings of both words are unrelated, the presence of overlapping subwords will result in similar vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7190c074-e8ce-42de-b42f-ad3439590c37",
   "metadata": {},
   "source": [
    "### Novel words\n",
    "\n",
    "Spelling errors can cause tokens to no longer match an entry in a vector table. But there are other reasons why you might see a word in practice that isn't listed. What if someone makes up a new word?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d34c9008-15da-4b09-92f0-0b4ca736e642",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a made up word that sounds like a disease\n",
    "word_1 = nlp_md.vocab[\"zerglingmatosis\"]\n",
    "# This is a word that actually exists which also comes with a vector! \n",
    "word_2 = nlp_md.vocab[\"neurofibromatosis\"]\n",
    "\n",
    "word_1.similarity(word_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bba5d117-192c-4fb4-a3af-39c3e9f54093",
   "metadata": {},
   "source": [
    "The made up word sounds like it might be related to a disease, but our static vector table doesn't have an entry for it. \n",
    "\n",
    "floret on the other hand... "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "442ca2d6-5be7-4c99-b164-e31207d282f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "word_1 = nlp_fl.vocab[\"zerglingmatosis\"]\n",
    "word_2 = nlp_fl.vocab[\"neurofibromatosis\"]\n",
    "word_3 = nlp_fl.vocab[\"osteochondrosis\"]\n",
    "\n",
    "word_1.similarity(word_2), word_1.similarity(word_3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69c2d78b-a56b-4f20-93f2-fca89e5d5fde",
   "metadata": {},
   "source": [
    "Even though the word \"zerglingmatosis\" does not exist, the \"matosis\" part of the word does imply that it might be about a disease. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3afd062f-ac29-4301-8254-20b9e7ba4e3e",
   "metadata": {},
   "source": [
    "## Notes about spaCy usage\n",
    "\n",
    "spaCy applies attributes to tokens, some of which are based on the [Vocab](https://spacy.io/api/vocab) object attached to the pipeline. One of these attributes is the `is_oov` property which flags tokens to be out of vocabulary. In models with default word embeddings this property is `True` if the token does not appear in the embedding table. If there is no entry, it is considered out of vocabulary. \n",
    "\n",
    "But the idea of \"out of vocabulary\" only makes sense for default vectors with a fixed vocabulary. So what happens when we use floret with subwords?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27a80d02-b815-4aae-9ff8-71a53e4604d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "word_md = nlp_md.vocab[\"superlongwordthatdoesnotexist\"]\n",
    "word_fl = nlp_fl.vocab[\"superlongwordthatdoesnotexist\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "669f4207-dd41-4cf0-bffe-1610f6309ed9",
   "metadata": {},
   "outputs": [],
   "source": [
    "word_md.is_oov, word_fl.is_oov"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bb81fb5-034c-4c69-a97e-b3a71279abac",
   "metadata": {},
   "source": [
    "The `.is_oov` property will always be `False` when you have a pipeline with floret embeddings.\n",
    "\n",
    "There are a few other operations that aren't supported with subwords tables as well. One of these is the `.vocab.vectors.most_similar` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6290271e-9754-4d08-872e-0242fae7d3ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "query_vec = np.array([nlp_fl.vocab[\"zerglingmatosis\"].vector])\n",
    "\n",
    "# This line is commented out because it will throw an error.\n",
    "# nlp_fl.vocab.vectors.most_similar(query_vec, n=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61cf5381-df65-49bf-bbc7-2623dd95ef56",
   "metadata": {},
   "source": [
    "## Explore with `floret` library\n",
    "\n",
    "If you'd like to dive a bit deeper into the floret subword embeddings you can also download the `.bin` files and use the `floret` library directly. These `.bin` files are fairly large but having them locally will allow us to explore the embeddings in more detail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb1b6413-7d74-41bf-9f57-120d77a57170",
   "metadata": {},
   "outputs": [],
   "source": [
    "! wget -nc https://github.com/explosion/spacy-vectors-builder/releases/download/en-3.4.0/en_vectors_floret_md.bin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b94a67c-3c7b-4c1a-b45c-f96faae9e8f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import floret \n",
    "\n",
    "model_fl = floret.load_model(\"en_vectors_floret_md.bin\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f3de897-716e-4a7e-a4af-56b906180ace",
   "metadata": {},
   "source": [
    "Now that we have a floret model loaded, we can explore the subwords."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43d257d4-08d3-47ea-b6cb-13eb3b71bcf1",
   "metadata": {},
   "outputs": [],
   "source": [
    "strings, indices = model_fl.get_subwords(\"univercities\")\n",
    "strings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c6a1315-1dc9-4edd-81e9-a2a7c98c5ef5",
   "metadata": {},
   "source": [
    "It also provides us with a `.get_nearest_neighbors` method that allows us to fetch nearby tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb0a5a39-974b-4cd5-b12c-e8ae60344b3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_fl.get_nearest_neighbors(\"univercities\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53389ff9-7f8e-41d0-8edf-16a826a36cb9",
   "metadata": {},
   "source": [
    "You can see how \"cities\" is considered similar to \"univercities\" here. \n",
    "\n",
    "### Compare nearest neighbors\n",
    "\n",
    "Let's now compare the nearest neighbors from the floret model and the default vectors from the `en_core_web_md` spaCy pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a932e05-66d5-4bcd-8f85-e28f13f2594f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "def default_vs_floret_most_similar(query):\n",
    "    # Get the spaCy tokens\n",
    "    vec = nlp_md.vocab[query].vector\n",
    "    hashes, _, scores = nlp_md.vocab.vectors.most_similar(np.array([vec]), n=11)\n",
    "    if nlp_md.vocab[query].has_vector:\n",
    "        items = [(nlp_md.vocab[h].text, s) for h, s in zip(hashes[0], scores[0])][1:]\n",
    "    else:\n",
    "        items = [(nlp_md.vocab[h].text, s) for h, s in zip(hashes[0], scores[0])][:10]\n",
    "    df_spacy = pd.DataFrame({\"token_default\": [t[0] for t in items], \"dist_default\": [t[1] for t in items]})\n",
    "\n",
    "    # Get the floret tokens\n",
    "    similar = model_fl.get_nearest_neighbors(query, k=10)\n",
    "    df_floret = pd.DataFrame({\"token_floret\": [t[1] for t in similar], \"dist_token_floret\": [t[0] for t in similar]})\n",
    "\n",
    "    return pd.concat([df_spacy, df_floret], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7329e26a",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Zerglingmatosis\"\n",
    "default_vs_floret_most_similar(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6978d6d4-307c-4a58-9183-af9932d10bc6",
   "metadata": {},
   "source": [
    "Note that whenever the `en_core_web_md` pipeline doesn't have an entry for a query, the nearest neighbors are going to be the vectors that are closest to the zero vector but these will typically all have a similarity score of zero. \n",
    "\n",
    "Feel free to play around! It can be very helpful to understand the difference in what the vectors represent. You'll notice that the spaCy pipeline does a pretty good job at capturing some popular mispellings of words, but not all of them. \n",
    "\n",
    "Some more words to try out:\n",
    "\n",
    "- univercity\n",
    "- hobbitshire\n",
    "- superduper"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6724a4f5-befa-46f6-bedb-29efa27f7112",
   "metadata": {},
   "source": [
    "## When does this make a difference? \n",
    "\n",
    "As of spaCy v3.3 we've started shipping some language models that support these floret embeddings. At the time of writing this notebook (spaCy v3.4) the medium and large pipelines for Croatian, Finnish, Korean and Swedish all use these embeddings.\n",
    "\n",
    "There's a reason why these embeddings might be more impactful in these languages. To use Finnish as an example, consider the following sentences. \n",
    "\n",
    "> **Talo** on helppo sana. -> **House** is an easy word.\n",
    ">\n",
    "> En pidä tämän **talon** väristä. -> I don't like this **house's** colour.\n",
    ">\n",
    "> Asun **talossa**. -> I live **in the house**.\n",
    ">\n",
    "> Nähdään **talolla**! -> See you **at the house**!\n",
    ">\n",
    "> On vaikeaa elää **talotta**. -> It's difficult to live **without a house**.\n",
    "\n",
    "Notice how, in English, you would use additional words to describe where the house is. But in Finnish, you would change the word \"house\" depending on the meaning. Now imagine that it's not just the word \"house\" that will have this property, but every noun! Image how big the vocabulary table would need to be if we wanted to support all of these words! \n",
    "\n",
    "And it's not just nouns either. Let's have a look at some of the verbs. \n",
    "\n",
    "> minä **kalastan** -> I **fish**\n",
    ">\n",
    "> sinä **kalastat** -> you **fish**\n",
    ">\n",
    "> me **kalastamme** -> we **fish**\n",
    ">\n",
    "> he **kalastavat** -> they **fish**\n",
    "\n",
    "\n",
    "Verb conjugations in many languages are more elaborate than for English, which again motivates the use of subword embeddings. \n",
    "\n",
    "## Should I use floret for English? \n",
    "\n",
    "You can download the [floret vectors](https://github.com/explosion/spacy-vectors-builder/releases/tag/en-3.4.0) if you'd like to try them out in your pipeline. It's plausible that if your domain includes noisy texts that you could benefit from robustness against spelling errors. But at the same time you also might not see any benefit from floret embeddings. The default English vector tables contain most words even if they only contain single token embeddings because English morphology is relatively simple.\n",
    "\n",
    "Feedback is always welcome in our [discussion forum](https://github.com/explosion/spaCy/discussions)!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "vscode": {
   "interpreter": {
    "hash": "4f946df053fbf2b937619d3c5458e7af74262f9a954d8797ba0b27400bcafe06"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}