Bag of Words Technique in Natural Language Processing: A Primer for Radiologists.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "BOW - Radiographics",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/RSNA/RadioGraphics/blob/main/Bag%20of%20Words%20Technique%20in%20Natural%20Language%20Processing%3A%20A%20Primer%20for%20Radiologists.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NL8pEoMIPtaK"
      },
      "source": [
        "# Bag of Words Technique in Natural Language Processing: A Primer for Radiologists: Hands on excercise\n",
        "\n",
        "\n",
        "\n",
        "[![RG Reference Logo.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAaIAAAAnCAYAAABNAUywAAAAAXNSR0IArs4c6QAAAAlwSFlzAAAXEgAAFxIBZ5/SUgAAKXNJREFUeAHtXQlgVNXVPve9mckKyUwIi4BoW4oQElalgCJLEkQquKHWv8qvbcW6VKmgbbFFVNRKVbQuVfu7UkWwigsgJGFTcEOBbOCOVXCBZAKBLDNv+b9zZ97kzWQmCRhA7LuQeffde+6595735px7lnuHyEkOBRwKOBRwKOBQ4AhSQBzBvp2uHQocNRQo2Vr9LQabfdQM2BmoQ4GjhAJul7uzcpSM1RmmQwGHAg4FHAr8QCngCKIf6IN1puVQwKGAQ4GjhQKOIDpanpQzTocCDgUcCvxAKeAIoh/og3Wm5VDAoYBDgaOFAq5yb+G/iMwsQaauk8A/k8fOHy4SSKbJN11NIQIqGV/oJn1EGi3Jqy1580hM8vNjJ3prawMPY1T1GCyGw8kUqqnuyPGvvPFIjCm2z9Iu+blKQDnFJCOHSHRgciIqxFQEVelE5SpplXXVHUqH0it1sW2Phvsy37jzTVOZjEkFMC2Dx4z3xG0o5uMDqlatOhrm4IzRoYBDge8PBVw66evBIAeZpJ7vMqkDs0wX9KSAYS4whLkNDF9VTSVZmDSAhHJWkqKQ8ND1Zb78DzzBlJP71L6y+3BO59P6/cFsxfMB2N9xGMkvVSJFMQUFhV6GcUQJorLO47oEDaVnkiBNN4INeVVrth3KsZZlFfwJ47o2TVOzNRAsYIq3yDQrSRF+CEyfJmiUx1Smu4SLdF+9WUYF23TVvGzgruI3DuW42hu3IpTdWAF8oRjmGLeqnMirAY8QVGear6OvAxZEG3oMT8mqT+2vKUowiEUF6LGpvcfs4HMo4FDg+0sB1wB/yYM8vC0Zhc8mqVTSCA1IGJBEhvn0AH/xSvvQTRrirszMWiYUyldMpU/A1fCRSVM6CVqMhf7hSWN2rdmHnv7MvZX6Cja5TOUuHUwf/EuuzO2jEJpyZaZQ/qwbghqI3kHdMHt9e+XLfIU9MYANblJ6uKH61Ova/CBpcwbVrKmJ7WMDTUnp6N07L4mUK6Fx9g1oZh5gjipBlFNVVIIx8x9h7l9DG+rC2jRenJCCyhUHkDr4U3snJXveIc0gLIyCaOo5gOYOqEMBhwJHOQUiPiJB7u3QeDAd8BJcFEVA2YhOgt4LelOrJhuA4T8PKZkV3prroqEO3x3G8KU0JbLwjLMjCpJJg2ZHQYyWFZRDMbLKzNN7wTr1OYyaPdzQzOoEXdy/pnh6PCHE/Y+gxfX9/SuuaiRjNTQ6MoShHYpxHS6cijC3s0H3uyQ8PROaEJ4TxJkQjd8Fl9PWoYBDgaOPAhFBlBpgq0h4QdvCuvaYne+xX6NEBefXWBwJOu9ITVtKSjmDsGcrZiAKhqdgnDrGCTOjK6b6O9+uptEuQ2jvC6EImDWp3gzemVu14uk2ITZprcoMHM63NsF/T4EMSWMeXAsvTStjNz2aqbAvUrBWazZbALXS3Kl2KOBQ4CinQEQQYYUeZoh8aYWpmOYnFhQgexxRGjQzyDWN5vma4O07OjamJPu2J3/pc49uqmmfXLbXdQsiOnwswAPC3NPfX3JDmzEL5b19pJeqJn3V5jbfU8CQQe7g5emL/tGVinxOn6Wmd/zY9z2dpjMshwIOBQ4RBaK1hAgvscRM/F4hfLAODgkrmGVakVrxcbRfaeKx3kRrNNoOxU2mj9uvS2CaQlNUU9TMZLOf21QQLGE8cCAd5PpXvgp4/jv6U+JH0Ka53UQ3GTdtZzfeDzttfGMV+at2NZtkckoq+bK70E9zBpLb47jHmhHoIAs+3lpGn31YScf++KfUp/+gg8Ry9DTbsGo5NdTX0diJ5xw9gw6PNFoQRURKJJNoQj0YQvo4yGyRw2/pMup4EUy5CNHhQ2D/PwZBECzCdsAHtbR/ddE/E3VglcMZfj7k3hTTEL1gxArAgPN+esC89Uf7S77R2TDIOp1NK3q7w7isFA+dQoYCnxBbediNDk+GGazNq1m1zsJrv1aknd5V9wSvBT8dZgqzM5TDIEx6W2BxeyGnasVLdlgrf3OWf7JqqGojrEkqy2JVfc6qO5hrmXfMRJWSTA3jDSWVVFN3eTq5VvT+eLn0m5RlFlwDuX8mpp2Fyptyq4pfsPqq7DS+m6FrF5lCGQZ6cPAE+1u+xoJhdU6N9z4roOT99FOyk5M8I7UIfRpBHw/oY+zLqylay/i20aQOQW/dX5AdDjQeAHzaqKp3Dt294n2rv6hrnNellCZ6lczAjVixnAi6JiOS4UvF7b45Z9eyzVbbcu/YMwzyYMJNc4bBV+T6i5daMLHXzd6CaYpJ+XBn9uE60zA/xZw/UoRWlFu9Kiq4hus/ogkd673aLLwmIzHMDAR9NuqmKId3bl3/au8C0OWQ+A6579j09EN/o/L334otjtz/6Kf9aO4/FlLnbj0iZU7m4Clwz02/p22l71GvH/ehx1598+ARHSUt/3Hnn6nq269/AIKozStbpRAMhtyI+w4YGvYhNU9l3vwHhCnOVYKiM/tPGoTYCCFSBuE10i3EiWh/ZnlWwf1uw3VWH//y5bEYNmaM/XGyor4NUZKFPSuhOAoAAeeIhiRxVVlS/iLTFE9atkUeOqc0l3IC/BaLk4Xi0kwWVQI8myioKPwmjpBA4Y+3fBM6phpaCQltaApiM+pMY6Giag+Z5GJBe7vLoIsRpr4HcRtTYwWSZooJIROmyUt5I/fb10rtuA80jz4fJKEfiwAQsGV23+ugr0q1X1PnLRljewpVfQMqWArLXLeKcHXT5CXeCxjfbGEq00zD6OZRXNRoGOVwuLwLFAOA6wyEkZ9R7quZV075v+1fXfxoUlJKrm7QwmRFJGns4BFuKa6DgtYD38mbffm3aNRwowvj4CUDb36CkDoxTTfPR1+LcquLsTCITaAW+7vCqcxb8By2nZ3HPjD+Y9MdghpPVDXtrFJf/vy86uLpDIrdX48mkdHFwHqC58WfuhDsg0zjensq9+bfDoDfI/zd0yDMCl0YD+O9UhGGcoVH0GQyXTPKfAXf5FYXdbXa8TvYKIJXpGAUDaaxAIuLlzTT7K6SeZ2L1KlbfTX/V07jb+lfvYKF7mFL9y9cSawFcdKxe29b2Xu0ZvkS2vTWOrrt+mk0/+mEcviwjfFo72j7R1ulEOqQkUmff/IBbXlnPQ04aeTRPq0f7PijNaI2TLMcjMplCg8HKoDp7RroX/WPuM2EmAqzHfiX4Rea2RsbYKssuIqs8ZM9pljSYBhJuqIvey89v9+QfcVbrfrKjj/vrSuNHzKjV8EQA0KfrruVZzwuXW9opCyXLq7xCPWKRtIngsFA3wETZF6I1N9ftP4moqR+NEX08+5ZC2VlJAIVWIBZy24JtzllbHfV1D5nOZWE5XWdoo8esDukETDAIppyb7/Mmo3YMzNQNc0lpd7xf8jzr/irbMwfggYyg0UYO5g1RcYeqT/ATJ6/qBf4vsBepHMR2r0ogAlxbKI72ewldPVd3isFmrNIhvbCTD8Urg7tYCrG0g2M3NgfdJ8waO+rH1ldb+0wbniS27Wh3jRV0OuRLdnjtuXsWrkK9SmrabTayetaiT1iY5g+kDY1FZ0K/pJiuG7cZ2rLhcu4TwTVOiiw81yKcpIGOrtN9TwIGX+uv+hyqw95Zfrjv6FTVak3fw32mp2Kd2MHxNwtQV3ZpLj0aR5SL8VeL6hG6rXlnca93H93yWoWGjzncm/B7zyqMj8IwYiB8EdUKvUWLofwOA3Cg/DMLwStnrUBzMc7WYz3ZBwoxoSRqdRb8Hc3hBQvVBqEcTb6ejFcxZd7y7yF6/HcRwRM3WsrPyzZ43r3pZTUJln7k765NO7nU+j80TlUsekd2r9vLyUnp9K/n3oo7ng6erPotLMulAuFxY/fHxdm8PDRdDz6aQ2HhhilF56O/gqrLjf5OnWmn51aSClp6aRrWqt4Ghvq6aVn/kn9B/+M+g08MWpMLy54VAoB1vg4BQKNVLn5XSkc+H74mNOod7881uDb1Be3aS0tfyG0Pp4+5x66+dpL6JXnHo8SRG2Z96pXn6fd334Vt6vsrt1pzOln09LFT9GxP/op5Q75WQSupfnV7a+lV597gkZPOKuZ5tsWOkc6OciMfbxteWZdu/eMO96KTW/Ld3XgsFEwKQ84yNE0NWsuiNhwhlWwxdibQLFnxJs/D4x3hoyWw/I9jZKH2uuj88IAI6B6od7ct3ZFRAgxDGsXYFgvuQR254PtJHnEHBRHou8MV2AlM1437CgNRvD0PD+Mn02JcV2J/TgzOmTWvIbovVFSBsmPENBN0li3mMqUwuUIaxsJZtrUOpxTU1xvIVJLBZOlOkO7Om938Vo70HnYG7XNNXJU0Ejdy+2ThHpHaeaYt/JqVks4aAtdWFQo+Af+GTU/O54DyeN7aFZkB19XdViygBuSUxMavQFz4ZJqf8rFJ9PLtdAoHkOAxCVaGDGGZrjBbWGzvN8uhLi6L06/KPUVzocGei00AVI05Q4U87LQHAP/GTbUvgXBNiYoeb8oMHRxUqBxb/e8uvU7uX04DYMmNA960wz4wUAHZdqH6fn3/tS2cOCIN2hoMFGKR/HIM+uD+vABe4vtNqh3YFYUeN4YNyIYdcEh/6sZP895axdzFTTncHfWNXSLvu/Ee3QauC7i77V5/f3FdiEkgfSgOSfZI8YFw4/507RJXYKi8Sqes0ZmZW5VlBCSbSDOH1chiEKBFuGuj+CFBVPHTB/tr91LDXV1+Aoq9PC82dStRy9Zbh9at57HRQQRw3Q5pidl+jrZQahL92Op+7HHt4ojEGiQMCwkfNkhZXLXNzvp84+3QZD1o7ufeoXcbnereOrr9ksY9nU99sqbxJqIlR6/by5NmzGHuI/6/fvo+l+fAy3wfcoJC6wFD82jISPG0B2PLqZAY2g8Lc3bwpvoykKm6KXniIXFqeMnU28wyteLXpH+OW9WtmzWlnlvevt1+mRbuYT/4rOPYZUR1KPXj+V9734DpCB69tH5NBYCyRJErc2vtqZG0onbx5pg22PuiWhildvH25ZnlpbWodl439uwhm684kIaOnIMnfXLaRbq73SNFkTyiwwDCQfQCX0YVqKEs3N64SCdidBuRoN/pXOgNBjk03n+4KWCiix+2GwQYNKVjWRm43SB4maVKFBU8Zyqi8l8AgH4Ra4FU+ErmALGeRzvKwmYxsYYIWSByf04FUrB/TBXjQKzsa2FIyC8YoRlj+eCMpssqvSOuwLtesg+DH1Xnr847rLyhN3rayF854IRzmKmRorrcWD6UbgHVk5CeIXRbqdL6AGt0WBbJv6jX8hRc2GOv/gia1Yww22Ff431BrcsE+LDRpYBhmlf8VvgGHNwiWp6rg2wxDepb1MF3FrC+AT4QB6mn9Brq2t7jaA36+0wnIc5bibMXpdDFUvXQQfDLf6I4ostOPm6YLw44mf7tqrik7CiiNI+GQ5S6EXVEJcE0B48dojVlq/BfThPKomB7KVEb6UN6wJNbmYAqpaKBUO/qupZ0RChO+w/2qqY7MrS5ftc464/IR2dMDrdMOIuElD9rYKB84LnSCZePX+z4z/QSh6mr77YTqwtZXXuKhk2j+s3182mU087s8UhTr3qDzT+rF80g2GmyKktOH55xUzJtC0kpRvfpOkXTaQNq5fTqYWT2oynetc3dP/cG+iPdz5soYq6zr5mKu38z2f0TPFmKSi40upr6aInI/6Ntow5CrHt5s01K2iPv4ouvExagKlw8gX0wG1/pOXPL6ALp4XKLPCW5n3dLfdaYPSnyy+QgSRz7nsqUhYv09r8ho4cG69ZVNl3mXsUojbetPbM7GgsIcRa7Kx5j5DqihYhdtgDycfFwpFg4Aoz2boPhsdf1XTwGRw+h9W/oMcGVK/8VWud5FUXNemqcYDBAOqZjzPDxYo/wg3Ap37HTB+rfmpQ9MfjNI0UyaFBG2Ac8i9SE5PhelvShfIbnhjPB0zsSVtVs6xmav9MVpJnBeGzSSHleARPjMitXrkBbA7dM2IMHblmDb9DAVvKQklQbSB5hnXH1/7+jLsrs3c9pFRlSyc7TJGn2+tj85BW+yR5gZNNH/YEkcAPWpo24afZGE8IReDhV4IG8Wv2X0E4FkbKZYaPhQJuU78/nhBiEE0YXyaB4jIZolMo0/JnWlKHy1lZQ79YHJmv8IbqeC2G1q7ZXZE9uoOheEKUc2F9BMEEszDBb3bKpo4FvQftLfrI3nZPcmaR2F+Xa3qMXfbyw5H/+ZCecbs5IXcwzbj1vrh1h7uQTWXMZOr37z+grk8/9yJa9vzTUniOGAs3qi3t8VfTe+tXSwbG2oqV8oYOp+tvewCC6Rir6Dtdl/97gWxfeGZIOLPZ8+F5f6FXFz1BF/zmGmzWZ6YRPx3svBnb4Zpf/JEffGlLz8yO1RJCp542mWbOvR/xWVjnt1OKK4iS4bjfpzaMHLRrjYxwKvcV3AUb/O/Z+Z9E6qVbOoy5c0Dt6g++0xh4zQwHh1y2Rhgv+IegkbwB1Q2mhdXw2632IWUmoKJ5rGwWj0wVNDodTvyBDVi083E8MDWuaqkPnJCwHeawb+GH6swmOjDskwC/AX+6rctW975s8RX8GkETP8FUm2kL3D88TR3Jpd6brOm7JEBYaHTmSluS0W+7KLTUtZUnykJbxMl2nKTmmQAsFCyQoFIW47u7El/fX4cWCUqX0oyJ3rw9S/2ykgUyTwzyKBEOd70eoCSr2vbA7Q242EZU4DyHj2+C9sqLlY120Nh8TujoJ1ls7N67ocHr0zBeF5sdPar4EJr27JzqoputdiO+XMyaX7l1fziv4+HfccHctReM+a21KykIrej8X/1Oai6xi4VVy16gTxGCbE9Dho+mvBOb4m7eXP1alC+jG8xyY39+bqRJW3CUQQPSeU87WnGI+Wr0qyBYxh4K3BY8+WecJ9vfPXu69Bd1zGxywW0rDT3CHPiRYpOl0VlaXFv6isXB9+zTeff1YmnS5FDmjypDMUQ/OSFXmgO5bhh8X1Zqy7wt2NaubZnfNzu+aA0NHezcW0WcAKClZ2Y12bhhDb3w1D8oLb0DzbjlvnYVQtyHxRms/nDFqwhmAJ4SYQn4Al8PYTSdl9TMiFS36wEA5tsatZh93zehn9sMngWsI2CbBzMWx7A2A5MSGLDsTLYvy5jw42SstzkAQUBIGfWuHS0i5koeJQ85MtqmFszQY4VRMEvp52HGiQb8z2MoCIpoOeHsvS8A3plNUIhpG8rQEEjfICLsuJAuh02trSTA5qD92RA4x3L3kRSafgNmXAVT4hOGcH8rXVwME2dOkXZxMtsyzjg+SPXnQIE4Gc17w9wGsyqZDQgSkKjiEYkXh1IvioPQXqQqX4vwWedudFBvalmoDgkiib0Ng7Umbp+/vY9medGDNVcBhojzAj9pVp2gYCg0p0ql8HcwvT3IrBVeKYacU+ErnINw75eTlL0X9a5+e2+C5oe8+Oob/xoJVviwYgvNuGQyPf/kgzRsVEGUQ50H8uX2T6huX23UmNg5bk8fbyujKpjErNSn/8AoQdQWHKuW/pt4HwoLIfZVMLP+v5fXU4bXFzETtgUPj+H3CBC49Izh9Pdbb6BZf3vEGhb5q0MW7LQOHSNliTJt7Su2fdGS56A9G/TVl5/T5eeMjq2mlxc+FiWIWpp3s8atFBzI/FpCdbBzbwlna3WJnpnVbiF8YSPHnU686Hnqgb/SpdfeaFW1yzWOIArjtWnkYDF6mSlugM/iTumwRoTSps5jRg76dvX6lkaByKVlMLFNMEwWXzi/RRirwJD/ppGocJNxEsKT79Ft3mLT09jd1GFMCnFNSBF/oCX8XMdKFcNbPC4+fBPnA/ZI4m60oNbMJxIBCGcgPHcCdgj3gXl04GJE0pVDSxrGpioIo1A4UGxD233/6pXTcTv9gw5ndDLdgV28UpfHJBlGSW5NcUSol2ac7E1W00Itm4Ztw9Q8iwiwJyFwLw7KPaGsPdA7YOAPgrRbXKrZw627npXnuMU2Zf4sCRgmYmy97V6v02s8yXhdJBEQui8QBG4lJmRbxsqCULbiBm1IMpoPywXgVoUe19eTCEu/qpUPQZNtQKTfY6xhQ8izSZHflUmN1HEPR/9pSergQV+/tj0RjsNRzhFH186+i+bO+A3dBP/JI0vWUXaXYyJdX8y+m1Z8RBf9dmZcH5GFpC04rpn9N+kj2rd3D8367S/oY2gSnmQOmmlKbcHD0BywcNWsv9LtCEVnM46VevfNk1kOhMgZxIaFpsTBC/yMjsOeH05t7asJQyi3/IUF0qQY8l/Yv+0Ilbx5Br29tgg+uS+oQ2YomKIt847tI9F9W+bny4q1cTTHdrBzb46p7SWJnpmF4eyLptGVf7qdHpt/Kz3zyD00cNgpNHj4qVb1d77yejiSQgtmMAkwydidHDgJYF5AGHvgsJYRUm7N1bTUiWAIZTbilG5EO+2H22ACsxwwgHsR8isGVJeM470sg9jHIpTtkgfaOBjMf1j+ogH64OIUPTv6TYrph28t90zLrK2plk1VIcmFxk3FcTDbigSoIYnDzm0MDAlN14aIZxL2ILkRKh29TLU1t2f5ZzPg8K/iAAHuHhphc12dK6w/e+M4eWiqnwPXxVwFbWsJ0xmhysPy/CUPDPQXv4HfkMJmTyDDX2jkNiQ8gXBdAouhDTicleMS8JexYTOcGHHUm2RVxFwZTrZqNhJUNMk1q1Vot3IoEAVLmSYbjwXQyhX7lR7v61/B+uBToHPYJBo2UQrTm9xofFbmHfePVtAc8mo2f3E4796aarrv5pmHvL+WOkjvmEF/uecxCgYbac41/yu1i5bgE9XlnzGF2Ec0H5tKLXMbbyzl/VMrlkQHPrIJbfZVF9HCf96bCF2byjnoYcfnn0rNkoX3yfkTo/44aIGFHfuKYlN7zPtQzy92zO19H++ZWX1Y/r6Lr7wBofYD5H63eKeEWPAHeo3SiMI8FjiYqzRnDKj/LcJ4n4FAImws7IcQbOzpKH4mttOkTN9S+ENSmeuA8S+yNjDGwoXumTOFGBN2+welu1kyTYP2pNZ3o/3UZHOIh4Cd5LwxMx5vk/DRFa7G5FocYCC75DZ16Vo21dHOeKitMshlH4svnM8Avq18xeVKTd0S4UtHjqMMsdpW1fG4+ZDrWkswFcFKxyt91l7M+H3zsCXTTowNP4PxBOh8LIs0rPrX4kSCs+JCc0wCP854+CKPOdaIGY0pJQXR+AwLojWYupm1J1nSIQKVCH8EgDPRzyKqit85wUGYTYME09gHKnn5OWHh0iZBH40zdDfAv3IqclMrsgrH4lE9iXwPXo9wKHmS4pq22Tvu1TS/p6g3hU6wiIfjUJddNesOem/DamkeW7fyZToxHF31GTZmZvqym3WfCwd/W9OB4uCovSv/dIfUaF5c8Aidfs4vZVcHimf6TXdLEx2byjixX+y6m+fT3JmXwdfQEXunzqUamOuW/OtRqt1bQ1f84TYJxx8t9ZUo2OC1cJBCweTzI3jsmdPO/h8p7DiYYcolV9mrZD523udcfHkzmJYK2jQ/fpmR+Pgh9sFZKSklhXqFTa4HM3cLD5tVb7+h+bh5X9r0OXdbYAmvsc8sFpDn+CdEy007+1S644bfypD7WL9mbJu23EcJotYaDMBGQmg6N8LB348j6/BC3IU2z9jbfZQ+IVtT9QIN33js3icjqP/NXh+VZ55j403BQNInujtkjePd9ti/2gcQMmAiql2zGyAKqVfNaiRjkyaeUFXOvuWVZe4C7pVNhaTq6gDkt8RpKIsAqOA4iD7sq+BfxmjUjfVccQKtry03818DgzwNwpZDl/np/102au1DiFow2GyeOyjU/BnI0eGDry0kzBrOfP7xHgg0Q7TwlvHCIqwQ2PExf+DvQohP2Gua5RtM5bgk0ACCGH2ZXx5zML8uy9qXFFi4xkty3rYKk96FNtdTar1m9KkYNqhm2Qqa4tG67MmsTgnsHbN9TYMFkFMlN/P2rOgw7hThUdexy4vfU0T8/YJ8tI6qsV/2ECbeh8Ip3heX97dcfv0tNG/W1dK38uiS1yXs0w/OI/6LTa++9wUlJafI4nj4whXy0hIOC28sDl4dF7+8SJpiThqVf1B42Nxz9Y130m0QPFbiIIo90PxWLllIzz/xAJzeLho8YjTNnv849kP1IN7bwqmlMds3A1t4mQGvxWlc7H/62WheEzZPPY//iQyg4GOW3llXJAFamveo8ZMiZlJpOLAtkppjD5W0Nr9vdn4pAfk4Hntiv9+Di0tk0YHO3cLDc2Ghz88tNqViP1AiQWSnQbNnxhPnZF2RZTpefsMtdO+cGbQYz/C8OEJdtjmAj+ZMsJXGvKEVYQzLeD8IHP1dyzLzr4aPI8KA65Mbjk0xkrDaBOODMElVkuB8T5DCjJZ1A4YYCrNVhS9/B+67M/MRpjoJxc8laI1icFHJ2NA8jCsubEwdSPsaWPME9nIgfzbaPBW3HQrLMwoGw/SWAi1AgmQE3aE3GHcIR5uLvTGnYb8TogmVfvgJ7YLc6pJIfSKcoUfLA29aEdlhQ/X4DGXsVZF8Zer4bvhV1PR6HhfgVE8Lm2rl/PkjDsII/eLURXpDS1MZzxSABgbtj5baqkJZlnUxdG4GIwsAFA+Ou48pV1TzKZzicTb/WKNbqIUwf2Zhk26rviLdVzU5I5i0SAmovMSeFTuOnNqS17E/7O8I37+au8R3N30Pjq2NhWvv+9aO7uEVO/9ZqWRrtZVNeG0Jhhl2S/UW0kQwvMHUSolgrPpEfbHWw3/2dNYvL8NGyMvkCRK8cZdPb7BSIjxWfaKrJymZlr4fYvKJYLj83n8ti1TnTzovkrdn7PO2yuc+tNDKRl0XrHw/6p5vWpofC9uWaNlSXbOOYgqeXP5uTEnzW/t4M7EJOl5/sc8sHsykCy4l/muvxOxDJiMFKoWUevhqQlNAYEIMWwjB5fhXLocQWh8+eoc3Gt63LbUg4l3VGpR6ye/wtWbn8H5FGxfuIurCYoYLYLHnz0jUGfjis/jVVTbp4Z9x4abM0SGvYlRreZglDGzG3zl4Qrp9eP9nG5Ph0u/gvSk4r4xNPpM3dBwe6T8WBaL3/iwDC1DRaOgvH79/2dcWTA5+4hvH0izkPU8sjHDWz3PbOo2UwQwWTNwrM3+edhzWp+OXjSzCs1enFjuK4+EQaip+3JX7lIwUR6OIgnhwTXSWnD7jCxoeWkYzcPjph4cSXyqGkWJhcL4MpcaqPhgw7rH6Cg2OByHFakhaW5W2q5ns6SyftZxznIkDRWgcIv0t37CO3DSnqvglbMSVPi4DczVc4kEbyoRZnHaBLbDoTSiR9zIWGOHoH3LEpnzlhahOV1P5qTjpMFKAzXN2IXQYuz4sXf3Q59eeRIwIIlUzuluOFj7LDOceJfwSg+tcwRFfknNgNMFks8Qa1OB9JZXQHr6RQQ1gB2CmD1Skje5q1fOV96BAhbyAj4Vh5gPG0dlikKkm3YrNjyjFkTQQFm7FvcbO3AEvNnUuGNno02oA8x/JXiHT4GvpwrjtiU8CYA6J6TDLiWh/ebtWrYNweRYRVdKi10FJe8Tezspv6pQ/Okm4JrF2BsamIZbtYqvOumJD6S/g8CmXTI3IiyOBvtiQNba7VR/vyr/ELhNPPiYlE2WG1gMIk2dJ5d4fN8ymb+2LVfBxbOZABI6IQ5q9OTN/kB3drj6TOui6em5ogzKCP7BsCHjTmhwOTGbQDtYpluIjcJLGI99mT2lanoaRIfrxfvjuMnAuHUyT+lMDa0s+tPpB805MXVTh9VF6W+WxV+zByuCHze8WTs5wbfTmZ0RgXApbF5FC4QnddHdEKAa1wER+S3iGLkU9r9Jb2EzDsfCY+KHCpjxPDaeCJEg4tW8Qh3UzjRHU+fC+3TsiJrwETZxihwIOBQ4RBURlp1O6mYHUNF3V57hJuZA1Ef5yaoKKTb3hstQUc1/vr1/fFdt/aWb+i2AoZwKOYD4hRDWtgTN/aqNarylaep9UhVaxJsFCgPGBkS8GY9gORnRquuo6qdYILIQAWorz5J7mjaJBYc7S9gefSe+R8lW9PzDBY6ovyhU/GC3jBzNdD9FUi1DqcanYybTf1B4B9ldSheuVRvSThD4Cipga3N+4zpXq0d04zBJh4iuwCQUHa2JOYJRBQ0xK8lC5GtxTxftIcNjmSx5FmcTyACcn3JODo2wwXMnVS3FgKI6lWQvNyY1AgPpgwD10MPxLsXSw7hF9BXqoZ3KgehKckPi11idEQNzZ6KadGVluyeTqA0qqtre+IwI+3gGtOoOxUkA37sqtKZqxGcILG5WT8QMU16PPyxiPC3NH348ITZ2HY38alW47vsmpqAhYfVamF/YXbrNMrv5RiA3ArJktBQeuRCDEiTApjq4z9Tew7L85mZSV7JyH8H9AS9bnqcbu3SLY+TycofeYfE6m+AxTT0kWrq4w9y1WFbEMzwUOITETGt8J/BxB72346Y6+vDBoDKZ1UlTjRAjg51iQsSA2FePDgK6eh9D2vTuyP9054eOPA5/4Cns0aiLJUI178R6czmNlqaMpxtU4kGeZC4cRwtR3Fx7PGUx4vIMsWOeS23hcMY392Kj6dWlG4RChGG9iYeJmVyAWKjtwSvcfoLV/xLQwDD0bz3k03q9f4Sw6b6lv7JQU4VnE9nK8V0+4g8kzOVqRYTmVZuVPTjXVJXy8FH4S4u6c6hXXhWoSf8I8wSbmJiGeGNSpcSjgUOAAKOB2uTvj5OP87dj42B1cRgcPkKcFMGMG3+HvfBKWsJsQjTUkFu9G/P6NRzd4oyczRigkkhfhlGfF3buqMaUywz0SDGaR2xQ+XnjL89HAXNHBJ/j9nl/1DZ90jTPMNqHNQPRPqdCA9upafs6eopKyzoUjwDWXwJGcDTnD45E4AoaxW1do+sCqogXl3sIL8PMCz0LbqQuPmVfEDAp2Jz80jE9m5SdW4rj1wGT4EH7L5woGqfCO+x9DKHeAqfHZc1h8m2Vw3nUC4z6mHgdaI6LhtYaaql/wJkmGbylt6TR2MHxoj8ImOQACDke5hQgDUyZ+NYJEMk7uY2d/PRuOTHOXqorPMdg/5FRhvt58bLAUOEWIZU/4OfBM+JcUQFucVu7G2X0TcfL0MvsYtnlPywNjXoHTB7pCt+EAEjn5AOk7IMiu7hc+8BN0fhV+rIks9FMhsPaScS5olwJiPM3CAX9r+g/QCso3u0vwrEZhkSGj+FkAARdb5RbiPZDhU9i3dBsE6R9h9uNnj1A3+YSBhaH52DjFXS+MfLdoqFSM1J0QgEHMg0/cMYBWPgqGQ5bh8YkwPEgxflTyOfIvd/ApT6bYhW0D3RjkMxqdXOt1PQnL4CQcnpqM5xL2n/KssVAxTT+E70v9a1ZcYvYYnlK+Pw2BNOIMLFx6pABVHegBolcBtCdOB/FiTv8xFfWK3N0rmvu7uMOY5AiiGII4tw4F2okCUhC1E66EaLb48n8GO0suWMxeVwoV993R3NkMJlkIo0wv+FveHVRTFBUlV545fgBiQQaxrwPH+b8/1F9clrCz71CBs8q6wsZ4kqno+GE8IwhtqnRgTfGmg0Fp4tdby7xVhdAFc+EQPxaCIhU7pD4zFOVrzQh+4K0x3jmemqK5DqaP2DYVmQUDEZWFTbc4j0Ix1vapWrUjFmaLd9wpLqH21bE/Oa+65M2KzPxLcP8YSxnNpNfz/CtHcZstVJimZOI8OUX4cP7qtyf4M5ZZP6wXi/NI3G/tVDgkoFF/yFw41JSvEKW5efDuop3xxrKMJiQdl9k4LEiu41XFhAVS8Qc8desGx9Hy47W3yhxBZFHCuToUaF8KHBZB1L5DdrC1JwVw8sAlbJrDCdWsVbzePyyI2rOPHwouRxD9UJ6kM4/vGwVYEMGO46T/VgpAETomZLeEcY0daU5yKOBQwKHAEaBAJMroCPTtdHmEKQAviwzk4J9Th7OmKYrtCI/L6d6hgEOB/y4KOILov+t5E3xh6YhsG66Y6k+wX2wqn2DBIQIIThiAg0DnImzhDdVlbOJotf8y0jjTdSjgUOAIUcARREeI8EeqW1N3jUQ49vOI/ahFAEgNwh39HLrG4eL497+I6rvSMJR5KJp7pMbo9OtQwKGAQwGHAg4FHAo4FHAo4FDAoYBDAYcCDgUcCjgUcCjgUMChgEMBhwIOBRwKHHIK/D8u4Ttt49UHWgAAAABJRU5ErkJggg==)](https://pubs.rsna.org/journal/radiographics)\n",
        "\n",
        "\n",
        "To enable machine learning (ML) techniques in NLP, free-form text must be converted to a numerical representation. This hands on exercise reviews the Bag of Words technique and variations that enable the representation of free text documents as feature vectors for training ML algorithms. We will review several preprocessing steps including: \n",
        "1. Tokenization\n",
        "2. Removal of stop words\n",
        "3. Token normalization\n",
        "4. Creation of a master dictionary\n",
        "\n",
        "We then demonstrate several numerical representations of free text, including:\n",
        "1. Simple Bag of Words\n",
        "2. Word Count\n",
        "3. Term Frequency\n",
        "4. Term Frequency * Inverse Document Frequency\n",
        "\n",
        "For this example, we will use a set of clinical statements provided by physicians when requesting radiological exams. Each clinical statement is considered a \"document\". For the purposes of simplicity and clarity, we will work with only three documents in this example, where each document is a clinical statement.\n",
        "\n",
        "##Let's get started!\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H29soKQkd2i-"
      },
      "source": [
        "#Import some basic python libraries and the Natural Language Toolkit\n",
        "'Pandas', 'Math', and the Natural Language Toolkit 'NLTK' are tools that we'll need. They can be readily imported into our project with a few lines of code. \n",
        "We also define here a function that we'll call, \"showDocs\" that we can use to see our documents as they go through the preprocessing steps. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EQOxIUu6MGuH"
      },
      "source": [
        "import pandas as pd\n",
        "import math \n",
        "import nltk\n",
        "nltk.download('stopwords')\n",
        "def showDocs(docs):\n",
        "    for idx in range(len(docs)):\n",
        "        print(f'Doc{idx+1}:', docs[idx])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "snuqxwWmgbiy"
      },
      "source": [
        "# Defining our documents\n",
        "For this exercise, we have three clinical statements, each an individual document. The documents are placed in an array called, \"docs\". The \"showDocs\" function that we defined earlier is then used to provide a clean display of our \"docs\" array. After going through this entire excercise the first time, come back to this step and try changing the text of the documents to see how it affects the following steps."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uSzm-WS6hEsd"
      },
      "source": [
        "docs = [\n",
        "        \"Pancreatic cancer with metastasis. Jaundice with transaminitis, evaluate for obstruction process.\",\n",
        "        \"Pancreatitis. Breast cancer. No output from enteric tube. Assess tube.\",\n",
        "        \"Metastatic pancreatic cancer. Acute renal failure, evaluate for hydronephrosis or obstructive uropathy.\"\n",
        "        ]\n",
        "\n",
        "showDocs(docs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LC38PX5jRcJI"
      },
      "source": [
        "# Tokenization\n",
        "In NLP and linguistics, the process of breaking text into individual meaningful components is known as “tokenization”. One form of tokenization is the identification of individual words in a sentence. The NLTK \"tokenizer\" function splits sentences into individual words and removes punctuation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MCuYLgYSMGuO"
      },
      "source": [
        "tokenizer = nltk.RegexpTokenizer(r\"\\w+\")\n",
        "docs = [tokenizer.tokenize(doc) for doc in docs]\n",
        "\n",
        "showDocs(docs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SXRIRwObRsNJ"
      },
      "source": [
        "# Remove stop words\n",
        "Stop words in NLP are words that occur with the highest frequency in speech and writing within the target language. They are commonly used when constructing sentences and typically don’t carry any special meaning, eg, “the,” “is”. Their removal, therefore, will greatly reduce the size of the dictionary of unique words, a desirable result. The NLTK library includes a list of 127 stop words, including, “from,” “with,” “for,” and “or”. Other words include, notably “no,” “not,” and “if,” which when removed, could drastically change meaning, and therefore caution should be exercised when removing stop words.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nPZZDFe3MGuP"
      },
      "source": [
        "from nltk.corpus import stopwords\n",
        "stopWords = set(stopwords.words('english'))\n",
        "docs = [[word for word in doc if word.lower() not in stopWords] for doc in docs]\n",
        "\n",
        "showDocs(docs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tp7Wn3a7R3yk"
      },
      "source": [
        "# Token normalization\n",
        "Token normalization in NLP is the process of reducing words to a root form so that variations of the same word are recognized as a single entity. One method is known as “stemming,” a rules-based approach in which markers for shades of meaning such as plurality and tense are modified or removed. The stem may be a word itself but need not be so. One of the most widely used stemming algorithms is known as the \"Porter Stemmer,\" that is also available in the NLTK library."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JHuJSFmLMGuQ"
      },
      "source": [
        "from nltk import PorterStemmer\n",
        "ps = PorterStemmer()\n",
        "docs = [([ps.stem(word) for word in doc]) for doc in docs]\n",
        "\n",
        "showDocs(docs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BL20NxzlWnlq"
      },
      "source": [
        "# Create master dictionary\n",
        "To define our dictionary, we take all the tokens remaining after preprocessing from all the documents, and place them into an array called, \"allTokens\". Our feature dictionary is the set of unique words in \"allTokens\" and is obtained by applying a Python function called, \"set\", and place it into a new array we call, \"Dictionary\". We then print the items in \"Dictionary\" and print the count of items in \"Dictionary\" using the \"len\" function.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nUm52LxtMGuQ"
      },
      "source": [
        "allTokens = [word for doc in docs for word in doc]\n",
        "Dictionary = set(allTokens)\n",
        "print(Dictionary)\n",
        "print('Count:' , len(Dictionary))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dZvVDZEnW2MA"
      },
      "source": [
        "# Basic bag of words representation: Term presence\n",
        "This is the simplest representation of the feature vectors, in which the features are given a binary value of \"0\" or \"1,\" depending on absence or presence in a given document.  An explanation of each step in the Python code is beyond the scope of this exercise. A student should appreciate the relatively few lines of code needed to generate the feature vectors and appreciate the various ways of assiging value to terms in documents. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H3zYT2-7Wttv"
      },
      "source": [
        "WordDictDocs = [dict.fromkeys(Dictionary, 0), dict.fromkeys(Dictionary, 0), dict.fromkeys(Dictionary, 0)]\n",
        "for idx in range(len(docs)):\n",
        "    for word in docs[idx]:\n",
        "        WordDictDocs[idx][word]=1\n",
        "arr = pd.DataFrame(WordDictDocs).sort_index(axis=1)\n",
        "arr.index = arr.index.map(lambda idx : 'Doc' + str(idx+1))\n",
        "display(arr)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CUB3a1VBW_7B"
      },
      "source": [
        "# Feature vectors with word counts\n",
        "Term count replaces the binary representation of a feature with a count of the number of times a term is present in the document. The assumption here is that the more times a word appears in a document, the more important it is. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vd7oHYDTMGuR"
      },
      "source": [
        "WordCountDocs = [dict.fromkeys(Dictionary, 0), dict.fromkeys(Dictionary, 0), dict.fromkeys(Dictionary, 0)]\n",
        "for idx in range(len(docs)):\n",
        "    for word in docs[idx]:\n",
        "        WordCountDocs[idx][word]+=1\n",
        "wc = pd.DataFrame(WordCountDocs).sort_index(axis=1)\n",
        "wc.index = wc.index.map(lambda idx : 'Doc' + str(idx+1))\n",
        "display(wc)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rj-QZzNwXbhH"
      },
      "source": [
        "# Feature vectors with Term Frequency (TF)\n",
        "TF is defined as the ratio of the number of times a term appears in a document to the number of terms in the document. TF therefore normalizes the importance given to a word with respect to the size of the document. Like term count, TF places a higher value on words that appear more often in a document, but as the size of a document increases, the TF for any given term will decrease. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WI2ZzqkaMGuR"
      },
      "source": [
        "def computeTF(wordDict, doc):\n",
        "    tfDict = {}\n",
        "    corpusCount = len(doc)\n",
        "    for word, count in wordDict.items():\n",
        "        tfDict[word] = round(count/float(corpusCount),2)\n",
        "    return(tfDict)\n",
        "TFs = []\n",
        "for idx in range(len(docs)):\n",
        "    TFs.append(computeTF(WordCountDocs[idx], docs[idx]))\n",
        "tfDisplay = pd.DataFrame(TFs).sort_index(axis=1)\n",
        "tfDisplay.index = tfDisplay.index.map(lambda idx : 'Doc' + str(idx+1))\n",
        "display(tfDisplay)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6vGAJ1WjXg_6"
      },
      "source": [
        "# Inverse document frequency calculation\n",
        "If a word appears in every document of the corpus, then that word does not help to identify the unique aspects of a given document. It is the unique aspects of a document that are essential to ML prediction models. Inverse document frequency (IDF) is a method used to increase the value of rare words and decrease the value of common words in the corpus.  \n",
        "\n",
        "> IDF = log(# documents in corpus/# documents where term appears)\n",
        "\n",
        "Unlike the previously described metrics, the IDF for any given term applies to the entire corpus, not to any individual document. Is there a word that appears in every document of our corpus? If so, this word will have an IDF value of '0'. Words that are rare will have higher IDF. Words that are common will have lower IDF. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Iklu78iTMGuS"
      },
      "source": [
        "def computeIDF(wordDicts):\n",
        "    idfDict = dict.fromkeys(Dictionary, 0)\n",
        "    N = len(wordDicts)\n",
        "    for wordDict in wordDicts:\n",
        "        for word, present in wordDict.items():\n",
        "            idfDict[word] += present\n",
        "    for word in idfDict.keys():\n",
        "        idfDict[word] = round(math.log10(N / float(idfDict[word])),2)\n",
        "    return(idfDict)\n",
        "IDFs = computeIDF(WordDictDocs)\n",
        "idfDisplay = pd.DataFrame([IDFs]).sort_index(axis=1)\n",
        "idfDisplay.index = idfDisplay.index.map(lambda idx : 'IDF')\n",
        "display(idfDisplay)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eTD6o_DMXnzy"
      },
      "source": [
        "# Feature vectors with Term frequency * Inverse document frequency (TFIDF)\n",
        "To use the IDF to weight the feature vector in any given document, we multiply it with its TF, to obtain a new metric simply described as TFIDF. Have a look at our sample documents again. A term that appears multiple times in one document, but few times in all the other documents will have the highest TFIDF. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MApu6tf2MGuS"
      },
      "source": [
        "def computeTFIDF(tfs, idfs):\n",
        "    tfidf = {}\n",
        "    for word, tf in tfs.items():\n",
        "        tfidf[word] = round(tf*idfs[word],2)\n",
        "    return(tfidf)\n",
        "TF_IDFs = []\n",
        "for idx in range(len(docs)):\n",
        "    TF_IDFs.append(computeTFIDF(TFs[idx], IDFs))\n",
        "dfidfDisplay = pd.DataFrame(TF_IDFs).sort_index(axis=1)\n",
        "dfidfDisplay.index = dfidfDisplay.index.map(lambda idx : 'Doc' + str(idx))\n",
        "display(dfidfDisplay)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LrbywyNkb8Pg"
      },
      "source": [
        "# Next Steps\n",
        "We have defined several numeric representations of free text, known as feature vectors, in three documents. To better understand how Bag of Words and other variations value words, go back to the 'Defining our documents' section and make changes to the documents, including adding or removing documents. Then re-run each step of the code and see how it affects the feature vectors that are generated. "
      ]
    }
  ]
}