radnlp_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RadNLP\n",
    "## *Rad*iology NLP or\n",
    "## *Rad* (as in cool) NLP or\n",
    "## *[Fill in the Blank]* NLP\n",
    "#### &copy; Brian E. Chapman, PhD\n",
    "\n",
    "RadNLP is a package that builds upon the [pyConTextNLP]() algorithm's sentence-level text processing to perform simple document-level classification. The package also contains a number of functions for identifying sections of reports, identifing and eliminating boiler-plate text, etc.\n",
    "\n",
    "In this notebook I will demonstrate radnlp's most basic functionality: given a text of interest create an overall document classification. The classifiction uses a simple maximum function: for each concept marked in a report the maximal schema value occurance is selected to characterize the report for that concept.\n",
    "\n",
    "## Report Schema and *maximal value*\n",
    "\n",
    "The document classification is based on schema that combines multiple concepts (e.g. existence, certitude, severity) into a single ordinal scale. The RadNLP GitHub repository includes a knowledge base directory (KBs) contains the schema we ahve previously developed for critical findings projects. It is included below:\n",
    "\n",
    "```Python\n",
    "# Lines that start with the # symbol are comments and are ignored\n",
    "# The schema consists of a numeric value, followed by a label (e.g. \"AMBIVALENT\"), \n",
    "# followed by a Python express that can evaluate to True or False\n",
    "# The Python expression uses LABELS from the rules. processReports.py will substitute \n",
    "# the LABEL with any matched values identified from \n",
    "# the corresponding rules\n",
    "1,AMBIVALENT,DISEASE_STATE == 2\n",
    "2,Negative/Certain/Acute,DISEASE_STATE == 0 and CERTAINTY_STATE == 1\n",
    "3,Negative/Uncertain/Chronic,DISEASE_STATE == 0 and CERTAINTY_STATE == 0 and ACUTE_STATE == 0\n",
    "4,Positive/Uncertain/Chronic,DISEASE_STATE == 1 and CERTAINTY_STATE == 0 and ACUTE_STATE == 0 \n",
    "5,Positive/Certain/Chronic,DISEASE_STATE == 1 and CERTAINTY_STATE == 1 and ACUTE_STATE == 0 \n",
    "6,Negative/Uncertain/Acute,DISEASE_STATE == 0 and CERTAINTY_STATE == 0 \n",
    "7,Positive/Uncertain/Acute,DISEASE_STATE == 1 and CERTAINTY_STATE == 0 and ACUTE_STATE == 1 \n",
    "8,Positive/Certain/Acute,DISEASE_STATE == 1 and CERTAINTY_STATE == 1 and ACUTE_STATE == 1 \n",
    "```\n",
    "\n",
    "A key idea is **\"a Python express that can evaluate to True or False\"**.\n",
    "\n",
    "The ``radnlp.schema`` subpackage contains functions for reading schema and applying the schema to the pyConTextNLP findings given ``rules`` specified by the user.\n",
    "\n",
    "There are two key functions in ``radnlp.schema``:\n",
    "\n",
    "```Python\n",
    "def instantiate_schema(values, rule):\n",
    "    \"\"\"\n",
    "    evaluates rule by substituting values into rule and evaluating the resulting literal.\n",
    "    This is currently insecure\n",
    "        * \"For security the ast.literal_eval() method should be used.\"\n",
    "    \"\"\"\n",
    "    r = rule\n",
    "    for k in values.keys():\n",
    "        r = r.replace(k, values[k].__str__())\n",
    "    #return ast.literal_eval(r)\n",
    "    return eval(r)\n",
    "\n",
    "def assign_schema(values, rules):\n",
    "    \"\"\"\n",
    "    \"\"\"\n",
    "    for k in rules.keys():\n",
    "        if instantiate_schema(values, rules[k][1]):\n",
    "            return k\n",
    "```\n",
    "\n",
    "For any given category (e.g. ``pulmonary_embolism``), the maximal schema score encountered in the report is selected to characterize that report."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ``radnlp`` Rules\n",
    "\n",
    "``radnlp`` uses rule files to specify rules that define how particular pyConTextNLP findings relate to radnlp concepts. For example, in the ``classificationRules3.csv`` provided in KBs, we provide a rules that state:\n",
    "\n",
    "* The default disease state is 1. \n",
    "* ``PROBABLE_EXISTENCE`` AND  ``DEFINITE_EXISTENCE`` map to a disease state of 1\n",
    "\n",
    "Rules as currently indicated are not quite general and reflect paraticular use cases we were working on.\n",
    "\n",
    "### Types of Rules\n",
    "\n",
    "#### We currently support three rules:\n",
    "\n",
    "1. ``CLASSIFICAITON_RULE``: these are the rules that relate to disease, temporality, and certainty\n",
    "1. ``CATEGORY_RULE``: these are only partially developed concepts that attempt to address combinatorics problems in pyConTextNLP by making default findings more general (e.g. ``infaract``) and then tries to create more specific findings by attaching anatomic locations to the findings (e.g. an ``infarct`` becomes a critical finding when attached to an anatomic concept like ``brain_anatomy`` or ``heart_anatomy``.\n",
    "1. ``SEVERITY_RULE``: Again, not fully developed but relates to extracting severity concepts (e.g. ``large`` or 4.3 cm).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```Python\n",
    "# Lines that start with the # symbol are comments and are ignored,,,,,,,,,,,,,\n",
    "# processReport current has three types of rules: @CLASSIFICATION_RULE, @CATEGORY_RULE, and @SEVERITY_RULE,,,,,,,,,,,\n",
    "# classification rules would be for things like disease_state, certainty_state, temporality state,,,,,,,,,,,\n",
    "# For each classification_rule set,\" there is a rule label (e.g. \"\"DISEASE_STATE\"\". This must match\",,,,,,,,,,,,\n",
    "# the terms used in the schema file,,,,,,,,,,,,,\n",
    "# Each rule set requires a DEFAULT which is the schema value to be returned if no rule conditions are satisifed,,,,,,,,,,,,,\n",
    "# Each rule set has zero or more rules consisting of a schema value to be returned if the rule evaluates to true,,,,,,,,,,,,,\n",
    "# A rule evalutes to true if the target is modified by one or more of the ConText CATEGORIES listed following,,,,,,,,,,,,,\n",
    "@CLASSIFICATION_RULE,DISEASE_STATE,RULE,0,DEFINITE_NEGATED_EXISTENCE,PROBABLE_NEGATED_EXISTENCE,FUTURE,INDICATION,PSEUDONEG,,,,,\n",
    "@CLASSIFICATION_RULE,DISEASE_STATE,RULE,2,AMBIVALENT_EXISTENCE,,,,,,,,,\n",
    "@CLASSIFICATION_RULE,DISEASE_STATE,RULE,1,PROBABLE_EXISTENCE,DEFINITE_EXISTENCE,,,,,,,,\n",
    "@CLASSIFICATION_RULE,DISEASE_STATE,DEFAULT,1,,,,,,,,,,\n",
    "@CLASSIFICATION_RULE,CERTAINTY_STATE,RULE,0,PROBABLE_NEGATED_EXISTENCE,AMBIVALENT_EXISTENCE,PROBABLE_EXISTENCE,,,,,,,\n",
    "@CLASSIFICATION_RULE,CERTAINTY_STATE,DEFAULT,1,,,,,,,,,,\n",
    "@CLASSIFICATION_RULE,ACUTE_STATE,RULE,0,HISTORICAL,,,,,,,,,\n",
    "@CLASSIFICATION_RULE,ACUTE_STATE,DEFAULT,1,,,,,,,,,,\n",
    "#CATEGORY_RULE rules specify what Findings (e.g. DVT) can have the category modified by the following ANATOMIC modifies,,,,,,,,,,,,,\n",
    "@CATEGORY_RULE,DVT,LOWER_DEEP_VEIN,UPPER_DEEP_VEIN,HEPATIC_VEIN,PORTAL_SYSTEM_VEIN,PULMONARY_VEIN,RENAL_VEIN,SINUS_VEIN,LOWER_SUPERFICIAL_VEIN,UPPER_SUPERFICIAL_VEIN,VARICOCELE,ARTERIAL,NON_VASCULAR\n",
    "@CATEGORY_RULE,INFARCT,BRAIN_ANATOMY,HEART_ANATOMY,OTHER_CRITICAL_ANATOMY,,,,,,,,,\n",
    "@CATEGORY_RULE,ANEURYSM,AORTIC_ANATOMY,,,,,,,,,,,\n",
    "#SEVERITY_RUlE specifiy which targets to try to obtain severity measures for,,,,,,,,,,,,,\n",
    "@SEVERITY_RULE,AORTIC_ANATOMY_ANEURYSM,SEVERITY,,,,,,,,,,,\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Licensing\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at\n",
    "\n",
    "[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software\n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "See the License for the specific language governing permissions and\n",
    "limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Program Description"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import radnlp.rules as rules\n",
    "import radnlp.schema as schema\n",
    "import radnlp.utils as utils\n",
    "import radnlp.classifier as classifier\n",
    "import radnlp.split as split\n",
    "\n",
    "from IPython.display import clear_output, display, HTML\n",
    "from IPython.html.widgets import interact, interactive, fixed\n",
    "import io\n",
    "from IPython.html import widgets # Widget definitions\n",
    "import pyConTextNLP.itemData as itemData\n",
    "\n",
    "from pyConTextNLP.display.html import mark_document_with_html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example Data\n",
    "\n",
    "Below are two example radiology reports pulled from the MIMIC2 demo data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reports = [\"\"\"1.  Pulmonary embolism with  filling defects noted within the upper and lower\n",
    "     lobar branches of the right main pulmonary artery.\n",
    "     2.  Bilateral pleural effusions, greater on the left.\n",
    "     3.  Ascites.\n",
    "     4.  There is edema of the gallbladder wall, without any evidence of\n",
    "     distention, intra- or extra-hepatic biliary dilatation.  This, along with\n",
    "     stranding within the mesentery, likely represents third spacing of fluid.\n",
    "     5.  There are several wedge shaped areas of decreased perfusion within the\n",
    "     spleen, which may represent splenic infarcts.\n",
    "     \n",
    "     Results were discussed with Dr. [**First Name8 (NamePattern2) 15561**] [**Last Name (NamePattern1) 13459**] \n",
    "     at 8 pm on [**3099-11-6**].\"\"\",\n",
    "           \n",
    "    \"\"\"1. Filling defects within the subsegmental arteries in the region\n",
    "     of the left lower lobe and lingula and within the right lower lobe consistent\n",
    "     with pulmonary emboli.\n",
    "     \n",
    "     2. Small bilateral pleural effusions with associated bibasilar atelectasis.\n",
    "     \n",
    "     3. Left anterior pneumothorax.\n",
    "     \n",
    "     4. No change in the size of the thoracoabdominal aortic aneurysm.\n",
    "     \n",
    "     5. Endotracheal tube 1.8 cm above the carina. NG tube within the stomach,\n",
    "     although the tip is pointed superiorly toward the fundus.\"\"\",\n",
    "           \n",
    "    \"\"\"1. There are no pulmonary emboli observed.\n",
    "     \n",
    "     2. Small bilateral pleural effusions with associated bibasilar atelectasis.\n",
    "     \n",
    "     3. Left anterior pneumothorax.\n",
    "     \n",
    "     4. No change in the size of the thoracoabdominal aortic aneurysm.\n",
    "     \n",
    "     5. Endotracheal tube 1.8 cm above the carina. NG tube within the stomach,\n",
    "     although the tip is pointed superiorly toward the fundus.\"\"\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#!python -m textblob.download_corpora"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define locations of knowledge, schema, and rules files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def getOptions():\n",
    "    \"\"\"Generates arguments for specifying database and other parameters\"\"\"\n",
    "    options = {}\n",
    "    options['lexical_kb'] = [\"https://raw.githubusercontent.com/chapmanbe/pyConTextNLP/master/KB/lexical_kb_04292013.tsv\", \n",
    "                             \"https://raw.githubusercontent.com/chapmanbe/pyConTextNLP/master/KB/criticalfinder_generalized_modifiers.tsv\"]\n",
    "    options['domain_kb'] = [\"https://raw.githubusercontent.com/chapmanbe/pyConTextNLP/master/KB/pe_kb.tsv\"]#[os.path.join(DATADIR2,\"pe_kb.tsv\")]\n",
    "    options[\"schema\"] = \"https://raw.githubusercontent.com/chapmanbe/RadNLP/master/KBs/schema2.csv\"#\"file specifying schema\"\n",
    "    options[\"rules\"] = \"https://raw.githubusercontent.com/chapmanbe/RadNLP/master/KBs/classificationRules3.csv\" # \"file specifying sentence level rules\")\n",
    "    return options\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define report analysis\n",
    "\n",
    "For every report we do two steps\n",
    "\n",
    "1. Markup all the sentences in the report based on the provided targets and modifiers\n",
    "1. Given this markup we apply our rules and schema to generate a document classification.\n",
    "\n",
    "``radnlp`` provides functions to do both of these steps:\n",
    "\n",
    "1. ``radnlp.utils.mark_report`` takes lists of modifiers and targets and generates a pyConTextNLP document graph\n",
    "1. ``radnlp.classify.classify_document_targets`` takes the document graph, rules, and schema and generates document classification for each identified concept.\n",
    "\n",
    "Because pyConTextNLP operates on sentences we split the report into sentences. In this function we use ``radnlp.split.get_sentences`` which is simply a wrapper around ``textblob`` for splitting the sentences.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def analyze_report(report, modifiers, targets, rules, schema):\n",
    "    \"\"\"\n",
    "    given an individual radiology report, creates a pyConTextGraph\n",
    "    object that contains the context markup\n",
    "    report: a text string containing the radiology reports\n",
    "    \"\"\"\n",
    "    \n",
    "    markup = utils.mark_report(split.get_sentences(report),\n",
    "                         modifiers,\n",
    "                         targets)\n",
    "    return  classifier.classify_document_targets(markup,\n",
    "                                          rules[0],\n",
    "                                          rules[1],\n",
    "                                          rules[2],\n",
    "                                          schema)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def process_report(report):\n",
    "    \n",
    "    options = getOptions()\n",
    "\n",
    "    _radnlp_rules = rules.read_rules(options[\"rules\"])\n",
    "    _schema = schema.read_schema(options[\"schema\"])\n",
    "    #_schema = readSchema(options[\"schema\"])\n",
    "    modifiers = itemData.itemData()\n",
    "    targets = itemData.itemData()\n",
    "    for kb in options['lexical_kb']:\n",
    "        modifiers.extend( itemData.instantiateFromCSVtoitemData(kb) )\n",
    "    for kb in options['domain_kb']:\n",
    "        targets.extend( itemData.instantiateFromCSVtoitemData(kb) )\n",
    "    return analyze_report(report, modifiers, targets, _radnlp_rules, _schema)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rslt_0 = process_report(reports[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``radnlp.classifier.classify_document_targets`` returns a dictionary with keys equal to the target category (e.g. ``pulmonary_embolism``) and the values a 3-tuple with the following values:\n",
    "\n",
    "1. The schema category (e.g. 8 or 2).\n",
    "1. The XML representation of the maximal schema node\n",
    "1. A list (usually empty (not really implemented yet)) of severity values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for key, value in rslt_0.items():\n",
    "    print((\"%s\"%key).center(42,\"-\"))\n",
    "    for v in value:\n",
    "        print(v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rslt_1 = main(reports[1])\n",
    "\n",
    "for key, value in rslt_1.items():\n",
    "    print((\"%s\"%key).center(42,\"-\"))\n",
    "    for v in value:\n",
    "        print(v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Negative Report\n",
    "\n",
    "For the third report I simply rewrote one of the findings to be negative for PE. We now see a change in the schema classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rslt_2 = main(reports[2])\n",
    "\n",
    "for key, value in rslt_2.items():\n",
    "    print((\"%s\"%key).center(42,\"-\"))\n",
    "    for v in value:\n",
    "        print(v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "keys = list(pec.markups.keys())\n",
    "keys.sort()\n",
    "\n",
    "pec.reports.insert(pec.reports.columns.get_loc(u'markup')+1,\n",
    "                   \"ConText Coding\",\n",
    "                   [codingKey.get(pec.markups[k][1].get(\"pulmonary_embolism\",[None])[0],\"NA\") for k in keys])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}