Medical_Sheet_ingestion_v1_AJ.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jarasch/medical_sheet_ingestion_via_GenAI/blob/main/Medical_Sheet_ingestion_v1_AJ.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "26f65668",
      "metadata": {
        "id": "26f65668"
      },
      "source": [
        "# LLM Powered Medical Case Sheet Ingestion\n",
        "## Outline\n",
        "1. Data Cleansing\n",
        "2. Prompt Definition\n",
        "3. Entity & Relationship Extraction\n",
        "4. Neo4j Cypher Generation\n",
        "5. Data Ingestion"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "61138c34",
      "metadata": {
        "id": "61138c34"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "%pip install graphdatascience\n",
        "%pip install openai=0.28\n",
        "%pip install python-dotenv\n",
        "%pip install retry"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c2c97f1b",
      "metadata": {
        "id": "c2c97f1b"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import openai\n",
        "from retry import retry\n",
        "import re\n",
        "from string import Template\n",
        "import json\n",
        "import ast\n",
        "import time\n",
        "import pandas as pd\n",
        "from graphdatascience import GraphDataScience\n",
        "import glob\n",
        "from timeit import default_timer as timer\n",
        "from dotenv import load_dotenv"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7d70b41f",
      "metadata": {
        "id": "7d70b41f"
      },
      "source": [
        "## Data Cleansing"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d89595dc",
      "metadata": {
        "id": "d89595dc"
      },
      "source": [
        "First, let's define a function that can help clean the input data. For the sake of simplicity, lets keep it simple. In the corpus, the data refers to some Figures like scan images. We dont have them and so will remove any such references."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "af73c9ce",
      "metadata": {
        "id": "af73c9ce"
      },
      "outputs": [],
      "source": [
        "def clean_text(text):\n",
        "  clean = \"\\n\".join([row for row in text.split(\"\\n\")])\n",
        "  clean = re.sub(r'\\(fig[^)]*\\)', '', clean, flags=re.IGNORECASE)\n",
        "  return clean"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5326d17d",
      "metadata": {
        "id": "5326d17d"
      },
      "source": [
        "Let's take this case sheet and extract entities and relations using LLM"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "09861c5f",
      "metadata": {
        "id": "09861c5f"
      },
      "outputs": [],
      "source": [
        "article_txt = \"\"\"The patient was a 34-yr-old man who presented with complaints of fever and a chronic cough.\n",
        "He was a smoker and had a history of pulmonary tuberculosis that had been treated and cured.\n",
        "A computed tomographic (CT) scan revealed multiple tiny nodules in both lungs.\n",
        "A thoracoscopic lung biopsy was taken from the right upper lobe.\n",
        "The microscopic examination revealed a typical LCH.\n",
        "The tumor cells had vesicular and grooved nuclei, and they formed small aggregations around the bronchioles (Fig.1).\n",
        "The tumor cells were strongly positive for S-100 protein, vimentin, CD68 and CD1a.\n",
        "There were infiltrations of lymphocytes and eosinophils around the tumor cells.\n",
        "With performing additional radiologic examinations, no other organs were thought to be involved.\n",
        "He quit smoking, but he received no other specific treatment.\n",
        "He was well for the following one year.\n",
        "After this, a follow-up CT scan was performed and it showed a 4 cm-sized mass in the left lower lobe, in addition to the multiple tiny nodules in both lungs (Fig.2).\n",
        "A needle biopsy specimen revealed the possibility of a sarcoma; therefore, a lobectomy was performed.\n",
        "Grossly, a 4 cm-sized poorly-circumscribed lobulated gray-white mass was found (Fig.3), and there were a few small satellite nodules around the main mass.\n",
        "Microscopically, the tumor cells were aggregated in large sheets and they showed an infiltrative growth.\n",
        "The cytologic features of some of the tumor cells were similar to those seen in a typical LCH.\n",
        "However, many tumor cells showed overtly malignant cytologic features such as pleomorphic/hyperchromatic nuclei and prominent nucleoli (Fig.4), and multinucleated tumor giant cells were also found.\n",
        "There were numerous mitotic figures ranging from 30 to 60 per 10 high power fields, and some of them were abnormal.\n",
        "A few foci of typical LCH remained around the main tumor mass.\n",
        "Immunohistochemically, the tumor cells were strongly positive for S-100 protein (Fig.5) and vimentin; they were also positive for CD68 (Dako N1577, Clone KPI), and focally positive for CD1a (Fig.6), and they were negative for cytokeratin, epithelial membrane antigen, CD3, CD20 and HMB45.\n",
        "The ultrastructural analysis failed to demonstrate any Birbeck granules in the cytoplasm of the tumor cells.\n",
        "Now, at five months after lobectomy, the patient is doing well with no significant change in the radiologic findings.\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bd3236df",
      "metadata": {
        "id": "bd3236df"
      },
      "source": [
        "## Prompt Definition"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "45c73e3d",
      "metadata": {
        "id": "45c73e3d"
      },
      "source": [
        "**⚠️** You need to duplicate `config.env.example` file in the left and rename as `config.env`. Edit the values in this file and provide the values for API keys and Neo4j credentials"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "os.environ[\"OPENAI_API_KEY\"] = 'PASTE YOUR OPENAi KEY HERE'\n",
        "openai.api_key = os.getenv('OPENAI_API_KEY')"
      ],
      "metadata": {
        "id": "IGjag-WrCa57"
      },
      "id": "IGjag-WrCa57",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f94b81b2",
      "metadata": {
        "id": "f94b81b2"
      },
      "outputs": [],
      "source": [
        "\n",
        "\n",
        "openai.api_key = os.getenv('OPENAI_API_KEY')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ff224199",
      "metadata": {
        "id": "ff224199"
      },
      "source": [
        "This is a helper function to talk to the LLM with our prompt and text input"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1d8592a3",
      "metadata": {
        "id": "1d8592a3"
      },
      "outputs": [],
      "source": [
        "# GPT-4 Prompt to complete\n",
        "@retry(tries=2, delay=5)\n",
        "def process_gpt(system,\n",
        "                prompt):\n",
        "\n",
        "    completion = openai.ChatCompletion.create(\n",
        "        # engine=\"gpt-3.5-turbo\",\n",
        "        model=\"gpt-4\",\n",
        "        max_tokens=2500,\n",
        "        # Try to be as deterministic as possible\n",
        "        temperature=0,\n",
        "        messages=[\n",
        "            {\"role\": \"system\", \"content\": system},\n",
        "            {\"role\": \"user\", \"content\": prompt},\n",
        "        ]\n",
        "    )\n",
        "    nlp_results = completion.choices[0].message.content\n",
        "    return nlp_results"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a52639cd",
      "metadata": {
        "id": "a52639cd"
      },
      "source": [
        "This is a simple prompt to start with. If the processing is very complex, you can also chain the prompts as and when required. I am going to use a single prompt here that helps me to extract the text strictly as per the Entities and Relationships defined. This is a simplification. In the real scenario, especially with medical records, you have to leverage on Domain experts to define the Ontology systematically and capture the important information. You might also be fine-tuning the model as and when required.\n",
        "\n",
        "Also, instead of one single large model, you can also consider chaining a number of smaller ones as per your needs.\n",
        "\n",
        "We are going with this Graph Schema for our Case Sheet:\n",
        "![schema.png](attachment:d16d5097-785c-4343-8a48-289517fca944.png)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "90379cd1",
      "metadata": {
        "id": "90379cd1"
      },
      "outputs": [],
      "source": [
        "prompt1=\"\"\"From the Case sheet for a patient below, extract the following Entities & relationships described in the mentioned format\n",
        "0. ALWAYS FINISH THE OUTPUT. Never send partial responses\n",
        "1. First, look for these Entity types in the text and generate as comma-separated format similar to entity type.\n",
        "   `id` property of each entity must be alphanumeric and must be unique among the entities. You will be referring this property to define the relationship between entities. Do not create new entity types that aren't mentioned below. Document must be summarized and stored inside Case entity under `summary` property.\n",
        "   You will have to generate as many entities as needed as per the types below:\n",
        "    Entity Types:\n",
        "    label:'Case',id:string,summary:string //Case\n",
        "    label:'Person',id:string,age:string,location:string,gender:string //Patient mentioned in the case\n",
        "    label:'Symptom',id:string,description:string //Symptom Entity; `id` property is the name of the symptom, in lowercase & camel-case & should always start with an alphabet\n",
        "    label:'Disease',id:string,name:string //Disease diagnosed now or previously as per the Case sheet; `id` property is the name of the disease, in lowercase & camel-case & should always start with an alphabet\n",
        "    label:'BodySystem',id:string,name:string //Body Part affected. Eg: Chest, Lungs; id property is the name of the part, in lowercase & camel-case & should always start with an alphabet\n",
        "    label:'Diagnosis',id:string,name:string,description:string,when:string //Diagnostic procedure conducted; `id` property is the summary of the Diagnosis, in lowercase & camel-case & should always start with an alphabet\n",
        "    label:'Biological',id:string,name:string,description:string //Results identified from Diagnosis; `id` property is the summary of the Biological, in lowercase & camel-case & should always start with an alphabet\n",
        "\n",
        "3. Next generate each relationships as triples of head, relationship and tail. To refer the head and tail entity, use their respective `id` property. Relationship property should be mentioned within brackets as comma-separated. They should follow these relationship types below. You will have to generate as many relationships as needed as defined below:\n",
        "    Relationship types:\n",
        "    case|FOR|person\n",
        "    person|HAS_SYMPTOM{when:string,frequency:string,span:string}|symptom //the properties inside HAS_SYMPTOM gets populated from the Case sheet\n",
        "    person|HAS_DISEASE{when:string}|disease //the properties inside HAS_DISEASE gets populated from the Case sheet\n",
        "    symptom|SEEN_ON|chest\n",
        "    disease|AFFECTS|heart\n",
        "    person|HAS_DIAGNOSIS|diagnosis\n",
        "    diagnosis|SHOWED|biological\n",
        "\n",
        "The output should look like :\n",
        "{\n",
        "    \"entities\": [{\"label\":\"Case\",\"id\":string,\"summary\":string}],\n",
        "    \"relationships\": [\"disease|AFFECTS|heart\"]\n",
        "}\n",
        "\n",
        "Case Sheet:\n",
        "$ctext\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a6a20e1c",
      "metadata": {
        "id": "a6a20e1c"
      },
      "source": [
        "Let's run our completion task with our LLM"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "98cbafb5",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "98cbafb5",
        "outputId": "364ddfb7-256a-4594-daaa-08309b5bc025"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 501 ms, sys: 68.6 ms, total: 570 ms\n",
            "Wall time: 1min 56s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "def run_completion(prompt, results, ctext):\n",
        "    try:\n",
        "      system = \"You are a helpful Medical Case Sheet expert who extracts relevant information and store them on a Neo4j Knowledge Graph\"\n",
        "      pr = Template(prompt).substitute(ctext=ctext)\n",
        "      res = process_gpt(system, pr)\n",
        "      results.append(json.loads(res.replace(\"\\'\", \"'\")))\n",
        "      return results\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "prompts = [prompt1]\n",
        "results = []\n",
        "for p in prompts:\n",
        "  results = run_completion(p, results, clean_text(article_txt))\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "results"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AamYyrvGbr66",
        "outputId": "5fd46704-9596-42bf-9768-0c7cdc17761b"
      },
      "id": "AamYyrvGbr66",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'entities': [{'label': 'Case',\n",
              "    'id': 'case1',\n",
              "    'summary': 'The patient was a 34-yr-old man who presented with complaints of fever and a chronic cough. He was a smoker and had a history of pulmonary tuberculosis that had been treated and cured. A computed tomographic (CT) scan revealed multiple tiny nodules in both lungs. A thoracoscopic lung biopsy was taken from the right upper lobe. The microscopic examination revealed a typical LCH. The tumor cells had vesicular and grooved nuclei, and they formed small aggregations around the bronchioles. The tumor cells were strongly positive for S-100 protein, vimentin, CD68 and CD1a. There were infiltrations of lymphocytes and eosinophils around the tumor cells. With performing additional radiologic examinations, no other organs were thought to be involved. He quit smoking, but he received no other specific treatment. He was well for the following one year. After this, a follow-up CT scan was performed and it showed a 4 cm-sized mass in the left lower lobe, in addition to the multiple tiny nodules in both lungs. A needle biopsy specimen revealed the possibility of a sarcoma; therefore, a lobectomy was performed. Grossly, a 4 cm-sized poorly-circumscribed lobulated gray-white mass was found , and there were a few small satellite nodules around the main mass. Microscopically, the tumor cells were aggregated in large sheets and they showed an infiltrative growth. The cytologic features of some of the tumor cells were similar to those seen in a typical LCH. However, many tumor cells showed overtly malignant cytologic features such as pleomorphic/hyperchromatic nuclei and prominent nucleoli , and multinucleated tumor giant cells were also found. There were numerous mitotic figures ranging from 30 to 60 per 10 high power fields, and some of them were abnormal. A few foci of typical LCH remained around the main tumor mass. Immunohistochemically, the tumor cells were strongly positive for S-100 protein  and vimentin; they were also positive for CD68 (Dako N1577, Clone KPI), and focally positive for CD1a , and they were negative for cytokeratin, epithelial membrane antigen, CD3, CD20 and HMB45. The ultrastructural analysis failed to demonstrate any Birbeck granules in the cytoplasm of the tumor cells. Now, at five months after lobectomy, the patient is doing well with no significant change in the radiologic findings.'},\n",
              "   {'label': 'Person',\n",
              "    'id': 'person1',\n",
              "    'age': '34',\n",
              "    'location': 'unknown',\n",
              "    'gender': 'male'},\n",
              "   {'label': 'Symptom', 'id': 'fever', 'description': 'fever'},\n",
              "   {'label': 'Symptom', 'id': 'chronicCough', 'description': 'chronic cough'},\n",
              "   {'label': 'Disease',\n",
              "    'id': 'pulmonaryTuberculosis',\n",
              "    'name': 'Pulmonary Tuberculosis'},\n",
              "   {'label': 'Disease', 'id': 'lch', 'name': 'LCH'},\n",
              "   {'label': 'Disease', 'id': 'sarcoma', 'name': 'Sarcoma'},\n",
              "   {'label': 'BodySystem', 'id': 'lungs', 'name': 'Lungs'},\n",
              "   {'label': 'Diagnosis',\n",
              "    'id': 'ctScan',\n",
              "    'name': 'CT Scan',\n",
              "    'description': 'Revealed multiple tiny nodules in both lungs',\n",
              "    'when': 'unknown'},\n",
              "   {'label': 'Diagnosis',\n",
              "    'id': 'thoracoscopicLungBiopsy',\n",
              "    'name': 'Thoracoscopic Lung Biopsy',\n",
              "    'description': 'Taken from the right upper lobe',\n",
              "    'when': 'unknown'},\n",
              "   {'label': 'Diagnosis',\n",
              "    'id': 'needleBiopsy',\n",
              "    'name': 'Needle Biopsy',\n",
              "    'description': 'Revealed the possibility of a sarcoma',\n",
              "    'when': 'unknown'},\n",
              "   {'label': 'Biological',\n",
              "    'id': 'tumorCells',\n",
              "    'name': 'Tumor Cells',\n",
              "    'description': 'Strongly positive for S-100 protein, vimentin, CD68 and CD1a'}],\n",
              "  'relationships': ['case1|FOR|person1',\n",
              "   \"person1|HAS_SYMPTOM{when:'unknown',frequency:'unknown',span:'unknown'}|fever\",\n",
              "   \"person1|HAS_SYMPTOM{when:'unknown',frequency:'unknown',span:'unknown'}|chronicCough\",\n",
              "   \"person1|HAS_DISEASE{when:'past'}|pulmonaryTuberculosis\",\n",
              "   \"person1|HAS_DISEASE{when:'present'}|lch\",\n",
              "   \"person1|HAS_DISEASE{when:'present'}|sarcoma\",\n",
              "   'fever|SEEN_ON|lungs',\n",
              "   'chronicCough|SEEN_ON|lungs',\n",
              "   'pulmonaryTuberculosis|AFFECTS|lungs',\n",
              "   'lch|AFFECTS|lungs',\n",
              "   'sarcoma|AFFECTS|lungs',\n",
              "   'person1|HAS_DIAGNOSIS|ctScan',\n",
              "   'person1|HAS_DIAGNOSIS|thoracoscopicLungBiopsy',\n",
              "   'person1|HAS_DIAGNOSIS|needleBiopsy',\n",
              "   'ctScan|SHOWED|tumorCells',\n",
              "   'thoracoscopicLungBiopsy|SHOWED|tumorCells',\n",
              "   'needleBiopsy|SHOWED|tumorCells']}]"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dc40d897",
      "metadata": {
        "id": "dc40d897"
      },
      "source": [
        "## Neo4j Cypher Generation"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ee2a861f",
      "metadata": {
        "id": "ee2a861f"
      },
      "source": [
        "The entities & relationships we got from the LLM have to be transformed to Cypher so we can ingest into Neo4j"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7b7fe8c3",
      "metadata": {
        "id": "7b7fe8c3"
      },
      "outputs": [],
      "source": [
        "#pre-processing results for uploading into Neo4j - helper function:\n",
        "def get_prop_str(prop_dict, _id):\n",
        "    s = []\n",
        "    for key, val in prop_dict.items():\n",
        "      if key != 'label' and key != 'id':\n",
        "         s.append(_id+\".\"+key+' = \"'+str(val).replace('\\\"', '\"').replace('\"', '\\\"')+'\"')\n",
        "    return ' ON CREATE SET ' + ','.join(s)\n",
        "\n",
        "def get_cypher_compliant_var(_id):\n",
        "    return \"_\"+ re.sub(r'[\\W_]', '', _id)\n",
        "\n",
        "def generate_cypher(in_json):\n",
        "    e_map = {}\n",
        "    e_stmt = []\n",
        "    r_stmt = []\n",
        "    e_stmt_tpl = Template(\"($id:$label{id:'$key'})\")\n",
        "    r_stmt_tpl = Template(\"\"\"\n",
        "      MATCH $src\n",
        "      MATCH $tgt\n",
        "      MERGE ($src_id)-[:$rel]->($tgt_id)\n",
        "    \"\"\")\n",
        "    for obj in in_json:\n",
        "      for j in obj['entities']:\n",
        "          props = ''\n",
        "          label = j['label']\n",
        "          id = j['id']\n",
        "          if label == 'Case':\n",
        "                id = 'c'+str(time.time_ns())\n",
        "          elif label == 'Person':\n",
        "                id = 'p'+str(time.time_ns())\n",
        "          varname = get_cypher_compliant_var(j['id'])\n",
        "          stmt = e_stmt_tpl.substitute(id=varname, label=label, key=id)\n",
        "          e_map[varname] = stmt\n",
        "          e_stmt.append('MERGE '+ stmt + get_prop_str(j, varname))\n",
        "\n",
        "      for st in obj['relationships']:\n",
        "          rels = st.split(\"|\")\n",
        "          src_id = get_cypher_compliant_var(rels[0].strip())\n",
        "          rel = rels[1].strip()\n",
        "          tgt_id = get_cypher_compliant_var(rels[2].strip())\n",
        "          stmt = r_stmt_tpl.substitute(\n",
        "              src_id=src_id, tgt_id=tgt_id, src=e_map[src_id], tgt=e_map[tgt_id], rel=rel)\n",
        "\n",
        "          r_stmt.append(stmt)\n",
        "\n",
        "    return e_stmt, r_stmt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "34f74bf0",
      "metadata": {
        "id": "34f74bf0"
      },
      "outputs": [],
      "source": [
        "ent_cyp, rel_cyp = generate_cypher(results)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "ent_cyp"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a8zDIAjYcDfH",
        "outputId": "ef07b70a-d710-4d23-c9ea-a0ee98ff6f33"
      },
      "id": "a8zDIAjYcDfH",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['MERGE (_case1:Case{id:\\'c1695915443385559648\\'}) ON CREATE SET _case1.summary = \"The patient was a 34-yr-old man who presented with complaints of fever and a chronic cough. He was a smoker and had a history of pulmonary tuberculosis that had been treated and cured. A computed tomographic (CT) scan revealed multiple tiny nodules in both lungs. A thoracoscopic lung biopsy was taken from the right upper lobe. The microscopic examination revealed a typical LCH. The tumor cells had vesicular and grooved nuclei, and they formed small aggregations around the bronchioles. The tumor cells were strongly positive for S-100 protein, vimentin, CD68 and CD1a. There were infiltrations of lymphocytes and eosinophils around the tumor cells. With performing additional radiologic examinations, no other organs were thought to be involved. He quit smoking, but he received no other specific treatment. He was well for the following one year. After this, a follow-up CT scan was performed and it showed a 4 cm-sized mass in the left lower lobe, in addition to the multiple tiny nodules in both lungs. A needle biopsy specimen revealed the possibility of a sarcoma; therefore, a lobectomy was performed. Grossly, a 4 cm-sized poorly-circumscribed lobulated gray-white mass was found , and there were a few small satellite nodules around the main mass. Microscopically, the tumor cells were aggregated in large sheets and they showed an infiltrative growth. The cytologic features of some of the tumor cells were similar to those seen in a typical LCH. However, many tumor cells showed overtly malignant cytologic features such as pleomorphic/hyperchromatic nuclei and prominent nucleoli , and multinucleated tumor giant cells were also found. There were numerous mitotic figures ranging from 30 to 60 per 10 high power fields, and some of them were abnormal. A few foci of typical LCH remained around the main tumor mass. Immunohistochemically, the tumor cells were strongly positive for S-100 protein  and vimentin; they were also positive for CD68 (Dako N1577, Clone KPI), and focally positive for CD1a , and they were negative for cytokeratin, epithelial membrane antigen, CD3, CD20 and HMB45. The ultrastructural analysis failed to demonstrate any Birbeck granules in the cytoplasm of the tumor cells. Now, at five months after lobectomy, the patient is doing well with no significant change in the radiologic findings.\"',\n",
              " 'MERGE (_person1:Person{id:\\'p1695915443385804393\\'}) ON CREATE SET _person1.age = \"34\",_person1.location = \"unknown\",_person1.gender = \"male\"',\n",
              " 'MERGE (_fever:Symptom{id:\\'fever\\'}) ON CREATE SET _fever.description = \"fever\"',\n",
              " 'MERGE (_chronicCough:Symptom{id:\\'chronicCough\\'}) ON CREATE SET _chronicCough.description = \"chronic cough\"',\n",
              " 'MERGE (_pulmonaryTuberculosis:Disease{id:\\'pulmonaryTuberculosis\\'}) ON CREATE SET _pulmonaryTuberculosis.name = \"Pulmonary Tuberculosis\"',\n",
              " 'MERGE (_lch:Disease{id:\\'lch\\'}) ON CREATE SET _lch.name = \"LCH\"',\n",
              " 'MERGE (_sarcoma:Disease{id:\\'sarcoma\\'}) ON CREATE SET _sarcoma.name = \"Sarcoma\"',\n",
              " 'MERGE (_lungs:BodySystem{id:\\'lungs\\'}) ON CREATE SET _lungs.name = \"Lungs\"',\n",
              " 'MERGE (_ctScan:Diagnosis{id:\\'ctScan\\'}) ON CREATE SET _ctScan.name = \"CT Scan\",_ctScan.description = \"Revealed multiple tiny nodules in both lungs\",_ctScan.when = \"unknown\"',\n",
              " 'MERGE (_thoracoscopicLungBiopsy:Diagnosis{id:\\'thoracoscopicLungBiopsy\\'}) ON CREATE SET _thoracoscopicLungBiopsy.name = \"Thoracoscopic Lung Biopsy\",_thoracoscopicLungBiopsy.description = \"Taken from the right upper lobe\",_thoracoscopicLungBiopsy.when = \"unknown\"',\n",
              " 'MERGE (_needleBiopsy:Diagnosis{id:\\'needleBiopsy\\'}) ON CREATE SET _needleBiopsy.name = \"Needle Biopsy\",_needleBiopsy.description = \"Revealed the possibility of a sarcoma\",_needleBiopsy.when = \"unknown\"',\n",
              " 'MERGE (_tumorCells:Biological{id:\\'tumorCells\\'}) ON CREATE SET _tumorCells.name = \"Tumor Cells\",_tumorCells.description = \"Strongly positive for S-100 protein, vimentin, CD68 and CD1a\"']"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "id": "13588341",
      "metadata": {
        "id": "13588341"
      },
      "source": [
        "## Data Ingestion"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "beacbd25",
      "metadata": {
        "id": "beacbd25"
      },
      "outputs": [],
      "source": [
        "connectionUrl = os.getenv('NEO4J_CONN_URL')\n",
        "username = os.getenv('NEO4J_USER')\n",
        "password = os.getenv('NEO4J_PASSWORD')"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Get your Sandbox credentials and enter them here below\n",
        "\n",
        "connectionUrl = 'bolt://44.200.0.59:7687'\n",
        "username = 'neo4j'\n",
        "password = 'digestion-dispatches-strikers'"
      ],
      "metadata": {
        "id": "p_ZlsuLtcj8C"
      },
      "id": "p_ZlsuLtcj8C",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "bfe9acfc",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "bfe9acfc",
        "outputId": "233fe066-029b-470d-b8d2-91f7734ac0d4"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'2.4.5'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 29
        }
      ],
      "source": [
        "gds = GraphDataScience(connectionUrl, auth=(username, password))\n",
        "gds.version()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cd579588",
      "metadata": {
        "id": "cd579588"
      },
      "source": [
        "Before loading the data, create constraints as below"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f1d546fd",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "id": "f1d546fd",
        "outputId": "41385a68-88f5-43b5-b4c2-be71479357cb"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Empty DataFrame\n",
              "Columns: []\n",
              "Index: []"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-4ebacdee-8be1-4342-91cb-afcda2d3d28c\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4ebacdee-8be1-4342-91cb-afcda2d3d28c')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-4ebacdee-8be1-4342-91cb-afcda2d3d28c button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-4ebacdee-8be1-4342-91cb-afcda2d3d28c');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ],
      "source": [
        "gds.run_cypher('CREATE CONSTRAINT unique_case_id IF NOT EXISTS FOR (n:Case) REQUIRE n.id IS UNIQUE')\n",
        "gds.run_cypher('CREATE CONSTRAINT unique_person_id IF NOT EXISTS FOR (n:Person) REQUIRE (n.id) IS UNIQUE')\n",
        "gds.run_cypher('CREATE CONSTRAINT unique_symptom_id IF NOT EXISTS FOR (n:Symptom) REQUIRE (n.id) IS UNIQUE')\n",
        "gds.run_cypher('CREATE CONSTRAINT unique_disease_id IF NOT EXISTS FOR (n:Disease) REQUIRE n.id IS UNIQUE')\n",
        "gds.run_cypher('CREATE CONSTRAINT unique_bodysys_id IF NOT EXISTS FOR (n:BodySystem) REQUIRE n.id IS UNIQUE')\n",
        "gds.run_cypher('CREATE CONSTRAINT unique_diag_id IF NOT EXISTS FOR (n:Diagnosis) REQUIRE n.id IS UNIQUE')\n",
        "gds.run_cypher('CREATE CONSTRAINT unique_biological_id IF NOT EXISTS FOR (n:Biological) REQUIRE n.id IS UNIQUE')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1435e1c9",
      "metadata": {
        "id": "1435e1c9"
      },
      "source": [
        "Ingest the entities"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "df1f033e",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "df1f033e",
        "outputId": "3f3e49d4-7a6a-4f9b-a0e1-4bf3d749d806"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 72 ms, sys: 5.68 ms, total: 77.7 ms\n",
            "Wall time: 8.11 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "for e in ent_cyp:\n",
        "    gds.run_cypher(e)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f52ce15b",
      "metadata": {
        "id": "f52ce15b"
      },
      "source": [
        "Ingest relationships now"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "684c1282",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "684c1282",
        "outputId": "151348ed-caaf-4e0b-9f9b-99c802cdf318"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 106 ms, sys: 13.7 ms, total: 120 ms\n",
            "Wall time: 14.6 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "for r in rel_cyp:\n",
        "    gds.run_cypher(r)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "36ce7424",
      "metadata": {
        "id": "36ce7424"
      },
      "source": [
        "This is a helper function to ingest all case sheets inside the `data/` directory"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "a15e3072",
      "metadata": {
        "id": "a15e3072"
      },
      "outputs": [],
      "source": [
        "def run_pipeline(count=191):\n",
        "    txt_files = glob.glob(\"data/*.txt\")[0:count]\n",
        "    print(f\"Running pipeline for {len(txt_files)} files\")\n",
        "    failed_files = process_pipeline(txt_files)\n",
        "    print(failed_files)\n",
        "    return failed_files\n",
        "\n",
        "def process_pipeline(files):\n",
        "    failed_files = []\n",
        "    for f in files:\n",
        "        try:\n",
        "            with open(f, 'r') as file:\n",
        "                print(f\"  {f}: Reading File...\")\n",
        "                data = file.read().rstrip()\n",
        "                text = clean_text(data)\n",
        "                print(f\"    {f}: Extracting E & R\")\n",
        "                results = extract_entities_relationships(f, text)\n",
        "                print(f\"    {f}: Generating Cypher\")\n",
        "                ent_cyp, rel_cyp = generate_cypher(results)\n",
        "                print(f\"    {f}: Ingesting Entities\")\n",
        "                for e in ent_cyp:\n",
        "                    gds.run_cypher(e)\n",
        "                print(f\"    {f}: Ingesting Relationships\")\n",
        "                for r in rel_cyp:\n",
        "                    gds.run_cypher(r)\n",
        "                print(f\"    {f}: Processing DONE\")\n",
        "        except Exception as e:\n",
        "            print(f\"    {f}: Processing Failed with exception {e}\")\n",
        "            failed_files.append(f)\n",
        "    return failed_files\n",
        "\n",
        "def extract_entities_relationships(f, text):\n",
        "    start = timer()\n",
        "    system = \"You are a helpful Medical Case Sheet expert who extracts relevant information and store them on a Neo4j Knowledge Graph\"\n",
        "    prompts = [prompt1]\n",
        "    all_cypher = \"\"\n",
        "    results = []\n",
        "    for p in prompts:\n",
        "      p = Template(p).substitute(ctext=text)\n",
        "      res = process_gpt(system, p)\n",
        "      results.append(json.loads(res))\n",
        "    end = timer()\n",
        "    elapsed = (end-start)\n",
        "    print(f\"    {f}: E & R took {elapsed}secs\")\n",
        "    return results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "92f1ebbe",
      "metadata": {
        "id": "92f1ebbe"
      },
      "outputs": [],
      "source": [
        "%%time\n",
        "failed_files = run_pipeline(200)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a559333e",
      "metadata": {
        "id": "a559333e"
      },
      "source": [
        "If processing failed for some files due to API Rate limit or some other error, you can retry as below"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "39031486",
      "metadata": {
        "id": "39031486"
      },
      "outputs": [],
      "source": [
        "%%time\n",
        "failed_files = process_pipeline(failed_files)\n",
        "failed_files"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "578d93fa",
      "metadata": {
        "id": "578d93fa"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "environment": {
      "kernel": "python3",
      "name": "common-cpu.m106",
      "type": "gcloud",
      "uri": "gcr.io/deeplearning-platform-release/base-cpu:m106"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.12"
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}