Exploring Score-based  Patient Matching.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring Score-based  Patient matching\n",
    "This notebook introduces key concepts of patient matching while demonstrating those concepts using python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*********************"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About this notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook follows  step by step the record linkage process , providing minimum exaplanation and assuming certain knowledge of the process. Our aim is to compare different score-based matching strategies using synthetic  data.\n",
    "For more detailed information, please consult the reference section"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import recordlinkage  as rl\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import warnings\n",
    "from recordlinkage.index import Block\n",
    "from recordlinkage.datasets import load_febrl4,load_febrl3,load_febrl2,load_febrl1\n",
    "from recordlinkage.preprocessing import phonetic,clean\n",
    "\n",
    "import matplotlib as mlp\n",
    "import matplotlib.pyplot as plt\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get version information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pandas version: 0.25.0 \n",
      "\n",
      "Python Record Linkage version: 0.14 \n",
      "\n",
      "Numpy version: 1.16.3 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Get Version information\n",
    "print(\"Pandas version: {0}\".format(pd.__version__),'\\n')\n",
    "print(\"Python Record Linkage version: {0}\".format(rl._version.get_versions()['version']),'\\n')\n",
    "print(\"Numpy version: {0}\".format(np.__version__),'\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing dataset to deduplicate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of records: 5000\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>given_name</th>\n",
       "      <th>surname</th>\n",
       "      <th>address_1</th>\n",
       "      <th>address_2</th>\n",
       "      <th>suburb</th>\n",
       "      <th>postcode</th>\n",
       "      <th>state</th>\n",
       "      <th>date_of_birth</th>\n",
       "      <th>soc_sec_id</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>rec-1496-org</th>\n",
       "      <td>mitchell</td>\n",
       "      <td>green</td>\n",
       "      <td>wallaby place</td>\n",
       "      <td>delmar</td>\n",
       "      <td>cleveland</td>\n",
       "      <td>2119</td>\n",
       "      <td>sa</td>\n",
       "      <td>19560409</td>\n",
       "      <td>1804974</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-552-dup-3</th>\n",
       "      <td>harley</td>\n",
       "      <td>mccarthy</td>\n",
       "      <td>pridhamstreet</td>\n",
       "      <td>milton</td>\n",
       "      <td>marsden</td>\n",
       "      <td>3165</td>\n",
       "      <td>nsw</td>\n",
       "      <td>19080419</td>\n",
       "      <td>6089216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-988-dup-1</th>\n",
       "      <td>madeline</td>\n",
       "      <td>mason</td>\n",
       "      <td>hoseason street</td>\n",
       "      <td>lakefront retrmnt vlge</td>\n",
       "      <td>granville</td>\n",
       "      <td>4881</td>\n",
       "      <td>nsw</td>\n",
       "      <td>19081128</td>\n",
       "      <td>2185997</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-1716-dup-1</th>\n",
       "      <td>isabelle</td>\n",
       "      <td>NaN</td>\n",
       "      <td>gundulu place</td>\n",
       "      <td>currin ga</td>\n",
       "      <td>utakarra</td>\n",
       "      <td>2193</td>\n",
       "      <td>wa</td>\n",
       "      <td>19921119</td>\n",
       "      <td>4314184</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-1213-org</th>\n",
       "      <td>taylor</td>\n",
       "      <td>hathaway</td>\n",
       "      <td>yuranigh court</td>\n",
       "      <td>brentwood vlge</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4220</td>\n",
       "      <td>nsw</td>\n",
       "      <td>19991207</td>\n",
       "      <td>9144092</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               given_name   surname        address_1               address_2  \\\n",
       "rec_id                                                                         \n",
       "rec-1496-org     mitchell     green    wallaby place                  delmar   \n",
       "rec-552-dup-3      harley  mccarthy    pridhamstreet                  milton   \n",
       "rec-988-dup-1    madeline     mason  hoseason street  lakefront retrmnt vlge   \n",
       "rec-1716-dup-1   isabelle       NaN    gundulu place               currin ga   \n",
       "rec-1213-org       taylor  hathaway   yuranigh court          brentwood vlge   \n",
       "\n",
       "                   suburb  postcode state date_of_birth  soc_sec_id  \n",
       "rec_id                                                               \n",
       "rec-1496-org    cleveland      2119    sa      19560409     1804974  \n",
       "rec-552-dup-3     marsden      3165   nsw      19080419     6089216  \n",
       "rec-988-dup-1   granville      4881   nsw      19081128     2185997  \n",
       "rec-1716-dup-1   utakarra      2193    wa      19921119     4314184  \n",
       "rec-1213-org          NaN      4220   nsw      19991207     9144092  "
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# file to deduplicate\n",
    "IMPORT_FILE_TO_DEDUPLICATE = './data/dataset_febrl3.csv'\n",
    "#df_a = pd.read_csv(IMPORT_FILE_TO_DEDUPLICATE)\n",
    "df_a = pd.read_csv(IMPORT_FILE_TO_DEDUPLICATE,\n",
    "                    index_col=\"rec_id\",\n",
    "                    sep=\",\",\n",
    "                    engine='c',\n",
    "                    skipinitialspace=True,\n",
    "                    encoding='utf-8',\n",
    "                    parse_dates=[\"date_of_birth\"])\n",
    "#df_a = df_a.set_index('rec_id')\n",
    "\n",
    "print(\"Total number of records:\", len(df_a))\n",
    "#df_a.sort_values('surname').head()\n",
    "df_a.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a function for preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _preprocessing(df_a):\n",
    "    df_a = df_a.copy()\n",
    "    df_a['given_name'] = clean(df_a['given_name'])\n",
    "    df_a['surname'] = clean(df_a['surname'])\n",
    "    df_a['date_of_birth'] = pd.to_datetime(df_a['date_of_birth'],format='%Y%m%d', errors='coerce')\n",
    "    df_a['YearB'] = df_a['date_of_birth'].dt.year.astype('Int64') \n",
    "    df_a['MonthB'] = df_a['date_of_birth'].dt.month.astype('Int64') \n",
    "    df_a['DayB'] = df_a['date_of_birth'].dt.day.astype('Int64')  \n",
    "    df_a['metaphone_given_name'] = phonetic(df_a['given_name'], method='metaphone')\n",
    "    df_a['metaphone_surname'] = phonetic(df_a['surname'], method='metaphone')  \n",
    "    \n",
    "    return df_a\n",
    "\n",
    "df_a= _preprocessing(df_a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a function for blokcing "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of candidate record pairs : 12833\n"
     ]
    }
   ],
   "source": [
    "###### Blocking\n",
    "def _blocking(df_a):\n",
    "    df_a = df_a.copy()\n",
    "    indexer = rl.Index()\n",
    "    # soundex firstname, methapone surname, exact date of birth\n",
    "    indexer.add(Block(['metaphone_given_name','metaphone_surname','date_of_birth']))\n",
    "    # soundex firstname , day of birth\n",
    "    indexer.add(Block(['metaphone_given_name','DayB']))\n",
    "    #soundex firstname , month of birth\n",
    "    indexer.add(Block(['metaphone_given_name','MonthB']))\n",
    "    # metaphone surname, year of birth \n",
    "    indexer.add(Block(['metaphone_surname','YearB']))\n",
    "    # ssn\n",
    "    indexer.add(Block(['soc_sec_id']))\n",
    "\n",
    "    candidate_record_pairs = indexer.index(df_a)\n",
    "\n",
    "    return candidate_record_pairs\n",
    "\n",
    "candidate_record_pairs =_blocking(df_a)\n",
    "print(\"Number of candidate record pairs :\", len(candidate_record_pairs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a function for comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "########## Comparison\n",
    "def _comparaison(candidate_record_pairs):   \n",
    "    df_candidates = candidate_record_pairs.copy() \n",
    "    compare_cl = rl.Compare()\n",
    "    compare_cl.string('given_name', 'given_name', method='jarowinkler', label='given_name')\n",
    "    compare_cl.string('surname', 'surname', method='jarowinkler',label='surname')\n",
    "    compare_cl.exact('date_of_birth', 'date_of_birth', label='date_of_birth')\n",
    "    compare_cl.exact('soc_sec_id', 'soc_sec_id', label='soc_sec_id')\n",
    "    compare_cl.string('address_1', 'address_1', method ='levenshtein' , label='address_1')\n",
    "    compare_cl.string('address_2', 'address_2', method ='levenshtein' , label='address_2')\n",
    "    compare_cl.string('suburb', 'suburb', method ='levenshtein', label='suburb')\n",
    "    compare_cl.exact('postcode', 'postcode', label='postcode')\n",
    "    compare_cl.exact('state', 'state', label='state')\n",
    "    features = compare_cl.compute(df_candidates, df_a)\n",
    "    \n",
    "    return features\n",
    "\n",
    "\n",
    "features = _comparaison(candidate_record_pairs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### An example of simple score-based matching : Similarity Sum (SimSum)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of matches with simsum  is : 6267 using a threshold of 4.\n"
     ]
    }
   ],
   "source": [
    "\n",
    "def simSum(features, threshold):\n",
    "    df_f = features.copy()\n",
    "    df_f['score'] = features.sum(axis=1)\n",
    "    #threshold or score based classification\n",
    "    matches = df_f[df_f['score'] >=threshold]\n",
    "    \n",
    "    return matches, df_f\n",
    "\n",
    "threshold = 4\n",
    "match , df_features = simSum(features,4)\n",
    "print(\"Number of matches with simsum  is : {} using a threshold of {}.\".format(len(match),threshold))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Histogram of similarity sum score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x14f31d89dd8>"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAATVElEQVR4nO3df5BdZX3H8fdX8EdMKiFibtMk0+CYQR13ENjBWGY6G2JbfjjGPwzFoZAw6eQfSmndGYz+4z/tGKcg4tRhmgEVWmpkIg4ZoFYnuHX8A0YD1KjRMaUpWRKJP0LqgtZu/faPexau2bu7d7P31z73/ZrZ2Xue8+w5zz57zmeffe45ZyMzkSSV5VW9boAkqf0Md0kqkOEuSQUy3CWpQIa7JBXIcJekArUU7hGxPCL2RsQPIuJQRLw7IlZExNci4kfV53OruhERn46IwxHxnYi4uLPfgiTpdK2O3O8EvpKZbwUuBA4BO4H9mbke2F8tA1wJrK8+dgB3tbXFkqQ5xVw3MUXEG4B/B96cDZUj4ofASGYej4hVwFhmXhAR/1C9/sLp9Tr2XUiSfsvZLdR5M/AT4HMRcSFwALgFqE0FdhXwK6v6q4GjDV8/XpXNGO7nnXderlu3bsYGvPjiiyxdurSFpg4W+2U6+2Q6+2S6UvrkwIEDP83MNzVb10q4nw1cDNycmU9ExJ28MgXTTDQpm/bnQUTsoD5tQ61W47bbbptxgxMTEyxbtqyFpg4W+2U6+2Q6+2S6Uvpk48aN/zXTulbCfRwYz8wnquW91MP9+YhY1TAtc6Kh/tqGr18DHDt9o5m5G9gNMDw8nCMjIzM2YGxsjNnWDyr7ZTr7ZDr7ZLpB6JM531DNzB8DRyPigqpoE/B9YB+wtSrbCjxUvd4H3FBdNbMBOOV8uyR1Vysjd4Cbgfsj4jXAM8CN1H8xPBAR24FngS1V3UeBq4DDwEtVXUlSF7UU7pn5NDDcZNWmJnUTuGmB7ZIkLYB3qEpSgQx3SSqQ4S5JBTLcJalAhrskFajVSyHVY+t2PvLy6yO7ru5hSyQtBo7cJalAhrskFchwl6QCGe6SVCDDXZIKZLhLUoEMd0kqkOEuSQUy3CWpQIa7JBXIxw/0scZHDkjSfDhyl6QCGe6SVCDDXZIKZLhLUoEMd0kqkOEuSQUy3CWpQIa7JBXIcJekAnmH6iI0defq6NAkI71tiqQ+1dLIPSKORMTBiHg6Ir5dla2IiK9FxI+qz+dW5RERn46IwxHxnYi4uJPfgCRpuvlMy2zMzHdm5nC1vBPYn5nrgf3VMsCVwPrqYwdwV7saK0lqzUKmZTbDy7MC9wJjwIer8vsyM4HHI2J5RKzKzOMLaagknYnGB/Ad2XV1D1vSXa2O3BP4akQciIgdVVltKrCrzyur8tXA0YavHa/KJEld0urI/bLMPBYRK4GvRcQPZqkbTcpyWqX6L4kdALVajbGxsRk3ODExMev6Uo0OTc66vraEgeyX2QzqsTKbQe+TxvNoqh8GoU9aCvfMPFZ9PhERXwYuBZ6fmm6JiFXAiar6OLC24cvXAMeabHM3sBtgeHg4R0ZGZtz/2NgYs60v1bY5nuc+OjTJNQPYL7MZ1GNlNoPeJ43n0ZHrRoDB6JM5p2UiYmlE/M7Ua+CPge8C+4CtVbWtwEPV633ADdVVMxuAU863S1J3tTJyrwFfjoip+v+cmV+JiG8BD0TEduBZYEtV/1HgKuAw8BJwY9tbXTD/+5Kkdpgz3DPzGeDCJuU/AzY1KU/gpra0TpJ0Rnz8gCQVyHCXpAL5bJlFblBv0JA0O0fuklQgw12SCmS4S1KBDHdJKpDhLkkFMtwlqUCGuyQVyHCXpAIZ7pJUIMNdkgpkuEtSgXy2jKSBMfUsptGhSUZ625SOc+QuSQUy3CWpQIa7JBXIcJekAhnuklQgw12SCmS4S1KBDHdJKpDhLkkFMtwlqUCGuyQVyHCXpAK1HO4RcVZEPBURD1fL50fEExHxo4j4YkS8pip/bbV8uFq/rjNNlyTNZD4j91uAQw3LnwDuyMz1wElge1W+HTiZmW8B7qjqqQvW7Xzk5Q9Jg62lcI+INcDVwN3VcgCXA3urKvcC769eb66WqdZvqupLkrqk1ZH7p4Bbgd9Uy28EXsjMyWp5HFhdvV4NHAWo1p+q6kuSumTOf9YREe8FTmTmgYgYmSpuUjVbWNe43R3ADoBarcbY2NiMbZiYmJh1fUlGhybnrlSpLZm5/qD01+kG6Vhp1aD3SbNzpLak/HOklf/EdBnwvoi4Cngd8AbqI/nlEXF2NTpfAxyr6o8Da4HxiDgbOAf4+ekbzczdwG6A4eHhHBkZmbEBY2NjzLa+JNvmMV8+OjTJ7Qeb/wiPXDfSphYtLoN0rLRq0Puk2Tk1OjTJNYX3yZzTMpn5kcxck5nrgGuBxzLzOuDrwAeqaluBh6rX+6plqvWPZea0kbskqXMWcp37h4EPRcRh6nPq91Tl9wBvrMo/BOxcWBMlSfM1r3+QnZljwFj1+hng0iZ1fgVsaUPbJElnyDtUJalAhrskFWhe0zLqDO8oldRujtwlqUCO3CUVx7+GDXdJ6orGXzhHdl3d8f05LSNJBTLcJalAhrskFchwl6QCGe6SVCDDXZIKZLhLUoEMd0kqkOEuSQUy3CWpQIa7JBXIcJekAvngsB7xqXWSOsmRuyQVyHCXpAI5LSNJXdaNZ7s7cpekAhnuklQgw12SCmS4S1KBDHdJKpBXy3SRNy5J6pY5wz0iXgd8A3htVX9vZn4sIs4H9gArgCeB6zPz1xHxWuA+4BLgZ8CfZuaRDrVfUh/rxiV/aq6VaZn/AS7PzAuBdwJXRMQG4BPAHZm5HjgJbK/qbwdOZuZbgDuqepKkLpoz3LNuolp8dfWRwOXA3qr8XuD91evN1TLV+k0REW1rsSRpTpGZc1eKOAs4ALwF+Azwd8Dj1eiciFgL/EtmviMivgtckZnj1br/AN6VmT89bZs7gB0AtVrtkj179sy4/4mJCZYtW3YG315/OfjcqbZur7YEnv/l3PWGVp/T1v32s1KOlXbqZZ80HvPdPA7nOtdqS2Dliu6eFzO1aSH9snHjxgOZOdxsXUtvqGbm/wHvjIjlwJeBtzWrVn1uNkqf9hskM3cDuwGGh4dzZGRkxv2PjY0x2/rFYlub31AdHZrk9oNz/wiPXDfS1v32s1KOlXbqZZ80HvPdPA7nOtdGhya5pst9MlObOtUv87oUMjNfAMaADcDyiJhKljXAser1OLAWoFp/DvDzdjRWktSaOcM9It5UjdiJiCXAe4BDwNeBD1TVtgIPVa/3VctU6x/LVuZ+JElt08q0zCrg3mre/VXAA5n5cER8H9gTEX8DPAXcU9W/B/jHiDhMfcR+bQfaLUmaxZzhnpnfAS5qUv4McGmT8l8BW9rSOknF8Jr37vLxA5JUIMNdkgpkuEtSgQx3SSqQ4S5JBTLcJalAhrskFchwl6QCGe6SVCDDXZIKZLhLUoH8B9kDwGd6SIPHcJdUhHVt/mc4i53TMpJUIMNdkgpkuEtSgZxzl9RWzn33B0fuklQgw12SCmS4S1KBDHdJKpDhLkkFMtwlqUBeCilJHdLLy0IduUtSgQx3SSqQ4S5JBZpzzj0i1gL3Ab8L/AbYnZl3RsQK4IvAOuAIcE1mnoyIAO4ErgJeArZl5pOdaX7/81ZsSb3Qysh9EhjNzLcBG4CbIuLtwE5gf2auB/ZXywBXAuurjx3AXW1vtSRpVnOGe2Yenxp5Z+YvgEPAamAzcG9V7V7g/dXrzcB9Wfc4sDwiVrW95ZKkGc1rzj0i1gEXAU8Atcw8DvVfAMDKqtpq4GjDl41XZZKkLonMbK1ixDLg34C/zcwHI+KFzFzesP5kZp4bEY8AH8/Mb1bl+4FbM/PAadvbQX3ahlqtdsmePXtm3PfExATLli2b57fWHw4+d6pj264tged/Ob+vGVp9Tmca0ycW87HSKd3uk1aO+U4ch/M512pLYOWKzp8Lne6LjRs3HsjM4WbrWrqJKSJeDXwJuD8zH6yKn4+IVZl5vJp2OVGVjwNrG758DXDs9G1m5m5gN8Dw8HCOjIzMuP+xsTFmW9/PtnXwDdXRoUluPzi/+9COXDfSmcb0icV8rHRKt/uklWO+E8fhfM610aFJrulCn/SqL6CFaZnq6pd7gEOZ+cmGVfuArdXrrcBDDeU3RN0G4NTU9I0kqTtaGfZdBlwPHIyIp6uyjwK7gAciYjvwLLClWvco9csgD1O/FPLGtrZ4EfDyR0m9Nme4V3PnMcPqTU3qJ3DTAtslSVoA71CVpAIZ7pJUIMNdkgpkuEtSgfxnHZpT49U/R3Zd3cOWSGqVI3dJKpDhLkkFMtwlqUCGuyQVyDdUpQI1vgn++SuW9rAl6hVH7pJUIEfukhYtH9I3M0fuklQgR+6aF29okhYHw12S2qSfpokM9wHjyFud0E+hpjrDXWfMXxRS//INVUkqkOEuSQUy3CWpQIa7JBXIN1Ql9SXfsF8Yw11S3zPo589wV1t48qlbvKa+Nc65S1KBDHdJKpDTMmrKP33VLU7pdcac4R4RnwXeC5zIzHdUZSuALwLrgCPANZl5MiICuBO4CngJ2JaZT3am6VooTyqpXK2M3D8P/D1wX0PZTmB/Zu6KiJ3V8oeBK4H11ce7gLuqz5L0Mv8y7Lw5wz0zvxER604r3gyMVK/vBcaoh/tm4L7MTODxiFgeEasy83i7GiypXIZ++5zpnHttKrAz83hErKzKVwNHG+qNV2WGe5/zpJLKEvVB9hyV6iP3hxvm3F/IzOUN609m5rkR8Qjw8cz8ZlW+H7g1Mw802eYOYAdArVa7ZM+ePTPuf2JigmXLls3n++q6g8+d6vo+a0vg+V92fbdzGlp9Ts/2vRiOlW5oPB7PP+esjvdJL47/hagtgZUr2n+cnkk/LOR82bhx44HMHG627kxH7s9PTbdExCrgRFU+DqxtqLcGONZsA5m5G9gNMDw8nCMjIzPubGxsjNnW94NtPRj5jg5NcvvB/rvg6ch1Iz3b92I4Vjrlt//6euW4+PwVSzveJ704/hdidGiSazrQJ2fSD506X870Ovd9wNbq9VbgoYbyG6JuA3DK+XZJ6r5WLoX8AvU3T8+LiHHgY8Au4IGI2A48C2ypqj9K/TLIw9QvhbyxA22WJM2hlatlPjjDqk1N6iZw00IbJUn9Zqb7Qvr1YoT+m7CV1Pf6NdD0Cp8tI0kFcuQuaSCV/vgNw10dVfoJ1Gudnh4ZxJ9fKd+z4S5p4M33l+RieM/BcF+AxfADlubL4/oVi7kvDHe13WI+IUpXypSD5ma4S2qJv7QXF8NdKtzB504tume/aOEMd2lAOUVTNsNdXWOYSN1juEtyPr1APn5AkgrkyF094RSN1FmO3CWpQI7cpTbzrxL1A0fuklQgR+7SIuDVLJovR+6SVCBH7vPkCErNeFyo3xjuUh/xzVi1i+Eu9Sn/GtBCGO5SB800EneErk7zDVVJKpAjd/XcoI9inX5RJxjuUpcY4uomp2UkqUAdGblHxBXAncBZwN2ZuasT+5G6zdG3Fou2h3tEnAV8BvgjYBz4VkTsy8zvt3tfneRJvPis2/kIo0OTbNv5yEDO3UuNOjFyvxQ4nJnPAETEHmAz0PfhbqD33nzfXJ3pZ9bKJYgzmW99qR91ItxXA0cblseBd3VgP8DCTmL1t3b9DOe7HY8dlSAys70bjNgC/Elm/nm1fD1waWbefFq9HcCOavEC4IezbPY84KdtbWgZ7Jfp7JPp7JPpSumT38/MNzVb0YmR+ziwtmF5DXDs9EqZuRvY3coGI+LbmTncnuaVw36Zzj6Zzj6ZbhD6pBOXQn4LWB8R50fEa4BrgX0d2I8kaQZtH7ln5mRE/AXwr9QvhfxsZn6v3fuRJM2sI9e5Z+ajwKNt3GRL0zcDyH6Zzj6Zzj6Zrvg+afsbqpKk3vPxA5JUoL4P94i4IiJ+GBGHI2Jnr9vTaxGxNiK+HhGHIuJ7EXFLr9vULyLirIh4KiIe7nVb+kFELI+IvRHxg+p4eXev29QPIuKvq3PnuxHxhYh4Xa/b1Al9He4NjzK4Eng78MGIeHtvW9Vzk8BoZr4N2ADcZJ+87BbgUK8b0UfuBL6SmW8FLsS+ISJWA38JDGfmO6hf9HFtb1vVGX0d7jQ8yiAzfw1MPcpgYGXm8cx8snr9C+on7Oretqr3ImINcDVwd6/b0g8i4g3AHwL3AGTmrzPzhd62qm+cDSyJiLOB19PkPpwS9Hu4N3uUwcAH2ZSIWAdcBDzR25b0hU8BtwK/6XVD+sSbgZ8An6umqu6OiKW9blSvZeZzwG3As8Bx4FRmfrW3reqMfg/3aFLm5T1ARCwDvgT8VWb+d6/b00sR8V7gRGYe6HVb+sjZwMXAXZl5EfAi4HtWEedS/+v/fOD3gKUR8We9bVVn9Hu4t/Qog0ETEa+mHuz3Z+aDvW5PH7gMeF9EHKE+dXd5RPxTb5vUc+PAeGZO/VW3l3rYD7r3AP+ZmT/JzP8FHgT+oMdt6oh+D3cfZXCaiAjq86iHMvOTvW5PP8jMj2TmmsxcR/0YeSwzixyNtSozfwwcjYgLqqJNLILHbnfBs8CGiHh9dS5totA3mvv6f6j6KIOmLgOuBw5GxNNV2Ueru4KlRjcD91cDo2eAG3vcnp7LzCciYi/wJPUrz56i0LtVvUNVkgrU79MykqQzYLhLUoEMd0kqkOEuSQUy3CWpQIa7JBXIcJekAhnuklSg/wdJYdwC8MDhuQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_features['score'].hist(bins=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### An example of weighted average score-based matching"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of matches with weighted average  is : 6287 using a threshold of 0.4.\n"
     ]
    }
   ],
   "source": [
    "#https://www.wikihow.com/Calculate-Weighted-Average\n",
    "from functools import reduce\n",
    "weigth_factor = {'given_name':2,\n",
    "                 'surname':2, \n",
    "                 'date_of_birth':1,\n",
    "                 'soc_sec_id':3,\n",
    "                 'address_1':1,\n",
    "                 'address_2':1,\n",
    "                 'suburb':1,\n",
    "                 'postcode':1,\n",
    "                 'state':1}\n",
    "\n",
    "def _weighted_average_classifier(features,threshold,weigth_factor):\n",
    "    \n",
    "    df_score = features.copy()\n",
    "    weighted_list =[]\n",
    "    factor_sum = 0\n",
    "    for col,wf in weigth_factor.items():\n",
    "        weighted_list.append(df_score[col]*wf)\n",
    "        factor_sum += wf\n",
    "    weighted_sum = reduce(lambda x, y: x.add(y, fill_value=0), weighted_list)\n",
    "    df_score['score'] = weighted_sum/factor_sum\n",
    "    matches = df_score[df_score['score'] >=threshold]\n",
    "    \n",
    "    return matches,df_score\n",
    "\n",
    "threshold = 0.4\n",
    "wa_matches, wa_df_score= _weighted_average_classifier(features,threshold,weigth_factor)\n",
    "print(\"Number of matches with weighted average  is : {} using a threshold of {}.\".format(len(wa_matches),threshold))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Histogram of weighted score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x14f31f0f588>"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAU0klEQVR4nO3df4xd51ng8e/ThLTF09pp046MbZigGmgVqyUZtUFI7EwNyE2run8kKChQuzJYQBoVNYgY+GN3WaSGXZVsK1UFi1RxEDAJgW6sJGW362ZUgnBoTEOcNkDcYNKJs/GWumYnoUDYhz/u63Rw7vieO/fX3Pd+P9LonvOed8553jnnPve97/kxkZlIkuryilEHIEnqP5O7JFXI5C5JFTK5S1KFTO6SVKGLRx0AwGWXXZYzMzON6j7//PNs2LBhsAGtM5PYZrDdk8Z2d+/YsWNfy8w3tFu2LpL7zMwMjzzySKO6i4uLzM3NDTagdWYS2wy2e9LY7u5FxN+ttsxhGUmqkMldkipkcpekCpncJalCJndJqpDJXZIqZHKXpAqZ3CWpQiZ3SarQurhDVcMzc+D+l6ZP3vruEUYiaZDsuUtShey5TzB78VK97LlLUoVM7pJUIYdlJsDK4RdJk8GeuyRVyOQuSRUyuUtShUzuklShRsk9IjZFxD0R8VcR8URE/EBEvC4iPhsRT5bXS0vdiIiPR8SJiHgsIq4cbBMkSedr2nP/GPDHmfl9wFuBJ4ADwJHM3A4cKfMA7wK2l5/9wCf7GrEkqaOOyT0iXgv8EHA7QGb+c2Z+A9gNHCrVDgHvK9O7gTuz5SiwKSI29z1ySdKqIjMvXCHibcBB4Mu0eu3HgA8Bz2TmphX1zmTmpRFxH3BrZj5Uyo8At2TmI+etdz+tnj3T09NXLSwsNAp4eXmZqamphs2rQ69tPv7M2Y51dmzZuOb1D8ok7muw3ZOml3bPz88fy8zZdsua3MR0MXAlcFNmPhwRH+NbQzDtRJuyl32CZOZBWh8azM7O5tzcXINQYHFxkaZ1a9Frm/c2uInp5A1rX/+gTOK+Bts9aQbV7iZj7kvAUmY+XObvoZXsnzs33FJeT6+ov23F728FTvUnXElSEx2Te2b+H+CrEfG9pWgnrSGaw8CeUrYHuLdMHwbeX66auRo4m5nP9jdsSdKFNH22zE3A70bEJcBTwAdofTDcHRH7gKeB60rdB4BrgBPAC6WuJGmIGiX3zHwUaDdov7NN3QRu7DEuSVIPvENVkipkcpekCpncJalCJndJqpDJXZIqZHKXpAqZ3CWpQiZ3SaqQyV2SKmRyl6QKmdwlqUImd0mqkMldkipkcpekCpncJalCJndJqpDJXZIqZHKXpAqZ3CWpQiZ3SaqQyV2SKnTxqAPQYMwcuH/UIUgaIXvuklShRsk9Ik5GxPGIeDQiHillr4uIz0bEk+X10lIeEfHxiDgREY9FxJWDbIAk6eW66bnPZ+bbMnO2zB8AjmTmduBImQd4F7C9/OwHPtmvYCVJzfQyLLMbOFSmDwHvW1F+Z7YcBTZFxOYetiNJ6lLT5J7A/4qIYxGxv5RNZ+azAOX1jaV8C/DVFb+7VMokSUMSmdm5UsR3ZOapiHgj8FngJuBwZm5aUedMZl4aEfcDH8nMh0r5EeAXM/PYeevcT2vYhunp6asWFhYaBby8vMzU1FSz1lViLW0+/szZrurv2LKxq/rDMIn7Gmz3pOml3fPz88dWDJX/O40uhczMU+X1dER8Gng78FxEbM7MZ8uwy+lSfQnYtuLXtwKn2qzzIHAQYHZ2Nufm5ho1ZnFxkaZ1a7GWNu/t8lLIkzd0t/5hmMR9DbZ70gyq3R2HZSJiQ0S85tw08KPA48BhYE+ptge4t0wfBt5frpq5Gjh7bvhGkjQcTXru08CnI+Jc/d/LzD+OiC8Ad0fEPuBp4LpS/wHgGuAE8ALwgb5HLUkDsvIGwJO3vnuEkfSmY3LPzKeAt7Yp/3tgZ5vyBG7sS3SSpDXxDlVJqpDJXZIqZHKXpAqZ3CWpQiZ3SaqQyV2SKmRyl6QKmdwlqUImd0mqkP9DVcDL/+fqON92LcmeuyRVyeQuSRUyuUtShUzuklQhk7skVcjkLkkVMrlLUoVM7pJUIW9ikjTxzr+JrwYmd0kaspUfJnfs2jCQbTgsI0kVMrlLUoVM7pJUIcfcJWkIhn3StnHPPSIuiogvRsR9Zf7yiHg4Ip6MiLsi4pJS/soyf6IsnxlM6JKk1XQzLPMh4IkV878O3JaZ24EzwL5Svg84k5lvAm4r9SRJQ9QouUfEVuDdwG+X+QDeCdxTqhwC3lemd5d5yvKdpb4kaUgiMztXirgH+AjwGuAXgL3A0dI7JyK2AZ/JzCsi4nFgV2YulWVfAd6RmV87b537gf0A09PTVy0sLDQKeHl5mampqWatq8Ra2nz8mbM9bXPHlo09/X4/TOK+Bts9Cqu9X/r5PlhtG5dvvGjN7Z6fnz+WmbPtlnU8oRoR7wFOZ+axiJg7V9ymajZY9q2CzIPAQYDZ2dmcm5s7v0pbi4uLNK1bi7W0eW+PJ29O3tDd9gZhEvc12O5RWO390s/3wWrbuGPXhoG0u8nVMj8IvDcirgFeBbwW+O/Apoi4ODNfBLYCp0r9JWAbsBQRFwMbga/3PXJJ0qo6jrln5i9l5tbMnAGuBz6XmTcADwLXlmp7gHvL9OEyT1n+uWwy9iNJ6ptebmK6BfhwRJwAXg/cXspvB15fyj8MHOgtRElSt7q6iSkzF4HFMv0U8PY2db4JXNeH2CRJa+TjBySpQiZ3SaqQyV2SKmRyl6QKmdwlqUImd0mqkMldkipkcpekCpncJalCJndJqpDJXZIqZHKXpAqZ3CWpQiZ3SaqQyV2SKmRyl6QKmdwlqUImd0mqUFf/Zk+SRm3mwP0vTZ+89d0jjGR9s+cuSRUyuUtShUzuklQhx9zXmV7GE1f+7ijjkDR6HXvuEfGqiPjziPjLiPhSRPznUn55RDwcEU9GxF0RcUkpf2WZP1GWzwy2CZKk8zUZlvkn4J2Z+VbgbcCuiLga+HXgtszcDpwB9pX6+4Azmfkm4LZST5I0RB2Te7Ysl9lvKz8JvBO4p5QfAt5XpneXecrynRERfYtYktRRZGbnShEXAceANwGfAP4bcLT0zomIbcBnMvOKiHgc2JWZS2XZV4B3ZObXzlvnfmA/wPT09FULCwuNAl5eXmZqaqph88bP8WfOvjS9Y8tGoHmbV/5uP52LY9hq39ersd0X1u490qvV3jv9PPZX28blGy9a8/6en58/lpmz7ZY1OqGamf8KvC0iNgGfBt7crlp5bddLf9knSGYeBA4CzM7O5tzcXJNQWFxcpGndcbR35UnR488DcPOOf+Wm98x197t9dPKGztsehNr39Wps94WtPM77dWyu9t7p57G/2jbu2LVhIPu7q0shM/MbwCJwNbApIs59OGwFTpXpJWAbQFm+Efh6P4KVJDXT5GqZN5QeOxHxauCHgSeAB4FrS7U9wL1l+nCZpyz/XDYZ+5Ek9U2TYZnNwKEy7v4K4O7MvC8ivgwsRMSvAV8Ebi/1bwd+JyJO0OqxXz+AuCVJF9AxuWfmY8D3tyl/Cnh7m/JvAtf1JTpJ0pr4+AFJqpDJXZIq5LNlJGkV4/yMJXvuklQhe+6SBmbQPd9x7lkPmj13SaqQyV2SKmRyl6QKmdwlqUImd0mqkMldkipkcpekCnmd+zowM6B/siHVwvdI9+y5S1KF7LmrI+8ClMaPyX0MmWwldWJyl9RXjo+vD465S1KF7LmPCXtDkrphz12SKmRyl6QKmdwlqUKOuUuaSLWfx+rYc4+IbRHxYEQ8ERFfiogPlfLXRcRnI+LJ8nppKY+I+HhEnIiIxyLiykE3QpL07zUZlnkRuDkz3wxcDdwYEW8BDgBHMnM7cKTMA7wL2F5+9gOf7HvUkqQL6jgsk5nPAs+W6f8XEU8AW4DdwFypdghYBG4p5XdmZgJHI2JTRGwu61Gf1f7VUhpno3x/RisHN6wcMQN8HrgCeDozN61YdiYzL42I+4BbM/OhUn4EuCUzHzlvXftp9eyZnp6+amFhoVEMy8vLTE1NNY55HBx/5uwFl0+/Gp77xyEF08GOLRuHtq0a93UT497u1Y7nTsfOhdrd6T3SZP1rWWcv62+6jcs3XrTm/T0/P38sM2fbLWt8QjUipoA/BH4+M/8hIlat2qbsZZ8gmXkQOAgwOzubc3NzjeJYXFykad1xsbfDp/vNO17ko8fXx7nvkzfMDW1bNe7rJsa93asdz52OnQu1u9N7pMn617LOXtbfdBt37NowkP3d6FLIiPg2Won9dzPzj0rxcxGxuSzfDJwu5UvAthW/vhU41Z9wJUlNNLlaJoDbgScy8zdWLDoM7CnTe4B7V5S/v1w1czVw1vF2SRquJt/1fxD4SeB4RDxayn4ZuBW4OyL2AU8D15VlDwDXACeAF4AP9DViSVJHTa6WeYj24+gAO9vUT+DGHuOSJPVgfZyl01ha7TIv/4GINHomd2lC9fM/enm/xfrjg8MkqUImd0mqkMMy6opfv8eb+29y2HOXpAqZ3CWpQiZ3SaqQyV2SKmRyl6QKebWMpCr086asGthzl6QKmdwlqUIOy6jv/HosjZ49d0mqkMldkirksMyI+IwPSYNkcpe0LtkB6o3DMpJUIZO7JFXI5C5JFTK5S1KFPKEqaWJM0knajsk9Ij4FvAc4nZlXlLLXAXcBM8BJ4Mcy80xEBPAx4BrgBWBvZv7FYELXOPBuVU2S9fTh0WRY5g5g13llB4AjmbkdOFLmAd4FbC8/+4FP9idMSVI3OvbcM/PzETFzXvFuYK5MHwIWgVtK+Z2ZmcDRiNgUEZsz89l+BSxp9NZTD1XtrfWE6vS5hF1e31jKtwBfXVFvqZRJkoYoWp3sDpVaPff7Voy5fyMzN61YfiYzL42I+4GPZOZDpfwI8IuZeazNOvfTGrphenr6qoWFhUYBLy8vMzU11ajuenb8mbON606/Gp77xwEGMyQ7tmzsqn4t+7pbg2z3asddt/umm+O36TbOb/dattFpW4NYZ6/rv3zjRWve3/Pz88cyc7bdsrVeLfPcueGWiNgMnC7lS8C2FfW2AqfarSAzDwIHAWZnZ3Nubq7RhhcXF2ladz3b28XX2pt3vMhHj4//hU0nb5jrqn4t+7pbg2z3asddt/umm+O36TbOb/datvGS489/a7srTuT3ss4mf6O1rP+OXRsGsr/XmjEOA3uAW8vrvSvKPxgRC8A7gLOOt0sCr5watiaXQv4+rZOnl0XEEvAfaSX1uyNiH/A0cF2p/gCtyyBP0LoU8gMDiFljyje3NDxNrpb58VUW7WxTN4Ebew1KkvplUq/sGf+BXKlSftNRL3y2jCRVyOQuSRVyWEaq3KSOOU86e+6SVCF77hoJTxZOttX2/3r+ljFux6w9d0mqkD13SSN1rkd8844XMSX1jz13SaqQH5NDtJ7HEyXVxeQurSMzB+7n5h0v9vZExD7EcM44nDhUew7LSFKF7LlLasRhxfbW69/F5C5VaL0mHA2PyV0aA4MeB/fDoD4m9wHzTTMZaj0J6fE7vkzuGrlaE6M0SiZ3aY361au1dzx+xmGfeSmkJFXInrvWlZU9ojt2bRhhJOvXOD5RUcNnz12SKmTPfQDsQakTx+s1aCZ3qQu9JFMTsYbJ5N4D36yDdfyZsy89QGucLpH00k6tBwNJ7hGxC/gYcBHw25l56yC2Mwom9PHTS7LtdX97vGhU+p7cI+Ii4BPAjwBLwBci4nBmfrnf2xoW36Djp8k+s4etmg2i5/524ERmPgUQEQvAbmAgyb3XN6iJe7x1u/9Wq+9xoNpEZvZ3hRHXArsy86fK/E8C78jMD55Xbz+wv8x+L/DXDTdxGfC1PoU7LiaxzWC7J43t7t53ZeYb2i0YRM892pS97BMkMw8CB7teecQjmTm7lsDG1SS2GWz3qOMYNtvdX4O4iWkJ2LZifitwagDbkSStYhDJ/QvA9oi4PCIuAa4HDg9gO5KkVfR9WCYzX4yIDwL/k9alkJ/KzC/1cRNdD+VUYBLbDLZ70tjuPur7CVVJ0uj54DBJqpDJXZIqtC6Te0Tsioi/jogTEXGgzfJXRsRdZfnDETEz/Cj7r0G7PxwRX46IxyLiSER81yji7LdO7V5R79qIyIio4nK5Ju2OiB8r+/xLEfF7w45xEBoc598ZEQ9GxBfLsX7NKOLsp4j4VEScjojHV1keEfHx8jd5LCKu7HmjmbmufmidhP0K8N3AJcBfAm85r87PAb9Zpq8H7hp13ENq9zzw7WX6Zyel3aXea4DPA0eB2VHHPaT9vR34InBpmX/jqOMeUrsPAj9bpt8CnBx13H1o9w8BVwKPr7L8GuAztO4Tuhp4uNdtrsee+0uPL8jMfwbOPb5gpd3AoTJ9D7AzItrdPDVOOrY7Mx/MzBfK7FFa9xCMuyb7G+C/AP8V+OYwgxugJu3+aeATmXkGIDNPDznGQWjS7gReW6Y3UsF9Mpn5eeDrF6iyG7gzW44CmyJicy/bXI/JfQvw1RXzS6WsbZ3MfBE4C7x+KNENTpN2r7SP1if9uOvY7oj4fmBbZt43zMAGrMn+/h7geyLiTyPiaHna6rhr0u7/BPxERCwBDwA3DSe0ker2/d/Renyee5PHFzR6xMGYadymiPgJYBb4DwONaDgu2O6IeAVwG7B3WAENSZP9fTGtoZk5Wt/S/iQirsjMbww4tkFq0u4fB+7IzI9GxA8Av1Pa/f8HH97I9D2nrceee5PHF7xUJyIupvXV7UJfecZBo8c2RMQPA78CvDcz/2lIsQ1Sp3a/BrgCWIyIk7TGIw9XcFK16XF+b2b+S2b+La2H620fUnyD0qTd+4C7ATLzz4BX0Xq4Vs36/tiW9Zjcmzy+4DCwp0xfC3wuy1mJMdax3WV44rdoJfYaxl+hQ7sz82xmXpaZM5k5Q+tcw3sz85HRhNs3TY7z/0HrJDoRcRmtYZqnhhpl/zVp99PAToCIeDOt5P5/hxrl8B0G3l+umrkaOJuZz/a0xlGfRb7AmeO/oXVW/VdK2a/SelNDa2f/AXAC+HPgu0cd85Da/b+B54BHy8/hUcc8jHafV3eRCq6Wabi/A/gNWv8L4Thw/ahjHlK73wL8Ka0raR4FfnTUMfehzb8PPAv8C61e+j7gZ4CfWbGvP1H+Jsf7cYz7+AFJqtB6HJaRJPXI5C5JFTK5S1KFTO6SVCGTuyRVyOQuSRUyuUtShf4N24MD+99P9F8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "wa_df_score['score'].hist(bins=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec_id_1</th>\n",
       "      <th>rec_id_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>rec-552-dup-1</th>\n",
       "      <th>rec-552-dup-3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">rec-552-dup-0</th>\n",
       "      <th>rec-552-dup-3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-552-dup-1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">rec-552-org</th>\n",
       "      <th>rec-552-dup-3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rec-552-dup-1</th>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: [(rec-552-dup-1, rec-552-dup-3), (rec-552-dup-0, rec-552-dup-3), (rec-552-dup-0, rec-552-dup-1), (rec-552-org, rec-552-dup-3), (rec-552-org, rec-552-dup-1)]"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# gold_ standard or known truth\n",
    "IMPORT_FILE_GOLD_STANDARD = './data/dataset_febrl3_true_links.csv'\n",
    "df_true_links = pd.read_csv(IMPORT_FILE_GOLD_STANDARD)\n",
    "df_true_links.set_index(['rec_id_1','rec_id_2'],inplace=True)\n",
    "df_true_links.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create function to caculate metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [],
   "source": [
    "def metrics(links_true,links_pred,pairs):\n",
    "    if len(links_pred) > 0 :\n",
    "        matrix  = rl.confusion_matrix(links_true, links_pred, len(pairs))\n",
    "            \n",
    "        # precision\n",
    "        precision  = rl.precision(links_true, links_pred)\n",
    "\n",
    "         #precision\n",
    "        recall  = rl.recall(links_true, links_pred)\n",
    "\n",
    "        # The F-score for this classification is\n",
    "        fscore = rl.fscore(links_true,links_pred)\n",
    "        \n",
    "        return {'precision':precision, 'recall':recall,'fscore':fscore}\n",
    "    else :\n",
    "        return {'precision':0, 'recall':0,'fscore':0}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Function to Print Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Create Function to Print Results\n",
    "def get_results(metrics):\n",
    "    print(\"\\n{0:20}    {1:6}    {2:6}    {3:6}\".format('Matching ','Precision','Recall','Fscore'))\n",
    "    print('------------------------------------------------------')\n",
    "    for i in metrics.keys():\n",
    "        print(\"{0:20}    {1:<6.4}      {2:<6.4}      {3:<6.4}\".format(i,metrics[i]['precision'],\n",
    "                                                                      metrics[i]['recall'],\n",
    "                                                                      metrics[i]['fscore']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate quality metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_score = {}\n",
    "\n",
    "results_score['SimSum'] =  metrics(df_true_links,match,features)\n",
    "results_score['WeightedAverage'] = metrics(df_true_links,wa_matches,features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Matching                Precision    Recall    Fscore\n",
      "------------------------------------------------------\n",
      "SimSum                  1.0         0.9586      0.9788\n",
      "WeightedAverage         0.9984      0.9601      0.9789\n"
     ]
    }
   ],
   "source": [
    "get_results(results_score)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}