C3M1_Assignment.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "C3M1_Assignment.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "sn8ODLuvXAyn"
      ],
      "toc_visible": true,
      "include_colab_link": true
    },
    "coursera": {
      "schema_names": [
        "AI4MC3-1"
      ]
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mailaucq/Data-Science/blob/master/C3M1_Assignment.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "B16h5bb8eFmw"
      },
      "source": [
        "Welcome to the first assignment of **AI for Medical Treatment**!\n",
        "\n",
        "You will be using different methods to evaluate the results of a [randomized control trial](https://en.wikipedia.org/wiki/Randomized_controlled_trial) (RCT).\n",
        "\n",
        "**You will learn:**\n",
        "- How to analyze data from a randomized control trial using both:\n",
        "  - traditional statistical methods\n",
        "  - and the more recent machine learning techniques\n",
        "- Interpreting Multivariate Models\n",
        "  - Quantifying treatment effect\n",
        "  - Calculating baseline risk\n",
        "  - Calculating predicted risk reduction\n",
        "- Evaluating Treatment Effect Models\n",
        "  - Comparing predicted and empirical risk reductions\n",
        "  - Computing C-statistic-for-benefit\n",
        "- Interpreting ML models for Treatment Effect Estimation\n",
        "  - Implement T-learner"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8jtwQ5FqTmm9",
        "colab_type": "text"
      },
      "source": [
        "### This assignment covers the folowing topics:\n",
        "\n",
        "- [1. Dataset](#1)\n",
        "  - [1.1 Why RCT?](#1-1)\n",
        "  - [1.2 Data Processing](#1-2)\n",
        "      - [Exercise 1](#ex-01)\n",
        "      - [Exercise 2](#ex-02)\n",
        "- [2. Modeling Treatment Effect](#2)\n",
        "  - [2.1 Constant Treatment Effect](#2-1)\n",
        "      - [Exercise 3](#ex-03)\n",
        "  - [2.2 Absolute Risk Reduction](#2-2)\n",
        "      - [Exercise 4](#ex-04)\n",
        "  - [2.3 Model Limitations](#2-3)\n",
        "      - [Exercise 5](#ex-05)\n",
        "      - [Exercise 6](#ex-06)\n",
        "- [3. Evaluation Metric](#3)\n",
        "  - [3.1 C-statistic-for-benefit](#3-1)\n",
        "      - [Exercise 7](#ex-07)\n",
        "      - [Exercise 8](#ex-08)\n",
        "- [4. Machine Learning Approaches](#4)\n",
        "  - [4.1 T-Learner](#4-1)\n",
        "      - [Exercise 9](#ex-09)\n",
        "      - [Exercise 10](#ex-10)\n",
        "      - [Exercise 11](#ex-11)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Tklnk8tneq2U"
      },
      "source": [
        "## Packages\n",
        "\n",
        "We'll first import all the packages that we need for this assignment. \n",
        "\n",
        "\n",
        "- `pandas` is what we'll use to manipulate our data\n",
        "- `numpy`  is a library for mathematical and scientific operations\n",
        "- `matplotlib` is a plotting library\n",
        "- `sklearn` contains a lot of efficient tools for machine learning and statistical modeling\n",
        "- `random` allows us to generate random numbers in python\n",
        "- `lifelines` is an open-source library that implements c-statistic\n",
        "- `itertools` will help us with hyperparameters searching\n",
        "\n",
        "## Import Packages\n",
        "\n",
        "Run the next cell to import all the necessary packages, dependencies and custom util functions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:29:41.602385Z",
          "start_time": "2020-04-04T15:29:39.274097Z"
        },
        "colab_type": "code",
        "id": "Z5zOXfAIH-41",
        "colab": {}
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import sklearn\n",
        "import random\n",
        "import lifelines\n",
        "import itertools\n",
        "\n",
        "plt.rcParams['figure.figsize'] = [10, 7]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pVEHJZ79mvQx"
      },
      "source": [
        "<a name=\"1\"></a>\n",
        "## 1 Dataset\n",
        "<a name=\"1-1\"></a>\n",
        "### 1.1 Why RCT?\n",
        "\n",
        "In this assignment, we'll be examining data from an RCT, measuring the effect of a particular drug combination on colon cancer. Specifically, we'll be looking the effect of [Levamisole](https://en.wikipedia.org/wiki/Levamisole) and [Fluorouracil](https://en.wikipedia.org/wiki/Fluorouracil) on patients who have had surgery to remove their colon cancer. After surgery, the curability of the patient depends on the remaining residual cancer. In this study, it was found that this particular drug combination had a clear beneficial effect, when compared with [Chemotherapy](https://en.wikipedia.org/wiki/Chemotherapy). \n",
        "<a name=\"1-2\"></a>\n",
        "### 1.2 Data Processing\n",
        "In this first section, we will load in the dataset and calculate basic statistics. Run the next cell to load the dataset. We also do some preprocessing to convert categorical features to one-hot representations."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:29:41.612018Z",
          "start_time": "2020-04-04T15:29:41.602385Z"
        },
        "colab_type": "code",
        "id": "QOV_BJGyLtjR",
        "colab": {}
      },
      "source": [
        "data = pd.read_csv(\"levamisole_data.csv\", index_col=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "RlqE8036sj3y"
      },
      "source": [
        "Let's look at our data to familiarize ourselves with the various fields. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:29:45.698204Z",
          "start_time": "2020-04-04T15:29:45.677460Z"
        },
        "colab_type": "code",
        "id": "RPS1stb7si4N",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "outputId": "a64b50c6-5df2-467a-abee-0d73f82d7825"
      },
      "source": [
        "print(f\"Data Dimensions: {data.shape}\")\n",
        "data.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Data Dimensions: (607, 14)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sex</th>\n",
              "      <th>age</th>\n",
              "      <th>obstruct</th>\n",
              "      <th>perfor</th>\n",
              "      <th>adhere</th>\n",
              "      <th>nodes</th>\n",
              "      <th>node4</th>\n",
              "      <th>outcome</th>\n",
              "      <th>TRTMT</th>\n",
              "      <th>differ_2.0</th>\n",
              "      <th>differ_3.0</th>\n",
              "      <th>extent_2</th>\n",
              "      <th>extent_3</th>\n",
              "      <th>extent_4</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>43</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>True</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>63</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>True</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0</td>\n",
              "      <td>71</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>7.0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>66</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>True</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1</td>\n",
              "      <td>69</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>22.0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   sex  age  obstruct  perfor  adhere  nodes  node4  outcome  TRTMT  \\\n",
              "1    1   43         0       0       0    5.0      1        1   True   \n",
              "2    1   63         0       0       0    1.0      0        0   True   \n",
              "3    0   71         0       0       1    7.0      1        1  False   \n",
              "4    0   66         1       0       0    6.0      1        1   True   \n",
              "5    1   69         0       0       0   22.0      1        1  False   \n",
              "\n",
              "   differ_2.0  differ_3.0  extent_2  extent_3  extent_4  \n",
              "1           1           0         0         1         0  \n",
              "2           1           0         0         1         0  \n",
              "3           1           0         1         0         0  \n",
              "4           1           0         0         1         0  \n",
              "5           1           0         0         1         0  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ctvm6IEhauEd"
      },
      "source": [
        "Below is a description of all the fields (one-hot means a different field for each level):\n",
        "- `sex (binary): 1 if Male, 0 otherwise`\n",
        "- `age (int): age of patient at start of the study`\n",
        "- `obstruct (binary): obstruction of colon by tumor`\n",
        "- `perfor (binary): perforation of colon`\n",
        "- `adhere (binary): adherence to nearby organs`\n",
        "- `nodes (int): number of lymphnodes with detectable cancer`\n",
        "- `node4 (binary): more than 4 positive lymph nodes`\n",
        "- `outcome (binary): 1 if died within 5 years`\n",
        "- `TRTMT (binary): treated with levamisole + fluoroucil`\n",
        "- `differ (one-hot): differentiation of tumor`\n",
        "- `extent (one-hot): extent of local spread`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "WTfGBXTOsq06"
      },
      "source": [
        "In particular pay attention to the `TRTMT` and `outcome` columns. Our primary endpoint for our analysis will be the 5-year survival rate, which is captured in the `outcome` variable."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Mz2uT46QMQPc"
      },
      "source": [
        "<a name='ex-01'></a>\n",
        "### Exercise 01\n",
        "\n",
        "Since this is an RCT, the treatment column is randomized. Let's warm up by finding what the treatment probability is.\n",
        "\n",
        "$$p_{treatment} = \\frac{n_{treatment}}{n}$$\n",
        "\n",
        "- $n_{treatment}$ is the number of patients where `TRTMT = True`\n",
        "- $n$ is the total number of patients."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "WKpz5E_CLKQy",
        "colab": {}
      },
      "source": [
        "# UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
        "def proportion_treated(df):\n",
        "    \"\"\"\n",
        "    Compute proportion of trial participants who have been treated\n",
        "\n",
        "    Args:\n",
        "        df (dataframe): dataframe containing trial results. Column\n",
        "                      'TRTMT' is 1 if patient was treated, 0 otherwise.\n",
        "  \n",
        "    Returns:\n",
        "        result (float): proportion of patients who were treated\n",
        "    \"\"\"\n",
        "    \n",
        "    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
        "\n",
        "    proportion = len(df[(df.TRTMT == 1)])/df.shape[0]\n",
        "    \n",
        "    ### END CODE HERE ###\n",
        "\n",
        "    return proportion"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5SxT1zqgTmoK",
        "colab_type": "text"
      },
      "source": [
        "**Test Case**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L0tWUlVZTmoM",
        "colab_type": "code",
        "colab": {},
        "outputId": "fbcf631d-b175-4a4e-c408-05472c7a1c30"
      },
      "source": [
        "print(\"dataframe:\\n\")\n",
        "example_df = pd.DataFrame(data =[[0, 0],\n",
        "                                 [1, 1], \n",
        "                                 [1, 1],\n",
        "                                 [1, 1]], columns = ['outcome', 'TRTMT'])\n",
        "print(example_df)\n",
        "print(\"\\n\")\n",
        "treated_proportion = proportion_treated(example_df)\n",
        "print(f\"Proportion of patient treated: computed {treated_proportion}, expected: 0.75\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "dataframe:\n",
            "\n",
            "   outcome  TRTMT\n",
            "0        0      0\n",
            "1        1      1\n",
            "2        1      1\n",
            "3        1      1\n",
            "\n",
            "\n",
            "Proportion of patient treated: computed 0.75, expected: 0.75\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "BtHs90CWLinQ"
      },
      "source": [
        "Next let's run it on our trial data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:30:00.222152Z",
          "start_time": "2020-04-04T15:30:00.219183Z"
        },
        "colab_type": "code",
        "id": "Oz9j9egVLh2k",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "3a2ce4a7-4747-4bce-efe1-f73bb8304910"
      },
      "source": [
        "p = proportion_treated(data)\n",
        "print(f\"Proportion Treated: {p} ~ {int(p*100)}%\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Proportion Treated: 0.49093904448105435 ~ 49%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "DWvZ4Qvun8p1"
      },
      "source": [
        "<a name='ex-02'></a>\n",
        "### Exercise 02\n",
        "\n",
        "Next, we can get a preliminary sense of the results by computing the empirical 5-year death probability for the treated arm versus the control arm. \n",
        "\n",
        "The probability of dying for patients who received the treatment is:\n",
        "\n",
        "$$p_{\\text{treatment, death}} = \\frac{n_{\\text{treatment,death}}}{n_{\\text{treatment}}}$$\n",
        "\n",
        "- $n_{\\text{treatment,death}}$ is the number of patients who received the treatment and died.\n",
        "- $n_{\\text{treatment}}$ is the number of patients who received treatment.\n",
        "\n",
        "The probability of dying for patients in the control group (who did not received treatment) is:\n",
        "\n",
        "$$p_{\\text{control, death}} = \\frac{n_{\\text{control,death}}}{n_{\\text{control}}}$$\n",
        "- $n_{\\text{control,death}}$ is the number of patients in the control group (did not receive the treatment) who died.\n",
        "- $n_{\\text{control}}$ is the number of patients in the control group (did not receive treatment).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "etNHvX3AKleg",
        "colab": {}
      },
      "source": [
        "# UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
        "def event_rate(df):\n",
        "    '''\n",
        "    Compute empirical rate of death within 5 years\n",
        "    for treated and untreated groups.\n",
        "\n",
        "    Args:\n",
        "        df (dataframe): dataframe containing trial results. \n",
        "                          'TRTMT' column is 1 if patient was treated, 0 otherwise. \n",
        "                            'outcome' column is 1 if patient died within 5 years, 0 otherwise.\n",
        "  \n",
        "    Returns:\n",
        "        treated_prob (float): empirical probability of death given treatment\n",
        "        untreated_prob (float): empirical probability of death given control\n",
        "    '''\n",
        "    \n",
        "    treated_prob = 0.0\n",
        "    control_prob = 0.0\n",
        "        \n",
        "    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
        "    \n",
        "    treated_prob = len(df[(df.TRTMT == 1) & (df.outcome == 1)])/len(df[(df.TRTMT == 1)])\n",
        "    control_prob = len(df[(df.TRTMT == 0) & (df.outcome == 1)])/len(df[(df.TRTMT == 0)])\n",
        "    \n",
        "    ### END CODE HERE ###\n",
        "\n",
        "    return treated_prob, control_prob"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7VHvqkjUTmow",
        "colab_type": "text"
      },
      "source": [
        "**Test Case**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_rf8YQVrTmoy",
        "colab_type": "code",
        "colab": {},
        "outputId": "f2cf28a1-4fd1-411e-b1bd-c64ca63ec630"
      },
      "source": [
        "print(\"TEST CASE\\ndataframe:\\n\")\n",
        "example_df = pd.DataFrame(data =[[0, 1],\n",
        "                                 [1, 1], \n",
        "                                 [1, 1],\n",
        "                                 [0, 1],\n",
        "                                 [1, 0],\n",
        "                                 [1, 0],\n",
        "                                 [1, 0],\n",
        "                                 [0, 0]], columns = ['outcome', 'TRTMT'])\n",
        "#print(\"dataframe:\\n\")\n",
        "print(example_df)\n",
        "print(\"\\n\")\n",
        "treated_prob, control_prob = event_rate(example_df)\n",
        "print(f\"Treated 5-year death rate, expected: 0.5, got: {treated_prob:.4f}\")\n",
        "print(f\"Control 5-year death rate, expected: 0.75, got: {control_prob:.4f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TEST CASE\n",
            "dataframe:\n",
            "\n",
            "   outcome  TRTMT\n",
            "0        0      1\n",
            "1        1      1\n",
            "2        1      1\n",
            "3        0      1\n",
            "4        1      0\n",
            "5        1      0\n",
            "6        1      0\n",
            "7        0      0\n",
            "\n",
            "\n",
            "Treated 5-year death rate, expected: 0.5, got: 0.5000\n",
            "Control 5-year death rate, expected: 0.75, got: 0.7500\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ShpX6ABSV_Pd"
      },
      "source": [
        "Now let's try the function on the real data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:30:17.279595Z",
          "start_time": "2020-04-04T15:30:17.273594Z"
        },
        "colab_type": "code",
        "id": "7rw2yKymV-WD",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "9daebe7b-d0d1-4654-d3d1-764312b598d2"
      },
      "source": [
        "treated_prob, control_prob = event_rate(data)\n",
        "\n",
        "print(f\"Death rate for treated patients: {treated_prob:.4f} ~ {int(treated_prob*100)}%\")\n",
        "print(f\"Death rate for untreated patients: {control_prob:.4f} ~ {int(control_prob*100)}%\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Death rate for treated patients: 0.3725 ~ 37%\n",
            "Death rate for untreated patients: 0.4822 ~ 48%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "yoTzaBUorB-3"
      },
      "source": [
        "On average, it seemed like treatment had a positive effect. \n",
        "\n",
        "#### Sanity checks\n",
        "It's important to compute these basic summary statistics as a sanity check for more complex models later on. If they strongly disagree with these robust summaries and there isn't a good reason, then there might be a bug. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "fywUHcbRnsQZ"
      },
      "source": [
        "### Train test split\n",
        "\n",
        "We'll now try to quantify the impact more precisely using statistical models.  Before we get started fitting models to analyze the data, let's split it using the `train_test_split` function from `sklearn`. While a hold-out test set isn't required for logistic regression, it will be useful for comparing its performance to the ML models later on."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:30:22.034397Z",
          "start_time": "2020-04-04T15:30:21.942443Z"
        },
        "colab_type": "code",
        "id": "FUBvTfF0mQuH",
        "colab": {}
      },
      "source": [
        "# As usual, split into dev and test set\n",
        "from sklearn.model_selection import train_test_split\n",
        "np.random.seed(18)\n",
        "random.seed(1)\n",
        "\n",
        "data = data.dropna(axis=0)\n",
        "y = data.outcome\n",
        "# notice we are dropping a column here. Now our total columns will be 1 less than before\n",
        "X = data.drop('outcome', axis=1) \n",
        "X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size = 0.25, random_state=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:30:23.071470Z",
          "start_time": "2020-04-04T15:30:23.068473Z"
        },
        "colab_type": "code",
        "id": "6EeBLbfeFVnk",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "bd02e605-335a-4007-f1c0-46906dc0522c"
      },
      "source": [
        "print(f\"dev set shape: {X_dev.shape}\")\n",
        "print(f\"test set shape: {X_test.shape}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "dev set shape: (455, 13)\n",
            "test set shape: (152, 13)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2c8mLTMQEZxD"
      },
      "source": [
        "<a name=\"2\"></a>\n",
        "## 2 Modeling Treatment Effect"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "QxHy4RGA0Goi"
      },
      "source": [
        "<a name=\"2-1\"></a>\n",
        "### 2.1 Constant Treatment Effect\n",
        "\n",
        "First, we will model the treatment effect using a standard logistic regression. If $x^{(i)}$ is the input vector, then this models the probability of death within 5 years as \n",
        "$$\\sigma(\\theta^T x^{(i)}) = \\frac{1}{1 + exp(-\\theta^T x^{(i)})},$$\n",
        "\n",
        "where $ \\theta^T x^{(i)} = \\sum_{j} \\theta_j x^{(i)}_j$ is an inner product. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rEM8qN5YTmpa",
        "colab_type": "text"
      },
      "source": [
        "For example, if we have three features, $TRTMT$, $AGE$, and  $SEX$, then our probability of death would be written as: \n",
        "\n",
        "$$\\sigma(\\theta^T x^{(i)}) = \\frac{1}{1 + exp(-\\theta_{TRTMT} x^{(i)}_{TRTMT} - \\theta_{AGE}x_{AGE}^{(i)} - \\theta_{SEX}x^{(i)}_{SEX})}.$$\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9N-V-hBYTmpc",
        "colab_type": "text"
      },
      "source": [
        "Another way to look at logistic regresion is as a linear model for the \"logit\" function, or \"log odds\": \n",
        "\n",
        "$$logit(p) = \\log \\left(\\frac{p}{1-p} \\right)= \\theta^T x^{(i)}$$\n",
        "\n",
        "- \"Odds\" is defined as the probability of an event divided by the probability of not having the event: $\\frac{p}{1-p}$.  \n",
        "\n",
        "- \"Log odds\", or \"logit\" function, is the natural log of the odds: $log \\left(\\frac{p}{1-p} \\right)$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nUNxsw5YTmpe",
        "colab_type": "text"
      },
      "source": [
        "In this example, $x^{(i)}_{TRTMT}$ is the treatment variable. Therefore, $\\theta_{TRTMT}$ tells you what the effect of treatment is. If $\\theta_{TRTMT}$ is negative, then having treatment reduces the log-odds of death, which means death is less likely than if you did not have treatment. \n",
        "\n",
        "Note that this assumes a constant relative treatment effect, since the impact of treatment does not depend on any other covariates. \n",
        "\n",
        "Typically, a randomized control trial (RCT) will seek to establish a negative $\\theta_{TRTMT}$ (because the treatment is intended to reduce risk of death), which corresponds to an odds ratio of less than 1.\n",
        "\n",
        "An odds ratio of less than one implies the probability of death is less than the probability of surviving.\n",
        "\n",
        "$$ \\frac{p}{1-p} < 1 \\rightarrow p < 1-p$$\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bQTPytiOTmpf",
        "colab_type": "text"
      },
      "source": [
        "Run the next cell to fit your logistic regression model. \n",
        "\n",
        "You can use the entire dev set (and do not need to reserve a separate validation set) because there is no need for hyperparameter tuning using a validation set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:30:26.980302Z",
          "start_time": "2020-04-04T15:30:26.884988Z"
        },
        "colab_type": "code",
        "id": "U-2hcHYycgFJ",
        "colab": {}
      },
      "source": [
        "from sklearn.linear_model import LogisticRegression\n",
        "\n",
        "lr = LogisticRegression(penalty='l2',solver='lbfgs', max_iter=10000).fit(X_dev, y_dev)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m6eWAk4NTmps",
        "colab_type": "text"
      },
      "source": [
        "### Calculating the Odds ratio\n",
        "\n",
        "You are interested in finding the odds for treatment relative to the odds for the baseline.\n",
        "\n",
        "$$ OddsRatio = \\frac{Odds_{treatment}}{Odds_{baseline}}$$\n",
        "\n",
        "where\n",
        "$$Odds_{treatment} = \\frac{p_{treatment}}{1-p_{treatment}}$$\n",
        "\n",
        "and \n",
        "\n",
        "$$Odds_{baseline} = \\frac{p_{baseline}}{1-p_{baseline}}$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kXl3mD8bTmpt",
        "colab_type": "text"
      },
      "source": [
        "If you look at the expression\n",
        "\n",
        "$$\\log \\left(\\frac{p}{1-p} \\right)= \\theta^T x^{(i)} = \\theta_{treatment} \\times x_{treatment}^{(i)} + \\theta_{age} \\times x_{age}^{(i)} + \\cdots$$\n",
        "\n",
        "Let's just let \"$\\theta \\times x_{age}^{(i)} + \\cdots$\" stand for all the other thetas and feature variables except for the treatment $\\theta_{treatment}^{(i)}$, and $x_{treatment}^{(i)}$ ."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hGNMsP8cTmpu",
        "colab_type": "text"
      },
      "source": [
        "#### Treatment\n",
        "To denote that the patient received treatment, we set $x_{treatment}^{(i)} = 1$.  Which means the log odds for a treated patient are:\n",
        "\n",
        "$$ log( Odds_{treatment}) = \\log \\left(\\frac{p_{treatment}}{1-p_{treatment}} \\right) = \\theta_{treatment} \\times 1 + \\theta_{age} \\times x_{age}^{(i)} + \\cdots$$\n",
        "\n",
        "To get odds from log odds, use exponentiation (raise to the power of e) to take the inverse of the natural log.\n",
        "\n",
        "$$Odds_{treatment} = e^{log( Odds_{treatment})} = \\left(\\frac{p_{treatment}}{1-p_{treatment}} \\right) = e^{\\theta_{treatment} \\times 1 + \\theta_{age} \\times x_{age}^{(i)} + \\cdots}$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ltMOoj60Tmpw",
        "colab_type": "text"
      },
      "source": [
        "#### Control (baseline)\n",
        "\n",
        "Similarly, when the patient has no treatment, this is denoted by $x_{treatment}^{(i)} = 0$.  So the log odds for the untreated patient is:\n",
        "\n",
        "$$log(Odds_{baseline}) = \\log \\left(\\frac{p_{baseline}}{1-p_{baseline}} \\right) = \\theta_{treatment} \\times 0 + \\theta_{age} \\times x_{age}^{(i)} + \\cdots$$\n",
        "\n",
        "$$ = 0 + \\theta_{age} \\times x_{age}^{(i)} + \\cdots$$\n",
        "\n",
        "To get odds from log odds, use exponentiation (raise to the power of e) to take the inverse of the natural log.\n",
        "\n",
        "$$Odds_{baseline} = e^{log(Odds_{baseline})} = \\left(\\frac{p_{baseline}}{1-p_{baseline}} \\right) = e^{0 + \\theta_{age} \\times x_{age}^{(i)} + \\cdots}$$\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PhEiehgBTmpy",
        "colab_type": "text"
      },
      "source": [
        "#### Odds Ratio\n",
        "\n",
        "The Odds ratio is:\n",
        "\n",
        "$$ OddsRatio = \\frac{Odds_{treatment}}{Odds_{baseline}}$$\n",
        "\n",
        "Doing some substitution:\n",
        "\n",
        "$$ OddsRatio = \\frac{e^{\\theta_{treatment} \\times 1 + \\theta_{age} \\times x_{age}^{(i)} + \\cdots}}{e^{0 + \\theta_{age} \\times x_{age}^{(i)} + \\cdots}}$$\n",
        "\n",
        "Notice that $e^{\\theta_{age} \\times x_{age}^{(i)} + \\cdots}$ cancels on top and bottom, so that:\n",
        "\n",
        "$$ OddsRatio = \\frac{e^{\\theta_{treatment} \\times 1}}{e^{0}}$$\n",
        "\n",
        "Since $e^{0} = 1$, This simplifies to:\n",
        "\n",
        "$$ OddsRatio = e^{\\theta_{treatment}}$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "JVUl6hTRzA-w"
      },
      "source": [
        "<a name='ex-03'></a>\n",
        "### Exercise 03: Extract the treatment effect\n",
        "\n",
        "Complete the `extract_treatment_effect` function to extract $\\theta_{treatment}$ and then calculate the odds ratio of treatment from the logistic regression model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:30:29.151352Z",
          "start_time": "2020-04-04T15:30:29.146349Z"
        },
        "colab_type": "code",
        "id": "vePgJgTWeclb",
        "colab": {}
      },
      "source": [
        "# UNQ_C3 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
        "def extract_treatment_effect(lr, data):\n",
        "    theta_TRTMT = 0.0\n",
        "    TRTMT_OR = 0.0\n",
        "    coeffs = {data.columns[i]:lr.coef_[0][i] for i in range(len(data.columns))}\n",
        "    \n",
        "    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
        "    # get the treatment coefficient\n",
        "    theta_TRTMT = coeffs[\"TRTMT\"]\n",
        "    \n",
        "    # calculate the Odds ratio for treatment\n",
        "    TRTMT_OR = np.exp(theta_TRTMT)\n",
        "    \n",
        "    ### END CODE HERE ###\n",
        "    return theta_TRTMT, TRTMT_OR\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bn3oKYXFTmp9",
        "colab_type": "text"
      },
      "source": [
        "#### Test"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W1-zToHyTmp9",
        "colab_type": "code",
        "colab": {},
        "outputId": "c63541ac-60fa-472a-deac-3266b28ffc7b"
      },
      "source": [
        "# Test extract_treatment_effect function\n",
        "theta_TRTMT, trtmt_OR = extract_treatment_effect(lr, X_dev)\n",
        "print(f\"Theta_TRTMT: {theta_TRTMT:.4f}\")\n",
        "print(f\"Treatment Odds Ratio: {trtmt_OR:.4f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Theta_TRTMT: -0.2885\n",
            "Treatment Odds Ratio: 0.7494\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0zBWaNE_TmqD",
        "colab_type": "text"
      },
      "source": [
        "### Expected Output\n",
        "\n",
        "```CPP\n",
        "Theta_TRTMT: -0.2885\n",
        "Treatment Odds Ratio: 0.7494\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "clf289SQtTzV"
      },
      "source": [
        "Based on this model, it seems that the treatment has a beneficial effect.  \n",
        "- The $\\theta_{treatment} = -0.29$ is a negative value, meaning that it has the effect of reducing risk of death.\n",
        "- In the code above, the $OddsRatio$ is stored in the variable `TRTMT_OR`.\n",
        "- The $OddsRatio = 0.75$, which is less than 1.  \n",
        "\n",
        "\n",
        "You can think of the $OddsRatio$ as a factor that is multiplied to the baseline odds $Odds_{baseline}$ in order to estimate the $Odds_{treatment}$.  You can think about the Odds Ratio as a rate, converting between baseline odds and treatment odds.\n",
        "\n",
        "$$Odds_{treatment} = OddsRatio \\times Odds_{baseline}$$\n",
        "\n",
        "In this case:\n",
        "\n",
        "$$Odds_{treatment} = 0.75 \\times Odds_{baseline}$$\n",
        "\n",
        "So you can interpret this to mean that the treatment reduces the odds of death by $(1 - OddsRatio) = 1 - 0.75 = 0.25$, or about 25%.\n",
        "\n",
        "You will see how well this model fits the data in the next few sections."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "kgv-HoPGsBP-"
      },
      "source": [
        "<a name=\"2-2\"></a>\n",
        "### 2.2 Absolute Risk Reduction"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hVhcO3t2yj-4"
      },
      "source": [
        "<a name='ex-04'></a>\n",
        "### Exercise 4: Calculate ARR\n",
        "\n",
        "A valuable quantity is the absolute risk reduction (ARR) of a treatment. If $p$ is the baseline probability of death, and $p_{treatment}$ is the probability of death if treated, then \n",
        "$$ARR = p_{baseline} - p_{treatment} $$\n",
        "\n",
        "In the case of logistic regression, here is how ARR can be computed:  \n",
        "Recall that the Odds Ratio is defined as:\n",
        "\n",
        "$$OR = Odds_{treatment} / Odds_{baseline}$$\n",
        "\n",
        "where the \"odds\" is the probability of the event over the probability of not having the event, or $p/(1-p)$. \n",
        "\n",
        "$$Odds_{trtmt} = \\frac{p_{treatment}}{1- p_{treatment}}$$\n",
        "and\n",
        "$$Odds_{baseline} = \\frac{p_{baseline}}{1- p_{baseline}}$$\n",
        "\n",
        "In the function below, compute the predicted absolute risk reduction (ARR) given\n",
        "- the odds ratio for treatment \"$OR$\", and\n",
        "- the baseline risk of an individual $p_{baseline}$\n",
        "\n",
        "If you get stuck, try reviewing the level 1 hints by clicking on the cell \"Hints Level 1\".  If you would like more help, please try viewing \"Hints Level 2\"."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZhnaToRFTmqK",
        "colab_type": "text"
      },
      "source": [
        "<details>    \n",
        "<summary>\n",
        "    <font size=\"3\" color=\"darkgreen\"><b>Hints Level 1</b></font>\n",
        "</summary>\n",
        "<p>\n",
        "<ul>\n",
        "    <li> Using the given $p$, compute the baseline odds of death.</li>\n",
        "    <li> Then, use the Odds Ratio to convert that to odds of death given treatment.</li>\n",
        "    <li> Finally, convert those odds back into a probability</li>\n",
        "</ul>\n",
        "</p>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ahGYH2AdTmqK",
        "colab_type": "text"
      },
      "source": [
        "<details>    \n",
        "<summary>\n",
        "    <font size=\"3\" color=\"darkgreen\"><b>Hints Level 2</b></font>\n",
        "</summary>\n",
        "<p>\n",
        "<ul>\n",
        "    <li> Solve for p_treatment starting with this expression: Odds_treatment = p_treatment / (1 - p_treatment).  You may want to do this on a piece of paper.</li>\n",
        "</ul>\n",
        "</p>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "CCCmR2lQjDzs",
        "colab": {}
      },
      "source": [
        "# UNQ_C4 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
        "def OR_to_ARR(p, OR):\n",
        "    \"\"\"\n",
        "    Compute ARR for treatment for individuals given\n",
        "    baseline risk and odds ratio of treatment.\n",
        "\n",
        "    Args:\n",
        "        p (float): baseline probability of risk (without treatment)\n",
        "        OR (float): odds ratio of treatment versus baseline\n",
        "\n",
        "    Returns:\n",
        "        ARR (float): absolute risk reduction for treatment \n",
        "      \"\"\"\n",
        "    \n",
        "    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
        "\n",
        "    # compute baseline odds from p\n",
        "    odds_baseline = p/(1 - p)\n",
        "\n",
        "    # compute odds of treatment using odds ratio\n",
        "    odds_trtmt = OR*odds_baseline\n",
        "\n",
        "    # compute new probability of death from treatment odds\n",
        "    p_trtmt = odds_trtmt/(1+odds_trtmt)\n",
        "\n",
        "    # compute ARR using treated probability and baseline probability \n",
        "    ARR = p - p_trtmt\n",
        "    \n",
        "    ### END CODE HERE ###\n",
        "    \n",
        "    return ARR"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7mlycVrDTmqR",
        "colab_type": "text"
      },
      "source": [
        "**Test Case**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VOAV2fXzTmqT",
        "colab_type": "code",
        "colab": {},
        "outputId": "55503251-a974-4d6a-c310-475ba0594a00"
      },
      "source": [
        "print(\"TEST CASES\")\n",
        "test_p, test_OR = (0.75, 0.5)\n",
        "print(f\"baseline p: {test_p}, OR: {test_OR}\")\n",
        "print(f\"Output: {OR_to_ARR(test_p, test_OR):.4f}, Expected: {0.15}\\n\")\n",
        "\n",
        "test_p, test_OR = (0.04, 1.2)\n",
        "print(f\"baseline p: {test_p}, OR: {test_OR}\")\n",
        "print(f\"Output: {OR_to_ARR(test_p, test_OR):.4f}, Expected: {-0.0076}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TEST CASES\n",
            "baseline p: 0.75, OR: 0.5\n",
            "Output: 0.1500, Expected: 0.15\n",
            "\n",
            "baseline p: 0.04, OR: 1.2\n",
            "Output: -0.0076, Expected: -0.0076\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "LLxmh1h92FFe"
      },
      "source": [
        "#### Visualize the treatment effect as baseline risk varies\n",
        "\n",
        "The logistic regression model assumes that treatment has a constant effect in terms of odds ratio and is independent of other covariates. \n",
        "\n",
        "However, this does not mean that absolute risk reduction is necessarily constant for any baseline risk $\\hat{p}$. To illustrate this, we can plot absolute risk reduction as a function of baseline predicted risk $\\hat{p}$. \n",
        "\n",
        "Run the next cell to see the relationship between ARR and baseline risk for the logistic regression model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:30:45.443881Z",
          "start_time": "2020-04-04T15:30:45.270615Z"
        },
        "colab_type": "code",
        "id": "eQdG21ogqTWy",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 458
        },
        "outputId": "16531142-20c9-459e-8dde-f239c1e31203"
      },
      "source": [
        "ps = np.arange(0.001, 0.999, 0.001)\n",
        "diffs = [OR_to_ARR(p, trtmt_OR) for p in ps]\n",
        "plt.plot(ps, diffs)\n",
        "plt.title(\"Absolute Risk Reduction for Constant Treatment OR\")\n",
        "plt.xlabel('Baseline Risk')\n",
        "plt.ylabel('Absolute Risk Reduction')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nOzdd3wUdfrA8c+TSksoIbQkEHqTHkBUFMGCiKBiAZUDz5NTT72f5TzvzrOdZznv9M4uiL1iR0WxgqD03iWEFmoINZCe5/fHTHQNm2SBbCabPO/Xa1+7OzM788zuzD7zne93viOqijHGGFNSmNcBGGOMqZosQRhjjPHLEoQxxhi/LEEYY4zxyxKEMcYYvyxBGGOM8csSRBlE5GUReaCC5zleRGZX5DyPl4i0FJEsEQkvZzpPYhaRZBFREYkIwrxXicigIMy3qYh8LyKHROQ/FT1/YyqTJQhARGaIyD4RifY6Fl8n+sfsfr7QTQIHRWSZiAwvHq+qW1S1nqoWVkzEICL3iki+u8z9IvKjiAyoqPkfZ0xHJXpV7aqqM4KwuAnAHiBWVW+riBmKSHMRmSwiO9zEs1ZE7hORuhUx/1KWOUNEfleB89skImeVMu5Kd3vJEpFsESnyeZ9VUTG4y1IRaVeR8zyGZd8rIq8HMN14EVkhIkdEZKeIPCsiDUrMp1L2sRqfIEQkGRgIKDDC02CCY46q1gMaAM8Ab/tubEHyjrvMxsB3wLtBXl5V0gpYrcdxBaq/kpKINALmALWBAaoaA5yN83u2PcFYqwRVfcM9UKkHnAdsL37vDvtZeaXdUCcitwGPAH8C6gMn42xTX4lIlM+klbOPqWqNfgB3Az8AjwGflhj3MvAc8BVwCJgJtHLHCfA4sBs4CKwATnLH1QdeBTKAzcBdQJg7bjww232djJOYInyWOQP4HdAZyAEKgSxgvzs+Gvg3sAXY5cZXu5R1+3lZ7vs67vL6+lu+O32au64bgStLmc+jwGygvp9l3gu87vO+i7uMeJ/vZjKwA9gGPACEu+PC3XXb48bxhxLxbQLOKmNZpwE/AvuBrW7cE4B8IM/9Hj8pOS/3O/0vsN19/BeIdscNAtKB29zfegdwdSnf98sllnVWgPP+M7ATeM3PPB/A2bbCytiGTwEWAAfc51NKbE//wNnGDwFfAo3dcbWA14FM9ztbADQF/omz3eW46/GUO/3/3O/1ILAIGFjit5iCs90fAlYBKe6414AiINud3x1lrMsgIL3Ed/osMA047H6nLYD3cfavjcDNPtP3w0mo+93f6ikgyh33Pc72dNiN43Kf3+AOn9/3QmAY8BOwF/irz/zDgDuBDe73NgVoVGJ/Goezf+4B/uaOG+puF/nuspf5WfdYd9xlJYbXc9f1t4HsYxX6/xiMP91QegCpwA1AH/fHa1pi4zwEnI6zo/+PX/7cz3V3kgY4yaIz0Nwd9yrwMRDjbjQ/Ade448YTQIIoOa3P+MeBqUAjd/6fAA+Vsm6+ywrH+cPNA5qUXD5QF2fH7+iOaw509Z2Pu3NMAqYDdUpZ5s8bLxAFPOzuKMV/8h8Cz7vLawLMB37vjrsOWAskuev3HQEmCJyjrEPAGCASiAN6+vyOD5SI8+d5AfcDc9144nGSzD/ccYOAAneaSJw/jiNAw1LW/1fLCnDej+BsX0clevez95Wx/TYC9gFj3d9xjPs+zmd72gB0wCmFzAAedsf9Hmf7qeNuH31wTo0Vf+53JZZ1lfu9RuAkzJ1ALZ/fIsf9fsKBh4C5/r7vcvbHQRydIA4Ap+Jsf3Vw9ru7cbavNjgHE+e60/fBOeqOwNm+1wD/5zM/BdqVWF6BO79I4FqcP+M3cfavrjiJrbU7/R/d3yTR/c2eB94qsT9Ncr/rHkAu0Lnk9lrKug91Y4nwM+4Vn+X8PB/87GMV+v9Y0TMMpQfOEWc+vxxRrQVuKbFxvu3zvh7OkVUSMBjnj/9kfI7u3J0jD+jiM+z3wAz39XiOM0HgJKLDQFufYQOAjaWs33h3g9vvrmc2PkcnHJ0g9gOjKPFH5c5nHvAOzpFbVBnf6b3u+u93v6tMYJA7rqm7w9T2mX4M8J37+lvgOp9x5xB4gvgL8GEpMb1M2QliAzDMZ9y5wCb39SD3e/P9jXYDJweyrADmnYf7J1vK/Nb7fid+xo8F5pcYNgcY77M93eUz7gbgC/f1b3ESVnc/851BiQThZ5p9QA+f3+Jrn3FdgGx/33c58xzE0QniVZ/3/YEtJT7zF+ClUub3f77bBf4TRDa/lGJj3Gn6+0yzCLjQfb0GGOIzrjnOvlWckBRI9Bk/HxhdcnstJdargJ2ljHsY+Kq8fayiHzW9DmIc8KWq7nHfv+kO87W1+IWqZuEUOVuo6rc4xdengd0iMlFEYnHOCUbinFoqthlIqIB443GPoNzKqf3AF+7w0sxV1QZAQ5ySx0B/E6nqYZwi93XADhH5TEQ6+UzSDhiJczSbV06cU9xlNgVW4hzVgXOUH+nOvzj+53GOrsE5dbDVZz6+32F5knD+jI9HC47+vVr4vM9U1QKf90dwDhYqYt4ZqppTxuczcf6EAp1/8TJ8t7edPq99Y38NpzT4tohsF5F/iUhkaQsSkdtFZI2IHHB/u/o423tpy6lVQS3QfLeJVkCL4u3HjeOvONsaItJBRD51K3cPAg+WiNGfTP2loUa2+7zLZ3w2v3xnrYAPfZa9BudPuqnP9KV93+XZAzQu5Ttr7o4vVto+VqFqbIIQkdrAZcAZ7sa0E7gF6CEiPXwmTfL5TD2cIv12AFV9QlX74BwtdcCpWNqDc0TRymceLXHOt5d02H2u4zOsmc9rLTH9HpyNtauqNnAf9bVERZ4/bnK7HhgrIr1KmWa6qp6NszGuxSkqF1sDXA18LiIdy1ueO789OHUA94pIc5wdPRenxFYcf6yqdnU/sgOf7xvne/N1mNK/q62UXmlb8nssaTtH/17by/lMoMqbd3mxfQ1cJCKl7asl51+8DH/b26+oar6q3qeqXXDqMYYDv/EXl4gMxDlPfxnO6bUGOKd+pLzl+JvfMfL97FacEnMDn0eMqg5zxz+Ls+22V9VYnOQRaIyB2AqcV2L5tVS13O+b8r+DOTj7x8W+A93/nfOAb46a4dH7WIWqsQkCpyKqEOfPvaf76AzM4pedBGCYiJzmtiD4B84R+VYR6Ssi/d0jrsM451+L3CORKcA/RSRGRFoBt+JUBv6Kqmbg7MhXiUi4iPyWX//J7QISi1svqGoRzp/24yLSBEBEEkTk3EBWWFX3Ai/gnG/9Fbf9/ki36WQuTmVZUYnPv4Wzw30tIgG1oFHVdThHqXeo6g6cStL/iEisiISJSFsROcOdfApws4gkikhDnMpAX0uB0SISKSIpwCU+494AzhKRy0QkQkTiRKSnO24Xzrnq0rwF3CUi8SLSGOf7Kbc5YoBOdN6P4VRevuJuS8W/+WMi0h2n8raDiFzhrvflONv0p+XNWETOFJFubsuggzgHNsW/ecnvLAbndGUGECEid7txBaq83yBQ84FDIvJnEant7jcniUhfnzgPAlluCfj6Co7jOZx9u/i3iBeRkQF+dheQXFqyV9UDwH3AkyIy1N3Ok3H2i3ScEp+/z/28jx3LigSiJieIcTjnLbeo6s7iB85poyt9inlvAvfgnFrqg3OeEJydYxLOedjNOKcCHnXH3YSTNNJwKnffBF4sJY5rcUoemTgVYj/6jPsWpzXIThEpLl7+Gadifa5bhP4aCOiI3vVfnKTXvcTwMJxEtt1d1zM4eudCVV/BqXj91t14A/EoMMFNar/BqVhbjfPdvccvp1CKK8CXAYuBD0rM5+84CXQfzo70pk9cW3AqSG9z41+KU0kITqupLu5pgY/8xPcAsBBYjtNiaLE7rCKc0LzdpH4Kzp/3PBE5hHMkeQBIVdVMnCP/23C2oTuA4T6nTcvSDOf7P4hTQpzJL39C/wMuEef6oCdwfpcvcOrdNuMcEG09ao6lewgnUe4XkduP4XO/4h6ADcc5oNuIU6p+Aed0F8DtwBU4DRYm4dSb+boXJ9nuF5HLjiOE/+Gcqv3S/S3m4tSLBKK4KWqmiCz2N4Gq/gvnIOzfOL/LPJzveYiq5pYxb999rMKIW+lhjDHG/EpNLkEYY4wpgyUIY4wxflmCMMYY45clCGOMMX5VeDfKXmncuLEmJyd7HYYxxoSURYsW7VFVvxfbVpsEkZyczMKFC70OwxhjQoqIlNpjgZ1iMsYY45clCGOMMX5ZgjDGGOOXJQhjjDF+WYIwxhjjlyUIY4wxflmCMMYY41e1uQ7CmFBxOLeAnQdzyMzKY09WLnsP55GTX0heYRH5BUqRKrWjwqkVEUbtqHAa1ImiaWwtmsREEx8TTWS4HdeZymEJwpggySsoYu3Ogyzbup/VOw6xcU8WaRmH2X2orG79yxYm0LJRHdo1iaF903p0ahZDn1YNSWhQG5GKvHGaMZYgjKkwuQWFLNq8j9nr9zAnLZNV2w6SV+jcoK1BnUjaxtfj9A7xtG5cl4QGtYmrF0XjetE0qhtFrYhwoiLCiIoI+3le2XmFZOcXsu9wPrsP5bDrYC47DmSzISOL9buymLFuNwVFzv1cmsXWok9yQ05t25gzO8XTvH5tz74HU31YgjDmBBzIzuer1bv4fMUOftyQSXZ+IeFhQs+kBlx9ajI9khrQI6kBLerXOqYj/DpREdSJcnbPxIbwyw3TfpFfWMS6nYdYtHkfizbvY+GmvXy2fAcAnZvHMqRTEy7o0YKOzWIqYlVNDRTUO8qJyFCcW/SFAy+o6sMlxkcDr+LcyjMTuFxVN4nIlTi34SzWHeitqktLW1ZKSopaX0ymMuTkF/L5yh18smwHs9ZnkF+oJDSozZDOTRjYPp6T2zQiplZkpcelqqTuzuLbtbv5Zu1uFm3eR2GR0rl5LBf3SmBkzxY0ia1V6XGZqk1EFqlqit9xwUoQ7o3QfwLOxrnh9gJgjKqu9pnmBqC7ql4nIqOBi1T18hLz6QZ8pKpty1qeJQgTbOt2HuKt+Vv4YHE6B3MKSGhQm2HdmjGsW3N6JjWocnUAe7Jy+XTZdj5cso1l6QeICBOGntSM8ack06dVwyoXr/FGWQkimKeY+uHcVD3NDeJtYCTOzeqLjcS5iTg4N09/SkREf521xgBvBzFOY0pVVKR8vWYXk2alsWDTPqLCwzj3pGaM6ZfEya3jCAurun+yjetFM/7U1ow/tTUbMrJ4a94W3lm4lU+X76Bri1gmnN6G4d1bEF6F18F4K5gliEuAoar6O/f9WKC/qt7oM81Kd5p09/0Gd5o9PtNsAEaq6ko/y5gATABo2bJln82bS+211phjkltQyEdLtvH892mkZRwmoUFtxp+SzKg+iTSqG+V1eMftSF4BHy7Zxks/bCJ1dxZt4uty0+B2XNC9BRHWfLZG8qoEccJEpD9wxF9yAFDVicBEcE4xVWZspnrKLyzi3YXpPPHNenYezKFri1ieGNOLYSc1qxZ/oHWiIriyfyvG9G3JF6t28sQ367nlnWU88U0qd5zbkaEnNbNTT+ZnwUwQ24Akn/eJ7jB/06SLSAROU41Mn/GjgbeCGKMxABQWKZ8s287jX//E5swj9G7ZgEcv7c5p7RpXyz/MsDBhWLfmDO3ajC9X7+Kxr9Zx/RuL6ZvckLvO70KPpAZeh2iqgGAmiAVAexFpjZMIRgNXlJhmKjAOmANcAnxbXP8gImHAZcDAIMZoDPPSMrn3k9Ws2XGQzs1jeXF8Cmd2bFItE0NJYW7F9VmdmzBlYTqPfbWOkU//wMW9E/jbsM7E1Yv2OkTjoaAlCFUtEJEbgek4zVxfVNVVInI/sFBVpwKTgddEJBXYi5NEip0ObC2u5Damom3fn82D09bw6fIdJDSozRNjejG8W/MqXfEcLBHhYVzRvyUX9GjO099tYPLsNL5bu5u/nd+FUb0TakSyNEcL6nUQlcmauZpA5RcWMfH7NJ78dj2qcP2gtvz+9LbUjgr3OrQq46ddh/jLBytYtHkfp7SN46GLu9Eqrq7XYZkg8OQ6iMpmCcIEYnn6fu54bzlrdx5iaNdm3DW8M4kN63gdVpVUVKS8OX8Lj3y+lkJV7r2gK5emJFppopoJ2VZMxlSU7LxC/vv1T0yalUZ8TDQTx/bhnK7NvA6rSgsLE646uRWDOzXhtinLuOP95Xy9ZhcPj+oe0k19TeCsBGGqvaVb93PLO0vZuOcwY/olced5nalfu/K7wghlRUXK5NkbeXT6OurXieSJ0b0Y0DbO67BMBSirBBH6DbuNKUVhkfLUt+sZ9eyP5BUU8ea1/Xno4u6WHI5DWJhw7elt+PjGU4mtFcGVL8zl2RkbqC4HmMY/SxCmWkrfd4QxE+fy7y9/Yli35kz740BOadvY67BCXufmsXx842mc1605j3yxlgmvLeJAdr7XYZkgsQRhqp3pq3Zy3v9msXrHQR67rAdPjO5ppYYKVC86gqfG9OLu4V34bu1uRj41mw0ZWV6HZYLAEoSpNgoKi3jo8zX8/rVFtG5cl2k3D+Ti3tbqJhhEhN+e1pq3JpzMoZwCLn7mR37csKf8D5qQYgnCVAsZh3IZO3k+z89M44r+LXn3ugG0jLPmq8HWN7kRH/3hVJrERPObyfOZsmCr1yGZCmQJwoS8xVv2MfzJWSzeso9/X9qDBy/qRnSEXfRWWZIa1eH9G05hQNs47nh/OQ9/vtYqr6sJSxAmpH28dBujJ84lOiKcD244hUv6JHodUo0UWyuSl8b35cr+LXlu5gbueG85Be79uE3osgvlTEhSVR7/ej1PfLOefq0b8fxVfWhoF295KiI8jAcuPInG9aL53zfrOZCdzxNjelEr0kpzocpKECbk5OQXctNbS3jim/Vc2ieR16/pb8mhihARbjm7A/dc0IUvV+/i6pcWcCjHmsGGKksQJqTsycrl8olz+WzFDu48rxP/uqQ7URG2GVc1V5/amv9e3pP5m/Zy1Qvz7FqJEGV7lgkZW/ce4dLn5rBu50Geu6oP153R1pqwVmEX9krg+av6sHrHQX4z2ZJEKLIEYULCmh0HGfXsj+w9nMcbv+vPudbRXkg4q0tTnr3STRIvzuegnW4KKZYgTJU3Ly2Ty56fQ5gI7143gD6tGnkdkjkGZ3VpytNX9Gb19gOMnWxJIpRYgjBV2lerdzH2xfnEx0Tz/g2n0KFpjNchmeNwTtdmPH1Fb1ZtO8D4F+dzJK/A65BMACxBmCrrs+U7uP71RXRuHst7151CQoPaXodkTsA5XZvx5JheLN26n+tfX0xegV0nUdUFNUGIyFARWSciqSJyp5/x0SLyjjt+nogk+4zrLiJzRGSViKwQkVrBjNVULR8t2cZNby2mV8sGvH5NP7tBTTVxXrfmPHhRN2b+lMFt7y6jqMiuuK7KgnahnIiEA08DZwPpwAIRmaqqq30muwbYp6rtRGQ08AhwuYhEAK8DY1V1mYjEAXbisoZ4b1E6f3pvGf1bN2LyuL7UjbbrOauT0f1asj87n4c/X0vDOpHcN6KrtUarooK55/UDUlU1DUBE3gZGAr4JYiRwr/v6PeApcbaUc4DlqroMQFUzgxinqULemr+Fv364glPbNmbSb1KoHWVX4VZH153Rln2H83j++zQa1Y3i/87q4HVIxo9gnmJKAHy7dkx3h/mdRlULgANAHNABUBGZLiKLReQOfwsQkQkislBEFmZkZFT4CpjK9db8LfzlgxWc0SGeF8ZZcqju7jyvE5f2SeS/X6/n/UXpXodj/KiqldQRwGnAle7zRSIypOREqjpRVVNUNSU+Pr6yYzQV6MMl6fz1Qyc5PD+2j/XfUwOICA9e3I1T28Vx5wfLmbPBThRUNcFMENuAJJ/3ie4wv9O49Q71gUyc0sb3qrpHVY8A04DeQYzVeOiLlTu4/d3l9G/diOfH9rGuumuQyPAwnrmyD63i6nLd64vsznRVTDATxAKgvYi0FpEoYDQwtcQ0U4Fx7utLgG/V6Uh+OtBNROq4ieMMfl13YaqJ79bu5qa3ltAjsT4vjOtrJYcaqH5tp6vwiDDh6pcWkJmV63VIxhW0BOHWKdyI82e/BpiiqqtE5H4RGeFONhmIE5FU4FbgTvez+4DHcJLMUmCxqn4WrFiNN35M3cN1ry+iY7MYXrq6H/WstVKNldSoDpPGpbDrYA7Xvb7IrpGoIqS63PkpJSVFFy5c6HUYJkBLt+7niklzSWxYm7cnDLDrHAwAU5dt5+a3lnDVyS154MJuXodTI4jIIlVN8TfODtlMpUvLyOK3Ly8grl4Ur1/T35KD+dmIHi1Yte0Az3+fRreE+lzet6XXIdVoVbUVk6mmdh/K4Tcvzgfg1d/2p0msXSBvfu2OoZ0Y2L4xf/9oFYu37PM6nBqt3AQhIvEi8lcRmSgiLxY/KiM4U70cysln/IsLyMzK46XxfWnduK7XIZkqKDxMeHJML5rWj+a61xax+2CO1yHVWIGUID7GaX76NfCZz8OYgOUVFHHd64v4adchnr2qNz2SGngdkqnCGtSJYuLYFA7lFPCHNxdTUGiV1l4IJEHUUdU/q+oUVX2/+BH0yEy1UVSk3P7uMn5IzeSRUd0Z1LGJ1yGZENC5eSwPj+rGgk37eOyrn7wOp0YKJEF8KiLDgh6JqbYe++onpi7bzp+HdmJUn0SvwzEhZGTPBMb0S+KZGRuYsW631+HUOIEkiD/iJIkcETnkPg4GOzBTPXywOJ2nvktldN8krjujjdfhmBB0zwVd6dQshlunLGPHgWyvw6lRyk0QqhqjqmGqWst9HaOqsZURnAltCzbt5c73V3BK2zj+ceFJ1qWzOS61IsN5+sre5OQXcvNbS6w+ohIF1MxVREaIyL/dx/BgB2VC3+bMw0x4dSGJDWvz7JV9iAy3FtXm+LWNr8eDF1l9RGULpJnrwzinmVa7jz+KyEPBDsyErgPZ+fz25QUo8OL4vtSvE+l1SKYauLBXAqP7JvHszA38uGGP1+HUCIEc1g0DzlbVF1X1RWAocH5wwzKhqqCwiD+8sZgte4/w/FV9SLZrHUwFuvuCLrSOq8ttU5Zx4IjdZDLYAi33+zZarx+MQEz18PDna5mduod/XtSN/m3ivA7HVDN1oiJ4/PKeZBzK5e8fr/Q6nGovkATxELBERF4WkVeARcA/gxuWCUUfL93GC7M3Mm5AKy5LSSr/A8Ychx5JDfjjkPZMXbadj5eWvMWMqUiBtGJ6CzgZ+AB4Hxigqu8EOzATWlZtP8Cf319Ov9aNuGt4F6/DMdXc9YPa0qdVQ+76aCXb9lvT12ApNUGISCf3uTfQHOcub+lAC3eYMQDsPZzHhFcX0bBOFM9c2dtaLJmgiwgP4/HLelJUpNz6zlIKi6rHbQuqmrK6+74VmAD8x884BQYHJSITUgoKi7jprcVkZOXy7u8H0LhetNchmRqiZVwd7hnRlTveW87LP27imtNaex1StVNqglDVCe7L81T1V90pioj10WwAeOSLtfyQmsmjl3S3DvhMpbu0TyJfrNzJo9PXMqRTE2s1V8ECORfwY4DDTA0zbcUOJs1yKqUvtUpp4wER4cGLuhEZHsYd7y+nyE41Vaiy6iCaiUgfoLaI9BKR3u5jEFCn0iI0VdKmPYe5473l9GrZgL+db5XSxjvN6tfi7+d3Yf7Gvbw+b7PX4VQrZZUgzgX+DSTi1EMUP24B/hrIzEVkqIisE5FUEbnTz/hoEXnHHT9PRJLd4ckiki0iS93Hc8e2WiaYcvILueGNxUSEC09d0ZuoCKuUNt66NCWR0zvE8/Dna9m694jX4VQbpe7ZqvqKqp4JjFfVwap6pvsYqaoflDdjEQkHngbOA7oAY0Sk5KHmNcA+VW0HPA484jNug6r2dB/XHeuKmeC575PVrN5xkMcv60lCg9peh2MMIsJDF3cjTIQ7P1iOqp1qqgiBHPr1EZGfax9FpKGIPBDA5/oBqaqapqp5wNvAyBLTjARecV+/BwwR6/KzSvtwSTpvzd/C9YPacmYnu/GPqToSGtTmL8M68UNqJu8s2Op1ONVCIAniPFXdX/xGVffh9M9UngTA91dKd4f5nUZVC4ADQHH/DK1FZImIzBSRgf4WICITRGShiCzMyMgIICRzItbvOsRfP1hJv+RG3HZ2B6/DMeYoV/RrycltGvHgtDVkHMr1OpyQF0iCCBeRnxu3i0htINiN3XcALVW1F871GG+KyFH3oFDViaqaoqop8fHxQQ6pZjuSV8ANbyymTlQ4T17Riwi7GM5UQSLCAxd2Izu/kH9+ttrrcEJeIHv5G8A3InKNiFwDfMUvp4XKsg3wbfuY6A7zO42IROB0BJipqrmqmgmgqouADYAdsnro/k9Wk5qRxX9H96RprF0GY6qudk3qcf0Zbflo6XZmr7duwU9EIH0xPYLTOV9n9/EPVf1XAPNeALQXkdYiEgWMBqaWmGYqMM59fQnwraqqiMS7ldyISBugPZAWyAqZivf5ih28vWAr15/RloHtraRmqr4bzmxHclwd/v7xSnLyC70OJ2QFdJ5AVT9X1dvdx/QAP1MA3AhMB9YAU1R1lYjcLyIj3MkmA3EikopzKqm4KezpwHIRWYpTeX2dqu4NfLVMRdm+P5s7P1hBj8T63GL1DiZE1IoM54ELu7Fxz2GembHB63BCVll9MQEgIodw+l4CiAIigcOB3JdaVacB00oMu9vndQ5wqZ/PvY/Tc6zxUGGRcss7S8kvLOJ/o3tZJ3wmpJzWvjEje7bguRkbGNGjBe2a1PM6pJATyCmmGFWNdRNCbWAU8EzQIzOee27mBuZt3Mv9I0+yPm5MSLrr/C7Uigzj7x+ttGsjjsMxHRKq4yOcq6xNNbZki3Nz+At6tGBU75Ktk40JDfEx0fzp3I7MSctk2oqdXocTcgI5xXSxz9swIAXIKWVyUw1k5Rbwx7eX0iy2Fg9ceBJ27aIJZVf0b8Wb87fyz89WM7hTE2pHhXsdUsgIpARxgc/jXOAQR18RbaqRez5eRfq+I/xvdE/q1470OhxjTkh4mHDvBV3YfiCHZ2dahfWxKLcEoapXV0Ygpmr4YuVO3l+czs2D25GS3MjrcGCjfn4AACAASURBVIypEP3bxDGiRwuem7mBS/skktTIOqQORKkJQkSe5JfWS0dR1ZuDEpHxzJ6sXP724QpOSojlpiHtvQ7HmAr1l2Gd+Gr1Lh74bDXPj03xOpyQUNYppoXAIqAW0BtY7z564jR3NdWIqvK3D1dwKLeAxy7raU1aTbXTvH5tbhzcjumrdjFrvfXdFojyuvt+BegODFLVJ1X1SWAITpIw1ciHS7YxfdUubj+nAx2axngdjjFBcc1prWkVV4f7PllNfmGR1+FUeYEcJjYEfC+Kq+cOM9XE9v3Z3DN1FX2TG3LNaW28DseYoKkVGc5d53chdXcWb8/f4nU4VV4gCeJhYImIvCwirwCLgQeDG5apLKrKn99fTmGR8u9LexAeZk1aTfV2VucmnNymEY9/vZ6DOfleh1OlBXIl9UtAf+BD4ANggHvqyVQDr8/bwqz1e/jrsM60irOrpU31JyL8bVgX9h7O4znrp6lM5SYI9w5vZwE9VPVjIEpE+gU9MhN0mzMP8+BnaxjYvjFX9m/pdTjGVJpuifW5qFcCk2dvZNv+bK/DqbICOcX0DDAAGOO+P4Rzr2kTwoqKnFNLEWHCvy7pbldLmxrn9nM7osB/pq/zOpQqK5AE0V9V/4DbvYZ7y1Fr5hri3l6wlblpe/nr+Z1pXr+21+EYU+kSGtTmmtNa88GSbazcdsDrcKqkQBJEvnvzHgUQkXjA2oeFsB0Hsnlo2hoGtIljdN+k8j9gTDV1/aC2NKobxQOfrbbeXv0IJEE8gVNB3URE/gnMBh4KalQmaFSVuz5cSX5REQ+P6manlkyNFlsrklvOas/ctL18vWa31+FUOYG0YnoDuAMnKewALgQ+C3JcJkimLtvON2t3c/s5Ha3VkjHA6H4tadO4Lv/6Yi2FRVaK8FVmghCRBBFJAdJU9WlgCjAWp8sNE2Iys3K575PV9EhqwNWntvY6HGOqhMjwMG47pyPrd2fx8dJtXodTpZSaIETk/4ClwJPAXBH5Hc69pWsDfQKZuYgMFZF1IpIqInf6GR8tIu+44+eJSHKJ8S1FJEtEbg98lUxp7v90NYdy8vnXqO52QZwxPs47qRldW8Ty+Nc/kVdgVazFyipBTAA6quoAnNNKTwHnqOotqrqjvBm7FdtPA+cBXYAxItKlxGTXAPtUtR3wOPBIifGPAZ8HtCamTN+s2cXHS7fzhzPb0bGZ9bVkjK+wMOFP53Zk695s3llgXXAUKytB5KjqXgBV3QKsU9VFxzDvfkCqqqapah7wNkffaGgkUHxV9nvAEPfCPETkQmAjsOoYlmn8OJSTz98+XEnHpjHcMKid1+EYUyWd0SGefsmNeOLbVLLzCr0Op0ooK0EkisgTxQ+geYn35UkAtvq8T3eH+Z1GVQuAA0CciNQD/gzcV9YCRGSCiCwUkYUZGdZ9b2n+8+VP7DqUwyOXdCcqwrrxNsYfEeFPQzuScSiXV+Zs8jqcKqGsO8r9qcT7Yyk9nKh7gcdVNausZpiqOhGYCJCSkmLND/xYnr6fV+dsYuzJreiZ1MDrcIyp0vomN+LMjvE8O2MDY/q1rPG33C01QVRAh3zbAN+rsBLdYf6mSReRCKA+kInTOeAlIvIvoAFQJCI5qvrUCcZUoxQWKX/7cCVx9aK5/dyOXodjTEi47ZyODH9yNi/MSuO2c2r2fhPM8w0LgPYi0lpEooDRwNQS00wFxrmvLwG+VcdAVU1W1WTgv8CDlhyO3atzNrFi2wHuHt6F2Fo1+0jImECdlFCf87s3Z/LsjezJyvU6HE8FLUG4dQo3AtNxmsdOUdVVInK/iIxwJ5uMU+eQCtwKHNUU1hyfnQdy+M+XP3F6h3iGd2/udTjGhJRbz+5ATn4hk75P8zoUT5VVBwGAiDQqbs3kM6y1qm4s77OqOg2YVmLY3T6vc4BLy5nHveUtxxzt/k9XkV9YxD9GdrXuNIw5Rm3j6zGiRwtenbOZa09vQ+N60V6H5IlAShCfiMjPtxx1r2X4JHghmRP13drdTFuxk5sGt7PuNIw5TjcObk9OQSGTZtXcUkQgCeJBnCRRT0T6AO8CVwU3LHO8svMK+fvHK2kbX5drT7f7SxtzvNo1cUoRr83ZzN7DeV6H44lAOuv7DOcq5y+Bl4GLVHVpkOMyx+mJb9eTvi+bf17UjeiIcK/DMSak3TS4Hdn5NbcUUWodhIg8iXsPCFd9YANwo4igqjcHOzhzbFJ3ZzHp+zRG9U7k5DZxXodjTMhr1ySG4d1b8MqPm7h2YBsa1a1Z90orq5J6YYn3lXmhnDlGqsp9n6yidlQ4fxnWyetwjKk2bh7cjk+Xb+eFWWncMbRm7VvHdKGciDQEklR1eVCjMsds+qpdzFq/h3su6FJjW1wYEwztm8ZwfrfmP5ciGtagUkS5dRAiMkNEYkWkEbAYmCQijwU/NBOo7LxC/vHpajo2jWHsya28DseYaufmIe05kl/IC7NrVl1EIK2Y6qvqQeBi4FVV7Q+cFdywzLF4duYGtu3P5r6RXYkIt874jKloHZrGMOyk5rz642YO5uR7HU6lCeTfJEJEmgOXAZ8GOR5zjLZkHuG5mRsY0aOFVUwbE0TXD2rLodwCXpuz2etQKk0gCeJ+nO4yUlV1gYi0wW45WmX847PVRIQJfx3W2etQjKnWTkqozxkd4nnph43k5NeM+0UEch3Eu6raXVVvcN+nqeqo4IdmyjNj3W6+Wr2Lmwa3p1n9Wl6HY0y1d8OgtuzJymPKwq3lT1wNlHUdxB2q+i8/10MA2HUQHsstKOS+T1bTpnFdfntastfhGFMj9GvdiD6tGvL8zDTG9GtJZDWv8ytr7da4zwtxroEo+TAeenH2JjbuOcw9I7raFdPGVBIR4foz2rJtfzafLNvudThBV9Z1EJ+4z/6uh2gZzKBM2XYfyuGpb9dzVuemnNEh3utwjKlRBndqQsemMTw7YwMX9kwgLKz69pZcZvlIRAaIyCUi0sR9311E3gR+qJTojF//mf4TeYVF/O18q5g2prKFhQnXD2rL+t1ZfL1ml9fhBFWpCUJEHgVeBEYBn4nIAzgd9s0D2ldOeKakVdsPMGXRVsYNSKZ1Y+vK2xgvDO/enKRGtXlmxgZUj6qirTbK6ovpfKCXqua4XWxsBU5S1U2VEpk5iqryj09X06B2JDcNsRxtjFciwsOYcHpb/v7RSuam7WVA2+p5DVJZp5hy3Du+oar7gPWWHLz15epdzE3by61nd6B+bbvHtDFeurRPInF1o6p1V+BllSDaiMhUn/etfd+r6gg/nzFBkltQyIPT1tC+ST3G9LM2AsZ4rVZkOGMHtOK/X68ndfch2jWJ8TqkCldWCWIk8B+fR8n35RKRoSKyTkRSReROP+OjReQdd/w8EUl2h/cTkaXuY5mIXHRsq1X9vPrjZjZnHuGu4V2svyVjqoixJ7ciOiKMybM3eh1KUJTVzHXmicxYRMKBp4GzgXRggYhMVdXVPpNdA+xT1XYiMhp4BLgcWAmkqGqB2w/UMhH5RFULTiSmUJWZlcsT365nUMd4a9ZqTBUSVy+ai3sn8v7idG4/pyNx1ayr/WAeivbD6b8pTVXzgLdxSiG+RgLF11m8BwwREVHVIz7JoBZ+ruSuSR7/+ieO5BVylzVrNabKuea01uQVFPHa3OrXiV8wE0QCTsunYunuML/TuAnhABAHICL9RWQVsAK4zl/pQUQmiMhCEVmYkZERhFXw3k+7DvHmvC1c1b9ltTzHaUyoa9ekHoM7NeG1OZurXSd+AScIEakTzEBKUtV5qtoV6Av8RUSO6o1OVSeqaoqqpsTHV89TLw98toZ60RH831kdvA7FGFOK3w1sTebhPD5css3rUCpUIHeUO0VEVgNr3fc9ROSZAOa9DUjyeZ/oDvM7jYhEAPWBTN8JVHUNkAWcFMAyq5XZ6/fw/U8Z3DykfY26zaExoWZAmzi6tojlhVlpFBVVnzPigZQgHgfOxf3jVtVlwOkBfG4B0F5EWotIFDAamFpimqnAOPf1JcC3qqruZyIARKQV0AnYFMAyq42iIuWhz9eQ2LA2YwfYbUSNqcpEhGsHtmFDxmFm/LTb63AqTECnmFS1ZOfn5Z5oc+sMbsS52dAaYIqqrhKR+0Wk+BqKyUCciKQCtwLFTWFPw2m5tBT4ELhBVfcEEmt1MXXZdlZtP8jt53S03lqNCQHnd29Os9havDCr+jR5LetCuWJbReQUQEUkEvgjv3QFXiZVnQZMKzHsbp/XOcClfj73GvBaIMuojnILCnl0+jq6tohlRI8WXodjjAlAZHgY409N5uHP17Jy2wFOSqjvdUgnLJASxHXAH3BaHG0DegI3BDOomu61OZvZtj+bv5zXuVp3JWxMdTOmX0tqR4bzyo+bvA6lQgSSIDqq6pWq2lRVm6jqVYA1yA+SA0fyefLbVE7vEM9p7Rt7HY4x5hjUrx3JRb0T+HjZdvYezvM6nBMWSIJ4MsBhpgI8MzOVgzn53Dm0k9ehGGOOw7gByeQVFPH2gi1eh3LCyron9QDgFCBeRG71GRULWK1pEGzbn81LP2ziol4JdGkR63U4xpjj0LFZDAPaxPH6nM1MGNgmpPtOKyvyKKAeThKJ8XkcxGmSairYY1/+BMBt53T0OBJjzIkYd0oy2w/khPwd58rrrG+miLysqtWvk5EqZs2Og3ywJJ1rB7YhoUFtr8MxxpyAszo3IaFBbV7+cRNDT2rudTjHLZBmri+LyFGXBqrq4CDEU2M9/PlaYmtF8odB7bwOxRhzgiLCwxg7oBUPf76WtTsP0qlZaJ4yDuTk2O3An9zH34GlwMJgBlXTzE3LZOZPGdwwqC3169id4oypDi5PSSI6IoxXfgzdEzDlliBUdVGJQT+IyPwgxVPjqCr/nr6OprHRjDsl2etwjDEVpGHdKC7smcBHS7Zx59BOIXnwF0hnfY18Ho1F5FycTvVMBZixLoOFm/dx0+D21Iq0xmHGVCfjTkkmO7+QKQtL9lYUGgKpg1iEc8MeAQqAjTh3gjMnqKhIeXT6Olo2qsNlKUnlf8AYE1K6tIilX3IjXpmzid+e1prwEOsZodwShKq2VtU27nN7VT1HVWdXRnDV3ecrd7J6x0FuObs9URGh21baGFO6cackk74vm5kh2MtrWRfKXVzWB1X1g4oPp+YoKCziP1+to0PTeozoUfJGe8aY6uKcrk2Jj4nmzXlbGNypqdfhHJOyTjFdUMY4BSxBnIAPlmwjLeMwz4/tE3LFTmNM4CLDw7gsJZFnZ2xg2/7skLrOqawL5a6uzEBqktyCQv739Xp6JNbnnC6hdURhjDl2o/u25JkZG3hn/hZuDaGeEgJpxVRfRB4TkYXu4z8iYq2YTsBb87awbX82fzq3EyJWejCmuktqVIdBHeJ5e8FW8guLvA4nYIHUjL4IHAIucx8HgZeCGVR1diSvgKe+S2VAmzhObRfndTjGmEpyZf9W7D6Uyzch1D9TIAmirareo6pp7uM+oE2wA6uuXvphE3uy8rj93I5WejCmBjmzUxNa1K/FG/NCpxvwQBJEtoicVvxGRE4FsgOZuYgMFZF1IpIqInf6GR8tIu+44+eJSLI7/GwRWSQiK9znatHv04HsfJ6fuYGzOjehT6uGXodjjKlE4WHC5X1bMmv9HjZnHvY6nIAEkiCuB54WkU0ishl4Cuc2pGUSkXDgaeA8oAswRkS6lJjsGmCfqrYDHgcecYfvAS5Q1W7AOKrJ/aknz0rjYE4Bt54dOpVUxpiKc3nfJMLDhDfnh0YpIpAL5Zaqag+gO9BNVXup6rIA5t0PSHVPS+UBbwMjS0wzEnjFff0eMERERFWXqOp2d/gqoLaIRAeyQlXV/iN5vPjDJoZ1a2Y3AzKmhmpWvxZndW7CuwvTyS0o9DqccgXSiumPIhKLU1H9mIgsFpFzAph3AuDbAUm6O8zvNKpaABwAStbcjgIWq2puAMussl6YtZHDeQX8cUgHr0Mxxnjoyv6t2Hs4j+mrqn5ldSCnmH6rqgeBc3D+vMcCDwc1KpeIdMU57fT7UsZPKG5+m5GRURkhHZd9h/N46YeNDOvWnI7NYrwOxxjjodPaNaZlozq8MbfqdwMeSIIobmozDHhVVVf5DCvLNsC3B7pEd5jfaUQkAqeX2Ez3fSLwIfAbVd3gbwGqOlFVU1Q1JT4+PoCQvDFpVhpH8gv5vyHtvQ7FGOOxsDBhTL+WzNu4lw0ZWV6HU6ZAEsQiEfkSJ0FMF5EYIJArPRYA7UWktYhEAaOBqSWmmYpTCQ3Ofa6/VVUVkQbAZ8CdqvpDICtSVe09nMcrP25iePcWtG9qpQdjDIzqk0B4mFT5bsADSRDXAHcCfVX1CBAFlNsNh1uncCMwHVgDTFHVVSJyv4iMcCebDMSJSCpwq7sc3M+1A+4WkaXuo8mxrFhVMfF7p/Rw82C7lagxxtEkphZndmzC+4u2VekrqwO5o1yRe33CVe69qWer6oeBzFxVpwHTSgy72+d1DnCpn889ADwQyDKqssysXF6ds4kLrPRgjCnh8r5JfL1mF9+t3c05XZt5HY5fgbRiegbnuocVwErg9yLydLADqw4mzkojJ7+Qm63uwRhTwpkd44mPia7Sp5kCuaPcYKCzqiqAiLwCrA5qVNXAnqxcXv1xMyN6tKBdk3peh2OMqWIiwsMY1TuRSbPS2H0whyaxtbwO6SiB1EGkAi193icB64MTTvUx8fs0cgus9GCMKd1lKYkUFinvLU73OhS/Sk0QIvKJiEwFYoA1IjJDRL7DqXC2E+plyDjk1D1c2DOBNvFWejDG+Ncmvh79khvx7sJ03JM0VUpZp5j+Xca4qrcmVcjE7zeQV1DEjdZyyRhTjsv6JnH7u8uYv3Ev/dtUrVsAlFqCUNWZ/h5AIXB55YUYWjKzcnlt7mYrPRhjAjKsWzPqRUfwThWsrA6kDgIR6SUij4rIJuAfOKeZjB+TZ28kt6CIG8600oMxpnx1oiIY0bMF01bs4GBOvtfh/EpZdRAdROQeEVkLPAlsAURVz1TVpyotwhBy4Eg+r87ZzLBuza3lkjEmYJenJJGTX8Qny7aXP3ElKqsEsRanietwVT1NVZ/EOb1kSvHyj5vIyi3gRis9GGOOQffE+nRqFsOUBVXrNFNZCeJiYAfwnYhMEpEhBNZJX42UlVvAiz9s5KzOTenc3O73YIwJnIhwSZ9ElqUfYP2uQ16H87OyKqk/UtXRQCfgO+D/gCYi8myA94OoUV6fu5kD2fnWcskYc1xG9nQ68Ht/cclOr70TyB3lDqvqm6p6AU6X3UuAPwc9shCSnVfIC7PSGNi+MT2TGngdjjEmBMXHRHNGh3g+WrKNwqKqcSVBQK2YiqnqPvceDEOCFVAoenvBFvZk5XHTYLtq2hhz/Eb1TmTnwRzmbMj0OhTgGBOEOVpuQSHPz0yjX+tG9GvdyOtwjDEhbEjnJsTWiuD9KtL1hiWIE/T+om3sPJjDTVb3YIw5QbUiwxneowVfrNxJVm6B1+FYgjgR+YVFPDMjlR5JDTitXWOvwzHGVAOjeieQnV/I5yt2eB2KJYgTMXXpdtL3ZXPTme0QsRbAxpgT17tlQ5Lj6vBBFWjNZAniOBUWKU/PSKVz81iGdA7Ju6EaY6ogEeHi3onMScskfd8RT2OxBHGcPl+5g7SMw9xopQdjTAW7qFcCAB8t8bYUEdQEISJDRWSdiKSKyJ1+xkeLyDvu+Hnuva8RkTgR+U5EskSkyvX7pKo8O2MDbRrXZehJVfNessaY0JXUqA79Wzfig8XbPL1PRNAShIiEA08D5wFdgDEi0qXEZNcA+1S1HfA48Ig7PAf4O3B7sOI7EbPW72HV9oP8/ow2hIdZ6cEYU/FG9U4kbc9hlmzd71kMwSxB9ANSVTVNVfOAt4GRJaYZCbzivn4PGCIi4l69PRsnUVQ5z83cQNPYaC50i4HGGFPRzuvWjFqRYby/yLtrIoKZIBIA364J091hfqdR1QLgABDwLZVEZIKILBSRhRkZGScYbmCWbd3Pjxsy+d1pbYiOCK+UZRpjap6YWpGc06UZn63YQV5BkScxhHQltdvtR4qqpsTHx1fKMp+buYHYWhGM6d+yUpZnjKm5LuzVgv1H8pm1vnIOgEsKZoLYBiT5vE90h/mdRkQigPpA1eiExI8NGVl8sWonvxmQTL3osm7nbYwxJ25g+3ga1onko6Xe3EgomAliAdBeRFqLSBQwGphaYpqpwDj39SXAt+pllX05Js5MIyo8jPGnJnsdijGmBogMD+P87s35avVODnvQ9UbQEoRbp3AjMB3nHtZTVHWViNwvIiPcySYDcSKSCtwK/NwU1r3/9WPAeBFJ99MCqlLtPJDDB0vSuSwlicb1or0MxRhTg4zsmUBOfhFfrd5V6csO6nkSVZ0GTCsx7G6f1znApaV8NjmYsR2rF3/YSGGRcu3ANl6HYoypQfq0bEhCg9p8vHRbpbecDOlK6spy4Eg+b8zdzPDuLWgZV8frcIwxNUhYmHBBjxZ8v34PmVm5lbvsSl1aiHp93mYO5xXy+zOs9GCMqXwX9mpBYZEyrZJ7eLUEUY6c/EJe+mEjZ3SIp2uL+l6HY4ypgTo1i6Vj0xg+ruTWTJYgyvHuonT2ZOVx3RltvQ7FGFODjejZgoWb97F1b+X18GoJogwFhUVM+j6NnkkNOLmN3U7UGOOdET1aAPDJ8sorRViCKMMXq3ayZe8RrjujrXXpbYzxVFKjOvRp1ZCPl1iC8JyqMun7NJLj6nB2l6Zeh2OMMVzYswXrdh1i7c6DlbI8SxClWLBpH8vSD3DNQOvS2xhTNQzr1pzwMKm0ympLEKWYNCuNhnUiuaR3otehGGMMAHH1ohnYvjFTl26nqCj4vRJZgvAjLSOLr9fsYuzJragdZV16G2OqjhE9WrBtf3al3EjIEoQfk2dvJDI8jLEDkr0OxRhjfuXsLk2Jigjj00pozWQJooS9h/N4b1E6F/VMID7GOuUzxlQtMbUiOaNDPNNW7Aj6aSZLECW8PnczuQVF/G5ga69DMcYYv4Z3b86ug7ks3LwvqMuxBOEjJ7+QV+ds4syO8bRvGuN1OMYY49dZnZsSXQmnmSxB+PhoyTb2ZOVZl97GmCqtbnQEgzs1YdqKnRQG8TSTJQhXUZHywuyNdG0Ry4C2cV6HY4wxZRrevQV7snKZtzF4d2m2BOGa8dNuUndnce3ANtathjGmyhvcqQm1I8P5dHnwugC3BOGa9P1Gmtevxfndm3sdijHGlKt2VDhDOjfhi5U7KSgsCsoygpogRGSoiKwTkVQRudPP+GgReccdP09Ekn3G/cUdvk5Ezg1mnCu3HWBOWiZXn5pMZLjlTGNMaBjevQV7D+cxJy04p5mC9m8oIuHA08B5QBdgjIh0KTHZNcA+VW0HPA484n62CzAa6AoMBZ5x5xcUk2alUS86gtH9WgZrEcYYU+EGdYynblQ4ny4LzmmmYB4u9wNSVTVNVfOAt4GRJaYZCbzivn4PGCJOBcBI4G1VzVXVjUCqO78Kt31/Np8u38HovknE1ooMxiKMMSYoakWGc8vZHTizU3xQ5h8RlLk6EoCtPu/Tgf6lTaOqBSJyAIhzh88t8dmEYAR5JK+Age0bc/VpdmGcMSb0/C6IzfKDmSCCTkQmABMAWrY8vtND7ZrE8PLVQSmcGGNMSAvmKaZtQJLP+0R3mN9pRCQCqA9kBvhZVHWiqqaoakp8fHCKWMYYU1MFM0EsANqLSGsRicKpdJ5aYpqpwDj39SXAt6qq7vDRbiun1kB7YH4QYzXGGFNC0E4xuXUKNwLTgXDgRVVdJSL3AwtVdSowGXhNRFKBvThJBHe6KcBqoAD4g6oWBitWY4wxRxPngD30paSk6MKFC70OwxhjQoqILFLVFH/j7KowY4wxflmCMMYY45clCGOMMX5ZgjDGGONXtamkFpEMYPNxfrwxsKcCwwkFts41g61zzXG8691KVf1eSFZtEsSJEJGFpdXiV1e2zjWDrXPNEYz1tlNMxhhj/LIEYYwxxi9LEI6JXgfgAVvnmsHWueao8PW2OghjjDF+WQnCGGOMX5YgjDHG+FWjEoSIDBWRdSKSKiJ3+hkfLSLvuOPniUhy5UdZsQJY51tFZLWILBeRb0SklRdxVqTy1tlnulEioiIS8k0iA1lnEbnM/a1XiciblR1jRQtg224pIt+JyBJ3+x7mRZwVSUReFJHdIrKylPEiIk+438lyEel9QgtU1RrxwOlyfAPQBogClgFdSkxzA/Cc+3o08I7XcVfCOp8J1HFfX18T1tmdLgb4HufWtilex10Jv3N7YAnQ0H3fxOu4K2GdJwLXu6+7AJu8jrsC1vt0oDewspTxw4DPAQFOBuadyPJqUgmiH5Cqqmmqmge8DYwsMc1I4BX39XvAEBGRSoyxopW7zqr6naoecd/Oxbl7XygL5HcG+AfwCJBTmcEFSSDrfC3wtKruA1DV3ZUcY0ULZJ0ViHVf1we2V2J8QaGq3+PcO6c0I4FX1TEXaCAizY93eTUpQSQAW33ep7vD/E6jqgXAASCuUqILjkDW2dc1OEcfoazcdXaL3Umq+lllBhZEgfzOHYAOIvKDiMwVkaGVFl1wBLLO9wJXiUg6MA24qXJC89Sx7vNlCtod5UxoEZGrgBTgDK9jCSYRCQMeA8Z7HEpli8A5zTQIp5T4vYh0U9X9nkYVXGOAl1X1PyIyAOfulSepapHXgYWKmlSC2AYk+bxPdIf5nUZEInCKpZmVEl1wBLLOiMhZwN+AEaqaW0mxBUt56xwDnATMEJFNOOdpp4Z4RXUgv3M6MFVV81V1I/ATTsIIVYGs8zXAFABVnQPUwunQrjoLaJ8PVE1KEAuA9iLSWkSicCqhp5aYZiowzn19CfCtujU/IarcdRaRXsDzOMkh1M9LQznrrKoHVLWxqiarajJOvcsI5ulZCgAABC9JREFUVQ3l+9UGsm1/hFN6QEQa45xySqvMICtYIOu8BRgCICKdcRJERqVGWfmmAr9xWzOdDBxQ1R3HO7Mac4pJVQtE5EZgOk4LiBdVdZWI3A8sVNWpwGScYmgqTkXQaO8iPnEBrvOjQD3gXbc+fouqjvAs6BMU4DpXKwGu83TgHBFZDRQCf1LVkC0dB7jOtwGTROQWnArr8SF+wIeIvIWT6Bu7dSv3AJEAqvocTl3LMCAVOAJcfULLC/HvyxhjTJDUpFNMxhhjjoElCGOMMX5ZgjDGGOOXJQhjjDF+WYIwxhjjlyUIU+2JSKGI/H979xNiVRnGcfz7TYRE0VoMYos0okgzGJxFWRQqleFUIhUtWhTRwpWLEFoErrR/Fv6poEULidoIJZQtKoTUpIkYm2kSpqBFi0KMkEQ0HPRp8b4XL9NpLjNN9Oc8Hxg4nHvOe96zuPOc95x7fu+IOqoeV2+f5fb3qQ/X5TfVFbPQ5hr119rvcfXlrs8e7JFS+4T62l/tQ0qteQ8itdr5iOgHUNcDz/M3RYpExFOz2NzRiLhfnQd8pR6IiGP1N/7/u/c50r9PjiBS2ywETgOoC+ocGMfVMXVjXT9f/bCOOL5RH63rB9TD6rD6UVNKpvppJ7ZDPavuqO0MqYvr+j71XfXL+nfHVB2OiPPACDV0rXuEoD5S+ziqHmnoz6D6eX17OqVpyRFEaoN56gglamEJsK6u/w3YFBFn6j/QIfV94D7gp4gYBFAXqXOBV4GNEfFzLRo7gCenOO58YCginlVfokRubwf2ALsi4jP1WsrbwMv/rBH1akpu0h8KALANWB8RP6pXTdpvE/A0sKET853SdGSBSG3QfYtpNfCWupIyqcpz6l3AJcoV+mJgDHhFfRE4GBFH6/YrgU9qJMkcoFfGzQXgYF0eBu6py3cDK7w81chCdUFEnJ20/53qKKU47I6Ikw3HOAbsU/cD73WtX0dJ5703Is706GdKjbJApFaJiM7tlj5KZk0fMBAREzXd9cqI+M4yZ8QGYLt6CDgAnIiI1dM43ERX9s9FLn/frgBui4hekxV1nkFcRxnd7I+IkUnns1m9FRgEhtWB+lFntrUbgf9yEGH6B+UziNQq6k2Uq/9fKHHup2pxWAssrdtcA5yLiLcpYYargG+BvjoCQZ2r3jzDbnxM1+Q1av9UG9d47heAZxrO5/qI+CIitlGSSjtRzz8AD1FGSzPtZ2q5HEGkNug8g4ByW+nxiLiovgN8oI5RrrLH6za3ADvVS8AEZV7jC/WnrHvVRZTvzm7gxAz6swV4Xf26tnME2NxjnzeAreqySet3qjfU8zpEmZu5HyAixtXHKEm9D0TE9zPoa2qxTHNNKaXUKG8xpZRSapQFIqWUUqMsECmllBplgUgppdQoC0RKKaVGWSBSSik1ygKRUkqp0e83K/S7jqow+AAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "OI4QLB5l2OyZ"
      },
      "source": [
        "Note that when viewed on an absolute scale, the treatment effect is not constant, despite the fact that you used a model with no interactions between the features (we didn't multiply two features together). \n",
        "\n",
        "As shown in the plot, when the baseline risk is either very low (close to zero) or very high (close to one), the Absolute Risk Reduction from treatment is fairly low.  When the baseline risk is closer to 0.5 the ARR of treatment is higher (closer to 0.10).\n",
        "\n",
        "It is always important to remember that baseline risk has a natural effect on absolute risk reduction."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "9bGTgLRkQZPR"
      },
      "source": [
        "<a name=\"2-3\"></a>\n",
        "### 2.3 Model Limitations\n",
        "\n",
        "We can now plot how closely the empirical (actual) risk reduction matches the risk reduction that is predicted by the logistic regression model. \n",
        "\n",
        "This is complicated by the fact that for each patient, we only observe one outcome (treatment or no treatment). \n",
        "- We can't give a patient treatment, then go back in time and measure an alternative scenario where the same patient did not receive the treatment.\n",
        "- Therefore, we will group patients into groups based on their baseline risk as predicted by the model, and then plot their empirical ARR within groups that have similar baseline risks.\n",
        "- The empirical ARR is the death rate of the untreated patients in that group minus the death rate of the treated patients in that group.\n",
        "\n",
        "$$ARR_{empirical} = p_{baseline} - p_{treatment}$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "y7sx9hZ85jNQ"
      },
      "source": [
        "<a name='ex-05'></a>\n",
        "### Exercise 5: Baseline Risk\n",
        "In the next cell, write a function to compute the baseline risk of each patient using the logistic regression model.\n",
        "\n",
        "The baseline risk is the model's predicted probability that the patient is predicted to die if they do not receive treatment.\n",
        "\n",
        "You will later use the baseline risk of each patient to organize patients into risk groups (that have similar baseline risks).  This will allow you to calculate the ARR within each risk group.\n",
        "\n",
        "$$p_{baseline} = logisticRegression(Treatment = False, Age = age_{i}, Obstruct = obstruct_{i}, \\cdots)$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TeXuReqQTmql",
        "colab_type": "text"
      },
      "source": [
        "<details>    \n",
        "<summary>\n",
        "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
        "</summary>\n",
        "<p>\n",
        "<ul>\n",
        "    <li> A patient receives treatment if their feature x_treatment is True, and does not receive treatment when their x_treatment is False.</li>\n",
        "    <li>For a patient who actually did receive treatment, you can ask the model to predict their risk without receiving treatment by setting the patient's x_treatment to False.</li>\n",
        "    <li>The logistic regression predict_proba() function returns a 2D array, one row for each patient, and one column for each possible outcome (each class).  In this case, the two outcomes are either no death (0), or death (1).  To find out which column contains the probability for death, check the order of the classes by using lr.classes_ </li>\n",
        "</ul>\n",
        "</p>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:30:49.614506Z",
          "start_time": "2020-04-04T15:30:49.580917Z"
        },
        "colab_type": "code",
        "id": "BrIYA-Ciu3EK",
        "colab": {}
      },
      "source": [
        "# UNQ_C5 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
        "def base_risks(X, lr_model):\n",
        "    \"\"\"\n",
        "    Compute baseline risks for each individual in X.\n",
        "\n",
        "    Args:\n",
        "        X (dataframe): data from trial. 'TRTMT' column\n",
        "                       is 1 if subject retrieved treatment, 0 otherwise\n",
        "        lr_model (model): logistic regression model\n",
        "    \n",
        "    Returns:\n",
        "        risks (np.array): array of predicted baseline risk\n",
        "                          for each subject in X\n",
        "    \"\"\"\n",
        "    \n",
        "    # first make a copy of the dataframe so as not to overwrite the original\n",
        "    X = X.copy(deep=True)\n",
        "    \n",
        "    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
        "\n",
        "    # Set the treatment variable to assume that the patient did not receive treatment\n",
        "    X.loc[:, \"TRTMT\"] = 0\n",
        "    \n",
        "    # Input the features into the model, and predict the probability of death.\n",
        "    risks = lr_model.predict_proba(X)[:,1]\n",
        "    \n",
        "    # END CODE HERE\n",
        "\n",
        "    return risks"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l2FA0PDjTmqw",
        "colab_type": "text"
      },
      "source": [
        "**Test Case**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ncWXo4pXTmqw",
        "colab_type": "code",
        "colab": {},
        "outputId": "0cc17748-291b-4024-f137-008c0559559d"
      },
      "source": [
        "example_df = pd.DataFrame(columns = X_dev.columns)\n",
        "example_df.loc[0, :] = X_dev.loc[X_dev.TRTMT == 1, :].iloc[0, :]\n",
        "example_df.loc[1, :] = example_df.iloc[0, :]\n",
        "example_df.loc[1, 'TRTMT'] = 0\n",
        "\n",
        "print(\"TEST CASE\")\n",
        "print(example_df)\n",
        "print(example_df.loc[:, ['TRTMT']])\n",
        "print('\\n')\n",
        "\n",
        "print(\"Base risks for both rows should be the same\")\n",
        "print(f\"Baseline Risks: {base_risks(example_df.copy(deep=True), lr)}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TEST CASE\n",
            "  sex age obstruct perfor adhere nodes node4 TRTMT differ_2.0 differ_3.0  \\\n",
            "0   1  60        0      0      0     3     0  True          1          0   \n",
            "1   1  60        0      0      0     3     0     0          1          0   \n",
            "\n",
            "  extent_2 extent_3 extent_4  \n",
            "0        0        1        0  \n",
            "1        0        1        0  \n",
            "  TRTMT\n",
            "0  True\n",
            "1     0\n",
            "\n",
            "\n",
            "Base risks for both rows should be the same\n",
            "Baseline Risks: [0.43115868 0.43115868]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SIUiFrVVTmq4",
        "colab_type": "text"
      },
      "source": [
        "#### Expected output\n",
        "\n",
        "```CPP\n",
        "Base risks for both rows should be the same\n",
        "Baseline Risks: [0.43115868 0.43115868]\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "JQsYKmVc6prz"
      },
      "source": [
        "<a name='ex-06'></a>\n",
        "### Exercise 6: ARR by quantile\n",
        "\n",
        "Since the effect of treatment varies depending on the baseline risk, it makes more sense to group patients who have similar baseline risks, and then look at the outcomes of those who receive treatment versus those who do not, to estimate the absolute risk reduction (ARR).\n",
        "\n",
        "You'll now implement the `lr_ARR_quantile` function to plot empirical average ARR for each quantile of base risk."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1r5wK47YTmq5",
        "colab_type": "text"
      },
      "source": [
        "<details>    \n",
        "<summary>\n",
        "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
        "</summary>\n",
        "<p>\n",
        "<ul>\n",
        "    <li>Use pandas.cut to define intervals of bins of equal size.  For example, pd.cut(arr,5) uses the values in the list or array 'arr' and returns the intervals of 5 bins.</li>\n",
        "    <li>Use pandas.DataFrame.groupby to group by a selected column of the dataframe.  Then select the desired variable and apply an aggregator function.  For example, df.groupby('col1')['col2'].sum() groups by column 1, and then calculates the sum of column 2 for each group. </li>\n",
        "</ul>\n",
        "</p>\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P7qxxb89Tmq6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# UNQ_C6 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
        "def lr_ARR_quantile(X, y, lr):\n",
        "    \n",
        "    # first make a deep copy of the features dataframe to calculate the base risks\n",
        "    X = X.copy(deep=True)\n",
        "    \n",
        "    # Make another deep copy of the features dataframe to store baseline risk, risk_group, and y\n",
        "    df = X.copy(deep=True)\n",
        "\n",
        "    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
        "    # Calculate the baseline risks (use the function that you just implemented)\n",
        "    baseline_risk = base_risks(X, lr)\n",
        "    \n",
        "    # bin patients into 10 risk groups based on their baseline risks\n",
        "    risk_groups = pd.cut(baseline_risk, 10)\n",
        "        \n",
        "    # Store the baseline risk, risk_groups, and y into the new dataframe\n",
        "    df.loc[:, 'baseline_risk'] = baseline_risk\n",
        "    df.loc[:, 'risk_group'] = risk_groups\n",
        "    df.loc[:, 'y'] = y\n",
        "\n",
        "    # select the subset of patients who did not actually receive treatment\n",
        "    df_baseline = df[(df[\"TRTMT\"] == False)]\n",
        "    \n",
        "    # select the subset of patients who did actually receive treatment\n",
        "    df_treatment = df[(df[\"TRTMT\"] == True)]\n",
        "    \n",
        "    # For baseline patients, group them by risk group, select their outcome 'y', and take the mean\n",
        "    baseline_mean_by_risk_group = df_baseline.groupby(\"risk_group\")[\"y\"].mean()\n",
        "    \n",
        "    # For treatment patients, group them by risk group, select their outcome 'y', and take the mean\n",
        "    treatment_mean_by_risk_group = df_treatment.groupby(\"risk_group\")[\"y\"].mean()\n",
        "    \n",
        "    # Calculate the absolute risk reduction by risk group (baseline minus treatment)\n",
        "    arr_by_risk_group = baseline_mean_by_risk_group - treatment_mean_by_risk_group\n",
        "    \n",
        "    # Set the index of the arr_by_risk_group dataframe to the average baseline risk of each risk group \n",
        "    # Use data for all patients to calculate the average baseline risk, grouped by risk group.\n",
        "    arr_by_risk_group.index = df.groupby(\"risk_group\")[\"baseline_risk\"].mean()\n",
        "\n",
        "    ### END CODE HERE ###\n",
        "    \n",
        "    # Set the name of the Series to 'ARR'\n",
        "    arr_by_risk_group.name = 'ARR'\n",
        "    \n",
        "\n",
        "    return arr_by_risk_group\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eTbE5q8ETmrA",
        "colab_type": "code",
        "colab": {},
        "outputId": "67e75a28-6301-406f-99d9-2c22b3696eb0"
      },
      "source": [
        "# Test\n",
        "abs_risks = lr_ARR_quantile(X_dev, y_dev, lr)\n",
        "\n",
        "# print the Series\n",
        "print(abs_risks)\n",
        "\n",
        "# just showing this as a Dataframe for easier viewing\n",
        "display(pd.DataFrame(abs_risks))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "baseline_risk\n",
            "0.231595    0.089744\n",
            "0.314713    0.042857\n",
            "0.386342   -0.014604\n",
            "0.458883    0.122222\n",
            "0.530568    0.142857\n",
            "0.626937   -0.104072\n",
            "0.693404    0.150000\n",
            "0.777353    0.293706\n",
            "0.836617    0.083333\n",
            "0.918884    0.200000\n",
            "Name: ARR, dtype: float64\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ARR</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>baseline_risk</th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0.231595</th>\n",
              "      <td>0.089744</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0.314713</th>\n",
              "      <td>0.042857</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0.386342</th>\n",
              "      <td>-0.014604</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0.458883</th>\n",
              "      <td>0.122222</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0.530568</th>\n",
              "      <td>0.142857</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0.626937</th>\n",
              "      <td>-0.104072</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0.693404</th>\n",
              "      <td>0.150000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0.777353</th>\n",
              "      <td>0.293706</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0.836617</th>\n",
              "      <td>0.083333</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0.918884</th>\n",
              "      <td>0.200000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                    ARR\n",
              "baseline_risk          \n",
              "0.231595       0.089744\n",
              "0.314713       0.042857\n",
              "0.386342      -0.014604\n",
              "0.458883       0.122222\n",
              "0.530568       0.142857\n",
              "0.626937      -0.104072\n",
              "0.693404       0.150000\n",
              "0.777353       0.293706\n",
              "0.836617       0.083333\n",
              "0.918884       0.200000"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S8y8ecFnTmrF",
        "colab_type": "text"
      },
      "source": [
        "##### Expected output\n",
        "```CPP\n",
        "baseline_risk\n",
        "0.231595    0.089744\n",
        "0.314713    0.042857\n",
        "0.386342   -0.014604\n",
        "0.458883    0.122222\n",
        "0.530568    0.142857\n",
        "0.626937   -0.104072\n",
        "0.693404    0.150000\n",
        "0.777353    0.293706\n",
        "0.836617    0.083333\n",
        "0.918884    0.200000\n",
        "Name: ARR, dtype: float64\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lBqIrGLwTmrG",
        "colab_type": "text"
      },
      "source": [
        "Plot the ARR grouped by baseline risk"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:31:27.365631Z",
          "start_time": "2020-04-04T15:31:27.190715Z"
        },
        "colab_type": "code",
        "id": "xtmp3BxtNR39",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 458
        },
        "outputId": "266dcffc-0c16-4456-c789-106465666b41"
      },
      "source": [
        "plt.scatter(abs_risks.index, abs_risks, label='empirical ARR')\n",
        "plt.title(\"Empirical Absolute Risk Reduction vs. Baseline Risk\")\n",
        "plt.ylabel(\"Absolute Risk Reduction\")\n",
        "plt.xlabel(\"Baseline Risk Range\")\n",
        "ps = np.arange(abs_risks.index[0]-0.05, abs_risks.index[-1]+0.05, 0.01)\n",
        "diffs = [OR_to_ARR(p, trtmt_OR) for p in ps]\n",
        "plt.plot(ps, diffs, label='predicted ARR')\n",
        "plt.legend(loc='upper right')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEWCAYAAABbgYH9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nOzdeXhV5bn38e8vIRMhhCkgBBBURmU0ohYHnACtxbFHrfZgbWttj23P2xarradarUcsnRx7aqvFts4TYqtFVGwdShUEZBZElISZEEggIdP9/rFW4k7IsEOys5Nwf65rX1nzuvfaO+ve63nWeh6ZGc4551xLS4h3AM455zomTzDOOediwhOMc865mPAE45xzLiY8wTjnnIsJTzDOOediwhNMGyKpSNJRDcz/P0n/08x9TJKU24z135D0tebEUMc2b5X0l5bc5qGSdKqktVEsF5eYm/v5NbLtBr9/h7vax17SSkmT4hDHlZJeiWK52ZJ+1hox1ccTTCMkbZRUHP7zVb3ui8W+zKyLmW1oYP51ZnZ7LPYdSYENklbFel9N1dwTe7h+Wfg5Fkh6R9LJVfPN7E0zG9Yy0Vbvc7ak0nCf+ZLmSxrekvs4hJgO+qHQ2PevLQnjLwmP6R5J/5Q0qjVjMLNjzeyNlt5uY98XM3vUzCa39H5jwRNMdL4Q/vNVva5v7QAkJbbi7k4DegNHSTqhFffbWp40sy5AL2AB8HQr7PPn4T6zgTzgoVbYZ0d3fXhMewBvAH+ObzgtqkN8XzzBNIOkqyW9LenX4a/hDZI+F07fJGm7pOkRy88Oi7nmSyqU9A9JR0bMN0nHRCz7W0kvSdoHnFH7klfSBZKWStor6SNJU8PpX5G0OtzHBknfaOJbmw68ALwUDtd2tKR3w/2+IKlHuN9USX+RtCs8Hu9J6hPO6ydpbviLbL2kr9dzTA8qAgqvIs8O39+PgMvCX3fLwvmZkh6StEVSnqSfRZOQzawceBTIlpRV1/4l/TDcZqGktZLOqiPmJEmPS3pWUnIj+ywGngLGRqzfL1x3h6SPJX0nYl5a+LnvDq8oayT8yO9MON7od0TSHcCpwH2KuCKv9f3LlPSnMKZPJN0sKSGcd7WktyT9IozrY0nn1vV+w+P3TK1pd0u6J2JbG8Lj+7GkKxs6fvUc0wrgCWBkxD4mSPpX+D3cIum+qs9GgV+H/597JS2XdFw4LyV8X59K2qbg/zWtnve2UdLZ4fCtkp4Kj1mhguKznIhl6/2MG3lvdX1frpb0VmPvpVasGZIWSLpHkqLZd0vwBNN8JwIfAD2Bxwi+6CcAxwBXEfwTd4lY/krgdoJfz0sJTnD1+RJwB5ABvBU5Q9IE4E/ADKAbwVXHxnD2duB8oCvwFeDXksZH82YkdQYuDeN6FLi8jpPmfwLXAH2BcuCecPp0IBMYQHA8rgOKw3lPALlAv3D7/yvpzGhiqmJmfwf+l/AKxMzGhLNmh3EcA4wDJgON1hOF7+s/gV3A7jrmDwOuB04wswxgCp8d46pl0oA5wAHgP8ystJF9pgNXAOvD8QTgRWAZwa/Vs4D/ljQlXOUW4OjwNYW6E359+6rzO2JmPwbeJLwCqOeK/F6Cz/Io4HSC4/SViPknAmsJvsc/Bx6q58T1BHCepIwwpkTgP4DHwmNxD3BueHw/R/A/0STh53glsDBicgXw/8L4TiY4rt8K500mOBZDw/f4HwTfAYCZ4fSxBN+nbOAnUYYyjeD9dgPmAlWJu7HPuKH3VuP7UoeG3kvVNnoCrwFvm9l3rDXbBzMzfzXwIjihFAEFEa+vh/OuBtZFLDsKMKBPxLRdwNhweDbwRMS8LgT/CAPCcQOOiVj2T7VimQ38LBz+HfDrKN/DHOC74fAkILeBZa8CdgCdgFRgD3BRxPw3gJkR4yOBUiCRIOm8A4yutc0B4fvMiJh2JzA7HL4V+Et98YWfwdm1lw3H+xCc3NMipl0BLKjn/d0axlsQxrQLmBQxv3r/BCeY7cDZQFId25kL/IPgJKkGjulsoCTcZyXwcdUxIjhRf1pr+ZuAP4bDG4CpEfOujTw+kd+ZpnxHws/xa7WmWfieE8NjNDJi3jeANyK+9+sj5nUO1z2inn29BfxnOHwO8FE4nB4ek0siP78ov9NvAPvD9Q8QfE/PamD5/waeD4fPBD4ETgISIpYRsA84OmLaycDHdX036/hevlrr/6I4ms+4Kd+XiOP/VkPvJWI7DwMrgBlNOb4t9fIrmOhcaGbdIl6/j5i3LWK4GMDMak+LvILZVDVgZkVAPsGv+rpsqmc6BCftj+qaIelcSQsVFEcVAOcR/JKLxnTgKTMrN7MS4FkO/tUcGdcnQFK4/T8D84AnJG2W9HNJSQTvL9/MCmutlx1lTA05Mtz/lrA4pIDgxNq7gXWeMrNuBMlpBXB8XQuZ2XqCE9OtwHZJT0iK/KxOAkYTJNzGfhX+ItznIILvRNWNBEcC/apiD+P/URgbBMeu9vGOVr3fkUb0Ijimkfuq/XltrRows/3hYOT3PNJjBEkfgqvyx8L19gGXEVzpbpH0NzXt5ofvhMc0jeCK/RlJowEkDZX0V0lbJe0luPLtFe73dYKri/sJPtcHJXUFsgiS5eKIz+Lv4fRobI0Y3g+kSupE459xXer7vtTQwHup8vnw+PxflO+hRXmCaX0DqgbCorMewOZ6lm3opLWJoNikBkkpBEnhFwRXUt0I6lIaLXeV1J/gF9FV4T/mVoLirPMkRSaoARHDA4EyYKeZlZnZT81sJEFxx/kERSubgR5VxSQR6+XVEcY+gn/yqpgSqfkPXvuYbCL4Bdsr4gdAVzM7trH3a2Y7Ca4IbpXUt55lHjOzUwhOEgbcFTH7FYIrsdcU1jVFsc9Pge8Cd4fFa5sIfiFH/oDJMLPzwlW2cPDxjrSfiOMFHBExXOd3pCqUBsLcSfCZHhkxrb7PKxpPA5PC79dFhAkGwMzmmdk5BMWta4Df172J+plZpZm9SVCMVHV31W/D7Q0xs64EJ3RFrHOPmR1PcKUxlKAYcSfByfzYiM8i04LK9uZo7DNu6L3V/r7UtUxd76XK7wmS5EthcVur8gTT+s6TdEpYbnw7sNDMGrpSqc9DwFcknSUpQVJ2+OsvGUghKOYqV1D5Gu0tjV8muNweRlAGPZbgC5vLZ79AIUhAI8P6mtuAZ8ysQtIZkkaFSWEvwUmqMnx/7wB3KrgRYDTwVaCu240/JPjl9/nw6ufm8P1U2QYMCsu1MbMtBCf6X0rqGh6LoyWdHs0bNrO1BFddN9SeJ2mYpDPDpF1CcPKprLX+zwlOmK/VSsIN7XM+QdK9FngXKFRQGZ4mKVHScfrs7r2ngJskdQ9P0N+utbmlwJfC9aYS1JdUqe87AsFxrPOZFwsqzZ8C7ggrh48Evkfdn1c073cHQZHWHwlOtKsBJPVRcBNCOsGPhCJqHd9oKbjVfCSwMpyUQfAdLArf8zcjlj1B0onh92sfwWdbaWaVBCfkX0vqHS6bHU1dSSMa+4wbVOv7UkN976XWYtcT1Je9WF+SihVPMNF5UTWfg3m+Gdt6jKDiNp+gaOaqQ9mImb1LWIFPUP78D+DIsBjqOwQniN0ERRJzo9zsdOABM9sa+SK4vI4sJvszQfnuVoJ6mqo7Yo4AniH4x14dxlR16+gVBJf7m4HngVvM7NU63tcegsrYPxD8Yt5HkOCqVN1SvEvS++HwfxIk1lXhe36G4BdxtGYB11adVCKkEFT67gzfa2+CsvPaMd9OUM/1qsI76qLc5w0EdV3nEyTzj8N9/YGgwhbgpwTFUx8TJNLat+J+F/gCQXn9lWEcVXHV+R0JZ98NXKrgLrB7ONi3CY79BoI6lMcIyvMP1WMEdVmPRUxLIEhcmwn+H04nTAQKHngtamSbVXfBFREcl5vN7OVw3g8IvvuFBEnjyYj1uobTdhMc210EnwfADwmuhBaGRWuvUk/xVLTChN3QZxyNWcAN4Y+dSA29l6r9G2HdHfCCpNRDeBuHRI0XHbuWImk2QSXhzfGOxTnnYs2vYJxzzsWEJxjnnHMx4UVkzjnnYsKvYJxzzsVEp3gH0Jp69eplgwYNincYzjnXbvTq1Yt58+bNM7OpTV33sEowgwYNYtGiRfEOwznn2pVon/GqzYvInHPOxYQnGOecczER1wSjoG+KtQr6B7mxjvnXKejfYKmC/ici+3u4KVxvbQs05eCcc66Fxa0OJmyv6n6C5rtzgfckzTWzyG56HzOz/wuXnwb8CpgaJprLgWMJWpt9VdLQsEkG51wbUlZWRm5uLiUlJfEOxTUiNTWV/v37k5SU1CLbi2cl/wSCPiU2AEh6AriAoD0pAMxsb8Ty6XzWAuwFBP2qHAA+lrQ+3N6/WiNw51z0cnNzycjIYNCgQaj1OlN0TWRm7Nq1i9zcXAYPHtwi24xngsmmZj8XuQQd89Qg6b8IGsRLJmhKvmrdyN7rcqmnbxFJ1xK2QjpwYO2Wzp1zsVZSUhKz5LJ7fynb9pRQWlFJcmICfTJT6d65wV6rXT0k0bNnT3bs2NFi22zzlfxmdr+ZHU3QymmTG4k0swfNLMfMcrKyou03yDnXkmKVXPJ2F1NaEbROX1pRSd7uYnbvb7DXateAlv6c4plg8qjZkVJ/Gu7Q6AngwkNc1znXwWzbU0JlraauKs3YtsfretqKeCaY94AhkgaHnW9dTq1+SyQNiRj9PLAuHJ4LXC4pRdJgYAhBpz7OucNE1ZVLtNNbyhtvvMH5558PwNy5c5k5c2a9yxYUFPDAAw80eR+33norv/jFL+qdP3bsWC6//PIa066++moGDx7M2LFjGTNmDK+99lr1vEmTJjFs2DDGjBnDCSecwNKlS5sc06GIW4Ixs3KCntbmEXRO9ZSZrZR0W3jHGMD1klZKWkpQDzM9XHclQYdaqwi6A/0vv4PMucNLcmLdp6/6pjemoqLpp5Bp06Zx440HPWFR7VATTENWr15NRUUFb775Jvv27asxb9asWSxdupTf/OY3XHfddTXmPfrooyxbtoxvfetbzJgxg9YQ1zoYM3vJzIaa2dFmdkc47SdmNjcc/q6ZHWtmY83sjDCxVK17R7jesIhe7Jxzh4k+makk1KozSJDok1mzw8aNGzcyfPhwrrzySkaMGMGll17K/v37gaD5qB/+8IeMHz+ep59+mldeeYWTTz6Z8ePH88UvfpGioqBTzb///e8MHz6c8ePH89xzz1Vve/bs2Vx//fUAbNu2jYsuuogxY8YwZswY3nnnHW688UY++ugjxo4dW31SnzVrFieccAKjR4/mlltuqd7WHXfcwdChQznllFNYu3Ztve/78ccf58tf/jKTJ0/mhRdeqHOZk08+mby8umsNGprX0g6rtsicc/H10xdXsmrz3sYXjFJ5pTGgexrXnDK4wbvI1q5dy0MPPcTEiRO55ppreOCBB/jBD34AQM+ePXn//ffZuXMnF198Ma+++irp6encdddd/OpXv+KGG27g61//Oq+//jrHHHMMl112WZ2xfOc73+H000/n+eefp6KigqKiImbOnMmKFSuqi6ReeeUV1q1bx7vvvouZMW3aNP75z3+Snp7OE088wdKlSykvL2f8+PEcf/zxde7nySefZP78+axZs4Z7772XL33pSwct8/e//50LL7ywjrUbntfSPME459qtTgmie3oyo/t3a3C5AQMGMHHiRACuuuoq7rnnnuoEU5UwFi5cyKpVq6qXKy0t5eSTT2bNmjUMHjyYIUOGVK//4IMPHrSP119/nT/96U8AJCYmkpmZye7du2ss88orr/DKK68wbtw4AIqKili3bh2FhYVcdNFFdO7cGQiK3uqyaNEievXqxcCBA8nOzuaaa64hPz+fHj16ADBjxgx+9KMfkZuby7/+VfOxwCuvvJLS0lKKioparQ7GE4xzrtXc8oVj47Lf2rffRo6np6cDwYOG55xzDo8//niNZVvyZGxm3HTTTXzjG9+oMf03v/lNVOs//vjjrFmzhqpuR/bu3cuzzz7L17/+dSAofrv00ku59957ueaaa1i8eHH1uo8++ijHH388M2bM4Nvf/naNor5YafPPwTjnXHN9+umn1b/oH3vsMU455ZSDljnppJN4++23Wb9+PQD79u3jww8/ZPjw4WzcuJGPPvoI4KAEVOWss87it7/9LRDcMLBnzx4yMjIoLCysXmbKlCk8/PDD1XU7eXl5bN++ndNOO405c+ZQXFxMYWEhL7744kHbr6ys5KmnnmL58uVs3LiRjRs38sILL9QZz/XXX09lZSXz5s2rMV0St99+OwsXLmTNmjWNHrfm8gTjnOvwhg0bxv3338+IESPYvXs33/zmNw9aJisri9mzZ3PFFVcwevTo6uKx1NRUHnzwQT7/+c8zfvx4evfuXec+7r77bhYsWMCoUaM4/vjjWbVqFT179mTixIkcd9xxzJgxg8mTJ/OlL32Jk08+mVGjRnHppZdSWFjI+PHjueyyyxgzZgznnnsuJ5xwwkHbf/PNN8nOzqZfv37V00477TRWrVrFli1baiwriZtvvpmf//znB20nLS2N73//+8yaNauph7HJZLUeVOrIcnJyzDscc651rV69mhEjRsRt/xs3buT8889nxYoVcYuhPanr85K02Mxymrotv4JxzjkXE55gnHMd2qBBg/zqJU48wTjnnIsJTzDOOediwhOMc865mPAE45xzLiY8wTjn3CH63Oc+d0jzGtJRmuoHTzDOOXfI3nnnnYOmlZeX1zuvudpTU/3gCcY518bMWZLHxJmvM/jGvzFx5uvMWdL8puX/8pe/MGHCBMaOHcs3vvGN6r5funTpwowZMzj22GM5++yzeffdd5k0aRJHHXUUc+cG/R/Onj2bCy64gEmTJjFkyBB++tOfVm+3S5cuQNAJ2amnnsq0adMYOXJkjXkAd911F6NGjWLMmDHV/cf8/ve/54QTTmDMmDFccskl1V0INKQ9NdUPnmCcc23InCV53PTccvIKijEgr6CYm55b3qwks3r1ap588knefvttli5dSmJiIo8++igQtDd25plnsnLlSjIyMrj55puZP38+zz//PD/5yU+qt/Huu+/y7LPP8sEHH/D0009TV4sg77//PnfffTcffvhhjekvv/wyL7zwAv/+979ZtmwZN9xwAwAXX3wx7733HsuWLWPEiBE89NBDjb6XJ598kssvv5wrrrii3jbR2kpT/RDn1pQlTQXuBhKBP5jZzFrzvwd8DSgHdgDXmNkn4bwKYHm46KdmVnf71s65dmPWvLUUl9XsWbK4rIJZ89Zy4bjsQ9rma6+9xuLFi6vb9youLq5uTyw5OZmpU6cCMGrUKFJSUkhKSmLUqFFs3LixehvnnHMOPXv2BILE8NZbb5GTU7PllAkTJjB48OCD9v/qq6/yla98pbop/qqm9VesWMHNN99MQUEBRUVFTJkypcH30d6a6oc4XsFISgTuB84FRgJXSBpZa7ElQI6ZjQaeASJbbisOe7oc68nFuY5hc0Fxk6ZHw8yYPn06S5cuZenSpaxdu5Zbb70VgKSkpOqm+xMSEkhJSakerqpLgYab+69S1ex/tK6++mruu+8+li9fzi233EJJSUmDy0c21X/00UdXN9VfZdasWXz44YfcddddXHPNNTXWffTRR9mwYQPTp0/n29/+dpPibI54FpFNANab2QYzKwWeAC6IXMDMFphZVcHkQqB/K8fonGtF/bqlNWl6NM466yyeeeYZtm/fDkB+fj6ffPJJk7Yxf/588vPzKS4uZs6cOdWdkkXjnHPO4Y9//GN1HUt+fj4AhYWF9O3bl7Kysuoiu/q0x6b6Ib4JJhvYFDGeG06rz1eBlyPGUyUtkrRQUr2FipKuDZdbtGPHjuZF7JyLqRlThpGWlFhjWlpSIjOmDDvkbY4cOZKf/exnTJ48mdGjR3POOecc1Lx9YyZMmMAll1zC6NGjueSSSw4qHmvI1KlTmTZtGjk5OYwdO7b6FuTbb7+dE088kYkTJzJ8+PAGt9Eem+qHODbXL+lSYKqZfS0c/zJwopldX8eyVwHXA6eb2YFwWraZ5Uk6CngdOMvMPmpon95cv3Otr6nN9c9ZkseseWvZXFBMv25pzJgy7JDrX1rC7NmzWbRoEffdd1/cYmhNLdlcfzwr+fOAARHj/cNpNUg6G/gxEckFwMzywr8bJL0BjAMaTDDOubbvwnHZcU0oruXEs4jsPWCIpMGSkoHLgbmRC0gaB/wOmGZm2yOmd5eUEg73AiYCq1otcufcYaOqMt41XdyuYMysXNL1wDyC25QfNrOVkm4DFpnZXGAW0AV4Orxro+p25BHA7yRVEiTJmWbmCca5NsrM6rzzyrUtLV1lEtfnYMzsJeClWtN+EjF8dj3rvQOMim10zrmWkJqayq5du+jZs6cnmTbMzNi1axepqaktts24JhjnXMfXv39/cnNz8bs4277U1FT692+5p0E8wTjnYiopKanOJ9xdx+dtkTnnnIsJTzDOOediwhOMc865mPAE45xzLiY8wTjnnIsJTzDOOediwhOMc865mPAE45xzLiY8wTjnnIsJTzDOOediwhOMc865mGi0LTJJWcDXgUGRy5vZNbELyznnXHsXTWOXLwBvAq8CFbENxznnXEcRTYLpbGY/jHkkzjnnOpRo6mD+Kum8WOxc0lRJayWtl3RjHfO/J2mVpA8kvSbpyIh50yWtC1/TYxGfc87F05wleUyc+TqDb/wbE2e+zpwlefEOqUmiSTDfJUgyJZIKw9fe5u5YUiJwP3AuMBK4QtLIWostAXLMbDTwDPDzcN0ewC3AicAE4BZJ3Zsbk3POtRVzluRx03PLySsoxoC8gmJuem55u0oyjSYYM8swswQzSw2HM8ysawvsewKw3sw2mFkp8ARwQa19LzCz/eHoQqCqq7UpwHwzyzez3cB8YGoLxOScc23CrHlrKS6rWe1dXFbBrHlr4xRR00XVo6WkacBp4egbZvbXFth3NrApYjyX4IqkPl8FXm5g3ey6VpJ0LXAtwMCBAw81Vueca1WbC4qbNL0tavQKRtJMgmKyVeHru5LujHVgtWK4CsgBZjV1XTN70MxyzCwnKyur5YNzzrkY6NctrUnT26Jo6mDOA84xs4fN7GGCoqjPt8C+84ABEeP9w2k1SDob+DEwzcwONGVd55xrr2ZMGUZaUmKNaWlJicyYMixOETVdtE/yd4sYzmyhfb8HDJE0WFIycDkwN3IBSeOA3xEkl+0Rs+YBkyV1Dyv3J4fTnHOuQ7hwXDZ3XjyK7G5pCMjulsadF4/iwnF11ga0SdHUwdwJLJG0ABBBXcxBtxQ3lZmVS7qeIDEkAg+b2UpJtwGLzGwuQZFYF+BpSQCfmtk0M8uXdDtBkgK4zczymxuTc861JReOy25XCaU2mVnjC0l9gRPC0XfNbGtMo4qRnJwcW7RoUbzDcM65dkXSYjPLaep69RaRSRoe/h0P9CW4UysX6BdOc8455+rVUBHZ9whu7/1lHfMMODMmETnnnOsQ6k0wZnZtOHiumZVEzpOUGtOonHPOtXvR3EX2TpTTnHPOuWr1XsFIOoLg6fi08HZhhbO6Ap1bITbnnHPtWEN1MFOAqwkeYvwlnyWYvcCPYhuWc8659q6hOphHgEckXWJmz7ZiTM455zqAaOpgjpdU/SR/+PT8z2IYk3POuQ4gmgRzrpkVVI2EzePHpAMy55xzHUc0CSZRUkrViKQ0IKWB5Z1zzrmo2iJ7FHhN0h/D8a8Aj8QuJOeccx1BownGzO6S9AFwVjjpdjPzloudc841KKoeLc3sZT7rTdI555xrVKMJRlIhQdtjAMlAErDPzLrGMjDnnHPtWzRFZBlVwwo6ZbkAOCmWQTnnnGv/ou3REgALzCF4yr/ZJE2VtFbSekkHdWIm6TRJ70sql3RprXkVkpaGr7m113XOORdf0RSRXRwxmgDkACX1LB41SYnA/cA5BP3MvCdprpmtiljsU4Lman5QxyaKzWxsc+Nwzh26OUvymDVvLZsLiunXLY0ZU4a16x4YXcuKppL/CxHD5cBGgmKy5poArDezDQCSngi3W51gzGxjOK+yBfbnnGtBc5bkcdNzyykuqwAgr6CYm55bDuBJxgHR1cF8JUb7zgY2RYznAic2Yf1USYsIkt7MsOjOOddKZs1bW51cqhSXVTBr3lpPMA5ouLn+e/ns7rGDmNl3YhJR9I40szxJRwGvS1puZh/VXkjStQQ9czJw4MDWjtF1cIdzEdHmguImTXeHn4Yq+RcBi4FUYDywLnyNJbhdubnygAER4/3DaVExs7zw7wbgDWBcPcs9aGY5ZpaTlZV16NE6V0tVEVFeQTHGZ0VEc5ZE/TVu1/p1S2vSdHf4qTfBmNkjYZP9o4FJZnavmd1L8ER/S1SuvwcMkTRYUjJwORDV3WBhi84p4XAvYCIRdTfOtYaGiogOBzOmDCMtKbHGtLSkRGZMGRaniFxbE81tyt0JerGs0iWc1ixmVg5cD8wDVgNPmdlKSbdJmgYg6QRJucAXgd9JWhmuPgJYJGkZsICgDsYTjGtVh3sR0YXjsrnz4lFkd0tDQHa3NO68eNRhU0ToGhfNXWQzgSWSFhD0ankacGtL7NzMXgJeqjXtJxHD7xEUndVe7x1gVEvE4Nyh6tctjbw6ksnhVER04bhsTyiuXo1ewZjZHwnu7noeeA44OSw6c+6w5kVEzjWs0QQTNg9zNjDGzF4AkiVNiHlkzrVxXkTkXMOiKSJ7AKgEzgRuAwqBZ4ETYhiXc+2CFxE5V79oEsyJZjZe0hIIukwO7/pyzjnn6hXNXWRlYbthBiApi+CKxjnnnKtXNFcw9xBU8PeWdAdwKfA/MY3KuUN0OD9Z71xbE01bZI9KWkzwgKWACwlaOXauTfHGF51rWxosIpOULSkH2GBm9wNPAV8maDLGuTblcH+y3rm2pt4EI+m/gaXAvcBCSV8jeOI+DTi+dcJzLnqH+5P1zrU1DRWRXQsMM7N8SQOBD4GJZra4dUJzrmn8yXrn2paGishKzCwfwMw+BdZ6cnFtmT9Z71zb0tAVTH9J90SM940cbwP9wThXQ1VFvt9F5lzb0L8VnwcAACAASURBVFCCmVFr3K9eXJvnT9Y713bUm2C8QUvnnHPNEc2T/M4551yTeYJxzjkXE9E019+jjmmDW2LnkqZKWitpvaQb65h/mqT3JZVLurTWvOmS1oWv6S0Rj3POuZYTzRXMi5Kqu0yWNBJ4sbk7DhvQvB84FxgJXBFuO9KnwNXAY7XW7QHcQtAR2gTgFknN7sbZOedcy4kmwfwvQZLpIul44GngqhbY9wRgvZltMLNS4AnggsgFzGyjmX3Awa03TwHmm1m+me0G5gNTWyAm55xzLSSaxi7/JikJeAXIAC4ysw9bYN/ZwKaI8VyCK5JDXdfvTXXOuTak3gQj6V7CPmBCmcBHwPWS2s2DlpKuJWj2hoEDB8Y5GuecO3w0dAWzqNZ4Sz9omQcMiBjvH06Ldt1JtdZ9o64FzexB4EGAnJwcq2sZ55xzLa9JD1qGFekDwnqR5noPGBLekZYHXA58Kcp15wH/G1GxPxm4qQViiinvDMs5dziJ5jblNyR1De/ceh/4vaRfNXfHZlYOXE+QLFYDT5nZSkm3SZoW7vsESbnAF4HfSVoZrpsP3E6QpN4DbqtqmLOtquoMK6+gGOOzzrDmLIn2os0559oXmTVcaiRpiZmNC/uDGWBmt0j6wMxGt06ILScnJ8cWLapd8tc6Js58vc6m5LO7pfH2jWfGISLnnIuOpMVmltPU9Rq9iwzoJKkv8B/Aj5scmQO8Myzn2iMv1m6eaJ6DuY2gGGu9mb0n6Si8y+Qmq6/TK+8My7m2yYu1m6/RBGNmT5vZaDP7Vji+wcwuiX1oHYt3huVc+zJr3lqKyypqTCsuq2DWvLVxiqj9aeg5mBvM7Od1PA8DeIdjTeWdYTnXvnixdvM1VAezOvwbn1rxDsg7w3Ku/ejXLa3OG3O8WDt6DT0H82L4t67nYfyReOdchzZjyjBuem55jWIyL9ZumgbrYCSdLOlSSb3D8dGSHgPebpXonHMuTi4cl82dF48iu1saInik4M6LR3kpRBM0VAczCzgfWAr8UNI84GvAncA1rROec7FXUWmUlFVwoLyS0qpXRQWl5UZ5ZSVlFZWUlhsVlUaFGRWVlVRUQqUZZkalQe3HySRIEEgiQSJBkJggOiUkBH8TRVJiAkmJIjkxgaTEBFKSEkjplEhypwRSOyXQKdH7A4w3L9ZunobqYD4PjDOzkrBJlk3AcWa2sVUic64BB8or2LO/jL0lZewpLmdvcTBcWFJO0YFyCkvKKCopZ19pBfsOBNP2l1awv7SC4tJguLisggNllZRW1O4Nom1IShSpSYmkJSXSOTmRtOROpCcnkpacSJeUTsErtRMZKZ3ISE2ia1onuqYm0TUticzw1a1zEl1SOiEp3m/HHYYaSjAlZlYCYGa7Ja3z5OJiobLSyN9fyq6iUnYVHWDnvlLyiw6Qv6+U/P2l7N5XRv6+UnbvL2VPcRkF+8sOun20tsQEkZ6cSEZqEp2TE0lP6UR6SiLdOyfTOTk4YacmVb0SSE1KJKXTZ1cQyZ0SSK6+ykigU+JnVx+JCSJRIiGB8OpE1D5/B1c3n/397OrHKK8IrozKK4zSiuAKqSrRlZZXVl9NlZQFSbA4TIb7DlSwv7ScvSXlbNlTEiTQA+UUlZYfdAVV+1h075xE987JdE9PpkfnZHp2SaZnlxR6dUmmZ3oKWRmfvdKTEz0huRbRUII5StLciPHBkeNmNi12YbmOoKyikh2FB9i2tyR8BcM7Cg+wPXztKDxA/r4DVNZxgpSge+dkunVOokfnZPp378yo7OBXebfOydW/1LumdiIzLSn4FZ8a/JpPTUo4bE6SlZVGUWk5hSXhlVxxWZCIi8vYs7+MguJSdu8vY/e+UvL3lbJ+RxHvbgwSdl2JKS0pkT5dU+jTNZU+XVM5IjOVI7qm0jczGO7XLY2sLikkJBwex9cduoYSzAW1xn8Zy0Bc+1JZaewsOkBuQTFbCkrYXFBMXkExW/eUsGVPMVv2lLCj6MBBJ7DEBJHVJYXeXVPol5nKmP6ZZGWk0KtLCj27JAd/05PpkZ5Mt87JJPpJrFEJCQqKxlKTyG7CLbQVlcbu/aXsLAoSfdVr294DbC8sYfveA3yQW8C8lSUcKK9ZjJiUKPpmppHdLY3s7mn0757GgO6d6d89jYE9O9MnI9UTkGvwNuV/tGYgrm0xM/L3lfJp/n427S5mU/5+cnfvJ3d3Mbm7g2RSWuukk56cSN9uafTNTGXYERkckZnGEV1TOSIzhd4Zwa/hnunJfuJpIxITRK8uQXIffkT9y5kZBfvL2BL+eNi8p4S83cXVPyreXLeDbXsP1FgnpVMCA3p05sgenTmyZzqDe1X9TadftzT/4XCYiKaxS9dBVVYa2wpL2LhzPxt37WPjrn18snM/n+TvZ1P+fooOlNdYvmd6Mv27pzGyb1cmj+xDdvc0+mUGv2D7dUuja6pXJndEkuieHtTfjOzXtc5lDpRXkLe7mE27i4MfJfn7+WTXPj7ZtZ93PtpVo84suVMCR/VK56isdI7O6sIxvbtwdFbwSktOrHP7rn3yBHMY2FNcxoYdRWzYsY8NO4v4eOc+NuwI/vlr/OMnJtC/RxpH9ujMiYN7MKBHZwaGr/7d00hP8a+Lq1tKp0SOyurCUVldDppnZmwvPMDHO/dVvz7aXsTqLYXMW7mNirACToL+3dMY2juDoUdkMLRPF4b2yeCY3l1I6eSJpz2K+owhqbOZ7Y9lMO7QmRk7Cg+wbnsR67YVsn5HEeu3F/HRjn3sKPys+CIxQQzs0ZnBvdKZeEwvBvVKZ3DPdI7s2dmLLlxMSKq+YeCko3rWmFdaXsnGXftYvz34vn64rZB124r4x4c7KA8TT2KCODorneFHdGVE366M7NeVkX27kpWREo+345qg0QQj6XPAH4AuwEBJY4BvVLWu3BySpgJ3A4nAH8xsZq35KcCfgOOBXcBlZrZR0iCCttKqmjVdaGbXNTee9mJn0QE+3FbIh1sLWbstSCjrthexp7isepmM1E4c07sLk4ZmcXRYBHFUVjoDe3QmyR/gc21EcqcEhvbJYGifjBrTqxLPmq2FrN26lzVbCln8yW7mLttcvUxWRgrH9uvKqOxMju2Xyaj+mfTLTPVi2jYkmiuYXwNTgLkAZrZM0mnN3bGkROB+4BwgF3hP0lwzWxWx2FeB3WZ2jKTLgbuAy8J5H5nZ2ObG0ZaVlFWwfnsRq7fsZc3WQtZs3cvarYXsLCqtXqZb5ySG9s7g/NF9q4sThvTuQlZGiv+juXarRuIZ0696+p79ZazaspdVW/aycvMeVm3ey5vrdlYXs/VMT2ZU/0xG9+/GmP6ZjBnQjV5d/EonXqIqIjOzTbVOVg0/5RadCQSdmG0AkPQEwa3RkQnmAuDWcPgZ4D510LPmrqID4T/NXlZt3svqLXvZsHNf9T9OSvgPd8aw3gw7IqP6ldXFE4k7fGR2TuLko3ty8tGfFbWVlFWwesteVuTt4YPc4PXPD9dVP1s1oEcaYwd0Z9yAbow/sjsj+3YluZNfxbeGaBLMprCYzCQlAd/ls6b8myOboPmZKrnAifUtY2blkvYAVd+swZKWAHuBm83szbp2Iula4FqAgQPj3wi0mbF1bwkr8oJ/iJWb97Aiby9b95ZUL9MvM5URfbsy5dgjGNG3K8P7ZjCoZ7rXjzhXh9SkRMYN7M64gd2rp+07UM6KvD0syy1gyacFLNqYz4th8VpKpwRG989k/JHdyTmyBzlHdqd7enK8wu/Qokkw1xHUk2QDecArQLPrX5ppCzDQzHZJOh6YI+lYM9tbe0EzexB4ECAnJ6eBBjViY+ueEj7ILWB53p7glbuHXfuCIq4EwdFZXTjpqB4c2y+TY/sFlZix/LJ7H+PucJCe0okTj+rJiRE3FWzdU8L7n+7m/U92s/jT3Tz81sf87h8bABjSuws5g3owYXB3Thzc0/t8aSHRJJhhZnZl5ARJE2l+k/15wICI8f7htLqWyZXUCcgEdpmZAQcAzGyxpI+AocS5c7Q9+8tYllvAsk0FLMvdwwe5BWwP7+BKTBBDenfhjOG9GZWdyXHZmYzom0Hn5Na79beqj/GqW5Or+hgHPMm4Du+IzFTOG9WX80b1BYKitQ9y9/DexnwWbcznrx9s5vF3PwWCYrUTB/fkpKOC4rimtJDgPhPN2e1eYHwU05rqPWCIpMEEieRy4Eu1lpkLTAf+BVwKvG5mJikLyDezCklHAUOADc2Mp0nKKipZs6WQJZt2s+TTIKls2Lmvev5RWcFtwKP7ZzK6fyYj+2bG/SGyhvoY9wTjDjepSYlMGNyDCYN7AEHTOWu27uXfG/J59+N8Xlu9jWcW5wIwsEdnTj6qJ587picTj+nlNw5EqaH+YE4GPgdkSfpexKyuBLcVN0tYp3I9MC/c3sNmtlLSbcAiM5sLPAT8WdJ6IJ8gCQGcBtwmqQyoBK4zs/zmxtSQ7XvDy+tPC3j/k90sz9tT3T5TVkYKYwd045Lj+zN2QDdG9c+ka2pSLMM5JN7HuHP1S0xQWFSdyTWnDKay0vhweyHvrN/Fvzbs4uUVW3hyUVBtPPyIDE45phenDs1iwqAecf/x2FbJ6mnnW9LpwCSCOpj/i5hVCLxoZutiHl0Ly8nJsUWLml6KdsF9b7Esdw8QPO1+XHbXsFKxG+MGdm83995PnPl6nX2MZ3dL4+0bz4xDRM61HxWVxoq8Pbz90U7eXr+T9zbuprS8kuROCUwY1INTh/Ri0rDeDO3TpV2cD5pC0mIzy2nyevUlmIgNH2lmnxxyZG3IoSaYB95YT3JiAuMGdue47K7tttmK2nUwEDTN7t3AOtd0JWUVvPtxPm+u28Gb63ayZmshAH0zUzl9aBaThmVxypAsunSAJpZimWAWAActZGbt7ifvoSaYjsTvInMuNrbuKeEfH27njbU7eGvdTgoPlJOUKCYM7sEZw3pz1og+DO6VHu8wD0ksE8zxEaOpwCVAuZnd0NSdxZsnGOdcayirqGTxJ7tZsGY7r6/ZzrrtRQAcnZXO2SP7cM6IPowb2L3dPNsWswRTz87eNbMJTV4xzjzBOOfiYVP+fl5bvY1XV29n4YZdlFcaPdOTOXtEHyYf24eJx/QiNantFr3H8gqmR8RoAkHDk/eY2bCm7izePME45+Jtb0kZ/1i7g/mrtrFgzXYKD5TTOTmRM4b1ZupxR3DG8N5trt7mUBNMNO9iMUEdjIBy4GOCRiidc841UdfUJL4wph9fGNOPA+UVLNyQz7yVW3ll5Tb+tnwLyZ0SOG1IFp8ffQRnj+hDRht85CFah1RE1l75FYxzrq2qqDQWbczn5RVbmbdyK1v2lJCcmMBpQ7M4f3Rfzh7ZJ25XNi1eRCbp4oZWNLPnmrqzePME45xrDyorjSWbdvO3D7by0vItbN1bQkqnBM4a0ZsvjO7HGcN7t2qdTSwSzB8bWM/M7Jqm7izePME459qbykpj8ae7+euyzfxt+RZ2FpXSJaUTU449ggvH9eNzR/eK+d1orXoXWXvlCcY5156VV1SycEM+c5fl8fLyrRQeKCcrI4UvjO7HxeOzObZf15i0IhDLu8gygVsI2v8C+Adwm5ntaXKUceYJxjnXUZSUVbBgzXbmLM1jwZodlFZUMqxPBheNz+bCsdkckZnaYvuKZYJ5FlgBPBJO+jIwxswarKNpizzBOOc6ooL9pfz1gy08934u739aQILglCFZfPH4/pwzsk+z62timWCWmtnYxqa1B55gnHMd3cc79/Hc+7k8uziXzXtKyExLYtqYfnx/8lC6dT60zgwPNcFE0zF1saRTInY0EfD23Z1zrg0a3Cud708exps/PJM/f3UCk4Zl8erqbXHpUiCam6q/CTwS1sWIoF+Wq2MZlHPOueZJTBCnDsni1CFZlFVUkpQYzfVEy2o0wZjZUmCMpK7h+EH93jvnnGu74pFcIIoiMknfDZNLIfArSe9LmtwSO5c0VdJaSesl3VjH/BRJT4bz/y1pUMS8m8LpayVNaYl4nHPOtZxo0to14VXLZKAnwV1kM5u7Y0mJwP3AucBI4ApJI2st9lVgt5kdA/wauCtcdyRB98nHAlOBB8LtOeecayOiSTBVT+2cB/zJzFZGTGuOCcB6M9tgZqXAE8AFtZa5gM9uj34GOEvBU0QXAE+Y2QEz+xhYH27POedcGxFNglks6RWCBDNPUgZQ2QL7zgY2RYznhtPqXMbMyoE9BFdR0awLgKRrJS2StGjHjh0tELZzzrloRJNgvgrcCJxgZvuBZOArMY2qBZnZg2aWY2Y5WVlZ8Q7HOecOG9HcRVYZVq5fJcmAt8zs+RbYdx4wIGK8fzitrmVyJXUCMoFdUa7rnHMujqK5i+wB4DpgOUGTMd+QdH8L7Ps9YIikwZKSCSrt59ZaZi4wPRy+FHjdgqYH5gKXh3eZDQaGAO+2QEzOOedaSDQPWp4JjAhP7Eh6BFjV3B2bWbmk64F5QCLwsJmtlHQbsMjM5gIPAX+WtJ7gAc/Lw3VXSnoqjKMc+C8zq2huTK7lzVmSx6x5a9lcUEy/bmnMmDKMC8fVWV3mnOtgokkw64GBwCfh+ABgXUvs3MxeAl6qNe0nEcMlwBfrWfcO4I6WiMPFxpwledz03HKKy4Lcn1dQzE3PLQfwJOPcYaDeIjJJL0qaC2QAqyW9IWkBsDqc5lyDZs1bW51cqhSXVTBr3to4ReSca00NXcH8ooF5h08vZe6QbS6ou03U+qY75zqWehOMmf2jrulhy8pXAP+MVVCuY+jXLY28OpJJv25pcYjGOdfaomoBTdI4SbMkbQRuJygmc65BM6YMI61WR0dpSYnMmDIsThE551pTvVcwkoYSXKlcAewEniTooOyMVorNtXNVFfl+F5lzh6eG6mDWAG8C55vZegBJ/69VonIdxoXjsj2hOHeYaqiI7GJgC7BA0u8lnUXLNHLpnHPuMFBvgjGzOWZ2OTAcWAD8N9Bb0m9bqj8Y55xzHVejlfxmts/MHjOzLxC0+bUE+GHMI3POOdeuNakfTTPbHbZOfFasAnLOOdcxxKejZueccx2eJxjnnHMx4QnGOedcTHiCcc45FxOeYJxzzsWEJxjnnHMxEZcEI6mHpPmS1oV/u9ez3PRwmXWSpkdMf0PSWklLw1fv1oveOedcNOJ1BXMj8JqZDQFeC8drkNQDuAU4EZgA3FIrEV1pZmPD1/bWCNo551z04pVgLgAeCYcfAS6sY5kpwHwzyzez3cB8YGorxeecc66Z4pVg+pjZlnB4K9CnjmWygU0R47nhtCp/DIvH/kdSvY1wSrpW0iJJi3bs2NHswJ1zzkWnoeb6m0XSq8ARdcz6ceSImZmkpnbBfKWZ5UnKAJ4Fvgz8qa4FzexB4EGAnJwc7+rZOedaScwSjJmdXd88Sdsk9TWzLZL6AnXVoeQBkyLG+wNvhNvOC/8WSnqMoI6mzgTjnHMuPuJVRDYXqLorbDrwQh3LzAMmS+oeVu5PBuZJ6iSpF4CkJOB8YEUrxOycc64J4pVgZgLnSFoHnB2OIylH0h8AzCwfuB14L3zdFk5LIUg0HwBLCa50ft/6b8E551xDZHb4VEvk5OTYokWL4h2Gc861K5IWm1lOU9fzJ/mdc87FhCcY55xzMeEJxjnnXEx4gnHOORcTnmCcc87FhCcY55xzMeEJxjnnXEx4gnHOORcTnmCcc87FhCcY55xzMeEJxjnnXEx4gnHOORcTnmCcc87FhCcY55xzMeEJxjnnXEx4gnHOORcTcUkwknpImi9pXfi3ez3L/V1SgaS/1po+WNK/Ja2X9KSk5NaJ3DnnXLTidQVzI/CamQ0BXgvH6zIL+HId0+8Cfm1mxwC7ga/GJErnnHOHLF4J5gLgkXD4EeDCuhYys9eAwshpkgScCTzT2PrOHa7mLMlj4szXGXzj35g483XmLMmLd0juMNQpTvvtY2ZbwuGtQJ8mrNsTKDCz8nA8F8iub2FJ1wLXAgwcOPAQQnWufZmzJI+bnltOcVkFAHkFxdz03HIALhxX77+Kcy0uZlcwkl6VtKKO1wWRy5mZARarOMzsQTPLMbOcrKysWO3GuTZj1ry11cmlSnFZBbPmrY1TRO5wFbMrGDM7u755krZJ6mtmWyT1BbY3YdO7gG6SOoVXMf0Bv/53LrS5oLhJ052LlXjVwcwFpofD04EXol0xvOJZAFx6KOs719H165bWpOnOxUq8EsxM4BxJ64Czw3Ek5Uj6Q9VCkt4EngbOkpQraUo464fA9yStJ6iTeahVo3euDZsxZRhpSYk1pqUlJTJjyrA4ReQOV3Gp5DezXcBZdUxfBHwtYvzUetbfAEyIWYDOtWNVFfmz5q1lc0Ex/bqlMWPKMK/gd60uXneROedi6MJx2Z5QXNx5UzHOOediwhOMc865mPAE45xzLiY8wTjnnIsJTzDOOediQsFzi4cHSTuAT2Kw6V7AzhhstyV4bIfGYzs0HtuhacuxpQK5Zja1qSseVgkmViQtMrOceMdRF4/t0Hhsh8ZjOzQdNTYvInPOORcTnmCcc87FhCeYlvFgvANogMd2aDy2Q+OxHZoOGZvXwTjnnIsJv4JxzjkXE55gnHPOxYQnmCaQNFXSWknrJd1Yx/zvSVol6QNJr0k6sg3Fdp2k5ZKWSnpL0si2ElvEcpdIMkmtdrtmFMftakk7wuO2VNLX6tpOPGILl/mP8Du3UtJjbSU2Sb+OOGYfSipoQ7ENlLRA0pLwf/W8NhTbkeG54wNJb0jq30pxPSxpu6QV9cyXpHvCuD+QND6qDZuZv6J4AYnAR8BRQDKwDBhZa5kzgM7h8DeBJ9tQbF0jhqcBf28rsYXLZQD/BBYCOW0lNuBq4L42+n0bAiwBuofjvdtKbLWW/zbwcFuJjaDS+pvh8EhgYxuK7Wlgejh8JvDnVortNGA8sKKe+ecBLwMCTgL+Hc12/QomehOA9Wa2wcxKgSeACyIXMLMFZrY/HF0ItMqvjyhj2xsxmg601t0djcYWuh24CyhppbiaEls8RBPb14H7zWw3gJltb0OxRboCeLxVIosuNgO6hsOZwOY2FNtI4PVweEEd82PCzP4J5DewyAXAnyywEOgmqW9j2/UEE71sYFPEeG44rT5fJcj4rSGq2CT9l6SPgJ8D32krsYWX2wPM7G+tFFOVaD/TS8JigWckDWid0KKKbSgwVNLbkhZKanJTHjGMDQiKfIDBfHbSjLVoYrsVuEpSLvASwRVWa4gmtmXAxeHwRUCGpJ6tEFtjmnr+AzzBxISkq4AcYFa8Y4lkZveb2dHAD4Gb4x0PgKQE4FfA9+MdSz1eBAaZ2WhgPvBInOOJ1ImgmGwSwVXC7yV1i2tEB7sceMbMKuIdSIQrgNlm1p+g6OfP4fewLfgBcLqkJcDpQB7Qlo5dk7SVg9oe5AGRv177h9NqkHQ28GNgmpkdaEuxRXgCuDCmEX2msdgygOOANyRtJCjfndtKFf2NHjcz2xXxOf4BOL4V4ooqNoJfkXPNrMzMPgY+JEg4bSG2KpfTesVjEF1sXwWeAjCzfxE05tirLcRmZpvN7GIzG0dwHsHMWu0GiQY09RwTaI0KpI7wIvi1uIHgcr+qgu7YWsuMI6jEG9IGYxsSMfwFYFFbia3W8m/QepX80Ry3vhHDFwEL21BsU4FHwuFeBEUYPdtCbOFyw4GNhA90t6Hj9jJwdTg8gqAOJuYxRhlbLyAhHL4DuK0Vj90g6q/k/zw1K/nfjWqbrRV8R3gRXE5/GCaRH4fTbiO4WgF4FdgGLA1fc9tQbHcDK8O4FjR0km/t2Got22oJJsrjdmd43JaFx214G4pNBMWLq4DlwOVtJbZw/FZgZmvF1ITjNhJ4O/xMlwKT21BslwLrwmX+AKS0UlyPA1uAMoIr468C1wHXRXzX7g/jXh7t/6g3FeOccy4mvA7GOedcTHiCcc45FxOeYJxzzsWEJxjnnHMx4QnGOedcTHiCce2apIqwxd5lkt6X9LkW3v5sSZeGw39oiVaoJU2StCeMe42kX0TMm9ZIi9NXS7qvke1HtgC9RtL/a27Mzh2KTvEOwLlmKjazsQCSphA8t3J6LHZkZi3ZVP+bZna+pDRgiaTnzextM5sLzG2B7T9pZteH7VitlfSMmW1qdC3nWpBfwbiOpCuwG0BSl7BfjffDfnAuCKenS/pbeMWzQtJl4fTjJf1D0mJJ8+pqKTbsnyMnHC6SdEe4nYWS+oTTsyQ9K+m98DWxoYDNrJjgYb/scP3qKxRJXwxjXCbpn3XE83lJ/5JUbzMnZrYLWA/0Ddf5SRjXCkkPSlLEe7tL0rth/y2nhtM7S3pKQZ8zz0v6d8QxmBzu/31JT0vq0tB7dYcfTzCuvUurKgoiePL59nB6CXCRmY0n6Kfnl+HJdCqw2czGmNlxwN8lJQH3Apea2fHAwwTNdDQknaDZmDEE/dh8PZx+N/BrMzsBuCSMqV6SuhO0H3ZQAgF+AkwJ9zGt1noXATcC55nZzga2P5Cgra0Pwkn3mdkJ4XtPA86PWLyTmU0A/hu4JZz2LWC3mY0E/oewLbYwqd0MnB0e40XA9xp6r+7w40Vkrr2LLCI7GfiTpOMImrb4X0mnAZUEVwh9CJq5+KWku4C/mtmb4fLHAfPDH/SJBM1mNKQU+Gs4vBg4Jxw+GxgZbgegq6QuZlZUa/1TJS0jSC6/MbOtdezjbWC2pKeA5yKmn0nQWvdkq9nPT6TLwvc+HLjezKr62TlD0g1AZ6AHQTM4L4bzqvaxmKBdKoBTCJImZrZCUlWiOomwyZXwvSYD/6onFneY8gTjOgwzqyouyiJo8ykLON7MysKWmlPN7EMF/c+cB/xM0mvA88BKMzu5Cbsrs8/aWargs/+lBOCkiBN6farqYAYDCyU9ZWZLa72f6ySdSNDQ4GJJVS05V/WKOJTgyqEuVXUwOcArkuYCBcADIcQuqwAAAapJREFUBO1IbZJ0K8HVTZWqVqMj3099BMw3sysaWc4dxryIzHUYkoYTXH3sIuipcHuYXM4AjgyX6QfsN/v/7d29K4VhHMbx75XOwmxXRxQpyab8FbIosRr4D1jOQHYpm5ckfwEpk3fDCYnRzGKz6Ge4b45OOedJPYNcn/E5d8+5n+nqfun3i21Sv54R4BHozisgJFUkDf5yGod8a2AlabjV4Ehl9pdJPXqav6caERcRsQg80yiX/kTafttsN8+IuAa2gAUaYfKSz0smCnzPCTCZ5zMADOXn58CYpN78W5ekvgLvs3/EAWN/3ecZTB3YI/Uzfwd2gFFJt8A08JDHDwGXefwSUIvUvnYCWMnbVnXgt9ed5/P/3ki6J1WkbWcdGJfU0/R8NV9QuANOSdV/AYiIB2AK2JdUbfP+FWCWtDLZAO6AA+CqwNzWSOF7D9RIW2qvEfEMzAC7edvsjLQdZ/bF1ZTN7EeSOoBKRLzlIDsC+nMom7XkMxgza6UTOM437QTMOVysKK9gzMysFD6DMTOzUjhgzMysFA4YMzMrhQPGzMxK4YAxM7NSfAAIRPUY8JXcNwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "fz8Es6q98Kjw"
      },
      "source": [
        "In the plot, the empirical absolute risk reduction is shown as circles, whereas the predicted risk reduction from the logistic regression model is given by the solid line.\n",
        "\n",
        "If ARR depended only on baseline risk, then if we plotted actual (empirical) ARR grouped by baseline risk, then it would follow the model's predictions closely (the dots would be near the line in most cases).\n",
        "\n",
        "However, you can see that the empirical absolute risk reduction (shown as circles) does not match the predicted risk reduction from the logistic regression model (given by the solid line). \n",
        "\n",
        "This may indicate that ARR may depend on more than simply the baseline risk.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "aAgIlK6Z8s2p"
      },
      "source": [
        "<a name=\"3\"></a>\n",
        "## 3 Evaluation Metric"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "oCASYrsI1EFI"
      },
      "source": [
        "<a name=\"3-1\"></a>\n",
        "### 3.1 C-statistic-for-benefit (C-for-benefit)\n",
        "\n",
        "You'll now use a measure to evaluate the discriminative power of your models for predicting ARR. Ideally, you could use something like the regular Concordance index (also called C-statistic) from Course 2. Proceeding by analogy, you'd like to estimate something like:\n",
        "\n",
        "$$P(A \\text{ has higher predicted ARR than } B| A \\text{ experienced a greater risk reduction than } B).$$\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xFgWJRP-TmrR",
        "colab_type": "text"
      },
      "source": [
        "#### The ideal data cannot be observed\n",
        "\n",
        "The fundamental problem is that for each person, you can only observe either their treatment outcome or their baseline outcome. \n",
        "- The patient either receives the treatment, or does not receive the treatment.  You can't go back in time to have the same patient undergo treatment and then not have treatment.\n",
        "- This means that you can't determine what their actual risk reduction was. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bj_L6AuwTmrS",
        "colab_type": "text"
      },
      "source": [
        "#### Estimate the treated/untreated patient using a pair of patients\n",
        "\n",
        "What you will do instead is match people across treatment and control arms based on predicted ARR. \n",
        "- Now, in each pair, you'll observe both outcomes, so you'll have an estimate of the true treatment effect.\n",
        "- In the pair of patients (A,B), \n",
        "    - Patient A receives the treatment \n",
        "    - Patient B does not receive the treatment.\n",
        "- Think of the pair of patients as a substitute for the the ideal data that has the same exact patient in both the treatment and control group."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vy9H4ji5TmrS",
        "colab_type": "text"
      },
      "source": [
        "#### The C-for-benefit\n",
        "\n",
        "$$P(\\text{$P_1$ has a predicted ARR greater than $P_2$} | \\text{$P_1$ experiences greater risk reduction than $P_2$}),$$\n",
        "\n",
        "- Pair 1 consists of two patients (A,B), where A receives treatment, B does not.\n",
        "- Pair 2 is another pair of two patients (A,B), where A receives treatment, B does not.\n",
        "\n",
        "The risk reduction for each pair is:\n",
        "- 1 if the treated person A survives and the untreated B person does not (treatment helps).  \n",
        "- -1 if the treated person A dies and the untreated person B doesn't (treatment harms)\n",
        "- 0 otherwise (treatment has no effect, because both patients in the pair live, or both die)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jccz_HJzTmrT",
        "colab_type": "text"
      },
      "source": [
        "#### Details for calculating C-for-benefit\n",
        "\n",
        "The c-for-benefit gives you a way to evaluate the ability of models to discriminate between patient profiles which are likely to experience greater benefit from treatment. \n",
        "- If you are better able to predict how likely a treatment can improve a patient's outcome, you can help the doctor and patient make a more informed decision when deciding whether to undergo treatment, considering the possible side-effects and other risks associated with treatment.\n",
        "\n",
        "Please complete the implementation of the C-statistic-for-benefit below.  \n",
        "\n",
        "The code to create the pairs is given to you. \n",
        "```CPP\n",
        "obs_benefit_dict = {\n",
        "        (0, 0): 0,\n",
        "        (0, 1): -1,\n",
        "        (1, 0): 1,\n",
        "        (1, 1): 0,\n",
        "    }\n",
        "```\n",
        "Here is the interpretation of this dictionary for a pair of patients, (A,B), where A receives treatment and B does not:  \n",
        "- When patient A does not die, and neither does patient B, `(0, 0)`, the observed benefit of treatment is 0.\n",
        "- When patient A does not die, but patient B does die, `(0, 1)`, the observed benefit is -1 (the treatment helped).\n",
        "- When patient A dies, but patient B does not die, `(1, 0)`, the observed benefit is 1 (the treatment was harmful)\n",
        "- When patient A dies and patient B dies, `(0, 0)`, the observed benefit of treatment is 0.\n",
        "\n",
        "Each patient in the pair is represented by a tuple `(ARR, y)`.\n",
        "- Index 0 contains the predicted ARR, which is the predicted benefit from treatment.\n",
        "- Index 1 contains the actual patient outcome: 0 for no death, 1 for death.\n",
        "\n",
        "So a pair of patients is represented as a tuple containing two tuples:\n",
        "\n",
        "For example, Pair_1 is `( (ARR_1_A, y_1_A),(ARR_1_B, y_1_B))`, and the data may look like:\n",
        "`( (0.60, 0),(0.40, 1))`.  \n",
        "- This means that patient A (who received treatment) has a predicted benefit of 0.60 and does not die.\n",
        "- Patient B (who did not receive treatment) has a predicted benefit of 0.40 and dies."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hdzxdG5FTmrT",
        "colab_type": "text"
      },
      "source": [
        "<a name='ex-07'></a>\n",
        "### Exercise 7: Calculate c for benefit score\n",
        "In `c_for_benefit_score`, you will compute the C-for-benefit given the matched pairs.\n",
        "\n",
        "$$\\text{c for benefit score} = \\frac{concordant + 0.5 \\times risk\\_ties}{permissible}$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OQNpIlO_TmrV",
        "colab_type": "text"
      },
      "source": [
        "<details>    \n",
        "<summary>\n",
        "    <font size=\"3\" color=\"darkgreen\"><b>Click here for Hints!</b></font>\n",
        "</summary>\n",
        "<p>\n",
        "<ul>\n",
        "    <li>A pair of patients in this case are two patients whose data are used to represent a single patient.</li>\n",
        "    <li> A pair of pairs is similar to what you think of as just a \"pair\" in the course 2 concordance index.  It's a pair of pairs of patients (four patients total).</li>\n",
        "    <li>Each patient is represented by a tuple of two values.  The first value is the predicted risk reduction, and the second is the patient's outcome.</li>\n",
        "    <li>observed benefit: for each patient pair, the first patient is assumed to be the one who received treatment, and second in the pair is the one who did not receive treatment.  Observed benefit is either 0 (no effect), -1 (treatment helped), 1 (treatment harmed)</li>\n",
        "    <li>predicted benefit: for each patient pair, take the mean of the two predicted benefits.  This is the first value in each patient's tuple.</li>\n",
        "    <li>permissible pair of pairs: observed benefit is different between the two pairs of pairs of patients.</li>\n",
        "    <li>concordant pair: the observed benefit and predicted benefit of pair 1 are both less than those for pair 2; or, the observed and predicted benefit of pair 1 are both greater than those for pair 2.  Also, it should be a permissible pair of pairs.</li>\n",
        "    <li>Risk tie: the predicted benefits of both pairs are equal, and it's also a permissible pair of pairs.</li>\n",
        "</ul>\n",
        "</p>\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "XYYwXThLOZKi",
        "colab": {}
      },
      "source": [
        "# UNQ_C7 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
        "def c_for_benefit_score(pairs):\n",
        "    \"\"\"\n",
        "    Compute c-statistic-for-benefit given list of\n",
        "    individuals matched across treatment and control arms. \n",
        "\n",
        "    Args:\n",
        "        pairs (list of tuples): each element of the list is a tuple of individuals,\n",
        "                                the first from the control arm and the second from\n",
        "                                the treatment arm. Each individual \n",
        "                                p = (pred_outcome, actual_outcome) is a tuple of\n",
        "                                their predicted outcome and actual outcome.\n",
        "    Result:\n",
        "        cstat (float): c-statistic-for-benefit computed from pairs.\n",
        "    \"\"\"\n",
        "    \n",
        "    # mapping pair outcomes to benefit\n",
        "    obs_benefit_dict = {\n",
        "        (0, 0): 0,\n",
        "        (0, 1): -1,\n",
        "        (1, 0): 1,\n",
        "        (1, 1): 0,\n",
        "    }\n",
        "    \n",
        "    ### START CODE HERE (REPLACE INSTANCES OF 'None', 'False', and 'pass' with your code) ###\n",
        "\n",
        "    # compute observed benefit for each pair\n",
        "    obs_benefit = [obs_benefit_dict[(pair[0][1],pair[1][1])] for pair in pairs]\n",
        "\n",
        "    # compute average predicted benefit for each pair\n",
        "    pred_benefit = [np.mean([pair[0][0], pair[1][0]]) for pair in pairs]\n",
        "\n",
        "    concordant_count, permissible_count, risk_tie_count = 0, 0, 0\n",
        "\n",
        "    # iterate over pairs of pairs\n",
        "    for i in range(len(pairs)):\n",
        "        for j in range(i + 1, len(pairs)):\n",
        "            \n",
        "            # if the observed benefit is different, increment permissible count\n",
        "            if obs_benefit[i] != obs_benefit[j]:\n",
        "\n",
        "                # increment count of permissible pairs\n",
        "                permissible_count += 1\n",
        "                \n",
        "                # if concordant, increment count\n",
        "                \n",
        "                if (obs_benefit[i] < obs_benefit[j]) == (pred_benefit[i] < pred_benefit[j]): # change to check for concordance\n",
        "                    \n",
        "                    concordant_count += 1\n",
        "\n",
        "                # if risk tie, increment count\n",
        "                if (pred_benefit[i] == pred_benefit[j]): #change to check for risk ties\n",
        "                    risk_tie_count += 1\n",
        "\n",
        "\n",
        "    # compute c-statistic-for-benefit\n",
        "    cstat = (concordant_count + 0.5*risk_tie_count)/permissible_count\n",
        "    \n",
        "    # END CODE HERE\n",
        "    \n",
        "    return cstat"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dGvjimwJTmre",
        "colab_type": "text"
      },
      "source": [
        "**Test Case**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CljNTQ1GTmrg",
        "colab_type": "code",
        "colab": {},
        "outputId": "35beccfd-0709-4926-9b6c-473d6ed8f8aa"
      },
      "source": [
        "print(\"TEST CASE\")\n",
        "tmp_pairs = [((0.64, 1), (0.54, 0)), \n",
        "             ((0.44, 0),(0.40, 1)), \n",
        "             ((0.56, 1), (0.74, 0)), \n",
        "             ((0.22,0),(0.22,1)), \n",
        "             ((0.22,1),(0.22,0))]\n",
        "print(f\"pairs: {tmp_pairs}\")\n",
        "tmp_cstat = c_for_benefit_score(tmp_pairs)\n",
        "print(f\"Output: {tmp_cstat:.4f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TEST CASE\n",
            "pairs: [((0.64, 1), (0.54, 0)), ((0.44, 0), (0.4, 1)), ((0.56, 1), (0.74, 0)), ((0.22, 0), (0.22, 1)), ((0.22, 1), (0.22, 0))]\n",
            "Output: 0.7500\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lt3YwStvTmrp",
        "colab_type": "text"
      },
      "source": [
        "##### Expected Output\n",
        "\n",
        "```CPP\n",
        "TEST CASE\n",
        "pairs: [((0.64, 1), (0.54, 0)), ((0.44, 0), (0.4, 1)), ((0.56, 1), (0.74, 0)), ((0.22, 0), (0.22, 1)), ((0.22, 1), (0.22, 0))]\n",
        "Output: 0.7500\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x0oLufqgTmrq",
        "colab_type": "text"
      },
      "source": [
        "<a name='ex-08'></a>\n",
        "### Exercise 8: Create patient pairs and calculate c-for-benefit\n",
        "\n",
        "You will implement the function `c_statistic`, which prepares the patient data and uses the c-for-benefit score function to calculate the c-for-benefit:\n",
        "\n",
        "- Take as input:\n",
        "    - The predicted risk reduction `pred_rr` (ARR)\n",
        "    - outcomes `y` (1 for death, 0 for no death)\n",
        "    - treatments `w` (1 for treatment, 0 for no treatment)\n",
        "- Collect the predicted risk reduction, outcomes and treatments into tuples, one tuple for each patient.\n",
        "- Filter one list of tuples where patients did not receive treatment.\n",
        "- Filter another list of tuples where patients received treatment.\n",
        "\n",
        "- Make sure that there is one treated patient for each untreated patient.\n",
        "    - If there are fewer treated patients, randomly sample a subset of untreated patients, one for each treated patient.\n",
        "    - If there are fewer untreated patients, randomly sample a subset of treated patients, one for each untreated patient.\n",
        "    \n",
        "- Sort treated patients by their predicted risk reduction, and similarly sort the untreated patients by predicted risk reduction.\n",
        "    - This allows you to match the treated patient with the highest predicted risk reduction with the untreated patient with the highest predicted risk reduction.  Similarly, the second highest treated patient is matched with the second highest untreated patient.\n",
        "    \n",
        "- Create pairs of treated and untreated patients."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KcxMqTK9Tmrq",
        "colab_type": "text"
      },
      "source": [
        "<details>    \n",
        "<summary>\n",
        "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
        "</summary>\n",
        "<p>\n",
        "<ul>\n",
        "    <li> Use zip(a,b,c) to create tuples from two or more lists of equal length, and use list(zip(a,b,c)) to store that as a list data type.</li>\n",
        "    <li> Use filter(lambda x: x[0] == True, some_list) to filter a list (such as a list of tuples) so that the 0th item in each tuple is equal to True. Cast the result as a list using list(filter(lambda x: x[0] == True, some_list)) </li>\n",
        "    <li>Use random.sample(some_list, sub_sample_length) to sample a subset from a list without replacement.</li>\n",
        "    <li>Use sorted(some_list, key=lambda x: x[1]) to sort a list of tuples by their value in index 1.</li>\n",
        "</ul>\n",
        "</p>\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u6lC5CNWTmrs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# UNQ_C8 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
        "def c_statistic(pred_rr, y, w, random_seed=0):\n",
        "    \"\"\"\n",
        "    Return concordance-for-benefit, the proportion of all matched pairs with\n",
        "    unequal observed benefit, in which the patient pair receiving greater\n",
        "    treatment benefit was predicted to do so.\n",
        "\n",
        "    Args: \n",
        "        pred_rr (array): array of predicted risk reductions\n",
        "        y (array): array of true outcomes\n",
        "        w (array): array of true treatments \n",
        "    \n",
        "    Returns: \n",
        "        cstat (float): calculated c-stat-for-benefit\n",
        "    \"\"\"\n",
        "    assert len(pred_rr) == len(w) == len(y)\n",
        "    random.seed(random_seed)\n",
        "    \n",
        "    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
        "    # Collect pred_rr, y, and w into tuples for each patient\n",
        "    tuples = tuple(zip(pred_rr,y,w))\n",
        "    \n",
        "    # Collect untreated patient tuples, stored as a list\n",
        "    untreated = list(filter(lambda x:x[2] == 0, tuples))\n",
        "    \n",
        "    # Collect treated patient tuples, stored as a list\n",
        "    treated = list(filter(lambda x:x[2] == 1, tuples))\n",
        "\n",
        "    # randomly subsample to ensure every person is matched\n",
        "    \n",
        "    # if there are more untreated than treated patients,\n",
        "    # randomly choose a subset of untreated patients, one for each treated patient.\n",
        "\n",
        "    if len(treated) < len(untreated):\n",
        "        untreated = random.sample(untreated, len(treated))\n",
        "        \n",
        "    # if there are more treated than untreated patients,\n",
        "    # randomly choose a subset of treated patients, one for each treated patient.\n",
        "    if len(untreated) < len(treated):\n",
        "        treated = random.sample(treated, len(untreated))\n",
        "        \n",
        "    assert len(untreated) == len(treated)\n",
        "\n",
        "    # Sort the untreated patients by their predicted risk reduction\n",
        "    untreated = sorted(untreated, key=lambda x:x[0])\n",
        "    \n",
        "    # Sort the treated patients by their predicted risk reduction\n",
        "    treated = sorted(treated, key=lambda x:x[0])\n",
        "    \n",
        "    # match untreated and treated patients to create pairs together\n",
        "    pairs = list(zip(untreated, treated))\n",
        "\n",
        "    # calculate the c-for-benefit using these pairs (use the function that you implemented earlier)\n",
        "    cstat = c_for_benefit_score(pairs)\n",
        "    \n",
        "    ### END CODE HERE ###\n",
        "    \n",
        "    return cstat"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GSI-0LJ8Tmrw",
        "colab_type": "code",
        "colab": {},
        "outputId": "55603a5a-def5-4fd1-8798-f19f79e83f62"
      },
      "source": [
        "# Test\n",
        "\n",
        "tmp_pred_rr = [0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9]\n",
        "tmp_y = [0,1,0,1,0,1,0,1,0]\n",
        "tmp_w = [0,0,0,0,1,1,1,1,1]\n",
        "\n",
        "tmp_cstat = c_statistic(tmp_pred_rr, tmp_y, tmp_w)\n",
        "\n",
        "print(f\"C-for-benefit calculated is {tmp_cstat}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "C-for-benefit calculated is 0.6\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lAHuSkK7Tmr0",
        "colab_type": "text"
      },
      "source": [
        "##### Expected output\n",
        "\n",
        "```CPP\n",
        "C-for-benefit calculated is 0.6\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "XH_yDTAq3D42"
      },
      "source": [
        "### Predicted risk reduction\n",
        "In order to compute the c-statistic-for-benefit for any of your models, you need to compute predicted risk reduction from treatment (predicted risk reduction is the input `pred_rr` to the c-statistic function).\n",
        "\n",
        "- The easiest way to do this in general is to create a version of the data where the treatment variable is False and a version where it is True.\n",
        "- Then take the difference $\\text{pred_RR} = p_{control} - p_{treatment}$\n",
        "\n",
        "We've implemented this for you."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:31:43.624458Z",
          "start_time": "2020-04-04T15:31:43.619458Z"
        },
        "colab_type": "code",
        "id": "arBYI7rR4lqr",
        "colab": {}
      },
      "source": [
        "def treatment_control(X):\n",
        "    \"\"\"Create treatment and control versions of data\"\"\"\n",
        "    X_treatment = X.copy(deep=True)\n",
        "    X_control = X.copy(deep=True)\n",
        "    X_treatment.loc[:, 'TRTMT'] = 1\n",
        "    X_control.loc[:, 'TRTMT'] = 0\n",
        "    return X_treatment, X_control\n",
        "\n",
        "def risk_reduction(model, data_treatment, data_control):\n",
        "    \"\"\"Compute predicted risk reduction for each row in data\"\"\"\n",
        "    treatment_risk = model.predict_proba(data_treatment)[:, 1]\n",
        "    control_risk = model.predict_proba(data_control)[:, 1]\n",
        "    return control_risk - treatment_risk"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "E4g3JazHF1G9"
      },
      "source": [
        "Now let's compute the predicted risk reductions of the logistic regression model on the test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qc2lHbSzTmr8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "X_test_treated, X_test_untreated = treatment_control(X_test)\n",
        "rr_lr = risk_reduction(lr, X_test_treated, X_test_untreated)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "uv0Yr96aGaeL"
      },
      "source": [
        "Before we evaluate the c-statistic-for-benefit, let's look at a histogram of predicted ARR."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:31:51.575460Z",
          "start_time": "2020-04-04T15:31:51.420183Z"
        },
        "colab_type": "code",
        "id": "Oa0gA4rCGZtU",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 444
        },
        "outputId": "8f8b1896-8276-4101-f488-1453389c62bc"
      },
      "source": [
        "plt.hist(rr_lr, bins='auto')\n",
        "plt.title(\"Histogram of Predicted ARR using logistic regression\")\n",
        "plt.ylabel(\"count of patients\")\n",
        "plt.xlabel(\"ARR\")\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEWCAYAAABhffzLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3deZgcZbn38e+PsASEEDAjBgIMmwtBCBgBEc9BFEUBQUQOiAgHNHJwQ0VBjud9we0FDosoCkaR1cMiHBRxQWQVQSDBJKxKgCBLQiZADBFEA/f7x/MM6XS6Z2p6prpnUr/PdfXVtdf9VFXf/fRT1VWKCMzMrDpW6nQAZmbWXk78ZmYV48RvZlYxTvxmZhXjxG9mVjFO/GZmFVO5xC/pXkm7dDqOTpL0AUmPSVosads2r/tGSR/L3QdJ+k0b1tktKSStXPa6yiTp7ZL+VNKyX9kvg1zOcZJ+2OK8iyVtOtgYhjtJZ0v6r07GsEIlfklzJL2rbtihkm7p7Y+IiRFxYz/LWSESRR9OAT4VEWtGxB/rR+ay/y1/EJ+QdJqkUUMdRET8OCLe3d90ko6XdNFQr7/Bem6U9Kyk1eqGnyfpH3l7PCPpWklvqBl/qKSX8vhFkmZK2nOo44uI30XE64d6uUMpIr4ZEf1+gTT6osnH48PlRTc8RMQREfG1TsawQiX+kWIYfKFsDNzbzzTbRMSawDuBDwMfr59gGJRjyEjqBt4OBPD+BpOcnLfHBsATwDl142/L48cC3wMukTS2tICtoaE+JlekY7xW5RJ/7a8CSdtLmpZraU9JOi1PdnN+X5hrcW+VtJKkr0h6VNJ8SRdIWrtmuR/N456W9F916zle0uWSLpK0CDg0r/s2SQslzZV0pqRVa5YXko6U9KCk5yR9TdJmkm7N8V5WO31dGRvGKmk1SYuBUcBMSQ/1t70i4gHgd8BWNb+EDpf0F+D6vL7DJN2fa8vXSNq4JpbdJD0g6a+SzgRUM26ZX2OSJuba9DN5fxwnaXfgOODf8r6YmaddW9I5eds9Ienrvb9KJI2SdIqkBZIeBvbor5zAR4E/AOcBh/SxPV4ALgMmNRn/MnAh8Cpgi0bT1Jc7DwtJm+fu90m6L+/3JyQdnYfvIunxmnnmSDpa0qy8fS+VNLpm/Jfy9nlS0sdq19GXITjWL8rdo/Mx/3Q+zu+UtJ6kb5C+ZM/M+/TMBttgdUmn5vX8VdItklZvEOsukh6XdIykecC5Of5jJT2U132ZpHUHEH/9Z7WvY21zSTflGBdIujQPl6TT8/ZbJOluSVvlcedJ+npNPB+XNDsf91dJWr/uuDhCKQ8slPRdSa98hloWESvMC5gDvKtu2KHALY2mAW4DDs7dawI75u5uUs1v5Zr5DgNmA5vmaf8XuDCP2xJYDOwMrEpqSvlnzXqOz/37kL5sVwfeDOwIrJzXdz9wVM36AvgZMAaYCLwIXJfXvzZwH3BIk+3QNNaaZW/ex3Z8ZXwu2zzg8JrtcgEpsa0O7J3X9cZclq8At+Z5xwHPAfsBqwCfA5YAH6vfN8BawFzgC8Do3L9Dzfa7qC7GK4Hv5zheA9wBfCKPOwJ4ANgQWBe4oX5/NijzbODIvF/+CaxXM+484Ou5+1WkxD6z0TFG+lL9JPAP4DVN1vXK9E22+Vzg7bl7HWC73L0L8HjdsXwHsH4u5/3AEXnc7nm/TQTWAC7qa78DN9bsl8Ee6xfl7k8AP8/rH5W37Zj69TXZBt/N02yQ590JWK1B3LuQjqmTgNVIx+RnSV/iE/Kw7wMXD+Kz2texdjHwn3na0cDOefh7gOmkX4AifT7GNziedgUWANvlWL8D3Fy3Ta7Oy9kI6AF2H3SuLDMRt/uVPwiLgYU1r+dpnvhvBk4AxtUtp5vlE/91wJE1/a/PB8jKwP/pPbDyuDVIH/zag+nmfmI/Criyboe/raZ/OnBMTf+pwLeaLKtprPUfsCbzB7AIeBZ4CPh6PrB7t8umNdP+Cji8pn+lvM03Jteia8YJeJzGif9A4I9N4jmemsQPrEf6Ily9ZtiBwA25+3pyAsz9767fn3XL3zlvn3G5/wHgczXjzwP+no+nl4FHgK1rxh9KSj4L83JeAPbvY/u+Uu66bd6b9P5CSppj6qbZheUT/0dq+k8Gzs7dPwL+X824zfva7yyb+Ad7rPcm/sOAW2u3VaP11W+DfAy9QGpu7O8zv0te/+iaYfcD76zpHz/A+GsTb3/H2gXAVGBCXVy7An8mVe5Wqht3HksT/zmkZsTecWvmWLtrtsnONeMvA47tb7v091oRm3r2iYixvS9SLa6Zw4HXAQ/kn6F9nZBbH3i0pv9R0oG0Xh73WO+IiHgeeLpu/sdqeyS9TtLVkubln5TfJNWQaz1V0/1Cg/41W4i1qO0iYp2I2CwivhKpCaNRWTYGzsg/QxcCz5AS/AYsv12ibt5aG5K+ZIrYmPQLYm7Ner9Pqo1Rv16W3RaNHAL8JiIW5P7/YfnmnlPy8dRN2vb1J1n/kMevA1xFaspo1QeB9wGP5maEt/Yx7bya7udZekzUb4Nm272RwR7rvS4EriGd73hS0smSVimw/nGk2nPR46EnIv5e078xcGXNsXE/8NIA4q8/vvs61r5EOt7vULpi8LC83OuBM0m/XOZLmippTIPYl9nWEbE4x7NBzTTN9nHLVsTEX1hEPBgRB5J24knA5ZJeRfqWrfck6SDotRGplvcU6af5hN4RuS3y1fWrq+s/i1Sz3CIixpDasQffdtd/rEOhtiyPkX72jq15rR4Rt5K2y4a9E+a2yQ1p7DFS00J/6+ud9kVSDb13nWMiYmIev8x6SeVvKO+r/YF/zV/C80hNUttI2ma5QCL+QmpKOKNRm3P+4P4HcLCaXyr7N1JNszeG19Yt486I2Jt0XP6UVMsbqGWOSZpv90YGe6wDEBH/jIgTImJLUlPNnqRfgdD4M9ZrAekX1mYF4210fLy37pgcHRFPFIy//vhueqxFxLyI+HhErE/6lfa93vMUEfHtiHgzqXnpdcAXG8S+zLbO+efVpAsISlPpxC/pI5K6cm12YR78Mqkd7WWWTUQXA5+TtImkNUk19EsjYglwObCXpJ2UTrgeT/9JfC1Sc8pipUsD/2OoytVPrEPtbODLkibCKyddP5TH/QKYKGlfpasjPgO8tslyrgbGSzpK6ST0WpJ2yOOeArolrQQQEXOB3wCnShqTT+ZtJulf8/SXAZ+RNEHSOsCxfcS/D6k2uCXphO0kUnvs71iapJYREdeSPrBTmox/BvghqVmhkZmk7TJJ6WTs8b0jJK2q9P+GtSPin6Rj5OUmy+nLZcC/S3qjpDWAgVw3PiTHuqR3SHpTPhG6iNSE0VuWp2jyRZ8/jz8CTpO0vtLJ+req7jLbPpwNfEP5IgNJXZL2zuMG9Fnt71iT9CFJvV8kz5K+NF6W9BZJO+RfOH8jfZE12o8Xk/bTpFy+bwK3R8ScgmVtSaUTP+kE2L1KV7qcARwQES/kn3/fAH6ff97tSDoQLySdF3iEtCM/DRAR9+buS0g1isXAfFJNoZmjSZdJPgf8ALh0CMvVNNahFhFXkn4tXZKbrO4B3pvHLQA+BJxI+vm6BfD7Jst5DtgN2Iv00/ZB4B159E/y+9OS7srdHyWdnLuP9IG7nNSWC2l7XkNKsHeRTk42cwhwbkT8Jdfe5kXEPNLP9IPU/HK+/wa+1Ecy+hbwPklbNyjrn4GvAr/N5bylbpKDgTl5ex4BHNRH/A1FxK+Ab5NObM8mneyEvo/JXkN1rL+WtF8WkZpbbsrLhfR520/pSrBvN5j3aOBu4E5S8+FJFM9XZ5Ca234j6TlS2XdoIf5efR1rbwFuzznkKuCzkf6LMIZ0HD5Lasp5mnTMLCMifkv6Ur4ix7MZcEDBcrZM+YSBDaFcS1pIasZ5pNPxmEl6I+lLebWh/OU30o/1kR5/q6pe4x8ykvaStEZuozuFVFuZ09morMqUbs2xWm7uOgn4+VAk/ZF+rI/0+IeCE//Q2ZvU7vskqUnjgPDPKeusT5CaMR4inccYqvNII/1YH+nxD5qbeszMKsY1fjOzihkRNyAaN25cdHd3dzoMM7MRZfr06Qsioqt++IhI/N3d3UybNq3TYZiZjSiSGv5r3U09ZmYV48RvZlYxTvxmZhXjxG9mVjFO/GZmFePEb2ZWMU78ZmYV48RvZlYxTvxmZhUzIv65a2Y2knQf+4shW9acE/cYsmX1co3fzKxinPjNzCrGid/MrGKc+M3MKsaJ38ysYkpP/JJGSfqjpKtz/yaSbpc0W9KlklYtOwYzM1uqHTX+zwL31/SfBJweEZsDzwKHtyEGMzPLSk38kiYAewA/zP0CdgUuz5OcD+xTZgxmZrassmv83wK+BLyc+18NLIyIJbn/cWCDkmMwM7MapSV+SXsC8yNieovzT5E0TdK0np6eIY7OzKy6yqzxvw14v6Q5wCWkJp4zgLGSem8VMQF4otHMETE1IiZHxOSuruUeEm9mZi0qLfFHxJcjYkJEdAMHANdHxEHADcB+ebJDgJ+VFYOZmS2vE9fxHwN8XtJsUpv/OR2Iwcysstpyd86IuBG4MXc/DGzfjvWamdny/M9dM7OKceI3M6sYJ34zs4px4jczqxgnfjOzinHiNzOrGCd+M7OKceI3M6sYJ34zs4px4jczqxgnfjOzinHiNzOrGCd+M7OKceI3M6uYttyW2cxsOOs+9hedDqGtXOM3M6uYMh+2PlrSHZJmSrpX0gl5+HmSHpE0I78mlRWDmZktr8ymnheBXSNisaRVgFsk/SqP+2JEXF7ius3MrInSEn9EBLA4966SX1HW+szMrJhS2/gljZI0A5gPXBsRt+dR35A0S9LpklZrMu8USdMkTevp6SkzTDOzSik18UfESxExCZgAbC9pK+DLwBuAtwDrAsc0mXdqREyOiMldXV1lhmlmViltuaonIhYCNwC7R8TcSF4EzgW2b0cMZmaWlHlVT5eksbl7dWA34AFJ4/MwAfsA95QVg5mZLa/Mq3rGA+dLGkX6grksIq6WdL2kLkDADOCIEmMwM7M6ZV7VMwvYtsHwXctap5mZ9c//3DUzqxgnfjOzinHiNzOrGCd+M7OKceI3M6sYJ34zs4px4jczqxgnfjOzinHiNzOrGCd+M7OKceI3M6sYJ34zs4px4jczqxgnfjOziinzfvxmZqXpPvYXnQ5hxHKN38ysYsp89OJoSXdIminpXkkn5OGbSLpd0mxJl0patawYzMxseQNK/JLWkbR1wclfBHaNiG2AScDuknYETgJOj4jNgWeBwwcSg5mZDU6/iV/SjZLGSFoXuAv4gaTT+psvksW5d5X8CmBX4PI8/HzSA9fNzKxNitT4146IRcC+wAURsQPwriILlzRK0gxgPnAt8BCwMCKW5EkeBzZoMu8USdMkTevp6SmyOjMzK6BI4l9Z0nhgf+DqgSw8Il6KiEnABGB74A0DmHdqREyOiMldXV0DWa2ZmfWhSOI/AbgGmB0Rd0raFHhwICuJiIXADcBbgbGSei8jnQA8MZBlmZnZ4BRJ/HMjYuuIOBIgIh4G+m3jl9QlaWzuXh3YDbif9AWwX57sEOBnrQRuZmatKZL4v1NwWL3xwA2SZgF3AtdGxNXAMcDnJc0GXg2cUzRYMzMbvKb/3JX0VmAnoEvS52tGjQFG9bfgiJgFbNtg+MOk9n4zM+uAvm7ZsCqwZp5mrZrhi1jaVGNmZiNM08QfETcBN0k6LyIebWNMZmZWoiI3aVtN0lSgu3b6iNi1rKDMzKw8RRL/T4CzgR8CL5UbjpmZla1I4l8SEWeVHomZmbVFkcs5fy7pSEnjJa3b+yo9MjMzK0WRGv8h+f2LNcMC2HTowzEzs7L1m/gjYpN2BGJmZu1R5LbMa0j6Sr6yB0lbSNqz/NDMzKwMRdr4zwX+QfoXL6Sbqn29tIjMzKxURRL/ZhFxMvBPgIh4HlCpUZmZWWmKJP5/5LtrBoCkzUiPVTQzsxGoyFU9/xf4NbChpB8DbwMOLTMoMzMrT5Greq6VdBewI6mJ57MRsaD0yMzMrBRNm3okvSG/bwdsDMwFngQ2ysPMzGwE6qvG/3lgCnBqg3EB9HmTNkkbAhcA6+Xpp0bEGZKOBz4O9D5B/biI+OUA4zYzsxb1dVvmKbnzvRHx99pxkkYXWPYS4AsRcZektYDpkq7N406PiFNaitjMzAalyFU9txYctoyImBsRd+Xu50jP291gYOGZmdlQ6+vRi68lJerVJW3L0mv3xwBrDGQlkrpJj2G8nXRV0KckfRSYRvpV8OyAIzczs5b01cb/HtJlmxOA02qGPwccV3QFktYErgCOiohFks4CvkZq9/8a6RzCYQ3mm0I6x8BGG21UdHVmZtaPvtr4zwfOl/TBiLiilYVLWoWU9H8cEf+bl/tUzfgfAFc3Wf9UYCrA5MmTo5X1m5nZ8opcx3+FpD2AicDomuFf7Ws+SQLOAe6PiNNqho+PiLm59wPAPa0EbmZmrek38Us6m9Sm/w7S4xf3A+4osOy3AQcDd0uakYcdBxwoaRKpqWcO8ImBh21mZq0qcsuGnSJia0mzIuIESacCv+pvpoi4hcY3c/M1+2ZmHVTkcs4X8vvzktYn3aVzfHkhmZlZmYrU+K+WNBb4b+AuUhPND0qNyszMSlPk5O7XcucVkq4GRkfEX8sNy8zMylLk5O5o4EhgZ1Jt/xZJZ9XfxsHMzEaGIk09F5D+tPWd3P9h4ELgQ2UFZWZm5SmS+LeKiC1r+m+QdF9ZAZmZWbmKXNVzl6Qde3sk7UC6x46ZmY1ARWr8bwZulfSX3L8R8CdJdwMREVuXFp2ZmQ25Iol/99KjMDOztilyOeej7QjEzMzao0gbv5mZrUD6etj6au0MxMzM2qOvGv9tAJIubFMsZmbWBn218a8q6cPATpL2rR/Z+2AVMzMbWfpK/EcABwFjgb3qxgXgxG9mNgL19ejFW0j35ZkWEee0MSYzMytRkat6LpT0GUmX59en87N0+yRpQ0k3SLpP0r2SPpuHryvpWkkP5vd1Bl0KMzMrrEji/x7p37vfy6/tgLMKzLcE+EK+z8+OwCclbQkcC1wXEVsA1+V+MzNrkyL/3H1LRGxT03+9pJn9zZQfqD43dz8n6X5gA2BvYJc82fnAjcAxA4jZzMwGoUiN/yVJm/X2SNoUeGkgK5HUDWwL3A6sl78UAOYB6zWZZ4qkaZKm9fT0DGR1ZmbWhyI1/i+SbsX8MOnh6RsD/150BZLWBK4AjoqIRdLS569HREiKRvNFxFRgKsDkyZMbTmNmZgNX5F4910naAnh9HvSniHixyMLzSeArgB/XXPf/lKTxETFX0nhgfiuBm5lZawrdqyciXoyIWflVNOkLOAe4PyJOqxl1FXBI7j4E+NlAAjYzs8Ep0tTTqrcBBwN3S5qRhx0HnAhcJulw4FFg/xJjMDOzOqUl/vwHMDUZ/c6y1mtmZn3rt6lH0nVFhpmZ2cjQtMYvaTSwBjAu/7u2t/Y+hnQ9vpmZjUB9NfV8AjgKWB+YztLEvwg4s+S4zMysJH3dpO0M4AxJn46I77QxJjMzK1GR6/i/I2knoLt2+oi4oMS4zMysJP0m/vwErs2AGSy9VUMATvxmZiNQkcs5JwNbRoRvm2BmtgIo8s/de4DXlh2ImZm1R5Ea/zjgPkl3AK/criEi3l9aVGZmVpoiif/4soMwM7P2KXJVz03tCMTMzNqjyFU9z5Gu4gFYFVgF+FtEjCkzMDMzK0eRGv9avd35Vst7k56ha2ZmI1Ch+/H3iuSnwHtKisfMzEpWpKln35relUjX9f+9tIjMzKxURa7q2aumewkwh9TcY2ZmI1CRNv7CD1avJelHwJ7A/IjYKg87Hvg40JMnOy4iftnK8s3MrDVFHsQyQdKVkubn1xWSJhRY9nnA7g2Gnx4Rk/LLSd/MrM2KnNw9l/SA9PXz6+d5WJ8i4mbgmUFFZ2ZmQ65I4u+KiHMjYkl+nQd0DWKdn5I0S9KP8pO9GpI0RdI0SdN6enqaTWZmZgNUJPE/Lekjkkbl10eAp1tc31mkWzxPAuYCpzabMCKmRsTkiJjc1TWY7xkzM6tVJPEfBuwPzCMl6/2Alk74RsRTEfFSRLwM/ADYvpXlmJlZ64pc1fMoMCR34pQ0PiLm5t4PkG75bGZmbVTkqp7zJY2t6V8nX6rZ33wXA7cBr5f0uKTDgZMl3S1pFvAO4HODiN3MzFpQ5A9cW0fEwt6eiHhW0rb9zRQRBzYYfM5AgjMzs6FXpI1/pdqrbyStS7EvDDMzG4aKJPBTgdsk/ST3fwj4RnkhmZlZmYqc3L1A0jRg1zxo34i4r9ywzMysLIWabHKid7I3M1sBDOh+/GZmNvI58ZuZVYwTv5lZxTjxm5lVjBO/mVnFOPGbmVWME7+ZWcU48ZuZVYwTv5lZxTjxm5lVjBO/mVnFlJb488PU50u6p2bYupKulfRgfm/6sHUzMytHmTX+84Dd64YdC1wXEVsA1+V+MzNro9ISf0TcDDxTN3hv4PzcfT6wT1nrNzOzxtrdxr9ezcPW5wHrNZtQ0hRJ0yRN6+npaU90ZmYV0LGTuxERQPQxfmpETI6IyV1dXW2MzMxsxdbuxP+UpPEA+X1+m9dvZlZ57U78VwGH5O5DgJ+1ef1mZpVX5uWcFwO3Aa+X9Likw4ETgd0kPQi8K/ebmVkbFXrmbisi4sAmo95Z1jptxdV97C+GbFlzTtxjyJZlNhL5n7tmZhXjxG9mVjFO/GZmFePEb2ZWMaWd3DWrAp90tpHINX4zs4px4jczqxgnfjOzinHiNzOrGCd+M7OKceI3M6sYJ34zs4px4jczqxgnfjOzivE/d82sbYbyn87WOtf4zcwqpiM1fklzgOeAl4AlETG5E3GYmVVRJ5t63hERCzq4fjOzSnJTj5lZxXQq8QfwG0nTJU1pNIGkKZKmSZrW09PT5vDMzFZcnUr8O0fEdsB7gU9K+pf6CSJiakRMjojJXV1d7Y/QzGwF1ZHEHxFP5Pf5wJXA9p2Iw8ysitqe+CW9StJavd3Au4F72h2HmVlVdeKqnvWAKyX1rv9/IuLXHYjDzKyS2p74I+JhYJt2r9es13D99+hwfX7vcN1e1jpfzmlmVjFO/GZmFePEb2ZWMU78ZmYV49syW2l8UrBzvO2tL67xm5lVjBO/mVnFOPGbmVWME7+ZWcU48ZuZVYwTv5lZxTjxm5lVjBO/mVnFOPGbmVXMCv/P3eF6q9vhyv/4NFvxucZvZlYxHUn8knaX9CdJsyUd24kYzMyqqhPP3B0FfBd4L7AlcKCkLdsdh5lZVXWixr89MDsiHo6IfwCXAHt3IA4zs0rqxMndDYDHavofB3aon0jSFGBK7l0s6U9NljcOWDCkETahk9qxln61rbzDRNXKC9Urs8vbh0HmnY0bDRy2V/VExFRgan/TSZoWEZPbENKw4PKu+KpWZpe3/TrR1PMEsGFN/4Q8zMzM2qATif9OYAtJm0haFTgAuKoDcZiZVVLbm3oiYomkTwHXAKOAH0XEvYNYZL/NQSsYl3fFV7Uyu7xtpojodAxmZtZG/ueumVnFOPGbmVXMsE78/d3aQdJqki7N42+X1J2H7yZpuqS78/uu7Y69FYMo7/aSZuTXTEkfaHfsrWi1vDXjN5K0WNLR7Yp5MAaxf7slvVCzj89ud+ytGMz+lbS1pNsk3Zs/x6PbGXsrBrF/D6rZtzMkvSxpUqnBRsSwfJFO/D4EbAqsCswEtqyb5kjg7Nx9AHBp7t4WWD93bwU80enylFzeNYCVc/d4YH5v/3B9Daa8NeMvB34CHN3p8pS8f7uBezpdhjaWd2VgFrBN7n81MKrTZSqrvHXTvAl4qOx4h3ONv8itHfYGzs/dlwPvlKSI+GNEPJmH3wusLmm1tkTdusGU9/mIWJKHjwZGwhn7lssLIGkf4BHS/h0JBlXeEWgw5X03MCsiZgJExNMR8VKb4m7VUO3fA/O8pRrOib/RrR02aDZNTnx/JdUOan0QuCsiXiwpzqEyqPJK2kHSvcDdwBE1XwTDVcvllbQmcAxwQhviHCqDPZ43kfRHSTdJenvZwQ6BwZT3dUBIukbSXZK+1IZ4B2uo8tW/AReXFOMrhu0tG4aCpInASaQaxAotIm4HJkp6I3C+pF9FxN87HVdJjgdOj4jFI7dCPCBzgY0i4mlJbwZ+KmliRCzqdGAlWRnYGXgL8DxwnaTpEXFdZ8Mql6QdgOcj4p6y1zWca/xFbu3wyjSSVgbWBp7O/ROAK4GPRsRDpUc7eIMqb6+IuB9YTDq3MZwNprw7ACdLmgMcBRyX/xQ4nLVc3oh4MSKeBoiI6aS25NeVHvHgDGb/Pg7cHBELIuJ54JfAdqVHPDhD8fk9gDbU9oFhfXJ3ZeBhYBOWniyZWDfNJ1n2ZMlluXtsnn7fTpejTeXdhKUndzcGngTGdbpMZZW3bprjGRkndwezf7vIJzdJJw+fANbtdJlKLO86wF3kixaA3wJ7dLpMZZU396+U9+umbYm30xusn435PuDPpBrOf+ZhXwXen7tHk67qmA3c0bvRgK8AfwNm1Lxe0+nylFjeg0knOWfkD8w+nS5LmeWtW8aISPyD3L8frNu/e3W6LGXvX+Ajucz3ACd3uixtKO8uwB/aFatv2WBmVjHDuY3fzMxK4MRvZlYxTvxmZhXjxG9mVjFO/GZmFePEb9YHSftICklvyP21d8q8T9IFklbJ43aR9Nc87gFJp3Q2erPGnPjN+nYgcEt+7/VQREwi3UlxArB/zbjf5XHbAntKelvbIjUryInfrIl8M7idgcNJ/7RcRqQ7Rt7B8jfjIiJeIP3harlxZp3mxG/W3N7AryPiz0DvDdJekR8OsgPw6/oZJa0DbAHc3I5AzQbCid+sudp7o1/C0uaezSTNAJ4C5kbErJp53i5pJum+K9dExLy2RWtW0Ap9W2azVklaF9gVeJOkID1hKYDvktv4JY0Dfi/p/RFxVZ71dxGxp6RNgD9IuiwiZnSkEGZNuMZv1th+wIURsXFEdEfEhqQnfr1y692IWAAcC3y5fuaIeAQ4kfTAGLNhxYnfrLEDSc9zqOs6AF8AAABISURBVHUFyyf5nwJrNHkq1tnAv9Q/JN6s03x3TjOzinGN38ysYpz4zcwqxonfzKxinPjNzCrGid/MrGKc+M3MKsaJ38ysYv4/VvcEkaUDhyYAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "rTI2xcriG4vi"
      },
      "source": [
        "Note that although it predicts different absolute risk reduction, it never predicts that the treatment will adversely impact risk. This is because the odds ratio of treatment is less than 1, so the model always predicts a decrease in the baseline risk. Run the next cell to compute the c-statistic-for-benefit on the test data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:31:53.895737Z",
          "start_time": "2020-04-04T15:31:53.880107Z"
        },
        "colab_type": "code",
        "id": "HTgU2BLbGX1B",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "44bd6144-31ca-4a02-e4ce-8f11f139f46d"
      },
      "source": [
        "tmp_cstat_test = c_statistic(rr_lr, y_test, X_test.TRTMT)\n",
        "print(f\"Logistic Regression evaluated by C-for-Benefit: {tmp_cstat_test:.4f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Logistic Regression evaluated by C-for-Benefit: 0.5412\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pb53C7vTTmsN",
        "colab_type": "text"
      },
      "source": [
        "##### Expected Output\n",
        "```CPP\n",
        "Logistic Regression evaluated by C-for-Benefit: 0.5412\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "o6YQq4LLZdBj"
      },
      "source": [
        "Recall that a c statistic ranges from 0 to 1, and is closer to when the model being evaluated is doing a good job with its predictions.\n",
        "\n",
        "You can see that the model is not doing a great job of predicting risk reduction, given a c-for-benefit of around 0.54."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g_n0dCNATmsR",
        "colab_type": "text"
      },
      "source": [
        "### Regular c-index\n",
        "Let's compare this with the regular C-index which you've applied in previous assignments. Note that the regular c-statistic does not look at pairs of pairs of patients, and just compares one patient to another when evaluating the model's performance.  So the regular c-index is evaluating the model's ability to predict overall patient risk, not necessarily measuring how well the model predicts benefit from treatment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:31:55.406270Z",
          "start_time": "2020-04-04T15:31:55.400272Z"
        },
        "colab_type": "code",
        "id": "JRtzA6qyJ5sn",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "4ada7ef3-b746-4ba1-c208-828cf6c8f674"
      },
      "source": [
        "from lifelines.utils import concordance_index\n",
        "tmp_regular_cindex = concordance_index(y_test, lr.predict_proba(X_test)[:, 1])\n",
        "print(f\"Logistic Regression evaluated by regular C-index: {tmp_regular_cindex:.4f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Logistic Regression evaluated by regular C-index: 0.7785\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qqe3gATmTmsa",
        "colab_type": "text"
      },
      "source": [
        "##### Expected output\n",
        "```CPP\n",
        "Logistic Regression evaluated by regular C-index: 0.7785\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qRYEhMCOLDjs"
      },
      "source": [
        "You can see that even though the model accurately predicts overall risk (regular c-index), it does not necessarily do a great job predicting benefit from treatment (c-for-benefit). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Z_4ogidoLqGd"
      },
      "source": [
        "You can also visually assess the discriminative ability of the model by checking if the people it thinks benefit the most from treatment empirically (actually) experience a benefit. \n",
        "\n",
        "Since you don't have counterfactual results from individuals, you'll need to aggregate patient information in some way. \n",
        "\n",
        "You can group patients by deciles (10 groups) of risk."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:32:02.277354Z",
          "start_time": "2020-04-04T15:32:02.107132Z"
        },
        "colab_type": "code",
        "id": "aP8ST7ycL-I6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 458
        },
        "outputId": "6c02ef30-8683-45b3-f3f1-dea8b39c4f79"
      },
      "source": [
        "def quantile_benefit(X, y, arr_hat):\n",
        "    df = X.copy(deep=True)\n",
        "    df.loc[:, 'y'] = y\n",
        "    df.loc[:, 'benefit'] = arr_hat\n",
        "    benefit_groups = pd.qcut(arr_hat, 10)\n",
        "    df.loc[:, 'benefit_groups'] = benefit_groups\n",
        "    empirical_benefit = df.loc[df.TRTMT == 0, :].groupby('benefit_groups').y.mean() - df.loc[df.TRTMT == 1].groupby('benefit_groups').y.mean()\n",
        "    avg_benefit = df.loc[df.TRTMT == 0, :].y.mean() - df.loc[df.TRTMT==1, :].y.mean()\n",
        "    return empirical_benefit, avg_benefit\n",
        "\n",
        "def plot_empirical_risk_reduction(emp_benefit, av_benefit, model):\n",
        "    plt.scatter(range(len(emp_benefit)), emp_benefit)\n",
        "    plt.xticks(range(len(emp_benefit)), range(1, len(emp_benefit) + 1))\n",
        "    plt.title(\"Empirical Risk Reduction vs. Predicted ({})\".format(model))\n",
        "    plt.ylabel(\"Empirical Risk Reduction\")\n",
        "    plt.xlabel(\"Predicted Risk Reduction Quantile\")\n",
        "    plt.plot(range(10), [av_benefit]*10, linestyle='--', label='average RR')\n",
        "    plt.legend(loc='lower right')\n",
        "    plt.show()\n",
        "\n",
        "emp_benefit, avg_benefit = quantile_benefit(X_test, y_test, rr_lr)\n",
        "plot_empirical_risk_reduction(emp_benefit, avg_benefit, \"Logistic Regression\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEWCAYAAACaBstRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3dd5wV5dn/8c/XFWWxIaJGUESjD4qGoGIvsYSAsWF7JMWoMZZEE00x0fxSTfLExCdFlNii0cQeHyVYEozdWAFBUZHYlcXYEEVcBPH6/TH3wuzu2bOzsGfPgf2+X6997Zm558xc0841c8/MPYoIzMzMilip2gGYmdnyw0nDzMwKc9IwM7PCnDTMzKwwJw0zMyvMScPMzArrdklD0nuSNi1TfoGkHy7jNPaUNHMZvv93SUcVGC4kbba001laku6W9JUKjPf7kv7Y2eNdHuXXbWdskwWnebSkf7UzzP2StqlgDLtLmrGU3y2036xIJH1B0m2dMJ71JU2XtGp7w9ZE0pD0oqTG9IPe9HdeJaYVEatHxPNlyk+MiJ9VYtpN0g/CvDSfDZJ+K6kuF8O+EXF5J05vYJpm07J9UdLpnTX+pYypVWKNiP+JiE5PRpWQ4v8oLc+5kmZIOqYS0yq6TVYqmefGfwAwNyKmpO6fSLqiM6cREfdFxKACsbSa9tLuN+3tj7UsIq6MiM90wnheA+4Cjm9v2JpIGskB6Qe96e/krg6gizeUT0bE6sCngCOAL3fBNHunaR4G/FDS8C6Y5opsVlqeawLfAy6WNLjlQJJW7vLIKuNE4C/VDqJCKr4/LgfbwZXACe0NVEtJo6R0yny/pN9JmiPpeUm7pP6vSHo9f0oq6bJ0Ov/PdAR4j6SNc+X50/7LJJ0v6VZJ84C9Ur+f54Y/SNJUSe9Kek7SyNT/mHQ6NzfF1O7CLiUingXuB4bmprn4iFHSZmke3pH0pqRr21hOu6XlsWeBaU4CnmwxzS+n+Xlb0oQWy2y4pKdTDOcBypU1O+LLndWsnLr7SPqTpFlp3OMkrQb8HeiXO/vpV2JcB0p6Mq33uyVtmSt7UdJ3JD2e4rpWUs8Sy2XV9P2tc/3WVXZmu56kvpJuTsPMlnSfpA7tF5EZB7wNDG6xzb4F/CTF8b+SXpb0WtpG63MxnSbp1bScmv1gFdkmJf0C2B04T7kzdUlbpH1htrKzof/OjWcdSePTeB4BPt7WPEpaBdgbuKfIMmln3W0raUrad/6a1t3PU1mzM1BJ31N29N90NrdP2ge/DxyR5vWxNGyzMy1Jx+X20ackbdte3G3sj/un5T1H0gOShnRkXtI8/Af4k6SVJJ2e1ttbkq6T1CcN31PSFan/HEkTJa2fyo5W9jszV9ILkr6Q6/+vXDy7pO+9k/7vkiu7W9LP0rY5V9JtkvrmZv9hYFPl9v22FlLV/4AXgU+3UXY08CFwDFAH/Bx4GRgLrAp8BpgLrJ6Gvyx175HKzwH+lRtfAJvlhn0H2JUsgfZM/X6eyndI5cNTeX9gi1S2H9lOJrKjk/eBbVPZnsDMMvObj2EL4FXgm7nyu4GvpM9XA/8vF99uLccDjAReAXZoY3oD07Arp+6dUrwHp+6DgGeBLYGVgR8AD6Syvml5Hgb0AL6Z1kdTfD8BrigzrVuAa4G10/c/1dYyyo8L+C9gXlr2PYDvphhXyW0zjwD9gD7AdODENub/UuAXue6TgH+kz78ELkjT6EH2w6sC2+zi+NO6ORhYCAxiyTb79bQ864HfAeNTrGsANwG/TN8fCbwGbA2sBlxF6+20yDZ5d9N6Sd2rpe3imBTHNsCbwOBUfg1wXRpua6CB3L7SYn63Aua1tb5a9G9z3aW/l4BTUtkhwILc/OWX66AUf7/ctvXxtqZN8/3m8DQ/25Pto5sBG3d0f0zL7HVgR7Lfn6PItr1VC87Lh8Cv0vD1adiHgA1TvwuBq9PwJ6Ttolea1nZkZ7GrAe8Cg9JwGwBb5X4f/5U+9yE7cDkyre/Ppe51csvnubR+6lP3WS2WxePAgWW3/c740V/Wv7QS3gPm5P6Oyy2UZ3LDfiKt5PVz/d4ChuZ2sGtyZasDi4CNSmwglwF/bhHLZbmVfiHwu4LzMA44peWGX2YjfZdsxwqyxLBqGxv/n4GLgA3bGM8ZacPdusz0BqZh5wCN6fP/kn4cyY76j80NvxJZUtkY+BLwUK5MwEwKJI20cX8ErF0iplbLiOZJ44fAdS1iagD2zG0zX8yV/xq4oI35/zTwXK77fuBL6fOZwN+atokObLN7pnmbA8wGpgKjc9vsyy2W2TzSD17qtzPwQvp8Kbmdl2ynbitptLlN0jppHAHc12KYC4Efk/0oLSQlnFT2P7SdNHYF/tPW+mrRv811R3Yw10AuMQP/onTS2IzsB/vTQI/2pk3z/WYCaX8ssC7b3B+B84GftRh+BtmBYpF5WQD0zJVPB/bJdW+Q1sPKZFViDwBDWkxvtbSdHQrUtyg7miVJ40jgkRblDwJH55bPD3JlXyMdPJXaN9r6q6XqqVER0Tv3d3Gu7LXc50ZYfOEm32/1XPcrTR8i4j2ynbpfG9N9pY3+ABuRZeZWJO0r6aF02j8H+CzZUXlR26aYjyA7ilmtjeG+S/aj80g63W9Z13oq2Q76RIFp9k3T/DbZBt0j9d8YOCedEjf9CIrsKLYfzZdnUH6Z5W0EzI6ItwsOn9ePLBk2TfejNN3+uWH+k/v8Ps23gby7gF6SdpQ0kKzq4cZUdjbZUfBt6fS/IzcIzErbap+IGBoR1+TK8stoXbKjx8m5ZfyP1B9aLGNy811Cm9tkCRsDOzZNM033C8DH0rRX7sB03yY7Qyqi3LrrBzSk7ahJye0psqqiU8kSxOuSrpHU1n7cUkeWE7S9P24MfLvFMtwozUeReXkjIubnujcGbsyNazrZQe36ZNeLJgDXpGrKX0vqERHzUlwnAq9KukXSFiXmodlyT16iY/vMGmQJqk21lDQ600ZNHyStTnbaNquNYaON/pBtAK3qeZXdlvZ/ZEfr60dEb+BWcnX9RUTmOrKjgR+1Mcx/IuK4iOhHdvr6BzW/zfZwYJSkUwpOc1FE/BaYT3akAdl8ntAiaddHxANkp+r55al8N9nRWa9c98dyn18B+kjqXSqUdkKdRbaDtZxuQzvfaz2hiEVk1TCfS383R8TcVDY3Ir4dEZsCBwLfkrRPR6dRarK5z2+SHdhslVu+a0V24RVaLGNgQJnxltwmS0yzadh7WqzX1SPiq8AbZFUnRaf7LNlq6F9mmCbl1t2rQP/Ur8lGtCEiroqI3dL4gqyqB9rffsotp7amVWp/fIWsajO/DHtFxNUF56XUOtm3xfh6RkRDRCyMiJ9GxGBgF2B/sjN9ImJCRAwnOzN5GriY1pot92QABfcZZdchNwMeKzfcipo0PqvswvAqwM/IqleKHh3nXQIcky6+rSSpf8rwq5DVR74BfChpX7JrK0vrLOA4SR9rWSDpcEkbps63yTbCj3KDzAL2AU6R9NUOTvO7yi4eXwCcIWmrNM21JB2ehrsF2ErSIWmj+gbNE8NUYA9JAyStRVZdBkBEvEpW9fUHSWtL6iFpj1T8GrBO+k4p1wH7pWXfg+zs6AOy0/elcRXZ0doX0mfSvO6v7GYDkV0rWETz5bvM0pH2xcDvJK2Xpttf0og0yHXA0ZIGS+pFVn3Ulra2SciWaf4ZpJuB/5J0ZFr2PSRtL2nLlEhvILtI30vZXV9HlZmHBcDtZNUyeSulC7hNf6tSft09SLaMT5a0sqSDyK7TtCJpkKS90zjnkyXepnXzGjBQbd+08EfgO5K2U2azdi/wLpHfHy8GTkxnqZK0mqT9JK3RkXnJuQD4RVMsym7KOCh93kvSJ5TdxfkuWbXVR8qeoThI2Q0kH5BV5ZfaRm8lW9+fT/EcAQwm2w6K2AF4MSLKnXHWVNK4Sc2f07ix/a+06SqyHW822cWkLy7NSCLiEbKLiL8j+0G5h+xi2lyyH8/ryH7IP092kXOpRMQ04F7gtBLF2wMPS3ovTeOUaPGcSUS8TJY4Tlfx+/RvSbEfFxE3kh3BXSPpXeAJYN807jfJzmbOIrt2tDlZvWfTtP9JdqH7cWAyrTfQI8k2/qfJ6qdPTd97mqzu+Pl0qt6s2iEiZpCtt3PJjtQPILste0HB+WsmIh4mOyvqR5bImmxO9mP4HtmPwB8i4i5Y/LDY95dmeiV8j+xo/aG0jG8nu9BLRPwd+D1wZxrmzjLzUXKbTMXnAIcpu0ttTNpOPwOMJju4+A9LLsoCnExWPfEfsusmf2pnHi4kW595nyP7MW/6e67cukvr7xDgWLJqkC+SbTMflJjeqmTb3ZspxvVYclDy1/T/LUmPtvxiRPwV+AXZb8FcsmuOfdqZv6bvLt4fI7vT8DjgPLL95Vmy6wh0cF6anEO2H98maS7ZRfEdU9nHgOvJEsZ0snX7F7Lf6W+RrcPZZIm71QFiRLxFdnbybbJ99bvA/mkfLuILZEmtrKYLoSsMSZeRXUj7QbVjMVvRSLofODnSA36dNM6HyW5iaC9p1bzldV7SGfA9wDYtrsG0UusPm5hZDYmIXZd1HJI+RXYH0ptkR7dDyG4MWO6sKPMSEa+T3XLfLicNM+tqg1jyfMjzwGHp+tfyaEWal0JWuOopMzOrnFq6EG5mZjVuuaue6tu3bwwcOLDaYZiZLVcmT578ZkSs2/6Q5S13SWPgwIFMmjSp2mGYmS1XJJV9/qIoV0+ZmVlhThpmZlaYk4aZmRXmpGFmZoU5aZiZWWHL3d1TZlY546Y0cPaEGcya00i/3vWcNmIQo7Yp0hq6dRdOGmYGZAnjjBum0bhwEQANcxo544ZpAE4ctpirp8wMgLMnzFicMJo0LlzE2RNmVCkiq0VOGmYGwKw5jR3qb92Tk4aZAdCvd32H+lv35KRhZgCcNmIQ9T3qmvWr71HHaSMGVSkiq0W+EG5mwJKL3b57yspx0jCzxUZt099Jwspy9ZSZmRXmMw0zszb4YcfWnDTMzErww46luXrKzKwEP+xYmpOGmVkJftixNCcNM7MS/LBjaU4aZmYl+GHH0nwh3MysBD/sWJqThplZG/ywY2uunjIzs8KcNMzMrDAnDTMzK8xJw8zMCnPSMDOzwpw0zMysMCcNMzMrzEnDzMwKc9IwM7PCnDTMzKywiiYNSSMlzZD0rKTTywx3qKSQNKyS8ZiZ2bKpWNtTkuqAscBwYCYwUdL4iHiqxXBrAKcAD1cqFrNy/EpPs+IqeaaxA/BsRDwfEQuAa4CDSgz3M+BXwPwKxmJWUtMrPRvmNBIseaXnuCkN1Q7NrCZVMmn0B17Jdc9M/RaTtC2wUUTcUsE4zNrkV3qadUzVLoRLWgn4LfDtAsMeL2mSpElvvPFG5YOzbsOv9DTrmEomjQZgo1z3hqlfkzWArYG7Jb0I7ASML3UxPCIuiohhETFs3XXXrWDI1t34lZ5mHVPJpDER2FzSJpJWAUYD45sKI+KdiOgbEQMjYiDwEHBgREyqYExmzfiVnmYdU7G7pyLiQ0knAxOAOuDSiHhS0pnApIgYX34MZpXnV3qadYwiotoxdMiwYcNi0iSfjJiZdYSkyRGxzM/C+YlwMzMrzEnDzMwKc9IwM7PCnDTMzKwwJw0zMyvMScPMzApz0jAzs8LafbhP0qrAocDA/PARcWblwjIzs1pU5InwvwHvAJOBDyobjpmZ1bIiSWPDiBhZ8UjMzKzmFbmm8YCkT1Q8EjMzq3lFzjR2A46W9AJZ9ZSAiIghFY3MzMxqTpGksW/FozAzs+VCu9VTEfES0Bs4IP31Tv3MzKybaTdpSDoFuBJYL/1dIenrlQ7MzMxqT5HqqWOBHSNiHoCkXwEPAudWMjAzM6s9Re6eErAo170o9TMzs26myJnGn4CHJd2YukcBl1QuJDMzq1XtJo2I+K2ku8luvQU4JiKmVDQqMzOrSW0mDUlrRsS7kvoAL6a/prI+ETG78uGZmVktKXemcRWwP1mbU5Hrr9S9aQXjMjOzGtRm0oiI/dP/TbouHDMzq2VFntO4o0g/MzNb8ZW7ptET6AX0lbQ2S26zXRPo3wWxmZlZjSl3TeME4FSgH9l1jaak8S5wXoXjMjOzGlTumsY5wDmSvh4RfvrbzMwKPRH+kaTeTR2S1pb0tQrGZGZmNapI0jguIuY0dUTE28BxlQvJzMxqVZGkUSdpcVtTkuqAVSoXkpmZ1aoibU/9A7hW0oWp+4TUz8zMupkiSeN7ZIniq6n7n8AfKxaRmZnVrCINFn4EnJ/+zMysG2s3aUh6geZtTwEQEW57ysysmylSPTUs97kncDjQpzLhmJlZLWv37qmIeCv31xARvwf2KzJySSMlzZD0rKTTS5SfKGmapKmS/iVp8FLMg5mtYMZNaWDXs+5kk9NvYdez7mTclIZqh2RJkeqpbXOdK5GdeRT5Xh0wFhgOzAQmShofEU/lBrsqIi5Iwx8I/BYYWTx8M1vRjJvSwBk3TKNxYfaW6YY5jZxxwzQARm3jZu+qrUj11G9ynz8kexnTfxf43g7AsxHxPICka4CDgMVJIyLezQ2/GiWunZhZ93L2hBmLE0aTxoWLOHvCDCeNGlDk7qm9lnLc/YFXct0zgR1bDiTpJOBbZA8M7l1qRJKOB44HGDBgwFKGY2bLg1lzGjvU37pWuabRv1XuixHx284IICLGAmMlfR74AXBUiWEuAi4CGDZsmM9GzFZg/XrX01AiQfTrXV+FaKylchfC10h/w8ge7Ouf/k4Eti3zvSYNwEa57g1Tv7ZcA4wqMF4zW4GdNmIQ9T3qmvWr71HHaSMGVSkiyyvXNPpPASTdC2wbEXNT90+AWwqMeyKwuaRNyJLFaODz+QEkbR4Rz6TO/YBnMLNurem6xdkTZjBrTiP9etdz2ohBvp5RI4pcCF8fWJDrXpD6lRURH0o6GZgA1AGXRsSTks4EJkXEeOBkSZ8GFgJvU6Jqysy6n1Hb9HeSqFFFksafgUck3Zi6RwGXFxl5RNwK3Nqi349yn08pGKeZmdWAIndP/ULS34HdU69jImJKZcMyM7NaVOR9GgC9gHfTK2BnpusUZmbWzbSbNCT9mKx59DNSrx7AFZUMyszMalORM42DgQOBeQARMYvsVlwzM+tmiiSNBRERpCY+JK1W2ZDMzKxWFbl76rr0qtfeko4DjmU5e3PfuCkNvufbzKwTFLl76n8lDQfeBQYBP4yIf1Y8sk7iFjPNzDpP2eopSXWS+kbEPyPiNOD7wCaSpndNeMuuXIuZZmbWMW0mDUmjgdnA45LukfQZ4HlgX+ALXRTfMnOLmWZmnadc9dQPgO0i4tn0IqYHgcMi4qauCa1zuMVMM7POU656akFEPAsQEY8CzyxvCQPcYqaZWWcqd6axXot3avTOd3fW+zQqzS1mmpl1nnJJ42KaP8TXsnu54RYzzcw6R7vv0zAzM2tStMFCMzMzJw0zMyuuSCu3q5bo16cy4ZiZWS0rcqZxg6QeTR2SNgCWm2ZEzMys8xRJGuPIGi2skzSQ7J3fZ5T9hpmZrZCKNFh4saRVyJLHQOCEiHig0oGZmVntaTNptHiwT8AAYCqwk6SdlpeH+8zMrPOUO9No+SDfDW30NzOzbqJDD/dJWglYPSLerWhUZmZWk4rccnuVpDXTa16fAJ6SdFrlQzMzs1pT5O6pwenMYhTwd2AT4MiKRmVmZjWpSNLokZ7TGAWMj4iFQFQ2LDMzq0VFksaFwIvAasC9kjYme1+4mZl1M0We0xgDjMn1eknSXpULyczMalW55zS+GBFXtHheI8/PaZiZdTPlzjRWS//9XIaZmQHln9O4MP0v9bzGaq2/YWZmK7qy1zQk9Qc2AB6PiAWS1gNOBY4G+lU+vM51xIUPtuq3/5ANOHLngTQuWMTRf3qkVflh223I4cM2Yva8BXz1ismtyr+408Yc8Ml+zJrTyDevndqq/LjdN+XTg9fnuTfe4/s3TGtV/vW9N2e3zfvy5Kx3OPOmp1qVf3fkILbbuA+TX5rNr/8xo1X5jw4YzFb91uJfz7zJuXc+06r8fw75BB9fd3Vuf+o1Lr7v+VblvztiKP1613PTY7O44qGXWpWf/8Xt6LPaKvx10itcP3lmq/LLjtmB+lXq+MuDL3Lz46+2Kr/2hJ0BuOje57hj+uvNynr2qOPyL+8AwJg7nuH+Z99sVr52r1W44MjtAPjVP57m0Zfebla+wVo9+f3obQD46U1P8tSs5vdnbLruavzykCEAnHHD4zz/xrxm5YP7rcmPD9gKgFOvmcKr78xvVr7txmvzvZFbAHDiXybz9vsLmpXvullfvrHP5gAcdekjzF+4qFn5Pluux/F7fBzwtudtr3O2vaZ5qqY2756SdCpZW1PnAg9J+gowHagHtuua8MzMrJYoovQjF5KeAnaLiNmSBgD/BnaNiNaHPF1o2LBhMWnSpGqGYGa23JE0OSKGLet4yj2nMT8iZgNExMvAjI4mDEkjJc2Q9Kyk00uUf0vSU5Iel3RHegbEzMxqVLlrGhtKyj+fsUG+OyK+UW7EkuqAscBwYCYwUdL4iMhXnk4BhkXE+5K+CvwaOKKjM2FmZl2jXNJo2ShhR6uldgCejYjnASRdAxwELE4aEXFXbviHgC92cBpmZtaFyt1ye/kyjrs/8EqueyawY5nhjyVrELEVSccDxwMMGDBgGcMyM7OlVaTtqYqT9EVgGHB2qfKIuCgihkXEsHXXXbdrgzMzs8XabXtqGTQAG+W6N0z9mpH0aeD/AZ+KiA8qGI+ZmS2jSp5pTAQ2l7SJpFWA0cD4/ACStiFrRffAiHi9xDjMzKyGlGuw8FzKvDejvbunIuJDSScDE4A64NKIeFLSmcCkiBhPVh21OvBXSQAvR8SBHZ8NMzPrCuWqp5b5CbqIuBW4tUW/H+U+f3pZp2FmZl2nkndPmZnZCqbdC+GS1gW+BwwGejb1j4i9KxiXmZnVoCIXwq8ka6hwE+CnZK9+nVjBmMzMrEYVSRrrRMQlwMKIuCcivgz4LMPMrBsq8pzGwvT/VUn7AbOAPpULyczMalWRpPFzSWsB3yZ7t8aawDcrGpWZmdWkdpNGRNycPr4D7FXZcMzMrJa1e01D0uWSeue615Z0aWXDMjOzWlTkQviQiJjT1BERbwPbVC4kMzOrVUWSxkqS1m7qkNSHyjZ0aGZmNarIj/9vgAcl/RUQcBjwi4pGZWZmNanIhfA/S5rEkmczDmnxylYzM+smyrVyu2ZEvJuqo/4DXJUr6xMRs7siQDMzqx3lzjSuAvYnezd4vol0pe5NKxiXmZnVoHKt3O6v7CUXn4qIl7swJjMzyxk3pYGzJ8xg1pxG+vWu57QRgxi1Tf+qxFL27qmICOCWLorFzMxaGDelgTNumEbDnEYCaJjTyBk3TGPclFZvz+4SRW65fVTS9hWPxMzMWjl7wgwaFy5q1q9x4SLOnjCjKvEUueV2R+ALkl4C5pGuaUTEkIpGZmZmzJrT2KH+lVYkaYyoeBRmZlZSv971NJRIEP1611chmjLVU5LWTB/ntvFnZmYVdtqIQdT3qGvWr75HHaeNGFSVeDpyy61yZb7l1sysCzTdJVUrd08pu0Fq+TFs2LCYNGlStcMwM1uuSJocEcOWdTyFGh6UdAiwG9kZxn0RMW5ZJ2xmZsufIu/T+ANwIjANeAI4UdLYSgdmZma1p8iZxt7AlulBPyRdDjxZ0ajMzKwmFXm471lgQK57o9TPzMy6mSJnGmsA0yU9krq3ByZJGg8QEQdWKjgzM6stRZLGjyoehZmZLReKvITpHlj8sN/Kuf5+n4aZWTfTbtKQdDxwJjAf+Ai/T8PMrNsqUj11GrB1RLxZ6WDMzKy2Fbl76jng/UoHYmZmta/ImcYZwAOSHgY+aOoZEd+oWFRmZlaTipxpXAjcCTxE1nhh01+7JI2UNEPSs5JOL1G+h6RHJX0o6bCOBG5mZl2vyJlGj4j4VkdHLKkOGAsMB2YCEyWNj4incoO9DBwNfKej4zczs65X5Ezj75KOl7SBpD5NfwW+twPwbEQ8HxELgGuAg/IDRMSLEfE42V1ZZmZW44qcaXwu/T8j16/ILbf9gVdy3TPJXh3bYem23+MBBgwY0M7QZmZWKUUe7tukKwJpJ4aLgIsge59GlcMxM+u2yr3u9bu5z4e3KPufAuNuIGvcsMmGqZ+ZmS2nyl3TGJ37fEaLspEFxj0R2FzSJpJWSeMb38H4zMyshpRLGmrjc6nuViLiQ+BkYAIwHbguIp6UdKakAwEkbS9pJnA4cKEkv6fDzKyGlbumEW18LtVdegQRtwK3tuj3o9zniWTVVmZmthwolzQ+KeldsrOK+vSZ1N2z4pGZmVnNaTNpRERdVwZiZma1r8jDfWZmZoCThpmZdYCThpmZFeakYWZmhTlpmJlZYU4aZmZWmJOGmZkV5qRhZmaFOWmYmVlhThpmZlaYk4aZmRXmpGFmZoU5aZiZWWFOGmZmVpiThpmZFeakYWZmhTlpmJlZYU4aZmZWmJOGmZkV5qRhZmaFOWmYmVlhThpmZlbYytUOwMxWXAsXLmTmzJnMnz+/2qF0Gz179mTDDTekR48eFRm/k4aZVczMmTNZY401GDhwIJKqHc4KLyJ46623mDlzJptssklFpuGkYVUzbkoDZ0+Ywaw5jfTrXc9pIwYxapv+1Q7LOtH8+fOdMLqQJNZZZx3eeOONik3DScOqYtyUBs64YRqNCxcB0DCnkTNumAbgxLGCccLoWpVe3r4QblVx9oQZixNGk8aFizh7wowqRWRmRThpWFXMmtPYof5m3UVdXR1Dhw5l66235oADDmDOnDkAvPjii9TX1zN06FAGDx7Ml770JRYuXNjl8TlpWFX0613fof5mtWTRokXtD7SU6uvrmTp1Kk888QR9+vRh7Nixi8s+/vGPM3XqVKZNm8bMmTO57rrrKhZHW3xNw6ritBGDml3TAKjvUcdpIwZVMSqrtCMufLBVv/2HbMCROw+kccEijv7TI63KD9tuQw4fthGz5y3gq1dMblZ27Qk7tzvNUaNG8corrzB//nxOOeUUjj/+eC644AKee+45zj77bAAuu0Au1WsAABCXSURBVOwyJk2axHnnnccVV1zBmDFjWLBgATvuuCN/+MMfqKurY/XVV+eEE07g9ttvZ+zYsdx5553cdNNNNDY2sssuu3DhhRciiYkTJ3Lsscey0korMXz4cP7+97/zxBNPsGjRIk4//XTuvvtuPvjgA0466SROOOGEsrHvvPPOPP74463619XVscMOO9DQ0NDu/Hc2n2lYVYzapj+/POQT9O9dj4D+vev55SGf8EVw63SXXnopkydPZtKkSYwZM4a33nqLQw89lBtvvHHxMNdeey2jR49m+vTpXHvttdx///1MnTqVuro6rrzySgDmzZvHjjvuyGOPPcZuu+3GySefzMSJE3niiSdobGzk5ptvBuCYY47hwgsvXPz9JpdccglrrbUWEydOZOLEiVx88cW88MILbca9aNEi7rjjDg488MBWZfPnz+fhhx9m5MiRnbWYCvOZhlXNqG36O0l0M+XODOpXqStb3me1VQqdWbQ0ZsyYxQnilVde4ZlnnmGnnXZi00035aGHHmLzzTfn6aefZtddd2Xs2LFMnjyZ7bffHoDGxkbWW289IDu6P/TQQxeP96677uLXv/4177//PrNnz2arrbZi9913Z+7cuey8cxbn5z//+cXJ5LbbbuPxxx/n+uuvB+Cdd97hmWeeafU8RWNjI0OHDqWhoYEtt9yS4cOHLy577rnnGDp0KC+88AL77bcfQ4YM6fDyWFYVTRqSRgLnAHXAHyPirBblqwJ/BrYD3gKOiIgXKxlTNdXKcwm1Eoct4XVSGXfffTe33347Dz74IL169WLPPfdc/HT66NGjue6669hiiy04+OCDkUREcNRRR/HLX/6y1bh69uy5+Mxh/vz5fO1rX2PSpElstNFG/OQnP2n3qfeI4Nxzz2XEiBFlh2u6pvH+++8zYsQIxo4dyze+8Q1gyTWNN998k1133ZXx48eXPBOppIpVT0mqA8YC+wKDgc9JGtxisGOBtyNiM+B3wK8qFU+1NT2X0DCnkWDJcwnjpnRtnWStxGFLeJ1UzjvvvMPaa69Nr169ePrpp3nooYcWlx188MH87W9/4+qrr2b06NEA7LPPPlx//fW8/vrrAMyePZuXXnqp1XibEkTfvn157733Fp899O7dmzXWWIOHH34YgGuuuWbxd0aMGMH555+/+I6nf//738ybN6/N2Hv16sWYMWP4zW9+w4cfftisrG/fvpx11lklk1ulVfKaxg7AsxHxfEQsAK4BDmoxzEHA5enz9cA+WkGfBKqV5xJqJQ5bwuukckaOHMmHH37Illtuyemnn85OO+20uGzttddmyy235KWXXmKHHXYAYPDgwfz85z/nM5/5DEOGDGH48OG8+uqrrcbbu3dvjjvuOLbeemtGjBixuDoLsmsXxx13HEOHDmXevHmstdZaAHzlK19h8ODBbLvttmy99daccMIJrZJBS9tssw1Dhgzh6quvblU2atQo3n//fe67776lWjZLSxFRmRFLhwEjI+IrqftIYMeIODk3zBNpmJmp+7k0zJstxnU8cDzAgAEDtiuV+WvdJqffQqklLeCFs/brdnHYEivyOpk+fTpbbrlltcPoUu+99x6rr746AGeddRavvvoq55xzTpfGUGq5S5ocEcOWddzLxd1TEXFRRAyLiGHrrrtutcNZKrXyXEKtxGFLeJ2sWG655ZbFD+fdd999/OAHP6h2SJ2qkkmjAdgo171h6ldyGEkrA2uRXRBf4Zw2YhD1Peqa9avGcwm1Eoct4XWyYjniiCMWP5x3yy23sLwe6LalkndPTQQ2l7QJWXIYDXy+xTDjgaOAB4HDgDujUvVlVdZ0J0y175CplThsiRV9nUSEGy3sQpX+Ca3YNQ0ASZ8Ffk92y+2lEfELSWcCkyJivKSewF+AbYDZwOiIeL7cOIcNGxaTJk2qWMxm1nleeOEF1lhjDdZZZx0nji7Q9D6NuXPntnr+o7OuaVQ0aVSCk4bZ8sNv7ut6bb25r7OShp8IN7OK6dGjR8XeIGfVsVzcPWVmZrXBScPMzApz0jAzs8KWuwvhkt4AlvaR8L7Am+0OVXmOoznHUVsxgONoaUWIY+OIWOaHRpa7pLEsJE3qjLsHHIfjWJFjcByOoxxXT5mZWWFOGmZmVlh3SxoXVTuAxHE05ziWqIUYwHG05DiSbnVNw8zMlk13O9MwM7Nl4KRhZmaFdYukIelSSa+nNwVWM46NJN0l6SlJT0o6pUpx9JT0iKTHUhw/rUYcKZY6SVMk3VzFGF6UNE3SVElVaw1TUm9J10t6WtJ0STtXIYZBaTk0/b0r6dSujiPF8s20fT4h6erUKnZXx3BKmv6TXbkcSv1mSeoj6Z+Snkn/1+6qePK6RdIALgNGVjsI4EPg2xExGNgJOEnS4CrE8QGwd0R8EhgKjJS0UzvfqZRTgOlVmnbeXhExtMr3wJ8D/CMitgA+SRWWS0TMSMthKLAd8D5wY1fHIak/8A1gWERsTfZ6hdFdHMPWwHHADmTrY39Jm3XR5C+j9W/W6cAdEbE5cEfq7nLdImlExL1k7+uodhyvRsSj6fNcsh+FLn/TTmTeS5090l+X3xEhaUNgP+CPXT3tWiNpLWAP4BKAiFgQEXOqGxX7AM9FxNK2wLCsVgbq01s9ewGzunj6WwIPR8T7EfEhcA9wSFdMuI3frIOAy9Pny4FRXRFLS90iadQiSQPJXj71cJWmXydpKvA68M+IqEYcvwe+C3xUhWnnBXCbpMmSjq9SDJsAbwB/StV1f5S0WpViaTIauLoaE46IBuB/gZeBV4F3IuK2Lg7jCWB3SetI6gV8luavsO5q60fEq+nzf4D1qxGEk0YVSFod+D/g1Ih4txoxRMSiVAWxIbBDOhXvMpL2B16PiMldOd027BYR2wL7klUZ7lGFGFYGtgXOj4htgHlUqfoBQNIqwIHAX6s0/bXJjqw3AfoBq0n6YlfGEBHTgV8BtwH/AKYCi7oyhrak12JX5XkJJ40uJqkHWcK4MiJuqHY8qQrkLrr+ms+uwIGSXgSuAfaWdEUXxwAsPqolIl4nq7/foQphzARm5s74ridLItWyL/BoRLxWpel/GnghIt6IiIXADcAuXR1ERFwSEdtFxB7A28C/uzqGnNckbQCQ/r9ejSCcNLqQspckXwJMj4jfVjGOdSX1Tp/rgeHA010ZQ0ScEREbRsRAsmqQOyOiS48kASStJmmNps/AZ8iqJbpURPwHeEXSoNRrH+Cpro4j53NUqWoqeRnYSVKvtN/sQxVuDJC0Xvo/gOx6xlVdHUPOeOCo9Pko4G/VCKJbvO5V0tXAnkBfSTOBH0fEJVUIZVfgSGBaup4A8P2IuLWL49gAuFxSHdmBw3URUbVbXqtsfeDG7HeJlYGrIuIfVYrl68CVqWroeeCYagSRkudw4IRqTB8gIh6WdD3wKNldh1OoThMa/ydpHWAhcFJX3ZxQ6jcLOAu4TtKxZK+H+O+uiKVVbG5GxMzMinL1lJmZFeakYWZmhTlpmJlZYU4aZmZWmJOGmZkV5qTRzUlalFozfULSX1NzCUs7rsskHZY+/7FcY4yS9pTU4Ye1Uou0fdvoP03S45LukbRxruyBdsb5XrnyNEx+Od3U9JxLB+JudxptfG+opM/mug+U1ClPikvaUNLfUqupz0s6T9KqnTHuFtNptq4lnSjpS+nz4m3Glg9OGtaYWjXdGlgAnJgvTI3FdVhEfCUiyj2ctied/4TvXhExBLgb+EEuls6YTn45zQZO6oRxFjGUrM0jACJifESctawjTQ/M3QCMS62mbg7UA79e1nGXsCe5dR0RF0TEnyswHesCThqWdx+wWToyvE/SeOCp1Ljh2ZImpiP5EyD74UlHpzMk3Q6s1zQiSXdLGpY+j5T0qLL3d9yRGms8EfhmOnrfPT2l/n9pGhMl7Zq+u46k25S9z+CPgArMx4PkWg9uOsqXtIGke3NnDLvnvySpr6QHJe1XdPySPi7pH8oaO7xP0hap/yZpXNMk/Tw3jT2Ve3dIWn5Hp8/bS3ogLadHlLV8eyZwRIr5CElHSzovDT9Q0p1pndyh7KnlpqP3MWlcz7dxJL83MD8i/gRZW2TAN4EvSVo9P500zpsl7Zk+ny9pklq8iyWd7f00retpkrZoY13/RNJ3WgYkabt0ljhZ0gSlJjOstjhpGLD4jGJfYFrqtS1wSkT8F3AsWSuj2wPbA8dJ2gQ4GBgEDAa+RIkzB0nrAhcDh6b3dxweES8CFwC/S0fv95G9S+J3aRqHsqS59B8D/4qIrcjahRpQYHZGAuNK9P88MCE11PhJsgbomuJcH7gF+FFE3NLWiJU9Rb8PWZMOkD2l/PWI2A74DvCH1P8cssYHP0HWSmtZyp4Cv5ZsmX+SrO2lecCPgGvTcrq2xdfOBS5PZ1dXAmNyZRsAuwH7kz1J3NJWQLPGIlPjmS8C7b0z4v+l944MAT4laUiu7M3U+OP5wHfaWNel5r9Hmp/D0rK8FPhFO3FYFXSLZkSsrHotadLkPrK2sXYBHomIF1L/zwBDckesa5FVZ+wBXJ2OUmdJurPE+HcC7m0aV0S09V6TTwODs1oTANZU1hrwHqR3GETELZLeLjMvd0nqA7wH/LBE+UTg0vQDNS4imua7B9lLbU6KiHvaGHfTcupP1gbSP1N8uwB/zcXddE1gV7LkB/AXstZSyxkEvBoRE2HxDzi58ZayM0ve7/AXmlctjYuIj8jOFDu7Ce3/VtaE/MpkyWkw8Hgqa2qEczIde/fEIGBrsuUK2UuX2k221vWcNKwxHXkvlnbaefleZEfTE1oM91k6z0rAThExv0QsRe0FzCE76v4p8K18YUTcq6zZ8/2AyyT9NtWtf0j2IzeC7EU7pTRGxFBlNwpMILumcRkwp+Xyy0+yRL8PaX6GX6lXmH6Q+1xqIT4FNKu2krQm8DFgBtkPeKs40xnmd4DtI+JtSZfRfB6apruIjv2+CHgyIrr8FbfWMa6esiImAF9NR+hI+i9ljdrdS1bfXpfqn/cq8d2HgD3Sjw3pTABgLrBGbrjbyBrsIw3X9EN8L1m1EpL2Bcq+Fzm9Ye1Usrr5PvkyZXdUvRYRF5NVfzU1PR7Al4EtJH2vnfG/T/Ya0m+TvQr1BUmHp/FL0ifToPez5PWkX8iN4iWyM6pVld2BtU/qPwPYQNL2aVxrpCrDlssp74EW0yhZ9dOGO4BeWnIXUx3wG+C8iGgkq6YaKmklSRuxpLn4NckOKN5JZzD7FphWuXloMgNYV+m96JJ6SNqqA/NjXcRJw4r4I9mR6aPKXnR/IdlR5I3AM6nsz2QXiJuJiDeA44EbJD1GVm8PcBNwcNPFUdL7oNNF3adYchfXT8mSzpNk1R0vtxdservZ1bS+w2lP4DFJU4AjyK47NH1nEVlz4HtL+lo7459CVh3zObIf62PTvD1J9uIgyN59fpKkaeQuykfEK8B1ZM2vX0fWeisRsSDFdG4a1z/JjuDvIksyUyUd0SKUrwPHSHqcrPXkU8ovmWbzEGTXpA6T9AzwFvBRRDRdR7gfeIFs3Y4ha22WiHgsxfw0WTPh9xeYXMt1XSqeBWRnPr9K8z+VKrw/w9rnVm7NDGXPUVwNHNz0HnuzUpw0zMysMFdPmZlZYU4aZmZWmJOGmZkV5qRhZmaFOWmYmVlhThpmZlbY/we33F5tjAjSEgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YZM3WZ2fPvOn"
      },
      "source": [
        "If the model performed well, then you would see patients in the higher deciles of predicted risk reduction (on the right) also have higher empirical risk reduction (to the top).  \n",
        "\n",
        "This model using logistic regression is far from perfect. \n",
        "\n",
        "Below, you'll see if you can do better using a more flexible machine learning approach."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "JL8ET3lk9r02"
      },
      "source": [
        "<a name=\"4\"></a>\n",
        "## 4 Machine Learning Approaches "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-oOkd5juz5To"
      },
      "source": [
        "<a name=\"4-1\"></a>\n",
        "### 4.1 T-Learner\n",
        "\n",
        "Now you will see how recent machine learning approaches compare to the more standard analysis. The approach we'll look at is called [T-learner](https://arxiv.org/pdf/1706.03461.pdf).\n",
        "- \"T\" stands for \"two\". \n",
        "- The T-learner learns two different models, one for treatment risk, and another model for control risk.\n",
        "- Then takes the difference of the two risk predictions to predict the risk reduction.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ec1S0C6qTmso",
        "colab_type": "text"
      },
      "source": [
        "<a name='ex-09'></a>\n",
        "### Exercise 9: Complete the TLearner class.  \n",
        "\n",
        "- The constructor `__init__()` sets the treatment and control estimators based on the given inputs to the constructor.\n",
        "- The `predict` function takes the features and uses each estimator to predict the risk of death.  Then it calculates the risk of death for the control estimator minus the risk of death from the treatment estimator, and returns this as the predicted risk reduction."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "38i0UWVKTmsp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# UNQ_C9 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
        "class TLearner():\n",
        "    \"\"\"\n",
        "    T-Learner class.\n",
        "\n",
        "    Attributes:\n",
        "      treatment_estimator (object): fitted model for treatment outcome\n",
        "      control_estimator (object): fitted model for control outcome\n",
        "    \"\"\"                               \n",
        "    def __init__(self, treatment_estimator, control_estimator):\n",
        "        \"\"\"\n",
        "        Initializer for TLearner class.\n",
        "        \"\"\"\n",
        "        ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
        "        # set the treatment estimator\n",
        "        self.treatment_estimator = treatment_estimator\n",
        "        \n",
        "        # set the control estimator \n",
        "        self.control_estimator = control_estimator\n",
        "        \n",
        "        ### END CODE HERE ###\n",
        "\n",
        "    def predict(self, X):\n",
        "        \"\"\"\n",
        "        Return predicted risk reduction for treatment for given data matrix.\n",
        "\n",
        "        Args:\n",
        "          X (dataframe): dataframe containing features for each subject\n",
        "    \n",
        "        Returns:\n",
        "          preds (np.array): predicted risk reduction for each row of X\n",
        "        \"\"\"\n",
        "        ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
        "        # predict the risk of death using the control estimator\n",
        "        risk_control = self.control_estimator.predict_proba(X)[:,1]\n",
        "        \n",
        "        # predict the risk of death using the treatment estimator\n",
        "        risk_treatment = self.treatment_estimator.predict_proba(X)[:,1]\n",
        "        \n",
        "        # the predicted risk reduction is control risk minus the treatment risk\n",
        "        pred_risk_reduction =  risk_control - risk_treatment\n",
        "        \n",
        "        ### END CODE HERE ###\n",
        "                \n",
        "        return pred_risk_reduction"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rpIuxXw0Tmsu",
        "colab_type": "text"
      },
      "source": [
        "### Tune the model with grid search\n",
        "\n",
        "In order to tune your two models, you will use grid search to find the desired parameters.\n",
        "- You will use a validation set to evaluate the model on different parameters, in order to avoid overfitting to the training set.\n",
        "\n",
        "To test models on all combinations of hyperparameters, you can first list out all of the values in a list of lists.\n",
        "For example:\n",
        "```CPP\n",
        "hyperparams = {\n",
        "    'n_estimators': [10, 20],\n",
        "    'max_depth': [2, 5],\n",
        "    'min_samples_leaf': [0.1, 0.2],\n",
        "    'random_state': [0]\n",
        "}\n",
        "```\n",
        "You can generate a list like this:\n",
        "```CPP\n",
        "[[10, 20],\n",
        " [2, 5], \n",
        " [0.1, 0.2]\n",
        "]\n",
        "```\n",
        "\n",
        "Next, you can get all combinations of the hyperparameter values:\n",
        "```CPP\n",
        "[(10, 2, 0.1),\n",
        " (10, 2, 0.2),\n",
        " (10, 5, 0.1),\n",
        " (10, 5, 0.2),\n",
        " (20, 2, 0.1),\n",
        " (20, 2, 0.2),\n",
        " (20, 5, 0.1),\n",
        " (20, 5, 0.2)]\n",
        "```\n",
        "\n",
        "To feed the hyperparameters into an random forest model, you can use a dictionary, so that you do not need to hard code the parameter names.\n",
        "For example, instead of\n",
        "```CPP\n",
        "RandomForestClassifier(n_estimators= 20, max_depth=5, min_samples_leaf=0.2)\n",
        "```\n",
        "\n",
        "You have more flexibility if you create a dictionary and pass it into the model.\n",
        "```CPP\n",
        "args_d = {'n_estimators': 20, 'max_depth': 5, 'min_samples_leaf': 0.2}\n",
        "RandomForestClassifier(**args_d)\n",
        "```\n",
        "This allows you to pass in a hyperparameter dictionary for any hyperpameters, not just `n_estimators`, `max_depth`, and `min_samples_leaf`.\n",
        "\n",
        "So you'll find a way to generate a list of dictionaries, like this:\n",
        "```CPP\n",
        "[{'n_estimators': 10, 'max_depth': 2, 'min_samples_leaf': 0.1},\n",
        " {'n_estimators': 10, 'max_depth': 2, 'min_samples_leaf': 0.2},\n",
        " {'n_estimators': 10, 'max_depth': 5, 'min_samples_leaf': 0.1},\n",
        " {'n_estimators': 10, 'max_depth': 5, 'min_samples_leaf': 0.2},\n",
        " {'n_estimators': 20, 'max_depth': 2, 'min_samples_leaf': 0.1},\n",
        " {'n_estimators': 20, 'max_depth': 2, 'min_samples_leaf': 0.2},\n",
        " {'n_estimators': 20, 'max_depth': 5, 'min_samples_leaf': 0.1},\n",
        " {'n_estimators': 20, 'max_depth': 5, 'min_samples_leaf': 0.2}]\n",
        "```\n",
        "\n",
        "Notice how the values in both the list of tuples and list of dictionaries are in the same order as the original hyperparams dictionary.  For example, the first value in each is n_estimarors, then max_depth, and then min_samples_leaf:\n",
        "```CPP\n",
        "# list of lists\n",
        "(10, 2, 0.1)\n",
        "\n",
        "# list of dictionaries\n",
        "{'n_estimators': 10, 'max_depth': 2, 'min_samples_leaf': 0.1}\n",
        "```\n",
        "\n",
        "\n",
        "\n",
        "Then for each dictionary of hyperparams:\n",
        "- Train a model.\n",
        "- Use the regular concordance index to compare their performances.  \n",
        "- Identify and return the best performing model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zAIB-9w8Tmsw",
        "colab_type": "text"
      },
      "source": [
        "<a name='ex-10'></a>\n",
        "### Exercise 10: hold out grid search\n",
        "\n",
        "Implement hold out grid search.  \n",
        "##### Note\n",
        "In this case, you are not going to apply k-fold cross validation.  Since `sklearn.model_selection.GridSearchCV()` applies k-fold cross validation, you won't be using this to perform grid search, and you will implement your own grid search.\n",
        "\n",
        "Please see the hints if you get stuck."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nTF9VHWgTmsw",
        "colab_type": "text"
      },
      "source": [
        "<details>    \n",
        "<summary>\n",
        "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
        "</summary>\n",
        "<p>\n",
        "<ul>\n",
        "    <li>You can use the .items() or .values() method of a dictionary to get its key, value pairs or just values.  Use a list() to store them inside a list.</li>\n",
        "    <li>To get all combinations of the hyperparams, you can use itertools.product(*args_list), where args_list is a list object.</li>\n",
        "    <li>To generate the list of dictionaries, loop through the list of tuples.  The position of each value</li>\n",
        "</ul>\n",
        "</p>\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QGfgTRUVTmsx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# UNQ_C10 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
        "def holdout_grid_search(clf, X_train_hp, y_train_hp, X_val_hp, y_val_hp, hyperparam, verbose=False):\n",
        "    '''\n",
        "    Conduct hyperparameter grid search on hold out validation set. Use holdout validation.\n",
        "    Hyperparameters are input as a dictionary mapping each hyperparameter name to the\n",
        "    range of values they should iterate over. Use the cindex function as your evaluation\n",
        "    function.\n",
        "    \n",
        "    Input:\n",
        "        clf: sklearn classifier\n",
        "        X_train_hp (dataframe): dataframe for training set input variables\n",
        "        y_train_hp (dataframe): dataframe for training set targets\n",
        "        X_val_hp (dataframe): dataframe for validation set input variables\n",
        "        y_val_hp (dataframe): dataframe for validation set targets\n",
        "        hyperparam (dict): hyperparameter dictionary mapping hyperparameter\n",
        "                                                names to range of values for grid search\n",
        "    \n",
        "    Output:\n",
        "        best_estimator (sklearn classifier): fitted sklearn classifier with best performance on\n",
        "                                                                                 validation set\n",
        "    '''\n",
        "    # Initialize best estimator\n",
        "    best_estimator = None\n",
        "    \n",
        "    # initialize best hyperparam\n",
        "    best_hyperparam = {}\n",
        "    \n",
        "    # initialize the c-index best score to zero\n",
        "    best_score = 0.0\n",
        "    \n",
        "    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
        "    \n",
        "    # Get the values of the hyperparam and store them as a list of lists\n",
        "    hyper_param_l = list(hyperparam.values())\n",
        "    \n",
        "    # Generate a list of tuples with all possible combinations of the hyperparams\n",
        "    combination_l_of_t = list(itertools.product(*hyper_param_l))\n",
        "    \n",
        "    # Initialize the list of dictionaries for all possible combinations of hyperparams\n",
        "    combination_l_of_d = []\n",
        "    \n",
        "    # loop through each tuple in the list of tuples\n",
        "    for val_tuple in combination_l_of_t: # complete this line\n",
        "        param_d = {}\n",
        "        \n",
        "        # Enumerate each key in the original hyperparams dictionary\n",
        "        for i, k in enumerate(hyperparam.keys()): # complete this line\n",
        "            \n",
        "            # add a key value pair to param_dict for each value in val_tuple\n",
        "            param_d[k] = val_tuple[i]\n",
        "        \n",
        "        # append the param_dict to the list of dictionaries\n",
        "        combination_l_of_d.append(param_d)\n",
        "        \n",
        "    \n",
        "    # For each hyperparam dictionary in the list of dictionaries:\n",
        "    for param_d in combination_l_of_d: # complete this line\n",
        "        \n",
        "        # Set the model to the given hyperparams\n",
        "        estimator = clf(**param_d)\n",
        "        \n",
        "        # Train the model on the training features and labels\n",
        "        estimator.fit(X_train_hp, y_train_hp)\n",
        "        \n",
        "        # Predict the risk of death using the validation features\n",
        "        preds = estimator.predict_proba(X_val_hp)[:,1]\n",
        "        \n",
        "        # Evaluate the model's performance using the regular concordance index\n",
        "        estimator_score = concordance_index(y_val_hp, preds)\n",
        "        \n",
        "        # if the model's c-index is better than the previous best:\n",
        "        if estimator_score > best_score: # complete this line\n",
        "\n",
        "            # save the new best score\n",
        "            best_score = estimator_score\n",
        "            \n",
        "            # same the new best estimator\n",
        "            best_estimator = estimator\n",
        "            \n",
        "            # save the new best hyperparams\n",
        "            best_hyperparam = param_d\n",
        "                \n",
        "    ### END CODE HERE ###\n",
        "\n",
        "    if verbose:\n",
        "        print(\"hyperparam:\")\n",
        "        display(hyperparam)\n",
        "        \n",
        "        print(\"hyper_param_l\")\n",
        "        display(hyper_param_l)\n",
        "        \n",
        "        print(\"combination_l_of_t\")\n",
        "        display(combination_l_of_t)\n",
        "        \n",
        "        print(f\"combination_l_of_d\")\n",
        "        display(combination_l_of_d)\n",
        "        \n",
        "        print(f\"best_hyperparam\")\n",
        "        display(best_hyperparam)\n",
        "        print(f\"best_score: {best_score:.4f}\")\n",
        "        \n",
        "    return best_estimator, best_hyperparam"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ocMk7MC_Tms2",
        "colab_type": "code",
        "colab": {},
        "outputId": "0bb7d9ea-be5f-432d-adc2-abcb3a55f99d"
      },
      "source": [
        "# Test\n",
        "n = X_dev.shape[0]\n",
        "tmp_X_train = X_dev.iloc[:int(n*0.8),:]\n",
        "tmp_X_val = X_dev.iloc[int(n*0.8):,:]\n",
        "tmp_y_train = y_dev[:int(n*0.8)]\n",
        "tmp_y_val = y_dev[int(n*0.8):]\n",
        "\n",
        "hyperparams = {\n",
        "    'n_estimators': [10, 20],\n",
        "    'max_depth': [2, 5],\n",
        "    'min_samples_leaf': [0.1, 0.2],\n",
        "    'random_state' : [0]\n",
        "}\n",
        "\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "control_model = holdout_grid_search(RandomForestClassifier,\n",
        "                                    tmp_X_train, tmp_y_train,\n",
        "                                    tmp_X_val, tmp_y_val, hyperparams, verbose=True)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "hyperparam:\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "{'n_estimators': [10, 20],\n",
              " 'max_depth': [2, 5],\n",
              " 'min_samples_leaf': [0.1, 0.2],\n",
              " 'random_state': [0]}"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "hyper_param_l\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "[[10, 20], [2, 5], [0.1, 0.2], [0]]"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "combination_l_of_t\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "[(10, 2, 0.1, 0),\n",
              " (10, 2, 0.2, 0),\n",
              " (10, 5, 0.1, 0),\n",
              " (10, 5, 0.2, 0),\n",
              " (20, 2, 0.1, 0),\n",
              " (20, 2, 0.2, 0),\n",
              " (20, 5, 0.1, 0),\n",
              " (20, 5, 0.2, 0)]"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "combination_l_of_d\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "[{'n_estimators': 10,\n",
              "  'max_depth': 2,\n",
              "  'min_samples_leaf': 0.1,\n",
              "  'random_state': 0},\n",
              " {'n_estimators': 10,\n",
              "  'max_depth': 2,\n",
              "  'min_samples_leaf': 0.2,\n",
              "  'random_state': 0},\n",
              " {'n_estimators': 10,\n",
              "  'max_depth': 5,\n",
              "  'min_samples_leaf': 0.1,\n",
              "  'random_state': 0},\n",
              " {'n_estimators': 10,\n",
              "  'max_depth': 5,\n",
              "  'min_samples_leaf': 0.2,\n",
              "  'random_state': 0},\n",
              " {'n_estimators': 20,\n",
              "  'max_depth': 2,\n",
              "  'min_samples_leaf': 0.1,\n",
              "  'random_state': 0},\n",
              " {'n_estimators': 20,\n",
              "  'max_depth': 2,\n",
              "  'min_samples_leaf': 0.2,\n",
              "  'random_state': 0},\n",
              " {'n_estimators': 20,\n",
              "  'max_depth': 5,\n",
              "  'min_samples_leaf': 0.1,\n",
              "  'random_state': 0},\n",
              " {'n_estimators': 20,\n",
              "  'max_depth': 5,\n",
              "  'min_samples_leaf': 0.2,\n",
              "  'random_state': 0}]"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "best_hyperparam\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "{'n_estimators': 10,\n",
              " 'max_depth': 2,\n",
              " 'min_samples_leaf': 0.1,\n",
              " 'random_state': 0}"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "best_score: 0.5928\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uh6fKMXWTms7",
        "colab_type": "text"
      },
      "source": [
        "T-Learner is a convenient framework because it does not restrict your choice of base learners.\n",
        "- You will use random forests as the base learners, but are able to choose another model as well."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RcXQSQJGTms8",
        "colab_type": "text"
      },
      "source": [
        "##### Expected output\n",
        "\n",
        "```CPP\n",
        "##### Expected output\n",
        "\n",
        "```CPP\n",
        "hyperparam:\n",
        "{'n_estimators': [10, 20],\n",
        " 'max_depth': [2, 5],\n",
        " 'min_samples_leaf': [0.1, 0.2],\n",
        " 'random_state': [0]}\n",
        "hyper_param_l\n",
        "[[10, 20], [2, 5], [0.1, 0.2], [0]]\n",
        "combination_l_of_t\n",
        "[(10, 2, 0.1, 0),\n",
        " (10, 2, 0.2, 0),\n",
        " (10, 5, 0.1, 0),\n",
        " (10, 5, 0.2, 0),\n",
        " (20, 2, 0.1, 0),\n",
        " (20, 2, 0.2, 0),\n",
        " (20, 5, 0.1, 0),\n",
        " (20, 5, 0.2, 0)]\n",
        "combination_l_of_d\n",
        "[{'n_estimators': 10,\n",
        "  'max_depth': 2,\n",
        "  'min_samples_leaf': 0.1,\n",
        "  'random_state': 0},\n",
        " {'n_estimators': 10,\n",
        "  'max_depth': 2,\n",
        "  'min_samples_leaf': 0.2,\n",
        "  'random_state': 0},\n",
        " {'n_estimators': 10,\n",
        "  'max_depth': 5,\n",
        "  'min_samples_leaf': 0.1,\n",
        "  'random_state': 0},\n",
        " {'n_estimators': 10,\n",
        "  'max_depth': 5,\n",
        "  'min_samples_leaf': 0.2,\n",
        "  'random_state': 0},\n",
        " {'n_estimators': 20,\n",
        "  'max_depth': 2,\n",
        "  'min_samples_leaf': 0.1,\n",
        "  'random_state': 0},\n",
        " {'n_estimators': 20,\n",
        "  'max_depth': 2,\n",
        "  'min_samples_leaf': 0.2,\n",
        "  'random_state': 0},\n",
        " {'n_estimators': 20,\n",
        "  'max_depth': 5,\n",
        "  'min_samples_leaf': 0.1,\n",
        "  'random_state': 0},\n",
        " {'n_estimators': 20,\n",
        "  'max_depth': 5,\n",
        "  'min_samples_leaf': 0.2,\n",
        "  'random_state': 0}]\n",
        "best_hyperparam\n",
        "{'n_estimators': 10,\n",
        " 'max_depth': 2,\n",
        " 'min_samples_leaf': 0.1,\n",
        " 'random_state': 0}\n",
        "best_score: 0.5928\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "O-BkhCwzIEYT"
      },
      "source": [
        "<a name='ex-11'></a>\n",
        "### Exercise 11: Training and validation, treatment and control splits\n",
        "\n",
        "- Unlike logistic regression, the machine learning algorithms used for base learners will generally require hyperparameter tuning, which means that you need to split your dev set into a training and validation set. \n",
        "- You need to also split each of the training and validation sets into *treatment* and *control* groups to train the treatment and control base learners of the T-Learner.\n",
        "\n",
        "The function below takes in a dev dataset and splits it into training and validation sets for treatment and control models, respectively. \n",
        "Complete the implementation. \n",
        "\n",
        "#### Note\n",
        "- The input X_train and X_val have the 'TRTMT' column.  Please remove the 'TRTMT' column from the treatment and control features that the function returns."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nZBoztjVTms-",
        "colab_type": "text"
      },
      "source": [
        "<details>    \n",
        "<summary>\n",
        "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
        "</summary>\n",
        "<p>\n",
        "<ul>\n",
        "    <li> To drop a column, set the axis to 1 when calling pandas.DataFrame.drop (axis=0 is used to drop a row by its index label)</li>\n",
        "    <li> </li>\n",
        "</ul>\n",
        "</p>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "QdVLM4Zxjd4L",
        "colab": {}
      },
      "source": [
        "# UNQ_C11 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
        "def treatment_dataset_split(X_train, y_train, X_val, y_val):\n",
        "    \"\"\"\n",
        "    Separate treated and control individuals in training\n",
        "    and testing sets. Remember that returned\n",
        "    datasets should NOT contain the 'TRMT' column!\n",
        "\n",
        "    Args:\n",
        "        X_train (dataframe): dataframe for subject in training set\n",
        "        y_train (np.array): outcomes for each individual in X_train\n",
        "        X_val (dataframe): dataframe for subjects in validation set\n",
        "        y_val (np.array): outcomes for each individual in X_val\n",
        "    \n",
        "    Returns:\n",
        "        X_treat_train (df): training set for treated subjects\n",
        "        y_treat_train (np.array): labels for X_treat_train\n",
        "        X_treat_val (df): validation set for treated subjects\n",
        "        y_treat_val (np.array): labels for X_treat_val\n",
        "        X_control_train (df): training set for control subjects\n",
        "        y_control_train (np.array): labels for X_control_train\n",
        "        X_control_val (np.array): validation set for control subjects\n",
        "        y_control_val (np.array): labels for X_control_val\n",
        "    \"\"\"\n",
        "    \n",
        "    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
        "    \n",
        "    # From the training set, get features of patients who received treatment\n",
        "    X_treat_train = X_train[X_train[\"TRTMT\"] == 1]\n",
        "    \n",
        "    # drop the 'TRTMT' column\n",
        "    X_treat_train = X_treat_train.drop(\"TRTMT\", axis=1)\n",
        "    \n",
        "    # From the training set, get the labels of patients who received treatment\n",
        "    y_treat_train = y_train[X_train[\"TRTMT\"]==1]\n",
        "\n",
        "    # From the validation set, get the features of patients who received treatment\n",
        "    X_treat_val = X_val[X_val[\"TRTMT\"] == 1]\n",
        "                        \n",
        "    # Drop the 'TRTMT' column\n",
        "    X_treat_val = X_treat_val.drop(\"TRTMT\", axis=1)\n",
        "                        \n",
        "    # From the validation set, get the labels of patients who received treatment\n",
        "    y_treat_val = y_val[X_val[\"TRTMT\"] == 1]\n",
        "                        \n",
        "# --------------------------------------------------------------------------------------------\n",
        "                        \n",
        "    # From the training set, get the features of patients who did not received treatment\n",
        "    X_control_train = X_train[X_train[\"TRTMT\"] == 0]\n",
        "                        \n",
        "    # Drop the TRTMT column\n",
        "    X_control_train = X_control_train.drop(\"TRTMT\", axis=1)\n",
        "                        \n",
        "    # From the training set, get the labels of patients who did not receive treatment\n",
        "    y_control_train = y_train[X_train[\"TRTMT\"]==0]\n",
        "    \n",
        "    # From the validation set, get the features of patients who did not receive treatment\n",
        "    X_control_val = X_val[X_val[\"TRTMT\"] == 0]\n",
        "    \n",
        "    # drop the 'TRTMT' column\n",
        "    X_control_val = X_control_val.drop(\"TRTMT\", axis=1)\n",
        "\n",
        "    # From the validation set, get teh labels of patients who did not receive treatment\n",
        "    y_control_val = y_val[X_val[\"TRTMT\"] == 0]\n",
        "    \n",
        "    ### END CODE HERE ###\n",
        "\n",
        "    return (X_treat_train, y_treat_train,\n",
        "            X_treat_val, y_treat_val,\n",
        "            X_control_train, y_control_train,\n",
        "            X_control_val, y_control_val)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iwQ-3tHdTmtH",
        "colab_type": "text"
      },
      "source": [
        "**Test Case**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z-VVCeW3TmtJ",
        "colab_type": "code",
        "colab": {},
        "outputId": "219e664a-8a8d-487f-eeaa-52ddc573e8fe"
      },
      "source": [
        "# Tests\n",
        "example_df = pd.DataFrame(columns = ['ID', 'TRTMT'])\n",
        "example_df.ID = range(100)\n",
        "example_df.TRTMT = np.random.binomial(n=1, p=0.5, size=100)\n",
        "treated_ids = set(example_df[example_df.TRTMT==1].ID)\n",
        "example_y = example_df.TRTMT.values\n",
        "\n",
        "example_train, example_val, example_y_train, example_y_val = train_test_split(\n",
        "    example_df, example_y, test_size = 0.25, random_state=0\n",
        ")\n",
        "\n",
        "\n",
        "(x_treat_train, y_treat_train,\n",
        " x_treat_val, y_treat_val,\n",
        " x_control_train, y_control_train,\n",
        " x_control_val, y_control_val) = treatment_dataset_split(example_train, example_y_train,\n",
        "                                                         example_val, example_y_val)\n",
        "\n",
        "print(\"Tests\")\n",
        "pass_flag = True\n",
        "pass_flag = (len(x_treat_train) + len(x_treat_val) + len(x_control_train) +\n",
        "             len(x_control_val) == 100)\n",
        "print(f\"\\nDidn't lose any subjects: {pass_flag}\")\n",
        "pass_flag = ((\"TRTMT\" not in x_treat_train) and (\"TRTMT\" not in x_treat_val) and\n",
        "             (\"TRTMT\" not in x_control_train) and (\"TRTMT\" not in x_control_val))\n",
        "print(f\"\\nTRTMT not in any splits: {pass_flag}\")\n",
        "split_treated_ids = set(x_treat_train.ID).union(set(x_treat_val.ID))\n",
        "pass_flag = (len(split_treated_ids.union(treated_ids)) == len(treated_ids))\n",
        "print(f\"\\nTreated splits have all treated patients: {pass_flag}\")\n",
        "split_control_ids = set(x_control_train.ID).union(set(x_control_val.ID))\n",
        "pass_flag = (len(split_control_ids.intersection(treated_ids)) == 0)\n",
        "print(f\"\\nAll subjects in control split are untreated: {pass_flag}\") \n",
        "pass_flag = (len(set(x_treat_train.ID).intersection(x_treat_val.ID)) == 0)\n",
        "print(f\"\\nNo overlap between treat_train and treat_val: {pass_flag}\")\n",
        "pass_flag = (len(set(x_control_train.ID).intersection(x_control_val.ID)) == 0)\n",
        "print(f\"\\nNo overlap between control_train and control_val: {pass_flag}\")\n",
        "print(f\"\\n--> Expected: All statements should be True\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tests\n",
            "\n",
            "Didn't lose any subjects: True\n",
            "\n",
            "TRTMT not in any splits: True\n",
            "\n",
            "Treated splits have all treated patients: True\n",
            "\n",
            "All subjects in control split are untreated: True\n",
            "\n",
            "No overlap between treat_train and treat_val: True\n",
            "\n",
            "No overlap between control_train and control_val: True\n",
            "\n",
            "--> Expected: All statements should be True\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2df59xufTmtk",
        "colab_type": "text"
      },
      "source": [
        "You will now train a T-learner model on the patient data, and evaluate its performance using the c-for-benefit.\n",
        "\n",
        "First, get the training and validation sets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jaIQMH2aTmtl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Import the random forest classifier to be used as the base learner\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "\n",
        "# Split the dev data into train and validation sets\n",
        "X_train, X_val, y_train, y_val = train_test_split(X_dev, \n",
        "                                                  y_dev, \n",
        "                                                  test_size = 0.25,\n",
        "                                                  random_state = 0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DmSZmSboTmto",
        "colab_type": "text"
      },
      "source": [
        "Split the training set into a treatment and control set.  \n",
        "Similarly, split the validation set into a treatment and control set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GcEhCBzNTmtp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# get treatment and control arms of training and validation sets\n",
        "(X_treat_train, y_treat_train, \n",
        " X_treat_val, y_treat_val,\n",
        " X_control_train, y_control_train,\n",
        " X_control_val, y_control_val) = treatment_dataset_split(X_train, y_train,\n",
        "                                                         X_val, y_val)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "twQpuUC7Tmtu",
        "colab_type": "text"
      },
      "source": [
        "Choose a set of hyperparameters to perform grid search and find the best model.  \n",
        "- Please first use these given hyperparameters so that you can get the same c-for-benefit calculation at the end of this exercise. \n",
        "- Afterwards, we encourage you to come back and try other ranges for these hyperparameters. \n",
        "\n",
        "```CPP\n",
        "# Given hyperparams to do grid search\n",
        "hyperparams = {\n",
        "    'n_estimators': [100, 200],\n",
        "    'max_depth': [2, 5, 10, 40, None],\n",
        "    'min_samples_leaf': [1, 0.1, 0.2],\n",
        "    'random_state': [0]\n",
        "}\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3zOiMengTmtv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# hyperparameter grid (we'll use the same one for both arms for convenience)\n",
        "# Note that we set random_state to zero\n",
        "# in order to make the output consistent each time it's run.\n",
        "hyperparams = {\n",
        "    'n_estimators': [100, 200],\n",
        "    'max_depth': [2, 5, 10, 40, None],\n",
        "    'min_samples_leaf': [1, 0.1, 0.2],\n",
        "    'random_state': [0]\n",
        "}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dlKKhaGiTmty",
        "colab_type": "text"
      },
      "source": [
        "Train the treatment base learner.  \n",
        "- Perform grid search to find a random forest classifier and associated hyperparameters with the best c-index (the regular c-index)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7ViKheAATmtz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# perform grid search with the treatment data to find the best model \n",
        "treatment_model, best_hyperparam_treat  = holdout_grid_search(RandomForestClassifier,\n",
        "                                      X_treat_train, y_treat_train,\n",
        "                                      X_treat_val, y_treat_val, hyperparams)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h7iIDn2qTmt4",
        "colab_type": "text"
      },
      "source": [
        "Train the control base learner."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JeCzocc2Tmt5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# perform grid search with the control data to find the best model \n",
        "control_model, best_hyperparam_ctrl = holdout_grid_search(RandomForestClassifier,\n",
        "                                    X_control_train, y_control_train,\n",
        "                                    X_control_val, y_control_val, hyperparams)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ZUx8A4UTmt8",
        "colab_type": "text"
      },
      "source": [
        "Combine the treatment and control base learners into the T-learner."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-umIv4w5Tmt9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Save the treatment and control models into an instance of the TLearner class\n",
        "t_learner = TLearner(treatment_model, control_model)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EvZemp2DTmt_",
        "colab_type": "text"
      },
      "source": [
        "For the validation set, predict each patient's risk reduction."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M_CI8RD9TmuA",
        "colab_type": "code",
        "colab": {},
        "outputId": "d14c295c-39cd-445b-9f00-9162819b8702"
      },
      "source": [
        "# Use the t-learner to predict the risk reduction for patients in the validation set\n",
        "rr_t_val = t_learner.predict(X_val.drop(['TRTMT'], axis=1))\n",
        "\n",
        "print(f\"X_val num of patients {X_val.shape[0]}\")\n",
        "print(f\"rr_t_val num of patient predictions {rr_t_val.shape[0]}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "X_val num of patients 114\n",
            "rr_t_val num of patient predictions 114\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xYX1rN1tIv4w"
      },
      "source": [
        "Now plot a histogram of your predicted risk reduction on the validation set. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:32:34.703743Z",
          "start_time": "2020-04-04T15:32:34.529749Z"
        },
        "colab_type": "code",
        "id": "XISgvb6IiXnl",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 444
        },
        "outputId": "6850488a-51aa-4bad-a151-1bcf9a7573bc"
      },
      "source": [
        "plt.hist(rr_t_val, bins='auto')\n",
        "plt.title(\"Histogram of Predicted ARR, T-Learner, validation set\")\n",
        "plt.xlabel('predicted risk reduction')\n",
        "plt.ylabel('count of patients')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEWCAYAAABhffzLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3dedgcVZn38e+PNQQCJCYiax6IgBO8EDAjCIhsgyAgiOCIyDLgGxEUEfU1o44GFd8IiKK8A6JiANlBlAEVEFlkU0JkX2QLQgwkbLJv8Z4/znmg0unup56luvOkfp/r6qtrOVXn7urqu0+fqq5SRGBmZvWxRLcDMDOzznLiNzOrGSd+M7OaceI3M6sZJ34zs5px4jczq5naJH5Jd0rauttxdJOkD0t6RNLzkjbucN1XSfpkHt5H0mUdqLNHUkhaquq6rLMkTZf07Tz8Pkn3lik7wLqel7TOQJdfFC0WiV/SLEnbN0w7QNK1veMRsUFEXNXHehb3RHEs8JmIWCEi/tI4M7/2F/KOPlvScZKWHOogIuKMiNihr3KSpkr6xVDX36SeqyQ9LWnZhunTJb2at8dTki6X9I7C/AMkzc/zn5V0q6RdStZ5Ul7u+VzHa4Xx3zYpv9A+bklE/DEi1h+KdRUbKIX1rxARDw7F+gdrqHLUYpH4h4tF4AtlPHBnH2XeFRErANsBHwf+T2OBReB1DBlJPcD7gAA+1KTI0Xl7rA7MBn7WMP+GPH9l4L+BsyWt3Fe9EXFwTigrAN8Bzukdj4idBvyCBmEov+QXp31kcVSbxF9sMUl6j6QZuZX2uKTjcrFr8vMzueX1XklLSPqapIclzZV0mqSVCuvdL897UtJ/NdQzVdL5kn4h6VnggFz3DZKekTRH0gmSlimsLyQdIuk+Sc9J+pakCZKuz/GeWyzf8BqbxippWUnPA0sCt0p6oK/tFRH3AH8E3lloZRwk6W/AH3J9B0q6O7eWL5U0vhDLv0m6R9I/JJ0AqDBvgV9jkjbIremn8vvxFUk7Al8B/j2/F7fmsitJ+lnedrMlfbs3YUlaUtKxkp6Q9CCwc1+vE9gPuBGYDuzfZnu8BJwLbNRi/j+B04HlgXVL1Dsk8ns+RdIDeR88V9KYwvzzJD2W34drJG1QmDdd0omSfiPpBWCbvP9+UdJteZlzJI0oLLOLpFvy/nu9pA0L82ZJ+rKk24AX+kr+ed/ZpTC+lKR5kjbpK/aG9Wwt6dHC+MaSZubPzzlAMf7Rki7O9Tydh9fI844iNQJOyPvcCXl6SHp7Hl4pf67m5c/Z1yQtkecdIOnavA8+LekhSS2/xPO2mp3jvFfSdiXe04VyVLtt3FJEDPsHMAvYvmHaAcC1zcoANwD75uEVgM3ycA+p5bdUYbkDgfuBdXLZXwKn53kTgeeBLYFlSF0prxXqmZrHdyd9yS4HvBvYDFgq13c3cHihvgB+DawIbAC8AlyR618JuAvYv8V2aBlrYd1vb7Md35ifX9tjwEGF7XIaKbEtB+yW6/qX/Fq+Blyflx0LPAfsCSwNfB54Hfhk43sDjALmAF8gfUBHAZsWtt8vGmK8EPhxjuOtwJ+BT+V5BwP3AGsCY4ArG9/PJq/5fuCQ/L68BqxSmDcd+HYeXp6U2G9tto+RvlQPBV4F3trP/Xeh11lmH8/TP0f64loDWDZvm7Ma9olRed4PgFsaXt8/gC1I++eIXM+fgdXyNrwbODiX3xiYC2yaX+/+ufyyhRhvydt/uRKv++vAGYXxnYG7+xF773uzNfBoHl4GeJi0zy1N2gdfK5R9C/ARYGRe93nArwrrvYq8n7b4XJxG+nyOIn0u/gocVNgfXiP9Sl4S+DTwd0BNXvv6wCPAaoXcM6Gv95QmOWpAOXMwCy8qj7zDPQ88U3i8SOvEfw1wJDC2YT0LbVRS0j2k4Q17jZTsvs6CH7KRpA9+MfFf00fshwMXNuxkWxTGbwa+XBj/HvCDFutqGWvjDtxi+QCeBZ4GHgC+TUoIvdtlnULZ3/bu8Hl8ibzNx5Nb0YV5Ah6leeLfG/hLi3imUkiIwCqkL8LlCtP2Bq7Mw38gJ6k8vkO7DwnpC/u13v2A9KXx+cL86cDLeX/6J/AQsGFh/gGkL7Rn8npeAj46gP13gdfZZh9vlvjvBrYrjK9afM8byq6ct8dKhdd3WpN6PlEYPxo4KQ+fCHyrofy9wPsLyx7Yj9f9dlIDYWQePwP4eouyzWJvlvi3oiHZAtf3lm2y3o2ApwvjV9Ei8ZOS+avAxMK8TwFXFfaH+wvzRuZl39bitc8FtgeWLvueMkSJf3Hq6tk9IlbufZBaca0cBKwH3CPpJrU/ILcaqQXR62HSG7BKnvdI74yIeBF4smH5R4ojktbLPy8fU+r++Q6phVz0eGH4pSbjKwwg1rI2iYjRETEhIr4WqQuj2WsZDxyff/I/AzxFSvCrs/B2iYZli9YkfcmUMZ7UiptTqPfHpJY/jfWy4LZoZn/gsoh4Io+fycLdPcfm/amHtO0bDyLemOePBi4idRUMiqTf6s0Dvfv0UXw8cGFhe9wNzAdWyV1f03KXwbOkxAwL7m/N3pfHCsMv8ub+Nh74Qm9dub41Sdu93fqaioj7c7y7ShpJOsZyJrzRbddX7M2sBszO+1yvN/YDSSMl/Th30zxLagSurHLHN8aS9r/Gz9jqhfE3tl3OB9Dk85pf++GkL/25ks6W1LsdW76nJWIsZXFK/KVFxH0RsTcpYXwXOF/S8qRv0kZ/J70RvdYitfIeJ3VRrNE7Q9JypJ+SC1TXMH4iqWW5bkSsSOrHFkOjXaxDofhaHiF1saxceCwXEdeTtsuavQUlqTje4BFS11Rf9fWWfYXUQu+tc8WI6O37XaBe0utvKr9XHwXen7+EHyN1D7xL0rsWCiTib6Sf4MfnZRvnP0/6ab+vBnmqbETsFG8e6D2jj+KPADs1vA8jImI26eD8bqRW5UqkLy9YcH9rts+3q+uohrpGRsRZA1wfwFmkX227AXflhEjJ2JuZA6ye97lexf3gC6Qv703z52+rhvW2i/8JUsu78TM2u4+YmoqIMyNiy7y+IOUiaP+e9nf7NlXLxC/pE5LG5dbsM3nyP4F5+bmYiM4CPi9pbUnFMzBeB84ntVY2VzrgOpW+d8xRpO6U55VODfz0UL2uPmIdaicB/9l7wC0f9Norz7sE2EDSHvkA32HA21qs52JgVUmHKx2EHiVp0zzvcaCn9+BZRMwBLgO+J2nFfBBsgqT35/LnAodJWkPSaGBKm/h3J7WiJpJ+7m9EOl7xR1JX1UIi4nLSl+vkFvOfAn5K6gIkb5dZkg5oE0d/LC1pROGxFOl9OEr5wLqkcZJ2y+VHkb4onyR1O3xnkPX/BDhY0qZKlpe0s6RRrRZQOoA8vc06zyZ1yX2a3NofZOw3kBo7h0laWtIewHsa1vsS6eDoGOAbDcs/TouGSETMJ+1jR+X9dDxwBNDvU44lrS9pW6VTiF/OMfX+um73njbLUf1Wy8QP7AjcqXSmy/HAxyLipfzT7CjguvwzazPgFNJBvWtIfbwvA58FiIg78/DZpJbG86R+u1fa1P1FUmvmOdIH6ZwhfF0tYx1qEXEhqYVydv7JfAewU573BLAXMI30wV0XuK7Fep4D/g3YlfQz+T5gmzz7vPz8pKSZeXg/0gG8u0jHIs4n9YFC2p6XArcCM0kHt1vZH/h5RPwtIh7rfQAnAPuo9RkpxwD/Vw3n/Bf8APigpA1zY+AtpAN1Q+E3pATR+5hK2n8vAi6T9Fyuq/eL8zRSV8Rs0vYaVBwRMYN04PIE0ra/n9Sv3c6atHjv8zrnkJL15iz4WRhQ7BHxKrBHjusp4N9ZcD/4AenkhCfyOn/XsIrjgT2Vzsr5YZMqPgu8ADwIXEv6sjqlTGwNliV9Pp4g7fdvBf6zEEPT97RFjuo3LdgVZoORW9nPkLpxHup2PNZdkrYEDs3dirWTv/huJR0Qf63b8dibnPgHSdKupLNpRDrjZlPSAVJvWDNbJNW1q2co7Ubq9/07qUvjY076ZrYoc4vfzKxm3OI3M6uZYXEhpbFjx0ZPT0+3wzAzG1ZuvvnmJyJiXOP0YZH4e3p6mDFjRrfDMDMbViQ1/fe6u3rMzGrGid/MrGac+M3MasaJ38ysZpz4zcxqxonfzKxmKkv8ktaUdKWkuyTdKelzefpUpftM3pIfH6wqBjMzW1iV5/G/DnwhImbm63XfLOnyPO/7EXFshXWbmVkLlSX+fJ3tOXn4OUl3s+AtyszMrAs68s9dST3AxsCfgC2Az0jaD5hB+lXwdJNlJpPvdLTWWi3voGcGQM+US7pS76xpO3elXrPBqPzgbr45yQXA4RHxLOmesxNIt7qbQ7qG/UIi4uSImBQRk8aNW+hSE2ZmNkCVJn5JS5OS/hkR8UuAiHg8Iubn+93+hAXvh2lmZhWr8qweAT8D7o6I4wrTVy0U+zDpXq1mZtYhVfbxbwHsC9wu6ZY87SvA3pI2AgKYBXyqwhjMzKxBlWf1XEu6D22j31RVp5mZ9c3/3DUzqxknfjOzmnHiNzOrGSd+M7OaceI3M6sZJ34zs5px4jczqxknfjOzmnHiNzOrGSd+M7OaceI3M6sZJ34zs5px4jczqxknfjOzmnHiNzOrGSd+M7OaceI3M6sZJ34zs5px4jczq5kqb7ZuZhXqmXJJV+qdNW3nrtRrQ8ctfjOzmnHiNzOrGSd+M7OaceI3M6sZJ34zs5px4jczqxknfjOzmvF5/DZkunVeuZn1j1v8ZmY148RvZlYzTvxmZjXjxG9mVjOVJX5Ja0q6UtJdku6U9Lk8fYykyyXdl59HVxWDmZktrMoW/+vAFyJiIrAZcKikicAU4IqIWBe4Io+bmVmHVJb4I2JORMzMw88BdwOrA7sBp+ZipwK7VxWDmZktrCN9/JJ6gI2BPwGrRMScPOsxYJUWy0yWNEPSjHnz5nUiTDOzWqg88UtaAbgAODwini3Oi4gAotlyEXFyREyKiEnjxo2rOkwzs9qoNPFLWpqU9M+IiF/myY9LWjXPXxWYW2UMZma2oCrP6hHwM+DuiDiuMOsiYP88vD/w66piMDOzhVV5rZ4tgH2B2yXdkqd9BZgGnCvpIOBh4KMVxmBmZg0qS/wRcS2gFrO3q6peMzNrz//cNTOrGSd+M7OaceI3M6sZJ34zs5px4jczqxknfjOzmnHiNzOrGSd+M7OaceI3M6sZJ34zs5rpV+KXNFrShlUFY2Zm1evzWj2SrgI+lMveDMyVdF1EHFFxbGaLvJ4pl3Q7BLN+K9PiXynfQGUP4LSI2BTYvtqwzMysKmUS/1L5hikfBS6uOB4zM6tYmcR/JHApcH9E3CRpHeC+asMyM7OqlLke/5yIeOOAbkQ8KOm4dguYmdmiq0yL/0clp5mZ2TDQssUv6b3A5sA4ScUzeFYElqw6MDMzq0a7rp5lgBVymVGF6c8Ce1YZlJmZVadl4o+Iq4GrJU2PiIc7GJOZmVWozMHdZSWdDPQUy0fEtlUFZWZm1SmT+M8DTgJ+CsyvNhwzM6tamcT/ekScWHkkZmbWEWVO5/wfSYdIWlXSmN5H5ZGZmVklyrT498/PXypMC2CdoQ/HzMyq1mfij4i1OxGImZl1Rp9dPZJGSvpaPrMHSetK2qX60MzMrApl+vh/DrxK+hcvwGzg25VFZGZmlSqT+CdExNHAawAR8SKgSqMyM7PKlEn8r0pajnRAF0kTgFcqjcrMzCpT5qyebwC/A9aUdAawBXBAlUGZmVl1ypzVc7mkmcBmpC6ez0XEE5VHZmZmlWjZ1SPpHfl5E2A8MAf4O7BWntaWpFMkzZV0R2HaVEmzJd2SHx8c/EswM7P+aNfiPwKYDHyvybwA+rpI23TgBOC0hunfj4hjywZoZmZDq91lmSfnwZ0i4uXiPEkj+lpxRFwjqWdQ0ZmZ2ZArc1bP9SWnlfUZSbflrqDRg1iPmZkNQLs+/rdJejewnKSNJW2SH1sDIwdY34nABGAj0jGDZt1IvfVPljRD0ox58+YNsDozM2vUro//A6TTNtcAjitMfw74ykAqi4jHe4cl/QS4uE3Zk4GTASZNmhQDqc/MzBbWro//VOBUSR+JiAuGojJJq0bEnDz6YeCOduXNzGzolTmP/wJJOwMbACMK07/ZbjlJZwFbA2MlPUr6I9jWkjYinRU0C/jUgCM3M7MB6TPxSzqJ1Ke/Den2i3sCf+5ruYjYu8nkn/U3QDMzG1plzurZPCL2A56OiCOB9wLrVRuWmZlVpUzifyk/vyhpNdJVOletLiQzM6tSmYu0XSxpZeAYYCapf/4nlUZlZmaVKXNw91t58AJJFwMjIuIf1YZlZmZVKXNwdwRwCLAlqbV/raQTGy/jYGZmw0OZrp7TSH/a+lEe/zhwOrBXVUGZmVl1yiT+d0bExML4lZLuqiogMzOrVpmzemZK2qx3RNKmwIzqQjIzsyqVafG/G7he0t/y+FrAvZJuByIiNqwsOjMzG3JlEv+OlUdhZmYdU+Z0zoc7EYiZmXVGmT5+MzNbjLS7EcuynQzEzMw6o12L/wYASad3KBYzM+uAdn38y0j6OLC5pD0aZ0bEL6sLy8zMqtIu8R8M7AOsDOzaMC8AJ34zs2Go3a0XryVdl2dGRPgGKmZmi4ky5/GfLukwYKs8fjVwUkS8Vl1YZmZWlTKJ/7+BpfMzwL7AicAnqwrKzMyqUybx/2tEvKsw/gdJt1YVkJmZVavMH7jmS5rQOyJpHWB+dSGZmVmVyrT4v0S6FPODgIDxwH9UGpWZmVWmzLV6rpC0LrB+nnRvRLxSbVhmZlaVMi1+cqK/reJYzMysA3yRNjOzmnHiNzOrmT4Tv6QrykwzM7PhoWUfv6QRwEhgrKTRpDN6AFYEVu9AbGZmVoF2B3c/BRwOrAbczJuJ/1nghIrjMjOzirS7SNvxwPGSPhsRP+pgTGZmVqEy5/H/SNLmQE+xfEScVmFcZmZWkT4Tf74D1wTgFt68VEMATvxmZsNQmT9wTQImRkRUHYyZmVWvzHn8dwBv6++KJZ0iaa6kOwrTxki6XNJ9+Xl0f9drZmaDUybxjwXuknSppIt6HyWWmw7s2DBtCnBFRKwLXJHHzcysg8p09UwdyIoj4hpJPQ2TdwO2zsOnAlcBXx7I+s3MbGDKnNVz9RDWt0pEzMnDjwGrtCooaTIwGWCttdYawhDMzOqtzCUbnpP0bH68LGm+pGcHW3E+WNzygHFEnBwRkyJi0rhx4wZbnZmZZWVa/KN6hyWJ1F2z2QDre1zSqhExR9KqwNwBrsfMzAaoX1fnjORXwAcGWN9FwP55eH/g1wNcj5mZDVCZP3DtURhdgnRe/8slljuLdCB3rKRHgW8A04BzJR0EPAx8dAAxm5nZIJQ5q2fXwvDrwCxSd09bEbF3i1nblajTzMwqUqaP3zdWNzNbjJQ5q2cNSRfmf+HOlXSBpDU6EZyZmQ29Mgd3f046KLtafvxPnmZmZsNQmcQ/LiJ+HhGv58d0wCfWm5kNU2US/5OSPiFpyfz4BPBk1YGZmVk1yiT+A0mnXT4GzAH2BHzA18xsmCpzVs/DwIc6EIuZmXVAmbN6TpW0cmF8tKRTqg3LzMyqUqarZ8OIeKZ3JCKeBjauLiQzM6tSmcS/RPFOWZLGUO4fv2Zmtggqk8C/B9wg6bw8vhdwVHUhmZlZlcoc3D1N0gxg2zxpj4i4q9qwzMysKqW6bHKid7I3M1sM9Ot6/GZmNvw58ZuZ1YwTv5lZzTjxm5nVjBO/mVnNOPGbmdWM/4G7GOqZckm3Q7DFWDf3r1nTdu5a3YsTt/jNzGrGid/MrGac+M3MasaJ38ysZpz4zcxqxonfzKxmnPjNzGpmsT+P3+ccm5ktyC1+M7OaceI3M6sZJ34zs5px4jczq5muHNyVNAt4DpgPvB4Rk7oRh5lZHXXzrJ5tIuKJLtZvZlZL7uoxM6uZbiX+AC6TdLOkyc0KSJosaYakGfPmzetweGZmi69uJf4tI2ITYCfgUElbNRaIiJMjYlJETBo3blznIzQzW0x1JfFHxOz8PBe4EHhPN+IwM6ujjid+SctLGtU7DOwA3NHpOMzM6qobZ/WsAlwoqbf+MyPid12Iw8ysljqe+CPiQeBdna7XzMwSn85pZlYzTvxmZjXjxG9mVjNO/GZmNePEb2ZWM078ZmY148RvZlYzTvxmZjXjxG9mVjNO/GZmNePEb2ZWM9289eJir2fKJd0OwWyx0q3P1KxpO3el3qq4xW9mVjNO/GZmNePEb2ZWM078ZmY148RvZlYzTvxmZjXjxG9mVjM+j9/MrA/d/E9OFf8hcIvfzKxmnPjNzGrGid/MrGac+M3MasaJ38ysZpz4zcxqxonfzKxmnPjNzGrGid/MrGac+M3MasaJ38ysZpz4zcxqxonfzKxmupL4Je0o6V5J90ua0o0YzMzqquOJX9KSwP8HdgImAntLmtjpOMzM6qobLf73APdHxIMR8SpwNrBbF+IwM6ulbtyIZXXgkcL4o8CmjYUkTQYm59HnJd07gLrGAk8MYLlFwXCOHYZ3/I69e4Zz/JXEru8OavHxzSYusnfgioiTgZMHsw5JMyJi0hCF1FHDOXYY3vE79u4ZzvEPp9i70dUzG1izML5GnmZmZh3QjcR/E7CupLUlLQN8DLioC3GYmdVSx7t6IuJ1SZ8BLgWWBE6JiDsrqm5QXUVdNpxjh+Edv2PvnuEc/7CJXRHR7RjMzKyD/M9dM7OaceI3M6uZYZ/4JY2RdLmk+/Lz6Bbl9s9l7pO0f2H6VfnyEbfkx1s7EHPbS1ZIWlbSOXn+nyT1FOb9Z55+r6QPVB1rk9gGFLukHkkvFbbzSZ2OPcfRV/xbSZop6XVJezbMa7oPdcogY59f2PYdP5miROxHSLpL0m2SrpA0vjBvUd/u7WLv6nZvKSKG9QM4GpiSh6cA321SZgzwYH4enYdH53lXAZM6GO+SwAPAOsAywK3AxIYyhwAn5eGPAefk4Ym5/LLA2nk9Sw6T2HuAO7q8r5SJvwfYEDgN2LPMPrSox57nPb+Ib/dtgJF5+NOF/WY4bPemsXd7u7d7DPsWP+lyD6fm4VOB3ZuU+QBweUQ8FRFPA5cDO3YovkZlLllRfE3nA9tJUp5+dkS8EhEPAffn9XXKYGJfFPQZf0TMiojbgH82LNvtfWgwsXdbmdivjIgX8+iNpP/3wPDY7q1iX2QtDol/lYiYk4cfA1ZpUqbZZSJWL4z/PP8U+68OJKm+YlmgTES8DvwDeEvJZas0mNgB1pb0F0lXS3pf1cE2MZjtNxy2fTsjJM2QdKOkZo2jKvU39oOA3w5w2aE2mNihu9u9pUX2kg1Fkn4PvK3JrK8WRyIiJPX3/NR9ImK2pFHABcC+pJ/KNrTmAGtFxJOS3g38StIGEfFstwOrifF5P18H+IOk2yPigW4H1UjSJ4BJwPu7HUt/tYh9kdzuw6LFHxHbR8Q7mzx+DTwuaVWA/Dy3ySpaXiYiInqfnwPOpPqukzKXrHijjKSlgJWAJ0suW6UBx567p54EiIibSf2m61UecYvYsv5sv+Gw7Vsq7OcPko5rbTyUwfWhVOyStic15j4UEa/0Z9kKDSb2bm/31rp9kGGwD+AYFjy4e3STMmOAh0gHh0bn4TGkXzxjc5mlSX3SB1cc71KkA1Rr8+bBog0ayhzKggdIz83DG7Dgwd0H6ezB3cHEPq43VtKBstnAmA7vK33GXyg7nYUP7i60Dw2T2EcDy+bhscB9NByg7HbspIT4ALBuw/RFfru3ib2r273t6+p2AEPwxrwFuCJv1N/37hSkn1w/LZQ7kHQw9H7gP/K05YGbgduAO4HjO5FIgQ8Cf807y1fztG+SWgsAI4Dzcqx/BtYpLPvVvNy9wE5d2N4Dih34SN7GtwAzgV27tL/0Ff+/kvpxXyD9yrqz3T40HGIHNgduz0nrduCgRTD23wOP5/3jFuCiYbTdm8a+KGz3Vg9fssHMrGaGRR+/mZkNHSd+M7OaceI3M6sZJ34zs5px4jczqxknflvkSdpa0sV5+EPNrpBYKLuypEMGUMdUSV8sUa6v+g+QdEJ/6y9D0ixJYwewXI+kjxfGJ0n64dBGZ8OJE791jaQl+7tMRFwUEdPaFFmZdIXQISdpqRL193udQ7WuNnqANxJ/RMyIiMM6UK8topz4bcjlFuY9ks6QdLek8yWNzPNmSfqupJnAXpJ2kHRDvo78eZJWyOV2zOuYCexRWPcbLWpJq0i6UNKt+bE5MA2YkC+6d0wu9yVJN+XrpR9ZWNdXJf1V0rXA+i1ey3RJJ0n6E3B0Q/17Sboj131Nk2V3zq9tbMP0qZJOl3QdcLqkcZIuyDHeJGmLXO4tki6TdKeknwIqbN87Cuv7oqSpefjtkn6fY5opaULeJu/L2+TzDb+gxkj6Vd42N0rasBDjKUr3q3hQkr8oFiPD4iJtNiytT/qn4nWSTiG1wo/N856MiE1yQvwlsH1EvCDpy8ARko4GfgJsS/q35jkt6vghcHVEfDj/eliBdNmOd0bERgCSdgDWJV2DScBFkrYi/bv1Y8BGpM/BTNK/uJtZA9g8IuZLOqAw/evAByJdhGvl4gKSPgwcAXww0uWEG00EtoyIlySdCXw/Iq6VtBZwKfAvwDeAayPim5J2Jl35sS9nANMi4kJJI0iNuynAFyNilxzb1oXyRwJ/iYjdJW1LukDhRnneO0jXmh8F3CvpxIh4rUQMtohz4reqPBIR1+XhXwCH8Wbi703km5ES4HVKV8NeBriBlHAeioj7ACT9ApjcpI5tgf0AImI+8A8tfAe2HfLjL3l8BdIXwSjgwsjXUVf7uyOdl9ff6DpguqRzSV9gxbgmATtE66uPXhQRL+Xh7YGJevOK4CvmXz5bkX/tRMQlkpp9gbxB6Qqzq0fEhXmZl/P0dottSbqcBhHxh/wrY8U875JIFxx7RdJc0iXPH223MhsenPitKo3XAimOv5CfRbrJxt7FgpI2YugI+H8R8eOGOg7vxzpeaDYxIg6WtCmwM3Cz0uWm4c07Nq0HzCixziWAzXoTdSHGVvG8zoLdtCPaRj9wrxSG5+N8sdhwH57aWfoAAAGMSURBVL9VZS1J783DHweubVLmRmALSW8HkLS8pPWAe4Ce3D8NsHeTZSFdnO/TedklJa0EPEdqzfe6FDiwcOxgdaX7Kl8D7C5pudxS3rW/L1DShIj4U0R8HZjHm5fvfZjUij5N0gYlVnUZ8NnCenu/+K4hH5SVtBPpao+QLgj21tw6XxbYBd64tPijyjf8ULr/8UgW3iZFfwT2yeW3Bp5o8yvFFhNO/FaVe4FDJd1NSlgnNhaIiHnAAcBZkm4jd/Pklu9k4JJ8cLfZPRYAPgdsI+l2Uv/8xEjX/L8uH3Q9JiIuI91n4YZc7nxgVETMJHU53Uq6Y9JNA3iNx0i6PR9ovT6vq/e13UNKqOcVvsBaOQyYlA+w3gUcnKcfCWwl6U5Sl8/f8rpfI10d8s+kWxHeU1jXvsBheXteT7qB0W3A/HzA9/MNdU8F3p3LTwM6fjNz6zxfndOGnKQe4OKIeGeXQzGzJtziNzOrGbf4zcxqxi1+M7OaceI3M6sZJ34zs5px4jczqxknfjOzmvlflXnXT1x+OtEAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "V89cP4pxQhNo"
      },
      "source": [
        "Notice when viewing the histogram that predicted risk reduction can be negative.\n",
        "- This means that for some patients, the T-learner predicts that treatment will actually increase their risk (negative risk reduction). \n",
        "- The T-learner is more flexible compared to the logistic regression model, which only predicts non-negative risk reduction for all patients (view the earlier histogram of the 'predicted ARR' histogram for the logistic regression model, and you'll see that the possible values are all non-negative)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "noMOc9kOI5cw"
      },
      "source": [
        "Now plot an empirical risk reduction plot for the validation set examples. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:32:38.119651Z",
          "start_time": "2020-04-04T15:32:37.941488Z"
        },
        "colab_type": "code",
        "id": "S-0nbpSkJFmZ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 458
        },
        "outputId": "13afaa75-71e8-4f7f-fa25-78da6cefe18a"
      },
      "source": [
        "empirical_benefit, avg_benefit = quantile_benefit(X_val, y_val, rr_t_val)\n",
        "plot_empirical_risk_reduction(empirical_benefit, avg_benefit, 'T Learner [val set]')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEWCAYAAABBvWFzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3deZxVdf3H8ddbxBjcEEETEKEiFTfQcYv0ZymBaYpaQVYumdhiaYulbT/rV0n5q36iZK5hWSr5Q8Qt3JefuTAIihvhLouJIi44xOLn98f5Dt65c+fOHeYuM/h+Ph7zmLPd8/3cc8+9n3O+55zvVxGBmZlZR21Q6wDMzGz94IRiZmZl4YRiZmZl4YRiZmZl4YRiZmZl4YRiZmZl4YRShKS3JH2gyPw/SPpxB8s4QNKCDrz+JknHlrBcSPrQupazriTdKenLFVjvDyRdXO71dkW5n2059skSyzxO0v+1scy9koZXOpbOStKZklal35GNy7C+in+HJX04xbum6Xsr6RuSflXK67tcQpH0nKTG9Kab/s6rRFkRsUlEPFNk/lci4r8qUXaTtBMtT+9zoaTfSuqWE8PBEXFZGcsblMps2rbPSTq9XOtfx5haJN2I+GVElD1RVUKK/520Pd+UNE/S8ZUoq9R9slKJPmf9nwLejIjZKck17U8rc35k35J0U4HXduggq5O5Kv2OLK91IIXkJ6mI+GdEbALck7PYRcDnJW3V1vq6XEJJPpU+pKa/k6sdQO6PehXslj7k/wDGAl+qQpm9UpmfBn4saWQVylyfLUrbczPg+8BFkobmLyRpw6pHVhlfAf4Ma5PcJun9/5J3f2Q3iYiDaxFcubfzevS5tRARK4CbgGPaWrarJpSC0mn4vZJ+J2mZpGckfSRNf1HSy7nVQ5Imp6OnW9KR412StsuZn1uVMFnS+ZJulLQc+Fia9vOc5Q+XNEfSG5KeljQ6TT9e0hOpjGcknbQu7y8ingLuBYbllLn2SFPSh9J7eF3SK5KuamU7fTRtjwNKKLMBeCyvzC+l9/OapBl522ykpCdTDOcBypl3pqTLc8abzoY2TOO9Jf1R0qK07mmpquAmoF/OUW2/Aus6TNJj6XO/U9KOOfOek/RdSY+kuK6S1KPAdnlfev3OOdP6Kjsj3kpSH0nXp2WWSrpHUru+Q5GZBrwGDM3bZ18Fzkxx/LekFyT9K+2jdTkxnSZpcdpOzQ4uStknJf0C2A84Tzln+JJ2SN+FpcrOoj6bs54tJU1P63kQ+GBr71HSRsDHgbvas21KkT77/5W0RNKzkr6ZM28vSfelz2expPNSLE3zQ9LXJc0H5iudCUn6jrLfhsXKOXMs9jnkvPb7kl4C/lhC7HtLekk5B6OSjpD0SCnxt7Hu45T9tryZtsvnc+YV/L5Kujst8nDaD8YWKeJO4JC24livEkqyN/AIsCXwV+BKYE/gQ8AXyL5Em+Qs/3ngv4A+wBzgL0XWfTTwC2BToFn9saS9gD8BpwG9gP2B59Lsl4FDyY5Ojwd+J2n39r4xSTuQ/RA81coi/wXcDGwBDADOLbCO0cAVwFERcWcJZe4D7NxUpqTDgR8ARwJ9yU6Nr0jz+gBTgR+Rbc+ngRGlvj+yI9qewE7AVsDvUlXBwaQj/PS3KC/GD6cYTk0x3Qhcl/dl/CwwGhgM7Aocl194RPw7xf+5vNfdFREvA98BFqQytk7boV1tF0naQNIRZPvI3DR5b+CZtM5fABOAD5Ml8Q8B/YGfpNePBr4LjASGAAcVKavgPhkRPyT73E5uOsNXlrhvIfvObAWMA36vd8+iJgErgG3IzpCLnSUPAd6JiLJWW6XkfR3wMNk2ORA4VdKotMga4Ftk+96+af7X8lYzhmx7N72v9wObp/WdAEyStEWa1+rnkPPa3sB2wPi24o+IB4DlZMm2ydFk27zU+FtIn91E4OCI2BT4CNlvWdHva0Tsn1axW9oPCh6AJk8Au7UVCxHRpf7IfqTfApbl/J2Y5h0HzM9ZdheyL/zWOdNeBYal4cnAlTnzNiH7ULdN4wF8KGfZP+XFMhn4eRq+gOwHsJT3MA04JQ0fACwosmwAb5DtiJF2hvflzL8T+HIa/hNwITCglfWcATwP7FykvEFp2WVAYxr+b0Bp/k3ACTnLbwC8TfalOga4P2eeyH6Am+I7E7i8QFkbkv1QvQNsUSCmFtsod13Aj4EpeTEtBA7I2We+kDP/18AfWnn/BwFP54zfCxyThn8GXNu0T7Rjnz0gvbdlwFKyL/u4nH32hbxtthz4YM60fYFn0/ClwISceR+m5X7a5j6Zu9+k8bHAPXnLXAD8J9ANWAXskDPvl8D/tbLuEcBLrcxrtg8U2V4tvhNkieCFvGlnAH9sZT2nAtfkfQc+nldOI7BhzrSXgX1K+BwOAFYCPYq8jxbvFfg5cGka3jSVsV074m+x7wEbp33rKKAub16r39c21pm/fwwB1rS1r3fVM5QxEdEr5++inHn/yhluBIiI/Gm5ZygvNg1ExFtkX/h+rZT7YivTAbYlOyJvQdLBku5PVQnLgE+SHYWUavcU81iyL1Vrd4x8j+yL8KCy6p/8o8hTyX54Hy2hzD6pzO+QfXm6p+nbAeek0/KmH0iRHb31o/n2DIpvs1zbAksj4rUSl8/VjyxRNpX7Tiq3f84yL+UMv03zfSDXHUDPVD0xiOzo9Jo072yyM7WbU/VCe25WWJT21d4RMSwirsyZl7uN+pKdpc3K2cZ/T9MhbxuT874LaHWfLGA7YO+mMlO5nyc7Cu9LlvRLLfc1sh/LctuOrOozN8YfkJ3ZNd2hdH2qVnqDLOnlf8/y98dXI2J1znjTvtHW5wCwJLLrC+3xV+BISe8jO2t4KCKeb0f8LUR2Fj+W7LrVYkk3pNoMKP59bY9NgdfbWqirJpRy2rZpIFWF9QYWtbJsseqNFylQr5x2nP8lO8rfOiJ6kVXJKH/ZYiIzBbiP5qfducu8FBEnRkQ/4CSyKovc2ww/A4yRdEqJZa6JiN+SVXU0nXq/CJyUl9DrIuIfwGKab0/ljpMdjfXMGX9/zvCLQG9JvQqF0kaoi8i+OPnlLmzjdS0LilgDTCGr9voccH1EvJnmvRkR34mIDwCHAd+WdGB7yyhUbM7wK2QHPTvlbN/NI7ugDXnbGBhYZL0F98kCZTYte1fe57pJRHwVWAKsbke5T5F9DO390WrLi2RnCLkxbhoRn0zzzweeBIZExGZkySb/e1ZqFWVbn0N71vXuCyIeJ0vGB9O8uqvU+Ftb74yIGEl2pv8k2Z1ZUPz72h47klU1FuWEAp9UdpF6I7JrEPdHRKlH1bkuAY6XdGCqJ++fjhI2At5H+lJKOhj4RAfinQCcKOn9+TMkfUbSgDT6GtkO/07OIovI6mVPkfTVdpb5PWUXsv8AnCFpp1Tm5pI+k5a7AdhJ0pHKLrR/k+ZJYw6wv6SBkjYnq64AICIWk52e/17SFpK6S2qq4/0XsGV6TSFTgEPStu9Odlb1b6C9X5omfyU74vs8OV94SYcqu/FBZEdra2i+fTssnV1dRHadbatUbv+c6wRTgOMkDZXUk6xKqjWt7ZOQbdPcZ6yuBz4s6Ytp23eXtKekHVOSnUp2w0DPdF2l1WefImIlcCvZXYnrTFKP3D/gQeBNZRfC6yR1k7SzpD3TSzYlqx5+K73P9uzj+e+hrc+hI/4KnEJ2TetvOdPXKX5JWyu7+WJjsv3+Ld7dL4t9X6HlftCa/yD7fhbVVRPKdWr+HMo1bb+kVX8l+1IuBfYgu3DfbhHxIOmCO9mPzV1k9ZRvkv2wTiH7kT8amL6uwUbEXOBusgut+fYEHpD0VirjlMh7jiYiXiBLKqer9OcQbkixnxgR1wC/Aq5Mp+WPkh1tERGvkJ0FTSC7VjWE7BpEU9m3AFeR3TQxi+xHLNcXyerqnySryz41ve5JsmtHz6RT92ZVkhExj+xzO5fsyPJTZLeWryzx/TUT71487UfzL9EQsh/Kt8jOFH8fEXfA2gdMf7Au5RXwfbKj/PvTNr4V2D7FdhPwP8DtaZnbi7yPgvtkmn0O8Glld/5MTPvpJ8guxi8iqyL8FdnBEMDJZFVBL5Fdp2nrrqYLyD7PddWf7Awh928w2c0tw4BnyT7ri8kuqkN2s8LRwJtkyaDYReZStPo5dNAVZD/Qt6fvTJN1jX8D4Ntkn9vStO6vAhT7viZnApel79VnKSAl808CbT7v1nSh9T1J0mSyi38/qnUsZusbSfeS3Uk2u9ax1IKkH5Gdha8C+kcnfbgxl6QhwEyympWvRcRkSd8gu1Hpe22+3gnFCcXMrBy6apWXmZl1Mu/pMxQzMysfn6GYmVlZrHcNmvXp0ycGDRpU6zDMzLqUWbNmvRIRfdtesnXrXUIZNGgQDQ0NtQ7DzKxLkVSs9YOSuMrLzMzKwgnFzMzKwgnFzMzKwgnFzMzKwgnFzMzKoqYJRdKlyrreLNg/hzITJT2lrPvWdvdy2JVMm72QERNuZ/DpNzBiwu1Mm93u1tfNzGqm1mcok8m6ZW3NwWQtvA4h62Lz/CrEVBPTZi/kjKlzWbiskQAWLmvkjKlznVTMrMuoaUKJiLvJmltuzeFk3e5GRNwP9JK0TXWiq66zZ8yjcdWaZtMaV63h7BnzahSRmVn71PoMpS39ad5l5wIKdF0pabykBkkNS5YsqVpw5bRoWWO7ppuZdTadPaGUJCIujIj6iKjv27dDLQfUTL9ede2abmbW2XT2hLKQ5v1YD2Ad+gnvCk4btT113bs1m1bXvRunjSpHB3FmZpXX2RPKdOCYdLfXPsDrqe/x9c6Y4f0568hd6N+rDgH9e9Vx1pG7MGZ4ixo+M7NOqaaNQ0q6AjgA6CNpAVnf7t0BIuIPwI1kfRk/BbxN1j/2emvM8P5OIGbWZdU0oUTE59qYH8DXqxSOmZl1QGev8jIzsy7CCcXMzMrCCcXMzMrCCcXMzMrCCcXMzMrCCcXMzMrCCcXMzMrCCcXMzMrCCcXMzMrCCcXMzMrCCcXMzMrCCcXMzMrCCcXMzMrCCcXMzMrCCcXMzMqipglF0mhJ8yQ9Jen0AvMHSrpD0mxJj0j6ZC3iNDOzttUsoUjqBkwCDgaGAp+TNDRvsR8BUyJiODAO+H11ozQzs1LV8gxlL+CpiHgmIlYCVwKH5y0TwGZpeHNgURXjMzOzdqhlQukPvJgzviBNy3Um8IXU3/yNwDcKrUjSeEkNkhqWLFlSiVjNzKwNnf2i/OeAyRExAPgk8GdJLWKOiAsjoj4i6vv27Vv1IM3MrLYJZSGwbc74gDQt1wnAFICIuA/oAfSpSnRmZtYutUwoM4EhkgZL2ojsovv0vGVeAA4EkLQjWUJxnZaZWSdUs4QSEauBk4EZwBNkd3M9Julnkg5Li30HOFHSw8AVwHEREbWJ2MzMitmwloVHxI1kF9tzp/0kZ/hxYES14zIzs/br7Bflzcysi3BCMTOzsnBCMTOzsnBCMTOzsnBCMTOzsnBCMTOzsnBCMTOzsnBCMTOzsnBCMTOzsnBCMTOzsmiz6RVJ7wOOAgblLh8RP6tcWGZm1tWU0pbXtcDrwCzg35UNx8zMuqpSEsqAiBhd8UjMzKxLK+Uayj8k7VLxSMzMrEsr5Qzlo8Bxkp4lq/ISEBGxa0UjMzOzLqWUhHJwpQqXNBo4B+gGXBwREwos81ngTCCAhyPi6ErFY2Zm667NhBIRz0vaDdgvTbonIh7uaMGSugGTgJHAAmCmpOmpU62mZYYAZwAjIuI1SVt1tFwzM6uMNq+hSDoF+AuwVfq7XNI3ylD2XsBTEfFMRKwErgQOz1vmRGBSRLwGEBEvl6FcMzOrgFKqvE4A9o6I5QCSfgXcB5zbwbL7Ay/mjC8A9s5b5sOpzHvJqsXOjIi/d7BcMzOrgFISioA1OeNr0rRq2BAYAhwADADulrRLRCxrFqA0HhgPMHDgwCqFZmZmuUpJKH8EHpB0TRofA1xShrIXAtvmjA9I03ItAB6IiFXAs5L+SZZgZuYuFBEXAhcC1NfXRxliMzOzdmrzGkpE/BY4Hlia/o6PiP8pQ9kzgSGSBkvaCBgHTM9bZhrZ2QmS+pBVgT1ThrLNzKzMWj1DkbRZRLwhqTfwXPprmtc7IpZ2pOCIWC3pZGAG2fWRSyPiMUk/AxoiYnqa9wlJj5NVtZ0WEa92pFwzM6sMRRSuIZJ0fUQcmh5ozF2o6cHGD1QjwPaqr6+PhoaGWodhZtalSJoVEfUdWUerZygRcWj6P7gjBZiZ2XtDKc+h3FbKNDMze28rdg2lB9AT6CNpC969VXgzsmdIzMzM1ip22/BJwKlAP7K+UJoSyhvAeRWOy8zMuphi11DOAc6R9I2I6OhT8WZmtp4rpT+UdyT1ahqRtIWkr1UwJjMz64JKSSgn5jZ1khpqPLFyIZmZWVdUSkLpJmlt212p2fmNKheSmZl1RaW05fV34CpJF6Txk9I0M7OamDZ7IWfPmMeiZY3061XHaaO2Z8xw33xaa6UklO+TJZGvpvFbgIsrFpGZWRHTZi/kjKlzaVyVNYK+cFkjZ0ydC+CkUmOl9Nj4DnB++jMzq6mzZ8xbm0yaNK5aw9kz5jmh1FibCaVAW14AdNa2vMxs/bZoWWO7plv1lFLlldtYWA/gM0DvyoRjZlZcv151LCyQPPr1qqtBNJarlP5QXs35W5j6QjmkCrGZmbVw2qjtqeverdm0uu7dOG3U9jWKyJqUUuW1e87oBmRnLKWc2ZiZlV3TdRLf5dX5lJIYfpMzvJqso63PViQaM7MSjBne3wmkEyrlLq+PVapwSaOBc8h6bLw4Iia0stxRwNXAnhHh3rPMzDqhYs3Xf7vYC1Nf8+ssPXE/CRgJLABmSpoeEY/nLbcpcArwQEfKMzOzyip2UX7T9FdP9lBj//T3FWD3Iq8r1V7AUxHxTESsBK4EDi+w3H8BvwJWlKFMMzOrkGLN1/8UQNLdwO4R8WYaPxO4oQxl9wdezBlfAOydu0C6IWDbiLhB0mmtrUjSeGA8wMCBA8sQmpmZtVcpjUNuDazMGV+ZplWUpA2A3wLfaWvZiLgwIuojor5v376VDs3MzAoo5S6vPwEPSromjY8BLitD2QuBbXPGB6RpTTYFdgbuTI0dvx+YLukwX5g3M+t8SrnL6xeSbgL2S5OOj4jZZSh7JjBE0mCyRDIOODqn3NeBPk3jku4EvutkYmbWOZVS5QXQE3gjdQu8ICWBDomI1cDJwAzgCWBKRDwm6WeSDuvo+s3MrLpKeVL+P8nu9Noe+CPQHbgcGNHRwiPiRuDGvGk/aWXZAzpanpnZ+qiz9A9TyjWUI4DhwEMAEbEoPRtiZmY11pn6hymlymtlRASpCXtJG1c2JDMzK1Wx/mGqrZSEMiV1/9tL0onAbbjHRjOzTqEz9Q9Tyl1e/y1pJPAG2XWUH0fELRWPzMzM2tSZ+ocpeoYiqZukPhFxS0ScBvwAGCzpieqEZ2ZmxXSm/mFaTSiSxgFLgUck3SXpE8AzwMHA56sUn5mZFTFmeH/OOnIX+veqQ0D/XnWcdeQune4urx8Be0TEU6lNrfuAT0fEddUJzczMStFZ+ocpVuW1MiKeAoiIh4D5TiZmZtaaYmcoW+X1idIrd7yj/aGYmdn6pVhCuYisgcbWxs3MzNZqsz8UMzOzUpTaOKSZmVlRTihmZlYWbSYUSe8rMK13ZcIxM7OuqpQzlKmSujeNSNoGcNMrZmbWTCkJZRpZA5HdJA0i6xDrjHIULmm0pHmSnpJ0eoH535b0uKRHJN0mabtylGtmZuVXSuOQF0naiCyxDAJOioh/dLRgSd2AScBIYAEwU9L0iHg8Z7HZQH1EvC3pq8CvgbEdLdvMzMqv1YSS91CjgIHAHGAfSfuU4cHGvYCnIuKZVN6VwOHA2oQSEXfkLH8/8IUOlmlmZhVS7Awl/yHGqa1MX1f9gRdzxhcAexdZ/gTgpkIzJI0HxgMMHDiwTOGZmVl7tOvBRkkbAJtExBsVjapluV8g69f+PwrNj4gLgQsB6uvro4qhmZlZUsptw3+VtFnq+vdR4HFJp5Wh7IXAtjnjA9K0/PIPAn4IHBYR/y5DuWZmVgGl3OU1NJ2RjCGrchoMfLEMZc8EhkganC76jwOm5y4gaThwAVkyebkMZZqZWYWUklC6p+dQxgDTI2IV0OFqpYhYDZxMdhvyE8CUiHhM0s8kHZYWOxvYBPibpDmSpreyOjMzq7E2bxsmO0N4DngYuDs9C1KWaygRcSNwY960n+QMH1SOcszMrPJKeQ5lIjAxZ9Lzkj5WuZDMzKwrKvYcyhci4vK851FyuYMtMzNbq9gZysbpvzvVMjOzNhV7DuWC9L/Q8ygbt3yFmZm9lxW9y0tSf0n16bZeJG0l6ZfA/KpEZ2ZmXUarCUXSqWRtd50L3C/py2S399YBe1QnPDMz6yqKXUMZD2wfEUslDQT+CYyIiFnVCc3MzLqSYlVeKyJiKUBEvADMczIxM7PWFDtDGSAp9/mTbXLHI+KblQvLzMy6mmIJJb8BSJ+dmJlZq4rdNnxZNQMxM7OurZTGIc3MzNrkhGJmZmXhhGJmZmVRrHHIcynS74nv8jIzs1zF7vJqqHThkkYD5wDdgIsjYkLe/PcBfyJ7Mv9VYGxEPFfpuMzMrP1qdpeXpG7AJGAksACYKWl6RDyes9gJwGsR8SFJ44BfAWMrGZeZma2bNjvYktQX+D4wFOjRND0iPt7BsvcCnoqIZ1I5VwKHA7kJ5XDgzDR8NXCeJEVEh7sgNjOz8irlovxfyBqFHAz8lKw74JllKLs/8GLO+II0reAyqQ/614Et81ckabykBkkNS5YsKUNoZmbWXqUklC0j4hJgVUTcFRFfAjp6dlJWEXFhRNRHRH3fvn1rHY6Z2XtSKQllVfq/WNIhkoYDvctQ9kJg25zxAWlawWUkbQhsTnZx3szMOpk2r6EAP5e0OfAdsr5RNgO+VYayZwJDJA0mSxzjgKPzlpkOHAvcB3wauN3XT8zMOqc2E0pEXJ8GXwc+Vq6CI2K1pJOBGWS3DV8aEY9J+hnQEBHTgUuAP0t6ClhKlnQqauwF97WYduiu2/DFfQfRuHINx/3xwRbzP73HAD5Tvy1Ll6/kq5e3bEPzC/tsx6d268eiZY1866o5LeafuN8HOGjo1jy95C1+MHVui/nf+PgQPjqkD48tep2fXfd4i/nfG709e2zXm1nPL+XXf5/XYv5PPjWUnfptzv/Nf4Vzb2/Z2eYvj9yFD/bdhFsf/xcX3fNMi/m/GzuMfr3quO7hRVx+//Mt5p//hT3ovfFG/K3hRa6etaDF/MnH70XdRt34833Pcf0ji1vMv+qkfQG48O6nue2Jl5vN69G9G5d9aS8AJt42n3ufeqXZ/C16bsQfvpj19/arvz/JQ8+/1mz+Npv34H/GDQfgp9c9xuOL3mg2/wN9N+asI3cF4Iypj/DMkuXN5g/ttxn/+amdADj1ytksfn1Fs/m7b7cF3x+9AwBf+fMsXnt7ZbP5Iz7Uh28eOASAYy99kBWr1jSbf+COWzF+/w8C3ve873V832t6P7VSyl1elwGnRMSyNL4F8Jt0LaVDIuJG4Ma8aT/JGV4BfKaj5ZiZWeWprRokSbMjYnhb0zqL+vr6aGio+DOZZmbrFUmzIqK+I+so5aL8BumspKnQ3pR27cXMzN5DSkkMvwHuk/Q3QGQXx39R0ajMzKzLKeWi/J8kNfDusydH5jWPYmZmVrS14c0i4o1UxfUS8Neceb0jYmk1AjQzs66h2BnKX4FDyfqSz71yrzT+gQrGZWZmXUyx1oYPlSTgPyLihSrGVBPTZi/k7BnzWLSskX696jht1PaMGZ7ftJiZmbWm6F1e6an0G6oUS81Mm72QM6bOZeGyRgJYuKyRM6bOZdrs/JZgzMysNaXcNvyQpD0rHkkNnT1jHo15TzA3rlrD2TNaPvlrZmaFlXLb8N7A5yU9DywnXUOJiF0rGlkVLVrW2K7pZmbWUikJZVTFo6ixfr3qWFggefTrVVeDaMzMuqZWq7wkbZYG32zlb71x2qjtqeverdm0uu7dOG3U9jWKyMys62nPbcPKmbde3TbcdDeX7/IyM1t3bTYO2dW4cUgzq5b16XGDcjQOWVIjj5KOBD5KdmZyT0RM60ihZmZdXdPjBk13iDY9bgB02aTSUW3eNizp98BXgLnAo8BXJE3qSKGSeku6RdL89H+LAssMk3SfpMckPSJpbEfKNDMrJz9u0FIpZygfB3Zs6no3dbj1WAfLPR24LSImSDo9jX8/b5m3gWMiYr6kfsAsSTOaOvoyq5b1qVrDysePG7RUSkJ5ChgINPW/uW2a1hGHAwek4cuAO8lLKBHxz5zhRZJeBvoCTigV5h/Qd7law1rjxw1aKuVJ+U2BJyTdKelO4HFgM0nTJU1fx3K3joimDp5fArYutrCkvYCNgKdbmT9eUoOkhiVLlqxjSAZuhiafqzWsNX7coKVSzlB+0vYiLUm6FXh/gVk/zB2JiJDU6q1mkrYB/gwcGxHvFFomIi4ELoTsLq91idcyxX5A34tH5K7WsNb4cYOWSulg6y5Y+6DjhjnTi/aHEhEHtTZP0r8kbRMRi1PCeLmV5TYja5zyhxFxf1uxWsf5B7Q5V2tYMWOG939PJ5B8pdzlNV7SS8AjQAPZg44dfdBjOnBsGj4WuLZAuRsB1wB/ioirO1ielai1H8r36g+oqzXMSlfKNZTTgJ0jYlBEfCAiBkdER5+SnwCMlDQfOCiNI6le0sVpmc8C+wPHSZqT/oZ1sFxrQ2f6AZ02eyEjJtzO4NNvYMSE22tyHWfM8P6cdeQu9O9Vh4D+veo468hdfFRqVkCbT8pL+jtZP/JvVyekjvGT8h3XGe7yyr+7CrLE5h9zs8qo1pPyZwD/kPQA8O+miRHxzY4UbJ1XZ6gX9s0BZl1PKQnlAuB2siflC95lZVZuvjnArOspJaF0j4hvVzwSsxy+u8qs6ynlovxN6U6vbVIbXL0l9a54ZPae1pluDjCz0pRyhvK59P+MnGnrVX8o1vn4oTGzrqeUBxsHVyMQs3yd4eYAMytdsS6Av5cz/Jm8eb+sZFBmZtb1FFFFImcAABPxSURBVLuGMi5n+Iy8eaMrEIuZmXVhxRKKWhkuNG5mZu9xxRJKtDJcaNzMzN7jil2U303SG2RnI3VpmDTeo+KRmZlZl9JqQomIbq3NMzMzy1fKg41mZmZtKuXBRjOztTpDa9TWOTmhmFnJ8rsVWLiskTOmzgVwUjFXeZlZ6Yp1K2BWk4SSGpi8RdL89H+LIstuJmmBpPOqGaOZteRuBayYWp2hnA7cFhFDgNvSeGv+C7i7KlGZWVGtdR/gbgUMapdQDgcuS8OXAWMKLSRpD2Br4OYqxWVmRbhbASumVgll64hYnIZfIksazUjaAPgN8N22Vpb6a2mQ1LBkyZLyRmpma40Z3p+zjtyF/r3qENC/Vx1nHbmLL8gbUMG7vCTdCry/wKwf5o5EREgq1JTL14AbI2KBVLzpsIi4ELgQoL6+3s3CmFWQuxWw1lQsoUTEQa3Nk/QvSdtExGJJ2wAvF1hsX2A/SV8DNgE2kvRWRBS73mJmZjVSq+dQpgPHAhPS/2vzF4iIzzcNSzoOqHcyMTPrvGp1DWUCMFLSfOCgNI6kekkX1ygmMzPrAEWsX5cc6uvro6GhodZhmJl1KZJmRUR9R9bhJ+XNzKwsnFDMzKwsnFDMzKwsnFDMzKwsnFDMzKwsnFDMzKwsnFDMzKwsnFDMzKwsnFDMzKws3Ke8mdXUqlWrWLBgAStWrKh1KO8JPXr0YMCAAXTv3r3s63ZCMbOaWrBgAZtuuimDBg2ira4qrGMigldffZUFCxYwePDgsq/fVV5mVlMrVqxgyy23dDKpAklsueWWFTsbdEIxs5pzMqmeSm5rJxQzMysLJxQzsy6kW7duDBs2jJ133plPfepTLFu2DIDnnnuOuro6hg0bxtChQznmmGNYtWpVVWOrSUKR1FvSLZLmp/9btLLcQEk3S3pC0uOSBlU3UjOz9luzZk3F1l1XV8ecOXN49NFH6d27N5MmTVo774Mf/CBz5sxh7ty5LFiwgClTplQsjkJqdZfX6cBtETFB0ulp/PsFlvsT8IuIuEXSJsA71QzSzKpv7AX3tZh26K7b8MV9B9G4cg3H/fHBFvM/vccAPlO/LUuXr+Srl89qNu+qk/YtWt6YMWN48cUXWbFiBaeccgrjx4/nD3/4A08//TRnn302AJMnT6ahoYHzzjuPyy+/nIkTJ7Jy5Ur23ntvfv/739OtWzc22WQTTjrpJG699VYmTZrE7bffznXXXUdjYyMf+chHuOCCC5DEzJkzOeGEE9hggw0YOXIkN910E48++ihr1qzh9NNP58477+Tf//43X//61znppJOKxr7vvvvyyCOPtJjerVs39tprLxYuXFj09eVWqyqvw4HL0vBlwJj8BSQNBTaMiFsAIuKtiHi7eiGa2XvBpZdeyqxZs2hoaGDixIm8+uqrHHXUUVxzzTVrl7nqqqsYN24cTzzxBFdddRX33nsvc+bMoVu3bvzlL38BYPny5ey99948/PDDfPSjH+Xkk09m5syZPProozQ2NnL99dcDcPzxx3PBBResfX2TSy65hM0335yZM2cyc+ZMLrroIp599tlW416zZg233XYbhx12WIt5K1as4IEHHmD06NHl2kwlqdUZytYRsTgNvwRsXWCZDwPLJE0FBgO3AqdHRItzSUnjgfEAAwcOrEzEZlYVxc4o6jbqVnR+7403avOMJN/EiRPXJo8XX3yR+fPns88++/CBD3yA+++/nyFDhvDkk08yYsQIJk2axKxZs9hzzz0BaGxsZKuttgKys4Kjjjpq7XrvuOMOfv3rX/P222+zdOlSdtppJ/bbbz/efPNN9t03i/Hoo49em2huvvlmHnnkEa6++moAXn/9debPn9/ieZHGxkaGDRvGwoUL2XHHHRk5cuTaeU8//TTDhg3j2Wef5ZBDDmHXXXdt17boqIolFEm3Au8vMOuHuSMREZIKdWy/IbAfMBx4AbgKOA64JH/BiLgQuBCyPuU7FLhZJzVt9kLOnjGPRcsa6derjtNGbc+Y4f1rHVaXduedd3Lrrbdy33330bNnTw444IC1z2iMGzeOKVOmsMMOO3DEEUcgiYjg2GOP5ayzzmqxrh49eqw941ixYgVf+9rXaGhoYNttt+XMM89s89mPiODcc89l1KhRRZdruoby9ttvM2rUKCZNmsQ3v/lN4N1rKK+88gojRoxg+vTpBc9gKqViVV4RcVBE7Fzg71rgX5K2AUj/Xy6wigXAnIh4JiJWA9OA3SsVr1lnNm32Qs6YOpeFyxoJYOGyRs6YOpdps6tbR76+ef3119liiy3o2bMnTz75JPfff//aeUcccQTXXnstV1xxBePGjQPgwAMP5Oqrr+bll7OfrKVLl/L888+3WG9T8ujTpw9vvfXW2rOOXr16semmm/LAAw8AcOWVV659zahRozj//PPX3pn1z3/+k+XLl7cae8+ePZk4cSK/+c1vWL16dbN5ffr0YcKECQUTXyXV6hrKdODYNHwscG2BZWYCvST1TeMfBx6vQmxmnc7ZM+bRuKp5bW/jqjWcPWNejSJaP4wePZrVq1ez4447cvrpp7PPPvusnbfFFluw44478vzzz7PXXnsBMHToUH7+85/ziU98gl133ZWRI0eyePHiFuvt1asXJ554IjvvvDOjRo1aW0UG2bWSE088kWHDhrF8+XI233xzAL785S8zdOhQdt99d3beeWdOOumkFoki3/Dhw9l111254oorWswbM2YMb7/9Nvfcc886bZt1oYjq1xBJ2hKYAgwEngc+GxFLJdUDX4mIL6flRgK/AQTMAsZHxMpi666vr4+GhoaKxm9WbYNPv4FC31QBz044pNrhlNUTTzzBjjvuWOswquatt95ik002AWDChAksXryYc845p6oxFNrmkmZFRH1H1luTi/IR8SpwYIHpDcCXc8ZvAap7VcmsE+rXq46FyxoLTreu5YYbbuCss85i9erVbLfddkyePLnWIZWNWxs26wJOG7U9Z0yd26zaq657N04btX0No7J1MXbsWMaOHVvrMCrCCcWsC2i6m2t9vcsrItxAZJVU8jKHE4pZFzFmeP/1JoHk6tGjB6+++qqbsK+Cpv5QevToUZH1O6GYWU0NGDCABQsWsGTJklqH8p7Q1GNjJTihmFlNde/evSK9B1r1ufl6MzMrCycUMzMrCycUMzMri5o8KV9JkpaQPX2/rvoAr5QpnK4cAziOfI6juc4QR2eIAdaPOLaLiL5tL9a69S6hdJSkho42P7A+xOA4HEdXiKMzxOA43uUqLzMzKwsnFDMzKwsnlJYurHUAdI4YwHHkcxzNdYY4OkMM4DgAX0MxM7My8RmKmZmVhROKmZmVhRNKIulSSS9LerSGMWwr6Q5Jj0t6TNIpNYqjh6QHJT2c4vhpLeJIsXSTNFvS9bWKIcXxnKS5kuZIqkmXoJJ6Sbpa0pOSnpC0bw1i2D5tg6a/NySdWu04UizfSvvno5KukFSZJnTbjuOUFMNj1dwWhX6zJPWWdIuk+en/FtWKB5xQck0GRtc4htXAdyJiKLAP8HVJQ2sQx7+Bj0fEbsAwYLSkfdp4TaWcAjxRo7LzfSwihtXwPv9zgL9HxA7AbtRgu0TEvLQNhgF7AG8D11Q7Dkn9gW8C9RGxM9ANGFeDOHYGTgT2IvtMDpX0oSoVP5mWv1mnA7dFxBDgtjReNU4oSUTcDSytcQyLI+KhNPwm2Q9G1TvAiMxbabR7+qv63RuSBgCHABdXu+zORtLmwP7AJQARsTIiltU2Kg4Eno6IjrRM0REbAnWSNgR6AotqEMOOwAMR8XZErAbuAo6sRsGt/GYdDlyWhi8DxlQjliZOKJ2UpEHAcOCBGpXfTdIc4GXgloioRRz/A3wPeKcGZecL4GZJsySNr0H5g4ElwB9TFeDFkjauQRy5xgFX1KLgiFgI/DfwArAYeD0ibq5BKI8C+0naUlJP4JPAtjWIo8nWEbE4Db8EbF3Nwp1QOiFJmwD/C5waEW/UIoaIWJOqNQYAe6VT+6qRdCjwckTMqma5RXw0InYHDiarity/yuVvCOwOnB8Rw4HlVLk6I5ekjYDDgL/VqPwtyI7GBwP9gI0lfaHacUTEE8CvgJuBvwNzgDXVjqOQyJ4JqWrNghNKJyOpO1ky+UtETK11PKla5Q6qf31pBHCYpOeAK4GPS7q8yjGslY6IiYiXya4Z7FXlEBYAC3LOFK8mSzC1cjDwUET8q0blHwQ8GxFLImIVMBX4SC0CiYhLImKPiNgfeA34Zy3iSP4laRuA9P/lahbuhNKJKOtQ+xLgiYj4bQ3j6CupVxquA0YCT1Yzhog4IyIGRMQgsqqV2yOi6kegAJI2lrRp0zDwCbKqjqqJiJeAFyVtnyYdCDxezRjyfI4aVXclLwD7SOqZvjcHUqObNyRtlf4PJLt+8tdaxJFMB45Nw8cC11azcHcBnEi6AjgA6CNpAfCfEXFJlcMYAXwRmJuuXwD8ICJurHIc2wCXSepGdtAxJSJqettujW0NXJP9brEh8NeI+HsN4vgG8JdU3fQMcHwNYmhKqiOBk2pRPkBEPCDpauAhsrsjZ1O7Zkf+V9KWwCrg69W6WaLQbxYwAZgi6QSybjw+W41Y1sbkplfMzKwcXOVlZmZl4YRiZmZl4YRiZmZl4YRiZmZl4YRiZmZl4YRiRUlak1qVfVTS31LzEuu6rsmSPp2GLy7W8KWkAyS1+0G11Cpwn1amz5X0iKS7JG2XM+8fbazzrWLz0zK52+m6pud42hF3m2W08rphkj6ZM36YpLI8QS9pgKRrU8u1z0g6T9L7yrHuvHKafdaSviLpmDS8dp+xzs8JxdrSmFqX3RlYCXwld2ZqmK/dIuLLEVHswbwDKP+Tzx+LiF2BO4Ef5cRSjnJyt9NS4OtlWGcphpG1HwVAREyPiAkdXWl6WHAqMC21XDsEqAN+3dF1F3AAOZ91RPwhIv5UgXKswpxQrD3uAT6UjijvkTQdeDw1JHm2pJnpDOAkyH6U0lHtPEm3Als1rUjSnZLq0/BoSQ8p63/lttQw5leAb6Wj/v3S0/v/m8qYKWlEeu2Wkm5W1hfFxYBKeB/3kdOKc9PZgaRtJN2dc6axX+6LJPWRdJ+kQ0pdv6QPSvq7skYl75G0Q5o+OK1rrqSf55RxgHL6fknb77g0vKekf6Tt9KCyFoh/BoxNMY+VdJyk89LygyTdnj6T25Q9yd101D8xreuZVs4APg6siIg/Qta2G/At4BhJm+SWk9Z5vaQD0vD5khqU15dOOkv8afqs50raoZXP+kxJ380PSNIe6exylqQZSk2MWOfhhGIlSWciBwNz06TdgVMi4sPACWStve4J7AmcKGkwcASwPTAUOIYCZxyS+gIXAUel/lc+ExHPAX8AfpeO+u8h6wvkd6mMo3i3Sfv/BP4vInYia2NrYAlvZzQwrcD0o4EZqVHM3cga+muKc2vgBuAnEXFDaytW1rrAgWRNYED29PY3ImIP4LvA79P0c8gaetyFrLXcopQ9HX8V2Tbfjawtq+XAT4Cr0na6Ku9l5wKXpbOyvwATc+ZtA3wUOJTs6ep8OwHNGuZMDZU+B7TV38cPU58xuwL/IWnXnHmvpEY2zwe+28pnXej9d0/v59NpW14K/KKNOKzK3PSKtaVO7zYDcw9ZW2MfAR6MiGfT9E8Au+Yc6W5OVkWyP3BFOrpdJOn2AuvfB7i7aV0R0VqfNAcBQ7OaGAA2U9Yq8/6k/ici4gZJrxV5L3dI6g28Bfy4wPyZwKXpx2taRDS97+5knRV9PSLuamXdTdupP1mbUrek+D4C/C0n7qZrECPIEiPAn8larC1me2BxRMyEtT/u5Ky3kH15t2+OP9O8umpaRLxDdoZZ7ibOP6usif8NyRLXUOCRNK+pwdNZtK/fkO2Bncm2K2QdarWZiK26nFCsLY3piH2t9IVenjuJ7Ch8Rt5yn6R8NgD2iYgVBWIp1ceAZWRH6z8Fvp07MyLuVtYs/SHAZEm/TXX5q8l+AEeRdaBUSGNEDFN208IMsmsok4Fl+dsvt8gC01bTvOagUt3a/jtnuNBGfBxoVhUmaTPg/cA8sh/3FnGmM9PvAntGxGuSJtP8PTSVu4b2/f4IeCwiqt7tsZXOVV5WDjOAr6YjeyR9WFkDgneT1e93S/XdHyvw2vuB/dMPEekMAuBNYNOc5W4maxyRtFzTj/TdZFVVSDoYKNqHdupV71SyawG9c+cpu/PrXxFxEVmVWlPz8AF8CdhB0vfbWP/bZF3Tfoese9xnJX0mrV+SdkuL3su7XdZ+PmcVz5Odib1P2Z1iB6bp84BtJO2Z1rVpqobM3065/pFXRsHqpFbcBvTUu3dbdQN+A5wXEY1kVV/DJG0gaVvebc5/M7KDjdfTmc/BJZRV7D00mQf0lbRviqe7pJ3a8X6sCpxQrBwuJjuifUjSo8AFZEef1wDz07w/kV2sbiYilgDjgamSHia7TgBwHXBE04VaUv/h6QLz47x7t9lPyRLSY2RVKC+0FWzq0e4KWt6JdQDwsKTZwFiy6xxNr1lD1mT7xyV9rY31zyar4vkc2Q/5Cem9PUbWKRTAKWQddc0l5waBiHgRmELWPP4UslZ0iYiVKaZz07puITvyv4MsAc2RNDYvlG8Ax0t6hKwV61OKb5lm7yHIroF9WtJ84FXgnYhoum5xL/As2Wc7kazVXyLi4RTzk2TNuN9bQnH5n3WheFaSnTH9Kr3/OdSo/xNrnVsbNrM2KXtO5ArgiIh4qNbxWOfkhGJmZmXhKi8zMysLJxQzMysLJxQzMysLJxQzMysLJxQzMysLJxQzMyuL/wdX6UB8ypJb/wAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "w8F2N-Zje8dB"
      },
      "source": [
        "Recall that the predicted risk reduction is along the horizontal axis and the vertical axis is the empirical (actual risk reduction).\n",
        "\n",
        "A good model would predict a lower risk reduction for patients with actual lower risk reduction.  Similarly, a good model would predict a higher risk reduction for patients with actual higher risk reduction (imagine a diagonal line going from the bottom left to the top right of the plot).\n",
        "\n",
        "The T-learner seems to be doing a bit better (compared to the logistic regression model) at differentiating between the people who would benefit most treatment and the people who would benefit least from treatment."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "CzcjvmxKJWlN"
      },
      "source": [
        "Compute the C-statistic-for-benefit on the validation set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:32:40.675054Z",
          "start_time": "2020-04-04T15:32:40.671084Z"
        },
        "colab_type": "code",
        "id": "blwOcph5JVnV",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "4f359278-db85-4296-a717-87d6175465cc"
      },
      "source": [
        "c_for_benefit_tlearner_val_set = c_statistic(rr_t_val, y_val, X_val.TRTMT)\n",
        "print(f\"C-for-benefit statistic of T-learner on val set: {c_for_benefit_tlearner_val_set:.4f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "C-for-benefit statistic of T-learner on val set: 0.5043\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mubWa6kvTmuT",
        "colab_type": "text"
      },
      "source": [
        "##### Expected output\n",
        "\n",
        "```CPP\n",
        "C-for-benefit statistic of T-learner on val set: 0.5043\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "yWo27MRmJoa0"
      },
      "source": [
        "Now or the test set, predict each patient's risk reduction"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nKYv_kFJTmuU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# predict the risk reduction for each of the patients in the test set\n",
        "rr_t_test = t_learner.predict(X_test.drop(['TRTMT'], axis=1))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0aAGGv8qTmuX",
        "colab_type": "text"
      },
      "source": [
        "Plot the histogram of risk reduction for the test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NzTLMVdUTmuY",
        "colab_type": "code",
        "colab": {},
        "outputId": "08875928-1ec4-4eb4-f48d-31823f57f0a1"
      },
      "source": [
        "# Plot a histogram of the predicted risk reduction\n",
        "plt.hist(rr_t_test, bins='auto')\n",
        "plt.title(\"Histogram of Predicted ARR for the T-learner on test set\")\n",
        "plt.xlabel(\"predicted risk reduction\")\n",
        "plt.ylabel(\"count of patients\")\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEWCAYAAABmE+CbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3deZwcVbn/8c+XsARIIAlEDEsYiIAGfiwaAQERAZFVlgvIIouiEVARcYvLvQTFa2TRy5V7QVAMIIIsoggqmyyyE8K+XQKGJQQIgUjYAgnP749zBirNdE/NpKt7Zvy+X69+Te3n6dM19VTVqT6tiMDMzGyxdgdgZmZ9gxOCmZkBTghmZpY5IZiZGeCEYGZmmROCmZkBAyghSLpf0lbtjqOdJO0u6UlJL0vaqMVlXyvp83l4f0lXtKDMDkkhafGqyypL0mGSns2fwQoVbL8t71nSZEnHtrJMa71+kRAkTZe0bc20gyXd0DkeEetGxLXdbKfPHUCa7ATgyxExJCLurJ2Z3/sr+WA1Q9JPJQ1qdhARcU5EbNfdcpImSvpNs8vvopxrJb0oaama6ZMlvZHr4wVJV0p6f2H+wZIW5PkvSbpb0s4NylkC+CmwXf4MZjch9nft+82Qk/bL+fWapLcK4y83u7yBIv8Pva8J26lk31/UY1y/SAj9RR9INKsD93ezzAYRMQTYBtgP+ELtAn3gfTSNpA7go0AAn+pikeNyfawCzAB+VTP/5jx/GPC/wHmShtUpbiVgMN1/Bl3FKUkt+3/MSXtIfm87AE93judpLdHsfa2KE5x/JQMmIRTPpCRtLGlKPqt7VtJP82LX579z8pnQRyQtJun7kh6X9JyksyQtX9jugXnebEn/XlPOREkXSvqNpJeAg3PZN0uaI2mmpJMlLVnYXkg6XNIjkuZK+qGkMZJuyvGeX1y+5j12GaukpfJZ3SDgbkmPdldfEfEQ8HdgvcJZxSGSngD+lsv7nKQH89n15ZJWL8TyCUkPSfqnpJMBFeYtdPUmad189v1C/jy+K2l74LvAp/NncXdednlJv8p1N0PSsZ3/5JIGSTpB0vOSHgN26u59AgcCtwCTgYMa1MdrwPnAhnXmvwWcDSwLrFU7X9LawMN5dI6kzjrcTNLtuZ5ul7RZYZ1rJf1I0o3Aq8CaNds8GxgN/CnX0bcKs/eX9ESui+8V1llM0gRJj+Z99nxJI+q9796StLOku/J+fpOk9QvzOsufK+kBSbsX5h0s6UZJP5M0G5iodKX2P5Iuy+vcKmlMYZ33F/afhyXtXZg3WdIpkv4s6RXg413EurKkS/L60yR9oTBvYq6js3LZ90saV+c9dx4/7s6fx6dL1MW38348N8e+Tb19v4vy3rVunt7oM37XMa7LD7CeiOjzL2A6sG3NtIOBG7paBrgZOCAPDwE2zcMdpDPFxQvrfQ6YRvpnHAL8Hjg7zxsLvAxsASxJuiXzZqGciXl8N1JyXRr4ELApsHgu70HgyEJ5AfwRWA5YF5gHXJ3LXx54ADioTj3UjbWw7fc1qMe35+f39gxwSKFeziId8JYGds1lfSC/l+8DN+V1VwTmAnsCSwBfA+YDn6/9bIChwEzg66Sz56HAJoX6+01NjBcDv8hxvAe4Dfhinnco8BCwGjACuKb28+ziPU8DDs+fy5vASoV5k4Fj8/CypAP+3V3tY6Rk+yXgDeA9dcrqKMaTY3wROCDX4b55fIU8/1rgibwfLA4s0d2+Xyjj9Pw5bZD3oQ/k+V8lJcBVgaVyXZ7bzf/XVsBT3SxTrKuNgOeATXK9HJTjXCrP3wtYmfQ/8WngFWBUoU7nA1/J73npvO3ZwMZ52jnAeYXP5Ungs3neRsDzwNhCXP8ENs/lDe4i9utJV3eDSQl/FrB1YR98Hdgxv5cfA7eU+R/qri6AdXLsKxc+uzH19v2achqtW/czpotjXI+Otb1ZqdWvXMEvA3MKr1epnxCuB44BVmz0D5unXQ0cXvNBvJl3vv+g8M8ELEM6IBQTwvXdxH4kcHHNDrV5YfwO4NuF8ROB/6qzrbqxdrWz1tmZXyIdlB4Fjs3/RJ31smZh2b8AhxTGF8t1vjr5rLswT8BTdJ0Q9gXurBPPQv8UpFsu84ClC9P2Ba7Jw38DDi3M267Rzk9K5G927gekZPK1wvzJpIPBHOAt4B/A+oX5B5MOXnPydl4D9m5QvwvtX6REcFvNMjcDB+fha4EflNj3u0oIqxam3Qbsk4cfBLYpzBtV3EfqlLEVPUsIpwA/rJn/MPCxOuveBexaqNMnutj2LwvjOwIP5eFPA3+vWf4XwNGFdc9qEPdqwAJgaGHaj4HJhX3wqsK8scBr3fwPFRNC3boA3kdKFttSk+zpPiE0WrfuZ1y7D/b01Z9uGe0WEcM6X6SzvnoOAdYGHsqX6XUbAklnMo8Xxh8nVexKed6TnTMi4lXSmUzRk8URSWtLulTSM0q3kf6TdEZd9Gxh+LUuxuvdw20Ua1kfjIjhETEmIr4f6VZIV+9ldeCkfBk8B3iBdOBfhXfXS9SsW7QaKfmUsTrpimNmodxfkK4UqC2XheuiKwcBV0TE83n8t7z7ttEJeX/qINX9OjXzb8nzhwOXkNojyqr9vDpjXqUwXq/euvNMYfhV3tlnVgcuLtTfg6QDYul9JN/S62xgPrWLRVYHvt5ZRi5nNdL77bzNeldh3nos/D/Q1Xtu9H42qSlrf+C93Wyv08rACxExtzCt9jOoLXuwyrdt1K2LiJhGOiGcCDwn6TxJK5fZaDfrLvJnXE9/SgilRcQjEbEv6UDyE+BCScuSMmetp0kV3Gk06azwWdKtjlU7Z0haGqh9lLB2m6eQzkTXiojlSPcKRXM0irUZiu/lSdKtmmGF19IRcROpXlbrXFCSiuM1nqTm3nid8jqXnUc6o+8sc7mIWDfPX6hc0vvvUv6s9gY+lpPzM6RbWxtI2uBdgUQ8QboUPymvWzv/ZeAw4ACVf6S39vPqjHlGcdPdbKO7+bWeBHao+dwGR8SMbtfsLDDiP+OdBuZD65Txo5oylomIc5XamU4Hvky6NTYMuI+F/wd68p6eBK6rKWtIRBxWcntPAyMkDS1Mq/0MFkXdugCIiN9GxBak/SBIx6PuYqabdRt9xj3dXxYyIBOCpM9IGpnPfufkyW+R7h2+xcIHqHOBr0laQ9IQ0hn97yJiPnAhsItSw+CSpGzd3cF9KOm2zMtKjzAe1s3yPdEo1mY7FfiOpHXh7cbevfK8y4B1Je2Rz6SOYOEztqJLgVGSjlRq/B4qaZM871mgQ/npmoiYCVwBnChpudx4NkbSx/Ly5wNHSFpV0nBgQoP4dyOdNY0l3TfekNQe8nfSLa93iYgrSQeQ8XXmvwD8knQrsYw/A2tL2k/S4rkRciypTsp6lvoJtSunAj/KB2YkjZS0aw/WL+N04FBJmyhZVtJO+aDbeeI1K5f/WdIVQm9dSqrDAyQtkV8flvSBMitHxJPATcCPJQ3ODb6HAL195LP286hbF5LWkbS10uPOr5OuQN8qbOftfb9WN+s2+oy7OsaVNiATArA9cL/Skzcnke6vvpZv+fwIuDFfbm0KnEFqTLyedA/5dVKDFxFxfx4+j3R2+jLpvt68BmV/g/Q451zSzvK7Jr6vurE2W0RcTDojOS/f+rqP9Hgi+RbMXsAk0i20tYAb62xnLvAJYBfSpfkjvPMkyAX572xJU/PwgaQG/AdIbR0Xku6RQqrPy4G7gamkRvV6DgJ+HRFPRMQznS/gZNITOvVuCRwPfEs131ko+C9gRxWeJKkn0vcQdiY1qM8GvgXsXLiFVcaPge/n/fUbJZY/iXRr6wpJc0mNj5s0XqVnImIK6XHlk0mf0TRS2wAR8QCpHexm0kHv/1Fn3yhZ1lxSW9E+pGT9DGm/rPf5dGVf0i3Bp0kPLRwdEVf1MqSJwJn589i7UV3kGCeRGsGfId2x+E6e19W+X9Ro3bqfcZ1jXGnKjRJWQj4rn0O6HfSPdsdjZtZMA/UKoWkk7SJpmdwGcQJwL+nJDzOzAcUJoXu7ki41nybdGtknfFllZgOQbxmZmRngKwQzM8sq68RM0mDS0zBL5XIujIijJa1BempnBdK3dA+IiDcabWvFFVeMjo6OqkI1MxuQ7rjjjucjYmTZ5avs1XIeqb+Ql5W6Bb5B0l+Ao4CfRcR5+VuQh5C+zFVXR0cHU6ZMqTBUM7OBR1J33+ZfSGW3jCLp7Fd9ifwKYGvSs+UAZ5K+QGRmZm1WaRuCUnfFd5G+zHUlqU+bOYVv1j7Fwn2KmJlZm1SaECJiQURsSOoPaGPg/d2s8jZJ45V+02DKrFmzKovRzMySljxlFBFzSH3XfwQYVug2YFXqdDIVEadFxLiIGDdyZOk2ETMz66XKEkLucGlYHl6a1J/Ng6TEsGde7CDSj8WYmVmbVfmU0ShSJ1CDSInn/Ii4VNIDpA7TjgXu5N2/YWtmZm1QWUKIiHtIPy9XO/0xUnuCmZn1If6mspmZAU4IZmaWVdmGYGZN0jHhsnaH0BTTJ+3U7hCsAV8hmJkZ4IRgZmaZE4KZmQFOCGZmljkhmJkZ4IRgZmaZE4KZmQFOCGZmljkhmJkZ4IRgZmaZE4KZmQFOCGZmljkhmJkZ4IRgZmaZE4KZmQFOCGZmljkhmJkZ4IRgZmaZE4KZmQFOCGZmljkhmJkZ4IRgZmaZE4KZmQFOCGZmllWWECStJukaSQ9Iul/SV/P0iZJmSLorv3asKgYzMytv8Qq3PR/4ekRMlTQUuEPSlXnezyLihArLNjOzHqosIUTETGBmHp4r6UFglarKMzOzRVPlFcLbJHUAGwG3ApsDX5Z0IDCFdBXxYhfrjAfGA4wePboVYZr1WMeEy9odglnTVN6oLGkIcBFwZES8BJwCjAE2JF1BnNjVehFxWkSMi4hxI0eOrDpMM7N/eZUmBElLkJLBORHxe4CIeDYiFkTEW8DpwMZVxmBmZuVU+ZSRgF8BD0bETwvTRxUW2x24r6oYzMysvCrbEDYHDgDulXRXnvZdYF9JGwIBTAe+WGEMZmZWUpVPGd0AqItZf66qTDMz6z1/U9nMzAAnBDMzy5wQzMwMcEIwM7PMCcHMzAAnBDMzy5wQzMwMcEIwM7PMCcHMzAAnBDMzy5wQzMwMcEIwM7PMCcHMzAAnBDMzy5wQzMwMcEIwM7PMCcHMzAAnBDMzy5wQzMwMcEIwM7PMCcHMzAAnBDMzy5wQzMwM6GFCkDRc0vpVBWNmZu3TbUKQdK2k5SSNAKYCp0v6afWhmZlZK5W5Qlg+Il4C9gDOiohNgG2rDcvMzFqtTEJYXNIoYG/g0orjMTOzNimTEI4BLgemRcTtktYEHuluJUmrSbpG0gOS7pf01Tx9hKQrJT2S/w5ftLdgZmbNUCYhzIyI9SPicICIeAwo04YwH/h6RIwFNgW+JGksMAG4OiLWAq7O42Zm1mZlEsLPS05bSETMjIipeXgu8CCwCrArcGZe7Exgt3KhmplZlRavN0PSR4DNgJGSjirMWg4Y1JNCJHUAGwG3AitFxMw86xlgpTrrjAfGA4wePbonxZmZWS80ukJYEhhCShpDC6+XgD3LFiBpCHARcGR+WultERFAdLVeRJwWEeMiYtzIkSPLFmdmZr1U9wohIq4DrpM0OSIe783GJS1BSgbnRMTv8+RnJY2KiJn56aXnerNtMzNrrroJoWApSacBHcXlI2LrRitJEvAr4MGIKDZCXwIcBEzKf//Yw5jNzKwCZRLCBcCpwC+BBT3Y9ubAAcC9ku7K075LSgTnSzoEeJz0/QYzM2uzMglhfkSc0tMNR8QNgOrM3qan2zMzs2qVeez0T5IOlzQqf6lsRO7XyMzMBpAyVwgH5b/fLEwLYM3mh2PWPB0TLmt3CGb9SrcJISLWaEUgZmbWXmW6v15G0vfzk0ZIWkvSztWHZmZmrVSmDeHXwBukby0DzACOrSwiMzNrizIJYUxEHAe8CRARr1L/6SEzM+unyiSENyQtTe5iQtIYYF6lUZmZWcuVecroaOCvwGqSziF94ezgKoMyM7PWK/OU0ZWSppJ+00DAVyPi+cojMzOzlqp7y0jS+/PfDwKrAzOBp4HReZqZmQ0gja4QjiL9HsGJXcwLoGHndmZm1r806v56fB7cISJeL86TNLjSqMzMrOXKPGV0U8lpZmbWjzX6Cc33kn4DeWlJG/HOdw+WA5ZpQWxmZtZCjdoQPkl6vHRVoPgDN3NJv2tgZmYDSKM2hDOBMyX9W0Rc1MKYzMysDcp8D+EiSTsB6wKDC9N/UGVgZmbWWmV6Oz0V+DTwFVI7wl6k7yWYmdkAUuYpo80i4kDgxYg4BvgIsHa1YZmZWauVSQiv5b+vSlqZ1OvpqOpCMjOzdijTud2lkoYBxwNTSd9SPr3SqMzMrOXKNCr/MA9eJOlSYHBE/LPasMzMrNW6TQi5m4rDgS1IVwc3SDqltjsLMzPr38rcMjqL9GW0n+fx/YCzSU8bmZnZAFEmIawXEWML49dIeqCqgMzMrD3KPGU0VdKmnSOSNgGmVBeSmZm1Q5krhA8BN0l6Io+PBh6WdC8QEbF+ZdGZmVnLlEkI2/dmw5LOAHYGnouI9fK0icAXgFl5se9GxJ97s30zM2uuMo+dPt7LbU8GTiY1Shf9LCJO6OU2zcysImXaEHolIq4HXqhq+2Zm1lx1E4KkpSoq88uS7pF0hqThDcofL2mKpCmzZs2qt5iZmTVJoyuEmwEknd3E8k4BxgAbAjOBE+stGBGnRcS4iBg3cuTIJoZgZmZdadSGsKSk/YDNJO1ROzMift/TwiLi2c5hSacDl/Z0G2ZmVo1GCeFQYH9gGLBLzbwAepwQJI2KiJl5dHfgvp5uw8zMqtHoJzRvIPVbNCUiftXTDUs6F9gKWFHSU8DRwFaSNiQllOnAF3sTtJmZNV+Z7yGcLekIYMs8fh1wakS82WiliNi3i8k9TixmZtYaZRLC/wJL5L8AB5Aahz9fVVBmZtZ6ZRLChyNig8L43yTdXVVA1l4dEy5rSTnTJ+3UknLMrLwyX0xbIGlM54ikNYEF1YVkZmbtUOYK4ZukLq8fAwSsDny20qjMzKzlyvRldLWktYB18qSHI2JetWGZmVmrlblCICeAeyqOxczM2qiyzu3MzKx/cUIwMzOgREKQdHWZaWZm1r/VbUOQNBhYhtT1xHDSE0YAywGrtCA2MzNroUaNyl8EjgRWBu7gnYTwEumX0MzMbABp1LndScBJkr4SET9vYUxmZtYGZb6H8HNJmwEdxeUjova3ks3MrB/rNiHkX0wbA9zFO11WBOCEYGY2gJT5Yto4YGxERNXBmJlZ+5RJCPcB7yX9BrKZWa+5N92+rUxCWBF4QNJtwNt9GEXEpyqLyszMWq5MQphYdRBmZtZ+ZZ4yuq4VgZiZWXuVecpoLumpIoAlST+n+UpELFdlYGZm1lplrhCGdg5LErArsGmVQZmZWev1qLfTSP4AfLKieMzMrE3K3DLaozC6GOl7Ca9XFpGZmbVFmaeMdikMzwemk24bmZnZAFKmDeGzrQjEzMzaq8wP5Kwq6WJJz+XXRZJWbUVwZmbWOmUalX8NXEL6XYSVgT/laQ1JOiMnkPsK00ZIulLSI/nv8N4GbmZmzVUmIYyMiF9HxPz8mgyMLLHeZGD7mmkTgKsjYi3g6jxuZmZ9QJmEMFvSZyQNyq/PALO7WykirgdeqJm8K3BmHj4T2K1H0ZqZWWXKJITPAXsDz5B6PN0T6G1D80oR0dlr6jPASvUWlDRe0hRJU2bNmtXL4szMrKwyTxk9DjS9Z9OICEl1f2MhIk4DTgMYN26cf4vBzKxiZZ4yOlPSsML4cEln9LK8ZyWNytsZBTzXy+2YmVmTlblltH5EzOkciYgXgY16Wd4lwEF5+CDgj73cjpmZNVmZhLBY8fFQSSMo1+XFucDNwDqSnpJ0CDAJ+ISkR4Bt87iZmfUBZbquOBG4WdIFeXwv4EfdrRQR+9aZtU3J2MzMrIXKNCqfJWkKsHWetEdEPFBtWGZm1mplrhDICcBJwMxsAOvR7yGYmdnA5YRgZmaAE4KZmWVOCGZmBjghmJlZ5oRgZmaAE4KZmWWlvodgZtafdEy4rPIypk/aqfIyWs1XCGZmBjghmJlZ5oRgZmaAE4KZmWVOCGZmBjghmJlZ5oRgZmaAE4KZmWVOCGZmBjghmJlZ5oRgZmaAE4KZmWVOCGZmBjghmJlZ5oRgZmaAE4KZmWVOCGZmBrTpF9MkTQfmAguA+RExrh1xmJnZO9r5E5ofj4jn21i+mZkV+JaRmZkB7btCCOAKSQH8IiJOq11A0nhgPMDo0aNbHJ5VrRU/gm5mPdOuK4QtIuKDwA7AlyRtWbtARJwWEeMiYtzIkSNbH6GZ2b+YtiSEiJiR/z4HXAxs3I44zMzsHS1PCJKWlTS0cxjYDriv1XGYmdnC2tGGsBJwsaTO8n8bEX9tQxxmZlbQ8oQQEY8BG7S6XDMza8yPnZqZGdDeL6ZZD/lRTbO+o1X/j9Mn7dSScsBXCGZmljkhmJkZ4IRgZmaZE4KZmQFOCGZmljkhmJkZ4IRgZmaZE4KZmQFOCGZmljkhmJkZ4IRgZmaZE4KZmQFOCGZmljkhmJkZ4O6vm8ZdU5tZf+crBDMzA5wQzMwsc0IwMzPACcHMzDInBDMzA5wQzMwsG/CPnfpxUDOzcnyFYGZmgBOCmZllTghmZga0KSFI2l7Sw5KmSZrQjhjMzGxhLU8IkgYB/wPsAIwF9pU0ttVxmJnZwtpxhbAxMC0iHouIN4DzgF3bEIeZmRW047HTVYAnC+NPAZvULiRpPDA+j74s6eEutrUi8HzTI6ye426t/ho39N/YHXeT6CelFqsX9+o9KavPfg8hIk4DTmu0jKQpETGuRSE1jeNurf4aN/Tf2B13azUr7nbcMpoBrFYYXzVPMzOzNmpHQrgdWEvSGpKWBPYBLmlDHGZmVtDyW0YRMV/Sl4HLgUHAGRFxfy831/CWUh/muFurv8YN/Td2x91aTYlbEdGM7ZiZWT/nbyqbmRnghGBmZlmfTwiSRki6UtIj+e/wOsv9VdIcSZfWTF9D0q25m4zf5YbsvhT3QXmZRyQdVJh+be7e4678ek/F8TbsTkTSUrn+puX67CjM+06e/rCkT1YZZ7PiltQh6bVC/Z7ax+LeUtJUSfMl7Vkzr8t9phUWMe4Fhfpu6YMkJeI+StIDku6RdLWk1Qvz2lbfufxFib1ndR4RffoFHAdMyMMTgJ/UWW4bYBfg0prp5wP75OFTgcP6StzACOCx/Hd4Hh6e510LjGtRrIOAR4E1gSWBu4GxNcscDpyah/cBfpeHx+bllwLWyNsZ1A/i7gDua+W+3MO4O4D1gbOAPcvsM3057jzv5T5c3x8HlsnDhxX2k7bV96LG3ps67/NXCKRuLc7Mw2cCu3W1UERcDcwtTpMkYGvgwu7Wr0CZuD8JXBkRL0TEi8CVwPYtiq+oTHcixfdzIbBNrt9dgfMiYl5E/AOYlrfX1+Nup27jjojpEXEP8FbNuu3cZxYl7nYqE/c1EfFqHr2F9P0oaP//6KLE3mP9ISGsFBEz8/AzwEo9WHcFYE5EzM/jT5G6zmiFMnF31Y1HMb5f50u9f6/4INZdHAstk+vzn6T6LbNuVRYlboA1JN0p6TpJH6062K5iynpSZ329vhsZLGmKpFskterEDHoe9yHAX3q5brMtSuzQwzrvE11XSLoKeG8Xs75XHImIkNRnnpOtOO79I2KGpKHARcABpMtwa46ZwOiImC3pQ8AfJK0bES+1O7ABbPW8T68J/E3SvRHxaLuDKpL0GWAc8LF2x9JTdWLvUZ33iYQQEdvWmyfpWUmjImKmpFHAcz3Y9GxgmKTF89lhU7vJaELcM4CtCuOrktoOiIgZ+e9cSb8lXTpWlRDKdCfSucxTkhYHlifVbzu7Iul13JFusM4DiIg7JD0KrA1MqTzqRauzuvtMCyzSZ13Ypx+TdC2wEen+eNVKxS1pW9LJ3MciYl5h3a1q1r22kii7tiix97zOW9U4sgiNKsezcOPscQ2W3Yp3NypfwMKNyof3lbhJDVX/IDVWDc/DI0iJesW8zBKke9+HVhjr4qTGsjV4p+Fq3ZplvsTCjbPn5+F1WbhR+TFa16i8KHGP7IyT1GA3AxjRV+IuLDuZdzcqv2uf6QdxDweWysMrAo9Q0zja5v2k80C5Vs30ttV3E2LvcZ235E0tYoWsAFyd38xVnR8G6dLol4Xl/g7MAl4j3Wf7ZJ6+JnAbqbHzgs4K6kNxfy7HNg34bJ62LHAHcA9wP3ASFR9kgR2B/8s71vfytB8An8rDg3P9Tcv1uWZh3e/l9R4Gdmjx/tGruIF/y3V7FzAV2KWPxf3hvB+/QroSu7/RPtPX4wY2A+4lHdDuBQ7pY3FfBTyb94e7gEv6Qn0vSuy9qXN3XWFmZkD/eMrIzMxawAnBzMwAJwQzM8ucEMzMDHBCMDOzzAnB+jxJWyn3YivpU131+FhYdpikw3tRxkRJ3yixXHflHyzp5J6WX4ak6ZJW7MV6HZL2K4yPk/TfzY3OBgInBGsbSYN6uk5EXBIRkxosMozUw2nT5W+8d1d+j7fZrG010AG8nRAiYkpEHNGCcq2fcUKwpstnpA9JOkfSg5IulLRMnjdd0k8kTQX2krSdpJtzH/oXSBqSl9s+b2MqsEdh22+fgUtaSdLFku7Or82AScCY3Cng8Xm5b0q6PfcXf0xhW9+T9H+SbgDWqfNeJks6VdKtwHE15e8l6b5c9vVdrLtTfm8r1kyfKOlsSTcCZ0saKemiHOPtkjbPy60g6QpJ90v6JaBC/d5X2N43JE3Mw++TdFWOaaqkMblOPprr5Gs1V1wjJP0h180tktYvxHiG0u9yPCbJCeRfQJ/oy8gGpHVI34y8UdIZpLP2E/K82RHxwXyg/D2wbUS8IunbwFGSjgNOJ3VdPg34XZ0y/hu4LiJ2z1cbQ0jdhKwXERsCSNoOWIvUF5SASyRtSfom7T7AhqT/g6mkb4d3ZVVgs4hYIOngwvT/IH0jfoakYcUVJO0OHAXsGKnb5FpjgS0i4rXcV9XPIuIGSci3YjoAAALeSURBVKOBy4EPAEcDN0TEDyTtROrJsjvnAJMi4mJJg0knfROAb0TEzjm2rQrLHwPcGRG7Sdqa1F/Whnne+0l97Q8FHpZ0SkS8WSIG66ecEKwqT0bEjXn4N8ARvJMQOg/wm5IOjDcq9e69JHAz6UD0j4h4BEDSb4DxXZSxNXAgQEQsAP6pd/8y3Xb5dWceH0JKEEOBiyP3I6/GvyZ1Qd5+rRuByZLOJyW2YlzjgO2ifu+pl0TEa3l4W2Cs3unhfLl8pbQl+eooIi6T1FVieZtSz7irRMTFeZ3X8/RGq21B6saDiPhbvipZLs+7LFJHafMkPUfqwv2pRhuz/s0JwapS2ydKcfyV/FekHx/Zt7igpA1pHgE/johf1JRxZA+28UpXEyPiUEmbADsBdyh1ow3v/MJVo95Ti9tcDNi08wBeiLFePPNZ+Hbv4IbR9968wvACfLwY8NyGYFUZLekjeXg/4IYulrkF2FzS+wAkLStpbeAhoCPf/wbYt4t1IXUeeFhed5Ck5Um/mje0sMzlwOcKbROrKP0+9fXAbpKWzmfWu/T0DUoaExG3RsR/kDpW7Oym+HHSWfdZktYtsakrgK8UttuZEK8nNwZL2oHUeyWkjszek8/mlwJ2htRVOqmb793yOksptd3U1knR34H98/JbAc83uKqxAc4JwaryMPAlSQ+SDmSn1C4QEbOAg4FzJd1Dvl2Uz5THA5flRuV6v4HxVeDjku4l3f8fGxGzSbeg7pN0fERcAfwWuDkvdyEwNCKmkm5d3U36hanbe/Eej5d0b27gvSlvq/O9PUQ60F5QSGz1HAGMyw27DwCH5unHAFtKup906+iJvO03Sb1d3kb6SceHCts6ADgi1+dNpB9wugdYkBuav1ZT9kTgQ3n5SUDLf0Te+g73dmpNJ6mD9LsU67U5FDPrAV8hmJkZ4CsEMzPLfIVgZmaAE4KZmWVOCGZmBjghmJlZ5oRgZmYA/H8bh0pwWkiZzgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XXQsiAfLTmuc",
        "colab_type": "text"
      },
      "source": [
        "Plot the predicted versus empircal risk reduction for the test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hj-NkTj3Tmud",
        "colab_type": "code",
        "colab": {},
        "outputId": "96544c0d-f5c7-4cca-bbad-13cc42066f1b"
      },
      "source": [
        "# Plot the predicted versus empirical risk reduction for the test set\n",
        "empirical_benefit, avg_benefit = quantile_benefit(X_test, y_test, rr_t_test)\n",
        "plot_empirical_risk_reduction(empirical_benefit, avg_benefit, 'T Learner (test set)')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZYAAAEWCAYAAABFSLFOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3debxVZb3H8c9XRAEnUNAEVLBMRSPQI2qm11ICc05LGtVMtDS1wdJbt2veSsvboEmOmZY5ZUqYJjnrNQcOSeJETiigJoo44CEGf/eP9WxcZ5999tmHs4dz4Pt+vc7r7DXs9fz22mvv336etdbzKCIwMzOrljUaHYCZma1anFjMzKyqnFjMzKyqnFjMzKyqnFjMzKyqnFjMzKyqnFiKSHpL0pZllp8v6b+6WMaekuZ24fl/kXR4BeuFpPetbDkrS9Kdkr5Ug+3+p6SLq73dnij/3lbjmKywzCMk/V8H69wraXStY+nOJA2S9ISkvo2OZWVJ2kTS45LWzs37qqQfV/L8HpFYJM2W1JK+9At/59airIhYNyKeKbP82Ij4n1qUXZC+NBal1zlP0s8k9crFsE9EXFbF8oalMgv7drakU6q1/ZWMqU3yjYgfRUTVE1YtpPjfSfvzTUmzJB1Zi7IqPSZrlfBz298feDMiHkrJrnA8LZG0NDf9lxLP7dKPrW7mFODSiGiB6u33Wu6j9JnfuzAdEf8C7gAm5la7CPispI072l6PSCzJ/ulLv/B3fL0DyH+518EHI2Jd4D+Aw4Av1qHM/qnMQ4H/kjS2DmWuyl5I+3N94NvARZJGFK8kac26R1YbxwK/gxXJbt30+n8EXJ377O7TiOCqvZ9LbS/9wj8cuLyaZTXI74FjChMRsRj4C/CFjp7YkxJLSal6fq+kn0taKOkZSR9K8+dIejnfbCTp0vRr6pb0S/IuSVvkluebGC6VdJ6kmyQtAj6S5v0gt/6BkmZIekPS05LGp/lHpqrkmymmY1gJEfEUcC8wKlfmil9Akt6XXsPrkl6RdHU7++nDaX/sWUGZzcCjRWV+Mb2e1yRNLdpnY1PV//VUk1Ru2WmSLs9NF2pHa6bpDSX9RtILaduTJa1DdgAPzv3KHVxiWwdIejS973dK2ja3bLakb0p6OMV1taQ+JfbL2un52+fmDVJWQ95Y0kBJf07rLJB0j6ROfW4iMxl4DRhRdMy+CpyW4vhfSc9L+lc6Rlc0pUg6WdKLaT+1+pFRyTEp6YfA7sC5ytX4JW2TPgsLlNWqPpXbzkaSpqTtPAi8t73XKGkt4KPAXZ3ZN5VI7/0fJc2X9KykE3LLxki6L70/L0o6N8VSWB6SjpP0JPCk0q9+Sd9Q9t3wonI1yXLvQ+6535b0EvCbEuHuDCyMiLnpOSuz3z8u6TFl3x3z0nFc8jNRYl+1eW5u2X7puFgo6W+SRqb5vwM2B25I2/1WesoDwJbKfdaBO4F9O3zTIqLb/wGzgb3bWXYEsAw4EugF/AB4HpgErA18DHgTWDetf2ma3iMtPxv4v9z2Anhfbt3Xgd3IknCfNO8HafmYtHxsWj4E2CYt25fsgyiyWsfbwA5p2Z7A3DKvNx/DNsCLwNdyy+8EvpQeXwl8Jxffh4u3A4wH5gBj2ilvWFp3zTS9S4r34DR9IPAUsC2wJvBd4G9p2cC0Pw8FegNfS+9HIb7TgMvLlHUjcDUwID3/P9rbR/ltAe8HFqV93xv4Vopxrdwx8yAwGNgQeBw4tp3Xfwnww9z0ccDN6fEZwPmpjN5kXxKq4JhdEX96bw4GlgJb8+4x+9W0P/sCPwempFjXA24AzkjPHw/8C9geWAe4grbHaSXH5J2F9yVNr5OOiyNTHKOBV4ARaflVwDVpve2BeeQ+K0WvdztgUTvLWh0DHe2vovlrANOB7wFrAVsCzwDj0vIdyY7XNcmOrceBk4o+A7ek/do3lbMMOD29nx8nO9YHpPXLvQ+F5/6Y7Lujb4l4jwNuLJrX2f3+IrB7ejyACr83OnjuaOBlssTXi6xWNRtYu9x3LPAwcEBuegdgQYfHf0crdIe/9KLfAhbm/o5Oy44Ansyt+4F0MG2Sm/cqMCr3Ibwqt2xdYDmwWe5AzH9gf1sUy6W8+yG+APh5ha9hMnBiJQdIiuENsi/OIEsea5c6UIHfAhcCQ9vZzqnAc8D2ZcobltZdCLSkx/9L+gIl+6V0VNGH/W1gC7Jq8f25ZQLmUkFiATYF3iF9qItiarOPaJ1Y/gu4piimecCeuWPmc7nlPwHOb+f17w08nZu+F/hCenw68KfCMdGJY3bP9NoWAguAGcCE3DH7fNE+WwS8NzdvV+DZ9PgS4MzcsvfTfmJp95ik7RfcYcA9RetcAPw32ZfPUlJSSst+RPuJZTfgpXaWtToGyuyvUoll5/y+SvNOBX7TznZOAq4v+gx8tKicFtIPmzTvZbLk1NH7sCewBOhT5nV8h9z3S2f3e3r8PFkT1PqV7KOiddp77nnA/xTNm8W7P+RmUzqxrPgspOmtgOUdHf89qSnsoIjon/u7KLfsX7nHLbDi5FN+3rq56TmFBxHxFtkHv021snjdEjYDni61QNI+ku5PVd2FZL+MBpbZVrEdUsyHkX241mlnvW+RfSAeTM1CxediTiL7An6kgjIHpjK/QXYQ907ztwDOTlXowhelyH4ND6b1/gzK77O8zch+/bxW4fp5g8kSZqHcd1K5Q3LrvJR7/Datj4G8O4B+knaWNIysCfD6tOwssprQX5U1aXbmooYX0rG6YUSMioircsvy+2gQ0A+YntvHN6f5ULSPyb3uEto9JkvYAti5UGYq97PAe1LZa3ai3NfIfuFX2xZkzT/5GP8T2ARA0vtTU+VLkt4gS37Fn7Pi4/HViFiWmy4cGx29DwDzIzvX0J5K9kO5/Q5wCNn3xXPKmrl37WB7ee09dwvgG0Vlbkb733sF65H9OMpPv95RED0psVTTZoUHktYlq/a+0M66UWY7cyjR7qzsBN4fyX71bxIR/YGbyJ17qERkrgHuI2sKKLXOSxFxdEQMJvul8iu1vsT4k8BBkk6ssMzlEfEzYDHwlTR7DnBMUWLvGxF/I6t65/en8tNkvwD75abfk3s8B9hQUv9SoXQQ6gtkH5bicud18Ly2BUUsJ2vy+XT6+3NEvJmWvRkR34iILYEDgK9L2quzZZQqNvf4FbIfP9vl9u8GkZ34hqJ9TNYe3p6Sx2SJMgvr3lX0vq4bEV8G5pM1+1Ra7lNkb8OQMuusjDlkNYZ8jOtFxMfT8vOAJ4CtImJ9sqRT/Dnr6Fgq6Oh9qGRbD5PVKMuVX26/ExHTIuJAYGOylo5rKn0dZZ47h6y5N19mv4i4sr1tKzsP+j7gH7nZ2xZNl7S6JpaPKzuZvRbwP2RNOZX+ys77NXCkpL0krSFpiKRtyNqC1yZ9OCXtQ3auZ2WdCRwt6T3FCyR9UtLQNPka2QHyTm6VF4C9gBMlfbmTZX5L2Qnv84FTJW2XytxA0ifTejcC20n6RDoQT6B18pgB7CFpc0kbkDVjABARL5I1s/1K0gBJvSXtkRb/C9goPaeUa4B9077vTVbL+jfwt068xrwryGqHn02PSa91P2UXSIjsl9pyWu/fLku1rYuAnytdypmOpXFplWuAIySNkNSPrKmqPe0dk5Dt0/w9Wn8G3i/p82nf95a0k6RtU7K9juzCgn7KrmY7vMxrWALcSnY+caVJ6pP/IztP9qayE+Z9JfWStL2kndJT1iNrNn4rvc7OHOPFr6Gj96ESDwL9ixJsxftd0lqSPitpg4hYml7bO7nttPuZ6OC5FwHHplq5JK0jaV9J6+W2XXz/3hhgdkTka6r/QfaZLasnJZbCFQuFv+s7fkq7riD7cC4gO/n3uZXZSEQ8SHYC7udkXzp3AVukX7snkH0hvAZ8huyE4EqJiJnA3cDJJRbvBDwg6a1UxolRdB9ORDxPllxOUeXX09+YYj86Iq4nO2F5VWpueATYJ237FbJa0Zlk57K2ImuXLZR9C9nJ+YfJTsL+uaicz5O15T9B1tZ9UnreE2Tnlp5JVfdWVfaImEX2vv2S7Jfm/mSXpC+p8PW1EhEPkNWuBtP6g7MV2RfmW2Q1x19FxB2w4kbV/1yZ8kr4Ntmv/vvTPr6V7EQ/EfEX4BfA7Wmd28u8jpLHZFp8NnCosqvvzknH6ceACWQ/QF7i3RPTAMeTNRG9RHYe5zcdvIYLyN7PlTWErMaQ/xsO7EfWPPks2Xt9MVD4cv0m2efrTbIvz5JXRXZCu+9DJdLxdymtv1M6u98/D8xO5R9L9mOnw89EB89tBo4GziX7XD9Fdq6v4Azgu2m7hSvJPkv2oxLIkj5ZM1uH99AVTs6uNiRdSnYC7LuNjsVsVSPpXuD4iHio0bE0iqRBwD3A6Eg3SfY0qcZ2F9lrWJzmfZXsIqdvlX0yTixmZlZlPakpzMzMeoDVrsZiZma15RqLmZlV1arS+d0KAwcOjGHDhjU6DDOzHmX69OmvRMSgjtfs2CqXWIYNG0Zzc3OjwzAz61EkletZoVPcFGZmZlXlxGJmZlXV0MSibJyIWZKeUjud+0n6lLLxBR6VdEWpdczMrPto2DkWZaMxTiIbN2IuME3SlIh4LLfOVmR9S+0WEa+pgiExzcyssRpZYxkDPBURz6T+da4iG1Aq72hgUqFb9Yh4uc4xmplZJzXyqrAhtB4nYS7ZuCN574cV/Q/1Ak6LiJuLNyRpIjARYPPNy/XsbZWY/NA8zpo6ixcWtjC4f19OHrc1B42udm/oZraq6u6XG69J1rvsnsBQ4G5JH4iI/MAzRMSFZKMo0tTU5K4EumDyQ/M49bqZtCxdDsC8hS2cet1MACcXM6tII5vC5tF6EKGhtB2kaS4wJSKWRsSzwD/JEo3VyFlTZ61IKgUtS5dz1tRZDYrIzHqaRiaWacBWkoanAbcm0HbMkslktRUkDSRrGnsGq5kXFpbu5bu9+WZmxRqWWNKY08cDU4HHycZlf1TS6ZIOSKtNBV6V9BjZuOQnR8SrjYl49TC4f99OzTczK9bQcywRcRPZWPD5ed/LPQ7g6+nP6uDkcVu3OscC0Ld3L04eV/Egema2muvuJ++tzgon6H1VmJmtLCcWa+Og0UOcSMxspbmvMDMzqyonFjMzqyonFjMzqyonFjMzqyonFjMzqyonFjMzqyonFjMzqyonFjMzqyonFjMzqyonFjMzqyonFjMzqyonFjMzqyonFjMzqyonFjMzqyonFjMzq6qGJhZJ4yXNkvSUpFNKLD9C0nxJM9LflxoRp5mZVa5hA31J6gVMAsYCc4FpkqZExGNFq14dEcfXPUAzM1spjayxjAGeiohnImIJcBVwYAPjMTOzKmhkYhkCzMlNz03zih0i6WFJ10rarD6hmZnZyuruJ+9vAIZFxEjgFuCyUitJmiipWVLz/Pnz6xqgmZm11sjEMg/I10CGpnkrRMSrEfHvNHkxsGOpDUXEhRHRFBFNgwYNqkmwZmZWmUYmlmnAVpKGS1oLmABMya8gadPc5AHA43WMz8zMVkLDrgqLiGWSjgemAr2ASyLiUUmnA80RMQU4QdIBwDJgAXBEo+I1M7PKKCIaHUNVNTU1RXNzc6PDMDPrUSRNj4imamyru5+8NzOzHsaJxczMqsqJxczMqsqJxczMqsqJxczMqsqJxczMqqph97GYma2syQ/N46yps3hhYQuD+/fl5HFbc9DoUl0NWiM4sZhZjzL5oXmcet1MWpYuB2DewhZOvW4mgJNLN+GmMDPrUc6aOmtFUiloWbqcs6bOalBEVsyJxcx6lBcWtnRqvtWfE4uZ9SiD+/ft1Hyrvw7PsUhaGzgEGJZfPyJOr11YZmalnTxu61bnWAD69u7FyeO2bmBUllfJyfs/Aa8D04F/d7CumVlNFU7Q+6qw7quSxDI0IsbXPBIzswodNHqIE0k3Vsk5lr9J+kDNIzEzs1VCJTWWDwNHSHqWrClMQKRx6M3MzFqpJLHsU/MozMxsldFhU1hEPAf0B/ZPf/3TPDMzszY6TCySTgR+D2yc/i6X9NVqFC5pvKRZkp6SdEqZ9Q6RFJKqMmymWWdMfmgeu515O8NPuZHdzrydyQ/Na3RIZt1aJU1hRwE7R8QiAEk/Bu4DftmVgiX1AiYBY4G5wDRJUyLisaL11gNOBB7oSnlmK8P9Upl1XiVXhQnId8yzPM3rqjHAUxHxTEQsAa4CDiyx3v8APwYWV6FMs05xv1RmnVdJYvkN8ICk0ySdBtwP/LoKZQ8B5uSm56Z5K0jaAdgsIm4styFJEyU1S2qeP39+FUIzy7hfKrPOq+Tk/c+AI4EF6e/IiPhFrQOTtAbwM+AbHa0bERdGRFNENA0aNKjWodlqxP1SmXVeu4lF0vrp/4bAbODy9PdcmtdV84DNctND07yC9YDtgTslzQZ2Aab4BL7V08njtqZv716t5rlfKrPyyp28vwLYj6yPsMjNV5resotlTwO2kjScLKFMAD5TWBgRrwMDVxQq3Ql8MyKau1iuWcXcL5VZ57WbWCJiv/R/eC0Kjohlko4HpgK9gEsi4lFJpwPNETGlFuWadZb7pTLrnEq6zb8tIvbqaN7KiIibgJuK5n2vnXX37Gp5ZmZWe+0mFkl9gH7AQEkDePcS4/UpunrLzMysoFyN5RjgJGAw2XmWQmJ5Azi3xnGZmVkPVe4cy9nA2ZK+GhFdusvezMxWH5XcIPmOpP6FCUkDJH2lhjGZmVkPVkliOToiFhYmIuI14OjahWRmZj1ZJYmll6QVfYOlziPXql1IZmbWk1XSu/HNwNWSLkjTx6R5ZmZmbVSSWL5Nlky+nKZvAS6uWURmZtajdZhYIuId4Lz0ZzU0+aF57jrEzHq8Su68f5bWfYUBEBFd7SvMcjyglJmtKippCsv3JtwH+CRQjd6NLafcgFJOLGbWk1TSFPZq0axfSJoOlOzTy1aOB5Qy63ncfF1aJU1hO+Qm1yCrwVRS07FOGNy/L/NKJBEPKGXWPbn5un2V3Mfy09zfGcCOwKdqGdTqyANKmfUs5ZqvV3eVNIV9pB6BrO48oJRZz+Lm6/aV6zb/6+WeGBE/q344qzcPKGXWc7j5un3lmsLWS39NZDdHDkl/xwI7lHmemdkqz83X7SvXbf73ASTdDewQEW+m6dOAG6tRuKTxwNlkQxNfHBFnFi0/FjgOWA68BUyMiMeqUbaZWVe4+bp9lVzdtQmwJDe9JM3rktSZ5SRgLDAXmCZpSlHiuCIizk/rHwD8DBjf1bLNzKrBzdelVZJYfgs8KOn6NH0QcFkVyh4DPBURzwBIugo4EFiRWCLijdz661CiBwAzM+teKrkq7IeS/gLsnmYdGREPVaHsIcCc3PRcYOfilSQdB3ydrKv+j5bakKSJwESAzTffvAqhmZnZyqrkPhaAfsAbabjiuZKG1zCmViJiUkS8l6yX5e+2s86FEdEUEU2DBg2qV2hmZlZCh4lF0n+Tfamfmmb1Bi6vQtnzgM1y00PTvPZcRdYMZ2Zm3VglNZaDgQOARQAR8QLZZchdNQ3YStJwSWsBE4Ap+RUkbZWb3Bd4sgrlmplZDVVy8n5JRISkAJC0TjUKjohlko4HppJdbnxJRDwq6XSgOSKmAMdL2htYCrwGHF6Nss3MrHYqSSzXpGGJ+0s6GjiKKo0gGRE3ATcVzfte7vGJ1SjHzMzqp5Krwv5X0ljgDWBr4L8i4paaR2ZmZj1S2cSSbmIckBLJLelcyBGSHo+IbesSoZmZ9SjtnryXNAFYADws6S5JHwOeAfYBPlun+MzMrIcpV2P5LrBjRDyVBvu6Dzg0Im6oT2hmZtYTlbvceElEPAUQEX8HnnRSMTOzjpSrsWxcNCZL//y0x2MxM7NSyiWWi2h9I2TxtJmZWRsdjsdiZmbWGZV2QmlmZlYRJxYzM6uqSno3XrvEvA1rE46ZmfV0ldRYrpPUuzAhaVPAXbqYmVlJlSSWyWQdUfaSNIysN+JTyz7DzMxWW5V0QnlR6iNsMjAMOCYi/lbrwMzMrGdqN7EU3RwpYHNgBrCLpF18g6SZmZVSrsZSfDPkde3MNzMzW6FTN0hKWgNYNyLeqGlUDXTYBfe1mbffyE35/K7DaFmynCN+82Cb5YfuOJRPNm3GgkVL+PLl09ss/9wuW7D/BwfzwsIWvnb1jDbLj959S/YesQlPz3+L/7xuZpvlX/3oVnx4q4E8+sLrnH7DY22Wf2v81uy4xYZMf24BP7l5Vpvl39t/BNsN3oD/e/IVfnl729Gdf/SJD/DeQety62P/4qJ7nmmz/OeHjWJw/77c8I8XuPz+59osP+9zO7LhOmvxh+Y5XDt9bpvllx45hr5r9eJ3983mzw+/2Gb51cfsCsCFdz/NbY+/3GpZn969uOyLYwA457YnufepV1otH9BvLc7//I4A/PjmJ/j7c6+1Wr7pBn34xYTRAHz/hkd57IXWh+6Wg9bhjE+MBODU6x7mmfmLWi0fMXh9/nv/7QA46aqHePH1xa2W77DFAL49fhsAjv3ddF57e0mr5bu9byAn7JWNsH34JQ+yeOnyVsv32nZjJu7xXsDHno+9rh97hdfTaJVcbnyFpPXTkMSPAI9JOrkahUsaL2mWpKcknVJi+dclPSbpYUm3SdqiGuWamVntKCLKryDNiIhRkj4L7ACcAkyPiJFdKjgbROyfwFhgLjAN+HREPJZb5yPAAxHxtqQvA3tGxGHlttvU1BTNzc1dCc3MbLUjaXpENFVjW5Vcbtw73cdyEDAlIpYC5bNRZcYAT0XEMxGxBLgKODC/QkTcERFvp8n7gaFVKNfMzGqoksRyATAbWAe4OzVHVeMcyxBgTm56bprXnqOAv5RaIGmipGZJzfPnz69CaGZmtrI6TCwRcU5EDImIj0fmOeAjdYhtBUmfA5qAs9qJ8cKIaIqIpkGDBtUzNDMzK1LuPpbPRcTlRfez5HX1PpZ5wGa56aFpXnEcewPfAf4jIv7dxTLNzKzGyt3Hsk76X6v7VqYBW0kaTpZQJgCfya8gaTRZU9z4iHi57SbMzKy7KXcfywXpf6n7WdZp+4zOiYhlko4n63usF3BJRDwq6XSgOSKmkDV9rQv8QRLA8xFxQFfLNjOz2inbV5ikIcCmwMMRsUTSxsBJwBHA4K4WHhE3ATcVzfte7vHeXS3DzMzqq92T95JOIusb7JfA/ZK+BDwO9AV2rE94ZmbW05SrsUwEto6IBZI2J7uZcbeIaNtvhJmZWVLucuPFEbEAICKeB2Y5qZiZWUfK1ViGSjonN71pfjoiTqhdWGZm1lOVSyzFHU26tmJmZh0qd7nxZfUMxMzMVg2V9BVmZmZWMScWMzOrKicWMzOrqnKdUP6SMuOu+KowMzMrpdxVYR6G0cysB5j80DzOmjqLFxa2MLh/X04etzUHjS43vFVt+aowM7MebPJD8zj1upm0LF0OwLyFLZx63UyAhiWXsp1QAkgaBHwbGAH0KcyPiI/WMC4zM6vAWVNnrUgqBS1Ll3PW1FkNSyyVnLz/PVnnk8OB75MNUzythjGZmVmFXljY0qn59VBJYtkoIn4NLI2IuyLii4BrK2Zm3cDg/n07Nb8eKkksS9P/FyXtm0Z13LCGMZmZWYVOHrc1fXv3ajWvb+9enDxu6wZFVME5FuAHkjYAvkE2Nsv6wNdqGpWZmVWkcB6lO10Vpoh2b1WpfeHSeOBssqGJL46IM4uW7wH8AhgJTIiIazvaZlNTUzQ3+0ppM7POkDQ9Ipqqsa0Om8IkXSapf256gKRLulqwpF7AJGAfsivOPi1pRNFqz5MNg3xFV8szM7P6qKQpbGRELCxMRMRr6TxLV40BnoqIZwAkXQUcCDyWK2t2WvZOFcozM7M6qOTk/RqSBhQmJG1IZQmpI0OAObnpuWlep0maKKlZUvP8+fOrEJqZma2sShLET4H7JP0BEHAo8MOaRtVJEXEhcCFk51gaHI6Z2Wqtw8QSEb+V1My79658IiIeK/ecCs0DNstND03zzMysByvXu/H6EfFGavp6idwJdEkbRsSCLpY9DdhK0nCyhDIB+EwXt2lmZg1WrsZyBbAf2Vj3+eYlpektu1JwRCyTdDwwlexy40si4lFJpwPNETFF0k7A9cAAYH9J34+I7bpSrpmZ1VbZ+1gkCdgsIp6vX0hd4/tYzMw6r273sUSWdW6sRkFmZrZ6qORy47+nJikzM7MOVXK58c7AZyU9BywinWOJiJE1jczMzHqkShLLuJpHYVZCdxtu1TJ+X6wjHV5uDLxZx3jMgO453Kr5fbHKlDvHUrhvZTrQnP5Pz02b1Uy54Vatcfy+WCXarbFExH7p//D6hWOW6Y7DrZrfF6tMRZ1JSvoE8GGyGyPviYjJNY3KVnuD+/dlXokvq0YOt2p+X6wylYzH8ivgWGAm8AhwrKRJtQ7MVm/dcbhV8/tilamkxvJRYNt0sySSLgMerWlUttrrjsOtmt8Xq0wlieUpYHPguTS9WZpnVlMHjR7iL6xuyO+LdaSSxLIe8LikB9P0TkCzpCkAEXFArYIzM7Oep5LE8r2aR2FmZquMSgb6uguyGybz61dhPBYzM1sFdZhYJE0ETgcWA+9QpfFYzMxs1VRJU9jJwPYR8UqtgzEzs56vkm7znwbernUgZma2aqikxnIq8DdJDwD/LsyMiBO6Wrik8cDZZEMTXxwRZxYtXxv4LbAj8CpwWETM7mq5Zj2RexW2nqKSxHIBcDvZnffvVKtgSb2AScBYYC4wTdKUiHgst9pRwGsR8T5JE4AfA4dVKwaznsK9CltPUkli6R0RX69B2WOApyLiGQBJVwEHAvnEciBwWnp8LXCuJBV6ATBbXZTrVdiJxbqbSs6x/EXSREmbStqw8FeFsocAc3LTc9O8kutExDLgdWCj4g2l+JolNc+fP78KoZl1L+5V2HqSSmosn07/T83N61aXG0fEhcCFAE1NTa7N2CrHvQpbT9JhjSUihpf4q0ZSmUfW71jB0DSv5DqS1gQ2IDuJb7Zaca/C1pO0m1gkfSv3+JNFy35UhbKnAVtJGi5pLWACMKVonSnA4enxocDtPr9iq6ODRg/hjE98gCH9+yJgSP++nPGJD/j8io8+zvwAABOuSURBVHVLau97WtLfI2KH4selple6cOnjwC/ILje+JCJ+KOl0oDkipkjqA/wOGA0sACYUTva3p6mpKZqbPXKymVlnSJoeEU3V2Fa5cyxq53Gp6ZUSETcBNxXN+17u8WLgk8XPMzOz7qvcOZZo53GpaTMzM6B8jeWDkt4gq530TY9J031qHpmZmfVI7SaWiOjV3jIzM7P2VHKDpJmZWcWcWMzMrKqcWMzMrKqcWMzMrKqcWMzMrKqcWMzMrKqcWMzMrKqcWMzMrKqcWMzMrKqcWMzMrKqcWMzMrKqcWMzMrKqcWMzMrKqcWMzMrKoaklgkbSjpFklPpv8D2lnvZkkLJf253jGamdnKaVSN5RTgtojYCrgtTZdyFvD5ukVlZmZd1qjEciBwWXp8GXBQqZUi4jbgzXoFZWZmXdeoxLJJRLyYHr8EbNKVjUmaKKlZUvP8+fO7Hp2Zma20cmPed4mkW4H3lFj0nfxERISk6EpZEXEhcCFAU1NTl7ZlZmZdU7PEEhF7t7dM0r8kbRoRL0raFHi5VnGYmVl9NaopbApweHp8OPCnBsVhZmZV1qjEciYwVtKTwN5pGklNki4urCTpHuAPwF6S5koa15BozcysYjVrCisnIl4F9ioxvxn4Um5693rGZWb1t3TpUubOncvixYsbHcpqoU+fPgwdOpTevXvXrIyGJBYzs4K5c+ey3nrrMWzYMCQ1OpxVWkTw6quvMnfuXIYPH16zctyli5k11OLFi9loo42cVOpAEhtttFHNa4dOLGbWcE4q9VOPfe3EYmZmVeXEYmbWg/Tq1YtRo0ax/fbbs//++7Nw4UIAZs+eTd++fRk1ahQjRozgC1/4AkuXLm1IjE4sZmZVtnz58pptu2/fvsyYMYNHHnmEDTfckEmTJq1Y9t73vpcZM2Ywc+ZM5s6dyzXXXFOzOMrxVWFm1q0cdsF9bebtN3JTPr/rMFqWLOeI3zzYZvmhOw7lk02bsWDREr58+fRWy64+Ztey5R100EHMmTOHxYsXc+KJJzJx4kTOP/98nn76ac466ywALr30Upqbmzn33HO5/PLLOeecc1iyZAk777wzv/rVr+jVqxfrrrsuxxxzDLfeeiuTJk3i9ttv54YbbqClpYUPfehDXHDBBUhi2rRpHHXUUayxxhqMHTuWv/zlLzzyyCMsX76cU045hTvvvJN///vfHHfccRxzzDFlY9911115+OGH28zv1asXY8aMYd68eWWfXyuusZjZau2SSy5h+vTpNDc3c8455/Dqq69yyCGHcP31169Y5+qrr2bChAk8/vjjXH311dx7773MmDGDXr168fvf/x6ARYsWsfPOO/OPf/yDD3/4wxx//PFMmzaNRx55hJaWFv7852xYqSOPPJILLrhgxfMLfv3rX7PBBhswbdo0pk2bxkUXXcSzzz7bbtzLly/ntttu44ADDmizbPHixTzwwAOMHz++WrupU1xjMbNupVwNo+9avcou33CdtTqsoRQ755xzViSROXPm8OSTT7LLLruw5ZZbcv/997PVVlvxxBNPsNtuuzFp0iSmT5/OTjvtBEBLSwsbb7wxkNUSDjnkkBXbveOOO/jJT37C22+/zYIFC9huu+3YfffdefPNN9l11yzGz3zmMysSzl//+lcefvhhrr32WgBef/11nnzyyTb3m7S0tDBq1CjmzZvHtttuy9ixY1cse/rppxk1ahTPPvss++67LyNHjuzUvqgWJ5Zk8kPzOGvqLF5Y2MLg/n05edzWHDR6SKPDMrMauvPOO7n11lu577776NevH3vuueeKezwmTJjANddcwzbbbMPBBx+MJCKCww8/nDPOOKPNtvr06bOiBrJ48WK+8pWv0NzczGabbcZpp53W4b0jEcEvf/lLxo0r33NV4RzL22+/zbhx45g0aRInnHAC8O45lldeeYXddtuNKVOmlKzR1JqbwsiSyqnXzWTewhYCmLewhVOvm8nkhxrTPmlm9fH6668zYMAA+vXrxxNPPMH999+/YtnBBx/Mn/70J6688komTJgAwF577cW1117Lyy9nHbIvWLCA5557rs12C0lk4MCBvPXWWytqIf3792e99dbjgQceAOCqq65a8Zxx48Zx3nnnrbiS65///CeLFi1qN/Z+/fpxzjnn8NOf/pRly5a1WjZw4EDOPPPMkgmwHpxYgLOmzqJlaeurOFqWLuesqbMaFJGZ1cP48eNZtmwZ2267Laeccgq77LLLimUDBgxg22235bnnnmPMmDEAjBgxgh/84Ad87GMfY+TIkYwdO5YXX3yxzXb79+/P0Ucfzfbbb8+4ceNWNJ1Bdi7l6KOPZtSoUSxatIgNNtgAgC996UuMGDGCHXbYge23355jjjmmTcIoNnr0aEaOHMmVV17ZZtlBBx3E22+/zT333LNS+6YrFLFqjYvV1NQUzc3NnXrO8FNupNReEPDsmftWJS4zK+3xxx9n2223bXQYdfPWW2+x7rrrAnDmmWfy4osvcvbZZ9c1hlL7XNL0iGiqxvZ9jgUY3L8v8xa2lJxvZlZNN954I2eccQbLli1jiy224NJLL210SFXnxAKcPG5rTr1uZqvmsL69e3HyuK0bGJWZrYoOO+wwDjvssEaHUVNOLLDi6i9fFWbWGBHhjijrpB6nP5xYkoNGD3EiMWuAPn368Oqrr7rr/DoojMfSp0+fmpbTkMQiaUPgamAYMBv4VES8VrTOKOA8YH1gOfDDiLi6vpGaWa0NHTqUuXPnMn/+/EaHsloojCBZS42qsZwC3BYRZ0o6JU1/u2idt4EvRMSTkgYD0yVNjYiF9Q7WzGqnd+/eNR3N0OqvUfexHAhclh5fBhxUvEJE/DMinkyPXwBeBgbVLUIzM1spjUosm0RE4a6il4BNyq0saQywFvB0rQMzM7OuqVlTmKRbgfeUWPSd/EREhKR2L1OQtCnwO+DwiHinnXUmAhMBNt9885WO2czMuq4hd95LmgXsGREvpsRxZ0S0uWlE0vrAncCPIuLaCrc9H2jbeU/lBgKvdOH51dAdYgDHUcxxtNYd4ugOMcCqEccWEVGV0w2NOnk/BTgcODP9/1PxCpLWAq4HfltpUgHo6o6R1Fytbg16cgyOw3H0hDi6QwyOo61GnWM5Exgr6Ulg7zSNpCZJF6d1PgXsARwhaUb6G9WYcM3MrFINqbFExKvAXiXmNwNfSo8vBy6vc2hmZtZF7ja/rQsbHQDdIwZwHMUcR2vdIY7uEAM4jlZWuW7zzcyssVxjMTOzqnJiMTOzqnJiSSRdIullSY80MIbNJN0h6TFJj0o6sUFx9JH0oKR/pDi+34g4Uiy9JD0k6c+NiiHFMVvSzHR1YueGKK1eDP0lXSvpCUmPS9q1ATFsnbtKc4akNySdVO84UixfS8fnI5KulFTbLnvbj+PEFMOj9dwXpb6zJG0o6RZJT6b/A+oVT54Ty7suBcY3OIZlwDciYgSwC3CcpBENiOPfwEcj4oPAKGC8pF06eE6tnAg83qCyi30kIkY18D6Bs4GbI2Ib4IM0YL9ExKy0D0YBO5J1Fnt9veOQNAQ4AWiKiO2BXsCEBsSxPXA0MIbsPdlP0vvqVPyltP3OKnTwuxVwW5quOyeWJCLuBhY0OIYXI+Lv6fGbZF8cdR8kJjJvpcne6a/uV3lIGgrsC1zc0bqrOkkbkN3X9WuAiFjSDXr63gt4OiK60tNFV6wJ9JW0JtAPeKEBMWwLPBARb0fEMuAu4BP1KLid76wOO/itByeWbkrSMGA08ECDyu8laQZZr9K3REQj4vgF8C2gZB9xdRbAXyVNT33T1dtwYD7wm9Q0eLGkdRoQR94E4MpGFBwR84D/BZ4HXgRej4i/NiCUR4DdJW0kqR/wcWCzBsRR0KkOfmvFiaUbkrQu8EfgpIh4oxExRMTy1NwxFBiTqvx1I2k/4OWImF7Pcsv4cETsAOxD1kS5R53LXxPYATgvIkYDi2hQMwes6HLpAOAPDSp/ANmv8+HAYGAdSZ+rdxwR8TjwY+CvwM3ADLKBCRsusntJGnI/iRNLNyOpN1lS+X1EXNfoeFJzyx3U//zTbsABkmYDVwEfldSwnhjSL2Qi4mWycwpj6hzCXGBuruZ4LVmiaZR9gL9HxL8aVP7ewLMRMT8ilgLXAR9qRCAR8euI2DEi9gBeA/7ZiDiSf6WOfQs9w7/ciCCcWLoRZQN+/xp4PCJ+1sA4Bknqnx73BcYCT9Qzhog4NSKGRsQwsiaX2yOi7r9IASStI2m9wmPgY2RNIHUTES8BcyQVegHfC3isnjEU+TQNagZLngd2kdQvfW72okEXeUjaOP3fnOz8yhWNiCMpdPAL7XTwWw+N6t2425F0JbAnMFDSXOC/I+LXdQ5jN+DzwMx0fgPgPyPipjrHsSlwmaReZD8+romIhl7u22CbANdn31+sCVwRETc3II6vAr9PzVDPAEc2IIZCch0LHNOI8gEi4gFJ1wJ/J7ua8iEa153JHyVtBCwFjqvXRRWlvrPIOvS9RtJRZMOHfKoesbSJzV26mJlZNbkpzMzMqsqJxczMqsqJxczMqsqJxczMqsqJxczMqsqJxcqStDz1YvuIpD+kbitWdluXSjo0Pb64XAebkvaU1Okb3lIvxAPbmT9T0sOS7pK0RW7Z3zrY5lvllqd18vvphsJ9QJ2Iu8My2nneKEkfz00fIKkqd+RLGirpT6mn3GcknStp7Wpsu6icVu+1pGMlfSE9XnHMWM/hxGIdaUm92W4PLAGOzS9MHQB2WkR8KSLK3eC3J9W/k/ojETESuBP4bi6WapST308LgOOqsM1KjCLrnwqAiJgSEWd2daPppsPrgMmpp9ytgL7AT7q67RL2JPdeR8T5EfHbGpRjdeLEYp1xD/C+9AvzHklTgMdSh5VnSZqWagTHQPbllH7lzpJ0K7BxYUOS7pTUlB6Pl/R3ZeO/3JY64DwW+FqqBeyeegP4YypjmqTd0nM3kvRXZWNhXAyogtdxH7leowu1BUmbSro7V/PYPf8kSQMl3Sdp30q3L+m9km5W1nnlPZK2SfOHp23NlPSDXBl7Kjf2TNp/R6THO0n6W9pPDyrr8fh04LAU82GSjpB0blp/mKTb03tym7I7wwu1gHPStp5pp0bwUWBxRPwGsr7jgK8BX5C0br6ctM0/S9ozPT5PUrOKxvJJtcbvp/d6pqRt2nmvT5P0zeKAJO2YapvTJU1V6rrEuh8nFqtIqpnsA8xMs3YAToyI9wNHkfUuuxOwE3C0pOHAwcDWwAjgC5SogUgaBFwEHJLGf/lkRMwGzgd+nmoB95CNRfLzVMYhvNuV/n8D/xcR25H14bV5BS9nPDC5xPzPAFNT55sfJOtQsBDnJsCNwPci4sb2Nqyst4K9yLrWgOxu8K9GxI7AN4Ffpflnk3Uo+QGy3nnLUna3/dVk+/yDZH1lLQK+B1yd9tPVRU/7JXBZqqX9Hjgnt2xT4MPAfmR3axfbDmjVAWjqEHU20NF4I99JY9aMBP5D0sjcsldSZ57nAd9s570u9fp7p9dzaNqXlwA/7CAOaxB36WId6at3u5e5h6wvsw8BD0bEs2n+x4CRuV++G5A1newBXJl+7b4g6fYS298FuLuwrYhob0ycvYERWQsNAOsr6wV6D9L4FxFxo6TXyryWOyRtCLwF/FeJ5dOAS9KX2OSIKLzu3mSDJh0XEXe1s+3CfhpC1mfVLSm+DwF/yMVdOEexG1mCBPgdWQ+55WwNvBgR02DFlzy57ZayK++ODfI7WjdjTY6Id8hqnNXuWv1TyoYWWJMsgY0AHk7LCh2rTqdz45ZsDWxPtl8hG9irw4RsjeHEYh1pSb/gV0gf7EX5WWS/yqcWrfdxqmcNYJeIWFwilkp9BFhI9uv9+8DX8wsj4m5l3eHvC1wq6WeprX8Z2RfhOLKBnEppiYhRyi5umEp2juVSYGHx/ssXWWLeMlq3JNRquN1/5x6X2omPAa2ayCStD7wHmEX2Jd8mzlRT/SawU0S8JulSWr+GQrnL6dz3j4BHI6LuwzFb57kpzKphKvDl9EsfSe9X1lHh3WTt/71Se/hHSjz3fmCP9IVEqlEAvAmsl1vvr2SdMJLWK3xZ303WhIWkfYCyY3ynUf5OIjtXsGF+mbIrxf4VEReRNbUVuqUP4IvANpK+3cH23yYbMvcbZMP2Pivpk2n7kvTBtOq9vDuU7mdzm3iOrGa2trIry/ZK82cBm0raKW1rvdQ8Wbyf8v5WVEbJZqZ23Ab007tXZ/UCfgqcGxEtZE1ioyStIWkz3h1GYH2yHx2vp5rQPhWUVe41FMwCBknaNcXTW9J2nXg9VkdOLFYNF5P9wv27pEeAC8h+jV4PPJmW/ZbspHYrETEfmAhcJ+kfZOcRAG4ADi6c0CWNb55ORD/Gu1enfZ8sMT1K1rTyfEfBphH2rqTtlVt7Av+Q9BBwGNl5kMJzlpN1Ff9RSV/pYPsPkTX9fJrsC/2o9NoeJRucCuBEsgHDZpK7kCAi5gDXkHXLfw1Zr71ExJIU0y/Ttm4hqwncQZaIZkg6rCiUrwJHSnqYrNfsE8vvmVavIcjOkR0q6UngVeCdiCic17gXeJbsvT2HrJdhIuIfKeYnyLqPv7eC4orf61LxLCGrQf04vf4ZNGj8FeuYezc2sw4pu8/kSuDgiPh7o+Ox7s2JxczMqspNYWZmVlVOLGZmVlVOLGZmVlVOLGZmVlVOLGZmVlVOLGZmVlX/D2NQZNDYILbcAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-SS52UIXTmul",
        "colab_type": "text"
      },
      "source": [
        "Evaluate the T-learner's performance using the test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-04T15:32:45.849067Z",
          "start_time": "2020-04-04T15:32:45.502487Z"
        },
        "colab_type": "code",
        "id": "tGFuQSpLJnym",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 970
        },
        "outputId": "6cc2307e-7abf-40be-df49-8be92147e4c1"
      },
      "source": [
        "# calculate the c-for-benefit of the t-learner on the test set\n",
        "c_for_benefit_tlearner_test_set = c_statistic(rr_t_test, y_test, X_test.TRTMT)\n",
        "print(f\"C-for-benefit statistic on test set: {c_for_benefit_tlearner_test_set:.4f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "C-for-benefit statistic on test set: 0.5250\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dPU-rBHmTmup",
        "colab_type": "text"
      },
      "source": [
        "##### Expected output\n",
        "\n",
        "```CPP\n",
        "C-for-benefit statistic on test set: 0.5250\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ihGyqKsEfJa0"
      },
      "source": [
        "The c-for-benefit of the two models were evaluated on different test sets.  However, we can compare their c-for-benefit scores to get a sense of how they perform:\n",
        "- logistic regression: 0.5412\n",
        "- T-learner: 0.5250\n",
        "\n",
        "The T-learner doesn't actually do better than the logistic regression in this case.  You can try to tune the hyperparameters of the T-Learner to see if you can improve it.\n",
        "\n",
        "### Note\n",
        "While the more flexible ML techniques may improve predictive power, the sample size is too small to be certain. \n",
        "- Models like the T-learner could still be helpful in identifying subgroups who will likely not be helped by treatment, or could even be harmed by treatment.  \n",
        "- So doctors can study these patients in more detail to find out how to improve their outcomes. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "PHjwt4UYoqy7"
      },
      "source": [
        "## Congratulations\n",
        "\n",
        "You've finished the assignment for Course 3 Module 1! We've seen that machine learning techniques can help determine when a treatment will have greater treatment effect for a particular patient."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kyxXbYDFTmur",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}