Decision Trees with package rpart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decision Trees with Package `rpart`\n",
    "====\n",
    "\n",
    "Package `rpart` is used to build a decision tree on the bodyfat data . Function `rpart()` is used to build a decision tree, and the tree with the minimum prediction error is selected. After that, it is applied to new data to make prediction with function `predict()`.\n",
    "\n",
    "\n",
    "\n",
    "`mboost` package - Model-Based Boosting\n",
    "\n",
    "`bodyfat` - For 71 healthy female subjects, body fat measurements and several anthropometric measurements are available for predictive modelling of body fat."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "#install.packages('mboost', dep=T)\n",
    "library(mboost)\n",
    "data(\"bodyfat\", package = \"TH.data\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "dim(bodyfat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "attributes(bodyfat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "bodyfat[1:5,]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split the data into training and test datasets. \n",
    "\n",
    "Build a decision tree on the training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "set.seed(1234)\n",
    "ind  <- sample(2,nrow(bodyfat), replace=T, prob=c(0.7, 0.3))\n",
    "bodyfat.train  <- bodyfat[ind == 1,]\n",
    "bodyfat.test  <- bodyfat[ind == 2,]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train a decision tree. \n",
    "\n",
    "`rpart` - Recursive Partitioning and Regression Trees\n",
    "\n",
    "`Recursive partitioning` is a statistical method for multivariable analysis. Recursive partitioning creates a decision tree that strives to correctly classify members of the population by splitting it into sub-populations based on several dichotomous (different) independent variables. The process is termed recursive because each sub-population may in turn be split an indefinite number of times until the splitting process terminates after a particular stopping criterion is reached.\n",
    "\n",
    "Regression Trees - \n",
    "In simple linear regression, a real-valued dependent variable Y is modeled as a linear function of a real-valued independent variable X plus noise:\n",
    "\n",
    "In multiple regression, we let there be multiple independent variables\n",
    "\n",
    "Linear regression is a **global model**, where there is a single predictive formula holding over the entire data-space. When the data has lots of features which interact in complicated, nonlinear ways, assembling a single global model can be very difficult, and hopelessly confusing when you do succeed. An alternative approach to nonlinear regression is to sub-divide, or partition, the space into smaller regions, where the interactions are more manageable. We then partition the sub-divisions again — this is called **recursive partitioning** — until finally we get to chunks of the space which are so tame that we can fit simple models to them. The **global model** thus has two parts: one is just the recursive partition, the other is a simple model for each cell of the partition.\n",
    "\n",
    "Prediction trees use the tree to represent the recursive partition. Each of the\n",
    "**terminal nodes**, or **leaves**, of the tree represents a cell of the partition, and has\n",
    "attached to it a simple model which applies in that cell only.\n",
    "\n",
    "\n",
    "\n",
    "- `control` - a list of options that control details of the rpart algorithm `??rpart.control`\n",
    "- `minsplit`  - the minimum number of observations that must exist in a node in order for a split to be attempted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "library(rpart)\n",
    "myFormula  <- DEXfat ~ age + waistcirc + waistcirc + elbowbreadth + kneebreadth\n",
    "bodyfat_rpart <- rpart(myFormula, data = bodyfat.train,\n",
    "                       control = rpart.control(minsplit = 10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "attributes(bodyfat_rpart)\n",
    "##??rpart.object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`cptable` - a matrix of information on the optimal prunings based on a complexity parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "print(bodyfat_rpart$cptable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "print(bodyfat_rpart)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot the tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "plot(bodyfat_rpart)\n",
    "text(bodyfat_rpart, use.n=T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select the tree with minimum prediction error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "opt  <- which.min(bodyfat_rpart$cptable[,\"xerror\"])\n",
    "cp  <- bodyfat_rpart$cptable[opt, \"CP\"]\n",
    "bodyfat_prune  <- prune(bodyfat_rpart, cp = cp)\n",
    "print(bodyfat_prune)\n",
    "#print(bodyfat_rpart)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "?prune -  Cost-complexity Pruning of an Rpart Object - Determines a nested sequence of subtrees of the supplied `rpart` object by recursively snipping off the least important splits, based on the complexity parameter (cp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "plot(bodyfat_prune)\n",
    "text(bodyfat_prune, use.n = T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the selected tree to make `prediciton` and compared with actual labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "DEXfat_pred  <- predict(bodyfat_prune, newdata= bodyfat.test)\n",
    "xlim  <- range(bodyfat$DEXfat)\n",
    "plot(DEXfat_pred ~ DEXfat, data = bodyfat.test, xlab=\"Observed\",\n",
    "     ylab=\"Predicted\", ylim=xlim, xlim=xlim)\n",
    "abline(a=0, b=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "References - \n",
    "- https://en.wikipedia.org/wiki/Recursive_partitioning\n",
    "- http://www.stat.cmu.edu/~cshalizi/350-2006/lecture-10.pdf"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 2
}