Student_MLE_MiniProject_Trees_and_Forests.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yzr5Eo_qxHQi"
      },
      "source": [
        "# Mini Project: Trees and Forests\n",
        "\n",
        "In this mini-project you'll be introduced to some fundamental concepts in machine learning: **Decision Trees**, **Random Forests**, **Boosting**, and **Bagging**. These techniques play a crucial role in building powerful and versatile machine learning models that can be applied to a wide range of tasks, from classification to regression.\n",
        "\n",
        "## Decision Trees\n",
        "\n",
        "Decision Trees are intuitive models that mimic human decision-making processes. Just like answering a series of questions to reach a conclusion, decision trees segment data based on a sequence of if-else questions. They partition the feature space into regions and assign a label to each region. Decision trees are particularly helpful in understanding the decision-making process behind predictions.\n",
        "\n",
        "## Random Forests\n",
        "\n",
        "Random Forests are an ensemble method that combines multiple decision trees to create a more robust and accurate model. Each tree in a random forest is trained on a random subset of the data with replacement (bootstrapping), and these trees are allowed to make individual predictions. The final prediction is determined through majority voting (classification) or averaging (regression) of the predictions made by the individual trees. Random Forests are known for their ability to handle noisy data, reduce overfitting, and capture complex relationships.\n",
        "\n",
        "## Bagging: Bootstrap Aggregating\n",
        "\n",
        "Bagging, short for Bootstrap Aggregating, is the technique behind Random Forests. It involves training multiple models on bootstrapped subsets of the training data. Bagging reduces the variance of the model by averaging out the noise and errors present in individual models. While Random Forests are a specific application of bagging to decision trees, bagging can also be applied to other base models.\n",
        "\n",
        "## Boosting\n",
        "\n",
        "Boosting is another ensemble technique that focuses on building a strong predictor by sequentially training multiple weak learners (often shallow decision trees). Boosting assigns weights to training instances, focusing more on those that the previous models misclassified. This iterative process helps the model correct its mistakes and improve its performance over time. Popular boosting algorithms include AdaBoost, Gradient Boosting, and XGBoost.\n",
        "\n",
        "In this mini-project, we will:\n",
        "\n",
        "1. Implement and visualize decision trees using the scikit-learn library.\n",
        "2. Explore the power of random forests and understand feature importance.\n",
        "3. Dive into the boosting technique with AdaBoost to enhance model accuracy."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "For this mini-project we'll be using the [Breast Cancer Wisconsin (Diagnostic) dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html). First, let's import all the libraries we'll be using."
      ],
      "metadata": {
        "id": "2JZGx1fBfi08"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.datasets import load_breast_cancer\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.tree import DecisionTreeClassifier, plot_tree\n",
        "from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier\n",
        "from sklearn.metrics import accuracy_score"
      ],
      "metadata": {
        "id": "5ynUME3XEH-Q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here are your tasks:\n",
        "\n",
        "1. Use [load_breast_cancer](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html) to load the Breast Cancer Wisconsin dataset as a Pandas dataframe.\n",
        "2. Split the dataset into training and test sets.\n",
        "3. Display the first five rows of data and make sure everything looks ok. You should have already explored the data a bit in the logistic regression mini-project so there's no need to conduct further EDA.  "
      ],
      "metadata": {
        "id": "81annADtOkJf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Load the Breast Cancer Wisconsin dataset"
      ],
      "metadata": {
        "id": "jN6cgyIREPGN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Split the dataset into training and testing sets"
      ],
      "metadata": {
        "id": "F7-6BwIiWhY1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Display first 5 rows"
      ],
      "metadata": {
        "id": "rOTFtVlxEjvN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's start by diving into decision trees. At a high level, decision trees are a machine learning algorithm used for both classification and regression tasks. They model decisions and decision-making processes by breaking down a complex decision into a sequence of simpler decisions or questions. Each decision leads to one of several possible outcomes, eventually leading to a prediction or classification.\n",
        "\n",
        "**Decision trees are built out of the following components:**\n",
        "- **Nodes:** Decision trees consist of nodes. The initial node is the root node, and the final nodes are called leaf nodes.\n",
        "- **Edges:** Edges connect nodes and represent the decision outcomes.\n",
        "- **Attributes/Features:** Each node involves a decision based on a specific attribute or feature.\n",
        "- **Splitting:** At each node, the dataset is partitioned into subsets based on an attribute's values.\n",
        "- **Leaf Nodes:** These nodes represent the final predicted class or value.\n",
        "\n",
        "**How Decision Trees Work:**\n",
        "1. **Selecting the Best Feature:** Decision trees start with the root node and select the feature that best separates the data.\n",
        "2. **Splitting:** The selected feature is used to split the data into subsets at each internal node.\n",
        "3. **Repeating:** The process continues recursively for each subset, selecting the best feature at each node.\n",
        "4. **Leaf Nodes:** The process stops when a certain stopping criterion is met, or when all instances at a node belong to the same class.\n",
        "\n",
        "**Advantages:**\n",
        "1. **Interpretability:** Decision trees are easy to understand and interpret. The path from the root to a leaf node can be visualized as a sequence of decisions.\n",
        "2. **Handles Numerical and Categorical Data:** Decision trees can handle both numerical and categorical data, making them versatile.\n",
        "3. **Non-Linearity:** They can capture non-linear relationships between features and the target variable.\n",
        "4. **Feature Importance:** Decision trees provide a measure of feature importance, helping in feature selection.\n",
        "\n",
        "**Challenges and Considerations:**\n",
        "1. **Overfitting:** Decision trees can create overly complex models that fit the training data too closely, leading to poor generalization.\n",
        "2. **Instability:** Small changes in the data can lead to different trees, which can make the model unstable.\n",
        "3. **Bias towards Majority Class:** In classification tasks, decision trees tend to favor the majority class if classes are imbalanced.\n",
        "\n",
        "**Improvements:**\n",
        "1. **Pruning:** Pruning is the process of removing branches that do not improve the model's performance on validation data, reducing overfitting.\n",
        "2. **Ensemble Methods:** Random Forests and Gradient Boosting Trees are ensemble methods that combine multiple decision trees to improve performance.\n",
        "\n",
        "There are a number of different metrics used to determine how to split a node in a decision tree. The Gini impurity is a popular criterion used in decision tree algorithms. It measures the degree of impurity in a dataset, where lower values indicate more pure subsets of data. In the context of decision trees, the Gini impurity is used to assess how often a randomly chosen element would be misclassified if it were randomly assigned to a class based on the distribution of class labels in a subset. The Gini impurity ranges between 0 (pure node, all instances belong to a single class) and 0.5 (impure node, instances are evenly distributed across classes)."
      ],
      "metadata": {
        "id": "bT64m51APSGh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here are your tasks:\n",
        "\n",
        "1. Use Scikit-Learn's [DecisionTreeClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html) to fit a model on the training data.\n",
        "2. Visualize the resulting tree using [plot_tree](https://scikit-learn.org/stable/modules/generated/sklearn.tree.plot_tree.html).\n",
        "3. Iterate on the first two steps by trying different inputs to the decision tree classifier. What happens if you change the max depth? How about the maximum number of lead nodes? From the visualization, make sure you're able to understand how to descend the decision tree to arrive at a prediction."
      ],
      "metadata": {
        "id": "PjRU4fdmTSF5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a Decision Tree model"
      ],
      "metadata": {
        "id": "41e16xuuWk_q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Visualize the Decision Tree"
      ],
      "metadata": {
        "id": "93kbxVvJEtFx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Random Forests are an extension of decision trees that leverage the power of ensemble learning to improve predictive accuracy and robustness. Instead of relying on a single decision tree, Random Forests combine multiple decision trees to make more accurate and stable predictions. Let's walk through the steps of going from decision trees to Random Forests:\n",
        "\n",
        "**Step 1: Building Individual Decision Trees**\n",
        "1. **Decision Tree Construction:** Start by building multiple individual decision trees. Each tree is trained on a bootstrapped subset of the training data, meaning that each tree sees a slightly different version of the data due to random sampling with replacement.\n",
        "2. **Feature Selection:** At each node of a decision tree, only a random subset of features is considered for splitting. This introduces randomness and diversity among the trees.\n",
        "\n",
        "**Step 2: Combining Trees in a Random Forest**\n",
        "1. **Predictions:** When you need to make a prediction using the Random Forest, each individual tree in the forest makes a prediction.\n",
        "2. **Majority Voting (Classification) or Averaging (Regression):** In the case of classification, the class that the majority of trees predict becomes the final prediction. In the case of regression, the average of predictions from all trees is taken as the final prediction.\n",
        "\n",
        "**Benefits of Random Forests:**\n",
        "1. **Reduced Overfitting:** The ensemble nature of Random Forests helps reduce overfitting. The diversity among the trees reduces the risk of capturing noise in the data.\n",
        "2. **Feature Importance:** Random Forests can provide an estimate of feature importance by tracking how much each feature contributes to improving the purity of the nodes.\n",
        "3. **Handles Noisy Data:** Random Forests can handle noisy and irrelevant features more effectively than individual decision trees.\n",
        "4. **Parallelization:** Training decision trees can be parallelized since they are independent of each other, which speeds up the training process."
      ],
      "metadata": {
        "id": "JbkDq7ApWF8m"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here are your tasks:\n",
        "\n",
        "1. Use your training data to train a Random Forest using [RandomForestClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html).\n",
        "2. Extract the feature importances from the trained model.\n",
        "3. Print the feature importances from largest to smallest."
      ],
      "metadata": {
        "id": "WJGS-FUzWiTz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Train random forest model"
      ],
      "metadata": {
        "id": "srHwa2q-WpWt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Extract feature importances"
      ],
      "metadata": {
        "id": "rLO_P-ivWc33"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Print feature importance"
      ],
      "metadata": {
        "id": "yc_9uoazFmxR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "AdaBoost, short for Adaptive Boosting, is a powerful ensemble learning technique used primarily for classification tasks. It is designed to enhance the performance of weak learners (typically shallow decision trees) by combining their predictions. AdaBoost focuses on learning from the mistakes of previous models and gives more weight to misclassified instances, allowing subsequent models to correct those mistakes.\n",
        "\n",
        "Here's an overview of how AdaBoost works:\n",
        "\n",
        "**Step 1: Building Weak Learners (Base Models)**\n",
        "1. **Initialization:** Each training instance is initially assigned equal weights.\n",
        "2. **Training Weak Learners:** A series of weak learners (e.g., shallow decision trees) are trained on the data. Each weak learner tries to minimize the weighted classification error, where the weight of an instance depends on its previous misclassifications. The first weak learner is trained on the original data.\n",
        "\n",
        "**Step 2: Weighting Instances**\n",
        "1. **Calculating Error:** The weighted classification error of each weak learner is computed as the sum of weights of misclassified instances.\n",
        "2. **Updating Weights:** Instances that were misclassified by the previous weak learner are given higher weights, making them more important for subsequent models. Instances that were correctly classified are given lower weights.\n",
        "\n",
        "**Step 3: Combining Weak Learners**\n",
        "1. **Predictions and Voting:** The predictions from all weak learners are combined using weighted majority voting. Weights are assigned to each weak learner based on its performance (lower error leads to higher weight).\n",
        "2. **Final Prediction:** The final prediction is made by taking a weighted majority vote of the weak learners' predictions.\n",
        "\n",
        "**Benefits of AdaBoost:**\n",
        "1. **Adaptive Learning:** AdaBoost focuses on instances that were misclassified by previous models, adapting to the complexity of the data.\n",
        "2. **Improves Weak Models:** Even if individual weak learners perform only slightly better than random guessing, AdaBoost can combine them to create a strong model.\n",
        "3. **Reduces Overfitting:** The iterative nature of AdaBoost allows it to focus on hard-to-classify instances and improve overall generalization."
      ],
      "metadata": {
        "id": "wBO7q_tdZ3iT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here are your tasks:\n",
        "\n",
        "1. Build and train an [AdaBoostClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html) on your training data using a decision tree of max depth equal to 1 as your weak learner.\n",
        "2. Evaluate your decision tree, random forest, and AdaBoost models by applying [accuracy_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html) to the test data. Which model performs the best? Experiment by changing the hyperparameters of these models.\n"
      ],
      "metadata": {
        "id": "T8IsKNfIaAC3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Build and train an AdaBoostClassifier on your training data."
      ],
      "metadata": {
        "id": "cEmNpWHLLZIQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Evaluate models"
      ],
      "metadata": {
        "id": "chOiS6grF4At"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}