4_Random_Forest_Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regression using random forest\n",
    "\n",
    "This example is adapted from an article [here](https://towardsdatascience.com/random-forest-in-python-24d0893d51c0). In this problem we want to predict the maximum temperature for any day in Seattle based on historical data from 2017. Temperature and precipitation data for Sydney may be accessed from [here](https://www.ncdc.noaa.gov/cdo-web/). For convenience we have downloaded the data in comma-separated-variable (CSV) format and placed it in ```DATA/temps.csv```.\n",
    "\n",
    "Our workflow for this section is:\n",
    "\n",
    " 1. Load the data into a Pandas dataframe\n",
    " 1. Clean the data as required\n",
    " 1. Prepare the data for ingestion by the machine learning system\n",
    " 1. Train the model and visualise the training and results\n",
    " 1. Make predictions against a test dataset and assess performance\n",
    " 1. Adjust hyperparameters and retrain\n",
    " 1. Visualise results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Load the data into a dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the Pandas module\n",
    "import pandas as pd\n",
    "\n",
    "# Load the data\n",
    "features = pd.read_csv('DATA/temps.csv')\n",
    "\n",
    "# Print the dimensions (shape) of the table\n",
    "print(\"Table dimensions [rows, columns]:\", features.shape)\n",
    "\n",
    "# Display the first 5 rows\n",
    "features.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display the last 5 rows\n",
    "features.tail(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The key columns for our analysis are:\n",
    "\n",
    " * temp_2: max temperature 2 days prior\n",
    " * temp_1: max temperature 1 day prior\n",
    " * average: historical average max temperature\n",
    " * actual: max temperature measurement\n",
    " * friend: your friend’s prediction, a random number between 20 below the average and 20 above the average"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Clean the data\n",
    "\n",
    "It is useful now to examine simple statistics for each column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print descriptive stats for each column\n",
    "features.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The statistics of the data don't show any unusual outliers.\n",
    "\n",
    "Next we want to plot the data for a visual examination. For convenience we will make a function that creates a nicely formatted plot with date on the x-axis and temperature on the y-axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Necessary imports and magic command to display plots inline\n",
    "import datetime\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "\n",
    "def plot_trend_ax(ax, df, columnNames = ['actual'], yLabel='Temperature'):\n",
    "    \"\"\"Plot multiple trend lines over each other\"\"\"\n",
    "    \n",
    "    # Select columns years, months and days\n",
    "    years = df['year'].astype(str)\n",
    "    months = df['month'].astype(str)\n",
    "    days = df['day'].astype(str)\n",
    "    \n",
    "    # Convert dates to a datetime object\n",
    "    dateStrLst = [\"-\".join(x) for x in zip(years, months, days)]\n",
    "    dates = [datetime.datetime.strptime(x, '%Y-%m-%d') for x in dateStrLst]\n",
    "    \n",
    "    # Plot the data\n",
    "    for colName in columnNames:\n",
    "        ax.step(dates, df[colName], where='mid', label=colName, linewidth=1)\n",
    "    ax.set_ylabel('Date')\n",
    "    ax.set_ylabel(yLabel)\n",
    "    ax.tick_params(axis='x', rotation=45)\n",
    "    \n",
    "    # Enable the legend\n",
    "    ax.legend(loc='best')\n",
    "    \n",
    "    return ax    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialise the figure\n",
    "fig = plt.figure(figsize=(12, 8))\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# Plot \n",
    "plot_trend_ax(ax, features, columnNames=['actual', 'temp_1', 'temp_2', 'friend'])"
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQ4AAABcCAIAAAAQ4sy6AAAQeUlEQVR4Ae1dfVBU1fs/C7q8LC8L7IIYNSFtEPZm2gwFyEsikqNpM4Ul5UsvMgSaYBkx08tM1h+VFOUIRDOmJFhNzWSWKMqiRhiLIr6USZKmNZUgA4iIsPc3fp+ZZw73cneBfbu/5dk/mOc85znn+ZzPvZ97zj132asSBIHRhxggBqwx4GEtgOqJAWLgBgMkFToPiIFRMUBSGRVNFEQMTLJAgUqlslBLVcSAezMguo23JBXGmChasdSoVCqlQVUgJLnDp2SorsImnSdoASZ3/pCfGBjGAEllGB1UIAbkGCCpyDFDfmJgGAMklWF0UIEYkGOApCLHDPmJgWEMkFSG0UEFYkCOAZKKHDPkJwaGMUBSGUYHFYgBOQZIKnLMkJ8YGMYASWUYHVQgBuQYIKnIMUN+YmAYAySVYXRQgRiQY2A8UqmtrVVxn82bN/O9l5SUcJUqo9HI1zrT/u6773gkI9pZWVnOhJSfnw8wCgsL+bwLFixAeCdPnsSqCxcugF+v19vyfdCIiAjoB3u2aoyePaPRCJ0vX77carf2CjCZTMjYiMaFCxcs5ELMoz8BxiMVEYK9e/fyntraWr5INs/A7NmzofjTTz+hf3BwsL6+Hot1dXVoY1hiYqL0u64YRoYTGLDyJfzRIKirqzObzR4eN1Q3NDTEH/XRNHdcjMFgKCoqwv5LS0s7OjoYYytXrgwPDwf/vffeiwFOMBISEiCLyWQaGhry9PRkjB0+fLinpwezG43G3NxcKDY2NoKBGsMwRxsKZG/EIev1+ueff15aFRAQIHWiJywsbOnSpYyx+Ph4dFoxBPkP/L+KtB6nEb1eD70fPnwYwhoaGsCDVXV1ddIe7O6Rg8onio6OBmxNTU2830G2HKQ77rgDYBw9ehRSv/766+AB5eh0OrPZDFUPPPAAVDU3N9uC86abboJ+RuxEDiofbIE9nAaXLVvGN7GLLYetqakJRjR9+nS7JBJ1Is1r0wIsKSkJ4OKiCw2sggA8Tp999llKSoper/f29jYYDC+++OJff/3FxyQkJKhUqilTpnR0dKxdu/bWW2/18vKKiop68803r1+/zkfaxe7t7YWV7p133okd9vf3gzMmJgadjLH6+vqMjAytVuvt7R0TE1NYWHj58mU+YDR2YmIihOHiCkjz9vZ+7LHHGGOXLl06ceIEY2xgYODo0aOMsYCAgHvuuQdatbe3r1ixIjw8XK1WR0RELFu27PTp06K8u3fvTkpK0mg0QUFBmZmZ7e3togDHFVtaWubPn+/v7x8QELBw4cLjx49jrnXr1gGrX331FTpfeeUVcFZXV6PTjkZlZSX2v2LFCo1Go9Vq33///XHcq9z4P0e5j1RYEImzypo1a6KiohhjKSkpUAUKMRgMa9asgQHjrNLf3z9v3jwpCyEhIQ0NDYgBJkQ/P79p06Yxxry8vLDJ2rVrMUxkyEHlw0a8LuLKh784Xb16FZJGR0djD+Xl5bDIRDyMsdtuu+3vv//GGN6Qg7Rt2zbo4emnnxYEobu7e9KkG8vgtLS0yspKqPrwww8FQcDVV0ZGBvRsMpm0Wi0PgDGm0WiQZEEQysvLRXc1er3ez88PWvEI0ZaDigGCIIzIHgTgrBIbG+vt7c3D02q1Fy9ehLCCggKo+vLLL7Hn9evXg7OqqgqdvCGHbZSzCrJ9yy23ILD9+/cj5qVLl/Lp0JbmtWlWAZEwxhoaGq5evdrX1wdXypSUFISFRmFh4e7duxljOp1uw4YN5eXlc+bMYYx1dHQ88sgjostzb2/v0NDQkSNH+vv7P//8c+ikrKxsYGAAO3Sm0dbW9sILL5jN5rCwsC1bthw6dOill15ijLW1ta1evXpMSHBWASUYjcbBwUHG2Jz/feAsh21DlArcqAwNDS1ZsqSrq8vT0/Ott946dOjQpk2b/P39r1y5kpWVBcz8+++/q1evhr2ylStX1tXVffrpp2azube3d0wgxxd86tSpxMTEvXv31tTUTJ8+nTHW1dWF+h9fn1ZbnTx5EuYN/q90++v8+fOPP/54ZWVlbm7uiEseq4lsnVWQiJqaGlACY6yqqko0q3R2dsL1Rq1Wnz59GrRrNpszMjIA4oYNG8CJt1nbtm1Did91110Q9scff6CTN6TXAL4W7BGvi6OcVfDiV11djT3Pnz+fMebp6dnZ2YlONCxAuvnmm2E4HR0dqDS4G4GRBgcHm83mzMxMCINZ94cffoBidnY2Znn33XfB+e233wqCUFpaCsW5c+diDDaUgyTnxx5GOavodLq+vj5o9cUXXwCSZ599FjwOmlUgi+jvn3/+CUlxVomIiBgcHMQRjWNWsXUHDCeQ2tpa3PhPTk7GKyKM4ccff+zv72eMzZs37/bbbwenSqXKy8uDA2k0Gl999VV+wHfffTcWQ0NDwYZO0O804+eff4ZcGzdu/OSTT8Bua2uDTb9jx44lJyePHkxiYuL27dsZY42NjXCjEhISMmPGDFiGHT9+vLOzs7W1FTj09fWdNWsWYwwx7Nu3DyZkmJMhb3Nz84IFC/De4NFHH0U8c+fO9fHxwVUl+u1uxMXF+fj4QLe4x9jX12f3RHyHer0+JyeH98DdnciTmJgIuyYi/+iLtkpl6tSpBoPhzJkzKJWYmJgpU6aIEFy6dAk8ERERfBUWYRuXr/L398cirObh4odO+xqoc8YYrIj4/nF9iOcrX9vZ2ckXrdoola+//vrUqVOMsdTUVFh6zZkzZ+PGjTAznzt3jjEWFxc3efJkxhhiOPO/jygLYLhy5Qr4cQfyxs8iengEBwdfvHhR1MTuxaCgIOwTD9nQ0BA6wbBMtSjYajE0NPSNN94YTZjVGMsBtkqFMZacnHzmzJmWlhbIhPMMnzgsLAyKokUkFvlDC5Gie1O+NzvamIW/C+ru7halQN2aTKaZM2eKasdaxNuVrVu3QlucJZKSktRq9cDAAH4HAp+oIIb33nsPFzOi1IGBgeBBXcH1BS9Vonj7FqXbHnz/o6Sab2JH29fX18bebL2tB6ngYheLIlj3338/XBprampg3QIBmzZtAuOhhx4SNXFO0cfHBw7wP//8g9c/6dSB8sD7McZYTk5Ofn5+RUXFWJeFsbGxISEhjDHc/kap+Pr6wrMUlCtKZUQMH3/8cXZ2dklJyfnz5xlj9913H/C2f/9+JPDgwYPXrl3DoqsMjUYDqfnHA7iR5WhUlmU8mux2kIpoGhlx1a7T6Z566inG2LVr1+Lj4995552Kior09PRdu3YxxsLDw1etWjUauHaP8fDwgF3pnp6evLw8k8m0detW6dPfZ555Bla6GzZs+Oijj44dO/baa69t3ry5uLi4oqJCtENqFaRKpcLdC8ZYZGQkYICGaWlp2INarY6Li4NiRkYGrFdra2vz8vKam5urqqpefvnlsrKyoqIiuElYvHgxrIKqq6tLS0v7+vpMJpN0ONi/Mw2DwQDpPvjgg507dzY0NDz55JO4GHEmknHmwj0BqYFzhaiKf64CVXinjk8nRDtggiD09vaOqKLQ0FD+8TmeQ+3t7Zg3PT0dhvfLL7+gkzfkoPIxI+6ACYKAm0jI4MyZMyMjIxlj/HMVaRhjLDQ09MSJE3wWtC1D4nt77rnnsBX/OIUx9uCDD/JV9fX1eN+MaD09PfmHEt988w3eJ0CMj48PSpHvDW3LUCFMjj1BEHA3iX9ajw9YMzMzoYeurq6pU6cibMaYSqXC7Qd+CAgM7mr4Ito4HeEph1W8gTtgRUVFvB8xO++5CowcJ5YRxQAxGo1m3759W7ZsSU1NDQkJUavVBoMhPz+/tbUVdnh4Ep1pFxQUlJSUREdHe3l5RUZGFhUVHThwQLq0Xbdu3Z49ex5++GGdTqdWq6dNm5adnd3U1AQPEMYKGG9X4IkK33zWrFn4nBFXXxAwe/bslpaW5cuXR0REqNXq8PDwhQsXHjx4cMmSJdjDokWLjEZjamqqr6+vVqtdvHhxY2Mj7rZjmPONwMDAAwcOLFq0KDAw0N/fPy0trb6+/oknnnA+kvFltPRTv676udhxjESBUBUISY5YJUN1FTZpXjvcq8gdAPITA+7EAEnFnY4mjcWBDJBUHEgude1ODJBU3Olo0lgcyABJxYHkUtfuxABJxZ2OJo3FgQyQVBxILnXtTgyQVNzpaNJYHMgAScWB5FLX7sQAScWdjiaNxYEMkFQcSC517U4MkFTc6WjSWBzIAEnFgeRS1+7EgJVvFrvTUGksxMCYGOB/A4AxZuV/60XRY8rkzGDpV6admX3EXAqENCJO+BcrxR5oV9GIvwSApNECDKkggxiwxABJxRI7VEcMIAMkFaSCDGLAEgMkFUvsUB0xgAyQVJAKMogBSwyQVCyxQ3XEADJAUkEqyCAGLDFAUrHEDtURA8gASQWpIIMYsMQAScUSO1RHDCADJBWkggxiwBIDJBVL7FAdMYAM2EEqxcXFKpXKOS+7QdxjMhoaGtLT03U6XUBAQEpKCr4xfEyd2DdYgZDkBqhkqE7Fxv+SvsgezbsEdu7cqVarGWP//fefqLkzixagfv/995MmTQoODs7Ozl61apVWq1WpVNu3b3c0PAVCkhuykqG6Cps073jeMIyMFxcXw7u4FCuV/v7+8PBwrVZ79uxZgH327Fm9Xh8UFHT58mUciCMMKdeQxYWQ5IapZKiuwibNO06ptLa2wtukoqOjo6KiFCsVeNni+vXr+bPk7bffZoyVlZXxTrvbUq4hhQshyY1RyVBdhU2ad5z3Krt27Tpy5EhBQUFzc7P0fcJyq17n++GViPwr4+CF14wx/m2JzgSmQEhyw1cyVOdjs/JfkHIkpqenZ2Vl4au05cJc7v/tt98YY/gWQsAD0+Cvv/7qEngKhCTHg5KhOh/bOKUyY8YMOX4V5e/o6GCMBQcH86jg9XFdXV2802m2AiHJjV3JUJ2PbZwLMDlyleaHt9F7eXnxwFQq1eTJk8f6Bm2+B1tsBUKSG46SoTofm5tLBd7HC7TiCSEIwvXr1/38/NDjTEOBkOSGr2Sozsfm5lKBpZdorQXFwMBAuVPEoX4FQpIbr5KhOh+bm0slJiaGMfb777/zZwMUY2NjeafTbAVCkhu7kqE6H5ubSyU5OVm6LwxfbElISJA7RRzqVyAkufEqGaoLsMk9loLfULNQi1Xx8fGKfQTZ09Oj0+mCg4Pb2toA8Llz50JDQ4OCgrq7u3EIjjCkz7AgiwshyQ1TyVBdhU2ad5xP63nSlSwVQRB27Njh4eGh1WpzcnJyc3NDQkI8PDx27NjBD8ERtpRrzOIqSAhAZCgZqquwSfO6v1QEQdizZ09SUpJGowkKCkpOTq6trRWdK44oSrnms7gEEg+At5UM1VXYpHmt/Ly3Yn/KVrS8dtUv24pg8EUFQuLh8baSoboKmzSvm9/W8ycE2cSALQyQVGxhj9pOIAZIKhPoYNNQbWGApGILe9R2AjFAUplAB5uGagsDJBVb2KO2E4gBksoEOtg0VFsYIKnYwh61nUAMkFQm0MGmodrCAEnFFvao7QRigKQygQ42DdUWBkgqtrBHbScQAySVCXSwaai2MGDlm8W2dE1tiYH/1wyIvlZv5XfARNGKHbn0K9Muh6pASHKcKBmqq7CpVCoRXf8HI4njsHsurBQAAAAASUVORK5CYII="
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) Preparing the data for the ML algorithm\n",
    "\n",
    "We will be using an implementation of the Random Forest algorithm within *scikit-learn*. Most machine learning algorithms require *catagorical data* to be encoded as a binary vector. For example, a column containing days of the week ```['Mon', 'Tue', 'Wed' ...]``` would be converted to a set of new columns that looks like this:\n",
    "\n",
    "![image.png](attachment:image.png)\n",
    "\n",
    "This is known as *one-hot encoding* and is simply a vector where the current catagory is indicated by a '1'. Pandas has a built-in function to convert all catagorical variables to new 'one-hot' columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# One-hot encode categorical variables\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the other columns (e.g., month, day) are already in numerical format and so are not converted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of this excersise is to predict the actual temperature using the other columns in the dataset. We want to split out the 'actual' column and put it in a separate array. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use numpy to create a new array from the 'actual' column\n",
    "\n",
    "# Then delete the column from the table\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scikit-learn does not accept pandas dataframes, so we want to convert the table into a standard Numpy array that contains only numerical values. During this process the column names will be lost, so we save these as a separate list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save a list of column headers for latter use\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the table to a numpy array\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Training and Test data**\n",
    "\n",
    "In supervised learning we want to set aside some data as an 'independent' dataset to test our predictions. This is usually some randomly-chosen fraction of the input data. Scikit-learn has a convenience function to make this split easy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using Skicit-learn to split data into training and testing sets\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split 25% of the data into a test set\n",
    "train_features, test_features, train_labels, test_labels = \\\n",
    "train_test_split(features, labels, test_size=0.25, random_state=42)\n",
    "\n",
    "print('Training Features Shape:', train_features.shape)\n",
    "print('Training Labels Shape:', train_labels.shape)\n",
    "print('Testing Features Shape:', test_features.shape)\n",
    "print('Testing Labels Shape:', test_labels.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4) Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the specific algorithm \n",
    "\n",
    "# Create an instance of the model with 1000 estimators\n",
    "\n",
    "# Train the model on training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the forest's predict method on the test data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The predictions are a temperature output for each date in the test dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5) Assess the performance of the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the absolute errors\n",
    "errors = abs(predictions - test_labels)\n",
    "\n",
    "# Print out the mean absolute error (mae)\n",
    "print('Mean Absolute Error:', round(np.mean(errors), 2), 'degrees.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate mean absolute percentage error (MAPE)\n",
    "mape = 100 * (errors / test_labels)\n",
    "\n",
    "# Calculate and display accuracy\n",
    "accuracy = 100 - np.mean(mape)\n",
    "print('Accuracy:', round(accuracy, 2), '%.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6) Adjusting the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_new = RandomForestRegressor(n_estimators = 100, \n",
    "                               criterion = 'mse', \n",
    "                               max_depth = None, \n",
    "                               min_samples_split = 2, \n",
    "                               min_samples_leaf = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7) Visualising the tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import tools needed for visualization\n",
    "from sklearn.tree import export_graphviz\n",
    "from IPython.display import Image\n",
    "import pydot\n",
    "\n",
    "# Pull out one tree from the forest\n",
    "tree = rf.estimators_[5]\n",
    "\n",
    "# Export the image to a dot file\n",
    "export_graphviz(tree, out_file = 'TMP/tree.dot', feature_names = feature_list, rounded = True, precision = 1)\n",
    "\n",
    "# Use dot file to create a graph\n",
    "(graph, ) = pydot.graph_from_dot_file('TMP/tree.dot')\n",
    "\n",
    "# Write graph to a png file and display\n",
    "graph.write_png('TMP/tree.png');\n",
    "Image('TMP/tree.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print the depth of the tree\n",
    "print('The depth of this tree is:', tree.tree_.max_depth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assess variable importance\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get numerical feature importances\n",
    "importances = list(rf.feature_importances_)\n",
    "\n",
    "# List of tuples with variable and importance\n",
    "feature_importances = [(feature, round(importance, 2)) for feature, importance in zip(feature_list, importances)]\n",
    "\n",
    "# Sort the feature importances by most important first\n",
    "feature_importances = sorted(feature_importances, key = lambda x: x[1], reverse = True)\n",
    "\n",
    "# Print out the feature and importances \n",
    "[print('Variable: {:20} Importance: {}'.format(*pair)) for pair in feature_importances];\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Model with 2 most important features only**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# New random forest with only the two most important variables\n",
    "rf_most_important = RandomForestRegressor(n_estimators= 1000, random_state=42)\n",
    "\n",
    "# Extract the two most important features\n",
    "important_indices = [feature_list.index('temp_1'), feature_list.index('average')]\n",
    "train_important = train_features[:, important_indices]\n",
    "test_important = test_features[:, important_indices]\n",
    "\n",
    "# Train the random forest\n",
    "rf_most_important.fit(train_important, train_labels)\n",
    "\n",
    "# Make predictions and determine the error\n",
    "predictions = rf_most_important.predict(test_important)\n",
    "\n",
    "errors = abs(predictions - test_labels)\n",
    "\n",
    "# Display the performance metrics\n",
    "print('Mean Absolute Error:', round(np.mean(errors), 2), 'degrees.')\n",
    "\n",
    "mape = np.mean(100 * (errors / test_labels))\n",
    "accuracy = 100 - mape\n",
    "\n",
    "print('Accuracy:', round(accuracy, 2), '%.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8) Visualisations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualise the importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dates of training values\n",
    "months = features[:, feature_list.index('month')]\n",
    "days = features[:, feature_list.index('day')]\n",
    "years = features[:, feature_list.index('year')]\n",
    "\n",
    "# List and then convert to datetime object\n",
    "dates = [str(int(year)) + '-' + str(int(month)) + '-' + str(int(day)) for year, month, day in zip(years, months, days)]\n",
    "dates = [datetime.datetime.strptime(date, '%Y-%m-%d') for date in dates]\n",
    "\n",
    "# Dataframe with true values and dates\n",
    "true_data = pd.DataFrame(data = {'date': dates, 'actual': labels})\n",
    "\n",
    "# Dates of predictions\n",
    "months = test_features[:, feature_list.index('month')]\n",
    "days = test_features[:, feature_list.index('day')]\n",
    "years = test_features[:, feature_list.index('year')]\n",
    "\n",
    "# Column of dates\n",
    "test_dates = [str(int(year)) + '-' + str(int(month)) + '-' + str(int(day)) for year, month, day in zip(years, months, days)]\n",
    "\n",
    "# Convert to datetime objects\n",
    "test_dates = [datetime.datetime.strptime(date, '%Y-%m-%d') for date in test_dates]\n",
    "\n",
    "# Dataframe with predictions and dates\n",
    "predictions_data = pd.DataFrame(data = {'date': test_dates, 'prediction': predictions})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the actual values\n",
    "plt.plot(true_data['date'], true_data['actual'], 'b-', label = 'actual')\n",
    "\n",
    "# Plot the predicted values\n",
    "plt.plot(predictions_data['date'], predictions_data['prediction'], 'ro', label = 'prediction')\n",
    "plt.xticks(rotation = '60'); \n",
    "plt.legend()\n",
    "\n",
    "# Graph labels\n",
    "plt.xlabel('Date'); plt.ylabel('Maximum Temperature (F)'); plt.title('Actual and Predicted Values');\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}