2_handling_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Handling data with python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Author:** Ties de Kok ([Personal Website](http://www.tiesdekok.com))  \n",
    "**Last updated:** 18 May 2018  \n",
    "**Python version:** Python 3.6  \n",
    "**License:** MIT License  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** Some features (like the ToC) will only work if you run it locally, use Binder, or use nbviewer by clicking this link: \n",
    "https://nbviewer.jupyter.org/github/TiesdeKok/LearnPythonforResearch/blob/master/2_handling_data.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *Introduction*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting your data ready for analysis (i.e. \"data wrangling\") is usually the most time-consuming part of a project. For data wrangling tasks I recommend `Pandas` and `Numpy`.\n",
    "\n",
    "What is `Pandas`?\n",
    "\n",
    "> Pandas is an open source, BSD-licensed library providing high-performance, easy-to-use  data structures and data analysis tools for the Python programming language.\n",
    "\n",
    "In other words, Pandas introduces a data structure (i.e. `dataframe`) that represents data as a table with columns and rows. Combining Python with `Pandas` yields a very powerful toolkit with which you can process any type of data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Format of this notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Pandas` library is massive and it is continuously expanding in functionality.  \n",
    "It is, therefore, impossible to keep it both comprehensive and cover everything in just one notebook.\n",
    "\n",
    "The goal of this notebook is to cover the basic functionality that I expect you to encounter for an average project. \n",
    "\n",
    "I have based parts on this notebook on a PyCon 2015 tutorial/talk by Brandon Rhodes. If you want to know more I highly recommend watching his talk and checking the accompanying GitHub page:\n",
    "  \n",
    "https://www.youtube.com/watch?v=5JnMutdy6Fw  \n",
    "https://github.com/brandon-rhodes/pycon-pandas-tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *Table of Contents* <a id='toc'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* [Import pandas](#import-pandas)   \n",
    "* [Create a dataframe](#create-dataframe)   \n",
    "* [Manipulate dataframe](#manipulate-dataframe)   \n",
    "* [Rename columns](#rename-columns)   \n",
    "* [View a dataframe using qgrid](#qgrid)   \n",
    "* [View (parts) of a dataframe using Pandas](#view-dataframe)   \n",
    "* [Dealing with datatypes](#datatypes)   \n",
    "* [Handling missing values](#missing-values)   \n",
    "* [Work with data in the dataframe](#work-with-data)   \n",
    "* [Combining dataframes](#combining-dataframes)   \n",
    "* [Group-by operations](#groupby)   \n",
    "* [Reshaping and Pivot Tables](#reshaping-pivot)   \n",
    "* [Dealing with dates](#dates)   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Import Pandas</span><a id='import-pandas'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* it is usually a good idea to also import `numpy` when you use `pandas`, their functionality is quite intertwined.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For convenience we also import `join` to easily create paths:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from os.path import join"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Path to our data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path = join(os.getcwd(), 'example_data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  <span style=\"text-decoration: underline;\">Create a dataframe</span><a id='create-dataframe'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create a dataframe in many ways. Below are a couple of situations:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Load file from drive into Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For details on opening files such as Excel, CSV, Stata, SAS, HDF see the `1_opening_files` notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_auto = pd.read_csv(join(data_path, 'auto_df.csv'), sep=';', index_col='Unnamed: 0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Create new dataframe and pass data to it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can pass many different types of data to the `pd.DataFrame()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col1</th>\n",
       "      <th>col2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   col1  col2\n",
       "0     1     5\n",
       "1     2     6\n",
       "2     3     7\n",
       "3     4     8"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = {'col1': [1,2,3,4], 'col2': [5,6,7,8]}\n",
    "df = pd.DataFrame(data=d)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0  1  2  3\n",
       "0  1  2  3  4\n",
       "1  5  6  7  8"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = [(1, 2 ,3 ,4), (5, 6, 7, 8)]\n",
    "df = pd.DataFrame(data=d)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) Create dataframe from a dictionary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also directly convert a dictionary to a dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      0  1  2  3\n",
       "row1  1  2  3  4\n",
       "row2  5  6  7  8"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = {'row1': [1,2,3,4], 'row2': [5,6,7,8]}\n",
    "df = pd.DataFrame.from_dict(d, orient='index')\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Manipulate dataframe</span><a id='manipulate-dataframe'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>col5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      0  1  2  3  col5\n",
       "row1  1  2  3  4    10\n",
       "row2  5  6  7  8    10"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['col5'] = [10, 10]\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>col5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row3</th>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       0   1   2   3  col5\n",
       "row1   1   2   3   4    10\n",
       "row2   5   6   7   8    10\n",
       "row3  11  12  13  14    15"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc['row3'] = [11, 12, 13, 14, 15]\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inverse the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>row1</th>\n",
       "      <th>row2</th>\n",
       "      <th>row3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>col5</th>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      row1  row2  row3\n",
       "0        1     5    11\n",
       "1        2     6    12\n",
       "2        3     7    13\n",
       "3        4     8    14\n",
       "col5    10    10    15"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row3</th>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       0   1   2   3\n",
       "row1   1   2   3   4\n",
       "row2   5   6   7   8\n",
       "row3  11  12  13  14"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.drop('col5', axis=1)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row3</th>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       0   1   2   3\n",
       "row2   5   6   7   8\n",
       "row3  11  12  13  14"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.drop('row1', axis=0)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row3</th>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       0   1   2   3\n",
       "row2   5   6   7   8\n",
       "row3  11  12  13  14"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     1   2   3\n",
       "0             \n",
       "5    6   7   8\n",
       "11  12  13  14"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.set_index(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* `Pandas` also allows a multi-index. These can be very powerful. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row3</th>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          1   2   3\n",
       "     0             \n",
       "row2 5    6   7   8\n",
       "row3 11  12  13  14"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.set_index(0, append=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reset index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can convert the index to a regular column using `reset_index()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>row2</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>row3</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  index   0   1   2   3\n",
       "0  row2   5   6   7   8\n",
       "1  row3  11  12  13  14"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Rename columns</span><a id='rename-columns'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can either manipulate `df.columns` directly or use `df.rename()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col1</th>\n",
       "      <th>col2</th>\n",
       "      <th>col3</th>\n",
       "      <th>col4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row3</th>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      col1  col2  col3  col4\n",
       "row2     5     6     7     8\n",
       "row3    11    12    13    14"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns = ['col1', 'col2', 'col3', 'col4']\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>column1</th>\n",
       "      <th>column2</th>\n",
       "      <th>col3</th>\n",
       "      <th>col4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row3</th>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      column1  column2  col3  col4\n",
       "row2        5        6     7     8\n",
       "row3       11       12    13    14"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.rename(columns={'col1' : 'column1', 'col2' : 'column2'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** The above creates a copy, it does not modify it in place!  \n",
    "We need to use either the `inplace=True` argument or assign it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.rename(columns={'col1' : 'column1', 'col2' : 'column2'})\n",
    "#or\n",
    "df.rename(columns={'col1' : 'column1', 'col2' : 'column2'}, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">View (parts) of a dataframe using `Pandas`</span><a id='view-dataframe'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can take some getting used to, but navigating your way around a dataframe is a very helpful skill. The ability sub-select parts of a dataframe is important for inspection purposes, analysis, exporting, and much more."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View entire dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* Pandas will only show the top and bottom parts if the dataframe is large."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AMC Concord</td>\n",
       "      <td>4099</td>\n",
       "      <td>22</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>11</td>\n",
       "      <td>2930</td>\n",
       "      <td>186</td>\n",
       "      <td>40</td>\n",
       "      <td>121</td>\n",
       "      <td>3.58</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AMC Pacer</td>\n",
       "      <td>4749</td>\n",
       "      <td>17</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>11</td>\n",
       "      <td>3350</td>\n",
       "      <td>173</td>\n",
       "      <td>40</td>\n",
       "      <td>258</td>\n",
       "      <td>2.53</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AMC Spirit</td>\n",
       "      <td>3799</td>\n",
       "      <td>22</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>12</td>\n",
       "      <td>2640</td>\n",
       "      <td>168</td>\n",
       "      <td>35</td>\n",
       "      <td>121</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Buick Century</td>\n",
       "      <td>4816</td>\n",
       "      <td>20</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.5</td>\n",
       "      <td>16</td>\n",
       "      <td>3250</td>\n",
       "      <td>196</td>\n",
       "      <td>40</td>\n",
       "      <td>196</td>\n",
       "      <td>2.93</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Buick Electra</td>\n",
       "      <td>7827</td>\n",
       "      <td>15</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>20</td>\n",
       "      <td>4080</td>\n",
       "      <td>222</td>\n",
       "      <td>43</td>\n",
       "      <td>350</td>\n",
       "      <td>2.41</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Buick LeSabre</td>\n",
       "      <td>5788</td>\n",
       "      <td>18</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>21</td>\n",
       "      <td>3670</td>\n",
       "      <td>218</td>\n",
       "      <td>43</td>\n",
       "      <td>231</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Buick Opel</td>\n",
       "      <td>4453</td>\n",
       "      <td>26</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>10</td>\n",
       "      <td>2230</td>\n",
       "      <td>170</td>\n",
       "      <td>34</td>\n",
       "      <td>304</td>\n",
       "      <td>2.87</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Buick Regal</td>\n",
       "      <td>5189</td>\n",
       "      <td>20</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>16</td>\n",
       "      <td>3280</td>\n",
       "      <td>200</td>\n",
       "      <td>42</td>\n",
       "      <td>196</td>\n",
       "      <td>2.93</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Buick Riviera</td>\n",
       "      <td>10372</td>\n",
       "      <td>16</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>17</td>\n",
       "      <td>3880</td>\n",
       "      <td>207</td>\n",
       "      <td>43</td>\n",
       "      <td>231</td>\n",
       "      <td>2.93</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Buick Skylark</td>\n",
       "      <td>4082</td>\n",
       "      <td>19</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>13</td>\n",
       "      <td>3400</td>\n",
       "      <td>200</td>\n",
       "      <td>42</td>\n",
       "      <td>231</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Cad. Deville</td>\n",
       "      <td>11385</td>\n",
       "      <td>14</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>20</td>\n",
       "      <td>4330</td>\n",
       "      <td>221</td>\n",
       "      <td>44</td>\n",
       "      <td>425</td>\n",
       "      <td>2.28</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Cad. Eldorado</td>\n",
       "      <td>14500</td>\n",
       "      <td>14</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>16</td>\n",
       "      <td>3900</td>\n",
       "      <td>204</td>\n",
       "      <td>43</td>\n",
       "      <td>350</td>\n",
       "      <td>2.19</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Cad. Seville</td>\n",
       "      <td>15906</td>\n",
       "      <td>21</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>13</td>\n",
       "      <td>4290</td>\n",
       "      <td>204</td>\n",
       "      <td>45</td>\n",
       "      <td>350</td>\n",
       "      <td>2.24</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Chev. Chevette</td>\n",
       "      <td>3299</td>\n",
       "      <td>29</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>9</td>\n",
       "      <td>2110</td>\n",
       "      <td>163</td>\n",
       "      <td>34</td>\n",
       "      <td>231</td>\n",
       "      <td>2.93</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Chev. Impala</td>\n",
       "      <td>5705</td>\n",
       "      <td>16</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>20</td>\n",
       "      <td>3690</td>\n",
       "      <td>212</td>\n",
       "      <td>43</td>\n",
       "      <td>250</td>\n",
       "      <td>2.56</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Chev. Malibu</td>\n",
       "      <td>4504</td>\n",
       "      <td>22</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>17</td>\n",
       "      <td>3180</td>\n",
       "      <td>193</td>\n",
       "      <td>31</td>\n",
       "      <td>200</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Chev. Monte Carlo</td>\n",
       "      <td>5104</td>\n",
       "      <td>22</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>16</td>\n",
       "      <td>3220</td>\n",
       "      <td>200</td>\n",
       "      <td>41</td>\n",
       "      <td>200</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667</td>\n",
       "      <td>24</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7</td>\n",
       "      <td>2750</td>\n",
       "      <td>179</td>\n",
       "      <td>40</td>\n",
       "      <td>151</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Chev. Nova</td>\n",
       "      <td>3955</td>\n",
       "      <td>19</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>13</td>\n",
       "      <td>3430</td>\n",
       "      <td>197</td>\n",
       "      <td>43</td>\n",
       "      <td>250</td>\n",
       "      <td>2.56</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Dodge Colt</td>\n",
       "      <td>3984</td>\n",
       "      <td>30</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8</td>\n",
       "      <td>2120</td>\n",
       "      <td>163</td>\n",
       "      <td>35</td>\n",
       "      <td>98</td>\n",
       "      <td>3.54</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Dodge Diplomat</td>\n",
       "      <td>4010</td>\n",
       "      <td>18</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>17</td>\n",
       "      <td>3600</td>\n",
       "      <td>206</td>\n",
       "      <td>46</td>\n",
       "      <td>318</td>\n",
       "      <td>2.47</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Dodge Magnum</td>\n",
       "      <td>5886</td>\n",
       "      <td>16</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>17</td>\n",
       "      <td>3600</td>\n",
       "      <td>206</td>\n",
       "      <td>46</td>\n",
       "      <td>318</td>\n",
       "      <td>2.47</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Dodge St. Regis</td>\n",
       "      <td>6342</td>\n",
       "      <td>17</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.5</td>\n",
       "      <td>21</td>\n",
       "      <td>3740</td>\n",
       "      <td>220</td>\n",
       "      <td>46</td>\n",
       "      <td>225</td>\n",
       "      <td>2.94</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389</td>\n",
       "      <td>28</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>9</td>\n",
       "      <td>1800</td>\n",
       "      <td>147</td>\n",
       "      <td>33</td>\n",
       "      <td>98</td>\n",
       "      <td>3.15</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Ford Mustang</td>\n",
       "      <td>4187</td>\n",
       "      <td>21</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>10</td>\n",
       "      <td>2650</td>\n",
       "      <td>179</td>\n",
       "      <td>43</td>\n",
       "      <td>140</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Linc. Continental</td>\n",
       "      <td>11497</td>\n",
       "      <td>12</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>22</td>\n",
       "      <td>4840</td>\n",
       "      <td>233</td>\n",
       "      <td>51</td>\n",
       "      <td>400</td>\n",
       "      <td>2.47</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Linc. Mark V</td>\n",
       "      <td>13594</td>\n",
       "      <td>12</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>18</td>\n",
       "      <td>4720</td>\n",
       "      <td>230</td>\n",
       "      <td>48</td>\n",
       "      <td>400</td>\n",
       "      <td>2.47</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Linc. Versailles</td>\n",
       "      <td>13466</td>\n",
       "      <td>14</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>15</td>\n",
       "      <td>3830</td>\n",
       "      <td>201</td>\n",
       "      <td>41</td>\n",
       "      <td>302</td>\n",
       "      <td>2.47</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Merc. Bobcat</td>\n",
       "      <td>3829</td>\n",
       "      <td>22</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>9</td>\n",
       "      <td>2580</td>\n",
       "      <td>169</td>\n",
       "      <td>39</td>\n",
       "      <td>140</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Merc. Cougar</td>\n",
       "      <td>5379</td>\n",
       "      <td>14</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>16</td>\n",
       "      <td>4060</td>\n",
       "      <td>221</td>\n",
       "      <td>48</td>\n",
       "      <td>302</td>\n",
       "      <td>2.75</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Plym. Sapporo</td>\n",
       "      <td>6486</td>\n",
       "      <td>26</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.5</td>\n",
       "      <td>8</td>\n",
       "      <td>2520</td>\n",
       "      <td>182</td>\n",
       "      <td>38</td>\n",
       "      <td>119</td>\n",
       "      <td>3.54</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>Plym. Volare</td>\n",
       "      <td>4060</td>\n",
       "      <td>18</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>16</td>\n",
       "      <td>3330</td>\n",
       "      <td>201</td>\n",
       "      <td>44</td>\n",
       "      <td>225</td>\n",
       "      <td>3.23</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Pont. Catalina</td>\n",
       "      <td>5798</td>\n",
       "      <td>18</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>20</td>\n",
       "      <td>3700</td>\n",
       "      <td>214</td>\n",
       "      <td>42</td>\n",
       "      <td>231</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934</td>\n",
       "      <td>18</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7</td>\n",
       "      <td>3470</td>\n",
       "      <td>198</td>\n",
       "      <td>42</td>\n",
       "      <td>231</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>Pont. Grand Prix</td>\n",
       "      <td>5222</td>\n",
       "      <td>19</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>16</td>\n",
       "      <td>3210</td>\n",
       "      <td>201</td>\n",
       "      <td>45</td>\n",
       "      <td>231</td>\n",
       "      <td>2.93</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>Pont. Le Mans</td>\n",
       "      <td>4723</td>\n",
       "      <td>19</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>17</td>\n",
       "      <td>3200</td>\n",
       "      <td>199</td>\n",
       "      <td>40</td>\n",
       "      <td>231</td>\n",
       "      <td>2.93</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>Pont. Phoenix</td>\n",
       "      <td>4424</td>\n",
       "      <td>19</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.5</td>\n",
       "      <td>13</td>\n",
       "      <td>3420</td>\n",
       "      <td>203</td>\n",
       "      <td>43</td>\n",
       "      <td>231</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Pont. Sunbird</td>\n",
       "      <td>4172</td>\n",
       "      <td>24</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7</td>\n",
       "      <td>2690</td>\n",
       "      <td>179</td>\n",
       "      <td>41</td>\n",
       "      <td>151</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>Audi 5000</td>\n",
       "      <td>9690</td>\n",
       "      <td>17</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>15</td>\n",
       "      <td>2830</td>\n",
       "      <td>189</td>\n",
       "      <td>37</td>\n",
       "      <td>131</td>\n",
       "      <td>3.20</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>Audi Fox</td>\n",
       "      <td>6295</td>\n",
       "      <td>23</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>11</td>\n",
       "      <td>2070</td>\n",
       "      <td>174</td>\n",
       "      <td>36</td>\n",
       "      <td>97</td>\n",
       "      <td>3.70</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>BMW 320i</td>\n",
       "      <td>9735</td>\n",
       "      <td>25</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>12</td>\n",
       "      <td>2650</td>\n",
       "      <td>177</td>\n",
       "      <td>34</td>\n",
       "      <td>121</td>\n",
       "      <td>3.64</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229</td>\n",
       "      <td>23</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6</td>\n",
       "      <td>2370</td>\n",
       "      <td>170</td>\n",
       "      <td>35</td>\n",
       "      <td>119</td>\n",
       "      <td>3.89</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>Datsun 210</td>\n",
       "      <td>4589</td>\n",
       "      <td>35</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8</td>\n",
       "      <td>2020</td>\n",
       "      <td>165</td>\n",
       "      <td>32</td>\n",
       "      <td>85</td>\n",
       "      <td>3.70</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>Datsun 510</td>\n",
       "      <td>5079</td>\n",
       "      <td>24</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>8</td>\n",
       "      <td>2280</td>\n",
       "      <td>170</td>\n",
       "      <td>34</td>\n",
       "      <td>119</td>\n",
       "      <td>3.54</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>Datsun 810</td>\n",
       "      <td>8129</td>\n",
       "      <td>21</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>8</td>\n",
       "      <td>2750</td>\n",
       "      <td>184</td>\n",
       "      <td>38</td>\n",
       "      <td>146</td>\n",
       "      <td>3.55</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>Fiat Strada</td>\n",
       "      <td>4296</td>\n",
       "      <td>21</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>16</td>\n",
       "      <td>2130</td>\n",
       "      <td>161</td>\n",
       "      <td>36</td>\n",
       "      <td>105</td>\n",
       "      <td>3.37</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>Honda Accord</td>\n",
       "      <td>5799</td>\n",
       "      <td>25</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>10</td>\n",
       "      <td>2240</td>\n",
       "      <td>172</td>\n",
       "      <td>36</td>\n",
       "      <td>107</td>\n",
       "      <td>3.05</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>Honda Civic</td>\n",
       "      <td>4499</td>\n",
       "      <td>28</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>5</td>\n",
       "      <td>1760</td>\n",
       "      <td>149</td>\n",
       "      <td>34</td>\n",
       "      <td>91</td>\n",
       "      <td>3.30</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>Mazda GLC</td>\n",
       "      <td>3995</td>\n",
       "      <td>30</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>11</td>\n",
       "      <td>1980</td>\n",
       "      <td>154</td>\n",
       "      <td>33</td>\n",
       "      <td>86</td>\n",
       "      <td>3.73</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>Peugeot 604</td>\n",
       "      <td>12990</td>\n",
       "      <td>14</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.5</td>\n",
       "      <td>14</td>\n",
       "      <td>3420</td>\n",
       "      <td>192</td>\n",
       "      <td>38</td>\n",
       "      <td>163</td>\n",
       "      <td>3.58</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>Renault Le Car</td>\n",
       "      <td>3895</td>\n",
       "      <td>26</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>10</td>\n",
       "      <td>1830</td>\n",
       "      <td>142</td>\n",
       "      <td>34</td>\n",
       "      <td>79</td>\n",
       "      <td>3.72</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>Subaru</td>\n",
       "      <td>3798</td>\n",
       "      <td>35</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>11</td>\n",
       "      <td>2050</td>\n",
       "      <td>164</td>\n",
       "      <td>36</td>\n",
       "      <td>97</td>\n",
       "      <td>3.81</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>Toyota Celica</td>\n",
       "      <td>5899</td>\n",
       "      <td>18</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>14</td>\n",
       "      <td>2410</td>\n",
       "      <td>174</td>\n",
       "      <td>36</td>\n",
       "      <td>134</td>\n",
       "      <td>3.06</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>Toyota Corolla</td>\n",
       "      <td>3748</td>\n",
       "      <td>31</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>9</td>\n",
       "      <td>2200</td>\n",
       "      <td>165</td>\n",
       "      <td>35</td>\n",
       "      <td>97</td>\n",
       "      <td>3.21</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>Toyota Corona</td>\n",
       "      <td>5719</td>\n",
       "      <td>18</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>11</td>\n",
       "      <td>2670</td>\n",
       "      <td>175</td>\n",
       "      <td>36</td>\n",
       "      <td>134</td>\n",
       "      <td>3.05</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>VW Dasher</td>\n",
       "      <td>7140</td>\n",
       "      <td>23</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>12</td>\n",
       "      <td>2160</td>\n",
       "      <td>172</td>\n",
       "      <td>36</td>\n",
       "      <td>97</td>\n",
       "      <td>3.74</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>VW Diesel</td>\n",
       "      <td>5397</td>\n",
       "      <td>41</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>15</td>\n",
       "      <td>2040</td>\n",
       "      <td>155</td>\n",
       "      <td>35</td>\n",
       "      <td>90</td>\n",
       "      <td>3.78</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>VW Rabbit</td>\n",
       "      <td>4697</td>\n",
       "      <td>25</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>15</td>\n",
       "      <td>1930</td>\n",
       "      <td>155</td>\n",
       "      <td>35</td>\n",
       "      <td>89</td>\n",
       "      <td>3.78</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>VW Scirocco</td>\n",
       "      <td>6850</td>\n",
       "      <td>25</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>16</td>\n",
       "      <td>1990</td>\n",
       "      <td>156</td>\n",
       "      <td>36</td>\n",
       "      <td>97</td>\n",
       "      <td>3.78</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>Volvo 260</td>\n",
       "      <td>11995</td>\n",
       "      <td>17</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>14</td>\n",
       "      <td>3170</td>\n",
       "      <td>193</td>\n",
       "      <td>37</td>\n",
       "      <td>163</td>\n",
       "      <td>2.98</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>74 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 make  price  mpg  rep78  headroom  trunk  weight  length  \\\n",
       "0         AMC Concord   4099   22    3.0       2.5     11    2930     186   \n",
       "1           AMC Pacer   4749   17    3.0       3.0     11    3350     173   \n",
       "2          AMC Spirit   3799   22    NaN       3.0     12    2640     168   \n",
       "3       Buick Century   4816   20    3.0       4.5     16    3250     196   \n",
       "4       Buick Electra   7827   15    4.0       4.0     20    4080     222   \n",
       "5       Buick LeSabre   5788   18    3.0       4.0     21    3670     218   \n",
       "6          Buick Opel   4453   26    NaN       3.0     10    2230     170   \n",
       "7         Buick Regal   5189   20    3.0       2.0     16    3280     200   \n",
       "8       Buick Riviera  10372   16    3.0       3.5     17    3880     207   \n",
       "9       Buick Skylark   4082   19    3.0       3.5     13    3400     200   \n",
       "10       Cad. Deville  11385   14    3.0       4.0     20    4330     221   \n",
       "11      Cad. Eldorado  14500   14    2.0       3.5     16    3900     204   \n",
       "12       Cad. Seville  15906   21    3.0       3.0     13    4290     204   \n",
       "13     Chev. Chevette   3299   29    3.0       2.5      9    2110     163   \n",
       "14       Chev. Impala   5705   16    4.0       4.0     20    3690     212   \n",
       "15       Chev. Malibu   4504   22    3.0       3.5     17    3180     193   \n",
       "16  Chev. Monte Carlo   5104   22    2.0       2.0     16    3220     200   \n",
       "17        Chev. Monza   3667   24    2.0       2.0      7    2750     179   \n",
       "18         Chev. Nova   3955   19    3.0       3.5     13    3430     197   \n",
       "19         Dodge Colt   3984   30    5.0       2.0      8    2120     163   \n",
       "20     Dodge Diplomat   4010   18    2.0       4.0     17    3600     206   \n",
       "21       Dodge Magnum   5886   16    2.0       4.0     17    3600     206   \n",
       "22    Dodge St. Regis   6342   17    2.0       4.5     21    3740     220   \n",
       "23        Ford Fiesta   4389   28    4.0       1.5      9    1800     147   \n",
       "24       Ford Mustang   4187   21    3.0       2.0     10    2650     179   \n",
       "25  Linc. Continental  11497   12    3.0       3.5     22    4840     233   \n",
       "26       Linc. Mark V  13594   12    3.0       2.5     18    4720     230   \n",
       "27   Linc. Versailles  13466   14    3.0       3.5     15    3830     201   \n",
       "28       Merc. Bobcat   3829   22    4.0       3.0      9    2580     169   \n",
       "29       Merc. Cougar   5379   14    4.0       3.5     16    4060     221   \n",
       "..                ...    ...  ...    ...       ...    ...     ...     ...   \n",
       "44      Plym. Sapporo   6486   26    NaN       1.5      8    2520     182   \n",
       "45       Plym. Volare   4060   18    2.0       5.0     16    3330     201   \n",
       "46     Pont. Catalina   5798   18    4.0       4.0     20    3700     214   \n",
       "47     Pont. Firebird   4934   18    1.0       1.5      7    3470     198   \n",
       "48   Pont. Grand Prix   5222   19    3.0       2.0     16    3210     201   \n",
       "49      Pont. Le Mans   4723   19    3.0       3.5     17    3200     199   \n",
       "50      Pont. Phoenix   4424   19    NaN       3.5     13    3420     203   \n",
       "51      Pont. Sunbird   4172   24    2.0       2.0      7    2690     179   \n",
       "52          Audi 5000   9690   17    5.0       3.0     15    2830     189   \n",
       "53           Audi Fox   6295   23    3.0       2.5     11    2070     174   \n",
       "54           BMW 320i   9735   25    4.0       2.5     12    2650     177   \n",
       "55         Datsun 200   6229   23    4.0       1.5      6    2370     170   \n",
       "56         Datsun 210   4589   35    5.0       2.0      8    2020     165   \n",
       "57         Datsun 510   5079   24    4.0       2.5      8    2280     170   \n",
       "58         Datsun 810   8129   21    4.0       2.5      8    2750     184   \n",
       "59        Fiat Strada   4296   21    3.0       2.5     16    2130     161   \n",
       "60       Honda Accord   5799   25    5.0       3.0     10    2240     172   \n",
       "61        Honda Civic   4499   28    4.0       2.5      5    1760     149   \n",
       "62          Mazda GLC   3995   30    4.0       3.5     11    1980     154   \n",
       "63        Peugeot 604  12990   14    NaN       3.5     14    3420     192   \n",
       "64     Renault Le Car   3895   26    3.0       3.0     10    1830     142   \n",
       "65             Subaru   3798   35    5.0       2.5     11    2050     164   \n",
       "66      Toyota Celica   5899   18    5.0       2.5     14    2410     174   \n",
       "67     Toyota Corolla   3748   31    5.0       3.0      9    2200     165   \n",
       "68      Toyota Corona   5719   18    5.0       2.0     11    2670     175   \n",
       "69          VW Dasher   7140   23    4.0       2.5     12    2160     172   \n",
       "70          VW Diesel   5397   41    5.0       3.0     15    2040     155   \n",
       "71          VW Rabbit   4697   25    4.0       3.0     15    1930     155   \n",
       "72        VW Scirocco   6850   25    4.0       2.0     16    1990     156   \n",
       "73          Volvo 260  11995   17    5.0       2.5     14    3170     193   \n",
       "\n",
       "    turn  displacement  gear_ratio   foreign  \n",
       "0     40           121        3.58  Domestic  \n",
       "1     40           258        2.53  Domestic  \n",
       "2     35           121        3.08  Domestic  \n",
       "3     40           196        2.93  Domestic  \n",
       "4     43           350        2.41  Domestic  \n",
       "5     43           231        2.73  Domestic  \n",
       "6     34           304        2.87  Domestic  \n",
       "7     42           196        2.93  Domestic  \n",
       "8     43           231        2.93  Domestic  \n",
       "9     42           231        3.08  Domestic  \n",
       "10    44           425        2.28  Domestic  \n",
       "11    43           350        2.19  Domestic  \n",
       "12    45           350        2.24  Domestic  \n",
       "13    34           231        2.93  Domestic  \n",
       "14    43           250        2.56  Domestic  \n",
       "15    31           200        2.73  Domestic  \n",
       "16    41           200        2.73  Domestic  \n",
       "17    40           151        2.73  Domestic  \n",
       "18    43           250        2.56  Domestic  \n",
       "19    35            98        3.54  Domestic  \n",
       "20    46           318        2.47  Domestic  \n",
       "21    46           318        2.47  Domestic  \n",
       "22    46           225        2.94  Domestic  \n",
       "23    33            98        3.15  Domestic  \n",
       "24    43           140        3.08  Domestic  \n",
       "25    51           400        2.47  Domestic  \n",
       "26    48           400        2.47  Domestic  \n",
       "27    41           302        2.47  Domestic  \n",
       "28    39           140        2.73  Domestic  \n",
       "29    48           302        2.75  Domestic  \n",
       "..   ...           ...         ...       ...  \n",
       "44    38           119        3.54  Domestic  \n",
       "45    44           225        3.23  Domestic  \n",
       "46    42           231        2.73  Domestic  \n",
       "47    42           231        3.08  Domestic  \n",
       "48    45           231        2.93  Domestic  \n",
       "49    40           231        2.93  Domestic  \n",
       "50    43           231        3.08  Domestic  \n",
       "51    41           151        2.73  Domestic  \n",
       "52    37           131        3.20   Foreign  \n",
       "53    36            97        3.70   Foreign  \n",
       "54    34           121        3.64   Foreign  \n",
       "55    35           119        3.89   Foreign  \n",
       "56    32            85        3.70   Foreign  \n",
       "57    34           119        3.54   Foreign  \n",
       "58    38           146        3.55   Foreign  \n",
       "59    36           105        3.37   Foreign  \n",
       "60    36           107        3.05   Foreign  \n",
       "61    34            91        3.30   Foreign  \n",
       "62    33            86        3.73   Foreign  \n",
       "63    38           163        3.58   Foreign  \n",
       "64    34            79        3.72   Foreign  \n",
       "65    36            97        3.81   Foreign  \n",
       "66    36           134        3.06   Foreign  \n",
       "67    35            97        3.21   Foreign  \n",
       "68    36           134        3.05   Foreign  \n",
       "69    36            97        3.74   Foreign  \n",
       "70    35            90        3.78   Foreign  \n",
       "71    35            89        3.78   Foreign  \n",
       "72    36            97        3.78   Foreign  \n",
       "73    37           163        2.98   Foreign  \n",
       "\n",
       "[74 rows x 12 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get top or bottom of dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AMC Concord</td>\n",
       "      <td>4099</td>\n",
       "      <td>22</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>11</td>\n",
       "      <td>2930</td>\n",
       "      <td>186</td>\n",
       "      <td>40</td>\n",
       "      <td>121</td>\n",
       "      <td>3.58</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AMC Pacer</td>\n",
       "      <td>4749</td>\n",
       "      <td>17</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>11</td>\n",
       "      <td>3350</td>\n",
       "      <td>173</td>\n",
       "      <td>40</td>\n",
       "      <td>258</td>\n",
       "      <td>2.53</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AMC Spirit</td>\n",
       "      <td>3799</td>\n",
       "      <td>22</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>12</td>\n",
       "      <td>2640</td>\n",
       "      <td>168</td>\n",
       "      <td>35</td>\n",
       "      <td>121</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "0  AMC Concord   4099   22    3.0       2.5     11    2930     186    40   \n",
       "1    AMC Pacer   4749   17    3.0       3.0     11    3350     173    40   \n",
       "2   AMC Spirit   3799   22    NaN       3.0     12    2640     168    35   \n",
       "\n",
       "   displacement  gear_ratio   foreign  \n",
       "0           121        3.58  Domestic  \n",
       "1           258        2.53  Domestic  \n",
       "2           121        3.08  Domestic  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>VW Rabbit</td>\n",
       "      <td>4697</td>\n",
       "      <td>25</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>15</td>\n",
       "      <td>1930</td>\n",
       "      <td>155</td>\n",
       "      <td>35</td>\n",
       "      <td>89</td>\n",
       "      <td>3.78</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>VW Scirocco</td>\n",
       "      <td>6850</td>\n",
       "      <td>25</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>16</td>\n",
       "      <td>1990</td>\n",
       "      <td>156</td>\n",
       "      <td>36</td>\n",
       "      <td>97</td>\n",
       "      <td>3.78</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>Volvo 260</td>\n",
       "      <td>11995</td>\n",
       "      <td>17</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>14</td>\n",
       "      <td>3170</td>\n",
       "      <td>193</td>\n",
       "      <td>37</td>\n",
       "      <td>163</td>\n",
       "      <td>2.98</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "71    VW Rabbit   4697   25    4.0       3.0     15    1930     155    35   \n",
       "72  VW Scirocco   6850   25    4.0       2.0     16    1990     156    36   \n",
       "73    Volvo 260  11995   17    5.0       2.5     14    3170     193    37   \n",
       "\n",
       "    displacement  gear_ratio  foreign  \n",
       "71            89        3.78  Foreign  \n",
       "72            97        3.78  Foreign  \n",
       "73           163        2.98  Foreign  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto.tail(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get an X amount of random rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>VW Rabbit</td>\n",
       "      <td>4697</td>\n",
       "      <td>25</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>15</td>\n",
       "      <td>1930</td>\n",
       "      <td>155</td>\n",
       "      <td>35</td>\n",
       "      <td>89</td>\n",
       "      <td>3.78</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>Datsun 210</td>\n",
       "      <td>4589</td>\n",
       "      <td>35</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8</td>\n",
       "      <td>2020</td>\n",
       "      <td>165</td>\n",
       "      <td>32</td>\n",
       "      <td>85</td>\n",
       "      <td>3.70</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Buick Regal</td>\n",
       "      <td>5189</td>\n",
       "      <td>20</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>16</td>\n",
       "      <td>3280</td>\n",
       "      <td>200</td>\n",
       "      <td>42</td>\n",
       "      <td>196</td>\n",
       "      <td>2.93</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AMC Spirit</td>\n",
       "      <td>3799</td>\n",
       "      <td>22</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>12</td>\n",
       "      <td>2640</td>\n",
       "      <td>168</td>\n",
       "      <td>35</td>\n",
       "      <td>121</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Olds Delta 88</td>\n",
       "      <td>4890</td>\n",
       "      <td>18</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>20</td>\n",
       "      <td>3690</td>\n",
       "      <td>218</td>\n",
       "      <td>42</td>\n",
       "      <td>231</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "71      VW Rabbit   4697   25    4.0       3.0     15    1930     155    35   \n",
       "56     Datsun 210   4589   35    5.0       2.0      8    2020     165    32   \n",
       "7     Buick Regal   5189   20    3.0       2.0     16    3280     200    42   \n",
       "2      AMC Spirit   3799   22    NaN       3.0     12    2640     168    35   \n",
       "37  Olds Delta 88   4890   18    4.0       4.0     20    3690     218    42   \n",
       "\n",
       "    displacement  gear_ratio   foreign  \n",
       "71            89        3.78   Foreign  \n",
       "56            85        3.70   Foreign  \n",
       "7            196        2.93  Domestic  \n",
       "2            121        3.08  Domestic  \n",
       "37           231        2.73  Domestic  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = 5\n",
    "df_auto.sample(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select column(s) based on name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* the below returns a pandas `Series` object, this is different than a pandas `Dataframe` object!   \n",
    "You can tell by the way that it looks when shown."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    AMC Concord\n",
       "1      AMC Pacer\n",
       "2     AMC Spirit\n",
       "Name: make, dtype: object"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['make'].head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the column name has no whitespace you can also use a dot followed with the column name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    AMC Concord\n",
       "1      AMC Pacer\n",
       "2     AMC Spirit\n",
       "Name: make, dtype: object"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto.make.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**If you want multiple columns you need to use double brackets:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AMC Concord</td>\n",
       "      <td>4099</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AMC Pacer</td>\n",
       "      <td>4749</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AMC Spirit</td>\n",
       "      <td>3799</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Buick Century</td>\n",
       "      <td>4816</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Buick Electra</td>\n",
       "      <td>7827</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Buick LeSabre</td>\n",
       "      <td>5788</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Buick Opel</td>\n",
       "      <td>4453</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Buick Regal</td>\n",
       "      <td>5189</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Buick Riviera</td>\n",
       "      <td>10372</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Buick Skylark</td>\n",
       "      <td>4082</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            make  price  mpg\n",
       "0    AMC Concord   4099   22\n",
       "1      AMC Pacer   4749   17\n",
       "2     AMC Spirit   3799   22\n",
       "3  Buick Century   4816   20\n",
       "4  Buick Electra   7827   15\n",
       "5  Buick LeSabre   5788   18\n",
       "6     Buick Opel   4453   26\n",
       "7    Buick Regal   5189   20\n",
       "8  Buick Riviera  10372   16\n",
       "9  Buick Skylark   4082   19"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto[['make', 'price', 'mpg']].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select  row based on index value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df_auto[['make', 'price', 'mpg', 'trunk', 'headroom']].set_index('make')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "price       10372.0\n",
       "mpg            16.0\n",
       "trunk          17.0\n",
       "headroom        3.5\n",
       "Name: Buick Riviera, dtype: float64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc['Buick Riviera']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* notice the appearance, this returned a pandas.Series object not a pandas.Dataframe object "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select row based on index position"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>trunk</th>\n",
       "      <th>headroom</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>make</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AMC Spirit</th>\n",
       "      <td>3799</td>\n",
       "      <td>22</td>\n",
       "      <td>12</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Buick Century</th>\n",
       "      <td>4816</td>\n",
       "      <td>20</td>\n",
       "      <td>16</td>\n",
       "      <td>4.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Buick Electra</th>\n",
       "      <td>7827</td>\n",
       "      <td>15</td>\n",
       "      <td>20</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               price  mpg  trunk  headroom\n",
       "make                                      \n",
       "AMC Spirit      3799   22     12       3.0\n",
       "Buick Century   4816   20     16       4.5\n",
       "Buick Electra   7827   15     20       4.0"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[2:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You can also include columns based on their column (!) index position:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>trunk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>make</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AMC Spirit</th>\n",
       "      <td>22</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Buick Century</th>\n",
       "      <td>20</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Buick Electra</th>\n",
       "      <td>15</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               mpg  trunk\n",
       "make                     \n",
       "AMC Spirit      22     12\n",
       "Buick Century   20     16\n",
       "Buick Electra   15     20"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[2:5, 1:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* In the example above the first `0:3` selects the first 3 rows, the second `1:3` selects the 2nd and 3rd column."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select based on condition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In many cases you want to filter rows based on a condition. You can do this in Pandas by putting the condition inside square brackets.  \n",
    "\n",
    "It is worth explaining the intuition behind this method as a lot of people find it confusing:  \n",
    "\n",
    "1. You request Pandas to filter a dataframe by putting a condition between square brackets: df[ `condition` ] \n",
    "2. The `condition` is a sequence of `True` or `False` values for each row (so the length of the `condition` always has to match the number of rows in the dataframe!)\n",
    "3. In Pandas you can generate a `True` or `False` value for each row by simply writing a boolean expression on the whole column. \n",
    "4. Pandas will then only show those rows where the value is `True`\n",
    "\n",
    "In more practical terms:\n",
    "\n",
    "`df_auto['price'] < 3800` will evaluate each row of `df_auto['price']` and return, for that row, whether the condition is `True` or `False`:\n",
    "\n",
    "``\n",
    "0     False\n",
    "1     False\n",
    "2      True\n",
    "3     False\n",
    "4     False\n",
    "5     False\n",
    "``\n",
    "\n",
    "By putting that condition in square brackets `df_auto[ df_auto['price'] < 3800 ]` pandas will first generate a sequence of `True` / `False` values and then only display the rows for which the value is `True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AMC Spirit</td>\n",
       "      <td>3799</td>\n",
       "      <td>22</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>12</td>\n",
       "      <td>2640</td>\n",
       "      <td>168</td>\n",
       "      <td>35</td>\n",
       "      <td>121</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Chev. Chevette</td>\n",
       "      <td>3299</td>\n",
       "      <td>29</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>9</td>\n",
       "      <td>2110</td>\n",
       "      <td>163</td>\n",
       "      <td>34</td>\n",
       "      <td>231</td>\n",
       "      <td>2.93</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667</td>\n",
       "      <td>24</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7</td>\n",
       "      <td>2750</td>\n",
       "      <td>179</td>\n",
       "      <td>40</td>\n",
       "      <td>151</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>Merc. Zephyr</td>\n",
       "      <td>3291</td>\n",
       "      <td>20</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>17</td>\n",
       "      <td>2830</td>\n",
       "      <td>195</td>\n",
       "      <td>43</td>\n",
       "      <td>140</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>Subaru</td>\n",
       "      <td>3798</td>\n",
       "      <td>35</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>11</td>\n",
       "      <td>2050</td>\n",
       "      <td>164</td>\n",
       "      <td>36</td>\n",
       "      <td>97</td>\n",
       "      <td>3.81</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>Toyota Corolla</td>\n",
       "      <td>3748</td>\n",
       "      <td>31</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>9</td>\n",
       "      <td>2200</td>\n",
       "      <td>165</td>\n",
       "      <td>35</td>\n",
       "      <td>97</td>\n",
       "      <td>3.21</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "2       AMC Spirit   3799   22    NaN       3.0     12    2640     168    35   \n",
       "13  Chev. Chevette   3299   29    3.0       2.5      9    2110     163    34   \n",
       "17     Chev. Monza   3667   24    2.0       2.0      7    2750     179    40   \n",
       "33    Merc. Zephyr   3291   20    3.0       3.5     17    2830     195    43   \n",
       "65          Subaru   3798   35    5.0       2.5     11    2050     164    36   \n",
       "67  Toyota Corolla   3748   31    5.0       3.0      9    2200     165    35   \n",
       "\n",
       "    displacement  gear_ratio   foreign  \n",
       "2            121        3.08  Domestic  \n",
       "13           231        2.93  Domestic  \n",
       "17           151        2.73  Domestic  \n",
       "33           140        3.08  Domestic  \n",
       "65            97        3.81   Foreign  \n",
       "67            97        3.21   Foreign  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto[ df_auto['price'] < 3800 ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also combine multiple conditions by just chaining the boolean expression.   \n",
    "\n",
    "* For an **AND** statement use: `&`\n",
    "* For an **OR** statement use: `|`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>Subaru</td>\n",
       "      <td>3798</td>\n",
       "      <td>35</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>11</td>\n",
       "      <td>2050</td>\n",
       "      <td>164</td>\n",
       "      <td>36</td>\n",
       "      <td>97</td>\n",
       "      <td>3.81</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>Toyota Corolla</td>\n",
       "      <td>3748</td>\n",
       "      <td>31</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>9</td>\n",
       "      <td>2200</td>\n",
       "      <td>165</td>\n",
       "      <td>35</td>\n",
       "      <td>97</td>\n",
       "      <td>3.21</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "65          Subaru   3798   35    5.0       2.5     11    2050     164    36   \n",
       "67  Toyota Corolla   3748   31    5.0       3.0      9    2200     165    35   \n",
       "\n",
       "    displacement  gear_ratio  foreign  \n",
       "65            97        3.81  Foreign  \n",
       "67            97        3.21  Foreign  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto[(df_auto['price'] < 3800) & (df_auto['foreign'] == 'Foreign')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** all the above return new dataframes that are removed if we do not assign them.  \n",
    "If we want to keep it as a separate dataframe we have to assign it like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>Subaru</td>\n",
       "      <td>3798</td>\n",
       "      <td>35</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>11</td>\n",
       "      <td>2050</td>\n",
       "      <td>164</td>\n",
       "      <td>36</td>\n",
       "      <td>97</td>\n",
       "      <td>3.81</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>Toyota Corolla</td>\n",
       "      <td>3748</td>\n",
       "      <td>31</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>9</td>\n",
       "      <td>2200</td>\n",
       "      <td>165</td>\n",
       "      <td>35</td>\n",
       "      <td>97</td>\n",
       "      <td>3.21</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "65          Subaru   3798   35    5.0       2.5     11    2050     164    36   \n",
       "67  Toyota Corolla   3748   31    5.0       3.0      9    2200     165    35   \n",
       "\n",
       "    displacement  gear_ratio  foreign  \n",
       "65            97        3.81  Foreign  \n",
       "67            97        3.21  Foreign  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto_small = df_auto[(df_auto.price < 3800) & (df_auto.foreign == 'Foreign')]\n",
    "df_auto_small"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sort dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229</td>\n",
       "      <td>23</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6</td>\n",
       "      <td>2370</td>\n",
       "      <td>170</td>\n",
       "      <td>35</td>\n",
       "      <td>119</td>\n",
       "      <td>3.89</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934</td>\n",
       "      <td>18</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7</td>\n",
       "      <td>3470</td>\n",
       "      <td>198</td>\n",
       "      <td>42</td>\n",
       "      <td>231</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Plym. Sapporo</td>\n",
       "      <td>6486</td>\n",
       "      <td>26</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.5</td>\n",
       "      <td>8</td>\n",
       "      <td>2520</td>\n",
       "      <td>182</td>\n",
       "      <td>38</td>\n",
       "      <td>119</td>\n",
       "      <td>3.54</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389</td>\n",
       "      <td>28</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>9</td>\n",
       "      <td>1800</td>\n",
       "      <td>147</td>\n",
       "      <td>33</td>\n",
       "      <td>98</td>\n",
       "      <td>3.15</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667</td>\n",
       "      <td>24</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7</td>\n",
       "      <td>2750</td>\n",
       "      <td>179</td>\n",
       "      <td>40</td>\n",
       "      <td>151</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "55      Datsun 200   6229   23    4.0       1.5      6    2370     170    35   \n",
       "47  Pont. Firebird   4934   18    1.0       1.5      7    3470     198    42   \n",
       "44   Plym. Sapporo   6486   26    NaN       1.5      8    2520     182    38   \n",
       "23     Ford Fiesta   4389   28    4.0       1.5      9    1800     147    33   \n",
       "17     Chev. Monza   3667   24    2.0       2.0      7    2750     179    40   \n",
       "\n",
       "    displacement  gear_ratio   foreign  \n",
       "55           119        3.89   Foreign  \n",
       "47           231        3.08  Domestic  \n",
       "44           119        3.54  Domestic  \n",
       "23            98        3.15  Domestic  \n",
       "17           151        2.73  Domestic  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto.sort_values(by=['headroom', 'trunk'], inplace=True)\n",
    "df_auto.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">View a dataframe using `qgrid`</span><a id='qgrid'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The primary workflow for inspecting parts of the data is to just create a new temporary dataframe with the data that you want to see. This is fairly quick once you get used to it, but in the beginning it can feel cumbersome. \n",
    "\n",
    "A neat workaround for beginners is to use a package called `qgrid` to quickly inspect your data:  \n",
    "\n",
    "This is the GitHub page for `qgrid`: https://github.com/quantopian/qgrid  \n",
    "You can install it by running the following two commands in your command line:  \n",
    "1. `pip install qgrid`   \n",
    "2. `jupyter nbextension enable --py --sys-prefix qgrid`\n",
    "\n",
    "Using it is simple:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**First make sure you import the `show_grid` function**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "from qgrid import show_grid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You can inspect a Dataframe as follows:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "73493d2a534c47d09f2849734b2372be",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>QgridWidget</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "QgridWidget(grid_options={'fullWidthRows': True, 'syncColumnCellResize': True, 'forceFitColumns': True, 'defaultColumnWidth': 150, 'rowHeight': 28, 'enableColumnReorder': False, 'enableTextSelectionOnCells': True, 'editable': True, 'autoEdit': False, 'explicitInitialization': True, 'maxVisibleRows': 15, 'minVisibleRows': 8, 'sortable': True, 'filterable': True, 'highlightSelectedCell': False, 'highlightSelectedRow': True}, precision=5)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_grid(df_auto)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Several things to note about qgrid:** \n",
    "- If you save the notebook with these `qgrids` it is going to increase the file-size dramatically, it essentially saves the data with the notebook. Try to avoid this, use it only for inspection.\n",
    "- Opening very big dataframes using `show_grid()` is not a good idea.  \n",
    "- These `qgrids` will only display locally, not on GitHub. Therefore, if you see this on GitHub, you will not see the actual `qgrid`.\n",
    "- There are a bunch of options you can use with `show_grid()`, however, I strongly discourage you to modify a dataframe using the qgrid toolbars. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Dealing with datatypes</span><a id='datatypes'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important to pay attention to the datatypes contained in a column. A lot of errors that you will encounter relate to wrong datatypes (e.g. because of data errors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show current datatypes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "make             object\n",
       "price             int64\n",
       "mpg               int64\n",
       "rep78           float64\n",
       "headroom        float64\n",
       "trunk             int64\n",
       "weight            int64\n",
       "length            int64\n",
       "turn              int64\n",
       "displacement      int64\n",
       "gear_ratio      float64\n",
       "foreign          object\n",
       "dtype: object"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert datatypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can convert the datatype of a column in two ways:  \n",
    "\n",
    "1. Loop over the values and convert them individually\n",
    "2. Use the build-in Pandas functions to convert the column in one go"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*1) Convert values individually*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('O')"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['length'].apply(lambda x: str(x)).dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: `'O'` stands for 'object'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('int64')"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['length'].apply(lambda x: int(x)).dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*2) Convert column directly*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to convert a column to `string` I recommend to use `.astype(str)`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('O')"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['length'].astype(str).dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to convert a column to `numeric` I recommend to use `df.to_numeric()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('int64')"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.to_numeric(df_auto['length']).dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The section `dealing with dates` will discuss how to convert a column with `dates`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Handling missing values</span><a id='missing-values'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dealing with missing values is easy in Pandas, as long as you are careful in defining them as `np.nan` (and **not** a string value like 'np.nan')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/missing_data.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add some missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* We define a missing value as `np.nan` so we can consistently select them!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_auto.loc['UvT_Car'] = [np.nan for x in range(0,len(df_auto.columns))]\n",
    "df_auto.loc['UvT_Bike'] = [np.nan for x in range(0,len(df_auto.columns))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>UvT_Car</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>UvT_Bike</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "UvT_Car   NaN    NaN  NaN    NaN       NaN    NaN     NaN     NaN   NaN   \n",
       "UvT_Bike  NaN    NaN  NaN    NaN       NaN    NaN     NaN     NaN   NaN   \n",
       "\n",
       "          displacement  gear_ratio foreign  \n",
       "UvT_Car            NaN         NaN     NaN  \n",
       "UvT_Bike           NaN         NaN     NaN  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto.loc[['UvT_Car', 'UvT_Bike']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select missing or non-missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Always use `pd.isnull()` or `pd.notnull()` as it is most reliable.  \n",
    "`df_auto.make == np.nan` will **not** work consistently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>UvT_Car</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>UvT_Bike</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "UvT_Car   NaN    NaN  NaN    NaN       NaN    NaN     NaN     NaN   NaN   \n",
       "UvT_Bike  NaN    NaN  NaN    NaN       NaN    NaN     NaN     NaN   NaN   \n",
       "\n",
       "          displacement  gear_ratio foreign  \n",
       "UvT_Car            NaN         NaN     NaN  \n",
       "UvT_Bike           NaN         NaN     NaN  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto[pd.isnull(df_auto.make)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2370.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>119.0</td>\n",
       "      <td>3.89</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7.0</td>\n",
       "      <td>3470.0</td>\n",
       "      <td>198.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>231.0</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Plym. Sapporo</td>\n",
       "      <td>6486.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.5</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2520.0</td>\n",
       "      <td>182.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>119.0</td>\n",
       "      <td>3.54</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1800.0</td>\n",
       "      <td>147.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>98.0</td>\n",
       "      <td>3.15</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2750.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price   mpg  rep78  headroom  trunk  weight  length  \\\n",
       "55      Datsun 200  6229.0  23.0    4.0       1.5    6.0  2370.0   170.0   \n",
       "47  Pont. Firebird  4934.0  18.0    1.0       1.5    7.0  3470.0   198.0   \n",
       "44   Plym. Sapporo  6486.0  26.0    NaN       1.5    8.0  2520.0   182.0   \n",
       "23     Ford Fiesta  4389.0  28.0    4.0       1.5    9.0  1800.0   147.0   \n",
       "17     Chev. Monza  3667.0  24.0    2.0       2.0    7.0  2750.0   179.0   \n",
       "\n",
       "    turn  displacement  gear_ratio   foreign  \n",
       "55  35.0         119.0        3.89   Foreign  \n",
       "47  42.0         231.0        3.08  Domestic  \n",
       "44  38.0         119.0        3.54  Domestic  \n",
       "23  33.0          98.0        3.15  Domestic  \n",
       "17  40.0         151.0        2.73  Domestic  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto[pd.notnull(df_auto.make)].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fill missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To fill missing values with something we can use `.fillna()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>UvT_Car</th>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>UvT_Bike</th>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             make    price      mpg    rep78 headroom    trunk   weight  \\\n",
       "UvT_Car   Missing  Missing  Missing  Missing  Missing  Missing  Missing   \n",
       "UvT_Bike  Missing  Missing  Missing  Missing  Missing  Missing  Missing   \n",
       "\n",
       "           length     turn displacement gear_ratio  foreign  \n",
       "UvT_Car   Missing  Missing      Missing    Missing  Missing  \n",
       "UvT_Bike  Missing  Missing      Missing    Missing  Missing  "
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df_auto.fillna('Missing')\n",
    "df.loc[['UvT_Car', 'UvT_Bike']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Drop rows with missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To drop missing values we can use `.dropna()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "45          Plym. Volare\n",
       "UvT_Car              NaN\n",
       "UvT_Bike             NaN\n",
       "Name: make, dtype: object"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['make'].tail(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36       Olds Cutlass\n",
       "22    Dodge St. Regis\n",
       "45       Plym. Volare\n",
       "Name: make, dtype: object"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df_auto.dropna(axis=0)\n",
    "df['make'].tail(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Work with data in the dataframe</span><a id='work-with-data'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combine columns (and output it to a new column)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Remember:* You can select a column using:\n",
    "1. `df_auto['price']`\n",
    "2. `df_auto.price` --> but this one only works if there are no spaces in the column name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>trunk</th>\n",
       "      <th>price_trunk_ratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>6229.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1038.166667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>4934.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>704.857143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>6486.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>810.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>4389.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>487.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>3667.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>523.857143</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     price  trunk  price_trunk_ratio\n",
       "55  6229.0    6.0        1038.166667\n",
       "47  4934.0    7.0         704.857143\n",
       "44  6486.0    8.0         810.750000\n",
       "23  4389.0    9.0         487.666667\n",
       "17  3667.0    7.0         523.857143"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['price_trunk_ratio'] = df_auto.price / df_auto.trunk\n",
    "df_auto[['price', 'trunk', 'price_trunk_ratio']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate a new column by iterating over the dataframe per row"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are multiple ways to iterate over rows.  \n",
    "They mainly different in their trade-off between ease-of-use, readability, and performance.  \n",
    "\n",
    "I will show the three main possibilities.\n",
    "\n",
    "For the sake of demonstration, let's say our goal is to achieve the following:    \n",
    "> If the car is a foreign brand, multiple the price by 1.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Option 1: use `.apply()` with `lambda`**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* `lambda` is a so-called anonymous function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>foreign</th>\n",
       "      <th>new_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>Foreign</td>\n",
       "      <td>9343.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>4934.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Plym. Sapporo</td>\n",
       "      <td>6486.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>6486.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>4389.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>3667.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price   foreign  new_price\n",
       "55      Datsun 200  6229.0   Foreign     9343.5\n",
       "47  Pont. Firebird  4934.0  Domestic     4934.0\n",
       "44   Plym. Sapporo  6486.0  Domestic     6486.0\n",
       "23     Ford Fiesta  4389.0  Domestic     4389.0\n",
       "17     Chev. Monza  3667.0  Domestic     3667.0"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logic = lambda x: x.price*1.5 if x.foreign == 'Foreign' else x.price\n",
    "df_auto['new_price'] = df_auto.apply(logic, axis=1)\n",
    "df_auto[['make', 'price', 'foreign', 'new_price']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Option 2: use `.apply()` with a function**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the example above we use an anonymous `lambda` function.  \n",
    "For more complex processing it is possible to use a defined function and call it in `.apply()`  \n",
    "\n",
    "**Personal note:** This is often my preferred method as it is the most flexible and a lot easier to read."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "def new_price_function(x):\n",
    "    if x.foreign == 'Foreign':\n",
    "        return x.price * 1.5\n",
    "    else:\n",
    "        return x.price"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>foreign</th>\n",
       "      <th>new_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>Foreign</td>\n",
       "      <td>9343.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>4934.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Plym. Sapporo</td>\n",
       "      <td>6486.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>6486.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>4389.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>3667.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price   foreign  new_price\n",
       "55      Datsun 200  6229.0   Foreign     9343.5\n",
       "47  Pont. Firebird  4934.0  Domestic     4934.0\n",
       "44   Plym. Sapporo  6486.0  Domestic     6486.0\n",
       "23     Ford Fiesta  4389.0  Domestic     4389.0\n",
       "17     Chev. Monza  3667.0  Domestic     3667.0"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['new_price'] = df_auto.apply(new_price_function, axis=1)\n",
    "df_auto[['make', 'price', 'foreign', 'new_price']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* make sure to include the `axis = 1` argument, this tells Pandas to iterate over the rows and not the columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Option 3: use a list comprehension:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>foreign</th>\n",
       "      <th>new_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>8129.0</td>\n",
       "      <td>Foreign</td>\n",
       "      <td>12193.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>4723.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>4723.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>4647.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>4647.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>4733.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>4733.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>10371.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>10371.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      price   foreign  new_price\n",
       "58   8129.0   Foreign    12193.5\n",
       "49   4723.0  Domestic     4723.0\n",
       "41   4647.0  Domestic     4647.0\n",
       "36   4733.0  Domestic     4733.0\n",
       "40  10371.0  Domestic    10371.0"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['new_price'] = [p*1.5 if f == 'Foreign' else p for p, f in zip(df_auto.price, df_auto.foreign)]\n",
    "df_auto[['price', 'foreign', 'new_price']].sample(5, random_state=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* `random_state=1` makes sure that we get the same random sample every time we run it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Combining dataframes</span><a id='combining-dataframes'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can combine dataframes in three ways:\n",
    "\n",
    "1. Merge\n",
    "2. Join\n",
    "3. Append"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I will demonstrate that using the following two datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_auto_p1 = df_auto[['make', 'price', 'mpg']]\n",
    "df_auto_p2 = df_auto[['make', 'headroom', 'trunk']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>23.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Plym. Sapporo</td>\n",
       "      <td>6486.0</td>\n",
       "      <td>26.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price   mpg\n",
       "55      Datsun 200  6229.0  23.0\n",
       "47  Pont. Firebird  4934.0  18.0\n",
       "44   Plym. Sapporo  6486.0  26.0"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto_p1.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Plym. Sapporo</td>\n",
       "      <td>1.5</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make  headroom  trunk\n",
       "55      Datsun 200       1.5    6.0\n",
       "47  Pont. Firebird       1.5    7.0\n",
       "44   Plym. Sapporo       1.5    8.0"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto_p2.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Merge datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.merge.html  \n",
    "The `.merge()` function is one of my personal favorites, it is really easy to use.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Plym. Sapporo</td>\n",
       "      <td>6486.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             make   price   mpg  headroom  trunk\n",
       "0      Datsun 200  6229.0  23.0       1.5    6.0\n",
       "1  Pont. Firebird  4934.0  18.0       1.5    7.0\n",
       "2   Plym. Sapporo  6486.0  26.0       1.5    8.0"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "merged_auto = pd.merge(df_auto_p1, df_auto_p2, how='left', on='make')\n",
    "merged_auto.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Join datasets on index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both dataframes need to have the same column set as the index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_auto_p1.set_index('make', inplace=True)\n",
    "df_auto_p2.set_index('make', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AMC Concord</td>\n",
       "      <td>4099.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AMC Pacer</td>\n",
       "      <td>4749.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AMC Spirit</td>\n",
       "      <td>3799.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          make   price   mpg  headroom  trunk\n",
       "0  AMC Concord  4099.0  22.0       2.5   11.0\n",
       "1    AMC Pacer  4749.0  17.0       3.0   11.0\n",
       "2   AMC Spirit  3799.0  22.0       3.0   12.0"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joined_auto = df_auto_p1.join(df_auto_p2)\n",
    "joined_auto.reset_index().head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) Append data to the dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See http://pandas.pydata.org/pandas-docs/stable/merging.html#concatenating-objects  \n",
    "\n",
    "*Note:* There is also a shortcut function called `.append()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_auto_i1 = df_auto.iloc[0:3]\n",
    "df_auto_i2 = df_auto.iloc[3:6]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "      <th>price_trunk_ratio</th>\n",
       "      <th>new_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2370.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>119.0</td>\n",
       "      <td>3.89</td>\n",
       "      <td>Foreign</td>\n",
       "      <td>1038.166667</td>\n",
       "      <td>9343.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7.0</td>\n",
       "      <td>3470.0</td>\n",
       "      <td>198.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>231.0</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>704.857143</td>\n",
       "      <td>4934.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Plym. Sapporo</td>\n",
       "      <td>6486.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.5</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2520.0</td>\n",
       "      <td>182.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>119.0</td>\n",
       "      <td>3.54</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>810.750000</td>\n",
       "      <td>6486.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price   mpg  rep78  headroom  trunk  weight  length  \\\n",
       "55      Datsun 200  6229.0  23.0    4.0       1.5    6.0  2370.0   170.0   \n",
       "47  Pont. Firebird  4934.0  18.0    1.0       1.5    7.0  3470.0   198.0   \n",
       "44   Plym. Sapporo  6486.0  26.0    NaN       1.5    8.0  2520.0   182.0   \n",
       "\n",
       "    turn  displacement  gear_ratio   foreign  price_trunk_ratio  new_price  \n",
       "55  35.0         119.0        3.89   Foreign        1038.166667     9343.5  \n",
       "47  42.0         231.0        3.08  Domestic         704.857143     4934.0  \n",
       "44  38.0         119.0        3.54  Domestic         810.750000     6486.0  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto_i1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "      <th>price_trunk_ratio</th>\n",
       "      <th>new_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1800.0</td>\n",
       "      <td>147.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>98.0</td>\n",
       "      <td>3.15</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>487.666667</td>\n",
       "      <td>4389.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2750.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>523.857143</td>\n",
       "      <td>3667.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Pont. Sunbird</td>\n",
       "      <td>4172.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2690.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>596.000000</td>\n",
       "      <td>4172.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             make   price   mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "23    Ford Fiesta  4389.0  28.0    4.0       1.5    9.0  1800.0   147.0  33.0   \n",
       "17    Chev. Monza  3667.0  24.0    2.0       2.0    7.0  2750.0   179.0  40.0   \n",
       "51  Pont. Sunbird  4172.0  24.0    2.0       2.0    7.0  2690.0   179.0  41.0   \n",
       "\n",
       "    displacement  gear_ratio   foreign  price_trunk_ratio  new_price  \n",
       "23          98.0        3.15  Domestic         487.666667     4389.0  \n",
       "17         151.0        2.73  Domestic         523.857143     3667.0  \n",
       "51         151.0        2.73  Domestic         596.000000     4172.0  "
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto_i2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the higher level function `concat()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "      <th>price_trunk_ratio</th>\n",
       "      <th>new_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2370.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>119.0</td>\n",
       "      <td>3.89</td>\n",
       "      <td>Foreign</td>\n",
       "      <td>1038.166667</td>\n",
       "      <td>9343.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7.0</td>\n",
       "      <td>3470.0</td>\n",
       "      <td>198.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>231.0</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>704.857143</td>\n",
       "      <td>4934.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Plym. Sapporo</td>\n",
       "      <td>6486.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.5</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2520.0</td>\n",
       "      <td>182.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>119.0</td>\n",
       "      <td>3.54</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>810.750000</td>\n",
       "      <td>6486.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1800.0</td>\n",
       "      <td>147.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>98.0</td>\n",
       "      <td>3.15</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>487.666667</td>\n",
       "      <td>4389.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2750.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>523.857143</td>\n",
       "      <td>3667.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Pont. Sunbird</td>\n",
       "      <td>4172.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2690.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>596.000000</td>\n",
       "      <td>4172.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price   mpg  rep78  headroom  trunk  weight  length  \\\n",
       "55      Datsun 200  6229.0  23.0    4.0       1.5    6.0  2370.0   170.0   \n",
       "47  Pont. Firebird  4934.0  18.0    1.0       1.5    7.0  3470.0   198.0   \n",
       "44   Plym. Sapporo  6486.0  26.0    NaN       1.5    8.0  2520.0   182.0   \n",
       "23     Ford Fiesta  4389.0  28.0    4.0       1.5    9.0  1800.0   147.0   \n",
       "17     Chev. Monza  3667.0  24.0    2.0       2.0    7.0  2750.0   179.0   \n",
       "51   Pont. Sunbird  4172.0  24.0    2.0       2.0    7.0  2690.0   179.0   \n",
       "\n",
       "    turn  displacement  gear_ratio   foreign  price_trunk_ratio  new_price  \n",
       "55  35.0         119.0        3.89   Foreign        1038.166667     9343.5  \n",
       "47  42.0         231.0        3.08  Domestic         704.857143     4934.0  \n",
       "44  38.0         119.0        3.54  Domestic         810.750000     6486.0  \n",
       "23  33.0          98.0        3.15  Domestic         487.666667     4389.0  \n",
       "17  40.0         151.0        2.73  Domestic         523.857143     3667.0  \n",
       "51  41.0         151.0        2.73  Domestic         596.000000     4172.0  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.concat([df_auto_i1, df_auto_i2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the shortcut fuction `append()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "      <th>price_trunk_ratio</th>\n",
       "      <th>new_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2370.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>119.0</td>\n",
       "      <td>3.89</td>\n",
       "      <td>Foreign</td>\n",
       "      <td>1038.166667</td>\n",
       "      <td>9343.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7.0</td>\n",
       "      <td>3470.0</td>\n",
       "      <td>198.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>231.0</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>704.857143</td>\n",
       "      <td>4934.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Plym. Sapporo</td>\n",
       "      <td>6486.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.5</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2520.0</td>\n",
       "      <td>182.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>119.0</td>\n",
       "      <td>3.54</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>810.750000</td>\n",
       "      <td>6486.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1800.0</td>\n",
       "      <td>147.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>98.0</td>\n",
       "      <td>3.15</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>487.666667</td>\n",
       "      <td>4389.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2750.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>523.857143</td>\n",
       "      <td>3667.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Pont. Sunbird</td>\n",
       "      <td>4172.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2690.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>596.000000</td>\n",
       "      <td>4172.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price   mpg  rep78  headroom  trunk  weight  length  \\\n",
       "55      Datsun 200  6229.0  23.0    4.0       1.5    6.0  2370.0   170.0   \n",
       "47  Pont. Firebird  4934.0  18.0    1.0       1.5    7.0  3470.0   198.0   \n",
       "44   Plym. Sapporo  6486.0  26.0    NaN       1.5    8.0  2520.0   182.0   \n",
       "23     Ford Fiesta  4389.0  28.0    4.0       1.5    9.0  1800.0   147.0   \n",
       "17     Chev. Monza  3667.0  24.0    2.0       2.0    7.0  2750.0   179.0   \n",
       "51   Pont. Sunbird  4172.0  24.0    2.0       2.0    7.0  2690.0   179.0   \n",
       "\n",
       "    turn  displacement  gear_ratio   foreign  price_trunk_ratio  new_price  \n",
       "55  35.0         119.0        3.89   Foreign        1038.166667     9343.5  \n",
       "47  42.0         231.0        3.08  Domestic         704.857143     4934.0  \n",
       "44  38.0         119.0        3.54  Domestic         810.750000     6486.0  \n",
       "23  33.0          98.0        3.15  Domestic         487.666667     4389.0  \n",
       "17  40.0         151.0        2.73  Domestic         523.857143     3667.0  \n",
       "51  41.0         151.0        2.73  Domestic         596.000000     4172.0  "
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto_i1.append(df_auto_i2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Group-by operations</span><a id='groupby'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often you want to perform an operation within a group, in Pandas you achieve this by using `.groupby()`.\n",
    "\n",
    "Pandas `.groupby()` is a process involving one or more of the following steps (paraphrasing from the docs):  \n",
    "1. **Splitting** the data into groups based on some criteria\n",
    "2. **Applying** a function to each group independently\n",
    "3. **Combining** the results into a data structure\n",
    "\n",
    "For the full documentation see: http://pandas.pydata.org/pandas-docs/stable/groupby.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split the dataframe by creating a group object:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Step 1 is to create a `group` object that specifies the groups that we want to create."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "col_list = ['price', 'mpg', 'headroom', 'trunk', 'weight', 'length']\n",
    "grouped = df_auto[col_list + ['foreign']].groupby(['foreign'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After creating a `group` object we can apply operations to it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying example 1) Compute mean summary statistic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foreign</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Domestic</th>\n",
       "      <td>6072.423077</td>\n",
       "      <td>19.826923</td>\n",
       "      <td>3.153846</td>\n",
       "      <td>14.750000</td>\n",
       "      <td>3317.115385</td>\n",
       "      <td>196.134615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Foreign</th>\n",
       "      <td>6384.681818</td>\n",
       "      <td>24.772727</td>\n",
       "      <td>2.613636</td>\n",
       "      <td>11.409091</td>\n",
       "      <td>2315.909091</td>\n",
       "      <td>168.545455</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                price        mpg  headroom      trunk       weight      length\n",
       "foreign                                                                       \n",
       "Domestic  6072.423077  19.826923  3.153846  14.750000  3317.115385  196.134615\n",
       "Foreign   6384.681818  24.772727  2.613636  11.409091  2315.909091  168.545455"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying example 2) Retrieve particular group:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>4934.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7.0</td>\n",
       "      <td>3470.0</td>\n",
       "      <td>198.0</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>6486.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2520.0</td>\n",
       "      <td>182.0</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>4389.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1800.0</td>\n",
       "      <td>147.0</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>3667.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2750.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>4172.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2690.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     price   mpg  headroom  trunk  weight  length   foreign\n",
       "47  4934.0  18.0       1.5    7.0  3470.0   198.0  Domestic\n",
       "44  6486.0  26.0       1.5    8.0  2520.0   182.0  Domestic\n",
       "23  4389.0  28.0       1.5    9.0  1800.0   147.0  Domestic\n",
       "17  3667.0  24.0       2.0    7.0  2750.0   179.0  Domestic\n",
       "51  4172.0  24.0       2.0    7.0  2690.0   179.0  Domestic"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.get_group('Domestic').head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying example 3) Iterate over the groups in the `group` object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By iterating over each group you get a lot of flexibility as you can do anything you want with each group.   \n",
    "\n",
    "It is worth noting that each group is a dataframe object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Domestic\n",
      "     price   mpg  headroom  trunk  weight  length   foreign\n",
      "47  4934.0  18.0       1.5    7.0  3470.0   198.0  Domestic\n",
      "44  6486.0  26.0       1.5    8.0  2520.0   182.0  Domestic\n",
      "23  4389.0  28.0       1.5    9.0  1800.0   147.0  Domestic\n",
      "17  3667.0  24.0       2.0    7.0  2750.0   179.0  Domestic\n",
      "51  4172.0  24.0       2.0    7.0  2690.0   179.0  Domestic\n",
      "Foreign\n",
      "     price   mpg  headroom  trunk  weight  length  foreign\n",
      "55  6229.0  23.0       1.5    6.0  2370.0   170.0  Foreign\n",
      "56  4589.0  35.0       2.0    8.0  2020.0   165.0  Foreign\n",
      "68  5719.0  18.0       2.0   11.0  2670.0   175.0  Foreign\n",
      "72  6850.0  25.0       2.0   16.0  1990.0   156.0  Foreign\n",
      "61  4499.0  28.0       2.5    5.0  1760.0   149.0  Foreign\n"
     ]
    }
   ],
   "source": [
    "for name, group in grouped:\n",
    "    print(name)\n",
    "    print(group.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### It is also possible to use the `.apply()` function on `group` objects:   \n",
    "\n",
    "Using a `lambda df: ....` with `.apply()` is a nice way to iterature over subsets of the data.\n",
    "\n",
    "For example, let's say we want to get the cheapest car within each \"trunk\" size category:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "      <th>price_trunk_ratio</th>\n",
       "      <th>new_price</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trunk</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5.0</th>\n",
       "      <td>Honda Civic</td>\n",
       "      <td>4499.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1760.0</td>\n",
       "      <td>149.0</td>\n",
       "      <td>34.0</td>\n",
       "      <td>91.0</td>\n",
       "      <td>3.30</td>\n",
       "      <td>Foreign</td>\n",
       "      <td>899.800000</td>\n",
       "      <td>6748.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6.0</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2370.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>119.0</td>\n",
       "      <td>3.89</td>\n",
       "      <td>Foreign</td>\n",
       "      <td>1038.166667</td>\n",
       "      <td>9343.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7.0</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2750.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>523.857143</td>\n",
       "      <td>3667.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8.0</th>\n",
       "      <td>Dodge Colt</td>\n",
       "      <td>3984.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2120.0</td>\n",
       "      <td>163.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>98.0</td>\n",
       "      <td>3.54</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>498.000000</td>\n",
       "      <td>3984.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9.0</th>\n",
       "      <td>Chev. Chevette</td>\n",
       "      <td>3299.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2110.0</td>\n",
       "      <td>163.0</td>\n",
       "      <td>34.0</td>\n",
       "      <td>231.0</td>\n",
       "      <td>2.93</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>366.555556</td>\n",
       "      <td>3299.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 make   price   mpg  rep78  headroom  trunk  weight  length  \\\n",
       "trunk                                                                         \n",
       "5.0       Honda Civic  4499.0  28.0    4.0       2.5    5.0  1760.0   149.0   \n",
       "6.0        Datsun 200  6229.0  23.0    4.0       1.5    6.0  2370.0   170.0   \n",
       "7.0       Chev. Monza  3667.0  24.0    2.0       2.0    7.0  2750.0   179.0   \n",
       "8.0        Dodge Colt  3984.0  30.0    5.0       2.0    8.0  2120.0   163.0   \n",
       "9.0    Chev. Chevette  3299.0  29.0    3.0       2.5    9.0  2110.0   163.0   \n",
       "\n",
       "       turn  displacement  gear_ratio   foreign  price_trunk_ratio  new_price  \n",
       "trunk                                                                          \n",
       "5.0    34.0          91.0        3.30   Foreign         899.800000     6748.5  \n",
       "6.0    35.0         119.0        3.89   Foreign        1038.166667     9343.5  \n",
       "7.0    40.0         151.0        2.73  Domestic         523.857143     3667.0  \n",
       "8.0    35.0          98.0        3.54  Domestic         498.000000     3984.0  \n",
       "9.0    34.0         231.0        2.93  Domestic         366.555556     3299.0  "
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto.groupby('trunk').apply(lambda df: df.sort_values('price').iloc[0]).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aggregate groupby object to new dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to aggregate each group to one row in the new dataframe you have many options, below a couple of examples:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) `grouped.sum()` and `grouped.mean()`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foreign</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Domestic</th>\n",
       "      <td>315766.0</td>\n",
       "      <td>1031.0</td>\n",
       "      <td>164.0</td>\n",
       "      <td>767.0</td>\n",
       "      <td>172490.0</td>\n",
       "      <td>10199.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Foreign</th>\n",
       "      <td>140463.0</td>\n",
       "      <td>545.0</td>\n",
       "      <td>57.5</td>\n",
       "      <td>251.0</td>\n",
       "      <td>50950.0</td>\n",
       "      <td>3708.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             price     mpg  headroom  trunk    weight   length\n",
       "foreign                                                       \n",
       "Domestic  315766.0  1031.0     164.0  767.0  172490.0  10199.0\n",
       "Foreign   140463.0   545.0      57.5  251.0   50950.0   3708.0"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foreign</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Domestic</th>\n",
       "      <td>6072.423077</td>\n",
       "      <td>19.826923</td>\n",
       "      <td>3.153846</td>\n",
       "      <td>14.750000</td>\n",
       "      <td>3317.115385</td>\n",
       "      <td>196.134615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Foreign</th>\n",
       "      <td>6384.681818</td>\n",
       "      <td>24.772727</td>\n",
       "      <td>2.613636</td>\n",
       "      <td>11.409091</td>\n",
       "      <td>2315.909091</td>\n",
       "      <td>168.545455</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                price        mpg  headroom      trunk       weight      length\n",
       "foreign                                                                       \n",
       "Domestic  6072.423077  19.826923  3.153846  14.750000  3317.115385  196.134615\n",
       "Foreign   6384.681818  24.772727  2.613636  11.409091  2315.909091  168.545455"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) `grouped.count()` and `grouped.size()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foreign</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Domestic</th>\n",
       "      <td>52</td>\n",
       "      <td>52</td>\n",
       "      <td>52</td>\n",
       "      <td>52</td>\n",
       "      <td>52</td>\n",
       "      <td>52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Foreign</th>\n",
       "      <td>22</td>\n",
       "      <td>22</td>\n",
       "      <td>22</td>\n",
       "      <td>22</td>\n",
       "      <td>22</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          price  mpg  headroom  trunk  weight  length\n",
       "foreign                                              \n",
       "Domestic     52   52        52     52      52      52\n",
       "Foreign      22   22        22     22      22      22"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "foreign\n",
       "Domestic    52\n",
       "Foreign     22\n",
       "dtype: int64"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) `grouped.first()` and `grouped.last()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foreign</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Domestic</th>\n",
       "      <td>4934.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7.0</td>\n",
       "      <td>3470.0</td>\n",
       "      <td>198.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Foreign</th>\n",
       "      <td>6229.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2370.0</td>\n",
       "      <td>170.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           price   mpg  headroom  trunk  weight  length\n",
       "foreign                                                \n",
       "Domestic  4934.0  18.0       1.5    7.0  3470.0   198.0\n",
       "Foreign   6229.0  23.0       1.5    6.0  2370.0   170.0"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.first()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foreign</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Domestic</th>\n",
       "      <td>4060.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>3330.0</td>\n",
       "      <td>201.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Foreign</th>\n",
       "      <td>12990.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>14.0</td>\n",
       "      <td>3420.0</td>\n",
       "      <td>192.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            price   mpg  headroom  trunk  weight  length\n",
       "foreign                                                 \n",
       "Domestic   4060.0  18.0       5.0   16.0  3330.0   201.0\n",
       "Foreign   12990.0  14.0       3.5   14.0  3420.0   192.0"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.last()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** 4) You can also use the `.agg()` function to specify which operations to perform for each column**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th colspan=\"2\" halign=\"left\">mpg</th>\n",
       "      <th colspan=\"2\" halign=\"left\">trunk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>mean</th>\n",
       "      <th>median</th>\n",
       "      <th>mean</th>\n",
       "      <th>&lt;lambda&gt;</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foreign</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Domestic</th>\n",
       "      <td>4934.0</td>\n",
       "      <td>19.826923</td>\n",
       "      <td>19.0</td>\n",
       "      <td>14.750000</td>\n",
       "      <td>1475.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Foreign</th>\n",
       "      <td>6229.0</td>\n",
       "      <td>24.772727</td>\n",
       "      <td>24.5</td>\n",
       "      <td>11.409091</td>\n",
       "      <td>1140.909091</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           price        mpg             trunk             \n",
       "           first       mean median       mean     <lambda>\n",
       "foreign                                                   \n",
       "Domestic  4934.0  19.826923   19.0  14.750000  1475.000000\n",
       "Foreign   6229.0  24.772727   24.5  11.409091  1140.909091"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.agg({'price' : 'first', 'mpg' : ['mean', 'median'], 'trunk' : ['mean', (lambda x: 100 * np.mean(x))]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### And a lot of other operations!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many-many more things you can do with Pandas `.groupby`, too much to show here.  \n",
    "Feel free to check out the comprehensive documentation:  \n",
    "https://pandas.pydata.org/pandas-docs/stable/groupby.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Reshaping and Pivot Tables</span><a id='reshaping-pivot'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Pandas includes a variety of tools that allow you to reshape your DataFrame.  \n",
    "These tools are very powerful but can be a bit confusing to use. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create some sample data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuples = [('bar', 'one',   1, 2),\n",
    "          ('bar', 'two',   3, 4),\n",
    "          ('bar', 'three', 5, 6),\n",
    "          ('baz', 'one',   1, 2),\n",
    "          ('baz', 'two',   3, 4),\n",
    "          ('baz', 'three', 5, 6),\n",
    "          ('foo', 'one',   1, 2),\n",
    "          ('foo', 'two',   3, 4),\n",
    "          ('foo', 'three', 5, 6)\n",
    "         ]\n",
    "df = pd.DataFrame(tuples)\n",
    "df.columns = ['first', 'second', 'A', 'B']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>second</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bar</td>\n",
       "      <td>one</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>bar</td>\n",
       "      <td>two</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>bar</td>\n",
       "      <td>three</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>baz</td>\n",
       "      <td>one</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>baz</td>\n",
       "      <td>two</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>baz</td>\n",
       "      <td>three</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>foo</td>\n",
       "      <td>one</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>foo</td>\n",
       "      <td>two</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>foo</td>\n",
       "      <td>three</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  first second  A  B\n",
       "0   bar    one  1  2\n",
       "1   bar    two  3  4\n",
       "2   bar  three  5  6\n",
       "3   baz    one  1  2\n",
       "4   baz    two  3  4\n",
       "5   baz  three  5  6\n",
       "6   foo    one  1  2\n",
       "7   foo    two  3  4\n",
       "8   foo  three  5  6"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1) Create a pivot table:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `pivot()` function:  \n",
    "http://pandas.pydata.org/pandas-docs/stable/reshaping.html#reshaping-by-pivoting-dataframe-objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>second</th>\n",
       "      <th>one</th>\n",
       "      <th>three</th>\n",
       "      <th>two</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>first</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bar</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>baz</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foo</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "second  one  three  two\n",
       "first                  \n",
       "bar       1      5    3\n",
       "baz       1      5    3\n",
       "foo       1      5    3"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.pivot(index='first', columns='second', values='A')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `pd.pivot_table()` function:  \n",
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.pivot_table.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">A</th>\n",
       "      <th colspan=\"3\" halign=\"left\">B</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>second</th>\n",
       "      <th>one</th>\n",
       "      <th>three</th>\n",
       "      <th>two</th>\n",
       "      <th>one</th>\n",
       "      <th>three</th>\n",
       "      <th>two</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>first</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bar</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>baz</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foo</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         A             B          \n",
       "second one three two one three two\n",
       "first                             \n",
       "bar      1     5   3   2     6   4\n",
       "baz      1     5   3   2     6   4\n",
       "foo      1     5   3   2     6   4"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.pivot_table(df, values=['A', 'B'], index='first', columns='second')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note 1:* the above illustrates that Pandas essentially has two indexes: the usual 'row index' but also a 'column index'  \n",
    "*Note 2:* pandas also has an \"unpivot\" function called `pandas.melt` (https://pandas.pydata.org/pandas-docs/stable/generated/pandas.melt.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2: Stack and Unstack"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Stack` and `Unstack` are higher level operators to reshape a dataframe based on a multi-level index."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the documentation:  \n",
    ">stack: “pivot” a level of the (possibly hierarchical) column labels, returning a DataFrame with an index with a new inner-most level of row labels.  \n",
    "unstack: inverse operation from stack: “pivot” a level of the (possibly hierarchical) row index to the column axis, producing a reshaped DataFrame with a new inner-most level of column labels.\n",
    "\n",
    "http://pandas.pydata.org/pandas-docs/stable/reshaping.html#reshaping-by-stacking-and-unstacking\n",
    "\n",
    "In other words:  \n",
    "**Stack** --> move the data \"down\"  \n",
    "**Unstack** --> move the data \"up\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">A</th>\n",
       "      <th colspan=\"3\" halign=\"left\">B</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>second</th>\n",
       "      <th>one</th>\n",
       "      <th>three</th>\n",
       "      <th>two</th>\n",
       "      <th>one</th>\n",
       "      <th>three</th>\n",
       "      <th>two</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>first</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bar</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>baz</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foo</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         A             B          \n",
       "second one three two one three two\n",
       "first                             \n",
       "bar      1     5   3   2     6   4\n",
       "baz      1     5   3   2     6   4\n",
       "foo      1     5   3   2     6   4"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pivot_df = pd.pivot_table(df, values=['A', 'B'], index='first', columns='second')\n",
    "pivot_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>first</th>\n",
       "      <th>second</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">bar</th>\n",
       "      <th>one</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>three</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">baz</th>\n",
       "      <th>one</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>three</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">foo</th>\n",
       "      <th>one</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>three</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              A  B\n",
       "first second      \n",
       "bar   one     1  2\n",
       "      three   5  6\n",
       "      two     3  4\n",
       "baz   one     1  2\n",
       "      three   5  6\n",
       "      two     3  4\n",
       "foo   one     1  2\n",
       "      three   5  6\n",
       "      two     3  4"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pivot_df.stack(level=['second'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note* We could also just use `pivot_df.stack()` as it will by default choose the 'last' level of the index."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unstack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.set_index(['first', 'second'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>first</th>\n",
       "      <th>second</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">bar</th>\n",
       "      <th>one</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>three</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">baz</th>\n",
       "      <th>one</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>three</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">foo</th>\n",
       "      <th>one</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>three</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              A  B\n",
       "first second      \n",
       "bar   one     1  2\n",
       "      two     3  4\n",
       "      three   5  6\n",
       "baz   one     1  2\n",
       "      two     3  4\n",
       "      three   5  6\n",
       "foo   one     1  2\n",
       "      two     3  4\n",
       "      three   5  6"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">A</th>\n",
       "      <th colspan=\"3\" halign=\"left\">B</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>first</th>\n",
       "      <th>bar</th>\n",
       "      <th>baz</th>\n",
       "      <th>foo</th>\n",
       "      <th>bar</th>\n",
       "      <th>baz</th>\n",
       "      <th>foo</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>second</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>one</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>three</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         A           B        \n",
       "first  bar baz foo bar baz foo\n",
       "second                        \n",
       "one      1   1   1   2   2   2\n",
       "three    5   5   5   6   6   6\n",
       "two      3   3   3   4   4   4"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.unstack(level=['first'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">A</th>\n",
       "      <th colspan=\"3\" halign=\"left\">B</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>second</th>\n",
       "      <th>one</th>\n",
       "      <th>three</th>\n",
       "      <th>two</th>\n",
       "      <th>one</th>\n",
       "      <th>three</th>\n",
       "      <th>two</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>first</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bar</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>baz</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foo</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         A             B          \n",
       "second one three two one three two\n",
       "first                             \n",
       "bar      1     5   3   2     6   4\n",
       "baz      1     5   3   2     6   4\n",
       "foo      1     5   3   2     6   4"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.unstack(level=['second'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Dealing with dates</span><a id='dates'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Pandas has a lot of build-in functionality to deal with timeseries data  \n",
    "http://pandas.pydata.org/pandas-docs/stable/timeseries.html\n",
    "\n",
    "A nice overview from the documentation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Class           | Remarks                        | How to create:                               |\n",
    "|-----------------|--------------------------------|----------------------------------------------|\n",
    "| `Timestamp`     | Represents a single time stamp | `to_datetime`, `Timestamp`                   |\n",
    "| `DatetimeIndex` | Index of `Timestamp`           | `to_datetime`, `date_range`, `DatetimeIndex` |\n",
    "| `Period`        | Represents a single time span  | `Period`                                     |\n",
    "| `PeriodIndex`   | Index of `Period`              | `period_range`, `PeriodIndex`                |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a range of dates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatetimeIndex(['2011-01-01', '2011-01-02', '2011-01-03', '2011-01-04',\n",
       "               '2011-01-05'],\n",
       "              dtype='datetime64[ns]', freq='D')"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "date_index = pd.date_range('1/1/2011', periods=len(df_auto.index), freq='D')\n",
    "date_index[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the sake of illustration, let's add these dates to our `df_auto`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df_ad = df_auto.copy()[['make', 'price']]\n",
    "df_ad['date'] = date_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>2011-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>2011-01-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Plym. Sapporo</td>\n",
       "      <td>6486.0</td>\n",
       "      <td>2011-01-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>2011-01-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>2011-01-05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price       date\n",
       "55      Datsun 200  6229.0 2011-01-01\n",
       "47  Pont. Firebird  4934.0 2011-01-02\n",
       "44   Plym. Sapporo  6486.0 2011-01-03\n",
       "23     Ford Fiesta  4389.0 2011-01-04\n",
       "17     Chev. Monza  3667.0 2011-01-05"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ad.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "make             object\n",
       "price           float64\n",
       "date     datetime64[ns]\n",
       "dtype: object"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ad.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Converting a `str` column to a `date` column**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In many cases you import data but it is not recognized as a date column. \n",
    "\n",
    "Let's 'sabotage' our date column and convert it to strings:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('O')"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ad['date'] = df_ad['date'].astype(str)\n",
    "df_ad['date'].dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We cannot perform any `datetime` operations on the column now because it has the wrong datatype!  \n",
    "\n",
    "Luckily we can fix it as such:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('<M8[ns]')"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.to_datetime(df_ad['date']).dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ad['date'] = df_ad['date'].apply(lambda x: pd.to_datetime(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select observation that fall within a certain range"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2011-02-01 00:00:00')"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Timestamp('2011-02-01')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2011-02-01 00:00:00')"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.to_datetime('01-02-2011', format='%d-%m-%Y')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* it is usually a good idea to explicitly include the format, to avoid unexpected behavior"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Olds Delta 88</td>\n",
       "      <td>4890.0</td>\n",
       "      <td>2011-03-08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Pont. Catalina</td>\n",
       "      <td>5798.0</td>\n",
       "      <td>2011-03-09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Buick LeSabre</td>\n",
       "      <td>5788.0</td>\n",
       "      <td>2011-03-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Olds Omega</td>\n",
       "      <td>4181.0</td>\n",
       "      <td>2011-03-11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Buick Century</td>\n",
       "      <td>4816.0</td>\n",
       "      <td>2011-03-12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Olds Cutlass</td>\n",
       "      <td>4733.0</td>\n",
       "      <td>2011-03-13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Dodge St. Regis</td>\n",
       "      <td>6342.0</td>\n",
       "      <td>2011-03-14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>Plym. Volare</td>\n",
       "      <td>4060.0</td>\n",
       "      <td>2011-03-15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>UvT_Car</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2011-03-16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>UvT_Bike</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2011-03-17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     make   price       date\n",
       "37          Olds Delta 88  4890.0 2011-03-08\n",
       "46         Pont. Catalina  5798.0 2011-03-09\n",
       "5           Buick LeSabre  5788.0 2011-03-10\n",
       "38             Olds Omega  4181.0 2011-03-11\n",
       "3           Buick Century  4816.0 2011-03-12\n",
       "36           Olds Cutlass  4733.0 2011-03-13\n",
       "22        Dodge St. Regis  6342.0 2011-03-14\n",
       "45           Plym. Volare  4060.0 2011-03-15\n",
       "UvT_Car               NaN     NaN 2011-03-16\n",
       "UvT_Bike              NaN     NaN 2011-03-17"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ad[df_ad.date > pd.to_datetime('07-03-2011', format='%d-%m-%Y')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the Pandas `.isin()` to use a `date_range` object instead"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Cad. Eldorado</td>\n",
       "      <td>14500.0</td>\n",
       "      <td>2011-02-20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Merc. Cougar</td>\n",
       "      <td>5379.0</td>\n",
       "      <td>2011-02-21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Buick Riviera</td>\n",
       "      <td>10372.0</td>\n",
       "      <td>2011-02-22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Chev. Malibu</td>\n",
       "      <td>4504.0</td>\n",
       "      <td>2011-02-23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>Merc. Zephyr</td>\n",
       "      <td>3291.0</td>\n",
       "      <td>2011-02-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>Olds Toronado</td>\n",
       "      <td>10371.0</td>\n",
       "      <td>2011-02-25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>Pont. Le Mans</td>\n",
       "      <td>4723.0</td>\n",
       "      <td>2011-02-26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Linc. Continental</td>\n",
       "      <td>11497.0</td>\n",
       "      <td>2011-02-27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Merc. Marquis</td>\n",
       "      <td>6165.0</td>\n",
       "      <td>2011-02-28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Dodge Diplomat</td>\n",
       "      <td>4010.0</td>\n",
       "      <td>2011-03-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Dodge Magnum</td>\n",
       "      <td>5886.0</td>\n",
       "      <td>2011-03-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Plym. Horizon</td>\n",
       "      <td>4482.0</td>\n",
       "      <td>2011-03-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Buick Electra</td>\n",
       "      <td>7827.0</td>\n",
       "      <td>2011-03-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Cad. Deville</td>\n",
       "      <td>11385.0</td>\n",
       "      <td>2011-03-05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Chev. Impala</td>\n",
       "      <td>5705.0</td>\n",
       "      <td>2011-03-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Olds 98</td>\n",
       "      <td>8814.0</td>\n",
       "      <td>2011-03-07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Olds Delta 88</td>\n",
       "      <td>4890.0</td>\n",
       "      <td>2011-03-08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Pont. Catalina</td>\n",
       "      <td>5798.0</td>\n",
       "      <td>2011-03-09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Buick LeSabre</td>\n",
       "      <td>5788.0</td>\n",
       "      <td>2011-03-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Olds Omega</td>\n",
       "      <td>4181.0</td>\n",
       "      <td>2011-03-11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 make    price       date\n",
       "11      Cad. Eldorado  14500.0 2011-02-20\n",
       "29       Merc. Cougar   5379.0 2011-02-21\n",
       "8       Buick Riviera  10372.0 2011-02-22\n",
       "15       Chev. Malibu   4504.0 2011-02-23\n",
       "33       Merc. Zephyr   3291.0 2011-02-24\n",
       "40      Olds Toronado  10371.0 2011-02-25\n",
       "49      Pont. Le Mans   4723.0 2011-02-26\n",
       "25  Linc. Continental  11497.0 2011-02-27\n",
       "30      Merc. Marquis   6165.0 2011-02-28\n",
       "20     Dodge Diplomat   4010.0 2011-03-01\n",
       "21       Dodge Magnum   5886.0 2011-03-02\n",
       "43      Plym. Horizon   4482.0 2011-03-03\n",
       "4       Buick Electra   7827.0 2011-03-04\n",
       "10       Cad. Deville  11385.0 2011-03-05\n",
       "14       Chev. Impala   5705.0 2011-03-06\n",
       "34            Olds 98   8814.0 2011-03-07\n",
       "37      Olds Delta 88   4890.0 2011-03-08\n",
       "46     Pont. Catalina   5798.0 2011-03-09\n",
       "5       Buick LeSabre   5788.0 2011-03-10\n",
       "38         Olds Omega   4181.0 2011-03-11"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ad[df_ad.date.isin(pd.date_range('2/20/2011', '3/11/2011', freq='D'))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select components of the dates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can extract, for example: `day`, `month`, `year`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See: http://pandas.pydata.org/pandas-docs/stable/timeseries.html#time-date-components"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>date</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>2011-01-01</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>2011-01-02</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Plym. Sapporo</td>\n",
       "      <td>6486.0</td>\n",
       "      <td>2011-01-03</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>2011-01-04</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>2011-01-05</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price       date  day\n",
       "55      Datsun 200  6229.0 2011-01-01    1\n",
       "47  Pont. Firebird  4934.0 2011-01-02    2\n",
       "44   Plym. Sapporo  6486.0 2011-01-03    3\n",
       "23     Ford Fiesta  4389.0 2011-01-04    4\n",
       "17     Chev. Monza  3667.0 2011-01-05    5"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ad['day'] = df_ad['date'].apply(lambda x: x.day)\n",
    "df_ad.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Manipulate (off-set) the date"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See: http://pandas.pydata.org/pandas-docs/stable/timeseries.html#dateoffset-objects  \n",
    "\n",
    "You can even take into consideration stuff like business days / hours, holidays etc.!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>date</th>\n",
       "      <th>day</th>\n",
       "      <th>new_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>2011-01-01</td>\n",
       "      <td>1</td>\n",
       "      <td>2012-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>2011-01-02</td>\n",
       "      <td>2</td>\n",
       "      <td>2012-01-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Plym. Sapporo</td>\n",
       "      <td>6486.0</td>\n",
       "      <td>2011-01-03</td>\n",
       "      <td>3</td>\n",
       "      <td>2012-01-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>2011-01-04</td>\n",
       "      <td>4</td>\n",
       "      <td>2012-01-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>2011-01-05</td>\n",
       "      <td>5</td>\n",
       "      <td>2012-01-05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price       date  day   new_date\n",
       "55      Datsun 200  6229.0 2011-01-01    1 2012-01-01\n",
       "47  Pont. Firebird  4934.0 2011-01-02    2 2012-01-02\n",
       "44   Plym. Sapporo  6486.0 2011-01-03    3 2012-01-03\n",
       "23     Ford Fiesta  4389.0 2011-01-04    4 2012-01-04\n",
       "17     Chev. Monza  3667.0 2011-01-05    5 2012-01-05"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ad['new_date'] = df_ad.date.apply(lambda x: x + pd.DateOffset(years=1))\n",
    "df_ad.head()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}