ES Module 3 Soln.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# ES Module 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Welcome to Module 3!\n",
    "\n",
    "Last time, we went over:\n",
    "    1. Strings and Intergers\n",
    "    2. Arrays\n",
    "    3. Tables\n",
    "    \n",
    "Today we will continue working with tables, and introduce a new procedure called filtering. Before you start, run the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Loading our libraries, i.e. tool box for our module\n",
    "import numpy as np\n",
    "from datascience import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Paired Programming \n",
    "Today we want to introduce a new system of work called paired programming. Wikipedia defines paired programming in the following way:\n",
    "\n",
    "Pair programming is an agile software development technique in which two programmers work together at one workstation. One, the driver, writes code while the other, the observer or navigator, reviews each line of code as it is typed in. The two programmers switch roles frequently.\n",
    "\n",
    "This methodolgy is quite known in the computer science realm, and we want to try and see how well it would work in our little class room. Hopefully we would all benefit from this, by closing the gap between more experienced coders and less so we could move forward to more advanced topics! Additionally, there is always the benefit of having a friend when all hell breaks loose (or the code just would not work..)\n",
    "\n",
    "So after this brief introduction, please team up with a class-mate, hopefully someone you did not know from before that is at a slightly different level of programming experience."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Please start now, as one takes the controls and the other is reviewing the code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 0. Comments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Comments are ways of making your code more human readable. It's good practice to add comments to your code so someone else reading your code can get an idea of what's going on. \n",
    "\n",
    "You can add a comment to your code by preceeding it with a `#` symbol. When the computer sees any line preceeded by a `#` symbol, it'll ignore it. Here's an example below: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Calculating the total number of pets in my house.\n",
    "num_cats = 4\n",
    "num_dogs = 10\n",
    "total = num_cats + num_dogs\n",
    "total"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now, write a comment in the cell below explaining what it is doing, then run the cell to see if you're correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "    <thead>\n",
       "        <tr>\n",
       "            <th>Animal</th> <th>Number of Legs</th>\n",
       "        </tr>\n",
       "    </thead>\n",
       "    <tbody>\n",
       "        <tr>\n",
       "            <td>Cat   </td> <td>4             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>Dog   </td> <td>4             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>Bird  </td> <td>2             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>Spider</td> <td>8             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "Animal | Number of Legs\n",
       "Cat    | 4\n",
       "Dog    | 4\n",
       "Bird   | 2\n",
       "Spider | 8"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "animals = make_array('Cat', 'Dog', 'Bird', 'Spider')\n",
    "num_legs = make_array(4, 4, 2, 8)\n",
    "my_table = Table().with_columns('Animal', animals,\n",
    "                               'Number of Legs', num_legs)\n",
    "my_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 1. Tables (Continued) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "It is time to practice tables again. We want to load the table files you have uploaded last module. This time, you do it by yourself. Load the table \"inmates_by_year.csv\" and \"correctional_population.csv\" and assign it to a variable. Remember, to load a table we use `Table.read_table()` and pass the name of the table as an argument to the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "inmates_by_year = Table.read_table('inmates_by_year.csv')\n",
    "correctional_population = Table.read_table('correctional_population.csv')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Good job! Now we have all the tables loaded. \n",
    "\n",
    "It is time to extract some information from these tables!\n",
    "In the next several cells, we would guide you through a quick manipulation that will allow us to extract information about the entire correctional population using both tables we have loaded above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In the correctional_population table, we are given data about the number of supervised per 100,000 U.S. adult residents. That means that if we want to have the approximated number of the entire population under supervision we need to multiply by 100,000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['3,060', '3,160', '3,190', '3,210', '3,160', '3,100', '3,000',\n",
       "       '2,930', '2,880', '2,830', '2,780', '2,710'], \n",
       "      dtype='<U5')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# First, extract the column name \"Number supervised per 100,000 U.S. adult residents/c\" from \n",
    "# the correctional_population table and assign it to the variable provided.\n",
    "c_p = correctional_population.column('Number supervised per 100,000 U.S. adult residents/c')\n",
    "c_p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### filtering\n",
    "When you run the cell above, you may notice that the values in our array are actually strings (you can tell because each value has quotation marks around it). However, we can't do mathematical operations on strings, so we'll have to convert this array first so it has integers instead of strings. This is called filtering, or cleaning the data, so we can actually do some work on it. In the following cells, when you see the `# filtering` sign, know that we have yet to cover this topic.\n",
    "\n",
    "Run the following cell to do clean the table. We'll go over how to do this in a later section of this module. If you have any questions about how it works, feel free to ask any of us!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# filtering\n",
    "def string_to_int(val):\n",
    "    return int(val.replace(',', ''))\n",
    "\n",
    "c_p = correctional_population.apply(string_to_int, 'Number supervised per 100,000 U.S. adult residents/c')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now, let's continue finding the real value of c_p. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([306000000, 316000000, 319000000, 321000000, 316000000, 310000000,\n",
       "       300000000, 293000000, 288000000, 283000000, 278000000, 271000000])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# In this cell, multiply the correctional population column name \"Number supervised per 100,000 U.S. adult residents/c\" \n",
    "# by 100000 and assign it to a new variable (c_p stands for correctional population)\n",
    "real_c_p = c_p * 100000\n",
    "real_c_p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Next we want to assign the Total column from inmates_by_year to a variable in order to be able to operate on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['747,500', '765,800', '780,200', '785,500', '767,400', '748,700',\n",
       "       '735,600', '744,500', '731,200', '744,600', '728,200'], \n",
       "      dtype='<U7')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "total_inmates = inmates_by_year.column('Total')\n",
    "total_inmates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Again, run the following line to convert the values in `total_inmates` to ints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([747500, 765800, 780200, 785500, 767400, 748700, 735600, 744500,\n",
       "       731200, 744600, 728200])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# filtering \n",
    "total_inmates = inmates_by_year.apply(string_to_int, 'Total')\n",
    "total_inmates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Switch position, the navigator now takes the wheel."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now that we have the variables holding all the information we want to manipulate, we can start digging into it.\n",
    "\n",
    "We want to come up with a scheme that will allow us to see the precentage of people that are incarcerated, from the total supervised population, by year.\n",
    "\n",
    "Before we do that, though, examine your two variables, `total_inmates` and `real_c_p` and their corresponding tables. Do you foresee any issues with directly comparing these two tables? \n",
    "\n",
    "The `correctional_population` table has a row corresponding to 2000, which `inmates_by_year` does not have. This not only means that the data from our two tables doesn't match up, but also that our arrays are two different lengths. Recall that we cannot do operations on arrays with different lengths. \n",
    "\n",
    "To fix this, run the following cell, in which we get rid of the value corresponding to the year 2000 from `real_c_p`. Again, if you have questions about how this works, feel free to ask us! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([316000000, 319000000, 321000000, 316000000, 310000000, 300000000,\n",
       "       293000000, 288000000, 283000000, 278000000, 271000000])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# filtering\n",
    "real_c_p = real_c_p.take(np.arange(1, real_c_p.size))\n",
    "real_c_p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now our arrays both correspond to data from the same years and we can do operations with both of them!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.23655063,  0.2400627 ,  0.24305296,  0.24857595,  0.24754839,\n",
       "        0.24956667,  0.25105802,  0.25850694,  0.25837456,  0.26784173,\n",
       "        0.26870849])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Write a short code that stores the precentage of people incarcerated from the supervised population \n",
    "# (rel stands for relative, c_p stands from correctional population)\n",
    "inmates_rel_c_p = (total_inmates / real_c_p) * 100\n",
    "inmates_rel_c_p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now, this actually gives us useful information!\n",
    "Why not write it down? Please write down what this information tells you about the judicial infrastructure - we are looking for more mathy/dry explanation (rather than observation of how poorly it is). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# A simple sentence will suffice, we want to see intuitive understanding. Please call a teacher when done to check!\n",
    "extract_information_shows = \"The percentage of people, supervisied by US adult correctional system, who are incarcerated\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "For a final touch, please sort inmates_rel_c_p by descending order in the next cell. We won't tell you how to sort, this time please check the last lab module on how to sort a table. It is an important quality of a programmer to be able to reuse code you already have.  \n",
    "\n",
    "Hint: Remember that you can only use `sort` on tables. How might you manipulate your array so that you can sort it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "    <thead>\n",
       "        <tr>\n",
       "            <th>Inmate_percentage</th>\n",
       "        </tr>\n",
       "    </thead>\n",
       "    <tbody>\n",
       "        <tr>\n",
       "            <td>0.236551         </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>0.240063         </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>0.243053         </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>0.248576         </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>0.247548         </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>0.249567         </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>0.251058         </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>0.258507         </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>0.258375         </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>0.267842         </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "</table>\n",
       "<p>... (1 rows omitted)</p"
      ],
      "text/plain": [
       "Inmate_percentage\n",
       "0.236551\n",
       "0.240063\n",
       "0.243053\n",
       "0.248576\n",
       "0.247548\n",
       "0.249567\n",
       "0.251058\n",
       "0.258507\n",
       "0.258375\n",
       "0.267842\n",
       "... (1 rows omitted)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Please sort inmates_rel_c_p in descending order and print it out\n",
    "inmates_rel_c_p = Table().with_columns('Inmate_percentage', inmates_rel_c_p)\n",
    "inmates_rel_c_p.sort('Inmate_percentage',descending = True)\n",
    "inmates_rel_c_p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Before starting, please switch positions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Filtering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Right now, we can't really get much extra information from our tables other than by sorting them. In this section, we'll learn how to filter our data so we can get more useful insights from it. This is especially useful when dealing with larger data sets!\n",
    "\n",
    "For example, say we wanted insights about the total number of inmates after 2012. We can find this out using the `where` function. Check out the cell below for an example of how to use this. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "    <thead>\n",
       "        <tr>\n",
       "            <th>Year</th> <th>Total</th> <th>Standard error/a</th>\n",
       "        </tr>\n",
       "    </thead>\n",
       "    <tbody>\n",
       "        <tr>\n",
       "            <td>2013</td> <td>731,200</td> <td>8,040           </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2014</td> <td>744,600</td> <td>8,380           </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2015</td> <td>728,200</td> <td>7,380           </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "Year | Total   | Standard error/a\n",
       "2013 | 731,200 | 8,040\n",
       "2014 | 744,600 | 8,380\n",
       "2015 | 728,200 | 7,380"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inmates_by_year.where('Year', are.above(2012))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Notice that `where` takes in two arguments: the name of the column, and the condition we are filtering by. \n",
    "\n",
    "Now, try it for yourself! In the cell below, filter `correctional_population` so it only includes years after 2008. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "    <thead>\n",
       "        <tr>\n",
       "            <th>Year</th> <th>Number supervised per 100,000 U.S. adult residents/c</th> <th>U.S. adult residents under correctional supervision </th>\n",
       "        </tr>\n",
       "    </thead>\n",
       "    <tbody>\n",
       "        <tr>\n",
       "            <td>2009</td> <td>3,100                                               </td> <td>1 in 32                                             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2010</td> <td>3,000                                               </td> <td>1 in 33                                             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2011</td> <td>2,930                                               </td> <td>1 in 34                                             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2012</td> <td>2,880                                               </td> <td>1 in 35                                             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2013</td> <td>2,830                                               </td> <td>1 in 35                                             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2014</td> <td>2,780                                               </td> <td>1 in 36                                             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2015</td> <td>2,710                                               </td> <td>1 in 37                                             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "Year | Number supervised per 100,000 U.S. adult residents/c | U.S. adult residents under correctional supervision\n",
       "2009 | 3,100                                                | 1 in 32\n",
       "2010 | 3,000                                                | 1 in 33\n",
       "2011 | 2,930                                                | 1 in 34\n",
       "2012 | 2,880                                                | 1 in 35\n",
       "2013 | 2,830                                                | 1 in 35\n",
       "2014 | 2,780                                                | 1 in 36\n",
       "2015 | 2,710                                                | 1 in 37"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "If you run the following cell, you'll find a complete description of all such conditions (which we'll call predicates) that you can pass into where. This information can also be found [here](https://www.inferentialthinking.com/chapters/05/2/selecting-rows.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "    <thead>\n",
       "        <tr>\n",
       "            <th>Predicate</th> <th>Description</th>\n",
       "        </tr>\n",
       "    </thead>\n",
       "    <tbody>\n",
       "        <tr>\n",
       "            <td>are.equal_to(Z)              </td> <td>Equal to Z                                             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>are.above(x)                 </td> <td>Greater than x                                         </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>are.above_or_equal_to(x)     </td> <td>Greater than or equal to x                             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>are.below(x)                 </td> <td>Below x                                                </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>are.below_or_equal_to(x)     </td> <td>Less than or equal to x                                </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>are.between(x, y)            </td> <td>Greater than or equal to x, and less than y            </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>are.strictly_between(x, y)   </td> <td>Greater than x and less than y                         </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>are.between_or_equal_to(x, y)</td> <td>Greater than or equal to x, and less than or equal to y</td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>are.containing(S)            </td> <td>Contains the string S                                  </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "Predicate                     | Description\n",
       "are.equal_to(Z)               | Equal to Z\n",
       "are.above(x)                  | Greater than x\n",
       "are.above_or_equal_to(x)      | Greater than or equal to x\n",
       "are.below(x)                  | Below x\n",
       "are.below_or_equal_to(x)      | Less than or equal to x\n",
       "are.between(x, y)             | Greater than or equal to x, and less than y\n",
       "are.strictly_between(x, y)    | Greater than x and less than y\n",
       "are.between_or_equal_to(x, y) | Greater than or equal to x, and less than or equal to y\n",
       "are.containing(S)             | Contains the string S"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "functions = make_array('are.equal_to(Z)', 'are.above(x)', 'are.above_or_equal_to(x)', 'are.below(x)', \n",
    "                       'are.below_or_equal_to(x)', 'are.between(x, y)', 'are.strictly_between(x, y)',\n",
    "                      'are.between_or_equal_to(x, y)', 'are.containing(S)')\n",
    "descriptions = make_array('Equal to Z', 'Greater than x', 'Greater than or equal to x', 'Below x', \n",
    "                         'Less than or equal to x', 'Greater than or equal to x, and less than y',\n",
    "                         'Greater than x and less than y', 'Greater than or equal to x, and less than or equal to y',\n",
    "                         'Contains the string S')\n",
    "predicates = Table().with_columns('Predicate', functions,\n",
    "                                 'Description', descriptions)\n",
    "predicates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now, we'll be using filtering to gain more insights about our two tables. Before we start, be sure to run the following cell so we can ensure every column we're working with is numerical."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "inmates_by_year = inmates_by_year.drop('Total').with_column('Total', total_inmates).select('Year', 'Total', 'Standard error/a')\n",
    "correctional_population = correctional_population.drop('Number supervised per 100,000 U.S. adult residents/c').with_column('Number supervised per 100,000 U.S. adult residents/c', c_p).select('Year', 'Number supervised per 100,000 U.S. adult residents/c', 'U.S. adult residents under correctional supervision ').relabel('U.S. adult residents under correctional supervision ', 'U.S. adult residents under correctional supervision')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "First, find the mean of the total number of inmates. Hint: You can use the `np.mean()` function on arrays to calculate this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "752654.54545454541"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "avg_inmates = np.mean(inmates_by_year.column('Total'))\n",
    "avg_inmates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now, filter `inmates_by_year` to find data for the years in which the number of total inmates was under the average."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "    <thead>\n",
       "        <tr>\n",
       "            <th>Year</th> <th>Total</th> <th>Standard error/a</th>\n",
       "        </tr>\n",
       "    </thead>\n",
       "    <tbody>\n",
       "        <tr>\n",
       "            <td>2005</td> <td>747500</td> <td> ~              </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2010</td> <td>748700</td> <td>5,430           </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2011</td> <td>735600</td> <td>6,010           </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2012</td> <td>744500</td> <td>7,680           </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2013</td> <td>731200</td> <td>8,040           </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2014</td> <td>744600</td> <td>8,380           </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2015</td> <td>728200</td> <td>7,380           </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "Year | Total  | Standard error/a\n",
       "2005 | 747500 |  ~\n",
       "2010 | 748700 | 5,430\n",
       "2011 | 735600 | 6,010\n",
       "2012 | 744500 | 7,680\n",
       "2013 | 731200 | 8,040\n",
       "2014 | 744600 | 8,380\n",
       "2015 | 728200 | 7,380"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filtered_inmates = inmates_by_year.where('Total', are.below(avg_inmates))\n",
    "filtered_inmates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "What does this tell you about the total inmate population? Write your answer in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "answer = \"YOUR TEXT HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Before continuing, please switch positions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now, similarly, find the average number of adults under correctional supervision, and filter the table to find the years in which the number of adults under correctional supervision was under the average."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "    <thead>\n",
       "        <tr>\n",
       "            <th>Year</th> <th>Number supervised per 100,000 U.S. adult residents/c</th> <th>U.S. adult residents under correctional supervision </th>\n",
       "        </tr>\n",
       "    </thead>\n",
       "    <tbody>\n",
       "        <tr>\n",
       "            <td>2010</td> <td>3000                                                </td> <td>1 in 33                                             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2011</td> <td>2930                                                </td> <td>1 in 34                                             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2012</td> <td>2880                                                </td> <td>1 in 35                                             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2013</td> <td>2830                                                </td> <td>1 in 35                                             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2014</td> <td>2780                                                </td> <td>1 in 36                                             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2015</td> <td>2710                                                </td> <td>1 in 37                                             </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "Year | Number supervised per 100,000 U.S. adult residents/c | U.S. adult residents under correctional supervision\n",
       "2010 | 3000                                                 | 1 in 33\n",
       "2011 | 2930                                                 | 1 in 34\n",
       "2012 | 2880                                                 | 1 in 35\n",
       "2013 | 2830                                                 | 1 in 35\n",
       "2014 | 2780                                                 | 1 in 36\n",
       "2015 | 2710                                                 | 1 in 37"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "avg = np.mean(correctional_population.column('Number supervised per 100,000 U.S. adult residents/c'))\n",
    "filtered_c_p = correctional_population.where('Number supervised per 100,000 U.S. adult residents/c', are.below(avg))\n",
    "filtered_c_p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Do the years match up? Does this make sense based on the proportions you calculated above in `inmates_rel_c_p`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "answer = \"YOUR TEXT HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now, from `correctional_population`, filter the table so the value of U.S. adult residents under correctional supervision is 1 in 31. Remember, the values in this column are strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "    <thead>\n",
       "        <tr>\n",
       "            <th>Year</th> <th>Number supervised per 100,000 U.S. adult residents/c</th> <th>U.S. adult residents under correctional supervision</th>\n",
       "        </tr>\n",
       "    </thead>\n",
       "    <tbody>\n",
       "        <tr>\n",
       "            <td>2006</td> <td>3190                                                </td> <td>1 in 31                                            </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "        <tr>\n",
       "            <td>2007</td> <td>3210                                                </td> <td>1 in 31                                            </td>\n",
       "        </tr>\n",
       "    </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "Year | Number supervised per 100,000 U.S. adult residents/c | U.S. adult residents under correctional supervision\n",
       "2006 | 3190                                                 | 1 in 31\n",
       "2007 | 3210                                                 | 1 in 31"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c_p_1_in_34 = correctional_population.where('U.S. adult residents under correctional supervision', are.containing('1 in 31'))\n",
    "c_p_1_in_34"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now, we have one last challenge exercise. Before doing this, finish the challenge exercises from last module. We highly encourage you to work with your partner on this one.\n",
    "\n",
    "In the following cell, find the year with the max number of supervised adults for which the proportion of US adult residents under correctional supervision was 1 in 32. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2008"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "one_in_32 = correctional_population.where('U.S. adult residents under correctional supervision', are.containing('1 in 32'))\n",
    "one_in_32_sorted = one_in_32.sort('Number supervised per 100,000 U.S. adult residents/c', descending = True)\n",
    "year = one_in_32_sorted.column('Year').item(0)\n",
    "year"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Congratulations, you're done with this module! Before you leave, please fill out this [link](https://docs.google.com/a/berkeley.edu/forms/d/1KQHzw-rh_E--lnQ7ItLrOcH7WJUTexDiKMLcuPwClzo/edit?usp=drive_web) to give us feedback on how we can make these modules more useful for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}