Assignment 3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "_You are currently looking at **version 1.5** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-data-analysis/resources/0dhYG) course resource._\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 3 - More Pandas\n",
    "This assignment requires more individual learning then the last one did - you are encouraged to check out the [pandas documentation](http://pandas.pydata.org/pandas-docs/stable/) to find functions or methods you might not have used yet, or ask questions on [Stack Overflow](http://stackoverflow.com/) and tag them as pandas and python related. And of course, the discussion forums are open for interaction with your peers and the course staff."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1 (20%)\n",
    "Load the energy data from the file `Energy Indicators.xls`, which is a list of indicators of [energy supply and renewable electricity production](Energy%20Indicators.xls) from the [United Nations](http://unstats.un.org/unsd/environment/excel_file_tables/2013/Energy%20Indicators.xls) for the year 2013, and should be put into a DataFrame with the variable name of **energy**.\n",
    "\n",
    "Keep in mind that this is an Excel file, and not a comma separated values file. Also, make sure to exclude the footer and header information from the datafile. The first two columns are unneccessary, so you should get rid of them, and you should change the column labels so that the columns are:\n",
    "\n",
    "`['Country', 'Energy Supply', 'Energy Supply per Capita', '% Renewable']`\n",
    "\n",
    "Convert `Energy Supply` to gigajoules (there are 1,000,000 gigajoules in a petajoule). For all countries which have missing data (e.g. data with \"...\") make sure this is reflected as `np.NaN` values.\n",
    "\n",
    "Rename the following list of countries (for use in later questions):\n",
    "\n",
    "```\"Republic of Korea\": \"South Korea\",\n",
    "\"United States of America\": \"United States\",\n",
    "\"United Kingdom of Great Britain and Northern Ireland\": \"United Kingdom\",\n",
    "\"China, Hong Kong Special Administrative Region\": \"Hong Kong\"```\n",
    "\n",
    "There are also several countries with numbers and/or parenthesis in their name. Be sure to remove these, \n",
    "\n",
    "e.g. \n",
    "\n",
    "`'Bolivia (Plurinational State of)'` should be `'Bolivia'`, \n",
    "\n",
    "`'Switzerland17'` should be `'Switzerland'`.\n",
    "\n",
    "<br>\n",
    "\n",
    "Next, load the GDP data from the file `world_bank.csv`, which is a csv containing countries' GDP from 1960 to 2015 from [World Bank](http://data.worldbank.org/indicator/NY.GDP.MKTP.CD). Call this DataFrame **GDP**. \n",
    "\n",
    "Make sure to skip the header, and rename the following list of countries:\n",
    "\n",
    "```\"Korea, Rep.\": \"South Korea\", \n",
    "\"Iran, Islamic Rep.\": \"Iran\",\n",
    "\"Hong Kong SAR, China\": \"Hong Kong\"```\n",
    "\n",
    "<br>\n",
    "\n",
    "Finally, load the [Sciamgo Journal and Country Rank data for Energy Engineering and Power Technology](http://www.scimagojr.com/countryrank.php?category=2102) from the file `scimagojr-3.xlsx`, which ranks countries based on their journal contributions in the aforementioned area. Call this DataFrame **ScimEn**.\n",
    "\n",
    "Join the three datasets: GDP, Energy, and ScimEn into a new dataset (using the intersection of country names). Use only the last 10 years (2006-2015) of GDP data and only the top 15 countries by Scimagojr 'Rank' (Rank 1 through 15). \n",
    "\n",
    "The index of this DataFrame should be the name of the country, and the columns should be ['Rank', 'Documents', 'Citable documents', 'Citations', 'Self-citations',\n",
    "       'Citations per document', 'H index', 'Energy Supply',\n",
    "       'Energy Supply per Capita', '% Renewable', '2006', '2007', '2008',\n",
    "       '2009', '2010', '2011', '2012', '2013', '2014', '2015'].\n",
    "\n",
    "*This function should return a DataFrame with 20 columns and 15 entries.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "umich_part_id": "009",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Rank</th>\n",
       "      <th>Documents</th>\n",
       "      <th>Citable documents</th>\n",
       "      <th>Citations</th>\n",
       "      <th>Self-citations</th>\n",
       "      <th>Citations per document</th>\n",
       "      <th>H index</th>\n",
       "      <th>Energy Supply</th>\n",
       "      <th>Energy Supply per Capita</th>\n",
       "      <th>% Renewable</th>\n",
       "      <th>2006</th>\n",
       "      <th>2007</th>\n",
       "      <th>2008</th>\n",
       "      <th>2009</th>\n",
       "      <th>2010</th>\n",
       "      <th>2011</th>\n",
       "      <th>2012</th>\n",
       "      <th>2013</th>\n",
       "      <th>2014</th>\n",
       "      <th>2015</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Country</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>China</th>\n",
       "      <td>1.0</td>\n",
       "      <td>127050.0</td>\n",
       "      <td>126767.0</td>\n",
       "      <td>597237.0</td>\n",
       "      <td>411683.0</td>\n",
       "      <td>4.70</td>\n",
       "      <td>138.0</td>\n",
       "      <td>127191000000</td>\n",
       "      <td>93.0</td>\n",
       "      <td>19.754910</td>\n",
       "      <td>3.992331e+12</td>\n",
       "      <td>4.559041e+12</td>\n",
       "      <td>4.997775e+12</td>\n",
       "      <td>5.459247e+12</td>\n",
       "      <td>6.039659e+12</td>\n",
       "      <td>6.612490e+12</td>\n",
       "      <td>7.124978e+12</td>\n",
       "      <td>7.672448e+12</td>\n",
       "      <td>8.230121e+12</td>\n",
       "      <td>8.797999e+12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>United States</th>\n",
       "      <td>2.0</td>\n",
       "      <td>96661.0</td>\n",
       "      <td>94747.0</td>\n",
       "      <td>792274.0</td>\n",
       "      <td>265436.0</td>\n",
       "      <td>8.20</td>\n",
       "      <td>230.0</td>\n",
       "      <td>90838000000</td>\n",
       "      <td>286.0</td>\n",
       "      <td>11.570980</td>\n",
       "      <td>1.479230e+13</td>\n",
       "      <td>1.505540e+13</td>\n",
       "      <td>1.501149e+13</td>\n",
       "      <td>1.459484e+13</td>\n",
       "      <td>1.496437e+13</td>\n",
       "      <td>1.520402e+13</td>\n",
       "      <td>1.554216e+13</td>\n",
       "      <td>1.577367e+13</td>\n",
       "      <td>1.615662e+13</td>\n",
       "      <td>1.654857e+13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Japan</th>\n",
       "      <td>3.0</td>\n",
       "      <td>30504.0</td>\n",
       "      <td>30287.0</td>\n",
       "      <td>223024.0</td>\n",
       "      <td>61554.0</td>\n",
       "      <td>7.31</td>\n",
       "      <td>134.0</td>\n",
       "      <td>18984000000</td>\n",
       "      <td>149.0</td>\n",
       "      <td>10.232820</td>\n",
       "      <td>5.496542e+12</td>\n",
       "      <td>5.617036e+12</td>\n",
       "      <td>5.558527e+12</td>\n",
       "      <td>5.251308e+12</td>\n",
       "      <td>5.498718e+12</td>\n",
       "      <td>5.473738e+12</td>\n",
       "      <td>5.569102e+12</td>\n",
       "      <td>5.644659e+12</td>\n",
       "      <td>5.642884e+12</td>\n",
       "      <td>5.669563e+12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>United Kingdom</th>\n",
       "      <td>4.0</td>\n",
       "      <td>20944.0</td>\n",
       "      <td>20357.0</td>\n",
       "      <td>206091.0</td>\n",
       "      <td>37874.0</td>\n",
       "      <td>9.84</td>\n",
       "      <td>139.0</td>\n",
       "      <td>7920000000</td>\n",
       "      <td>124.0</td>\n",
       "      <td>10.600470</td>\n",
       "      <td>2.419631e+12</td>\n",
       "      <td>2.482203e+12</td>\n",
       "      <td>2.470614e+12</td>\n",
       "      <td>2.367048e+12</td>\n",
       "      <td>2.403504e+12</td>\n",
       "      <td>2.450911e+12</td>\n",
       "      <td>2.479809e+12</td>\n",
       "      <td>2.533370e+12</td>\n",
       "      <td>2.605643e+12</td>\n",
       "      <td>2.666333e+12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Russian Federation</th>\n",
       "      <td>5.0</td>\n",
       "      <td>18534.0</td>\n",
       "      <td>18301.0</td>\n",
       "      <td>34266.0</td>\n",
       "      <td>12422.0</td>\n",
       "      <td>1.85</td>\n",
       "      <td>57.0</td>\n",
       "      <td>30709000000</td>\n",
       "      <td>214.0</td>\n",
       "      <td>17.288680</td>\n",
       "      <td>1.385793e+12</td>\n",
       "      <td>1.504071e+12</td>\n",
       "      <td>1.583004e+12</td>\n",
       "      <td>1.459199e+12</td>\n",
       "      <td>1.524917e+12</td>\n",
       "      <td>1.589943e+12</td>\n",
       "      <td>1.645876e+12</td>\n",
       "      <td>1.666934e+12</td>\n",
       "      <td>1.678709e+12</td>\n",
       "      <td>1.616149e+12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Canada</th>\n",
       "      <td>6.0</td>\n",
       "      <td>17899.0</td>\n",
       "      <td>17620.0</td>\n",
       "      <td>215003.0</td>\n",
       "      <td>40930.0</td>\n",
       "      <td>12.01</td>\n",
       "      <td>149.0</td>\n",
       "      <td>10431000000</td>\n",
       "      <td>296.0</td>\n",
       "      <td>61.945430</td>\n",
       "      <td>1.564469e+12</td>\n",
       "      <td>1.596740e+12</td>\n",
       "      <td>1.612713e+12</td>\n",
       "      <td>1.565145e+12</td>\n",
       "      <td>1.613406e+12</td>\n",
       "      <td>1.664087e+12</td>\n",
       "      <td>1.693133e+12</td>\n",
       "      <td>1.730688e+12</td>\n",
       "      <td>1.773486e+12</td>\n",
       "      <td>1.792609e+12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Germany</th>\n",
       "      <td>7.0</td>\n",
       "      <td>17027.0</td>\n",
       "      <td>16831.0</td>\n",
       "      <td>140566.0</td>\n",
       "      <td>27426.0</td>\n",
       "      <td>8.26</td>\n",
       "      <td>126.0</td>\n",
       "      <td>13261000000</td>\n",
       "      <td>165.0</td>\n",
       "      <td>17.901530</td>\n",
       "      <td>3.332891e+12</td>\n",
       "      <td>3.441561e+12</td>\n",
       "      <td>3.478809e+12</td>\n",
       "      <td>3.283340e+12</td>\n",
       "      <td>3.417298e+12</td>\n",
       "      <td>3.542371e+12</td>\n",
       "      <td>3.556724e+12</td>\n",
       "      <td>3.567317e+12</td>\n",
       "      <td>3.624386e+12</td>\n",
       "      <td>3.685556e+12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>India</th>\n",
       "      <td>8.0</td>\n",
       "      <td>15005.0</td>\n",
       "      <td>14841.0</td>\n",
       "      <td>128763.0</td>\n",
       "      <td>37209.0</td>\n",
       "      <td>8.58</td>\n",
       "      <td>115.0</td>\n",
       "      <td>33195000000</td>\n",
       "      <td>26.0</td>\n",
       "      <td>14.969080</td>\n",
       "      <td>1.265894e+12</td>\n",
       "      <td>1.374865e+12</td>\n",
       "      <td>1.428361e+12</td>\n",
       "      <td>1.549483e+12</td>\n",
       "      <td>1.708459e+12</td>\n",
       "      <td>1.821872e+12</td>\n",
       "      <td>1.924235e+12</td>\n",
       "      <td>2.051982e+12</td>\n",
       "      <td>2.200617e+12</td>\n",
       "      <td>2.367206e+12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>France</th>\n",
       "      <td>9.0</td>\n",
       "      <td>13153.0</td>\n",
       "      <td>12973.0</td>\n",
       "      <td>130632.0</td>\n",
       "      <td>28601.0</td>\n",
       "      <td>9.93</td>\n",
       "      <td>114.0</td>\n",
       "      <td>10597000000</td>\n",
       "      <td>166.0</td>\n",
       "      <td>17.020280</td>\n",
       "      <td>2.607840e+12</td>\n",
       "      <td>2.669424e+12</td>\n",
       "      <td>2.674637e+12</td>\n",
       "      <td>2.595967e+12</td>\n",
       "      <td>2.646995e+12</td>\n",
       "      <td>2.702032e+12</td>\n",
       "      <td>2.706968e+12</td>\n",
       "      <td>2.722567e+12</td>\n",
       "      <td>2.729632e+12</td>\n",
       "      <td>2.761185e+12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>South Korea</th>\n",
       "      <td>10.0</td>\n",
       "      <td>11983.0</td>\n",
       "      <td>11923.0</td>\n",
       "      <td>114675.0</td>\n",
       "      <td>22595.0</td>\n",
       "      <td>9.57</td>\n",
       "      <td>104.0</td>\n",
       "      <td>11007000000</td>\n",
       "      <td>221.0</td>\n",
       "      <td>2.279353</td>\n",
       "      <td>9.410199e+11</td>\n",
       "      <td>9.924316e+11</td>\n",
       "      <td>1.020510e+12</td>\n",
       "      <td>1.027730e+12</td>\n",
       "      <td>1.094499e+12</td>\n",
       "      <td>1.134796e+12</td>\n",
       "      <td>1.160809e+12</td>\n",
       "      <td>1.194429e+12</td>\n",
       "      <td>1.234340e+12</td>\n",
       "      <td>1.266580e+12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Italy</th>\n",
       "      <td>11.0</td>\n",
       "      <td>10964.0</td>\n",
       "      <td>10794.0</td>\n",
       "      <td>111850.0</td>\n",
       "      <td>26661.0</td>\n",
       "      <td>10.20</td>\n",
       "      <td>106.0</td>\n",
       "      <td>6530000000</td>\n",
       "      <td>109.0</td>\n",
       "      <td>33.667230</td>\n",
       "      <td>2.202170e+12</td>\n",
       "      <td>2.234627e+12</td>\n",
       "      <td>2.211154e+12</td>\n",
       "      <td>2.089938e+12</td>\n",
       "      <td>2.125185e+12</td>\n",
       "      <td>2.137439e+12</td>\n",
       "      <td>2.077184e+12</td>\n",
       "      <td>2.040871e+12</td>\n",
       "      <td>2.033868e+12</td>\n",
       "      <td>2.049316e+12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Spain</th>\n",
       "      <td>12.0</td>\n",
       "      <td>9428.0</td>\n",
       "      <td>9330.0</td>\n",
       "      <td>123336.0</td>\n",
       "      <td>23964.0</td>\n",
       "      <td>13.08</td>\n",
       "      <td>115.0</td>\n",
       "      <td>4923000000</td>\n",
       "      <td>106.0</td>\n",
       "      <td>37.968590</td>\n",
       "      <td>1.414823e+12</td>\n",
       "      <td>1.468146e+12</td>\n",
       "      <td>1.484530e+12</td>\n",
       "      <td>1.431475e+12</td>\n",
       "      <td>1.431673e+12</td>\n",
       "      <td>1.417355e+12</td>\n",
       "      <td>1.380216e+12</td>\n",
       "      <td>1.357139e+12</td>\n",
       "      <td>1.375605e+12</td>\n",
       "      <td>1.419821e+12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iran</th>\n",
       "      <td>13.0</td>\n",
       "      <td>8896.0</td>\n",
       "      <td>8819.0</td>\n",
       "      <td>57470.0</td>\n",
       "      <td>19125.0</td>\n",
       "      <td>6.46</td>\n",
       "      <td>72.0</td>\n",
       "      <td>9172000000</td>\n",
       "      <td>119.0</td>\n",
       "      <td>5.707721</td>\n",
       "      <td>3.895523e+11</td>\n",
       "      <td>4.250646e+11</td>\n",
       "      <td>4.289909e+11</td>\n",
       "      <td>4.389208e+11</td>\n",
       "      <td>4.677902e+11</td>\n",
       "      <td>4.853309e+11</td>\n",
       "      <td>4.532569e+11</td>\n",
       "      <td>4.445926e+11</td>\n",
       "      <td>4.639027e+11</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Australia</th>\n",
       "      <td>14.0</td>\n",
       "      <td>8831.0</td>\n",
       "      <td>8725.0</td>\n",
       "      <td>90765.0</td>\n",
       "      <td>15606.0</td>\n",
       "      <td>10.28</td>\n",
       "      <td>107.0</td>\n",
       "      <td>5386000000</td>\n",
       "      <td>231.0</td>\n",
       "      <td>11.810810</td>\n",
       "      <td>1.021939e+12</td>\n",
       "      <td>1.060340e+12</td>\n",
       "      <td>1.099644e+12</td>\n",
       "      <td>1.119654e+12</td>\n",
       "      <td>1.142251e+12</td>\n",
       "      <td>1.169431e+12</td>\n",
       "      <td>1.211913e+12</td>\n",
       "      <td>1.241484e+12</td>\n",
       "      <td>1.272520e+12</td>\n",
       "      <td>1.301251e+12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Brazil</th>\n",
       "      <td>15.0</td>\n",
       "      <td>8668.0</td>\n",
       "      <td>8596.0</td>\n",
       "      <td>60702.0</td>\n",
       "      <td>14396.0</td>\n",
       "      <td>7.00</td>\n",
       "      <td>86.0</td>\n",
       "      <td>12149000000</td>\n",
       "      <td>59.0</td>\n",
       "      <td>69.648030</td>\n",
       "      <td>1.845080e+12</td>\n",
       "      <td>1.957118e+12</td>\n",
       "      <td>2.056809e+12</td>\n",
       "      <td>2.054215e+12</td>\n",
       "      <td>2.208872e+12</td>\n",
       "      <td>2.295245e+12</td>\n",
       "      <td>2.339209e+12</td>\n",
       "      <td>2.409740e+12</td>\n",
       "      <td>2.412231e+12</td>\n",
       "      <td>2.319423e+12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Rank  Documents  Citable documents  Citations  \\\n",
       "Country                                                             \n",
       "China                1.0   127050.0           126767.0   597237.0   \n",
       "United States        2.0    96661.0            94747.0   792274.0   \n",
       "Japan                3.0    30504.0            30287.0   223024.0   \n",
       "United Kingdom       4.0    20944.0            20357.0   206091.0   \n",
       "Russian Federation   5.0    18534.0            18301.0    34266.0   \n",
       "Canada               6.0    17899.0            17620.0   215003.0   \n",
       "Germany              7.0    17027.0            16831.0   140566.0   \n",
       "India                8.0    15005.0            14841.0   128763.0   \n",
       "France               9.0    13153.0            12973.0   130632.0   \n",
       "South Korea         10.0    11983.0            11923.0   114675.0   \n",
       "Italy               11.0    10964.0            10794.0   111850.0   \n",
       "Spain               12.0     9428.0             9330.0   123336.0   \n",
       "Iran                13.0     8896.0             8819.0    57470.0   \n",
       "Australia           14.0     8831.0             8725.0    90765.0   \n",
       "Brazil              15.0     8668.0             8596.0    60702.0   \n",
       "\n",
       "                    Self-citations  Citations per document  H index  \\\n",
       "Country                                                               \n",
       "China                     411683.0                    4.70    138.0   \n",
       "United States             265436.0                    8.20    230.0   \n",
       "Japan                      61554.0                    7.31    134.0   \n",
       "United Kingdom             37874.0                    9.84    139.0   \n",
       "Russian Federation         12422.0                    1.85     57.0   \n",
       "Canada                     40930.0                   12.01    149.0   \n",
       "Germany                    27426.0                    8.26    126.0   \n",
       "India                      37209.0                    8.58    115.0   \n",
       "France                     28601.0                    9.93    114.0   \n",
       "South Korea                22595.0                    9.57    104.0   \n",
       "Italy                      26661.0                   10.20    106.0   \n",
       "Spain                      23964.0                   13.08    115.0   \n",
       "Iran                       19125.0                    6.46     72.0   \n",
       "Australia                  15606.0                   10.28    107.0   \n",
       "Brazil                     14396.0                    7.00     86.0   \n",
       "\n",
       "                   Energy Supply  Energy Supply per Capita  % Renewable  \\\n",
       "Country                                                                   \n",
       "China               127191000000                      93.0    19.754910   \n",
       "United States        90838000000                     286.0    11.570980   \n",
       "Japan                18984000000                     149.0    10.232820   \n",
       "United Kingdom        7920000000                     124.0    10.600470   \n",
       "Russian Federation   30709000000                     214.0    17.288680   \n",
       "Canada               10431000000                     296.0    61.945430   \n",
       "Germany              13261000000                     165.0    17.901530   \n",
       "India                33195000000                      26.0    14.969080   \n",
       "France               10597000000                     166.0    17.020280   \n",
       "South Korea          11007000000                     221.0     2.279353   \n",
       "Italy                 6530000000                     109.0    33.667230   \n",
       "Spain                 4923000000                     106.0    37.968590   \n",
       "Iran                  9172000000                     119.0     5.707721   \n",
       "Australia             5386000000                     231.0    11.810810   \n",
       "Brazil               12149000000                      59.0    69.648030   \n",
       "\n",
       "                            2006          2007          2008          2009  \\\n",
       "Country                                                                      \n",
       "China               3.992331e+12  4.559041e+12  4.997775e+12  5.459247e+12   \n",
       "United States       1.479230e+13  1.505540e+13  1.501149e+13  1.459484e+13   \n",
       "Japan               5.496542e+12  5.617036e+12  5.558527e+12  5.251308e+12   \n",
       "United Kingdom      2.419631e+12  2.482203e+12  2.470614e+12  2.367048e+12   \n",
       "Russian Federation  1.385793e+12  1.504071e+12  1.583004e+12  1.459199e+12   \n",
       "Canada              1.564469e+12  1.596740e+12  1.612713e+12  1.565145e+12   \n",
       "Germany             3.332891e+12  3.441561e+12  3.478809e+12  3.283340e+12   \n",
       "India               1.265894e+12  1.374865e+12  1.428361e+12  1.549483e+12   \n",
       "France              2.607840e+12  2.669424e+12  2.674637e+12  2.595967e+12   \n",
       "South Korea         9.410199e+11  9.924316e+11  1.020510e+12  1.027730e+12   \n",
       "Italy               2.202170e+12  2.234627e+12  2.211154e+12  2.089938e+12   \n",
       "Spain               1.414823e+12  1.468146e+12  1.484530e+12  1.431475e+12   \n",
       "Iran                3.895523e+11  4.250646e+11  4.289909e+11  4.389208e+11   \n",
       "Australia           1.021939e+12  1.060340e+12  1.099644e+12  1.119654e+12   \n",
       "Brazil              1.845080e+12  1.957118e+12  2.056809e+12  2.054215e+12   \n",
       "\n",
       "                            2010          2011          2012          2013  \\\n",
       "Country                                                                      \n",
       "China               6.039659e+12  6.612490e+12  7.124978e+12  7.672448e+12   \n",
       "United States       1.496437e+13  1.520402e+13  1.554216e+13  1.577367e+13   \n",
       "Japan               5.498718e+12  5.473738e+12  5.569102e+12  5.644659e+12   \n",
       "United Kingdom      2.403504e+12  2.450911e+12  2.479809e+12  2.533370e+12   \n",
       "Russian Federation  1.524917e+12  1.589943e+12  1.645876e+12  1.666934e+12   \n",
       "Canada              1.613406e+12  1.664087e+12  1.693133e+12  1.730688e+12   \n",
       "Germany             3.417298e+12  3.542371e+12  3.556724e+12  3.567317e+12   \n",
       "India               1.708459e+12  1.821872e+12  1.924235e+12  2.051982e+12   \n",
       "France              2.646995e+12  2.702032e+12  2.706968e+12  2.722567e+12   \n",
       "South Korea         1.094499e+12  1.134796e+12  1.160809e+12  1.194429e+12   \n",
       "Italy               2.125185e+12  2.137439e+12  2.077184e+12  2.040871e+12   \n",
       "Spain               1.431673e+12  1.417355e+12  1.380216e+12  1.357139e+12   \n",
       "Iran                4.677902e+11  4.853309e+11  4.532569e+11  4.445926e+11   \n",
       "Australia           1.142251e+12  1.169431e+12  1.211913e+12  1.241484e+12   \n",
       "Brazil              2.208872e+12  2.295245e+12  2.339209e+12  2.409740e+12   \n",
       "\n",
       "                            2014          2015  \n",
       "Country                                         \n",
       "China               8.230121e+12  8.797999e+12  \n",
       "United States       1.615662e+13  1.654857e+13  \n",
       "Japan               5.642884e+12  5.669563e+12  \n",
       "United Kingdom      2.605643e+12  2.666333e+12  \n",
       "Russian Federation  1.678709e+12  1.616149e+12  \n",
       "Canada              1.773486e+12  1.792609e+12  \n",
       "Germany             3.624386e+12  3.685556e+12  \n",
       "India               2.200617e+12  2.367206e+12  \n",
       "France              2.729632e+12  2.761185e+12  \n",
       "South Korea         1.234340e+12  1.266580e+12  \n",
       "Italy               2.033868e+12  2.049316e+12  \n",
       "Spain               1.375605e+12  1.419821e+12  \n",
       "Iran                4.639027e+11           NaN  \n",
       "Australia           1.272520e+12  1.301251e+12  \n",
       "Brazil              2.412231e+12  2.319423e+12  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_one():\n",
    "    \n",
    "    energy = pd.read_excel(\"Energy Indicators.xls\",skip_footer=38,skip_header=1,skiprows=17) # Skip header and footer\n",
    "\n",
    "    #energy.head()\n",
    "    #type(energy) #DataFrame\n",
    "\n",
    "    energy.drop(energy.columns[[0,1]],axis=1,inplace=True) # Drop first 2 columns\n",
    "\n",
    "    energy.columns = ['Country', 'Energy Supply', 'Energy Supply per Capita', '% Renewable']\n",
    "    \n",
    "    energy['Energy Supply'] *= 1000000 # Convert from petajoues to gigajoules.\n",
    "\n",
    "    energy.dropna() # Drop rows with NaN values.\n",
    "\n",
    "    energy['Country'] = energy['Country'].str.replace(r'\\(.*\\)', '') # Remove contents within parenthesis.\n",
    "    energy['Country'] = energy['Country'].str.replace('\\d+', '') # Remove digits from names\n",
    "    \n",
    "    # This brings the Iran energy values back! Iran is listed as \"Iran (Islamic Republic of)\" in the energy data! So removing\n",
    "    # the parentheses still leaves a space after Iran (\"Iran \") which is why Iran's energy values don't appear properly.\n",
    "    energy['Country'] = energy['Country'].str.strip()\n",
    "    \n",
    "    # Turn blank values into NaN\n",
    "    for col in energy:\n",
    "        energy[col] = energy[col].replace('...',np.nan)\n",
    "\n",
    "    energy['Country'] = energy['Country'].str.replace('Republic of Korea','South Korea')\n",
    "    energy['Country'] = energy['Country'].str.replace('United States of America','United States')\n",
    "    energy['Country'] = energy['Country'].str.replace('United Kingdom of Great Britain and Northern Ireland','United Kingdom')\n",
    "    energy['Country'] = energy['Country'].str.replace('China, Hong Kong Special Administrative Region','Hong Kong')\n",
    "    \n",
    "\n",
    "    # GDP:\n",
    "\n",
    "    GDP = pd.read_csv('world_bank.csv', skiprows=3) # Skip header\n",
    "\n",
    "    # Make first row the column names\n",
    "    new_header = GDP.iloc[0]\n",
    "    GDP = GDP[1:]\n",
    "    GDP.columns = new_header\n",
    "\n",
    "    #GDP = GDP.rename(index=str,columns = {\"Country Name\":\"Country\"})\n",
    "\n",
    "    GDP['Country Name'] = GDP['Country Name'].str.replace('Korea, Rep.','South Korea')\n",
    "    GDP['Country Name'] = GDP['Country Name'].str.replace('Iran, Islamic Rep.','Iran')\n",
    "    GDP['Country Name'] = GDP['Country Name'].str.replace('Hong Kong SAR, China','Hong Kong')\n",
    "\n",
    "    # Change column name from 'Country Name' to 'Country' for merging 3 files on country name.\n",
    "    names = GDP.columns.tolist()\n",
    "    names[names.index('Country Name')] = 'Country'\n",
    "    GDP.columns = names\n",
    "\n",
    "    # Only keep the columns from 2006-15. Drop column number 1 to 50. Don't need country code, etc.\n",
    "    GDP = GDP.drop(GDP.iloc[:,1:50], axis=1)\n",
    "    \n",
    "    GDP.columns = GDP.columns.astype(str).str.split('.').str[0] # Remove '.0' at the end of the year columns.    \n",
    "    \n",
    "    # SCIMEN:\n",
    "\n",
    "    ScimEn = pd.read_excel('scimagojr-3.xlsx')\n",
    "    #ScimEn = ScimEn[:15] # Only top 15 countries\n",
    "    #ScimEn\n",
    "\n",
    "    # MERGING:\n",
    "\n",
    "    master = pd.merge(ScimEn,energy,how='outer',left_on='Country',right_on='Country')\n",
    "    master1 = pd.merge(master,GDP,how='outer',left_on='Country',right_on='Country')\n",
    "\n",
    "    master1 = master1[:15] # Limit to top 15 entries.\n",
    "\n",
    "    master2 = master1.set_index('Country') # Just making 'Country' the index column.\n",
    "    \n",
    "    return master2\n",
    "\n",
    "answer_one()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2 (6.6%)\n",
    "The previous question joined three datasets then reduced this to just the top 15 entries. When you joined the datasets, but before you reduced this to the top 15 items, how many entries did you lose?\n",
    "\n",
    "*This function should return a single number.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<svg width=\"800\" height=\"300\">\n",
       "  <circle cx=\"150\" cy=\"180\" r=\"80\" fill-opacity=\"0.2\" stroke=\"black\" stroke-width=\"2\" fill=\"blue\" />\n",
       "  <circle cx=\"200\" cy=\"100\" r=\"80\" fill-opacity=\"0.2\" stroke=\"black\" stroke-width=\"2\" fill=\"red\" />\n",
       "  <circle cx=\"100\" cy=\"100\" r=\"80\" fill-opacity=\"0.2\" stroke=\"black\" stroke-width=\"2\" fill=\"green\" />\n",
       "  <line x1=\"150\" y1=\"125\" x2=\"300\" y2=\"150\" stroke=\"black\" stroke-width=\"2\" fill=\"black\" stroke-dasharray=\"5,3\"/>\n",
       "  <text  x=\"300\" y=\"165\" font-family=\"Verdana\" font-size=\"35\">Everything but this!</text>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%HTML\n",
    "<svg width=\"800\" height=\"300\">\n",
    "  <circle cx=\"150\" cy=\"180\" r=\"80\" fill-opacity=\"0.2\" stroke=\"black\" stroke-width=\"2\" fill=\"blue\" />\n",
    "  <circle cx=\"200\" cy=\"100\" r=\"80\" fill-opacity=\"0.2\" stroke=\"black\" stroke-width=\"2\" fill=\"red\" />\n",
    "  <circle cx=\"100\" cy=\"100\" r=\"80\" fill-opacity=\"0.2\" stroke=\"black\" stroke-width=\"2\" fill=\"green\" />\n",
    "  <line x1=\"150\" y1=\"125\" x2=\"300\" y2=\"150\" stroke=\"black\" stroke-width=\"2\" fill=\"black\" stroke-dasharray=\"5,3\"/>\n",
    "  <text  x=\"300\" y=\"165\" font-family=\"Verdana\" font-size=\"35\">Everything but this!</text>\n",
    "</svg>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "umich_part_id": "010",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "156"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_two():\n",
    "    \n",
    "    import pandas as pd\n",
    "    import numpy as np\n",
    "    \n",
    "    energy = pd.read_excel(\"Energy Indicators.xls\",skip_footer=38,skip_header=1,skiprows=17) # Skip header and footer\n",
    "\n",
    "    energy.drop(energy.columns[[0,1]],axis=1,inplace=True) # Drop first 2 columns\n",
    "\n",
    "    energy.columns = ['Country', 'Energy Supply', 'Energy Supply per Capita', '% Renewable']\n",
    "\n",
    "    energy.dropna() # Drop rows with NaN values.\n",
    "\n",
    "    energy['Country'] = energy['Country'].str.replace(r'\\(.*\\)', '') # Remove contents within parenthesis.\n",
    "    energy['Country'] = energy['Country'].str.replace('\\d+', '') # Remove digits from names\n",
    "    \n",
    "    energy['Country'] = energy['Country'].str.strip() # This brings the Iran energy values back!\n",
    "    \n",
    "\n",
    "    # Turn blank values into NaN\n",
    "    for col in energy:\n",
    "        energy[col] = energy[col].replace('...',np.nan)\n",
    "\n",
    "    energy['Country'] = energy['Country'].str.replace('Republic of Korea','South Korea')\n",
    "    energy['Country'] = energy['Country'].str.replace('United States of America','United States')\n",
    "    energy['Country'] = energy['Country'].str.replace('United Kingdom of Great Britain and Northern Ireland','United Kingdom')\n",
    "    energy['Country'] = energy['Country'].str.replace('China, Hong Kong Special Administrative Region','Hong Kong')\n",
    "\n",
    "\n",
    "    # GDP:\n",
    "\n",
    "    GDP = pd.read_csv('world_bank.csv', skiprows=3) # Skip header\n",
    "\n",
    "    # Make first row the column names\n",
    "    new_header = GDP.iloc[0]\n",
    "    GDP = GDP[1:]\n",
    "    GDP.columns = new_header\n",
    "\n",
    "    #GDP = GDP.rename(index=str,columns = {\"Country Name\":\"Country\"})\n",
    "\n",
    "    GDP['Country Name'] = GDP['Country Name'].str.replace('Korea, Rep.','South Korea')\n",
    "    GDP['Country Name'] = GDP['Country Name'].str.replace('Iran, Islamic Rep.','Iran')\n",
    "    GDP['Country Name'] = GDP['Country Name'].str.replace('Hong Kong SAR, China','Hong Kong')\n",
    "\n",
    "    # Change column name from 'Country Name' to 'Country' for merging 3 files on country name.\n",
    "    names = GDP.columns.tolist()\n",
    "    names[names.index('Country Name')] = 'Country'\n",
    "    GDP.columns = names\n",
    "\n",
    "    # Only keep the columns from 2006-15. Drop column number 1 to 50. Don't need country code, etc.\n",
    "    GDP = GDP.drop(GDP.iloc[:,1:50], axis=1)\n",
    "    \n",
    "    GDP.columns = GDP.columns.astype(str).str.split('.').str[0] # Remove '.0' at the end of the year columns.    \n",
    "\n",
    "\n",
    "    # SCIMEN:\n",
    "    ScimEn = pd.read_excel('scimagojr-3.xlsx')\n",
    "    \n",
    "    # LOST ENTRIES = LEN(OUTER JOIN) - LEN(INNER JOIN)\n",
    "    \n",
    "    # Need unique entries in all 3 sets so use concat. Can't do that with a left or right outer join!\n",
    "    num_outer = len(pd.concat([ScimEn['Country'],energy['Country'],GDP['Country']]).unique())\n",
    "    \n",
    "    num_inter = (GDP.merge(energy, left_on='Country', right_on='Country', how='inner').merge(ScimEn, left_on='Country', right_on='Country', how='inner').shape[0])\n",
    "\n",
    "    return num_outer-num_inter\n",
    "\n",
    "answer_two()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Answer the following questions in the context of only the top 15 countries by Scimagojr Rank (aka the DataFrame returned by `answer_one()`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3 (6.6%)\n",
    "What is the average GDP over the last 10 years for each country? (exclude missing values from this calculation.)\n",
    "\n",
    "*This function should return a Series named `avgGDP` with 15 countries and their average GDP sorted in descending order.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true,
    "umich_part_id": "011",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Country\n",
       "United States         1.536434e+13\n",
       "China                 6.348609e+12\n",
       "Japan                 5.542208e+12\n",
       "Germany               3.493025e+12\n",
       "France                2.681725e+12\n",
       "United Kingdom        2.487907e+12\n",
       "Brazil                2.189794e+12\n",
       "Italy                 2.120175e+12\n",
       "India                 1.769297e+12\n",
       "Canada                1.660647e+12\n",
       "Russian Federation    1.565459e+12\n",
       "Spain                 1.418078e+12\n",
       "Australia             1.164043e+12\n",
       "South Korea           1.106715e+12\n",
       "Iran                  4.441558e+11\n",
       "dtype: float64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_three():\n",
    "    \n",
    "    import pandas as pd\n",
    "    Top15 = answer_one()\n",
    "    #Top15.columns = Top15.columns.astype(str) # Turn year column names from numbers to string so you can call it.\n",
    "    \n",
    "    Top15 = Top15.iloc[:,10:] # Cut out all columns except the years.\n",
    "    avgGDP = Top15.mean(axis=1)\n",
    "    return avgGDP.sort_values(ascending=False)\n",
    "\n",
    "answer_three()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4 (6.6%)\n",
    "By how much had the GDP changed over the 10 year span for the country with the 6th largest average GDP?\n",
    "\n",
    "*This function should return a single number.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true,
    "umich_part_id": "012",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "246702696075.3999"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_four():\n",
    "    top10 = answer_three()\n",
    "    top15 = answer_one()\n",
    "    \n",
    "    # Return the country (index) with the 6th largest average GDP. Need to use boolean masking to return the index value.\n",
    "    target_country = top10[top10 == top10[5]].index[0]\n",
    "    \n",
    "    # Change = GDP_2015 - GDP_2006\n",
    "    diff = top15.loc[target_country].iloc[19] - top15.loc[target_country].iloc[10]\n",
    "\n",
    "    return diff\n",
    "answer_four()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 5 (6.6%)\n",
    "What is the mean `Energy Supply per Capita`?\n",
    "\n",
    "*This function should return a single number.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "umich_part_id": "013",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "157.6"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_five():\n",
    "    Top15 = answer_one()\n",
    "    \n",
    "    # Need to return float not numpy 64 float!\n",
    "    return Top15.iloc[:,8].mean()\n",
    "\n",
    "answer_five()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 6 (6.6%)\n",
    "What country has the maximum % Renewable and what is the percentage?\n",
    "\n",
    "*This function should return a tuple with the name of the country and the percentage.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "umich_part_id": "014",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Brazil', 69.648030000000006)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_six():\n",
    "    Top15 = answer_one()\n",
    "    return (Top15.iloc[:,9].idxmax(), Top15.iloc[:,9].max())\n",
    "answer_six()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 7 (6.6%)\n",
    "Create a new column that is the ratio of Self-Citations to Total Citations. \n",
    "What is the maximum value for this new column, and what country has the highest ratio?\n",
    "\n",
    "*This function should return a tuple with the name of the country and the ratio.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "umich_part_id": "015",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('China', 0.68931261793894216)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_seven():\n",
    "    Top15 = answer_one()\n",
    "    Top15['Ratio'] = Top15['Self-citations']/Top15['Citations']\n",
    "    return (Top15['Ratio'].idxmax(), Top15['Ratio'].max())\n",
    "    #return Top15\n",
    "answer_seven()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 8 (6.6%)\n",
    "\n",
    "Create a column that estimates the population using Energy Supply and Energy Supply per capita. \n",
    "What is the third most populous country according to this estimate?\n",
    "\n",
    "*This function should return a single string value.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "umich_part_id": "016",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'United States'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_eight():\n",
    "    Top15 = answer_one()\n",
    "    Top15['Population Estimate'] = Top15['Energy Supply']/Top15['Energy Supply per Capita']\n",
    "    return Top15['Population Estimate'].sort_values(ascending=False).index[2]\n",
    "    #return Top15\n",
    "answer_eight()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 9 (6.6%)\n",
    "Create a column that estimates the number of citable documents per person. \n",
    "What is the correlation between the number of citable documents per capita and the energy supply per capita? Use the `.corr()` method, (Pearson's correlation).\n",
    "\n",
    "*This function should return a single number.*\n",
    "\n",
    "*(Optional: Use the built-in function `plot9()` to visualize the relationship between Energy Supply per Capita vs. Citable docs per Capita)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "umich_part_id": "017",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.79400104354429424"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_nine():\n",
    "    Top15 = answer_one()\n",
    "    Top15['Population Estimate'] = Top15['Energy Supply']/Top15['Energy Supply per Capita']\n",
    "    Top15['Citable Documents per Capita'] = Top15['Citable documents']/Top15['Population Estimate']\n",
    "    \n",
    "    # Need to convert both to float for this to work.\n",
    "    Top15['Citable Documents per Capita'] = Top15['Citable Documents per Capita'].astype(float)\n",
    "    Top15['Energy Supply per Capita'] = Top15['Energy Supply per Capita'].astype(float)\n",
    "    \n",
    "    # Pearson's method is the default one.\n",
    "    return Top15['Citable Documents per Capita'].corr(Top15['Energy Supply per Capita'])\n",
    "answer_nine()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot9():\n",
    "    import matplotlib as plt\n",
    "    %matplotlib inline\n",
    "    \n",
    "    Top15 = answer_one()\n",
    "    Top15['PopEst'] = Top15['Energy Supply'] / Top15['Energy Supply per Capita']\n",
    "    Top15['Citable docs per Capita'] = Top15['Citable documents'] / Top15['PopEst']\n",
    "    Top15.plot(x='Citable docs per Capita', y='Energy Supply per Capita', kind='scatter', xlim=[0, 0.0006])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#plot9() # Be sure to comment out plot9() before submitting the assignment!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 10 (6.6%)\n",
    "Create a new column with a 1 if the country's % Renewable value is at or above the median for all countries in the top 15, and a 0 if the country's % Renewable value is below the median.\n",
    "\n",
    "*This function should return a series named `HighRenew` whose index is the country name sorted in ascending order of rank.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "umich_part_id": "018",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Country\n",
       "China                 1\n",
       "United States         0\n",
       "Japan                 0\n",
       "United Kingdom        0\n",
       "Russian Federation    1\n",
       "Canada                1\n",
       "Germany               1\n",
       "India                 0\n",
       "France                1\n",
       "South Korea           0\n",
       "Italy                 1\n",
       "Spain                 1\n",
       "Iran                  0\n",
       "Australia             0\n",
       "Brazil                1\n",
       "Name: HighRenew, dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_ten():\n",
    "    import numpy as np\n",
    "    Top15 = answer_one()\n",
    "    Top15['HighRenew'] = np.where(Top15['% Renewable'] >= np.median(Top15['% Renewable']),1,0)\n",
    "    \n",
    "    # Dataframe is already sorted by rank.\n",
    "    return Top15['HighRenew']\n",
    "answer_ten()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 11 (6.6%)\n",
    "Use the following dictionary to group the Countries by Continent, then create a dateframe that displays the sample size (the number of countries in each continent bin), and the sum, mean, and std deviation for the estimated population of each country.\n",
    "\n",
    "```python\n",
    "ContinentDict  = {'China':'Asia', \n",
    "                  'United States':'North America', \n",
    "                  'Japan':'Asia', \n",
    "                  'United Kingdom':'Europe', \n",
    "                  'Russian Federation':'Europe', \n",
    "                  'Canada':'North America', \n",
    "                  'Germany':'Europe', \n",
    "                  'India':'Asia',\n",
    "                  'France':'Europe', \n",
    "                  'South Korea':'Asia', \n",
    "                  'Italy':'Europe', \n",
    "                  'Spain':'Europe', \n",
    "                  'Iran':'Asia',\n",
    "                  'Australia':'Australia', \n",
    "                  'Brazil':'South America'}\n",
    "```\n",
    "\n",
    "*This function should return a DataFrame with index named Continent `['Asia', 'Australia', 'Europe', 'North America', 'South America']` and columns `['size', 'sum', 'mean', 'std']`*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "umich_part_id": "019",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>size</th>\n",
       "      <th>sum</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Continent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Asia</th>\n",
       "      <td>5.0</td>\n",
       "      <td>2.898666e+09</td>\n",
       "      <td>5.797333e+08</td>\n",
       "      <td>6.790979e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Australia</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.331602e+07</td>\n",
       "      <td>2.331602e+07</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Europe</th>\n",
       "      <td>6.0</td>\n",
       "      <td>4.579297e+08</td>\n",
       "      <td>7.632161e+07</td>\n",
       "      <td>3.464767e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>North America</th>\n",
       "      <td>2.0</td>\n",
       "      <td>3.528552e+08</td>\n",
       "      <td>1.764276e+08</td>\n",
       "      <td>1.996696e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>South America</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.059153e+08</td>\n",
       "      <td>2.059153e+08</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               size           sum          mean           std\n",
       "Continent                                                    \n",
       "Asia            5.0  2.898666e+09  5.797333e+08  6.790979e+08\n",
       "Australia       1.0  2.331602e+07  2.331602e+07           NaN\n",
       "Europe          6.0  4.579297e+08  7.632161e+07  3.464767e+07\n",
       "North America   2.0  3.528552e+08  1.764276e+08  1.996696e+08\n",
       "South America   1.0  2.059153e+08  2.059153e+08           NaN"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_eleven():\n",
    "    import numpy as np\n",
    "    \n",
    "    Top15 = answer_one()\n",
    "    \n",
    "    ContinentDict  = {'China':'Asia', \n",
    "                  'United States':'North America', \n",
    "                  'Japan':'Asia', \n",
    "                  'United Kingdom':'Europe', \n",
    "                  'Russian Federation':'Europe', \n",
    "                  'Canada':'North America', \n",
    "                  'Germany':'Europe', \n",
    "                  'India':'Asia',\n",
    "                  'France':'Europe', \n",
    "                  'South Korea':'Asia', \n",
    "                  'Italy':'Europe', \n",
    "                  'Spain':'Europe', \n",
    "                  'Iran':'Asia',\n",
    "                  'Australia':'Australia', \n",
    "                  'Brazil':'South America'}\n",
    "    \n",
    "    # Map the dictionary values with the index values.\n",
    "    Top15['Continent'] = Top15.index.to_series().map(ContinentDict)\n",
    "    \n",
    "    # Set to float for numerical operations in the last step.\n",
    "    Top15['Population Estimate'] = (Top15['Energy Supply']/Top15['Energy Supply per Capita']).astype(float)\n",
    "    \n",
    "    # agg appies functions to a group of values.\n",
    "    final = Top15.set_index('Continent').groupby(level = 0)['Population Estimate'].agg({'size':np.size, 'sum':np.sum, 'mean':np.mean, 'std':np.std})\n",
    "    return final\n",
    "\n",
    "answer_eleven()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 12 (6.6%)\n",
    "Cut % Renewable into 5 bins. Group Top15 by the Continent, as well as these new % Renewable bins. How many countries are in each of these groups?\n",
    "\n",
    "*This function should return a __Series__ with a MultiIndex of `Continent`, then the bins for `% Renewable`. Do not include groups with no countries.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Continent      Bins            \n",
       "Asia           (2.212, 15.753]     4\n",
       "               (15.753, 29.227]    1\n",
       "Australia      (2.212, 15.753]     1\n",
       "Europe         (2.212, 15.753]     1\n",
       "               (15.753, 29.227]    3\n",
       "               (29.227, 42.701]    2\n",
       "North America  (2.212, 15.753]     1\n",
       "               (56.174, 69.648]    1\n",
       "South America  (56.174, 69.648]    1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_twelve():\n",
    "    import pandas as pd\n",
    "    Top15 = answer_one()\n",
    "    \n",
    "    ContinentDict  = {'China':'Asia', \n",
    "                  'United States':'North America', \n",
    "                  'Japan':'Asia', \n",
    "                  'United Kingdom':'Europe', \n",
    "                  'Russian Federation':'Europe', \n",
    "                  'Canada':'North America', \n",
    "                  'Germany':'Europe', \n",
    "                  'India':'Asia',\n",
    "                  'France':'Europe', \n",
    "                  'South Korea':'Asia', \n",
    "                  'Italy':'Europe', \n",
    "                  'Spain':'Europe', \n",
    "                  'Iran':'Asia',\n",
    "                  'Australia':'Australia', \n",
    "                  'Brazil':'South America'}\n",
    "    \n",
    "    # Map the dictionary values with the index values.\n",
    "    Top15['Continent'] = Top15.index.to_series().map(ContinentDict)\n",
    "    \n",
    "    Top15['Bins'] = pd.cut(Top15['% Renewable'],5) # The ENTIRE range of % Renewable is split into 5 bins.\n",
    "    \n",
    "    return Top15.groupby(['Continent','Bins']).size() # The entire range of bins is then grouped by continent.\n",
    "\n",
    "answer_twelve()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 13 (6.6%)\n",
    "Convert the Population Estimate series to a string with thousands separator (using commas). Do not round the results.\n",
    "\n",
    "e.g. 317615384.61538464 -> 317,615,384.61538464\n",
    "\n",
    "*This function should return a Series `PopEst` whose index is the country name and whose values are the population estimate string.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true,
    "umich_part_id": "020",
    "umich_partlist_id": "003"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Country\n",
       "China                 1,367,645,161.2903225\n",
       "United States          317,615,384.61538464\n",
       "Japan                  127,409,395.97315437\n",
       "United Kingdom         63,870,967.741935484\n",
       "Russian Federation            143,500,000.0\n",
       "Canada                  35,239,864.86486486\n",
       "Germany                 80,369,696.96969697\n",
       "India                 1,276,730,769.2307692\n",
       "France                  63,837,349.39759036\n",
       "South Korea            49,805,429.864253394\n",
       "Italy                  59,908,256.880733944\n",
       "Spain                    46,443,396.2264151\n",
       "Iran                    77,075,630.25210084\n",
       "Australia              23,316,017.316017315\n",
       "Brazil                 205,915,254.23728815\n",
       "Name: PopEst, dtype: object"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_thirteen():\n",
    "    Top15 = answer_one()\n",
    "    \n",
    "    Top15['PopEst'] = (Top15['Energy Supply']/Top15['Energy Supply per Capita'])\n",
    "    \n",
    "    # Formula for adding comma separators.\n",
    "    return Top15['PopEst'].apply(lambda x: '{0:,}'.format(x)).astype(str)\n",
    "\n",
    "answer_thirteen()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optional\n",
    "\n",
    "Use the built in function `plot_optional()` to see an example visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def plot_optional():\n",
    "    import matplotlib as plt\n",
    "    %matplotlib inline\n",
    "    Top15 = answer_one()\n",
    "    ax = Top15.plot(x='Rank', y='% Renewable', kind='scatter', \n",
    "                    c=['#e41a1c','#377eb8','#e41a1c','#4daf4a','#4daf4a','#377eb8','#4daf4a','#e41a1c',\n",
    "                       '#4daf4a','#e41a1c','#4daf4a','#4daf4a','#e41a1c','#dede00','#ff7f00'], \n",
    "                    xticks=range(1,16), s=6*Top15['2014.0']/10**10, alpha=.75, figsize=[16,6]);\n",
    "\n",
    "    for i, txt in enumerate(Top15.index):\n",
    "        ax.annotate(txt, [Top15['Rank'][i], Top15['% Renewable'][i]], ha='center')\n",
    "\n",
    "    print(\"This is an example of a visualization that can be created to help understand the data. \\\n",
    "This is a bubble chart showing % Renewable vs. Rank. The size of the bubble corresponds to the countries' \\\n",
    "2014 GDP, and the color corresponds to the continent.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#plot_optional() # Be sure to comment out plot_optional() before submitting the assignment!"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "coursera": {
   "course_slug": "python-data-analysis",
   "graded_item_id": "zAr06",
   "launcher_item_id": "KSSjT",
   "part_id": "SL3fU"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  },
  "umich": {
   "id": "Assignment 3",
   "version": "1.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}