introduction-lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Goals of the Laboratory\n",
    "In this introductory laboratory, we expect students to:\n",
    "\n",
    "1. Acquire basic knowledge about Python and Matplotlib\n",
    "2. Gain familiarity with Juypter Notebooks\n",
    "3. Gain familiarity with the PySpark API\n",
    "\n",
    "To achieve such goals, we will go through the following steps:\n",
    "\n",
    "1. In section 1, **IPython** and **Jupyter Notebooks** are introduced to help students understand the environment used to work on projects, including those that are part of the CLOUDS course.\n",
    "\n",
    "2. In section 2, we briefly overview **Python** and its syntax. In addition, we cover **Matplotlib**, a very powerful library to plot figures in Python. Finally, we introduce **Pandas**, a python library that is very helpful when manipulating data.\n",
    "\n",
    "3. In section 3 we cover the **PySpark** APIs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Python, IPython and Jupyter Notebooks\n",
    "\n",
    "**Python** is a high-level, dynamic, object-oriented programming language. It is a general purpose language, which is designed to be easy to use and easy to read.\n",
    "\n",
    "**IPython** (Interactive Python) is originally developed for Python. Now, it is a command shell for interactive computing supporting multiple programming languages. It offers rich media, shell syntax, tab completion, and history. IPython is based on an architecture that provides parallel and distributed computing. IPython enables parallel applications to be developed, executed, debugged and monitored interactively.\n",
    "\n",
    "**Jupyter Notebooks** are a web-based interactive computational environment for creating IPython notebooks. An IPython notebook is a JSON document containing an ordered list of input/output cells which can contain code, text, mathematics, plots and rich media. Notebooks make data analysis easier to perform, understand and reproduce. All laboratories in this course are prepared as Notebooks. As you can see, in this Notebook, we can put text, images, hyperlinks, source code... The Notebooks can be converted to a number of open standard output formats (HTML, HTML presentation slides, LaTeX, PDF, ReStructuredText, Markdown, Python) through `File` -> `Download As` in the web interface. In addition, Jupyter manages the notebooks' versions through a `checkpoint` mechanism. You can create checkpoint anytime via `File -> Save and Checkpoint`. \n",
    "\n",
    "**NOTE on Checkpointing:** in this course, we use a peculiar environment to work. We don't have a Notebook server: instead, we create on demand clusters with a Notebook front-end. Since your clusters are **ephemeral** (they are terminated after a predefined amount of time), checkpointing is of little use, for anything else than saving your notebook in your ephemeral environment. It is far better to download regularly your notebooks, and to push them to your git repository."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1. Tab completion\n",
    "\n",
    "Tab completion is a convenient way to explore the structure of any object you're dealing with. Simply type object_name.<TAB> to view the suggestion for object's attributes. Besides Python objects and keywords, tab completion also works on file and directory names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "s = \"test function of tab completion\"\n",
    "\n",
    "# type s.<TAB> to see the suggestions\n",
    "\n",
    "# Show your experiments working on a string. \n",
    "# Try splitting a string into its constituent words, and count the number of words.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2. System shell commands\n",
    "\n",
    "To run any command in the system shell, simply prefix it with `!`. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# list all file and directories in the current folder\n",
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3. Magic functions\n",
    "\n",
    "IPython has a set of predefined `magic functions` that you can call with a command line style syntax. There are two types of magics, line-oriented and cell-oriented. \n",
    "\n",
    "**Line magics** are prefixed with the `%` character and work much like OS command-line calls: they get as an argument the rest of the line, *where arguments are passed without parentheses or quotes*. \n",
    "\n",
    "**Cell magics** are prefixed with a double `%%`, and they are functions that get as an argument not only the rest of the line, but also the lines below it in a separate argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%timeit range(1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%timeit x = range(10000)\n",
    "max(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information, you can follow this [link](http://nbviewer.jupyter.org/github/ipython/ipython/blob/1.x/examples/notebooks/Cell%20Magics.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4. Debugging\n",
    "\n",
    "Whenever an exception occurs, the call stack is printed out to help you to track down the true source of the problem. It is important to gain familiarity with the call stack, especially when using the PySpark API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for i in [4,3,2,0]:\n",
    "    print(5/i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.5. Additional features\n",
    "\n",
    "Jupyter also supports viewing the status of the cluster and interact with the real shell environment.\n",
    "\n",
    "To do that, you can click on the Logo Jupyter in the up-left corner of each notebook to go to the dashboard:\n",
    "\n",
    "<img src=\"https://farm2.staticflickr.com/1488/24681339931_733acb3494_b.jpg\" width=\"600px\" />\n",
    "\n",
    "You can easily find out how to use these features, so you're invited to play around!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Python + Pandas + Matplotlib: A great environment for Data Science\n",
    "\n",
    "This section aims to help students gain a basic understanding of the python programming language and some of its libraries, including `Pandas` or `Matplotlib`. \n",
    "\n",
    "When working with a small dataset (one that can comfortably fit into a single machine), Pandas and Matplotlib, together with Python are valid alternatives to other popular tools such as R and Matlab. Using such libraries allows to inherit from the simple and clear Python syntax, achieve very good performance, enjoy superior memory management,  error handling, and good package management \\[[1](http://ajminich.com/2013/06/22/9-reasons-to-switch-from-matlab-to-python/)\\].\n",
    "\n",
    "\n",
    "## 2.1. Python syntax\n",
    "\n",
    "(This section is for students who did not program in Python before. If you're familiar with Python, please move to the next section: 1.2. Numpy)\n",
    "\n",
    "When working with Python, the code seems to be simpler than (many) other languages. In this laboratory, we compare the Python syntax to that of Java - another very common language.\n",
    "\n",
    "```java\n",
    "// java syntax\n",
    "int i = 10;\n",
    "string s = \"advanced machine learning\";\n",
    "System.out.println(i);\n",
    "System.out.println(s);\n",
    "// you must not forget the semicolon at the end of each sentence\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# python syntax\n",
    "i = 10\n",
    "s = \"advanced machine learning\"\n",
    "print(i)\n",
    "print(s)\n",
    "# forget about the obligation of commas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indentation & If-else syntax\n",
    "In python, we don't use `{` and `}` to define blocks of codes: instead, we use indentation to do that. **The code within the same block must have the same indentation**. For example, in java, we write:\n",
    "```java\n",
    "string language = \"Python\";\n",
    "\n",
    "// the block is surrounded by { and }\n",
    "// the condition is in ( and )\n",
    "if (language == \"Python\") {\n",
    "    int x = 1;\n",
    "    x += 10;\n",
    "       int y = 5; // a wrong indentation isn't problem\n",
    "    y = x + y;\n",
    "    System.out.println(x + y);\n",
    "    \n",
    "    // a statement is broken into two line\n",
    "    x = y\n",
    "        + y;\n",
    "    \n",
    "    // do some stuffs\n",
    "}\n",
    "else if (language == \"Java\") {\n",
    "    // another block\n",
    "}\n",
    "else {\n",
    "    // another block\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "45\n"
     ]
    }
   ],
   "source": [
    "language = \"Python\"\n",
    "if language == \"Python\":\n",
    "    x = 10\n",
    "    x += 10\n",
    "    y = 5 # all statements in the same block must have the same indentation\n",
    "    y = (\n",
    "        x + y\n",
    "    ) # statements can be on multiple lines, using ( )\n",
    "    print (x \n",
    "           + y)\n",
    "    \n",
    "    # statements can also be split on multiple lines by using \\ at the END of each line\n",
    "    x = y \\\n",
    "        + y\n",
    "    \n",
    "    # do some other stuffs\n",
    "elif language == \"Java\":\n",
    "    # another block\n",
    "    pass\n",
    "else:\n",
    "    # another block\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ternary conditional operator\n",
    "In python, we often see ternary conditional operator, which is used to assign a value to a variable based on some condition. For example, in java, we write:\n",
    "\n",
    "```java\n",
    "int x = 10;\n",
    "// if x > 10, assign y = 5, otherwise, y = 15\n",
    "int y = (x > 10) ? 5 : 15;\n",
    "\n",
    "int z;\n",
    "if (x > 10)\n",
    "    z = 5; // it's not necessary to have { } when the block has only one statement\n",
    "else\n",
    "    z = 15;\n",
    "```\n",
    "\n",
    "Of course, although we can easily write these lines of code in an `if else` block to get the same result, people prefer ternary conditional operator because of simplicity.\n",
    "\n",
    "In python, we write:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = 10\n",
    "# a very natural way\n",
    "y = 5 if x > 10 else 15\n",
    "print(y)\n",
    "\n",
    "# another way\n",
    "y = x > 10 and 5 or 15\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lists and For loops\n",
    "Another syntax that we should revisit is the `for loop`. In java, we can write:\n",
    "\n",
    "```java\n",
    "// init an array with 10 integer numbers\n",
    "int[] array = new int[]{1, 2, 3, 4, 5, 6, 7, 8, 9, 10};\n",
    "for (int i = 0; i < array.length; i++){\n",
    "    // print the i-th element of array\n",
    "    System.out.println(array[i]);\n",
    "}\n",
    "```\n",
    "\n",
    "In Python, instead of using an index to help indicating an element, we can access the element directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "array = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]\n",
    "# Python has no built-in array data structure\n",
    "# instead, it uses \"list\" which is much more general \n",
    "# and can be used as a multidimensional array quite easily.\n",
    "for element in array:\n",
    "    print(element)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the code is very clean. If you need the index of each element, here's what you should do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for (index, element) in enumerate(array):\n",
    "    print(index, element)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Actually, Python has no built-in array data structure. It uses the `list` data structure, which is much more general and can be used as a multidimensional array quite easily. In addition, elements in a list can be retrieved in a very concise way. For example, we create a 2d-array with 4 rows. Each row has 3 elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 2-dimentions array with 4 rows, 3 columns\n",
    "twod_array = [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]]\n",
    "for index, row in enumerate(twod_array):\n",
    "    print(\"row \", index, \":\", row)\n",
    "\n",
    "# print row 1 until row 3\n",
    "print(\"row 1 until row 3: \", twod_array[1:3])\n",
    "\n",
    "# all rows from row 2\n",
    "print(\"all rows from row 2: \", twod_array[2:])\n",
    "\n",
    "# all rows until row 2\n",
    "print(\"all rows until row 2:\", twod_array[:2])\n",
    "\n",
    "# all rows from the beginning with step of 2. \n",
    "print(\"all rows from the beginning with step of 2:\", twod_array[::2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dictionaries\n",
    "Another useful data structure in Python is a `dictionary`, which we use to store (key, value) pairs. Here's some example usage of dictionaries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "d = {'key1': 'value1', 'key2': 'value2'}  # Create a new dictionary with some data\n",
    "print(d['key1'])       # Get an entry from a dictionary; prints \"value1\"\n",
    "print('key1' in d)     # Check if a dictionary has a given key; prints \"True\"\n",
    "d['key3'] = 'value3'    # Set an entry in a dictionary\n",
    "print(d['key3'])      # Prints \"value3\"\n",
    "# print(d['key9'])  # KeyError: 'key9' not a key of d\n",
    "print(d.get('key9', 'custom_default_value'))  # Get an element with a default; prints \"custom_default_value\"\n",
    "print(d.get('key3', 'custom_default_value'))    # Get an element with a default; prints \"value3\"\n",
    "del d['key3']        # Remove an element from a dictionary\n",
    "print(d.get('key3', 'custom_default_value')) # \"fish\" is no longer a key; prints \"custom_default_value\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Functions\n",
    "In Python, we can define a function by using keyword `def`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def square(x):\n",
    "    return x*x\n",
    "\n",
    "print(square(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can apply a function to each element of a list/array by using `lambda` function. For example, we want to square elements in a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "array = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]\n",
    "\n",
    "# apply function \"square\" on each element of \"array\"\n",
    "print(list(map(lambda x: square(x), array)))\n",
    "\n",
    "# or using a for loop, and a list comprehension\n",
    "print([square(x) for x in array])\n",
    "\n",
    "print(\"orignal array:\", array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These two above syntaxes are used very often. \n",
    "\n",
    "If you are not familiar with **list comprehensions**, follow this [link](http://python-3-patterns-idioms-test.readthedocs.io/en/latest/Comprehensions.html]).\n",
    "\n",
    "We can also put a function `B` inside a function `A` (that is, we can have nested functions). In that case, function `B` is only accessed inside function `A` (the scope that it's declared). For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# select only the prime number in array\n",
    "# and square them\n",
    "def filterAndSquarePrime(arr):\n",
    "    \n",
    "    # a very simple function to check a number is prime or not\n",
    "    def checkPrime(number):\n",
    "        for i in range(2, int(number/2)):\n",
    "            if number % i == 0:\n",
    "                return False\n",
    "        return True\n",
    "    \n",
    "    primeNumbers = filter(lambda x: checkPrime(x), arr)\n",
    "    return map(lambda x: square(x), primeNumbers)\n",
    "\n",
    "# we can not access checkPrime from here\n",
    "# checkPrime(5)\n",
    "\n",
    "result = filterAndSquarePrime(array)\n",
    "list(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing modules, functions\n",
    "Modules in Python are packages of code. Putting code into modules helps increasing the reusability and maintainability.\n",
    "The modules can be nested.\n",
    "To import a module, we simple use syntax: `import <module_name>`. Once it is imported, we can use any functions, classes inside it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# import module 'math' to uses functions for calculating\n",
    "import math\n",
    "\n",
    "# print the square root of 16\n",
    "print(math.sqrt(16))\n",
    "\n",
    "# we can create alias when import a module\n",
    "import numpy as np\n",
    "\n",
    "print(np.sqrt(16))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, you only need to import some functions inside a module to avoid loading the whole module into memory. To do that, we can use syntax: `from <module> import <function>`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# only import function 'sin' in package 'math'\n",
    "from math import sin\n",
    "\n",
    "# use the function\n",
    "print(sin(60))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's quite enough for Python. Now, let's practice a little bit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1\n",
    "#### Question 1.1\n",
    "<div class=\"alert alert-info\">\n",
    "Write a function `checkSquareNumber` to check if a integer number is a square number or not. For example, 16 and 9 are square numbers. 15 isn't square number.\n",
    "Requirements:\n",
    "\n",
    "- Input: an integer number\n",
    "\n",
    "- Output: `True` or `False`\n",
    "\n",
    "HINT: If the square root of a number is an integer number, it is a square number.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "import math\n",
    "\n",
    "def checkSquareNumber(x):\n",
    "    # calculate the square root of x\n",
    "    # return True if square root is integer, \n",
    "    # otherwise, return False\n",
    "    return ...\n",
    "\n",
    "print(checkSquareNumber(16))\n",
    "print(checkSquareNumber(250))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 1.2\n",
    "<div class=\"alert alert-info\">\n",
    "A list `list_numbers` which contains the numbers from 1 to 9999 can be constructed from: \n",
    "\n",
    "```python\n",
    "list_numbers = range(0, 10000)\n",
    "```\n",
    "\n",
    "Extract the square numbers in `list_numbers` using function `checkSquareNumber` from question 1.1. How many elements in the extracted list ?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "list_numbers = ...\n",
    "square_numbers = # try to use the filter method\n",
    "print(square_numbers)\n",
    "print(len(square_numbers))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 1.3\n",
    "<div class=\"alert alert-info\">\n",
    "Using array slicing, select the elements of the list square_numbers, whose index is from 5 to 20 (zero-based index).\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "print(square_numbers[...])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will take a quick look on Numpy - a powerful module of Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2. Numpy\n",
    "Numpy is the core library for scientific computing in Python. It provides a high-performance multidimensional array object, and tools for working with these arrays.\n",
    "### 2.2.1. Array\n",
    "A numpy array is a grid of values, all of **the same type**, and is indexed by a tuple of nonnegative integers. Thanks to the same type property, Numpy has the benefits of [locality of reference](https://en.wikipedia.org/wiki/Locality_of_reference). Besides, many other Numpy operations are implemented in C, avoiding the general cost of loops in Python, pointer indirection and per-element dynamic type checking. So, the speed of Numpy is often faster than using built-in datastructure of Python. When working with massive data with computationally expensive tasks, you should consider to use Numpy. \n",
    "\n",
    "The number of dimensions is the `rank` of the array; the `shape` of an array is a tuple of integers giving the size of the array along each dimension.\n",
    "\n",
    "We can initialize numpy arrays from nested Python lists, and access elements using square brackets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Create a rank 1 array\n",
    "rank1_array = np.array([1, 2, 3])\n",
    "print(\"type of rank1_array:\", type(rank1_array))\n",
    "print(\"shape of rank1_array:\", rank1_array.shape)\n",
    "print(\"elements in rank1_array:\", rank1_array[0], rank1_array[1], rank1_array[2])\n",
    "\n",
    "# Create a rank 2 array\n",
    "rank2_array = np.array([[1,2,3],[4,5,6]])\n",
    "print(\"shape of rank2_array:\", rank2_array.shape)\n",
    "print(rank2_array[0, 0], rank2_array[0, 1], rank2_array[1, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.2. Array slicing\n",
    "Similar to Python lists, numpy arrays can be sliced. The different thing is that you must specify a slice for each dimension of the array because arrays may be multidimensional."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "m_array = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])\n",
    "\n",
    "# Use slicing to pull out the subarray consisting of the first 2 rows\n",
    "# and columns 1 and 2\n",
    "b = m_array[:2, 1:3]\n",
    "print(b)\n",
    "\n",
    "# we can only use this syntax with numpy array, not python list\n",
    "print(\"value at row 0, column 1:\", m_array[0, 1])\n",
    "\n",
    "# Rank 1 view of the second row of m_array  \n",
    "print(\"the second row of m_array:\", m_array[1, :])\n",
    "\n",
    "# print element at position (0,2) and (1,3)\n",
    "print(m_array[[0,1], [2,3]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.3. Boolean array indexing\n",
    "We can use boolean array indexing to check whether each element in the array satisfies a condition or use it to do filtering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "m_array = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])\n",
    "\n",
    "# Find the elements of a that are bigger than 2\n",
    "# this returns a numpy array of Booleans of the same\n",
    "# shape as m_array, where each value of bool_idx tells\n",
    "# whether that element of a is > 3 or not\n",
    "bool_idx = (m_array > 3)\n",
    "print(bool_idx , \"\\n\")\n",
    "\n",
    "# We use boolean array indexing to construct a rank 1 array\n",
    "# consisting of the elements of a corresponding to the True values\n",
    "# of bool_idx\n",
    "print(m_array[bool_idx], \"\\n\")\n",
    "\n",
    "# We can combine two statements\n",
    "print(m_array[m_array > 3], \"\\n\")\n",
    "\n",
    "# select elements with multiple conditions\n",
    "print(m_array[(m_array > 3) & (m_array % 2 == 0)])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.4. Datatypes\n",
    "Remember that the elements in a numpy array have the same type. When constructing arrays, Numpy tries to guess a datatype when you create an array However, we can specify the datatype explicitly via an optional argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# let Numpy guess the datatype\n",
    "x1 = np.array([1, 2])\n",
    "print(x1.dtype)\n",
    "\n",
    "# force the datatype be float64\n",
    "x2 = np.array([1, 2], dtype=np.float64)\n",
    "print(x2.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.5. Array math\n",
    "Similar to Matlab or R, in Numpy, basic mathematical functions operate elementwise on arrays, and are available both as operator overloads and as functions in the numpy module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = np.array([[1,2],[3,4]], dtype=np.float64)\n",
    "y = np.array([[5,6],[7,8]], dtype=np.float64)\n",
    "# mathematical function is used as operator\n",
    "print(\"x + y =\", x + y, \"\\n\")\n",
    "\n",
    "# mathematical function is used as function\n",
    "print(\"np.add(x, y)=\", np.add(x, y), \"\\n\")\n",
    "\n",
    "# Unlike MATLAB, * is elementwise multiplication\n",
    "# not matrix multiplication\n",
    "print(\"x * y =\", x * y , \"\\n\")\n",
    "print(\"np.multiply(x, y)=\", np.multiply(x, y), \"\\n\")\n",
    "print(\"x*2=\", x*2, \"\\n\")\n",
    "\n",
    "# to multiply two matrices, we use dot function\n",
    "print(\"x.dot(y)=\", x.dot(y), \"\\n\")\n",
    "print(\"np.dot(x, y)=\", np.dot(x, y), \"\\n\")\n",
    "\n",
    "# Elementwise square root\n",
    "print(\"np.sqrt(x)=\", np.sqrt(x), \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that unlike MATLAB, `*` is elementwise multiplication, not matrix multiplication. We instead use the `dot` function to compute inner products of vectors, to multiply a vector by a matrix, and to multiply matrices. In what follows, we work on a few more examples to reiterate the concept."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# declare two vectors\n",
    "v = np.array([9,10])\n",
    "w = np.array([11, 12])\n",
    "\n",
    "# Inner product of vectors\n",
    "print(\"v.dot(w)=\", v.dot(w))\n",
    "print(\"np.dot(v, w)=\", np.dot(v, w))\n",
    "\n",
    "# Matrix / vector product\n",
    "print(\"x.dot(v)=\", x.dot(v))\n",
    "print(\"np.dot(x, v)=\", np.dot(x, v))\n",
    "\n",
    "# Matrix / matrix product\n",
    "print(\"x.dot(y)=\", x.dot(y))\n",
    "print(\"np.dot(x, y)=\", np.dot(x, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, we can do other aggregation computations on arrays such as `sum`, `nansum`, or `T`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = np.array([[1,2], [3,4]])\n",
    "\n",
    "# Compute sum of all elements\n",
    "print(np.sum(x))\n",
    "\n",
    "# Compute sum of each column\n",
    "print(np.sum(x, axis=0))\n",
    "\n",
    "# Compute sum of each row\n",
    "print(np.sum(x, axis=1))\n",
    "\n",
    "# transpose the matrix\n",
    "print(x.T)\n",
    "\n",
    "# Note that taking the transpose of a rank 1 array does nothing:\n",
    "v = np.array([1,2,3])\n",
    "print(v.T)  # Prints \"[1 2 3]\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2\n",
    "\n",
    "Given a 2D array:\n",
    "\n",
    "```\n",
    " 1  2  3  4\n",
    " 5  6  7  8 \n",
    " 9 10 11 12\n",
    "13 14 15 16\n",
    "```\n",
    "\n",
    "\n",
    "#### Question 2.1\n",
    "<div class=\"alert alert-info\">\n",
    "Print the all odd numbers in this array using `Boolean array indexing`.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "array_numbers = np.array([\n",
    "        [1, 2, 3, 4],\n",
    "        [5, 6, 7, 8],\n",
    "        [9, 10, 11, 12],\n",
    "        [13, 14, 15, 16]\n",
    "    ])\n",
    "\n",
    "print(...)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 2.2\n",
    "<div class=\"alert alert-info\">\n",
    "Extract the second row and the third column in this array using `array slicing`.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "print(array_numbers[...])\n",
    "print(array_numbers[...])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 2.3\n",
    "<div class=\"alert alert-info\">\n",
    "Calculate the sum of diagonal elements.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "sum = 0\n",
    "for i in range(0, ...):\n",
    "    sum += array_numbers...\n",
    "    \n",
    "print(sum)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 2.4\n",
    "<div class=\"alert alert-info\">\n",
    "Print elementwise multiplication of the first row and the last row using numpy's functions.\n",
    "\n",
    "Print the inner product of these two rows.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "print(...)\n",
    "print(...)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3. Matplotlib\n",
    "\n",
    "As its name indicates, Matplotlib is a plotting library. It provides both a very quick way to visualize data from Python and publication-quality figures in many formats. The most important function in matplotlib is `plot`, which allows you to plot 2D data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.plot([1,2,3,4])\n",
    "plt.ylabel('custom y label')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, we provide a single list or array to the `plot()` command, matplotlib assumes it is a sequence of y values, and automatically generates the x values for us. Since python ranges start with 0, the default x vector has the same length as y but starts with 0. Hence the x data are [0,1,2,3].\n",
    "\n",
    "In the next example, we plot figure with both x and y data. Besides, we want to draw dashed lines instead of the solid in default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.plot([1, 2, 3, 4], [1, 4, 9, 16], 'r--')\n",
    "plt.show()\n",
    "\n",
    "plt.bar([1, 2, 3, 4], [1, 4, 9, 16], align='center')\n",
    "# labels of each column bar\n",
    "x_labels = [\"Type 1\", \"Type 2\", \"Type 3\", \"Type 4\"]\n",
    "# assign labels to the plot\n",
    "plt.xticks([1, 2, 3, 4], x_labels)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to merge two figures into a single one, subplot is the best way to do that. For example, we want to put two figures in a stack vertically, we should define a grid of plots with 2 rows and 1 column. Then, in each row, a single figure is plotted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Set up a subplot grid that has height 2 and width 1,\n",
    "# and set the first such subplot as active.\n",
    "plt.subplot(2, 1, 1)\n",
    "plt.plot([1, 2, 3, 4], [1, 4, 9, 16], 'r--')\n",
    "\n",
    "# Set the second subplot as active, and make the second plot.\n",
    "plt.subplot(2, 1, 2)\n",
    "plt.bar([1, 2, 3, 4], [1, 4, 9, 16])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more examples, please visit the [homepage](http://matplotlib.org/1.5.1/examples/index.html) of Matplotlib."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Question 3\n",
    "Given a list of numbers from 0 to 9999.\n",
    "\n",
    "\n",
    "#### Question 3.1\n",
    "<div class=\"alert alert-info\">\n",
    "Calculate the histogram of numbers divisible by 3, 7, 11 in the list respectively.\n",
    "\n",
    "( Or in other words, how many numbers divisible by 3, 7, 11 in the list respectively ?)\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "arr = np.array(...)\n",
    "divisors = [3, 7, 11]\n",
    "histogram = list(...)\n",
    "print(histogram)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 3.2\n",
    "<div class=\"alert alert-info\">\n",
    "Plot the histogram in a line chart.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# simple line chart\n",
    "plt.plot(histogram)\n",
    "x_indexes = ...\n",
    "x_names = list(...)\n",
    "plt.xticks(x_indexes, x_names)\n",
    "plt.show()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 3.3\n",
    "<div class=\"alert alert-info\">\n",
    "Plot the histogram in a bar chart.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# char chart with x-lables\n",
    "x_indexes = range(...)\n",
    "x_names = list(...)\n",
    "plt.bar( x_indexes, histogram, align='center')\n",
    "plt.xticks(x_indexes, x_names)\n",
    "plt.show()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.4. Pandas\n",
    "\n",
    "Pandas is an open source library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. Indeed, it is great for data manipulation, data analysis, and data visualization.\n",
    "\n",
    "### 2.4.1. Data structures\n",
    "Pandas introduces two useful (and powerful) structures: `Series` and `DataFrame`, both of which are built on top of NumPy.\n",
    "\n",
    "#### Series\n",
    "A `Series` is a one-dimensional object similar to an array, list, or even column in a table. It assigns a *labeled index* to each item in the Series. By default, each item will receive an index label from `0` to `N-1`, where `N` is the number items of `Series`.\n",
    "\n",
    "We can create a Series by passing a list of values, and let pandas create a default integer index.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# create a Series with an arbitrary list\n",
    "s = pd.Series([3, 'Machine learning', 1.414259, -65545, 'Happy coding!'])\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, an index can be used explicitly when creating the `Series`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "s = pd.Series([3, 'Machine learning', 1.414259, -65545, 'Happy coding!'],\n",
    "             index=['Col1', 'Col2', 'Col3', 4.1, 5])\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `Series` can be constructed from a dictionary too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "s = pd.Series({\n",
    "        'Col1': 3, 'Col2': 'Machine learning', \n",
    "        'Col3': 1.414259, 4.1: -65545, \n",
    "        5: 'Happy coding!'\n",
    "    })\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access items in a `Series` in a same way as `Numpy`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "s = pd.Series({\n",
    "        'Col1': 3, 'Col2': -10, \n",
    "        'Col3': 1.414259, \n",
    "        4.1: -65545, \n",
    "        5: 8\n",
    "    })\n",
    "\n",
    "# get element which has index='Col1'\n",
    "print(\"s['Col1']=\", s['Col1'], \"\\n\")\n",
    "\n",
    "# get elements whose index is in a given list\n",
    "print(\"s[['Col1', 'Col3', 4.5]]=\", s[['Col1', 'Col3', 4.5]], \"\\n\")\n",
    "\n",
    "# use boolean indexing for selection\n",
    "print(s[s > 0], \"\\n\")\n",
    "\n",
    "# modify elements on the fly using boolean indexing\n",
    "s[s > 0] = 15\n",
    "\n",
    "print(s, \"\\n\")\n",
    "\n",
    "# mathematical operations can be done using operators and functions.\n",
    "print(s*10,  \"\\n\")\n",
    "print(np.square(s), \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### DataFrame\n",
    "A DataFrame is a tabular data structure comprised of rows and columns, akin to database table, or R's data.frame object. In a loose way, we can also think of a DataFrame as a group of Series objects that share an index (the column names).\n",
    "\n",
    "We can create a DataFrame by passing a dict of objects that can be converted to series-like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = {'year': [2013, 2014, 2015, 2013, 2014, 2015, 2013, 2014],\n",
    "        'team': ['Manchester United', 'Chelsea', 'Asernal', 'Liverpool', 'West Ham', 'Newcastle', 'Machester City', 'Tottenham'],\n",
    "        'wins': [11, 8, 10, 15, 11, 6, 10, 4],\n",
    "        'losses': [5, 8, 6, 1, 5, 10, 6, 12]}\n",
    "football = pd.DataFrame(data, columns=['year', 'team', 'wins', 'losses'])\n",
    "football"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can store data as a CSV file, or read data from a CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# save data to a csv file without the index\n",
    "football.to_csv('football.csv', index=False)\n",
    "\n",
    "from_csv = pd.read_csv('football.csv')\n",
    "from_csv.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To read a CSV file with a custom delimiter between values and custom columns' names, we can use parameters `sep` and `names` relatively.\n",
    "Moreover, Pandas also supports to read and write to [Excel file](http://pandas.pydata.org/pandas-docs/stable/io.html#io-excel) , sqlite database file, URL,  or even clipboard.\n",
    "\n",
    "We can have an overview on the data by using functions `info` and `describe`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(football.info(), \"\\n\")\n",
    "football.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy's regular slicing syntax works as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(football[0:2], \"\\n\")\n",
    "\n",
    "# show only the teams that have won more than 10 matches from 2014\n",
    "print(football[(football.year >= 2014) & (football.wins >= 10)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An important feature that Pandas supports is `JOIN`. Very often, the data comes from multiple sources, in multiple files. For example, we have 2 CSV files, one contains the information of Artists, the other contains information of Songs. If we want to query the artist name and his/her corresponding songs, we have to do joining two dataframe.\n",
    "\n",
    "Similar to SQL, in Pandas, you can do inner join, left outer join, right outer join and full outer join. Let's see a small example. Assume that we have two dataset of singers and songs. The relationship between two datasets is maintained by a constrain on `singer_code`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "singers = pd.DataFrame({'singer_code': range(5), \n",
    "                           'singer_name': ['singer_a', 'singer_b', 'singer_c', 'singer_d', 'singer_e']})\n",
    "songs = pd.DataFrame({'singer_code': [2, 2, 3, 4, 5], \n",
    "                           'song_name': ['song_f', 'song_g', 'song_h', 'song_i', 'song_j']})\n",
    "print(singers)\n",
    "print('\\n')\n",
    "print(songs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# inner join\n",
    "pd.merge(singers, songs, on='singer_code', how='inner')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# left join\n",
    "pd.merge(singers, songs, on='singer_code', how='left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# right join\n",
    "pd.merge(singers, songs, on='singer_code', how='right')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# outer join (full join)\n",
    "pd.merge(singers, songs, on='singer_code', how='outer')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also concatenate two dataframes vertically or horizontally via function `concat` and parameter `axis`. This function is useful when we need to append two similar datasets or to put them side by site"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# concat vertically\n",
    "pd.concat([singers, songs])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# concat horizontally\n",
    "pd.concat([singers, songs], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When computing descriptive statistic, we usually need to aggregate data by each group. For example, to answer the question \"how many songs each singer has?\", we have to group data by each singer, and then calculate the number of songs in each group. Not that the result must contain the statistic of all singers in database (even if some of them have no song)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = pd.merge(singers, songs, on='singer_code', how='left')\n",
    "\n",
    "# count the values of each column in group\n",
    "print(data.groupby('singer_code').count())\n",
    "\n",
    "print(\"\\n\")\n",
    "\n",
    "# count only song_name\n",
    "print(data.groupby('singer_code').song_name.count())\n",
    "\n",
    "print(\"\\n\")\n",
    "\n",
    "# count song name but ignore duplication, and order the result\n",
    "print(data.groupby('singer_code').song_name.nunique().sort_values(ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4\n",
    "We have two datasets about music: [song](https://github.com/michiard/AML-COURSE/blob/master/data/song.tsv) and [album](https://github.com/michiard/AML-COURSE/blob/master/data/album.tsv).\n",
    "\n",
    "In the following questions, you **have to** use Pandas to load data and write code to answer these questions.\n",
    "\n",
    "\n",
    "#### Question 4.1\n",
    "<div class=\"alert alert-info\">\n",
    "Load both dataset into two dataframes and print the information of each dataframe\n",
    "\n",
    "**HINT**: \n",
    "\n",
    "- You can click button `Raw` on the github page of each dataset and copy the URL of the raw file.\n",
    "- The dataset can be load by using function `read_table`. For example: `df = pd.read_table(raw_url, sep='\\t')`\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "import pandas as pd\n",
    "\n",
    "songdb_url = 'https://raw.githubusercontent.com/DistributedSystemsGroup/Algorithmic-Machine-Learning/master/data/song.tsv'\n",
    "albumdb_url = 'https://raw.githubusercontent.com/DistributedSystemsGroup/Algorithmic-Machine-Learning/master/data/album.tsv'\n",
    "song_df = pd...\n",
    "album_df = pd...\n",
    "\n",
    "print(song_df...)\n",
    "print(album_df...)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 4.2\n",
    "<div class=\"alert alert-info\">\n",
    "How many albums in this datasets ?\n",
    "\n",
    "How many songs in this datasets ?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "print(\"number of albums:\", album_df....count())\n",
    "print(\"number of songs:\", song_df.Song...)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 4.3\n",
    "<div class=\"alert alert-info\">\n",
    "How many distinct singers in this dataset ?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "print(\"number distinct singers:\", len(...))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 4.4\n",
    "<div class=\"alert alert-info\">\n",
    "Is there any song that doesn't belong to any album ?\n",
    "\n",
    "Is there any album that has no song ?\n",
    "\n",
    "**HINT**: \n",
    "\n",
    "- To join two datasets on different key names, we use `left_on=` and `right_on=` instead of `on=`.\n",
    "- Funtion `notnull` and `isnull` help determining the value of a column is missing or not. For example:\n",
    "`df['song'].isnull()`.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "fulldf = pd.merge(song_df, album_df, how='outer', left_on='Album', right_on='Album code')\n",
    "fulldf[fulldf['Song'].... & fulldf['Album']....]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "fulldf[fulldf['Song'].... & fulldf['Album code']....]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 4.5\n",
    "<div class=\"alert alert-info\">\n",
    "How many songs in each albums of Michael Jackson ?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "\n",
    "# Try thinking like as for map reduce word count!!\n",
    "\n",
    "fulldf[fulldf['Singer']=='Michael Jackson']....\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. PySpark\n",
    "\n",
    "Spark is an alternative framework to Hadoop MapReduce, designed to make it easier and quicker to build and run distributed data manipulation algorithms. Spark comes with a library for machine learning (MLLib) and graph algorithms, and also supports real-time streaming and SQL syntax, via Spark Streaming and SparkSQL, respectively. Spark exposes the Spark programming model to Java, Scala, or Python. In Python, we use the PySpark API to interact with Spark.\n",
    "\n",
    "As discussed in the CLOUDS lectures, every Spark application has a Spark driver. It is the program that declares the transformations and actions on RDDs of data and submits such requests to the cluster manager. Actually, the driver is the program that creates the `SparkContext`, connecting to a given cluster manager such as  Spark Master, YARN or others. The executors run user code, run computations and can cache data for your application. The `SparkContext` will create a job that is broken into stages. The stages are broken into tasks which are scheduled by the SparkContext on an executor.\n",
    "\n",
    "When starting PySpark with command `pyspark` or using a well configured notebook (such as this one), `SparkContext` is created automatically in variable `sc`. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <div>\n",
       "            <p><b>SparkContext</b></p>\n",
       "\n",
       "            <p><a href=\"http://10.0.0.25:4040\">Spark UI</a></p>\n",
       "\n",
       "            <dl>\n",
       "              <dt>Version</dt>\n",
       "                <dd><code>v2.2.0</code></dd>\n",
       "              <dt>Master</dt>\n",
       "                <dd><code>spark://spark-master0-1677-prod2:7077</code></dd>\n",
       "              <dt>AppName</dt>\n",
       "                <dd><code>pyspark-shell</code></dd>\n",
       "            </dl>\n",
       "        </div>\n",
       "        "
      ],
      "text/plain": [
       "<SparkContext master=spark://spark-master0-1677-prod2:7077 appName=pyspark-shell>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PySpark uses PySpark RDDs which  are just RDDs of Python objects: like Python lists, they can store objects with mixed types (actually all the objects are instances of `PyObject`).\n",
    "\n",
    "When PySpark is started, it also starts a JVM, which is accessible through a socket. PySpark uses `Py4J` to handle this communication. The JVM works as the actual Spark driver, and loads a `JavaSparkContext` that communicates with the Spark executors across the cluster. Python API calls to the Spark Context object are then **translated into Java API calls** to the JavaSparkContext. For example, the implementation of PySpark's `sc.textFile()` dispatches a call to the `.textFile` method of the `JavaSparkContext`, which ultimately communicates with the Spark executor JVMs to load the text data from HDFS. \n",
    "\n",
    "![](http://i.imgur.com/YlI8AqEl.png)\n",
    "\n",
    "The Spark executors on the cluster start a Python interpreter for each core, with which they communicate data through a pipe when they need to execute user-code. A Python RDD in the local PySpark client corresponds to a `PythonRDD` object in the local JVM. The data associated with the RDD actually lives in the Spark JVMs as Java objects. For example, running `sc.textFile()` in the Python interpreter will call the `JavaSparkContexts` `textFile` method, which loads the data as Java String objects in the cluster.\n",
    "\n",
    "\n",
    "When an API call is made on the `PythonRDD`, any associated code (e.g., Python lambda function) **is serialized and distributed to the executors**. The data is then converted from Java objects to a Python-compatible representation (e.g., pickle objects) and streamed to executor-associated Python interpreters through a pipe. Any necessary Python processing is executed in the interpreter, and the resulting data is stored back as an RDD (as pickle objects by default) in the JVMs. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is read easily by using functions of Spark Context. For example, to read a text file and count the number of lines, we can write:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of lines in the input file is: 745613\n"
     ]
    }
   ],
   "source": [
    "# each line is stored as an element in 'input_file' - a PythonRDD.\n",
    "input_file = sc.textFile(\"/datasets/gutenberg/gutenberg_tiny.txt\")\n",
    "num_lines = input_file.count()\n",
    "print(\"The number of lines in the input file is:\", num_lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.0. Preliminaries: working with HDFS and using the Apache Spark Web UI to inspect jobs, tasks and many more metrics\n",
    "\n",
    "Alright, so we have run our first Apache Spark program, and we learned how many lines it contains. But can we obtain some more information about this file? This is what we are going to do next.\n",
    "\n",
    "As a side note, the reason why we are interested in learning more about our file is that it allows us to understand how the Apache Spark job operates on it. Specifically, we want to learn about its tasks, their performance, and many more interesting metrics and figures, such as for example the Directed Acyclic Graph representing the job. This kind of exercise is highly connected to the content we learned in class, and it's a good way to put in practice some notions that would otherwise remain only theoretical.\n",
    "\n",
    "Since Apache Spark jobs generally operate on an input file, and that the way this file is stored determines to a large extent the degree of parallelism we can achieve with our Spark program, let's first focus on a few commands to interrogate the HDFS Name Node."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with HDFS\n",
    "A comfortable way to work with HDFS, albeit a bit geeky, is to use the command line. The nice thing about your Jupyter environment is that you can launch a terminal, which will give you access to one of the containers running your environment (the one where the Jupyter Notebook is running).\n",
    "\n",
    "Next, we will look at our home directory in HDFS. Open up a terminal, or alternatively use some magic functions from the Notebook, and type the following command:\n",
    "\n",
    "```\n",
    "! hdfs dfs ls\n",
    "```\n",
    "\n",
    "NOTE: you'll probably land on an empty directory, as this is the first time you're using our system."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll have a look at the directory we've used for the simple line count example from above:\n",
    "\n",
    "```\n",
    "! hdfs dfs ls /datasets/gutenberg\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3.1\n",
    "<div class=\"alert alert-info\">\n",
    "Using the hdfs command, inspect the file we've used before, located in /datasets/gutenberg/gutenberg_tiny.txt\n",
    "</div>\n",
    "\n",
    "\n",
    "* How many HDFS blocks constitute our file?\n",
    "* Where are they located?\n",
    "* What is the replication factor of our file?\n",
    "\n",
    "**Hint**: you should lookup for the HDFS command line documentation to learn how to answer the questions above. Look for the sub-command called ```fsck```."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with the Apache Spark Web UI\n",
    "Next, we'll spend some time trying to understand the execution of our job. This requires a good grip on the notions you've learned in class, so you're invited to revise the lecture on Apache Spark.\n",
    "\n",
    "The Apache Spark Web UI can be used to dissect the life of a job execution.\n",
    "\n",
    "* Using the link available from the endpoints that Zoe created for you, open the Web UI: to be precise, we will use the Spark Mater Web UI, which gives you a broad view of your cluster, executors, and jobs.\n",
    "* You'll see information about the worker machines of your cluster (number of cores, available memory, ...), and a list of applications. You'll see one running application, corresponding to the pyspark shell connected to your Jupyter Notebook. Click on its Application ID to inspect all the jobs this application has run.\n",
    "* You will land on a general page with a summary of your pyspark shell application. Follow the *Application Detail UI*.\n",
    "\n",
    "If this is the first time you execute the Notebook cell with the simple \"line counting\" job, you will see a single item in the job list. Pay attention to the job name: it corresponds to the name of the **Action** that fired the job, that is, the ```count``` action. At this level, you only see a coarse job summary: its submission time, its duration, and so on.\n",
    "\n",
    "* Click on the job description. You will land on a page with a great deal of details about your job, starting with its stages (remember, jobs are made of stages, that are in turn made of tasks).\n",
    "* Expand the sections about the **Event Timeline** and the **DAG visualization**.\n",
    "* Next, click on the job stage with the same name as the **Action** ```count``` we specified in our code\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3.2\n",
    "<div class=\"alert alert-info\">\n",
    "Using the Apache Spark Web UI, as explained above, answer the following questions:\n",
    "</div>\n",
    "\n",
    "* How many tasks were launched?\n",
    "* What is the duration of each task?\n",
    "* Hown many input bytes where processed by each task? How does this relate to the input file size?\n",
    "* How many workers in total does your cluster have? How many workers were involved in your \"line count\" job?\n",
    "* Given that we know how many HDFS blocks compose our input file (from Question 3.1), explain the number of tasks your job is broken into.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1. Wordcount example\n",
    "In the example below, we are interested in the top-10 words in terms of frequency of occurrence. To do so, we use a small text file as an input, and we wish to plot the term frequency of such top-10 words using Matplotlib.\n",
    "\n",
    "First, using the method `textFile` from the SparkContext `sc`, we create a RDD of strings. Each string in the RDD is representative for a line in the text file. In a loose way, we can think the first RDD is a RDD of lines. \n",
    "\n",
    "Because we work on the scope of words, we have to transform **a line** of the current RDD into **multiple words**, each word is an object of the new RDD. This is done by using `flatMap` function. \n",
    "\n",
    "Then, a `map` function transforms **each word** in the RDD into **a single** tuple with 2 components: the word itself and the count of 1. As you might have guessed, this is a PairRDD, where each object is a key-value pair. \n",
    "\n",
    "We can take advantage of function `reduceByKey` to sum all frequencies of the same word. Now, each element in the RDD is in the form of: (word, total_frequency). To sort the words by frequency of occurrence, we can use many approaches. One of the simplest approach is swap each tuple such that the frequency becomes the key, and use the `sortByKey` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "words = (\n",
    "            # read the text file\n",
    "            sc.textFile(\"/datasets/gutenberg/gutenberg_small.txt\").repartition(16)\n",
    "            \n",
    "            # construct words from lines\n",
    "            .flatMap(lambda line: line.split())\n",
    "            \n",
    "            # map each word to (word, 1)\n",
    "            .map(lambda x: (x, 1))\n",
    "    \n",
    "            # reduce by key: accumulate sum the freq of the same word\n",
    "            .reduceByKey(lambda freq1, freq2: freq1 + freq2)\n",
    "            \n",
    "            # swap (word, freq) to (freq, word)\n",
    "            .map(lambda x: (x[1], x[0]))\n",
    "    \n",
    "            # sort result by key DESC\n",
    "            .sortByKey(False)\n",
    "         )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3.3\n",
    "<div class=\"alert alert-info\">\n",
    "Using both the terminal and the Apache Spark Web UI, as explained above, answer the following questions:\n",
    "</div>\n",
    "\n",
    "* What is the size of the input file?\n",
    "* How many blocks the input file is broken into? What is the block size?\n",
    "* How many tasks execute in parallel?\n",
    "* How many jobs were launched upon the execution of the code above? Why (note that there are no actions!)?\n",
    "* What does a **\"skipped\"** stage mean?\n",
    "* What is the number of shuffled bytes? How does this compare to the number of input bytes?\n",
    "* Do you think Spark is doing a good job in balancing the load across the workers? What can go wrong with load balancing?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the top-10 words are collected and sent back to the driver by using function `take`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(29556020, 'the'), (18043957, 'of'), (14914748, 'and'), (12892052, 'to'), (9779124, 'a'), (8568854, 'in'), (5143447, 'that'), (4824456, 'was'), (4156700, 'I'), (3915358, 'with')]\n"
     ]
    }
   ],
   "source": [
    "# top 10 words:\n",
    "top10 = words.take(10)\n",
    "print(top10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The action `collect` gathers all elements of the RDD (that reside on multiple machines) to the driver (which is running in a single machine), and cast it as a list.\n",
    "\n",
    "**ATTENTION** collecting an RDD in the driver can be problematic: indeed, an RDD can be very big in size (this is why they are distributed across several machines in the first place!) and thus it could deplete the RAM available in the machine running the driver!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# collect results from executors to the driver\n",
    "# results = words.collect()\n",
    "# If you want to have a look at the results, don't print them all, for otherwise the notebook size will be too big"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that there are two kinds of functions in Spark: **transformations** and **actions**. All functions `map`, `flatMap`, `reduceByKey`, `sortByKey` are transformation functions. They are not executed right away when called. Indeed, Spark is lazily evaluated, so nothing gets executed unless the driver invokes actions such as `count`, `take`, `collect`...\n",
    "\n",
    "RDD transformations allow us to create dependencies between RDDs. Each RDD in the lineage chain (string of dependencies) has a function for calculating its data and has a pointer (dependency) to its parent RDD. Every time we use an RDD, dependencies are computed again from the beginning, which can be costly. Fortunately, we can use the function `cache` to instruct Spark to checkpoint in RAM (but eventually also on disk) a particular RDD."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now use the previous results from the execution of our simple Spark word count job, to plot word frequency information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEACAYAAACpoOGTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH4dJREFUeJzt3XmUVdWd9vHvA4U40YA4AwoqICjGEqf1ascyEITEgaQd\nMGlBIIlvTEdbk3TQtVrJ2DGJbUyn41p5BSwHiEajiYkDOJTG1UGigiJDgzYglIKIglOrIL/3j33K\nusK1Jqrq3Lr1fNa6q07te86pXyHy3L332ecoIjAzM9tel7wLMDOz0uSAMDOzohwQZmZWlAPCzMyK\nckCYmVlRDggzMyuqwYCQ1F/So5IWS3pe0iVZ+zRJayUtyF5jC465QtIKScskjS5oHyFpUfbe9QXt\n3SXdnrXPk3RwwXsTJS3PXhNa91c3M7OGqKF1EJL2B/aPiIWS9gSeBsYB5wJvRcS/b7f/MGAWcBzQ\nF3gIGBQRIWk+8E8RMV/SfcAvI+IBSRcDR0bExZLOA74QEeMl7QX8DRiRnf5pYEREbGrF39/MzD5B\ngz2IiFgXEQuz7beBpaR/+AFU5JCzgNkRsSUiVgEvACdIOgDoERHzs/1uJgUNwJlAdbZ9FzAy2z4N\nmBMRm7JQmAuMaebvZ2ZmLdTkOQhJA4BKYF7W9E1Jz0qaLqlX1nYgsLbgsLWkQNm+vZb6oOkLrAGI\niK3AZkl9GjiXmZm1gyYFRDa8dCdwadaTuAEYCBwNvAJc22YVmplZLioa20FSN9LQz60RcQ9ARLxa\n8P6NwL3Zt7VA/4LD+5E++ddm29u31x1zEPCypAqgZ0RslFQLVBUc0x94pEh9vpmUmVkLRESxqYKP\nNHYVk4DpwJKI+EVB+wEFu30BWJRt/xEYL2kXSQOBQcD8iFgHvCnphOycFwB/KDhmYrZ9NvBwtj0H\nGC2pl6TewGeBBz/hlyyp19VXX517DR2hplKtyzW5ps5QV1M01oM4CfhH4DlJC7K2K4HzJR0NBLAS\nuCj7h3qJpDuAJcBW4OKor+Ri4CZgN+C+iHgga58O3CJpBbARGJ+d63VJPyBdyQTwvfAVTGZm7abB\ngIiIJyjey7i/gWN+DPy4SPvTwPAi7e+TLpstdq6ZwMyGagRYuxb69WtsLzMza46yWEl90015V/Bx\nVVVVeZewg1KsCUqzLtfUNK6p6Uq1rsY0uFCuI5AUAwcGL7wAXcoi7szM2p4kYmcmqTuKHj2gpibv\nKszMyktZBMSUKTB9et5VmJmVl7IYYnrtteDQQ2HlSujdO++KzMxKX6cZYurTB8aMgVmz8q7EzKx8\nlEVAQBpmuvHGvKswMysfZRMQI0fCG2/AM8/kXYmZWXkom4Do0gUmTfJktZlZaymLSeq63+Gll6Cy\nMq2s3m23nAszMythnWaSus5BB8Gxx8Lvf593JWZmHV9ZBQR4TYSZWWspqyEmgPffTzfumzcPDj00\nx8LMzEpYpxtiAujeHb78ZZjZ6D1gzcysIWXXgwBYtAjGjoXVq6Fr15wKMzMrYZ2yBwEwfDgceCA8\nWPT5c2Zm1hRlGRDgyWozs51VlkNMAJs3w8EHw/LlsO++ORRmZlbCOu0QE0DPnjBuHNxyS96VmJl1\nTGUbEFA/zNTBO0lmZrko64A4+WT48MO0JsLMzJqnrANCgsmTPVltZtYSZTtJXWfdOhg6FNasgT33\nbMfCzMxKWKeepK6z//7w6U/DHXfkXYmZWcdS9gEBXhNhZtYSnSIgPvc5WLkSli7NuxIzs46jUwRE\nRQVMmAAzZuRdiZlZx1H2k9R1li9PcxFr1kC3bu1QmJlZCfMkdYHBg9PrT3/KuxIzs46h0wQEeLLa\nzKw5Os0QE8A770D//ul5EX37tnFhZmYlzENM29ljDzjnHKiuzrsSM7PS16l6EADz58P558OKFdCl\nU8WjmVk99yCKOO442H13eOyxvCsxMyttnS4gJE9Wm5k1RacbYgJ47TU47DBYtQp69WqbuszMStlO\nDzFJ6i/pUUmLJT0v6ZKsfS9JcyUtlzRHUq+CY66QtELSMkmjC9pHSFqUvXd9QXt3Sbdn7fMkHVzw\n3sTsZyyXNKElfwjF7L03jB4Ns2a11hnNzMpPY0NMW4DLIuII4ETgG5KGAlOBuRExGHg4+x5Jw4Dz\ngGHAGODXkuoS6gZgSkQMAgZJGpO1TwE2Zu3XAddk59oLuAo4PntdXRhEO8vDTGZmDWswICJiXUQs\nzLbfBpYCfYEzgbqLRauBcdn2WcDsiNgSEauAF4ATJB0A9IiI+dl+NxccU3iuu4CR2fZpwJyI2BQR\nm4C5pNBpFaNGwYYNsHBha53RzKy8NHmSWtIAoBJ4EtgvItZnb60H9su2DwTWFhy2lhQo27fXZu1k\nX9cARMRWYLOkPg2cq1V07QqTJrkXYWb2SSqaspOkPUmf7i+NiLfqR40gIkJSrjPd06ZN+2i7qqqK\nqqqqJh03aRIceyz87Gew665tU5uZWSmoqamhpqamWcc0GhCSupHC4ZaIuCdrXi9p/4hYlw0fvZq1\n1wL9Cw7vR/rkX5ttb99ed8xBwMuSKoCeEbFRUi1QVXBMf+CRYjUWBkRzDBgAlZVw991p8ZyZWbna\n/sPz9773vUaPaewqJgHTgSUR8YuCt/4ITMy2JwL3FLSPl7SLpIHAIGB+RKwD3pR0QnbOC4A/FDnX\n2aRJb4A5wGhJvST1Bj4LPNjob9RMnqw2MyuuwXUQkk4GHgeeA+p2vAKYD9xB+uS/Cjg3m0hG0pXA\nZGAraUjqwax9BHATsBtwX0TUXTLbHbiFNL+xERifTXAjaRJwZfZzfxgRO9xFqSXrIAq99x706wd/\n+xsMHNji05iZdShNWQfRKRfKbe+SS9KCue9/v5WKMjMrcQ6IJnr2WTj99LSyumvX1qnLzKyU+WZ9\nTfSpT8F++8HcuXlXYmZWOhwQGU9Wm5l9nIeYMps2pcteV6yAffbZ+brMzEqZh5iaoVcvOOMMuPXW\nvCsxMysNDogCdcNMHbxTZWbWKhwQBU45Ja2LmD+/8X3NzMqdA6KABJMne7LazAw8Sb2D2lo48khY\nuxb22KPVTmtmVlI8Sd0CffvCySfD736XdyVmZvlyQBThNRFmZh5iKmrLFujfHx57DIYMadVTm5mV\nBA8xtVC3bjBhAsyYkXclZmb5cQ/iEyxbBlVVsGZNCgwzs3LiHsROOPxwOOww+POf867EzCwfDogG\neLLazDozDzE14O2302T14sVw4IFt8iPMzHLhIaadtOeecPbZUL3Dg07NzMqfexCNmDcPLrgAli9P\nt+IwMysH7kG0ghNOgF12gccfz7sSM7P25YBohOTJajPrnDzE1AQbNsCgQbB6NfTs2aY/ysysXXiI\nqZXssw+MGgWzZ+ddiZlZ+3FANJGHmcyss3FANNHo0bBuHTz3XN6VmJm1DwdEE3XtChde6F6EmXUe\nnqRuhv/5n3TZ69q10L17u/xIM7M24UnqVnbIIXDUUXDPPXlXYmbW9hwQzeTJajPrLDzE1Ez/+7/Q\nrx888wwcfHC7/Vgzs1blIaY2sNtucP75MHNm3pWYmbUt9yBaYMECGDcuTVp37dquP9rMrFW4B9FG\nKiuhTx94+OG8KzEzazsOiBbyZLWZlTsPMbXQG2/AwIHw4oupN2Fm1pG0yhCTpBmS1ktaVNA2TdJa\nSQuy19iC966QtELSMkmjC9pHSFqUvXd9QXt3Sbdn7fMkHVzw3kRJy7PXhOb88m2td2/4/Ofh1lvz\nrsTMrG00ZYhpJjBmu7YA/j0iKrPX/QCShgHnAcOyY34tffQcthuAKRExCBgkqe6cU4CNWft1wDXZ\nufYCrgKOz15XS+rVwt+zTdQNM3XwTpiZWVGNBkRE/AV4o8hbxbomZwGzI2JLRKwCXgBOkHQA0CMi\n5mf73QyMy7bPBOqe+nwXMDLbPg2YExGbImITMJcdgypXVVXw9tvw1FN5V2Jm1vp2ZpL6m5KelTS9\n4JP9gcDagn3WAn2LtNdm7WRf1wBExFZgs6Q+DZyrZHTpApMne7LazMpTSwPiBmAgcDTwCnBtq1XU\nwVx4IdxxB7zzTt6VmJm1roqWHBQRr9ZtS7oRuDf7thboX7BrP9In/9pse/v2umMOAl6WVAH0jIiN\nkmqBqoJj+gOPFKtn2rRpH21XVVVRVVVVbLc20a8fnHgi3HknTJzYbj/WzKxZampqqKmpadYxTbrM\nVdIA4N6IGJ59f0BEvJJtXwYcFxFfyiapZ5EmlfsCDwGHRURIehK4BJgP/Bn4ZUQ8IOliYHhEfF3S\neGBcRIzPJqmfAo4hzXc8DRyTzUcU1pbLZa6F7roLrr8eHn881zLMzJqsKZe5NhoQkmYDpwB7A+uB\nq0mf7I8mXc20ErgoItZn+18JTAa2ApdGxINZ+wjgJmA34L6IuCRr7w7cAlQCG4Hx2QQ3kiYBV2al\n/DAi6iazC+vLPSA++CD1JJ54AgYPzrUUM7MmaZWAKHWlEBAA3/oWdOsGP/lJ3pWYmTXOAdGOliyB\nkSNhzRqoaNHMjplZ+/HN+trRsGEwYADcd1/elZiZtQ4HRCv6yle8JsLMyoeHmFrRW2/BQQel4aYD\nDsi7GjOzT+Y5iBxMmZLmIL72NRg6FHbfPe+KzMx25IDIwYsvwlVXwfPPw/Ll0LcvHHnkx1+DB8Mu\nu+RdqZl1Zg6InG3dCi+8kMKi8LV6NRx6aH1gHHFE+nrIIX6EqZm1DwdEiXrvPVi2rD4wFi9OX199\nFQ4/fMceR79+oAb/M5qZNY8DooN56600wb19j+Pdd+t7GYWvfffNu2Iz66gcEGXitdfqexl1Xxct\nSiu3tx+mOuII6FVSj1Uys1LkgChjEfDKKzv2NpYsSY9DLexpHHFEWsjnK6rMrI4DohPati1Ngm8f\nHCtXwumnp+dXjBzpyXCzzs4BYR/ZuBFmz4abboL162HChPT8Ct991qxzckBYUYsWQXU13Hprutz2\nwgvh3HOhZ8+8KzOz9uKAsAZt2QIPPggzZ8LDD8PnPw+TJsGpp3oIyqzcOSCsyV57rX4IasOG+iGo\nQYPyrszM2oIDwlrkuefqh6AGDaofgvq7v8u7MjNrLQ4I2ylbtsD996dexSOPwBlnpLA49VTo4hvF\nm3VoDghrNRs21A9BbdxYPwR12GF5V2ZmLeGAsDbx7LMpKG67DYYMSRPb55wDPXrkXZmZNZUDwtrU\nBx/UD0E9+iiceWYagqqq8hCUWalzQFi72bABZs1KYfH662n4aeLEtM7CzEqPA8JysXBhCopZs9JT\n9S68EM4+20NQZqXEAWG5+uADuO++FBY1NXDWWSksTjnFQ1BmeXNAWMl49dXUo5g5EzZvrh+COuSQ\nvCsz65wcEFaSCoeghg1LQeEhKLP25YCwkrb9EJSvgjJrPw4I6zBefbV+Id7rr3shnllbc0BYh/Ts\ns+leULfdlp5XMXGi7wVl1tocENah1d0Lqro63Y789NNTWHzmM74dudnOckBY2ai7HXl1tZ+IZ9Ya\nHBBWluqeiHfbbTBwYAqK886DXr3yrsys43BAWFnbujU9Ea+6GubMgbFjU1h89rMegjJrjAPCOo3X\nX4ff/jaFxdq1cMEFKSyGDs27MrPS5ICwTmnJkvon4vXrl9ZWjB8PvXvnXZlZ6XBAWKe2dSs89FBa\nW/HAAzB6dAqL0aOhoiLv6szy1ZSAaHS9qqQZktZLWlTQtpekuZKWS5ojqVfBe1dIWiFpmaTRBe0j\nJC3K3ru+oL27pNuz9nmSDi54b2L2M5ZLmtCcX96sogLGjElDTytXwsiR8IMfQP/+8J3vwPPP512h\nWWlryg0NZgJjtmubCsyNiMHAw9n3SBoGnAcMy475taS6hLoBmBIRg4BBkurOOQXYmLVfB1yTnWsv\n4Crg+Ox1dWEQmTVH795w0UXw17+m23p065bC49hj4Ve/So9RNbOPazQgIuIvwBvbNZ8JVGfb1cC4\nbPssYHZEbImIVcALwAmSDgB6RMT8bL+bC44pPNddwMhs+zRgTkRsiohNwFx2DCqzZhsyBH78Y1i9\nOn3961/Tg43+4R/g3nvT0JSZNa0HUcx+EbE+214P7JdtHwisLdhvLdC3SHtt1k72dQ1ARGwFNkvq\n08C5zFpF165pPuK221JYjB0LP/pRusNsdbWDwmyn75mZzRB7ltg6tJ494StfSb2J3/wmBcSQITB9\nerrrrFln1NJrOdZL2j8i1mXDR69m7bVA/4L9+pE++ddm29u31x1zEPCypAqgZ0RslFQLVBUc0x94\npFgx06ZN+2i7qqqKqqqqYruZNUpKtxuvqoInnkiT2j/4AUydCpMmQffueVdo1jI1NTXU1NQ065gm\nXeYqaQBwb0QMz77/KWli+RpJU4FeETE1m6SeRZpU7gs8BBwWESHpSeASYD7wZ+CXEfGApIuB4RHx\ndUnjgXERMT6bpH4KOAYQ8DRwTDYfUVibL3O1NjVvXgqJ556D73439TR23TXvqsx2Tqusg5A0GzgF\n2Js033AV8AfgDtIn/1XAuXX/cEu6EpgMbAUujYgHs/YRwE3AbsB9EXFJ1t4duAWoBDYC47MJbiRN\nAq7MSvlhRNRNZhfW54CwdvHUU/DDH8L8+eky2Ysugt13z7sqs5bxQjmzNrBwYQqKJ56Ayy+Hiy+G\nPffMuyqz5mmVhXJm9nFHHw133plWaS9YAIccki6XffPNvCsza10OCLMWOvLI9IyKxx+HpUvTWorv\nfx82bWr8WLOOwAFhtpMOPxxuuQX+679g1ar0HO1//VevzraOzwFh1koGDYIZM9Ik9vr16Wl3U6fC\nhg15V2bWMg4Is1Z2yCFpsd2CBfDWW2nB3be/DevW5V2ZWfM4IMzayEEHwX/+Z3pE6pYt6RYel14K\ntbV5V2bWNA4IszbWty9cf316kFG3bjB8OHzjG/DSS3lXZtYwB4RZO9l/f/j5z2HZMujRAyor4Wtf\nS8+qMCtFDgizdrbvvvCTn8Dy5bDffnDccTB5MrzwQt6VmX2cA8IsJ336pHs8rVgBBx8MJ54IF1yQ\nehhmpcABYZaz3r3h6qvhxRdh6FD49Kfh/PNh8eK8K7POzgFhViJ69oQrr0xBUVmZnqF99tnw7LN5\nV2adlW/WZ1ai3nknraf42c/SXEVlZboP1NFHw6c+lQLFrKV8N1ezMvD++6kXsXBhei1YkNZW7Ldf\nfWDUhUffvumhR2aNcUCYlakPP0yT23WhURcc27bVh0ZdcAweDBUtfXaklS0HhFknEpFu57FgwcdD\n4+WX4YgjPj5EddRRsMceeVdseXJAmBlvvZUel1oYHEuWpFuBFA5PHX10GrayzsEBYWZFbdmS1lvU\n9TLqgqN7948HRmVles5FF1/vWHYcEGbWZBHp/lCFw1MLF6bnWhx11MeD48gjYddd867YdoYDwsx2\n2htv7DgZvmIFnHRSugS3sjLvCq0lHBBm1ibeew+qq9MK8NNPhx/9yPMXHU1TAsIji2bWbLvuChdd\nlOYxevZMV0n99KdpzYaVDweEmbVYr15w7bXpedxPPJGC4p570nyGdXweYjKzVjNnDlx+eRpu+sUv\n0sORrDR5iMnM2tXo0Wki+4tfhFGj4Otfhw0b8q7KWsoBYWatqqIiPVJ16VLYZZf0LO7rroMPPsi7\nMmsuB4SZtYm99krP4n788TT0NHw4/OlPnp/oSDwHYWbt4v774bLL0tPzrrsu9SwsP56DMLOSMXZs\nuk355z4Hp5wC3/wmvP563lVZQxwQZtZuunWDSy9N8xPbtsHhh8N//Ee6N5SVHg8xmVlunn8+DTvV\n1qZhp9NOy7uizsO32jCzkheRJq8vvxyGDEkL74YMybuq8uc5CDMreRKccQYsXgynngonn5x6FW+8\nkXdl5oAws5Kwyy7wrW+loHj33TQ/ccMNsHVr3pV1Xh5iMrOS9Oyz8M//DK+9lm7bMXJk3hWVF89B\nmFmHFgF33w3f/nZ6aNHPfw6HHZZ3VeWhzecgJK2S9JykBZLmZ217SZorabmkOZJ6Fex/haQVkpZJ\nGl3QPkLSouy96wvau0u6PWufJ+ngnanXzDoWKd3XackSOPHE9PqXf4E338y7ss5hZ+cgAqiKiMqI\nOD5rmwrMjYjBwMPZ90gaBpwHDAPGAL+WVJdeNwBTImIQMEjSmKx9CrAxa78OuGYn6zWzDmjXXWHq\n1LTQbuPGdJXTjTfChx/mXVl5a41J6u27KGcC1dl2NTAu2z4LmB0RWyJiFfACcIKkA4AeETE/2+/m\ngmMKz3UX4FFIs07sgANg+vR0WWx1NRx7bLrXk7WN1uhBPCTpKUlfzdr2i4j12fZ6oO5BhAcCawuO\nXQv0LdJem7WTfV0DEBFbgc2S9trJms2sgxsxIgXDFVfAhAlwzjmwcmXeVZWfip08/qSIeEXSPsBc\nScsK34yIkNTmM8jTpk37aLuqqoqqqqq2/pFmljMJzj03raG49lo47jj40pfghBNg6NA0DLXHHnlX\nWTpqamqoqalp1jGtdhWTpKuBt4GvkuYl1mXDR49GxOGSpgJExE+y/R8ArgZWZ/sMzdrPBz4dEV/P\n9pkWEfMkVQCvRMQ+2/1cX8VkZtTWpuGnxYvTvZ5WrEhPths6NL0OP7x+e++98642f216mauk3YGu\nEfGWpD2AOcD3gFGkieVrslDoFRFTs0nqWcDxpKGjh4DDsl7Gk8AlwHzgz8AvI+IBSRcDw7OwGA+M\ni4jx29XhgDCzHXz4YRp2Wro0vZYtq9/u1q0+LAoDpH9/6NJJlg+3dUAMBO7Ovq0AbouIf8vmCO4A\nDgJWAedGxKbsmCuBycBW4NKIeDBrHwHcBOwG3BcRl2Tt3YFbgEpgIzA+m+AurMMBYWZNFgHr1tWH\nRWF4bN6chqa2D49DD00rvcuJF8qZmTXD5s0f72nUhcdLL8GAAcV7HXvumXfVLeOAMDNrBe+/n+Y0\nCoNj6VJYvjzNZxTOb9S99tknTaSXKgeEmVkb2rYNVq/eMTiWLk3hMHQoDBqUFvpJn/zq0qXh99vi\nuMsuc0CYmbW7CNiwof5qqg8+SG2f9Nq2reH32+LY6693QJiZWRF+YJCZmbWYA8LMzIpyQJiZWVEO\nCDMzK8oBYWZmRTkgzMysKAeEmZkV5YAwM7OiHBBmZlaUA8LMzIpyQJiZWVEOCDMzK8oBYWZmRTkg\nzMysKAeEmZkV5YAwM7OiHBBmZlaUA8LMzIpyQJiZWVEOCDMzK8oBYWZmRTkgzMysKAeEmZkV5YAw\nM7OiHBBmZlaUA8LMzIpyQJiZWVEOCDMzK8oBYWZmRTkgzMysKAeEmZkVVfIBIWmMpGWSVkj6bt71\nmJl1FiUdEJK6Ar8CxgDDgPMlDc23qsbV1NTkXcIOSrEmKM26XFPTuKamK9W6GlPSAQEcD7wQEasi\nYgvwW+CsnGtqVCn+ZSjFmqA063JNTeOamq5U62pMqQdEX2BNwfdrszYzM2tjpR4QkXcBZmadlSJK\n999gSScC0yJiTPb9FcC2iLimYJ/S/QXMzEpYRKih90s9ICqA/wZGAi8D84HzI2JproWZmXUCFXkX\n0JCI2Crpn4AHga7AdIeDmVn7KOkehJmZ5afUJ6k/USkuoJM0Q9J6SYvyrqWOpP6SHpW0WNLzki4p\ngZp2lfSkpIVZTdPyrqmOpK6SFki6N+9a6khaJem5rK75edcDIKmXpDslLZW0JJsvzLOeIdmfT91r\nc4n8Xb8s+zu+SNIsSd1LoKZLs3qel3Rpg/t2xB5EtoDuv4FRQC3wN0pgbkLS3wNvAzdHxPA8a6kj\naX9g/4hYKGlP4GlgXAn8We0eEe9m80xPAJdGxJN51pTVdTkwAugREWfmXQ+ApJXAiIh4Pe9a6kiq\nBh6LiBnZf8M9ImJz3nUBSOpC+nfh+IhY09j+bVhHX+AvwNCIeF/S7cB9EVGdY01HArOB44AtwAPA\n/42IF4vt31F7ECW5gC4i/gK8kXcdhSJiXUQszLbfBpYCB+ZbFUTEu9nmLkA3YFuO5QAgqR/wOeBG\noMGrO3JQMvVI6gn8fUTMgDRXWCrhkBkFvJhnOBSoAHbPQnR3UnDl6XDgyYh4LyI+BB4DvvhJO3fU\ngPACuhaQNACoBErhk3oXSQuB9cCciPhb3jUB1wHfoQTCajsBPCTpKUlfzbsYYCCwQdJMSc9I+n+S\nds+7qALjgVl5FxERtcC1wEukqzA3RcRD+VbF88DfS9or+2/2eaDfJ+3cUQOi442L5SwbXrqTNJTz\ndt71RMS2iDia9JfzBElH5FmPpNOBVyNiASX0aT1zUkRUAmOBb2RDmXmqAI4Bfh0RxwDvAFPzLSmR\ntAtwBvC7EqilN3AmMIDUa99T0pfzrCkilgHXAHOA+4EFNPCBqKMGRC3Qv+D7/qRehBUhqRtwF3Br\nRNyTdz2FsqGJR0k3ZMzT/wHOzMb7ZwOfkXRzzjUBEBGvZF83AHeThljztBZYW9Dru5MUGKVgLPB0\n9meVt1HAyojYGBFbgd+T/p7lKiJmRMSxEXEKsIk0n1tURw2Ip4BBkgZknxjOA/6Yc00lSZKA6cCS\niPhF3vUASNpbUq9sezfgs6S5kdxExJUR0T8iBpKGKB6JiAl51gRpMl9Sj2x7D2A0kOtVchGxDlgj\naXDWNApYnGNJhc4nBXwpWA2cKGm37P/DUcCSnGtC0r7Z14OAL9DAcFxJL5T7JKW6gE7SbOAUoI+k\nNcBVETEz57JOAv4ReE7Sgqztioh4IMeaDgCqs6vRugC3R8R9OdZTTKkMY+4H3J3+faECuC0i5uRb\nEgDfBG7LPqC9CEzKuZ66AB0FlMI8DRExX9KdwDPA1uzrb/KtCoA7JfUhXcV0cUS8+Uk7dsjLXM3M\nrO111CEmMzNrYw4IMzMrygFhZmZFOSDMzKwoB4SZmRXlgDAzs6IcEGZmVpQDwszMivr/o6AlkhyT\nfjkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f3d7a1e8f98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# extract the frequencies from the result\n",
    "frequencies = [x[0] for x in top10]\n",
    "\n",
    "# plot the frequencies\n",
    "plt.plot(frequencies)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2. Night flights example\n",
    "We have a CSV file which contains the information about flights that took place in the US in 1994.\n",
    "The data in this file has 29 columns such as `year`, `month`, `day_of_month`, `scheduled_departure_time`,...\n",
    "We can have a quick look on the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Year,Month,DayofMonth,DayOfWeek,DepTime,CRSDepTime,ArrTime,CRSArrTime,UniqueCarrier,FlightNum,TailNum,ActualElapsedTime,CRSElapsedTime,AirTime,ArrDelay,DepDelay,Origin,Dest,Distance,TaxiIn,TaxiOut,Cancelled,CancellationCode,Diverted,CarrierDelay,WeatherDelay,NASDelay,SecurityDelay,LateAircraftDelay\r\n",
      "1994,1,7,5,858,900,954,1003,US,227,NA,56,63,NA,-9,-2,CLT,ORF,290,NA,NA,0,NA,0,NA,NA,NA,NA,NA\r\n",
      "1994,1,8,6,859,900,952,1003,US,227,NA,53,63,NA,-11,-1,CLT,ORF,290,NA,NA,0,NA,0,NA,NA,NA,NA,NA\r\n",
      "1994,1,10,1,935,900,1023,1003,US,227,NA,48,63,NA,20,35,CLT,ORF,290,NA,NA,0,NA,0,NA,NA,NA,NA,NA\r\n",
      "1994,1,11,2,903,900,1131,1003,US,227,NA,148,63,NA,88,3,CLT,ORF,290,NA,NA,0,NA,0,NA,NA,NA,NA,NA\r\n",
      "1994,1,12,3,933,900,1024,1003,US,227,NA,51,63,NA,21,33,CLT,ORF,290,NA,NA,0,NA,0,NA,NA,NA,NA,NA\r\n",
      "1994,1,13,4,NA,900,NA,1003,US,227,NA,NA,63,NA,NA,NA,CLT,ORF,290,NA,NA,1,NA,0,NA,NA,NA,NA,NA\r\n",
      "1994,1,14,5,903,900,1005,1003,US,227,NA,62,63,NA,2,3,CLT,ORF,290,NA,NA,0,NA,0,NA,NA,NA,NA,NA\r\n",
      "1994,1,15,6,859,900,1004,1003,US,227,NA,65,63,NA,1,-1,CLT,ORF,290,NA,NA,0,NA,0,NA,NA,NA,NA,NA\r\n",
      "1994,1,17,1,859,900,955,1003,US,227,NA,56,63,NA,-8,-1,CLT,ORF,290,NA,NA,0,NA,0,NA,NA,NA,NA,NA\r\n",
      "cat: Unable to write to output stream.\r\n"
     ]
    }
   ],
   "source": [
    "! hdfs dfs -cat /datasets/airline/1994.csv | head -n 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this simple example, we are interested only in columns: `CRSDepTime` (scheduled departure time) and `UniqueCarrier` (carrier identifier). The values of `CRSDepTime` are expressed in the format: hhmm (hour-minute).\n",
    "Assume that a flight is considered as a 'night flight' if its scheduled departed time `CRSDepTime` is later than 18:00.\n",
    "\n",
    "Questions:\n",
    "\n",
    "- How many night flights do we have in our data ?\n",
    "- How many night flights per unique carrier ? Plot the top-5 of them, in terms of volume\n",
    "\n",
    "First, we read the data and remove the header. Then, from the lines, we extract the information of scheduled departure time and carrier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# read the data\n",
    "data = sc.textFile('/datasets/airline/1994.csv')\n",
    "\n",
    "# extract information about scheduled departure time and carrier\n",
    "# note that the scheduled time must be convert from string to interger number\n",
    "def extract_CRSDepTime_Carier(line):\n",
    "    cols = line.split(\",\")\n",
    "    return (int(cols[5]), cols[8])\n",
    "\n",
    "header = data.first()\n",
    "\n",
    "# remove header\n",
    "data_without_header = data.filter(lambda line: line != header)\n",
    "\n",
    "# create a new RDD with only scheduled departure time and carrier information\n",
    "# cache it for later usage\n",
    "newdata = (\n",
    "            data_without_header\n",
    "               .map(extract_CRSDepTime_Carier)\n",
    "               .cache()\n",
    "          )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Function `filter` helps us select only the objects that satisfy a condition. In this case, it creates a new RDD by filtering out the header. We can also use it to select the night flights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(2030, 'US'), (2030, 'US'), (2030, 'US')]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "night_flights = newdata.filter(lambda f: f[0] > 1800).cache()\n",
    "night_flights.take(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use `cache` because we don't want to recalculate `night_flights` from the beginning every time of using it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1078203\n"
     ]
    }
   ],
   "source": [
    "# filter and count the night flights\n",
    "num_night_flights = night_flights.count()\n",
    "print(num_night_flights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('DL', 208224), ('US', 170397), ('AA', 142832), ('WN', 124024), ('UA', 113640)]\n"
     ]
    }
   ],
   "source": [
    "# group by carrier\n",
    "night_flights_by_carrier = night_flights.groupBy(lambda x: x[1]).mapValues(lambda flights: len(flights))\n",
    "\n",
    "# take top 5 carriers\n",
    "top5_carriers = night_flights_by_carrier.takeOrdered(5, key=lambda x: -x[1])\n",
    "\n",
    "print(top5_carriers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use `groupBy` to cluster flights which belong to the same carrier into a group. In this example, to select the top-5 carriers, we use a method based on the function `takeOrder`. This function takes the top-`k` objects ordered by an index: the trick is that we instruct it to use the cumulative counts as the key.\n",
    "\n",
    "Let's plot a bar chart using Matplotlib. To draw a bar chart, we use function `bar` which requires two parameters. Each parameter is a list of float values in each dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEACAYAAABcXmojAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAE7xJREFUeJzt3XuM3eV95/H3p3HCshsUxzRLuJjLqs5u3CBx2UIu1XYq\nBDhqFxwtG0y3weq6F8WbEmW30kKqjY2ymy2VUkpahUpbEi7bUFBpE1CJsSGZ3WQTcIKgMXUIRsG7\ntgGDTEwSrdTFzXf/OI8fjgZ7ZhjP+MyM3y/paJ7z/V3meWZGv8/5/Z7zO5OqQpIkgJ8adQckSfOH\noSBJ6gwFSVJnKEiSOkNBktQZCpKkbtJQSLI8yVeT/G2SJ5Jc0+obk+xO8lh7vH9om+uS7EjyZJJL\nhurnJ9nWlt00VD8uyV2t/nCSM4aWrU3yVHtcPbtDlyRNlMnuU0jyduDtVfV4kjcDjwKrgQ8CP6qq\nP5iw/krgC8DPAacCDwIrqqqSbAU+UlVbk9wPfKaqNiVZD7yrqtYnuRL4QFWtSbIM+BZwftv9o8D5\nVbV/FscvSRoy6ZlCVT1fVY+39o+B7zI42APkEJtcDtxZVa9U1U7gaeDCJCcDJ1TV1rbe7QzCBeAy\n4LbWvge4qLUvBTZX1f4WBFuAVa9zfJKk12HacwpJzgTOBR5upd9O8jdJbkmytNVOAXYPbbabQYhM\nrO/h1XA5FdgFUFUHgJeTnDjJviRJc2RaodAuHf0F8NF2xnAzcBZwDvAc8Ok566Ek6ahZMtUKSd7I\n4LLOf6+qLwJU1QtDy/8UuK893QMsH9r8NAav8Pe09sT6wW1OB55NsgR4S1XtS7IHGBvaZjnwlUP0\nzw9vkqQZqKrXTANM9e6jALcA26vqD4fqJw+t9gFgW2vfC6xJ8qYkZwErgK1V9TzwwyQXtn1+CPjS\n0DZrW/sK4KHW3gxckmRpkrcCFwMPHGZgi/axYcOGkffB8Tk2x7f4Hocz1ZnC+4BfBb6T5LFW+zhw\nVZJzgAKeAX6rHZy3J7kb2A4cANbXq999PXArcDxwf1VtavVbgDuS7AD2AWvavl5K8kkG70ACuL58\n55EkzalJQ6Gqvs6hzya+PMk2nwI+dYj6o8DZh6j/HYO3uB5qX58HPj9ZHyVJs8c7mue5sbGxUXdh\nTi3m8S3msYHjW6wmvXltIUhSC30MknS0JaFe70SzJOnYYihIkjpDQZLUGQqSpM5QkCR1hoIkqTMU\nJEmdoSBJ6gwFSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSpMxQkSZ2hIEnqDAVJUmcoSJI6Q0GS1BkK\nkqTOUJAkdYaCJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLUGQqSpM5QkCR1hoIkqTMUJEndklF3YDFI\nMuouzEhVjboLkuYZQ2HWLLQD7MIMMklzy8tHkqRu0lBIsjzJV5P8bZInklzT6suSbEnyVJLNSZYO\nbXNdkh1JnkxyyVD9/CTb2rKbhurHJbmr1R9OcsbQsrXtezyV5OrZHbokaaKpzhReAT5WVT8LvBv4\nd0neCVwLbKmqdwAPteckWQlcCawEVgGfzasX3G8G1lXVCmBFklWtvg7Y1+o3Aje0fS0DPgFc0B4b\nhsNHkjT7Jg2Fqnq+qh5v7R8D3wVOBS4Dbmur3Qasbu3LgTur6pWq2gk8DVyY5GTghKra2ta7fWib\n4X3dA1zU2pcCm6tqf1XtB7YwCBpJ0hyZ9pxCkjOBc4FHgJOqam9btBc4qbVPAXYPbbabQYhMrO9p\nddrXXQBVdQB4OcmJk+xLkjRHpvXuoyRvZvAq/qNV9aPht2BWVSUZ6VtvNm7c2NtjY2OMjY2NrC+S\nNB+Nj48zPj4+5XpThkKSNzIIhDuq6outvDfJ26vq+XZp6IVW3wMsH9r8NAav8Pe09sT6wW1OB55N\nsgR4S1XtS7IHGBvaZjnwlUP1cTgUJEmvNfEF8/XXX3/I9aZ691GAW4DtVfWHQ4vuBda29lrgi0P1\nNUnelOQsYAWwtaqeB36Y5MK2zw8BXzrEvq5gMHENsBm4JMnSJG8FLgYemKy/kqQjk8nuak3y88D/\nBL7Dq3dnXQdsBe5m8Ap/J/DBNhlMko8D/xY4wOBy0wOtfj5wK3A8cH9VHXx763HAHQzmK/YBa9ok\nNUl+Dfh4+77/uaoOTkgP97FGfWfuIOcW3s1ro/65SRqdJFTVa+5inTQUFgJDYaYMBelYdrhQ8I5m\nSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSpMxQkSZ2hIEnqDAVJUmcoSJI6Q0GS1BkKkqTOUJAkdYaC\nJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLUGQqSpM5QkCR1hoIkqVsy6g5o/ksy6i7MSFWNugvSgmMo\naJoW2gF2YQaZNGpePpIkdYaCJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLUGQqSpM5QkCR1hoIkqTMU\nJEndlKGQ5HNJ9ibZNlTbmGR3ksfa4/1Dy65LsiPJk0kuGaqfn2RbW3bTUP24JHe1+sNJzhhatjbJ\nU+1x9ewMWZJ0ONM5U/g8sGpCrYA/qKpz2+PLAElWAlcCK9s2n82rH7F5M7CuqlYAK5Ic3Oc6YF+r\n3wjc0Pa1DPgEcEF7bEiydIbjlCRNw5ShUFVfA35wiEWH+hjKy4E7q+qVqtoJPA1cmORk4ISq2trW\nux1Y3dqXAbe19j3ARa19KbC5qvZX1X5gC68NJ0nSLDqSOYXfTvI3SW4ZegV/CrB7aJ3dwKmHqO9p\nddrXXQBVdQB4OcmJk+xLkjRHZhoKNwNnAecAzwGfnrUeSZJGZkb/ZKeqXjjYTvKnwH3t6R5g+dCq\npzF4hb+ntSfWD25zOvBskiXAW6pqX5I9wNjQNsuBrxyqPxs3buztsbExxsbGDrWaJB2zxsfHGR8f\nn3K9TOdfFiY5E7ivqs5uz0+uquda+2PAz1XVr7SJ5i8wmBg+FXgQ+JmqqiSPANcAW4G/Bj5TVZuS\nrAfOrqoPJ1kDrK6qNW2i+dvAeQzmLx4FzmvzC8N9q1H/28XBXPrC+89k0/25LfbxSceiJFTVa+aG\npzxTSHIn8AvATyfZBWwAxpKcw+BI8QzwWwBVtT3J3cB24ACwfuiIvR64FTgeuL+qNrX6LcAdSXYA\n+4A1bV8vJfkk8K223vUTA0GSNLumdaYwn3mmMFOeKUjHssOdKXhHsySpMxQkSd2M3n0kLSav3nS/\nsHh5THPBUJCAhThnIs0FLx9JkjpDQZLUGQqSpM5QkCR1hoIkqTMUJEmdoSBJ6gwFSVJnKEiSOkNB\nktQZCpKkzlCQJHWGgiSpMxQkSZ2hIEnq/H8K0iK3EP+JkP9AaHQMBemYsJAOsgsvxBYTLx9JkjpD\nQZLUGQqSpM5QkCR1hoIkqTMUJEmdoSBJ6rxPQdKC5s15s8tQkLQIzN+D7GvN7xDz8pEkqTMUJEmd\noSBJ6gwFSVJnKEiSOkNBktRNGQpJPpdkb5JtQ7VlSbYkeSrJ5iRLh5Zdl2RHkieTXDJUPz/Jtrbs\npqH6cUnuavWHk5wxtGxt+x5PJbl6doYsSTqc6ZwpfB5YNaF2LbClqt4BPNSek2QlcCWwsm3z2bx6\nZ8nNwLqqWgGsSHJwn+uAfa1+I3BD29cy4BPABe2xYTh8JEmzb8pQqKqvAT+YUL4MuK21bwNWt/bl\nwJ1V9UpV7QSeBi5McjJwQlVtbevdPrTN8L7uAS5q7UuBzVW1v6r2A1t4bThJkmbRTOcUTqqqva29\nFziptU8Bdg+ttxs49RD1Pa1O+7oLoKoOAC8nOXGSfUmS5sgRTzTX4EM8FtI95pKkw5jpZx/tTfL2\nqnq+XRp6odX3AMuH1juNwSv8Pa09sX5wm9OBZ5MsAd5SVfuS7AHGhrZZDnzlUJ3ZuHFjb4+NjTE2\nNnao1STpmDU+Ps74+PiU62U6n9aX5Ezgvqo6uz3/fQaTwzckuRZYWlXXtonmLzCYGD4VeBD4maqq\nJI8A1wBbgb8GPlNVm5KsB86uqg8nWQOsrqo1baL528B5DD5B6lHgvDa/MNy3GvUnDg7m0hfayVKm\n/UmNjm8+Wszjm/7YYPGPb856kVBVr/l0vinPFJLcCfwC8NNJdjF4R9DvAXcnWQfsBD4IUFXbk9wN\nbAcOAOuHjtjrgVuB44H7q2pTq98C3JFkB7APWNP29VKSTwLfautdPzEQJEmza1pnCvOZZwoztZhf\naYLjG1pzwY3PM4Wj0ovDnCl4R7MkqTMUJEmdoSBJ6gwFSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSp\nMxQkSZ2hIEnqDAVJUmcoSJI6Q0GS1BkKkqTOUJAkdYaCJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLU\nGQqSpM5QkCR1hoIkqTMUJEmdoSBJ6gwFSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSpMxQkSZ2hIEnq\nDAVJUndEoZBkZ5LvJHksydZWW5ZkS5KnkmxOsnRo/euS7EjyZJJLhurnJ9nWlt00VD8uyV2t/nCS\nM46kv5KkyR3pmUIBY1V1blVd0GrXAluq6h3AQ+05SVYCVwIrgVXAZ5OkbXMzsK6qVgArkqxq9XXA\nvla/EbjhCPsrSZrEbFw+yoTnlwG3tfZtwOrWvhy4s6peqaqdwNPAhUlOBk6oqq1tvduHthne1z3A\nRbPQX0nSYczGmcKDSb6d5Dda7aSq2tvae4GTWvsUYPfQtruBUw9R39PqtK+7AKrqAPBykmVH2GdJ\n0mEsOcLt31dVzyV5G7AlyZPDC6uqktQRfo8pbdy4sbfHxsYYGxub628pSQvK+Pg44+PjU66Xqtk5\nZifZAPwY+A0G8wzPt0tDX62qf5bkWoCq+r22/iZgA/C/2zrvbPWrgH9RVR9u62ysqoeTLAGeq6q3\nTfi+NVtjmKnB1Mho+/D6hen+3BzffLSYxzf9scHiH9+c9SKhqiZe/p/55aMk/zDJCa39j4BLgG3A\nvcDattpa4IutfS+wJsmbkpwFrAC2VtXzwA+TXNgmnj8EfGlom4P7uoLBxLUkaY4cyeWjk4C/am8g\nWgL8WVVtTvJt4O4k64CdwAcBqmp7kruB7cABYP3QS/z1wK3A8cD9VbWp1W8B7kiyA9gHrDmC/kqS\npjBrl49GxctHM7WYLz+A4xtac8GNz8tHR6UXs335SJK0+BgKkqTOUJAkdYaCJKkzFCRJnaEgSeoM\nBUlSZyhIkjpDQZLUGQqSpM5QkCR1hoIkqTMUJEmdoSBJ6gwFSVJnKEiSOkNBktQZCpKkzlCQJHWG\ngiSpMxQkSZ2hIEnqDAVJUmcoSJI6Q0GS1BkKkqTOUJAkdYaCJKkzFCRJnaEgSeoMBUlSZyhIkjpD\nQZLUGQqSpM5QkCR18z4UkqxK8mSSHUn+46j7I0mL2bwOhSRvAP4YWAWsBK5K8s7R9upoGx91B+bY\n+Kg7MIfGR92BOTY+6g7MsfFRd2Ak5nUoABcAT1fVzqp6Bfhz4PIR9+koGx91B+bY+Kg7MIfGR92B\nOTY+6g7MsfFRd2Ak5nsonArsGnq+u9UkSXNgvodCjboDknQsSdX8Pe4meTewsapWtefXAT+pqhuG\n1pm/A5CkeayqMrE230NhCfA94CLgWWArcFVVfXekHZOkRWrJqDswmao6kOQjwAPAG4BbDARJmjvz\n+kxBknR0zfeJ5mNGkr9P8liSJ5I8nuTfJ0lbNpbkvlH3caaSnJlk24TaxiT/Icm7kzzSxr49yYZR\n9fNIJFmd5CdJ/umE+jmtfumo+jZTSW5M8tGh5w8k+W9Dzz+d5GNtfB8Zqv9xkrVHu78zMdnfZmsv\nSfJikv86mh4efYbC/PF/q+rcqnoXcDHwfmBBHiCn6eAp6q3Ar1fVucDPAnePrEdH5irga+3rdOoL\nwdeB9wIk+SngRAY3kR70HuAbwAvANUne2OoL/fLDcP8vZjCv+a9H1JejzlCYh6rqReA3gY9Mte4i\n8I+B5wFqYMHNGSV5M/A+4NeBNUP1AFcAvwZcnOS40fRwxr7J4MAPg8B+AvhRkqVtLO8EXgJeBB4C\nFsTZwTQdDIargJuA/5PkPZOsv2gYCvNUVT0DvCHJ20bdlzl2I/C9JH+Z5DcX4IETBnfZf7mqdgD7\nkpzX6u8Fvl9V32dwe+wvjah/M1JVzwIHkixnEA7fZPAOwPcA/xzYBvy/tvrvA7/TzigWhST/gME7\nH+8F7mRhnu29bovmF6h57XCXE6qqPsngALMZ+BVg01Hr1ey5Crirte/i1YPH4eoLyTcYhNt7GYTC\nN1v7PQwuLwH9RcwjDH6HC8lkl7p+GfhqVf0d8JfA6oPzfIvZvH5L6rEsyT8B/r6qXlwEf4f7gLdO\nqJ0IfB+gvZL+kzaJ+WKSt1bVD45yH2ckyTLgF4F3tRsp3wD8pH2i778CLkvyu0CAZUneXFU/Hl2P\nX7f/xeDS2NkMzgx2Ab8DvAx8jsG4DvoU8BfA/zjKfTwSh/rbXAY8wyDE35fkmaH6RcCDR697R59n\nCvNQu2T0J8Afjbovs6EdBJ9L8ovQD6SXAl9P8ktDr77eARwA9o+mpzNyBXB7VZ1ZVWdV1enATuB3\ngcer6vRWP5PBq80PjK6rM/INBq+Y97U5nx8AS3l1krmrqu8B24F/yQKZbD7M3+Yq4HHg54Hl7fd3\nFoM5voV4tve6GArzx/EH35IKbAE2VdX1bVkBFyXZNfS4cHRdnZGrgf+U5DEGk5Ib2yWHXwWebPXb\ngX9TC+vmmTXAX02o3QOcdZj6GhaWJxic1T08VPsOsL+qXmrPh39f/wU47Sj1bba85m8TOAd4qH06\n80H3Ar889C6rRcmb1yRJnWcKkqTOUJAkdYaCJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLU/X+UzNO2\n3YI5gQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f3d78ea3710>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# extract the number of flights which will be used as y-values\n",
    "num_flights = [ x[1] for x in top5_carriers]\n",
    "\n",
    "# extract the carriers' names\n",
    "carrier_names = [x[0] for x in top5_carriers]\n",
    "\n",
    "# create `virtual indexes for carriers which will be used as x-values`\n",
    "carrier_indexes = range(0, len(carrier_names))\n",
    "\n",
    "# plot\n",
    "plt.bar(carrier_indexes, num_flights, align=\"center\")\n",
    "\n",
    "# put x-labels for the plot\n",
    "plt.xticks(carrier_indexes, carrier_names)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 5\n",
    "\n",
    "\n",
    "### Question 5.1\n",
    "<div class=\"alert alert-info\">\n",
    "Compute how many flights have a scheduled departure time later than 09:00 and before 14:00.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# read the data\n",
    "data = sc.textFile('/datasets/airline/1994.csv')\n",
    "\n",
    "# extract information about scheduled departure time and carrier\n",
    "# note that the scheduled time must be convert from string to interger number\n",
    "def extract_CRSDepTime_Carier(line):\n",
    "    ...\n",
    "    ...\n",
    "    return (int(cols[5]), cols[16])\n",
    "\n",
    "header = data.first()\n",
    "\n",
    "# remove header\n",
    "data_without_header = data.filter(...)\n",
    "\n",
    "# create RDD with only scheduled departure time and carrier information\n",
    "# cache it for later usages\n",
    "newdata = (\n",
    "            data_without_header\n",
    "               .map(extract_CRSDepTime_Carier)\n",
    "               ...\n",
    "          )\n",
    "\n",
    "flights = newdata.filter(...).cache()\n",
    "\n",
    "print(flights.count())\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 5.2\n",
    "<div class=\"alert alert-info\">\n",
    "Compute the cumulative number flights that have a scheduled departure time after 09:00 and before 14:00, for each source airport (origin). Plot the top-5 of such airports.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "flights_per_carrier = flights.groupBy(...).mapValues(...)\n",
    "\n",
    "# take top 5 source airports\n",
    "top5_source_airport = flights_per_carrier.takeOrdered(...)\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# extract the number of flights which will be used as y-values\n",
    "# This is called list comprehension\n",
    "num_flights = [ x[1] for x in top5_source_airport]\n",
    "\n",
    "# create `virtual indexes for carriers which will be used as x-values`\n",
    "airport_indexes = range(0, len(top5_source_airport))\n",
    "\n",
    "# plot\n",
    "plt.bar(airport_indexes, num_flights, align=\"center\")\n",
    "\n",
    "# extract the carriers' names\n",
    "airport_names = [ x[0] for x in top5_source_airport]\n",
    "\n",
    "# put x-labels for the plot\n",
    "plt.xticks(airport_indexes, airport_names)\n",
    "plt.show()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary\n",
    "\n",
    "In this lecture, we gained familiarity with the Jupyter Notebook environment, the Python programming language and its modules. In particular, we covered the Python syntax, Numpy - the core library for scientific computing, Matplotlib - a module to plot graphs, Pandas - a data analysis module. Besides, we started to gain practical experience with PySpark, using, for the moment, small data sets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References\n",
    "This notebook is inspired from:\n",
    "\n",
    "- [Python Numpy tutorial](http://cs231n.github.io/python-numpy-tutorial/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}