Chapter 2 - The basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 2: The basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-- *A Python Course for the Humanities by Folgert Karsdorp and Maarten van Gompel*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Type casting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous chapter you have seen that each variable as a data type. Sometimes it is necessary to convert one type into the other. Consider this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = \"5\"\n",
    "y = 5\n",
    "print(x == y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you see why these are not equal? *x* is an string, and *y* is an integer. Because of this, the following will result in an error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(x + y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What would we have expected? 10 or 55? Addition of integers or concatenation of strings? We can do both, but we need to explicitly convert one type to the other:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = \"5\"\n",
    "y = 5\n",
    "print(\"String concatenation: \" + x + str(y))\n",
    "print(\"Integer addition: \" + str(int(x) + y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how we convert the result of the integer addition to a string so we can print it! \n",
    "\n",
    "Other types of conversions are possible as well, an interesting one is the fact that lots of types can be converted to a boolean value (True or False). Any non-zero integer will convert to True, zero to False. Any non-empty string or list will be True, an empty one will be False. You can use `bool()` to do such a conversion explicitly, but in conditions like if statements this will be done implicitly. We have seen in the last chapter that we can do things such as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fruits = ['banana','apple','pear']\n",
    "if fruits:\n",
    "    print(\"We have \" + str(len(fruits)) + \" fruits\")\n",
    "else:\n",
    "    print(\"We have no fruits :'(\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Quiz!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Edit the example above, initialise fruits to an empty list and see what happens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# insert your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## String Manipulation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have already seen several forms of string manipulation, such as concatenation using the `+` operator and the use of indices and splices with the `[]` operator. We will now look at some further ways in which strings can be manipulated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `split` method is very convenient in splitting a string into a list. It takes one parameter, the character to split on. Without a parameter, `split` splits a string on spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fruitstring = \"banana,pear,apple\"\n",
    "fruitlist = fruitstring.split(',')\n",
    "print(fruitlist)\n",
    "print(fruitlist[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split the following quote by Alan Turing into a list of words:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quotestring = \"A computer would deserve to be called intelligent if it could deceive a human into believing that it was human\"\n",
    "# insert your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, count the number of two-letter words in this list. (Tip: think about the loops you learnt in the previous chapter). Do you see why making this a list makes it easier?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numberoftwoletterwords = 0\n",
    "quotelist = quotestring.split()\n",
    "for word in quotelist:\n",
    "    # insert your code here \n",
    "print(numberoftwoletterwords == 5) # This will be True if you've done it right"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### join()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reverse of `split()` function can be accomplished with `join()`, it turns a list into a string, with a specific delimiter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fruitlist = ['banana', 'pear', 'apple']\n",
    "delimiter = \",\"\n",
    "fruitstring = delimiter.join(fruitlist)\n",
    "print(fruitstring)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above four lines can be accomplished in a single line if code, can you figure out how? (Tip: replace all variables by their values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# insert your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### find() *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to find a certain substring in a string, you can use `find()` on the string. It will return the index at which the substring was found, if the substring is not found it will return `-1`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"We spend our time searching for security and hate it when we get it\"\n",
    "found = text.find('security')\n",
    "\n",
    "if found == -1:\n",
    "    print(\"No security found! :(\")\n",
    "else:\n",
    "    print(\"Security found at position \" + str(found))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read a text file and see if a specific keyword can be found in the data. We will explain more about reading text files later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# insert your code here\n",
    "keyword = \"mistress\"\n",
    "keywordfound = False\n",
    "f = open('data/austen-emma-excerpt.txt','rt', encoding='utf-8')\n",
    "for line in f:\n",
    "    # insert your code here\n",
    "        \n",
    "f.close()\n",
    "if keywordfound:\n",
    "    # insert your code here\n",
    "else:\n",
    "    # insert your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### replace()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `replace()` function replaces occurrences of a specified substring with another.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"You can not compare apples and pears\"\n",
    "text = text.replace(\"pears\", \"apples\")\n",
    "text = text.replace(\"not\", \"\")\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take out all the vowels using the `.replace()`.`"
   ]
  },

  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"Research has shown that it is often still possible to understand text even if all vowels are removed\"\n",
    "# insert your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### strip() and newlines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did you notice the double space in the previous example? Whilst this one occurs in the middle. It often happens that a string has extra leading or trailing spaces that you want to get rid of. The `strip()` function does this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = \"   strip me!    \"\n",
    "print(s.strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also strip any other characters, the following will strip spaces and exclamation marks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(s.strip(\" !\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When no parameters are passed, `strip()` applies to spaces, tabs, and newline characters. A newline character is internally represented as `\"\\n\"` (that is one character) and visualised as if enter was pressed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = \"This is the first line.\\nThis is the second line.\"\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Manipulate `s` (use the existing version) so the two sentences are not on separate lines but on just one line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# insert your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### lower() and upper() - Changing Case"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python has two functions for changing the case of a string. `lower()` turns a string to lower case and `upper()` returns an upper-case version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "teachers = \"Folgert Karsdorp and Maarten van Gompel\"\n",
    "print(teachers.lower())\n",
    "print(teachers.upper())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Count the number of times some form of *bla*, regardless of case, occurs. Making use of one of the two functions you just learnt makes this job easier!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = [\"BlA\", \"plop\", \"blA\", \"Bla\", \"PLoP\", \"bLa\", \"pLop\", \"BLA\"]\n",
    "# insert your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iterables, Iteration & Loops"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuples & Sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lists and dictionaries are hugely important data structures and you will see a lot of them. They are almost always combined with the power of iteration using either **for** loops or other methods.\n",
    "\n",
    "In addition to these data structures, there are two others which should be mentioned, these are *tuples* and *sets*. Lists, sets, tuples, dictionaries and even strings are often called *iterables*, as they are all collections over which can be iterated. \n",
    "\n",
    "Tuples are ordered collections like lists, but they are *immutable*. Once created, nothing can be added, inserted, deleted or substituted. The syntax for creating a tuple is `()`. Example:\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fruittuple = ('banana','apple','pear')\n",
    "print(fruittuple[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following would have worked for lists, but does not for tuples (nor strings as we have seen before):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fruittuple[0] = 'orange' # will raise an error, tuples are immutable!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A set is an **unordered** data collection in which an each element can only occur once, elements can be appended or deleted. It is unordered so this implies that you never really know in what order you get the elements when you iterate over the set. The syntax for creating a set is `{}`. this is not to be confused with dictionaries, those take `key: value`  pairs where sets just take single elements.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fruitset = {'banana', 'apple', 'pear'}\n",
    "fruitset.add('banana') # will have no effect, banana already exists\n",
    "print(fruitset)\n",
    "fruitset.add('orange')\n",
    "print(fruitset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the order of the elements maybe different from the order you initialised the `fruitset`. This is because order has no meaning in the context of sets and neither in that of dictionaries.\n",
    "\n",
    "In the previous chapter you learnt how to convert strings that contains numbers to integers, and how to turn integers and floats to strings. Such a kind of type casting can also be done for iterables, allowing you to turn almost any iterable into any other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fruitlist = ['banana', 'apple', 'pear']\n",
    "fruitset = set(fruitlist)\n",
    "fruittuple = tuple(fruitlist)\n",
    "fruitlist = list(fruitset)\n",
    "print(fruitset)\n",
    "print(fruittuple)\n",
    "print(fruitlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### enumerate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Various functions exists to aid in iteration over any iterable. One such function is `enumerate()`, for each item it returns, it will also return its index number, the first element always receiving index `0`. When using enumarate in a for loop, you have to catch both these variables as shown in the example below.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Advanced background:** enumerate() acts as if it returns a list of 2-tuples, where for each 2-tuple the first element is the index number, and the second element the item itself. By specifying multiple variables in a for loop, the tuple is automatically *unpacked*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, fruit in enumerate(fruittuple):\n",
    "    print(\"Fruit #\" + str(i+1) + \" is a \" + fruit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did you see that output? \"A apple?\" That's not proper English! Correct the script so it uses \"an\" when the word starts with a vowel.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vowels = (\"a\",\"i\",\"e\",\"o\",\"u\")\n",
    "# insert your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### zip() *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This has nothing to do with file archives (zip files). The zip() function is used to join together two elements from separate lists of the same length. Just visualise a zipper and the way the two sides magically come together whenever you close your jacket, each chain in the zipper former a nice pair with its counterpart.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Advanced background:** zip() acts as if it returns a list of 2-tuples, where for each 2-tuple the first element is the element from the first list, and the second element comes from the second list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fruits = ['banana', 'apple', 'pear']\n",
    "prices = [0.89, 0.34, 0.44]\n",
    "for fruit, price in zip(fruits, prices):\n",
    "    print(fruit + \"s, now only \" + str(price) + \" each!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we use two lists, but we can also rewrite this whole code to use one single data structure, very suited for such a task. Try!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fruits = {'banana':0.89, 'apple':0.34,'pear': 0.44}\n",
    "for fruit,price in fruits.items():\n",
    "    print(fruit + \"s, now only \" + str(price) + \" each!\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### sorted() and reverse()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often you want to iterate over your data in a sorted manner. The `sorted()` function will take any iterable and return the elements in sorted order. For strings this is alphabetical order, for numbers this is numerical order.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fruits = ['banana', 'apple', 'pear']\n",
    "for fruit in sorted(fruits):\n",
    "    print(fruit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reverse order is also possible by using `reversed()`, which simply returns the elements in any iterable in reverse order:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for fruit in reversed(fruits):\n",
    "    print(fruit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reversed can be applied to strings just as well. But as this function returns something that evaluates to a list, not a string, we need some extra instrument to glue together all letters in reverse and make it a proper string. The `join()` statement we saw earlier is very suited for this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fruit = \"banana\"\n",
    "print(\"\".join(reversed(fruit)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given a list of words, output only the ones that are palindromes. As you know, a palindrome is a word that does not change when read backwards. For extra challenge, output the palindromes in alphabetic order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = ['bicycle', 'radar', 'origin', 'tie', 'level', 'poop', 'solar', 'nun']\n",
    "# insert your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bonus Exercise *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are also words which are not palindromes, but when reversed another existing word emerges. Consider the words *stressed* and *desserts*. We give you a text that contains a few of them, output all pairs that occur in the text (regardless of case), but exclude words of length one.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"I just live for desserts , I really love them . My dog does too . I saw he ate mine . I was very stressed because of that . If dogs steal desserts God can't be real , for it is pure evil .\" \n",
    "# insert your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the punctuation in the text we provided is detached from the words? This is tokenisation and made the exercise easier. Tokenisation is a very necessary first step in most NLP applications. We will get back to it later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### min(), max() and sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When dealing with lists of numbers, there are three functions which come in handy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numbers = [1, 2, 3, 4, 5]\n",
    "print(min(numbers))\n",
    "print(max(numbers))\n",
    "print(sum(numbers))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the average of `numbers`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# insert your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List comprehensions *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "List comprehensions are a powerful tool in Python to quickly manipulate lists. A list comprehension can replace a more explicit **for** loop on a single line. Consider the following example: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numbers = [1, 2, 3, 4, 5]\n",
    "doublenumbers = [number*2 for number in numbers]\n",
    "print(doublenumbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is not the same as the normal **for** loop you are hopefully accustomed to by now, but similar. The entire expression always takes place within square brackets `[]` , indicating it is a list we want. Anything you can do with list comprehensions you can also do without, but they add a certain elegance and are a quick way of working."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numbers = [1, 2, 3, 4, 5]\n",
    "# insert your code here, it must be on one line and must be \n",
    "# a list comprehension using numbers. If you did well, the following\n",
    "# will output True. Study this outcome to find out what's asked of you here!\n",
    "\n",
    "print(numbers == [3, 4, 5, 6, 7])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing external modules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the things we have seen thus far are all built in. Often however you need to use external modules in your code. A lot of external modules are available in the Python Standard Library, for a wide variety of tasks. There are also countless third-party providers of Python modules. \n",
    "\n",
    "To use an external module in your code you need to explicitly import it. Consider for example the module `random` from the standard library, which contains functions for generating random numbers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "print(random.randint(0, 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can import an entire module or import only a specific function in the module. We could have also imported as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from random import randint\n",
    "print(randint(0, 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can even change the names of the functions you import:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from random import randint as gimme_a_random_number_please\n",
    "print(gimme_a_random_number_please(0, 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## While loop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is also another form of loop in Python, and most other programming languages. This is the `while` loop which will run as long the specified condition is satisfied."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "number = 0\n",
    "while number != 42:\n",
    "    number = random.randint(0, 100)\n",
    "    print(number)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## File Input/Output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Input for your programs often comes from files on disk, such as corpora. Likewise, you often want output to be written back to files on disk as well. Reading and writing files is often an essential part of programming. The following example reads a file from disk:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open('data/austen-emma-excerpt.txt', 'rt', encoding='utf-8')\n",
    "text = f.read()\n",
    "f.close()\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we read the entire text into a single variable. In it you will notice things such as newlines, previously discussed in the string. \n",
    "\n",
    "First we discuss the three arguments to open():\n",
    "\n",
    " * the filename to open\n",
    " * the mode, a combination of characters, 'r' represents read-mode, and 't' represent plain text-mode. This indicates we are reading a plain text file.\n",
    " * the last argument, a named argument (encoding), specifies the encoding of the text file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may wonder what an encoding is and what *utf-8* is, for anyone working with texts and computers this is vital to know. Internally, a computer knows no letters whatsoever, all are represented as numbers (which in turn are represented in binary as zeroes and ones). An encoding specifies which numbers represent which letters. A famous ancient encoding scheme is ASCII, in which for example the letter 'A' is 65. ASCII however has only very limited spaces and can not encode any other writing systems. A modern-day encoding supporting countless writing systems is unicode, utf-8 is a form of a unicode. This is what we use for our data. Whenever you have a choice, you should use unicode as well!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reading an entire file in one string is not always desireable, especially not with huge files. The following example reads up until a newline everytime, and returns one line at a time. The newline is included which each line, which is why you will extra empty lines in the output!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open('data/austen-emma-excerpt.txt','rt', encoding='utf-8')\n",
    "for line in f:\n",
    "    print(line)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Quiz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adapt the example above so the newline character is stripped off. You may only change the print line.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# insert your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rather than just printing it, we can of course do whatever we want, let's count the number of lines. Do note that a line need not correspond with a sentence!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "count = 0\n",
    "f = open('data/austen-emma-excerpt.txt', 'rt', encoding='utf-8')\n",
    "# insert your code here\n",
    "f.close()\n",
    "print(count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Quiz »"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the file `data/austen-emma-excerpt.txt` and compute the average length (in words) over all lines. Do not worry about tokenisation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open('data/austen-emma-excerpt.txt', 'rt', encoding='utf-8')\n",
    "# insert your code here\n",
    "    \n",
    "print(round(avlength,2) == 6.74)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- - -"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we mastered the art of reading files, let's move on to write files, which follows a similar structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open('data/testouput.txt', 'wt', encoding='utf-8')\n",
    "f.write(\"Hello world!\")\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want data on multiple lines, you need to take care to explicitly encode the newlines. Instead of:\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open('data/testouput.txt','wt', encoding='utf-8')\n",
    "f.write(\"Hello world!\")\n",
    "f.write(\"Hello world!\")\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You need to write:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open('data/testoutput.txt','wt', encoding='utf-8')\n",
    "f.write(\"Hello world!\\n\")\n",
    "f.write(\"Hello world!\")\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Otherwise your file would have `Hello world!Hello world!` in it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides read-mode and write-mode, there is also append-mode. Write-mode would overwrite any existing file, append-mode adds to the end of the file. Use 2nd argument 'at' for append mode."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise »"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the file `data/austen-emma-excerpt-tokenised.txt` , and write to a file `words.txt` all words occuring in this text (without duplicates!!), with one word per line. Thus effectively creating a lexicon based on this text. (Tip: use sets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open('data/austen-emma-excerpt-tokenised.txt','rt',encoding='utf-8')\n",
    "# insert your code here\n",
    "f.close()\n",
    "\n",
    "f = open('words.txt','wt',encoding='utf-8')\n",
    "# insert your code here\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check your output by viewing the `words.txt` file in a text editor such as Sublime Text 2. (Windows users: do not use Notepad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this chapter you have already seen lots of functions. A function does something, often based on some input you pass to it, and generally returns a result. You are not just limited to using functions available built in, in the standard library, or by external parties, but you can also write your own functions.\n",
    "\n",
    "In fact, you *must* write your own functions. Separating your problem into sub-problems and writing a function for each of those is an immensely important part of well-structured programming.\n",
    "\n",
    "We start with a trivial function, functions are defined using the `def` keyword, they take a name and optionally a number of parameters. \n",
    "\n",
    "    def some_name(optional_parameters):\n",
    "\n",
    "The `return` statement returns a value back to the caller and always ends the execution of the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def multiply(x, y):\n",
    "    return x * y\n",
    "\n",
    "z = multiply(2, 5)\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now something more substantial, this function counts the number of articles (the, a, an) in a list of words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_articles(tokens):\n",
    "    count = 0\n",
    "    for token in tokens:\n",
    "        if token.lower() in ('the', 'a', 'an'):\n",
    "            count += 1\n",
    "    return count\n",
    "   \n",
    "text = \"A bit less trivial , this function counts the number of articles ( the , a , an ) in a list of words\"\n",
    "print(count_articles(text.split(\" \")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also have a function return multiple values by combining them in a tuple, Python has a nice way of 'unpacking' such a tuple in assignment. For **return** and assignment, the usual parentheses associated with tuples are not required. We have seen a similar construction in the **for** loop with `enumerate()` and `zip()`. Consider the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count(text):\n",
    "    words = text.split(\" \")\n",
    "    word_count = len(words)\n",
    "    character_count = len(text)\n",
    "    return word_count, character_count\n",
    "\n",
    "word_count, character_count = count(\"To be or not to be , that is the question .\")\n",
    "print(word_count)\n",
    "print(character_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Variable scope"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Any variables you declare in a function, as well as the parameters that are passed to a function. Will only exist within the scope of that function. The following will thus produce an error, because the variable does not exist outside the function:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def setx():\n",
    "    x = 1\n",
    "\n",
    "setx()\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also consider this:\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = 0\n",
    "def setx():\n",
    "    x = 1\n",
    "setx()\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = 0\n",
    "def setx(x):\n",
    "    x = 1\n",
    "setx(x)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is however possible to read a global variable from within a function, in a strictly read-only fashion. But as soon as you assign something, the variable will be a local copy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = 1\n",
    "def getx():\n",
    "    print(x)\n",
    "    \n",
    "getx()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write a function `calculate_average(numbers)`, taking one parameter, a list (or other iterable) containing only numbers (you may assume that), and returning the average/mean:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your function calculate_average here\n",
    "\n",
    " \n",
    "\n",
    "# do not modify the code below, for testing only\n",
    "numbers = [1, 2, 3, 4, 5]\n",
    "av = calculate_average(numbers)\n",
    "assert av == 3 # should be True if you did it correctly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bonus Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write a function `textstats(filename)` that, given a filename to load return a three-tuple with the following general statistics on the file: number of lines, number of words/tokens, number of characters. Don't worry about tokenisation. Test it on a text file.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your function textstats here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ignore this, it's only here to make the page pretty:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.display import HTML\n",
    "def css_styling():\n",
    "    styles = open(\"styles/custom.css\", \"r\").read()\n",
    "    return HTML(styles)\n",
    "css_styling()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "by Folgert Karsdorp (Meertens Instituut) en Maarten van Gompel (Radboud University Nijmegen)\n",
    "\n",
    "Licensed under the [GNU Free Documentation License](http://www.gnu.org/copyleft/fdl.html)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}