Lectures/Week-03-Accumulators.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Handy vizualization tool:  http://www.pythontutor.com/visualize.html#mode=edit\n",
    "\n",
    "# Week 3:  Acumulators and numbers\n",
    "\n",
    "## Announcements\n",
    "\n",
    "* There is a \"quiz\" in moodle to collect your optional due dates for the essays.  Remember that there is the default due date, but you can shift things plus or minus one week.  The quiz in an easy way for us to collect that info.  The new due dates will still be on Saturdays.  Once you select a due date in there, you will be held to that due date.  The quiz closes Sept 22, when the default due dates will lock in if you haven't made a selection.\n",
    "* Essay 1's prompt is ready for you to review, the other two are really close.  Remember that there are 3 prompts and your lowest is dropped, so you can skip this one if you want.  However, please believe me when I say that you don't want to have to be writing an essay near the end of the semester.\n",
    "* The David Beazley extra credit is also coming up due later this month, and you can complete this at anytime between now and the due date.  Note that late submissions for this extra credit will not be accepted for grading.  He'll do a lot of things that will be over your head, but you'll recognize pieces of everything.  This is a fun assignment where you got to see Python used in context for a project.  Pay attention to the patterns we've been talking about, and watch for them.  Even if you don't know exactly what's going on with the code, you should see these patterns.\n",
    "\n",
    "Some general reminders and points on things for this week:\n",
    "\n",
    "* I will often put in extra requirements into the homework text in moodle, and these will be graded requirements. \n",
    "* Even a narrative only answer may still need you to run and deal with code.\n",
    "* Please read all the hints for each problem that I've given in the homework problems, and reference back to them as you have questions.\n",
    "* Please start to test your programs that take in inputs.  I've given you some reminders to test your code with many inputs, but I won't have that there the entire semester.\n",
    "\n",
    "This week's chapter really focuses on numbers, both generating numbers, manipulating their contents, and converting them to suit your needs.  The only math we'll be doing is addition and division.  Nothing more.  Try to focus on understanding the code within the chapter and largely ignore the math that it's trying to talk about.  Along with numbers, the chapter introduces very important pattern that will be in many of your programs:  accumulators and counters.  This is where I'll be focusing my lecture on.  You will still be strongly needing the book with other content.   Any math that you need to know for your homework is explained in the hints area.  \n",
    "\n",
    "As you are thinking of accumulators and counters, keep them separate in your head.  They're going to look really similar in the code, but their purpose and usage will be very different!  You can have multiple of both inside you program, but commonly you'll have an accumulator to collect up many values, and then a counter to keep track of how many things you have counter.  There is an argument to be made that counters are a special type of accumulator, but I prefer to think of them as siblings rather than parent/child.\n",
    "\n",
    "We'll be talking about thinking through the application of many of these tools.  Not just how to generate the numbers that you want but how to think through solving problems with numbers and integrating those solutions into larger needs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The two patterns:  accumulators vs counters\n",
    "\n",
    "You'll note that the words much and many aren't specific here.  So when you are thinking about these questions, think about the complete question.\n",
    "\n",
    "In particular, counting questions can be tricky here.  A counter literally has the name in there, but you may still want to use an accumulator.\n",
    "\n",
    "The first question that you need to ask is:  what am I summing here?  Are you counting a series of uniform values?  Or are you counting up a series of different values?  The variability of the values is a big distinction here.\n",
    "\n",
    "For example, we have a class of 24 students.  \n",
    "\n",
    "**We can count the number of students.**  Each person here counts as \"1\".  This is a uniform value across all entities.  This is the core of a counter pattern.  Inside the loop you'll see that the incrementing value is often hard coded. \n",
    "\n",
    "**We can sum up the total age.** Each person has some age, which may repeat but is variable across everyone in here.  This value must be accessed each time through, and that value is added up to create a final value representing the total number of years old all the students are in this room.  \n",
    "\n",
    "We can use them together.  If I know a total sum of a property for a set of entities, and the total number of entities, what does that sound like to you?\n",
    "\n",
    "This is how we calculate the mean or average value.  So if we've got a loop counting these two things together, we can calculate some summary statistics about it.  \n",
    "\n",
    "Can you think of some examples of things you might want to count up versus things you might want to sum up? \n",
    "\n",
    "So that's what we can do and how we will be mixing these up together.  Let's take a step back and look at some code for making these things happen.  \n",
    "\n",
    "\n",
    "# Minimal counter\n",
    "\n",
    "A minimal counter will be the easiest to see this mechanism in action.\n",
    "\n",
    "\n",
    "## Counting up by a set number\n",
    "\n",
    "For this task, we want to count how many times a loop executes.  This isn't a trick or anything fancy.  \n",
    "\n",
    "Remember our steps to a for loop?  We need:\n",
    "\n",
    "1. determine your pattern\n",
    "2. identify your sequence\n",
    "3. give your iterable a good name\n",
    "\n",
    "Answering questions 1 and 2 here often go in circles.  Pick a pattern, think about how it'll work with the sequences available to you, and either rethink or continue forward.  As you build up more experience, you'll know what you need faster and begin to have a set of patterns that you are the most comfortable with.  As you are getting started, take note of the patterns that make the most sense to you and latch onto those.\n",
    "\n",
    "Let's start with an range loop, because what we need is to make a for loop that can execute some number of times so that we can count how many times that is.  True, we can run a counter in every type of for loop out there, but let's go with a range loop to start with.\n",
    "\n",
    "### Review a base range loop\n",
    "\n",
    "``` python\n",
    "for number in range(5):\n",
    "    print(\"hello\")\n",
    "```\n",
    "\n",
    "This block of code will print \"hello\" five times.  \n",
    "\n",
    "We're going to start with this base, and change it so that we are counting how many times this runs.  We know it will be 5 times, but out end goal will be to have a variable holding this value.\n",
    "\n",
    "\n",
    "### There are 4 steps to a counter\n",
    "\n",
    "Here are the 4 steps:\n",
    "\n",
    "1. Establish you base variable at the initial state (usually 0)\n",
    "2. Start a for loop for whatever you're repeating\n",
    "3. Do your stuff, whatever you need to do in that loop\n",
    "4. Update your counter, usually by 1.  This is commonly the last thing in your loop.\n",
    "\n",
    "The reference pattern is:\n",
    "\n",
    "```python \n",
    "counter = 0 # 1: your counter variable at the initial state\n",
    "\n",
    "for something in sequence: # 2: loop over something!\n",
    "    do stuff # 3: do all the things\n",
    "    counter = 1 + counter # 4: update the counter, replace 1 and/or the operator with whatever you need\n",
    "```\n",
    " \n",
    "My nitpick about the Zelle formula is that it leaves out the `do stuff` item that goes between your for loop and incrementing up your counter variable.  You are often having to do more than just update your counter variable, and many times you need that value in the process.  This means that you'll need to increment up your counter as the last thing you do in your loop. There will be exceptions to this, but they are rare.  More generally, all your incrementers should be at the end of your for loop, and the internal order doesn't always matter.\n",
    "\n",
    "This particular example is counting up by 1, but you can always change what that number is or what the operator is.  For example, if you were counting up by 10, you would have 10 in step 4.\n",
    "\n",
    "\n",
    "#### Worked example of a counter\n",
    "\n",
    "\n",
    "##### Step 1: have a base\n",
    "\n",
    "So let's start adapting things.  We need to start with a base.\n",
    "\n",
    "``` python\n",
    "count = 0\n",
    "```\n",
    "\n",
    "##### Step 2: have a for loop in there\n",
    "\n",
    "Now we're going to add in our for loop.\n",
    "\n",
    "``` python\n",
    "count = 0\n",
    "\n",
    "for number in range(5):\n",
    "    print(\"hello\")\n",
    "```\n",
    "You see that I haven't changed up the for loop's content, we'll get to that next.\n",
    "\n",
    "##### Step 3: do stuff to get the count that you need in that for loop\n",
    "\n",
    "Very often with counters you won't need to do anything here, because you're just brute force counting by a certain number each time.  But this is when you'd address any decision structures or data extraction methods to get out the value that you need to count by.\n",
    "\n",
    "``` python\n",
    "count = 0\n",
    "\n",
    "for number in range(5):\n",
    "    # you'd do stuff here\n",
    "    # or maybe you have more counters do do\n",
    "    # either way, all the fun stuff not counter related goes right after the for loop starts\n",
    "    print(\"hello\") # and likely print it out here instead of \"hello\"\n",
    "```\n",
    "\n",
    "##### Step 4:  increment up your counter\n",
    "\n",
    "Now that we've got all our pieces in order and we've done our business, we can finally update our counter.  This is the last step because this is ofter the last thing your for loop.  One of the most common problems that I see is when the increment is put at the beginning of the for loop or in the middle of the action, and everything gets weird.  Almost always it will need to be the last line of code within your for loop.\n",
    "\n",
    "``` python\n",
    "count = 0\n",
    "\n",
    "for number in range(5):\n",
    "    count = count + 1 #here's the update\n",
    "    \n",
    "print(base)\n",
    "```\n",
    "\n",
    "Note our use of `base = base + 1` here.  We are updating a variable that already exists (this is why we define a base), and since we want to reuse it and have the value retained even when the for loop resets, we need to a) keep that base on the outside of our for loop and b) reuse the variable name.  So that repetition of the base variable name is purposeful and absolutely necessary.  \n",
    "\n",
    "What would happen if we didn't include the `print()` there at the end?  (nothing would have printed out)\n",
    "\n",
    "What would happen if the `print(base)` was indented inside the for loop? (we would see a print out for it each time).  Would we see count with the base value of 0?  Would we see count with the final value of 5?  Why?\n",
    "\n",
    "\n",
    "# Minimal Accumulator:  summing variable/non-uniform values\n",
    "\n",
    "\n",
    "Think like snow measurement, we say that there is an \"accumulation of 3 inches\".  You don't know how big the individual time moments are (like if it snowed heavily at once or if it was a slow trickle), but you do know the end total of how much snow ended up there.  You can also measure the amount incrementally over time, but once something is added in you can't know how big that individual piece is.\n",
    "\n",
    "An accumulator is highlighted by the incremental value going in being variable in size.  Think of this like a running total where you are only keeping track of the current value, and once an individual value is added in, that value is lost.  \n",
    "\n",
    "## There are 5 steps to an accumulator\n",
    "\n",
    "Here are the 5 steps (they should look very familiar to you with the counter steps)\n",
    "\n",
    "1. Establish you accumulator base at the initial state (usually 0)\n",
    "2. Start a for loop for whatever you're repeating\n",
    "3. Do your stuff, whatever you need to do in that loop\n",
    "4. Determine your value to accumulate\n",
    "5. Update your base variable by whatever your incremental value is.  This is commonly the last thing in your loop.\n",
    "\n",
    "The reference pattern is:\n",
    "\n",
    "```python \n",
    "total_something = 0 # 1: your counter variable at the initial state\n",
    "\n",
    "for something in sequence: # 2: loop over something!\n",
    "    do stuff # 3: do all the things\n",
    "    value_you_want = ?? # 4: determine the value you want to add to your accumulator\n",
    "    total_something = value + total_something # 5: update the counter, replace 1 and/or the operator with whatever you need\n",
    "```\n",
    "\n",
    "## Worked example\n",
    "\n",
    "Let's sum up all the numbers produced in a range loop.  Yes, this is as silly example, but it keeps the clutter away so you can see the inner workings better.\n",
    "\n",
    "Reminder, here's our range loop printing out 10 numbers, starting at 0.  We will be adding an accumulator pattern within and around this for loop.\n",
    "\n",
    "``` python\n",
    "for num in range(10):\n",
    "    print(num)\n",
    "```\n",
    "\n",
    "\n",
    "### 1: add the base\n",
    "\n",
    "We want to start at 0 because we're counting like humans.\n",
    "\n",
    "``` python\n",
    "\n",
    "total_sum = 0  # 1\n",
    "\n",
    "for num in range(10):\n",
    "    print(num)\n",
    "```\n",
    "\n",
    "### 2 & 3: looping and doing things\n",
    "\n",
    "There's not much to do because we already have our loop set up for us.  Likewise, there's no transformation or lookup to do there.  \n",
    "\n",
    "``` python\n",
    "\n",
    "total_sum = 0 \n",
    "\n",
    "for num in range(10): # 2\n",
    "    # 3: you'd do stuff here to transform or extract the number that you need\n",
    "    print(num) \n",
    "```\n",
    "\n",
    "### 4: add the incrementing value\n",
    "\n",
    "In this case, we're counting how many times the loop has executed.  \n",
    "\n",
    "So how much do we want to count by each time?  1.\n",
    "\n",
    "And where do we need to put the incrementing expression?  When in doubt, put it at the end and see what happens. In our case, we aren't doing much of anything, so we can put it anywhere.\n",
    "\n",
    "Here's what the final form should look like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "45\n"
     ]
    }
   ],
   "source": [
    "total_sum = 0 # 1\n",
    "\n",
    "for num in range(10):\n",
    "    # we can get rid of the print now\n",
    "    total_sum = num + total_sum # 5\n",
    "    \n",
    "print(total_sum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# You can 'accumulate' more than numbers!!!!!\n",
    "\n",
    "Anything that can 'hold' stuff can be accumulated into.  You'll be using this pattern a lot moving forward.\n",
    "\n",
    "## Collecting letters\n",
    "\n",
    "When a data type has the ability to be 'added' to, either through the `+` operator or a method (lists do this, but we aren't there yet), you can use this same kind of pattern.  Instead of setting the counter to `0`, you have an 'empty' whatever-it-is.  Since we've been playing with strings lately, let's use that as an example.\n",
    "\n",
    "```python \n",
    "new_string = \"\" # 1 your empty string\n",
    "\n",
    "for letter in \"the quick brown fox jumped over the lazy dog\": # 2 loop over something!\n",
    "    do stuff # 3 maybe you're filtering for letter characters\n",
    "    new_string = new_string + letter # 4 update the new string to include this letter\n",
    "```\n",
    "\n",
    "Other than not having numbers, you'll notice just one difference in this example:  I'm using `new_string = new_string + letter` whereas the counter had `thingtoadd = 1 + thingtoadd`.  So whereas numercial addition can have any order (Commutative Property) there will be times that order does matter.  In this case, it does with string concatenation.\n",
    "\n",
    "`\"fizzy\" + \"pop\"` makes `\"fizzypop\"` and `\"pop\" + \"fizzy\"` makes `\"popfizzy\"`.  So the order that you decide to write the operation will determine what you get and should be part of your design process.  Most of the time you'll want to add new things on to the end, so:  `current + new`\n",
    "\n",
    "## Generalizing the accumulator pattern beyond numbers\n",
    "\n",
    "You should still keep your idea of the counter as something separate for now.  But we can come up with a more general pattern to serve you beyond basic operator (`+`) accumulation.\n",
    "\n",
    "This pattern will be used all over as we move forward in class.  What will change is what the collection item is (a string, a sum, a file, list, etc), what you're adding to the thing, and what you need to do to create the thing you want to add to your collection.  Here's a more abstract formula:\n",
    "\n",
    "\n",
    "```python \n",
    "collection_thing = ?? # declare an empty version of your collection thing \n",
    "# thing may also be opening an empty file to write out to\n",
    "\n",
    "for thing in sequence: # loop over something!\n",
    "    do stuff # transform, filter, etc\n",
    "    ??? # put your new data in your collection thing\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# common pitfalls\n",
    "\n",
    "We're going to switch gears a bit and go over some of the common errors that you'll see related to these patterns.\n",
    "\n",
    "## the accumulator value isn't adding or only has the last value\n",
    "\n",
    "This happens when your counter variable is placed inside your for loop and ends up starting over each run. It will still perform the operations each time, but your counter will be reset each time through.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the number is: 0\n",
      "the number is: 1\n",
      "the number is: 2\n",
      "the number is: 3\n",
      "the number is: 4\n",
      "the final result is: 4\n"
     ]
    }
   ],
   "source": [
    "for i in range(5):\n",
    "    print(\"the number is:\", i)\n",
    "    counter = 0\n",
    "    counter = counter + i\n",
    "    \n",
    "print(\"the final result is:\", counter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the pieces you need are in there, just in the wrong place.  I've fixed the code below by only moving the location of the `counter = 0` line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the number is: 0\n",
      "the number is: 1\n",
      "the number is: 2\n",
      "the number is: 3\n",
      "the number is: 4\n",
      "the final result is: 10\n"
     ]
    }
   ],
   "source": [
    "counter = 0 # look at me at the top now\n",
    "\n",
    "for i in range(5):\n",
    "    print(\"the number is:\", i)\n",
    "    counter = counter + i\n",
    "    \n",
    "print(\"the final result is:\", counter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## your output looks like a pyramid or shows a correct but incremental version of the final output\n",
    "\n",
    "This happens when your final result statement is inside your for loop, causing it to be executed each loop.  The output this produces a characteristic pyramid effect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the final string is: Ϩ\n",
      "the final string is: Ϩϩ\n",
      "the final string is: ϨϩϪ\n",
      "the final string is: ϨϩϪϫ\n",
      "the final string is: ϨϩϪϫϬ\n",
      "the final string is: ϨϩϪϫϬϭ\n",
      "the final string is: ϨϩϪϫϬϭϮ\n",
      "the final string is: ϨϩϪϫϬϭϮϯ\n",
      "the final string is: ϨϩϪϫϬϭϮϯϰ\n",
      "the final string is: ϨϩϪϫϬϭϮϯϰϱ\n"
     ]
    }
   ],
   "source": [
    "s = \"\"\n",
    "\n",
    "for i in range(1000, 1010):\n",
    "    s = s + chr(i) # don't worry about what's going on in here yet\n",
    "    print(\"the final string is:\", s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See how that result is growing?  The print statement is inside the for loop.  So we're seeing the correct result, but we're watching it grow.  This sort of thing may also cause odd errors when you are dealing with creating other kinds of data structures, where you are adding your incremental results to the full data stucture rather than collecting up a sub result and then adding that.  You'll likely still see a pyramid effect in there somewhere.  The solution is to move the print statement out of the for loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the final string is: ϨϩϪϫϬϭϮϯϰϱ\n"
     ]
    }
   ],
   "source": [
    "s = \"\"\n",
    "\n",
    "for i in range(1000, 1010):\n",
    "    s = s + chr(i)\n",
    "\n",
    "print(\"the final string is:\", s) # did you see me move to the left?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "raven = \"\"\"Once upon a midnight dreary, while I pondered, weak and weary,\n",
    "Over many a quaint and curious volume of forgotten lore—\n",
    "    While I nodded, nearly napping, suddenly there came a tapping,\n",
    "As of some one gently rapping, rapping at my chamber door.\n",
    "“’Tis some visitor,” I muttered, “tapping at my chamber door—\n",
    "            Only this and nothing more.”\n",
    "\n",
    "    Ah, distinctly I remember it was in the bleak December;\n",
    "And each separate dying ember wrought its ghost upon the floor.\n",
    "    Eagerly I wished the morrow;—vainly I had sought to borrow\n",
    "    From my books surcease of sorrow—sorrow for the lost Lenore—\n",
    "For the rare and radiant maiden whom the angels name Lenore—\n",
    "            Nameless here for evermore.\n",
    "\n",
    "    And the silken, sad, uncertain rustling of each purple curtain\n",
    "Thrilled me—filled me with fantastic terrors never felt before;\n",
    "    So that now, to still the beating of my heart, I stood repeating\n",
    "    “’Tis some visitor entreating entrance at my chamber door—\n",
    "Some late visitor entreating entrance at my chamber door;—\n",
    "            This it is and nothing more.”\n",
    "\n",
    "    Presently my soul grew stronger; hesitating then no longer,\n",
    "“Sir,” said I, “or Madam, truly your forgiveness I implore;\n",
    "    But the fact is I was napping, and so gently you came rapping,\n",
    "    And so faintly you came tapping, tapping at my chamber door,\n",
    "That I scarce was sure I heard you”—here I opened wide the door;—\n",
    "            Darkness there and nothing more.\n",
    "\n",
    "    Deep into that darkness peering, long I stood there wondering, fearing,\n",
    "Doubting, dreaming dreams no mortal ever dared to dream before;\n",
    "    But the silence was unbroken, and the stillness gave no token,\n",
    "    And the only word there spoken was the whispered word, “Lenore?”\n",
    "This I whispered, and an echo murmured back the word, “Lenore!”—\n",
    "            Merely this and nothing more.\n",
    "\n",
    "    Back into the chamber turning, all my soul within me burning,\n",
    "Soon again I heard a tapping somewhat louder than before.\n",
    "    “Surely,” said I, “surely that is something at my window lattice;\n",
    "      Let me see, then, what thereat is, and this mystery explore—\n",
    "Let my heart be still a moment and this mystery explore;—\n",
    "            ’Tis the wind and nothing more!”\n",
    "\n",
    "    Open here I flung the shutter, when, with many a flirt and flutter,\n",
    "In there stepped a stately Raven of the saintly days of yore;\n",
    "    Not the least obeisance made he; not a minute stopped or stayed he;\n",
    "    But, with mien of lord or lady, perched above my chamber door—\n",
    "Perched upon a bust of Pallas just above my chamber door—\n",
    "            Perched, and sat, and nothing more.\n",
    "\n",
    "Then this ebony bird beguiling my sad fancy into smiling,\n",
    "By the grave and stern decorum of the countenance it wore,\n",
    "“Though thy crest be shorn and shaven, thou,” I said, “art sure no craven,\n",
    "Ghastly grim and ancient Raven wandering from the Nightly shore—\n",
    "Tell me what thy lordly name is on the Night’s Plutonian shore!”\n",
    "            Quoth the Raven “Nevermore.”\n",
    "\n",
    "    Much I marvelled this ungainly fowl to hear discourse so plainly,\n",
    "Though its answer little meaning—little relevancy bore;\n",
    "    For we cannot help agreeing that no living human being\n",
    "    Ever yet was blessed with seeing bird above his chamber door—\n",
    "Bird or beast upon the sculptured bust above his chamber door,\n",
    "            With such name as “Nevermore.”\n",
    "\n",
    "    But the Raven, sitting lonely on the placid bust, spoke only\n",
    "That one word, as if his soul in that one word he did outpour.\n",
    "    Nothing farther then he uttered—not a feather then he fluttered—\n",
    "    Till I scarcely more than muttered “Other friends have flown before—\n",
    "On the morrow he will leave me, as my Hopes have flown before.”\n",
    "            Then the bird said “Nevermore.”\n",
    "\n",
    "    Startled at the stillness broken by reply so aptly spoken,\n",
    "“Doubtless,” said I, “what it utters is its only stock and store\n",
    "    Caught from some unhappy master whom unmerciful Disaster\n",
    "    Followed fast and followed faster till his songs one burden bore—\n",
    "Till the dirges of his Hope that melancholy burden bore\n",
    "            Of ‘Never—nevermore’.”\n",
    "\n",
    "    But the Raven still beguiling all my fancy into smiling,\n",
    "Straight I wheeled a cushioned seat in front of bird, and bust and door;\n",
    "    Then, upon the velvet sinking, I betook myself to linking\n",
    "    Fancy unto fancy, thinking what this ominous bird of yore—\n",
    "What this grim, ungainly, ghastly, gaunt, and ominous bird of yore\n",
    "            Meant in croaking “Nevermore.”\n",
    "\n",
    "    This I sat engaged in guessing, but no syllable expressing\n",
    "To the fowl whose fiery eyes now burned into my bosom’s core;\n",
    "    This and more I sat divining, with my head at ease reclining\n",
    "    On the cushion’s velvet lining that the lamp-light gloated o’er,\n",
    "But whose velvet-violet lining with the lamp-light gloating o’er,\n",
    "            She shall press, ah, nevermore!\n",
    "\n",
    "    Then, methought, the air grew denser, perfumed from an unseen censer\n",
    "Swung by Seraphim whose foot-falls tinkled on the tufted floor.\n",
    "    “Wretch,” I cried, “thy God hath lent thee—by these angels he hath sent thee\n",
    "    Respite—respite and nepenthe from thy memories of Lenore;\n",
    "Quaff, oh quaff this kind nepenthe and forget this lost Lenore!”\n",
    "            Quoth the Raven “Nevermore.”\n",
    "\n",
    "    “Prophet!” said I, “thing of evil!—prophet still, if bird or devil!—\n",
    "Whether Tempter sent, or whether tempest tossed thee here ashore,\n",
    "    Desolate yet all undaunted, on this desert land enchanted—\n",
    "    On this home by Horror haunted—tell me truly, I implore—\n",
    "Is there—is there balm in Gilead?—tell me—tell me, I implore!”\n",
    "            Quoth the Raven “Nevermore.”\n",
    "\n",
    "    “Prophet!” said I, “thing of evil!—prophet still, if bird or devil!\n",
    "By that Heaven that bends above us—by that God we both adore—\n",
    "    Tell this soul with sorrow laden if, within the distant Aidenn,\n",
    "    It shall clasp a sainted maiden whom the angels name Lenore—\n",
    "Clasp a rare and radiant maiden whom the angels name Lenore.”\n",
    "            Quoth the Raven “Nevermore.”\n",
    "\n",
    "    “Be that word our sign of parting, bird or fiend!” I shrieked, upstarting—\n",
    "“Get thee back into the tempest and the Night’s Plutonian shore!\n",
    "    Leave no black plume as a token of that lie thy soul hath spoken!\n",
    "    Leave my loneliness unbroken!—quit the bust above my door!\n",
    "Take thy beak from out my heart, and take thy form from off my door!”\n",
    "            Quoth the Raven “Nevermore.”\n",
    "\n",
    "    And the Raven, never flitting, still is sitting, still is sitting\n",
    "On the pallid bust of Pallas just above my chamber door;\n",
    "    And his eyes have all the seeming of a demon’s that is dreaming,\n",
    "    And the lamp-light o’er him streaming throws his shadow on the floor;\n",
    "And my soul from out that shadow that lies floating on the floor\n",
    "            Shall be lifted—nevermore!\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem statement:\n",
    "\n",
    "Write a program that prints out just the first line of each stanza in the raven, but these lines should be collected into a single string (a common need when writing results out to a file).\n",
    "\n",
    "Two choices here:\n",
    "\n",
    "1. Do something fancy and detect the true meaning of what it means to be the first line of a stanza.\n",
    "2. Exploit the known structure of the poem and do a little math about where each stanza should be.\n",
    "\n",
    "For now, let's start with the second option and see if this exploit will do the trick for us.  Hopefully it will and we can avoid some complication about the true meaning of a stanza.\n",
    "\n",
    "## First question:  what is the granularity of our question?\n",
    "\n",
    "For example, are we concerned about characters, words, lines, larger chunks, entire files, etc?  Answering this question will help you slim down your choices of which patterns, functions, and tools will be helpful. This is also how you should internally organize these things in your head.\n",
    "\n",
    "For example, you have already learned that there are ways to slice out chunks of strings if characters are the granularity of interest.\n",
    "\n",
    "You could say that the individual stanza is our granularity of interest.  Which isn't untrue, but python doesn't really have a sense of the meaning of stanzas.  This means that we would need to figure out how to define a stanza from a technical perspective.  Which we can, but isn't entirely necessary.  We'll explore how to do this in later weeks.\n",
    "\n",
    "Taking another look at our problem statement, we can reconsider what we really want out of this thing.  Which is a bunch of individual lines.  We are going to encode our understanding of stanzas in a later step.\n",
    "\n",
    "The fact that we care about lines of the poem and that the poem is already stored as a series of lines makes processing it so much easier.\n",
    "\n",
    "So, we care about lines.  This opens us up to some nice processing options because lines of data are so normal that there are some great tools to work with them.\n",
    "\n",
    "On a technical basis, we can use the `my_string.split('\\n')` to split a string with newlines on the newline characters.  This will create a list where each line is an individual line, and thus we can loop over the individual lines.\n",
    "\n",
    "You'll be using `.split()` in a lot of your programs, so get comfortable with it!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Second question:  how can we determine which lines that we want?\n",
    "\n",
    "I'll go ahead and give you a spoiler:  we can easily look up the line of text that we want if we know which line number that we want.\n",
    "\n",
    "So given that, we need to sort out how to correctly predict where those lines are.\n",
    "\n",
    "Given that the structure of the poem is so steady (18 stanzas with 6 lines each), we should be able to reasonably construct a little formula for where the first line of every stanza is.\n",
    "\n",
    "For example, if we say that I'm going to start with my left little finger and touch every other finger you can likely sort out which ones that will be without having to watch me in action.\n",
    "\n",
    "You know the following things:\n",
    "\n",
    "1. I have 10 fingers.\n",
    "2. This is couting, so there will be only integer numbers.\n",
    "3. 'Every other' is counting by two.\n",
    "\n",
    "Let's remind ourselves of the basic use of range:\n",
    "\n",
    "1. `range(exclusive_stop)`\n",
    "2. `range(inclusive_start, exclusive_stop)`\n",
    "3. `range(inclusive_start, exclusive_stop, step)`\n",
    "\n",
    "The definition of these things are the same as what you learned for string slicing.\n",
    "\n",
    "Let's count like python and say that my left hand has fingers 0-4, and my right hand has fingers 5-9.  Since I'm starting at 0 and need ten numbers after and including 0, which form can I use?\n",
    "\n",
    "Normally we'd say that form 1 would be fine, because `range(10)` will give us 0-9 inclusive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n"
     ]
    }
   ],
   "source": [
    "print(list(range(10)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "But remember that we need to go by every other.  So we need to specify a step of 2.  The only form that allows us to specify a step is form 3, which also requires that we spefify a start."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 2, 4, 6, 8]\n"
     ]
    }
   ],
   "source": [
    "print(list(range(0, 10, 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So you can predict that I'll point to my left little finger, my left middle finger, left thumb, right pointer, and right ring.  You can correctly make this prediction because my hands have an orderly dependable structure and I need a steady pattern.  This kind of system will also work on most (not all!) humans.\n",
    "\n",
    "Our treatment of the stanzas will be no different.  These mechanisms that exploit the dependability of data file can be quite elegant, but come with risk:  you need to be very certain that your data is actually dependable.  \n",
    "\n",
    "So here we're going to talk a lot about non coding things, because we shouldn't forget:  the code is just our tool.  We need to be able to think of these solutions and approaches that we can use code for, but the coding comes second to all that.\n",
    "\n",
    "We can explore how to check that dependability.  Let's pop up to our poem text and look at it from a technical perspective, then try to come up with some checks to see if our file is on track.\n",
    "\n",
    "Things we know (as in, the things that will inform us about what we need to do and how we might handle things):\n",
    "\n",
    "* there's nothing before or after the poem, as in:  the first line of the poem is the first line, and the last line is the last line.  So nothing to snip out before or after.\n",
    "* each stanza is 6 lines followed by a single empty line, except for the last stanza.\n",
    "\n",
    "Can we build up a rudamentary method of calculaing how many lines there will be is our list of lines?  Yes we can!  We're going to define some known variables.\n",
    "\n",
    "So if stanza length is 6:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stanza_len = 6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And there are 18 stanzas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_stanzas = 18"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of buffer lines between the individual stanzas is 1 less than the total number of stanzas (because remember that there are no trailing empty lines after the poem, but this will not always be the case):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_blanks = num_stanzas - 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we put them all together (with some multiplicaton):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "125\n"
     ]
    }
   ],
   "source": [
    "expected_poem_len = (stanza_len * num_stanzas) + num_blanks\n",
    "print(expected_poem_len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we've used python as a calculator, where we encoded our core formula for length, gave it some values, and then made the final calculation.  \n",
    "\n",
    "What we need to do is check what the actual length is.  We first need to do the split on the poem string and then we can run it through `len()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "125\n"
     ]
    }
   ],
   "source": [
    "raven_lines = raven.split('\\n')\n",
    "\n",
    "print(len(raven_lines))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excellent!  Our prediction based off observations in the data worked.\n",
    "\n",
    "What would a result of 6,646 have meant?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6646\n"
     ]
    }
   ],
   "source": [
    "print(len(raven)) # that we were calculating the length of the entire string rather than the number of lines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# List slicing basics\n",
    "\n",
    "Let's take a break here and learn a little about list slicing.  The basic syntax is the same as what you learned about how to slice strings.  Except that it is working at the level of element (line in this case, because that's what we built our list with when we used `.split()`) rather than character.\n",
    "\n",
    "I can give it a single postion number and get back the corresponding lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    While I nodded, nearly napping, suddenly there came a tapping,\n"
     ]
    }
   ],
   "source": [
    "print(raven_lines[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So let's get this range going.  We can slice out the beginning of our list to give ourselves something to base an initial try at.  This is the power of using lists to sort our data.  When we can split it apart such that the position numbers can have meaning, either directly via their numbers or indirectly but referencable), we can make our code clean and meaningful.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Once upon a midnight dreary, while I pondered, weak and weary,', 'Over many a quaint and curious volume of forgotten lore—', '    While I nodded, nearly napping, suddenly there came a tapping,', 'As of some one gently rapping, rapping at my chamber door.', '“’Tis some visitor,” I muttered, “tapping at my chamber door—', '            Only this and nothing more.”', '', '    Ah, distinctly I remember it was in the bleak December;', 'And each separate dying ember wrought its ghost upon the floor.', '    Eagerly I wished the morrow;—vainly I had sought to borrow', '    From my books surcease of sorrow—sorrow for the lost Lenore—', 'For the rare and radiant maiden whom the angels name Lenore—', '            Nameless here for evermore.', '', '    And the silken, sad, uncertain rustling of each purple curtain', 'Thrilled me—filled me with fantastic terrors never felt before;', '    So that now, to still the beating of my heart, I stood repeating', '    “’Tis some visitor entreating entrance at my chamber door—', 'Some late visitor entreating entrance at my chamber door;—', '            This it is and nothing more.”', '']\n"
     ]
    }
   ],
   "source": [
    "print(raven_lines[:(stanza_len * 3) + 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is not fun to read, let's put it into a for loop so everything prints out left jusified, and we're going to use a counter for funzies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 : Once upon a midnight dreary, while I pondered, weak and weary,\n",
      "1 : Over many a quaint and curious volume of forgotten lore—\n",
      "2 :     While I nodded, nearly napping, suddenly there came a tapping,\n",
      "3 : As of some one gently rapping, rapping at my chamber door.\n",
      "4 : “’Tis some visitor,” I muttered, “tapping at my chamber door—\n",
      "5 :             Only this and nothing more.”\n",
      "6 : \n",
      "7 :     Ah, distinctly I remember it was in the bleak December;\n",
      "8 : And each separate dying ember wrought its ghost upon the floor.\n",
      "9 :     Eagerly I wished the morrow;—vainly I had sought to borrow\n",
      "10 :     From my books surcease of sorrow—sorrow for the lost Lenore—\n",
      "11 : For the rare and radiant maiden whom the angels name Lenore—\n",
      "12 :             Nameless here for evermore.\n",
      "13 : \n",
      "14 :     And the silken, sad, uncertain rustling of each purple curtain\n",
      "15 : Thrilled me—filled me with fantastic terrors never felt before;\n",
      "16 :     So that now, to still the beating of my heart, I stood repeating\n",
      "17 :     “’Tis some visitor entreating entrance at my chamber door—\n",
      "18 : Some late visitor entreating entrance at my chamber door;—\n",
      "19 :             This it is and nothing more.”\n",
      "20 : \n"
     ]
    }
   ],
   "source": [
    "line_num = 0\n",
    "\n",
    "for line in raven_lines[:(stanza_len * 3) + 3]:\n",
    "    print(line_num, \":\", line)\n",
    "    line_num += 1 # a new notation!  equivalent to line_num = line_num + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cool cool, now we have our lines with the numbers here for reference.\n",
    "\n",
    "Looks like we would want line 0, 7, and 14.  So maybe a step of 7 will do the trick?  We can try it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91, 98, 105, 112, 119]\n"
     ]
    }
   ],
   "source": [
    "print(list(range(0, len(raven_lines), 7)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far so good.  Looks like the last one is 119.  Can we do a slicing trick to see the last few lines of the poem?\n",
    "\n",
    "We'll start a few stanzas before the end of the poem. The place to start is given by the last line number less the number of lines that we want to show. We could also have used a negative step, but that can be harder to wrap your head around."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "104 : \n",
      "105 :     “Prophet!” said I, “thing of evil!—prophet still, if bird or devil!\n",
      "106 : By that Heaven that bends above us—by that God we both adore—\n",
      "107 :     Tell this soul with sorrow laden if, within the distant Aidenn,\n",
      "108 :     It shall clasp a sainted maiden whom the angels name Lenore—\n",
      "109 : Clasp a rare and radiant maiden whom the angels name Lenore.”\n",
      "110 :             Quoth the Raven “Nevermore.”\n",
      "111 : \n",
      "112 :     “Be that word our sign of parting, bird or fiend!” I shrieked, upstarting—\n",
      "113 : “Get thee back into the tempest and the Night’s Plutonian shore!\n",
      "114 :     Leave no black plume as a token of that lie thy soul hath spoken!\n",
      "115 :     Leave my loneliness unbroken!—quit the bust above my door!\n",
      "116 : Take thy beak from out my heart, and take thy form from off my door!”\n",
      "117 :             Quoth the Raven “Nevermore.”\n",
      "118 : \n",
      "119 :     And the Raven, never flitting, still is sitting, still is sitting\n",
      "120 : On the pallid bust of Pallas just above my chamber door;\n",
      "121 :     And his eyes have all the seeming of a demon’s that is dreaming,\n",
      "122 :     And the lamp-light o’er him streaming throws his shadow on the floor;\n",
      "123 : And my soul from out that shadow that lies floating on the floor\n",
      "124 :             Shall be lifted—nevermore!\n"
     ]
    }
   ],
   "source": [
    "num_of_lines = ((stanza_len * 3) + 3) # we'll want the last three stanzas, plus a few empty lines\n",
    "starting_line_num = len(raven_lines) - num_of_lines # subtract the number of lines we want from the total length\n",
    "\n",
    "for line in raven_lines[starting_line_num:]: # how convenient we've already done the starting calculation\n",
    "    print(line_num, \":\", line)\n",
    "    line_num += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So based on this, we want the last few items to be lines 105, 112, and 119.  Which we have in our list of items above!  So let's just do this.\n",
    "\n",
    "We've got our range that produces the line numbers that we want, so we can loop over those numbers and use them (one at a time) to look up the lines that we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 : Once upon a midnight dreary, while I pondered, weak and weary,\n",
      "7 :     Ah, distinctly I remember it was in the bleak December;\n",
      "14 :     And the silken, sad, uncertain rustling of each purple curtain\n",
      "21 :     Presently my soul grew stronger; hesitating then no longer,\n",
      "28 :     Deep into that darkness peering, long I stood there wondering, fearing,\n",
      "35 :     Back into the chamber turning, all my soul within me burning,\n",
      "42 :     Open here I flung the shutter, when, with many a flirt and flutter,\n",
      "49 : Then this ebony bird beguiling my sad fancy into smiling,\n",
      "56 :     Much I marvelled this ungainly fowl to hear discourse so plainly,\n",
      "63 :     But the Raven, sitting lonely on the placid bust, spoke only\n",
      "70 :     Startled at the stillness broken by reply so aptly spoken,\n",
      "77 :     But the Raven still beguiling all my fancy into smiling,\n",
      "84 :     This I sat engaged in guessing, but no syllable expressing\n",
      "91 :     Then, methought, the air grew denser, perfumed from an unseen censer\n",
      "98 :     “Prophet!” said I, “thing of evil!—prophet still, if bird or devil!—\n",
      "105 :     “Prophet!” said I, “thing of evil!—prophet still, if bird or devil!\n",
      "112 :     “Be that word our sign of parting, bird or fiend!” I shrieked, upstarting—\n",
      "119 :     And the Raven, never flitting, still is sitting, still is sitting\n"
     ]
    }
   ],
   "source": [
    "for i in range(0, len(raven_lines), 7):\n",
    "    print(i, \":\", raven_lines[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that this has worked, and also another pattern:  most of these start with an indent, but not all.  Going back into the poem, we can see that even though lines 0 and 49 do not have these indents, but they are indeed the first lines.  So we are correct in our computaion of where things are even though the poem has errors.  This is a good case of data having the appearance of regularity but it being inconsistent.  I'm glad we didn't depend on that!\n",
    "\n",
    "One interesting thing to note is that we could have skipped the for loop entirely if we only wanted the line text out.  Using our slicing skills we can more directly ask for every 7th item in the list.  The advantage of using range is that we have the line numbers to use for referece or reporting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Once upon a midnight dreary, while I pondered, weak and weary,',\n",
       " '    Ah, distinctly I remember it was in the bleak December;',\n",
       " '    And the silken, sad, uncertain rustling of each purple curtain',\n",
       " '    Presently my soul grew stronger; hesitating then no longer,',\n",
       " '    Deep into that darkness peering, long I stood there wondering, fearing,',\n",
       " '    Back into the chamber turning, all my soul within me burning,',\n",
       " '    Open here I flung the shutter, when, with many a flirt and flutter,',\n",
       " 'Then this ebony bird beguiling my sad fancy into smiling,',\n",
       " '    Much I marvelled this ungainly fowl to hear discourse so plainly,',\n",
       " '    But the Raven, sitting lonely on the placid bust, spoke only',\n",
       " '    Startled at the stillness broken by reply so aptly spoken,',\n",
       " '    But the Raven still beguiling all my fancy into smiling,',\n",
       " '    This I sat engaged in guessing, but no syllable expressing',\n",
       " '    Then, methought, the air grew denser, perfumed from an unseen censer',\n",
       " '    “Prophet!” said I, “thing of evil!—prophet still, if bird or devil!—',\n",
       " '    “Prophet!” said I, “thing of evil!—prophet still, if bird or devil!',\n",
       " '    “Be that word our sign of parting, bird or fiend!” I shrieked, upstarting—',\n",
       " '    And the Raven, never flitting, still is sitting, still is sitting']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raven_lines[::7]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But going back to our problem statement, we can see that we're not all done.  We need to collect all the lines into a single string.  Right now we have a way to get all the lines we want, we just need to figure out how to get them all into a single string.\n",
    "\n",
    "Let's look at a few ways to do this.\n",
    "\n",
    "* What we have:  a bunch of strings\n",
    "* What we need:  a single string\n",
    "* What we need to decide:  what content should go between them.\n",
    "\n",
    "When we need to collect up a string we need to decide what our final form looks like.  Do we want it to look like the original?  So each 'line' as an actual line.  Connected like something else?\n",
    "\n",
    "Chances are you can presume pretty correctly that when we break something apart,  we reconnect the pieces we want  the reconstruction to look like the original.  \n",
    "\n",
    "So what is the meaning of a 'line'?\n",
    "\n",
    "* Technically:  some not newline stuff (including nothing) followed by a newline character (\\n)\n",
    "* Humanly:  normal looking text, likely on top of or below some other normal looking text\n",
    "\n",
    "So if we want to reconstruct these pieces of text back into something that looks like a poem, we need to consider those newline characters.  Sometimes they are retained in the lines when we break things apart, but sometimes they are lost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Once upon a midnight dreary, while I pondered, weak and weary,',\n",
       " '    Ah, distinctly I remember it was in the bleak December;',\n",
       " '    And the silken, sad, uncertain rustling of each purple curtain',\n",
       " '    Presently my soul grew stronger; hesitating then no longer,',\n",
       " '    Deep into that darkness peering, long I stood there wondering, fearing,',\n",
       " '    Back into the chamber turning, all my soul within me burning,',\n",
       " '    Open here I flung the shutter, when, with many a flirt and flutter,',\n",
       " 'Then this ebony bird beguiling my sad fancy into smiling,',\n",
       " '    Much I marvelled this ungainly fowl to hear discourse so plainly,',\n",
       " '    But the Raven, sitting lonely on the placid bust, spoke only',\n",
       " '    Startled at the stillness broken by reply so aptly spoken,',\n",
       " '    But the Raven still beguiling all my fancy into smiling,',\n",
       " '    This I sat engaged in guessing, but no syllable expressing',\n",
       " '    Then, methought, the air grew denser, perfumed from an unseen censer',\n",
       " '    “Prophet!” said I, “thing of evil!—prophet still, if bird or devil!—',\n",
       " '    “Prophet!” said I, “thing of evil!—prophet still, if bird or devil!',\n",
       " '    “Be that word our sign of parting, bird or fiend!” I shrieked, upstarting—',\n",
       " '    And the Raven, never flitting, still is sitting, still is sitting']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raven_lines[::7]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Taking another look here, we don't see any telltale `'\\n'` at the end of the strings nor are extra lines being printed out, so we no longer have our newlines in our text.  They were removed when we split the original string apart with `.split()`.  This method is what we call 'consumptive', meaning that it consumes (gets rid of) whatever string you pass it.  \n",
    "\n",
    "So `.split()` removed our newline character, and now we need to add it back in at the end of each line.  We want it to look like a poem again.\n",
    "\n",
    "Your appoach for how to do this depends on what all you need to do with the content.  \n",
    "\n",
    "* Just the newline?  You can use a method like `.join()`\n",
    "* More stuff or need to further process the lines?  Do an accumulator loop.\n",
    "\n",
    "Let's demo the easy one first, with `.join()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## `.join()`\n",
    "\n",
    "The syntax for this method is one of the oddest that you'll find, and I still find it a little boggling to this day.  Anyhow, homework 1 required you to make it work, even if you didn't quite understand it.  So you've seen this before, but here's a reminder of the syntax:\n",
    "\n",
    "Essential definitons:\n",
    "\n",
    "* `t` is the string (either the literal string or a variable) that you want to be in the middle of things.\n",
    "* `sequence` in the stuff that you want to join together.  This will most likely be a list, but could be something else like a string. Any sequence that you could use a for loop on will work here, and the pieces that it picks out of the sequence will be what a for loop can access.  This caveat will make sense if you try to use other things like a dictionary in here, but most of the time you wont and don't need to worry.\n",
    "\n",
    "`t.join(sequence)`\n",
    "\n",
    "Yes, you are reading that right.  You call the method on the string that you want to join the sequence with.  I understand and agree with the technical reasons for this, but that has never stopped me from always giving this a side eye.\n",
    "\n",
    "Let's see it in action.\n",
    "\n",
    "* What do we want to join? `raven_lines[::7]`  (this is our sequence)\n",
    "* What do we want to appear between each line?  A `\\n` character.  (this is our t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Once upon a midnight dreary, while I pondered, weak and weary,\n",
      "    Ah, distinctly I remember it was in the bleak December;\n",
      "    And the silken, sad, uncertain rustling of each purple curtain\n",
      "    Presently my soul grew stronger; hesitating then no longer,\n",
      "    Deep into that darkness peering, long I stood there wondering, fearing,\n",
      "    Back into the chamber turning, all my soul within me burning,\n",
      "    Open here I flung the shutter, when, with many a flirt and flutter,\n",
      "Then this ebony bird beguiling my sad fancy into smiling,\n",
      "    Much I marvelled this ungainly fowl to hear discourse so plainly,\n",
      "    But the Raven, sitting lonely on the placid bust, spoke only\n",
      "    Startled at the stillness broken by reply so aptly spoken,\n",
      "    But the Raven still beguiling all my fancy into smiling,\n",
      "    This I sat engaged in guessing, but no syllable expressing\n",
      "    Then, methought, the air grew denser, perfumed from an unseen censer\n",
      "    “Prophet!” said I, “thing of evil!—prophet still, if bird or devil!—\n",
      "    “Prophet!” said I, “thing of evil!—prophet still, if bird or devil!\n",
      "    “Be that word our sign of parting, bird or fiend!” I shrieked, upstarting—\n",
      "    And the Raven, never flitting, still is sitting, still is sitting\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(raven_lines[::7]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "And if we had gotten it the wrong way around?  We get an error, becuase we are trying to call the join method on a data type that doesn't have it.  So while you can pass join (remember that passing is where you put stuff in the `()`) any sequence, you can only call `.join()` on a string.  Meaning that the thing with the `.join()` directly after must be a string (literal of variable)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'list' object has no attribute 'join'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-23-04083c7a7204>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mraven_lines\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;36m7\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'\\n'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m: 'list' object has no attribute 'join'"
     ]
    }
   ],
   "source": [
    "print(raven_lines[::7].join('\\n'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So that was the easy way, but you can sort of see where the limitation is.  Our line content still has that weird formatting issue where most, but not all, of the lines have the indent.  Without direct access to the content in this process, we're kind of stuck with that spacing.  So we would have needed to do the cleanup earlier in our process (sometimes we've already been in there messing with stuff before we do our filtering, so this isn't a concern in the final prep preocess) but other times we haven't needed to touch the content before this point (which is our current situation).  This is a place where your personal style will come into play a lot.  You may have already dealt with this, or had a better opportunity to deal with this, depending on what you might have needed to do in your program before this point.  In this case, we haven't needed to do much else and just need to clean up a few things, so combining it in with our final collection process makes sense.  There will be other time when you'll want to keep everything separate.\n",
    "\n",
    "Let's do this again with an accumulator to have more direct control.\n",
    "\n",
    "## Accumulator\n",
    "\n",
    "Following with our usual pattern, we need to have:\n",
    "\n",
    "* an empty string to collect stuff into\n",
    "* a for loop over our desired lines\n",
    "* a cleaned up line with a newline to add to our collection (which we'll make during our loop)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clean_lines = \"\" # our empty collection string\n",
    "\n",
    "for line in raven_lines[::7]:\n",
    "    cleaned_line = line.strip() # this will get rid of the indent\n",
    "    cleaned_line += '\\n' # now we add the newline\n",
    "    clean_lines += cleaned_line # now add the clean line to the end of the collection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can see what we've done!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Once upon a midnight dreary, while I pondered, weak and weary,\n",
      "Ah, distinctly I remember it was in the bleak December;\n",
      "And the silken, sad, uncertain rustling of each purple curtain\n",
      "Presently my soul grew stronger; hesitating then no longer,\n",
      "Deep into that darkness peering, long I stood there wondering, fearing,\n",
      "Back into the chamber turning, all my soul within me burning,\n",
      "Open here I flung the shutter, when, with many a flirt and flutter,\n",
      "Then this ebony bird beguiling my sad fancy into smiling,\n",
      "Much I marvelled this ungainly fowl to hear discourse so plainly,\n",
      "But the Raven, sitting lonely on the placid bust, spoke only\n",
      "Startled at the stillness broken by reply so aptly spoken,\n",
      "But the Raven still beguiling all my fancy into smiling,\n",
      "This I sat engaged in guessing, but no syllable expressing\n",
      "Then, methought, the air grew denser, perfumed from an unseen censer\n",
      "“Prophet!” said I, “thing of evil!—prophet still, if bird or devil!—\n",
      "“Prophet!” said I, “thing of evil!—prophet still, if bird or devil!\n",
      "“Be that word our sign of parting, bird or fiend!” I shrieked, upstarting—\n",
      "And the Raven, never flitting, still is sitting, still is sitting\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(clean_lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have this pattern, we can play with changing other things, such as adding a line number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stanza 01: Once upon a midnight dreary, while I pondered, weak and weary,\n",
      "Stanza 02: Ah, distinctly I remember it was in the bleak December;\n",
      "Stanza 03: And the silken, sad, uncertain rustling of each purple curtain\n",
      "Stanza 04: Presently my soul grew stronger; hesitating then no longer,\n",
      "Stanza 05: Deep into that darkness peering, long I stood there wondering, fearing,\n",
      "Stanza 06: Back into the chamber turning, all my soul within me burning,\n",
      "Stanza 07: Open here I flung the shutter, when, with many a flirt and flutter,\n",
      "Stanza 08: Then this ebony bird beguiling my sad fancy into smiling,\n",
      "Stanza 09: Much I marvelled this ungainly fowl to hear discourse so plainly,\n",
      "Stanza 10: But the Raven, sitting lonely on the placid bust, spoke only\n",
      "Stanza 11: Startled at the stillness broken by reply so aptly spoken,\n",
      "Stanza 12: But the Raven still beguiling all my fancy into smiling,\n",
      "Stanza 13: This I sat engaged in guessing, but no syllable expressing\n",
      "Stanza 14: Then, methought, the air grew denser, perfumed from an unseen censer\n",
      "Stanza 15: “Prophet!” said I, “thing of evil!—prophet still, if bird or devil!—\n",
      "Stanza 16: “Prophet!” said I, “thing of evil!—prophet still, if bird or devil!\n",
      "Stanza 17: “Be that word our sign of parting, bird or fiend!” I shrieked, upstarting—\n",
      "Stanza 18: And the Raven, never flitting, still is sitting, still is sitting\n",
      "\n"
     ]
    }
   ],
   "source": [
    "clean_lines = \"\" # our empty collection string\n",
    "\n",
    "counter = 1 # starting at 1 because we're human counting\n",
    "\n",
    "for line in raven_lines[::7]:\n",
    "    cleaned_line = \"Stanza \" + str(counter).zfill(2) + \": \"\n",
    "    cleaned_line += line.strip() # note this is += now\n",
    "    cleaned_line += '\\n' # this can stay as +=\n",
    "    clean_lines += cleaned_line # now add the clean line to the end of the collection\n",
    "    counter +=1 # increment the counter\n",
    "    \n",
    "print(clean_lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this final form you can see that we're using both a counter accumulator and a collection accumulator.  We haven't gone back to our original use of range here because we haven't needed to use the line numbers.  In the previous line we only printed out the staza number, so using a simple counter (or enumerate) along with the string slicing have done the trick just fine.  I would need to add my range back in if I wanted that line number, but I could leave my counter in so I would have both the line and the stanza.  \n",
    "\n",
    "Can you experiment with making that work?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stanza 01 line 000: Once upon a midnight dreary, while I pondered, weak and weary,\n",
      "Stanza 02 line 007: Ah, distinctly I remember it was in the bleak December;\n",
      "Stanza 03 line 014: And the silken, sad, uncertain rustling of each purple curtain\n",
      "Stanza 04 line 021: Presently my soul grew stronger; hesitating then no longer,\n",
      "Stanza 05 line 028: Deep into that darkness peering, long I stood there wondering, fearing,\n",
      "Stanza 06 line 035: Back into the chamber turning, all my soul within me burning,\n",
      "Stanza 07 line 042: Open here I flung the shutter, when, with many a flirt and flutter,\n",
      "Stanza 08 line 049: Then this ebony bird beguiling my sad fancy into smiling,\n",
      "Stanza 09 line 056: Much I marvelled this ungainly fowl to hear discourse so plainly,\n",
      "Stanza 10 line 063: But the Raven, sitting lonely on the placid bust, spoke only\n",
      "Stanza 11 line 070: Startled at the stillness broken by reply so aptly spoken,\n",
      "Stanza 12 line 077: But the Raven still beguiling all my fancy into smiling,\n",
      "Stanza 13 line 084: This I sat engaged in guessing, but no syllable expressing\n",
      "Stanza 14 line 091: Then, methought, the air grew denser, perfumed from an unseen censer\n",
      "Stanza 15 line 098: “Prophet!” said I, “thing of evil!—prophet still, if bird or devil!—\n",
      "Stanza 16 line 105: “Prophet!” said I, “thing of evil!—prophet still, if bird or devil!\n",
      "Stanza 17 line 112: “Be that word our sign of parting, bird or fiend!” I shrieked, upstarting—\n",
      "Stanza 18 line 119: And the Raven, never flitting, still is sitting, still is sitting\n",
      "\n"
     ]
    }
   ],
   "source": [
    "clean_lines = \"\" # our empty collection string\n",
    "\n",
    "counter = 1 # starting at 1 becuase we're human counting\n",
    "\n",
    "for i in range(0, len(raven_lines), 7):\n",
    "    cleaned_line = \"Stanza \" + str(counter).zfill(2) + \" line \" + str(i).zfill(3) + \": \" #adding line num\n",
    "    cleaned_line += raven_lines[i].strip() # add the lookup here\n",
    "    cleaned_line += '\\n' # this can stay as +=\n",
    "    clean_lines += cleaned_line # now add the clean line to the end of the collection\n",
    "    counter +=1 # increment the counter\n",
    "    \n",
    "print(clean_lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's adapt our current program to take user input.  You prompt the user to ask which line of each stanza they would like.\n",
    "\n",
    "What's our first step?  Before we start mucking about with our current program, we need to think about how we can change our start, stop, step to accomplish this.  We can think back to our example about the hands.  Suppose that I lose my left pinky finger, but you give me the same rules and I abide by the same number convention despite lacking one of my fingers.  How would you change our original range statement? The only change that we have in play is a change in our start position.\n",
    "\n",
    "So a few facts:\n",
    "\n",
    "* no matter which line in each stanza is requested, the consistent nature of the poem means that it'll always be every 7 lines.\n",
    "    * so our step value of 7 can remain the same\n",
    "* no matter which line in each stanza is requested, we want to always go to the end.\n",
    "    * so we can leave our stop value as unspecified\n",
    "* the offset number should be between 0 and 6 (0-5 are the 6 lines and offset 6 is the blank line)\n",
    "    \n",
    "We need to go consistently every 7 places, but starting at a different point.  So our start position can be whatever the offset is.\n",
    "\n",
    "Actually, that's all we need to do.  Once we shift the starting point, all the others will fall in the right place.  Let's test this first.\n",
    "\n",
    "(we're going to use the word offset to mean the number of spaces that are off our normal position)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stanza 01: Over many a quaint and curious volume of forgotten lore—\n",
      "Stanza 02: And each separate dying ember wrought its ghost upon the floor.\n",
      "Stanza 03: Thrilled me—filled me with fantastic terrors never felt before;\n",
      "Stanza 04: “Sir,” said I, “or Madam, truly your forgiveness I implore;\n",
      "Stanza 05: Doubting, dreaming dreams no mortal ever dared to dream before;\n",
      "Stanza 06: Soon again I heard a tapping somewhat louder than before.\n",
      "Stanza 07: In there stepped a stately Raven of the saintly days of yore;\n",
      "Stanza 08: By the grave and stern decorum of the countenance it wore,\n",
      "Stanza 09: Though its answer little meaning—little relevancy bore;\n",
      "Stanza 10: That one word, as if his soul in that one word he did outpour.\n",
      "Stanza 11: “Doubtless,” said I, “what it utters is its only stock and store\n",
      "Stanza 12: Straight I wheeled a cushioned seat in front of bird, and bust and door;\n",
      "Stanza 13: To the fowl whose fiery eyes now burned into my bosom’s core;\n",
      "Stanza 14: Swung by Seraphim whose foot-falls tinkled on the tufted floor.\n",
      "Stanza 15: Whether Tempter sent, or whether tempest tossed thee here ashore,\n",
      "Stanza 16: By that Heaven that bends above us—by that God we both adore—\n",
      "Stanza 17: “Get thee back into the tempest and the Night’s Plutonian shore!\n",
      "Stanza 18: On the pallid bust of Pallas just above my chamber door;\n",
      "\n"
     ]
    }
   ],
   "source": [
    "offset = 1\n",
    "\n",
    "clean_lines = \"\" # our empty collection string\n",
    "\n",
    "counter = 1 # starting at 1 becuase we're human counting\n",
    "\n",
    "for line in raven_lines[offset::7]:\n",
    "    cleaned_line = \"Stanza \" + str(counter).zfill(2) + \": \"\n",
    "    cleaned_line += line.strip() # note this is += now\n",
    "    cleaned_line += '\\n' # this can stay as +=\n",
    "    clean_lines += cleaned_line # now add the clean line to the end of the collection\n",
    "    counter +=1 # increment the counter\n",
    "    \n",
    "print(clean_lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can add in an input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Which line of each stanza do you want?:3\n",
      "Stanza 01: As of some one gently rapping, rapping at my chamber door.\n",
      "Stanza 02: From my books surcease of sorrow—sorrow for the lost Lenore—\n",
      "Stanza 03: “’Tis some visitor entreating entrance at my chamber door—\n",
      "Stanza 04: And so faintly you came tapping, tapping at my chamber door,\n",
      "Stanza 05: And the only word there spoken was the whispered word, “Lenore?”\n",
      "Stanza 06: Let me see, then, what thereat is, and this mystery explore—\n",
      "Stanza 07: But, with mien of lord or lady, perched above my chamber door—\n",
      "Stanza 08: Ghastly grim and ancient Raven wandering from the Nightly shore—\n",
      "Stanza 09: Ever yet was blessed with seeing bird above his chamber door—\n",
      "Stanza 10: Till I scarcely more than muttered “Other friends have flown before—\n",
      "Stanza 11: Followed fast and followed faster till his songs one burden bore—\n",
      "Stanza 12: Fancy unto fancy, thinking what this ominous bird of yore—\n",
      "Stanza 13: On the cushion’s velvet lining that the lamp-light gloated o’er,\n",
      "Stanza 14: Respite—respite and nepenthe from thy memories of Lenore;\n",
      "Stanza 15: On this home by Horror haunted—tell me truly, I implore—\n",
      "Stanza 16: It shall clasp a sainted maiden whom the angels name Lenore—\n",
      "Stanza 17: Leave my loneliness unbroken!—quit the bust above my door!\n",
      "Stanza 18: And the lamp-light o’er him streaming throws his shadow on the floor;\n",
      "\n"
     ]
    }
   ],
   "source": [
    "offset = eval(input(\"Which line of each stanza do you want?:\"))\n",
    "\n",
    "clean_lines = \"\" # our empty collection string\n",
    "\n",
    "counter = 1 # starting at 1 becuase we're human counting\n",
    "\n",
    "for line in raven_lines[offset::7]:\n",
    "    cleaned_line = \"Stanza \" + str(counter).zfill(2) + \": \"\n",
    "    cleaned_line += line.strip() # note this is += now\n",
    "    cleaned_line += '\\n' # this can stay as +=\n",
    "    clean_lines += cleaned_line # now add the clean line to the end of the collection\n",
    "    counter +=1 # increment the counter\n",
    "    \n",
    "print(clean_lines)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}