scraper-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scraping multiple pages from a live site\n",
    "\n",
    "By now we're feeling pretty good about how to structure a Python script that scrapes data out of HTML, and in our [previous exercise](scraper-1.ipynb) we even pulled that HTML straight off the internet. We're going to go even one step further here, and get our script ready to handle multiple pages in one scrape.\n",
    "\n",
    "It's pretty common to see websites paginate long lists of data or search results, so this technique will really come in handy. We'll need the same libraries we used before (`requests`, `BeautifulSoup`, and `csv`), and then we'll add Python's built-in `time` to help us be good citizens later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the Python libraries we need\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our target data this time [lives here](https://www.fda.gov/ICECI/EnforcementActions/WarningLetters/2018/default.htm), in a big list of FDA Warning Letters. Let's take a second to inspect the HTML, and notice a couple things we'll care about in just a minute.\n",
    "\n",
    "* The first row of every table is a list of headers, not data. And we probably won't want those same headers sprinkled throughout our CSV at the end.\n",
    "* Below each table is a set of links to each page in the dataset. There's also a link called \"Next\" that doesn't work on the very last page. Interesting.\n",
    "\n",
    "Now that we have a feel for the HTML, we'll set up just like last time, with `URL` and `HEADERS` variables that we pass into a `get()` request. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the URL we want to scrape\n",
    "\n",
    "\n",
    "# define the headers our scraper will pass, so we look like a browser\n",
    "# https://developers.whatismybrowser.com/useragents/explore/\n",
    "\n",
    "\n",
    "# use requests to fetch that URL\n",
    "\n",
    "\n",
    "# and store the page content in a Python variable\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If some of this is starting to feel familiar our third time through, then good! Just wait till you write your 30th scraper, or your 300th. Your scripts like this will probably always have a lot in common, with tweaks here and there to target different kinds of HTML tags, or to handle different edge cases. That's not a _bad_ thing either—it means you don't have to think about every single piece each time you write a new scraper. Just copy, paste, and make changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use BeautifulSoup to parse our page\n",
    "\n",
    "\n",
    "# make ourselves an empty list to hold data for a CSV\n",
    "\n",
    "\n",
    "# use BeautifulSoup to find the table in our parsed HTML\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here's our first big change: We need to set up for handling multiple pages. We still need an empty list to hold our row data, but let's give ourselves a couple new variables too. As we loop through the pages that hold our dataset, we'll need to keep track of what `page_num` we're on, and whether there are `more_pages` left to scrape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make ourselves an empty list to hold data for a CSV\n",
    "\n",
    "\n",
    "# we'll be scraping multiple pages, so start tracking what page we're on\n",
    "\n",
    "\n",
    "# and set up our loop to run until we tell it to stop\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our loop is going to look a little different too. In fact, we'll have a loop inside of a loop, handling the pages we're scraping and then the rows of data on each one of them.\n",
    "\n",
    "Python's `while` statement will help us out here. It starts a loop that runs until its condition fails to be met, so our first loop will just keep on going until we flip a switch and tell it not to. As long as there are `more_pages`, we'll keep scraping. Inside that loop, we'll have three main pieces of logic:\n",
    "\n",
    "* The internal loop that we've written before, looking at each `<tr>`, grabbing data, and sticking it in our list for later\n",
    "* An `if` statement that tests for a link called \"Next\" on our current page. If it exists, we still have scraping to do! So we should use our `page_num` to concoct a new URL to request, and `time` to pause for a second and give the server a break. If there's no link called \"Next,\" we're on the last page, and we should use ...\n",
    "* An `else` statement that gets us out of our `while` loop\n",
    "\n",
    "This part of our script might look more unwieldy than our previous scrapers, but breaking down what we need to do like this maps things out nicely."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "ename": "IndentationError",
     "evalue": "unexpected indent (<ipython-input-5-8bfcaa7ea6ae>, line 43)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-5-8bfcaa7ea6ae>\"\u001b[0;36m, line \u001b[0;32m43\u001b[0m\n\u001b[0;31m    else:\u001b[0m\n\u001b[0m    ^\u001b[0m\n\u001b[0;31mIndentationError\u001b[0m\u001b[0;31m:\u001b[0m unexpected indent\n"
     ]
    }
   ],
   "source": [
    "# start our loop to scrape multiple pages\n",
    "\n",
    "\n",
    "    # loop through the rows in our current table using BeautifulSoup\n",
    "    # (we noticed that the first row is empty, so we can use Python's slice to skip it)\n",
    "\n",
    "        # create an empty list each time through, to hold cell data\n",
    "\n",
    "\n",
    "        # loop through each cell in this table row\n",
    "\n",
    "        \n",
    "            # grab the text from that cell\n",
    "\n",
    "            \n",
    "            # and append it to our list\n",
    "\n",
    "        \n",
    "        # when we're done with this table row, append its data to our list of rows\n",
    "\n",
    "    \n",
    "    # look to see if there's a \"next page\" link on our current page\n",
    "\n",
    "\n",
    "        # we have another page! fetch a new table and send it back through the loop\n",
    "        # adjust the URL we're scraping (in this case, by incrementing the page number)\n",
    "\n",
    "\n",
    "        \n",
    "        # use requests to fetch our new URL\n",
    "\n",
    "        \n",
    "        # as above, get the page content into Python\n",
    "        # then use BeautifulSoup to parse it and find our table\n",
    "\n",
    "\n",
    "\n",
    "        \n",
    "        # pause for a second to be kind to the server\n",
    "\n",
    "\n",
    "    # if there's no \"next page\" link, there are no more pages, so drop out of our loop\n",
    "    else:\n",
    "        # our `while` loop only runs until `more_pages` is no longer True, so...\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And because we carried forward some code from last time, we have _all_ that data, from multiple pages, tucked nicely into one Python list. So we don't have to change anything about our CSV output other than the filename."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use Python's CSV library to create our output file\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}