Code
import numpy as np
@@ -390,7 +396,7 @@
5.1.1.1 CSV
CSVs, which stand for Comma-Separated Values, are a common tabular data format. In the past two pandas lectures, we briefly touched on the idea of file format: the way data is encoded in a file for storage. Specifically, our elections and babynames datasets were stored and loaded as CSVs:
-
+
pd.read_csv("data/elections.csv").head(5)
@@ -461,7 +467,7 @@
+
with open("data/elections.csv", "r") as table:
i = 0
for row in table:
@@ -482,7 +488,7 @@ 5.1.1.2 TSV
Another common file type is TSV (Tab-Separated Values). In a TSV, records are still delimited by a newline \n, while fields are delimited by \t tab character.
Let’s check out the first few rows of the raw TSV file. Again, we’ll use the repr() function so that print shows the special characters.
-
+
with open("data/elections.txt", "r") as table:
i = 0
for row in table:
@@ -498,7 +504,7 @@ (documentation).
-
+
pd.read_csv("data/elections.txt", sep='\t').head(3)
@@ -555,7 +561,7 @@
5.1.1.3 JSON
JSON (JavaScript Object Notation) files behave similarly to Python dictionaries. A raw JSON is shown below.
-
+
with open("data/elections.json", "r") as table:
i = 0
for row in table:
@@ -585,7 +591,7 @@
+
pd.read_json('data/elections.json').head(3)
@@ -640,7 +646,7 @@
5.1.1.3.1 EDA with JSON: Berkeley COVID-19 Data
The City of Berkeley Open Data website has a dataset with COVID-19 Confirmed Cases among Berkeley residents by date. Let’s download the file and save it as a JSON (note the source URL file type is also a JSON). In the interest of reproducible data science, we will download the data programatically. We have defined some helper functions in the ds100_utils.py file that we can reuse these helper functions in many different notebooks.
-
+
from ds100_utils import fetch_and_cache
covid_file = fetch_and_cache(
@@ -659,7 +665,7 @@ 5.1.1.3.1.1 File Size
Let’s start our analysis by getting a rough estimate of the size of the dataset to inform the tools we use to view the data. For relatively small datasets, we can use a text editor or spreadsheet. For larger datasets, more programmatic exploration or distributed computing tools may be more fitting. Here we will use Python tools to probe the file.
Since there seem to be text files, let’s investigate the number of lines, which often corresponds to the number of records
-
+
import os
print(covid_file, "is", os.path.getsize(covid_file) / 1e6, "MB")
@@ -677,7 +683,7 @@ As part of the EDA workflow, Unix commands can come in very handy. In fact, there’s an entire book called “Data Science at the Command Line” that explores this idea in depth! In Jupyter/IPython, you can prefix lines with ! to execute arbitrary Unix commands, and within those lines, you can refer to Python variables and expressions with the syntax {expr}.
Here, we use the ls command to list files, using the -lh flags, which request “long format with information in human-readable form.” We also use the wc command for “word count,” but with the -l flag, which asks for line counts instead of words.
These two give us the same information as the code above, albeit in a slightly different form:
-
+
!ls -lh {covid_file}
!wc -l {covid_file}
@@ -689,7 +695,7 @@
5.1.1.3.1.3 File Contents
Let’s explore the data format using Python.
-
+
with open(covid_file, "r") as f:
for i, row in enumerate(f):
print(repr(row)) # print raw strings
@@ -703,7 +709,7 @@
We can use the head Unix command (which is where pandas’ head method comes from!) to see the first few lines of the file:
-
+
!head -5 {covid_file}
{
@@ -714,21 +720,21 @@
In order to load the JSON file into pandas, Let’s first do some EDA with Oython’s json package to understand the particular structure of this JSON file so that we can decide what (if anything) to load into pandas. Python has relatively good support for JSON data since it closely matches the internal python object model. In the following cell we import the entire JSON datafile into a python dictionary using the json package.
-
+
import json
with open(covid_file, "rb") as f:
covid_json = json.load(f)
The covid_json variable is now a dictionary encoding the data in the file:
-
+
type(covid_json)
dict
We can examine what keys are in the top level JSON object by listing out the keys.
-
+
covid_json.keys()
dict_keys(['meta', 'data'])
@@ -736,14 +742,14 @@
Observation: The JSON dictionary contains a meta key which likely refers to metadata (data about the data). Metadata is often maintained with the data and can be a good source of additional information.
We can investigate the metadata further by examining the keys associated with the metadata.
-
+
covid_json['meta'].keys()
dict_keys(['view'])
The meta key contains another dictionary called view. This likely refers to metadata about a particular “view” of some underlying database. We will learn more about views when we study SQL later in the class.
-
+
covid_json['meta']['view'].keys()
dict_keys(['id', 'name', 'assetType', 'attribution', 'averageRating', 'category', 'createdAt', 'description', 'displayType', 'downloadCount', 'hideFromCatalog', 'hideFromDataJson', 'newBackend', 'numberOfComments', 'oid', 'provenance', 'publicationAppendEnabled', 'publicationDate', 'publicationGroup', 'publicationStage', 'rowsUpdatedAt', 'rowsUpdatedBy', 'tableId', 'totalTimesRated', 'viewCount', 'viewLastModified', 'viewType', 'approvals', 'columns', 'grants', 'metadata', 'owner', 'query', 'rights', 'tableAuthor', 'tags', 'flags'])
@@ -763,7 +769,7 @@
There is a key called description in the view sub dictionary. This likely contains a description of the data:
-
+
print(covid_json['meta']['view']['description'])
Counts of confirmed COVID-19 cases among Berkeley residents by date.
@@ -773,7 +779,7 @@
5.1.1.3.1.4 Examining the Data Field for Records
We can look at a few entries in the data field. This is what we’ll load into pandas.
-
+
for i in range(3):
print(f"{i:03} | {covid_json['data'][i]}")
@@ -784,7 +790,7 @@
+
type(covid_json['meta']['view']['columns'])
list
@@ -811,7 +817,7 @@
+
# Load the data from JSON and assign column titles
covid = pd.DataFrame(
covid_json['data'],
@@ -924,7 +930,7 @@ 5.1.2 Primary and Foreign Keys
Last time, we introduced .merge as the pandas method for joining multiple DataFrames together. In our discussion of joins, we touched on the idea of using a “key” to determine what rows should be merged from each table. Let’s take a moment to examine this idea more closely.
The primary key is the column or set of columns in a table that uniquely determine the values of the remaining columns. It can be thought of as the unique identifier for each individual row in the table. For example, a table of Data 100 students might use each student’s Cal ID as the primary key.
-
+
@@ -970,7 +976,7 @@ is a foreign key referencing the previous table.
-
+
@@ -1063,7 +1069,7 @@
5.2.3.1 Temporality with pandas’ dt accessors
Let’s briefly look at how we can use pandas’ dt accessors to work with dates/times in a dataset using the dataset you’ll see in Lab 3: the Berkeley PD Calls for Service dataset.
-
+
Code
calls = pd.read_csv("data/Berkeley_PD_-_Calls_for_Service.csv")
@@ -1170,11 +1176,11 @@
+
calls["EVENTDT"] = pd.to_datetime(calls["EVENTDT"])
calls.head()
-/var/folders/ks/dgd81q6j5b7ghm1zc_4483vr0000gn/T/ipykernel_73700/874729699.py:1: UserWarning:
+/var/folders/ks/dgd81q6j5b7ghm1zc_4483vr0000gn/T/ipykernel_83785/874729699.py:1: UserWarning:
Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
@@ -1279,7 +1285,7 @@
+
calls["EVENTDT"].dt.month.head()
0 4
@@ -1291,7 +1297,7 @@
+
calls["EVENTDT"].dt.dayofweek.head()
0 3
@@ -1303,7 +1309,7 @@
+
calls.sort_values("EVENTDT").head()
@@ -1450,7 +1456,7 @@ <
We can then explore the CSV (which is a text file, and does not contain binary-encoded data) in many ways: 1. Using a text editor like emacs, vim, VSCode, etc. 2. Opening the CSV directly in DataHub (read-only), Excel, Google Sheets, etc. 3. The Python file object 4. pandas, using pd.read_csv()
To try out options 1 and 2, you can view or download the Tuberculosis from the lecture demo notebook under the data folder in the left hand menu. Notice how the CSV file is a type of rectangular data (i.e., tabular data) stored as comma-separated values.
Next, let’s try out option 3 using the Python file object. We’ll look at the first four lines:
-
+
Code
with open("data/cdc_tuberculosis.csv", "r") as f:
@@ -1475,7 +1481,7 @@ <
Whoa, why are there blank lines interspaced between the lines of the CSV?
You may recall that all line breaks in text files are encoded as the special newline character \n. Python’s print() prints each string (including the newline), and an additional newline on top of that.
If you’re curious, we can use the repr() function to return the raw string with all special characters:
-
+
Code
with open("data/cdc_tuberculosis.csv", "r") as f:
@@ -1494,7 +1500,7 @@ <
Finally, let’s try option 4 and use the tried-and-true Data 100 approach: pandas.
-
+
tb_df = pd.read_csv("data/cdc_tuberculosis.csv")
tb_df.head()
@@ -1574,7 +1580,7 @@ <
You may notice some strange things about this table: what’s up with the “Unnamed” column names and the first row?
Congratulations — you’re ready to wrangle your data! Because of how things are stored, we’ll need to clean the data a bit to name our columns better.
A reasonable first step is to identify the row with the right header. The pd.read_csv() function (documentation) has the convenient header parameter that we can set to use the elements in row 1 as the appropriate columns:
-
+
tb_df = pd.read_csv("data/cdc_tuberculosis.csv", header=1) # row index
tb_df.head(5)
@@ -1653,7 +1659,7 @@ <
Wait…but now we can’t differentiate betwen the “Number of TB cases” and “TB incidence” year columns. pandas has tried to make our lives easier by automatically adding “.1” to the latter columns, but this doesn’t help us, as humans, understand the data.
We can do this manually with df.rename() (documentation):
-
+
rename_dict = {'2019': 'TB cases 2019',
'2020': 'TB cases 2020',
'2021': 'TB cases 2021',
@@ -1743,7 +1749,7 @@ Row 0 is what we call a rollup record, or summary record. It’s often useful when displaying tables to humans. The granularity of record 0 (Totals) vs the rest of the records (States) is different.
Okay, EDA step two. How was the rollup record aggregated?
Let’s check if Total TB cases is the sum of all state TB cases. If we sum over all rows, we should get 2x the total cases in each of our TB cases by year (why do you think this is?).
-
+
Code
tb_df.sum(axis=0)
@@ -1760,7 +1766,7 @@
Whoa, what’s going on with the TB cases in 2019, 2020, and 2021? Check out the column types:
-
+
Code
tb_df.dtypes
@@ -1778,7 +1784,7 @@
Since there are commas in the values for TB cases, the numbers are read as the object datatype, or storage type (close to the Python string datatype), so pandas is concatenating strings instead of adding integers (recall that Python can “sum”, or concatenate, strings together: "data" + "100" evaluates to "data100").
Fortunately read_csv also has a thousands parameter (documentation):
-
+
# improve readability: chaining method calls with outer parentheses/line breaks
tb_df = (
pd.read_csv("data/cdc_tuberculosis.csv", header=1, thousands=',')
@@ -1859,7 +1865,7 @@
-
+
tb_df.sum()
U.S. jurisdiction TotalAlabamaAlaskaArizonaArkansasCaliforniaCol...
@@ -1874,7 +1880,7 @@
The total TB cases look right. Phew!
Let’s just look at the records with state-level granularity:
-
+
Code
state_tb_df = tb_df[1:]
@@ -1959,7 +1965,7 @@ 5.4.3 Gather Census Data
U.S. Census population estimates source (2019), source (2020-2021).
Running the below cells cleans the data. There are a few new methods here: * df.convert_dtypes() (documentation) conveniently converts all float dtypes into ints and is out of scope for the class. * df.drop_na() (documentation) will be explained in more detail next time.
-
+
Code
# 2010s census data
@@ -2083,7 +2089,7 @@ or use iPython magic which will intelligently import code when files change:
%load_ext autoreload
%autoreload 2
-
+
Code
# census 2020s data
@@ -2160,7 +2166,7 @@
5.4.4 Joining Data (Merging DataFrames)
Time to merge! Here we use the DataFrame method df1.merge(right=df2, ...) on DataFrame df1 (documentation). Contrast this with the function pd.merge(left=df1, right=df2, ...) (documentation). Feel free to use either.
-
+
# merge TB DataFrame with two US census DataFrames
tb_census_df = (
tb_df
@@ -2335,7 +2341,7 @@
+
# try merging again, but cleaner this time
tb_census_df = (
tb_df
@@ -2448,7 +2454,7 @@ \[\text{TB incidence} = \frac{\text{TB cases in population}}{\text{groups in population}} = \frac{\text{TB cases in population}}{\text{population}/100000} \]
\[= \frac{\text{TB cases in population}}{\text{population}} \times 100000\]
Let’s try this for 2019:
-
+
tb_census_df["recompute incidence 2019"] = tb_census_df["TB cases 2019"]/tb_census_df["2019"]*100000
tb_census_df.head(5)
@@ -2551,7 +2557,7 @@ documentation).
-
+
# recompute incidence for all years
for year in [2019, 2020, 2021]:
tb_census_df[f"recompute incidence {year}"] = tb_census_df[f"TB cases {year}"]/tb_census_df[f"{year}"]*100000
@@ -2667,7 +2673,7 @@
+
tb_census_df.describe()
@@ -2828,7 +2834,7 @@
+
Code
tb_df = tb_df.set_index("U.S. jurisdiction")
@@ -2911,7 +2917,7 @@
+
census_2010s_df = census_2010s_df.set_index("Geographic Area")
census_2010s_df.head(5)
@@ -3019,7 +3025,7 @@
+
census_2020s_df = census_2020s_df.set_index("Geographic Area")
census_2020s_df.head(5)
@@ -3079,7 +3085,7 @@
+
tb_df.head()
@@ -3160,7 +3166,7 @@
+
census_2010s_df.head(5)
@@ -3269,7 +3275,7 @@ documentation):
-
+
# rename rolled record for 2010s
census_2010s_df.rename(index={'United States':'Total'}, inplace=True)
census_2010s_df.head(5)
@@ -3378,7 +3384,7 @@
+
# same, but for 2020s rename rolled record
census_2020s_df.rename(index={'United States':'Total'}, inplace=True)
census_2020s_df.head(5)
@@ -3440,7 +3446,7 @@ documentation).
-
+
tb_census_df = (
tb_df
.merge(right=census_2010s_df[["2019"]],
@@ -3537,7 +3543,7 @@
+
# recompute incidence for all years
for year in [2019, 2020, 2021]:
tb_census_df[f"recompute incidence {year}"] = tb_census_df[f"TB cases {year}"]/tb_census_df[f"{year}"]*100000
@@ -3652,21 +3658,21 @@ \(A\) to \(B\) is computed as \(\text{percent change} = \frac{B - A}{A} \times 100\).
-
+
incidence_2020 = tb_census_df.loc['Total', 'recompute incidence 2020']
incidence_2020
np.float64(2.1637257652759883)
-
+
incidence_2021 = tb_census_df.loc['Total', 'recompute incidence 2021']
incidence_2021
np.float64(2.3672448914298068)
-
+
difference = (incidence_2021 - incidence_2020)/incidence_2020 * 100
difference
@@ -3678,7 +3684,7 @@
5.5 EDA Demo 2: Mauna Loa CO2 Data – A Lesson in Data Faithfulness
Mauna Loa Observatory has been monitoring CO2 concentrations since 1958.
-
+
co2_file = "data/co2_mm_mlo.txt"
Let’s do some EDA!!
@@ -3703,7 +3709,7 @@
+
co2 = pd.read_csv(
co2_file, header = None, skiprows = 72,
sep = r'\s+' #delimiter for continuous whitespace (stay tuned for regex next lecture))
@@ -3791,7 +3797,7 @@ 5.5.2 Exploring Variable Feature Types
The NOAA webpage might have some useful tidbits (in this case it doesn’t).
Using this information, we’ll rerun pd.read_csv, but this time with some custom column names.
-
+
co2 = pd.read_csv(
co2_file, header = None, skiprows = 72,
sep = '\s+', #regex for continuous whitespace (next lecture)
@@ -3807,7 +3813,7 @@
5.5.3 Visualizing CO2
Scientific studies tend to have very clean data, right…? Let’s jump right in and make a time series plot of CO2 monthly averages.
-
+
Code