diff --git a/sql_II/sql_II.html b/sql_II/sql_II.html index 6c2ad45c..d1266d15 100644 --- a/sql_II/sql_II.html +++ b/sql_II/sql_II.html @@ -57,7 +57,7 @@ - + @@ -67,9 +67,6 @@ - - - @@ -83,15 +80,19 @@

SQL II

@@ -138,125 +139,34 @@

SQL II

In this lecture, we’ll continue our work from last time to introduce some advanced SQL syntax.

-

First, let’s load in the database from last lecture.

-
-
# Load the SQL Alchemy Python library
-import sqlalchemy
-import pandas as pd
+

First, let’s load in the basic_examples.db database.

+
+
%load_ext sql
-
-
%load_ext sql
-
-
The sql extension is already loaded. To reload it, use:
-  %reload_ext sql
+
+
import duckdb
+conn = duckdb.connect()
+conn.query("INSTALL sqlite")
-
-
-
%%sql
-sqlite:///data/basic_examples.db
+
+
%sql duckdb:///data/basic_examples.db --alias basic

Aggregating with GROUP BY

At this point, we’ve seen that SQL offers much of the same functionality that was given to us by pandas. We can extract data from a table, filter it, and reorder it to suit our needs.

In pandas, much of our analysis work relied heavily on being able to use .groupby() to aggregate across the rows of our dataset. SQL’s answer to this task is the (very conveniently named) GROUP BY clause. While the outputs of GROUP BY are similar to those of .groupby() —— in both cases, we obtain an output table where some column has been used for grouping —— the syntax and logic used to group data in SQL are fairly different to the pandas implementation.

-

To illustrate GROUP BY, we will consider the Dish table from the basic_examples.db database.

-
-
%%sql
-SELECT * 
-FROM Dish;
-
-
 * sqlite:///data/basic_examples.db
-   sqlite:///data/imdbmini.db
-Done.
-
-
-
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
nametypecost
raviolientree10
ramenentree13
tacoentree7
edamameappetizer4
friesappetizer4
potstickerappetizer4
ice creamdessert5
-
-
-
-

Say we wanted to find the total costs of dishes of a certain type. To accomplish this, we would write the following code.

-
-
%%sql
-SELECT type, SUM(cost)
-FROM Dish
-GROUP BY type;
-
-
 * sqlite:///data/basic_examples.db
-   sqlite:///data/imdbmini.db
-Done.
-
-
-
- - - - - - - - - - - - - - - - - - - - - -
typeSUM(cost)
appetizer12
dessert5
entree30
-
+

To illustrate GROUP BY, we will consider the Dish table from our database.

+
+
%%sql
+SELECT * 
+FROM Dish;
+

Notice that there are multiple dishes of the same type. What if we wanted to find the total costs of dishes of a certain type? To accomplish this, we would write the following code.

+
+
%%sql
+SELECT type, SUM(cost)
+FROM Dish
+GROUP BY type;

What is going on here? The statement GROUP BY type tells SQL to group the data based on the value contained in the type column (whether a record is an appetizer, entree, or dessert). SUM(cost) sums up the costs of dishes in each type and displays the result in the output table.

You may be wondering: why does SUM(cost) come before the command to GROUP BY type? Don’t we need to form groups before we can count the number of entries in each? Remember that SQL is a declarative programming language —— a SQL programmer simply states what end result they would like to see, and leaves the task of figuring out how to obtain this result to SQL itself. This means that SQL queries sometimes don’t follow what a reader sees as a “logical” sequence of thought. Instead, SQL requires that we follow its set order of operations when constructing queries. So long as we follow this order, SQL will handle the underlying logic.

@@ -270,166 +180,30 @@

Aggregating with
  • AVG: find the average value of each group

    • We can easily compute multiple aggregations all at once (a task that was very tricky in pandas).

    -
    -
    %%sql
-SELECT type, SUM(cost), MIN(cost), MAX(name)
-FROM Dish
-GROUP BY type;
    -
    -
     * sqlite:///data/basic_examples.db
-   sqlite:///data/imdbmini.db
-Done.
    -
    -
    -
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    typeSUM(cost)MIN(cost)MAX(name)
    appetizer124potsticker
    dessert55ice cream
    entree307taco
    -
    -
    +
    +
    %%sql
+SELECT type, SUM(cost), MIN(cost), MAX(name)
+FROM Dish
+GROUP BY type;

    To count the number of rows associated with each group, we use the COUNT keyword. Calling COUNT(*) will compute the total number of rows in each group, including rows with null values. Its pandas equivalent is .groupby().size().

    Recall the Dragon table from the previous lecture:

    -
    -
    %%sql
-SELECT * FROM Dragon;
    -
    -
     * sqlite:///data/basic_examples.db
-   sqlite:///data/imdbmini.db
-Done.
    -
    -
    -
    - - - - - - - - - - - - - - - - - - - - - - - - - -
    nameyearcute
    hiccup201010
    drogon2011-100
    dragon 220190
    -
    -
    -
    -

    Notice that COUNT(*) and COUNT(cute) result in different outputs/

    -
    -
    %%sql
-SELECT year, COUNT(*)
-FROM Dragon
-GROUP BY year;
    -
    -
     * sqlite:///data/basic_examples.db
-   sqlite:///data/imdbmini.db
-Done.
    -
    -
    -
    - - - - - - - - - - - - - - - - - - - - - -
    yearCOUNT(*)
    20101
    20111
    20191
    -
    -
    -
    -
    -
    %%sql
-SELECT year, COUNT(cute)
-FROM Dragon
-GROUP BY year;
    -
    -
     * sqlite:///data/basic_examples.db
-   sqlite:///data/imdbmini.db
-Done.
    -
    -
    -
    - - - - - - - - - - - - - - - - - - - - - -
    yearCOUNT(cute)
    20101
    20111
    20191
    +
    +
    %%sql
+SELECT * FROM Dragon;
    +

    Notice that COUNT(*) and COUNT(cute) result in different outputs.

    +
    +
    %%sql
+SELECT year, COUNT(*)
+FROM Dragon
+GROUP BY year;
    +
    +
    %%sql
+SELECT year, COUNT(cute)
+FROM Dragon
+GROUP BY year;

    With this definition of GROUP BY in hand, let’s update our SQL order of operations. Remember: every SQL query must list clauses in this order.

    SELECT <column expression list>
@@ -441,97 +215,31 @@ 
    Aggregating with
 [OFFSET <number of rows>];

    Note that we can use the AS keyword to rename columns during the selection process and that column expressions may include aggregation functions (MAX, MIN, etc.).

    -
    -

    Summary

    -

    Let’s summarize what we’ve learned so far. We know that SELECT and FROM are the fundamental building blocks of any SQL query. We can augment these two keywords with additional clauses to refine the data in our output table.

    -

    Any clauses that we include must follow a strict ordering within the query:

    -
    SELECT <column list>
+
    
+
    Filtering Groups
    
+
    Now, what if we only want groups that meet a certain condition? HAVING filters groups by applying some condition across all rows in each group. We interpret it as a way to keep only the groups HAVING some condition. Note the difference between WHERE and HAVING: we use WHERE to filter rows, whereas we use HAVING to filter groups. WHERE precedes HAVING in terms of how SQL executes a query.
    
+
    Let’s take a look at the Dish table to see how we can use HAVING. Say we want to group dishes with a cost greater than 4 by type and only keep groups where the max cost is less than 10.
    
+
    
+
    %%sql
+SELECT type, COUNT(*)
+FROM Dish
+WHERE cost > 4
+GROUP BY type
+HAVING MAX(cost) <  10;
    
+
    
+
    Here, we first use WHERE to filter for rows with a cost greater than 4. We then group our values by type before applying the HAVING operator. With HAVING, we can filter our groups based on if the max cost is less than 10.
    
+
    
+
    
+
    Summary: SQL
    
+
    With this definition of GROUP BY and HAVING in hand, let’s update our SQL order of operations. Remember: every SQL query must list clauses in this order.
    
+
    SELECT <column expression list>
 FROM <table>
 [WHERE <predicate>]
 [GROUP BY <column list>]
 [ORDER BY <column list>]
 [LIMIT <number of rows>]
-[OFFSET <number of rows>]
    
-
    Here, any clause contained in square brackets [ ] is optional —— we only need to use the keyword if it is relevant to the table operation we want to perform. Also note that by convention, we use all caps for keywords in SQL statements and use newlines to make code more readable.
    
-
    
-
    
-
    Filtering Groups
    
-
    HAVING filters groups by applying some condition across all rows in each group. We interpret it as a a way to keep only the groups HAVING some condition. Note the difference between WHERE and HAVING: we use WHERE to filter rows, whereas we use HAVING to filter groups. WHERE precedes HAVING in terms of how SQL executes a query.
    
-
    Let’s take a look at the Dish table to see how we can use HAVING.
    
-
    The code below groups the different dishes by type, and only keeps those groups wherein the max cost is still less than 8.
    
-
    
-
    %%sql
-SELECT type, COUNT(*)
-FROM Dish
-GROUP BY type
-HAVING MAX(cost) < 8;
    
-
    
-
     * sqlite:///data/basic_examples.db
-   sqlite:///data/imdbmini.db
-Done.
    
-
    
-
    
-
    
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
    typeCOUNT(*)
    appetizer3
    dessert1
    
-
    
-
    
-
    
-
    In contrast, the code below first filters for rows where the cost is less than 8, and then does the grouping. Note the difference in outputs - in this case, “taco” is also included, whereas other entries in the same group having cost greater than or equal to 8 are not included.
    
-
    
-
    %%sql
-SELECT type, COUNT(*)
-FROM Dish
-WHERE cost < 8
-GROUP BY type;
    
-
    
-
     * sqlite:///data/basic_examples.db
-Done.
    
-
    
-
    
-
    
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
    typeCOUNT(*)
    appetizer3
    dessert1
    entree1
    
-
    
-
    
-
    
-
    In general, to filter rows, we use WHERE, while to filter groups, we use HAVING. Note that WHERE precedes HAVING when they are both used.
    
+[OFFSET <number of rows>];
    +

    Note that we can use the AS keyword to rename columns during the selection process and that column expressions may include aggregation functions (MAX, MIN, etc.).

    EDA in SQL

    @@ -539,253 +247,80 @@

    EDA in SQL

    Our typical workflow when working with “big data” is:

    1. Use SQL to query data from a database
      2. -
    2. Use python (with pandas) to analyze this data in detail
      3. +
    3. Use Python (with pandas) to analyze this data in detail
    -

    We can, however, still perform simple data cleaning and re-structuring using SQL directly. To do so, we’ll use the Title table from the imdbmini database.

    +

    We can, however, still perform simple data cleaning and re-structuring using SQL directly. To do so, we’ll use the Title table from the imdb_duck database, which contains information about movies and actors.

    +

    Let’s load in the imdb_duck database.

    +
    +
    import os
+if os.path.exists("/home/jovyan/shared/sql/imdb_duck.db"):
+    imdbpath = "duckdb:////home/jovyan/shared/sql/imdb_duck.db"
+elif os.path.exists("data/imdb_duck.db"):
+    imdbpath =  "duckdb:///data/imdb_duck.db"
+else:
+    import gdown
+    url = 'https://drive.google.com/uc?id=10tKOHGLt9QoOgq5Ii-FhxpB9lDSQgl1O'
+    output_path = 'data/imdb_duck.db'
+    gdown.download(url, output_path, quiet=False)
+    imdbpath = "duckdb:///data/imdb_duck.db"
+print(imdbpath)
    +
    +
    +
    from sqlalchemy import create_engine
+imdb_engine = create_engine(imdbpath, connect_args={'read_only': True})
+%sql imdb_engine --alias imdb
    +
    +

    Since we’ll be working with the Title table, let’s take a quick look at what it contains.

    +
    +
    %%sql imdb 
+    
+SELECT *
+FROM Title
+WHERE primaryTitle IN ('Ginny & Georgia', 'What If...?', 'Succession', 'Veep', 'Tenet')
+LIMIT 10;
    +

    Matching Text using LIKE

    One common task we encountered in our first look at EDA was needing to match string data. For example, we might want to remove entries beginning with the same prefix as part of the data cleaning process.

    In SQL, we use the LIKE operator to (you guessed it) look for strings that are like a given string pattern.

    -
    -
    %%sql
-sqlite:///data/imdbmini.db
    -
    -
    -
    %%sql
-SELECT titleType, primaryTitle
-FROM Title
-WHERE primaryTitle LIKE "Star Wars: Episode I - The Phantom Menace"
    -
    -
       sqlite:///data/basic_examples.db
- * sqlite:///data/imdbmini.db
-Done.
    -
    -
    -
    - - - - - - - - - - - - - -
    titleTypeprimaryTitle
    movieStar Wars: Episode I - The Phantom Menace
    -
    -
    -
    -

    What if we wanted to find all Star Wars movies? % is the wildcard operator, it means “look for any character, any number of times”. This makes it helpful for identifying strings that are similar to our desired pattern, even when we don’t know the full text of what we aim to extract. In contrast, _ means “look for exactly 1 character”, as you can see in the Harry Potter example that follows.

    -
    -
    %%sql
-SELECT titleType, primaryTitle
-FROM Title
-WHERE primaryTitle LIKE "%Star Wars%"
-LIMIT 10;
    -
    -
       sqlite:///data/basic_examples.db
- * sqlite:///data/imdbmini.db
-Done.
    -
    -
    -
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    titleTypeprimaryTitle
    movieStar Wars: Episode IV - A New Hope
    movieStar Wars: Episode V - The Empire Strikes Back
    movieStar Wars: Episode VI - Return of the Jedi
    movieStar Wars: Episode I - The Phantom Menace
    movieStar Wars: Episode II - Attack of the Clones
    movieStar Wars: Episode III - Revenge of the Sith
    tvSeriesStar Wars: Clone Wars
    tvSeriesStar Wars: The Clone Wars
    movieStar Wars: The Clone Wars
    movieStar Wars: Episode VII - The Force Awakens
    -
    -
    -
    -
    -
    %%sql
-SELECT titleType, primaryTitle
-FROM Title
-WHERE primaryTitle LIKE "Harry Potter and the Deathly Hallows: Part _"
    -
    -
       sqlite:///data/basic_examples.db
- * sqlite:///data/imdbmini.db
-Done.
    -
    -
    -
    - - - - - - - - - - - - - - - - - -
    titleTypeprimaryTitle
    movieHarry Potter and the Deathly Hallows: Part 1
    movieHarry Potter and the Deathly Hallows: Part 2
    -
    -
    +
    +
    %%sql
+SELECT titleType, primaryTitle
+FROM Title
+WHERE primaryTitle LIKE 'Star Wars: Episode I - The Phantom Menace'
    +
    +

    What if we wanted to find all Star Wars movies? % is the wildcard operator, it means “look for any character, any number of times”. This makes it helpful for identifying strings that are similar to our desired pattern, even when we don’t know the full text of what we aim to extract.

    +
    +
    %%sql
+SELECT titleType, primaryTitle
+FROM Title
+WHERE primaryTitle LIKE '%Star Wars%'
+LIMIT 10;
    +
    +

    Alternatively, we can use RegEx! DuckDB and most real DBMSs allow for this. Note that here, we have to use the SIMILAR TO operater rather than LIKE.

    +
    +
    %%sql
+SELECT titleType, primaryTitle
+FROM Title
+WHERE primaryTitle SIMILAR TO '.*Star Wars*.'
+LIMIT 10;

    CASTing Data Types

    A common data cleaning task is converting data to the correct variable type. The CAST keyword is used to generate a new output column. Each entry in this output column is the result of converting the data in an existing column to a new data type. For example, we may wish to convert numeric data stored as a string to an integer.

    -
    -
    %%sql
-SELECT primaryTitle, CAST(runtimeMinutes AS INT), CAST(startYear AS INT)
-FROM Title
-LIMIT 5
    -
    -
       sqlite:///data/basic_examples.db
- * sqlite:///data/imdbmini.db
-Done.
    -
    -
    -
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    primaryTitleCAST(runtimeMinutes AS INT)CAST(startYear AS INT)
    A Trip to the Moon131902
    The Birth of a Nation1951915
    The Cabinet of Dr. Caligari761920
    The Kid681921
    Nosferatu941922
    -
    -
    +
    +
    %%sql
+SELECT primaryTitle, CAST(runtimeMinutes AS INT)
+FROM Title;

    We use CAST when SELECTing colunns for our output table. In the example above, we want to SELECT the columns of integer year and runtime data that is created by the CAST.

    SQL will automatically name a new column according to the command used to SELECT it, which can lead to unwieldy column names. We can rename the CASTed column using the AS keyword.

    -
    -
    %%sql
-SELECT primaryTitle AS title, CAST(runtimeMinutes AS INT) AS minutes, CAST(startYear AS INT) AS year
-FROM Title
-LIMIT 5;
    -
    -
       sqlite:///data/basic_examples.db
- * sqlite:///data/imdbmini.db
-Done.
    -
    -
    -
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    titleminutesyear
    A Trip to the Moon131902
    The Birth of a Nation1951915
    The Cabinet of Dr. Caligari761920
    The Kid681921
    Nosferatu941922
    -
    -
    +
    +
    %%sql
+SELECT primaryTitle AS title, CAST(runtimeMinutes AS INT) AS minutes, CAST(startYear AS INT) AS year
+FROM Title
+LIMIT 5;
    @@ -797,98 +332,27 @@

    Usi ... ELSE <yet another value> END -

    Scanning through the skeleton code above, you can see that the logic is similar to that of an if statement in python. The conditional statement is first opened by calling CASE. Each new condition is specified by WHEN, with THEN indicating what value should be filled if the condition is met. ELSE specifies the value that should be filled if no other conditions are met. Lastly, END indicates the end of the conditional statement; once END has been called, SQL will continue evaluating the query as usual.

    +

    Scanning through the skeleton code above, you can see that the logic is similar to that of an if statement in Python. The conditional statement is first opened by calling CASE. Each new condition is specified by WHEN, with THEN indicating what value should be filled if the condition is met. ELSE specifies the value that should be filled if no other conditions are met. Lastly, END indicates the end of the conditional statement; once END has been called, SQL will continue evaluating the query as usual.

    Let’s see this in action. In the example below, we give the new column created by the CASE statement the name movie_age.

    -
    -
    %%sql
-/* If a movie was filmed before 1950, it is "old"
-Otherwise, if a movie was filmed before 2000, it is "mid-aged"
-Else, a movie is "new" */
-
-SELECT titleType, startYear,
-CASE WHEN startYear < 1950 THEN "old"
-     WHEN startYear < 2000 THEN "mid-aged"
-     ELSE "new"
-     END AS movie_age
-FROM Title
-LIMIT 10;
    -
    -
       sqlite:///data/basic_examples.db
- * sqlite:///data/imdbmini.db
-Done.
    -
    -
    -
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    titleTypestartYearmovie_age
    short1902old
    movie1915old
    movie1920old
    movie1921old
    movie1922old
    movie1924old
    movie1925old
    movie1925old
    movie1927old
    movie1926old
    -
    -
    +
    +
    %%sql
+/* If a movie was filmed before 1950, it is "old"
+Otherwise, if a movie was filmed before 2000, it is "mid-aged"
+Else, a movie is "new" */
+
+SELECT titleType, startYear,
+CASE WHEN startYear < 1950 THEN 'old'
+     WHEN startYear < 2000 THEN 'mid-aged'
+     ELSE 'new'
+     END AS movie_age
+FROM Title;

    JOINing Tables

    -

    At this point, we’re well-versed in using SQL as a tool to clean, manipulate, and transform data in a table. Notice that this sentence referred to one table, specifically. What happens if the data we need is distributed across multiple tables? This is an important consideration when using SQL – recall that we first introduced SQL as a language to query from databases. Databases often store data in a multidimensional structure. In other words, information is stored across several tables, with each table containing a small subset of all the data housed by the database.

    -

    A common way of organizing a database is by using a star schema. A star schema is composed of two types of tables. A fact table is the central table of the database – it contains the information needed to link entries across several dimension tables, which contain more detailed information about the data.

    +

    At this point, we’re well-versed in using SQL as a tool to clean, manipulate, and transform data in a table. Notice that this sentence referred to one table, specifically. What happens if the data we need is distributed across multiple tables? This is an important consideration when using SQL —— recall that we first introduced SQL as a language to query from databases. Databases often store data in a multidimensional structure. In other words, information is stored across several tables, with each table containing a small subset of all the data housed by the database.

    +

    A common way of organizing a database is by using a star schema. A star schema is composed of two types of tables. A fact table is the central table of the database —— it contains the information needed to link entries across several dimension tables, which contain more detailed information about the data.

    Say we were working with a database about boba offerings in Berkeley. The dimension tables of the database might contain information about tea varieties and boba toppings. The fact table would be used to link this information across the various dimension tables.

    multidimensional

    @@ -908,37 +372,85 @@

    JOINing Tables JOIN table_2 ON key_1 = key_2;

    We also need to specify what column from each table should be used to determine matching entries. By defining these keys, we provide SQL with the information it needs to pair rows of data together.

    -

    In a cross join, all possible combinations of rows appear in the output table, regardless of whether or not rows share a matching key. Because all rows are joined, even if there is no matching key, it is not necessary to specify what keys to consider in an ON statement. A cross join is also known as a cartesian product.

    -
    -

    cross

    -

    The most commonly used type of SQL JOIN is the inner join. It turns out you’re already familiar with what an inner join does, and how it works – this is the type of join we’ve been using in pandas all along! In an inner join, we combine every row in our first table with its matching entry in the second table. If a row from either table does not have a match in the other table, it is omitted from the output.

    inner

    -

    Another way of interpreting the inner join: perform a cross join, then remove all rows that do not share a matching key. Notice that the output of the inner join above contains all rows of the cross join example that contain a single color across the entire row.

    -

    In a full outer join, all rows that have a match between the two tables are joined together. If a row has no match in the second table, then the values of the columns for that second table are filled with null. In other words, a full outer join performs an inner join while still keeping rows that have no match in the other table. This is best understood visually:

    +

    In a cross join, all possible combinations of rows appear in the output table, regardless of whether or not rows share a matching key. Because all rows are joined, even if there is no matching key, it is not necessary to specify what keys to consider in an ON statement. A cross join is also known as a cartesian product.

    -

    full

    +

    cross

    -

    We have kept the same output achieved using an inner join, with the addition of partially null rows for entries in s and t that had no match in the second table. Note that FULL OUTER JOIN is not supported by SQLite, the “flavor” of SQL that will be used in lab and homework.

    -

    A left outer join is similar to a full outer join. In a left outer join, all rows in the left table are kept in the output table. If a row in the right table shares a match with the left table, this row will be kept; otherwise, the rows in the right table are omitted from the output.

    +

    Conceptually, we can interpret an inner join as a cross join, followed by removing all rows that do not share a matching key. Notice that the output of the inner join above contains all rows of the cross join example that contain a single color across the entire row.

    +

    In a left outer join, all rows in the left table are kept in the output table. If a row in the right table shares a match with the left table, this row will be kept; otherwise, the rows in the right table are omitted from the output. We can fill in any missing values with NULL.

    left

    -

    A right outer join keeps all rows in the right table. Rows in the left table are only kept if they share a match in the right table. Right outer joins are not supported by SQLite.

    +

    A right outer join keeps all rows in the right table. Rows in the left table are only kept if they share a match in the right table. Again, we can fill in any missing values with NULL.

    right

    -

    In the examples above, we performed our joins by checking for equality between the two tables (i.e., by setting s.id = t.id). SQL also supports joining rows on inequalities, which is something we weren’t able to do when working in pandas. Consider a new dataset that contains information about students and teachers.

    +

    In a full outer join, all rows that have a match between the two tables are joined together. If a row has no match in the second table, then the values of the columns for that second table are filled with NULL. In other words, a full outer join performs an inner join while still keeping rows that have no match in the other table. This is best understood visually:

    -

    student

    +

    full

    -

    Often, we wish to compare the relative values of rows in different tables, rather than check that they are exactly equal. For example, we may want to join rows where students are older than the corresponding teacher. We can do so by specifying an inequality in our ON statement.

    -
    -

    inequality

    +

    We have kept the same output achieved using an inner join, with the addition of partially null rows for entries in s and t that had no match in the second table.

    +
    +

    Aliasing in JOINs

    +

    When joining tables, we often create aliases for table names (similarly to what we did with column names in the last lecture). We do this as it is typically easier to refer to aliases, especially when we are working with long table names. We can even reference columns using aliased table names.

    +

    Let’s say we want to determine the average rating of various movies:

    +
    +
    %%sql
+
+SELECT primaryTitle, averageRating
+FROM Title AS T INNER JOIN Rating AS R
+ON T.tconst = R.tconst;
    +
    +

    Note that the AS is actually optional! We can create aliases for our tables even without it, but we usually include it for clarity.

    +
    +
    %%sql
+
+SELECT primaryTitle, averageRating
+FROM Title T INNER JOIN Rating R
+ON T.tconst = R.tconst;
    +
    +

    Common Table Expression

    +

    For more sophisticated data problems, the queries can become very complex. Common Table Expressions allow us to break down these complex queries into more manageable parts. This involves creating temporary tables which correspond to different aspects of the problem and then referencing them in the final query. The following format is an example of how we can create two temporary tables and then use them for further querying:

    +
    WITH 
+table_name1 AS ( 
+    SELECT ...
+),
+table_name2 AS ( 
+    SELECT ...
+)
+SELECT ... 
+FROM 
+table_name1, 
+table_name2, ...
    +

    Let’s say we want to identify the top 10 action movies that are highly rated (with an average rating greater than 7) and popular (having more than 5000 votes), along with the primary actors who are the most popular, we can use Common Table Expression to break this query down into separate problems. Initially, we can filter to find good action movies and prolific actors separately. This way, in our final join, we only need to change the order.

    +
    +
    %%sql
+WITH 
+good_action_movies AS (
+    SELECT *
+    FROM Title T JOIN Rating R ON T.tconst = R.tconst  
+    WHERE genres LIKE '%Action%' AND averageRating > 7 AND numVotes > 5000
+),
+prolific_actors AS (
+    SELECT N.nconst, primaryName, COUNT(*) as numRoles
+    FROM Name N JOIN Principal P ON N.nconst = P.nconst
+    WHERE category = 'actor'
+    GROUP BY N.nconst, primaryName
+)
+SELECT primaryTitle, primaryName, numRoles, ROUND(averageRating) AS rating
+FROM good_action_movies m, prolific_actors a, principal p
+WHERE p.tconst = m.tconst AND p.nconst = a.nconst
+ORDER BY rating DESC, numRoles DESC
+LIMIT 10;
    +
    +
    +

    diff --git a/sql_II/sql_II.ipynb b/sql_II/sql_II.ipynb new file mode 100644 index 00000000..843f0947 --- /dev/null +++ b/sql_II/sql_II.ipynb @@ -0,0 +1,772 @@ +{ + "cells": [ + { + "cell_type": "raw", + "metadata": {}, + "source": [ + "---\n", + "title: SQL II\n", + "execute:\n", + " echo: true\n", + "format:\n", + " html:\n", + " code-fold: false\n", + " code-tools: true\n", + " toc: true\n", + " toc-title: SQL II\n", + " page-layout: full\n", + " theme:\n", + " - cosmo\n", + " - cerulean\n", + " callout-icon: false\n", + "---" + ], + "id": "80b16079" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "::: {.callout-note collapse=\"false\"}\n", + "## Learning Outcomes\n", + "* Perform aggregations using `GROUP BY`\n", + "* Introduce the ability to filter groups\n", + "* Perform data cleaning and text manipulation in SQL\n", + "* Join data across tables\n", + ":::\n", + "\n", + "In this lecture, we'll continue our work from last time to introduce some advanced SQL syntax. \n", + "\n", + "First, let's load in the `basic_examples.db` database.\n" + ], + "id": "c17a3026" + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%load_ext sql" + ], + "id": "85480f1c", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "import duckdb\n", + "conn = duckdb.connect()\n", + "conn.query(\"INSTALL sqlite\")" + ], + "id": "1cd6b32b", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%sql duckdb:///data/basic_examples.db --alias basic" + ], + "id": "caeae893", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Aggregating with `GROUP BY`\n", + "\n", + "At this point, we've seen that SQL offers much of the same functionality that was given to us by `pandas`. We can extract data from a table, filter it, and reorder it to suit our needs.\n", + "\n", + "In `pandas`, much of our analysis work relied heavily on being able to use `.groupby()` to aggregate across the rows of our dataset. SQL's answer to this task is the (very conveniently named) `GROUP BY` clause. While the outputs of `GROUP BY` are similar to those of `.groupby()` —— in both cases, we obtain an output table where some column has been used for grouping —— the syntax and logic used to group data in SQL are fairly different to the `pandas` implementation.\n", + "\n", + "To illustrate `GROUP BY`, we will consider the `Dish` table from our database.\n" + ], + "id": "8f4bf9ff" + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%%sql\n", + "SELECT * \n", + "FROM Dish;" + ], + "id": "41ab1891", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Notice that there are multiple dishes of the same `type`. What if we wanted to find the total costs of dishes of a certain `type`? To accomplish this, we would write the following code.\n" + ], + "id": "810fd2b3" + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%%sql\n", + "SELECT type, SUM(cost)\n", + "FROM Dish\n", + "GROUP BY type;" + ], + "id": "4854e3ef", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "What is going on here? The statement `GROUP BY type` tells SQL to group the data based on the value contained in the `type` column (whether a record is an appetizer, entree, or dessert). `SUM(cost)` sums up the costs of dishes in each `type` and displays the result in the output table.\n", + "\n", + "You may be wondering: why does `SUM(cost)` come before the command to `GROUP BY type`? Don't we need to form groups before we can count the number of entries in each? Remember that SQL is a *declarative* programming language —— a SQL programmer simply states what end result they would like to see, and leaves the task of figuring out *how* to obtain this result to SQL itself. This means that SQL queries sometimes don't follow what a reader sees as a \"logical\" sequence of thought. Instead, SQL requires that we follow its set order of operations when constructing queries. So long as we follow this order, SQL will handle the underlying logic.\n", + "\n", + "In practical terms: our goal with this query was to output the total `cost`s of each `type`. To communicate this to SQL, we say that we want to `SELECT` the `SUM`med `cost` values for each `type` group. \n", + "\n", + "There are many aggregation functions that can be used to aggregate the data contained in each group. Some common examples are:\n", + "\n", + "* `COUNT`: count the number of rows associated with each group\n", + "* `MIN`: find the minimum value of each group\n", + "* `MAX`: find the maximum value of each group\n", + "* `SUM`: sum across all records in each group\n", + "* `AVG`: find the average value of each group\n", + "\n", + "We can easily compute multiple aggregations all at once (a task that was very tricky in `pandas`).\n" + ], + "id": "1a3f4e19" + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%%sql\n", + "SELECT type, SUM(cost), MIN(cost), MAX(name)\n", + "FROM Dish\n", + "GROUP BY type;" + ], + "id": "8cde2b58", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To count the number of rows associated with each group, we use the `COUNT` keyword. Calling `COUNT(*)` will compute the total number of rows in each group, including rows with null values. Its `pandas` equivalent is `.groupby().size()`.\n", + "\n", + "Recall the `Dragon` table from the previous lecture:\n" + ], + "id": "dca9824a" + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%%sql\n", + "SELECT * FROM Dragon;" + ], + "id": "1496e271", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Notice that `COUNT(*)` and `COUNT(cute)` result in different outputs.\n" + ], + "id": "3b502331" + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%%sql\n", + "SELECT year, COUNT(*)\n", + "FROM Dragon\n", + "GROUP BY year;" + ], + "id": "f03ba4d8", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%%sql\n", + "SELECT year, COUNT(cute)\n", + "FROM Dragon\n", + "GROUP BY year;" + ], + "id": "b2874e72", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "With this definition of `GROUP BY` in hand, let's update our SQL order of operations. Remember: *every* SQL query must list clauses in this order. \n", + "\n", + " SELECT \n", + " FROM \n", + " [WHERE ]\n", + " [GROUP BY ]\n", + " [ORDER BY ]\n", + " [LIMIT ]\n", + " [OFFSET ];\n", + "\n", + "Note that we can use the `AS` keyword to rename columns during the selection process and that column expressions may include aggregation functions (`MAX`, `MIN`, etc.).\n", + "\n", + "## Filtering Groups\n", + "\n", + "Now, what if we only want groups that meet a certain condition? `HAVING` filters groups by applying some condition across all rows in each group. We interpret it as a way to keep only the groups `HAVING` some condition. Note the difference between `WHERE` and `HAVING`: we use `WHERE` to filter rows, whereas we use `HAVING` to filter *groups*. `WHERE` precedes `HAVING` in terms of how SQL executes a query.\n", + "\n", + "Let's take a look at the `Dish` table to see how we can use `HAVING`. Say we want to group dishes with a cost greater than 4 by `type` and only keep groups where the max cost is less than 10.\n" + ], + "id": "60ad115f" + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%%sql\n", + "SELECT type, COUNT(*)\n", + "FROM Dish\n", + "WHERE cost > 4\n", + "GROUP BY type\n", + "HAVING MAX(cost) < 10;" + ], + "id": "1c4b2891", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here, we first use `WHERE` to filter for rows with a cost greater than 4. We then group our values by `type` before applying the `HAVING` operator. With `HAVING`, we can filter our groups based on if the max cost is less than 10.\n", + "\n", + "## Summary: SQL\n", + "With this definition of `GROUP BY` and `HAVING` in hand, let's update our SQL order of operations. Remember: *every* SQL query must list clauses in this order. \n", + "\n", + " SELECT \n", + " FROM
    \n", + " [WHERE ]\n", + " [GROUP BY ]\n", + " [ORDER BY ]\n", + " [LIMIT ]\n", + " [OFFSET ];\n", + "\n", + "Note that we can use the `AS` keyword to rename columns during the selection process and that column expressions may include aggregation functions (`MAX`, `MIN`, etc.).\n", + "\n", + "## EDA in SQL\n", + "In the last lecture, we mostly worked under the assumption that our data had already been cleaned. However, as we saw in our first pass through the data science lifecycle, we're very unlikely to be given data that is free of formatting issues. With this in mind, we'll want to learn how to clean and transform data in SQL. \n", + "\n", + "Our typical workflow when working with \"big data\" is:\n", + "\n", + "1. Use SQL to query data from a database\n", + "2. Use Python (with `pandas`) to analyze this data in detail\n", + "\n", + "We can, however, still perform simple data cleaning and re-structuring using SQL directly. To do so, we'll use the `Title` table from the `imdb_duck` database, which contains information about movies and actors.\n", + "\n", + "Let's load in the `imdb_duck` database.\n" + ], + "id": "6fabb79e" + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "import os\n", + "if os.path.exists(\"/home/jovyan/shared/sql/imdb_duck.db\"):\n", + " imdbpath = \"duckdb:////home/jovyan/shared/sql/imdb_duck.db\"\n", + "elif os.path.exists(\"data/imdb_duck.db\"):\n", + " imdbpath = \"duckdb:///data/imdb_duck.db\"\n", + "else:\n", + " import gdown\n", + " url = 'https://drive.google.com/uc?id=10tKOHGLt9QoOgq5Ii-FhxpB9lDSQgl1O'\n", + " output_path = 'data/imdb_duck.db'\n", + " gdown.download(url, output_path, quiet=False)\n", + " imdbpath = \"duckdb:///data/imdb_duck.db\"\n", + "print(imdbpath)" + ], + "id": "eeef2080", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "from sqlalchemy import create_engine\n", + "imdb_engine = create_engine(imdbpath, connect_args={'read_only': True})\n", + "%sql imdb_engine --alias imdb" + ], + "id": "1b29dd9d", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Since we'll be working with the `Title` table, let's take a quick look at what it contains. \n" + ], + "id": "fd0e63fc" + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%%sql imdb \n", + " \n", + "SELECT *\n", + "FROM Title\n", + "WHERE primaryTitle IN ('Ginny & Georgia', 'What If...?', 'Succession', 'Veep', 'Tenet')\n", + "LIMIT 10;" + ], + "id": "9e2ad6b1", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Matching Text using `LIKE`\n", + "\n", + "One common task we encountered in our first look at EDA was needing to match string data. For example, we might want to remove entries beginning with the same prefix as part of the data cleaning process.\n", + "\n", + "In SQL, we use the `LIKE` operator to (you guessed it) look for strings that are *like* a given string pattern. \n" + ], + "id": "f62e7365" + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%%sql\n", + "SELECT titleType, primaryTitle\n", + "FROM Title\n", + "WHERE primaryTitle LIKE 'Star Wars: Episode I - The Phantom Menace'" + ], + "id": "0ec5992b", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "What if we wanted to find *all* Star Wars movies? `%` is the wildcard operator, it means \"look for any character, any number of times\". This makes it helpful for identifying strings that are similar to our desired pattern, even when we don't know the full text of what we aim to extract.\n" + ], + "id": "0a385f94" + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%%sql\n", + "SELECT titleType, primaryTitle\n", + "FROM Title\n", + "WHERE primaryTitle LIKE '%Star Wars%'\n", + "LIMIT 10;" + ], + "id": "22e90c15", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Alternatively, we can use RegEx! DuckDB and most real DBMSs allow for this. Note that here, we have to use the `SIMILAR TO` operater rather than `LIKE`.\n" + ], + "id": "64b16995" + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%%sql\n", + "SELECT titleType, primaryTitle\n", + "FROM Title\n", + "WHERE primaryTitle SIMILAR TO '.*Star Wars*.'\n", + "LIMIT 10;" + ], + "id": "dd81b7fa", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### `CAST`ing Data Types\n", + "\n", + "A common data cleaning task is converting data to the correct variable type. The `CAST` keyword is used to generate a new output column. Each entry in this output column is the result of converting the data in an existing column to a new data type. For example, we may wish to convert numeric data stored as a string to an integer.\n" + ], + "id": "a1275863" + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%%sql\n", + "SELECT primaryTitle, CAST(runtimeMinutes AS INT)\n", + "FROM Title;" + ], + "id": "9f8f2499", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We use `CAST` when `SELECT`ing colunns for our output table. In the example above, we want to `SELECT` the columns of integer year and runtime data that is created by the `CAST`. \n", + "\n", + "SQL will automatically name a new column according to the command used to `SELECT` it, which can lead to unwieldy column names. We can rename the `CAST`ed column using the `AS` keyword.\n" + ], + "id": "cac7d842" + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%%sql\n", + "SELECT primaryTitle AS title, CAST(runtimeMinutes AS INT) AS minutes, CAST(startYear AS INT) AS year\n", + "FROM Title\n", + "LIMIT 5;" + ], + "id": "bb1f7d18", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Using Conditional Statements with `CASE`\n", + "\n", + "When working with `pandas`, we often ran into situations where we wanted to generate new columns using some form of conditional statement. For example, say we wanted to describe a film title as \"old,\" \"mid-aged,\" or \"new,\" depending on the year of its release.\n", + "\n", + "In SQL, conditional operations are performed using a `CASE` clause. Conceptually, `CASE` behaves much like the `CAST` operation: it creates a new column that we can then `SELECT` to appear in the output. The syntax for a `CASE` clause is as follows:\n", + "\n", + " CASE WHEN THEN \n", + " WHEN THEN \n", + " ...\n", + " ELSE \n", + " END\n", + "\n", + "Scanning through the skeleton code above, you can see that the logic is similar to that of an `if` statement in Python. The conditional statement is first opened by calling `CASE`. Each new condition is specified by `WHEN`, with `THEN` indicating what value should be filled if the condition is met. `ELSE` specifies the value that should be filled if no other conditions are met. Lastly, `END` indicates the end of the conditional statement; once `END` has been called, SQL will continue evaluating the query as usual. \n", + "\n", + "Let's see this in action. In the example below, we give the new column created by the `CASE` statement the name `movie_age`.\n" + ], + "id": "06d784bf" + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%%sql\n", + "/* If a movie was filmed before 1950, it is \"old\"\n", + "Otherwise, if a movie was filmed before 2000, it is \"mid-aged\"\n", + "Else, a movie is \"new\" */\n", + "\n", + "SELECT titleType, startYear,\n", + "CASE WHEN startYear < 1950 THEN 'old'\n", + " WHEN startYear < 2000 THEN 'mid-aged'\n", + " ELSE 'new'\n", + " END AS movie_age\n", + "FROM Title;" + ], + "id": "3a658a84", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## `JOIN`ing Tables\n", + "\n", + "At this point, we're well-versed in using SQL as a tool to clean, manipulate, and transform data in a table. Notice that this sentence referred to one *table*, specifically. What happens if the data we need is distributed across multiple tables? This is an important consideration when using SQL —— recall that we first introduced SQL as a language to query from databases. Databases often store data in a multidimensional structure. In other words, information is stored across several tables, with each table containing a small subset of all the data housed by the database. \n", + "\n", + "A common way of organizing a database is by using a **star schema**. A star schema is composed of two types of tables. A **fact table** is the central table of the database —— it contains the information needed to link entries across several **dimension tables**, which contain more detailed information about the data. \n", + "\n", + "Say we were working with a database about boba offerings in Berkeley. The dimension tables of the database might contain information about tea varieties and boba toppings. The fact table would be used to link this information across the various dimension tables.\n", + "\n", + "
    \n", + "multidimensional\n", + "
    \n", + "\n", + "If we explicitly mark the relationships between tables, we start to see the star-like structure of the star schema.\n", + "\n", + "
    \n", + "star\n", + "
    \n", + "\n", + "To join data across multiple tables, we'll use the (creatively named) `JOIN` keyword. We'll make things easier for now by first considering the simpler `cats` dataset, which consists of the tables `s` and `t`.\n", + "\n", + "
    \n", + "cats\n", + "
    \n", + "\n", + "To perform a join, we amend the `FROM` clause. You can think of this as saying, \"`SELECT` my data `FROM` tables that have been `JOIN`ed together.\" \n", + "\n", + "Remember: SQL does not consider newlines or whitespace when interpreting queries. The indentation given in the example below is to help improve readability. If you wish, you can write code that does not follow this formatting.\n", + "\n", + " SELECT \n", + " FROM table_1 \n", + " JOIN table_2 \n", + " ON key_1 = key_2;\n", + "\n", + "We also need to specify what column from each table should be used to determine matching entries. By defining these keys, we provide SQL with the information it needs to pair rows of data together.\n", + "\n", + "\n", + "The most commonly used type of SQL `JOIN` is the **inner join**. It turns out you're already familiar with what an inner join does, and how it works – this is the type of join we've been using in `pandas` all along! In an inner join, we combine every row in our first table with its matching entry in the second table. If a row from either table does not have a match in the other table, it is omitted from the output. \n", + "\n", + "
    \n", + "inner\n", + "
    \n", + "\n", + "In a **cross join**, *all* possible combinations of rows appear in the output table, regardless of whether or not rows share a matching key. Because all rows are joined, even if there is no matching key, it is not necessary to specify what keys to consider in an `ON` statement. A cross join is also known as a cartesian product.\n", + "\n", + "
    \n", + "cross\n", + "
    \n", + "\n", + "Conceptually, we can interpret an inner join as a cross join, followed by removing all rows that do not share a matching key. Notice that the output of the inner join above contains all rows of the cross join example that contain a single color across the entire row.\n", + "\n", + "In a **left outer join**, *all* rows in the left table are kept in the output table. If a row in the right table shares a match with the left table, this row will be kept; otherwise, the rows in the right table are omitted from the output. We can fill in any missing values with `NULL`.\n", + "\n", + "
    \n", + "left\n", + "
    \n", + "\n", + "A **right outer join** keeps all rows in the right table. Rows in the left table are only kept if they share a match in the right table. Again, we can fill in any missing values with `NULL`. \n", + "\n", + "
    \n", + "right\n", + "
    \n", + "\n", + "In a **full outer join**, all rows that have a match between the two tables are joined together. If a row has no match in the second table, then the values of the columns for that second table are filled with `NULL`. In other words, a full outer join performs an inner join *while still keeping* rows that have no match in the other table. This is best understood visually:\n", + "\n", + "
    \n", + "full\n", + "
    \n", + "\n", + "We have kept the same output achieved using an inner join, with the addition of partially null rows for entries in `s` and `t` that had no match in the second table. \n", + "\n", + "### Aliasing in `JOIN`s\n", + "\n", + "When joining tables, we often create aliases for table names (similarly to what we did with column names in the last lecture). We do this as it is typically easier to refer to aliases, especially when we are working with long table names. We can even reference columns using aliased table names.\n", + "\n", + "Let's say we want to determine the average rating of various movies:\n" + ], + "id": "70547df7" + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%%sql\n", + "\n", + "SELECT primaryTitle, averageRating\n", + "FROM Title AS T INNER JOIN Rating AS R\n", + "ON T.tconst = R.tconst;" + ], + "id": "d16ee02a", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Note that the `AS` is actually optional! We can create aliases for our tables even without it, but we usually include it for clarity.\n" + ], + "id": "fc095aea" + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%%sql\n", + "\n", + "SELECT primaryTitle, averageRating\n", + "FROM Title T INNER JOIN Rating R\n", + "ON T.tconst = R.tconst;" + ], + "id": "036e84f0", + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Common Table Expression\n", + "\n", + "For more sophisticated data problems, the queries can become very complex. Common Table Expressions allow us to break down these complex queries into more manageable parts. This involves creating temporary tables which correspond to different aspects of the problem and then referencing them in the final query. The following format is an example of how we can create two temporary tables and then use them for further querying:\n", + "\n", + " WITH \n", + " table_name1 AS ( \n", + " SELECT ...\n", + " ),\n", + " table_name2 AS ( \n", + " SELECT ...\n", + " )\n", + " SELECT ... \n", + " FROM \n", + " table_name1, \n", + " table_name2, ...\n", + "\n", + "Let's say we want to identify the top 10 action movies that are highly rated (with an average rating greater than 7) and popular (having more than 5000 votes), along with the primary actors who are the most popular, we can use Common Table Expression to break this query down into separate problems. Initially, we can filter to find good action movies and prolific actors separately. This way, in our final join, we only need to change the order.\n" + ], + "id": "f5c99bbe" + }, + { + "cell_type": "code", + "metadata": { + "vscode": { + "languageId": "python" + } + }, + "source": [ + "%%sql\n", + "WITH \n", + "good_action_movies AS (\n", + " SELECT *\n", + " FROM Title T JOIN Rating R ON T.tconst = R.tconst \n", + " WHERE genres LIKE '%Action%' AND averageRating > 7 AND numVotes > 5000\n", + "),\n", + "prolific_actors AS (\n", + " SELECT N.nconst, primaryName, COUNT(*) as numRoles\n", + " FROM Name N JOIN Principal P ON N.nconst = P.nconst\n", + " WHERE category = 'actor'\n", + " GROUP BY N.nconst, primaryName\n", + ")\n", + "SELECT primaryTitle, primaryName, numRoles, ROUND(averageRating) AS rating\n", + "FROM good_action_movies m, prolific_actors a, principal p\n", + "WHERE p.tconst = m.tconst AND p.nconst = a.nconst\n", + "ORDER BY rating DESC, numRoles DESC\n", + "LIMIT 10;" + ], + "id": "4cde9f1c", + "execution_count": null, + "outputs": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "jupytext": { + "text_representation": { + "extension": ".qmd", + "format_name": "quarto", + "format_version": "1.0", + "jupytext_version": "1.16.1" + } + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/sql_II/sql_II.qmd b/sql_II/sql_II.qmd index d5ed31ca..b1174400 100644 --- a/sql_II/sql_II.qmd +++ b/sql_II/sql_II.qmd @@ -39,16 +39,19 @@ In this lecture, we'll continue our work from last time to introduce some advanc First, let's load in the `basic_examples.db` database. ```{python} +#| vscode: {languageId: python} %load_ext sql ``` ```{python} +#| vscode: {languageId: python} import duckdb conn = duckdb.connect() conn.query("INSTALL sqlite") ``` ```{python} +#| vscode: {languageId: python} %sql duckdb:///data/basic_examples.db --alias basic ``` @@ -61,6 +64,7 @@ In `pandas`, much of our analysis work relied heavily on being able to use `.gro To illustrate `GROUP BY`, we will consider the `Dish` table from our database. ```{python} +#| vscode: {languageId: python} %%sql SELECT * FROM Dish; @@ -69,6 +73,7 @@ FROM Dish; Notice that there are multiple dishes of the same `type`. What if we wanted to find the total costs of dishes of a certain `type`? To accomplish this, we would write the following code. ```{python} +#| vscode: {languageId: python} %%sql SELECT type, SUM(cost) FROM Dish @@ -92,6 +97,7 @@ There are many aggregation functions that can be used to aggregate the data cont We can easily compute multiple aggregations all at once (a task that was very tricky in `pandas`). ```{python} +#| vscode: {languageId: python} %%sql SELECT type, SUM(cost), MIN(cost), MAX(name) FROM Dish @@ -103,6 +109,7 @@ To count the number of rows associated with each group, we use the `COUNT` keywo Recall the `Dragon` table from the previous lecture: ```{python} +#| vscode: {languageId: python} %%sql SELECT * FROM Dragon; ``` @@ -110,6 +117,7 @@ SELECT * FROM Dragon; Notice that `COUNT(*)` and `COUNT(cute)` result in different outputs. ```{python} +#| vscode: {languageId: python} %%sql SELECT year, COUNT(*) FROM Dragon @@ -117,6 +125,7 @@ GROUP BY year; ``` ```{python} +#| vscode: {languageId: python} %%sql SELECT year, COUNT(cute) FROM Dragon @@ -142,6 +151,7 @@ Now, what if we only want groups that meet a certain condition? `HAVING` filters Let's take a look at the `Dish` table to see how we can use `HAVING`. Say we want to group dishes with a cost greater than 4 by `type` and only keep groups where the max cost is less than 10. ```{python} +#| vscode: {languageId: python} %%sql SELECT type, COUNT(*) FROM Dish @@ -178,6 +188,7 @@ We can, however, still perform simple data cleaning and re-structuring using SQL Let's load in the `imdb_duck` database. ```{python} +#| vscode: {languageId: python} import os if os.path.exists("/home/jovyan/shared/sql/imdb_duck.db"): imdbpath = "duckdb:////home/jovyan/shared/sql/imdb_duck.db" @@ -193,6 +204,7 @@ print(imdbpath) ``` ```{python} +#| vscode: {languageId: python} from sqlalchemy import create_engine imdb_engine = create_engine(imdbpath, connect_args={'read_only': True}) %sql imdb_engine --alias imdb @@ -201,6 +213,7 @@ imdb_engine = create_engine(imdbpath, connect_args={'read_only': True}) Since we'll be working with the `Title` table, let's take a quick look at what it contains. ```{python} +#| vscode: {languageId: python} %%sql imdb SELECT * @@ -216,6 +229,7 @@ One common task we encountered in our first look at EDA was needing to match str In SQL, we use the `LIKE` operator to (you guessed it) look for strings that are *like* a given string pattern. ```{python} +#| vscode: {languageId: python} %%sql SELECT titleType, primaryTitle FROM Title @@ -225,6 +239,7 @@ WHERE primaryTitle LIKE 'Star Wars: Episode I - The Phantom Menace' What if we wanted to find *all* Star Wars movies? `%` is the wildcard operator, it means "look for any character, any number of times". This makes it helpful for identifying strings that are similar to our desired pattern, even when we don't know the full text of what we aim to extract. ```{python} +#| vscode: {languageId: python} %%sql SELECT titleType, primaryTitle FROM Title @@ -235,6 +250,7 @@ LIMIT 10; Alternatively, we can use RegEx! DuckDB and most real DBMSs allow for this. Note that here, we have to use the `SIMILAR TO` operater rather than `LIKE`. ```{python} +#| vscode: {languageId: python} %%sql SELECT titleType, primaryTitle FROM Title @@ -247,6 +263,7 @@ LIMIT 10; A common data cleaning task is converting data to the correct variable type. The `CAST` keyword is used to generate a new output column. Each entry in this output column is the result of converting the data in an existing column to a new data type. For example, we may wish to convert numeric data stored as a string to an integer. ```{python} +#| vscode: {languageId: python} %%sql SELECT primaryTitle, CAST(runtimeMinutes AS INT) FROM Title; @@ -257,6 +274,7 @@ We use `CAST` when `SELECT`ing colunns for our output table. In the example abov SQL will automatically name a new column according to the command used to `SELECT` it, which can lead to unwieldy column names. We can rename the `CAST`ed column using the `AS` keyword. ```{python} +#| vscode: {languageId: python} %%sql SELECT primaryTitle AS title, CAST(runtimeMinutes AS INT) AS minutes, CAST(startYear AS INT) AS year FROM Title @@ -280,6 +298,7 @@ Scanning through the skeleton code above, you can see that the logic is similar Let's see this in action. In the example below, we give the new column created by the `CASE` statement the name `movie_age`. ```{python} +#| vscode: {languageId: python} %%sql /* If a movie was filmed before 1950, it is "old" Otherwise, if a movie was filmed before 2000, it is "mid-aged" @@ -365,11 +384,12 @@ We have kept the same output achieved using an inner join, with the addition of ### Aliasing in `JOIN`s -Generally, when joining tables, we often create aliases for table names (similarly to what we did with column names in the last lecture). It is typically easier to refer to aliases, especially when we are working with long table names, and can even reference columns using aliased table names. +When joining tables, we often create aliases for table names (similarly to what we did with column names in the last lecture). We do this as it is typically easier to refer to aliases, especially when we are working with long table names. We can even reference columns using aliased table names. Let's say we want to determine the average rating of various movies: ```{python} +#| vscode: {languageId: python} %%sql SELECT primaryTitle, averageRating @@ -380,9 +400,52 @@ ON T.tconst = R.tconst; Note that the `AS` is actually optional! We can create aliases for our tables even without it, but we usually include it for clarity. ```{python} +#| vscode: {languageId: python} %%sql SELECT primaryTitle, averageRating FROM Title T INNER JOIN Rating R ON T.tconst = R.tconst; ``` + +### Common Table Expression + +For more sophisticated data problems, the queries can become very complex. Common Table Expressions allow us to break down these complex queries into more manageable parts. This involves creating temporary tables which correspond to different aspects of the problem and then referencing them in the final query. The following format is an example of how we can create two temporary tables and then use them for further querying: + + WITH + table_name1 AS ( + SELECT ... + ), + table_name2 AS ( + SELECT ... + ) + SELECT ... + FROM + table_name1, + table_name2, ... + +Let's say we want to identify the top 10 action movies that are highly rated (with an average rating greater than 7) and popular (having more than 5000 votes), along with the primary actors who are the most popular, we can use Common Table Expression to break this query down into separate problems. Initially, we can filter to find good action movies and prolific actors separately. This way, in our final join, we only need to change the order. + +```{python} +#| vscode: {languageId: python} +%%sql +WITH +good_action_movies AS ( + SELECT * + FROM Title T JOIN Rating R ON T.tconst = R.tconst + WHERE genres LIKE '%Action%' AND averageRating > 7 AND numVotes > 5000 +), +prolific_actors AS ( + SELECT N.nconst, primaryName, COUNT(*) as numRoles + FROM Name N JOIN Principal P ON N.nconst = P.nconst + WHERE category = 'actor' + GROUP BY N.nconst, primaryName +) +SELECT primaryTitle, primaryName, numRoles, ROUND(averageRating) AS rating +FROM good_action_movies m, prolific_actors a, principal p +WHERE p.tconst = m.tconst AND p.nconst = a.nconst +ORDER BY rating DESC, numRoles DESC +LIMIT 10; +``` + +