https://derring-do.github.io/syllabus/
This is a cursorily-curated, ever-evolving brain dump of resources related to the kinds of work I do and teach others to do {:.test}
Paywalls, subscriptions, newsletters
- We Launched a Paywall. It Worked! Mostly.
- Be smart: How Axios drives engagement with its email newsletters through user-level data
- One subscriber or 48,000 pageviews: Why every journalist should know the “unit economics” of their content
- Digital Pay-Meter Playbook: How Today’s News Publishers Can Use Data, Best Practices, and Test-and-Learn Tactics To Build Better Pay-Meters
- COPE: Create Once, Publish Everywhere (NPR)
- The next business revolution: American business schools are reinventing the MBA
-
Head, K. & Harsin, A. (2017). Quasi-experimental design. In M. Allen (Ed.), The sage encyclopedia of communication research methods (pp. 1384-1387). Thousand Oaks, CA: SAGE Publications, Inc doi: 10.4135/9781483381411.n478: link: Overview of quasi-experimental designs
-
Salganik, M. (2018). Bit by bit : Social research in the digital age. Princeton, New Jersey: Princeton University Press. Quick read: <8 hours
Publisher Summary
An innovative and accessible guide to doing social research in the digital age. In just the past several years, we have witnessed the birth and rapid spread of social media, mobile phones, and numerous other digital marvels. In addition to changing how we live, these tools enable us to collect and process data about human behavior on a scale never before imaginable, offering entirely new approaches to core questions about social behavior. Bit by Bit is the key to unlocking these powerful methods-a landmark book that will fundamentally change how the next generation of social scientists and data scientists explores the world around us. Bit by Bit is the essential guide to mastering the key principles of doing social research in this fast-evolving digital age. In this comprehensive yet accessible book, Matthew Salganik explains how the digital revolution is transforming how social scientists observe behavior, ask questions, run experiments, and engage in mass collaborations. He provides a wealth of real-world examples throughout, and also lays out a principles-based approach to handling ethical challenges in the era of social media. Bit by Bit is an invaluable resource for social scientists who want to harness the research potential of big data and a must-read for data scientists interested in applying the lessons of social science to tomorrow's technologies.
Introductory text to online social research. Chapters are concise and accessible with mathematical notes and "What to Read Next" references and exercises at end. The online platform also has Hypothesis embedded, which is a generally useful plugin for all-purpose/cross-Internet annotation. (free account required)
-
Hall, E. (2014). Just Enough Research (1st ed.). A Book Apart. Very quick read: <3 hours
Publisher Summary
Design research is a hard slog that takes years to learn and time away from the real work of design, right? Wrong. Good research is about asking more and better questions, and thinking critically about the answers. It’s something every member of your team can and should do, and which everyone can learn, quickly. And done well, it will save you time and money by reducing unknowns and creating a solid foundation to build the right thing, in the most effective way. In Just Enough Research, co-founder of Mule Design Erika Hall distills her experience into a brief cookbook of research methods. Learn how to discover your competitive advantages, spot your own blind spots and biases, understand and harness your findings, and why you should never, ever hold a focus group. You’ll start doing good research faster than you can plan your next pitch.
-
So You Think You Can Test? (Simulation game by Lukas Vermeer): "Decision making under uncertainty is complicated business. This game aims to make decision makers more aware of the complex trade off between indecision and acting on insufficient information." (Read the instructions and play a few rounds in simulation mode)
-
Three key checklists and remedies for trustworthy analysis of online controlled experiments at scale or summary blog post
"experiment analysis has traditionally been done by experienced data analysts and scientists that closely monitored experiments throughout their lifecycle. Depending solely on scarce experts, however, is neither scalable nor bulletproof. To democratize experimentation, analysis should be streamlined and meticulously performed by engineers, managers, or others responsible for the development of a product."
-
Getting Past Statistical Significance: Foundations of AB Testing and Experimentation
-
Chasing Statistical Ghosts in Experimentation (3-part series):
-
Leaky Abstractions In Online Experimentation Platforms: Categorising Common Challenges: How things go awry in online A/B testing
-
Understanding Experimentation Platforms: Optimizely white paper
-
Top Challenges from the first Practical Online Controlled Experiments Summit
-
One of the most useful indicators of a variety of data quality issues is a Sample Ratio Mismatch (SRM) ? the situation when the observed sample ratio in the experiment is different from the expected. Just like fever is a symptom for multiple types of illness, an SRM is a symptom for a variety of data quality issues. While a simple statistical check is used to detect an SRM, correctly identifying the root cause and preventing it from happening in the future is often extremely challenging and time consuming. Ignoring the SRM without knowing the root cause may result in a bad product modification appearing to be good and getting shipped to users, or vice versa. The goal of this paper is to make diagnosing, fixing, and preventing SRMs easier.
-
When the Revolution Came for Amy Cuddy
- "As a young social psychologist, she played by the rules and won big: an influential study, a viral TED talk, a prestigious job at Harvard. Then, suddenly, the rules changed."
-
Suffering from a Non-inferiority Complex?: Good, simplified examples and illustrations
- "Seaching 80% of your room for your backpack is a bit like performing a test with 80% power. If you don't find your backpack after searching only 80% of the room, should you conclude the backpack is not in your room?"
- How Etsy Handles Peeking in A/B Testing
- Berman, Ron and Pekelis, Leonid and Scott, Aisling and Van den Bulte, Christophe, p-Hacking and False Discovery in A/B Testing (December 11, 2018). Available at SSRN: https://ssrn.com/abstract=3204791 or http://dx.doi.org/10.2139/ssrn.3204791
- Non-Inferiority Designs in A/B Testing
- Alternatives to the Randomized Controlled Trial
- Walker, E., & Nowacki, A. S. (2010). Understanding equivalence and noninferiority testing. Journal of general internal medicine, 26(2), 192–196. doi:10.1007/s11606-010-1513-8: link
- Reducing A/B test measurement variance by 30%+ (TripAdvisor)
- How Booking.com increases the power of online experiments with CUPED
- Statistics for people in a hurry
- Glossary of Statistical Terms (Frank Harrell)
- StatQuest: Videos
- Causal Inference Animated Plots: This outlines some non-experimental designs and methods to visualize relationships in situations where a randomized control experiment isn't feasible.
- The Book of Why: The New Science of Cause and Effect. 2018. Judea Pearl and Dana Mackenzie: Gives historical context to why the language of frequentist hypothesis testing seems so constrained re: causal claims.
- Andrew Gelman's Applied Regression exam questions and answers
-
Adobe’s performanceTiming plugin, with some improvements and an explanation
-
33 Tangents – Episode #29 – Publishing Analytics with Todd Schauman of The Christian Science Monitor (50 min)
-
Digital Analytics Power Hour podcast: Most episodes have transcripts. I'd just work your way through most of their catalog over the next six months -- make note of episodes that are particularly helpful or surprisingly not, and we can add them here.
-
Neumann, Nico and Tucker, Catherine E. and Whitfield, Timothy, How Effective Is Black-box Digital Consumer Profiling and Audience Delivery?: Evidence from Field Studies (June 25, 2018). Available at SSRN: https://ssrn.com/abstract=3203131 or http://dx.doi.org/10.2139/ssrn.3203131: link: This is about buying third-party data from brokers, but provides a good overview of the digital tracking technology in general
-
#measure Slack: free community for people working in digital analytics, broadly defined; includes channels for specific Adobe products, general data science, R and statistics, Google Analytics, testing methodology, etc. Must submit form to receive invitation.
-
This is how it works in the advertising context: A marketer or agency loads a virgin computer or device – one that’s never been connected to the internet before – with its first-party data, like an email list, for example. A second clean machine on the Facebook side is loaded with impression-level and non-PII campaign data.
The machines can’t ever have touched the internet so that there’s no possibility of outside interference or interception of the data.
The proprietary data coming from Facebook is then combined with the advertiser’s first-party data, matches are made and the data is analyzed to extract aggregated insights that one or the other couldn’t gather on its own.
- Pew Research Center Methods 101 YouTube Playlist: Currently 3 videos all ~5 min or less
- Harvard University Program on Survey Research
- Representative Samples: Does Sample Size Really Matter?
- Margin of error
- 10 Best Practices in Survey Screening: Don't terminate respondents after sensitive demographic questions
- Yes, There is a Right and Wrong Way to Number Rating Scales (SurveyMonkey): Understand unipolar vs. bipolar scales
- How much do the numbers used in survey scales really matter?
- These outcomes suggest that when a scale is easily divided in half — for example, when the maximum value is 6 rather than 7 — it’s more likely for respondents to select the midpoint. Previous research has found that respondents are likely to assume that half the top endpoint is a scale’s midpoint, so when half the top endpoint is not an answer option (e.g., 3.5 on the 1–7 scale), respondents seeking the central point on the scale may sometime choose 3 (not the scale midpoint) and sometimes choose 4 (the actual scale midpoint). Those who received the 0–6 scale could more easily find the midpoint (3) by halving the top endpoint.
- Comparing Forced-Choice and Select-All Online Survey Responses - Pew Research Center Methods
- The new study found compelling evidence that forced-choice questions yield more accurate results than select-all-that-apply lists.
- Survey rating scales: numbered vs worded lists (SurveyMonkey)
- What is a Likert scale? (SurveyMonkey)
- A survey resolution: Stop throwing away your variance! (SurveyMonkey)
- What’s the best way to design a matrix question? (SurveyMonkey)
- Let’s agree NOT to use agree/disagree questions (SurveyMonkey)
- Why (and how!) to ask survey questions on sexual orientation and gender identity (SurveyMonkey)
- Huang, Jason & Bowling, Nathan & Liu, Mengqiao & Li, Yuhui. (2014). Detecting Insufficient Effort Responding with an Infrequency Scale: Evaluating Validity and Participant Reactions. Journal of Business and Psychology. 10.1007/s10869-014-9357-6: link: Understand what IER is, what it might look like, pros and cons of various countermeasures (longstring, psychometric antonym/synonym, Mahalanobis distance, manipulation/attention check)
- Net Promoter Score Pros and Cons: Why Use NPS? (SurveyMonkey)
- Net Promoter Score (Wikipedia)
- Respectful Collection of Demographic Data
- SurveyMonkey's survey science blog
- Andrew Gelman's Design and Analysis of Sample Surveys exam questions and answers
- "18. A survey is taken of 100 undergraduates, 100 graduate students, and 100 continuing education students at a university. Assume a simple random sample within each group. Each student is asked to rate his or her satisfaction (on a 1–10 scale) with his or her experiences. Write the estimate and standard error of the average satisfaction of all the students at the university. Introduce notation as necessary for all the information needed to solve the problem."
- Questionnaire design and translation (Pew)
- The Unique Challenges of Surveying U.S. Latinos (Pew):
- Kleiner, B., Pan, Y., & Bouic, J. (2009). The Impact of Instructions on Survey Translation: An Experimental Study. Survey Research Methods, 3(3), 113-122. doi: http://dx.doi.org/10.18148/srm/2009.v3i3.1563: link: Includes translator instructions template
- Gallup Worldwide Research Methodology and Codebook (2017)
- ESOMAR 28 Questions to Help Research Buyers of Online Samples: Overview of terminology and considerations re: panel samples. Every reputable sample provider should have a document answering these questions.
- Anson, I. G. (2018). Taking the time? Explaining effortful participation among low-cost online survey participants. Research & Politics. https://doi.org/10.1177/2053168018785483: link
- Working knowledge of HTML, CSS, JavaScript, and other internet things for designing tests, debugging, customizing surveys and intercepts.
- Understand how to navigate the DOM to select elements to change and make needed changes
- Understand our stylesheet and when/how to use/preserve existing helper classes, esp. for mobile breakpoints (e.g., hide-for-small vs. small-only)
- Write code that is slightly more efficient than your average bear to minimize burden on the site; faster site = less likely that what we observe is due to difference in site loading time.
- Don't break the site
- How a webpage is loaded and displayed
- Introduction to HTML
- Introduction to CSS
- CSS Diner: Interactive game to get used to CSS selectors
- Responsive CSS
- Flexbox Froggy: Interactive game to understand the flex framework
- JavaScript
- The Vanilla JS Toolkit: "A quick reference for commonly used JavaScript methods and browser APIs."
- JavaScript control flow and error handling: Write code that breaks usefully
- JavaScript Regex:
- Principles of Accessible and Universal Design
- More JavaScript: Types, functions, loops/iterators, truthy/falsy, regex, IIFE/self-executing anonymous functions, DOM navigation
document.designMode = 'on': demo- Browser Detection (and What to Do Instead)
- Why is getElementsByTagName() faster than querySelectorAll()?
- James Mickens -- "To Wash It All Away"
Anyways, my point is that browsers are too complex to be trusted. Unfortunately, youth is always wasted on the young, and the current generation of software developers is convinced that browsers need more features, not fewer. So, we are encouraged to celebrate the fact that browsers turn our computers into little Star Wars cantinas where everyone is welcome and you can drink a blue drink if you want to drink a blue drink and if something bad happens, then maybe a Jedi will save you, and if not, HEY IT’S A STAR WARS CANTINA YESSSSS. Space cantinas are fun, but they’re just a fantasy; they’re just a series of outlandish details stitched together to amuse and entertain. You have to open your eyes and see that in the real, non-hyperbolic world that you actually inhabit, your browser will frequently stop playing a video and then display flashing epilepsy pixels while making the sound that TVs make in Japanese horror movies before a pasty salamander child steps out of the screen and voids your warranty. That’s a thing which could actually happen, and we should wash it all away
- Understanding Specificity in CSS - Jess Mitchell
Specificity can sometimes feeling counter-intuitive, especially if you’re stuck in a downward spiral of adding more and more selectors to your CSS. One way I find helpful to think about specificity is by thinking of a company with clear seniority levels
Web privacy technology and policies are evolving, and, with them, analytics platforms adapt with various levels of transparency. What we might observe as a steep decline in visitors might actually be attributable to new browser policies. Some primers:
- What Does Private Browsing Mode Do?
- GDPR
- ITP 2.1 and Web Analytics: Intelligent Tracking Prevention (ITP) is Apple's version of privacy features for Safari
- Google announces two new privacy-focused features for Chrome at the I/O 2019 developer conference
- The nascent DETOUR ACT in Congress calls for data-collecting restrictions on websites with over 100 million active users
Generally, the site is up to date on the latest vulnerabilities, but when we write code for experiments or update (or fail to update) vendor libraries, we should try not to override any of the guardrails... Relevant topics from the OWASP Cheat Sheet Series ("Life is too short -- AppSec is hard -- Cheat!"):
Best practices for organizing and visualizing; tutorials for scripted languages
- Tukey, Design Thinking, and Better Questions - Roger Peng
- Corollary: Tukey, John W. The Future of Data Analysis. Ann. Math. Statist. 33 (1962), no. 1, 1--67. doi:10.1214/aoms/1177704711. https://projecteuclid.org/euclid.aoms/1177704711: link
- 11 Tips for Improving Productivity using OneNote
- How Data Scientists Think - A Mini Case Study
- Wickham, H. (2014). Tidy Data. Journal of Statistical Software, 59(10), 1 - 23. doi:http://dx.doi.org/10.18637/jss.v059.i10: How to organize data
- Data Organization in Spreadsheets
- Designing Projects: directory layout example
- Markdown
- Command line crash course
- Pro Git: Clone, commit, push, checkout branches, pull requests
- The Big Book of Dashboards
- Ch. 1: Data Visualization: A Primer
- Ch. 6: Ranking by Now, Comparing with Then
- Ch. 24: Showing Churn or Turnover
- Glossary of Chart Types (good reference for Google searches, e.g. "[type of chart] in [programming language/viz tool]"
- Mistakes, we've drawn a few: Learning from our errors in data visualization (The Economist)
- The laws of shitty dashboards
- Stephen Few's "Show Me the Numbers: Designing Tables & Graphs to Enlighten" is available in hard copy (2012) and an older version is available as workshop slides here (they are decently annotated and standalone)
- CRAP Talks #12 – Annabel St John-Lyle: Design Principles for Dashboards: ~30 minute video
- Fundamentals of Data Visualization (Claus Wilke)
- Avoid bar plots for continuous data! Do this instead: (Paul van der Laken)
- TidyTuesday: Weekly social data project in R organized by the R4DS online learning community. Each week, they post a raw dataset for people to explore and share via the #tidytuesday Twitter hashtag.
- Uncertainty + Visualization, Explained
Generally, having a well-defined project is the best way to learn. You'll end up searching for and finding more specific tutorials for the task at hand, but here are some places to get started:
- 20-minute end-to-end data analysis by Hadley Wickham: Crash course/example workflow; code/data linked in video description
- UBC's STAT545 curriculum
- Practical R for Mass Communication and Journalism: Chapters 1-2 will get you through installation/setup
- Wrangling data in the Tidyverse - Part 1 (1 hr 15 min)
- How to Use R with Excel
- Intro to R for Microsoft Excel Users
- Regular Expressions in R – Part 1: Introduction and base R functions
- RMarkdown
- Utah R Users Group YouTube channel, starting with:
- Exploratory data analysis with the tidyverse (40 min)
- Intro to data visualization with R (1 hour)
- Branding and Automating Your Work with R Markdown (47 min)
- Debugging and Profiling (1 hour)
- The lazy and easily distracted report writer: Using rmarkdown and parameterised reports - Mike K Smith (15 min)
- Happy Git and GitHub for the useR
- Building an A/B Testing Analytics System with R and Shiny (Emily Robinson , rstudio::conf 2019)
- Data Science in a Box: All lecture slides in "Exploring data" section
- R for Marketing Research and Analytics. Chapman & Feit 2015: Book with exercises
- rstudio-conf 2019 Workshop materials
- plotly: D3.js wrapper to make interactives from ggplot graphics
Unless otherwise noted, these are free and/or open source with no account creation required. For some of these, a tradeoff of the ease of use is the lack of security re: proprietary data as well as limited persistence, so be vigilant!
- Jira/JQL + Confluence: tk
- VisBug: Chrome Extension that adds a design toolbar to any webpage for easier WYSIWIG mock-ups and inspection. Demo: GIF
- Aggie and AWW App: Collaborative painting/whiteboarding
- GlooMaps: Visual Sitemap Tool
- wireframe.cc: Minimal wireframing
- draw.io: Flowcharts and UML diagrams; for mobile, I like DrawExpress, which has an intuitive gesture system
- Document Design Mode in any browser: Edit a web page like it's a Word doc
- mermaidJS: Markdown-based flowcharts and gantt charts; has VSCode extension for live preview; can be published in Github .MDs by sourcing the JS library. Here's an example illustrating my reference management system:
- VLC media player is a free and open source media player that runs on all platforms, plays all formats, and has useful playback features, e.g., speed up/slow down
- ShareX (Windows): screenshots, video/GIF recording; install ffmpeg when prompted in GUI
- LosslessCut is a fast, cross-platform tool for quick video editing (doesn't re-encode video after changes, so edited video is available near-instantly)
- ffmpeg is a command line tool that can screen and video capture, encode/decode, convert, and more -- can be baked into automated workflows
Last-ditch effort to provide novel information to savvy readers: