tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following two boxes first! By \"running\", we mean click on the box of code, and then press `Shift+Return` on your keyboard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from causality_simulation import *\n",
    "\n",
    "sample_size = 100\n",
    "\n",
    "init_data = {\n",
    "    'Height (cm)': np.array(sorted(np.append(np.random.normal(167, 8, sample_size //2), \n",
    "                                             np.random.normal(160, 8, sample_size//2 + sample_size % 2))))\n",
    "}\n",
    "basketball.init(init_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a short tutorial to teach you how to use the RCT simulator and view the data.\n",
    "\n",
    "You will also see how systematic and statistical errors creep into experimental settings, and you will practice using the most common techniques that scientists use to reduce these errors. \n",
    "\n",
    "# High school basketball study\n",
    "\n",
    "You are a high school PE teacher. You are interested to find out about how much a student's height and the amount of practice time affect his/her basketball skills. To do this, you perform an __observational study__ by passively recording the success rate of various kinds of shots made by every student in your PE class. For every student, his/her height is recorded, as well as the number of hours he/she spends each week playing basketball. You have a total of 100 students.\n",
    "\n",
    "We seek to measure the effect size of height and practice time on success rate. In this experiment, we operationalize \"effect size\" to be the simple [correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) between our two variables (height and success rate or practice time and success rate). As is the case whenever we are taking measurements, there will be possible sources of both systematic error and statistical error that we will seek to mitigate. Systematic and statistical errors are two distinct problems, and must be handled in different ways by different techniques, as we will see below.\n",
    "\n",
    "Run the following box.\n",
    "\n",
    "To begin, you must first assign the students to experimental groups. Since you are merely observing the students, there is only one group, consisting of all 100 students. In the following interface, you may type in __Observation__ for \"Name\" and __1-100__ for \"Assigned samples\". This means that student number 1 through student number 100 are all assigned to this group to be observed. Then, click on \"Visualise assignment\" to finalise the group assignment.\n",
    "\n",
    "In the \"Student Group Assignments\" chart, each vertical bar represents a single student. The length of the bar indicates the height of that student. The students are lined up from shortest on the left to tallest on the right. The shortest student is numbered 1, and the tallest student is numbered 100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "experiment = BasketballExperiment()\n",
    "experiment.assignment()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experimental setup\n",
    "\n",
    "Now we must specify how the experiment is done on the single group assigned above. Run the following box. In the interface, leave all the lines on __No intervention__. This means that you are not performing any experimental intervention that may change the behaviour of your students. In other words, you are sitting back and observing the students behave in their natural setting.\n",
    "\n",
    "Once you have made the settings, click on \"Perform experiment\". You should then see \"Data from experiment collected!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment.setting(show=['Shot type', 'Hours practised per week', 'Success rate'], disable=['Success rate'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data visualisation\n",
    "\n",
    "Now you would like to see the relationship between the student's height and the success rate of his/her shots. First, run the following box. Use the dropdown menu to select `Height (cm)` as the x-axis variable and `Success Rate` as the y-axis variable. You should see a scatter plot where each point represents the data from one student."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "experiment.plot(show=['Shot type', 'Hours practised per week', 'Height (cm)', 'Success rate'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 1\n",
    "\n",
    "__1a) Is there a correlation between `Success Rate` and the `Height`?__\n",
    "\n",
    "\n",
    "Now try to change the x-axis and y-axis variables using the dropdown menu. Using these charts, answer the following questions.\n",
    "\n",
    "__1b) Is there a correlation between `Success Rate` and the `Hours Practised Per Week`?__\n",
    "\n",
    "__1c) Which `Shot Type` has the highest `Success Rate`?__\n",
    "\n",
    "You would also like to find out how much time students are spending on basketball practice. Select `Hours Practised per Week` for x-axis variable and `None (Distributions Only)` for y-axis variable. You should now see a histogram. __1d) How many students practise 1 to 3 hours per week?__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Double-click this box to type your answer. Press Shift+Return when you are done.`\n",
    "\n",
    "1a)\n",
    "\n",
    "1b)\n",
    "\n",
    "1c)\n",
    "\n",
    "1d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experimental intervention\n",
    "\n",
    "Now you would like to perform an experiment to see if practising more hours per week will improve students' success rate. You give every student an assignment to practise exactly some number of hours per week. Some students are asked not to practise at all, some are asked to practise 14 hours a week, while others are assigned something in between. After a few weeks, you observe the students' performance during your PE lesson.\n",
    "\n",
    "Just as before, we are only setting up one experimental group. Run the following box. In \"Name\", type __Intervention__. In \"Assigned samples\", type __1-100__, so that all 100 students will participate in this experiment. Then, click on \"Visualise assignment\". The bar chart should look the same as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment2 = BasketballExperiment()\n",
    "experiment2.assignment()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following box. To assign a range of practice hours to the students, select __Range__ for \"Hours Practised per Week\" and type in __from 0 to 14__. Leave \"Shot Type\" on __No intervention__, so that the students can still freely choose how they want to make their shots without your intervention. Once you're done, click on \"Perform experiment\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment2.setting(show=['Shot type', 'Hours practised per week', 'Success rate'], disable='Success rate')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following box. To find out the relationship between hours practised and success rate, select __Hours Practised per Week__ for the x-axis variable and __Success Rate__ for the y-axis variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment2.plot(show=['Shot type', 'Hours practised per week', 'Height (cm)', 'Success rate'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__2a) What is the correlation between success rate and number of hours practised per week?__\n",
    "\n",
    "__2b) Looking at the p-value, is the correlation statistically significant?__ Recall that the p-value tells you the probability that this level of correlation can be produced simply by random chance. The lower the p-value, the stronger the statistical significance of the observed correlation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Type your answer here`\n",
    "\n",
    "2a)\n",
    "\n",
    "2b)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiple experimental groups\n",
    "\n",
    "To improve your experiment even further, you decide to repeat the above experiment while separately recording the success rates of three different types of shots: above head, layup, and hook shot. For this, we need to create 3 experimental groups.\n",
    "\n",
    "Run the following box. To add two new groups, click on __Add another group__ two times. Each group should be named __Above head__, __Layup__, and __Hook shot__, respectively. You then need to assign the students to each of the groups. For example, you can assign __1-33__ to the first group, __34-66__ to the second group, and __67-100__ to the third group. (Remember that 1 is the shortest student, and 100 is the tallest student.)\n",
    "\n",
    "Once you're done, click on \"Visualise assignment\". You should see a bar chart with three different colours, each corresponding to a group that you've assigned. Notice that the shortest students are all assigned to one group, and the tallest students are all assigned to a different group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment3 = BasketballExperiment()\n",
    "experiment3.assignment()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following box. In each of the three groups, select __Range__ for \"Hours Practised per Week\" and type __from 0 to 14__. Next, select __Fixed__ for \"Shot Type\" and use the dropdown menu to select the shot type corresponding to the group. Once you're done, click on \"Perform experiment\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment3.setting(show=['Shot type', 'Hours practised per week', 'Success rate'], disable='Success rate')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following box. Select __Hours Practised per Week__ as the x-axis variable and __Success Rate__ as the y-axis variable. You should see three sets of data points in three different colours, corresponding to the three experimental groups or shot types. To show/hide a single experimental group, click on the items in the legend on the right side of the graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment3.plot(show=['Shot type', 'Hours practised per week', 'Height (cm)', 'Success rate'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__3a) According to this graph, which shot type is the most successful?__\n",
    "\n",
    "__3b) Is there a correlation between hours practised per week and the success rate? Which shot type has the strongest correlation?__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Type your answer here`\n",
    "\n",
    "3a)\n",
    "\n",
    "3b)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Randomise assignment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may be unconvinced by the above comparison of the success rates of different shot types, as one shot type is assigned to only short people, while another shot type is assigned to only tall people. We have introduced a source of _systematic bias_ into our experiment! \n",
    "\n",
    "To mitigate this problem, we will repeat the previous experiment, but we will randomise the assignment of students into each of the three experimental groups. As we have discussed in class, if sources of error skew in random directions between groups, we can average out these effects, and we are left with only statistical uncertainty, rather than systematic uncertainty.\n",
    "\n",
    "Run the following box. Just as before, create two more groups and give each of the three groups a descriptive name. Instead of typing a simple range of numbers into \"Assigned samples\", click on \"Randomise assignment\". You'll see that the textboxes are automatically filled with many numbers. These are the students that are randomly assigned to each group. Now, every group has short and tall people.\n",
    "\n",
    "Once you're done, click on \"Visualise assignment\". You should see a bar chart where the three colours are all mixed up, rather than neatly separated by height. Again, you can click on the legend items to show/hide each group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment4 = BasketballExperiment()\n",
    "experiment4.assignment()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following box. Just as before, select __Hours Practised per Week__ as the x-axis variable and __Success Rate__ as the y-axis variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment4.setting(show=['Shot type', 'Hours practised per week', 'Success rate'], disable='Success rate')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment4.plot(show=['Shot type', 'Hours practised per week', 'Height (cm)', 'Success rate'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__4a) According to this graph, which shot type is the most successful?__\n",
    "\n",
    "__4b) Is there a correlation between hours practised per week and the success rate? Which shot type has the strongest correlation?__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Type your answer here`\n",
    "\n",
    "4a)\n",
    "\n",
    "4b)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Statistical Uncertainty/Error\n",
    "Statistical uncertainty/error arises due to random chance in selecting our sample. For example, suppose we only selected 5 players to observe. It could very well be that we got an unrepresentative sample where tall players are worse at basketball and the shorter ones happen to be very good. We would have a larger degree of uncertainty due to the small sample size. In general, we can reduce statistical uncertainty by increasing our sample size. In this experiment, that means selecting more players.\n",
    "\n",
    "One very common way that statisticians convey the amount of statistical uncertainty is with a [confidence interval](https://en.wikipedia.org/wiki/Confidence_interval). A wider confidence interval implies more statistical uncertainty, and a narrower confidence interval implies less statistical uncertainty.\n",
    "\n",
    "In the very first cell, there is a variable called `sample_size`. Try playing around with taking very large and very small sample sizes, and re-running experiments to see how the confidence intervals change. You can find the confidence interval for the correlation value printed below the plot.\n",
    "\n",
    "__5a) How does increasing the sample size affect the width of our confidence intervals for the correlation coefficient?__\n",
    "\n",
    "__5b) What does this imply about how sample size affects the amount of statistical uncertainty?__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Type your answer here`\n",
    "\n",
    "5a)\n",
    "\n",
    "5b)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now it's your turn\n",
    "\n",
    "Now that you have learnt how to use the interface, you may design your own experiment! You may notice that the `Success Rate` setting is always fixed on __No intervention__, with the other options greyed out. This is because it is a dependent variable that you as the experimenter cannot intervene on. Play with it and familiarise yourself with it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment5 = BasketballExperiment()\n",
    "experiment5.assignment()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment5.setting(show=['Shot type', 'Hours practised per week', 'Success rate'], disable='Success rate')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment5.plot(show=['Shot type', 'Hours practised per week', 'Height (cm)', 'Success rate'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Causal network\n",
    "\n",
    "To visualise the causal relationship between height, hours practised, and shot success rate, we can draw a causal network graph. \n",
    "\n",
    "An arrow indicates the direction of a causal effect. In this example, we see \"hours practiced per week\" affects \"ability\" which in turns affects \"success rate.\" \n",
    "\n",
    "One way that systematic errors can slip into a measure of causal effects is if there are causal paths from other uncontrolled variables to our dependent variable. Here, for example, we see that we are interested in the path between \"hours practiced per week\" and \"success rate.\" We see however that other things can also cause differences in \"success rate\": shot type and height! So we need to control for these other variables to reduce systematic bias. One very common way to handle this in experimental settings is to randomize the assignments of these other variables between the control and treatment groups. We did this above by randomizing height assignments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "basketball.draw('Success rate')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}