estimate.src.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "urth": {
     "dashboard": {
      "layout": {
       "col": 0,
       "height": 5,
       "row": 0,
       "width": 12
      }
     }
    }
   },
   "source": [
    "# Estimate of Public Jupyter Notebooks on GitHub\n",
    "\n",
    "This notebook shows the historical count and future estimate of the number of `*.ipynb` files on GitHub. The daily count comes from executing the query [extension:ipynb nbformat_minor](https://github.com/search?utf8=%E2%9C%93&q=extension%3Aipynb+nbformat_minor)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "urth": {
     "dashboard": {
      "layout": {
       "col": 0,
       "height": 9,
       "row": 5,
       "width": 12
      }
     }
    }
   },
   "source": [
    "## Data Collection\n",
    "\n",
    "* Late-2014 to mid-2016: I wrote a script that scrapes the GitHub web search UI for the count, appends to a CSV, executes a notebook, and stores the results in a gist at https://gist.github.com/parente/facb555dfbae28e817e0. I scheduled the script to run daily.\n",
    "* Mid-2106 to late-2016: The GitHub web search UI started requiring authentication to see global search results. I stopped collecting data.\n",
    "* Late-2016 to early-2019: I rewrote the process to include a human-in-the-loop who entered the hit count after viewing the search results page. I moved the CSV, notebook, and scripts to this repo, and sporadically ran the script.\n",
    "* Early-2019: I found out that the GitHub search API now supports global search by file type. I automated the entire collection process again and set it to run on TravisCI on a daily schedule."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "urth": {
     "dashboard": {
      "layout": {
       "col": 0,
       "height": 9,
       "row": 5,
       "width": 12
      }
     }
    }
   },
   "source": [
    "## Assumptions\n",
    "\n",
    "1. That the search query hits and notebooks on GitHub are in 1:1 correspondence.\n",
    "1. That GitHub is accurately reporting the total number of `*.ipynb` file hits.\n",
    "1. That the result is **not** inflated due to GitHub forks.\n",
    "    * Evidence: We do not see the tutorial notebooks from the ipython/ipython GitHub repository duplicated in the search results because of the 2,000+ forks of the ipython/ipython repo.\n",
    "1. That the result **is** inflated a tiny bit by manually created duplicates of notebooks.\n",
    "    * Evidence: Some people seem to download their favorite notebooks and then upload them into their own git repositories for safe keeping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "urth": {
     "dashboard": {}
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.simplefilter('ignore', FutureWarning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "urth": {
     "dashboard": {}
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "urth": {
     "dashboard": {}
    }
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "import fbprophet\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "urth": {
     "dashboard": {}
    }
   },
   "outputs": [],
   "source": [
    "mpl.style.use('ggplot')\n",
    "figsize = (14,7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "now = datetime.datetime.utcnow()\n",
    "print(f'This notebook was last rendered at {now} UTC')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "urth": {
     "dashboard": {
      "hidden": true
     }
    }
   },
   "source": [
    "## Raw Hits\n",
    "\n",
    "First, let's load the historical data into a DataFrame indexed by date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "urth": {
     "dashboard": {}
    }
   },
   "outputs": [],
   "source": [
    "hits_df = pd.read_csv('ipynb_counts.csv', index_col=0, header=0, parse_dates=True)\n",
    "hits_df.reset_index(inplace=True)\n",
    "hits_df.drop_duplicates(subset='date', inplace=True)\n",
    "hits_df.set_index('date', inplace=True)\n",
    "hits_df.sort_index(ascending=True, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "urth": {
     "dashboard": {
      "hidden": true
     }
    }
   },
   "outputs": [],
   "source": [
    "hits_df.tail(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "urth": {
     "dashboard": {
      "hidden": true
     }
    }
   },
   "source": [
    "There might be missing counts for days that we failed to sample. We build up the expected date range and insert NaNs for dates we missed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "urth": {
     "dashboard": {}
    }
   },
   "outputs": [],
   "source": [
    "til_today = pd.date_range(hits_df.index[0], hits_df.index[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "urth": {
     "dashboard": {}
    }
   },
   "outputs": [],
   "source": [
    "hits_df = hits_df.reindex(til_today)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "urth": {
     "dashboard": {
      "hidden": true
     }
    }
   },
   "source": [
    "Now we plot the known notebook counts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=figsize)\n",
    "ax.set_title(f'GitHub search hits for {len(hits_df)} days')\n",
    "ax.plot(hits_df.hits, 'ko', markersize=1, label='hits')\n",
    "ax.legend(loc='upper left')\n",
    "ax.set_xlabel('Date')\n",
    "ax.set_ylabel('# of ipynb files');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change\n",
    "\n",
    "Next, let's look at various measurements of change.\n",
    "\n",
    "The total change in the number of `*.ipynb` hits between the first day we have data and today is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "urth": {
     "dashboard": {
      "hidden": true
     }
    }
   },
   "outputs": [],
   "source": [
    "total_delta_nbs = hits_df.iloc[-1] - hits_df.iloc[0]\n",
    "total_delta_nbs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "urth": {
     "dashboard": {
      "hidden": true
     }
    }
   },
   "source": [
    "The mean daily change for the entire duration is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "urth": {
     "dashboard": {
      "hidden": true
     }
    }
   },
   "outputs": [],
   "source": [
    "avg_delta_nbs = total_delta_nbs / len(hits_df)\n",
    "avg_delta_nbs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "urth": {
     "dashboard": {
      "hidden": true
     }
    }
   },
   "source": [
    "The change in hit count between any two consecutive days for which we have data looks like the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "urth": {
     "dashboard": {}
    }
   },
   "outputs": [],
   "source": [
    "daily_deltas = (hits_df.hits - hits_df.hits.shift())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=figsize)\n",
    "ax.plot(daily_deltas, 'ko', markersize=2)\n",
    "ax.set_xlabel('Date')\n",
    "ax.set_ylabel('$\\Delta$ # of ipynb files')\n",
    "ax.set_title('Day-to-Day Change');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The large jumps in the data are from GitHub reporting drastically different counts from one day to the next. Maybe GitHub was rebuilding a search index when we queried or had a search broker out-of-sync with the others?\n",
    "\n",
    "Let's drop outliers defined as values more than two standard deviations away from a centered 180 day rolling mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "daily_delta_rolling = daily_deltas.rolling(window=180, min_periods=0, center=True)\n",
    "outliers = abs(daily_deltas - daily_delta_rolling.mean()) > 2*daily_delta_rolling.std()\n",
    "outliers.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleaned_hits_df = hits_df.copy()\n",
    "cleaned_hits_df[outliers] = np.NaN\n",
    "cleaned_daily_deltas = (cleaned_hits_df.hits - cleaned_hits_df.hits.shift())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=figsize)\n",
    "ax.plot(cleaned_daily_deltas, 'ko', markersize=2)\n",
    "ax.set_xlabel('Date')\n",
    "ax.set_ylabel('$\\Delta$ # of ipynb files')\n",
    "ax.set_title('Day-to-Day Change Sans Outliers');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's do a simple linear interpolation for missing values and then look at the rolling mean of change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filled_df = cleaned_hits_df.interpolate(method='time')\n",
    "smoothed_daily_deltas = (filled_df.hits - filled_df.hits.shift()).rolling(window=30, min_periods=0, center=False).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=figsize)\n",
    "ax.plot(smoothed_daily_deltas, 'r-')\n",
    "ax.set_xlabel('Date')\n",
    "ax.set_ylabel('$\\Delta$ # of ipynb files')\n",
    "ax.set_title('30-Day Rolling Mean of Day-to-Day Change');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "render": false,
    "urth": {
     "dashboard": {
      "hidden": true
     }
    }
   },
   "source": [
    "## Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "render": false,
    "urth": {
     "dashboard": {
      "hidden": true
     }
    }
   },
   "source": [
    "Now let's use [fbprophet](https://facebook.github.io/prophet/) to forecast growth for the upcoming year. First, we'll try to forecast based on the raw search hit data with outliers removed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "periods = 365"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forecast(df):\n",
    "    # 95% confidence intervals\n",
    "    m = fbprophet.Prophet(interval_width=0.95, seasonality_mode='multiplicative')\n",
    "    df = df.reset_index().rename(columns={'index': 'ds', 'hits': 'y'})\n",
    "    m.fit(df)\n",
    "    future = m.make_future_dataframe(periods=periods)\n",
    "    return m, m.predict(future)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model, forecast_df = forecast(cleaned_hits_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_forecast(m, fc):\n",
    "    fig, ax = plt.subplots(figsize=figsize)\n",
    "    m.plot(fc, ax=ax)\n",
    "    ax.set_xlabel('Date')\n",
    "    ax.set_ylabel('# ipynb files')\n",
    "    ax.minorticks_on()\n",
    "    ax.legend(loc='upper left')\n",
    "    ax.set_title(f'GitHub search hits predicted until {fc.iloc[-1].ds.date()} (95% confidence interval)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model appears to favor seasonality effects in the early data and replicate them throughout the forecast period. The density of early data versus the sparsity of later data is a likely cause."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_forecast(model, forecast_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's fit and predict again, but this time on the interpolated data so that there's better balance between early and later observations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_filled, forecast_filled_df = forecast(filled_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This forecast looks much more sensible with respect to the true measurements and overall trend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note: Using model instead of model_filled to see observations without imputed missing values\n",
    "plot_forecast(model, forecast_filled_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can plot the components of the model. The weekly component appears to track the work week while the yearly component seems to track with a traditional academic year in the northern hemisphere."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_df = fbprophet.plot.plot_components(model_filled, forecast_filled_df, figsize=figsize,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use Prophet's cross validation function to measure the root mean square error for forecasts overlapping with past data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_df = fbprophet.diagnostics.cross_validation(model_filled, horizon='365 days', initial='730 days', period='90 days')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=figsize)\n",
    "fbprophet.plot.plot_cross_validation_metric(cv_df, metric='rmse', ax=ax)\n",
    "ax.set_title('Root Mean Square Error')\n",
    "ax.minorticks_on();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Milestones\n",
    "\n",
    "Finally, it's nice to celebrate million-notebook milestones. We can use our model to predict when they're going to occur."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "combined_df = pd.concat([filled_df.reset_index(drop=True).rename(columns={'hits': 'y'}), forecast_filled_df], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rows = []\n",
    "cols = {'y': 'actual', 'yhat_upper': 'optimistic', 'yhat': 'predicted', 'yhat_lower': 'conservative'}\n",
    "for i in range(1, 11):\n",
    "    milestone = i * 1e6\n",
    "    row = {'milestone': milestone}\n",
    "\n",
    "    for col in cols:\n",
    "        gt_df = combined_df[combined_df[col] > milestone]\n",
    "        if len(gt_df):\n",
    "            row[col] = gt_df.iloc[0].ds\n",
    "    rows.append(row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(rows, columns=['milestone']+list(cols.keys())).rename(columns=cols)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "urth": {
   "dashboard": {
    "cellMargin": 10,
    "defaultCellHeight": 20,
    "layoutStrategy": "packed",
    "maxColumns": 12
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}