notebooks/parameter_tuning_randomized_search.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hyperparameter tuning by randomized-search\n",
    "\n",
    "In the previous notebook, we showed how to use a grid-search approach to\n",
    "search for the best hyperparameters maximizing the statistical performance\n",
    "of a predictive model.\n",
    "\n",
    "However, a grid-search approach has limitations. It does not scale when\n",
    "the number of parameters to tune is increasing. Also, the grid will imposed\n",
    "a regularity during the search which might be problematic.\n",
    "\n",
    "In this notebook, we will present the another method to tune hyperparameters\n",
    "called randomized search."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Our predictive model\n",
    "\n",
    "Let us reload the dataset as we did previously:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import set_config\n",
    "\n",
    "set_config(display=\"diagram\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "adult_census = pd.read_csv(\"../datasets/adult-census.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We extract the column containing the target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_name = \"class\"\n",
    "target = adult_census[target_name]\n",
    "target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We drop from our data the target and the `\"education-num\"` column which\n",
    "duplicates the information with `\"education\"` columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = adult_census.drop(columns=[target_name, \"education-num\"])\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the dataset is loaded, we split it into a training and testing sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data_train, data_test, target_train, target_test = train_test_split(\n",
    "    data, target, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will create the same predictive pipeline as seen in the grid-search\n",
    "section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.preprocessing import OrdinalEncoder\n",
    "from sklearn.compose import make_column_selector as selector\n",
    "\n",
    "categorical_columns_selector = selector(dtype_include=object)\n",
    "categorical_columns = categorical_columns_selector(data)\n",
    "\n",
    "categorical_preprocessor = OrdinalEncoder(handle_unknown=\"use_encoded_value\",\n",
    "                                          unknown_value=-1)\n",
    "preprocessor = ColumnTransformer([\n",
    "    ('cat-preprocessor', categorical_preprocessor, categorical_columns)],\n",
    "    remainder='passthrough', sparse_threshold=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for the moment this line is required to import HistGradientBoostingClassifier\n",
    "from sklearn.experimental import enable_hist_gradient_boosting\n",
    "from sklearn.ensemble import HistGradientBoostingClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "model = Pipeline([\n",
    "    (\"preprocessor\", preprocessor),\n",
    "    (\"classifier\", HistGradientBoostingClassifier(random_state=42, max_leaf_nodes=4)),\n",
    "])\n",
    "\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tuning using a randomized-search\n",
    "\n",
    "With the `GridSearchCV` estimator, the parameters need to be specified\n",
    "explicitly. We already mentioned that exploring a large number of values for\n",
    "different parameters will be quickly untractable.\n",
    "\n",
    "Instead, we can randomly generate the parameter candidates. Indeed,\n",
    "such approach avoids the regularity of the grid. Hence, adding more\n",
    "evaluations can increase the resolution in each direction. This is the\n",
    "case in the frequent situation where the choice of some hyperparameters\n",
    "is not very important, as for hyperparameter 2 in the figure below.\n",
    "\n",
    "![Randomized vs grid search](../figures/grid_vs_random_search.svg)\n",
    "\n",
    "Indeed, the number of evaluation points need to be divided across the\n",
    "two different hyperparameters. With a grid, the danger is that the\n",
    "region of good hyperparameters fall between the line of the grid: this\n",
    "region is aligned with the grid given that hyperparameter 2 has a weak\n",
    "influence. Rather, stochastic search will sample hyperparameter 1\n",
    "independently from hyperparameter 2 and find the optimal region.\n",
    "\n",
    "The `RandomizedSearchCV` class allows for such stochastic search. It is\n",
    "used similarly to the `GridSearchCV` but the sampling distributions\n",
    "need to be specified instead of the parameter values. For instance, we\n",
    "will draw candidates using a log-uniform distribution because the parameters\n",
    "we are interested in take positive values with a natural log scaling (.1 is\n",
    "as close to 1 as 10 is).\n",
    "\n",
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">Random search (with <tt class=\"docutils literal\">RandomizedSearchCV</tt>) is typically beneficial compared\n",
    "to grid search (with <tt class=\"docutils literal\">GridSearchCV</tt>) to optimize 3 or more\n",
    "hyperparameters.</p>\n",
    "</div>\n",
    "\n",
    "We will optimize 3 other parameters in addition to the ones we\n",
    "optimized in the notebook presenting the `GridSearchCV`:\n",
    "\n",
    "* `l2_regularization`: it corresponds to the constant to regularized the loss\n",
    "  function\n",
    "* `min_samples_leaf`: it corresponds to the minimum number of samples\n",
    "  required in a leaf;\n",
    "* `max_bins`: it corresponds to the maximum number of bins to construct the\n",
    "  histograms.\n",
    "\n",
    "We recall the meaning of the 2 remaining parameters:\n",
    "\n",
    "* `learning_rate`: it corresponds to the speed at which the gradient-boosting\n",
    "  will correct the residuals at each boosting iteration;\n",
    "* `max_leaf_nodes`: it corresponds to the maximum number of leaves for each\n",
    "  tree in the ensemble.\n",
    "\n",
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\"><tt class=\"docutils literal\">scipy.stats.loguniform</tt> can be used to generate floating numbers. To\n",
    "generate random values for integer-valued parameters (e.g.\n",
    "<tt class=\"docutils literal\">min_samples_leaf</tt>) we can adapt is as follows:</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import loguniform\n",
    "\n",
    "\n",
    "class loguniform_int:\n",
    "    \"\"\"Integer valued version of the log-uniform distribution\"\"\"\n",
    "    def __init__(self, a, b):\n",
    "        self._distribution = loguniform(a, b)\n",
    "\n",
    "    def rvs(self, *args, **kwargs):\n",
    "        \"\"\"Random variable sample\"\"\"\n",
    "        return self._distribution.rvs(*args, **kwargs).astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Now, we can define the randomized search using the different distributions.\n",
    "Executing 10 iterations of 5-fold cross-validation for random\n",
    "parametrizations of this model on this dataset can take from 10 seconds to\n",
    "several minutes, depending on the speed of the host computer and the number\n",
    "of available processors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "\n",
    "param_distributions = {\n",
    "    'classifier__l2_regularization': loguniform(1e-6, 1e3),\n",
    "    'classifier__learning_rate': loguniform(0.001, 10),\n",
    "    'classifier__max_leaf_nodes': loguniform_int(2, 256),\n",
    "    'classifier__min_samples_leaf': loguniform_int(1, 100),\n",
    "    'classifier__max_bins': loguniform_int(2, 255),\n",
    "}\n",
    "\n",
    "model_random_search = RandomizedSearchCV(\n",
    "    model, param_distributions=param_distributions, n_iter=10,\n",
    "    cv=5, verbose=1,\n",
    ")\n",
    "model_random_search.fit(data_train, target_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can compute the accuracy score on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy = model_random_search.score(data_test, target_test)\n",
    "\n",
    "print(f\"The test accuracy score of the best model is \"\n",
    "      f\"{accuracy:.2f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "print(\"The best parameters are:\")\n",
    "pprint(model_random_search.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "source": [
    "\n",
    "We can inspect the results using the attributes `cv_results` as we did\n",
    "previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def shorten_param(param_name):\n",
    "    if \"__\" in param_name:\n",
    "        return param_name.rsplit(\"__\", 1)[1]\n",
    "    return param_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the parameter names\n",
    "column_results = [\n",
    "    f\"param_{name}\" for name in param_distributions.keys()]\n",
    "column_results += [\n",
    "    \"mean_test_score\", \"std_test_score\", \"rank_test_score\"]\n",
    "\n",
    "cv_results = pd.DataFrame(model_random_search.cv_results_)\n",
    "cv_results = cv_results[column_results].sort_values(\n",
    "    \"mean_test_score\", ascending=False)\n",
    "cv_results = cv_results.rename(shorten_param, axis=1)\n",
    "cv_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice, a randomized hyperparameter search is usually run with a large\n",
    "number of iterations. In order to avoid the computation cost and still make a\n",
    "decent analysis, we load the results obtained from a similar search with 200\n",
    "iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model_random_search = RandomizedSearchCV(\n",
    "#     model, param_distributions=param_distributions, n_iter=200,\n",
    "#     n_jobs=2, cv=5)\n",
    "# model_random_search.fit(data_train, target_train)\n",
    "# cv_results =  pd.DataFrame(model_random_search.cv_results_)\n",
    "# cv_results.to_csv(\"../figures/randomized_search_results.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_results = pd.read_csv(\"../figures/randomized_search_results.csv\",\n",
    "                         index_col=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we have more than 2 parameters in our grid-search, we cannot visualize the\n",
    "results using a heatmap. However, we can us a parallel coordinates plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(cv_results[column_results].rename(\n",
    "    shorten_param, axis=1).sort_values(\"mean_test_score\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import plotly.express as px\n",
    "\n",
    "fig = px.parallel_coordinates(\n",
    "    cv_results.rename(shorten_param, axis=1).apply({\n",
    "        \"learning_rate\": np.log10,\n",
    "        \"max_leaf_nodes\": np.log2,\n",
    "        \"max_bins\": np.log2,\n",
    "        \"min_samples_leaf\": np.log10,\n",
    "        \"l2_regularization\": np.log10,\n",
    "        \"mean_test_score\": lambda x: x}),\n",
    "    color=\"mean_test_score\",\n",
    "    color_continuous_scale=px.colors.sequential.Viridis,\n",
    ")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The parallel coordinates plot will display the values of the hyperparameters\n",
    "on different columns while the performance metric is color coded. Thus, we\n",
    "are able to quickly inspect if there is a range of hyperparameters which is\n",
    "working or not.\n",
    "\n",
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">We <strong>transformed most axis values by taking a log10 or log2</strong> to\n",
    "spread the active ranges and improve the readability of the plot.</p>\n",
    "</div>\n",
    "\n",
    "In particular for this hyper-parameter search, it is interesting to see that\n",
    "the yellow lines (top performing models) all reach intermediate values for\n",
    "the learning rate, that is, tick values between -2 and 0 which correspond to\n",
    "learning rate values of 0.01 to 1.0 once we invert the log10 transform for\n",
    "that axis.\n",
    "\n",
    "It is possible to **select a range of results by clicking and holding on any\n",
    "axis** of the parallel coordinate plot. You can then slide (move) the range\n",
    "selection and cross two selections to see the intersections. You can undo a\n",
    "selection by clicking once again on the same axis.\n",
    "\n",
    "We also observe that it is not possible to select the highest performing\n",
    "models by selecting lines of on the `max_bins` axis with tick values between\n",
    "1 and 3.\n",
    "\n",
    "The other hyper-parameters are not very sensitive. We can check that if we\n",
    "select the `learning_rate` axis tick values between -1.5 and -0.5 and\n",
    "`max_bins` tick values between 5 and 8, we always select top performing\n",
    "models, whatever the values of the other hyper-parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "In this notebook, we have seen how randomized search offer a valuable\n",
    "alternative to grid-search when the number of hyperparameters to tune is more\n",
    "than two. It also alleviates the regularity imposed by the grid that might be\n",
    "problematic sometimes."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "main_language": "python",
   "notebook_metadata_filter": "-all"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}