hyperparameter_tuning/analyze_results/HPO_Analyze_TuningJob_Results.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyze Results of a Hyperparameter Tuning job\n",
    "\n",
    "Once you have completed a tuning job, (or even while the job is still running) you can use this notebook to analyze the results to understand how each hyperparameter effects the quality of the model.\n",
    "\n",
    "---\n",
    "## Set up the environment\n",
    "To start the analysis, you must pick the name of the hyperparameter tuning job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "import os\n",
    "\n",
    "region = boto3.Session().region_name\n",
    "sage_client = boto3.Session().client('sagemaker')\n",
    "\n",
    "## You must have already run a hyperparameter tuning job to analyze it here.\n",
    "## The Hyperparameter tuning jobs you have run are listed in the Training section on your SageMaker dashboard.\n",
    "## Copy the name of a completed job you want to analyze from that list.\n",
    "## For example: tuning_job_name = 'mxnet-training-201007-0054'.\n",
    "tuning_job_name = 'YOUR-HYPERPARAMETER-TUNING-JOB-NAME'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Track hyperparameter tuning job progress\n",
    "After you launch a tuning job, you can see its progress by calling describe_tuning_job API. The output from describe-tuning-job is a JSON object that contains information about the current state of the tuning job. You can call list_training_jobs_for_tuning_job to see a detailed list of the training jobs that the tuning job launched."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# run this cell to check current status of hyperparameter tuning job\n",
    "tuning_job_result = sage_client.describe_hyper_parameter_tuning_job(HyperParameterTuningJobName=tuning_job_name)\n",
    "\n",
    "status = tuning_job_result['HyperParameterTuningJobStatus']\n",
    "if status != 'Completed':\n",
    "    print('Reminder: the tuning job has not been completed.')\n",
    "    \n",
    "job_count = tuning_job_result['TrainingJobStatusCounters']['Completed']\n",
    "print(\"%d training jobs have completed\" % job_count)\n",
    "    \n",
    "is_minimize = (tuning_job_result['TrainingJobDefinitions'][0]['TuningObjective']['Type'] != 'Maximize')\n",
    "objective_name = tuning_job_result['TrainingJobDefinitions'][0]['TuningObjective']['MetricName']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "if tuning_job_result.get('BestTrainingJob',None):\n",
    "    print(\"Best model found so far:\")\n",
    "    pprint(tuning_job_result['BestTrainingJob'])\n",
    "else:\n",
    "    print(\"No training jobs have reported results yet.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fetch all results as DataFrame\n",
    "We can list hyperparameters and objective metrics of all training jobs and pick up the training job with the best objective metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "tuner = sagemaker.HyperparameterTuningJobAnalytics(tuning_job_name)\n",
    "\n",
    "full_df = tuner.dataframe()\n",
    "\n",
    "if len(full_df) > 0:\n",
    "    df = full_df[full_df['FinalObjectiveValue'] > -float('inf')]\n",
    "    if len(df) > 0:\n",
    "        df = df.sort_values('FinalObjectiveValue', ascending=is_minimize)\n",
    "        print(\"Number of training jobs with valid objective: %d\" % len(df))\n",
    "        print({\"lowest\":min(df['FinalObjectiveValue']),\"highest\": max(df['FinalObjectiveValue'])})\n",
    "        pd.set_option('display.max_colwidth', -1)  # Don't truncate TrainingJobName        \n",
    "    else:\n",
    "        print(\"No training jobs have reported valid results yet.\")\n",
    "        \n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## See TuningJob results vs time\n",
    "Next we will show how the objective metric changes over time, as the tuning job progresses.  For Bayesian strategy, you should expect to see a general trend towards better results, but this progress will not be steady as the algorithm needs to balance _exploration_ of new areas of parameter space against _exploitation_ of known good areas.  This can give you a sense of whether or not the number of training jobs is sufficient for the complexity of your search space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import bokeh\n",
    "import bokeh.io\n",
    "bokeh.io.output_notebook()\n",
    "from bokeh.plotting import figure, show\n",
    "from bokeh.models import HoverTool\n",
    "\n",
    "class HoverHelper():\n",
    "\n",
    "    def __init__(self, tuning_analytics):\n",
    "        self.tuner = tuning_analytics\n",
    "\n",
    "    def hovertool(self):\n",
    "        tooltips = [\n",
    "            (\"FinalObjectiveValue\", \"@FinalObjectiveValue\"),\n",
    "            (\"TrainingJobName\", \"@TrainingJobName\"),\n",
    "        ]\n",
    "        for k in self.tuner.tuning_ranges[tuning_job_name].keys():\n",
    "            tooltips.append( (k, \"@{%s}\" % k) )\n",
    "\n",
    "        ht = HoverTool(tooltips=tooltips)\n",
    "        return ht\n",
    "\n",
    "    def tools(self, standard_tools='pan,crosshair,wheel_zoom,zoom_in,zoom_out,undo,reset'):\n",
    "        return [self.hovertool(), standard_tools]\n",
    "\n",
    "hover = HoverHelper(tuner)\n",
    "\n",
    "p = figure(plot_width=900, plot_height=400, tools=hover.tools(), x_axis_type='datetime')\n",
    "p.circle(source=df, x='TrainingStartTime', y='FinalObjectiveValue')\n",
    "show(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyze the correlation between objective metric and individual hyperparameters \n",
    "Now you have finished a tuning job, you may want to know the correlation between your objective metric and individual hyperparameters you've selected to tune. Having that insight will help you decide whether it makes sense to adjust search ranges for certain hyperparameters and start another tuning job. For example, if you see a positive trend between objective metric and a numerical hyperparameter, you probably want to set a higher tuning range for that hyperparameter in your next tuning job.\n",
    "\n",
    "The following cell draws a graph for each hyperparameter to show its correlation with your objective metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "ranges = tuner.tuning_ranges[tuning_job_name]\n",
    "figures = []\n",
    "for hp_name, hp_range in ranges.items():\n",
    "    categorical_args = {}\n",
    "    if hp_range.get('Values'):\n",
    "        # This is marked as categorical.  Check if all options are actually numbers.\n",
    "        def is_num(x):\n",
    "            try:\n",
    "                float(x)\n",
    "                return 1\n",
    "            except:\n",
    "                return 0           \n",
    "        vals = hp_range['Values']\n",
    "        if sum([is_num(x) for x in vals]) == len(vals):\n",
    "            # Bokeh has issues plotting a \"categorical\" range that's actually numeric, so plot as numeric\n",
    "            print(\"Hyperparameter %s is tuned as categorical, but all values are numeric\" % hp_name)\n",
    "        else:\n",
    "            # Set up extra options for plotting categoricals.  A bit tricky when they're actually numbers.\n",
    "            categorical_args['x_range'] = vals\n",
    "\n",
    "    # Now plot it\n",
    "    p = figure(plot_width=500, plot_height=500, \n",
    "               title=\"Objective vs %s\" % hp_name,\n",
    "               tools=hover.tools(),\n",
    "               x_axis_label=hp_name, y_axis_label=objective_name,\n",
    "               **categorical_args)\n",
    "    p.circle(source=df, x=hp_name, y='FinalObjectiveValue')\n",
    "    figures.append(p)\n",
    "show(bokeh.layouts.Column(*figures))"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 2
}