09A - Reviewing Automated Machine Learning Explanations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reviewing Automated Machine Learning Explanations\n",
    "\n",
    "As machine learning becomes more and more and more prevelant, the predictions made by models have greater influence over many aspects of our society. For example, machine learning models are an increasingly significant factor in how banks decide to grant loans or doctors prioritise treatments. The ability to interpret and explain models is increasingly important, so that the rationale for the predictions made by machine learning models can be explained and justified, and any inadvertant bias in the model can be identified.\n",
    "\n",
    "When you use automated machine learning to train a model, you have the option to generate explanations of feature importance that quantify the extent to which each feature (including engineered features that were generated by preprocessing transformations) influences label prediction. In this lab, you'll explore the explanations generated by an automated machine learning experiment.\n",
    "\n",
    "\n",
    "## Connect to Your Workspace\n",
    "\n",
    "The first thing you need to do is to connect to your workspace using the Azure ML SDK.\n",
    "\n",
    "> **Note**: If the authenticated session with your Azure subscription has expired since you completed the previous exercise, you'll be prompted to reauthenticate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import azureml.core\n",
    "from azureml.core import Workspace\n",
    "\n",
    "# Load the workspace from the saved config file\n",
    "ws = Workspace.from_config()\n",
    "print('Ready to use Azure ML {} to work with {}'.format(azureml.core.VERSION, ws.name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run an Automated Machine Learning Experiment\n",
    "\n",
    "To reduce time in this lab, you'll run an automated machine learning experiment on local compute with only three iterations.\n",
    "\n",
    "Note that the **model_explainability** configuration option is set to **True**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from azureml.train.automl import AutoMLConfig\n",
    "from azureml.core.experiment import Experiment\n",
    "from azureml.widgets import RunDetails\n",
    "\n",
    "# Load the data\n",
    "train_data = pd.read_csv('data/diabetes.csv')\n",
    "\n",
    "# Configure Auto ML\n",
    "automl_config = AutoMLConfig(name='Automated ML Experiment',\n",
    "                             task='classification',\n",
    "                             compute_target='local',\n",
    "                             training_data = train_data,\n",
    "                             n_cross_validations = 2,\n",
    "                             label_column_name='Diabetic',\n",
    "                             iterations=3,\n",
    "                             primary_metric = 'AUC_weighted',\n",
    "                             max_concurrent_iterations=3,\n",
    "                             model_explainability=True # Generate feature importance!\n",
    "                             )\n",
    "\n",
    "# Run the Auto ML experiment\n",
    "automl_experiment = Experiment(ws, 'diabetes_automl')\n",
    "automl_run = automl_experiment.submit(automl_config)\n",
    "RunDetails(automl_run).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View Feature Importance\n",
    "\n",
    "When the experiment has completed in the widget above, click the run that produced the best result to see its details. Thens scroll to the bottom of the visualizations to see the relative feature importance.\n",
    "\n",
    "You can also view feature importance for the best model produced by the experiment in [Azure Machine Learning studio](https://ml.azure.com) by viewing the **Explanations** tab of the best model details, or by using the **ExplanationClient** class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.contrib.interpret.explanation.explanation_client import ExplanationClient\n",
    "\n",
    "# Get the best model (2nd item in outputs)\n",
    "best_run, fitted_model = automl_run.get_output()\n",
    "\n",
    "# Get the feature explanations\n",
    "client = ExplanationClient.from_run(best_run)\n",
    "engineered_explanations = client.download_model_explanation()\n",
    "feature_importances = engineered_explanations.get_feature_importance_dict()\n",
    "\n",
    "# Overall feature importance\n",
    "print('Feature\\tImportance')\n",
    "for key, value in feature_importances.items():\n",
    "    print(key, '\\t', value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View Importance for Auto-Engineered Features\n",
    "\n",
    "Automated machine learning includes the option to try preprocessing the data, which often performs *feature engieering* to create new, derived features from which the model is trained. Let's enable that option and re-run the automated machine learning experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from azureml.train.automl import AutoMLConfig\n",
    "from azureml.core.experiment import Experiment\n",
    "from azureml.widgets import RunDetails\n",
    "\n",
    "# Load the data\n",
    "train_data = pd.read_csv('data/diabetes.csv')\n",
    "\n",
    "# Configure Auto ML\n",
    "automl_config = AutoMLConfig(name='Automated ML Experiment',\n",
    "                             task='classification',\n",
    "                             compute_target='local',\n",
    "                             training_data = train_data,\n",
    "                             n_cross_validations = 2,\n",
    "                             label_column_name='Diabetic',\n",
    "                             iterations=3,\n",
    "                             primary_metric = 'AUC_weighted',\n",
    "                             max_concurrent_iterations=3,\n",
    "                             featurization='auto', # Enable feature engineering\n",
    "                             model_explainability=True # Generate feature importance\n",
    "                             )\n",
    "\n",
    "# Run the Auto ML experiment\n",
    "automl_experiment = Experiment(ws, 'diabetes_automl')\n",
    "automl_run = automl_experiment.submit(automl_config)\n",
    "RunDetails(automl_run).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Featurization is accomplished through the use of [Scikit-Learn transformation pipelines](https://scikit-learn.org/stable/modules/compose.html#combining-estimators) (not to be confused with Azure Machine Learning pipelines!). These produce models that include steps to transform the data before inferencing.\n",
    "\n",
    "Run the following code to view the steps in the model pipeline, and the importance of the engineered features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.contrib.interpret.explanation.explanation_client import ExplanationClient\n",
    "\n",
    "# Get the best model (2nd item in outputs)\n",
    "best_run, fitted_model = automl_run.get_output()\n",
    "\n",
    "print('Model pipeline steps:')\n",
    "for step in fitted_model.named_steps:\n",
    "    print('-',step)\n",
    "    \n",
    "# Get the feature explanations\n",
    "client = ExplanationClient.from_run(best_run)\n",
    "engineered_explanations = client.download_model_explanation(raw=False)\n",
    "feature_importances = engineered_explanations.get_feature_importance_dict()\n",
    "\n",
    "# Overall feature importance\n",
    "print('\\nFeature\\tImportance')\n",
    "for key, value in feature_importances.items():\n",
    "    print(key, '\\t', value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **More Information**: For more information Automated machine Learning, see the [Azure ML documentation](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-auto-train)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}