aml-pipelines-with-automated-machine-learning-step.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/NotebookVM/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-with-automated-machine-learning-step.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Azure Machine Learning Pipeline with AutoMLStep (Udacity Course 2)\n",
    "This notebook demonstrates the use of AutoMLStep in Azure Machine Learning Pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "In this example we showcase how you can use AzureML Dataset to load data for AutoML via AML Pipeline. \n",
    "\n",
    "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you have executed the [configuration](https://aka.ms/pl-config) before running this notebook.\n",
    "\n",
    "In this notebook you will learn how to:\n",
    "1. Create an `Experiment` in an existing `Workspace`.\n",
    "2. Create or Attach existing AmlCompute to a workspace.\n",
    "3. Define data loading in a `TabularDataset`.\n",
    "4. Configure AutoML using `AutoMLConfig`.\n",
    "5. Use AutoMLStep\n",
    "6. Train the model using AmlCompute\n",
    "7. Explore the results.\n",
    "8. Test the best fitted model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Azure Machine Learning and Pipeline SDK-specific imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SDK version: 1.18.0\n"
     ]
    }
   ],
   "source": [
    "import logging\n",
    "import os\n",
    "import csv\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn import datasets\n",
    "import pkg_resources\n",
    "\n",
    "import azureml.core\n",
    "from azureml.core.experiment import Experiment\n",
    "from azureml.core.workspace import Workspace\n",
    "from azureml.train.automl import AutoMLConfig\n",
    "from azureml.core.dataset import Dataset\n",
    "\n",
    "from azureml.pipeline.steps import AutoMLStep\n",
    "\n",
    "# Check core SDK version number\n",
    "print(\"SDK version:\", azureml.core.VERSION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize Workspace\n",
    "Initialize a workspace object from persisted configuration. Make sure the config file is present at .\\config.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "quick-starts-ws-128228\n",
      "aml-quickstarts-128228\n",
      "southcentralus\n",
      "dafd62fe-51d3-4671-af50-6bac2f001b5d\n"
     ]
    }
   ],
   "source": [
    "ws = Workspace.from_config()\n",
    "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create an Azure ML experiment\n",
    "Let's create an experiment named \"automlstep-classification\" and a folder to hold the training scripts. The script runs will be recorded under the experiment in Azure.\n",
    "\n",
    "The best practice is to use separate folders for scripts and its dependent files for each step and specify that folder as the `source_directory` for the step. This helps reduce the size of the snapshot created for the step (only the specific folder is snapshotted). Since changes in any files in the `source_directory` would trigger a re-upload of the snapshot, this helps keep the reuse of the step when there are no changes in the `source_directory` of the step.\n",
    "\n",
    "*Udacity Note:* There is no need to create an Azure ML experiment, this needs to re-use the experiment that was already created\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"width:100%\"><tr><th>Name</th><th>Workspace</th><th>Report Page</th><th>Docs Page</th></tr><tr><td>bankMarketing-exp</td><td>quick-starts-ws-128228</td><td><a href=\"https://ml.azure.com/experiments/bankMarketing-exp?wsid=/subscriptions/dafd62fe-51d3-4671-af50-6bac2f001b5d/resourcegroups/aml-quickstarts-128228/workspaces/quick-starts-ws-128228\" target=\"_blank\" rel=\"noopener\">Link to Azure Machine Learning studio</a></td><td><a href=\"https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.experiment.Experiment?view=azure-ml-py\" target=\"_blank\" rel=\"noopener\">Link to Documentation</a></td></tr></table>"
      ],
      "text/plain": [
       "Experiment(Name: bankMarketing-exp,\n",
       "Workspace: quick-starts-ws-128228)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Choose a name for the run history container in the workspace.\n",
    "# NOTE: update these to match your existing experiment name\n",
    "experiment_name = 'bankMarketing-exp'\n",
    "project_folder = './pipeline-project'\n",
    "\n",
    "experiment = Experiment(ws, experiment_name)\n",
    "experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create or Attach an AmlCompute cluster\n",
    "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for your AutoML run. In this tutorial, you get the default `AmlCompute` as your training compute resource.\n",
    "\n",
    "**Udacity Note** There is no need to create a new compute target, it can re-use the previous cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found existing cluster, use it.\n",
      "Succeeded\n",
      "AmlCompute wait for completion finished\n",
      "\n",
      "Minimum number of nodes requested have been provisioned\n"
     ]
    }
   ],
   "source": [
    "from azureml.core.compute import AmlCompute\n",
    "from azureml.core.compute import ComputeTarget\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "\n",
    "# NOTE: update the cluster name to match the existing cluster\n",
    "# Choose a name for your CPU cluster\n",
    "amlcompute_cluster_name = \"instance01\"\n",
    "\n",
    "# Verify that cluster does not exist already\n",
    "try:\n",
    "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
    "    print('Found existing cluster, use it.')\n",
    "except ComputeTargetException:\n",
    "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',# for GPU, use \"STANDARD_NC6\"\n",
    "                                                           #vm_priority = 'lowpriority', # optional\n",
    "                                                           max_nodes=4)\n",
    "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
    "\n",
    "compute_target.wait_for_completion(show_output=True, min_node_count = 1, timeout_in_minutes = 10)\n",
    "# For a more detailed view of current AmlCompute status, use get_status()."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "\n",
    "**Udacity note:** Make sure the `key` is the same name as the dataset that is uploaded, and that the description matches. If it is hard to find or unknown, loop over the `ws.datasets.keys()` and `print()` them.\n",
    "If it *isn't* found because it was deleted, it can be recreated with the link that has the CSV "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found existing dataset\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>duration</th>\n",
       "      <th>campaign</th>\n",
       "      <th>pdays</th>\n",
       "      <th>previous</th>\n",
       "      <th>emp.var.rate</th>\n",
       "      <th>cons.price.idx</th>\n",
       "      <th>cons.conf.idx</th>\n",
       "      <th>euribor3m</th>\n",
       "      <th>nr.employed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>32950.000000</td>\n",
       "      <td>32950.000000</td>\n",
       "      <td>32950.000000</td>\n",
       "      <td>32950.000000</td>\n",
       "      <td>32950.000000</td>\n",
       "      <td>32950.000000</td>\n",
       "      <td>32950.000000</td>\n",
       "      <td>32950.000000</td>\n",
       "      <td>32950.000000</td>\n",
       "      <td>32950.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>40.040212</td>\n",
       "      <td>257.335205</td>\n",
       "      <td>2.561730</td>\n",
       "      <td>962.174780</td>\n",
       "      <td>0.174780</td>\n",
       "      <td>0.076228</td>\n",
       "      <td>93.574243</td>\n",
       "      <td>-40.518680</td>\n",
       "      <td>3.615654</td>\n",
       "      <td>5166.859608</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>10.432313</td>\n",
       "      <td>257.331700</td>\n",
       "      <td>2.763646</td>\n",
       "      <td>187.646785</td>\n",
       "      <td>0.496503</td>\n",
       "      <td>1.572242</td>\n",
       "      <td>0.578636</td>\n",
       "      <td>4.623004</td>\n",
       "      <td>1.735748</td>\n",
       "      <td>72.208448</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>17.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-3.400000</td>\n",
       "      <td>92.201000</td>\n",
       "      <td>-50.800000</td>\n",
       "      <td>0.634000</td>\n",
       "      <td>4963.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>32.000000</td>\n",
       "      <td>102.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>999.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-1.800000</td>\n",
       "      <td>93.075000</td>\n",
       "      <td>-42.700000</td>\n",
       "      <td>1.344000</td>\n",
       "      <td>5099.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>38.000000</td>\n",
       "      <td>179.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>999.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.100000</td>\n",
       "      <td>93.749000</td>\n",
       "      <td>-41.800000</td>\n",
       "      <td>4.857000</td>\n",
       "      <td>5191.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>47.000000</td>\n",
       "      <td>318.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>999.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.400000</td>\n",
       "      <td>93.994000</td>\n",
       "      <td>-36.400000</td>\n",
       "      <td>4.961000</td>\n",
       "      <td>5228.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>98.000000</td>\n",
       "      <td>4918.000000</td>\n",
       "      <td>56.000000</td>\n",
       "      <td>999.000000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>1.400000</td>\n",
       "      <td>94.767000</td>\n",
       "      <td>-26.900000</td>\n",
       "      <td>5.045000</td>\n",
       "      <td>5228.100000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                age      duration      campaign         pdays      previous  \\\n",
       "count  32950.000000  32950.000000  32950.000000  32950.000000  32950.000000   \n",
       "mean      40.040212    257.335205      2.561730    962.174780      0.174780   \n",
       "std       10.432313    257.331700      2.763646    187.646785      0.496503   \n",
       "min       17.000000      0.000000      1.000000      0.000000      0.000000   \n",
       "25%       32.000000    102.000000      1.000000    999.000000      0.000000   \n",
       "50%       38.000000    179.000000      2.000000    999.000000      0.000000   \n",
       "75%       47.000000    318.000000      3.000000    999.000000      0.000000   \n",
       "max       98.000000   4918.000000     56.000000    999.000000      7.000000   \n",
       "\n",
       "       emp.var.rate  cons.price.idx  cons.conf.idx     euribor3m   nr.employed  \n",
       "count  32950.000000    32950.000000   32950.000000  32950.000000  32950.000000  \n",
       "mean       0.076228       93.574243     -40.518680      3.615654   5166.859608  \n",
       "std        1.572242        0.578636       4.623004      1.735748     72.208448  \n",
       "min       -3.400000       92.201000     -50.800000      0.634000   4963.600000  \n",
       "25%       -1.800000       93.075000     -42.700000      1.344000   5099.100000  \n",
       "50%        1.100000       93.749000     -41.800000      4.857000   5191.000000  \n",
       "75%        1.400000       93.994000     -36.400000      4.961000   5228.100000  \n",
       "max        1.400000       94.767000     -26.900000      5.045000   5228.100000  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Try to load the dataset from the Workspace. Otherwise, create it from the file\n",
    "# NOTE: update the key to match the dataset name\n",
    "found = False\n",
    "key = \"Bank-marketing\"\n",
    "description_text = \"Bank Marketing DataSet for Udacity Course 2\"\n",
    "\n",
    "if key in ws.datasets.keys():\n",
    "        print(\"Found existing dataset\")\n",
    "        found = True\n",
    "        dataset = ws.datasets[key] \n",
    "\n",
    "if not found:\n",
    "        print(\"Downloading from URL\")\n",
    "        # Create AML Dataset and register it into Workspace\n",
    "        example_data = 'https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/bankmarketing_train.csv'\n",
    "        dataset = Dataset.Tabular.from_delimited_files(example_data)        \n",
    "        #Register Dataset in Workspace\n",
    "        dataset = dataset.register(workspace=ws,\n",
    "                                   name=key,\n",
    "                                   description=description_text)\n",
    "\n",
    "\n",
    "df = dataset.to_pandas_dataframe()\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Review the Dataset Result\n",
    "\n",
    "You can peek the result of a TabularDataset at any range using `skip(i)` and `take(j).to_pandas_dataframe()`. Doing so evaluates only `j` records for all the steps in the TabularDataset, which makes it fast even against large datasets.\n",
    "\n",
    "`TabularDataset` objects are composed of a list of transformation steps (optional)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>job</th>\n",
       "      <th>marital</th>\n",
       "      <th>education</th>\n",
       "      <th>default</th>\n",
       "      <th>housing</th>\n",
       "      <th>loan</th>\n",
       "      <th>contact</th>\n",
       "      <th>month</th>\n",
       "      <th>day_of_week</th>\n",
       "      <th>...</th>\n",
       "      <th>campaign</th>\n",
       "      <th>pdays</th>\n",
       "      <th>previous</th>\n",
       "      <th>poutcome</th>\n",
       "      <th>emp.var.rate</th>\n",
       "      <th>cons.price.idx</th>\n",
       "      <th>cons.conf.idx</th>\n",
       "      <th>euribor3m</th>\n",
       "      <th>nr.employed</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>57</td>\n",
       "      <td>technician</td>\n",
       "      <td>married</td>\n",
       "      <td>high.school</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>cellular</td>\n",
       "      <td>may</td>\n",
       "      <td>mon</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>1</td>\n",
       "      <td>failure</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>92.893</td>\n",
       "      <td>-46.2</td>\n",
       "      <td>1.299</td>\n",
       "      <td>5099.1</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>55</td>\n",
       "      <td>unknown</td>\n",
       "      <td>married</td>\n",
       "      <td>unknown</td>\n",
       "      <td>unknown</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>telephone</td>\n",
       "      <td>may</td>\n",
       "      <td>thu</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.860</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>33</td>\n",
       "      <td>blue-collar</td>\n",
       "      <td>married</td>\n",
       "      <td>basic.9y</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>may</td>\n",
       "      <td>fri</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>1</td>\n",
       "      <td>failure</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>92.893</td>\n",
       "      <td>-46.2</td>\n",
       "      <td>1.313</td>\n",
       "      <td>5099.1</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>36</td>\n",
       "      <td>admin.</td>\n",
       "      <td>married</td>\n",
       "      <td>high.school</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>telephone</td>\n",
       "      <td>jun</td>\n",
       "      <td>fri</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.4</td>\n",
       "      <td>94.465</td>\n",
       "      <td>-41.8</td>\n",
       "      <td>4.967</td>\n",
       "      <td>5228.1</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>27</td>\n",
       "      <td>housemaid</td>\n",
       "      <td>married</td>\n",
       "      <td>high.school</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>jul</td>\n",
       "      <td>fri</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.4</td>\n",
       "      <td>93.918</td>\n",
       "      <td>-42.7</td>\n",
       "      <td>4.963</td>\n",
       "      <td>5228.1</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   age          job  marital    education  default housing loan    contact  \\\n",
       "0   57   technician  married  high.school       no      no  yes   cellular   \n",
       "1   55      unknown  married      unknown  unknown     yes   no  telephone   \n",
       "2   33  blue-collar  married     basic.9y       no      no   no   cellular   \n",
       "3   36       admin.  married  high.school       no      no   no  telephone   \n",
       "4   27    housemaid  married  high.school       no     yes   no   cellular   \n",
       "\n",
       "  month day_of_week  ...  campaign  pdays  previous     poutcome emp.var.rate  \\\n",
       "0   may         mon  ...         1    999         1      failure         -1.8   \n",
       "1   may         thu  ...         2    999         0  nonexistent          1.1   \n",
       "2   may         fri  ...         1    999         1      failure         -1.8   \n",
       "3   jun         fri  ...         4    999         0  nonexistent          1.4   \n",
       "4   jul         fri  ...         2    999         0  nonexistent          1.4   \n",
       "\n",
       "   cons.price.idx  cons.conf.idx  euribor3m  nr.employed   y  \n",
       "0          92.893          -46.2      1.299       5099.1  no  \n",
       "1          93.994          -36.4      4.860       5191.0  no  \n",
       "2          92.893          -46.2      1.313       5099.1  no  \n",
       "3          94.465          -41.8      4.967       5228.1  no  \n",
       "4          93.918          -42.7      4.963       5228.1  no  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.take(5).to_pandas_dataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train\n",
    "This creates a general AutoML settings object.\n",
    "**Udacity notes:** These inputs must match what was used when training in the portal. `label_column_name` has to be `y` for example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_settings = {\n",
    "    \"experiment_timeout_minutes\": 20,\n",
    "    \"max_concurrent_iterations\": 5,\n",
    "    \"primary_metric\" : 'accuracy'\n",
    "}\n",
    "automl_config = AutoMLConfig(compute_target=compute_target,\n",
    "                             task = \"classification\",\n",
    "                             training_data=dataset,\n",
    "                             label_column_name=\"y\",   \n",
    "                             path = project_folder,\n",
    "                             enable_early_stopping= True,\n",
    "                             featurization= 'auto',\n",
    "                             debug_log = \"automl_errors.log\",\n",
    "                             **automl_settings\n",
    "                            )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Pipeline and AutoMLStep\n",
    "\n",
    "You can define outputs for the AutoMLStep using TrainingOutput."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.pipeline.core import PipelineData, TrainingOutput\n",
    "\n",
    "ds = ws.get_default_datastore()\n",
    "metrics_output_name = 'metrics_output'\n",
    "best_model_output_name = 'best_model_output'\n",
    "\n",
    "metrics_data = PipelineData(name='metrics_data',\n",
    "                           datastore=ds,\n",
    "                           pipeline_output_name=metrics_output_name,\n",
    "                           training_output=TrainingOutput(type='Metrics'))\n",
    "model_data = PipelineData(name='model_data',\n",
    "                           datastore=ds,\n",
    "                           pipeline_output_name=best_model_output_name,\n",
    "                           training_output=TrainingOutput(type='Model'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an AutoMLStep."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": [
     "automlstep-remarks-sample1"
    ]
   },
   "outputs": [],
   "source": [
    "automl_step = AutoMLStep(\n",
    "    name='automl_module',\n",
    "    automl_config=automl_config,\n",
    "    outputs=[metrics_data, model_data],\n",
    "    allow_reuse=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": [
     "automlstep-remarks-sample2"
    ]
   },
   "outputs": [],
   "source": [
    "from azureml.pipeline.core import Pipeline\n",
    "pipeline = Pipeline(\n",
    "    description=\"pipeline_with_automlstep\",\n",
    "    workspace=ws,    \n",
    "    steps=[automl_step])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created step automl_module [6bf916dc][cfacd8ad-2d53-425b-953e-3d9b2cca9846], (This step will run and generate new outputs)\n",
      "Submitted PipelineRun 61aaa4dd-39f7-470e-ba84-2c9c885067d3\n",
      "Link to Azure Machine Learning Portal: https://ml.azure.com/experiments/bankMarketing-exp/runs/61aaa4dd-39f7-470e-ba84-2c9c885067d3?wsid=/subscriptions/dafd62fe-51d3-4671-af50-6bac2f001b5d/resourcegroups/aml-quickstarts-128228/workspaces/quick-starts-ws-128228\n"
     ]
    }
   ],
   "source": [
    "pipeline_run = experiment.submit(pipeline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ca6415c4c9754a848cadf31f122d2cf4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_PipelineWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/aml.mini.widget.v1": "{\"status\": \"Completed\", \"workbench_run_details_uri\": \"https://ml.azure.com/experiments/bankMarketing-exp/runs/61aaa4dd-39f7-470e-ba84-2c9c885067d3?wsid=/subscriptions/dafd62fe-51d3-4671-af50-6bac2f001b5d/resourcegroups/aml-quickstarts-128228/workspaces/quick-starts-ws-128228\", \"run_id\": \"61aaa4dd-39f7-470e-ba84-2c9c885067d3\", \"run_properties\": {\"run_id\": \"61aaa4dd-39f7-470e-ba84-2c9c885067d3\", \"created_utc\": \"2020-11-28T18:11:46.065828Z\", \"properties\": {\"azureml.runsource\": \"azureml.PipelineRun\", \"runSource\": \"SDK\", \"runType\": \"SDK\", \"azureml.parameters\": \"{}\"}, \"tags\": {\"azureml.pipelineComponent\": \"pipelinerun\"}, \"end_time_utc\": \"2020-11-28T18:49:15.911579Z\", \"status\": \"Completed\", \"log_files\": {\"logs/azureml/executionlogs.txt\": \"https://mlstrg128228.blob.core.windows.net/azureml/ExperimentRun/dcid.61aaa4dd-39f7-470e-ba84-2c9c885067d3/logs/azureml/executionlogs.txt?sv=2019-02-02&sr=b&sig=PnhVWrhM0NkicHGSHWnjwYLkgoqHcysBhmUXC0YNQhQ%3D&st=2020-11-28T19%3A02%3A27Z&se=2020-11-29T03%3A12%3A27Z&sp=r\", \"logs/azureml/stderrlogs.txt\": \"https://mlstrg128228.blob.core.windows.net/azureml/ExperimentRun/dcid.61aaa4dd-39f7-470e-ba84-2c9c885067d3/logs/azureml/stderrlogs.txt?sv=2019-02-02&sr=b&sig=QePaNplLw%2FxO4ton%2FUCZX6FqsIM2cXx9OzkjTPwbcsc%3D&st=2020-11-28T19%3A02%3A27Z&se=2020-11-29T03%3A12%3A27Z&sp=r\", \"logs/azureml/stdoutlogs.txt\": \"https://mlstrg128228.blob.core.windows.net/azureml/ExperimentRun/dcid.61aaa4dd-39f7-470e-ba84-2c9c885067d3/logs/azureml/stdoutlogs.txt?sv=2019-02-02&sr=b&sig=2ne4Xqbfa8MDnJsWCFl76P5ruv%2BjYkJoUKxZC7ebnd4%3D&st=2020-11-28T19%3A02%3A27Z&se=2020-11-29T03%3A12%3A27Z&sp=r\"}, \"log_groups\": [[\"logs/azureml/executionlogs.txt\", \"logs/azureml/stderrlogs.txt\", \"logs/azureml/stdoutlogs.txt\"]], \"run_duration\": \"0:37:29\"}, \"child_runs\": [{\"run_id\": \"c904e5fb-b0d2-472d-b0c0-dea0bc123c48\", \"name\": \"automl_module\", \"status\": \"Finished\", \"start_time\": \"2020-11-28T18:12:25.257217Z\", \"created_time\": \"2020-11-28T18:11:55.503519Z\", \"end_time\": \"2020-11-28T18:43:55.497401Z\", \"duration\": \"0:31:59\", \"run_number\": 63, \"metric\": null, \"run_type\": \"azureml.StepRun\", \"training_percent\": null, \"created_time_dt\": \"2020-11-28T18:11:55.503519Z\", \"is_reused\": \"\"}], \"children_metrics\": {\"categories\": null, \"series\": null, \"metricName\": null}, \"run_metrics\": [], \"run_logs\": \"\\nRun is completed.\", \"graph\": {\"datasource_nodes\": {\"c7554563\": {\"node_id\": \"c7554563\", \"name\": \"Bank-marketing\"}}, \"module_nodes\": {\"6bf916dc\": {\"node_id\": \"6bf916dc\", \"name\": \"automl_module\", \"status\": \"Finished\", \"_is_reused\": false, \"run_id\": \"c904e5fb-b0d2-472d-b0c0-dea0bc123c48\"}}, \"edges\": [{\"source_node_id\": \"c7554563\", \"source_node_name\": \"Bank-marketing\", \"source_name\": \"data\", \"target_name\": \"training_data\", \"dst_node_id\": \"6bf916dc\", \"dst_node_name\": \"automl_module\"}], \"child_runs\": [{\"run_id\": \"c904e5fb-b0d2-472d-b0c0-dea0bc123c48\", \"name\": \"automl_module\", \"status\": \"Finished\", \"start_time\": \"2020-11-28T18:12:25.257217Z\", \"created_time\": \"2020-11-28T18:11:55.503519Z\", \"end_time\": \"2020-11-28T18:43:55.497401Z\", \"duration\": \"0:31:59\", \"run_number\": 63, \"metric\": null, \"run_type\": \"azureml.StepRun\", \"training_percent\": null, \"created_time_dt\": \"2020-11-28T18:11:55.503519Z\", \"is_reused\": \"\"}]}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": false, \"log_level\": \"INFO\", \"sdk_version\": \"1.18.0\"}, \"loading\": false}"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from azureml.widgets import RunDetails\n",
    "RunDetails(pipeline_run).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_run.wait_for_completion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examine Results\n",
    "\n",
    "### Retrieve the metrics of all child runs\n",
    "Outputs of above run can be used as inputs of other steps in pipeline. In this tutorial, we will examine the outputs by retrieve output data and running some tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading azureml/c904e5fb-b0d2-472d-b0c0-dea0bc123c48/metrics_data\n",
      "Downloaded azureml/c904e5fb-b0d2-472d-b0c0-dea0bc123c48/metrics_data, 1 files out of an estimated total of 1\n"
     ]
    }
   ],
   "source": [
    "metrics_output = pipeline_run.get_pipeline_output(metrics_output_name)\n",
    "num_file_downloaded = metrics_output.download('.', show_progress=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_6</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_19</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_21</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_29</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_34</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_9</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_23</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_25</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_35</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_26</th>\n",
       "      <th>...</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_32</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_0</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_14</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_16</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_4</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_5</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_8</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_15</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_20</th>\n",
       "      <th>c904e5fb-b0d2-472d-b0c0-dea0bc123c48_38</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>precision_score_weighted</th>\n",
       "      <td>[0.8811157550701237]</td>\n",
       "      <td>[0.8850009147375308]</td>\n",
       "      <td>[0.9076871902108772]</td>\n",
       "      <td>[0.8732450758932834]</td>\n",
       "      <td>[0.788565560086672]</td>\n",
       "      <td>[0.8876166057383225]</td>\n",
       "      <td>[0.9150928962708703]</td>\n",
       "      <td>[0.788565560086672]</td>\n",
       "      <td>[0.8872411026406429]</td>\n",
       "      <td>[0.899969838742595]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.788565560086672]</td>\n",
       "      <td>[0.9072720074188747]</td>\n",
       "      <td>[0.8842474652444061]</td>\n",
       "      <td>[0.788565560086672]</td>\n",
       "      <td>[0.9182355798069128]</td>\n",
       "      <td>[0.8826993014565436]</td>\n",
       "      <td>[0.8823379905566089]</td>\n",
       "      <td>[0.918526645726982]</td>\n",
       "      <td>[0.8781641870148528]</td>\n",
       "      <td>[0.909448627526437]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>average_precision_score_micro</th>\n",
       "      <td>[0.9703567488474186]</td>\n",
       "      <td>[0.9686589213387464]</td>\n",
       "      <td>[0.8753334963872397]</td>\n",
       "      <td>[0.9744028647017335]</td>\n",
       "      <td>[0.9708849513936123]</td>\n",
       "      <td>[0.8075627173222304]</td>\n",
       "      <td>[0.9203306408744603]</td>\n",
       "      <td>[0.971504365730406]</td>\n",
       "      <td>[0.9749716762937735]</td>\n",
       "      <td>[0.9773969896823556]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.9688737022895029]</td>\n",
       "      <td>[0.9806603102489483]</td>\n",
       "      <td>[0.8193241774793902]</td>\n",
       "      <td>[0.9629364555606664]</td>\n",
       "      <td>[0.8317277043239888]</td>\n",
       "      <td>[0.9681835493215114]</td>\n",
       "      <td>[0.9618011068804218]</td>\n",
       "      <td>[0.8704031509845976]</td>\n",
       "      <td>[0.8054037059077446]</td>\n",
       "      <td>[0.9797107523500044]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>average_precision_score_weighted</th>\n",
       "      <td>[0.9320889711178237]</td>\n",
       "      <td>[0.9376133198890253]</td>\n",
       "      <td>[0.9337610621269242]</td>\n",
       "      <td>[0.9457035885693491]</td>\n",
       "      <td>[0.9351375094525163]</td>\n",
       "      <td>[0.9238108629450296]</td>\n",
       "      <td>[0.93697807507535]</td>\n",
       "      <td>[0.938037137377674]</td>\n",
       "      <td>[0.941035720886307]</td>\n",
       "      <td>[0.9480034685313283]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.9294913358914507]</td>\n",
       "      <td>[0.9531771295804466]</td>\n",
       "      <td>[0.9160128044475222]</td>\n",
       "      <td>[0.9224727710016692]</td>\n",
       "      <td>[0.9370750303308232]</td>\n",
       "      <td>[0.9364668231804184]</td>\n",
       "      <td>[0.9256523104186373]</td>\n",
       "      <td>[0.9369762229683729]</td>\n",
       "      <td>[0.909723441391043]</td>\n",
       "      <td>[0.9523248695577932]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>f1_score_macro</th>\n",
       "      <td>[0.6132139688332033]</td>\n",
       "      <td>[0.6937746140605427]</td>\n",
       "      <td>[0.6610225794993417]</td>\n",
       "      <td>[0.47852705178786353]</td>\n",
       "      <td>[0.4703423886834914]</td>\n",
       "      <td>[0.6536071116194577]</td>\n",
       "      <td>[0.7470138530892498]</td>\n",
       "      <td>[0.4703423886834914]</td>\n",
       "      <td>[0.674881533645934]</td>\n",
       "      <td>[0.7431757525868666]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.4703423886834914]</td>\n",
       "      <td>[0.7653697272147331]</td>\n",
       "      <td>[0.6250946364070289]</td>\n",
       "      <td>[0.4703423886834914]</td>\n",
       "      <td>[0.7105954377506608]</td>\n",
       "      <td>[0.6616134730891533]</td>\n",
       "      <td>[0.6170656133421111]</td>\n",
       "      <td>[0.7200647870031351]</td>\n",
       "      <td>[0.6043570770314957]</td>\n",
       "      <td>[0.7677334161916184]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>log_loss</th>\n",
       "      <td>[0.2285875257414639]</td>\n",
       "      <td>[0.3029335482501969]</td>\n",
       "      <td>[0.4822721441310681]</td>\n",
       "      <td>[0.2485995933230933]</td>\n",
       "      <td>[0.22693806377246428]</td>\n",
       "      <td>[0.5224355614991127]</td>\n",
       "      <td>[0.7695856566025591]</td>\n",
       "      <td>[0.33626916244298677]</td>\n",
       "      <td>[0.2119826616116293]</td>\n",
       "      <td>[0.19664471940167458]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.25637830013935325]</td>\n",
       "      <td>[0.17775706110025447]</td>\n",
       "      <td>[0.6093462584593584]</td>\n",
       "      <td>[0.2749583271733524]</td>\n",
       "      <td>[0.46651601264873727]</td>\n",
       "      <td>[0.23145009573598296]</td>\n",
       "      <td>[0.2618000004386081]</td>\n",
       "      <td>[1.0528420699356784]</td>\n",
       "      <td>[0.5722209570104337]</td>\n",
       "      <td>[0.2388149173745338]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>matthews_correlation</th>\n",
       "      <td>[0.3149058510980744]</td>\n",
       "      <td>[0.4062162676409269]</td>\n",
       "      <td>[0.4400892283056553]</td>\n",
       "      <td>[0.07053425306980969]</td>\n",
       "      <td>[0.0]</td>\n",
       "      <td>[0.38061454428956815]</td>\n",
       "      <td>[0.536663013911126]</td>\n",
       "      <td>[0.0]</td>\n",
       "      <td>[0.3938665324111809]</td>\n",
       "      <td>[0.49213857715421105]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.0]</td>\n",
       "      <td>[0.5323740218566827]</td>\n",
       "      <td>[0.3482629755918812]</td>\n",
       "      <td>[0.0]</td>\n",
       "      <td>[0.5114480241575082]</td>\n",
       "      <td>[0.36945708081714224]</td>\n",
       "      <td>[0.3218684594572277]</td>\n",
       "      <td>[0.520875607466786]</td>\n",
       "      <td>[0.31550798797377594]</td>\n",
       "      <td>[0.5404101114596196]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>precision_score_macro</th>\n",
       "      <td>[0.7982595199638411]</td>\n",
       "      <td>[0.7549849939275037]</td>\n",
       "      <td>[0.6493038973221901]</td>\n",
       "      <td>[0.8193938012762079]</td>\n",
       "      <td>[0.4440060698027314]</td>\n",
       "      <td>[0.6357931251078711]</td>\n",
       "      <td>[0.7085242050507593]</td>\n",
       "      <td>[0.4440060698027314]</td>\n",
       "      <td>[0.7863945498280172]</td>\n",
       "      <td>[0.7756218523915434]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.4440060698027314]</td>\n",
       "      <td>[0.7819118765348991]</td>\n",
       "      <td>[0.6193021454122402]</td>\n",
       "      <td>[0.4440060698027314]</td>\n",
       "      <td>[0.6811761566448443]</td>\n",
       "      <td>[0.774300365133433]</td>\n",
       "      <td>[0.8017560826299885]</td>\n",
       "      <td>[0.6873597929249353]</td>\n",
       "      <td>[0.6064710240250706]</td>\n",
       "      <td>[0.798907733703278]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>f1_score_weighted</th>\n",
       "      <td>[0.871192710611544]</td>\n",
       "      <td>[0.8887173812479593]</td>\n",
       "      <td>[0.8115904641932854]</td>\n",
       "      <td>[0.8373926979576183]</td>\n",
       "      <td>[0.8353395018439429]</td>\n",
       "      <td>[0.82076651627607]</td>\n",
       "      <td>[0.8805375927431068]</td>\n",
       "      <td>[0.8353395018439429]</td>\n",
       "      <td>[0.8864458469569265]</td>\n",
       "      <td>[0.9028591783574997]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.8353395018439429]</td>\n",
       "      <td>[0.9091539479147899]</td>\n",
       "      <td>[0.7947997493884662]</td>\n",
       "      <td>[0.8353395018439429]</td>\n",
       "      <td>[0.848890702599696]</td>\n",
       "      <td>[0.8823695777855658]</td>\n",
       "      <td>[0.8722950091956113]</td>\n",
       "      <td>[0.8566040394576435]</td>\n",
       "      <td>[0.7781380114280563]</td>\n",
       "      <td>[0.9117102130976235]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>balanced_accuracy</th>\n",
       "      <td>[0.5831203100137632]</td>\n",
       "      <td>[0.6617856540834718]</td>\n",
       "      <td>[0.8243025338660768]</td>\n",
       "      <td>[0.5038941589005774]</td>\n",
       "      <td>[0.5]</td>\n",
       "      <td>[0.7667061222902045]</td>\n",
       "      <td>[0.8452922772563338]</td>\n",
       "      <td>[0.5]</td>\n",
       "      <td>[0.635417071874068]</td>\n",
       "      <td>[0.7196853923426452]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.5]</td>\n",
       "      <td>[0.7513392683482543]</td>\n",
       "      <td>[0.7541595118616942]</td>\n",
       "      <td>[0.5]</td>\n",
       "      <td>[0.8609457864913577]</td>\n",
       "      <td>[0.6244060817231549]</td>\n",
       "      <td>[0.5858303371140341]</td>\n",
       "      <td>[0.8620192387843222]</td>\n",
       "      <td>[0.7337379850216821]</td>\n",
       "      <td>[0.7442585584434108]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>recall_score_weighted</th>\n",
       "      <td>[0.898937784522003]</td>\n",
       "      <td>[0.8998482549317147]</td>\n",
       "      <td>[0.7720789074355083]</td>\n",
       "      <td>[0.888619119878604]</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.787556904400607]</td>\n",
       "      <td>[0.8640364188163885]</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.9034901365705614]</td>\n",
       "      <td>[0.9080424886191198]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.9116843702579667]</td>\n",
       "      <td>[0.7526555386949925]</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.8203338391502276]</td>\n",
       "      <td>[0.9007587253414264]</td>\n",
       "      <td>[0.8995447647951441]</td>\n",
       "      <td>[0.8306525037936268]</td>\n",
       "      <td>[0.7311077389984826]</td>\n",
       "      <td>[0.9159332321699545]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AUC_micro</th>\n",
       "      <td>[0.969436286643901]</td>\n",
       "      <td>[0.9694322339683293]</td>\n",
       "      <td>[0.8735382851195423]</td>\n",
       "      <td>[0.9731035896113345]</td>\n",
       "      <td>[0.969774040310306]</td>\n",
       "      <td>[0.8275269698651334]</td>\n",
       "      <td>[0.9293920756376632]</td>\n",
       "      <td>[0.9715119933867703]</td>\n",
       "      <td>[0.9737232805487691]</td>\n",
       "      <td>[0.9762771109028486]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.9673541324626221]</td>\n",
       "      <td>[0.979695082216353]</td>\n",
       "      <td>[0.8270461751722962]</td>\n",
       "      <td>[0.9640629914732627]</td>\n",
       "      <td>[0.8694183719757484]</td>\n",
       "      <td>[0.9698026853580978]</td>\n",
       "      <td>[0.9612656321598227]</td>\n",
       "      <td>[0.8888591487999704]</td>\n",
       "      <td>[0.8269354634441757]</td>\n",
       "      <td>[0.9788009146151915]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>f1_score_micro</th>\n",
       "      <td>[0.8989377845220029]</td>\n",
       "      <td>[0.8998482549317147]</td>\n",
       "      <td>[0.7720789074355083]</td>\n",
       "      <td>[0.888619119878604]</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.787556904400607]</td>\n",
       "      <td>[0.8640364188163885]</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.9034901365705614]</td>\n",
       "      <td>[0.90804248861912]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.9116843702579667]</td>\n",
       "      <td>[0.7526555386949924]</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.8203338391502276]</td>\n",
       "      <td>[0.9007587253414264]</td>\n",
       "      <td>[0.8995447647951441]</td>\n",
       "      <td>[0.8306525037936268]</td>\n",
       "      <td>[0.7311077389984825]</td>\n",
       "      <td>[0.9159332321699545]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>norm_macro_recall</th>\n",
       "      <td>[0.16624062002752638]</td>\n",
       "      <td>[0.32357130816694357]</td>\n",
       "      <td>[0.6486050677321535]</td>\n",
       "      <td>[0.007788317801154854]</td>\n",
       "      <td>[0.0]</td>\n",
       "      <td>[0.533412244580409]</td>\n",
       "      <td>[0.6905845545126676]</td>\n",
       "      <td>[0.0]</td>\n",
       "      <td>[0.2708341437481361]</td>\n",
       "      <td>[0.43937078468529034]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.0]</td>\n",
       "      <td>[0.5026785366965085]</td>\n",
       "      <td>[0.5083190237233883]</td>\n",
       "      <td>[0.0]</td>\n",
       "      <td>[0.7218915729827153]</td>\n",
       "      <td>[0.24881216344630985]</td>\n",
       "      <td>[0.1716606742280682]</td>\n",
       "      <td>[0.7240384775686444]</td>\n",
       "      <td>[0.46747597004336416]</td>\n",
       "      <td>[0.48851711688682165]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>precision_score_micro</th>\n",
       "      <td>[0.898937784522003]</td>\n",
       "      <td>[0.8998482549317147]</td>\n",
       "      <td>[0.7720789074355083]</td>\n",
       "      <td>[0.888619119878604]</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.787556904400607]</td>\n",
       "      <td>[0.8640364188163885]</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.9034901365705614]</td>\n",
       "      <td>[0.9080424886191198]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.9116843702579667]</td>\n",
       "      <td>[0.7526555386949925]</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.8203338391502276]</td>\n",
       "      <td>[0.9007587253414264]</td>\n",
       "      <td>[0.8995447647951441]</td>\n",
       "      <td>[0.8306525037936268]</td>\n",
       "      <td>[0.7311077389984826]</td>\n",
       "      <td>[0.9159332321699545]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weighted_accuracy</th>\n",
       "      <td>[0.9773467207242611]</td>\n",
       "      <td>[0.9589527592379402]</td>\n",
       "      <td>[0.7591131936179908]</td>\n",
       "      <td>[0.9841359210553396]</td>\n",
       "      <td>[0.9843450583187134]</td>\n",
       "      <td>[0.7927335895945071]</td>\n",
       "      <td>[0.8686900821453001]</td>\n",
       "      <td>[0.9843450583187134]</td>\n",
       "      <td>[0.9700454272809962]</td>\n",
       "      <td>[0.954806460651324]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.9843450583187134]</td>\n",
       "      <td>[0.9514937218005303]</td>\n",
       "      <td>[0.7522821428394862]</td>\n",
       "      <td>[0.9843450583187134]</td>\n",
       "      <td>[0.8102509911600128]</td>\n",
       "      <td>[0.9693696115393181]</td>\n",
       "      <td>[0.9774315713566799]</td>\n",
       "      <td>[0.8228649919512622]</td>\n",
       "      <td>[0.7304547200578732]</td>\n",
       "      <td>[0.9585554099349054]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>accuracy</th>\n",
       "      <td>[0.898937784522003]</td>\n",
       "      <td>[0.8998482549317147]</td>\n",
       "      <td>[0.7720789074355083]</td>\n",
       "      <td>[0.888619119878604]</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.787556904400607]</td>\n",
       "      <td>[0.8640364188163885]</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.9034901365705614]</td>\n",
       "      <td>[0.9080424886191198]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.9116843702579667]</td>\n",
       "      <td>[0.7526555386949925]</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.8203338391502276]</td>\n",
       "      <td>[0.9007587253414264]</td>\n",
       "      <td>[0.8995447647951441]</td>\n",
       "      <td>[0.8306525037936268]</td>\n",
       "      <td>[0.7311077389984826]</td>\n",
       "      <td>[0.9159332321699545]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>average_precision_score_macro</th>\n",
       "      <td>[0.7402297763729253]</td>\n",
       "      <td>[0.7602770155641473]</td>\n",
       "      <td>[0.7513286103482077]</td>\n",
       "      <td>[0.7904534536669628]</td>\n",
       "      <td>[0.7522873325529215]</td>\n",
       "      <td>[0.727208050549709]</td>\n",
       "      <td>[0.7546005621322331]</td>\n",
       "      <td>[0.7633894156044256]</td>\n",
       "      <td>[0.7710658035358414]</td>\n",
       "      <td>[0.7984255787742488]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.7319347041742593]</td>\n",
       "      <td>[0.8151093723721079]</td>\n",
       "      <td>[0.7045642649395899]</td>\n",
       "      <td>[0.7206954310055973]</td>\n",
       "      <td>[0.7554261028392691]</td>\n",
       "      <td>[0.7525938317909414]</td>\n",
       "      <td>[0.7375304402096373]</td>\n",
       "      <td>[0.7558790923089123]</td>\n",
       "      <td>[0.7063419739896816]</td>\n",
       "      <td>[0.8138331417012676]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AUC_macro</th>\n",
       "      <td>[0.9076553171546753]</td>\n",
       "      <td>[0.9192243357840277]</td>\n",
       "      <td>[0.900182366485319]</td>\n",
       "      <td>[0.9278193636345112]</td>\n",
       "      <td>[0.911084529505582]</td>\n",
       "      <td>[0.8636085779859849]</td>\n",
       "      <td>[0.9219853032433263]</td>\n",
       "      <td>[0.9198226534555161]</td>\n",
       "      <td>[0.9242836396238193]</td>\n",
       "      <td>[0.9319154315945073]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.8989176562989143]</td>\n",
       "      <td>[0.9450464668693166]</td>\n",
       "      <td>[0.8467551917487732]</td>\n",
       "      <td>[0.8823703753100416]</td>\n",
       "      <td>[0.9198800771329655]</td>\n",
       "      <td>[0.920452461530767]</td>\n",
       "      <td>[0.8662885039650123]</td>\n",
       "      <td>[0.9211651634629812]</td>\n",
       "      <td>[0.8180188090329297]</td>\n",
       "      <td>[0.9409666072053748]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AUC_weighted</th>\n",
       "      <td>[0.9076553171546753]</td>\n",
       "      <td>[0.9192243357840276]</td>\n",
       "      <td>[0.900182366485319]</td>\n",
       "      <td>[0.9278193636345111]</td>\n",
       "      <td>[0.911084529505582]</td>\n",
       "      <td>[0.8636085779859849]</td>\n",
       "      <td>[0.9219853032433264]</td>\n",
       "      <td>[0.9198226534555161]</td>\n",
       "      <td>[0.9242836396238192]</td>\n",
       "      <td>[0.9319154315945072]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.8989176562989144]</td>\n",
       "      <td>[0.9450464668693167]</td>\n",
       "      <td>[0.8467551917487733]</td>\n",
       "      <td>[0.8823703753100416]</td>\n",
       "      <td>[0.9198800771329654]</td>\n",
       "      <td>[0.920452461530767]</td>\n",
       "      <td>[0.8662885039650123]</td>\n",
       "      <td>[0.9211651634629813]</td>\n",
       "      <td>[0.8180188090329298]</td>\n",
       "      <td>[0.9409666072053747]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>recall_score_micro</th>\n",
       "      <td>[0.898937784522003]</td>\n",
       "      <td>[0.8998482549317147]</td>\n",
       "      <td>[0.7720789074355083]</td>\n",
       "      <td>[0.888619119878604]</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.787556904400607]</td>\n",
       "      <td>[0.8640364188163885]</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.9034901365705614]</td>\n",
       "      <td>[0.9080424886191198]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.9116843702579667]</td>\n",
       "      <td>[0.7526555386949925]</td>\n",
       "      <td>[0.8880121396054628]</td>\n",
       "      <td>[0.8203338391502276]</td>\n",
       "      <td>[0.9007587253414264]</td>\n",
       "      <td>[0.8995447647951441]</td>\n",
       "      <td>[0.8306525037936268]</td>\n",
       "      <td>[0.7311077389984826]</td>\n",
       "      <td>[0.9159332321699545]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>recall_score_macro</th>\n",
       "      <td>[0.5831203100137632]</td>\n",
       "      <td>[0.6617856540834718]</td>\n",
       "      <td>[0.8243025338660768]</td>\n",
       "      <td>[0.5038941589005774]</td>\n",
       "      <td>[0.5]</td>\n",
       "      <td>[0.7667061222902045]</td>\n",
       "      <td>[0.8452922772563338]</td>\n",
       "      <td>[0.5]</td>\n",
       "      <td>[0.635417071874068]</td>\n",
       "      <td>[0.7196853923426452]</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.5]</td>\n",
       "      <td>[0.7513392683482543]</td>\n",
       "      <td>[0.7541595118616942]</td>\n",
       "      <td>[0.5]</td>\n",
       "      <td>[0.8609457864913577]</td>\n",
       "      <td>[0.6244060817231549]</td>\n",
       "      <td>[0.5858303371140341]</td>\n",
       "      <td>[0.8620192387843222]</td>\n",
       "      <td>[0.7337379850216821]</td>\n",
       "      <td>[0.7442585584434108]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>21 rows × 38 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_6  \\\n",
       "precision_score_weighted                           [0.8811157550701237]   \n",
       "average_precision_score_micro                      [0.9703567488474186]   \n",
       "average_precision_score_weighted                   [0.9320889711178237]   \n",
       "f1_score_macro                                     [0.6132139688332033]   \n",
       "log_loss                                           [0.2285875257414639]   \n",
       "matthews_correlation                               [0.3149058510980744]   \n",
       "precision_score_macro                              [0.7982595199638411]   \n",
       "f1_score_weighted                                   [0.871192710611544]   \n",
       "balanced_accuracy                                  [0.5831203100137632]   \n",
       "recall_score_weighted                               [0.898937784522003]   \n",
       "AUC_micro                                           [0.969436286643901]   \n",
       "f1_score_micro                                     [0.8989377845220029]   \n",
       "norm_macro_recall                                 [0.16624062002752638]   \n",
       "precision_score_micro                               [0.898937784522003]   \n",
       "weighted_accuracy                                  [0.9773467207242611]   \n",
       "accuracy                                            [0.898937784522003]   \n",
       "average_precision_score_macro                      [0.7402297763729253]   \n",
       "AUC_macro                                          [0.9076553171546753]   \n",
       "AUC_weighted                                       [0.9076553171546753]   \n",
       "recall_score_micro                                  [0.898937784522003]   \n",
       "recall_score_macro                                 [0.5831203100137632]   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_19  \\\n",
       "precision_score_weighted                            [0.8850009147375308]   \n",
       "average_precision_score_micro                       [0.9686589213387464]   \n",
       "average_precision_score_weighted                    [0.9376133198890253]   \n",
       "f1_score_macro                                      [0.6937746140605427]   \n",
       "log_loss                                            [0.3029335482501969]   \n",
       "matthews_correlation                                [0.4062162676409269]   \n",
       "precision_score_macro                               [0.7549849939275037]   \n",
       "f1_score_weighted                                   [0.8887173812479593]   \n",
       "balanced_accuracy                                   [0.6617856540834718]   \n",
       "recall_score_weighted                               [0.8998482549317147]   \n",
       "AUC_micro                                           [0.9694322339683293]   \n",
       "f1_score_micro                                      [0.8998482549317147]   \n",
       "norm_macro_recall                                  [0.32357130816694357]   \n",
       "precision_score_micro                               [0.8998482549317147]   \n",
       "weighted_accuracy                                   [0.9589527592379402]   \n",
       "accuracy                                            [0.8998482549317147]   \n",
       "average_precision_score_macro                       [0.7602770155641473]   \n",
       "AUC_macro                                           [0.9192243357840277]   \n",
       "AUC_weighted                                        [0.9192243357840276]   \n",
       "recall_score_micro                                  [0.8998482549317147]   \n",
       "recall_score_macro                                  [0.6617856540834718]   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_21  \\\n",
       "precision_score_weighted                            [0.9076871902108772]   \n",
       "average_precision_score_micro                       [0.8753334963872397]   \n",
       "average_precision_score_weighted                    [0.9337610621269242]   \n",
       "f1_score_macro                                      [0.6610225794993417]   \n",
       "log_loss                                            [0.4822721441310681]   \n",
       "matthews_correlation                                [0.4400892283056553]   \n",
       "precision_score_macro                               [0.6493038973221901]   \n",
       "f1_score_weighted                                   [0.8115904641932854]   \n",
       "balanced_accuracy                                   [0.8243025338660768]   \n",
       "recall_score_weighted                               [0.7720789074355083]   \n",
       "AUC_micro                                           [0.8735382851195423]   \n",
       "f1_score_micro                                      [0.7720789074355083]   \n",
       "norm_macro_recall                                   [0.6486050677321535]   \n",
       "precision_score_micro                               [0.7720789074355083]   \n",
       "weighted_accuracy                                   [0.7591131936179908]   \n",
       "accuracy                                            [0.7720789074355083]   \n",
       "average_precision_score_macro                       [0.7513286103482077]   \n",
       "AUC_macro                                            [0.900182366485319]   \n",
       "AUC_weighted                                         [0.900182366485319]   \n",
       "recall_score_micro                                  [0.7720789074355083]   \n",
       "recall_score_macro                                  [0.8243025338660768]   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_29  \\\n",
       "precision_score_weighted                            [0.8732450758932834]   \n",
       "average_precision_score_micro                       [0.9744028647017335]   \n",
       "average_precision_score_weighted                    [0.9457035885693491]   \n",
       "f1_score_macro                                     [0.47852705178786353]   \n",
       "log_loss                                            [0.2485995933230933]   \n",
       "matthews_correlation                               [0.07053425306980969]   \n",
       "precision_score_macro                               [0.8193938012762079]   \n",
       "f1_score_weighted                                   [0.8373926979576183]   \n",
       "balanced_accuracy                                   [0.5038941589005774]   \n",
       "recall_score_weighted                                [0.888619119878604]   \n",
       "AUC_micro                                           [0.9731035896113345]   \n",
       "f1_score_micro                                       [0.888619119878604]   \n",
       "norm_macro_recall                                 [0.007788317801154854]   \n",
       "precision_score_micro                                [0.888619119878604]   \n",
       "weighted_accuracy                                   [0.9841359210553396]   \n",
       "accuracy                                             [0.888619119878604]   \n",
       "average_precision_score_macro                       [0.7904534536669628]   \n",
       "AUC_macro                                           [0.9278193636345112]   \n",
       "AUC_weighted                                        [0.9278193636345111]   \n",
       "recall_score_micro                                   [0.888619119878604]   \n",
       "recall_score_macro                                  [0.5038941589005774]   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_34  \\\n",
       "precision_score_weighted                             [0.788565560086672]   \n",
       "average_precision_score_micro                       [0.9708849513936123]   \n",
       "average_precision_score_weighted                    [0.9351375094525163]   \n",
       "f1_score_macro                                      [0.4703423886834914]   \n",
       "log_loss                                           [0.22693806377246428]   \n",
       "matthews_correlation                                               [0.0]   \n",
       "precision_score_macro                               [0.4440060698027314]   \n",
       "f1_score_weighted                                   [0.8353395018439429]   \n",
       "balanced_accuracy                                                  [0.5]   \n",
       "recall_score_weighted                               [0.8880121396054628]   \n",
       "AUC_micro                                            [0.969774040310306]   \n",
       "f1_score_micro                                      [0.8880121396054628]   \n",
       "norm_macro_recall                                                  [0.0]   \n",
       "precision_score_micro                               [0.8880121396054628]   \n",
       "weighted_accuracy                                   [0.9843450583187134]   \n",
       "accuracy                                            [0.8880121396054628]   \n",
       "average_precision_score_macro                       [0.7522873325529215]   \n",
       "AUC_macro                                            [0.911084529505582]   \n",
       "AUC_weighted                                         [0.911084529505582]   \n",
       "recall_score_micro                                  [0.8880121396054628]   \n",
       "recall_score_macro                                                 [0.5]   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_9  \\\n",
       "precision_score_weighted                           [0.8876166057383225]   \n",
       "average_precision_score_micro                      [0.8075627173222304]   \n",
       "average_precision_score_weighted                   [0.9238108629450296]   \n",
       "f1_score_macro                                     [0.6536071116194577]   \n",
       "log_loss                                           [0.5224355614991127]   \n",
       "matthews_correlation                              [0.38061454428956815]   \n",
       "precision_score_macro                              [0.6357931251078711]   \n",
       "f1_score_weighted                                    [0.82076651627607]   \n",
       "balanced_accuracy                                  [0.7667061222902045]   \n",
       "recall_score_weighted                               [0.787556904400607]   \n",
       "AUC_micro                                          [0.8275269698651334]   \n",
       "f1_score_micro                                      [0.787556904400607]   \n",
       "norm_macro_recall                                   [0.533412244580409]   \n",
       "precision_score_micro                               [0.787556904400607]   \n",
       "weighted_accuracy                                  [0.7927335895945071]   \n",
       "accuracy                                            [0.787556904400607]   \n",
       "average_precision_score_macro                       [0.727208050549709]   \n",
       "AUC_macro                                          [0.8636085779859849]   \n",
       "AUC_weighted                                       [0.8636085779859849]   \n",
       "recall_score_micro                                  [0.787556904400607]   \n",
       "recall_score_macro                                 [0.7667061222902045]   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_23  \\\n",
       "precision_score_weighted                            [0.9150928962708703]   \n",
       "average_precision_score_micro                       [0.9203306408744603]   \n",
       "average_precision_score_weighted                      [0.93697807507535]   \n",
       "f1_score_macro                                      [0.7470138530892498]   \n",
       "log_loss                                            [0.7695856566025591]   \n",
       "matthews_correlation                                 [0.536663013911126]   \n",
       "precision_score_macro                               [0.7085242050507593]   \n",
       "f1_score_weighted                                   [0.8805375927431068]   \n",
       "balanced_accuracy                                   [0.8452922772563338]   \n",
       "recall_score_weighted                               [0.8640364188163885]   \n",
       "AUC_micro                                           [0.9293920756376632]   \n",
       "f1_score_micro                                      [0.8640364188163885]   \n",
       "norm_macro_recall                                   [0.6905845545126676]   \n",
       "precision_score_micro                               [0.8640364188163885]   \n",
       "weighted_accuracy                                   [0.8686900821453001]   \n",
       "accuracy                                            [0.8640364188163885]   \n",
       "average_precision_score_macro                       [0.7546005621322331]   \n",
       "AUC_macro                                           [0.9219853032433263]   \n",
       "AUC_weighted                                        [0.9219853032433264]   \n",
       "recall_score_micro                                  [0.8640364188163885]   \n",
       "recall_score_macro                                  [0.8452922772563338]   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_25  \\\n",
       "precision_score_weighted                             [0.788565560086672]   \n",
       "average_precision_score_micro                        [0.971504365730406]   \n",
       "average_precision_score_weighted                     [0.938037137377674]   \n",
       "f1_score_macro                                      [0.4703423886834914]   \n",
       "log_loss                                           [0.33626916244298677]   \n",
       "matthews_correlation                                               [0.0]   \n",
       "precision_score_macro                               [0.4440060698027314]   \n",
       "f1_score_weighted                                   [0.8353395018439429]   \n",
       "balanced_accuracy                                                  [0.5]   \n",
       "recall_score_weighted                               [0.8880121396054628]   \n",
       "AUC_micro                                           [0.9715119933867703]   \n",
       "f1_score_micro                                      [0.8880121396054628]   \n",
       "norm_macro_recall                                                  [0.0]   \n",
       "precision_score_micro                               [0.8880121396054628]   \n",
       "weighted_accuracy                                   [0.9843450583187134]   \n",
       "accuracy                                            [0.8880121396054628]   \n",
       "average_precision_score_macro                       [0.7633894156044256]   \n",
       "AUC_macro                                           [0.9198226534555161]   \n",
       "AUC_weighted                                        [0.9198226534555161]   \n",
       "recall_score_micro                                  [0.8880121396054628]   \n",
       "recall_score_macro                                                 [0.5]   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_35  \\\n",
       "precision_score_weighted                            [0.8872411026406429]   \n",
       "average_precision_score_micro                       [0.9749716762937735]   \n",
       "average_precision_score_weighted                     [0.941035720886307]   \n",
       "f1_score_macro                                       [0.674881533645934]   \n",
       "log_loss                                            [0.2119826616116293]   \n",
       "matthews_correlation                                [0.3938665324111809]   \n",
       "precision_score_macro                               [0.7863945498280172]   \n",
       "f1_score_weighted                                   [0.8864458469569265]   \n",
       "balanced_accuracy                                    [0.635417071874068]   \n",
       "recall_score_weighted                               [0.9034901365705614]   \n",
       "AUC_micro                                           [0.9737232805487691]   \n",
       "f1_score_micro                                      [0.9034901365705614]   \n",
       "norm_macro_recall                                   [0.2708341437481361]   \n",
       "precision_score_micro                               [0.9034901365705614]   \n",
       "weighted_accuracy                                   [0.9700454272809962]   \n",
       "accuracy                                            [0.9034901365705614]   \n",
       "average_precision_score_macro                       [0.7710658035358414]   \n",
       "AUC_macro                                           [0.9242836396238193]   \n",
       "AUC_weighted                                        [0.9242836396238192]   \n",
       "recall_score_micro                                  [0.9034901365705614]   \n",
       "recall_score_macro                                   [0.635417071874068]   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_26  ...  \\\n",
       "precision_score_weighted                             [0.899969838742595]  ...   \n",
       "average_precision_score_micro                       [0.9773969896823556]  ...   \n",
       "average_precision_score_weighted                    [0.9480034685313283]  ...   \n",
       "f1_score_macro                                      [0.7431757525868666]  ...   \n",
       "log_loss                                           [0.19664471940167458]  ...   \n",
       "matthews_correlation                               [0.49213857715421105]  ...   \n",
       "precision_score_macro                               [0.7756218523915434]  ...   \n",
       "f1_score_weighted                                   [0.9028591783574997]  ...   \n",
       "balanced_accuracy                                   [0.7196853923426452]  ...   \n",
       "recall_score_weighted                               [0.9080424886191198]  ...   \n",
       "AUC_micro                                           [0.9762771109028486]  ...   \n",
       "f1_score_micro                                        [0.90804248861912]  ...   \n",
       "norm_macro_recall                                  [0.43937078468529034]  ...   \n",
       "precision_score_micro                               [0.9080424886191198]  ...   \n",
       "weighted_accuracy                                    [0.954806460651324]  ...   \n",
       "accuracy                                            [0.9080424886191198]  ...   \n",
       "average_precision_score_macro                       [0.7984255787742488]  ...   \n",
       "AUC_macro                                           [0.9319154315945073]  ...   \n",
       "AUC_weighted                                        [0.9319154315945072]  ...   \n",
       "recall_score_micro                                  [0.9080424886191198]  ...   \n",
       "recall_score_macro                                  [0.7196853923426452]  ...   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_32  \\\n",
       "precision_score_weighted                             [0.788565560086672]   \n",
       "average_precision_score_micro                       [0.9688737022895029]   \n",
       "average_precision_score_weighted                    [0.9294913358914507]   \n",
       "f1_score_macro                                      [0.4703423886834914]   \n",
       "log_loss                                           [0.25637830013935325]   \n",
       "matthews_correlation                                               [0.0]   \n",
       "precision_score_macro                               [0.4440060698027314]   \n",
       "f1_score_weighted                                   [0.8353395018439429]   \n",
       "balanced_accuracy                                                  [0.5]   \n",
       "recall_score_weighted                               [0.8880121396054628]   \n",
       "AUC_micro                                           [0.9673541324626221]   \n",
       "f1_score_micro                                      [0.8880121396054628]   \n",
       "norm_macro_recall                                                  [0.0]   \n",
       "precision_score_micro                               [0.8880121396054628]   \n",
       "weighted_accuracy                                   [0.9843450583187134]   \n",
       "accuracy                                            [0.8880121396054628]   \n",
       "average_precision_score_macro                       [0.7319347041742593]   \n",
       "AUC_macro                                           [0.8989176562989143]   \n",
       "AUC_weighted                                        [0.8989176562989144]   \n",
       "recall_score_micro                                  [0.8880121396054628]   \n",
       "recall_score_macro                                                 [0.5]   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_0  \\\n",
       "precision_score_weighted                           [0.9072720074188747]   \n",
       "average_precision_score_micro                      [0.9806603102489483]   \n",
       "average_precision_score_weighted                   [0.9531771295804466]   \n",
       "f1_score_macro                                     [0.7653697272147331]   \n",
       "log_loss                                          [0.17775706110025447]   \n",
       "matthews_correlation                               [0.5323740218566827]   \n",
       "precision_score_macro                              [0.7819118765348991]   \n",
       "f1_score_weighted                                  [0.9091539479147899]   \n",
       "balanced_accuracy                                  [0.7513392683482543]   \n",
       "recall_score_weighted                              [0.9116843702579667]   \n",
       "AUC_micro                                           [0.979695082216353]   \n",
       "f1_score_micro                                     [0.9116843702579667]   \n",
       "norm_macro_recall                                  [0.5026785366965085]   \n",
       "precision_score_micro                              [0.9116843702579667]   \n",
       "weighted_accuracy                                  [0.9514937218005303]   \n",
       "accuracy                                           [0.9116843702579667]   \n",
       "average_precision_score_macro                      [0.8151093723721079]   \n",
       "AUC_macro                                          [0.9450464668693166]   \n",
       "AUC_weighted                                       [0.9450464668693167]   \n",
       "recall_score_micro                                 [0.9116843702579667]   \n",
       "recall_score_macro                                 [0.7513392683482543]   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_14  \\\n",
       "precision_score_weighted                            [0.8842474652444061]   \n",
       "average_precision_score_micro                       [0.8193241774793902]   \n",
       "average_precision_score_weighted                    [0.9160128044475222]   \n",
       "f1_score_macro                                      [0.6250946364070289]   \n",
       "log_loss                                            [0.6093462584593584]   \n",
       "matthews_correlation                                [0.3482629755918812]   \n",
       "precision_score_macro                               [0.6193021454122402]   \n",
       "f1_score_weighted                                   [0.7947997493884662]   \n",
       "balanced_accuracy                                   [0.7541595118616942]   \n",
       "recall_score_weighted                               [0.7526555386949925]   \n",
       "AUC_micro                                           [0.8270461751722962]   \n",
       "f1_score_micro                                      [0.7526555386949924]   \n",
       "norm_macro_recall                                   [0.5083190237233883]   \n",
       "precision_score_micro                               [0.7526555386949925]   \n",
       "weighted_accuracy                                   [0.7522821428394862]   \n",
       "accuracy                                            [0.7526555386949925]   \n",
       "average_precision_score_macro                       [0.7045642649395899]   \n",
       "AUC_macro                                           [0.8467551917487732]   \n",
       "AUC_weighted                                        [0.8467551917487733]   \n",
       "recall_score_micro                                  [0.7526555386949925]   \n",
       "recall_score_macro                                  [0.7541595118616942]   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_16  \\\n",
       "precision_score_weighted                             [0.788565560086672]   \n",
       "average_precision_score_micro                       [0.9629364555606664]   \n",
       "average_precision_score_weighted                    [0.9224727710016692]   \n",
       "f1_score_macro                                      [0.4703423886834914]   \n",
       "log_loss                                            [0.2749583271733524]   \n",
       "matthews_correlation                                               [0.0]   \n",
       "precision_score_macro                               [0.4440060698027314]   \n",
       "f1_score_weighted                                   [0.8353395018439429]   \n",
       "balanced_accuracy                                                  [0.5]   \n",
       "recall_score_weighted                               [0.8880121396054628]   \n",
       "AUC_micro                                           [0.9640629914732627]   \n",
       "f1_score_micro                                      [0.8880121396054628]   \n",
       "norm_macro_recall                                                  [0.0]   \n",
       "precision_score_micro                               [0.8880121396054628]   \n",
       "weighted_accuracy                                   [0.9843450583187134]   \n",
       "accuracy                                            [0.8880121396054628]   \n",
       "average_precision_score_macro                       [0.7206954310055973]   \n",
       "AUC_macro                                           [0.8823703753100416]   \n",
       "AUC_weighted                                        [0.8823703753100416]   \n",
       "recall_score_micro                                  [0.8880121396054628]   \n",
       "recall_score_macro                                                 [0.5]   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_4  \\\n",
       "precision_score_weighted                           [0.9182355798069128]   \n",
       "average_precision_score_micro                      [0.8317277043239888]   \n",
       "average_precision_score_weighted                   [0.9370750303308232]   \n",
       "f1_score_macro                                     [0.7105954377506608]   \n",
       "log_loss                                          [0.46651601264873727]   \n",
       "matthews_correlation                               [0.5114480241575082]   \n",
       "precision_score_macro                              [0.6811761566448443]   \n",
       "f1_score_weighted                                   [0.848890702599696]   \n",
       "balanced_accuracy                                  [0.8609457864913577]   \n",
       "recall_score_weighted                              [0.8203338391502276]   \n",
       "AUC_micro                                          [0.8694183719757484]   \n",
       "f1_score_micro                                     [0.8203338391502276]   \n",
       "norm_macro_recall                                  [0.7218915729827153]   \n",
       "precision_score_micro                              [0.8203338391502276]   \n",
       "weighted_accuracy                                  [0.8102509911600128]   \n",
       "accuracy                                           [0.8203338391502276]   \n",
       "average_precision_score_macro                      [0.7554261028392691]   \n",
       "AUC_macro                                          [0.9198800771329655]   \n",
       "AUC_weighted                                       [0.9198800771329654]   \n",
       "recall_score_micro                                 [0.8203338391502276]   \n",
       "recall_score_macro                                 [0.8609457864913577]   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_5  \\\n",
       "precision_score_weighted                           [0.8826993014565436]   \n",
       "average_precision_score_micro                      [0.9681835493215114]   \n",
       "average_precision_score_weighted                   [0.9364668231804184]   \n",
       "f1_score_macro                                     [0.6616134730891533]   \n",
       "log_loss                                          [0.23145009573598296]   \n",
       "matthews_correlation                              [0.36945708081714224]   \n",
       "precision_score_macro                               [0.774300365133433]   \n",
       "f1_score_weighted                                  [0.8823695777855658]   \n",
       "balanced_accuracy                                  [0.6244060817231549]   \n",
       "recall_score_weighted                              [0.9007587253414264]   \n",
       "AUC_micro                                          [0.9698026853580978]   \n",
       "f1_score_micro                                     [0.9007587253414264]   \n",
       "norm_macro_recall                                 [0.24881216344630985]   \n",
       "precision_score_micro                              [0.9007587253414264]   \n",
       "weighted_accuracy                                  [0.9693696115393181]   \n",
       "accuracy                                           [0.9007587253414264]   \n",
       "average_precision_score_macro                      [0.7525938317909414]   \n",
       "AUC_macro                                           [0.920452461530767]   \n",
       "AUC_weighted                                        [0.920452461530767]   \n",
       "recall_score_micro                                 [0.9007587253414264]   \n",
       "recall_score_macro                                 [0.6244060817231549]   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_8  \\\n",
       "precision_score_weighted                           [0.8823379905566089]   \n",
       "average_precision_score_micro                      [0.9618011068804218]   \n",
       "average_precision_score_weighted                   [0.9256523104186373]   \n",
       "f1_score_macro                                     [0.6170656133421111]   \n",
       "log_loss                                           [0.2618000004386081]   \n",
       "matthews_correlation                               [0.3218684594572277]   \n",
       "precision_score_macro                              [0.8017560826299885]   \n",
       "f1_score_weighted                                  [0.8722950091956113]   \n",
       "balanced_accuracy                                  [0.5858303371140341]   \n",
       "recall_score_weighted                              [0.8995447647951441]   \n",
       "AUC_micro                                          [0.9612656321598227]   \n",
       "f1_score_micro                                     [0.8995447647951441]   \n",
       "norm_macro_recall                                  [0.1716606742280682]   \n",
       "precision_score_micro                              [0.8995447647951441]   \n",
       "weighted_accuracy                                  [0.9774315713566799]   \n",
       "accuracy                                           [0.8995447647951441]   \n",
       "average_precision_score_macro                      [0.7375304402096373]   \n",
       "AUC_macro                                          [0.8662885039650123]   \n",
       "AUC_weighted                                       [0.8662885039650123]   \n",
       "recall_score_micro                                 [0.8995447647951441]   \n",
       "recall_score_macro                                 [0.5858303371140341]   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_15  \\\n",
       "precision_score_weighted                             [0.918526645726982]   \n",
       "average_precision_score_micro                       [0.8704031509845976]   \n",
       "average_precision_score_weighted                    [0.9369762229683729]   \n",
       "f1_score_macro                                      [0.7200647870031351]   \n",
       "log_loss                                            [1.0528420699356784]   \n",
       "matthews_correlation                                 [0.520875607466786]   \n",
       "precision_score_macro                               [0.6873597929249353]   \n",
       "f1_score_weighted                                   [0.8566040394576435]   \n",
       "balanced_accuracy                                   [0.8620192387843222]   \n",
       "recall_score_weighted                               [0.8306525037936268]   \n",
       "AUC_micro                                           [0.8888591487999704]   \n",
       "f1_score_micro                                      [0.8306525037936268]   \n",
       "norm_macro_recall                                   [0.7240384775686444]   \n",
       "precision_score_micro                               [0.8306525037936268]   \n",
       "weighted_accuracy                                   [0.8228649919512622]   \n",
       "accuracy                                            [0.8306525037936268]   \n",
       "average_precision_score_macro                       [0.7558790923089123]   \n",
       "AUC_macro                                           [0.9211651634629812]   \n",
       "AUC_weighted                                        [0.9211651634629813]   \n",
       "recall_score_micro                                  [0.8306525037936268]   \n",
       "recall_score_macro                                  [0.8620192387843222]   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_20  \\\n",
       "precision_score_weighted                            [0.8781641870148528]   \n",
       "average_precision_score_micro                       [0.8054037059077446]   \n",
       "average_precision_score_weighted                     [0.909723441391043]   \n",
       "f1_score_macro                                      [0.6043570770314957]   \n",
       "log_loss                                            [0.5722209570104337]   \n",
       "matthews_correlation                               [0.31550798797377594]   \n",
       "precision_score_macro                               [0.6064710240250706]   \n",
       "f1_score_weighted                                   [0.7781380114280563]   \n",
       "balanced_accuracy                                   [0.7337379850216821]   \n",
       "recall_score_weighted                               [0.7311077389984826]   \n",
       "AUC_micro                                           [0.8269354634441757]   \n",
       "f1_score_micro                                      [0.7311077389984825]   \n",
       "norm_macro_recall                                  [0.46747597004336416]   \n",
       "precision_score_micro                               [0.7311077389984826]   \n",
       "weighted_accuracy                                   [0.7304547200578732]   \n",
       "accuracy                                            [0.7311077389984826]   \n",
       "average_precision_score_macro                       [0.7063419739896816]   \n",
       "AUC_macro                                           [0.8180188090329297]   \n",
       "AUC_weighted                                        [0.8180188090329298]   \n",
       "recall_score_micro                                  [0.7311077389984826]   \n",
       "recall_score_macro                                  [0.7337379850216821]   \n",
       "\n",
       "                                 c904e5fb-b0d2-472d-b0c0-dea0bc123c48_38  \n",
       "precision_score_weighted                             [0.909448627526437]  \n",
       "average_precision_score_micro                       [0.9797107523500044]  \n",
       "average_precision_score_weighted                    [0.9523248695577932]  \n",
       "f1_score_macro                                      [0.7677334161916184]  \n",
       "log_loss                                            [0.2388149173745338]  \n",
       "matthews_correlation                                [0.5404101114596196]  \n",
       "precision_score_macro                                [0.798907733703278]  \n",
       "f1_score_weighted                                   [0.9117102130976235]  \n",
       "balanced_accuracy                                   [0.7442585584434108]  \n",
       "recall_score_weighted                               [0.9159332321699545]  \n",
       "AUC_micro                                           [0.9788009146151915]  \n",
       "f1_score_micro                                      [0.9159332321699545]  \n",
       "norm_macro_recall                                  [0.48851711688682165]  \n",
       "precision_score_micro                               [0.9159332321699545]  \n",
       "weighted_accuracy                                   [0.9585554099349054]  \n",
       "accuracy                                            [0.9159332321699545]  \n",
       "average_precision_score_macro                       [0.8138331417012676]  \n",
       "AUC_macro                                           [0.9409666072053748]  \n",
       "AUC_weighted                                        [0.9409666072053747]  \n",
       "recall_score_micro                                  [0.9159332321699545]  \n",
       "recall_score_macro                                  [0.7442585584434108]  \n",
       "\n",
       "[21 rows x 38 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "with open(metrics_output._path_on_datastore) as f:\n",
    "    metrics_output_result = f.read()\n",
    "    \n",
    "deserialized_metrics_output = json.loads(metrics_output_result)\n",
    "df = pd.DataFrame(deserialized_metrics_output)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve the Best Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading azureml/c904e5fb-b0d2-472d-b0c0-dea0bc123c48/model_data\n",
      "Downloaded azureml/c904e5fb-b0d2-472d-b0c0-dea0bc123c48/model_data, 1 files out of an estimated total of 1\n"
     ]
    }
   ],
   "source": [
    "# Retrieve best model from Pipeline Run\n",
    "best_model_output = pipeline_run.get_pipeline_output(best_model_output_name)\n",
    "num_file_downloaded = best_model_output.download('.', show_progress=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PipelineWithYTransformations(Pipeline={'memory': None,\n",
       "                                       'steps': [('datatransformer',\n",
       "                                                  DataTransformer(enable_dnn=None,\n",
       "                                                                  enable_feature_sweeping=None,\n",
       "                                                                  feature_sweeping_config=None,\n",
       "                                                                  feature_sweeping_timeout=None,\n",
       "                                                                  featurization_config=None,\n",
       "                                                                  force_text_dnn=None,\n",
       "                                                                  is_cross_validation=None,\n",
       "                                                                  is_onnx_compatible=None,\n",
       "                                                                  logger=None,\n",
       "                                                                  observer=None,\n",
       "                                                                  task=None,\n",
       "                                                                  working_dir=None))...\n",
       "                                                                                                                                   min_weight_fraction_leaf=0.0,\n",
       "                                                                                                                                   n_estimators=50,\n",
       "                                                                                                                                   n_jobs=1,\n",
       "                                                                                                                                   oob_score=True,\n",
       "                                                                                                                                   random_state=None,\n",
       "                                                                                                                                   verbose=0,\n",
       "                                                                                                                                   warm_start=False))],\n",
       "                                                                                                      verbose=False))],\n",
       "                                                                                flatten_transform=None,\n",
       "                                                                                weights=[0.35714285714285715,\n",
       "                                                                                         0.07142857142857142,\n",
       "                                                                                         0.14285714285714285,\n",
       "                                                                                         0.07142857142857142,\n",
       "                                                                                         0.14285714285714285,\n",
       "                                                                                         0.07142857142857142,\n",
       "                                                                                         0.14285714285714285]))],\n",
       "                                       'verbose': False},\n",
       "                             y_transformer={},\n",
       "                             y_transformer_name='LabelEncoder')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pickle\n",
    "\n",
    "with open(best_model_output._path_on_datastore, \"rb\" ) as f:\n",
    "    best_model = pickle.load(f)\n",
    "best_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('datatransformer',\n",
       "  DataTransformer(enable_dnn=None, enable_feature_sweeping=None,\n",
       "                  feature_sweeping_config=None, feature_sweeping_timeout=None,\n",
       "                  featurization_config=None, force_text_dnn=None,\n",
       "                  is_cross_validation=None, is_onnx_compatible=None, logger=None,\n",
       "                  observer=None, task=None, working_dir=None)),\n",
       " ('prefittedsoftvotingclassifier',\n",
       "  PreFittedSoftVotingClassifier(classification_labels=None,\n",
       "                                estimators=[('0',\n",
       "                                             Pipeline(memory=None,\n",
       "                                                      steps=[('maxabsscaler',\n",
       "                                                              MaxAbsScaler(copy=True)),\n",
       "                                                             ('lightgbmclassifier',\n",
       "                                                              LightGBMClassifier(boosting_type='gbdt',\n",
       "                                                                                 class_weight=None,\n",
       "                                                                                 colsample_bytree=1.0,\n",
       "                                                                                 importance_type='split',\n",
       "                                                                                 learning_rate=0.1,\n",
       "                                                                                 max_depth=-1,\n",
       "                                                                                 min_child_samples=20,\n",
       "                                                                                 min_child_weight=0.001,\n",
       "                                                                                 min_spl...\n",
       "                                                                                   min_samples_leaf=0.01,\n",
       "                                                                                   min_samples_split=0.01,\n",
       "                                                                                   min_weight_fraction_leaf=0.0,\n",
       "                                                                                   n_estimators=50,\n",
       "                                                                                   n_jobs=1,\n",
       "                                                                                   oob_score=True,\n",
       "                                                                                   random_state=None,\n",
       "                                                                                   verbose=0,\n",
       "                                                                                   warm_start=False))],\n",
       "                                                      verbose=False))],\n",
       "                                flatten_transform=None,\n",
       "                                weights=[0.35714285714285715, 0.07142857142857142,\n",
       "                                         0.14285714285714285, 0.07142857142857142,\n",
       "                                         0.14285714285714285, 0.07142857142857142,\n",
       "                                         0.14285714285714285]))]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_model.steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "datatransformer\n",
      "{'enable_dnn': None,\n",
      " 'enable_feature_sweeping': None,\n",
      " 'feature_sweeping_config': None,\n",
      " 'feature_sweeping_timeout': None,\n",
      " 'featurization_config': None,\n",
      " 'force_text_dnn': None,\n",
      " 'is_cross_validation': None,\n",
      " 'is_onnx_compatible': None,\n",
      " 'logger': None,\n",
      " 'observer': None,\n",
      " 'task': None,\n",
      " 'working_dir': None}\n",
      "\n",
      "prefittedsoftvotingclassifier\n",
      "{'estimators': ['0', '26', '1', '27', '35', '21', '20'],\n",
      " 'weights': [0.35714285714285715,\n",
      "             0.07142857142857142,\n",
      "             0.14285714285714285,\n",
      "             0.07142857142857142,\n",
      "             0.14285714285714285,\n",
      "             0.07142857142857142,\n",
      "             0.14285714285714285]}\n",
      "\n",
      "0 - maxabsscaler\n",
      "{'copy': True}\n",
      "\n",
      "0 - lightgbmclassifier\n",
      "{'boosting_type': 'gbdt',\n",
      " 'class_weight': None,\n",
      " 'colsample_bytree': 1.0,\n",
      " 'importance_type': 'split',\n",
      " 'learning_rate': 0.1,\n",
      " 'max_depth': -1,\n",
      " 'min_child_samples': 20,\n",
      " 'min_child_weight': 0.001,\n",
      " 'min_split_gain': 0.0,\n",
      " 'n_estimators': 100,\n",
      " 'n_jobs': 1,\n",
      " 'num_leaves': 31,\n",
      " 'objective': None,\n",
      " 'random_state': None,\n",
      " 'reg_alpha': 0.0,\n",
      " 'reg_lambda': 0.0,\n",
      " 'silent': True,\n",
      " 'subsample': 1.0,\n",
      " 'subsample_for_bin': 200000,\n",
      " 'subsample_freq': 0,\n",
      " 'verbose': -10}\n",
      "\n",
      "26 - maxabsscaler\n",
      "{'copy': True}\n",
      "\n",
      "26 - lightgbmclassifier\n",
      "{'boosting_type': 'gbdt',\n",
      " 'class_weight': None,\n",
      " 'colsample_bytree': 0.99,\n",
      " 'importance_type': 'split',\n",
      " 'learning_rate': 0.04211105263157895,\n",
      " 'max_bin': 250,\n",
      " 'max_depth': 4,\n",
      " 'min_child_samples': 1023,\n",
      " 'min_child_weight': 10,\n",
      " 'min_split_gain': 0.894736842105263,\n",
      " 'n_estimators': 100,\n",
      " 'n_jobs': 1,\n",
      " 'num_leaves': 212,\n",
      " 'objective': None,\n",
      " 'random_state': None,\n",
      " 'reg_alpha': 0.7894736842105263,\n",
      " 'reg_lambda': 0.631578947368421,\n",
      " 'silent': True,\n",
      " 'subsample': 0.8415789473684211,\n",
      " 'subsample_for_bin': 200000,\n",
      " 'subsample_freq': 0,\n",
      " 'verbose': -10}\n",
      "\n",
      "1 - maxabsscaler\n",
      "{'copy': True}\n",
      "\n",
      "1 - xgboostclassifier\n",
      "{'base_score': 0.5,\n",
      " 'booster': 'gbtree',\n",
      " 'colsample_bylevel': 1,\n",
      " 'colsample_bynode': 1,\n",
      " 'colsample_bytree': 1,\n",
      " 'gamma': 0,\n",
      " 'learning_rate': 0.1,\n",
      " 'max_delta_step': 0,\n",
      " 'max_depth': 3,\n",
      " 'min_child_weight': 1,\n",
      " 'missing': nan,\n",
      " 'n_estimators': 100,\n",
      " 'n_jobs': 1,\n",
      " 'nthread': None,\n",
      " 'objective': 'binary:logistic',\n",
      " 'random_state': 0,\n",
      " 'reg_alpha': 0,\n",
      " 'reg_lambda': 1,\n",
      " 'scale_pos_weight': 1,\n",
      " 'seed': None,\n",
      " 'silent': None,\n",
      " 'subsample': 1,\n",
      " 'tree_method': 'auto',\n",
      " 'verbose': -10,\n",
      " 'verbosity': 0}\n",
      "\n",
      "27 - sparsenormalizer\n",
      "{'copy': True, 'norm': 'max'}\n",
      "\n",
      "27 - lightgbmclassifier\n",
      "{'boosting_type': 'goss',\n",
      " 'class_weight': None,\n",
      " 'colsample_bytree': 0.99,\n",
      " 'importance_type': 'split',\n",
      " 'learning_rate': 0.08947473684210526,\n",
      " 'max_bin': 320,\n",
      " 'max_depth': 8,\n",
      " 'min_child_samples': 1330,\n",
      " 'min_child_weight': 2,\n",
      " 'min_split_gain': 0.9473684210526315,\n",
      " 'n_estimators': 25,\n",
      " 'n_jobs': 1,\n",
      " 'num_leaves': 44,\n",
      " 'objective': None,\n",
      " 'random_state': None,\n",
      " 'reg_alpha': 0.05263157894736842,\n",
      " 'reg_lambda': 0.5789473684210527,\n",
      " 'silent': True,\n",
      " 'subsample': 1,\n",
      " 'subsample_for_bin': 200000,\n",
      " 'subsample_freq': 0,\n",
      " 'verbose': -10}\n",
      "\n",
      "35 - standardscalerwrapper\n",
      "{'class_name': 'StandardScaler',\n",
      " 'copy': True,\n",
      " 'module_name': 'sklearn.preprocessing._data',\n",
      " 'with_mean': False,\n",
      " 'with_std': True}\n",
      "\n",
      "35 - lightgbmclassifier\n",
      "{'boosting_type': 'gbdt',\n",
      " 'class_weight': None,\n",
      " 'colsample_bytree': 0.3966666666666666,\n",
      " 'importance_type': 'split',\n",
      " 'learning_rate': 0.04737368421052632,\n",
      " 'max_bin': 270,\n",
      " 'max_depth': 3,\n",
      " 'min_child_samples': 2557,\n",
      " 'min_child_weight': 5,\n",
      " 'min_split_gain': 0.7894736842105263,\n",
      " 'n_estimators': 100,\n",
      " 'n_jobs': 1,\n",
      " 'num_leaves': 242,\n",
      " 'objective': None,\n",
      " 'random_state': None,\n",
      " 'reg_alpha': 0.7894736842105263,\n",
      " 'reg_lambda': 0.631578947368421,\n",
      " 'silent': True,\n",
      " 'subsample': 0.6931578947368422,\n",
      " 'subsample_for_bin': 200000,\n",
      " 'subsample_freq': 0,\n",
      " 'verbose': -10}\n",
      "\n",
      "21 - maxabsscaler\n",
      "{'copy': True}\n",
      "\n",
      "21 - randomforestclassifier\n",
      "{'bootstrap': True,\n",
      " 'ccp_alpha': 0.0,\n",
      " 'class_weight': 'balanced',\n",
      " 'criterion': 'entropy',\n",
      " 'max_depth': None,\n",
      " 'max_features': 'sqrt',\n",
      " 'max_leaf_nodes': None,\n",
      " 'max_samples': None,\n",
      " 'min_impurity_decrease': 0.0,\n",
      " 'min_impurity_split': None,\n",
      " 'min_samples_leaf': 0.035789473684210524,\n",
      " 'min_samples_split': 0.01,\n",
      " 'min_weight_fraction_leaf': 0.0,\n",
      " 'n_estimators': 10,\n",
      " 'n_jobs': 1,\n",
      " 'oob_score': True,\n",
      " 'random_state': None,\n",
      " 'verbose': 0,\n",
      " 'warm_start': False}\n",
      "\n",
      "20 - maxabsscaler\n",
      "{'copy': True}\n",
      "\n",
      "20 - extratreesclassifier\n",
      "{'bootstrap': True,\n",
      " 'ccp_alpha': 0.0,\n",
      " 'class_weight': 'balanced',\n",
      " 'criterion': 'gini',\n",
      " 'max_depth': None,\n",
      " 'max_features': 'log2',\n",
      " 'max_leaf_nodes': None,\n",
      " 'max_samples': None,\n",
      " 'min_impurity_decrease': 0.0,\n",
      " 'min_impurity_split': None,\n",
      " 'min_samples_leaf': 0.01,\n",
      " 'min_samples_split': 0.01,\n",
      " 'min_weight_fraction_leaf': 0.0,\n",
      " 'n_estimators': 50,\n",
      " 'n_jobs': 1,\n",
      " 'oob_score': True,\n",
      " 'random_state': None,\n",
      " 'verbose': 0,\n",
      " 'warm_start': False}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Code below is used to gather parameters of the fitted model.\n",
    "# Source: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-features#scaling-and-normalization \n",
    "from pprint import pprint\n",
    "\n",
    "def print_model(model, prefix=\"\"):\n",
    "    for step in model.steps:\n",
    "        print(prefix + step[0])\n",
    "        if hasattr(step[1], 'estimators') and hasattr(step[1], 'weights'):\n",
    "            pprint({'estimators': list(\n",
    "                e[0] for e in step[1].estimators), 'weights': step[1].weights})\n",
    "            print()\n",
    "            for estimator in step[1].estimators:\n",
    "                print_model(estimator[1], estimator[0] + ' - ')\n",
    "        else:\n",
    "            pprint(step[1].get_params())\n",
    "            print()\n",
    "\n",
    "print_model(best_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the Model\n",
    "#### Load Test Data\n",
    "For the test data, it should have the same preparation step as the train data. Otherwise it might get failed at the preprocessing step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "#from azureml.data.dataset_factory import TabularDatasetFactory\n",
    "import pandas as pd\n",
    "df_test = pd.read_csv('./bankmarketing_train.csv')\n",
    "#dataset_test=ws.datasets.get('Bank-marketing')\n",
    "#df_test = dataset_test.to_pandas_dataframe()\n",
    "df_test = df_test[pd.notnull(df_test['y'])]\n",
    "\n",
    "y_test = df_test['y']\n",
    "X_test = df_test.drop(['y'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing Our Best Fitted Model\n",
    "\n",
    "We will use confusion matrix to see how our model works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "ypred = best_model.predict(X_test)\n",
    "cm = confusion_matrix(y_test, ypred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_e0bb4c74_31ac_11eb_a337_000d3aeeb0e6row0_col0 {\n",
       "            background-color:  #63a8d3;\n",
       "            color:  #000000;\n",
       "        }    #T_e0bb4c74_31ac_11eb_a337_000d3aeeb0e6row0_col1 {\n",
       "            background-color:  #f7fbff;\n",
       "            color:  #000000;\n",
       "        }    #T_e0bb4c74_31ac_11eb_a337_000d3aeeb0e6row1_col0 {\n",
       "            background-color:  #f7fbff;\n",
       "            color:  #000000;\n",
       "        }    #T_e0bb4c74_31ac_11eb_a337_000d3aeeb0e6row1_col1 {\n",
       "            background-color:  #63a8d3;\n",
       "            color:  #000000;\n",
       "        }</style><table id=\"T_e0bb4c74_31ac_11eb_a337_000d3aeeb0e6\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >0</th>        <th class=\"col_heading level0 col1\" >1</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_e0bb4c74_31ac_11eb_a337_000d3aeeb0e6level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "                        <td id=\"T_e0bb4c74_31ac_11eb_a337_000d3aeeb0e6row0_col0\" class=\"data row0 col0\" >28392</td>\n",
       "                        <td id=\"T_e0bb4c74_31ac_11eb_a337_000d3aeeb0e6row0_col1\" class=\"data row0 col1\" >866</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_e0bb4c74_31ac_11eb_a337_000d3aeeb0e6level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "                        <td id=\"T_e0bb4c74_31ac_11eb_a337_000d3aeeb0e6row1_col0\" class=\"data row1 col0\" >1531</td>\n",
       "                        <td id=\"T_e0bb4c74_31ac_11eb_a337_000d3aeeb0e6row1_col1\" class=\"data row1 col1\" >2161</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f31b2abd198>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Visualize the confusion matrix\n",
    "pd.DataFrame(cm).style.background_gradient(cmap='Blues', low=0, high=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Publish and run from REST endpoint\n",
    "\n",
    "Run the following code to publish the pipeline to your workspace. In your workspace in the portal, you can see metadata for the pipeline including run history and durations. You can also run the pipeline manually from the portal.\n",
    "\n",
    "Additionally, publishing the pipeline enables a REST endpoint to rerun the pipeline from any HTTP library on any platform.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"width:100%\"><tr><th>Name</th><th>Id</th><th>Status</th><th>Endpoint</th></tr><tr><td>Bankmarketing Train</td><td><a href=\"https://ml.azure.com/pipelines/656d5c65-ad70-4830-82fe-d23e27f1a9df?wsid=/subscriptions/dafd62fe-51d3-4671-af50-6bac2f001b5d/resourcegroups/aml-quickstarts-128228/workspaces/quick-starts-ws-128228\" target=\"_blank\" rel=\"noopener\">656d5c65-ad70-4830-82fe-d23e27f1a9df</a></td><td>Active</td><td><a href=\"https://southcentralus.api.azureml.ms/pipelines/v1.0/subscriptions/dafd62fe-51d3-4671-af50-6bac2f001b5d/resourceGroups/aml-quickstarts-128228/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-128228/PipelineRuns/PipelineSubmit/656d5c65-ad70-4830-82fe-d23e27f1a9df\" target=\"_blank\" rel=\"noopener\">REST Endpoint</a></td></tr></table>"
      ],
      "text/plain": [
       "Pipeline(Name: Bankmarketing Train,\n",
       "Id: 656d5c65-ad70-4830-82fe-d23e27f1a9df,\n",
       "Status: Active,\n",
       "Endpoint: https://southcentralus.api.azureml.ms/pipelines/v1.0/subscriptions/dafd62fe-51d3-4671-af50-6bac2f001b5d/resourceGroups/aml-quickstarts-128228/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-128228/PipelineRuns/PipelineSubmit/656d5c65-ad70-4830-82fe-d23e27f1a9df)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "published_pipeline = pipeline_run.publish_pipeline(\n",
    "    name=\"Bankmarketing Train\", description=\"Training bankmarketing pipeline\", version=\"1.0\")\n",
    "\n",
    "published_pipeline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Authenticate once again, to retrieve the `auth_header` so that the endpoint can be used"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.authentication import InteractiveLoginAuthentication\n",
    "\n",
    "interactive_auth = InteractiveLoginAuthentication()\n",
    "auth_header = interactive_auth.get_authentication_header()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the REST url from the endpoint property of the published pipeline object. You can also find the REST url in your workspace in the portal. Build an HTTP POST request to the endpoint, specifying your authentication header. Additionally, add a JSON payload object with the experiment name and the batch size parameter. As a reminder, the process_count_per_node is passed through to ParallelRunStep because you defined it is defined as a PipelineParameter object in the step configuration.\n",
    "\n",
    "Make the request to trigger the run. Access the Id key from the response dict to get the value of the run id.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "rest_endpoint = published_pipeline.endpoint\n",
    "response = requests.post(rest_endpoint, \n",
    "                         headers=auth_header, \n",
    "                         json={\"ExperimentName\": \"pipeline-rest-endpoint\"}\n",
    "                        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submitted pipeline run:  43e128ba-8cc8-4fbb-a221-8097f86c7ce3\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    response.raise_for_status()\n",
    "except Exception:    \n",
    "    raise Exception(\"Received bad response from the endpoint: {}\\n\"\n",
    "                    \"Response Code: {}\\n\"\n",
    "                    \"Headers: {}\\n\"\n",
    "                    \"Content: {}\".format(rest_endpoint, response.status_code, response.headers, response.content))\n",
    "\n",
    "run_id = response.json().get('Id')\n",
    "print('Submitted pipeline run: ', run_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the run id to monitor the status of the new run. This will take another 10-15 min to run and will look similar to the previous pipeline run, so if you don't need to see another pipeline run, you can skip watching the full output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "85a7d9057632410caebbfaff32be7546",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_PipelineWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/aml.mini.widget.v1": "{\"loading\": true}"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from azureml.pipeline.core.run import PipelineRun\n",
    "from azureml.widgets import RunDetails\n",
    "\n",
    "published_pipeline_run = PipelineRun(ws.experiments[\"pipeline-rest-endpoint\"], run_id)\n",
    "RunDetails(published_pipeline_run).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "sanpil"
   }
  ],
  "categories": [
   "how-to-use-azureml",
   "machine-learning-pipelines",
   "intro-to-pipelines"
  ],
  "category": "tutorial",
  "compute": [
   "AML Compute"
  ],
  "datasets": [
   "Custom"
  ],
  "deployment": [
   "None"
  ],
  "exclude_from_index": false,
  "framework": [
   "Automated Machine Learning"
  ],
  "friendly_name": "How to use AutoMLStep with AML Pipelines",
  "kernel_info": {
   "name": "python3"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  },
  "order_index": 11,
  "star_tag": [
   "featured"
  ],
  "tags": [
   "None"
  ],
  "task": "Demonstrates the use of AutoMLStep"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}