1-training-and-evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mask Detection Demo - Training and Evaluation (1 / 3)\n",
    "\n",
    "The following example demonstrates a training workflow that builds and trains a model that classifies whether a person is wearing a mask or not. The training is auto-logged to both Tensorboard and MLRun, and easily distributed using Horovod. Post training, we will run an evaluation to check our model's performance, updating the logging as a part of a routine test.\n",
    "\n",
    "1. [Setup the Project](#section_1)\n",
    "2. [Download the Data](#section_2)\n",
    "3. [Write the Training and Evaluation Code](#section_3)\n",
    "4. [Create the MLRun Function](#section_4)\n",
    "5. [Run Training and Evaluation](#section_5)\n",
    "6. [Run Distributed Training Using Horovod](#section_6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we continue, we need to install MLRun and the framework of choice (comment and uncomment the framework you wish to use):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install mlrun typing-extensions\n",
    "\n",
    "########## For TF.Keras: ##########\n",
    "!pip install mlrun tensorflow~=2.9.0\n",
    "\n",
    "######### For PyTorch:  ##########\n",
    "# !pip install mlrun torch~=1.13 torchvision~=0.14"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"section_1\"></a>\n",
    "## 1. Setup the Project\n",
    "\n",
    "Create a project using `mlrun.get_or_create_project` (make sure to load it in case it already exists), creating the paths where we'll store the project's artifacts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "import mlrun\n",
    "import os\n",
    "\n",
    "# Set our project's name:\n",
    "project_name = \"mask-detection\"\n",
    "\n",
    "# Create the project:\n",
    "project = mlrun.get_or_create_project(name=project_name, context=\"./\", user_project=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A project in MLRun is based on the MLRun Functions it can run. In this notebook we will see two ways to create a MLRun Function:\n",
    "* `mlrun.code_to_function`: Create our own MLRun Function from code (will be used for training and evaluation in [section 4](#section_4)).\n",
    "* `mlrun.import_function`: Import from [MLRun's functions marketplace](https://docs.mlrun.org/en/latest/load-from-marketplace.html) - a functions hub intended to be a centralized location for open source contributions of function components (will be used for downloading the data in [section 2](#section_2)).\n",
    " \n",
    "Before we continue, **please select the desired framework** (comment and uncomment the below lines as needed):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "framework = \"tf-keras\"\n",
    "# framework = \"pytorch\"\n",
    "\n",
    "# If you wish to train on gpu, set this variable to 'True', otherwise 'False':\n",
    "use_gpu = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Build project image\n",
    "Building the image to satisfy the requirements of our project, according to the selection above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if framework=='tf-keras':\n",
    "    commands = ['pip install mlrun tensorflow~=2.9.0',\n",
    "                'pip install mlrun horovod==0.25.0']\n",
    "    builder_env = {'HOROVOD_WITH_MPI':'1', 'HOROVOD_WITH_TENSORFLOW':'1'}\n",
    "else:\n",
    "    commands = ['pip install mlrun torch==1.13.0+cpu torchvision==0.14.0+cpu -f https://download.pytorch.org/whl/torch_stable.html',\n",
    "                'pip install mlrun tensorboard==2.9.1',\n",
    "                'pip install mlrun onnx~=1.15.0',\n",
    "                'pip install mlrun horovod==0.25.0']\n",
    "    builder_env={'HOROVOD_WITH_MPI':'1',\n",
    "                 'HOROVOD_WITH_PYTORCH': '1'}\n",
    "\n",
    "if not use_gpu: # install horovod dependency (already installed in mlrun/mlrun-gpu image)\n",
    "    commands = ['apt update -qqq --fix-missing \\\n",
    "            && apt upgrade -y \\\n",
    "            && apt install -y \\\n",
    "            build-essential \\\n",
    "            cmake \\\n",
    "            gcc \\\n",
    "            && apt clean \\\n",
    "            && rm -rf /var/lib/apt/lists/*',\n",
    "            ] + commands\n",
    "    builder_env['HOROVOD_WITHOUT_GLOO'] = '1'\n",
    "    \n",
    "img = project.build_image(image=f'.mask-detection-{framework}',\n",
    "                          base_image='mlrun/mlrun-gpu' if use_gpu else 'mlrun/mlrun',\n",
    "                          commands=commands,\n",
    "                          builder_env=builder_env,\n",
    "                          overwrite_build_params=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"section_2\"></a>\n",
    "## 2. Download the Data\n",
    "\n",
    "### 2.1. Import a Function\n",
    "\n",
    "We will download the images using `open_archive` - a function from MLRun's functions marketplace. We will import the function using `mlrun.import_function` and describe it to get the function's documentation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function: open-archive\n",
      "Open a file/object archive into a target directory\n",
      "default handler: open_archive\n",
      "entry points:\n",
      "  open_archive: Open a file/object archive into a target directory\n",
      "Currently supports zip and tar.gz\n",
      "    context(MLClientCtx)  - function execution context, default=\n",
      "    archive_url(DataItem)  - url of archive file , default=\n",
      "    subdir(str)  - path within artifact store where extracted files are stored, default=content/\n",
      "    key(str)  - key of archive contents in artifact store, default=content\n",
      "    target_path(str)  - file system path to store extracted files, default=None\n"
     ]
    }
   ],
   "source": [
    "# Import the function:\n",
    "open_archive_function = mlrun.import_function(\"hub://open_archive\")\n",
    "\n",
    "# Print the function's documentation:\n",
    "open_archive_function.doc()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2. Run the Function - Download the Images\n",
    "\n",
    "* **Function handlers**: We'll download the images by running the function using the `open_archive` handler as noted in the function's documentation. MLRun function is a collection of code and the handlers are the function headers inside. Every function with a context (type: `mlrun.MLClientCtx`) can be used as a handler.\n",
    "* **Passing parameters**: MLRun function expects two types of parameters: inputs (type: `mlrun.DataItem`) and parameters. As noted in the function's documentation, we can see the `archive_url` is an `mlrun.DataItem`, so it should be passed in the `inputs` attribute of the `run` function. The others are passed via the `parameters` attribute.\n",
    "* We will use the `local` argument and pass it as `True`. That means we will run the function locally and not on a pod. Using `local` is a convenient way to debug the code.\n",
    "\n",
    "For more information regarding MLRun functions, context and data items, refer to [MLRun's documentation](https://docs.mlrun.org/en/latest/index.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-09-05 10:21:14,209 [info] Storing function: {'name': 'download-data', 'uid': 'af37dcded51a455ea43c8a2ef0941936', 'db': 'http://mlrun-api:8080'}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Names with underscore '_' are about to be deprecated, use dashes '-' instead. Replacing underscores with dashes.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-09-05 10:21:14,387 [info] downloading https://s3.wasabisys.com/iguazio/data/prajnasb-generated-mask-detection/prajnasb_generated_mask_detection.zip to local temp file\n",
      "> 2023-09-05 10:21:30,074 [info] Logging artifact to /User/test/demos/mask-detection/data/\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>mask-detection-iguazio</td>\n",
       "      <td><div title=\"af37dcded51a455ea43c8a2ef0941936\"><a href=\"https://dashboard.default-tenant.app.vmdev94.lab.iguazeng.com/mlprojects/mask-detection-iguazio/jobs/monitor/af37dcded51a455ea43c8a2ef0941936/overview\" target=\"_blank\" >...f0941936</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Sep 05 10:21:14</td>\n",
       "      <td>completed</td>\n",
       "      <td>download-data</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=iguazio</div><div class=\"dictlist\">kind=</div><div class=\"dictlist\">owner=iguazio</div><div class=\"dictlist\">host=jupyter-6f96d6d666-q5zdg</div></td>\n",
       "      <td><div title=\"https://s3.wasabisys.com/iguazio/data/prajnasb-generated-mask-detection/prajnasb_generated_mask_detection.zip\">archive_url</div></td>\n",
       "      <td><div class=\"dictlist\">target_path=/User/test/demos/mask-detection/data/</div></td>\n",
       "      <td></td>\n",
       "      <td><div title=\"/User/test/demos/mask-detection/data\">content</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result8eae72b8-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result8eae72b8-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result8eae72b8\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result8eae72b8-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.vmdev94.lab.iguazeng.com/mlprojects/mask-detection-iguazio/jobs/monitor/af37dcded51a455ea43c8a2ef0941936/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-09-05 10:21:30,271 [info] Run execution finished: {'status': 'completed', 'name': 'download-data'}\n"
     ]
    }
   ],
   "source": [
    "# Setup the archive url for downloading the dataset images:\n",
    "archive_url = mlrun.get_sample_path(\"data/prajnasb-generated-mask-detection/prajnasb_generated_mask_detection.zip\")\n",
    "\n",
    "# Set the path to download the images data to:\n",
    "# Make sure environment variable `PWD` points to a valid accessible path e.g. `/v3io/projects/mask-detection/`\n",
    "dataset_path = os.environ.get('PWD', None)+'/data/'\n",
    "\n",
    "# Run the function using the 'open_archive' handler:\n",
    "open_archive_run = open_archive_function.run(\n",
    "    name='download_data',\n",
    "    handler='open_archive',\n",
    "    inputs={'archive_url': archive_url},\n",
    "    params={'target_path': dataset_path},\n",
    "    local=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"section_3\"></a>\n",
    "## 3. Write the Training and Evaluation Code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TF.Keras\n",
    "\n",
    "The code is taken from the python file [training-and-evaluation.py](tf-keras/training-and-evaluation.py), which is classic and straightforward. We: \n",
    "1. Use `_get_datasets` to get the training and validation datasets (on evaluation - the evaluation dataset).\n",
    "2. Use `_get_model` to build our classifier - simple transfer learning from MobileNetV2 (`keras.applications`).\n",
    "3. Call `train` to train the model.\n",
    "4. Call `evaluate` to evaluate the model.\n",
    "\n",
    "Taking this code one step further is **MLRun**'s framework for `tf.keras`:\n",
    "\n",
    "```python\n",
    "# Apply MLRun's interface for tf.keras:\n",
    "mlrun_tf_keras.apply_mlrun(model=model, context=context, ...)\n",
    "```\n",
    "\n",
    "With just one line of code, it seamlessly provides **automatic logging** (for both MLRun and Tensorboard) and **distributed training** by wrapping the `fit` and `evaluate` methods of `tf.keras.Model`.\n",
    "\n",
    "In addition, there is the `TFKerasModelHandler` class that is being returned from `apply_mlrun`. This class supports loading, saving and logging `tf.keras` models with ease, enabling easy versioning of the model and his results, artifacts and custom objects. Pay attention that by default, the model is loaded and logged automatically by `apply_mlrun`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PyTorch\n",
    "\n",
    "The code is taken from the python file [training-and-evaluation.py](pytorch/training-and-evaluation.py), which is classic and straightforward. We:\n",
    "1. Use `_get_datasets` to get the training and validation datasets (on evaluation - the evaluation dataset). The function is initiazliing a `MaskDetectionDataset` to handle our images.\n",
    "2. Initialize our `MaskDetector` classifier class - a simple transfer learning from MobileNetV2 (`torchvision.models`).\n",
    "3. Call `train` to train the model.\n",
    "4. Call `evaluate` to evaluate the model.\n",
    "\n",
    "Taking this code one step further is **MLRun**'s framework for `torch`:\n",
    "\n",
    "```python\n",
    "import mlrun.frameworks.pytorch as mlrun_torch\n",
    "```\n",
    "\n",
    "`mlrun_torch` is providing what we call \"shortcut functions\" for using PyTorch with ease:\n",
    "* `train` - Training a model.\n",
    "* `evaluate` - Evaluating a model.\n",
    "\n",
    "Both functions enable **automatic logging** (for both MLRun and Tensorboard) and **distributed training** by simply passing the following parameters: `auto_log: bool` and `use_horovod: bool`.\n",
    "\n",
    "In addition, you can choose to use our classes directly:\n",
    "* `PyTorchMLRunInterface` - the interface for training, evaluating and predicting a PyTorch model. Our code is highly generic and should fit for any type of model.\n",
    "* If you wish to use your own training code, to get automatic logging you will simply need to use our callback mechanism with `CallbackHandler`.\n",
    "* `PyTorchModelHandler` - supports loading, saving and logging `torch` models with ease, enabling easy versioning of the model and his results, artifacts and custom objects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both **TF.Keras** and **PyTorch** has the same features regarding MLRun's automatic logging and distributed training orchastration:\n",
    "* **Automatic logging**: auto-log your training and model to both **Tensorboard** and **MLRun**. Additional settings can be passed onto this method to gain extra logging capabilities, like:\n",
    "  * Weights histograms and distributions\n",
    "  * Weights statistics\n",
    "  * Weights images (working in progress)\n",
    "  * Edit static and dynamic hyperparameters tracking\n",
    "  * Logging frequency and more\n",
    "* **Distributed training with Horovod**: Horovod will be initialized and used automatically if the MLRun Function's `kind` attribute is equal to `\"mpijob\"`, there won't be any additional changes needed to the original code! More on that later in [section 6](#section_6)\n",
    "\n",
    "We suggest reading the documentation for further use, or like in this example, use the default settings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"section_4\"></a>\n",
    "## 4. Create the MLRun Function\n",
    "\n",
    "We will use MLRun's `mlrun.code_to_function` to create a MLRun Function from our code in the above mentioned python file. Notice our MLRun Function will have two handlers: `train` and `evaluate`.\n",
    "\n",
    "We wish to run the training first as a Job, so we will set the `kind` parameter to `\"job\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the function parsing the given file code using 'code_to_function':\n",
    "training_and_evaluation_function = mlrun.code_to_function(\n",
    "    filename=os.path.join(framework, \"training-and-evaluation.py\"),\n",
    "    name=\"training-and-evaluation\",\n",
    "    kind=\"job\",\n",
    "    image=f'.mask-detection-{framework}'\n",
    ")\n",
    "\n",
    "# Mount it:\n",
    "training_and_evaluation_function.apply(mlrun.auto_mount())\n",
    "if os.getenv('V3IO_ACCESS_KEY','False')=='False':\n",
    "    training_and_evaluation_function.spec.disable_auto_mount=False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"section_5\"></a>\n",
    "## 5. Run Training and Evaluation\n",
    "\n",
    "### 5.1. Train the Model\n",
    "\n",
    "We will run the training using the `train` handler. We will pass the desired hyperparameters and keep the returning run object in order to pass the trained model to the evaluation (more on that can be seen later on). \n",
    "\n",
    "Unlike running the `open_archive` function, the training will be performed as a **job** on the **cluster** and not locally. \n",
    "\n",
    "> **Notice** now the `local` attribute is `False` (this is its default value) which means it will run the function on the cluster. To run the training locally, simply pass `local=True` as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-09-05 10:21:30,471 [info] Storing function: {'name': 'training', 'uid': 'e80c00ef3e69420fbb2e29a17fa053de', 'db': 'http://mlrun-api:8080'}\n",
      "> 2023-09-05 10:21:30,780 [info] Job is running in the background, pod: training-5stfr\n",
      "2023-09-05 10:22:25.921701: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
      "2023-09-05 10:22:25.921744: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "2023-09-05 10:22:28.127332: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory\n",
      "2023-09-05 10:22:28.127398: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "2023-09-05 10:22:28.127430: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (training-5stfr): /proc/driver/nvidia/version does not exist\n",
      "WARNING:tensorflow:`input_shape` is undefined or non-square, or `rows` is not in [96, 128, 160, 192, 224]. Weights for input shape (224, 224) will be loaded as the default.\n",
      "2023-09-05 10:22:38.289811: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/mobilenet_v2/mobilenet_v2_weights_tf_dim_ordering_tf_kernels_1.0_224_no_top.h5\n",
      "9406464/9406464 [==============================] - 0s 0us/step\n",
      "The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "Epoch 1/3\n",
      "35/35 [==============================] - 58s 2s/step - loss: 0.5089 - accuracy: 0.7700 - val_loss: 0.1534 - val_accuracy: 0.9565 - lr: 1.0000e-04\n",
      "Epoch 2/3\n",
      "35/35 [==============================] - 54s 2s/step - loss: 0.1461 - accuracy: 0.9527 - val_loss: 0.0633 - val_accuracy: 0.9891 - lr: 1.0000e-04\n",
      "Epoch 3/3\n",
      "35/35 [==============================] - 54s 2s/step - loss: 0.0995 - accuracy: 0.9718 - val_loss: 0.0390 - val_accuracy: 0.9928 - lr: 1.0000e-04\n",
      "WARNING:absl:Found untraced functions such as _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op while saving (showing 5 of 52). These functions will not be directly callable after loading.\n",
      "> 2023-09-05 10:25:45,109 [info] Run execution finished: {'status': 'completed', 'name': 'training'}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>mask-detection-iguazio</td>\n",
       "      <td><div title=\"e80c00ef3e69420fbb2e29a17fa053de\"><a href=\"https://dashboard.default-tenant.app.vmdev94.lab.iguazeng.com/mlprojects/mask-detection-iguazio/jobs/monitor/e80c00ef3e69420fbb2e29a17fa053de/overview\" target=\"_blank\" >...7fa053de</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Sep 05 10:22:25</td>\n",
       "      <td>completed</td>\n",
       "      <td>training</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=iguazio</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=iguazio</div><div class=\"dictlist\">mlrun/client_version=1.5.0-rc9</div><div class=\"dictlist\">mlrun/client_python_version=3.9.16</div><div class=\"dictlist\">host=training-5stfr</div></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">dataset_path=/User/test/demos/mask-detection/data/</div><div class=\"dictlist\">batch_size=32</div><div class=\"dictlist\">lr=0.0001</div><div class=\"dictlist\">epochs=3</div></td>\n",
       "      <td><div class=\"dictlist\">dataset_path=/User/test/demos/mask-detection/data/</div><div class=\"dictlist\">batch_size=32</div><div class=\"dictlist\">epochs=3</div><div class=\"dictlist\">lr=9.999999747378752e-05</div><div class=\"dictlist\">training_loss=0.05770158767700195</div><div class=\"dictlist\">training_accuracy=0.9686927795410156</div><div class=\"dictlist\">validation_loss=0.039046022627088756</div><div class=\"dictlist\">validation_accuracy=0.9927536646525065</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulteada4ed1\" title=\"files/v3io/projects/mask-detection-iguazio/artifacts/training/0/training_loss.html.html\">training_loss.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulteada4ed1\" title=\"files/v3io/projects/mask-detection-iguazio/artifacts/training/0/training_accuracy.html.html\">training_accuracy.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulteada4ed1\" title=\"files/v3io/projects/mask-detection-iguazio/artifacts/training/0/validation_loss.html.html\">validation_loss.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulteada4ed1\" title=\"files/v3io/projects/mask-detection-iguazio/artifacts/training/0/validation_accuracy.html.html\">validation_accuracy.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulteada4ed1\" title=\"files/v3io/projects/mask-detection-iguazio/artifacts/training/0/loss_summary.html.html\">loss_summary.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulteada4ed1\" title=\"files/v3io/projects/mask-detection-iguazio/artifacts/training/0/accuracy_summary.html.html\">accuracy_summary.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulteada4ed1\" title=\"files/v3io/projects/mask-detection-iguazio/artifacts/training/0/lr_values.html.html\">lr_values.html</div><div title=\"v3io:///projects/mask-detection-iguazio/artifacts/training/0/model/\">model</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resulteada4ed1-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resulteada4ed1-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resulteada4ed1\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resulteada4ed1-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.vmdev94.lab.iguazeng.com/mlprojects/mask-detection-iguazio/jobs/monitor/e80c00ef3e69420fbb2e29a17fa053de/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-09-05 10:25:46,645 [info] Run execution finished: {'status': 'completed', 'name': 'training'}\n"
     ]
    }
   ],
   "source": [
    "training_run = training_and_evaluation_function.run(\n",
    "    name=\"training\",\n",
    "    handler=\"train\",\n",
    "    params={\n",
    "        \"dataset_path\": dataset_path,\n",
    "        \"batch_size\": 32,\n",
    "        \"lr\": 1e-4,\n",
    "        \"epochs\": 3\n",
    "    },\n",
    "    local=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the training is done, there will be a list of all the <span style=\"background:lightgreen\">artifacts created</span> in MLRun during the training run. All the (hopfully smooth) loss and metrics graphs we all love will be in both MLRun and Tensorboard, as well as the model weights and custom objects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2. Evaluate the Model\n",
    "\n",
    "Evaluating the model requires, you guessed it, the trained model. In order to get the model we just trained, we will use the training run object `training_run` that was returned from calling `run` on the MLRun function. Then, to get the model artifact, we will use `training_run.outputs` - a dictionary of all the function's artifacts that can be accessed by their names. So, to get the model, we will use the key \"model\" (as seen in the artifacts list generated above)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-09-05 10:25:46,679 [info] Storing function: {'name': 'evaluating', 'uid': 'a899d1c8d9b5458bbfb6bef7f8e7c119', 'db': 'http://mlrun-api:8080'}\n",
      "> 2023-09-05 10:25:46,979 [info] Job is running in the background, pod: evaluating-jzts5\n",
      "2023-09-05 10:26:12.017860: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
      "2023-09-05 10:26:12.017917: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "2023-09-05 10:26:14.243557: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory\n",
      "2023-09-05 10:26:14.243608: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "2023-09-05 10:26:14.243641: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (evaluating-jzts5): /proc/driver/nvidia/version does not exist\n",
      "2023-09-05 10:26:24.272542: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "43/43 [==============================] - 49s 1s/step - loss: 0.0429 - accuracy: 0.9927\n",
      "> 2023-09-05 10:27:21,316 [info] Run execution finished: {'status': 'completed', 'name': 'evaluating'}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>mask-detection-iguazio</td>\n",
       "      <td><div title=\"a899d1c8d9b5458bbfb6bef7f8e7c119\"><a href=\"https://dashboard.default-tenant.app.vmdev94.lab.iguazeng.com/mlprojects/mask-detection-iguazio/jobs/monitor/a899d1c8d9b5458bbfb6bef7f8e7c119/overview\" target=\"_blank\" >...f8e7c119</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Sep 05 10:26:11</td>\n",
       "      <td>completed</td>\n",
       "      <td>evaluating</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=iguazio</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=iguazio</div><div class=\"dictlist\">mlrun/client_version=1.5.0-rc9</div><div class=\"dictlist\">mlrun/client_python_version=3.9.16</div><div class=\"dictlist\">host=evaluating-jzts5</div></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">model_path=store://artifacts/mask-detection-iguazio/mask_detector:e80c00ef3e69420fbb2e29a17fa053de</div><div class=\"dictlist\">dataset_path=/User/test/demos/mask-detection/data/</div><div class=\"dictlist\">batch_size=32</div></td>\n",
       "      <td><div class=\"dictlist\">model_path=store://artifacts/mask-detection-iguazio/mask_detector:e80c00ef3e69420fbb2e29a17fa053de</div><div class=\"dictlist\">dataset_path=/User/test/demos/mask-detection/data/</div><div class=\"dictlist\">batch_size=32</div><div class=\"dictlist\">evaluation_loss=0.042869803517363796</div><div class=\"dictlist\">evaluation_accuracy=0.9927325581395349</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result1b09d639\" title=\"files/v3io/projects/mask-detection-iguazio/artifacts/evaluating/0/evaluation_loss.html.html\">evaluation_loss.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result1b09d639\" title=\"files/v3io/projects/mask-detection-iguazio/artifacts/evaluating/0/evaluation_accuracy.html.html\">evaluation_accuracy.html</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result1b09d639-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result1b09d639-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result1b09d639\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result1b09d639-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.vmdev94.lab.iguazeng.com/mlprojects/mask-detection-iguazio/jobs/monitor/a899d1c8d9b5458bbfb6bef7f8e7c119/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-09-05 10:27:25,654 [info] Run execution finished: {'status': 'completed', 'name': 'evaluating'}\n"
     ]
    }
   ],
   "source": [
    "evaluation_run = training_and_evaluation_function.run(\n",
    "    name=\"evaluating\",\n",
    "    handler=\"evaluate\",\n",
    "    params={\n",
    "        \"model_path\": training_run.outputs['model'],  # <- Take the model we trained from the previous MLRun function via the run object.\n",
    "        \"dataset_path\": dataset_path,\n",
    "        \"batch_size\": 32\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"section_6\"></a>\n",
    "## 6. Run Distributed Training Using Horovod\n",
    "\n",
    "Now we can see the second benefit of MLRun, we can **distribute** our model **training** across **multiple workers** (i.e., perform distributed training), assign **GPUs**, and more. We don't need to bother with Dockerfiles or K8s YAML configuration files — MLRun does all of this for us. We will simply create our function with `kind=\"mpijob\"`.\n",
    "\n",
    "> **Notice**: for this demo, in order to use GPUs in training, set the `use_gpu` variable to `True`. This will later assign the required configurations to use the GPUs and pass the correct image to support GPUs (image with CUDA libraries)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the MLRun Function:\n",
    "mpi_training_and_evaluation_function = mlrun.code_to_function(\n",
    "    filename=os.path.join(framework, \"training-and-evaluation.py\"),\n",
    "    name=\"mpi-training-and-evaluation\",\n",
    "    handler=\"train\",\n",
    "    kind=\"mpijob\",\n",
    "    image=f'.mask-detection-{framework}',\n",
    "    with_doc=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can set additional configurations for our run like image, workers, gpus and more (if `use_gpu` is `True` we will setup 2 workers with 1 GPU per worker):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mlrun.runtimes.mpijob.v1.MpiRuntimeV1 at 0x7fbd2b0007c0>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Setup the desired configurations:\n",
    "mpi_training_and_evaluation_function.spec.replicas = 1\n",
    "if use_gpu:\n",
    "    # Select the number of GPUs per replica:\n",
    "    mpi_training_and_evaluation_function.with_limits(gpus=1)\n",
    "else:\n",
    "    mpi_training_and_evaluation_function.with_requests(cpu=2)\n",
    "\n",
    "# Mount it:\n",
    "mpi_training_and_evaluation_function.apply(mlrun.auto_mount())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# community edition mount support\n",
    "if os.getenv('V3IO_ACCESS_KEY','False')=='False':\n",
    "    mpi_training_and_evaluation_function.spec.disable_auto_mount=False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Call run, and notice each epoch is shorter as we now have 2 workers instead of 1. As the 2 workers will print a lot of outputs we would rather wait for completion and then show the results. For that, we will pass `watch=False` and use the run objects function `wait_for_completion` and `show`. \n",
    "\n",
    "To see the logs, you can go into the UI by clicking the blue hyperlink \"<span style=\"color:blue\">**click here**</span>\" after running the function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-09-05 10:27:25,906 [info] Storing function: {'name': 'distributed-training', 'uid': '0cb4e7dbe2a2438788038915e789d505', 'db': 'http://mlrun-api:8080'}\n",
      "> 2023-09-05 10:28:17,832 [info] MpiJob distributed-training-d3ae842b launcher pod distributed-training-d3ae842b-launcher state active\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>mask-detection-iguazio</td>\n",
       "      <td><div title=\"0cb4e7dbe2a2438788038915e789d505\"><a href=\"https://dashboard.default-tenant.app.vmdev94.lab.iguazeng.com/mlprojects/mask-detection-iguazio/jobs/monitor/0cb4e7dbe2a2438788038915e789d505/overview\" target=\"_blank\" >...e789d505</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Sep 05 10:27:26</td>\n",
       "      <td>running</td>\n",
       "      <td>distributed-training</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=iguazio</div><div class=\"dictlist\">kind=mpijob</div><div class=\"dictlist\">owner=iguazio</div><div class=\"dictlist\">mlrun/client_version=1.5.0-rc9</div><div class=\"dictlist\">mlrun/client_python_version=3.9.16</div></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">dataset_path=/User/test/demos/mask-detection/data/</div><div class=\"dictlist\">batch_size=32</div><div class=\"dictlist\">lr=0.0001</div><div class=\"dictlist\">epochs=3</div></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resultfa765f31-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resultfa765f31-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resultfa765f31\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resultfa765f31-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.vmdev94.lab.iguazeng.com/mlprojects/mask-detection-iguazio/jobs/monitor/0cb4e7dbe2a2438788038915e789d505/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-09-05 10:28:17,853 [info] Run execution finished: {'status': 'running', 'name': 'distributed-training'}\n",
      "> 2023-09-05 10:28:17,877 [info] run distributed-training is not completed yet, waiting for it to complete: {'current_state': 'running'}\n",
      "+ POD_NAME=distributed-training-d3ae842b-worker-0\n",
      "+ shift\n",
      "+ /opt/kube/kubectl exec distributed-training-d3ae842b-worker-0 -- /bin/sh -c  orted -mca ess \"env\" -mca ess_base_jobid \"2329411584\" -mca ess_base_vpid 1 -mca ess_base_num_procs \"2\" -mca orte_node_regex \"distributed-training-d[1:3]ae842b-launcher,distributed-training-d[1:3]ae842b-worker-0@0(2)\" -mca orte_hnp_uri \"2329411584.0;tcp://10.233.67.47:37575\" -mca plm \"rsh\" --tree-spawn -mca routed \"radix\" -mca orte_parent_uri \"2329411584.0;tcp://10.233.67.47:37575\" -mca plm_rsh_agent \"/etc/mpi/kubexec.sh\" -mca orte_default_hostfile \"/etc/mpi/hostfile\" -mca pmix \"^s1,s2,cray,isolated\"\n",
      "2023-09-05 10:28:22.201858: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
      "2023-09-05 10:28:22.201904: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "2023-09-05 10:28:24.415602: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory\n",
      "2023-09-05 10:28:24.415655: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "2023-09-05 10:28:24.415691: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (distributed-training-d3ae842b-worker-0): /proc/driver/nvidia/version does not exist\n",
      "WARNING:tensorflow:`input_shape` is undefined or non-square, or `rows` is not in [96, 128, 160, 192, 224]. Weights for input shape (224, 224) will be loaded as the default.\n",
      "2023-09-05 10:28:33.778812: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/mobilenet_v2/mobilenet_v2_weights_tf_dim_ordering_tf_kernels_1.0_224_no_top.h5\n",
      "9406464/9406464 [==============================] - 0s 0us/step\n",
      "The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "Horovod worker #0 is using CPU\n",
      "Epoch 1/3\n",
      "35/35 [==============================] - 96s 3s/step - loss: 0.5084 - accuracy: 0.7736 - val_loss: 0.1132 - val_accuracy: 0.9783 - lr: 1.0000e-04\n",
      "Epoch 2/3\n",
      "35/35 [==============================] - 89s 3s/step - loss: 0.1301 - accuracy: 0.9618 - val_loss: 0.0609 - val_accuracy: 0.9855 - lr: 1.0000e-04\n",
      "Epoch 3/3\n",
      "35/35 [==============================] - 88s 3s/step - loss: 0.0730 - accuracy: 0.9782 - val_loss: 0.0465 - val_accuracy: 0.9855 - lr: 1.0000e-04\n",
      "WARNING:absl:Found untraced functions such as _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op while saving (showing 5 of 52). These functions will not be directly callable after loading.\n",
      "> 2023-09-05 10:33:28,738 [info] Run execution finished: {'status': 'running', 'name': 'distributed-training'}\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>mask-detection-iguazio</td>\n",
       "      <td><div title=\"0cb4e7dbe2a2438788038915e789d505\"><a href=\"https://dashboard.default-tenant.app.vmdev94.lab.iguazeng.com/mlprojects/mask-detection-iguazio/jobs/monitor/0cb4e7dbe2a2438788038915e789d505/overview\" target=\"_blank\" >...e789d505</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Sep 05 10:28:21</td>\n",
       "      <td>completed</td>\n",
       "      <td>distributed-training</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=iguazio</div><div class=\"dictlist\">kind=mpijob</div><div class=\"dictlist\">owner=iguazio</div><div class=\"dictlist\">mlrun/client_version=1.5.0-rc9</div><div class=\"dictlist\">mlrun/client_python_version=3.9.16</div><div class=\"dictlist\">mlrun/job=distributed-training-d3ae842b</div><div class=\"dictlist\">host=distributed-training-d3ae842b-worker-0</div></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">dataset_path=/User/test/demos/mask-detection/data/</div><div class=\"dictlist\">batch_size=32</div><div class=\"dictlist\">lr=0.0001</div><div class=\"dictlist\">epochs=3</div></td>\n",
       "      <td><div class=\"dictlist\">dataset_path=/User/test/demos/mask-detection/data/</div><div class=\"dictlist\">batch_size=32</div><div class=\"dictlist\">epochs=3</div><div class=\"dictlist\">lr=9.999999747378752e-05</div><div class=\"dictlist\">training_loss=0.040120527148246765</div><div class=\"dictlist\">training_accuracy=1.0004091262817383</div><div class=\"dictlist\">validation_loss=0.04653192311525345</div><div class=\"dictlist\">validation_accuracy=0.9855072498321533</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultabaa14dc\" title=\"files/v3io/projects/mask-detection-iguazio/artifacts/distributed-training/0/training_loss.html.html\">training_loss.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultabaa14dc\" title=\"files/v3io/projects/mask-detection-iguazio/artifacts/distributed-training/0/training_accuracy.html.html\">training_accuracy.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultabaa14dc\" title=\"files/v3io/projects/mask-detection-iguazio/artifacts/distributed-training/0/validation_loss.html.html\">validation_loss.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultabaa14dc\" title=\"files/v3io/projects/mask-detection-iguazio/artifacts/distributed-training/0/validation_accuracy.html.html\">validation_accuracy.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultabaa14dc\" title=\"files/v3io/projects/mask-detection-iguazio/artifacts/distributed-training/0/loss_summary.html.html\">loss_summary.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultabaa14dc\" title=\"files/v3io/projects/mask-detection-iguazio/artifacts/distributed-training/0/accuracy_summary.html.html\">accuracy_summary.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultabaa14dc\" title=\"files/v3io/projects/mask-detection-iguazio/artifacts/distributed-training/0/lr_values.html.html\">lr_values.html</div><div title=\"v3io:///projects/mask-detection-iguazio/artifacts/distributed-training/0/model/\">model</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resultabaa14dc-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resultabaa14dc-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resultabaa14dc\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resultabaa14dc-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Run the training job:\n",
    "distributed_training_run = mpi_training_and_evaluation_function.run(\n",
    "    name=\"distributed-training\",\n",
    "    params={\n",
    "        \"dataset_path\": dataset_path,\n",
    "        \"batch_size\": 32,\n",
    "        \"lr\": 1e-4,\n",
    "        \"epochs\": 3,\n",
    "    },\n",
    "    watch=False,  # <- Turn off the logs.\n",
    ")\n",
    "\n",
    "# Wait for complition and show the results. \n",
    "distributed_training_run.wait_for_completion()\n",
    "distributed_training_run.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mlrun-base",
   "language": "python",
   "name": "conda-env-mlrun-base-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}