examples/tutorial_pytorch_instance_segmentation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right\"> \n",
    "<!-- Add icon library -->\n",
    "<link rel=\"stylesheet\" href=\"https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css\"> <a href=\"https://gitcdn.link/repo/rediscovery-io/remo-python/master/examples/tutorial_pytorch_instance_segmentation.ipynb\" target=\"_blank\"> <button style=\"background-color: #4caf50;border: none;border-radius:2px; position: relative; top: -6px;color: white; padding: 2px 4px;cursor: pointer;height: 20px;font-size: 12px;width: 95px;\"> <i class=\"fa fa-download\"></i> Download </button></a> <a href=\"http://colab.research.google.com/github/rediscovery-io/remo-python/blob/master/examples/google-colab/tutorial_pytorch_instance_segmentation.ipynb\" target=\"_blank\"><button style=\"background-image: url('https://colab.research.google.com/assets/colab-badge.svg');height: 20px; width:117px;border: none; outline: none;)\"></button></a>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vM54r6jlKTII"
   },
   "source": [
    "# Instance Segmentation with Detectron2 and Remo\n",
    "\n",
    "In this tutorial, we do transfer learning on a MaskRCNN model from Detectron2. \n",
    "We use Remo to facilitate exploring, accessing and managing the dataset.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jxqxkISHjEjg"
   },
   "source": [
    "In particular, we will:\n",
    "\n",
    "* Browse through our images and annotations\n",
    "* Quickly visualize the main properties of the dataset and make some initial observations\n",
    "* Create a train, test, valid split without moving data around, using Remo image tags.\n",
    "* Fine tune a pre-trained MaskRCNN model from Detectron2 and do some inference\n",
    "* Visually compare Mask predictions with the ground truth, and draw possible conclusions on how to improve performance\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2swBcOUSja50"
   },
   "source": [
    "**Along the way, we will see how browsing images, annotations and predictions helps to gather insights on the dataset and on the model.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3QGTs985jhMo"
   },
   "source": [
    "Before proceeding, we need to install the required dependencies. \n",
    "\n",
    "This can be done by executing the next cell. Once complete, **restart your runtime** to ensure that the installed packages can be detected.\n",
    "\n",
    "This tutorial is supported to run only on a **CUDA enabled GPU** locally or on Google Colab.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9_FzH13EjseR",
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install imantics\n",
    "!pip install git+https://github.com/facebookresearch/fvcore.git\n",
    "!git clone https://github.com/facebookresearch/detectron2 detectron2_repo\n",
    "!pip install -e detectron2_repo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HJ0_LPkjj9Xe"
   },
   "source": [
    "Let us then import the required packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 1280,
     "status": "ok",
     "timestamp": 1602098681523,
     "user": {
      "displayName": "Sree Harsha Nelaturu",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghyid0sxSGIX-vJDOaqVZv_5TgJ_muLwOFbqtQLZQ=s64",
      "userId": "01158735529041422505"
     },
     "user_tz": -330
    },
    "id": "ZyAvNCJMmvFF",
    "tags": []
   },
   "outputs": [],
   "source": [
    "import remo\n",
    "remo.set_viewer('jupyter')\n",
    "\n",
    "import numpy as np\n",
    "import os\n",
    "from PIL import Image\n",
    "import glob\n",
    "import random\n",
    "random.seed(600)\n",
    "\n",
    "from imantics import Polygons, Mask\n",
    "\n",
    "import torch, torchvision\n",
    "\n",
    "# Detectron 2 files\n",
    "import detectron2\n",
    "from detectron2.utils.logger import setup_logger\n",
    "setup_logger()\n",
    "from detectron2.engine import DefaultPredictor\n",
    "from detectron2.config import get_cfg\n",
    "from detectron2.utils.visualizer import Visualizer\n",
    "from detectron2.data import MetadataCatalog, DatasetCatalog\n",
    "from detectron2.engine import DefaultTrainer\n",
    "from detectron2.data.datasets import register_coco_instances\n",
    "from detectron2.evaluation import COCOEvaluator, inference_on_dataset\n",
    "from detectron2.data import build_detection_test_loader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cZ68o1uzkJAc"
   },
   "source": [
    "## Adding Data to Remo\n",
    "\n",
    "* The Dataset used is a subset of the <a href=\"https://cocodataset.org/#home\">MS COCO Dataset</a>.\n",
    "* The directory structure of the dataset is:\n",
    "```\n",
    "├── coco_instance_segmentation_dataset\n",
    "    ├── images\n",
    "        ├── image_1.jpg\n",
    "        ├── image_2.jpg\n",
    "        ├── ...\n",
    "    ├── annotations\n",
    "        ├── coco_instance_segmentation.json\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "uXtPd40lbx0N",
    "tags": []
   },
   "outputs": [],
   "source": [
    "if not os.path.exists('coco_instance_segmentation_dataset.zip'):\n",
    "  !wget https://s-3.s3-eu-west-1.amazonaws.com/coco_instance_segmentation_dataset.zip\n",
    "  !unzip coco_instance_segmentation_dataset.zip\n",
    "else:\n",
    "  print('Files already downloaded')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 1252,
     "status": "ok",
     "timestamp": 1602098700505,
     "user": {
      "displayName": "Sree Harsha Nelaturu",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghyid0sxSGIX-vJDOaqVZv_5TgJ_muLwOFbqtQLZQ=s64",
      "userId": "01158735529041422505"
     },
     "user_tz": -330
    },
    "id": "0Hz7k5N0cW6U"
   },
   "outputs": [],
   "source": [
    "# The path to the folders\n",
    "\n",
    "path_to_annotations = 'coco_instance_segmentation_dataset/annotations/'\n",
    "path_to_images = 'coco_instance_segmentation_dataset/images/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "R4c2IKhvk4tP"
   },
   "source": [
    "## Train / test split\n",
    "In Remo, we can use tags to organise our images. Among other things, this allows us to generate train / test splits without the need to move image files around.\n",
    "\n",
    "To do this, we just need to pass a dictionary (mapping tags to the relevant images paths) to the function ```remo.generate_image_tags()```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "executionInfo": {
     "elapsed": 691,
     "status": "ok",
     "timestamp": 1602098700509,
     "user": {
      "displayName": "Sree Harsha Nelaturu",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghyid0sxSGIX-vJDOaqVZv_5TgJ_muLwOFbqtQLZQ=s64",
      "userId": "01158735529041422505"
     },
     "user_tz": -330
    },
    "id": "tvlQdeFIgtGu",
    "outputId": "401be35c-5a68-4754-ae3e-c3b920bfc12b",
    "tags": []
   },
   "outputs": [],
   "source": [
    "im_list = [i for i in glob.glob(path_to_images + '/**/*.jpg', recursive=True)]\n",
    "im_list = random.sample(im_list, len(im_list))\n",
    "\n",
    "train_idx = round(len(im_list) * 0.8)\n",
    "test_idx  = train_idx + round(len(im_list) * 0.2)\n",
    "\n",
    "tags_dict =  {'train' : im_list[0:train_idx], \n",
    "              'test' : im_list[train_idx:test_idx]}\n",
    "\n",
    "train_test_split_file_path = os.path.join(path_to_annotations, 'images_tags.csv') \n",
    "remo.generate_image_tags(tags_dictionary  = tags_dict, \n",
    "                         output_file_path = train_test_split_file_path, \n",
    "                         append_path = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jyCLB5AdlA9q"
   },
   "source": [
    "### Create a dataset\n",
    "\n",
    "To create a dataset we can use ```remo.create_dataset()```, specifying the path to data and annotations.\n",
    "\n",
    "For a complete list of formats supported, you can <a href=\"https://remo.ai/docs/annotation-formats/\"> refer to the docs</a>.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "b6yii4wdZ3C6",
    "tags": []
   },
   "outputs": [],
   "source": [
    "coco_instance_segmentation_dataset = remo.create_dataset(name = 'coco_instance_segmentation_dataset', local_files = [path_to_annotations, path_to_images], annotation_task='Instance Segmentation')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yhtI7LI5lTda"
   },
   "source": [
    "**Visualizing the dataset**\n",
    "\n",
    "To view and explore images and labels, we can use Remo directly from the notebook. We just need to call ```dataset.view()```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vM85hXjDZ3C_",
    "tags": []
   },
   "outputs": [],
   "source": [
    "coco_instance_segmentation_dataset.view()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![instance_segmentation_view](assets/instance_dataset_view.png)\n",
    "\n",
    "**Looking at the dataset, we notice some interesting points:**\n",
    "\n",
    "- Pictures of the animals can be taken from different angles\n",
    "\n",
    "- In some cases, there is an overlap of classes such that it represents an occlusion i.e Zebra right beside a Giraffe.\n",
    "\n",
    "- The actual pose of the same object varies across instances."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kiQ4iX-MmBym"
   },
   "source": [
    "**Dataset Statistics**\n",
    "\n",
    "Using Remo, we can quickly visualize some key Dataset properties that can help us with our modelling, without needing to write extra boilerplate code.\n",
    "\n",
    "This can be done either from code, or using the visual interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "mPuePhS7mDho"
   },
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "[{'AnnotationSet ID': 430,\n  'AnnotationSet name': 'Instance segmentation',\n  'n_images': 6,\n  'n_classes': 2,\n  'n_objects': 22,\n  'top_3_classes': [{'name': 'Zebra', 'count': 13},\n   {'name': 'Giraffe', 'count': 9}],\n  'creation_date': None,\n  'last_modified_date': '2020-11-04T11:35:11.415574Z'}]"
     },
     "metadata": {},
     "execution_count": 9
    }
   ],
   "source": [
    "coco_instance_segmentation_dataset.get_annotation_statistics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PcTk9j6GmFb5",
    "tags": []
   },
   "outputs": [],
   "source": [
    "coco_instance_segmentation_dataset.view_annotation_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![instance_view_annotations](assets/instance_view_annotations.png)\n",
    "\n",
    "**Looking at the statistics we can gain some useful insights like:**\n",
    "\n",
    "- The highest number of instances per image is that of Zebra. This means we have a somewhat unbalanced dataset, and we might expect to see the model perfom better on Zebras\n",
    "\n",
    "- The data distribution looks similar in both train and test dataset. This is good!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kvRkcddYlXlF"
   },
   "source": [
    "**Exporting the dataset**\n",
    "\n",
    "To export annotations according to the train, test split in a format accepted by the model, we use the ```dataset.export_annotations_to_file()``` method, and filter by the desired tag.\n",
    "\n",
    "For a complete list of formats supported, you can <a href=\"https://remo.ai/docs/annotation-formats/\"> refer to the docs</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 1344,
     "status": "ok",
     "timestamp": 1602098960894,
     "user": {
      "displayName": "Sree Harsha Nelaturu",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghyid0sxSGIX-vJDOaqVZv_5TgJ_muLwOFbqtQLZQ=s64",
      "userId": "01158735529041422505"
     },
     "user_tz": -330
    },
    "id": "1nco9a1liIjG"
   },
   "outputs": [],
   "source": [
    "path_to_train = path_to_annotations + 'coco_instance_segmentation_train.json'\n",
    "path_to_test = path_to_annotations + 'coco_instance_segmentation_test.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 2110,
     "status": "ok",
     "timestamp": 1602098961788,
     "user": {
      "displayName": "Sree Harsha Nelaturu",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghyid0sxSGIX-vJDOaqVZv_5TgJ_muLwOFbqtQLZQ=s64",
      "userId": "01158735529041422505"
     },
     "user_tz": -330
    },
    "id": "X522eyE0Z3DE"
   },
   "outputs": [],
   "source": [
    "coco_instance_segmentation_dataset.export_annotations_to_file(path_to_train, annotation_format='coco', filter_by_tags=['train'], export_tags=False, append_path=False)\n",
    "coco_instance_segmentation_dataset.export_annotations_to_file(path_to_test, annotation_format='coco', filter_by_tags=['test'], export_tags=False, append_path=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bcsvm-6ZZ3DJ"
   },
   "source": [
    "# Detectron2\n",
    "\n",
    "Here we will start working with the ```Detectron2``` framework written in PyTorch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d6zVCtb1mwNt"
   },
   "source": [
    "## Feeding Data into Detectron2\n",
    "\n",
    "To use Detectron2, you are required to register your dataset.\n",
    "\n",
    "The ```register_coco_instances``` method takes in the following parameters:\n",
    "\n",
    "* **path_to_annotations:** Path to annotation files. Format: COCO JSON.\n",
    "\n",
    "* **path_to_images:** Path to the folder containing the images.\n",
    "\n",
    "This then allows to store the metadata for future operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 922,
     "status": "ok",
     "timestamp": 1602098994157,
     "user": {
      "displayName": "Sree Harsha Nelaturu",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghyid0sxSGIX-vJDOaqVZv_5TgJ_muLwOFbqtQLZQ=s64",
      "userId": "01158735529041422505"
     },
     "user_tz": -330
    },
    "id": "Lnkg1PByUjGQ"
   },
   "outputs": [],
   "source": [
    "register_coco_instances('coco_instance_segmentation_train', {}, path_to_train, path_to_images)\n",
    "register_coco_instances('coco_instance_segmentation_test', {}, path_to_test, path_to_images)\n",
    "\n",
    "train_metadata = MetadataCatalog.get('coco_instance_segmentation_train')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4OTMNIVZnTUQ"
   },
   "source": [
    "## Training the Model\n",
    "\n",
    "For the sake of the tutorial, our ```Mask RCNN``` architecture will have a ```ResNet-50 Backbone```, pre-trained on on COCO train2017. This can be loaded directly from Detectron2.\n",
    "\n",
    "To train the model, we specify the following details:\n",
    "\n",
    "- **model_yaml_path:** Configuration file for the Mask RCNN model.\n",
    "\n",
    "- **model_weights_path**: Symbolic link to the desired Mask RCNN architecture.\n",
    "\n",
    "The parameters can be tweaked by overriding the correspodning variable in the ```cfg```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7unkuuiqLdqd"
   },
   "outputs": [],
   "source": [
    "model_yaml_path = './detectron2_repo/configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml'\n",
    "model_weights_path = 'detectron2://COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x/137849600/model_final_f10217.pkl'\n",
    "\n",
    "cfg = get_cfg()\n",
    "cfg.merge_from_file(model_yaml_path)\n",
    "cfg.DATASETS.TRAIN = ('coco_instance_segmentation_train',)\n",
    "cfg.DATASETS.TEST = ()\n",
    "cfg.DATALOADER.NUM_WORKERS = 2\n",
    "cfg.MODEL.WEIGHTS = model_weights_path # initialize from model zoo\n",
    "cfg.SOLVER.IMS_PER_BATCH = 2\n",
    "cfg.SOLVER.BASE_LR = 0.02\n",
    "cfg.SOLVER.MAX_ITER = 150    # 300 iterations seems good enough, but you can certainly train longer\n",
    "cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128   # faster, and good enough for this toy dataset\n",
    "cfg.MODEL.ROI_HEADS.NUM_CLASSES = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YRqEn61uovp2"
   },
   "source": [
    "**Instantiating the Trainer**\n",
    "\n",
    "We instatiate the trainer with the required configuration, and finally kick-off the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "prcuWITMou0w",
    "tags": []
   },
   "outputs": [],
   "source": [
    "os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)\n",
    "trainer = DefaultTrainer(cfg)\n",
    "trainer.resume_or_load(resume=False)\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Performance Metrics**\n",
    "\n",
    "In order to evaluate the model performance on metrics such as Average Precision (AP) and Mean Average Precision (mAP), we will use the ```COCOEvaluator``` in the detectron2 package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "\u001b[32m[11/18 21:18:15 d2.data.datasets.coco]: \u001b[0mLoaded 1 images in COCO format from coco_instance_segmentation_dataset/annotations/coco_instance_segmentation_test.json\n\u001b[32m[11/18 21:18:15 d2.data.common]: \u001b[0mSerializing 1 elements to byte tensors and concatenating them all ...\n\u001b[32m[11/18 21:18:15 d2.data.common]: \u001b[0mSerialized dataset takes 0.00 MiB\n\u001b[32m[11/18 21:18:15 d2.data.dataset_mapper]: \u001b[0mAugmentations used in training: [ResizeShortestEdge(short_edge_length=(800, 800), max_size=1333, sample_style='choice')]\n\u001b[32m[11/18 21:18:15 d2.evaluation.evaluator]: \u001b[0mStart inference on 1 images\n\u001b[32m[11/18 21:18:15 d2.evaluation.evaluator]: \u001b[0mInference done 1/1. 0.4105 s / img. ETA=0:00:00\n\u001b[32m[11/18 21:18:15 d2.evaluation.evaluator]: \u001b[0mTotal inference time: 0:00:00.440677 (0.440677 s / img per device, on 1 devices)\n\u001b[32m[11/18 21:18:15 d2.evaluation.evaluator]: \u001b[0mTotal inference pure compute time: 0:00:00 (0.410518 s / img per device, on 1 devices)\n\u001b[32m[11/18 21:18:15 d2.evaluation.coco_evaluation]: \u001b[0mPreparing results for COCO format ...\n\u001b[32m[11/18 21:18:15 d2.evaluation.coco_evaluation]: \u001b[0mSaving results to ./output/coco_instances_results.json\n\u001b[32m[11/18 21:18:15 d2.evaluation.coco_evaluation]: \u001b[0mEvaluating predictions ...\nLoading and preparing results...\nDONE (t=0.00s)\ncreating index...\nindex created!\nRunning per image evaluation...\nEvaluate annotation type *bbox*\nCOCOeval_opt.evaluate() finished in 0.00 seconds.\nAccumulating evaluation results...\nCOCOeval_opt.accumulate() finished in 0.00 seconds.\n Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.800\n Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 1.000\n Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 1.000\n Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.800\n Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000\n Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.400\n Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.800\n Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.800\n Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.800\n Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000\n Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n\u001b[32m[11/18 21:18:15 d2.evaluation.coco_evaluation]: \u001b[0mEvaluation results for bbox: \n|   AP   |  AP50   |  AP75   |  APs   |  APm  |  APl  |\n|:------:|:-------:|:-------:|:------:|:-----:|:-----:|\n| 80.000 | 100.000 | 100.000 | 80.000 |  nan  |  nan  |\n\u001b[32m[11/18 21:18:15 d2.evaluation.coco_evaluation]: \u001b[0mSome metrics cannot be computed and is shown as NaN.\nLoading and preparing results...\nDONE (t=0.00s)\ncreating index...\nindex created!\nRunning per image evaluation...\nEvaluate annotation type *segm*\nCOCOeval_opt.evaluate() finished in 0.00 seconds.\nAccumulating evaluation results...\nCOCOeval_opt.accumulate() finished in 0.00 seconds.\n Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.800\n Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 1.000\n Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 1.000\n Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.800\n Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000\n Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.400\n Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.800\n Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.800\n Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.800\n Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000\n Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n\u001b[32m[11/18 21:18:15 d2.evaluation.coco_evaluation]: \u001b[0mEvaluation results for segm: \n|   AP   |  AP50   |  AP75   |  APs   |  APm  |  APl  |\n|:------:|:-------:|:-------:|:------:|:-----:|:-----:|\n| 80.000 | 100.000 | 100.000 | 80.000 |  nan  |  nan  |\n\u001b[32m[11/18 21:18:15 d2.evaluation.coco_evaluation]: \u001b[0mSome metrics cannot be computed and is shown as NaN.\nOrderedDict([('bbox', {'AP': 80.0, 'AP50': 100.0, 'AP75': 100.0, 'APs': 80.0, 'APm': nan, 'APl': nan}), ('segm', {'AP': 80.0, 'AP50': 100.0, 'AP75': 100.0, 'APs': 80.0, 'APm': nan, 'APl': nan})])\n"
    }
   ],
   "source": [
    "evaluator = COCOEvaluator(\"coco_instance_segmentation_test\", cfg, False, output_dir=\"./output/\")\n",
    "val_loader = build_detection_test_loader(cfg, \"coco_instance_segmentation_test\")\n",
    "print(inference_on_dataset(trainer.model, val_loader, evaluator))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "N0aT_wYKpIGu"
   },
   "source": [
    "## Visualizing Predictions\n",
    "\n",
    "Using Remo, we can easily browse our predictions and compare them with the ground-truth.\n",
    "\n",
    "We will do this by uploading the model predictions to a new ```AnnotationSet```, which we call `model_predictions`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Ya5nEuMELeq8",
    "tags": []
   },
   "outputs": [],
   "source": [
    "cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, 'model_final.pth')\n",
    "cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5   # set the testing threshold for this model\n",
    "cfg.DATASETS.TEST = ('coco_instance_segmentation_test', )\n",
    "predictor = DefaultPredictor(cfg)\n",
    "test_dataset_dicts = DatasetCatalog.get('coco_instance_segmentation_test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To visualize the labels as strings rather than IDs, we can use a dictionary mapping the two of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 962,
     "status": "ok",
     "timestamp": 1602099003560,
     "user": {
      "displayName": "Sree Harsha Nelaturu",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghyid0sxSGIX-vJDOaqVZv_5TgJ_muLwOFbqtQLZQ=s64",
      "userId": "01158735529041422505"
     },
     "user_tz": -330
    },
    "id": "UJ6K3CCHZ3DV"
   },
   "outputs": [],
   "source": [
    "mapping = {k: v for k, v in enumerate(train_metadata.thing_classes)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 1066,
     "status": "ok",
     "timestamp": 1602098310327,
     "user": {
      "displayName": "Sree Harsha Nelaturu",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghyid0sxSGIX-vJDOaqVZv_5TgJ_muLwOFbqtQLZQ=s64",
      "userId": "01158735529041422505"
     },
     "user_tz": -330
    },
    "id": "OuHpdPcNomu4"
   },
   "outputs": [],
   "source": [
    "for d in test_dataset_dicts:    \n",
    "    im = np.array(Image.open(d['file_name']))\n",
    "    outputs = predictor(im)\n",
    "    pred_classes = outputs['instances'].get('pred_classes').cpu().numpy()\n",
    "    masks = outputs['instances'].get('pred_masks').cpu().permute(1, 2, 0).numpy()\n",
    "    image_name = d['file_name']\n",
    "    annotations = []\n",
    "    \n",
    "    if masks.shape[2] != 0:\n",
    "        for i in range(masks.shape[2]):\n",
    "            polygons = Mask(masks[:, :, i]).polygons()\n",
    "            annotation = remo.Annotation()\n",
    "            annotation.img_filename = image_name\n",
    "            annotation.classes = mapping[pred_classes[i]]\n",
    "            annotation.segment = polygons.segmentation[0]\n",
    "            annotations.append(annotation)\n",
    "    else:\n",
    "        polygons = Mask(masks[:, :, 0]).polygons()\n",
    "        annotation = remo.Annotation()\n",
    "        annotation.img_filename = image_name\n",
    "        annotation.classes = mapping[pred_classes[0]]\n",
    "        annotation.segment = polygons.segmentation[0]\n",
    "        annotations.append(annotation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 3292,
     "status": "ok",
     "timestamp": 1602097960944,
     "user": {
      "displayName": "Sree Harsha Nelaturu",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghyid0sxSGIX-vJDOaqVZv_5TgJ_muLwOFbqtQLZQ=s64",
      "userId": "01158735529041422505"
     },
     "user_tz": -330
    },
    "id": "VrzA6mcJdRQ0",
    "outputId": "311ed274-4bf7-49c7-acfe-b9e6ca51bba4",
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_predictions = coco_instance_segmentation_dataset.create_annotation_set(annotation_task = 'Instance Segmentation', name = 'model_predictions')\n",
    "\n",
    "coco_instance_segmentation_dataset.add_annotations(annotations, annotation_set_id=model_predictions.id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "DfazUplreHoN",
    "tags": []
   },
   "outputs": [],
   "source": [
    "coco_instance_segmentation_dataset.view()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nqYP2rlEpjJw"
   },
   "source": [
    "![instance_model_predictions](assets/instance_model_predictions.png)\n",
    "\n",
    "By visualizing the predicted masks against the ground truth, we can go past summary performance metrics, and visually inspect model biases and iterate to improve it.\n",
    "\n",
    "**Looking at once picture, we notice:** \n",
    "\n",
    "- The giraffe legs are not picked up by the model, this might be due to the tree occlusion\n",
    "\n",
    "- A part of the gazelle's body (not present in the annotation) is mistaken as part of a zebra, possibly due to feature similarity in pose.\n",
    "\n",
    "- The model is able to distinguish the zebra in the background, which was quite occluded. This is good!\n",
    "\n",
    "In reality, we would look at all the pictures and at the model performance by class before drawing conclusions. Based on one picture, we can already come up with the following:\n",
    "\n",
    "**Potential improvements**\n",
    "\n",
    "- Add trees to the model training data, and give it more occluded examples. The occluded examples could be labelled as a different class initially, so we can see the count. And then experiments would say whether it's better to have it as unique class or not\n",
    "\n",
    "- Annotate Gazelles as a separate class.\n",
    "\n",
    "- Obvious one: train for more epochs and with more data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "- [How to train Detectron2 with Custom COCO Datasets](https://medium.com/@chengweizhang2012/how-to-train-detectron2-with-custom-coco-datasets-4d5170c9f389)\n",
    "- [Detectron2](https://github.com/facebookresearch/detectron2)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "colab_instance_segmentation_tutorial.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.6.12 64-bit ('pt_36': conda)",
   "language": "python",
   "name": "python_defaultSpec_1604489680091"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10-final"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}