05c - Transfer Learning (Tensorflow).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transfer Learning\n",
    "\n",
    "A Convolutional Neural Network (CNN) for image classification is made up of multiple layers that extract features, such as edges, corners, etc; and then use a final fully-connected layer to classify objects based on these features. You can visualize this like this:\n",
    "\n",
    "<table>\n",
    "    <tr><td rowspan=2 style='border: 1px solid black;'>&#x21d2;</td><td style='border: 1px solid black;'>Convolutional Layer</td><td style='border: 1px solid black;'>Pooling Layer</td><td style='border: 1px solid black;'>Convolutional Layer</td><td style='border: 1px solid black;'>Pooling Layer</td><td style='border: 1px solid black;'>Fully Connected Layer</td><td rowspan=2 style='border: 1px solid black;'>&#x21d2;</td></tr>\n",
    "    <tr><td colspan=4 style='border: 1px solid black; text-align:center;'>Feature Extraction</td><td style='border: 1px solid black; text-align:center;'>Classification</td></tr>\n",
    "</table>\n",
    "\n",
    "*Transfer Learning* is a technique where you can take an existing trained model and re-use its feature extraction layers, replacing its final classification layer with a fully-connected layer trained on your own custom images. With this technique, your model benefits from the feature extraction training that was performed on the base model (which may have been based on a larger training dataset than you have access to) to build a classification model for your own specific set of object classes.\n",
    "\n",
    "How does this help? Well, think of it this way. Suppose you take a professional tennis player and a complete beginner, and try to teach them both how to play raquetball. It's reasonable to assume that the professional tennis player will be easier to train, because many of the underlying skills involved in raquetball are already learned. Similarly, a pre-trained CNN model may be easier to train to classify specific set of objects because it's already learned how to identify the features of common objects, such as edges and corners. Fundamentally, a pre-trained model can be a great way to produce an effective classifier even when you have limited data with which to train it.\n",
    "\n",
    "In this notebook, we'll see how to implement transfer learning for a classification model using TensorFlow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install and import TensorFlow libraries\n",
    "\n",
    "Let's start by ensuring that we have the latest version of the **TensorFlow** package installed and importing the Tensorflow libraries we're going to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already up-to-date: tensorflow in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (2.10.0)\n",
      "Requirement already satisfied, skipping upgrade: opt-einsum>=2.3.2 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (3.3.0)\n",
      "Requirement already satisfied, skipping upgrade: protobuf<3.20,>=3.9.2 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (3.19.6)\n",
      "Requirement already satisfied, skipping upgrade: termcolor>=1.1.0 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (1.1.0)\n",
      "Requirement already satisfied, skipping upgrade: absl-py>=1.0.0 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (1.3.0)\n",
      "Requirement already satisfied, skipping upgrade: google-pasta>=0.1.1 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (0.2.0)\n",
      "Requirement already satisfied, skipping upgrade: h5py>=2.9.0 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (3.7.0)\n",
      "Requirement already satisfied, skipping upgrade: tensorflow-estimator<2.11,>=2.10.0 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (2.10.0)\n",
      "Requirement already satisfied, skipping upgrade: packaging in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (21.3)\n",
      "Requirement already satisfied, skipping upgrade: six>=1.12.0 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (1.16.0)\n",
      "Requirement already satisfied, skipping upgrade: keras-preprocessing>=1.1.1 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (1.1.2)\n",
      "Requirement already satisfied, skipping upgrade: libclang>=13.0.0 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (14.0.6)\n",
      "Requirement already satisfied, skipping upgrade: tensorflow-io-gcs-filesystem>=0.23.1 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (0.27.0)\n",
      "Requirement already satisfied, skipping upgrade: flatbuffers>=2.0 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (2.0)\n",
      "Requirement already satisfied, skipping upgrade: tensorboard<2.11,>=2.10 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (2.10.1)\n",
      "Requirement already satisfied, skipping upgrade: gast<=0.4.0,>=0.2.1 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (0.3.3)\n",
      "Requirement already satisfied, skipping upgrade: setuptools in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (49.6.0)\n",
      "Requirement already satisfied, skipping upgrade: numpy>=1.20 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (1.21.6)\n",
      "Requirement already satisfied, skipping upgrade: typing-extensions>=3.6.6 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (4.3.0)\n",
      "Requirement already satisfied, skipping upgrade: astunparse>=1.6.0 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (1.6.3)\n",
      "Requirement already satisfied, skipping upgrade: keras<2.11,>=2.10.0 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (2.10.0)\n",
      "Requirement already satisfied, skipping upgrade: wrapt>=1.11.0 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (1.12.1)\n",
      "Requirement already satisfied, skipping upgrade: grpcio<2.0,>=1.24.3 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorflow) (1.47.0)\n",
      "Requirement already satisfied, skipping upgrade: pyparsing!=3.0.5,>=2.0.2 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from packaging->tensorflow) (3.0.9)\n",
      "Requirement already satisfied, skipping upgrade: wheel>=0.26 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorboard<2.11,>=2.10->tensorflow) (0.37.1)\n",
      "Requirement already satisfied, skipping upgrade: google-auth-oauthlib<0.5,>=0.4.1 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorboard<2.11,>=2.10->tensorflow) (0.4.6)\n",
      "Requirement already satisfied, skipping upgrade: tensorboard-plugin-wit>=1.6.0 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorboard<2.11,>=2.10->tensorflow) (1.8.1)\n",
      "Requirement already satisfied, skipping upgrade: requests<3,>=2.21.0 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorboard<2.11,>=2.10->tensorflow) (2.28.1)\n",
      "Requirement already satisfied, skipping upgrade: werkzeug>=1.0.1 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorboard<2.11,>=2.10->tensorflow) (1.0.1)\n",
      "Requirement already satisfied, skipping upgrade: google-auth<3,>=1.6.3 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorboard<2.11,>=2.10->tensorflow) (2.9.1)\n",
      "Requirement already satisfied, skipping upgrade: tensorboard-data-server<0.7.0,>=0.6.0 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorboard<2.11,>=2.10->tensorflow) (0.6.1)\n",
      "Requirement already satisfied, skipping upgrade: markdown>=2.6.8 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from tensorboard<2.11,>=2.10->tensorflow) (3.4.1)\n",
      "Requirement already satisfied, skipping upgrade: requests-oauthlib>=0.7.0 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.11,>=2.10->tensorflow) (1.3.1)\n",
      "Requirement already satisfied, skipping upgrade: charset-normalizer<3,>=2 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<2.11,>=2.10->tensorflow) (2.1.0)\n",
      "Requirement already satisfied, skipping upgrade: urllib3<1.27,>=1.21.1 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<2.11,>=2.10->tensorflow) (1.26.9)\n",
      "Requirement already satisfied, skipping upgrade: idna<4,>=2.5 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<2.11,>=2.10->tensorflow) (3.3)\n",
      "Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<2.11,>=2.10->tensorflow) (2022.6.15)\n",
      "Requirement already satisfied, skipping upgrade: pyasn1-modules>=0.2.1 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard<2.11,>=2.10->tensorflow) (0.2.8)\n",
      "Requirement already satisfied, skipping upgrade: rsa<5,>=3.1.4; python_version >= \"3.6\" in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard<2.11,>=2.10->tensorflow) (4.9)\n",
      "Requirement already satisfied, skipping upgrade: cachetools<6.0,>=2.0.0 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard<2.11,>=2.10->tensorflow) (5.2.0)\n",
      "Requirement already satisfied, skipping upgrade: importlib-metadata>=4.4; python_version < \"3.10\" in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from markdown>=2.6.8->tensorboard<2.11,>=2.10->tensorflow) (4.12.0)\n",
      "Requirement already satisfied, skipping upgrade: oauthlib>=3.0.0 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.11,>=2.10->tensorflow) (3.2.0)\n",
      "Requirement already satisfied, skipping upgrade: pyasn1<0.5.0,>=0.4.6 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.11,>=2.10->tensorflow) (0.4.8)\n",
      "Requirement already satisfied, skipping upgrade: zipp>=0.5 in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from importlib-metadata>=4.4; python_version < \"3.10\"->markdown>=2.6.8->tensorboard<2.11,>=2.10->tensorflow) (3.8.1)\n"
     ]
    }
   ],
   "source": [
    "!pip install --upgrade tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "gather": {
     "logged": 1667461462827
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-11-03 07:44:15.511351: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2022-11-03 07:44:16.094454: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2022-11-03 07:44:18.837429: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory\n",
      "2022-11-03 07:44:18.837578: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory\n",
      "2022-11-03 07:44:18.837595: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow version: 2.10.0\n",
      "Keras version: 2.10.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow\n",
    "from  tensorflow import keras\n",
    "print('TensorFlow version:',tensorflow.__version__)\n",
    "print('Keras version:',keras.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare the base model\n",
    "\n",
    "To use transfer learning, we need a base model from which we can use the trained feature extraction layers. The ***resnet*** model is an CNN-based image classifier that has been pre-trained using a huge dataset of 3-color channel images of 224x224 pixels. Let's create an instance of it with some pretrained weights, excluding its final (top) prediction layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "gather": {
     "logged": 1667461477392
    },
    "tags": [
     "outputPrepend"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-11-03 07:44:23.392477: E tensorflow/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected\n",
      "2022-11-03 07:44:23.392559: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (emmanguyen06021): /proc/driver/nvidia/version does not exist\n",
      "2022-11-03 07:44:23.393417: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/resnet/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5\n",
      "94765736/94765736 [==============================] - 5s 0us/step\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
      "Model: \"resnet50\"\n",
      "__________________________________________________________________________________________________\n",
      " Layer (type)                   Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      " input_1 (InputLayer)           [(None, 224, 224, 3  0           []                               \n",
      "                                )]                                                                \n",
      "                                                                                                  \n",
      " conv1_pad (ZeroPadding2D)      (None, 230, 230, 3)  0           ['input_1[0][0]']                \n",
      "                                                                                                  \n",
      " conv1_conv (Conv2D)            (None, 112, 112, 64  9472        ['conv1_pad[0][0]']              \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv1_bn (BatchNormalization)  (None, 112, 112, 64  256         ['conv1_conv[0][0]']             \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv1_relu (Activation)        (None, 112, 112, 64  0           ['conv1_bn[0][0]']               \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " pool1_pad (ZeroPadding2D)      (None, 114, 114, 64  0           ['conv1_relu[0][0]']             \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " pool1_pool (MaxPooling2D)      (None, 56, 56, 64)   0           ['pool1_pad[0][0]']              \n",
      "                                                                                                  \n",
      " conv2_block1_1_conv (Conv2D)   (None, 56, 56, 64)   4160        ['pool1_pool[0][0]']             \n",
      "                                                                                                  \n",
      " conv2_block1_1_bn (BatchNormal  (None, 56, 56, 64)  256         ['conv2_block1_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block1_1_relu (Activatio  (None, 56, 56, 64)  0           ['conv2_block1_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv2_block1_2_conv (Conv2D)   (None, 56, 56, 64)   36928       ['conv2_block1_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv2_block1_2_bn (BatchNormal  (None, 56, 56, 64)  256         ['conv2_block1_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block1_2_relu (Activatio  (None, 56, 56, 64)  0           ['conv2_block1_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv2_block1_0_conv (Conv2D)   (None, 56, 56, 256)  16640       ['pool1_pool[0][0]']             \n",
      "                                                                                                  \n",
      " conv2_block1_3_conv (Conv2D)   (None, 56, 56, 256)  16640       ['conv2_block1_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv2_block1_0_bn (BatchNormal  (None, 56, 56, 256)  1024       ['conv2_block1_0_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block1_3_bn (BatchNormal  (None, 56, 56, 256)  1024       ['conv2_block1_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block1_add (Add)         (None, 56, 56, 256)  0           ['conv2_block1_0_bn[0][0]',      \n",
      "                                                                  'conv2_block1_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv2_block1_out (Activation)  (None, 56, 56, 256)  0           ['conv2_block1_add[0][0]']       \n",
      "                                                                                                  \n",
      " conv2_block2_1_conv (Conv2D)   (None, 56, 56, 64)   16448       ['conv2_block1_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv2_block2_1_bn (BatchNormal  (None, 56, 56, 64)  256         ['conv2_block2_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block2_1_relu (Activatio  (None, 56, 56, 64)  0           ['conv2_block2_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv2_block2_2_conv (Conv2D)   (None, 56, 56, 64)   36928       ['conv2_block2_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv2_block2_2_bn (BatchNormal  (None, 56, 56, 64)  256         ['conv2_block2_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block2_2_relu (Activatio  (None, 56, 56, 64)  0           ['conv2_block2_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv2_block2_3_conv (Conv2D)   (None, 56, 56, 256)  16640       ['conv2_block2_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv2_block2_3_bn (BatchNormal  (None, 56, 56, 256)  1024       ['conv2_block2_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block2_add (Add)         (None, 56, 56, 256)  0           ['conv2_block1_out[0][0]',       \n",
      "                                                                  'conv2_block2_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv2_block2_out (Activation)  (None, 56, 56, 256)  0           ['conv2_block2_add[0][0]']       \n",
      "                                                                                                  \n",
      " conv2_block3_1_conv (Conv2D)   (None, 56, 56, 64)   16448       ['conv2_block2_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv2_block3_1_bn (BatchNormal  (None, 56, 56, 64)  256         ['conv2_block3_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block3_1_relu (Activatio  (None, 56, 56, 64)  0           ['conv2_block3_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv2_block3_2_conv (Conv2D)   (None, 56, 56, 64)   36928       ['conv2_block3_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv2_block3_2_bn (BatchNormal  (None, 56, 56, 64)  256         ['conv2_block3_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block3_2_relu (Activatio  (None, 56, 56, 64)  0           ['conv2_block3_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv2_block3_3_conv (Conv2D)   (None, 56, 56, 256)  16640       ['conv2_block3_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv2_block3_3_bn (BatchNormal  (None, 56, 56, 256)  1024       ['conv2_block3_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block3_add (Add)         (None, 56, 56, 256)  0           ['conv2_block2_out[0][0]',       \n",
      "                                                                  'conv2_block3_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv2_block3_out (Activation)  (None, 56, 56, 256)  0           ['conv2_block3_add[0][0]']       \n",
      "                                                                                                  \n",
      " conv3_block1_1_conv (Conv2D)   (None, 28, 28, 128)  32896       ['conv2_block3_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv3_block1_1_bn (BatchNormal  (None, 28, 28, 128)  512        ['conv3_block1_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block1_1_relu (Activatio  (None, 28, 28, 128)  0          ['conv3_block1_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv3_block1_2_conv (Conv2D)   (None, 28, 28, 128)  147584      ['conv3_block1_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv3_block1_2_bn (BatchNormal  (None, 28, 28, 128)  512        ['conv3_block1_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block1_2_relu (Activatio  (None, 28, 28, 128)  0          ['conv3_block1_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv3_block1_0_conv (Conv2D)   (None, 28, 28, 512)  131584      ['conv2_block3_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv3_block1_3_conv (Conv2D)   (None, 28, 28, 512)  66048       ['conv3_block1_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv3_block1_0_bn (BatchNormal  (None, 28, 28, 512)  2048       ['conv3_block1_0_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block1_3_bn (BatchNormal  (None, 28, 28, 512)  2048       ['conv3_block1_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block1_add (Add)         (None, 28, 28, 512)  0           ['conv3_block1_0_bn[0][0]',      \n",
      "                                                                  'conv3_block1_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv3_block1_out (Activation)  (None, 28, 28, 512)  0           ['conv3_block1_add[0][0]']       \n",
      "                                                                                                  \n",
      " conv3_block2_1_conv (Conv2D)   (None, 28, 28, 128)  65664       ['conv3_block1_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv3_block2_1_bn (BatchNormal  (None, 28, 28, 128)  512        ['conv3_block2_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block2_1_relu (Activatio  (None, 28, 28, 128)  0          ['conv3_block2_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv3_block2_2_conv (Conv2D)   (None, 28, 28, 128)  147584      ['conv3_block2_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv3_block2_2_bn (BatchNormal  (None, 28, 28, 128)  512        ['conv3_block2_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block2_2_relu (Activatio  (None, 28, 28, 128)  0          ['conv3_block2_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv3_block2_3_conv (Conv2D)   (None, 28, 28, 512)  66048       ['conv3_block2_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv3_block2_3_bn (BatchNormal  (None, 28, 28, 512)  2048       ['conv3_block2_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block2_add (Add)         (None, 28, 28, 512)  0           ['conv3_block1_out[0][0]',       \n",
      "                                                                  'conv3_block2_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv3_block2_out (Activation)  (None, 28, 28, 512)  0           ['conv3_block2_add[0][0]']       \n",
      "                                                                                                  \n",
      " conv3_block3_1_conv (Conv2D)   (None, 28, 28, 128)  65664       ['conv3_block2_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv3_block3_1_bn (BatchNormal  (None, 28, 28, 128)  512        ['conv3_block3_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block3_1_relu (Activatio  (None, 28, 28, 128)  0          ['conv3_block3_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv3_block3_2_conv (Conv2D)   (None, 28, 28, 128)  147584      ['conv3_block3_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv3_block3_2_bn (BatchNormal  (None, 28, 28, 128)  512        ['conv3_block3_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block3_2_relu (Activatio  (None, 28, 28, 128)  0          ['conv3_block3_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv3_block3_3_conv (Conv2D)   (None, 28, 28, 512)  66048       ['conv3_block3_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv3_block3_3_bn (BatchNormal  (None, 28, 28, 512)  2048       ['conv3_block3_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block3_add (Add)         (None, 28, 28, 512)  0           ['conv3_block2_out[0][0]',       \n",
      "                                                                  'conv3_block3_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv3_block3_out (Activation)  (None, 28, 28, 512)  0           ['conv3_block3_add[0][0]']       \n",
      "                                                                                                  \n",
      " conv3_block4_1_conv (Conv2D)   (None, 28, 28, 128)  65664       ['conv3_block3_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv3_block4_1_bn (BatchNormal  (None, 28, 28, 128)  512        ['conv3_block4_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block4_1_relu (Activatio  (None, 28, 28, 128)  0          ['conv3_block4_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv3_block4_2_conv (Conv2D)   (None, 28, 28, 128)  147584      ['conv3_block4_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv3_block4_2_bn (BatchNormal  (None, 28, 28, 128)  512        ['conv3_block4_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block4_2_relu (Activatio  (None, 28, 28, 128)  0          ['conv3_block4_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv3_block4_3_conv (Conv2D)   (None, 28, 28, 512)  66048       ['conv3_block4_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv3_block4_3_bn (BatchNormal  (None, 28, 28, 512)  2048       ['conv3_block4_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block4_add (Add)         (None, 28, 28, 512)  0           ['conv3_block3_out[0][0]',       \n",
      "                                                                  'conv3_block4_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv3_block4_out (Activation)  (None, 28, 28, 512)  0           ['conv3_block4_add[0][0]']       \n",
      "                                                                                                  \n",
      " conv4_block1_1_conv (Conv2D)   (None, 14, 14, 256)  131328      ['conv3_block4_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv4_block1_1_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block1_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block1_1_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block1_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block1_2_conv (Conv2D)   (None, 14, 14, 256)  590080      ['conv4_block1_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv4_block1_2_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block1_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block1_2_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block1_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block1_0_conv (Conv2D)   (None, 14, 14, 1024  525312      ['conv3_block4_out[0][0]']       \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block1_3_conv (Conv2D)   (None, 14, 14, 1024  263168      ['conv4_block1_2_relu[0][0]']    \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block1_0_bn (BatchNormal  (None, 14, 14, 1024  4096       ['conv4_block1_0_conv[0][0]']    \n",
      " ization)                       )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block1_3_bn (BatchNormal  (None, 14, 14, 1024  4096       ['conv4_block1_3_conv[0][0]']    \n",
      " ization)                       )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block1_add (Add)         (None, 14, 14, 1024  0           ['conv4_block1_0_bn[0][0]',      \n",
      "                                )                                 'conv4_block1_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv4_block1_out (Activation)  (None, 14, 14, 1024  0           ['conv4_block1_add[0][0]']       \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block2_1_conv (Conv2D)   (None, 14, 14, 256)  262400      ['conv4_block1_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv4_block2_1_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block2_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block2_1_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block2_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block2_2_conv (Conv2D)   (None, 14, 14, 256)  590080      ['conv4_block2_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv4_block2_2_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block2_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block2_2_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block2_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block2_3_conv (Conv2D)   (None, 14, 14, 1024  263168      ['conv4_block2_2_relu[0][0]']    \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block2_3_bn (BatchNormal  (None, 14, 14, 1024  4096       ['conv4_block2_3_conv[0][0]']    \n",
      " ization)                       )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block2_add (Add)         (None, 14, 14, 1024  0           ['conv4_block1_out[0][0]',       \n",
      "                                )                                 'conv4_block2_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv4_block2_out (Activation)  (None, 14, 14, 1024  0           ['conv4_block2_add[0][0]']       \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block3_1_conv (Conv2D)   (None, 14, 14, 256)  262400      ['conv4_block2_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv4_block3_1_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block3_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block3_1_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block3_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block3_2_conv (Conv2D)   (None, 14, 14, 256)  590080      ['conv4_block3_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv4_block3_2_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block3_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block3_2_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block3_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block3_3_conv (Conv2D)   (None, 14, 14, 1024  263168      ['conv4_block3_2_relu[0][0]']    \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block3_3_bn (BatchNormal  (None, 14, 14, 1024  4096       ['conv4_block3_3_conv[0][0]']    \n",
      " ization)                       )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block3_add (Add)         (None, 14, 14, 1024  0           ['conv4_block2_out[0][0]',       \n",
      "                                )                                 'conv4_block3_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv4_block3_out (Activation)  (None, 14, 14, 1024  0           ['conv4_block3_add[0][0]']       \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block4_1_conv (Conv2D)   (None, 14, 14, 256)  262400      ['conv4_block3_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv4_block4_1_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block4_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block4_1_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block4_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block4_2_conv (Conv2D)   (None, 14, 14, 256)  590080      ['conv4_block4_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv4_block4_2_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block4_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block4_2_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block4_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block4_3_conv (Conv2D)   (None, 14, 14, 1024  263168      ['conv4_block4_2_relu[0][0]']    \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block4_3_bn (BatchNormal  (None, 14, 14, 1024  4096       ['conv4_block4_3_conv[0][0]']    \n",
      " ization)                       )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block4_add (Add)         (None, 14, 14, 1024  0           ['conv4_block3_out[0][0]',       \n",
      "                                )                                 'conv4_block4_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv4_block4_out (Activation)  (None, 14, 14, 1024  0           ['conv4_block4_add[0][0]']       \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block5_1_conv (Conv2D)   (None, 14, 14, 256)  262400      ['conv4_block4_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv4_block5_1_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block5_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block5_1_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block5_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block5_2_conv (Conv2D)   (None, 14, 14, 256)  590080      ['conv4_block5_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv4_block5_2_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block5_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block5_2_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block5_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block5_3_conv (Conv2D)   (None, 14, 14, 1024  263168      ['conv4_block5_2_relu[0][0]']    \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block5_3_bn (BatchNormal  (None, 14, 14, 1024  4096       ['conv4_block5_3_conv[0][0]']    \n",
      " ization)                       )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block5_add (Add)         (None, 14, 14, 1024  0           ['conv4_block4_out[0][0]',       \n",
      "                                )                                 'conv4_block5_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv4_block5_out (Activation)  (None, 14, 14, 1024  0           ['conv4_block5_add[0][0]']       \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block6_1_conv (Conv2D)   (None, 14, 14, 256)  262400      ['conv4_block5_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv4_block6_1_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block6_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block6_1_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block6_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block6_2_conv (Conv2D)   (None, 14, 14, 256)  590080      ['conv4_block6_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv4_block6_2_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block6_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block6_2_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block6_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block6_3_conv (Conv2D)   (None, 14, 14, 1024  263168      ['conv4_block6_2_relu[0][0]']    \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block6_3_bn (BatchNormal  (None, 14, 14, 1024  4096       ['conv4_block6_3_conv[0][0]']    \n",
      " ization)                       )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block6_add (Add)         (None, 14, 14, 1024  0           ['conv4_block5_out[0][0]',       \n",
      "                                )                                 'conv4_block6_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv4_block6_out (Activation)  (None, 14, 14, 1024  0           ['conv4_block6_add[0][0]']       \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv5_block1_1_conv (Conv2D)   (None, 7, 7, 512)    524800      ['conv4_block6_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv5_block1_1_bn (BatchNormal  (None, 7, 7, 512)   2048        ['conv5_block1_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block1_1_relu (Activatio  (None, 7, 7, 512)   0           ['conv5_block1_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv5_block1_2_conv (Conv2D)   (None, 7, 7, 512)    2359808     ['conv5_block1_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv5_block1_2_bn (BatchNormal  (None, 7, 7, 512)   2048        ['conv5_block1_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block1_2_relu (Activatio  (None, 7, 7, 512)   0           ['conv5_block1_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv5_block1_0_conv (Conv2D)   (None, 7, 7, 2048)   2099200     ['conv4_block6_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv5_block1_3_conv (Conv2D)   (None, 7, 7, 2048)   1050624     ['conv5_block1_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv5_block1_0_bn (BatchNormal  (None, 7, 7, 2048)  8192        ['conv5_block1_0_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block1_3_bn (BatchNormal  (None, 7, 7, 2048)  8192        ['conv5_block1_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block1_add (Add)         (None, 7, 7, 2048)   0           ['conv5_block1_0_bn[0][0]',      \n",
      "                                                                  'conv5_block1_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv5_block1_out (Activation)  (None, 7, 7, 2048)   0           ['conv5_block1_add[0][0]']       \n",
      "                                                                                                  \n",
      " conv5_block2_1_conv (Conv2D)   (None, 7, 7, 512)    1049088     ['conv5_block1_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv5_block2_1_bn (BatchNormal  (None, 7, 7, 512)   2048        ['conv5_block2_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block2_1_relu (Activatio  (None, 7, 7, 512)   0           ['conv5_block2_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv5_block2_2_conv (Conv2D)   (None, 7, 7, 512)    2359808     ['conv5_block2_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv5_block2_2_bn (BatchNormal  (None, 7, 7, 512)   2048        ['conv5_block2_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block2_2_relu (Activatio  (None, 7, 7, 512)   0           ['conv5_block2_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv5_block2_3_conv (Conv2D)   (None, 7, 7, 2048)   1050624     ['conv5_block2_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv5_block2_3_bn (BatchNormal  (None, 7, 7, 2048)  8192        ['conv5_block2_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block2_add (Add)         (None, 7, 7, 2048)   0           ['conv5_block1_out[0][0]',       \n",
      "                                                                  'conv5_block2_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv5_block2_out (Activation)  (None, 7, 7, 2048)   0           ['conv5_block2_add[0][0]']       \n",
      "                                                                                                  \n",
      " conv5_block3_1_conv (Conv2D)   (None, 7, 7, 512)    1049088     ['conv5_block2_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv5_block3_1_bn (BatchNormal  (None, 7, 7, 512)   2048        ['conv5_block3_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block3_1_relu (Activatio  (None, 7, 7, 512)   0           ['conv5_block3_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv5_block3_2_conv (Conv2D)   (None, 7, 7, 512)    2359808     ['conv5_block3_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv5_block3_2_bn (BatchNormal  (None, 7, 7, 512)   2048        ['conv5_block3_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block3_2_relu (Activatio  (None, 7, 7, 512)   0           ['conv5_block3_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv5_block3_3_conv (Conv2D)   (None, 7, 7, 2048)   1050624     ['conv5_block3_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv5_block3_3_bn (BatchNormal  (None, 7, 7, 2048)  8192        ['conv5_block3_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block3_add (Add)         (None, 7, 7, 2048)   0           ['conv5_block2_out[0][0]',       \n",
      "                                                                  'conv5_block3_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv5_block3_out (Activation)  (None, 7, 7, 2048)   0           ['conv5_block3_add[0][0]']       \n",
      "                                                                                                  \n",
      "==================================================================================================\n",
      "Total params: 23,587,712\n",
      "Trainable params: 23,534,592\n",
      "Non-trainable params: 53,120\n",
      "__________________________________________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "base_model = keras.applications.resnet.ResNet50(weights='imagenet', include_top=False, input_shape=(224,224,3))\n",
    "print(base_model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare the image data\n",
    "\n",
    "The pretrained model has many layers, starting with a convolutional layer that starts the feature extraction process from image data.\n",
    "\n",
    "For feature extraction to work with our own images, we  need to ensure that the image data we use to train our prediction layer has the same number of features (pixel values) as the images originally used to train the feature extraction layers, so we need data loaders for color images that are 224x224 pixels in size.\n",
    "\n",
    "Tensorflow includes functions for loading and transforming data. We'll use these to create a generator for training data, and a second generator for test data (which we'll use to validate the trained model). The loaders will transform the image data to match the format used to train the original resnet CNN model and normalize them.\n",
    "\n",
    "Run the following cell to define the data generators and list the classes for our images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "gather": {
     "logged": 1667461477571
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Getting Data...\n",
      "Preparing training dataset...\n",
      "Found 840 images belonging to 3 classes.\n",
      "Preparing validation dataset...\n",
      "Found 360 images belonging to 3 classes.\n",
      "class names:  ['circle', 'square', 'triangle']\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
    "\n",
    "data_folder = 'data/shapes'\n",
    "pretrained_size = (224,224)\n",
    "batch_size = 30\n",
    "\n",
    "print(\"Getting Data...\")\n",
    "datagen = ImageDataGenerator(rescale=1./255, # normalize pixel values\n",
    "                             validation_split=0.3) # hold back 30% of the images for validation\n",
    "\n",
    "print(\"Preparing training dataset...\")\n",
    "train_generator = datagen.flow_from_directory(\n",
    "    data_folder,\n",
    "    target_size=pretrained_size, # resize to match model expected input\n",
    "    batch_size=batch_size,\n",
    "    class_mode='categorical',\n",
    "    subset='training') # set as training data\n",
    "\n",
    "print(\"Preparing validation dataset...\")\n",
    "validation_generator = datagen.flow_from_directory(\n",
    "    data_folder,\n",
    "    target_size=pretrained_size, # resize to match model expected input\n",
    "    batch_size=batch_size,\n",
    "    class_mode='categorical',\n",
    "    subset='validation') # set as validation data\n",
    "\n",
    "classnames = list(train_generator.class_indices.keys())\n",
    "print(\"class names: \", classnames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a prediction layer\n",
    "\n",
    "We downloaded the complete *resnet* model excluding its final prediction layer, so need to combine these layers with a fully-connected (*dense*) layer that takes the flattened outputs from the feature extraction layers and generates a prediction for each of our image classes.\n",
    "\n",
    "We also need to freeze the feature extraction layers to retain the trained weights. Then when we train the model using our images, only the final prediction layer will learn new weight and bias values - the pre-trained weights already learned for feature extraction will remain the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "gather": {
     "logged": 1667461479931
    },
    "tags": [
     "outputPrepend"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      " Layer (type)                   Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      " input_1 (InputLayer)           [(None, 224, 224, 3  0           []                               \n",
      "                                )]                                                                \n",
      "                                                                                                  \n",
      " conv1_pad (ZeroPadding2D)      (None, 230, 230, 3)  0           ['input_1[0][0]']                \n",
      "                                                                                                  \n",
      " conv1_conv (Conv2D)            (None, 112, 112, 64  9472        ['conv1_pad[0][0]']              \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv1_bn (BatchNormalization)  (None, 112, 112, 64  256         ['conv1_conv[0][0]']             \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv1_relu (Activation)        (None, 112, 112, 64  0           ['conv1_bn[0][0]']               \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " pool1_pad (ZeroPadding2D)      (None, 114, 114, 64  0           ['conv1_relu[0][0]']             \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " pool1_pool (MaxPooling2D)      (None, 56, 56, 64)   0           ['pool1_pad[0][0]']              \n",
      "                                                                                                  \n",
      " conv2_block1_1_conv (Conv2D)   (None, 56, 56, 64)   4160        ['pool1_pool[0][0]']             \n",
      "                                                                                                  \n",
      " conv2_block1_1_bn (BatchNormal  (None, 56, 56, 64)  256         ['conv2_block1_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block1_1_relu (Activatio  (None, 56, 56, 64)  0           ['conv2_block1_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv2_block1_2_conv (Conv2D)   (None, 56, 56, 64)   36928       ['conv2_block1_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv2_block1_2_bn (BatchNormal  (None, 56, 56, 64)  256         ['conv2_block1_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block1_2_relu (Activatio  (None, 56, 56, 64)  0           ['conv2_block1_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv2_block1_0_conv (Conv2D)   (None, 56, 56, 256)  16640       ['pool1_pool[0][0]']             \n",
      "                                                                                                  \n",
      " conv2_block1_3_conv (Conv2D)   (None, 56, 56, 256)  16640       ['conv2_block1_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv2_block1_0_bn (BatchNormal  (None, 56, 56, 256)  1024       ['conv2_block1_0_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block1_3_bn (BatchNormal  (None, 56, 56, 256)  1024       ['conv2_block1_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block1_add (Add)         (None, 56, 56, 256)  0           ['conv2_block1_0_bn[0][0]',      \n",
      "                                                                  'conv2_block1_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv2_block1_out (Activation)  (None, 56, 56, 256)  0           ['conv2_block1_add[0][0]']       \n",
      "                                                                                                  \n",
      " conv2_block2_1_conv (Conv2D)   (None, 56, 56, 64)   16448       ['conv2_block1_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv2_block2_1_bn (BatchNormal  (None, 56, 56, 64)  256         ['conv2_block2_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block2_1_relu (Activatio  (None, 56, 56, 64)  0           ['conv2_block2_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv2_block2_2_conv (Conv2D)   (None, 56, 56, 64)   36928       ['conv2_block2_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv2_block2_2_bn (BatchNormal  (None, 56, 56, 64)  256         ['conv2_block2_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block2_2_relu (Activatio  (None, 56, 56, 64)  0           ['conv2_block2_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv2_block2_3_conv (Conv2D)   (None, 56, 56, 256)  16640       ['conv2_block2_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv2_block2_3_bn (BatchNormal  (None, 56, 56, 256)  1024       ['conv2_block2_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block2_add (Add)         (None, 56, 56, 256)  0           ['conv2_block1_out[0][0]',       \n",
      "                                                                  'conv2_block2_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv2_block2_out (Activation)  (None, 56, 56, 256)  0           ['conv2_block2_add[0][0]']       \n",
      "                                                                                                  \n",
      " conv2_block3_1_conv (Conv2D)   (None, 56, 56, 64)   16448       ['conv2_block2_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv2_block3_1_bn (BatchNormal  (None, 56, 56, 64)  256         ['conv2_block3_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block3_1_relu (Activatio  (None, 56, 56, 64)  0           ['conv2_block3_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv2_block3_2_conv (Conv2D)   (None, 56, 56, 64)   36928       ['conv2_block3_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv2_block3_2_bn (BatchNormal  (None, 56, 56, 64)  256         ['conv2_block3_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block3_2_relu (Activatio  (None, 56, 56, 64)  0           ['conv2_block3_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv2_block3_3_conv (Conv2D)   (None, 56, 56, 256)  16640       ['conv2_block3_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv2_block3_3_bn (BatchNormal  (None, 56, 56, 256)  1024       ['conv2_block3_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv2_block3_add (Add)         (None, 56, 56, 256)  0           ['conv2_block2_out[0][0]',       \n",
      "                                                                  'conv2_block3_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv2_block3_out (Activation)  (None, 56, 56, 256)  0           ['conv2_block3_add[0][0]']       \n",
      "                                                                                                  \n",
      " conv3_block1_1_conv (Conv2D)   (None, 28, 28, 128)  32896       ['conv2_block3_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv3_block1_1_bn (BatchNormal  (None, 28, 28, 128)  512        ['conv3_block1_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block1_1_relu (Activatio  (None, 28, 28, 128)  0          ['conv3_block1_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv3_block1_2_conv (Conv2D)   (None, 28, 28, 128)  147584      ['conv3_block1_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv3_block1_2_bn (BatchNormal  (None, 28, 28, 128)  512        ['conv3_block1_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block1_2_relu (Activatio  (None, 28, 28, 128)  0          ['conv3_block1_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv3_block1_0_conv (Conv2D)   (None, 28, 28, 512)  131584      ['conv2_block3_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv3_block1_3_conv (Conv2D)   (None, 28, 28, 512)  66048       ['conv3_block1_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv3_block1_0_bn (BatchNormal  (None, 28, 28, 512)  2048       ['conv3_block1_0_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block1_3_bn (BatchNormal  (None, 28, 28, 512)  2048       ['conv3_block1_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block1_add (Add)         (None, 28, 28, 512)  0           ['conv3_block1_0_bn[0][0]',      \n",
      "                                                                  'conv3_block1_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv3_block1_out (Activation)  (None, 28, 28, 512)  0           ['conv3_block1_add[0][0]']       \n",
      "                                                                                                  \n",
      " conv3_block2_1_conv (Conv2D)   (None, 28, 28, 128)  65664       ['conv3_block1_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv3_block2_1_bn (BatchNormal  (None, 28, 28, 128)  512        ['conv3_block2_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block2_1_relu (Activatio  (None, 28, 28, 128)  0          ['conv3_block2_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv3_block2_2_conv (Conv2D)   (None, 28, 28, 128)  147584      ['conv3_block2_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv3_block2_2_bn (BatchNormal  (None, 28, 28, 128)  512        ['conv3_block2_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block2_2_relu (Activatio  (None, 28, 28, 128)  0          ['conv3_block2_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv3_block2_3_conv (Conv2D)   (None, 28, 28, 512)  66048       ['conv3_block2_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv3_block2_3_bn (BatchNormal  (None, 28, 28, 512)  2048       ['conv3_block2_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block2_add (Add)         (None, 28, 28, 512)  0           ['conv3_block1_out[0][0]',       \n",
      "                                                                  'conv3_block2_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv3_block2_out (Activation)  (None, 28, 28, 512)  0           ['conv3_block2_add[0][0]']       \n",
      "                                                                                                  \n",
      " conv3_block3_1_conv (Conv2D)   (None, 28, 28, 128)  65664       ['conv3_block2_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv3_block3_1_bn (BatchNormal  (None, 28, 28, 128)  512        ['conv3_block3_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block3_1_relu (Activatio  (None, 28, 28, 128)  0          ['conv3_block3_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv3_block3_2_conv (Conv2D)   (None, 28, 28, 128)  147584      ['conv3_block3_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv3_block3_2_bn (BatchNormal  (None, 28, 28, 128)  512        ['conv3_block3_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block3_2_relu (Activatio  (None, 28, 28, 128)  0          ['conv3_block3_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv3_block3_3_conv (Conv2D)   (None, 28, 28, 512)  66048       ['conv3_block3_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv3_block3_3_bn (BatchNormal  (None, 28, 28, 512)  2048       ['conv3_block3_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block3_add (Add)         (None, 28, 28, 512)  0           ['conv3_block2_out[0][0]',       \n",
      "                                                                  'conv3_block3_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv3_block3_out (Activation)  (None, 28, 28, 512)  0           ['conv3_block3_add[0][0]']       \n",
      "                                                                                                  \n",
      " conv3_block4_1_conv (Conv2D)   (None, 28, 28, 128)  65664       ['conv3_block3_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv3_block4_1_bn (BatchNormal  (None, 28, 28, 128)  512        ['conv3_block4_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block4_1_relu (Activatio  (None, 28, 28, 128)  0          ['conv3_block4_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv3_block4_2_conv (Conv2D)   (None, 28, 28, 128)  147584      ['conv3_block4_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv3_block4_2_bn (BatchNormal  (None, 28, 28, 128)  512        ['conv3_block4_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block4_2_relu (Activatio  (None, 28, 28, 128)  0          ['conv3_block4_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv3_block4_3_conv (Conv2D)   (None, 28, 28, 512)  66048       ['conv3_block4_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv3_block4_3_bn (BatchNormal  (None, 28, 28, 512)  2048       ['conv3_block4_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv3_block4_add (Add)         (None, 28, 28, 512)  0           ['conv3_block3_out[0][0]',       \n",
      "                                                                  'conv3_block4_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv3_block4_out (Activation)  (None, 28, 28, 512)  0           ['conv3_block4_add[0][0]']       \n",
      "                                                                                                  \n",
      " conv4_block1_1_conv (Conv2D)   (None, 14, 14, 256)  131328      ['conv3_block4_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv4_block1_1_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block1_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block1_1_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block1_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block1_2_conv (Conv2D)   (None, 14, 14, 256)  590080      ['conv4_block1_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv4_block1_2_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block1_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block1_2_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block1_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block1_0_conv (Conv2D)   (None, 14, 14, 1024  525312      ['conv3_block4_out[0][0]']       \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block1_3_conv (Conv2D)   (None, 14, 14, 1024  263168      ['conv4_block1_2_relu[0][0]']    \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block1_0_bn (BatchNormal  (None, 14, 14, 1024  4096       ['conv4_block1_0_conv[0][0]']    \n",
      " ization)                       )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block1_3_bn (BatchNormal  (None, 14, 14, 1024  4096       ['conv4_block1_3_conv[0][0]']    \n",
      " ization)                       )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block1_add (Add)         (None, 14, 14, 1024  0           ['conv4_block1_0_bn[0][0]',      \n",
      "                                )                                 'conv4_block1_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv4_block1_out (Activation)  (None, 14, 14, 1024  0           ['conv4_block1_add[0][0]']       \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block2_1_conv (Conv2D)   (None, 14, 14, 256)  262400      ['conv4_block1_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv4_block2_1_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block2_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block2_1_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block2_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block2_2_conv (Conv2D)   (None, 14, 14, 256)  590080      ['conv4_block2_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv4_block2_2_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block2_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block2_2_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block2_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block2_3_conv (Conv2D)   (None, 14, 14, 1024  263168      ['conv4_block2_2_relu[0][0]']    \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block2_3_bn (BatchNormal  (None, 14, 14, 1024  4096       ['conv4_block2_3_conv[0][0]']    \n",
      " ization)                       )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block2_add (Add)         (None, 14, 14, 1024  0           ['conv4_block1_out[0][0]',       \n",
      "                                )                                 'conv4_block2_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv4_block2_out (Activation)  (None, 14, 14, 1024  0           ['conv4_block2_add[0][0]']       \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block3_1_conv (Conv2D)   (None, 14, 14, 256)  262400      ['conv4_block2_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv4_block3_1_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block3_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block3_1_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block3_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block3_2_conv (Conv2D)   (None, 14, 14, 256)  590080      ['conv4_block3_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv4_block3_2_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block3_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block3_2_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block3_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block3_3_conv (Conv2D)   (None, 14, 14, 1024  263168      ['conv4_block3_2_relu[0][0]']    \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block3_3_bn (BatchNormal  (None, 14, 14, 1024  4096       ['conv4_block3_3_conv[0][0]']    \n",
      " ization)                       )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block3_add (Add)         (None, 14, 14, 1024  0           ['conv4_block2_out[0][0]',       \n",
      "                                )                                 'conv4_block3_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv4_block3_out (Activation)  (None, 14, 14, 1024  0           ['conv4_block3_add[0][0]']       \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block4_1_conv (Conv2D)   (None, 14, 14, 256)  262400      ['conv4_block3_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv4_block4_1_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block4_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block4_1_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block4_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block4_2_conv (Conv2D)   (None, 14, 14, 256)  590080      ['conv4_block4_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv4_block4_2_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block4_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block4_2_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block4_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block4_3_conv (Conv2D)   (None, 14, 14, 1024  263168      ['conv4_block4_2_relu[0][0]']    \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block4_3_bn (BatchNormal  (None, 14, 14, 1024  4096       ['conv4_block4_3_conv[0][0]']    \n",
      " ization)                       )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block4_add (Add)         (None, 14, 14, 1024  0           ['conv4_block3_out[0][0]',       \n",
      "                                )                                 'conv4_block4_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv4_block4_out (Activation)  (None, 14, 14, 1024  0           ['conv4_block4_add[0][0]']       \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block5_1_conv (Conv2D)   (None, 14, 14, 256)  262400      ['conv4_block4_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv4_block5_1_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block5_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block5_1_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block5_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block5_2_conv (Conv2D)   (None, 14, 14, 256)  590080      ['conv4_block5_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv4_block5_2_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block5_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block5_2_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block5_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block5_3_conv (Conv2D)   (None, 14, 14, 1024  263168      ['conv4_block5_2_relu[0][0]']    \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block5_3_bn (BatchNormal  (None, 14, 14, 1024  4096       ['conv4_block5_3_conv[0][0]']    \n",
      " ization)                       )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block5_add (Add)         (None, 14, 14, 1024  0           ['conv4_block4_out[0][0]',       \n",
      "                                )                                 'conv4_block5_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv4_block5_out (Activation)  (None, 14, 14, 1024  0           ['conv4_block5_add[0][0]']       \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block6_1_conv (Conv2D)   (None, 14, 14, 256)  262400      ['conv4_block5_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv4_block6_1_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block6_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block6_1_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block6_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block6_2_conv (Conv2D)   (None, 14, 14, 256)  590080      ['conv4_block6_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv4_block6_2_bn (BatchNormal  (None, 14, 14, 256)  1024       ['conv4_block6_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv4_block6_2_relu (Activatio  (None, 14, 14, 256)  0          ['conv4_block6_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv4_block6_3_conv (Conv2D)   (None, 14, 14, 1024  263168      ['conv4_block6_2_relu[0][0]']    \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block6_3_bn (BatchNormal  (None, 14, 14, 1024  4096       ['conv4_block6_3_conv[0][0]']    \n",
      " ization)                       )                                                                 \n",
      "                                                                                                  \n",
      " conv4_block6_add (Add)         (None, 14, 14, 1024  0           ['conv4_block5_out[0][0]',       \n",
      "                                )                                 'conv4_block6_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv4_block6_out (Activation)  (None, 14, 14, 1024  0           ['conv4_block6_add[0][0]']       \n",
      "                                )                                                                 \n",
      "                                                                                                  \n",
      " conv5_block1_1_conv (Conv2D)   (None, 7, 7, 512)    524800      ['conv4_block6_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv5_block1_1_bn (BatchNormal  (None, 7, 7, 512)   2048        ['conv5_block1_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block1_1_relu (Activatio  (None, 7, 7, 512)   0           ['conv5_block1_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv5_block1_2_conv (Conv2D)   (None, 7, 7, 512)    2359808     ['conv5_block1_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv5_block1_2_bn (BatchNormal  (None, 7, 7, 512)   2048        ['conv5_block1_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block1_2_relu (Activatio  (None, 7, 7, 512)   0           ['conv5_block1_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv5_block1_0_conv (Conv2D)   (None, 7, 7, 2048)   2099200     ['conv4_block6_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv5_block1_3_conv (Conv2D)   (None, 7, 7, 2048)   1050624     ['conv5_block1_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv5_block1_0_bn (BatchNormal  (None, 7, 7, 2048)  8192        ['conv5_block1_0_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block1_3_bn (BatchNormal  (None, 7, 7, 2048)  8192        ['conv5_block1_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block1_add (Add)         (None, 7, 7, 2048)   0           ['conv5_block1_0_bn[0][0]',      \n",
      "                                                                  'conv5_block1_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv5_block1_out (Activation)  (None, 7, 7, 2048)   0           ['conv5_block1_add[0][0]']       \n",
      "                                                                                                  \n",
      " conv5_block2_1_conv (Conv2D)   (None, 7, 7, 512)    1049088     ['conv5_block1_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv5_block2_1_bn (BatchNormal  (None, 7, 7, 512)   2048        ['conv5_block2_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block2_1_relu (Activatio  (None, 7, 7, 512)   0           ['conv5_block2_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv5_block2_2_conv (Conv2D)   (None, 7, 7, 512)    2359808     ['conv5_block2_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv5_block2_2_bn (BatchNormal  (None, 7, 7, 512)   2048        ['conv5_block2_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block2_2_relu (Activatio  (None, 7, 7, 512)   0           ['conv5_block2_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv5_block2_3_conv (Conv2D)   (None, 7, 7, 2048)   1050624     ['conv5_block2_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv5_block2_3_bn (BatchNormal  (None, 7, 7, 2048)  8192        ['conv5_block2_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block2_add (Add)         (None, 7, 7, 2048)   0           ['conv5_block1_out[0][0]',       \n",
      "                                                                  'conv5_block2_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv5_block2_out (Activation)  (None, 7, 7, 2048)   0           ['conv5_block2_add[0][0]']       \n",
      "                                                                                                  \n",
      " conv5_block3_1_conv (Conv2D)   (None, 7, 7, 512)    1049088     ['conv5_block2_out[0][0]']       \n",
      "                                                                                                  \n",
      " conv5_block3_1_bn (BatchNormal  (None, 7, 7, 512)   2048        ['conv5_block3_1_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block3_1_relu (Activatio  (None, 7, 7, 512)   0           ['conv5_block3_1_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv5_block3_2_conv (Conv2D)   (None, 7, 7, 512)    2359808     ['conv5_block3_1_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv5_block3_2_bn (BatchNormal  (None, 7, 7, 512)   2048        ['conv5_block3_2_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block3_2_relu (Activatio  (None, 7, 7, 512)   0           ['conv5_block3_2_bn[0][0]']      \n",
      " n)                                                                                               \n",
      "                                                                                                  \n",
      " conv5_block3_3_conv (Conv2D)   (None, 7, 7, 2048)   1050624     ['conv5_block3_2_relu[0][0]']    \n",
      "                                                                                                  \n",
      " conv5_block3_3_bn (BatchNormal  (None, 7, 7, 2048)  8192        ['conv5_block3_3_conv[0][0]']    \n",
      " ization)                                                                                         \n",
      "                                                                                                  \n",
      " conv5_block3_add (Add)         (None, 7, 7, 2048)   0           ['conv5_block2_out[0][0]',       \n",
      "                                                                  'conv5_block3_3_bn[0][0]']      \n",
      "                                                                                                  \n",
      " conv5_block3_out (Activation)  (None, 7, 7, 2048)   0           ['conv5_block3_add[0][0]']       \n",
      "                                                                                                  \n",
      " flatten (Flatten)              (None, 100352)       0           ['conv5_block3_out[0][0]']       \n",
      "                                                                                                  \n",
      " dense (Dense)                  (None, 3)            301059      ['flatten[0][0]']                \n",
      "                                                                                                  \n",
      "==================================================================================================\n",
      "Total params: 23,888,771\n",
      "Trainable params: 301,059\n",
      "Non-trainable params: 23,587,712\n",
      "__________________________________________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras import applications\n",
    "from tensorflow.keras import Model\n",
    "from tensorflow.keras.layers import Flatten, Dense\n",
    "\n",
    "# Freeze the already-trained layers in the base model\n",
    "for layer in base_model.layers:\n",
    "    layer.trainable = False\n",
    "\n",
    "# Create prediction layer for classification of our images\n",
    "x = base_model.output\n",
    "x = Flatten()(x)\n",
    "prediction_layer = Dense(len(classnames), activation='softmax')(x) \n",
    "model = Model(inputs=base_model.input, outputs=prediction_layer)\n",
    "\n",
    "# Compile the model\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "# Now print the full model, which will include the layers of the base model plus the dense layer we added\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the Model\n",
    "\n",
    "With the layers of the CNN defined, we're ready to train it using our image data. The weights used in the feature extraction layers from the base resnet model will not be changed by training, only the final dense layer that maps the features to our shape classes will be trained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "gather": {
     "logged": 1667462283593
    },
    "scrolled": false,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/3\n",
      "28/28 [==============================] - 282s 10s/step - loss: 2.6059 - accuracy: 0.5357 - val_loss: 0.3406 - val_accuracy: 0.8722\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
      "Epoch 2/3\n",
      "28/28 [==============================] - 262s 9s/step - loss: 0.3097 - accuracy: 0.8702 - val_loss: 0.1868 - val_accuracy: 0.9556\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
      "Epoch 3/3\n",
      "28/28 [==============================] - 261s 9s/step - loss: 0.1482 - accuracy: 0.9798 - val_loss: 0.1528 - val_accuracy: 0.9528\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
     ]
    }
   ],
   "source": [
    "# Train the model over 3 epochs\n",
    "num_epochs = 3\n",
    "history = model.fit(\n",
    "    train_generator,\n",
    "    steps_per_epoch = train_generator.samples // batch_size,\n",
    "    validation_data = validation_generator, \n",
    "    validation_steps = validation_generator.samples // batch_size,\n",
    "    epochs = num_epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View the loss history\n",
    "\n",
    "We tracked average training and validation loss for each epoch. We can plot these to verify that the loss reduced over the training process and to detect *over-fitting* (which is indicated by a continued drop in training loss after validation loss has levelled out or started to increase)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "gather": {
     "logged": 1667462284640
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXxU9b3/8ddnJpM9kJV9VdlkTwJi0VarVtTWfcEFsFfkavVa23r7U9tKbbW3t9darysFtFeoG8W1Fde6UjcSBGQVFFRAloQlKwkJ398fM8AQEphgZk6SeT8fj3nkzDlnZj5zOMxnvuf7ne/HnHOIiEj88nkdgIiIeEuJQEQkzikRiIjEOSUCEZE4p0QgIhLnErwOoLlyc3Ndnz59vA5DRKRNKS4uLnHO5TW2rc0lgj59+lBUVOR1GCIibYqZfdHUNl0aEhGJc0oEIiJxTolARCTOtbk+AhFpX3bv3s369evZtWuX16G0C8nJyfTo0YNAIBDxY5QIRMRT69evJyMjgz59+mBmXofTpjnnKC0tZf369fTt2zfix+nSkIh4ateuXeTk5CgJtAAzIycnp9mtKyUCEfGckkDLOZJjGTeJYEv5Lm7/+zJq6/Z4HYqISKsSN4mgeN12/vKvdUx9YSmqwSAie+3YsYMHH3yw2Y8788wz2bFjxyH3ue2223j99dePNLSYiZtEcMbQrlx38tE88dFXPPreOq/DEZFWoqlEUFdXd8jHzZs3j8zMzEPu85vf/IZTTz31G8UXC3GTCAB+dtoATju2M799cQXzV5d4HY6ItAI333wzn332GSNGjGDUqFGceOKJnH322Rx77LEAnHvuuRQUFDB48GCmT5++73F9+vShpKSEdevWMWjQIK6++moGDx7M9773PaqrqwG48sormTt37r79p06dSn5+PkOHDmXlypUAbN26ldNOO43BgwczefJkevfuTUlJbD+f4mr4qM9n/OmSEVzw4Hv86LFinr/+BPrmpnkdloiE3P73ZSzfWNaiz3lstw5M/cHgJrf//ve/Z+nSpSxatIi33nqLs846i6VLl+4bfvnII4+QnZ1NdXU1o0aN4oILLiAnJ+eA51i9ejVPPPEEM2bM4OKLL+bpp5/miiuuOOi1cnNzWbhwIQ8++CB33XUXM2fO5Pbbb+e73/0ut9xyCy+//DIPP/xwi77/SMRViwAgPSmBmZMKSfD7mPzoAsp27fY6JBFpRUaPHn3AGPx7772X4cOHM2bMGL766itWr1590GP69u3LiBEjACgoKGDdunWNPvf5559/0D7z589n/PjxAIwbN46srKwWfDeRiasWwV49s1N58PJ8rpj5If/x+Mc8cuUo/D4NXxPx2qG+ucdKWtr+qwRvvfUWr7/+Ou+//z6pqamcdNJJjY7RT0pK2rfs9/v3XRpqaj+/33/YPohYilqLwMx6mtmbZrbczJaZ2Y8b2eckM9tpZotCt9uiFU9DY47K4fZzBvP2p1v575dXxuplRaSVycjIoLy8vNFtO3fuJCsri9TUVFauXMkHH3zQ4q8/duxY5syZA8Crr77K9u3bW/w1DieaLYI64GfOuYVmlgEUm9lrzrnlDfZ71zn3/SjG0aTLj+vNqk3lTH/nc/p3zuDCgh5ehCEiHsrJyWHs2LEMGTKElJQUOnfuvG/buHHjmDZtGoMGDWLAgAGMGTOmxV9/6tSpXHrppcyePZvjjz+eLl26kJGR0eKvcygWqzH1ZvY8cL9z7rWwdScBNzUnERQWFrqWLEyzu34Pkx75iKJ123liyhgKesf++pxIPFuxYgWDBg3yOgzP1NTU4Pf7SUhI4P333+faa69l0aJF3+g5GzumZlbsnCtsbP+YdBabWR9gJPBhI5uPN7PFZvaSmTV6gdDMpphZkZkVbd26tUVjC/h9PHh5Pl0zk/n32cVs3NH4tT0RkWj48ssvGTVqFMOHD+eGG25gxowZMY8h6onAzNKBp4EbnXMNx4UtBHo754YD9wHPNfYczrnpzrlC51xhXl6jJTe/kczURGZOLGTX7nqmzC6iura+xV9DRKQx/fr14+OPP2bx4sUsWLCAUaNGxTyGqCYCMwsQTAKPOeeeabjdOVfmnKsILc8DAmaWG82YmtKvcwb3XTqSZRvLuGnuYk1DISJxI5qjhgx4GFjhnLu7iX26hPbDzEaH4imNVkyHc/LATtw8biAvLvma+95Y41UYIiIxFc1RQ2OBCcAnZra35+NWoBeAc24acCFwrZnVAdXAeOfxV/Ep3z6KVZvKufu1T+nfOZ1xQ7p6GY6ISNRFLRE45+YDh/yVlnPufuD+aMVwJMyM350/lLWllfzkqcX0yk7j2G4dvA5LRCRq4m6KiUgkB/z8+YoCOqYEuHpWESUVNV6HJCKtRHp6OgAbN27kwgsvbHSfk046icMNc7/nnnuoqqradz+Saa2jRYmgCZ06JDNjYiGllTVc+9diFbQRkQN069Zt38yiR6JhIohkWutoUSI4hKE9OvI/Fw5nwbrt/Oo5FbQRaY9uvvlmHnjggX33f/3rX3PHHXdwyimn7Jsy+vnnnz/ocevWrWPIkCEAVFdXM378eAYNGsR55513wFxD1157LYWFhQwePJipU6cCwYnsNm7cyMknn8zJJ58M7J/WGuDuu+9myJAhDBkyhHvuuWff6zU13fU3FZeTzjXHD4Z349PN5dz3xhoGdMng307oe/gHiciReelm2PRJyz5nl6Fwxu+b3HzJJZdw4403ct111wEwZ84cXnnlFW644QY6dOhASUkJY8aM4eyzz26yHvBDDz1EamoqK1asYMmSJeTn5+/bduedd5KdnU19fT2nnHIKS5Ys4YYbbuDuu+/mzTffJDf3wBHzxcXF/OUvf+HDDz/EOcdxxx3Hd77zHbKysiKe7rq51CKIwE9O7c/pgztzx4vLeefTlv1ls4h4a+TIkWzZsoWNGzeyePFisrKy6NKlC7feeivDhg3j1FNPZcOGDWzevLnJ53jnnXf2fSAPGzaMYcOG7ds2Z84c8vPzGTlyJMuWLWP58obTrR1o/vz5nHfeeaSlpZGens7555/Pu+++C0Q+3XVzqUUQAZ/PuPviEVzw0Htc//hCnrtuLEflpXsdlkj7c4hv7tF00UUXMXfuXDZt2sQll1zCY489xtatWykuLiYQCNCnT59Gp58+nLVr13LXXXexYMECsrKyuPLKK4/oefaKdLrr5lKLIEJpSQnMmLi3oE0RO6tV0Eakvbjkkkt48sknmTt3LhdddBE7d+6kU6dOBAIB3nzzTb744otDPv7b3/42jz/+OABLly5lyZIlAJSVlZGWlkbHjh3ZvHkzL7300r7HNDX99Yknnshzzz1HVVUVlZWVPPvss5x44okt+G4PpkTQDD2zU5l2RQFfbqviP574mLp6jSQSaQ8GDx5MeXk53bt3p2vXrlx++eUUFRUxdOhQZs2axcCBAw/5+GuvvZaKigoGDRrEbbfdRkFBAQDDhw9n5MiRDBw4kMsuu4yxY8fue8yUKVMYN27cvs7ivfLz87nyyisZPXo0xx13HJMnT2bkyJEt/6bDxGwa6pbS0tNQH4knPvqSW575hMkn9OWX3z/W01hE2rp4n4Y6Gpo7DbX6CI7ApaN7sWpTOTPnr2VAlwwuKuzpdUgiIkdMl4aO0C/PGsQJx+Tyi2eXUvzFNq/DERE5YkoERyjB7+P+y0bSLVTQZoMK2ogcsbZ2ibo1O5JjqUTwDWSmJjJz0ihqdu/h6keLqKqt8zokkTYnOTmZ0tJSJYMW4JyjtLSU5OTkZj1OfQTf0DGd0rn3spFc9X8LuOlvi7n/0nx8vkNOuioiYXr06MH69etp6TK08So5OZkePXo06zFKBC3g5AGduOWMQdw5bwX3dl7Njaf29zokkTYjEAjQt6+mbvGSEkELmXxiX1ZtLuee11czoHMGZwxVQRsRaRvUR9BCzIw7zxtCfq9MfjpnMcs27vQ6JBGRiCgRtKCkBD/TJhSQmRrg6keL2FqugjYi0vopEbSwThnBgjbbqmq55q/F1NTVex2SiMghKRFEwZDuHfnjRSMo/mI7v3xWBW1EpHVTIoiSs4Z15YZT+vG34vU8PH+t1+GIiDRJiSCKbjylH2cM6cLv5q3gbRW0EZFWSokginw+448XD2dAlw5c//hCPtta4XVIIiIHUSKIstTEBGZMLCBxb0GbKhW0EZHWRYkgBnpkpTJtQgHrt1dx/RMLVdBGRFoVJYIYGdUnmzvOHcK7q0v43byVXocjIrKPppiIoUtG9WLlpnIe+ddaBnbJ4OJRKmgjIt5TiyDGfnHmIE7sl8svnvuEBetU0EZEvKdEEGMJfh/3X5pPz6xUrpldzPrtVV6HJCJxTonAAx1TA8yYVEht/R6unlVMZY0K2oiId5QIPHJ0Xjr3X5bPqk1l/GzOYvbs0TQUIuKNqCUCM+tpZm+a2XIzW2ZmP25kHzOze81sjZktMbP8aMXTGn2nfx63njmIl5dt4p5/rvY6HBGJU9EcNVQH/Mw5t9DMMoBiM3vNObc8bJ8zgH6h23HAQ6G/ceOqE/qyalM59/4zWNDmrGEqaCMisRW1FoFz7mvn3MLQcjmwAujeYLdzgFku6AMg08zi6pPQzLjjvCEU9M7iZ39bxNINKmgjIrEVkz4CM+sDjAQ+bLCpO/BV2P31HJwsMLMpZlZkZkXtscB1UoKfaVcUkJ2ayNWzithSvsvrkEQkjkQ9EZhZOvA0cKNzruxInsM5N905V+icK8zLy2vZAFuJvIwkZkwqZEfVbq6ZrYI2IhI7UU0EZhYgmAQec84908guG4Dwn9f2CK2LS4O7deTui4ez8Msd3PqMCtqISGxEc9SQAQ8DK5xzdzex2wvAxNDooTHATufc19GKqS04Y2hXbjy1H08vXM/Md1XQRkSiL5qjhsYCE4BPzGxRaN2tQC8A59w0YB5wJrAGqAJ+GMV42owbvtuPTzeX818vreCYzumcPKCT1yGJSDsWtUTgnJsP2GH2ccB10YqhrfL5jLsuGs66kipuePxjnr3uWxzTKcPrsESkndIvi1up1MQEZkwqJCkQLGizo6rW65BEpJ1SImjFumemMO2KAjbsqOb6xz9WQRsRiQolglausE82d543lPlrSrjjxRVehyMi7ZAK07QBFxf2ZNWmch6eHyxoM350L69DEpF2RC2CNuKWMwby7f55/Or5pXy0VgVtRKTlKBG0EQl+H/ddOpKe2alc89divtqmgjYi0jKUCNqQjikBZk4spK5+D1fPKlJBGxFpEUoEbcxRoYI2n24u5ydPLVJBGxH5xpQI2qBv98/jl2cdy6vLN/On1z/1OhwRaeM0aqiN+uHYPqzaVM59b6yhf+cMfjC8m9chiUgbpRZBG2Vm/PbcIYzqk8VNf1vMJ+tV0EZEjowSQRuWmODjoSsKyE1PCha0KVNBGxFpPiWCNi43PYkZEwsp27WbKbOL2bVbBW1EpHmUCNqBY7t14O6LR7Doqx3c+swnKmgjIs2iRNBOjBvShZ+e1p9nPt7A9Hc+9zocEWlDlAjakf/47jGcNawrv395JW+s3Ox1OCLSRigRtCNmxl0XDmdwtw7c8MQiVm8u9zokEWkDlAjamZREP9MnFJIc8DN5VhHbK1XQRkQOTYmgHeqWmcKfJxTw9Y5dXPf4QnaroI2IHIISQTtV0DuL350/lPc+K+WOfyz3OhwRacU0xUQ7dmFBD1ZtKmPGu2sZ0KUDlx2ngjYicjC1CNq5m88YxEkD8rjt+aV88Hmp1+GISCukRNDO+X3GvZeOpHdOKteqoI2INEKJIA50SA4wc9Io9jiY/GgRFSpoIyJhlAjiRN/cNB64LJ81WytU0EZEDqBEEEdO6JfLr84axGvLN/PH11Z5HY6ItBIaNRRnJn2rD6s2l/PAm5/Rv3MG54zo7nVIIuIxtQjijJlx+9lDGN03m5/PXcLir3Z4HZKIeEyJIA4lJvh46PJ88jKSmDK7iM0qaCMS15QI4lROqKBN+a46FbQRiXNKBHFsUNcO/OmSESz+agc3P71EBW1E4lTUEoGZPWJmW8xsaRPbTzKznWa2KHS7LVqxSNNOH9yFm77Xn+cWbWTa2ypoIxKPojlq6P+A+4FZh9jnXefc96MYg0TgupOPYdXmCv7wykr6dUrn1GM7ex2SiMRQ1FoEzrl3gG3Ren5pOWbGHy4YxpBuHfnxkx/zqQraiMQVr/sIjjezxWb2kpkNbmonM5tiZkVmVrR169ZYxhc3UhL9TJ9YQGpSApMfVUEbkXjiZSJYCPR2zg0H7gOea2pH59x051yhc64wLy8vZgHGm64dgwVtNpXt4trHilXQRiROeJYInHNlzrmK0PI8IGBmuV7FI0H5vbL4/flD+eDzbfzm7ypoIxIPPJtiwsy6AJudc87MRhNMSpowvxU4P78HqzaV8+d3PmdAlwyuGNPb65BEJIoiahGY2Y/NrIMFPWxmC83se4d5zBPA+8AAM1tvZleZ2TVmdk1olwuBpWa2GLgXGO80kL3V+Pm4gXx3YCd+/cIy3v9M+VmkPbNIPnvNbLFzbriZnQ78O/ArYLZzLj/aATZUWFjoioqKYv2ycal8127Oe/A9SipqeOG6E+iVk+p1SCJyhMys2DlX2Ni2SPsILPT3TIIJYFnYOmmnMpIDzJxYiHMwedYCynft9jokEYmCSBNBsZm9SjARvGJmGYCGlMSBPrlpPHR5Pp9treQnTy2iXgVtRNqdSBPBVcDNwCjnXBUQAH4YtaikVfnWMblM/cGxvL5iC3e9qoI2Iu1NpIngeGCVc26HmV0B/BLYGb2wpLWZMKY3lx3Xi4fe+oznPt7gdTgi0oIiTQQPAVVmNhz4GfAZh55DSNqZYEGbwRzXN5ufP72ERSpoI9JuRJoI6kJDO88B7nfOPQBkRC8saY0Cfh8PXVFA5w5JTJlVxKadKmgj0h5EmgjKzewWYALwopn5CPYTSJzJTktk5sRRVNbUMWV2kQraiLQDkSaCS4Aa4N+cc5uAHsD/RC0qadUGdMngnvEj+WTDTn4+VwVtRNq6iBJB6MP/MaCjmX0f2OWcUx9BHDvt2M7c9L0BvLB4Iw++9ZnX4YjINxDpFBMXAx8BFwEXAx+a2YXRDExavx+ddDTnjOjGXa+u4rXlm70OR0SOUKSXhn5B8DcEk5xzE4HRBKeZkDhmZvz3BcMY1r0jNz75MSs3lXkdkogcgUgTgc85tyXsfmkzHivtWHLAz58nFJIWKmizTQVtRNqcSD/MXzazV8zsSjO7EngRmBe9sKQt6dIxmekTC9lSXsO1fy2mtk6zj4i0JZF2Fv8nMB0YFrpNd879v2gGJm3LiJ6Z/OGCYXy4dhu3/32Z1+GISDNEXJjGOfc08HQUY5E27tyR3Vm5qZxpb3/GwC4ZTDi+j9chiUgEDpkIzKwcaGyQuAHOOdchKlFJm/Wfpw9g9eZyfv335Rydl863jlH1UZHW7pCXhpxzGc65Do3cMpQEpDF+n3HP+BEcnZfGjx5fyBellV6HJCKHoZE/0uKCBW1GAXDVo0UqaCPSyikRSFT0yknlwcvzWVdSyY+fVEEbkdZMiUCi5ltH5zL17MG8sXILf3hlpdfhiEgTIh41JHIkJozpzapNZfz57c8Z0DmD8/N7eB2SiDSgFoFE3dQfDOb4o3K4+ZlP+PjL7V6HIyINKBFI1AX8Ph68PJ8uHZKZMruYr3dWex2SiIRRIpCYyEpLZOakQqpr65kyq5jqWhW0EWktlAgkZvp3zuB/x49g6cad/PxpFbQRaS2UCCSmThnUmZ+fPpC/L97IA2+u8TocEUGjhsQD13znKD7dXM5dr35Kv84ZnD64i9chicQ1tQgk5syM/zp/KMN7ZvKTpxax4msVtBHxkhKBeCI54Gf6hAIykoMFbUorarwOSSRuKRGIZzp3SGb6hEJKKmq49rGFKmgj4hElAvHU8J6Z/OHCYXy0dhtTX1imkUQiHohaIjCzR8xsi5ktbWK7mdm9ZrbGzJaYWX60YpHW7ZwR3fnRSUfzxEdfMuv9L7wORyTuRLNF8H/AuENsPwPoF7pNAR6KYizSyt30vQGcOqgzv/nHcv61psTrcETiStQSgXPuHWDbIXY5B5jlgj4AMs2sa7TikdbNFypoc0xeOj96bCFrS1TQRiRWvOwj6A58FXZ/fWjdQcxsipkVmVnR1q1bYxKcxF56UgIzJxXiM5j86ALKVNBGJCbaRGexc266c67QOVeYl5fndTgSRT2zU3noigK+KK3ihic+VkEbkRjwMhFsAHqG3e8RWidxbsxROdx+zmDeWrWV/35ZBW1Eos3LRPACMDE0emgMsNM597WH8UgrcvlxvZl4fG+mv/M5c4vXex2OSLsWtbmGzOwJ4CQg18zWA1OBAIBzbhowDzgTWANUAT+MVizSNv3q+8eyZksFtz7zCX1z0yjoneV1SCLtkrW1H/AUFha6oqIir8OQGNlRVcs5D/yLypp6Xrh+LN0yU7wOSaRNMrNi51xhY9vaRGexxK/M1ERmTixk1+56pswuUkEbkShQIpBWr1/nDO67dCTLNpZx09zFmoZCpIUpEUibcPLATtw8biAvLvma+95QQRuRlqTCNNJmTPn2UazaVM7dr31K/87pjBuiH6KLtAS1CKTNMDN+d/5QRvTM5CdPLWb5RhW0EWkJSgTSpuwtaNMxJcDVs4ooUUEbkW9MiUDanE4dkpk+sSBY0OavxSpoI/INKRFImzSsRyZ3XTScBeu2c9vzSzWSSOQbUGextFk/GN6NVZvKuf/NNQzoksEPx/b1OiSRNkktAmnTfnpaf753bGd++4/lvLtaU5SLHAklAmnTfD7jT5eMoH/nDK57bCGfb63wOiSRNkeJQNq8tKQEZkwsJMHvY/KsInZWq6CNSHMoEUi70DM7lYcuz+dLFbQRaTYlAmk3jjsqh9+eO4S3P93Kf81b4XU4Im2GRg1Ju3Lp6F6s2lTOzPlrGdAlg4sKex7+QSJxTi0CaXd+edYgTjgml188u5TiL7Z5HY5Iq6dEIO1Ogt/H/ZeNpFtmMv8+u5gNO6q9DkmkVVMikHYpMzWRmZNGUbN7D1c/WkRVbZ3XIYm0WkoE0m4d0ymdey8bycpNZdz0t8Xs0UgikUYpEUi7dvKATtxyxiDmfbKJe99Y7XU4Iq2SRg1Juzf5xL6s2lzOPa+vZkDnDM4YqoI2IuHUIpB2z8y487wh5PfK5KdzFrNs406vQxJpVZQIJC4kJfiZNqGAzNQAVz9axNZyFbQR2UuJQOJGp4xkZkwsZFtVLdf8tZiaunqvQxJpFZQIJK4M6d6Ruy4aTvEX2/nVcypoIwLqLJY49P1h3fh0Uzn3vrGGAV06cNUJKmgj8U0tAolLN57an9MHd+bOF5fz9qcqaCPxTYlA4pLPZ9x9cbCgzfWPL+QzFbSROKZEIHErLSmBmZMKSfT7uPrRInZWqaCNxCclAolrPbJSmTahgK+2V3H9Ewupq9/jdUgiMadEIHFvVJ9s7jh3CO+uLuF381Z6HY5IzEU1EZjZODNbZWZrzOzmRrZfaWZbzWxR6DY5mvGINOWSUb344dg+PPKvtcxZ8JXX4YjEVNSGj5qZH3gAOA1YDywwsxecc8sb7PqUc+76aMUhEqlfnDmINVsq+MVzn9A3L41RfbK9DkkkJqLZIhgNrHHOfe6cqwWeBM6J4uuJfCMJfh/3X5pPz6xUrpldzPrtVV6HJBIT0UwE3YHwNvb60LqGLjCzJWY218xUYFY81TE1wIxJhdTW7+HqWcVU1qigjbR/XncW/x3o45wbBrwGPNrYTmY2xcyKzKxo61b9+Eei6+i8dO6/LJ9Vm8r42RwVtJH2L5qJYAMQ/g2/R2jdPs65Uufc3mkgZwIFjT2Rc266c67QOVeYl5cXlWBFwn2nfx63njmIl5dt4p5/qqCNtG/RTAQLgH5m1tfMEoHxwAvhO5hZeIWQs4EVUYxHpFmuOqEvFxX04N5/rubFJV97HY5I1ERt1JBzrs7MrgdeAfzAI865ZWb2G6DIOfcCcIOZnQ3UAduAK6MVj0hzmRl3nDeEz0sq+dnfFtE7J5Uh3Tt6HZZIi7O2Ng1vYWGhKyoq8joMiSNby2s45/75OOD568fSKSPZ65BEms3Mip1zhY1t87qzWKTVy8tIYvrEQnZU7eaa2SpoI+2PEoFIBIZ078gfLx7Owi938ItnVdBG2hclApEInTm0Kz8+pR9zi9fz8Py1Xocj0mKUCESa4cen9OOMIV343bwVvLlqi9fhiLQIJQKRZvD5jD9ePJyBXTpww+Mfs2aLCtpI26dEINJMqYkJzJhUSFLAx9WzVNBG2r74SQRlG2Hli/Dlh1D6GVTvAHX4yRHqnpnCtCsKWL+9iuseV0Ebadui9oOyVueL9+Dpqw5c50uA1BxIzYW0vX9zQ+tyQsth61KywR8/h0wOrbBPNneeN5Sfz13CHS+u4NdnD/Y6JJEjEj+fav1OgylvQWUpVJVAZUnY323B5a8XB//u2tnEkxikZB4+YYSvC+jHR+3ZxYU9WbWpnIfnr2VglwzGj+7ldUgizRY/iSC5I3QbGdm+9bv3J4d9CSOUQKpKQ+tKg5eYvvowuK9r4kdGiemNJIyGrY+wdUkZYNZy71ui7pYzBrJ6SwW/en4pzy3aQE56ErlpiWSnJZGTnkhueiI56UnkpAX/dkhOwPRvLK1I/CSC5vAHIKNz8BaJPXtg146wJFGyP1mEryvfBJuXB5frdjXx2olNXK7KhdTsBq2P3GALxedvufcuzZbg93HfpSO588XlrCupYsXXZWyrrGVHE53IAb+RnZZIzr5EsT9JBP/uX85NTyIlUf++El2aa8gLzkFtZRMtjSZaHzVljT+X+SAl68AWxgEtjUbWJSTG9v3Gqd31e9heWUtJRS2llTWUVtRSWllLacXe5ZoDtlXVNt6qTE30BxNHqKXRMFFkh9btXQ7442cMiETuUHMNqUXgBTNISg/esvpE9pi6mgYtjCZaH1tXwRf/Cl6uookkn9ShkYTRsPWRs39dYpouVx2BgN9Hpw7JdOoQWT9RdW19WMIIJYmKWraF1pVU1vXyt3QAAAxXSURBVLKpbBfLNpZRWlnD7vrG/307pgSCySKs1ZGTnhS8RLX3fiixZKYE8Pn0bxvvlAjaioQk6NAteIvEnvrgENmDOsYbXK7auSHYSV5ZAnuaGA+fkNzEpakmWh/JmeDTt9LmSkn00yMxlR5ZqYfd1zlH2a46toVaGAe0OipqQi2PWj4vqWDBulq2VdU2Olra7zOyUvf2Y4T6NdIO7tfITU8kOy2R9CT1b7RHSgTtlc8f/EaflgN5Aw6/v3NQU37gpal9iaPButLPgutrm/hVrfmDSeOghNGgpRHe9+EPtOz7b+fMjI4pATqmBOibm3bY/ev3OLZX1e5rbYQnjJKw5aUbdlJSUUP5rsZrNScm+EKXp/a2LA5sYeSkJ5IbWpedlkhyQP0bbYESgQSZQXKH4C37qMges7v6wNZF1bbGWx+blwXXVW9v+rmSOx6YHNJyGrQ0GqxLPPy3ZtnP7zNy05PITU8CMg67f01dfai1UUtJRc3+5QZJZPXmCkoqaqipa/wHdelJCQckir2Xp8L7NfYmlKzUAAnq3/CEEoEcuUAKdOwRvEWivg6qtzVoaTToFK8qge3rYENR8P6exr+ZEkg9sG8j/DLVAa2PvZerOqqfoxmSEvx07ZhC144ph93XOUdVbX2jiSK8BfLVtioWfbWDbZW11O85+DqVGWSlJoZGVB2YJBprdXRI0WWqlqJEILHjT4D0TsFbJJwL/rjvoI7xhq2PrbB1ZfB+XXXjz+ULhCWG7MO3PlKzNSw3QmZGWlICaUkJ9Mo5fEttzx5H2a7dB1yS2tvPsa1yf0f5yk1llB5iGG6CzyK6PLU3oaQm6uOuKToy0npZ6JfcKZmQc3Rkj6mtarqlEb7u60XB5UP+ijzr8C2N8M7zhKQWe+vtmc9nZKYmkpmayDGd0g+7/+76Pfv7Nw4YURU2JLeyhi9KqyitqKGyiWG4KQH//hFUaYn7h+SGOsr3J5TgpavEhPi5TKVEIO1LYiok9oLMCKd6qN/doJ+jtJFpSMJ/RV4KrokJ5gJpwWTgDwTnsfIlhC0Hgi2Mw24LrfMn7N8WvrxvWyDsOfwN7iccvC18/b5tDWNq5LVaweivgN9Hp4zkiGtF7x2GG97HEf77jZLKWjaX72L512WUVtRS28SEgR2SEw55eSonLWnf6Kq2PgxXiUDimz8AGV2Ct0js/RX5QS2N0OWq+tpgctlTHxyOu6cudL/u4OXd1fuX920LPTZ8v/BtsWa+CBNSeOKKMDk1uq2R5eYkLn8CKb4EevgC9EgJQJofugbAlw7+rIOe3wHlNXX7+jX2XZ7aN6KqZt8w3KIvgtsa6d7AZxzwa/H9P/hrMNVIaLm1DcNVIhBpDp8v1MeQHfvXdi7YGtmXJHYHO+D3Lu+pC90PX967X2PJqT5sWwQJKfx1D9jWMI56qKuFPZWNxHGIeJsaGBBFZn46+AN08CXQ91BJJyUB0hJwvgB1+Kl1Pmqdj5o9wduueh/V9UZVvVFValRuNirqoLrOKCOBbfhYgZ/dLoF6fDhfgKTERJKSk0hJSiIlOYnU5GTSkpNITU0hPSWZjNRkMtJS6ZCaTGJiUjCOjK7QsXuLHwclApG2wiz4G4322ontXGQtqYgSV/j9xhLjoRJo0y06q99NYE89gT27SavfDdQFb1YHtht8dRCoA99uSKjDhT3eGk5MWQ9Uhm4RWtz7Sob/8H9b7piHKBGISOtgFrzU045qfhxw8WfPnuAsxU20slx9HdU1uyirqGZHZTVlldWUV1VTXllNRfUuKqt3ccwx0al50X6OuIhIa+bzAb4mf0VvQGroFmGPVYvxfkiAiIh4SolARCTOKRGIiMQ5JQIRkTinRCAiEueUCERE4pwSgYhInFMiEBGJc+YaK2TaipnZVuCLI3x4LlDSguG0lNYaF7Te2BRX8yiu5mmPcfV2zuU1tqHNJYJvwsyKnHOFXsfRUGuNC1pvbIqreRRX88RbXLo0JCIS55QIRETiXLwlguleB9CE1hoXtN7YFFfzKK7miau44qqPQEREDhZvLQIREWlAiUBEJM61i0RgZo+Y2RYzW9rEdjOze81sjZktMbP8sG2TzGx16DYpxnFdHornEzN7z8yGh21bF1q/yMyKWjKuCGM7ycx2hl5/kZndFrZtnJmtCh3Pm2MY03+GxbPUzOrNLDu0LWrHy8x6mtmbZrbczJaZ2Y8b2Sfm51iEccX8HIswLi/Or0ji8uocSzazj8xscSi22xvZJ8nMngodlw/NrE/YtltC61eZ2enNDsA51+ZvwLeBfGBpE9vPBF4iWARoDPBhaH028Hnob1ZoOSuGcX1r7+sBZ+yNK3R/HZDr4TE7CfhHI+v9wGfAUUAisBg4NhYxNdj3B8AbsTheQFcgP7ScAXza8D17cY5FGFfMz7EI4/Li/DpsXB6eYwakh5YDwIfAmAb7/AiYFloeDzwVWj42dJySgL6h4+dvzuu3ixaBc+4dYNshdjkHmOWCPgAyzawrcDrwmnNum3NuO/AaMC5WcTnn3gu9LsAHQI+Weu3DieCYNWU0sMY597lzrhZ4kuDxjXVMlwJPtMTrHo5z7mvn3MLQcjmwAujeYLeYn2ORxOXFORbh8WpKNM+v5sYVy3PMOecqQncDoVvDkTznAI+GlucCp5iZhdY/6Zyrcc6tBdYQPI4RaxeJIALdga/C7q8PrWtqvReuIviNci8HvGpmxWY2xaOYjg81VV8ys71Vsz0/ZmaWSvDD9Omw1TE5XqHm+EiC39jCeXqOHSKucDE/xw4Tl2fn1+GOlxfnmJn5zWwRsIXgl4cmzzHnXB2wE8ihBY6Zite3AmZ2MsH/pCeErT7BObfBzDoBr5nZytA35lhZSHBukgozOxN4DugXw9c/lB8A/3LOhbceon68zCyd4AfDjc65spZ87m8ikri8OMcOE5dn51eE/44xP8ecc/XACDPLBJ41syHOuUb7y1pavLQINgA9w+73CK1ran3MmNkwYCZwjnOudO9659yG0N8twLM0s6n3TTnnyvY2VZ1z84CAmeXSCo4ZweujBzTZo328zCxA8MPjMefcM43s4sk5FkFcnpxjh4vLq/MrkuMVEvNzLOx1dgBvcvAlxH3HxswSgI5AKS1xzKLR8eHFDehD0x2fZ3FgR95HofXZwFqCnXhZoeXsGMbVi+D1vG81WJ8GZIQtvweMi/Ex68L+HxyOBr4MHb8Egh2efdnfmTc4FjGFtnck2I+QFqvjFXrfs4B7DrFPzM+xCOOK+TkWYVwxP78iicvDcywPyAwtpwDvAt9vsM91HNhZPCe0PJgDO4s/p5mdxe3i0pCZPUFwFEKuma0HphLsbME5Nw2YR3BUxxqgCvhhaNs2M/stsCD0VL9xBzYFox3XbQSv8T0Y7POhzgVnFuxMsGkIwf8YjzvnXm6puCKM7ULgWjOrA6qB8S541tWZ2fXAKwRHeDzinFsWo5gAzgNedc5Vhj002sdrLDAB+CR0DRfgVoIfsl6eY5HE5cU5FklcMT+/IowLvDnHugKPmpmf4JWaOc65f5jZb4Ai59wLwMPAbDNbQzBRjQ/FvczM5gDLgTrgOhe8zBQxTTEhIhLn4qWPQEREmqBEICIS55QIRETinBKBiEicUyIQEYlzSgQiMRSadfMfXschEk6JQEQkzikRiDTCzK4IzQ+/yMz+HJoQrMLM/hSaL/6fZpYX2neEmX1gwXn/nzWzrND6Y8zs9dDEagvN7OjQ06eb2VwzW2lmj4VmkBTxjBKBSANmNgi4BBjrnBsB1AOXE5xaoMg5Nxh4m+AvnyE4bcH/c84NAz4JW/8Y8IBzbjjBugBfh9aPBG4kOI/8UQR/8SrimXYxxYRICzsFKAAWhL6spxCcGngP8FRon78Cz5hZR4JzxLwdWv8o8DczywC6O+eeBXDO7QIIPd9Hzrn1ofuLCM6vND/6b0ukcUoEIgcz4FHn3C0HrDT7VYP9jnR+lpqw5Xr0/1A8pktDIgf7J3BhaN55zCzbzHoT/P9yYWify4D5zrmdwHYzOzG0fgLwtgtWwFpvZueGniMpVOxEpNXRNxGRBpxzy83slwSrUfmA3QSnAK4ERoe2bSHYjwAwCZgW+qD/nNDMowSTwp9DM0juBi6K4dsQiZhmHxWJkJlVOOfSvY5DpKXp0pCISJxTi0BEJM6pRSAiEueUCERE4pwSgYhInFMiEBGJc0oEIiJx7v8D14gROoKpZUQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "epoch_nums = range(1,num_epochs+1)\n",
    "training_loss = history.history[\"loss\"]\n",
    "validation_loss = history.history[\"val_loss\"]\n",
    "plt.plot(epoch_nums, training_loss)\n",
    "plt.plot(epoch_nums, validation_loss)\n",
    "plt.xlabel('epoch')\n",
    "plt.ylabel('loss')\n",
    "plt.legend(['training', 'validation'], loc='upper right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate model performance\n",
    "\n",
    "We can see the final accuracy based on the test data, but typically we'll want to explore performance metrics in a little more depth. Let's plot a confusion matrix to see how well the model is predicting each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "gather": {
     "logged": 1667462295856
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating predictions from validation data...\n",
      "1/1 [==============================] - 9s 9s/step\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVIAAAEkCAYAAAB9rrkzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deZhcVZ3/8fenG0IghM2AgqwiiyGyRkVQDMvwRFlVFHBBBxVUBGTEEUefHzA6jvOAjggzSsQFFZFRUBFnCIsEUIEkQAghgCK7oBB2krAkfH9/nNOm0nR33eqq6ntv9+fFc5+uunXr3m8VnW+f7Z6jiMDMzIavp+wAzMzqzonUzKxNTqRmZm1yIjUza5MTqZlZm5xIzczatErZAVSJVlk9NG5i2WFU1k6v27TsEKzm7rvvXhYtWqR2ztG71mYRy5YWOjaWPjozIqa3c70inEgbaNxEVtvmvWWHUVm/v+GsskOwmtv9TVPbPkcse47Vtj2s0LHP3XzmpLYvWIATqZnViwC1VajtOCdSM6sfVat7x4nUzOrHJVIzs3YIenrLDmIlTqRmVi/CVXszs/bIVXszs7a5RGpm1iaXSM3M2iGXSM3M2iLca29m1p7qlUirFY2ZWRE9KrY1Iel7kh6RtKBh32mS7pA0X9IvJK3TNJw2P46Z2cjqG0daZGvuB0D/2aEuB6ZExPbAH4HPNzuJE6mZ1Y9UbGsiIq4BHu+377KIWJafXg9s3Ow8biM1s5oZ0TbSI4ELmh3kRGpm9VO8136SpLkNz2dExIwib5T0BWAZcF6zY51IzaxeClbbs0UR0fJs0pI+DOwP7B0R0ex4J1Izq58uVu0lTQf+GXhbRCwp8h53NplZ/XSos0nS+cB1wDaSHpT0EeAsYCJwuaR5kr7d7DwukZpZzXSusykiDh9g93dbPY8TqZnVi28RNTNrV/VuEXUiNbP68TR6ZmZtconUzKxNLpGambVBbiM1M2ubepxIzcyGTYBctTcza4PyViFOpGZWM6pcibRaDQ39SPq4pCNaOH6apEu6GZOZlU9SoW2kVLpEGhEDThYgaZWGGazNbIypWom0Uok0lz5PBAKYD/wZeDYiTpc0C5gHvAU4X9I1wBnABOB5YO9+55oAnAlMAVYFTomIX43QRzGzbhGowMJ2I6kyiVTSdsAXgd0iYpGk9YDj+h02LiKmShoH3AEcGhFzJK0FLO137BeA30bEkXkVwNmSroiIxd3+LGbWPapgG2llEimwF/CziFgEEBGPD/Bl9a2dsg3wcETMycc+DS8r7u8LHCjpxPx8PLApcHvjQZKOAo4CYNU1O/RRzKybnEjb00ppUsC7I+LOoQ7K67fMAOhZY4OmSwqYWfmqlkir1Gv/W+A9kl4BkKv2g7kT2FDSG/KxEyX1/6MwEzhW+RuXtFMXYjazErjXfhARcZukfwOulrQcuBm4d5BjX5B0KHCmpNVJ7aP79DvsS8A3gPmSeoB7SItZmVmdubNpaBFxLnDuIK9N6/d8DrBrv8Nm5Y2IWAoc3ekYzaxc7mwyM+sAJ1Izs3ZVK486kZpZzcglUjOztjmRmpm1QYieik3sXK1ozMyKUMGt2Wmk70l6RNKChn3rSbpc0p/yz3WbnceJ1MzqRR0dkP8DYHq/fScBV0bEVsCV+fmQnEjNrHY6lUgj4hrg8X67D2LFePZzgYObncdtpGZWO13ubHplRDycH/8VeGWzNziRmlnttHCL6CRJcxuez8gTFRUSESGp6WRGTqRmVistTkiyKCKmtniJv0naMCIelrQh8EizN7iN1Mxqp8uzP10MfCg//hDQdGUNl0jNrHY61UYq6XxgGqkJ4EHgZOCrwP9I+ghwH/DeZudxIjWz+ulQX1NEHD7IS3sPsn9ATqRmVju+RdTMrA0S9HhiZzOzdnhiZzOztlUsjzqRmln9uERqZtYOuURqZtYW4c4mM7O2OZGambXDVXszs/YIdzaZmbXJ40jNzNpWsTzqRGpmNeNbRM3M2uM2UjOzDqhYHnUiNbP6cYnUzKxNFcujTqSNdnrdpvz+hrPKDqOyDjz7+rJDqLyLj9617BBGP7lEambWFiH32puZtatiBVInUjOrH1ftzczaUcFJS3qKHCRpdUnbdDsYM7Nm+gbkF9lGStNEKukAYB5waX6+o6SLux2YmdlgapdIgVOANwJPAkTEPGCLLsZkZjaknh4V2kYsngLHvBgRT/XbF90IxsysqdxGWmQrdDrpBEm3SVog6XxJ41sNqUgivU3S+4BeSVtJOhP4Q6sXMjPrBFGsWl+kai/p1cBxwNSImAL0Aoe1GlORRHossB3wPHA+8DTw6VYvZGbWKZ0skZJGL60uaRVgDeChVuNpOvwpIpYAX5D0H+lpPNPqRczMOqmnQx1JEfEXSacD9wNLgcsi4rKW42l2gKQ3SLoVmA/cKukWSbu0HLGZWQdILXU2TZI0t2E7auVzaV3gIFIH+kbABEkfaDWmIgPyvwt8MiKuzRd+C/B9YPtWL2Zm1gktdMgvioipQ7y+D3BPRDwKIOkiYDfgx63EUySRLu9LogAR8TtJy1q5iJlZJ3VwjOj9wK6S1iBV7fcG5rZ6kiKJ9GpJZ5M6mgI4FJglaWeAiLip1YuambWjU3k0Im6Q9HPgJmAZcDMwo9XzFEmkO+SfJ/fbvxMpse7V6kXNzIZLpCFQnRIRJ/Py/NaSIr32e7ZzATOzTqvYdKTFZn+StB9pLOnfR/xHxL92Kygzs0GphhM7S/o2aZDqnsA5wCHA7C7HZWY2ING5caSdUuTOpt0i4gjgiYg4FXgzsHV3wzIzG1yH72xqW5Gq/dL8c4mkjYDHgA27F5KZ2dDqOEP+JZLWAU4jDREIUhXfzGzEjXRps4givfZfyg8vlHQJMH6AafXMzEZM1dpIi/ba7wZs3ne8JCLih12My8xsULVLpJJ+BGxJWm5ked4dgBOpmY241GtfdhQrK1IinQpMjgjPim9m5Rvh9ZiKKDL8aQHwqm4HYmZWVG2GP0n6NakKPxFYKGk2aZZ8ACLiwO6HZ2b2clUrkQ5VtT99xKIwMytIQG/FGkkHTaQRcXXjc0mvAPYA7o+IG7sdmJnZYKqVRodoI5V0iaQp+fGGpLbSI4EfSfLid2ZWCikNfyqyjZShOpu2iIgF+fE/ApdHxAHAm0gJ1cysFFXrbBoqkb7Y8Hhv4H8B8iqiL3UzKDOzoXRqXftOGaqz6QFJxwIPAjsDlwJIWh1YdQRiGzZJvRGxvPmRZlZHFeu0H7JE+hHSZM4fBg6NiCfz/l1Jq4i2RNIESb/JyzkvkHSopOmS7pB0k6Rv5nv5kXSKpBMb3rtA0ub58S8l3SjptsalVSU9K+lrkm4B3izpA5JmS5on6WxJva3GbGbVI4nenmLbSBmq1/4R4OMD7L8KuGoY15oOPBQR+wFIWpvUgbUXcBdwQcHzHBkRj+eS8RxJF0bEY8AE4IaI+Iyk1wGfA3aPiBcl/Tfwfnxbq9moULVxpEXubOqUW4F/kPQfkt4KbEFaT/pP+fbToutIH5dLndcDmwBb5f3LgQvz472BXUiJdl5+/pqBTibpKElzJc19dNGjw/pgZjayegpuI6XQ7E+dEBF/zEs4vwP4MnDlEIcvY+XvYTyApGnAPsCbI2KJpFmsWEfquYZ2UQHnRsTnC8Q1g7z86i67TPV8AmYVJ8ZwiTTPrr8kIn5MmiR6N2BzSVvmQw5vOPxeUgcXOflukfevTVryZImkbUnttQO5EjhE0gb5HOtJ2qyTn8fMytOjYttIGepe+zNJ99oPKCKOa/FarwdOk/QSaWjVJ4BJwG8kLQGuJd3XD6mKfoSk24AbgD/m/ZcCH5d0O3AnqXo/UGwLJX0RuExST77eMcB9LcZsZhVUsTtEh6zaz+3khSJiJjBzgJe2hb9X20/Mxy4F9h3kVG8f5Pxr9nt+AcU7sMysJqTO3mufl1I6B5hCKjweGRHXtXKOoXrtz20vPDOz7uhwE+kZwKURcYikcaTl51tSZIb89UlDiSazomOHiNir1YsNJSJmAbM6eU4zG306ua59Hoa5B2m8PBHxAvBCq+cp0tl0HnA7qcPnVFJH0JxWL2Rm1ikdHP60BfAo8H1JN0s6R9KE4cTTzCsi4rvAixFxdUQcSRpEb2ZWihYmLZnUN048b0f1O9UqpBFC34qInYDFwEmtxlNkHGnf5CUPS9oPeAhYr9ULmZl1Qt8togUtioipQ7z+IPBgRNyQn/+cLiXSL+d2hM8AZwJrASe0eiEzs07pVKd9RPxV0gOStomIO0l3QS5s9TxNE2lEXJIfPgXs2eoFzMw6qZOdTdmxwHm5x/5u0vzLLSnSa/99BhiYn9tKzcxGXCfzaETMIy07P2xFqvaXNDweD7yT1E5qZjbyRvj2zyKKVO0vbHwu6Xzgd12LyMysCVVs+bvhzP60FbBBpwMxMytCwCojOUdeAUXaSJ9h5TbSv5LudDIzK0XVptErUrWf2OwYM7ORknrty45iZU0LyJJeNgHzQPvMzEZEwbuaRrLQOtR8pONJs6BMkrQu/L11dy3g1SMQm5nZgDo8jrRtQ1XtjwY+DWwE3MiKRPo0cFaX4zIzG5CA3rp0NkXEGcAZko6NiDNHMCYzsyGInooNfyqS11/KM0gDIGldSZ/sYkxmZoNKi99Vq420SCL9WEQ82fckIp4APta9kMzMhlBw4btKLH7XoFeS8trzSOoFxnU3LDOzwdWps6nPpcAFks7Oz4/O+8zMRlxf1b5KiiTSzwFHkZZPBrgc+E7XIjIza6KTq4h2QtM20oh4KSK+HRGHRMQhpElP3YtvZqUQHV2zqSMKTVoiaSfgcOC9wD3ARd0MysxsUKrRvfaStiYlz8OBRcAFgCLCs+SbWamqlUaHLpHeAVwL7B8RdwFI8lpNZlaqLiw10rahmhHeBTwMXCXpO5L2pnp/CMxsDFLBbaQMmkgj4pcRcRiwLXAV6b77DSR9S9K+IxWgmdnKRE9PsW2kFOm1XxwRP4mIA4CNgZvxxM5mVpIq9tq3dK2IeCIiZkTE3t0KyMysGUmFtpEynDWbRq3nl73EfYuWlB1GZV189K5lh1B5R//P/LJDqLT7nljakfNUrbPGidTM6qVO40jNzKpIQG/FEmnF5pk2M2uuk8OfJPVKulnSJcONxyVSM6udDhdIjwduJ61HNywukZpZraThTyq0NT2XtDGwH3BOOzG5RGpmtdNCiXSSpLkNz2dExIyG598A/hmY2E48TqRmVjNCxQdALYqIqQOeRdofeCQibpQ0rZ2InEjNrFY62Gu/O3CgpHcA44G1JP04Ij7Q6oncRmpm9VJwBdFmuTYiPh8RG0fE5sBhwG+Hk0TBJVIzq6GKDSN1IjWz+mmhjbSQiJgFzBru+51IzaxW0sTOZUexMidSM6udTpdI2+VEama1U7WlRpxIzaxWXLU3M2tbSwPyR4QTqZnVS4ExoiPNidTMaqdiedSJ1MzqpYoTOzuRmln9VCuPOpGaWf24s8nMrE0Vq9k7kZpZ/VQsjzqRmlkNVSyTOpGaWa1IvkXUzKxt1UqjTqRmVkcVy6ROpGZWM77X3sysbRVrInUiNbN6EZWr2TuRmln9qGJFUidSM6udiuXR7qxrL2kdSZ8c4vU/dOGa0yRd0unzmln1qOA2UrqSSIF1gJclUkmrAETEbl26rpmNdkWz6Ahm0m4l0q8CW0qaJ2mOpGslXQwsBJD0bP65pqQrJd0k6VZJB+X9m0u6XdJ3JN0m6TJJq+fX3iBpfj73aZIW9L+4pAmSvidptqSb+85rZqODCv43UrqVSE8C/hwROwKfBXYGjo+Irfsd9xzwzojYGdgT+JpWtCJvBfxXRGwHPAm8O+//PnB0PvfyQa7/BeC3EfHGfN7TJE3o0GczsxL1LX5XZGt6LmkTSVdJWpgLbccPJ6ZuJdL+ZkfEPQPsF/AVSfOBK4BXA6/Mr90TEfPy4xuBzSWtA0yMiOvy/p8Mcr19gZMkzQNmAeOBTQc6UNJRkuZKmvvEY4ta/VxmVobOVe2XAZ+JiMnArsAxkia3Gs5I9dovHmT/+4H1gV0i4kVJ95KSHsDzDcctB1Zv4XoC3h0RdzY7MCJmADMApuywc7RwDTMrSaeq7RHxMPBwfvyMpNtJBbqFrZynWyXSZ4CJBY5bG3gkJ9E9gc2GOjgingSekfSmvOuwQQ6dCRzb10wgaadiYZtZHUjFttbOqc2BnYAbWo2nKyXSiHhM0u9zR9BS4G+DHHoe8GtJtwJzgTsKnP4jwHckvQRcDTw1wDFfAr4BzJfUA9wD7N/ixzCzimohR06SNLfh+YxcC135fNKawIXApyPi6Vbj6VrVPiLeN8Rra+afi4A3D3LYlIbjT2/Yf1tEbA8g6SRSAiYiZpHaQ4mIpcDRw4/ezCqteCZdFBFThzyVtCopiZ4XERcNJ5w63tm0n6TPk2K/D/hwueGY2Ujq5MTOufnvu8DtEfH14Z6ndok0Ii4ALig7DjMrTwdHiO4OfBC4NY/yAfiXiPjfVk5Su0RqZtapTBoRv+vE2ZxIzaxmPLGzmVnbqjb7kxOpmdWKcCI1M2ubq/ZmZm1yidTMrE0Vy6NOpGZWM8O4j77bnEjNrIaqlUmdSM2sVvomdq4SJ1Izqx1X7c3M2uThT2Zm7apWHnUiNbP6qVgedSI1s3oZzjIi3eZEama1o4plUidSM6udaqVRJ1Izq6GKFUidSM2sbjyxs5lZWzwfqZlZBziRmpm1yVV7M7N2eBypmVl7hIc/mZm1r2KZ1InUzGqnam2kPWUHYGbWqh4V24qQNF3SnZLuknTSsOIZzpvMzEqlgluz00i9wH8BbwcmA4dLmtxqOE6kZlY7KvhfAW8E7oqIuyPiBeCnwEEtxxMRrb5n1JL0KHBf2XE0mAQsKjuIivN3NLSqfT+bRcT67ZxA0qWkz1XEeOC5huczImJGw7kOAaZHxEfz8w8Cb4qIT7USkzubGrT7P7jTJM2NiKllx1Fl/o6GNhq/n4iYXnYM/blqb2Zj2V+ATRqeb5z3tcSJ1MzGsjnAVpK2kDQOOAy4uNWTuGpfbTOaHzLm+Tsamr+fIUTEMkmfAmYCvcD3IuK2Vs/jziYzsza5am9m1iYnUjOzNjmRmo1SqtpSm6OYE6nVkpNEc+EOkBHjRFpBkiZIOlzSMZIm5n2rlh1XlURESJoi6XUAksY5ua4gaR1JJ0r6uqT1lbxKkv/Nd4G/1Gr6IvAPwNeB8ZJeBVwiqehtcaOapLUkHQ2cBByfd08mfWdjWsMfk68AE4GPAhPy9jNWHnxuHeJxpNX0rojYRtK2wNMR8bykjYBnyw6sInYA9gGuJc3aAylBHANcJqknIl4qK7gyNVTn3xYR20l6G/BURDybazdPlxjeqOUSacVIWh14QNLmwGo5ib4SWB4Rzw355rHjtcAdwC+BR/K+VVkxOceYruLn6vuDkrYDJkbEE5JeA7wUEU+UHN6o5BJphTRUy84DPg2sIelA4H3Ar0sLrHruALYGTgOWStoUmAb8scygqiAn0V7gDOCTwCRJxwFvAS4oM7bRzHc2VZSk9wK7AesDv4iIn5ccUmXktuI9gEOBZaT2vzuBf4+IJ8uMrUok7QrsTGr2uCIiriw5pFHLibQicjXsSOBeUhX1T8DfgKeAABaP1Xa//iR9CbgUmE8qad0ZEXeXG1X58giGk0m/Q48DdwMPAY+SfoeeiohHSwtwFHPVvlpeBF4NTAH2B9YgtWNPIvW4fqO80CplGbBeRDwD/F/ZwVTIYuA6UtV+A2ALUs/9aqTfocuBr4zlzrhucYm0InLb1urAC8A4YG1SB8p40j+GRyLi/vIirA5JVwHbkpLoAuABUqfTHyLixTJjK5ukcXnJjMZ9awDrAUsj4rFyIhvdnEgrRtJ+wMMRcVPDvnWBZ8d6kugj6WDSuNFNSSWv9YEtgR0j4pGh3jua9ZU0Jb0beDNwD2m40wOkKv7jwGO+46nznEgrQtIqeW7EXwE/jYjzJY2PiOck/RT4SUS0POGsjR0NifSTwKdIHXBLgO1J1f17gTMiYmZ5UY5ObiOtjr42q4nkBfgaxo1OpFoLmJVK0l7AEaRmkCdJHSm3RsSPSw2sOrYGTgF+AywFpgLvJzWDHC/p8YiYU154o48H5FdEQ+P/ucA/SjpY0g6SppPaSR8qL7rqkLQxcAKpR/pAUjJ9D3Bwfn1MD8bP9iItMbw4Il6KiNmkoXQXkjrq3NHUYS6RVs+dwE7AB0idT5sAH42Ie8sMqkI2IZWyvgbsFhFflPQT4LP5dZFKqGNOwx/j04CTJS0gtZP23Rq6hDTm9m/lRDh6OZFWSJ7h6XMR8U5JqwHj8hAfW2FdUlLYFOjNg853I/VKwxi/PTS7gFTqfD1pnG0P8KH82pnAwyXFNWo5kVbLeOAlSdsDCyPiGY/5e5kbSJ0m9wC/ILUFPgX8NL8+JkujjSLiBUkzgbmkiW4eA9bIbe6/LDW4Ucq99hUgSXl+zY2Bi0gDq/+PdEfK08ADuZ3L+pG0CWlIz5KyY6kCSWuTpmF8Nen3p4c0HvmhiPjXMmMbzVwirYCGcX2LgbNIbaOvIc1ytDFwO+BECkj6DGnqvLtI4yIXAU9KmhkRfyk1uBL1/TEm3agwnTQP6XjS79LaeArGrnIirYA8Zd540sDpOaSSxLOeNm9AV5AS6DrAq4B3kTrnDgD+4qYQeoGLIuKGsgMZS5xIq+E1pGEpvaTp8xYDIek5UrXs0oi4osT4KiMibgFuadwn6T/JHShjOIn2jVbYEPi4pK1I7cmPkP7w3BoRHkLXJU6k1XBTRDwp6V2kOTVvJZW4VicNrl5aZnBVIuntpKTxNOl7eZZUlT2rzLjK1vAHZCHwbdIkJVNI1fqtgW8BZ0vqjYjl5UQ5ejmRVkDDHJpvA66MiEv7XpN0MmnsnyX7kGbFWp63VwFX4RsWAIiI20lT6a2k70YFJ9HucCKtgL777Enj/vrPYr4z8IeRj6qyTgXWbNieiog/lxtSdeQJbo4l9do/Rmom+gtphQXP/NQlTqTV0FdKuAp4v6SXgL+SEsXauLQFgKQ1SXcw3Q08R2r/Wy5pCvAE8MRYHQbV0Ml2GunuuIOAWaQ5SScDNwKPNfTuWwd5HGmF5CU0TiWN/VtMumf6DODHrpJBXgTwR6Q/MHNJQ8O2IJW4VgXuj4iPlRdheRrGIt8UETtLuiwi9s2vnQN8MSL+WnKYo5ZLpBUSEYuAY3KP61rAqb5FdCVbkkqhx5DuZtqAtDDgM6QVBMaVF1q5GkqZyoPyl0s6AvgdsCfusOwqJ9IKiog/lR1DRU0mLVHd9/08Imk28L6I+Pe8ysCY1FBlP4HUVPRN0lSD+wBXR8RTZcY32rlqb7UhaUvgy6S1ra4ltZPuTpoy7nQP7VlZbgpZNyLuKDuW0c6J1GojJ9IdgG2AVwKHAT8kzZg1Zn+RJb0eOBH4J1Kn0lzSVHl3kJpC7o+I68qLcPRzIrXakHQG6fbHqyWdBGwGPAjMGMvLDOcmjfVIw5v2Iw3A35g0BGorYEFEHOESe/e4jdTqZDvgW3m40/akoT5fBa4HrhyrQ3vysKdFkg4gLVP99f7H5O/GSbRLxmzjvNXSUuCdwPeACyPiZtJkL3+DlXqux5SGTrbNSM0eLzNWv5uR4hKp1clnSYn0cuBySb2kmbLG+ozvfasCPAccLGkiqa308bzNj4inywpuLHAbqdWKpHER8UJ+vAawdkSM6UTa1/Yp6RukOWzvAlYjJdi3Ap+JiEs9xWD3uERqtdKXRPPjJaQF3ca0hrbPDUkjGG7re03SRawYjO9SU5c4kZrVnKSDgY+RxtSuKukOUg/+vaRFAh8Ft5N2kxOpWf39njTJzRnAdaRpBrckVet/TZrkxbrIbaRmo4Sk8V6ephxOpGZmbfI4UjOzNjmRmpm1yYnUhiRpuaR5khZI+lkeuzncc/1A0iH58TmSJg9x7DRJuw3jGvfmCbL77z9S0q2S5ufPclDeP0vS1FavY9bIidSaWRoRO0bEFOAF4OONL0oa1siPiPhoRCwc4pBpQMuJdCCSNga+ALwlIrYHdgXmd+LcZuBEaq25FnhtLi1eK+liYKGkXkmnSZqTS3xHQ5ooQ9JZku6UdAVpRnvya38vCUqaLukmSbdIulLS5qSEfUIuDb9V0vqSLszXmCNp9/zeV0i6TNJteUkN8XIbkGbRfxYgIp6NiHsaXn+PpNmS/ijprfm8m+fPeFPedsv7p0m6RtJv8uf6dt+97pL2lXRdPv5neY0pGwsiwpu3QTfg2fxzFeBXwCdIpcXFwBb5taNIawJBujVxLmktpXeR7ovvBTYCngQOycfNAqYC6wMPNJxrvfzzFODEhjh+QipRQhpkfnt+/E3g/+XH+5Hu3pnU7zP0AjOB+4HvAwc0vDYL+Fp+/A7givx4DWB8frwVMDc/nka6p/01+byXA4eQ1pG/BpiQj/tcX1zeRv/mAfnWzOqS5uXH1wLfJVW5Z8eKUt2+wPZ97Z+klU+3AvYAzo90C+NDkn47wPl3Ba7pO1dEPD5IHPsAk6W/FzjXyiW+PUgJm4j4jaQn+r8x0n3o04E3AHsD/ylpl4g4JR9yUf55I7B5frwqcJakHUlLd2zdcMrZEXE3gKTzgbeQkutk4Pc5xnGkwfE2BjiRWjNLI2LHxh05USxu3AUcGxEz+x33jg7G0QPsGv0GnDck1iFFRACzgdmSLieVTE/JLz+ffy5nxb+JE0jT8+2Qr9143f6Dr4P0HVweEYcXCshGFbeRWifMBD4haVUASVtLmkCq6h6a21A3JK1m2d/1wB6StsjvXS/vfwaY2HDcZcCxfU9ySZF8jfflfW8H1u1/AUkbSdq5YdeOwH1NPtPawMORZkv6IKka3+eNkrbIbaOHklbqvB7YXdJr8zUnSNq6/0ltdHIitU44B1gI3CRpAXA2qWT3C+BP+bUfMkBVN9ISIUcBF0m6Bbggv/Rr4J19nU3AccDU3Jm1kBWjB04lJeLbSFX8+weIb1XgdEl35GaKQ4Hjm3ym/wY+lGPalpVL4HOAs4DbgXuAX+TP8WHgfEnz82fdtsk1bJTwLaGgJT4AAABASURBVKJmLZA0jdQJtn/ZsVh1uERqZtYml0jNzNrkEqmZWZucSM3M2uREambWJidSM7M2OZGambXJidTMrE3/HxATBGg6h4XOAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Tensorflow doesn't have a built-in confusion matrix metric, so we'll use SciKit-Learn\n",
    "import numpy as np\n",
    "from sklearn.metrics import confusion_matrix\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "print(\"Generating predictions from validation data...\")\n",
    "# Get the image and label arrays for the first batch of validation data\n",
    "x_test = validation_generator[0][0]\n",
    "y_test = validation_generator[0][1]\n",
    "\n",
    "# Use the model to predict the class\n",
    "class_probabilities = model.predict(x_test)\n",
    "\n",
    "# The model returns a probability value for each class\n",
    "# The one with the highest probability is the predicted class\n",
    "predictions = np.argmax(class_probabilities, axis=1)\n",
    "\n",
    "# The actual labels are hot encoded (e.g. [0 1 0], so get the one with the value 1\n",
    "true_labels = np.argmax(y_test, axis=1)\n",
    "\n",
    "# Plot the confusion matrix\n",
    "cm = confusion_matrix(true_labels, predictions)\n",
    "plt.imshow(cm, interpolation=\"nearest\", cmap=plt.cm.Blues)\n",
    "plt.colorbar()\n",
    "tick_marks = np.arange(len(classnames))\n",
    "plt.xticks(tick_marks, classnames, rotation=85)\n",
    "plt.yticks(tick_marks, classnames)\n",
    "plt.xlabel(\"Predicted Shape\")\n",
    "plt.ylabel(\"Actual Shape\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use the trained model\n",
    "\n",
    "Now that we've trained the model, we can use it to predict the class of an image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "gather": {
     "logged": 1667462296678
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 292ms/step\n",
      "circle\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAG+ElEQVR4nO3dy4uddx3H8c+5zP2S2yQzmQabi0lMkzS3TjKTaSyVUqqUUmoJXqBFSynBNvkfLCoUFwUXii4KulDQTRe6qaBYiuimpatCV6FWqYI1ba3LcTHdqGMml5lzvuec12s5z+L5MZw35/L8nu/TWFlZCVBPs9sLANYmTihKnFCUOKEocUJR7XWO+ykXNl9jrT9654SixAlFiROKEicUJU4oSpxQlDihKHFCUeKEosQJRYkTihInFCVOKEqcUJQ4oShxQlHihKLWm4SwIX599f387k/XOnGqnnBsZiIXD+/s9jIoriNxTgy18vs/f5BX3/2gE6cr74sHZ8TJujrysXZpfjrPL+/N8vx0J04HfaFj3zkX56fznQv7srh7qlOnhJ7W0R+Ezu6eynfv25+FOYHCejr+a+09c1P53ucO5PSuyU6fGnpKVy6lnNw1mR8+eDDHZya6cXroCV27znlsZiI/+cLhHNk+1q0lQGld3YTwme3j+cUjd+XTW0e7uQwoqes7hA5sHcsvHzuWO6dHur0UKKXrcSbJndOj+c3FuzM/OdztpUAZJeJMkvnJkfzhKyezc2yo20uBEsrEmSQ7x4fz5pNnsnWkI7sKobRScSbJttF23n5qIZND5ZYGHVWygKnhVt555lxG2yWXBx1R9tU/1m7lr5cWM9Rc87mi0PfKxpkkw61m/vHs+eiTQVQ6ziRpNRv56PLy2s/lhj5WPs4kaTYa+fiKQBksPRFnkjQ+CdRHXAZFz8SZrAb64XPLaSuUAdBTcSar30Hf/8ZSRloCpb/1XJxJMtRq5r1LS5mwUYE+1rOv7tF2M1efPpetI61uLwU2Rc/GmSSTw6289bWFzIzZi0v/6ek4k2TraDtvPHEmuyfcbkZ/6fk4k2RmbCivffmEG7bpK30RZ7J6P+grjx838oS+0TdxJsmnpkfz8qNHDQ2jL/RVnMnqTKKfPnzE2E16Xt/FmaxO9XvpoUM5tUug9K6+jDNZnYv7gwcOZmHOZHl6U9/GmSQndk3mxfsPeHgSPamv40ySM7NTeeG+/R4/SM/p+ziT5OzcVL51YW/uvUOg9I6BiDNJFndP55vLe3NBoPSIgYkzSc7PC5TeMVBxJsmSQOkRAxdnshro88t7syxQChvIOJNkcX463753b5bmXWahpoGNM0nO7Z7OC5/dn3NzAqWegY4zSRbmpvLi/QdyZtZOImoZ+DiT5NTsZL7/wMGc2GkvLnWI8xN375zISw8dztEd491eCiQR53+4a8d4fvbwkRza5n5Quk+c/+XgtrG8/OjR7NtiogLdJc417NsymlceP549U4aG0T3i/D/2TI3ktS+dzK7xoW4vhQElzuuYnRjOG0+czvZRc3HpPHGuY/voUN76+j2ZGjZZns4S5w2YHm7n6tNnM97276JzvNpu0PhQK3+5tJhhjx+kQ8R5E0Zazfz92fPx9EE6QZw3qd1s5MPLy9Enm02ct6DZaOTjK8vdXgZ9Tpy3qNFo5F9XvIOyecR5GxqNRv55Zdl3UDaFOG9Ts9HIteeWM6xQNpg4N0C72cjfLi25DsqG8mraICPtZt555lym7SRig4hzA00MtfL2UwvZYS8uG0CcG2zLSDtvPnkms+5m4TaJcxPsGBvKH796yv2g3BZxbpLZieH89uIJExW4ZeLcRHumRvKrx47lsJlE3AJxbrJ9W0bz80eOmOrHTRNnBxzaNp4ff/6wubjcFHF2yNGZifzowUMmy3PDGisrK9c7ft2D3LzX3/sor757LZdP39HtpVDHmns/xQndt2acPtZCUeKEosQJRYkTihInFCVOKEqcUJQ4oShxQlHihKLECUWJE4oSJxQlTihKnFCUOKEocUJR4oSixAlFiROKEicUJU4oSpxQlDihKHFCUeKEosQJRYkTihInFCVOKEqcUJQ4oShxQlHihKLECUWJE4oSJxQlTihKnFCUOKEocUJR4oSixAlFiROKEicUJU4oSpxQlDihKHFCUeKEosQJRYkTihInFCVOKEqcUJQ4oShxQlHihKLECUWJE4oSJxQlTihKnFCUOKEocUJR4oSixAlFiROKEicUJU4oSpxQlDihKHFCUeKEosQJRYkTihInFCVOKEqcUJQ4oShxQlHihKLECUWJE4oSJxQlTihKnFCUOKEocUJR4oSixAlFiROKEicUJU4oSpxQlDihKHFCUeKEosQJRYkTihInFCVOKEqcUJQ4oShxQlHihKLECUWJE4oSJxQlTihKnFCUOKEocUJR4oSixAlFiROKEicUJU4oSpxQlDihKHFCUeKEosQJRYkTihInFCVOKEqcUJQ4oShxQlHihKLa6xxvdGQVwP/wzglFiROKEicUJU4oSpxQlDihqH8DLvSMgR9+QmUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from tensorflow.keras import models\n",
    "import numpy as np\n",
    "from random import randint\n",
    "import os\n",
    "%matplotlib inline\n",
    "\n",
    "# Function to predict the class of an image\n",
    "def predict_image(classifier, image):\n",
    "    from tensorflow import convert_to_tensor\n",
    "    # The model expects a batch of images as input, so we'll create an array of 1 image\n",
    "    imgfeatures = image.reshape(1, image.shape[0], image.shape[1], image.shape[2])\n",
    "\n",
    "    # We need to format the input to match the training data\n",
    "    # The generator loaded the values as floating point numbers\n",
    "    # and normalized the pixel values, so...\n",
    "    imgfeatures = imgfeatures.astype('float32')\n",
    "    imgfeatures /= 255\n",
    "    \n",
    "    # Use the model to predict the image class\n",
    "    class_probabilities = classifier.predict(imgfeatures)\n",
    "    \n",
    "    # Find the class predictions with the highest predicted probability\n",
    "    index = int(np.argmax(class_probabilities, axis=1)[0])\n",
    "    return index\n",
    "\n",
    "# Function to create a random image (of a square, circle, or triangle)\n",
    "def create_image (size, shape):\n",
    "    from random import randint\n",
    "    import numpy as np\n",
    "    from PIL import Image, ImageDraw\n",
    "    \n",
    "    xy1 = randint(10,40)\n",
    "    xy2 = randint(60,100)\n",
    "    col = (randint(0,200), randint(0,200), randint(0,200))\n",
    "\n",
    "    img = Image.new(\"RGB\", size, (255, 255, 255))\n",
    "    draw = ImageDraw.Draw(img)\n",
    "    \n",
    "    if shape == 'circle':\n",
    "        draw.ellipse([(xy1,xy1), (xy2,xy2)], fill=col)\n",
    "    elif shape == 'triangle':\n",
    "        draw.polygon([(xy1,xy1), (xy2,xy2), (xy2,xy1)], fill=col)\n",
    "    else: # square\n",
    "        draw.rectangle([(xy1,xy1), (xy2,xy2)], fill=col)\n",
    "    del draw\n",
    "    \n",
    "    return np.array(img)\n",
    "\n",
    "# Create a random test image\n",
    "classnames = os.listdir(os.path.join('data', 'shapes'))\n",
    "classnames.sort()\n",
    "img = create_image ((224,224), classnames[randint(0, len(classnames)-1)])\n",
    "plt.axis('off')\n",
    "plt.imshow(img)\n",
    "\n",
    "# Use the classifier to predict the class\n",
    "class_idx = predict_image(model, img)\n",
    "print (classnames[class_idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learn More\n",
    "\n",
    "* [Tensorflow Documentation](https://www.tensorflow.org/tutorials/images/transfer_learning)"
   ]
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "python38-azureml"
  },
  "kernelspec": {
   "display_name": "Python 3.8 - AzureML",
   "language": "python",
   "name": "python38-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}