Support Vector Machine-Application 1-Image Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Support Vector Machine - Image Classification\n",
    "\n",
    "\n",
    "In this notebook we will train SVM classifiers to classify images. \n",
    "\n",
    "For a comparative understanding, we will compare the performance of the SVM with the Logistic Regression Softmax classifier.\n",
    "\n",
    "We will use dimensionality reduction technique (Principle Component Analysis) to project the features into a smaller dimension to expedite the training time.\n",
    "\n",
    "\n",
    "Generally **images are linearly non-separable**. Based on this we formulate the following hypotheses:\n",
    "- The kernelized SVM models will perform significantly better than the linear SVM model.\n",
    "- The RBF Kernel based SVM will perform better than Softmax regression classifier.\n",
    "- Dimensionaly reduction (by retaining maximum variance) should improve the performance.\n",
    "\n",
    "We will investige these hypotheses by conducting the following experiments.\n",
    "\n",
    "\n",
    "## Experiments\n",
    "\n",
    "- Experiment 1: Support Vector Machine (LinearSVC) + PCA\n",
    "- Experiment 2: Support Vector Machine (SVC with RBF Kernel) + PCA\n",
    "- Experiment 3: Support Vector Machine (SVC with RBF Kernel) \n",
    "- Experiment 4: Logistic Regression (Softmax Regression) + PCA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset: MNIST\n",
    "\n",
    "\n",
    "We will use the MNIST dataset, which is a set of 70,000 small images of digits handwritten by high school students and employees of the US Census Bureau. Each image is labeled with the digit it represents.\n",
    "\n",
    "\n",
    "There are 70,000 images. Each image is 28x28 pixels, and each feature simply represents one pixel’s intensity, from 0 (white) to 255 (black).\n",
    "\n",
    "Thus, each image has 784 features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import time\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "from sklearn.datasets import fetch_openml\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import LinearSVC, SVC\n",
    "\n",
    "from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, classification_report\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.decomposition import PCA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data and Create Data Matrix (X) and the Label Vector (y)\n",
    "\n",
    "First load the data and explore the feature names, target names, etc.\n",
    "\n",
    "We may load the data from a local folder or load it directly from cloud using Scikit-Learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "No. of Samples:  (70000, 784)\n",
      "No. of Labels:  (70000,)\n",
      "\n",
      "X Type:  float64\n",
      "y Type int64\n"
     ]
    }
   ],
   "source": [
    "# Load the data from the local folder \"data\"\n",
    "# mnist = loadmat('data/mnist-original.mat')\n",
    "\n",
    "# Create the data Matrix X and the target vector y\n",
    "# X = mnist[\"data\"].T.astype('float64')\n",
    "# y = mnist[\"label\"][0].astype('int64')\n",
    "\n",
    "\n",
    "# Load data using Scikit-Learn\n",
    "mnist = fetch_openml('mnist_784', cache=False)\n",
    "\n",
    "X = mnist[\"data\"]\n",
    "y = mnist[\"target\"].astype('int64')\n",
    "\n",
    "\n",
    "print(\"\\nNo. of Samples: \", X.shape)\n",
    "print(\"No. of Labels: \", y.shape)\n",
    "\n",
    "\n",
    "print(\"\\nX Type: \", X.dtype)\n",
    "print(\"y Type\", y.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split Data Into Training and Test Sets\n",
    "\n",
    "We use sklearn's train_test_split function to spilt the dataset into training and test subsets. The data is shuffled by default before splitting.\n",
    "\n",
    "This function splits arrays or matrices into **random** train and test subsets.\n",
    "\n",
    "For the **reproducibility of the results**, we need to use the same seed for the random number generator. The seed is set by the \"random_state\" parameter of the split function. \n",
    "\n",
    "However, in repeated experiments if we don't want to use the same train and test subsets, then we drop the \"random_state\" parameter from the funtion.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimization Using Dimensionaly Reduction\n",
    "\n",
    "We can optimize the running-time of the Logistic Regression algorithm by reducing the number of features. Our assumption is that the essence or core content of the data does not span along all dimensions. The technique for reducing the dimension of data is known as dimensionality reduction.\n",
    "\n",
    "For a gentle introduction to various dimensionality reduction technique, see the notebook \"Dimensionality Reduction\" in the Github repository.\n",
    "\n",
    "We will use the Principle Component Analysis (PCA) dimensionality reduction technique to project the MNIST dataset (784 features) to a lower dimensional space by retaining maximum variance. \n",
    "\n",
    "The goal is to see the improvement in training time due to this dimensionality reduction.\n",
    "\n",
    "Before we apply the PCA, we need to standardize the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Standardize the Data\n",
    "\n",
    "PCA is influenced by scale of the data. Thus we need to scale the features of the data before applying PCA. \n",
    "\n",
    "For understanding the negative effect of not scaling the data, see the following post:\n",
    "\n",
    "https://scikit-learn.org/stable/auto_examples/preprocessing/plot_scaling_importance.html#sphx-glr-auto-examples-preprocessing-plot-scaling-importance-py\n",
    "\n",
    "Note that we fit the scaler on the training set and transform on the training and test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "\n",
    "# Fit on training set only.\n",
    "scaler.fit(X_train)\n",
    "\n",
    "# Apply transform to both the training set and the test set.\n",
    "X_train = scaler.transform(X_train)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Apply PCA\n",
    "\n",
    "While applying PCA we can set the number of principle components by the \"n_components\" attribute. But more importantly, we can use this attribute to determine the % of variance we want to retain in the extracted features.\n",
    "\n",
    "For example, if we set it to 0.95, sklearn will choose the **minimum number of principal components** such that 95% of the variance is retained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 6.66 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "PCA(copy=True, iterated_power='auto', n_components=0.95, random_state=None,\n",
       "    svd_solver='auto', tol=0.0, whiten=False)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "pca = PCA(n_components=0.95)\n",
    "\n",
    "pca.fit(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Number of Principle Components\n",
    "\n",
    "We can find how many components PCA chose after fitting the model by using the following attribute: n_components_\n",
    "\n",
    "We will see that 95% of the variance amounts to **315 principal components**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Principle Components:  327\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of Principle Components: \", pca.n_components_)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Apply the Mapping (Transform) to both the Training Set and the Test Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_pca = pca.transform(X_train)\n",
    "X_test_pca = pca.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiments\n",
    "\n",
    "We will conduct the following experiments.\n",
    "\n",
    "- Experiment 1: Support Vector Machine (LinearSVC) + PCA\n",
    "- Experiment 2: Support Vector Machine (SVC with RBF Kernel) + PCA\n",
    "- Experiment 3: Support Vector Machine (SVC with RBF Kernel) \n",
    "- Experiment 4: Logistic Regression (Softmax Regression) + PCA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Support Vector Machine: Model Selection via Hyperparameter Tuning\n",
    "\n",
    "Note that we are not performing grid search (which we should have). \n",
    "\n",
    "We are simply using the best values for the two hyperparameters ($\\gamma$ and $C$) for the SVC from prior grid search. However, it is advised that one should perform grid search to fine tune the hyperparameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment 1: LinearSVC + PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Complete in 182.533s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\program files\\python37\\lib\\site-packages\\sklearn\\svm\\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n"
     ]
    }
   ],
   "source": [
    "t0 = time()\n",
    "\n",
    "linear_svc_pca = LinearSVC(loss='hinge', C=1, random_state=42)\n",
    "linear_svc_pca.fit(X_train_pca, y_train)\n",
    "\n",
    "training_LinearSVC = time() - t0\n",
    "\n",
    "print(\"Training Complete in %0.3fs\" % training_LinearSVC)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment 1: Evaluate LinearSVC + PCA on Test Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Test Accuracy:  0.9075714285714286\n",
      "\n",
      "Test Confusion Matrix:\n",
      "[[1353    1    4    1    1    8   15    0    4    0]\n",
      " [   0 1538    9    6    3    8    2    3   11    0]\n",
      " [  17   16 1290   11   20    2   23   19   41    4]\n",
      " [   6    8   62 1233    3   48    5   15   33   22]\n",
      " [   5    3   11    1 1264    5    5    8    6   42]\n",
      " [  11    8   11   28   16 1079   29    7   33    9]\n",
      " [  13    8   15    7   12   25 1297    2    5    3]\n",
      " [   8    9   32    8   22    7    3 1328    7   34]\n",
      " [  10   40   15   27   12   52    8    5 1169   30]\n",
      " [   8    6   16   26   65   16    1   54   14 1155]]\n",
      "\n",
      "Classification Report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      0.98      0.96      1387\n",
      "           1       0.94      0.97      0.96      1580\n",
      "           2       0.88      0.89      0.89      1443\n",
      "           3       0.91      0.86      0.89      1435\n",
      "           4       0.89      0.94      0.91      1350\n",
      "           5       0.86      0.88      0.87      1231\n",
      "           6       0.93      0.94      0.93      1387\n",
      "           7       0.92      0.91      0.92      1458\n",
      "           8       0.88      0.85      0.87      1368\n",
      "           9       0.89      0.85      0.87      1361\n",
      "\n",
      "    accuracy                           0.91     14000\n",
      "   macro avg       0.91      0.91      0.91     14000\n",
      "weighted avg       0.91      0.91      0.91     14000\n",
      "\n",
      "Wall time: 53 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "y_test_predicted = linear_svc_pca.predict(X_test_pca)\n",
    "\n",
    "accuracy_score_test_LinearSVC = np.mean(y_test_predicted == y_test)\n",
    "print(\"\\nTest Accuracy: \", accuracy_score_test_LinearSVC)\n",
    "\n",
    "print(\"\\nTest Confusion Matrix:\")\n",
    "print(confusion_matrix(y_test, y_test_predicted))\n",
    "\n",
    "print(\"\\nClassification Report:\")\n",
    "print(classification_report(y_test, y_test_predicted))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment 2: SVC (RBF Kernel) + PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Complete in 180.884s\n"
     ]
    }
   ],
   "source": [
    "t0 = time()\n",
    "\n",
    "svm_clf_pca = SVC(C=1, gamma=0.001)\n",
    "svm_clf_pca.fit(X_train_pca, y_train)\n",
    "\n",
    "training_SVC_rbf_pca = time() - t0\n",
    "\n",
    "print(\"Training Complete in %0.3fs\" % training_SVC_rbf_pca)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment 2: Evaluate SVC (RBF Kernel) + PCA on Test Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Test Accuracy:  0.9612857142857143\n",
      "\n",
      "Test Confusion Matrix:\n",
      "[[1364    0    4    0    1    2    8    3    5    0]\n",
      " [   0 1564    5    0    2    0    0    6    2    1]\n",
      " [   6    4 1389    7    6    1    2   14   11    3]\n",
      " [   1    7   21 1350    2   18    0   13   20    3]\n",
      " [   2    2   11    0 1305    1    2    5    1   21]\n",
      " [   3    4    8   12    2 1172   13    8    8    1]\n",
      " [   6    1    5    0    6    9 1344   13    3    0]\n",
      " [   2    7   16    1    9    0    0 1402    1   20]\n",
      " [   1   15    6    8    5   14    5    5 1296   13]\n",
      " [   1    0    6   17   28    5    0   26    6 1272]]\n",
      "\n",
      "Classification Report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.98      0.98      1387\n",
      "           1       0.98      0.99      0.98      1580\n",
      "           2       0.94      0.96      0.95      1443\n",
      "           3       0.97      0.94      0.95      1435\n",
      "           4       0.96      0.97      0.96      1350\n",
      "           5       0.96      0.95      0.96      1231\n",
      "           6       0.98      0.97      0.97      1387\n",
      "           7       0.94      0.96      0.95      1458\n",
      "           8       0.96      0.95      0.95      1368\n",
      "           9       0.95      0.93      0.94      1361\n",
      "\n",
      "    accuracy                           0.96     14000\n",
      "   macro avg       0.96      0.96      0.96     14000\n",
      "weighted avg       0.96      0.96      0.96     14000\n",
      "\n",
      "Wall time: 55.5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "y_test_predicted = svm_clf_pca.predict(X_test_pca)\n",
    "\n",
    "accuracy_score_test_svc_rbf_pca = np.mean(y_test_predicted == y_test)\n",
    "print(\"\\nTest Accuracy: \", accuracy_score_test_svc_rbf_pca)\n",
    "\n",
    "print(\"\\nTest Confusion Matrix:\")\n",
    "print(confusion_matrix(y_test, y_test_predicted))\n",
    "\n",
    "print(\"\\nClassification Report:\")\n",
    "print(classification_report(y_test, y_test_predicted))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment 3: SVC (RBF Kernel) \n",
    "\n",
    "We experiment with the SVC (RBF Kernel) without applying dimensionaly reducion on the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Complete in 391.316s\n"
     ]
    }
   ],
   "source": [
    "t0 = time()\n",
    "\n",
    "svm_clf = SVC(gamma=0.001)\n",
    "svm_clf.fit(X_train, y_train)\n",
    "\n",
    "training_SVC_rbf = time() - t0\n",
    "\n",
    "print(\"Training Complete in %0.3fs\" % training_SVC_rbf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment 3: Evaluate SVC (RBF Kernel) on Test Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy:  0.9604285714285714\n",
      "\n",
      "Test Confusion Matrix:\n",
      "[[1364    0    4    0    1    2    8    3    5    0]\n",
      " [   0 1564    5    0    2    0    0    5    3    1]\n",
      " [   4    4 1389    8    7    1    2   14   11    3]\n",
      " [   1    7   19 1350    1   19    0   15   20    3]\n",
      " [   2    2   10    0 1303    1    2    5    1   24]\n",
      " [   3    4    8   13    2 1170   13    9    8    1]\n",
      " [   6    1    4    0    7    9 1343   14    3    0]\n",
      " [   2    6   16    1    9    0    0 1401    2   21]\n",
      " [   1   16    6    8    4   14    4    5 1297   13]\n",
      " [   2    0    7   17   29    6    1   28    6 1265]]\n",
      "\n",
      "Classification Report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.98      0.98      1387\n",
      "           1       0.98      0.99      0.98      1580\n",
      "           2       0.95      0.96      0.95      1443\n",
      "           3       0.97      0.94      0.95      1435\n",
      "           4       0.95      0.97      0.96      1350\n",
      "           5       0.96      0.95      0.95      1231\n",
      "           6       0.98      0.97      0.97      1387\n",
      "           7       0.93      0.96      0.95      1458\n",
      "           8       0.96      0.95      0.95      1368\n",
      "           9       0.95      0.93      0.94      1361\n",
      "\n",
      "    accuracy                           0.96     14000\n",
      "   macro avg       0.96      0.96      0.96     14000\n",
      "weighted avg       0.96      0.96      0.96     14000\n",
      "\n",
      "Wall time: 2min 13s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "y_test_predicted = svm_clf.predict(X_test)\n",
    "\n",
    "accuracy_score_test_svc_rbf = np.mean(y_test_predicted == y_test)\n",
    "print(\"\\nAccuracy: \", accuracy_score_test_svc_rbf)\n",
    "\n",
    "\n",
    "print(\"\\nTest Confusion Matrix:\")\n",
    "print(confusion_matrix(y_test, y_test_predicted))\n",
    "\n",
    "\n",
    "print(\"\\nClassification Report:\")\n",
    "print(classification_report(y_test, y_test_predicted))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment 4: Logistic Regression (Softmax Regression) + PCA\n",
    "\n",
    "We use the best performing solver (i.e., lbfgs) from previous notebook to train the logistic regression model on the PCA transformed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Complete in 7.420s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\program files\\python37\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n",
      "  \"of iterations.\", ConvergenceWarning)\n"
     ]
    }
   ],
   "source": [
    "t0 = time()\n",
    "\n",
    "softmax_reg_pca = LogisticRegression(solver='lbfgs', multi_class='multinomial')\n",
    "\n",
    "softmax_reg_pca.fit(X_train_pca, y_train)\n",
    "\n",
    "training_logistic = time() - t0\n",
    "\n",
    "print(\"Training Complete in %0.3fs\" % training_logistic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment 4: Evaluate Softmax Regression + PCA on Test Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No. of Iterations: [100]\n",
      "\n",
      "Test Accuracy:  0.9187142857142857\n",
      "\n",
      "Test Confusion Matrix:\n",
      "[[1347    0    4    1    2   10   16    0    6    1]\n",
      " [   0 1537   10    2    2    6    2    3   16    2]\n",
      " [   7   23 1291   23   19    6   16   17   36    5]\n",
      " [   2    8   51 1264    2   45    2   15   32   14]\n",
      " [   1    4    9    2 1261    2   18   11    5   37]\n",
      " [  10    6   16   40    7 1085   19    5   35    8]\n",
      " [  14    7   13    1   13   16 1315    0    8    0]\n",
      " [   3    6   22    6   16    5    0 1341    6   53]\n",
      " [   5   36   11   28   10   37   10    3 1209   19]\n",
      " [   5    3    6   15   49   11    1   48   11 1212]]\n",
      "\n",
      "Classification Report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.97      0.97      1387\n",
      "           1       0.94      0.97      0.96      1580\n",
      "           2       0.90      0.89      0.90      1443\n",
      "           3       0.91      0.88      0.90      1435\n",
      "           4       0.91      0.93      0.92      1350\n",
      "           5       0.89      0.88      0.88      1231\n",
      "           6       0.94      0.95      0.94      1387\n",
      "           7       0.93      0.92      0.92      1458\n",
      "           8       0.89      0.88      0.89      1368\n",
      "           9       0.90      0.89      0.89      1361\n",
      "\n",
      "    accuracy                           0.92     14000\n",
      "   macro avg       0.92      0.92      0.92     14000\n",
      "weighted avg       0.92      0.92      0.92     14000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"No. of Iterations:\", softmax_reg_pca.n_iter_ )\n",
    "\n",
    "\n",
    "y_test_predicted = softmax_reg_pca.predict(X_test_pca)\n",
    "#print(y_test_predict)\n",
    "\n",
    "accuracy_score_test_logistic = np.mean(y_test_predicted == y_test)\n",
    "print(\"\\nTest Accuracy: \", accuracy_score_test_logistic)\n",
    "\n",
    "\n",
    "print(\"\\nTest Confusion Matrix:\")\n",
    "print(confusion_matrix(y_test, y_test_predicted))\n",
    "\n",
    "\n",
    "print(\"\\nClassification Report:\")\n",
    "print(classification_report(y_test, y_test_predicted))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary of Results from 4 Experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Classifier</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Running-Time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>LinearSVC + PCA</td>\n",
       "      <td>0.907571</td>\n",
       "      <td>182.532686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>SVM(RBF) + PCA</td>\n",
       "      <td>0.961286</td>\n",
       "      <td>180.884422</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>SVM(RBF)</td>\n",
       "      <td>0.960429</td>\n",
       "      <td>391.315884</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>Softmax + PCA</td>\n",
       "      <td>0.918714</td>\n",
       "      <td>7.419869</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Classifier  Accuracy  Running-Time\n",
       "0  LinearSVC + PCA  0.907571    182.532686\n",
       "1   SVM(RBF) + PCA  0.961286    180.884422\n",
       "2         SVM(RBF)  0.960429    391.315884\n",
       "3    Softmax + PCA  0.918714      7.419869"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = [[\"LinearSVC + PCA\", accuracy_score_test_LinearSVC, training_LinearSVC], \n",
    "        [\"SVM(RBF) + PCA\", accuracy_score_test_svc_rbf_pca, training_SVC_rbf_pca],\n",
    "        [\"SVM(RBF)\", accuracy_score_test_svc_rbf, training_SVC_rbf],\n",
    "        [\"Softmax + PCA\", accuracy_score_test_logistic, training_logistic]]\n",
    "\n",
    "pd.DataFrame(data, columns=[\"Classifier\", \"Accuracy\", \"Running-Time\"])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparative Understanding\n",
    "\n",
    "We have done 4 experiments using SVM and Logistic Regression classifiers.\n",
    "\n",
    "The first 3 experiements are done using 2 SVM algorithms, with the effect of PCA.\n",
    "\n",
    "The experimental results confirm our hypotheses:\n",
    "- The kernelized SVM models will perform significantly better than the linear SVM model.\n",
    "- The RBF Kernel based SVM will perform better than Softmax regression classifier.\n",
    "- Dimensionaly reduction (by retaining maximum variance) should improve the performance.\n",
    "\n",
    "We make following observations.\n",
    "- The SVM classifiers perform **significantly** better than the Softmax classifier.\n",
    "- The SVM classifier training and prediction time is **longer**.\n",
    "- The RBF kernel based SVM classifier performs better than the linear SVM classifier. It indicates that for this non-linear image classsification problem the kernelized SVM is the most suitable algorithm.\n",
    "- Dimensionality reduction improves the performance slightly on the RBF kernel based SVM.\n",
    "\n",
    "### Thus, for image classification problems RBF kernel based SVM model should be used with dimensionality reduction."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}