P3_PerformanceEstimation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9d85ec88",
   "metadata": {},
   "source": [
    "# Part 3 - Performance estimation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d973b1d7",
   "metadata": {},
   "source": [
    "## Performance estimation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3929defa",
   "metadata": {},
   "source": [
    "In this part the gathered standardized features are used. <br>\n",
    "The performance of each model is estimated using nested cross validation, 10-fold cross validation for outer and <br>\n",
    "5-fold repeated cross validation with 3 repetitions for inner loop.  <br> \n",
    "The best model is selected in the inner loop using the hyperparameter combinations and ranges defined in the Part 2. <br>\n",
    "For each model, calculate the accuracy and the confusion matrix. <br> \n",
    "Finally I report which hyperparameter/hyperparameter combination is most often chosen as the best one for each classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "f562b9f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.model_selection import RepeatedKFold, GridSearchCV, KFold, train_test_split, cross_validate\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn import metrics\n",
    "import matplotlib.pyplot as plt \n",
    "import seaborn as sb\n",
    "from statistics import mode\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c28f5204",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_b</th>\n",
       "      <th>var_b</th>\n",
       "      <th>skew_b</th>\n",
       "      <th>kurt_b</th>\n",
       "      <th>entr_b</th>\n",
       "      <th>mean_g</th>\n",
       "      <th>var_g</th>\n",
       "      <th>skew_g</th>\n",
       "      <th>kurt_g</th>\n",
       "      <th>entr_g</th>\n",
       "      <th>...</th>\n",
       "      <th>kurt_r</th>\n",
       "      <th>entr_r</th>\n",
       "      <th>major_axis_length</th>\n",
       "      <th>minor_axis_length</th>\n",
       "      <th>area</th>\n",
       "      <th>perimeter</th>\n",
       "      <th>roundness</th>\n",
       "      <th>aspect_ratio</th>\n",
       "      <th>class</th>\n",
       "      <th>class_int</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>237.701602</td>\n",
       "      <td>235.550005</td>\n",
       "      <td>-0.916309</td>\n",
       "      <td>0.337702</td>\n",
       "      <td>5.498779</td>\n",
       "      <td>231.293858</td>\n",
       "      <td>301.677048</td>\n",
       "      <td>-0.746753</td>\n",
       "      <td>-0.196393</td>\n",
       "      <td>5.868242</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.155548</td>\n",
       "      <td>5.841225</td>\n",
       "      <td>146.282043</td>\n",
       "      <td>72.265778</td>\n",
       "      <td>8084.5</td>\n",
       "      <td>371.404109</td>\n",
       "      <td>0.736495</td>\n",
       "      <td>2.024223</td>\n",
       "      <td>Arb</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>236.918079</td>\n",
       "      <td>157.647628</td>\n",
       "      <td>-0.768407</td>\n",
       "      <td>0.436432</td>\n",
       "      <td>5.485904</td>\n",
       "      <td>232.733608</td>\n",
       "      <td>191.903693</td>\n",
       "      <td>-0.752232</td>\n",
       "      <td>0.325161</td>\n",
       "      <td>5.688247</td>\n",
       "      <td>...</td>\n",
       "      <td>0.349649</td>\n",
       "      <td>5.657413</td>\n",
       "      <td>137.059006</td>\n",
       "      <td>65.893982</td>\n",
       "      <td>6933.5</td>\n",
       "      <td>354.232535</td>\n",
       "      <td>0.694361</td>\n",
       "      <td>2.079993</td>\n",
       "      <td>Arb</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>216.536679</td>\n",
       "      <td>259.919503</td>\n",
       "      <td>-0.839951</td>\n",
       "      <td>-0.140669</td>\n",
       "      <td>5.501954</td>\n",
       "      <td>210.975351</td>\n",
       "      <td>283.247363</td>\n",
       "      <td>-0.772167</td>\n",
       "      <td>-0.337009</td>\n",
       "      <td>5.604155</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.316691</td>\n",
       "      <td>5.572969</td>\n",
       "      <td>132.384201</td>\n",
       "      <td>67.964844</td>\n",
       "      <td>6929.5</td>\n",
       "      <td>351.178713</td>\n",
       "      <td>0.706082</td>\n",
       "      <td>1.947834</td>\n",
       "      <td>Arb</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>212.078894</td>\n",
       "      <td>167.887738</td>\n",
       "      <td>-0.647664</td>\n",
       "      <td>0.068170</td>\n",
       "      <td>5.637232</td>\n",
       "      <td>207.893328</td>\n",
       "      <td>195.452218</td>\n",
       "      <td>-0.596941</td>\n",
       "      <td>-0.158326</td>\n",
       "      <td>5.740592</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.192698</td>\n",
       "      <td>5.729030</td>\n",
       "      <td>149.286682</td>\n",
       "      <td>71.326492</td>\n",
       "      <td>8045.5</td>\n",
       "      <td>384.232536</td>\n",
       "      <td>0.684817</td>\n",
       "      <td>2.093005</td>\n",
       "      <td>Arb</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>233.318557</td>\n",
       "      <td>211.242371</td>\n",
       "      <td>-0.424428</td>\n",
       "      <td>-0.734968</td>\n",
       "      <td>5.665341</td>\n",
       "      <td>221.524494</td>\n",
       "      <td>225.958931</td>\n",
       "      <td>-0.303483</td>\n",
       "      <td>-0.890377</td>\n",
       "      <td>5.749867</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.871835</td>\n",
       "      <td>5.735764</td>\n",
       "      <td>140.248886</td>\n",
       "      <td>71.801796</td>\n",
       "      <td>7663.5</td>\n",
       "      <td>365.060963</td>\n",
       "      <td>0.722614</td>\n",
       "      <td>1.953278</td>\n",
       "      <td>Arb</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       mean_b       var_b    skew_b    kurt_b    entr_b      mean_g  \\\n",
       "0  237.701602  235.550005 -0.916309  0.337702  5.498779  231.293858   \n",
       "1  236.918079  157.647628 -0.768407  0.436432  5.485904  232.733608   \n",
       "2  216.536679  259.919503 -0.839951 -0.140669  5.501954  210.975351   \n",
       "3  212.078894  167.887738 -0.647664  0.068170  5.637232  207.893328   \n",
       "4  233.318557  211.242371 -0.424428 -0.734968  5.665341  221.524494   \n",
       "\n",
       "        var_g    skew_g    kurt_g    entr_g  ...    kurt_r    entr_r  \\\n",
       "0  301.677048 -0.746753 -0.196393  5.868242  ... -0.155548  5.841225   \n",
       "1  191.903693 -0.752232  0.325161  5.688247  ...  0.349649  5.657413   \n",
       "2  283.247363 -0.772167 -0.337009  5.604155  ... -0.316691  5.572969   \n",
       "3  195.452218 -0.596941 -0.158326  5.740592  ... -0.192698  5.729030   \n",
       "4  225.958931 -0.303483 -0.890377  5.749867  ... -0.871835  5.735764   \n",
       "\n",
       "   major_axis_length  minor_axis_length    area   perimeter  roundness  \\\n",
       "0         146.282043          72.265778  8084.5  371.404109   0.736495   \n",
       "1         137.059006          65.893982  6933.5  354.232535   0.694361   \n",
       "2         132.384201          67.964844  6929.5  351.178713   0.706082   \n",
       "3         149.286682          71.326492  8045.5  384.232536   0.684817   \n",
       "4         140.248886          71.801796  7663.5  365.060963   0.722614   \n",
       "\n",
       "   aspect_ratio  class  class_int  \n",
       "0      2.024223    Arb          0  \n",
       "1      2.079993    Arb          0  \n",
       "2      1.947834    Arb          0  \n",
       "3      2.093005    Arb          0  \n",
       "4      1.953278    Arb          0  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# I got this box and the next from my own report of Exercise2\n",
    "# I am using the output of previous part directly\n",
    "\n",
    "data_path = '../training_data/rice_feature_data.parquet' # path of data\n",
    "data = pd.read_parquet(data_path)               # reading parquet file\n",
    "data.head()                                     # printing a few lines to see if everything is ok"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "437256cd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_b</th>\n",
       "      <th>var_b</th>\n",
       "      <th>skew_b</th>\n",
       "      <th>kurt_b</th>\n",
       "      <th>entr_b</th>\n",
       "      <th>mean_g</th>\n",
       "      <th>var_g</th>\n",
       "      <th>skew_g</th>\n",
       "      <th>kurt_g</th>\n",
       "      <th>entr_g</th>\n",
       "      <th>...</th>\n",
       "      <th>kurt_r</th>\n",
       "      <th>entr_r</th>\n",
       "      <th>major_axis_length</th>\n",
       "      <th>minor_axis_length</th>\n",
       "      <th>area</th>\n",
       "      <th>perimeter</th>\n",
       "      <th>roundness</th>\n",
       "      <th>aspect_ratio</th>\n",
       "      <th>class</th>\n",
       "      <th>class_int</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.702268</td>\n",
       "      <td>1.296104</td>\n",
       "      <td>-0.518657</td>\n",
       "      <td>-0.603753</td>\n",
       "      <td>0.737087</td>\n",
       "      <td>1.381771</td>\n",
       "      <td>1.453728</td>\n",
       "      <td>-0.918784</td>\n",
       "      <td>-0.539606</td>\n",
       "      <td>1.002590</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.548778</td>\n",
       "      <td>0.967070</td>\n",
       "      <td>-0.662878</td>\n",
       "      <td>1.468420</td>\n",
       "      <td>0.858662</td>\n",
       "      <td>-0.382661</td>\n",
       "      <td>1.466364</td>\n",
       "      <td>-1.133834</td>\n",
       "      <td>Arb</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.637647</td>\n",
       "      <td>0.349329</td>\n",
       "      <td>-0.344452</td>\n",
       "      <td>-0.576530</td>\n",
       "      <td>0.711366</td>\n",
       "      <td>1.527036</td>\n",
       "      <td>0.185969</td>\n",
       "      <td>-0.925793</td>\n",
       "      <td>-0.351919</td>\n",
       "      <td>0.566264</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.335125</td>\n",
       "      <td>0.485281</td>\n",
       "      <td>-0.936849</td>\n",
       "      <td>0.908895</td>\n",
       "      <td>-0.000719</td>\n",
       "      <td>-0.702840</td>\n",
       "      <td>1.079154</td>\n",
       "      <td>-1.080698</td>\n",
       "      <td>Arb</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1.043300</td>\n",
       "      <td>1.592275</td>\n",
       "      <td>-0.428719</td>\n",
       "      <td>-0.735656</td>\n",
       "      <td>0.743432</td>\n",
       "      <td>-0.668283</td>\n",
       "      <td>1.240886</td>\n",
       "      <td>-0.951299</td>\n",
       "      <td>-0.590208</td>\n",
       "      <td>0.362418</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.616928</td>\n",
       "      <td>0.263944</td>\n",
       "      <td>-1.075714</td>\n",
       "      <td>1.090743</td>\n",
       "      <td>-0.003706</td>\n",
       "      <td>-0.759781</td>\n",
       "      <td>1.186873</td>\n",
       "      <td>-1.206614</td>\n",
       "      <td>Arb</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-1.410954</td>\n",
       "      <td>0.473781</td>\n",
       "      <td>-0.202237</td>\n",
       "      <td>-0.678072</td>\n",
       "      <td>1.013700</td>\n",
       "      <td>-0.979246</td>\n",
       "      <td>0.226950</td>\n",
       "      <td>-0.727107</td>\n",
       "      <td>-0.525907</td>\n",
       "      <td>0.693155</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.564490</td>\n",
       "      <td>0.672997</td>\n",
       "      <td>-0.573624</td>\n",
       "      <td>1.385939</td>\n",
       "      <td>0.829544</td>\n",
       "      <td>-0.143464</td>\n",
       "      <td>0.991443</td>\n",
       "      <td>-1.068301</td>\n",
       "      <td>Arb</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.340778</td>\n",
       "      <td>1.000685</td>\n",
       "      <td>0.060698</td>\n",
       "      <td>-0.899523</td>\n",
       "      <td>1.069858</td>\n",
       "      <td>0.396082</td>\n",
       "      <td>0.579269</td>\n",
       "      <td>-0.351640</td>\n",
       "      <td>-0.789344</td>\n",
       "      <td>0.715637</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.851704</td>\n",
       "      <td>0.690646</td>\n",
       "      <td>-0.842093</td>\n",
       "      <td>1.427676</td>\n",
       "      <td>0.544327</td>\n",
       "      <td>-0.500935</td>\n",
       "      <td>1.338797</td>\n",
       "      <td>-1.201427</td>\n",
       "      <td>Arb</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     mean_b     var_b    skew_b    kurt_b    entr_b    mean_g     var_g  \\\n",
       "0  0.702268  1.296104 -0.518657 -0.603753  0.737087  1.381771  1.453728   \n",
       "1  0.637647  0.349329 -0.344452 -0.576530  0.711366  1.527036  0.185969   \n",
       "2 -1.043300  1.592275 -0.428719 -0.735656  0.743432 -0.668283  1.240886   \n",
       "3 -1.410954  0.473781 -0.202237 -0.678072  1.013700 -0.979246  0.226950   \n",
       "4  0.340778  1.000685  0.060698 -0.899523  1.069858  0.396082  0.579269   \n",
       "\n",
       "     skew_g    kurt_g    entr_g  ...    kurt_r    entr_r  major_axis_length  \\\n",
       "0 -0.918784 -0.539606  1.002590  ... -0.548778  0.967070          -0.662878   \n",
       "1 -0.925793 -0.351919  0.566264  ... -0.335125  0.485281          -0.936849   \n",
       "2 -0.951299 -0.590208  0.362418  ... -0.616928  0.263944          -1.075714   \n",
       "3 -0.727107 -0.525907  0.693155  ... -0.564490  0.672997          -0.573624   \n",
       "4 -0.351640 -0.789344  0.715637  ... -0.851704  0.690646          -0.842093   \n",
       "\n",
       "   minor_axis_length      area  perimeter  roundness  aspect_ratio  class  \\\n",
       "0           1.468420  0.858662  -0.382661   1.466364     -1.133834    Arb   \n",
       "1           0.908895 -0.000719  -0.702840   1.079154     -1.080698    Arb   \n",
       "2           1.090743 -0.003706  -0.759781   1.186873     -1.206614    Arb   \n",
       "3           1.385939  0.829544  -0.143464   0.991443     -1.068301    Arb   \n",
       "4           1.427676  0.544327  -0.500935   1.338797     -1.201427    Arb   \n",
       "\n",
       "   class_int  \n",
       "0          0  \n",
       "1          0  \n",
       "2          0  \n",
       "3          0  \n",
       "4          0  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# creating the columns for standardazing\n",
    "columns = data.columns.tolist()\n",
    "columns.remove('class')      # deleting class and class_int from features' columns\n",
    "columns.remove('class_int')\n",
    "\n",
    "# I am using z-score for standarizing. The output of this standarizing is between -3 and +3.\n",
    "# In z-score standarization, we use mean and standard deviation of the data set with observed sample.\n",
    "# The score determines how far a sample is from the mean, in scale of -3 to +3.\n",
    "\n",
    "scale= StandardScaler()                            # creating an instance of standardizer\n",
    "data[columns] = scale.fit_transform(data[columns]) # applying the transformation \n",
    "data.head()                                        # printing the data to check it"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "413e34ad",
   "metadata": {},
   "source": [
    "### KNN Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "fc551c95",
   "metadata": {},
   "outputs": [],
   "source": [
    "# I used Exercise 2 solution as a reference here\n",
    "# Also this website \n",
    "# https://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html\n",
    "\n",
    "y = data['class'].values       # creating labels. each sample belongs to a different class.\n",
    "X_train = data[columns].values # seperating classes and class values from features for training\n",
    "\n",
    "k_range=range(1,30)            # creating a range for K to be tested\n",
    "knn = KNeighborsClassifier()   # creating the knn model\n",
    "parameters={'n_neighbors': k_range}  # assigning k_range to be the parameter for knn\n",
    "\n",
    "outer_cv = KFold(n_splits=10)  # creating the outer cross validation with 10 folds\n",
    "inner_cv = RepeatedKFold(n_splits=5, n_repeats=3) # creating the inner cross validation with 5 folds\n",
    "                                                  # and 3 repetitions\n",
    "\n",
    "clf = GridSearchCV(knn, parameters, cv=inner_cv)  # creating a cross validation with knn estimator\n",
    "# evaluating our model with cross validation and saving the scores and estimator\n",
    "cv_final = cross_validate(clf, X=X_train, y=y, cv=outer_cv, return_estimator=True)\n",
    "\n",
    "cross_val_score = cv_final['test_score']          # saving the scores seperately\n",
    "cross_val_estimators = cv_final['estimator']      # saving the estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "dedaa8ef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.        , 1.        , 1.        , 1.        , 0.96666667,\n",
       "       0.93333333, 0.96666667, 1.        , 0.93333333, 0.93333333])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score  # printing the scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "06007050",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_neighbors': 1}\n",
      "{'n_neighbors': 1}\n",
      "{'n_neighbors': 3}\n",
      "{'n_neighbors': 1}\n",
      "{'n_neighbors': 1}\n",
      "{'n_neighbors': 3}\n",
      "{'n_neighbors': 2}\n",
      "{'n_neighbors': 1}\n",
      "{'n_neighbors': 1}\n",
      "{'n_neighbors': 3}\n"
     ]
    }
   ],
   "source": [
    "# printing the best params\n",
    "for es in cross_val_estimators:\n",
    "    print(es.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4ec5e58",
   "metadata": {},
   "source": [
    "#### result\n",
    "\n",
    "Most often hyper parametr: n_neighbors = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2ca2261f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9833333333333333\n",
      "Confusion Matrix:\n",
      "[[19  0  0]\n",
      " [ 0 25  0]\n",
      " [ 1  0 15]]\n",
      "Classification Report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.97        19\n",
      "           1       1.00      1.00      1.00        25\n",
      "           2       1.00      0.94      0.97        16\n",
      "\n",
      "    accuracy                           0.98        60\n",
      "   macro avg       0.98      0.98      0.98        60\n",
      "weighted avg       0.98      0.98      0.98        60\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# I am using the code from my own solution for part 2\n",
    "\n",
    "# spiliting the features and classes into X_train, X_test, y_train, y_test\n",
    "X_train, X_test, y_train, y_test = train_test_split(data[columns], data['class_int'], test_size=0.2)\n",
    "\n",
    "classifier = KNeighborsClassifier(n_neighbors=1) # best K\n",
    "classifier.fit(X_train, y_train)                 # training the classifier\n",
    "y_pred = classifier.predict(X_test)              # make predictions by the model\n",
    "print(metrics.accuracy_score(y_test,y_pred))     # printing the accuracy\n",
    "result = metrics.confusion_matrix(y_test, y_pred)# getting the confusion matrix\n",
    "\n",
    "# printing the result in an understandable format\n",
    "print(\"Confusion Matrix:\")\n",
    "print(result)\n",
    "result1 = metrics.classification_report(y_test, y_pred)\n",
    "print(\"Classification Report:\",)\n",
    "print (result1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe9442be",
   "metadata": {},
   "source": [
    "### Random Forest Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2e0d2163",
   "metadata": {},
   "outputs": [],
   "source": [
    "# I used Exercise 2 solution as a reference here\n",
    "\n",
    "y = data['class'].values       # creating labels. each sample belongs to a different class.\n",
    "X_train = data[columns].values # seperating classes and class values from features for training\n",
    "\n",
    "max_depths = [2, 4, 6, 8, 10, 12]    # Random Forest Params\n",
    "max_features = [2, 3, 4, 5, 6, 7, 8]\n",
    "parameters={'max_depth': max_depths,\n",
    "           'max_features': max_features}\n",
    "\n",
    "r_f_c = RandomForestClassifier(random_state=20)\n",
    "outer_cv = KFold(n_splits=10)                    # creating outer folds of cross validation\n",
    "inner_cv = RepeatedKFold(n_splits=5, n_repeats=3)# creating inner folds of cross validation\n",
    "\n",
    "clf = GridSearchCV(r_f_c, parameters, cv=inner_cv) # creating inner cross validation\n",
    "# creating outter cross validation\n",
    "cv_final = cross_validate(clf, X=X_train, y=y, cv=outer_cv, return_estimator=True)\n",
    "\n",
    "cross_val_score = cv_final['test_score']\n",
    "cross_val_estimators = cv_final['estimator']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d9b14a99",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.        , 1.        , 0.96666667, 0.96666667, 1.        ,\n",
       "       0.96666667, 1.        , 1.        , 1.        , 0.86666667])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score  # printing the scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "22bc1c21",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'max_depth': 4, 'max_features': 8}\n",
      "{'max_depth': 4, 'max_features': 7}\n",
      "{'max_depth': 4, 'max_features': 5}\n",
      "{'max_depth': 6, 'max_features': 5}\n",
      "{'max_depth': 6, 'max_features': 6}\n",
      "{'max_depth': 4, 'max_features': 4}\n",
      "{'max_depth': 4, 'max_features': 5}\n",
      "{'max_depth': 4, 'max_features': 5}\n",
      "{'max_depth': 8, 'max_features': 3}\n",
      "{'max_depth': 2, 'max_features': 6}\n"
     ]
    }
   ],
   "source": [
    "# printing the best params\n",
    "for es in cross_val_estimators:\n",
    "    print(es.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e418cef",
   "metadata": {},
   "source": [
    "#### result\n",
    "\n",
    "Most often hyper parametr: max_depth: 4, max_features: 7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "19f89d63",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9666666666666667\n",
      "Confusion Matrix:\n",
      "[[21  0  0]\n",
      " [ 0 16  0]\n",
      " [ 0  2 21]]\n",
      "Classification Report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00        21\n",
      "           1       0.89      1.00      0.94        16\n",
      "           2       1.00      0.91      0.95        23\n",
      "\n",
      "    accuracy                           0.97        60\n",
      "   macro avg       0.96      0.97      0.97        60\n",
      "weighted avg       0.97      0.97      0.97        60\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# I used Exercise 2 solution as a reference here\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(data[columns], data['class_int'], test_size=0.2)\n",
    "clf = RandomForestClassifier(max_depth=4, max_features=7) # best params\n",
    "clf.fit(X_train, y_train)\n",
    "y_pred = clf.predict(X_test)\n",
    "print(metrics.accuracy_score(y_test,y_pred))     # printing the accuracy\n",
    "result = metrics.confusion_matrix(y_test, y_pred)# getting the confusion matrix\n",
    "\n",
    "# printing the result in an understandable format\n",
    "print(\"Confusion Matrix:\")\n",
    "print(result)\n",
    "result1 = metrics.classification_report(y_test, y_pred)\n",
    "print(\"Classification Report:\",)\n",
    "print (result1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81483515",
   "metadata": {},
   "source": [
    "### MLP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b33b5bcf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# I used Exercise 2 solution as a reference here\n",
    "\n",
    "y = data['class'].values\n",
    "X_train = data[columns].values\n",
    "\n",
    "# creating MLP classifier\n",
    "mlp = MLPClassifier(max_iter=500, early_stopping=True, random_state=20)\n",
    "outer_cv = KFold(n_splits=10)                    # creating outer folds of cross validation\n",
    "inner_cv = RepeatedKFold(n_splits=5, n_repeats=3)# creating inner folds of cross validation\n",
    "\n",
    "# creating parameters for MLP classifier\n",
    "hidden_layer_sizes = [(i,) for i in range(3,22)]\n",
    "activation = ['logistic', 'relu']\n",
    "solver = ['sgd', 'adam']\n",
    "val_fr = [0.1, 0.5]\n",
    "\n",
    "# creating parameters for MLP classifier\n",
    "parameter_space = {\n",
    "    'hidden_layer_sizes': hidden_layer_sizes,\n",
    "    'activation': activation,\n",
    "    'solver': solver,\n",
    "    'validation_fraction': val_fr\n",
    "}\n",
    "\n",
    "clf = GridSearchCV(mlp, parameter_space, cv=inner_cv) # creating inner cross validation\n",
    "# creating outter cross validation\n",
    "cv_final = cross_validate(clf, X=X_train, y=y, cv=outer_cv, return_estimator=True)\n",
    "cross_val_score = cv_final['test_score']\n",
    "cross_val_estimators = cv_final['estimator']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6dc6072d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.93333333, 1.        , 0.96666667, 1.        , 1.        ,\n",
       "       0.9       , 0.9       , 1.        , 0.93333333, 0.83333333])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score  # printing the scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "54c026b7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'activation': 'relu', 'hidden_layer_sizes': (16,), 'solver': 'sgd', 'validation_fraction': 0.5}\n",
      "{'activation': 'relu', 'hidden_layer_sizes': (20,), 'solver': 'adam', 'validation_fraction': 0.5}\n",
      "{'activation': 'relu', 'hidden_layer_sizes': (16,), 'solver': 'adam', 'validation_fraction': 0.5}\n",
      "{'activation': 'relu', 'hidden_layer_sizes': (16,), 'solver': 'adam', 'validation_fraction': 0.5}\n",
      "{'activation': 'relu', 'hidden_layer_sizes': (16,), 'solver': 'adam', 'validation_fraction': 0.5}\n",
      "{'activation': 'relu', 'hidden_layer_sizes': (16,), 'solver': 'adam', 'validation_fraction': 0.5}\n",
      "{'activation': 'relu', 'hidden_layer_sizes': (20,), 'solver': 'adam', 'validation_fraction': 0.5}\n",
      "{'activation': 'relu', 'hidden_layer_sizes': (16,), 'solver': 'adam', 'validation_fraction': 0.5}\n",
      "{'activation': 'relu', 'hidden_layer_sizes': (16,), 'solver': 'adam', 'validation_fraction': 0.5}\n",
      "{'activation': 'relu', 'hidden_layer_sizes': (16,), 'solver': 'adam', 'validation_fraction': 0.5}\n"
     ]
    }
   ],
   "source": [
    "# printing the best params\n",
    "for es in cross_val_estimators:\n",
    "    print(es.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3109388a",
   "metadata": {},
   "source": [
    "#### result\n",
    "\n",
    "Most often hyper parametr: 'activation': 'relu', 'hidden_layer_sizes': (20,), 'solver': 'adam', 'validation_fraction': 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "fb329832",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9166666666666666\n",
      "Confusion Matrix:\n",
      "[[18  0  0]\n",
      " [ 0 21  2]\n",
      " [ 1  2 16]]\n",
      "Classification Report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.97        18\n",
      "           1       0.91      0.91      0.91        23\n",
      "           2       0.89      0.84      0.86        19\n",
      "\n",
      "    accuracy                           0.92        60\n",
      "   macro avg       0.92      0.92      0.92        60\n",
      "weighted avg       0.92      0.92      0.92        60\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/maryamt/.local/lib/python3.8/site-packages/sklearn/base.py:443: UserWarning: X has feature names, but MLPClassifier was fitted without feature names\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "# I used Exercise 2 solution as a reference here\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(data[columns], data['class_int'], test_size=0.2)\n",
    "clf = MLPClassifier( hidden_layer_sizes = 20, activation = 'relu', solver = 'adam',\n",
    "                    validation_fraction = 0.5, early_stopping=True).fit(X_train.values, y_train)\n",
    "y_pred = clf.predict(X_test)\n",
    "print(metrics.accuracy_score(y_test,y_pred))     # printing the accuracy\n",
    "result = metrics.confusion_matrix(y_test, y_pred)# getting the confusion matrix\n",
    "\n",
    "# printing the result in an understandable format\n",
    "print(\"Confusion Matrix:\")\n",
    "print(result)\n",
    "result1 = metrics.classification_report(y_test, y_pred)\n",
    "print(\"Classification Report:\",)\n",
    "print (result1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85f896a1",
   "metadata": {},
   "source": [
    "## Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18c6e2e9",
   "metadata": {},
   "source": [
    "Discussing my results\n",
    "\n",
    "- Which model performs the best? Why?\n",
    "- Ponder the limitations and generalization of the models. How well will the classifiers perform for data outside this data set?\n",
    "- Compare your results with the original article. Are they comparable?\n",
    "- Ponder applications for these type of models (classifying rice or other plant species), who could benefit from them? Ponder also what would be interesting to study more on this area?\n",
    "- What did you learn? What was difficult? Could you improve your own working process in some way?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "555d4b66",
   "metadata": {},
   "source": [
    "### My answers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5166c9f",
   "metadata": {},
   "source": [
    "- Based on my results Random Forest is working the best. The accuracy of the knn is higher, but precision and recal of the Random Forest match better and the results by this model are more trustworthy.\n",
    "  \n",
    "- Speaking of the same species with a data set capable of similar features, we evaluated and tested the model (untimately) with unseen data. The expectation is to perform with an accuracy near to what we have. Although, for other species or a different kind of data set, the result can not be reliable.\n",
    "\n",
    "\n",
    "- The highest accuracy reported on the paper is 99.91% which is much higher than ours! Nevertheless, we skipped some parts of the data set because of the limited computation capacity. This might affect the results too.\n",
    "\n",
    "- Rice is not something dangerous, but these classifications can be used to classify poisonous or non-poisonous species (for example in muchrooms). Its interesting for me that I can command machines do something that I dont have any ideas about :) . I am sure I can not guess the species my self.\n",
    "\n",
    "- I have never worked on image data as an anlytics task before. All I had was computer vision. It gave me a different prespective and was very injoyable as the template and hints were reallt helpful."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "896ca16f81e75fad27ddb3763c7bbbef4434df0538473c6ce1b19bddc2f51f11"
  },
  "kernelspec": {
   "display_name": "Python [conda env:env1]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.15"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "232px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}