Ankur_ML_Logistic_regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# This notebook explains logistic regression and how it is used for binary classsfication. \n",
    "\n",
    "## For the sake of this task, we will use the same dataset that we used in KNN- Breast Cancer prediction . As the dataset is about whether the cancer is miligant or benign i.e binary classification (1 or 0)\n",
    "\n",
    "### we choose this dataset beacuse there is not much preprocessing required so it allow us to explain more about the logistic regression and we use our logit model to predict the cancer is benign or milgant.\n",
    "\n",
    "Features are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass. They describe characteristics of the cell nuclei present in the image.\n",
    "in the 3-dimensional space is that described in: [K. P. Bennett and O. L. Mangasarian: \"Robust Linear Programming Discrimination of Two Linearly Inseparable Sets\", Optimization Methods and Software 1, 1992, 23-34].\n",
    "\n",
    "## Attribute Information:\n",
    "\n",
    "1) ID number\n",
    "\n",
    "2) Diagnosis (M = malignant, B = benign)\n",
    "\n",
    "Ten real-valued features are computed for each cell nucleus:\n",
    "\n",
    "a) radius (mean of distances from center to points on the perimeter)\n",
    "\n",
    "b) texture (standard deviation of gray-scale values)\n",
    "\n",
    "c) perimeter\n",
    "\n",
    "d) area\n",
    "\n",
    "e) smoothness (local variation in radius lengths)\n",
    "\n",
    "f) compactness (perimeter^2 / area - 1.0)\n",
    "\n",
    "g) concavity (severity of concave portions of the contour)\n",
    "\n",
    "h) concave points (number of concave portions of the contour)\n",
    "i) symmetry\n",
    "\n",
    "j) fractal dimension (\"coastline approximation\" - 1)\n",
    "\n",
    "The mean, standard error and \"worst\" or largest (mean of the three\n",
    "largest values) of these features were computed for each image,\n",
    "resulting in 30 features. For instance, field 3 is Mean Radius, field\n",
    "13 is Radius SE, field 23 is Worst Radius.\n",
    "\n",
    "All feature values are recoded with four significant digits.\n",
    "\n",
    "Missing attribute values: none\n",
    "\n",
    "Class distribution: 357 benign, 212 malignant\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing the relevant libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set()\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 569 entries, 0 to 568\n",
      "Data columns (total 33 columns):\n",
      "id                         569 non-null int64\n",
      "diagnosis                  569 non-null object\n",
      "radius_mean                569 non-null float64\n",
      "texture_mean               569 non-null float64\n",
      "perimeter_mean             569 non-null float64\n",
      "area_mean                  569 non-null float64\n",
      "smoothness_mean            569 non-null float64\n",
      "compactness_mean           569 non-null float64\n",
      "concavity_mean             569 non-null float64\n",
      "concave points_mean        569 non-null float64\n",
      "symmetry_mean              569 non-null float64\n",
      "fractal_dimension_mean     569 non-null float64\n",
      "radius_se                  569 non-null float64\n",
      "texture_se                 569 non-null float64\n",
      "perimeter_se               569 non-null float64\n",
      "area_se                    569 non-null float64\n",
      "smoothness_se              569 non-null float64\n",
      "compactness_se             569 non-null float64\n",
      "concavity_se               569 non-null float64\n",
      "concave points_se          569 non-null float64\n",
      "symmetry_se                569 non-null float64\n",
      "fractal_dimension_se       569 non-null float64\n",
      "radius_worst               569 non-null float64\n",
      "texture_worst              569 non-null float64\n",
      "perimeter_worst            569 non-null float64\n",
      "area_worst                 569 non-null float64\n",
      "smoothness_worst           569 non-null float64\n",
      "compactness_worst          569 non-null float64\n",
      "concavity_worst            569 non-null float64\n",
      "concave points_worst       569 non-null float64\n",
      "symmetry_worst             569 non-null float64\n",
      "fractal_dimension_worst    569 non-null float64\n",
      "Unnamed: 32                0 non-null float64\n",
      "dtypes: float64(31), int64(1), object(1)\n",
      "memory usage: 144.5+ KB\n"
     ]
    }
   ],
   "source": [
    "# Loading our dataset: \n",
    "data =  pd.read_csv(r'Kaggle_BreastCancer.csv')\n",
    "\n",
    "data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>diagnosis</th>\n",
       "      <th>radius_mean</th>\n",
       "      <th>texture_mean</th>\n",
       "      <th>perimeter_mean</th>\n",
       "      <th>area_mean</th>\n",
       "      <th>smoothness_mean</th>\n",
       "      <th>compactness_mean</th>\n",
       "      <th>concavity_mean</th>\n",
       "      <th>concave points_mean</th>\n",
       "      <th>...</th>\n",
       "      <th>texture_worst</th>\n",
       "      <th>perimeter_worst</th>\n",
       "      <th>area_worst</th>\n",
       "      <th>smoothness_worst</th>\n",
       "      <th>compactness_worst</th>\n",
       "      <th>concavity_worst</th>\n",
       "      <th>concave points_worst</th>\n",
       "      <th>symmetry_worst</th>\n",
       "      <th>fractal_dimension_worst</th>\n",
       "      <th>Unnamed: 32</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>842302</td>\n",
       "      <td>M</td>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>...</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>842517</td>\n",
       "      <td>M</td>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.0869</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>...</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>84300903</td>\n",
       "      <td>M</td>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.1974</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>...</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>84348301</td>\n",
       "      <td>M</td>\n",
       "      <td>11.42</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.2414</td>\n",
       "      <td>0.10520</td>\n",
       "      <td>...</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.2098</td>\n",
       "      <td>0.8663</td>\n",
       "      <td>0.6869</td>\n",
       "      <td>0.2575</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>84358402</td>\n",
       "      <td>M</td>\n",
       "      <td>20.29</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.1980</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>...</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 33 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         id diagnosis  radius_mean  texture_mean  perimeter_mean  area_mean  \\\n",
       "0    842302         M        17.99         10.38          122.80     1001.0   \n",
       "1    842517         M        20.57         17.77          132.90     1326.0   \n",
       "2  84300903         M        19.69         21.25          130.00     1203.0   \n",
       "3  84348301         M        11.42         20.38           77.58      386.1   \n",
       "4  84358402         M        20.29         14.34          135.10     1297.0   \n",
       "\n",
       "   smoothness_mean  compactness_mean  concavity_mean  concave points_mean  \\\n",
       "0          0.11840           0.27760          0.3001              0.14710   \n",
       "1          0.08474           0.07864          0.0869              0.07017   \n",
       "2          0.10960           0.15990          0.1974              0.12790   \n",
       "3          0.14250           0.28390          0.2414              0.10520   \n",
       "4          0.10030           0.13280          0.1980              0.10430   \n",
       "\n",
       "   ...  texture_worst  perimeter_worst  area_worst  smoothness_worst  \\\n",
       "0  ...          17.33           184.60      2019.0            0.1622   \n",
       "1  ...          23.41           158.80      1956.0            0.1238   \n",
       "2  ...          25.53           152.50      1709.0            0.1444   \n",
       "3  ...          26.50            98.87       567.7            0.2098   \n",
       "4  ...          16.67           152.20      1575.0            0.1374   \n",
       "\n",
       "   compactness_worst  concavity_worst  concave points_worst  symmetry_worst  \\\n",
       "0             0.6656           0.7119                0.2654          0.4601   \n",
       "1             0.1866           0.2416                0.1860          0.2750   \n",
       "2             0.4245           0.4504                0.2430          0.3613   \n",
       "3             0.8663           0.6869                0.2575          0.6638   \n",
       "4             0.2050           0.4000                0.1625          0.2364   \n",
       "\n",
       "   fractal_dimension_worst  Unnamed: 32  \n",
       "0                  0.11890          NaN  \n",
       "1                  0.08902          NaN  \n",
       "2                  0.08758          NaN  \n",
       "3                  0.17300          NaN  \n",
       "4                  0.07678          NaN  \n",
       "\n",
       "[5 rows x 33 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dropping the id and unnamed_32 columns as they are not very important.\n",
    "\n",
    "data1 = data.copy()\n",
    "\n",
    "\n",
    "data = data.drop(['id','Unnamed: 32'],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# defiining our variable of interest,\n",
    "# diagnosis is our target variable, which we have to predict and,\n",
    "# the remaining are our features which are independent and we see there's no null values!\n",
    "Y = data['diagnosis']\n",
    "\n",
    "X = data.drop('diagnosis',axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((569, 30), (569,))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape , Y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_reg = LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on class LogisticRegression in module sklearn.linear_model._logistic:\n",
      "\n",
      "class LogisticRegression(sklearn.base.BaseEstimator, sklearn.linear_model._base.LinearClassifierMixin, sklearn.linear_model._base.SparseCoefMixin)\n",
      " |  LogisticRegression(penalty='l2', *, dual=False, tol=0.0001, C=1.0, fit_intercept=True, intercept_scaling=1, class_weight=None, random_state=None, solver='lbfgs', max_iter=100, multi_class='auto', verbose=0, warm_start=False, n_jobs=None, l1_ratio=None)\n",
      " |  \n",
      " |  Logistic Regression (aka logit, MaxEnt) classifier.\n",
      " |  \n",
      " |  In the multiclass case, the training algorithm uses the one-vs-rest (OvR)\n",
      " |  scheme if the 'multi_class' option is set to 'ovr', and uses the\n",
      " |  cross-entropy loss if the 'multi_class' option is set to 'multinomial'.\n",
      " |  (Currently the 'multinomial' option is supported only by the 'lbfgs',\n",
      " |  'sag', 'saga' and 'newton-cg' solvers.)\n",
      " |  \n",
      " |  This class implements regularized logistic regression using the\n",
      " |  'liblinear' library, 'newton-cg', 'sag', 'saga' and 'lbfgs' solvers. **Note\n",
      " |  that regularization is applied by default**. It can handle both dense\n",
      " |  and sparse input. Use C-ordered arrays or CSR matrices containing 64-bit\n",
      " |  floats for optimal performance; any other input format will be converted\n",
      " |  (and copied).\n",
      " |  \n",
      " |  The 'newton-cg', 'sag', and 'lbfgs' solvers support only L2 regularization\n",
      " |  with primal formulation, or no regularization. The 'liblinear' solver\n",
      " |  supports both L1 and L2 regularization, with a dual formulation only for\n",
      " |  the L2 penalty. The Elastic-Net regularization is only supported by the\n",
      " |  'saga' solver.\n",
      " |  \n",
      " |  Read more in the :ref:`User Guide <logistic_regression>`.\n",
      " |  \n",
      " |  Parameters\n",
      " |  ----------\n",
      " |  penalty : {'l1', 'l2', 'elasticnet', 'none'}, default='l2'\n",
      " |      Used to specify the norm used in the penalization. The 'newton-cg',\n",
      " |      'sag' and 'lbfgs' solvers support only l2 penalties. 'elasticnet' is\n",
      " |      only supported by the 'saga' solver. If 'none' (not supported by the\n",
      " |      liblinear solver), no regularization is applied.\n",
      " |  \n",
      " |      .. versionadded:: 0.19\n",
      " |         l1 penalty with SAGA solver (allowing 'multinomial' + L1)\n",
      " |  \n",
      " |  dual : bool, default=False\n",
      " |      Dual or primal formulation. Dual formulation is only implemented for\n",
      " |      l2 penalty with liblinear solver. Prefer dual=False when\n",
      " |      n_samples > n_features.\n",
      " |  \n",
      " |  tol : float, default=1e-4\n",
      " |      Tolerance for stopping criteria.\n",
      " |  \n",
      " |  C : float, default=1.0\n",
      " |      Inverse of regularization strength; must be a positive float.\n",
      " |      Like in support vector machines, smaller values specify stronger\n",
      " |      regularization.\n",
      " |  \n",
      " |  fit_intercept : bool, default=True\n",
      " |      Specifies if a constant (a.k.a. bias or intercept) should be\n",
      " |      added to the decision function.\n",
      " |  \n",
      " |  intercept_scaling : float, default=1\n",
      " |      Useful only when the solver 'liblinear' is used\n",
      " |      and self.fit_intercept is set to True. In this case, x becomes\n",
      " |      [x, self.intercept_scaling],\n",
      " |      i.e. a \"synthetic\" feature with constant value equal to\n",
      " |      intercept_scaling is appended to the instance vector.\n",
      " |      The intercept becomes ``intercept_scaling * synthetic_feature_weight``.\n",
      " |  \n",
      " |      Note! the synthetic feature weight is subject to l1/l2 regularization\n",
      " |      as all other features.\n",
      " |      To lessen the effect of regularization on synthetic feature weight\n",
      " |      (and therefore on the intercept) intercept_scaling has to be increased.\n",
      " |  \n",
      " |  class_weight : dict or 'balanced', default=None\n",
      " |      Weights associated with classes in the form ``{class_label: weight}``.\n",
      " |      If not given, all classes are supposed to have weight one.\n",
      " |  \n",
      " |      The \"balanced\" mode uses the values of y to automatically adjust\n",
      " |      weights inversely proportional to class frequencies in the input data\n",
      " |      as ``n_samples / (n_classes * np.bincount(y))``.\n",
      " |  \n",
      " |      Note that these weights will be multiplied with sample_weight (passed\n",
      " |      through the fit method) if sample_weight is specified.\n",
      " |  \n",
      " |      .. versionadded:: 0.17\n",
      " |         *class_weight='balanced'*\n",
      " |  \n",
      " |  random_state : int, RandomState instance, default=None\n",
      " |      Used when ``solver`` == 'sag', 'saga' or 'liblinear' to shuffle the\n",
      " |      data. See :term:`Glossary <random_state>` for details.\n",
      " |  \n",
      " |  solver : {'newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga'},             default='lbfgs'\n",
      " |  \n",
      " |      Algorithm to use in the optimization problem.\n",
      " |  \n",
      " |      - For small datasets, 'liblinear' is a good choice, whereas 'sag' and\n",
      " |        'saga' are faster for large ones.\n",
      " |      - For multiclass problems, only 'newton-cg', 'sag', 'saga' and 'lbfgs'\n",
      " |        handle multinomial loss; 'liblinear' is limited to one-versus-rest\n",
      " |        schemes.\n",
      " |      - 'newton-cg', 'lbfgs', 'sag' and 'saga' handle L2 or no penalty\n",
      " |      - 'liblinear' and 'saga' also handle L1 penalty\n",
      " |      - 'saga' also supports 'elasticnet' penalty\n",
      " |      - 'liblinear' does not support setting ``penalty='none'``\n",
      " |  \n",
      " |      Note that 'sag' and 'saga' fast convergence is only guaranteed on\n",
      " |      features with approximately the same scale. You can\n",
      " |      preprocess the data with a scaler from sklearn.preprocessing.\n",
      " |  \n",
      " |      .. versionadded:: 0.17\n",
      " |         Stochastic Average Gradient descent solver.\n",
      " |      .. versionadded:: 0.19\n",
      " |         SAGA solver.\n",
      " |      .. versionchanged:: 0.22\n",
      " |          The default solver changed from 'liblinear' to 'lbfgs' in 0.22.\n",
      " |  \n",
      " |  max_iter : int, default=100\n",
      " |      Maximum number of iterations taken for the solvers to converge.\n",
      " |  \n",
      " |  multi_class : {'auto', 'ovr', 'multinomial'}, default='auto'\n",
      " |      If the option chosen is 'ovr', then a binary problem is fit for each\n",
      " |      label. For 'multinomial' the loss minimised is the multinomial loss fit\n",
      " |      across the entire probability distribution, *even when the data is\n",
      " |      binary*. 'multinomial' is unavailable when solver='liblinear'.\n",
      " |      'auto' selects 'ovr' if the data is binary, or if solver='liblinear',\n",
      " |      and otherwise selects 'multinomial'.\n",
      " |  \n",
      " |      .. versionadded:: 0.18\n",
      " |         Stochastic Average Gradient descent solver for 'multinomial' case.\n",
      " |      .. versionchanged:: 0.22\n",
      " |          Default changed from 'ovr' to 'auto' in 0.22.\n",
      " |  \n",
      " |  verbose : int, default=0\n",
      " |      For the liblinear and lbfgs solvers set verbose to any positive\n",
      " |      number for verbosity.\n",
      " |  \n",
      " |  warm_start : bool, default=False\n",
      " |      When set to True, reuse the solution of the previous call to fit as\n",
      " |      initialization, otherwise, just erase the previous solution.\n",
      " |      Useless for liblinear solver. See :term:`the Glossary <warm_start>`.\n",
      " |  \n",
      " |      .. versionadded:: 0.17\n",
      " |         *warm_start* to support *lbfgs*, *newton-cg*, *sag*, *saga* solvers.\n",
      " |  \n",
      " |  n_jobs : int, default=None\n",
      " |      Number of CPU cores used when parallelizing over classes if\n",
      " |      multi_class='ovr'\". This parameter is ignored when the ``solver`` is\n",
      " |      set to 'liblinear' regardless of whether 'multi_class' is specified or\n",
      " |      not. ``None`` means 1 unless in a :obj:`joblib.parallel_backend`\n",
      " |      context. ``-1`` means using all processors.\n",
      " |      See :term:`Glossary <n_jobs>` for more details.\n",
      " |  \n",
      " |  l1_ratio : float, default=None\n",
      " |      The Elastic-Net mixing parameter, with ``0 <= l1_ratio <= 1``. Only\n",
      " |      used if ``penalty='elasticnet'``. Setting ``l1_ratio=0`` is equivalent\n",
      " |      to using ``penalty='l2'``, while setting ``l1_ratio=1`` is equivalent\n",
      " |      to using ``penalty='l1'``. For ``0 < l1_ratio <1``, the penalty is a\n",
      " |      combination of L1 and L2.\n",
      " |  \n",
      " |  Attributes\n",
      " |  ----------\n",
      " |  \n",
      " |  classes_ : ndarray of shape (n_classes, )\n",
      " |      A list of class labels known to the classifier.\n",
      " |  \n",
      " |  coef_ : ndarray of shape (1, n_features) or (n_classes, n_features)\n",
      " |      Coefficient of the features in the decision function.\n",
      " |  \n",
      " |      `coef_` is of shape (1, n_features) when the given problem is binary.\n",
      " |      In particular, when `multi_class='multinomial'`, `coef_` corresponds\n",
      " |      to outcome 1 (True) and `-coef_` corresponds to outcome 0 (False).\n",
      " |  \n",
      " |  intercept_ : ndarray of shape (1,) or (n_classes,)\n",
      " |      Intercept (a.k.a. bias) added to the decision function.\n",
      " |  \n",
      " |      If `fit_intercept` is set to False, the intercept is set to zero.\n",
      " |      `intercept_` is of shape (1,) when the given problem is binary.\n",
      " |      In particular, when `multi_class='multinomial'`, `intercept_`\n",
      " |      corresponds to outcome 1 (True) and `-intercept_` corresponds to\n",
      " |      outcome 0 (False).\n",
      " |  \n",
      " |  n_iter_ : ndarray of shape (n_classes,) or (1, )\n",
      " |      Actual number of iterations for all classes. If binary or multinomial,\n",
      " |      it returns only 1 element. For liblinear solver, only the maximum\n",
      " |      number of iteration across all classes is given.\n",
      " |  \n",
      " |      .. versionchanged:: 0.20\n",
      " |  \n",
      " |          In SciPy <= 1.0.0 the number of lbfgs iterations may exceed\n",
      " |          ``max_iter``. ``n_iter_`` will now report at most ``max_iter``.\n",
      " |  \n",
      " |  See Also\n",
      " |  --------\n",
      " |  SGDClassifier : Incrementally trained logistic regression (when given\n",
      " |      the parameter ``loss=\"log\"``).\n",
      " |  LogisticRegressionCV : Logistic regression with built-in cross validation.\n",
      " |  \n",
      " |  Notes\n",
      " |  -----\n",
      " |  The underlying C implementation uses a random number generator to\n",
      " |  select features when fitting the model. It is thus not uncommon,\n",
      " |  to have slightly different results for the same input data. If\n",
      " |  that happens, try with a smaller tol parameter.\n",
      " |  \n",
      " |  Predict output may not match that of standalone liblinear in certain\n",
      " |  cases. See :ref:`differences from liblinear <liblinear_differences>`\n",
      " |  in the narrative documentation.\n",
      " |  \n",
      " |  References\n",
      " |  ----------\n",
      " |  \n",
      " |  L-BFGS-B -- Software for Large-scale Bound-constrained Optimization\n",
      " |      Ciyou Zhu, Richard Byrd, Jorge Nocedal and Jose Luis Morales.\n",
      " |      http://users.iems.northwestern.edu/~nocedal/lbfgsb.html\n",
      " |  \n",
      " |  LIBLINEAR -- A Library for Large Linear Classification\n",
      " |      https://www.csie.ntu.edu.tw/~cjlin/liblinear/\n",
      " |  \n",
      " |  SAG -- Mark Schmidt, Nicolas Le Roux, and Francis Bach\n",
      " |      Minimizing Finite Sums with the Stochastic Average Gradient\n",
      " |      https://hal.inria.fr/hal-00860051/document\n",
      " |  \n",
      " |  SAGA -- Defazio, A., Bach F. & Lacoste-Julien S. (2014).\n",
      " |      SAGA: A Fast Incremental Gradient Method With Support\n",
      " |      for Non-Strongly Convex Composite Objectives\n",
      " |      https://arxiv.org/abs/1407.0202\n",
      " |  \n",
      " |  Hsiang-Fu Yu, Fang-Lan Huang, Chih-Jen Lin (2011). Dual coordinate descent\n",
      " |      methods for logistic regression and maximum entropy models.\n",
      " |      Machine Learning 85(1-2):41-75.\n",
      " |      https://www.csie.ntu.edu.tw/~cjlin/papers/maxent_dual.pdf\n",
      " |  \n",
      " |  Examples\n",
      " |  --------\n",
      " |  >>> from sklearn.datasets import load_iris\n",
      " |  >>> from sklearn.linear_model import LogisticRegression\n",
      " |  >>> X, y = load_iris(return_X_y=True)\n",
      " |  >>> clf = LogisticRegression(random_state=0).fit(X, y)\n",
      " |  >>> clf.predict(X[:2, :])\n",
      " |  array([0, 0])\n",
      " |  >>> clf.predict_proba(X[:2, :])\n",
      " |  array([[9.8...e-01, 1.8...e-02, 1.4...e-08],\n",
      " |         [9.7...e-01, 2.8...e-02, ...e-08]])\n",
      " |  >>> clf.score(X, y)\n",
      " |  0.97...\n",
      " |  \n",
      " |  Method resolution order:\n",
      " |      LogisticRegression\n",
      " |      sklearn.base.BaseEstimator\n",
      " |      sklearn.linear_model._base.LinearClassifierMixin\n",
      " |      sklearn.base.ClassifierMixin\n",
      " |      sklearn.linear_model._base.SparseCoefMixin\n",
      " |      builtins.object\n",
      " |  \n",
      " |  Methods defined here:\n",
      " |  \n",
      " |  __init__(self, penalty='l2', *, dual=False, tol=0.0001, C=1.0, fit_intercept=True, intercept_scaling=1, class_weight=None, random_state=None, solver='lbfgs', max_iter=100, multi_class='auto', verbose=0, warm_start=False, n_jobs=None, l1_ratio=None)\n",
      " |      Initialize self.  See help(type(self)) for accurate signature.\n",
      " |  \n",
      " |  fit(self, X, y, sample_weight=None)\n",
      " |      Fit the model according to the given training data.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      X : {array-like, sparse matrix} of shape (n_samples, n_features)\n",
      " |          Training vector, where n_samples is the number of samples and\n",
      " |          n_features is the number of features.\n",
      " |      \n",
      " |      y : array-like of shape (n_samples,)\n",
      " |          Target vector relative to X.\n",
      " |      \n",
      " |      sample_weight : array-like of shape (n_samples,) default=None\n",
      " |          Array of weights that are assigned to individual samples.\n",
      " |          If not provided, then each sample is given unit weight.\n",
      " |      \n",
      " |          .. versionadded:: 0.17\n",
      " |             *sample_weight* support to LogisticRegression.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      self\n",
      " |          Fitted estimator.\n",
      " |      \n",
      " |      Notes\n",
      " |      -----\n",
      " |      The SAGA solver supports both float64 and float32 bit arrays.\n",
      " |  \n",
      " |  predict_log_proba(self, X)\n",
      " |      Predict logarithm of probability estimates.\n",
      " |      \n",
      " |      The returned estimates for all classes are ordered by the\n",
      " |      label of classes.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      X : array-like of shape (n_samples, n_features)\n",
      " |          Vector to be scored, where `n_samples` is the number of samples and\n",
      " |          `n_features` is the number of features.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      T : array-like of shape (n_samples, n_classes)\n",
      " |          Returns the log-probability of the sample for each class in the\n",
      " |          model, where classes are ordered as they are in ``self.classes_``.\n",
      " |  \n",
      " |  predict_proba(self, X)\n",
      " |      Probability estimates.\n",
      " |      \n",
      " |      The returned estimates for all classes are ordered by the\n",
      " |      label of classes.\n",
      " |      \n",
      " |      For a multi_class problem, if multi_class is set to be \"multinomial\"\n",
      " |      the softmax function is used to find the predicted probability of\n",
      " |      each class.\n",
      " |      Else use a one-vs-rest approach, i.e calculate the probability\n",
      " |      of each class assuming it to be positive using the logistic function.\n",
      " |      and normalize these values across all the classes.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      X : array-like of shape (n_samples, n_features)\n",
      " |          Vector to be scored, where `n_samples` is the number of samples and\n",
      " |          `n_features` is the number of features.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      T : array-like of shape (n_samples, n_classes)\n",
      " |          Returns the probability of the sample for each class in the model,\n",
      " |          where classes are ordered as they are in ``self.classes_``.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from sklearn.base.BaseEstimator:\n",
      " |  \n",
      " |  __getstate__(self)\n",
      " |  \n",
      " |  __repr__(self, N_CHAR_MAX=700)\n",
      " |      Return repr(self).\n",
      " |  \n",
      " |  __setstate__(self, state)\n",
      " |  \n",
      " |  get_params(self, deep=True)\n",
      " |      Get parameters for this estimator.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      deep : bool, default=True\n",
      " |          If True, will return the parameters for this estimator and\n",
      " |          contained subobjects that are estimators.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      params : mapping of string to any\n",
      " |          Parameter names mapped to their values.\n",
      " |  \n",
      " |  set_params(self, **params)\n",
      " |      Set the parameters of this estimator.\n",
      " |      \n",
      " |      The method works on simple estimators as well as on nested objects\n",
      " |      (such as pipelines). The latter have parameters of the form\n",
      " |      ``<component>__<parameter>`` so that it's possible to update each\n",
      " |      component of a nested object.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      **params : dict\n",
      " |          Estimator parameters.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      self : object\n",
      " |          Estimator instance.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors inherited from sklearn.base.BaseEstimator:\n",
      " |  \n",
      " |  __dict__\n",
      " |      dictionary for instance variables (if defined)\n",
      " |  \n",
      " |  __weakref__\n",
      " |      list of weak references to the object (if defined)\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from sklearn.linear_model._base.LinearClassifierMixin:\n",
      " |  \n",
      " |  decision_function(self, X)\n",
      " |      Predict confidence scores for samples.\n",
      " |      \n",
      " |      The confidence score for a sample is the signed distance of that\n",
      " |      sample to the hyperplane.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      X : array_like or sparse matrix, shape (n_samples, n_features)\n",
      " |          Samples.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      array, shape=(n_samples,) if n_classes == 2 else (n_samples, n_classes)\n",
      " |          Confidence scores per (sample, class) combination. In the binary\n",
      " |          case, confidence score for self.classes_[1] where >0 means this\n",
      " |          class would be predicted.\n",
      " |  \n",
      " |  predict(self, X)\n",
      " |      Predict class labels for samples in X.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      X : array_like or sparse matrix, shape (n_samples, n_features)\n",
      " |          Samples.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      C : array, shape [n_samples]\n",
      " |          Predicted class label per sample.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from sklearn.base.ClassifierMixin:\n",
      " |  \n",
      " |  score(self, X, y, sample_weight=None)\n",
      " |      Return the mean accuracy on the given test data and labels.\n",
      " |      \n",
      " |      In multi-label classification, this is the subset accuracy\n",
      " |      which is a harsh metric since you require for each sample that\n",
      " |      each label set be correctly predicted.\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      X : array-like of shape (n_samples, n_features)\n",
      " |          Test samples.\n",
      " |      \n",
      " |      y : array-like of shape (n_samples,) or (n_samples, n_outputs)\n",
      " |          True labels for X.\n",
      " |      \n",
      " |      sample_weight : array-like of shape (n_samples,), default=None\n",
      " |          Sample weights.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      score : float\n",
      " |          Mean accuracy of self.predict(X) wrt. y.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from sklearn.linear_model._base.SparseCoefMixin:\n",
      " |  \n",
      " |  densify(self)\n",
      " |      Convert coefficient matrix to dense array format.\n",
      " |      \n",
      " |      Converts the ``coef_`` member (back) to a numpy.ndarray. This is the\n",
      " |      default format of ``coef_`` and is required for fitting, so calling\n",
      " |      this method is only required on models that have previously been\n",
      " |      sparsified; otherwise, it is a no-op.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      self\n",
      " |          Fitted estimator.\n",
      " |  \n",
      " |  sparsify(self)\n",
      " |      Convert coefficient matrix to sparse format.\n",
      " |      \n",
      " |      Converts the ``coef_`` member to a scipy.sparse matrix, which for\n",
      " |      L1-regularized models can be much more memory- and storage-efficient\n",
      " |      than the usual numpy.ndarray representation.\n",
      " |      \n",
      " |      The ``intercept_`` member is not converted.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      self\n",
      " |          Fitted estimator.\n",
      " |      \n",
      " |      Notes\n",
      " |      -----\n",
      " |      For non-sparse models, i.e. when there are not many zeros in ``coef_``,\n",
      " |      this may actually *increase* memory usage, so use this method with\n",
      " |      care. A rule of thumb is that the number of zero elements, which can\n",
      " |      be computed with ``(coef_ == 0).sum()``, must be more than 50% for this\n",
      " |      to provide significant benefits.\n",
      " |      \n",
      " |      After calling this method, further fitting with the partial_fit\n",
      " |      method (if any) will not work until you call densify.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(LogisticRegression)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# splitting the dataset into train and test set with a ratio if 80:20 \n",
    "\n",
    "# on train set we train our model and on test we test our predictions and how well our model has perfomed.\n",
    "\n",
    "# defining random_state shuffles the data, each time when we excute the entire notebook the model will get new shuffled train & set data.\n",
    "# It is very useful specially dealing with huge amount of data.\n",
    "\n",
    "xtrain,xtest, ytrain,ytest  = train_test_split(X ,Y,random_state=204,test_size=0.02)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((557, 30), (557,), (12, 30), (12,))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xtrain.shape , ytrain.shape , xtest.shape, ytest.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the Logistic Model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\anktr\\appdata\\local\\programs\\python\\python38-32\\lib\\site-packages\\sklearn\\linear_model\\_sag.py:329: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
      "  warnings.warn(\"The max_iter was reached which means \"\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression(penalty='l1', solver='saga')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# As explained earlier we will use Logistic Regression as our model and will se how well it fits the data.\n",
    "\n",
    "# We also evaluate our model's precision and performance.\n",
    "\n",
    "log_reg = LogisticRegression(solver='saga',penalty='l1')\n",
    "\n",
    "# Coming back to logistic regression: \n",
    "\n",
    "# it has  several hypermaters such as 'penalty', 'solver', 'max_iter','verbose','fit_interept',etc.\n",
    "\n",
    "# Fitting our model:\n",
    "\n",
    "log_reg.fit(xtrain, ytrain)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>radius_mean</th>\n",
       "      <th>texture_mean</th>\n",
       "      <th>perimeter_mean</th>\n",
       "      <th>area_mean</th>\n",
       "      <th>smoothness_mean</th>\n",
       "      <th>compactness_mean</th>\n",
       "      <th>concavity_mean</th>\n",
       "      <th>concave points_mean</th>\n",
       "      <th>symmetry_mean</th>\n",
       "      <th>fractal_dimension_mean</th>\n",
       "      <th>...</th>\n",
       "      <th>radius_worst</th>\n",
       "      <th>texture_worst</th>\n",
       "      <th>perimeter_worst</th>\n",
       "      <th>area_worst</th>\n",
       "      <th>smoothness_worst</th>\n",
       "      <th>compactness_worst</th>\n",
       "      <th>concavity_worst</th>\n",
       "      <th>concave points_worst</th>\n",
       "      <th>symmetry_worst</th>\n",
       "      <th>fractal_dimension_worst</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>506</th>\n",
       "      <td>12.220</td>\n",
       "      <td>20.04</td>\n",
       "      <td>79.47</td>\n",
       "      <td>453.1</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.11520</td>\n",
       "      <td>0.08175</td>\n",
       "      <td>0.02166</td>\n",
       "      <td>0.2124</td>\n",
       "      <td>0.06894</td>\n",
       "      <td>...</td>\n",
       "      <td>13.160</td>\n",
       "      <td>24.17</td>\n",
       "      <td>85.13</td>\n",
       "      <td>515.3</td>\n",
       "      <td>0.1402</td>\n",
       "      <td>0.2315</td>\n",
       "      <td>0.3535</td>\n",
       "      <td>0.08088</td>\n",
       "      <td>0.2709</td>\n",
       "      <td>0.08839</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>268</th>\n",
       "      <td>12.870</td>\n",
       "      <td>16.21</td>\n",
       "      <td>82.38</td>\n",
       "      <td>512.2</td>\n",
       "      <td>0.09425</td>\n",
       "      <td>0.06219</td>\n",
       "      <td>0.03900</td>\n",
       "      <td>0.01615</td>\n",
       "      <td>0.2010</td>\n",
       "      <td>0.05769</td>\n",
       "      <td>...</td>\n",
       "      <td>13.900</td>\n",
       "      <td>23.64</td>\n",
       "      <td>89.27</td>\n",
       "      <td>597.5</td>\n",
       "      <td>0.1256</td>\n",
       "      <td>0.1808</td>\n",
       "      <td>0.1992</td>\n",
       "      <td>0.05780</td>\n",
       "      <td>0.3604</td>\n",
       "      <td>0.07062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>435</th>\n",
       "      <td>13.980</td>\n",
       "      <td>19.62</td>\n",
       "      <td>91.12</td>\n",
       "      <td>599.5</td>\n",
       "      <td>0.10600</td>\n",
       "      <td>0.11330</td>\n",
       "      <td>0.11260</td>\n",
       "      <td>0.06463</td>\n",
       "      <td>0.1669</td>\n",
       "      <td>0.06544</td>\n",
       "      <td>...</td>\n",
       "      <td>17.040</td>\n",
       "      <td>30.80</td>\n",
       "      <td>113.90</td>\n",
       "      <td>869.3</td>\n",
       "      <td>0.1613</td>\n",
       "      <td>0.3568</td>\n",
       "      <td>0.4069</td>\n",
       "      <td>0.18270</td>\n",
       "      <td>0.3179</td>\n",
       "      <td>0.10550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132</th>\n",
       "      <td>16.160</td>\n",
       "      <td>21.54</td>\n",
       "      <td>106.20</td>\n",
       "      <td>809.8</td>\n",
       "      <td>0.10080</td>\n",
       "      <td>0.12840</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>0.05613</td>\n",
       "      <td>0.2160</td>\n",
       "      <td>0.05891</td>\n",
       "      <td>...</td>\n",
       "      <td>19.470</td>\n",
       "      <td>31.68</td>\n",
       "      <td>129.70</td>\n",
       "      <td>1175.0</td>\n",
       "      <td>0.1395</td>\n",
       "      <td>0.3055</td>\n",
       "      <td>0.2992</td>\n",
       "      <td>0.13120</td>\n",
       "      <td>0.3480</td>\n",
       "      <td>0.07619</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>324</th>\n",
       "      <td>12.200</td>\n",
       "      <td>15.21</td>\n",
       "      <td>78.01</td>\n",
       "      <td>457.9</td>\n",
       "      <td>0.08673</td>\n",
       "      <td>0.06545</td>\n",
       "      <td>0.01994</td>\n",
       "      <td>0.01692</td>\n",
       "      <td>0.1638</td>\n",
       "      <td>0.06129</td>\n",
       "      <td>...</td>\n",
       "      <td>13.750</td>\n",
       "      <td>21.38</td>\n",
       "      <td>91.11</td>\n",
       "      <td>583.1</td>\n",
       "      <td>0.1256</td>\n",
       "      <td>0.1928</td>\n",
       "      <td>0.1167</td>\n",
       "      <td>0.05556</td>\n",
       "      <td>0.2661</td>\n",
       "      <td>0.07961</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>514</th>\n",
       "      <td>15.050</td>\n",
       "      <td>19.07</td>\n",
       "      <td>97.26</td>\n",
       "      <td>701.9</td>\n",
       "      <td>0.09215</td>\n",
       "      <td>0.08597</td>\n",
       "      <td>0.07486</td>\n",
       "      <td>0.04335</td>\n",
       "      <td>0.1561</td>\n",
       "      <td>0.05915</td>\n",
       "      <td>...</td>\n",
       "      <td>17.580</td>\n",
       "      <td>28.06</td>\n",
       "      <td>113.80</td>\n",
       "      <td>967.0</td>\n",
       "      <td>0.1246</td>\n",
       "      <td>0.2101</td>\n",
       "      <td>0.2866</td>\n",
       "      <td>0.11200</td>\n",
       "      <td>0.2282</td>\n",
       "      <td>0.06954</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>538</th>\n",
       "      <td>7.729</td>\n",
       "      <td>25.49</td>\n",
       "      <td>47.98</td>\n",
       "      <td>178.8</td>\n",
       "      <td>0.08098</td>\n",
       "      <td>0.04878</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.1870</td>\n",
       "      <td>0.07285</td>\n",
       "      <td>...</td>\n",
       "      <td>9.077</td>\n",
       "      <td>30.92</td>\n",
       "      <td>57.17</td>\n",
       "      <td>248.0</td>\n",
       "      <td>0.1256</td>\n",
       "      <td>0.0834</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.3058</td>\n",
       "      <td>0.09938</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>269</th>\n",
       "      <td>10.710</td>\n",
       "      <td>20.39</td>\n",
       "      <td>69.50</td>\n",
       "      <td>344.9</td>\n",
       "      <td>0.10820</td>\n",
       "      <td>0.12890</td>\n",
       "      <td>0.08448</td>\n",
       "      <td>0.02867</td>\n",
       "      <td>0.1668</td>\n",
       "      <td>0.06862</td>\n",
       "      <td>...</td>\n",
       "      <td>11.690</td>\n",
       "      <td>25.21</td>\n",
       "      <td>76.51</td>\n",
       "      <td>410.4</td>\n",
       "      <td>0.1335</td>\n",
       "      <td>0.2550</td>\n",
       "      <td>0.2534</td>\n",
       "      <td>0.08600</td>\n",
       "      <td>0.2605</td>\n",
       "      <td>0.08701</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>245</th>\n",
       "      <td>10.480</td>\n",
       "      <td>19.86</td>\n",
       "      <td>66.72</td>\n",
       "      <td>337.7</td>\n",
       "      <td>0.10700</td>\n",
       "      <td>0.05971</td>\n",
       "      <td>0.04831</td>\n",
       "      <td>0.03070</td>\n",
       "      <td>0.1737</td>\n",
       "      <td>0.06440</td>\n",
       "      <td>...</td>\n",
       "      <td>11.480</td>\n",
       "      <td>29.46</td>\n",
       "      <td>73.68</td>\n",
       "      <td>402.8</td>\n",
       "      <td>0.1515</td>\n",
       "      <td>0.1026</td>\n",
       "      <td>0.1181</td>\n",
       "      <td>0.06736</td>\n",
       "      <td>0.2883</td>\n",
       "      <td>0.07748</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>14.780</td>\n",
       "      <td>23.94</td>\n",
       "      <td>97.40</td>\n",
       "      <td>668.3</td>\n",
       "      <td>0.11720</td>\n",
       "      <td>0.14790</td>\n",
       "      <td>0.12670</td>\n",
       "      <td>0.09029</td>\n",
       "      <td>0.1953</td>\n",
       "      <td>0.06654</td>\n",
       "      <td>...</td>\n",
       "      <td>17.310</td>\n",
       "      <td>33.39</td>\n",
       "      <td>114.60</td>\n",
       "      <td>925.1</td>\n",
       "      <td>0.1648</td>\n",
       "      <td>0.3416</td>\n",
       "      <td>0.3024</td>\n",
       "      <td>0.16140</td>\n",
       "      <td>0.3321</td>\n",
       "      <td>0.08911</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>557 rows × 30 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     radius_mean  texture_mean  perimeter_mean  area_mean  smoothness_mean  \\\n",
       "506       12.220         20.04           79.47      453.1          0.10960   \n",
       "268       12.870         16.21           82.38      512.2          0.09425   \n",
       "435       13.980         19.62           91.12      599.5          0.10600   \n",
       "132       16.160         21.54          106.20      809.8          0.10080   \n",
       "324       12.200         15.21           78.01      457.9          0.08673   \n",
       "..           ...           ...             ...        ...              ...   \n",
       "514       15.050         19.07           97.26      701.9          0.09215   \n",
       "538        7.729         25.49           47.98      178.8          0.08098   \n",
       "269       10.710         20.39           69.50      344.9          0.10820   \n",
       "245       10.480         19.86           66.72      337.7          0.10700   \n",
       "65        14.780         23.94           97.40      668.3          0.11720   \n",
       "\n",
       "     compactness_mean  concavity_mean  concave points_mean  symmetry_mean  \\\n",
       "506           0.11520         0.08175              0.02166         0.2124   \n",
       "268           0.06219         0.03900              0.01615         0.2010   \n",
       "435           0.11330         0.11260              0.06463         0.1669   \n",
       "132           0.12840         0.10430              0.05613         0.2160   \n",
       "324           0.06545         0.01994              0.01692         0.1638   \n",
       "..                ...             ...                  ...            ...   \n",
       "514           0.08597         0.07486              0.04335         0.1561   \n",
       "538           0.04878         0.00000              0.00000         0.1870   \n",
       "269           0.12890         0.08448              0.02867         0.1668   \n",
       "245           0.05971         0.04831              0.03070         0.1737   \n",
       "65            0.14790         0.12670              0.09029         0.1953   \n",
       "\n",
       "     fractal_dimension_mean  ...  radius_worst  texture_worst  \\\n",
       "506                 0.06894  ...        13.160          24.17   \n",
       "268                 0.05769  ...        13.900          23.64   \n",
       "435                 0.06544  ...        17.040          30.80   \n",
       "132                 0.05891  ...        19.470          31.68   \n",
       "324                 0.06129  ...        13.750          21.38   \n",
       "..                      ...  ...           ...            ...   \n",
       "514                 0.05915  ...        17.580          28.06   \n",
       "538                 0.07285  ...         9.077          30.92   \n",
       "269                 0.06862  ...        11.690          25.21   \n",
       "245                 0.06440  ...        11.480          29.46   \n",
       "65                  0.06654  ...        17.310          33.39   \n",
       "\n",
       "     perimeter_worst  area_worst  smoothness_worst  compactness_worst  \\\n",
       "506            85.13       515.3            0.1402             0.2315   \n",
       "268            89.27       597.5            0.1256             0.1808   \n",
       "435           113.90       869.3            0.1613             0.3568   \n",
       "132           129.70      1175.0            0.1395             0.3055   \n",
       "324            91.11       583.1            0.1256             0.1928   \n",
       "..               ...         ...               ...                ...   \n",
       "514           113.80       967.0            0.1246             0.2101   \n",
       "538            57.17       248.0            0.1256             0.0834   \n",
       "269            76.51       410.4            0.1335             0.2550   \n",
       "245            73.68       402.8            0.1515             0.1026   \n",
       "65            114.60       925.1            0.1648             0.3416   \n",
       "\n",
       "     concavity_worst  concave points_worst  symmetry_worst  \\\n",
       "506           0.3535               0.08088          0.2709   \n",
       "268           0.1992               0.05780          0.3604   \n",
       "435           0.4069               0.18270          0.3179   \n",
       "132           0.2992               0.13120          0.3480   \n",
       "324           0.1167               0.05556          0.2661   \n",
       "..               ...                   ...             ...   \n",
       "514           0.2866               0.11200          0.2282   \n",
       "538           0.0000               0.00000          0.3058   \n",
       "269           0.2534               0.08600          0.2605   \n",
       "245           0.1181               0.06736          0.2883   \n",
       "65            0.3024               0.16140          0.3321   \n",
       "\n",
       "     fractal_dimension_worst  \n",
       "506                  0.08839  \n",
       "268                  0.07062  \n",
       "435                  0.10550  \n",
       "132                  0.07619  \n",
       "324                  0.07961  \n",
       "..                       ...  \n",
       "514                  0.06954  \n",
       "538                  0.09938  \n",
       "269                  0.08701  \n",
       "245                  0.07748  \n",
       "65                   0.08911  \n",
       "\n",
       "[557 rows x 30 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xtrain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predicting the class using predict method.\n",
    "\n",
    "pred = log_reg.predict(xtest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['M', 'M', 'B', 'B', 'B', 'B', 'M', 'B', 'B', 'B', 'B', 'B'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This 'sklearn's metric' is used for model evaluation, it shows how well our model predicted the output and how many false values/outputs it produced.\n",
    "\n",
    "from sklearn.metrics import classification_report,accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           B       0.89      1.00      0.94         8\n",
      "           M       1.00      0.75      0.86         4\n",
      "\n",
      "    accuracy                           0.92        12\n",
      "   macro avg       0.94      0.88      0.90        12\n",
      "weighted avg       0.93      0.92      0.91        12\n",
      "\n"
     ]
    }
   ],
   "source": [
    "predicted = log_reg.predict(xtest)\n",
    "\n",
    "report = classification_report(ytest, pred)\n",
    "\n",
    "print(report)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring the Classification report: \n",
    "A Classification report is used to measure the quality of predictions from a classification algorithm. How many predictions are True and how many are False. More specifically, True Positives, False Positives, True negatives and False Negatives are used to predict the metrics of a classification report as shown above.\n",
    "\n",
    "\n",
    "**In predictive analytics, when deciding between two models it is important to pick a single performance metric. As you can see here, there are many that you can choose from (accuracy, recall, precision, f1-score, AUC, etc). Ultimately, you should use the performance metric that is most suitable for the business problem at hand.** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[8 0]\n",
      " [1 3]]\n"
     ]
    }
   ],
   "source": [
    "# Creating the confusion matrix: \n",
    "\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score\n",
    "\n",
    "# accuracy_score tells how well our model predict the right class,as in our case: 'cancer type'.\n",
    "cm = confusion_matrix(ytest, pred)\n",
    "\n",
    "print(cm)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The above confusion matrix explain these four elements: \n",
    "\n",
    "* True Positives : The cases in which we predicted YES and the actual output was also YES.\n",
    "* True Negatives : The cases in which we predicted NO and the actual output was NO.\n",
    "* False Positives : The cases in which we predicted YES and the actual output was NO.\n",
    "* False Negatives : The cases in which we predicted NO and the actual output was YES."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy score: 0.9166666666666666\n"
     ]
    }
   ],
   "source": [
    "\n",
    "print(\"Accuracy score:\",accuracy_score(ytest,pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}