Questions and Answers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Enron Submission Free-Response Questions\n",
    "\n",
    "A critical part of machine learning is making sense of your analysis process and communicating it to others. The questions below will help us understand your decision-making process and allow us to give feedback on your project. Please answer each question; your answers should be about 1-2 paragraphs per question. If you find yourself writing much more than that, take a step back and see if you can simplify your response!\n",
    "\n",
    "When your evaluator looks at your responses, he or she will use a specific list of rubric items to assess your answers. Here is the link to that rubric: [Link] Each question has one or more specific rubric items associated with it, so before you submit an answer, take a look at that part of the rubric. If your response does not meet expectations for all rubric points, you will be asked to revise and resubmit your project. Make sure that your responses are detailed enough that the evaluator will be able to understand the steps you took and your thought processes as you went through the data analysis.\n",
    "Once you’ve submitted your responses, your coach will take a look and may ask a few more focused follow-up questions on one or more of your answers.  \n",
    "\n",
    "We can’t wait to see what you’ve put together for this project!\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** QUESTION 1 ** \n",
    "\n",
    "Summarize for us the goal of this project and how machine learning is useful in trying to accomplish it. As part of your answer, give some background on the dataset and how it can be used to answer the project question. Were there any outliers in the data when you got it, and how did you handle those?  [relevant rubric items: “data exploration”, “outlier investigation”]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** ANSWER **\n",
    "\n",
    "The goal of this project is to develop a predictive model to identify Persons of Interest (POI) in the Enron Fraud case. Machine learning is useful here, in a supervised environment, to use computerized algorithms to identify/extract patterns in the complex available  data/features and use this knowledge in order to make predictions in new cases. Machine Learning is also useful here by allowing for training and testing in separate sub-sets of the data, giving confidence that the model would hold with cases outside this dataset.\n",
    "\n",
    "The dataset provided contains financial information (14 features) and e-mail communication information (6 features) from 146 Enron employees, along with the label classifying the known POIs (18 POIs in the data set). \n",
    "\n",
    "*OUTLIERS*\n",
    "\n",
    "I spotted 3 keys that needed to tbe removed: *TOTAL* and *THE TRAVEL AGENCY IN THE PARK*. These keys are clearly not persons and will only disturb any analysis.\n",
    "\n",
    "I noticed also that *LOCKHART EUGENE E* has only 0s or *NaN* values, therefore he will also be removed. I opted to remove it manually, but this would not be stricly necessary if using *features_format* function since by default (option *remove_all_zeroes=True*) this person would be removed. However for consisteny and initial analysis, it is cleaner to do so.\n",
    "\n",
    "I also noitced that Kenneth Lay has a gigantic outliers value for *total payments* so I considered removing him or his value for total payments. However he is a real outlier and should be identified by the model. Therefore I did not do remove it.\n",
    "\n",
    "In further investigation using the Tukey's Inter Quartile Range method, I found that, as expected, *LAY KENNETH L* and *SKILLING JEFFREY K* show up often as outliers when looking feature by feature. However, a few new names are also there: *FREVERT MARK A*, *BELDEN TIMOTHY N*, *LAVORATO JOHN J*. As they all seem like real outliers, so I will keep them. \n",
    "\n",
    "*DATA ERRORS*\n",
    "\n",
    "In exploring the data I also noticed that 2 entries had errors, with one column shift: *BHATNAGAR SANJAY* and *BELFER ROBERT*. I noticed this since *BELFER ROBERT* had a negative value for a feature where no negative values should be expected. To find all problem cases I computed the *Total Payments* and *Total Stock Value* based on the other columns and compared to the original values. After finding the problem cases I manually typed the correct values based on the PDF."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** QUESTION 2** \n",
    "\n",
    "What features did you end up using in your POI identifier, and what selection process did you use to pick them? Did you have to do any scaling? Why or why not? As part of the assignment, you should attempt to engineer your own feature that does not come ready-made in the dataset -- explain what feature you tried to make, and the rationale behind it. (You do not necessarily have to use it in the final analysis, only engineer and test it.) In your feature selection step, if you used an algorithm like a decision tree, please also give the feature importances of the features that you use, and if you used an automated feature selection function like SelectKBest, please report the feature scores and reasons for your choice of parameter values.  [relevant rubric items: “create new features”, “intelligently select features”, “properly scale features”]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** ANSWER ** \n",
    "\n",
    "I tried and compared two processes. \n",
    "\n",
    "*First Process*\n",
    "\n",
    "I initially listed the number of *NaN* in each feature (broken down by within POI and non-POIs) and decided to drop any feature that had less than 50% valid points. Therefore I kept only the 14 features:\n",
    "\n",
    "['poi','salary', 'to_messages', 'total_payments', 'exercised_stock_options', 'bonus', 'restricted_stock', 'shared_receipt_with_poi', 'total_stock_value', 'expenses', 'from_messages', 'other', 'from_this_person_to_poi', 'from_poi_to_this_person'].\n",
    "\n",
    "After that I decided to convert convert all the to/from messages features related to pois into share of the total to/from messages. The rationale was that I did not expect the features *to/from_messages* to be informative, and the to/from poi in absolute terms ignores the relative effect. Therefore I re-scaled all to their shares and removed the absolute ones. \n",
    "\n",
    "The updated list with 12 features was:\n",
    "['poi', 'salary', 'total_payments', 'exercised_stock_options', 'bonus', 'restricted_stock', 'total_stock_value', 'expenses', 'other', 'share_shared_receipt_with_poi', 'share_from_this_person_to_poi', 'share_from_poi_to_this_person']\n",
    "\n",
    "I decided to not do any re-scaling since I was not planning on using any classifier that was sensitive to that. In particular, my initial plan was to test and select among NaiveBayes, SVM, Decision Tree and Adaboost, which should not be sensitive to the scaling, except SVM. Therefore I decided to re-visit scaling only if I opted to further investigate SVM, which was not the case.\n",
    "\n",
    "After tuning and training it, the performance in the *tester.py* was:\n",
    "    \n",
    "    Accuracy: 0.87300\tPrecision: 0.53251\tRecall: 0.38900\tF1: 0.44958\tF2: 0.41116\n",
    "\n",
    "I compared with the performance after manually removing the 3 features with near 0 importance, and re-tunning the model:\n",
    "\n",
    "features = ['poi', 'salary', 'total_payments', 'exercised_stock_options', 'bonus', 'restricted_stock', 'expenses', 'other', 'share_shared_receipt_with_poi']\n",
    "\n",
    "    Accuracy: 0.88767\tPrecision: 0.63496\tRecall: 0.37050\tF1: 0.46795\tF2: 0.40417\n",
    "    \n",
    "I attempted to remove an additional feature (*other*), but performance dropped in *Recall* and *F1*:\n",
    "\n",
    "    Accuracy: 0.89007\tPrecision: 0.66924\tRecall: 0.34700\tF1: 0.45703\tF2: 0.38398\n",
    "\n",
    "\n",
    "\n",
    "*Second Process*\n",
    "\n",
    "In the second process I opted to use recursive feature elimination (RFE) to select features. I started from all features, including the 3 additional computed features I created and used RFE to select the 8 best (since there where 8 features + poi in the process above).\n",
    "\n",
    "I tested these selection on the  *tester.py* and obtained a superior performance. \n",
    "\n",
    "With the list:\n",
    "\n",
    "['poi', 'total_payments', 'exercised_stock_options', 'bonus', 'restricted_stock', 'expenses', 'other', 'share_shared_receipt_with_poi', 'share_from_this_person_to_poi']\n",
    "\n",
    "Performance was:\n",
    "\n",
    "    Accuracy: 0.88933\tPrecision: 0.64912\tRecall: 0.37000\tF1: 0.47134\tF2: 0.40481\n",
    "\n",
    "I noticed again that features had near 0 performance, and tried to remove via RFE another 2 features.\n",
    "\n",
    "With the list:\n",
    "['poi', 'exercised_stock_options', 'bonus', 'expenses', 'other', 'share_shared_receipt_with_poi', 'share_from_this_person_to_poi']\n",
    "\n",
    "Performance was:\n",
    "\n",
    "    Accuracy: 0.88943\tPrecision: 0.72027\tRecall: 0.36950\tF1: 0.48843\tF2: 0.40937\n",
    "    \n",
    "I re-tuned this model, obtaining:\n",
    "\n",
    "    Accuracy: 0.87736\tPrecision: 0.58408\tRecall: 0.49150\tF1: 0.53380\tF2: 0.50759\n",
    "\n",
    "I tried removing additional features via RFE or manually, but performance did not improve in the direction I wanted.\n",
    "\n",
    "*Final selection*\n",
    "\n",
    "I compared the best performing classifier from the two processes above and selected the best one, which was the 6 best features selected via RFE.\n",
    "\n",
    "**List:**\n",
    "\n",
    "['poi', 'exercised_stock_options', 'bonus', 'expenses', 'other', 'share_shared_receipt_with_poi', 'share_from_this_person_to_poi']\n",
    "\n",
    "**Performance:**\n",
    "\n",
    "    Accuracy: 0.87736\tPrecision: 0.58408\tRecall: 0.49150\tF1: 0.53380\tF2: 0.5075\n",
    "    \n",
    "**Features importance:**\n",
    "* exercised_stock_options: 0.69\n",
    "* bonus: 0.23\n",
    "* expenses: 0.08\n",
    "* other: 0.0\n",
    "* share_shared_receipt_with_poi: 0.0\n",
    "* share_from_this_person_to_poi: 0.0\n",
    "\n",
    "(even though there are some near 0 importances, removing then cleary reduced performance)\n",
    "\n",
    "** IMPACT OF CREATED FEATURES **\n",
    "To check the impact of my created features, I computed the performance of the classifier without them.\n",
    "\n",
    "With the list:\n",
    "['poi', 'exercised_stock_options', 'bonus', 'expenses', 'other']\n",
    "\n",
    "Performance was:\n",
    "    Accuracy: 0.85186\tPrecision: 0.47068\tRecall: 0.29700\tF1: 0.36419\tF2: 0.32067\n",
    "    \n",
    "Which clearly shows the impact of my created features!\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** QUESTION 3** \n",
    "\n",
    "What algorithm did you end up using? What other one(s) did you try? How did model performance differ between algorithms?  [relevant rubric item: “pick an algorithm”]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** ANSWER **\n",
    "\n",
    "I initially compared 4 algorithms with their default parameters:\n",
    "* Naive Bays\n",
    "* SVM\n",
    "* Decision Tree\n",
    "* Adaboost\n",
    "\n",
    "These where the initial performances:\n",
    "\n",
    "GaussianNB(priors=None)\n",
    "\n",
    "\tAccuracy: 0.84707\tPrecision: 0.37105\tRecall: 0.21150\tF1: 0.26943\tF2: 0.23140    \n",
    "\tTotal predictions: 15000\tTrue positives:  423\tFalse positives:  717\tFalse negatives: 1577\tTrue negatives: 12283\n",
    "\n",
    "Got a divide by zero when trying out: SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
    "  decision_function_shape=None, degree=3, gamma='auto', kernel='rbf',\n",
    "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
    "  tol=0.001, verbose=False)\n",
    "Precision or recall may be undefined due to a lack of true positive predicitons.\n",
    "\n",
    "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
    "            max_features=None, max_leaf_nodes=None,\n",
    "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
    "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
    "            presort=False, random_state=None, splitter='best')\n",
    "            \n",
    "\tAccuracy: 0.81073\tPrecision: 0.30370\tRecall: 0.32450\tF1: 0.31375\tF2: 0.32011\n",
    "\tTotal predictions: 15000\tTrue positives:  649\tFalse positives: 1488\tFalse negatives: 1351\tTrue negatives: 11512\n",
    "\n",
    "AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None,\n",
    "          learning_rate=1.0, n_estimators=50, random_state=None)\n",
    "          \n",
    "\tAccuracy: 0.85473\tPrecision: 0.44526\tRecall: 0.36400\tF1: 0.40055\tF2: 0.37779\n",
    "\tTotal predictions: 15000\tTrue positives:  728\tFalse positives:  907\tFalse negatives: 1272\tTrue negatives: 12093\n",
    "\n",
    "\n",
    "Adaboost clearly had the superior performance, therefore I opted to drop the other ones and pursue tuning of Adaboost along side the selection of features.\n",
    "\n",
    "In the final model, as indicate in *QUESTION 2* performance was:\n",
    "\n",
    "    Accuracy: 0.87736\tPrecision: 0.58408\tRecall: 0.49150\tF1: 0.53380\tF2: 0.5075\n",
    "  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** QUESTION 4** \n",
    "\n",
    "What does it mean to tune the parameters of an algorithm, and what can happen if you don’t do this well?  How did you tune the parameters of your particular algorithm? What parameters did you tune? (Some algorithms do not have parameters that you need to tune -- if this is the case for the one you picked, identify and briefly explain how you would have done it for the model that was not your final choice or a different model that does utilize parameter tuning, e.g. a decision tree classifier).  [relevant rubric items: “discuss parameter tuning”, “tune the algorithm”]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** ANSWER **\n",
    "\n",
    "Tuning the parameters of an algorithm is the *art* of choosing the right values in order to allow the model to get its best performance. If you do not do this well your model will underperform and you will have worse results than you could.\n",
    "\n",
    "In this case I optimized AdaBoost using the built-in sklearn function GridSearchCV. In particular I did the grid search on\n",
    "\n",
    "*learning_rate*\n",
    "\n",
    "[0.1, 0.2, 0.3, 0.5, 0.7,0.8, 0.9, 1, **2**,3, 5, 10]\n",
    "\n",
    "and *n_estimators*\n",
    "\n",
    "[1,5,8,10,11, 12, **13**, 14, 15, 16, 17, 18,50,100,1000, 2000].\n",
    "\n",
    "Moreover I used  *F1* as the scoring metric and 10 folds for the stratified cross-validating in the GridSearch.\n",
    "\n",
    "The results indicated **2** and **13** and the best parameters for *learning_rate* and *n_estimators* respectively.\n",
    "\n",
    "I obtained this tuning using the final selection of features. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** QUESTION 5** \n",
    "\n",
    "What is validation, and what’s a classic mistake you can make if you do it wrong? How did you validate your analysis?  [relevant rubric items: “discuss validation”, “validation strategy”]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** ANSWER **\n",
    "\n",
    "Validation is the process of veryfing that your model works well (i.e., has good performance) outside the training set. The classic mistake is to over fit. This means your model will have very good performance in a training set, but once tested in out-of-sample or new data it will have poor performance. This would mean that the model nows very well the training set, but is not capable of handling new situations.\n",
    "\n",
    "In my case I used 2 approaches for cross validation. First and foremost, I used the *tester.py* code to infer performance. This code applied a Shuffled Stratified K Fold with multiple re-sampling (see [here](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedShuffleSplit.html)). This implies repeating the process of randomly picking a sample sub-set from the data within strata (i.e., guaranteeing that POIs will be included) for training and testing performance on the left over data. Because this dataset contains only 18 POIs simple K-folding would not work, since there is a high chance no POI would be present in a given test set. Therefore *tester.py* repeats the process of 1000 times random shuffling and spliting with 10% testing set, where in both the training and testing sets the percentage of POIs mimics the share in the total data set.\n",
    "\n",
    "The other related approach was to built-in the GridSearchCV. This function, in doing the gridseach applies the cross-validation with Stratified K-fold, for which I select 10. This also chooses the best parameters form dividing the sample in stratified folds and traiing in one set and computing performance in another set. The stratification also guarantees the presence of POIs in both the training and testing sets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** QUESTION 6** \n",
    "\n",
    "Give at least 2 evaluation metrics and your average performance for each of them.  Explain an interpretation of your metrics that says something human-understandable about your algorithm’s performance. [relevant rubric item: “usage of evaluation metrics”]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** ANSWER **\n",
    "\n",
    "The final model had the performance:\n",
    "\n",
    "    Accuracy: 0.87700\tPrecision: 0.58264\tRecall: 0.49000\tF1: 0.53232\tF2: 0.50609\n",
    "\tTotal predictions: 15000\tTrue positives:  851\tFalse positives:  534\tFalse negatives: 1149\tTrue negatives: 12466\n",
    "    \n",
    "*Accuracy*:\n",
    "\n",
    "Accuracy is the rate in which the model gives right predictions, i.e., it compares the predicted label with the real label. It means that the model is right 88% of the time. It is a general metric for performance which is quite informative if the labels are close to even split. In this case however, there are only 18 POIs in the 143 valid observations. That means, for example, that if you would if you classify all persons as non-POIs, your accuracy would be 87.4. Does this model this better than that!\n",
    "\n",
    "*Precision*:\n",
    "\n",
    "Precision is the rate to which the POIs identification are correct, i.e, once a person is classified as POI, this is correct 58% of the time. This gives an indication of how much \"trust\" you can give to a result from a model. It is the rate of true positive over positive classifications.\n",
    "\n",
    "*Recall*:\n",
    "\n",
    "Recall is the rate to which POIs are \"caught\" by the model, i.e., once a person is a POI, the model correctly label her as a POI in 49% of the time. This gives an indication of the power of the model. It is the rate of classification as POI over all classifications received by POIs.\n",
    "\n",
    "*F1*:\n",
    "\n",
    "F1 combines the Precision with Recall. It is  way to average Precision and Recall, trying to improve the best of both!\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}