In this work, we address the task of predicting if an individual is earning 50K or more a year. It is a binary classification problem. This work is a part of an in-class Kaggle Competition for Knowledge Discovery and Data Mining Course offered at NUS, Singapore in Semester 2 of AY 2019-2020.
Our final binary F1-score is 86,946% on the private leaderboard and we were ranked 10th / 55.
To clone this github repository, use the following command:
$ git clone https://github.com/hanaecarrie/CS5228_kaggle_income50K_classification.git
Make sure you have Python 3.6 or above installed, as well as the following packages:
- numpy
- csv
- os
- pandas
- matplotlib
- seaborn
- sklearn
- pickle
- scipy
- lightgbm
- catboost
- xgboost
- IPython
- subprocess
- keras
- collections
- math
- time
- glob
- re
The Kaggle dataset consists of a separate training and test dataset, both consisting of 24,421 records each. The training dataset suffers from class imbalance, with 75.15% of the samples being from the negative class (≤ $50K, label=0) and the remaining 24.85% being positive samples (> $50K, label=1). The dataset consists of 13 attributes described below:
Here are some plots from the report to summarise the dataset features and look at their relationships.
The full report is available here
The description of the different preprocessed data:
