src/sagemaker/01.FD_SL_Process_IEEE-CIS_Dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Preprocessing and Feature Engineering\n",
    "\n",
    "Upload raw data to S3\n",
    "The dataset we use is the IEEE-CIS Fraud Detection dataset which is a typical example of financial transactions dataset that many companies have. The dataset consists of two tables:\n",
    "\n",
    "Transactions: Records transactions and metadata about transactions between two users. Examples of columns include the product code for the transaction and features on the card used for the transaction.\n",
    "Identity: Contains information about the identity users performing transactions. Examples of columns here include the device type and device ids used.\n",
    "We will go over the specific data schema in subsequent cells but now let's move the raw data to a convenient location in the S3 bucket for this proejct, where it will be picked up by the preprocessing job and training job.\n",
    "\n",
    "If you would like to use your own dataset for this demonstration. Replace the raw_data_location with the s3 path or local path of your dataset, and modify the data preprocessing step as needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prerequisites\n",
    "\n",
    "- AWS account\n",
    "- make sure your env with memory 32G+\n",
    "- install python 3.6+, boto3, sagemaker>=2.41.0, pyyaml, pandas, requests(such as `pip install boto3 'sagemaker>=2.41.0,<3.0' pyyaml pandas requests`)\n",
    "- install supported version of [awscli](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html)\n",
    "- configure credential of aws cli with s3, sagemaker permissions\n",
    "- install tar and gzip in your OS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "import boto3\n",
    "import sagemaker\n",
    "import tempfile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_data_location = 's3://aws-gcr-solutions-assets/open-dataset/ieee-fraud-detection/'\n",
    "\n",
    "session_prefix = 'realtime-fraud-detection-on-dgl'\n",
    "\n",
    "dest_dir = tempfile.mkdtemp()\n",
    "\n",
    "transaction_source = f'{raw_data_location}train_transaction.csv'\n",
    "transaction_dest = f'{dest_dir}/transaction.csv'\n",
    "\n",
    "!aws s3 cp $transaction_source $transaction_dest\n",
    "\n",
    "identity_source = f'{raw_data_location}train_identity.csv'\n",
    "identity_dest = f'{dest_dir}/identity.csv'\n",
    "\n",
    "!aws s3 cp $identity_source $identity_dest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_dir = tempfile.mkdtemp()\n",
    "\n",
    "! python ./data-preprocessing/graph_data_preprocessor.py --data-dir $dest_dir --output-dir $output_dir --id-cols 'card1,card2,card3,card4,card5,card6,ProductCD,addr1,addr2,P_emaildomain,R_emaildomain' '--cat-cols' 'M1,M2,M3,M4,M5,M6,M7,M8,M9'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "def detect_running_region():\n",
    "    \"\"\"Dynamically determine the region).\"\"\"\n",
    "    easy_checks = [\n",
    "        # check if set through ENV vars\n",
    "        os.environ.get('AWS_REGION'),\n",
    "        os.environ.get('AWS_DEFAULT_REGION'),\n",
    "        # else check if set in config or in boto already\n",
    "        boto3.DEFAULT_SESSION.region_name if boto3.DEFAULT_SESSION else None,\n",
    "        boto3.Session().region_name,\n",
    "    ]\n",
    "    for region in easy_checks:\n",
    "        if region:\n",
    "            return region\n",
    "\n",
    "    # else query an external service\n",
    "    # https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-identity-documents.html\n",
    "    r = requests.get(\"http://169.254.169.254/latest/dynamic/instance-identity/document\")\n",
    "    response_json = r.json()\n",
    "    return response_json.get('region')\n",
    "\n",
    "current_region = detect_running_region()\n",
    "print(f'current AWS region is {current_region}')\n",
    "\n",
    "default_bucket = sagemaker.session.Session(boto3.session.Session(region_name=current_region)).default_bucket()\n",
    "\n",
    "processed_data = f's3://{default_bucket}/{session_prefix}/processed-data'\n",
    "print(processed_data)\n",
    "\n",
    "! export AWS_DEFAULT_REGION=$current_region && aws s3 sync $output_dir $processed_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store processed_data\n",
    "%store default_bucket\n",
    "%store current_region"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_latest_p36",
   "language": "python",
   "name": "conda_pytorch_latest_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}