t81_558_class6_preprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Class 6: Preprocessing.**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), School of Engineering and Applied Science, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Why is Preprocessing Necessary\n",
    "\n",
    "The feature vector, the input to a model (such as a neural network), must be completely numeric. Converting non-numeric data into numeric is one major component of preprocessing.  It is also often important to preprocess numeric values.  Scikit-learn provides a large number of preprocessing functions: \n",
    "\n",
    "* [Scikit-Learn Preprocessing](http://scikit-learn.org/stable/modules/preprocessing.html)\n",
    "\n",
    "However, this is just the beginning.  The success of your neural network's predictions is often directly tied to the data representation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing Functions\n",
    "\n",
    "The following functions will be used in conjunction with TensorFlow to help preprocess the data.  Some of these were [covered previously](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class2_tensor_flow.ipynb), some are new.\n",
    "\n",
    "It is okay to just use them. For better understanding, try to see how they work.\n",
    "\n",
    "These functions allow you to build the feature vector for a neural network. Consider the following:\n",
    "\n",
    "* Predictors/Inputs \n",
    "    * Fill any missing inputs with the median for that column.  Use **missing_median**.\n",
    "    * Encode textual/categorical values with **encode_text_dummy** or more creative means (see last part of this class session). \n",
    "    * Encode numeric values with **encode_numeric_zscore**, **encode_numeric_binary** or **encode_numeric_range**. \n",
    "    * Consider removing outliers: **remove_outliers**\n",
    "* Output\n",
    "    * Discard rows with missing outputs.\n",
    "    * Encode textual/categorical values with **encode_text_index**. \n",
    "    * Do not encode output numeric values.\n",
    "    * Consider removing outliers: **remove_outliers**\n",
    "* Produce final feature vectors (x) and expected output (y) with **to_xy**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Complete Set of Preprocessing Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import sklearn.preprocessing\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "\n",
    "# Encode text values to dummie variables(i.e. [1,0,0],[0,1,0],[0,0,1] for red,green,blue)    \n",
    "def encode_text_dummy(df,name):\n",
    "    dummies = pd.get_dummies(df[name])\n",
    "    for x in dummies.columns:\n",
    "        dummy_name = \"{}-{}\".format(name,x)\n",
    "        df[dummy_name] = dummies[x]\n",
    "    df.drop(name, axis=1, inplace=True)\n",
    "    \n",
    "# Encode text values to indexes(i.e. [1],[2],[3] for red,green,blue)    \n",
    "def encode_text_index(df,name): \n",
    "    le = preprocessing.LabelEncoder()\n",
    "    df[name] = le.fit_transform(df[name])\n",
    "    return le.classes_\n",
    "                \n",
    "# Encode a numeric column as zscores    \n",
    "def encode_numeric_zscore(df,name,mean=None,sd=None):\n",
    "    if mean is None:\n",
    "        mean = df[name].mean()\n",
    "        \n",
    "    if sd is None:\n",
    "        sd = df[name].std()\n",
    "        \n",
    "    df[name] = (df[name]-mean)/sd\n",
    "    \n",
    "# Convert all missing values in the specified column to the median\n",
    "def missing_median(df, name):\n",
    "    med = df[name].median()\n",
    "    df[name] = df[name].fillna(med)\n",
    "    \n",
    "# Remove all rows where the specified column is +/- sd standard deviations\n",
    "def remove_outliers(df, name, sd):\n",
    "    drop_rows = df.index[(np.abs(df[name]-df[name].mean())>=(sd*df[name].std()))]\n",
    "    df.drop(drop_rows,axis=0,inplace=True)\n",
    "    \n",
    "# Encode a column to a range between normalized_low and normalized_high.\n",
    "def encode_numeric_range(df, name, normalized_low =-1, normalized_high =1, \n",
    "                         data_low=None, data_high=None):\n",
    "    \n",
    "    if data_low is None:\n",
    "        data_low = min(df[name])\n",
    "        data_high = max(df[name])\n",
    "    \n",
    "    df[name] = ((df[name] - data_low) / (data_high - data_low)) \\\n",
    "                * (normalized_high - normalized_low) + normalized_low\n",
    "\n",
    "# Convert a Pandas dataframe to the x,y inputs that TensorFlow needs\n",
    "def to_xy(df,target):\n",
    "    result = []\n",
    "    for x in df.columns:\n",
    "        if x != target:\n",
    "            result.append(x)\n",
    "\n",
    "    # find out the type of the target column.  Is it really this hard? :(\n",
    "    target_type = df[target].dtypes\n",
    "    target_type = target_type[0] if hasattr(target_type, '__iter__') else target_type\n",
    "    \n",
    "    # Encode to int for classification, float otherwise. TensorFlow likes 32 bits.\n",
    "    if target_type in (np.int64, np.int32):\n",
    "        # Classification\n",
    "        return df.as_matrix(result).astype(np.float32),df.as_matrix([target]).astype(np.int32)\n",
    "    else:\n",
    "        # Regression\n",
    "        return df.as_matrix(result).astype(np.float32),df.as_matrix([target]).astype(np.float32)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyzing a Dataset\n",
    "\n",
    "The following script can be used to give a high level overview of how a dataset appears."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ENCODING = 'utf-8'\n",
    "\n",
    "def expand_categories(values):\n",
    "    result = []\n",
    "    s = values.value_counts()\n",
    "    t = float(len(values))\n",
    "    for v in s.index:\n",
    "        result.append(\"{}:{}%\".format(v,round(100*(s[v]/t),2)))\n",
    "    return \"[{}]\".format(\",\".join(result))\n",
    "        \n",
    "def analyze(filename):\n",
    "    print()\n",
    "    print(\"Analyzing: {}\".format(filename))\n",
    "    df = pd.read_csv(filename,encoding=ENCODING)\n",
    "    cols = df.columns.values\n",
    "    total = float(len(df))\n",
    "\n",
    "    print(\"{} rows\".format(int(total)))\n",
    "    for col in cols:\n",
    "        uniques = df[col].unique()\n",
    "        unique_count = len(uniques)\n",
    "        if unique_count>100:\n",
    "            print(\"** {}:{} ({}%)\".format(col,unique_count,int(((unique_count)/total)*100)))\n",
    "        else:\n",
    "            print(\"** {}:{}\".format(col,expand_categories(df[col])))\n",
    "            expand_categories(df[col])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The analyze script can be run on the MPG dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Analyzing: ./data/auto-mpg.csv\n",
      "398 rows\n",
      "** mpg:129 (32%)\n",
      "** cylinders:[4:51.26%,8:25.88%,6:21.11%,3:1.01%,5:0.75%]\n",
      "** displacement:[97.0:5.28%,350.0:4.52%,98.0:4.52%,250.0:4.27%,318.0:4.27%,140.0:4.02%,400.0:3.27%,225.0:3.27%,91.0:3.02%,121.0:2.76%,302.0:2.76%,232.0:2.76%,151.0:2.51%,120.0:2.26%,200.0:2.01%,351.0:2.01%,85.0:2.01%,90.0:2.01%,231.0:2.01%,122.0:1.76%,105.0:1.76%,304.0:1.76%,79.0:1.51%,156.0:1.51%,119.0:1.51%,258.0:1.26%,89.0:1.26%,107.0:1.26%,108.0:1.26%,135.0:1.26%,360.0:1.01%,86.0:1.01%,134.0:1.01%,116.0:1.01%,112.0:1.01%,305.0:1.01%,70.0:0.75%,113.0:0.75%,455.0:0.75%,307.0:0.75%,168.0:0.75%,198.0:0.75%,146.0:0.75%,260.0:0.75%,173.0:0.75%,429.0:0.75%,199.0:0.5%,141.0:0.5%,163.0:0.5%,262.0:0.5%,71.0:0.5%,440.0:0.5%,383.0:0.5%,88.0:0.25%,97.5:0.25%,340.0:0.25%,144.0:0.25%,390.0:0.25%,83.0:0.25%,96.0:0.25%,80.0:0.25%,78.0:0.25%,76.0:0.25%,72.0:0.25%,81.0:0.25%,104.0:0.25%,267.0:0.25%,100.0:0.25%,101.0:0.25%,110.0:0.25%,111.0:0.25%,183.0:0.25%,181.0:0.25%,114.0:0.25%,115.0:0.25%,171.0:0.25%,155.0:0.25%,130.0:0.25%,131.0:0.25%,145.0:0.25%,454.0:0.25%,68.0:0.25%]\n",
      "** horsepower:[150:5.53%,90:5.03%,88:4.77%,110:4.52%,100:4.27%,75:3.52%,95:3.52%,105:3.02%,67:3.02%,70:3.02%,65:2.51%,85:2.26%,97:2.26%,145:1.76%,80:1.76%,140:1.76%,72:1.51%,84:1.51%,68:1.51%,78:1.51%,92:1.51%,?:1.51%,115:1.26%,130:1.26%,180:1.26%,175:1.26%,60:1.26%,170:1.26%,71:1.26%,86:1.26%,165:1.01%,52:1.01%,120:1.01%,83:1.01%,76:1.01%,48:0.75%,125:0.75%,190:0.75%,112:0.75%,69:0.75%,74:0.75%,63:0.75%,215:0.75%,96:0.75%,225:0.75%,139:0.5%,153:0.5%,87:0.5%,198:0.5%,53:0.5%,79:0.5%,129:0.5%,98:0.5%,155:0.5%,160:0.5%,81:0.5%,58:0.5%,62:0.5%,46:0.5%,66:0.25%,91:0.25%,142:0.25%,102:0.25%,116:0.25%,89:0.25%,167:0.25%,137:0.25%,135:0.25%,49:0.25%,149:0.25%,61:0.25%,133:0.25%,148:0.25%,220:0.25%,103:0.25%,77:0.25%,64:0.25%,158:0.25%,54:0.25%,208:0.25%,132:0.25%,152:0.25%,122:0.25%,210:0.25%,93:0.25%,200:0.25%,94:0.25%,108:0.25%,230:0.25%,193:0.25%,138:0.25%,107:0.25%,113:0.25%,82:0.25%]\n",
      "** weight:351 (88%)\n",
      "** acceleration:[14.5:5.78%,15.5:5.28%,16.0:4.02%,14.0:4.02%,13.5:3.77%,15.0:3.52%,17.0:3.52%,16.5:3.27%,13.0:3.02%,19.0:3.02%,12.0:2.51%,16.4:2.26%,12.5:2.01%,18.0:2.01%,11.0:1.76%,11.5:1.76%,14.9:1.76%,15.8:1.76%,13.2:1.51%,19.5:1.51%,17.3:1.26%,21.0:1.26%,18.5:1.26%,14.4:1.26%,18.2:1.26%,14.7:1.26%,16.9:1.01%,15.7:1.01%,17.6:1.01%,18.6:1.01%,10.0:1.01%,15.4:1.01%,16.2:1.01%,17.5:1.01%,16.7:0.75%,12.8:0.75%,20.5:0.75%,19.4:0.75%,15.3:0.75%,17.7:0.75%,14.2:0.75%,16.6:0.75%,14.8:0.75%,15.2:0.75%,19.2:0.75%,13.4:0.5%,13.7:0.5%,18.7:0.5%,9.5:0.5%,12.6:0.5%,8.5:0.5%,20.1:0.5%,12.2:0.5%,12.9:0.5%,19.6:0.5%,14.3:0.5%,11.4:0.5%,15.1:0.5%,17.2:0.5%,22.2:0.5%,13.6:0.5%,16.8:0.5%,17.4:0.5%,13.8:0.5%,15.9:0.5%,13.9:0.5%,17.8:0.5%,24.8:0.25%,10.5:0.25%,9.0:0.25%,23.5:0.25%,11.3:0.25%,16.1:0.25%,20.4:0.25%,14.1:0.25%,18.8:0.25%,11.2:0.25%,12.1:0.25%,18.1:0.25%,21.8:0.25%,21.5:0.25%,20.7:0.25%,19.9:0.25%,17.1:0.25%,24.6:0.25%,8.0:0.25%,23.7:0.25%,22.1:0.25%,21.7:0.25%,11.1:0.25%,18.3:0.25%,11.6:0.25%,17.9:0.25%,15.6:0.25%,21.9:0.25%]\n",
      "** year:[73:10.05%,78:9.05%,76:8.54%,82:7.79%,75:7.54%,81:7.29%,80:7.29%,79:7.29%,70:7.29%,77:7.04%,72:7.04%,71:7.04%,74:6.78%]\n",
      "** origin:[1:62.56%,3:19.85%,2:17.59%]\n",
      "** name:305 (76%)\n"
     ]
    }
   ],
   "source": [
    "import tensorflow.contrib.learn as skflow\n",
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from scipy.stats import zscore\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "analyze(filename_read)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing Examples\n",
    "\n",
    "The above preprocessing functions can be used in a variety of ways."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length before MPG outliers dropped: 398\n",
      "Length after MPG outliers dropped: 388\n",
      "      mpg  cylinders  displacement  horsepower    weight  acceleration  year  \\\n",
      "0    18.0        1.0      0.617571    0.672271  0.630077     -1.293870    70   \n",
      "1    15.0        1.0      0.728682    1.587959  0.853259     -1.475181    70   \n",
      "2    18.0        1.0      0.645995    1.195522  0.549778     -1.656492    70   \n",
      "3    16.0        1.0      0.609819    1.195522  0.546236     -1.293870    70   \n",
      "4    17.0        1.0      0.604651    0.933897  0.565130     -1.837804    70   \n",
      "5    15.0        1.0      0.932817    2.451322  1.618455     -2.019115    70   \n",
      "6    14.0        1.0      0.997416    3.026898  1.633806     -2.381737    70   \n",
      "7    14.0        1.0      0.961240    2.896085  1.584210     -2.563048    70   \n",
      "8    14.0        1.0      1.000000    3.157710  1.717647     -2.019115    70   \n",
      "9    15.0        1.0      0.832041    2.242022  1.038654     -2.563048    70   \n",
      "10   15.0        1.0      0.813953    1.718772  0.699747     -2.019115    70   \n",
      "11   14.0        1.0      0.702842    1.457147  0.754067     -2.744360    70   \n",
      "12   15.0        1.0      0.857881    1.195522  0.933557     -2.200426    70   \n",
      "13   14.0        1.0      1.000000    3.157710  0.136478     -2.019115    70   \n",
      "14   24.0        0.2      0.116279   -0.243417 -0.706655     -0.206002    70   \n",
      "15   22.0        0.6      0.335917   -0.243417 -0.162279     -0.024691    70   \n",
      "16   18.0        0.6      0.338501   -0.191092 -0.231950     -0.024691    70   \n",
      "17   21.0        0.6      0.341085   -0.505042 -0.452770      0.156620    70   \n",
      "18   27.0        0.2      0.074935   -0.426554 -0.992422     -0.387314    70   \n",
      "19   26.0        0.2      0.074935   -1.525380 -1.340775      1.788421    70   \n",
      "20   25.0        0.2      0.108527   -0.452717 -0.352397      0.700554    70   \n",
      "21   24.0        0.2      0.100775   -0.374229 -0.638165     -0.387314    70   \n",
      "22   25.0        0.2      0.093023   -0.243417 -0.703112      0.700554    70   \n",
      "23   26.0        0.2      0.136951    0.227509 -0.869613     -1.112559    70   \n",
      "24   21.0        0.6      0.338501   -0.374229 -0.380738     -0.206002    70   \n",
      "25   10.0        1.0      0.754522    2.896085  1.942010     -0.568625    70   \n",
      "26   10.0        1.0      0.617571    2.503648  1.659785     -0.206002    70   \n",
      "27   11.0        1.0      0.645995    2.765273  1.666870     -0.749936    70   \n",
      "28    9.0        1.0      0.609819    2.320510  2.080171      1.063176    70   \n",
      "29   27.0        0.2      0.074935   -0.426554 -0.992422     -0.387314    71   \n",
      "..    ...        ...           ...         ...       ...           ...   ...   \n",
      "367  28.0        0.2      0.113695   -0.426554 -0.431515      1.462061    82   \n",
      "368  27.0        0.2      0.113695   -0.426554 -0.390185      1.099439    82   \n",
      "369  34.0        0.2      0.113695   -0.426554 -0.679495      0.881865    82   \n",
      "370  31.0        0.2      0.113695   -0.505042 -0.466940      0.229145    82   \n",
      "371  29.0        0.2      0.173127   -0.531204 -0.525983      0.156620    82   \n",
      "372  27.0        0.2      0.214470   -0.374229 -0.278003      0.881865    82   \n",
      "373  24.0        0.2      0.186047   -0.321904 -0.124492      0.301669    82   \n",
      "374  23.0        0.2      0.214470   -0.282660  0.076254      1.788421    82   \n",
      "375  36.0        0.2      0.095607   -0.792829 -1.169551     -0.097216    82   \n",
      "376  37.0        0.2      0.059432   -0.949804 -1.116412      0.954390    82   \n",
      "377  31.0        0.2      0.059432   -0.949804 -1.181360      0.736816    82   \n",
      "378  38.0        0.2      0.095607   -1.080617 -0.998327     -0.314789    82   \n",
      "379  36.0        0.2      0.077519   -0.897479 -0.998327      0.628029    82   \n",
      "380  36.0        0.2      0.134367   -0.426554 -0.956997     -0.387314    82   \n",
      "381  36.0        0.2      0.100775   -0.766667 -0.903858     -0.387314    82   \n",
      "382  34.0        0.2      0.103359   -0.897479 -0.856624      0.482980    82   \n",
      "383  38.0        0.2      0.059432   -0.975967 -1.187264     -0.206002    82   \n",
      "384  32.0        0.2      0.059432   -0.975967 -1.187264      0.047833    82   \n",
      "385  38.0        0.2      0.059432   -0.975967 -1.151838      0.229145    82   \n",
      "386  25.0        0.6      0.291990    0.149021 -0.030023      0.301669    82   \n",
      "387  38.0        0.6      0.501292   -0.505042  0.052637      0.519243    82   \n",
      "388  26.0        0.2      0.227390   -0.321904 -0.455132     -0.387314    82   \n",
      "389  22.0        0.6      0.423773    0.201346 -0.159917     -0.314789    82   \n",
      "390  32.0        0.2      0.196382   -0.217254 -0.360663     -0.604887    82   \n",
      "391  36.0        0.2      0.173127   -0.531204 -0.709016     -0.931247    82   \n",
      "392  27.0        0.2      0.214470   -0.374229 -0.024119      0.628029    82   \n",
      "393  27.0        0.2      0.186047   -0.478879 -0.213056      0.011571    82   \n",
      "395  32.0        0.2      0.173127   -0.531204 -0.797581     -1.438919    82   \n",
      "396  28.0        0.2      0.134367   -0.662017 -0.407897      1.099439    82   \n",
      "397  31.0        0.2      0.131783   -0.583529 -0.295716      1.389537    82   \n",
      "\n",
      "     origin  \n",
      "0         1  \n",
      "1         1  \n",
      "2         1  \n",
      "3         1  \n",
      "4         1  \n",
      "5         1  \n",
      "6         1  \n",
      "7         1  \n",
      "8         1  \n",
      "9         1  \n",
      "10        1  \n",
      "11        1  \n",
      "12        1  \n",
      "13        1  \n",
      "14        3  \n",
      "15        1  \n",
      "16        1  \n",
      "17        1  \n",
      "18        3  \n",
      "19        2  \n",
      "20        2  \n",
      "21        2  \n",
      "22        2  \n",
      "23        2  \n",
      "24        1  \n",
      "25        1  \n",
      "26        1  \n",
      "27        1  \n",
      "28        1  \n",
      "29        3  \n",
      "..      ...  \n",
      "367       1  \n",
      "368       1  \n",
      "369       1  \n",
      "370       1  \n",
      "371       1  \n",
      "372       1  \n",
      "373       1  \n",
      "374       1  \n",
      "375       2  \n",
      "376       3  \n",
      "377       3  \n",
      "378       1  \n",
      "379       1  \n",
      "380       3  \n",
      "381       3  \n",
      "382       3  \n",
      "383       3  \n",
      "384       3  \n",
      "385       3  \n",
      "386       1  \n",
      "387       1  \n",
      "388       1  \n",
      "389       1  \n",
      "390       3  \n",
      "391       1  \n",
      "392       1  \n",
      "393       1  \n",
      "395       1  \n",
      "396       1  \n",
      "397       1  \n",
      "\n",
      "[388 rows x 8 columns]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow.contrib.learn as skflow\n",
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from scipy.stats import zscore\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "\n",
    "# create feature vector\n",
    "missing_median(df, 'horsepower')\n",
    "df.drop('name',1,inplace=True)\n",
    "encode_numeric_zscore(df, 'horsepower')\n",
    "encode_numeric_zscore(df, 'weight')\n",
    "encode_numeric_range(df, 'cylinders',0,1)\n",
    "encode_numeric_range(df, 'displacement',0,1)\n",
    "encode_numeric_zscore(df, 'acceleration')\n",
    "#encode_numeric_binary(df,'mpg',20)\n",
    "#df['origin'] = df['origin'].astype(str)\n",
    "#encode_text_tfidf(df, 'origin')\n",
    "\n",
    "# Drop outliers in horsepower\n",
    "print(\"Length before MPG outliers dropped: {}\".format(len(df)))\n",
    "remove_outliers(df,'mpg',2)\n",
    "print(\"Length after MPG outliers dropped: {}\".format(len(df)))\n",
    "\n",
    "print(df)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Other Examples: Dealing with Addresses\n",
    "\n",
    "Addresses can be difficult to encode into a neural network.  There are many different approaches, and you must consider how you can transform the address into something more meaningful.  Map coordinates can be a good approach.  [Latitude and longitude](https://en.wikipedia.org/wiki/Geographic_coordinate_system) can be a useful encoding.  Thanks to the power of the Internet, it is relatively easy to transform an address into its latitude and longitude values.  The following code determines the coordinates of [Washington University](https://wustl.edu/):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'lat': 38.6470653, 'lng': -90.30263459999999}\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "address = \"1 Brookings Dr, St. Louis, MO 63130\"\n",
    "\n",
    "response = requests.get('https://maps.googleapis.com/maps/api/geocode/json?address='+address)\n",
    "\n",
    "resp_json_payload = response.json()\n",
    "\n",
    "print(resp_json_payload['results'][0]['geometry']['location'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If latitude and longitude are simply fed into the neural network as two features, they might not be overly helpful.  These two values would allow your neural network to cluster locations on a map.  Sometimes cluster locations on a map can be useful.  Consider the percentage of the population that smokes in the USA by state:\n",
    "\n",
    "![Smokers by State](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_6_smokers.png \"Smokers by State\")\n",
    "\n",
    "The above map shows that certian behaviors, like smoking, can be clustered by global region. \n",
    "\n",
    "However, often you will want to transform the coordinates into distances.  It is reasonably easy to estimate the distance between any two points on Earth by using the [great circle distance](https://en.wikipedia.org/wiki/Great-circle_distance) between any two points on a sphere:\n",
    "\n",
    "The following code implements this formula:\n",
    "\n",
    "$\\Delta\\sigma=\\arccos\\bigl(\\sin\\phi_1\\cdot\\sin\\phi_2+\\cos\\phi_1\\cdot\\cos\\phi_2\\cdot\\cos(\\Delta\\lambda)\\bigr)$\n",
    "\n",
    "$d = r \\, \\Delta\\sigma$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distance, St. Louis, MO to Ft. Lauderdale, FL: 1685.0833252717607 km\n"
     ]
    }
   ],
   "source": [
    "from math import sin, cos, sqrt, atan2, radians\n",
    "\n",
    "# Distance function\n",
    "def distance_lat_lng(lat1,lng1,lat2,lng2):\n",
    "    # approximate radius of earth in km\n",
    "    R = 6373.0\n",
    "\n",
    "    # degrees to radians (lat/lon are in degrees)\n",
    "    lat1 = radians(lat1)\n",
    "    lng1 = radians(lng1)\n",
    "    lat2 = radians(lat2)\n",
    "    lng2 = radians(lng2)\n",
    "\n",
    "    dlng = lng2 - lng1\n",
    "    dlat = lat2 - lat1\n",
    "\n",
    "    a = sin(dlat / 2)**2 + cos(lat1) * cos(lat2) * sin(dlng / 2)**2\n",
    "    c = 2 * atan2(sqrt(a), sqrt(1 - a))\n",
    "\n",
    "    return R * c\n",
    "\n",
    "# Find lat lon for address\n",
    "def lookup_lat_lng(address):\n",
    "    response = requests.get('https://maps.googleapis.com/maps/api/geocode/json?address='+address)\n",
    "    json = response.json()\n",
    "    if len(json['results']) == 0:\n",
    "        print(\"Can't find: {}\".format(address))\n",
    "        return 0,0\n",
    "    map = json['results'][0]['geometry']['location']\n",
    "    return map['lat'],map['lng']\n",
    "\n",
    "\n",
    "# Distance between two locations\n",
    "\n",
    "import requests\n",
    "\n",
    "address1 = \"1 Brookings Dr, St. Louis, MO 63130\" \n",
    "address2 = \"3301 College Ave, Fort Lauderdale, FL 33314\"\n",
    "\n",
    "lat1, lng1 = lookup_lat_lng(address1)\n",
    "lat2, lng2 = lookup_lat_lng(address2)\n",
    "\n",
    "print(\"Distance, St. Louis, MO to Ft. Lauderdale, FL: {} km\".format(\n",
    "        distance_lat_lng(lat1,lng1,lat2,lng2)))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Distances can be useful to encode addresses as.  You must consider what distance might be useful for your dataset.  Consider:\n",
    "\n",
    "* Distance to major metropolitan area\n",
    "* Distance to competitor\n",
    "* Distance to distribution center\n",
    "* Distance to retail outlet\n",
    "\n",
    "The following code calculates the distance between 10 universities and washu:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "School 'Princeton', distance to wustl is: 1354.209708261112\n",
      "School 'Harvard', distance to wustl is: 1670.48400266576\n",
      "School 'University of Chicago', distance to wustl is: 418.0768183943189\n",
      "School 'Yale', distance to wustl is: 1504.9478116980558\n",
      "School 'Columbia University', distance to wustl is: 1021.5557486863092\n",
      "School 'Stanford', distance to wustl is: 2781.0358215314873\n",
      "School 'MIT', distance to wustl is: 1671.8200768854172\n",
      "School 'Duke University', distance to wustl is: 1047.4669155948627\n",
      "School 'University of Pennsylvania', distance to wustl is: 1306.6967081436705\n",
      "School 'Johns Hopkins', distance to wustl is: 1185.939948468073\n"
     ]
    }
   ],
   "source": [
    "# Encoding other universities by their distance to Washington University\n",
    "\n",
    "schools = [\n",
    "    [\"Princeton University, Princeton, NJ 08544\", 'Princeton'],\n",
    "    [\"Massachusetts Hall, Cambridge, MA 02138\", 'Harvard'],\n",
    "    [\"5801 S Ellis Ave, Chicago, IL 60637\", 'University of Chicago'],\n",
    "    [\"Yale, New Haven, CT 06520\", 'Yale'],\n",
    "    [\"116th St & Broadway, New York, NY 10027\", 'Columbia University'],\n",
    "    [\"450 Serra Mall, Stanford, CA 94305\", 'Stanford'],\n",
    "    [\"77 Massachusetts Ave, Cambridge, MA 02139\", 'MIT'],\n",
    "    [\"Duke University, Durham, NC 27708\", 'Duke University'],\n",
    "    [\"University of Pennsylvania, Philadelphia, PA 19104\", 'University of Pennsylvania'],\n",
    "    [\"Johns Hopkins University, Baltimore, MD 21218\", 'Johns Hopkins']\n",
    "]\n",
    "\n",
    "lat1, lng1 = lookup_lat_lng(\"1 Brookings Dr, St. Louis, MO 63130\")\n",
    "\n",
    "for address, name in schools:\n",
    "    lat2,lng2 = lookup_lat_lng(address)\n",
    "    dist = distance_lat_lng(lat1,lng1,lat2,lng2)\n",
    "    print(\"School '{}', distance to wustl is: {}\".format(name,dist))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Other Examples: Bag of Words\n",
    "\n",
    "The Bag of Words algorithm is a common means of encoding strings. (Harris, 1954) Each input represents the count of one particular word. The entire input vector would contain one value for each unique word. Consider the following strings.\n",
    "\n",
    "```\n",
    "Of Mice and Men\n",
    "Three Blind Mice\n",
    "Blind Man’s Bluff\n",
    "Mice and More Mice\n",
    "```\n",
    "\n",
    "We have the following unique words. This is our “dictionary.”\n",
    "\n",
    "```\n",
    "Input 0 : and\n",
    "Input 1 : blind\n",
    "Input 2 : bluff\n",
    "Input 3 : man’s\n",
    "Input 4 : men\n",
    "Input 5 : mice\n",
    "Input 6 : more\n",
    "Input 7 : of\n",
    "Input 8 : three\n",
    "```\n",
    "\n",
    "The four lines above would be encoded as follows.\n",
    "\n",
    "```\n",
    "Of Mice and Men [ 0 4 5 7 ]\n",
    "Three Blind Mice [ 1 5 8 ]\n",
    "Blind Man ’ s Bl u f f [ 1 2 3 ]\n",
    "Mice and More Mice [ 0 5 6 ]\n",
    "```\n",
    "\n",
    "Of course we have to fill in the missing words with zero, so we end up with\n",
    "the following.\n",
    "\n",
    "* Of Mice and Men [ 1 , 0 , 0 , 0 , 1 , 1 , 0 , 1 , 0 ]\n",
    "* Three Blind Mice [ 0 , 1 , 0 , 0 , 0 , 1 , 0 , 0 , 1 ]\n",
    "* Blind Man’s Bluff [ 0 , 1 , 1 , 1 , 0 , 0 , 0 , 0 , 0 ]\n",
    "* Mice and More Mice [ 1 , 0 , 0 , 0 , 0 , 2 , 1 , 0 , 0 ]\n",
    "\n",
    "Notice that we now have a consistent vector length of nine. Nine is the total\n",
    "number of words in our “dictionary”. Each component number in the vector is\n",
    "an index into our dictionary of available words. At each vector component is\n",
    "stored a count of the number of words for that dictionary entry. Each string\n",
    "will usually contain only a small subset of the dictionary. As a result, most of\n",
    "the vector values will be zero.\n",
    "\n",
    "As you can see, one of the most difficult aspects of machine learning programming\n",
    "is translating your problem into a fixed-length array of floating point\n",
    "numbers. The following section shows how to translate several examples.\n",
    "\n",
    "\n",
    "* [CountVectorizer](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mapping\n",
      "{'third': 7, 'this': 8, 'the': 6, 'first': 2, 'document': 1, 'second': 5, 'one': 4, 'is': 3, 'and': 0}\n",
      "\n",
      "Encoded\n",
      "[[0 1 1 1 0 0 1 0 1]\n",
      " [0 1 0 1 0 2 1 0 1]\n",
      " [1 0 0 0 1 0 1 1 0]\n",
      " [0 1 1 1 0 0 1 0 1]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "corpus = [\n",
    "    'This is the first document.',\n",
    "    'This is the second second document.',\n",
    "    'And the third one.',\n",
    "    'Is this the first document?']\n",
    "\n",
    "vectorizer = CountVectorizer(min_df=1)\n",
    "\n",
    "vectorizer.fit(corpus)\n",
    "\n",
    "print(\"Mapping\")\n",
    "print(vectorizer.vocabulary_)\n",
    "\n",
    "print()\n",
    "print(\"Encoded\")\n",
    "x = vectorizer.transform(corpus)\n",
    "print(x.toarray())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mapping\n",
      "{'99e': 56, 'gran': 151, 'c20': 77, 'valiant': 284, 'peugeot': 215, 'concours': 104, 'ciera': 97, 'vokswagen': 288, 'new': 206, '100': 1, 'liftback': 176, 'tc3': 271, 'arrow': 64, 'beetle': 70, 'premier': 223, 'matador': 190, 'b210': 69, 'skylark': 251, 'seville': 248, '2000': 21, 'volvo': 291, 'nissan': 208, 'astro': 66, 'dart': 121, 'luxus': 181, 'rabbit': 225, 'rampage': 226, 'capri': 80, 'cvcc': 118, 'grand': 153, 'c10': 76, 'v8': 283, 'se': 245, 'tr7': 278, 'camaro': 79, 'lemans': 174, '304': 35, 'sunbird': 265, 'zx': 299, '124b': 9, 'classic': 100, 'ford': 143, 'concord': 103, 'nova': 209, 'challenger': 88, 'supreme': 267, 'air': 60, 'chevy': 95, 'accord': 59, 'prix': 224, 'gx': 158, '5000s': 46, 'lebaron': 172, 'chevette': 92, 'aspen': 65, 'wagon': 293, 'audi': 67, 'cressida': 112, '244dl': 28, 'hatchback': 160, '1500': 16, '280s': 32, '131': 13, 'granada': 152, 'cuda': 115, 'woody': 294, 'fiesta': 141, 'thunderbird': 273, 'colt': 102, '111': 3, 'starlet': 261, 'cordoba': 105, 'monte': 202, '1200d': 7, 'amc': 62, 'sport': 254, 'maxima': 193, '610': 51, '200sx': 23, 'reliant': 231, 'ltd': 180, 'corolla': 106, 'hardtop': 159, 'type': 281, 'maxda': 192, 'chevrolet': 94, '340': 39, 'mazda': 194, 'duster': 132, '240d': 27, 'pickup': 217, 'pontiac': 221, 'manta': 187, '210': 24, 'skyhawk': 250, 'chevelle': 91, 'stanza': 259, 'citation': 98, 'xe': 296, 'pinto': 218, 'jetta': 170, 'auto': 68, 'mercury': 197, 'phoenix': 216, 'eldorado': 133, '810': 54, 'cruiser': 114, 'monarch': 201, 'datsun': 123, 'saab': 239, 'f108': 137, 'coupe': 111, '5000': 45, 'sj': 249, 'torino': 274, 'miser': 198, 'opel': 213, 'gt': 156, 'galaxie': 147, 'fox': 144, 'lj': 178, 'yorker': 297, 'pl510': 219, 'cavalier': 85, 'dpl': 131, '144ea': 14, 'fairmont': 139, 'sst': 257, '1131': 4, 'hi': 161, 'cobra': 101, 'prelude': 222, 'omni': 212, 'gl': 149, 'cadillac': 78, 'vista': 287, 'volare': 289, 'rx2': 237, '1900': 19, '12tl': 11, 'cricket': 113, 'turbo': 280, 'maverick': 191, 'volkswagen': 290, 'super': 266, '411': 42, 'lx': 182, 'j2000': 169, 'subaru': 263, '18i': 18, 'estate': 136, 'lynx': 183, '310': 36, 'century': 87, '10': 0, 'st': 258, 'dl': 128, 'd100': 119, '505s': 48, 'marquis': 189, 'town': 275, 'runabout': 235, 'f250': 138, 'landau': 171, 'civic': 99, 'toyota': 276, 'brougham': 74, 'sedan': 247, '2h': 33, 'monza': 203, 'buick': 75, 'impala': 167, 'iii': 166, 'squire': 256, 'renault': 232, '245': 29, 'diesel': 126, '200': 20, '4w': 43, 'caprice': 81, 'honda': 162, 'ls': 179, 'satellite': 243, 'ii': 165, 'medallion': 195, 'ambassador': 61, 'magnum': 184, 'monaco': 200, 'special': 252, '1600': 17, 'sportabout': 255, 'aries': 63, '280': 31, 'tc': 270, 'fury': 145, '510': 49, '99le': 58, '145e': 15, 'regal': 229, 'triumph': 279, '710': 53, 'oldsmobile': 210, 'lesabre': 175, '626': 52, '350': 40, '100ls': 2, 'ventura': 286, 'celica': 86, 'sapporo': 242, 'newport': 207, 'mark': 188, '12': 5, 'royale': 234, 'charger': 90, 'gremlin': 154, '2002': 22, 'plymouth': 220, '604sl': 50, 'royal': 233, '504': 47, 'bmw': 73, '320': 37, 'rebel': 228, 'strada': 262, 'delta': 124, '88': 55, 'chevroelt': 93, 'escort': 135, 'sx': 269, 'man': 186, 'carina': 82, '264gl': 30, 'x1': 295, 'electra': 134, 'ranger': 227, 'sebring': 246, '124': 8, 'tercel': 272, 'regis': 230, '300d': 34, 'glc': 150, 'deluxe': 125, 'salon': 241, 'fiat': 140, '99gle': 57, 'carlo': 83, 'v6': 282, 'mpg': 204, 'model': 199, '128': 10, 'vega': 285, 'malibu': 185, '500': 44, 'zephyr': 298, 'cougar': 109, 'toyouta': 277, 'chrysler': 96, 'starfire': 260, '225': 25, 'vw': 292, 'champ': 89, 'coronet': 108, 'safari': 240, 'pacer': 214, 'lecar': 173, 'gs': 155, 'mustang': 205, 'bel': 71, 'horizon': 163, '4000': 41, 'corona': 107, '1300': 12, 'omega': 211, 'rx': 236, 'isuzu': 168, 'cutlass': 117, 'hornet': 164, '1200': 6, 'dasher': 122, 'ghia': 148, 'door': 130, '320i': 38, 'gtl': 157, '2300': 26, 'd200': 120, 'custom': 116, 'futura': 146, 'firebird': 142, 'sw': 268, 'scirocco': 244, 'country': 110, 'suburb': 264, 'benz': 72, 'diplomat': 127, 'rx3': 238, 'limited': 177, 'catalina': 84, 'spirit': 253, 'dodge': 129, 'mercedes': 196}\n",
      "\n",
      "Encoded\n",
      "[[0 0 0 ..., 0 0 0]\n",
      " [0 0 0 ..., 0 0 0]\n",
      " [0 0 0 ..., 0 0 0]\n",
      " ..., \n",
      " [0 0 0 ..., 0 0 0]\n",
      " [0 0 0 ..., 0 0 0]\n",
      " [1 0 0 ..., 0 0 0]]\n",
      "300\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "\n",
    "corpus = df['name']\n",
    "\n",
    "vectorizer = CountVectorizer(min_df=1)\n",
    "\n",
    "vectorizer.fit(corpus)\n",
    "\n",
    "print(\"Mapping\")\n",
    "print(vectorizer.vocabulary_)\n",
    "\n",
    "print()\n",
    "print(\"Encoded\")\n",
    "x = vectorizer.transform(corpus)\n",
    "print(x.toarray())\n",
    "\n",
    "print(len(vectorizer.vocabulary_))\n",
    "\n",
    "# reverse lookup for columns\n",
    "bag_cols = [0] * len(vectorizer.vocabulary_)\n",
    "for i,key in enumerate(vectorizer.vocabulary_):\n",
    "    bag_cols[i] = key\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature ranking:\n",
      "1. 99e (0.069858186873746)\n",
      "2. gran (0.0657962678049988)\n",
      "3. c20 (0.051008883870636394)\n",
      "4. valiant (0.04581221475540706)\n",
      "5. peugeot (0.04198230223754908)\n",
      "6. concours (0.040399764416811264)\n",
      "7. ciera (0.032608231867797544)\n",
      "8. vokswagen (0.028083396361630455)\n",
      "9. new (0.025365169053742708)\n",
      "10. 100 (0.020822620576517337)\n",
      "11. liftback (0.02065610924024198)\n",
      "12. tc3 (0.019690892181255654)\n",
      "13. arrow (0.019012669846068458)\n",
      "14. beetle (0.01750032394359091)\n",
      "15. premier (0.014938020006637028)\n",
      "16. matador (0.014369710481955338)\n",
      "17. b210 (0.01252127394359739)\n",
      "18. skylark (0.010751429685292812)\n",
      "19. seville (0.010702428667434061)\n",
      "20. 2000 (0.010542674460379622)\n",
      "21. volvo (0.01025593869126505)\n",
      "22. nissan (0.00986759162389823)\n",
      "23. astro (0.009133861291584486)\n",
      "24. dart (0.008772300799934196)\n",
      "25. luxus (0.008600514953250088)\n",
      "26. rabbit (0.007821470049179373)\n",
      "27. rampage (0.007294524170328526)\n",
      "28. capri (0.007132502583116468)\n",
      "29. cvcc (0.007099310584072058)\n",
      "30. grand (0.007078688726990171)\n",
      "31. c10 (0.006707488660010955)\n",
      "32. v8 (0.006467280930991713)\n",
      "33. se (0.006412846067598118)\n",
      "34. tr7 (0.006102704354248814)\n",
      "35. camaro (0.006065784714276979)\n",
      "36. lemans (0.005627410767934647)\n",
      "37. 304 (0.00516947845622533)\n",
      "38. sunbird (0.005069313937771301)\n",
      "39. zx (0.004891430604387839)\n",
      "40. 124b (0.0048092522191788886)\n",
      "41. classic (0.004780536832539579)\n",
      "42. ford (0.004777631333172296)\n",
      "43. concord (0.004716392725717725)\n",
      "44. nova (0.004541898767747425)\n",
      "45. challenger (0.004414682290267593)\n",
      "46. supreme (0.004409132210097985)\n",
      "47. air (0.00435238709020031)\n",
      "48. chevy (0.004323360548335858)\n",
      "49. accord (0.004213496746988927)\n",
      "50. prix (0.004143793068001074)\n",
      "51. gx (0.0039554358634945)\n",
      "52. 5000s (0.0038994215363000865)\n",
      "53. lebaron (0.0038913497104200736)\n",
      "54. chevette (0.0037898467657427794)\n",
      "55. aspen (0.0037731221796846654)\n",
      "56. wagon (0.003684782256319336)\n",
      "57. audi (0.0036779690717969136)\n",
      "58. cressida (0.0036411312749935355)\n",
      "59. 244dl (0.003640013972001834)\n",
      "60. hatchback (0.0035686123303577694)\n",
      "61. 1500 (0.003521342109599453)\n",
      "62. 280s (0.00335845979581203)\n",
      "63. 131 (0.003257560971158881)\n",
      "64. granada (0.003229361061909285)\n",
      "65. cuda (0.003227396835646353)\n",
      "66. woody (0.003186537784297465)\n",
      "67. fiesta (0.003120437759660097)\n",
      "68. thunderbird (0.003087949376346005)\n",
      "69. colt (0.0030263112449864164)\n",
      "70. 111 (0.003016106870573076)\n",
      "71. starlet (0.00276047271574072)\n",
      "72. cordoba (0.0026978828592652574)\n",
      "73. monte (0.002688867170056436)\n",
      "74. 1200d (0.0026693369993517064)\n",
      "75. amc (0.0026662244569612805)\n",
      "76. sport (0.0026445934235147816)\n",
      "77. maxima (0.0026216297561914865)\n",
      "78. 610 (0.0025598751661845055)\n",
      "79. 200sx (0.002390764949718666)\n",
      "80. reliant (0.002364731483091029)\n",
      "81. ltd (0.0023635763066745387)\n",
      "82. corolla (0.002332253684585771)\n",
      "83. hardtop (0.0022248933570720693)\n",
      "84. type (0.002216468454990445)\n",
      "85. maxda (0.0021936686750714434)\n",
      "86. chevrolet (0.002184004329724791)\n",
      "87. 340 (0.002183691148679262)\n",
      "88. mazda (0.0021542648811175066)\n",
      "89. duster (0.002108274734943049)\n",
      "90. 240d (0.002086924858157528)\n",
      "91. pickup (0.0020084606226251963)\n",
      "92. pontiac (0.0019475447287013933)\n",
      "93. manta (0.0018696525404742726)\n",
      "94. 210 (0.0018685131367991584)\n",
      "95. skyhawk (0.0018599256451845906)\n",
      "96. chevelle (0.0018423867732761472)\n",
      "97. stanza (0.001793781963878932)\n",
      "98. citation (0.0017536657673044524)\n",
      "99. xe (0.0016924739724648039)\n",
      "100. pinto (0.0016821146168706524)\n",
      "101. jetta (0.0015973634676595821)\n",
      "102. auto (0.001567377423866302)\n",
      "103. mercury (0.001553247828919427)\n",
      "104. phoenix (0.0015515934732918935)\n",
      "105. eldorado (0.0015470946435324406)\n",
      "106. 810 (0.0015367331183517652)\n",
      "107. cruiser (0.0015015773127053352)\n",
      "108. monarch (0.0014946977354463977)\n",
      "109. datsun (0.001480305299049996)\n",
      "110. saab (0.0014415054475451015)\n",
      "111. f108 (0.00143893424957812)\n",
      "112. coupe (0.0014242645444977073)\n",
      "113. 5000 (0.001412836473478591)\n",
      "114. sj (0.0013666167020922157)\n",
      "115. torino (0.0013331243291685993)\n",
      "116. miser (0.0013027672109378438)\n",
      "117. opel (0.0012971440668855733)\n",
      "118. gt (0.0012731269372647292)\n",
      "119. galaxie (0.0012646122310959564)\n",
      "120. fox (0.0012379424623078653)\n",
      "121. lj (0.0012011138723334507)\n",
      "122. yorker (0.0011934146136674873)\n",
      "123. pl510 (0.0011922624040994222)\n",
      "124. cavalier (0.0011726956119608016)\n",
      "125. dpl (0.0011673732265750612)\n",
      "126. 144ea (0.0011424368454951254)\n",
      "127. fairmont (0.0011180427415947566)\n",
      "128. sst (0.0010939718737583688)\n",
      "129. 1131 (0.0010900464617843424)\n",
      "130. hi (0.0010895775853801042)\n",
      "131. cobra (0.0010683834496410622)\n",
      "132. prelude (0.0010680718093904213)\n",
      "133. omni (0.0010621788920972875)\n",
      "134. gl (0.001042064702062226)\n",
      "135. cadillac (0.0010259708332757275)\n",
      "136. vista (0.0010223971383224748)\n",
      "137. volare (0.0010195827581511107)\n",
      "138. rx2 (0.0010178621729532158)\n",
      "139. 1900 (0.00101029552067883)\n",
      "140. 12tl (0.0009779689627034522)\n",
      "141. cricket (0.0009766221620128143)\n",
      "142. turbo (0.0009264485726357364)\n",
      "143. maverick (0.000911111257415155)\n",
      "144. volkswagen (0.0009044219263565262)\n",
      "145. super (0.0008865789383651361)\n",
      "146. 411 (0.0008796177789892935)\n",
      "147. lx (0.0008774564959933761)\n",
      "148. j2000 (0.0008766206603768816)\n",
      "149. subaru (0.000829124566083776)\n",
      "150. 18i (0.0008199241801540197)\n",
      "151. estate (0.0008189603660715666)\n",
      "152. lynx (0.0008175928174699019)\n",
      "153. 310 (0.000811079939525088)\n",
      "154. century (0.0007918129610193799)\n",
      "155. 10 (0.0007799838827481231)\n",
      "156. st (0.0007763841628620652)\n",
      "157. dl (0.0007650194191378492)\n",
      "158. d100 (0.0007564940897592104)\n",
      "159. 505s (0.0007560080585707475)\n",
      "160. marquis (0.0007446462138476084)\n",
      "161. town (0.0007377487862693738)\n",
      "162. runabout (0.0007271955691454409)\n",
      "163. f250 (0.000693953567409178)\n",
      "164. landau (0.0006919151700483239)\n",
      "165. civic (0.0006918116301433932)\n",
      "166. toyota (0.0006769789279974073)\n",
      "167. brougham (0.0006757742311926802)\n",
      "168. sedan (0.0006439371075384423)\n",
      "169. 2h (0.0006365554362269344)\n",
      "170. monza (0.0006246597107919899)\n",
      "171. buick (0.0006208528466254748)\n",
      "172. impala (0.0006124576210108167)\n",
      "173. iii (0.0006122265760668993)\n",
      "174. squire (0.0005985094949514362)\n",
      "175. renault (0.0005928129188422803)\n",
      "176. 245 (0.0005433360409432352)\n",
      "177. diesel (0.0005413759188962672)\n",
      "178. 200 (0.000514768693542608)\n",
      "179. 4w (0.0005041481719848354)\n",
      "180. caprice (0.0004961670385032416)\n",
      "181. honda (0.0004934322940419157)\n",
      "182. ls (0.0004896013460710245)\n",
      "183. satellite (0.0004855042133954258)\n",
      "184. ii (0.00047324355013794007)\n",
      "185. medallion (0.0004636553214286276)\n",
      "186. ambassador (0.0004583510713887751)\n",
      "187. magnum (0.0004556646549686148)\n",
      "188. monaco (0.00044915951815501903)\n",
      "189. special (0.0004486808906455319)\n",
      "190. 1600 (0.00044772894391571034)\n",
      "191. sportabout (0.0004405517411809396)\n",
      "192. aries (0.0004392824811697179)\n",
      "193. 280 (0.0004341001012296668)\n",
      "194. tc (0.0004339938486215451)\n",
      "195. fury (0.00042944096844703015)\n",
      "196. 510 (0.0004151566108737538)\n",
      "197. 99le (0.000410060989628179)\n",
      "198. 145e (0.0004085692340215761)\n",
      "199. regal (0.00039946490616565633)\n",
      "200. triumph (0.0003982892161702556)\n",
      "201. 710 (0.0003979445287500527)\n",
      "202. oldsmobile (0.0003958079997130763)\n",
      "203. lesabre (0.0003684072183124499)\n",
      "204. 626 (0.00036449207456034375)\n",
      "205. 350 (0.0003588571469675093)\n",
      "206. 100ls (0.0003568015050362267)\n",
      "207. ventura (0.00035052586151976863)\n",
      "208. celica (0.0003473851206705483)\n",
      "209. sapporo (0.0003420213936211421)\n",
      "210. newport (0.00034062927837360987)\n",
      "211. mark (0.0003402629380177425)\n",
      "212. 12 (0.0003332124313259237)\n",
      "213. royale (0.00032545795440500015)\n",
      "214. charger (0.0003173012054950901)\n",
      "215. gremlin (0.00030902126148182505)\n",
      "216. 2002 (0.0003032583621705214)\n",
      "217. plymouth (0.0003010952820093363)\n",
      "218. 604sl (0.0002940042075516602)\n",
      "219. royal (0.00029374738198989053)\n",
      "220. 504 (0.0002792313697584064)\n",
      "221. bmw (0.0002788360337060003)\n",
      "222. 320 (0.0002776152110544088)\n",
      "223. rebel (0.0002774495469971049)\n",
      "224. strada (0.00027288389555120234)\n",
      "225. delta (0.00026900054477813404)\n",
      "226. 88 (0.0002574910179518917)\n",
      "227. chevroelt (0.00025436609597443165)\n",
      "228. escort (0.00025392270248017105)\n",
      "229. sx (0.00025287550566753503)\n",
      "230. man (0.0002460656549029872)\n",
      "231. carina (0.0002450060312576913)\n",
      "232. 264gl (0.00024106493351103313)\n",
      "233. x1 (0.00023721685152842815)\n",
      "234. electra (0.00023212875850958526)\n",
      "235. ranger (0.00022830094784139393)\n",
      "236. sebring (0.00022690295668001563)\n",
      "237. 124 (0.00022469345165969845)\n",
      "238. tercel (0.0002163029280049272)\n",
      "239. regis (0.00021596266573046011)\n",
      "240. 300d (0.0002105909042520773)\n",
      "241. glc (0.00020924414977481807)\n",
      "242. deluxe (0.0002015104248796245)\n",
      "243. salon (0.00020056501666563048)\n",
      "244. fiat (0.00019906354190628216)\n",
      "245. 99gle (0.00019792059846661947)\n",
      "246. carlo (0.00019479349075041875)\n",
      "247. v6 (0.00019367029509640167)\n",
      "248. mpg (0.0001911594739921858)\n",
      "249. model (0.00018016122768815094)\n",
      "250. 128 (0.0001765273314254329)\n",
      "251. vega (0.0001580482579352968)\n",
      "252. malibu (0.00015585857008904678)\n",
      "253. 500 (0.0001556943298045007)\n",
      "254. zephyr (0.00015504465520757156)\n",
      "255. cougar (0.00015464540361501868)\n",
      "256. toyouta (0.00015218257590328093)\n",
      "257. chrysler (0.00015182895665000887)\n",
      "258. starfire (0.00014846194290711842)\n",
      "259. 225 (0.0001444900598378759)\n",
      "260. vw (0.0001402926387993275)\n",
      "261. champ (0.00013473028324044773)\n",
      "262. coronet (0.0001342501138446045)\n",
      "263. safari (0.00013309297767526)\n",
      "264. pacer (0.00013158916135857065)\n",
      "265. lecar (0.00012430880252108634)\n",
      "266. gs (0.00011286568384797587)\n",
      "267. mustang (0.00011265494612653889)\n",
      "268. bel (0.00010911453069107237)\n",
      "269. horizon (0.00010552292383302249)\n",
      "270. 4000 (0.00010289041566846265)\n",
      "271. corona (0.00010238460442714537)\n",
      "272. 1300 (0.00010020265098662577)\n",
      "273. omega (8.85696524480241e-05)\n",
      "274. rx (8.607245210090368e-05)\n",
      "275. isuzu (8.106721774674678e-05)\n",
      "276. cutlass (7.986084576140837e-05)\n",
      "277. hornet (7.891780273251315e-05)\n",
      "278. 1200 (7.491853450617297e-05)\n",
      "279. dasher (7.43822112435414e-05)\n",
      "280. ghia (7.172640477721485e-05)\n",
      "281. door (7.170628259704008e-05)\n",
      "282. 320i (6.916089743903923e-05)\n",
      "283. gtl (6.851609951694874e-05)\n",
      "284. 2300 (6.650895674851617e-05)\n",
      "285. d200 (6.599607563771107e-05)\n",
      "286. custom (6.51773180411194e-05)\n",
      "287. futura (6.064394497659656e-05)\n",
      "288. firebird (5.8273667662948626e-05)\n",
      "289. sw (5.533106575103077e-05)\n",
      "290. scirocco (5.377920959814423e-05)\n",
      "291. country (5.099831329442585e-05)\n",
      "292. suburb (4.991227840855166e-05)\n",
      "293. benz (4.626001848766871e-05)\n",
      "294. diplomat (4.574829401594722e-05)\n",
      "295. rx3 (4.3489798089526794e-05)\n",
      "296. limited (3.324011018817211e-05)\n",
      "297. catalina (3.112809553723385e-05)\n",
      "298. spirit (2.978749370637539e-05)\n",
      "299. dodge (1.4736605968426178e-05)\n",
      "300. mercedes (5.9328904773682955e-06)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  49 tasks       | elapsed:    0.3s\n",
      "[Parallel(n_jobs=1)]: Done  50 out of  50 | elapsed:    0.3s finished\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "\n",
    "#x = x.toarray() #.as_matrix()\n",
    "y = df['mpg'].as_matrix()\n",
    "\n",
    "# Build a forest and compute the feature importances\n",
    "forest = RandomForestRegressor(n_estimators=50,\n",
    "                              random_state=0, verbose = True)\n",
    "forest.fit(x, y)\n",
    "importances = forest.feature_importances_\n",
    "std = np.std([tree.feature_importances_ for tree in forest.estimators_],\n",
    "             axis=0)\n",
    "indices = np.argsort(importances)[::-1]\n",
    "\n",
    "# Print the feature ranking\n",
    "print(\"Feature ranking:\")\n",
    "\n",
    "for f in range(x.shape[1]):\n",
    "    print(\"{}. {} ({})\".format(f + 1, bag_cols[f], importances[indices[f]]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Other Examples: Time Series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Time series data will need to be encoded for a regular feedforward neural network.  In a few classes we will see how to use a recurrent neural network to find patterns over time.  For now, we will encode the series into input neurons.\n",
    "\n",
    "Financial forecasting is a very popular form of temporal algorithm. A temporal algorithm is one that accepts input for values that range over time. If the algorithm supports short term memory (internal state) then ranges over time are supported automatically. If your algorithm does not have an internal state then you should use an input window and a prediction window. Most algorithms do not have an internal state. To see how to use these windows, consider if you would like the algorithm to predict the stock market. You begin with the closing price for a stock over several days:\n",
    "\n",
    "```\n",
    "Day 1 : $45\n",
    "Day 2 : $47\n",
    "Day 3 : $48\n",
    "Day 4 : $40\n",
    "Day 5 : $41\n",
    "Day 6 : $43\n",
    "Day 7 : $45\n",
    "Day 8 : $57\n",
    "Day 9 : $50\n",
    "Day 10 : $41\n",
    "```\n",
    "\n",
    "The first step is to normalize the data. This is necessary whether your algorithm has internal state or not. To normalize, we want to change each number into the percent movement from the previous day. For example, day 2 would become 0.04, because there is a 4% difference between $45 and $47. Once you perform this calculation for every day, the data set will look like the following:\n",
    "\n",
    "```\n",
    "Day 2 : 0. 04\n",
    "Day 3 : 0. 02\n",
    "Day 4:−0.16\n",
    "Day 5 : 0. 02\n",
    "Day 6 : 0. 04\n",
    "Day 7 : 0. 04\n",
    "Day 8 : 0. 04\n",
    "Day 9:−0.12\n",
    "Day 10:−0.18\n",
    "```\n",
    "\n",
    "In order to create an algorithm that will predict the next day’s values, we need to think about how to encode this data to be presented to the algorithm. The encoding depends on whether the algorithm has an internal state. The internal state allows the algorithm to use the last few values inputted to help establish trends.\n",
    "\n",
    "Many machine learning algorithms have no internal state. If this is the case, then you will typically use a sliding window algorithm to encode the data. To do this, we use the last three prices to predict the next one. The inputs would be the last three-day prices, and the output would be the fourth day. The above data could be organized in the following way to provide training data.\n",
    "\n",
    "These cases specified the ideal output for the given inputs:\n",
    "\n",
    "```\n",
    "[ 0.04 , 0.02 , −0.16 ] −> 0.02\n",
    "[ 0.02 , −0.16 , 0.02 ] −> 0.04\n",
    "[ −0.16 , 0.02 , 0.04 ] −> 0.04\n",
    "[ 0.02 , 0.04 , 0.04 ] −> 0. 26\n",
    "[ 0.04 , 0.04 , 0.26 ] −> −0.12\n",
    "[ 0.04 , 0.26 , −0.12 ] −> −0.18\n",
    "```\n",
    "\n",
    "The above encoding would require that the algorithm have three inputs and one output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normalized price history:\n",
      "[0.044444444444444446, 0.02127659574468085, -0.16666666666666666, 0.025, 0.04878048780487805, 0.046511627906976744, 0.26666666666666666, -0.12280701754385964, -0.18]\n",
      "\n",
      "Rounded normalized price history:\n",
      "[ 0.04  0.02 -0.17  0.02  0.05  0.05  0.27 -0.12 -0.18]\n",
      "\n",
      "Time Boxed(time series encoded):\n",
      "[ 0.04  0.02 -0.17] -> [ 0.02]\n",
      "[ 0.02 -0.17  0.02] -> [ 0.05]\n",
      "[-0.17  0.02  0.05] -> [ 0.05]\n",
      "[ 0.02  0.05  0.05] -> [ 0.27]\n",
      "[ 0.05  0.05  0.27] -> [-0.12]\n",
      "[ 0.05  0.27 -0.12] -> [-0.18]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "def normalize_price_change(history):\n",
    "    last = None\n",
    "    \n",
    "    result = []\n",
    "    for price in history:\n",
    "        if last is not None:\n",
    "            result.append( float(price-last)/last )\n",
    "        last = price\n",
    "\n",
    "    return result\n",
    "\n",
    "def encode_timeseries_window(source, lag_size, lead_size):\n",
    "    \"\"\"\n",
    "    Encode raw data to a time-series window.\n",
    "    :param source: A 2D array that specifies the source to be encoded.\n",
    "    :param lag_size: The number of rows uses to predict.\n",
    "    :param lead_size: The number of rows to be predicted\n",
    "    :return: A tuple that contains the x (input) & y (expected output) for training.\n",
    "    \"\"\"\n",
    "    result_x = []\n",
    "    result_y = []\n",
    "\n",
    "    output_row_count = len(source) - (lag_size + lead_size) + 1\n",
    "    \n",
    "\n",
    "    for raw_index in range(output_row_count):\n",
    "        encoded_x = []\n",
    "\n",
    "        # Encode x (predictors)\n",
    "        for j in range(lag_size):\n",
    "            encoded_x.append(source[raw_index+j])\n",
    "\n",
    "        result_x.append(encoded_x)\n",
    "\n",
    "        # Encode y (prediction)\n",
    "        encoded_y = []\n",
    "\n",
    "        for j in range(lead_size):\n",
    "            encoded_y.append(source[lag_size+raw_index+j])\n",
    "\n",
    "        result_y.append(encoded_y)\n",
    "\n",
    "    return result_x, result_y\n",
    "\n",
    "\n",
    "price_history = [ 45, 47, 48, 40, 41, 43, 45, 57, 50, 41 ]\n",
    "norm_price_history = normalize_price_change(price_history)\n",
    "\n",
    "print(\"Normalized price history:\")\n",
    "print(norm_price_history)\n",
    "\n",
    "print()\n",
    "print(\"Rounded normalized price history:\")\n",
    "norm_price_history = np.round(norm_price_history,2)\n",
    "print(norm_price_history)\n",
    "\n",
    "\n",
    "print()\n",
    "print(\"Time Boxed(time series encoded):\")\n",
    "x, y = encode_timeseries_window(norm_price_history, 3, 1)\n",
    "\n",
    "for x_row, y_row in zip(x,y):\n",
    "    print(\"{} -> {}\".format(np.round(x_row,2), np.round(y_row,2)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:wustl]",
   "language": "python",
   "name": "conda-env-wustl-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}