machine_learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning\n",
    "Katie Thomas  \n",
    "kdt13@pitt.edu\n",
    "\n",
    "Now onto the fun stuff!\n",
    "\n",
    "## Table of Contents\n",
    "1. DataFrames used:\n",
    "    - [All posts and responses](#Posts-and-responses-all-together,-with-known-genders): a quick summary of the file I will be using to predict gender, regardless of whether they are the poster or responder\n",
    "    - [Both genders visible](#Responses-to-posts,-genders-visible): a quick summary of the file I will be using to predict both genders, given the response text only\n",
    "2. [Machine Learning plan](#Solidifying-ML-plan): a discussion of my plan for the different classifiers, as well as a cleanup of the file to be used for this\n",
    "3. Machine Learning:\n",
    "    - [Predicting singular gender](#Predict-gender-of-poster/responder): attempting to predict the gender, regardless of whether the text was from a post or response\n",
    "        - also an attempt to add more features, other than just text\n",
    "    - [Predicting both genders](#Predict-both-genders): attempting to predict the gender of both the poster and responder, just give the responder's text\n",
    "    - [Predicting poster's gender](#Predict-poster-gender): attempting to predict the gender of the poster, given the response text\n",
    "\n",
    "### Setting up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import things\n",
    "import pickle\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Posts and responses all together, with known genders\n",
    "This dataframe will be used to predict gender, regardless of whether they were the poster or responder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>text</th>\n",
       "      <th>text_length</th>\n",
       "      <th>avg_slen</th>\n",
       "      <th>hedges</th>\n",
       "      <th>questions</th>\n",
       "      <th>source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>M</td>\n",
       "      <td>President Trump has hit the ground running. As...</td>\n",
       "      <td>27</td>\n",
       "      <td>13.500000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>fb_congress_post</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>M</td>\n",
       "      <td>It's time to put Obama in the history books. W...</td>\n",
       "      <td>37</td>\n",
       "      <td>18.500000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>fb_congress_post</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>M</td>\n",
       "      <td>Since its inception, Obamacare has been an abs...</td>\n",
       "      <td>28</td>\n",
       "      <td>9.333333</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>fb_congress_post</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>M</td>\n",
       "      <td>It's time. Like if you agree.</td>\n",
       "      <td>9</td>\n",
       "      <td>4.500000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>fb_congress_post</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>M</td>\n",
       "      <td>In 15 days we will restore conservative values...</td>\n",
       "      <td>17</td>\n",
       "      <td>8.500000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>fb_congress_post</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  gender                                               text  text_length  \\\n",
       "0      M  President Trump has hit the ground running. As...           27   \n",
       "1      M  It's time to put Obama in the history books. W...           37   \n",
       "2      M  Since its inception, Obamacare has been an abs...           28   \n",
       "3      M                      It's time. Like if you agree.            9   \n",
       "4      M  In 15 days we will restore conservative values...           17   \n",
       "\n",
       "    avg_slen  hedges  questions            source  \n",
       "0  13.500000       0          0  fb_congress_post  \n",
       "1  18.500000       0          0  fb_congress_post  \n",
       "2   9.333333       0          0  fb_congress_post  \n",
       "3   4.500000       0          0  fb_congress_post  \n",
       "4   8.500000       0          0  fb_congress_post  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_posts_responses = pd.read_pickle(\"all_posts_responses.pkl\")\n",
    "all_posts_responses.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "M    121244\n",
       "W     80019\n",
       "Name: gender, dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# gender distribution\n",
    "all_posts_responses.gender.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>text_length</th>\n",
       "      <th>avg_slen</th>\n",
       "      <th>hedges</th>\n",
       "      <th>questions</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>source</th>\n",
       "      <th>gender</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">fb_congress_post</th>\n",
       "      <th>M</th>\n",
       "      <td>59.149368</td>\n",
       "      <td>18.556301</td>\n",
       "      <td>0.030742</td>\n",
       "      <td>0.000683</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W</th>\n",
       "      <td>64.457930</td>\n",
       "      <td>19.234648</td>\n",
       "      <td>0.029615</td>\n",
       "      <td>0.000277</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">fb_wiki_post</th>\n",
       "      <th>M</th>\n",
       "      <td>44.733200</td>\n",
       "      <td>12.706150</td>\n",
       "      <td>0.062865</td>\n",
       "      <td>0.001668</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W</th>\n",
       "      <td>33.098703</td>\n",
       "      <td>11.506994</td>\n",
       "      <td>0.028190</td>\n",
       "      <td>0.000077</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">fitocracy_post</th>\n",
       "      <th>M</th>\n",
       "      <td>17.085614</td>\n",
       "      <td>9.309599</td>\n",
       "      <td>0.030264</td>\n",
       "      <td>0.000110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W</th>\n",
       "      <td>18.663198</td>\n",
       "      <td>9.305174</td>\n",
       "      <td>0.036183</td>\n",
       "      <td>0.000132</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">fitocracy_response</th>\n",
       "      <th>M</th>\n",
       "      <td>16.178226</td>\n",
       "      <td>8.090028</td>\n",
       "      <td>0.040457</td>\n",
       "      <td>0.000492</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W</th>\n",
       "      <td>35.309405</td>\n",
       "      <td>12.356293</td>\n",
       "      <td>0.032651</td>\n",
       "      <td>0.000273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">reddit_post</th>\n",
       "      <th>M</th>\n",
       "      <td>40.154875</td>\n",
       "      <td>13.579069</td>\n",
       "      <td>0.132194</td>\n",
       "      <td>0.000278</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W</th>\n",
       "      <td>47.093379</td>\n",
       "      <td>13.990707</td>\n",
       "      <td>0.171737</td>\n",
       "      <td>0.000287</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">reddit_response</th>\n",
       "      <th>M</th>\n",
       "      <td>37.967346</td>\n",
       "      <td>13.012099</td>\n",
       "      <td>0.149037</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W</th>\n",
       "      <td>45.557087</td>\n",
       "      <td>13.339783</td>\n",
       "      <td>0.209646</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           text_length   avg_slen    hedges  questions\n",
       "source             gender                                             \n",
       "fb_congress_post   M         59.149368  18.556301  0.030742   0.000683\n",
       "                   W         64.457930  19.234648  0.029615   0.000277\n",
       "fb_wiki_post       M         44.733200  12.706150  0.062865   0.001668\n",
       "                   W         33.098703  11.506994  0.028190   0.000077\n",
       "fitocracy_post     M         17.085614   9.309599  0.030264   0.000110\n",
       "                   W         18.663198   9.305174  0.036183   0.000132\n",
       "fitocracy_response M         16.178226   8.090028  0.040457   0.000492\n",
       "                   W         35.309405  12.356293  0.032651   0.000273\n",
       "reddit_post        M         40.154875  13.579069  0.132194   0.000278\n",
       "                   W         47.093379  13.990707  0.171737   0.000287\n",
       "reddit_response    M         37.967346  13.012099  0.149037   0.000000\n",
       "                   W         45.557087  13.339783  0.209646   0.000000"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# group by source and gender\n",
    "all_posts_responses.groupby(['source','gender'])['text_length', 'avg_slen', 'hedges', 'questions'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text_length</th>\n",
       "      <th>avg_slen</th>\n",
       "      <th>hedges</th>\n",
       "      <th>questions</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gender</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>M</th>\n",
       "      <td>33.522228</td>\n",
       "      <td>12.297163</td>\n",
       "      <td>0.070766</td>\n",
       "      <td>0.000412</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W</th>\n",
       "      <td>34.515802</td>\n",
       "      <td>12.197646</td>\n",
       "      <td>0.053075</td>\n",
       "      <td>0.000200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        text_length   avg_slen    hedges  questions\n",
       "gender                                             \n",
       "M         33.522228  12.297163  0.070766   0.000412\n",
       "W         34.515802  12.197646  0.053075   0.000200"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# group by gender\n",
    "all_posts_responses.groupby('gender')['text_length', 'avg_slen', 'hedges', 'questions'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of these stats are very similar, regardless of gender (even the hedges, which I predicted would be more prominent in female posters). This makes me believe that simply looking at text would be our best bet here.\n",
    "\n",
    "## Responses to posts, genders visible\n",
    "This dataframe will be used to predict gender of poster and responder, based on responder's reaction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>responder_gender</th>\n",
       "      <th>post_text</th>\n",
       "      <th>response_text</th>\n",
       "      <th>source</th>\n",
       "      <th>post_tokens</th>\n",
       "      <th>response_tokens</th>\n",
       "      <th>post_length</th>\n",
       "      <th>response_length</th>\n",
       "      <th>post_avg_slen</th>\n",
       "      <th>response_avg_slen</th>\n",
       "      <th>post_hedges</th>\n",
       "      <th>response_hedges</th>\n",
       "      <th>post_questions</th>\n",
       "      <th>response_questions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>102</td>\n",
       "      <td>W</td>\n",
       "      <td>1196122</td>\n",
       "      <td>M</td>\n",
       "      <td>Thanks for the follow!  I followed back :)  I ...</td>\n",
       "      <td>One day at a time! =]</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>[Thanks, for, the, follow, !, I, followed, bac...</td>\n",
       "      <td>[One, day, at, a, time, !, =, ]]</td>\n",
       "      <td>25</td>\n",
       "      <td>8</td>\n",
       "      <td>12.5</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>104</td>\n",
       "      <td>W</td>\n",
       "      <td>5867</td>\n",
       "      <td>W</td>\n",
       "      <td>I've decided I have this crazy goal of running...</td>\n",
       "      <td>Crazy is synonymous with awesome in this case.</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>[I, 've, decided, I, have, this, crazy, goal, ...</td>\n",
       "      <td>[Crazy, is, synonymous, with, awesome, in, thi...</td>\n",
       "      <td>13</td>\n",
       "      <td>9</td>\n",
       "      <td>13.0</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>104</td>\n",
       "      <td>W</td>\n",
       "      <td>1635</td>\n",
       "      <td>M</td>\n",
       "      <td>Umm, yeah, so those are assisted pull-ups. Bef...</td>\n",
       "      <td>You can enter assisted pullups. Select pullups...</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>[Umm, ,, yeah, ,, so, those, are, assisted, pu...</td>\n",
       "      <td>[You, can, enter, assisted, pullups, ., Select...</td>\n",
       "      <td>19</td>\n",
       "      <td>57</td>\n",
       "      <td>9.5</td>\n",
       "      <td>11.400000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>117</td>\n",
       "      <td>M</td>\n",
       "      <td>8520</td>\n",
       "      <td>W</td>\n",
       "      <td>dam gurl lookin mad tone in dat pp holla bb</td>\n",
       "      <td>Mirin 3% bodyfat? Yeah, you are.</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>[dam, gurl, lookin, mad, tone, in, dat, pp, ho...</td>\n",
       "      <td>[Mirin, 3, %, bodyfat, ?, Yeah, ,, you, are, .]</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>10.0</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>117</td>\n",
       "      <td>M</td>\n",
       "      <td>29126</td>\n",
       "      <td>M</td>\n",
       "      <td>What's up there bear mode?</td>\n",
       "      <td>Hey! I just started a new job, so things are s...</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>[What, 's, up, there, bear, mode, ?]</td>\n",
       "      <td>[Hey, !, I, just, started, a, new, job, ,, so,...</td>\n",
       "      <td>7</td>\n",
       "      <td>26</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  op_id op_gender responder_id responder_gender  \\\n",
       "0   102         W      1196122                M   \n",
       "1   104         W         5867                W   \n",
       "2   104         W         1635                M   \n",
       "3   117         M         8520                W   \n",
       "4   117         M        29126                M   \n",
       "\n",
       "                                           post_text  \\\n",
       "0  Thanks for the follow!  I followed back :)  I ...   \n",
       "1  I've decided I have this crazy goal of running...   \n",
       "2  Umm, yeah, so those are assisted pull-ups. Bef...   \n",
       "3        dam gurl lookin mad tone in dat pp holla bb   \n",
       "4                         What's up there bear mode?   \n",
       "\n",
       "                                       response_text     source  \\\n",
       "0                              One day at a time! =]  fitocracy   \n",
       "1     Crazy is synonymous with awesome in this case.  fitocracy   \n",
       "2  You can enter assisted pullups. Select pullups...  fitocracy   \n",
       "3                   Mirin 3% bodyfat? Yeah, you are.  fitocracy   \n",
       "4  Hey! I just started a new job, so things are s...  fitocracy   \n",
       "\n",
       "                                         post_tokens  \\\n",
       "0  [Thanks, for, the, follow, !, I, followed, bac...   \n",
       "1  [I, 've, decided, I, have, this, crazy, goal, ...   \n",
       "2  [Umm, ,, yeah, ,, so, those, are, assisted, pu...   \n",
       "3  [dam, gurl, lookin, mad, tone, in, dat, pp, ho...   \n",
       "4               [What, 's, up, there, bear, mode, ?]   \n",
       "\n",
       "                                     response_tokens  post_length  \\\n",
       "0                   [One, day, at, a, time, !, =, ]]           25   \n",
       "1  [Crazy, is, synonymous, with, awesome, in, thi...           13   \n",
       "2  [You, can, enter, assisted, pullups, ., Select...           19   \n",
       "3    [Mirin, 3, %, bodyfat, ?, Yeah, ,, you, are, .]           10   \n",
       "4  [Hey, !, I, just, started, a, new, job, ,, so,...            7   \n",
       "\n",
       "   response_length  post_avg_slen  response_avg_slen  post_hedges  \\\n",
       "0                8           12.5           4.000000            0   \n",
       "1                9           13.0           9.000000            0   \n",
       "2               57            9.5          11.400000            0   \n",
       "3               10           10.0           5.000000            0   \n",
       "4               26            7.0           8.666667            0   \n",
       "\n",
       "   response_hedges  post_questions  response_questions  \n",
       "0                0               0                   0  \n",
       "1                0               0                   0  \n",
       "2                0               0                   0  \n",
       "3                0               0                   0  \n",
       "4                0               0                   0  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gender_visible_df = pd.read_pickle(\"gender_visible_df.pkl\")\n",
    "gender_visible_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 396766 entries, 0 to 396765\n",
      "Data columns (total 17 columns):\n",
      "op_id                 396766 non-null object\n",
      "op_gender             396766 non-null object\n",
      "responder_id          396766 non-null object\n",
      "responder_gender      396766 non-null object\n",
      "post_text             396766 non-null object\n",
      "response_text         396766 non-null object\n",
      "source                396766 non-null object\n",
      "post_tokens           396766 non-null object\n",
      "response_tokens       396766 non-null object\n",
      "post_length           396766 non-null int64\n",
      "response_length       396766 non-null int64\n",
      "post_avg_slen         396766 non-null float64\n",
      "response_avg_slen     396766 non-null float64\n",
      "post_hedges           396766 non-null int64\n",
      "response_hedges       396766 non-null int64\n",
      "post_questions        396766 non-null int64\n",
      "response_questions    396766 non-null int64\n",
      "dtypes: float64(2), int64(6), object(9)\n",
      "memory usage: 51.5+ MB\n"
     ]
    }
   ],
   "source": [
    "gender_visible_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everything looks great! Let's recap some basic stats:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "M    237339\n",
       "W    159427\n",
       "Name: op_gender, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# poster gender distribution\n",
    "gender_visible_df.op_gender.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "M    217639\n",
       "W    179127\n",
       "Name: responder_gender, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# responder gender distribution\n",
    "gender_visible_df.responder_gender.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>post_length</th>\n",
       "      <th>response_length</th>\n",
       "      <th>post_avg_slen</th>\n",
       "      <th>response_avg_slen</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>source</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>responder_gender</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">fitocracy</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">M</th>\n",
       "      <th>M</th>\n",
       "      <td>19.023550</td>\n",
       "      <td>16.612086</td>\n",
       "      <td>9.677136</td>\n",
       "      <td>8.384321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W</th>\n",
       "      <td>15.503597</td>\n",
       "      <td>31.527746</td>\n",
       "      <td>8.910398</td>\n",
       "      <td>11.419609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">W</th>\n",
       "      <th>M</th>\n",
       "      <td>19.160420</td>\n",
       "      <td>15.814585</td>\n",
       "      <td>9.066472</td>\n",
       "      <td>7.888464</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W</th>\n",
       "      <td>18.134494</td>\n",
       "      <td>40.720844</td>\n",
       "      <td>9.586728</td>\n",
       "      <td>13.561374</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">reddit</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">M</th>\n",
       "      <th>M</th>\n",
       "      <td>48.399651</td>\n",
       "      <td>38.106992</td>\n",
       "      <td>13.993324</td>\n",
       "      <td>12.982770</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W</th>\n",
       "      <td>62.564599</td>\n",
       "      <td>48.993869</td>\n",
       "      <td>15.221700</td>\n",
       "      <td>13.782006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">W</th>\n",
       "      <th>M</th>\n",
       "      <td>55.883847</td>\n",
       "      <td>50.545774</td>\n",
       "      <td>14.828193</td>\n",
       "      <td>14.161640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W</th>\n",
       "      <td>58.729640</td>\n",
       "      <td>47.042140</td>\n",
       "      <td>15.947841</td>\n",
       "      <td>14.290182</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      post_length  response_length  \\\n",
       "source    op_gender responder_gender                                 \n",
       "fitocracy M         M                   19.023550        16.612086   \n",
       "                    W                   15.503597        31.527746   \n",
       "          W         M                   19.160420        15.814585   \n",
       "                    W                   18.134494        40.720844   \n",
       "reddit    M         M                   48.399651        38.106992   \n",
       "                    W                   62.564599        48.993869   \n",
       "          W         M                   55.883847        50.545774   \n",
       "                    W                   58.729640        47.042140   \n",
       "\n",
       "                                      post_avg_slen  response_avg_slen  \n",
       "source    op_gender responder_gender                                    \n",
       "fitocracy M         M                      9.677136           8.384321  \n",
       "                    W                      8.910398          11.419609  \n",
       "          W         M                      9.066472           7.888464  \n",
       "                    W                      9.586728          13.561374  \n",
       "reddit    M         M                     13.993324          12.982770  \n",
       "                    W                     15.221700          13.782006  \n",
       "          W         M                     14.828193          14.161640  \n",
       "                    W                     15.947841          14.290182  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gender_visible_df.groupby(['source','op_gender','responder_gender'])['post_length','response_length',\n",
    "                                                                     'post_avg_slen','response_avg_slen'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So Reddit posts seem to be longer overall than posts on Fitocracy. This makes me believe that this may not be such a good statistic to use for machine learning. The more important factor may be the actual content of the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "percentage male posters with hedges: 0.0597668314099242\n",
      "percentage female posters with hedges: 0.04752018165053598\n",
      "percentage male responders with hedges: 0.06502051562449744\n",
      "percentage female responders with hedges: 0.043739916372182865\n"
     ]
    }
   ],
   "source": [
    "# hedges\n",
    "\n",
    "# male posters with hedges\n",
    "male_hedges_post = gender_visible_df[(gender_visible_df.op_gender == 'M') & (gender_visible_df.post_hedges != 0)]\n",
    "# male responders with hedges\n",
    "male_hedges_response = gender_visible_df[(gender_visible_df.responder_gender == 'M') & (gender_visible_df.response_hedges != 0)]\n",
    "\n",
    "# female posters with hedges\n",
    "female_hedges_post = gender_visible_df[(gender_visible_df.op_gender == 'W') & (gender_visible_df.post_hedges != 0)]\n",
    "# female responders with hedges\n",
    "female_hedges_response = gender_visible_df[(gender_visible_df.responder_gender == 'W') & (gender_visible_df.response_hedges != 0)]\n",
    "\n",
    "print('percentage male posters with hedges:',\n",
    "      len(male_hedges_post.index) / gender_visible_df.op_gender.value_counts()['M'])\n",
    "\n",
    "print('percentage female posters with hedges:',\n",
    "      len(female_hedges_post.index) / gender_visible_df.op_gender.value_counts()['W'])\n",
    "\n",
    "print('percentage male responders with hedges:',\n",
    "      len(male_hedges_response.index) / gender_visible_df.responder_gender.value_counts()['M'])\n",
    "\n",
    "print('percentage female responders with hedges:',\n",
    "      len(female_hedges_response.index) / gender_visible_df.responder_gender.value_counts()['W'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is actually the opposite trend that I wanted to see. It was my belief coming in that female posters/responders would use more hedges, but that is in fact not the case. But Reddit was actually skewed in the direction I wanted! What does just Reddit data look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "percentage male posters with hedges: 0.14174305435449128\n",
      "percentage female posters with hedges: 0.19084073820915926\n",
      "percentage male responders with hedges: 0.12842243547637738\n",
      "percentage female responders with hedges: 0.1791044776119403\n"
     ]
    }
   ],
   "source": [
    "reddit = gender_visible_df[gender_visible_df.source == 'reddit']\n",
    "\n",
    "# male posters with hedges\n",
    "male_hedges_post = reddit[(reddit.op_gender == 'M') & (reddit.post_hedges != 0)]\n",
    "# male responders with hedges\n",
    "male_hedges_response = reddit[(reddit.responder_gender == 'M') & (reddit.response_hedges != 0)]\n",
    "\n",
    "# female posters with hedges\n",
    "female_hedges_post = reddit[(reddit.op_gender == 'W') & (reddit.post_hedges != 0)]\n",
    "# female responders with hedges\n",
    "female_hedges_response = reddit[(reddit.responder_gender == 'W') & (reddit.response_hedges != 0)]\n",
    "\n",
    "print('percentage male posters with hedges:',\n",
    "      len(male_hedges_post.index) / reddit.op_gender.value_counts()['M'])\n",
    "\n",
    "print('percentage female posters with hedges:',\n",
    "      len(female_hedges_post.index) / reddit.op_gender.value_counts()['W'])\n",
    "\n",
    "print('percentage male responders with hedges:',\n",
    "      len(male_hedges_response.index) / reddit.responder_gender.value_counts()['M'])\n",
    "\n",
    "print('percentage female responders with hedges:',\n",
    "      len(female_hedges_response.index) / reddit.responder_gender.value_counts()['W'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So this looks how I'd expect. It seems to be the Fitocracy data that is messing with my hypothesis. Thus, using the hedge data in machine learning will not work for this combined data set. Maybe we should just focus on the text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Solidifying ML plan\n",
    "\n",
    "So what do I actually want to be able to predict?\n",
    "\n",
    "1. Simply the gender of the poster/responder, based on the text alone (ignoring whether they are the poster or responder).\n",
    "2. A more interesting and complicated topic would be analyzing the responder's text and predicting both their own gender and the original poster's gender. This way, we could see if different genders really do respond differently given who they are responding to. Let's make a dataframe that would fit this purpose:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_gender</th>\n",
       "      <th>responder_gender</th>\n",
       "      <th>response_text</th>\n",
       "      <th>response_tokens</th>\n",
       "      <th>response_length</th>\n",
       "      <th>response_avg_slen</th>\n",
       "      <th>source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>W</td>\n",
       "      <td>M</td>\n",
       "      <td>One day at a time! =]</td>\n",
       "      <td>[One, day, at, a, time, !, =, ]]</td>\n",
       "      <td>8</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>fitocracy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>W</td>\n",
       "      <td>W</td>\n",
       "      <td>Crazy is synonymous with awesome in this case.</td>\n",
       "      <td>[Crazy, is, synonymous, with, awesome, in, thi...</td>\n",
       "      <td>9</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>fitocracy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>W</td>\n",
       "      <td>M</td>\n",
       "      <td>You can enter assisted pullups. Select pullups...</td>\n",
       "      <td>[You, can, enter, assisted, pullups, ., Select...</td>\n",
       "      <td>57</td>\n",
       "      <td>11.400000</td>\n",
       "      <td>fitocracy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>M</td>\n",
       "      <td>W</td>\n",
       "      <td>Mirin 3% bodyfat? Yeah, you are.</td>\n",
       "      <td>[Mirin, 3, %, bodyfat, ?, Yeah, ,, you, are, .]</td>\n",
       "      <td>10</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>fitocracy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>Hey! I just started a new job, so things are s...</td>\n",
       "      <td>[Hey, !, I, just, started, a, new, job, ,, so,...</td>\n",
       "      <td>26</td>\n",
       "      <td>8.666667</td>\n",
       "      <td>fitocracy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  op_gender responder_gender  \\\n",
       "0         W                M   \n",
       "1         W                W   \n",
       "2         W                M   \n",
       "3         M                W   \n",
       "4         M                M   \n",
       "\n",
       "                                       response_text  \\\n",
       "0                              One day at a time! =]   \n",
       "1     Crazy is synonymous with awesome in this case.   \n",
       "2  You can enter assisted pullups. Select pullups...   \n",
       "3                   Mirin 3% bodyfat? Yeah, you are.   \n",
       "4  Hey! I just started a new job, so things are s...   \n",
       "\n",
       "                                     response_tokens  response_length  \\\n",
       "0                   [One, day, at, a, time, !, =, ]]                8   \n",
       "1  [Crazy, is, synonymous, with, awesome, in, thi...                9   \n",
       "2  [You, can, enter, assisted, pullups, ., Select...               57   \n",
       "3    [Mirin, 3, %, bodyfat, ?, Yeah, ,, you, are, .]               10   \n",
       "4  [Hey, !, I, just, started, a, new, job, ,, so,...               26   \n",
       "\n",
       "   response_avg_slen     source  \n",
       "0           4.000000  fitocracy  \n",
       "1           9.000000  fitocracy  \n",
       "2          11.400000  fitocracy  \n",
       "3           5.000000  fitocracy  \n",
       "4           8.666667  fitocracy  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# getting just the columns we want\n",
    "responder_df = gender_visible_df[['op_gender','responder_gender','response_text',\n",
    "                                  'response_tokens','response_length','response_avg_slen','source']]\n",
    "\n",
    "responder_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:3: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_gender</th>\n",
       "      <th>responder_gender</th>\n",
       "      <th>response_text</th>\n",
       "      <th>response_tokens</th>\n",
       "      <th>response_length</th>\n",
       "      <th>response_avg_slen</th>\n",
       "      <th>source</th>\n",
       "      <th>gender_info</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>W</td>\n",
       "      <td>M</td>\n",
       "      <td>One day at a time! =]</td>\n",
       "      <td>[One, day, at, a, time, !, =, ]]</td>\n",
       "      <td>8</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>WM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>W</td>\n",
       "      <td>W</td>\n",
       "      <td>Crazy is synonymous with awesome in this case.</td>\n",
       "      <td>[Crazy, is, synonymous, with, awesome, in, thi...</td>\n",
       "      <td>9</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>WW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>W</td>\n",
       "      <td>M</td>\n",
       "      <td>You can enter assisted pullups. Select pullups...</td>\n",
       "      <td>[You, can, enter, assisted, pullups, ., Select...</td>\n",
       "      <td>57</td>\n",
       "      <td>11.400000</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>WM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>M</td>\n",
       "      <td>W</td>\n",
       "      <td>Mirin 3% bodyfat? Yeah, you are.</td>\n",
       "      <td>[Mirin, 3, %, bodyfat, ?, Yeah, ,, you, are, .]</td>\n",
       "      <td>10</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>MW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>Hey! I just started a new job, so things are s...</td>\n",
       "      <td>[Hey, !, I, just, started, a, new, job, ,, so,...</td>\n",
       "      <td>26</td>\n",
       "      <td>8.666667</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>MM</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  op_gender responder_gender  \\\n",
       "0         W                M   \n",
       "1         W                W   \n",
       "2         W                M   \n",
       "3         M                W   \n",
       "4         M                M   \n",
       "\n",
       "                                       response_text  \\\n",
       "0                              One day at a time! =]   \n",
       "1     Crazy is synonymous with awesome in this case.   \n",
       "2  You can enter assisted pullups. Select pullups...   \n",
       "3                   Mirin 3% bodyfat? Yeah, you are.   \n",
       "4  Hey! I just started a new job, so things are s...   \n",
       "\n",
       "                                     response_tokens  response_length  \\\n",
       "0                   [One, day, at, a, time, !, =, ]]                8   \n",
       "1  [Crazy, is, synonymous, with, awesome, in, thi...                9   \n",
       "2  [You, can, enter, assisted, pullups, ., Select...               57   \n",
       "3    [Mirin, 3, %, bodyfat, ?, Yeah, ,, you, are, .]               10   \n",
       "4  [Hey, !, I, just, started, a, new, job, ,, so,...               26   \n",
       "\n",
       "   response_avg_slen     source gender_info  \n",
       "0           4.000000  fitocracy          WM  \n",
       "1           9.000000  fitocracy          WW  \n",
       "2          11.400000  fitocracy          WM  \n",
       "3           5.000000  fitocracy          MW  \n",
       "4           8.666667  fitocracy          MM  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's combine op_gender and responder_gender into one column, because this is what we want to predict\n",
    "# original poster's gender is first character, responder's gender is second character\n",
    "responder_df['gender_info'] = responder_df.op_gender + responder_df.responder_gender\n",
    "responder_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.7/site-packages/pandas/core/frame.py:3697: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  errors=errors)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_gender</th>\n",
       "      <th>response_text</th>\n",
       "      <th>response_tokens</th>\n",
       "      <th>response_length</th>\n",
       "      <th>response_avg_slen</th>\n",
       "      <th>source</th>\n",
       "      <th>gender_info</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>W</td>\n",
       "      <td>One day at a time! =]</td>\n",
       "      <td>[One, day, at, a, time, !, =, ]]</td>\n",
       "      <td>8</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>WM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>W</td>\n",
       "      <td>Crazy is synonymous with awesome in this case.</td>\n",
       "      <td>[Crazy, is, synonymous, with, awesome, in, thi...</td>\n",
       "      <td>9</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>WW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>W</td>\n",
       "      <td>You can enter assisted pullups. Select pullups...</td>\n",
       "      <td>[You, can, enter, assisted, pullups, ., Select...</td>\n",
       "      <td>57</td>\n",
       "      <td>11.400000</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>WM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>M</td>\n",
       "      <td>Mirin 3% bodyfat? Yeah, you are.</td>\n",
       "      <td>[Mirin, 3, %, bodyfat, ?, Yeah, ,, you, are, .]</td>\n",
       "      <td>10</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>MW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>M</td>\n",
       "      <td>Hey! I just started a new job, so things are s...</td>\n",
       "      <td>[Hey, !, I, just, started, a, new, job, ,, so,...</td>\n",
       "      <td>26</td>\n",
       "      <td>8.666667</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>MM</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  op_gender                                      response_text  \\\n",
       "0         W                              One day at a time! =]   \n",
       "1         W     Crazy is synonymous with awesome in this case.   \n",
       "2         W  You can enter assisted pullups. Select pullups...   \n",
       "3         M                   Mirin 3% bodyfat? Yeah, you are.   \n",
       "4         M  Hey! I just started a new job, so things are s...   \n",
       "\n",
       "                                     response_tokens  response_length  \\\n",
       "0                   [One, day, at, a, time, !, =, ]]                8   \n",
       "1  [Crazy, is, synonymous, with, awesome, in, thi...                9   \n",
       "2  [You, can, enter, assisted, pullups, ., Select...               57   \n",
       "3    [Mirin, 3, %, bodyfat, ?, Yeah, ,, you, are, .]               10   \n",
       "4  [Hey, !, I, just, started, a, new, job, ,, so,...               26   \n",
       "\n",
       "   response_avg_slen     source gender_info  \n",
       "0           4.000000  fitocracy          WM  \n",
       "1           9.000000  fitocracy          WW  \n",
       "2          11.400000  fitocracy          WM  \n",
       "3           5.000000  fitocracy          MW  \n",
       "4           8.666667  fitocracy          MM  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# drop the columns (keep op_gender in there for later)\n",
    "responder_df.drop(labels=['responder_gender'], axis=1, inplace=True)\n",
    "responder_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predict gender of poster/responder\n",
    "Let's start simple! Given our `all_posts_responses` dataframe, can we predict the gender of the poster/responder?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>text</th>\n",
       "      <th>text_length</th>\n",
       "      <th>avg_slen</th>\n",
       "      <th>hedges</th>\n",
       "      <th>questions</th>\n",
       "      <th>source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>M</td>\n",
       "      <td>President Trump has hit the ground running. As...</td>\n",
       "      <td>27</td>\n",
       "      <td>13.500000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>fb_congress_post</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>M</td>\n",
       "      <td>It's time to put Obama in the history books. W...</td>\n",
       "      <td>37</td>\n",
       "      <td>18.500000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>fb_congress_post</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>M</td>\n",
       "      <td>Since its inception, Obamacare has been an abs...</td>\n",
       "      <td>28</td>\n",
       "      <td>9.333333</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>fb_congress_post</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>M</td>\n",
       "      <td>It's time. Like if you agree.</td>\n",
       "      <td>9</td>\n",
       "      <td>4.500000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>fb_congress_post</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>M</td>\n",
       "      <td>In 15 days we will restore conservative values...</td>\n",
       "      <td>17</td>\n",
       "      <td>8.500000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>fb_congress_post</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  gender                                               text  text_length  \\\n",
       "0      M  President Trump has hit the ground running. As...           27   \n",
       "1      M  It's time to put Obama in the history books. W...           37   \n",
       "2      M  Since its inception, Obamacare has been an abs...           28   \n",
       "3      M                      It's time. Like if you agree.            9   \n",
       "4      M  In 15 days we will restore conservative values...           17   \n",
       "\n",
       "    avg_slen  hedges  questions            source  \n",
       "0  13.500000       0          0  fb_congress_post  \n",
       "1  18.500000       0          0  fb_congress_post  \n",
       "2   9.333333       0          0  fb_congress_post  \n",
       "3   4.500000       0          0  fb_congress_post  \n",
       "4   8.500000       0          0  fb_congress_post  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_posts_responses.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Welcome! Fitocracy is a great place to track your workouts and keep consistent, with a very supportive community. To help you find your way around:                Getting Started Guides:        On the web - http://bit.ly/I5PoXP        On iPhone - http://bit.ly/N2IXHE                If you need help finding an exercise name, have questions about training, etc--leave me a message on my profile or ask the Welcome to Fitocracy group:        http://ftcy.co/UpSig0        Glad to know ya ;)    4416\n",
       "just joined the WTF - Welcome to Fitocracy group!                                                                                                                                                                                                                                                                                                                                                                                                                                                           3601\n",
       "Hello and welcome! Fitocracy is a great place to track your workouts and keep consistent, with a very supportive community.        Start by earning points for any exercise...even walking up the stairs. To start:                1. Tap on the \"track\" icon on the bottom center of your screen.        2. Tap on \"start a workout\"        3. Tap on \"Add an exercise\"        If you need help finding an exercise, have questions about training, etc--drop me a line ;)                                 1518\n",
       "Thanks for the follow!                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       815\n",
       "just joined the No Barbie Weights Here group!                                                                                                                                                                                                                                                                                                                                                                                                                                                                654\n",
       "Name: text, dtype: int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_posts_responses.text.value_counts().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Something to note... it seems like a lot of these messages are auto-generated when joining some group, or there's a lot of duplicates because it's something simple like \"thanks for the follow!\" Let's drop all duplicates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_posts_responses.drop_duplicates(subset='text', keep=False, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "So, do you enjoy having the occasional drink now? And, I totally understand why itd not be fun.                  1\n",
       "I just got dumped this morning, I hope it's okay if I skipped SS Set A and get face smashed drunk instead. :/    1\n",
       "im gonna win in the sitting competition                                                                          1\n",
       "Hey thanks for the follow-back and props. Love the tiara :)                                                      1\n",
       "How do I get my points up on our group?                                                                          1\n",
       "Name: text, dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make sure\n",
    "all_posts_responses.text.value_counts().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Good - no more duplicates!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "M    0.634469\n",
       "W    0.365531\n",
       "Name: gender, dtype: float64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_posts_responses.gender.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So our baseline is 63% here.\n",
    "\n",
    "### Using Naive-Bayes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "138004\n",
      "34501\n"
     ]
    }
   ],
   "source": [
    "X = list(all_posts_responses.text)\n",
    "y = list(all_posts_responses.gender)\n",
    "\n",
    "# create train test split\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\n",
    "\n",
    "print(len(y_train))\n",
    "print(len(y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fit to model\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.decomposition import TruncatedSVD\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "# include punctuation in tokenization\n",
    "import nltk\n",
    "model = make_pipeline(TfidfVectorizer(max_df=0.5, min_df=5, ngram_range=(1,2), tokenizer=nltk.word_tokenize),\n",
    "                      MultinomialNB())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fit and predict\n",
    "model.fit(X_train, y_train)\n",
    "y_pred = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAEKCAYAAABwhMxUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmUFNX5xvHvM4OACsimuACCiqIiuIvGBTVRXBIwRiOaSFyCiVtc45YoakxiYkzUKIYoLokKxuUnKkpQcYmKW3DBIAGXCCIgIAiiwsD7+6NqcIDpmZ5mema6+/mcU2e6blV1vXVg3rm37q1bigjMzGxNZY0dgJlZU+UEaWaWgROkmVkGTpBmZhk4QZqZZeAEaWaWgROkmVkGTpBmZhk4QZqZZdCssQPI5Cdq40d8CsiNj1zT2CFYDsoPG6K1Ob4uv6c3x2drda7G4BqkmVkGTbYGaWZNX7HXsJwgzSxnzVRwreY6cYI0s5yVFXd+dII0s9y5iW1mlkGZm9hmZtVzDdLMLAPfgzQzy6DcTWwzs+q5iW1mloGb2GZmGbgGaWaWQbEP8yn2PwBmlkfNlP1SG0kjJM2RNKlK2Y6SJkh6XdKrknZPyyXpeknTJL0paecqxwyWNDVdBlcp30XSW+kx10u1Z3cnSDPLWVkdlizcDvRfrex3wOURsSNwaboOcAjQI12GAMMAJLUHLgP2AHYHLpPULj1mWLpv5XGrn6va6zMzy0kZynqpTUQ8C8xfvRhok37eAJiZfh4A3BmJCUBbSZsABwPjImJ+RHwKjAP6p9vaRMSLERHAncDA2mLyPUgzy1lderElDSGpwVUaHhHDaznsLGCspGtIKnR7peWbAdOr7DcjLaupfEY15TVygjSznNWlCZomw9oS4up+CpwdEfdLOhq4FfgmVFsljRzKa+QmtpnlrEzZLzkaDDyQfv4HyX1FSGqAXars15mk+V1TeedqymvkBGlmOWsmZb3kaCawX/r5AGBq+nk0cHzam90XWBgRHwNjgYMktUs7Zw4CxqbbFknqm/ZeHw88VOv15Rq1mVl91rAk3QP0AzpKmkHSG/1j4DpJzYAv+foe5hjgUGAasAQ4ASAi5ku6Engl3e+KiKjs+PkpSU/5usBj6VIjJ0gzy1l9PmoYEYMybNqlmn0DOC3D94wARlRT/irQqy4xOUGaWc6yGb5TyJwgzSxnnqzCzCyDcidIM7PquYltZpaBm9hmZhkU+0BqJ0gzy1mRVyCdIM0sd8U+Ya4TpJnlzE1sM7MMirv+6ARpZmshi7cWFDQnSDPLWXGnRydIM1sLvgdpZpZBkbewnSDNLHd+1NDMLIPiTo9OkGa2FvwstplZBiryOqQTpJnlrLjToxOkma0FN7HNzDJwL7aZWQbFnR6dIM1sLXiguJlZBkWeH50gzSx3HuZjZpaBX/tqZpZBkedHJ8hc/fDWG9nh8P4smvMJV+7QF4DNevfiuJv/RItW6zPvgw8ZcdzJfLloEdt+c38G/nYozZo3p2LpUh44/5dMGf8sALsc/V0OueQ8ysrLmfToWB644FIAjrr2N2y9/z4ANF9vPVpv1JFz2nVtnIstMu/Pmc85dz6ycn3GvIWc0X8vZi9czNP/eZd1ysvp0qEtVw06mDbrtgRg+BMvcf9LkygvExcfcQB79+zGV8sqOP7Po1hasZyKFSs4qE8Pzuj/jca6rEZR7E1sRURjx1Ctn6hN0wwstdU+e/HV4s/50Z1/WZkgL3z5ae4/7xKmPvs8e53wAzp078bDl/6KLjv25rPZc1j48Sw23X5bzhz7IBd27sn67dtzycTn+PUu+7J47jwG334zE+68hylPPbPKufqdfgpddurN3046rRGuNDs3PnJNY4eQk+UrVtDv8r8w8mfH8cEn89ljq640Ky/jDw8nf8DO/fa+TJs1j/P/9iijzj6WOQs/56Sb/8GYi06kTGLJ0mWs36I5y5Yv5wc3jOTigfvTp9umjXxV2Ss/bMhaZbgXNu6S9e/pXrOmF1w2Lfb5LvNm2nMvsGT+p6uUddpmK6Y++zwAk8eNZ+cjvwPA9NffZOHHswCY+fZkmrVsSbPmzem4RTdm/3cai+fOA+CdJ55m5yMHrHGu3QZ9j1fvuS+fl1OyJkz9kK4d2rJZ+zZ8Y5tuNCtPfiX6bL4JsxYuAuCpSdM4ZKdtaN6sGZ07bEDXjm1568NZSGL9Fs0BqFi+gorlK4p/3MtqyuqwFKJCjbtJmjlpMn2+cygAOx81kHZdNltjn52PHMD0iW9QsXQpn0x7j417bk2HzbtSVl5On4GHrXFM+65d6Nh9c95ZrVZp9WPMxHc4dKeea5Q/8PIk9unZHYA5CxezcdvWK7d12qA1sxcuBpIa6BHX3Mnelw5jr603p8/mmzRM4E2E6rDU+l3SCElzJE2qUjZU0keSXk+XQ6tsu0jSNElTJB1cpbx/WjZN0oVVyrtLeknSVEmjJDWvLaa8JEhJo2ta8nHOpuDOE09lv9OGcNGrz9CydWsqli5bZfsm2/XkiKuv4K5TzgJgyYIF3P3Tszl51O2c99xY5n3wISsqKlY5ZtdjjuTf9z1ErFjRYNdRKpZWLGf82+9y8I5br1J+87gJlJeV8e1dtgUgWLMVWfkLX15WxoPnHc/4y4bw1oezmPrx3HyH3aSUSVkvWbgd6F9N+R8jYsd0GQMgaTvgGGD79JibJJVLKgduBA4BtgMGpfsCXJ1+Vw/gU+Ck2gLKVyfNnsB04B7gJbLs7JI0BBgCsA8t2I5aE3yTMnvKVK4/eCAAG/XYih0OW/lHjbabbcpPHryb248fwtz33l9Z/tYjj/PWI48DsPePf0QsX77Kd+56zJGMPO3cBoi+9Dz3zvtst1knOrZef2XZ/73yNs/85z1G/PSolW/s67RBa2YtWLRyn9kLF7HRBq1W+a4267Zkt60689w779Njk44NcwFNQH3eUIiIZyV1y3L3AcDIiPgKeF/SNGD3dNu0iHgPQNJIYICkycABwLHpPncAQ4FhNZ0kX03sjYGLgV7AdcC3gLkR8UxEZGwrRsTwiNg1InYttOQI0HrD5BdDEof+4nyevflWANbdYANOf/Qf/N9FQ3n3hZeqPWa9tm3Z79ST+dctd67c1mnrrVi/XVvee/HlBrqC0jLm3+9w6M5fN6+fm/w+tzz1MjeeNJB1m6+zsnz/Xlvy2MQpLK2oYMa8hfzvkwXs0HVj5i9ewmdffAnAl0uX8eJ/P2SLjdo3+HU0JklZL2vhdElvpk3wdmnZZiSVsEoz0rJM5R2ABRFRsVp5jfJSg4yI5cDjwOOSWgCDgKclXRERN+TjnA3tpLtHsHW/vWnVsQO/mT6Zhy/7NS1btWK/034MwMQHRvPCbX8HoN/pQ9hwqy049Jc/59Bf/hyA6w8ayKJP5nL0db+jc59eADx6xdXMmTpt5Tl2G3QUr4y8v4GvrDR8sXQZL/z3fww96lsry371wFMsW17BSTcnHWJ9Nt+EoUd9ix4bd+TgHbfm21ffTnlZGb848kDKy8r45LPPueiex1ixIlgRQf8+29Bv+y0b65IaRV2mO6vaQkwNj4jhtRw2DLgSiPTnH4ATqb7yGlRf6Ysa9q855nwN80kT42EkybEbMBoYEREfZXN8Ux/mY6sq1GE+pW5th/m83qVb1r+nO07/oNZzpU3sRyKiV03bJF0EEBG/SbeNJWkyAwyNiIPT8ovSst8CnwAbR0SFpD2r7pdJXmqQku4gaV4/BlweEZNqOcTMClBZnsfBSNokIj5OV48AKnPJaOBuSdcCmwI9gJdJaoo9JHUHPiLpyDk2IkLSeOB7wEhgMPBQbefPVyfND4HPga2BM6vcfxAQEdEmT+c1swa0lvcWV/+ue4B+QEdJM4DLgH6SdiRpDn8AnAIQEW9Luhf4D1ABnJbe2kPS6cBYoJyk1fp2eooLgJGSfgVMBG6tLaZ83YP0+EqzElCf4+IjYlA1xRmTWERcBVxVTfkYYEw15e/xdU93VvwstpnlrD5rkE2RE6SZ5azI86MTpJnlLssnZAqWE6SZ5aysyN/76gRpZjlTkXfHOkGaWc7cSWNmlkGR50cnSDPLnWuQZmYZFHl+dII0s9yVuxfbzKx6JdvEllTjzJ8RMb/+wzGzQlLk+bHGGuRr1DzR5BZ5icjMCkbJJsiI6N6QgZhZ4VGp34NUcpPhOKB7RFwpqSvJrLx+UYpZiSv2TppsHhS6ieQthZVvA1tE8lpFMytxUvZLIcqmF3uPiNhZ0kSAiPg0mxdum1nxK9le7CqWpS/jDgBJGwJ+i72ZFWzNMFvZJMjrgQeBTpKuInnpzS/yGpWZFYSSr0FGxF2SXgMOTIsGRsTk/IZlZoWgyPNj1k/SrEfyhrAA1s1fOGZWSMrKiztD1tqLLelS4A6gPdARuE2Sm9hmhqSsl0KUTQ1yELBTRHwJIOm3wL+BX+UzMDMrAEU+DjKbBPkB0BL4Ml1vAbybr4DMrIAUaM0wWzVNVnEDyT3Hr4C3JY1L178F/KthwjOzpqxQm87ZqqkG+Wr68zWSYT6Vns5bNGZWWMqL+61dNU1WcUdDBmJmhceTVUg9gN8A25HciwQgIjzdmVmpK/Imdjb149uAYUAFsD9wJ/C3fAZlZoVBZcp6KUTZJMh1I+JJQBHxv4gYChyQ37DMrCAU+XQ+2STILyWVAVMlnS7pCGCjPMdlZoWgTNkvtZA0QtIcSZOqlP1e0juS3pT0oKS2VbZdJGmapCmSDq5S3j8tmybpwirl3SW9JGmqpFHZzEqWTYI8i+RRwzOBXYAfAoOzOM7MipzKy7JesnA70H+1snFAr4joDfwXuAhA0nbAMcD26TE3SSpPZx67ETiEpN9kULovwNXAHyOiB/ApcFJtAdUadUS8EhGLI2JGRJwQEd+NiAm1X6uZFb16bGJHxLPA/NXK/hkRFenqBKBz+nkAMDIivoqI94FpwO7pMi0i3ouIpcBIYED6ZoQDgPvS4+8ABtYWU00DxR8mnQMyw8V8p7YvN7PipoYdBnkiMCr9vBlJwqw0Iy0DmL5a+R5AB2BBlWRbdf+Mahrmc00WAZtZKatD54ukIcCQKkXDI2J4lsdeQjKS5q7Komp2C6pvFdf0dtYa1TRQ/JnaDjaz0laX4TtpMswqIa5yDmkwcDhwYERUJrUZQJcqu3UGZqafqyufC7SV1CytRVbdP6Pifk7IzPIrz8N8JPUHLgC+ExFLqmwaDRwjqYWk7kAP4GXgFaBH2mPdnKQjZ3SaWMeTvBEBko7mh2o7f7YT5pqZrSHL3unsvku6B+gHdJQ0A7iMpNe6BTAunRhjQkT8JCLelnQv8B+SpvdpEbE8/Z7TgbEkk3yPiIi301NcAIyU9CtgInBrbTE5QZpZ7urxCZmIGFRNccYkFhFXAVdVUz4GGFNN+XskvdxZcy+2meWuQJ+QyVY2vdjfBTYG/p6uDyKZRNfMSlzJzgdZ2Yst6cqI2LfKpoclPZv3yMys6SvQSSiylc09yA0lbZG230l7jDbMb1hmVgjqs5OmKcomQZ4NPC3pvXS9G3BK3iIys8JRqk3sShHxeDppbs+06J2I+Cq/YZlZISjUeR6zlc17sdcDzgdOj4g3gK6SDs97ZGbW9Hk+SG4DlgJ7pusz8DuxzQzqdT7Ipiibe5BbRsT3JQ0CiIgv1AB9+zd/Pr32nazJiOmTGzsEawQlO8yniqWS1iUdNC5pS5J3ZZtZqXMvNkOBx4Euku4CvgGckM+gzKxAlHoNMiL+Kek1oC/JnGo/i4i5eY/MzJq+Uk+Qkp6MiAOBR6spM7NSVlaiTWxJLUle1tVRUju+npG3DbBpA8RmZk1dCdcgTyF5o+GmwGt8nSA/I3lrmJmVulJNkBFxHXCdpDMi4oYGjMnMCkV5eWNHkFfZ3EBYsdrLuttJOjWPMZlZofCTNPw4IhZUrkTEp8CP8xeSmRWMIk+Q2YyDLJOkyreJSSoHmuc3LDMrCAWa+LKVTYIcC9wr6WaSp2l+QjJw3MxKXakO86niApIe7Z+S9GT/E7gln0GZWYEo9QQZESuAYeliZva1Um1iS7o3Io6W9BbVvN0wInrnNTIza/JUwjXIn6U/PTmumVWvVGuQEfFx+vN/DReOmRWUUk2QkhZRTdO6UkS0yUtEZlY4SjVBRkRrAElXALOAv5H0Yh8HtG6Q6MysaSvyRw2zGeZzcETsUWV9mKSXgN/lKSYzKxRFXoPMpgtquaTjJJVLKpN0HLA834GZWQEo8kcNs0mQxwJHA7PT5ai0zMxKXVlZ9ksBymag+AfAgPyHYmYFp0BrhtmqNa1L2lrSk5Impeu9Jf0i/6GZWZNXz01sST+TNEnS25LOSsvaSxonaWr6s11aLknXS5om6U1JO1f5nsHp/lMlDc718rKp9/4VuAhYBhARbwLH5HpCMysi5eXZL7WQ1ItkKsXdgT7A4ZJ6ABcCT0ZED+DJdB3gEKBHugwhfRxaUnvgMmCP9Lsuq0yqdZVNglwvIl5erawil5OZWZGp3xrktsCEiFgSERXAM8ARJLf47kj3uQMYmH4eANwZiQlAW0mbAAcD4yJifjp/7Tigfy6Xl02CnCtpS9JB45K+B3ycy8nMrMjUIUFKGiLp1SrLkNW+bRKwr6QOktYDDgW6AJ2qPNn3MbBRuv9mwPQqx89IyzKV11k24yBPA4YDPSV9BLxPMljczEpdHXqnI2I4SS7JtH2ypKtJanyLgTeoubVaXbU0aiivsxqvTlIZsGtEfBPYEOgZEXv7+WwzA+q9kyYibo2InSNiX2A+MBWYnTadSX/OSXefQVLDrNQZmFlDeZ3VmCDTuSBPTz9/HhGLcjmJmRWpsvLslyxI2ij92RX4LnAPMBqo7IkeDDyUfh4NHJ/2ZvcFFqZN8LHAQekLBtsBB6VldZZNE3ucpPOAUcDnlYURMT+XE5pZESmr93GQ90vqQDJq5rSI+FTSb0le+3IS8CHJwyoAY0juU04DlgAnQJKbJF0JvJLud0Wu+SqbBHli+vO0KmUBbJHLCc2siKh+n5CJiH2qKZsHHFhNebBqXqq6bQQwYm3jyeZJmu5rexIzK1JF/iRNrQlSUkvgVGBvkprjc8DNEfFlnmMzs6auQJ+xzlY2Tew7gUXADen6IJK5IY/KeISZlYZSr0EC20REnyrr4yW9ka+AzKyAZNk7XaiyqR9PTLvQAZC0B/B8/kIys4JR6tOdkTzwfbykD9P1rsDkytfB+vWvZiXMTezcHvI2sxJQz8N8mppshvn4sUIzq179DxRvUrKpQZqZVa/IO2mcIM0sd6XexDYzy8hNbDOzDNyLbWaWgZvYZmYZuIltZpaBe7HNzDJwE9vMLAM3sc3MMnAN0mrz8azZ/PyXQ5k7bx5lEkcfeQSDjz2Gsy64mPc/SJ7UXLRoMa1bt+KhUXfx/ISX+MP1N7Js2TLWWWcdzj/rDPbcfTcAxowdx7Bbb2PF8uXst883+PlZZzbmpRWtA04+l/XXbUl5WRnl5WXcf+3l/O62kYx/+XXWaVZO10024tdnnkybVuvz8NMvcOuDj608dsoH03ngj5fTbbONOevqG/nw4zmUl4n9d9+Jcwcf3YhX1QiKfJiPktc6NEFLFjbRwNY055O5fDJ3Lttv25PFn3/Okccez43X/p6ttvz6tT2//cOfaNWqFaefcjL/eWcKHdq3p9NGG/Lfae9y0qln8tw/H+XTBQs4YtAPeeCuO2nfvh0X/HIoAw8/lD332L0Rry47MX1yY4dQJwecfC73XzuUdm1aryz718S36Nt7O5qVl3PN7aMAOO9H31/luCkfTOe0q67jib9ewxdffcUbU96jb+9tWbqsghN+eTWnHHU4++7Sh0KhbfquVYZb/tgtWf+elh9ycsFl07zUjyWdJWk3SSVRQ91ow45sv21PAFqtvz5bdO/O7E8+Wbk9Inhs3BMc3v8gALbruQ2dNtoQgB5bbsHSpV+xdOlSpn80k25du9K+fTsA9txjd8Y+Ob6Br6Z07b3TDjQrT3pl+2yzJbPmfbrGPo8+O4HD9k2mR123RQv69t4WgObrNGO7LTdn1tw1jylq9fza16YmXzcQOgPXAXMkPS3p15IOk9Q+T+drMmbMnMnkKVPo02v7lWWv/nsiHdq3p9vmXdfYf+wTT7HtNtvQvHlzNu/Smfc++B8zZs6koqKCJ8c/w6zZsxsy/JIh4KRLf893z76UUY+v+Ufo/ieeY9+dd1ij/LF/vbQyQVb12eLPGf/y6+zZZ7t8hNt0SdkvBSgvNbyIOA9AUnNgV2AvktfH/lXSgoio9n+RpCHAEIC/3PAnhpz4o3yElzefL1nCmeddyMXnnUOrVq1Wlj/y+D85vP/Ba+w/9d13ueb6PzPipuR1Pxu0acPQiy/g7AsuoUxipz69mf7RRw0Wfym5++pf0KlDO+Yt+IwTL/0dW3TehN16Ja2Am+8dTbPyMr7db69Vjnljyru0bNGCrTfvvEp5xfLlnHvNMH54+LfosvFGDXYNTUKBzhSerXw3gdcF2gAbpMtM4K1MO0fEcGA4UFD3IAGWLavgzPMu4NuHHMxBB+6/sryiooJxTz3NA3ffscr+s2bP5vRzfs7VVw6la5evf+EO2G8fDtgveTXwqPsfpKy8uP8DNpZOHZLbGB3atuGbfXfhzanvsVuvnjz45L8Y/8rr3P6rC9BqtZ4xz03gsH3WrD1e+ufb2HzTjRk8YM0/gkWvQGuG2crXPcjhkp4HRgF7Ai8AR0XErhFxQj7O2Zgigksuv5ItunfnhB8et8q2F156hS26bc7GnTqtLPts0SKGnHE255xxGrvsuOoN/Xnz5wOw8LPPuPve+zjqiAH5v4ASs+TLr1i85IuVn59/fRJbd+3Mc6+9yS0PPMqwX5zFui1arHLMihUrePz5Vzhs3z1WKf/T3+9j0ZIvuPjkYxss/iZFZdkvBShfNciuQAtgKvARMANYkKdzNbrXXn+Dhx59jK17bMWA7ycJ8pzTT2W/fb7BmLH/5LC0c6bS30fey4fTZ3DTX2/lpr/eCsCIYTfQoX17rvrdtbzz36kAnDbkJLpvvnnDXkwJmLdgIaf/+noAli9fzuH77ck+u/TmoCHns7SighMv/T2QdNRcfuqPAHjl7Sls3KH9Kk3oWXPnc/O9D7NF50347tmXAXDcYQdy1EH9GvR6GlWBdr5kK2/DfJS0T7Ynuf+4F9ALmA+8GBGX1foFBdbELnWFNszHEms9zOfZUdkP89n3+wXXHs/bPchIMu8kSQuAhelyOLA7UHuCNLOmr0CbztnKS4KUdCZJrfEbwDKS92i/CIyghk4aMyswRd5Jk68aZDfgPuDsiPg4T+cws8ZW5DXIvFxdRJwTEfc5OZoVN0lZL1l+X1tJ90l6R9JkSXtKai9pnKSp6c926b6SdL2kaZLelLRzle8ZnO4/VdLgXK+vuNO/meVXWbPsl+xcBzweET2BPsBk4ELgyYjoATyZrgMcAvRIlyHAMID0ib3LgD1I+zwqk2qdLy+Xg8zMgGQ+yGyXWkhqA+wL3AoQEUsjYgEwAKh80uIOYGD6eQBwZyQmAG0lbQIcDIyLiPkR8SkwDuif0+XlcpCZGVDfA8W3AD4BbpM0UdItktYHOlXerkt/Vg5G3QyYXuX4GWlZpvI6c4I0s9zVYbIKSUMkvVplGbLatzUDdgaGRcROwOd83Zyu9uzVlEUN5XVWEtORmVme1KEXe5W5Fqo3A5gRES+l6/eRJMjZkjaJiI/TJvScKvt3qXJ8Z5L5HmYA/VYrfzrrQKtwDdLMcleP051FxCxguqRt0qIDgf8Ao4HKnujBwEPp59HA8Wlvdl9gYdoEHwscJKld2jlzUFpWZ65Bmlnuyuv9WewzgLvSqRLfA04gqcjdK+kk4EPgqHTfMcChwDRgSbovETFf0pXAK+l+V0TE/FyC8SsXrF74WezCtLbPYq94c3zWv6dlvfcvuMduXIM0s9z5UUMzswyK/FFDJ0gzy51rkGZmGZQXdwop7qszs7zKdhKKQuUEaWa58z1IM7MMXIM0M8vANUgzswxcgzQzy6D+HzVsUpwgzSx3bmKbmWXgJraZWSZOkGZm1XMN0swsAydIM7MM3EljZpZBcVcgnSDNbG0Ud4Z0gjSz3PkepJlZBk6QZmYZuJPGzCwT1yDNzKrnJraZWQZOkGZmmThBmplVyy/tMjPLxL3YZmYZuAZpZpaBE6SZWSZOkGZm1SvyGmRx32E1s/xSHZbavkpqKellSW9IelvS5Wl5d0kvSZoqaZSk5ml5i3R9Wrq9W5XvuigtnyLp4FwvzwnSzHKnsuyX2n0FHBARfYAdgf6S+gJXA3+MiB7Ap8BJ6f4nAZ9GxFbAH9P9kLQdcAywPdAfuElSTu+ndYI0s9xJ2S+1iMTidHWddAngAOC+tPwOYGD6eUC6Trr9QCUDMwcAIyPiq4h4H5gG7J7L5TlBmtlayL6NLWmIpFerLEPW+DapXNLrwBxgHPAusCAiKtJdZgCbpZ83A6YDpNsXAh2qlldzTJ24k8bMcleHTpqIGA4Mr2Wf5cCOktoCDwLbVrdb5dkzbMtUXmeuQZpZ7uqxiV1VRCwAngb6Am0lVVbmOgMz088zgC5JGGoGbADMr1pezTF14gRpZrmrx04aSRumNUckrQt8E5gMjAe+l+42GHgo/Tw6XSfd/lRERFp+TNrL3R3oAbyc0+Ul32cNSdKQtLlhBcD/Xg1DUm+STpdyksrbvRFxhaQtgJFAe2Ai8IOI+EpSS+BvwE4kNcdjIuK99LsuAU4EKoCzIuKxnGJygmx4kl6NiF0bOw7Ljv+9Speb2GZmGThBmpll4ATZOHw/q7D436tE+R6kmVkGrkGamWXgBNkAJIWkv1VZbybpE0mPNGZcVj1Jf5R0VpX1sZJuqbL+B0nnNE501pCcIBvG50CvdPArwLeAjxoxHqvZC8BeAJLKgI4kM8NU2gt4vhHisgbmBNlwHgMOSz8PAu5pxFisZs+TJkiSxDgJWCSpnaQWJM8HT2ys4KzhOEE2nJEkjz+1BHoDLzVyPJZBRMwEKiR1JUmUL5L8e+0J7Aq8GRFLGzFEayCezaeBRMSb6YzHg4AxjRvvAXEzAAACoUlEQVSNZaGyFrkXcC3JdFl7kUyp9UIjxmUNyDXIhjUauAY3rwtB5X3IHUia2BNIapC+/1hCnCAb1gjgioh4q7EDsVo9DxwOzI+I5RExH2hLkiRfbNTIrME4QTagiJgREdc1dhyWlbdIeq8nrFa2MCLmNk5I1tD8JI2ZWQauQZqZZeAEaWaWgROkmVkGTpBmZhk4QZqZZeAEWUIktZV0ah6//0eS/lzLPkMlnVfH7128dpGZ5cYJsrS0BapNkJLKGzgWsybPCbK0/BbYUtLrkn4vqZ+k8ZLuBt6S1E3SpMqdJZ0naWj6eUtJj0t6TdJzknrWdCJJ35b0kqSJkp6Q1KnK5j6SnpI0VdKPqxxzvqRXJL0p6fL6vXSzuvNkFaXlQqBXROwIIKkfsHta9n46mUYmw4GfRMRUSXsANwEH1LD/v4C+ERGSTgZ+DpybbusN9AXWByZKehToRfKC990BAaMl7RsRz+Z0pWb1wAnSXo6I92vaQVIrkkka/iGpsrhFLd/bGRglaROgOVD1HA9FxBfAF5LGkyTFvYGD+HqexVYkCdMJ0hqNE6R9XuVzBavedmmZ/iwDFlTWPLN0A3BtRIxOa6pDq2xb/fnWIKk1/iYi/lKHc5jlle9BlpZFQOsats8GNpLUIZ05+3CAiPgMeF/SUQBK9KnlXBvw9WslBq+2bYCklpI6AP2AV4CxwIlpbRVJm0naKPtLM6t/rkGWkIiYJ+n5tCPmMeDR1bYvk3QFyezZ7wPvVNl8HDBM0i+AdUhmSH+jhtMNJWmSf0QyI073KtteTs/dFbgyncF7pqRtgRfTZvxi4AfAnBwv12yteTYfM7MM3MQ2M8vACdLMLAMnSDOzDJwgzcwycII0M8vACdLMLAMnSDOzDJwgzcwy+H++P8R/VvQdDwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.7121242862525724\n"
     ]
    }
   ],
   "source": [
    "# confusion matrix\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score\n",
    "\n",
    "mat = confusion_matrix(y_test, y_pred)\n",
    "\n",
    "labels = sorted(set(y_test))\n",
    "\n",
    "# plot it\n",
    "sns.heatmap(mat.T, square=True, annot=True, fmt='d', cmap=\"Reds\", xticklabels=labels, yticklabels=labels)\n",
    "plt.xlabel('true label')\n",
    "plt.ylabel('predicted label')\n",
    "\n",
    "# save\n",
    "plt.savefig('images/predict_1gender_cm.png')\n",
    "\n",
    "# show\n",
    "plt.show()\n",
    "\n",
    "print('accuracy:', accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: Including nltk's tokenizer actually helps!\n",
    "\n",
    "Note 2: before dropping duplicates, my accuracy was 70%. Dropping them actually helped a little, which is good!\n",
    "\n",
    "So we get an accuracy of 71%! And I'm okay (ish) with a 71% accuracy when baseline was 63% - we are at least 8% better than guessing, and this is with the simplest algorithm! So what is going on with punctuation? Does one gender use more than the other? Do they use it differently?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'!' classified as W with probabilities: [[0.49185065 0.50814935]]\n",
      "'!!' classified as W with probabilities: [[0.37611985 0.62388015]]\n",
      "'?' classified as M with probabilities: [[0.6797264 0.3202736]]\n",
      "'??' classified as M with probabilities: [[0.55419085 0.44580915]]\n",
      "'...' classified as M with probabilities: [[0.61347976 0.38652024]]\n"
     ]
    }
   ],
   "source": [
    "# examine punctuation\n",
    "print(\"'!' classified as\", model.predict(['!'])[0], 'with probabilities:', model.predict_proba(['!']))\n",
    "print(\"'!!' classified as\", model.predict(['!!'])[0], 'with probabilities:', model.predict_proba(['!!']))\n",
    "print(\"'?' classified as\", model.predict(['?'])[0], 'with probabilities:', model.predict_proba(['?']))\n",
    "print(\"'??' classified as\", model.predict(['??'])[0], 'with probabilities:', model.predict_proba(['??']))\n",
    "print(\"'...' classified as\", model.predict(['...'])[0], 'with probabilities:', model.predict_proba(['...']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'hi' classified as W with probabilities: [[0.43441307 0.56558693]]\n",
      "'hi.' classified as W with probabilities: [[0.45861247 0.54138753]]\n",
      "'hi!' classified as W with probabilities: [[0.28461163 0.71538837]]\n",
      "--------\n",
      "'hello' classified as W with probabilities: [[0.45067681 0.54932319]]\n",
      "'hello.' classified as M with probabilities: [[0.59753053 0.40246947]]\n",
      "'hello!' classified as W with probabilities: [[0.25852562 0.74147438]]\n"
     ]
    }
   ],
   "source": [
    "print(\"'hi' classified as\", model.predict(['hi'])[0], 'with probabilities:', model.predict_proba(['hi']))\n",
    "print(\"'hi.' classified as\", model.predict(['hi.'])[0], 'with probabilities:', model.predict_proba(['hi.']))\n",
    "print(\"'hi!' classified as\", model.predict(['hi!'])[0], 'with probabilities:', model.predict_proba(['hi!']))\n",
    "print('--------')\n",
    "print(\"'hello' classified as\", model.predict(['hello'])[0], 'with probabilities:', model.predict_proba(['hello']))\n",
    "print(\"'hello.' classified as\", model.predict(['hello.'])[0], 'with probabilities:', model.predict_proba(['hello.']))\n",
    "print(\"'hello!' classified as\", model.predict(['hello!'])[0], 'with probabilities:', model.predict_proba(['hello!']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems like female posters/responders may like exclamation points more (especially prominent in saying \"hello!\").\n",
    "\n",
    "Let's try to examine the most informative features!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Important words in M:\n",
      ", \t 2732.133245178166\n",
      "to \t 2300.6123683956034\n",
      "! \t 2286.0752908099303\n",
      "i \t 2063.741711751211\n",
      "a \t 1902.9847520120068\n",
      "and \t 1862.810250795262\n",
      "you \t 1838.9974076494518\n",
      "of \t 1594.8305643917747\n",
      "? \t 1499.5804509005725\n",
      "for \t 1497.0295612173697\n",
      "in \t 1346.8265801783116\n",
      "it \t 1339.201895086304\n",
      "is \t 1318.753473680547\n",
      "that \t 1297.4917023906848\n",
      ": \t 1231.7872669068488\n",
      "on \t 974.4419363331431\n",
      "thanks \t 946.0001536278753\n",
      "this \t 867.5257833048455\n",
      ") \t 841.9990822699219\n",
      "my \t 833.755747564762\n",
      "\n",
      "Important words in W:\n",
      "! \t 2693.4109582484616\n",
      "i \t 1422.364239308098\n",
      ", \t 1330.5693690725939\n",
      ": \t 1297.6021240497912\n",
      "you \t 1273.722223917261\n",
      "to \t 1258.9628002652757\n",
      ") \t 1004.4584497088891\n",
      "! ! \t 1002.1789772219872\n",
      "and \t 995.4762711573597\n",
      "for \t 976.5123780756081\n",
      "a \t 968.021485672453\n",
      "? \t 805.2533498385291\n",
      ": ) \t 794.8870142698498\n",
      "thanks \t 766.2237765659911\n",
      "of \t 751.1488239649971\n",
      "it \t 734.5256155797815\n",
      "my \t 673.8216713551694\n",
      "in \t 657.4193701963476\n",
      "# \t 653.973860541869\n",
      "is \t 589.2986451495458\n"
     ]
    }
   ],
   "source": [
    "# from stack overflow:\n",
    "# https://stackoverflow.com/questions/11116697/how-to-get-most-informative-features-for-scikit-learn-classifiers\n",
    "\n",
    "def important_features(vectorizer, classifier, n):\n",
    "    class_labels = classifier.classes_\n",
    "    feature_names =vectorizer.get_feature_names()\n",
    "    topn_class1 = sorted(zip(classifier.feature_count_[0], feature_names),reverse=True)[:n]\n",
    "    topn_class2 = sorted(zip(classifier.feature_count_[1], feature_names),reverse=True)[:n]\n",
    "    print(\"Important words in\", class_labels[0] + ':')\n",
    "    for coef, feat in topn_class1:\n",
    "        print(feat, '\\t', coef)\n",
    "    print()\n",
    "    print(\"Important words in\", class_labels[1] + ':')\n",
    "    for coef, feat in topn_class2:\n",
    "        print(feat, '\\t', coef) \n",
    "    \n",
    "vzr = model.steps[0][1] \n",
    "clf = model.steps[1][1] \n",
    "\n",
    "important_features(vzr, clf, 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'm not sure how telling this code is, so I'm not going to put too much faith in the results. Also, it doesn't tell us much since I'm also not sure how to modify this code to omit stop words. If we analyze this at face value, it seems like female posters may use more exclamation points (note both single and double exclamation points!). Also, the smiley face :) is present in the important words for female posters and not in male posters. Still interesting to look at!\n",
    "\n",
    "### Try adding other features\n",
    "\n",
    "I'm going to try adding text length and sentence length (and maybe hedges).\n",
    "\n",
    "What I've tried:\n",
    "1. First, I tried to do this with the whole `all_posts_responses` dataframe. But I then realized that adding these features didn't mean much since there was so much discrepency between Fitocracy and Reddit data.\n",
    "2. I then tried to narrow it down to only Reddit data. However, I was still getting a ML accuracy that was lower than my baseline.\n",
    "3. Finally, I tried with just Reddit posts, to try to narrow down discrepencies even more.\n",
    "\n",
    "Unfortunately, the results were still lower than the baseline. I'm going to keep the code in here anyways, so that the reader can understand my thought process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>text</th>\n",
       "      <th>text_length</th>\n",
       "      <th>avg_slen</th>\n",
       "      <th>hedges</th>\n",
       "      <th>questions</th>\n",
       "      <th>source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>146664</th>\n",
       "      <td>M</td>\n",
       "      <td>Glad to see youre playing 07 still!</td>\n",
       "      <td>8</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>reddit_post</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146665</th>\n",
       "      <td>M</td>\n",
       "      <td>I used to go down there all of the time and co...</td>\n",
       "      <td>44</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>reddit_post</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146666</th>\n",
       "      <td>M</td>\n",
       "      <td>Ive heard of people getting 70+ dry streaks.  ...</td>\n",
       "      <td>15</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>reddit_post</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146667</th>\n",
       "      <td>M</td>\n",
       "      <td>Stun alching is faster Magic xp per hour and I...</td>\n",
       "      <td>45</td>\n",
       "      <td>22.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>reddit_post</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146668</th>\n",
       "      <td>M</td>\n",
       "      <td>Calm down child. This conversation went from J...</td>\n",
       "      <td>80</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>reddit_post</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       gender                                               text  text_length  \\\n",
       "146664      M                Glad to see youre playing 07 still!            8   \n",
       "146665      M  I used to go down there all of the time and co...           44   \n",
       "146666      M  Ive heard of people getting 70+ dry streaks.  ...           15   \n",
       "146667      M  Stun alching is faster Magic xp per hour and I...           45   \n",
       "146668      M  Calm down child. This conversation went from J...           80   \n",
       "\n",
       "        avg_slen  hedges  questions       source  \n",
       "146664       8.0       0          0  reddit_post  \n",
       "146665      11.0       0          0  reddit_post  \n",
       "146666       5.0       0          0  reddit_post  \n",
       "146667      22.5       0          0  reddit_post  \n",
       "146668      20.0       0          0  reddit_post  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# just reddit posts\n",
    "reddit = all_posts_responses[all_posts_responses.source == 'reddit_post']\n",
    "reddit.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "M    0.790612\n",
       "W    0.209388\n",
       "Name: gender, dtype: float64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reddit.gender.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Baseline is 79% here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "39692\n",
      "9924\n"
     ]
    }
   ],
   "source": [
    "# train test split\n",
    "\n",
    "X = list(reddit.text)\n",
    "y = list(reddit.gender)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\n",
    "\n",
    "print(len(y_train))\n",
    "print(len(y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# first vectorize text\n",
    "text_vec = TfidfVectorizer(max_df=0.5, min_df=5, ngram_range=(1,2), tokenizer=nltk.word_tokenize)\n",
    "\n",
    "Xtrain_fit_transf = text_vec.fit_transform(X_train)\n",
    "Xtest_fit_transf = text_vec.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(39692, 51609)\n",
      "(9924, 51609)\n"
     ]
    }
   ],
   "source": [
    "print(Xtrain_fit_transf.shape)\n",
    "print(Xtest_fit_transf.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now make into sparse matrix and combine features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.sparse import coo_matrix, hstack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(39692, 7)\n",
      "(9924, 7)\n"
     ]
    }
   ],
   "source": [
    "# need to find the text length and sentence length that correspond to our train test split\n",
    "\n",
    "# training chunk of df\n",
    "train_chunk = reddit.isin(X_train)\n",
    "# getting this chunk from original\n",
    "train_chunk = reddit[train_chunk.text]\n",
    "# print to make sure same size\n",
    "print(train_chunk.shape)\n",
    "\n",
    "# testing chunk of df\n",
    "test_chunk = reddit.isin(X_test)\n",
    "# getting this chunk from original\n",
    "test_chunk = reddit[test_chunk.text]\n",
    "# print to make sure same size\n",
    "print(test_chunk.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(39692, 1)\n",
      "(9924, 1)\n"
     ]
    }
   ],
   "source": [
    "# new features\n",
    "sparse_stats_train = coo_matrix(train_chunk[['text_length']], dtype='float64')\n",
    "sparse_stats_test = coo_matrix(test_chunk[['text_length']], dtype='float64')\n",
    "\n",
    "# making sure we're keeping same sizes\n",
    "print(sparse_stats_train.shape)\n",
    "print(sparse_stats_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(39692, 51610)\n",
      "(9924, 51610)\n"
     ]
    }
   ],
   "source": [
    "# combine features\n",
    "X_train = hstack([Xtrain_fit_transf, sparse_stats_train])\n",
    "X_test = hstack([Xtest_fit_transf, sparse_stats_test])\n",
    "\n",
    "# making sure we're keeping same sizes\n",
    "print(X_train.shape)\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = MultinomialNB()\n",
    "\n",
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUIAAAEKCAYAAABnplydAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAHNhJREFUeJzt3XmUFtWdxvHv082+CbihgAIObhi3g4hLPESjEDVixhU1YiSgiUZjEkedyajRmGgm0VETUTQormicJBI1Ku4bChoVEZOAgoKyRBEFVKCb3/zxVmODvRRNv/12Vz2fc+r0W7fuW3WLPv3j3rq37lVEYGaWZ2WlLoCZWak5EJpZ7jkQmlnuORCaWe45EJpZ7jkQmlnuORCaWe45EJpZ7jkQmlnutSp1AWpzurr4lZcW5LqpE0tdBGuAsr0O1cZ8f0P+Tq+PTzbqWsXkGqGZ5V6zrRGaWfOXlZqUA6GZNVgrNdvW7gZxIDSzBivLRhx0IDSzhnPT2Mxyr8xNYzPLO9cIzSz3/IzQzHKv3E1jM8s7N43NLPfcNDaz3HON0Mxyz8NnzCz3WmUjDjoQmlnDuWlsZrlXRjaqhA6EZtZg7jU2s9xz09jMcs81QjPLPU/Mama556axmeWem8ZmlnsePmNmuecaoZnlXrkDoZnlnZvGZpZ7bhqbWe55+IyZ5V5GKoSZCehmVgJlUuotDUldJd0r6e+S3pS0j6TukiZLmpX87JbklaRrJM2WNF3SntXOMzLJP0vSyHrvo8H/AmaWe2UbsKV0NfBQROwI7Aa8CZwPPBYR/YHHkn2AbwD9k20MMBZAUnfgImBvYBBwUVXwrOs+zMwaRBuw1XsuqQtwAPB7gIhYFRFLgeHAhCTbBODI5PNw4NYoeAHoKmkrYCgwOSKWRMRHwGRgWF3XdiA0swaTlHpLoR/wL+BmSa9IuklSR2DLiFgAkPzcIsnfE5hX7fvzk7Ta0mvlQGhmDbYhNUJJYyS9VG0bs97pWgF7AmMjYg9gBV80g2u7/PqijvRaudfYzBpsQ2pSETEOGFdHlvnA/Ih4Mdm/l0IgXCRpq4hYkDR9F1fL37va93sB7yfpQ9ZLf7KusrlGaGYNJqXf6hMRC4F5knZIkg4CZgKTgKqe35HAfcnnScDJSe/xYODjpOn8MHCIpG5JJ8khSVqtXCM0swYrwit2PwDukNQGeBv4DoUK2z2SRgHvAsckeR8EDgVmA58meYmIJZIuBaYl+S6JiCV1XdSB0MwarLHDYES8Cgys4dBBNeQN4IxazjMeGJ/2ug6EZtZgftfYzHJPGXnJzoHQzBosG2HQgdDMNoKbxmaWe56Y1cxyLxth0IHQzDZCRtZ3dyA0s4bLSBx0IDSzhvPwGTPLPS/naWa5l5E46EDYGLbc/t/47t23rN3frF8f/nLhL3jh1rsYfffNbNpnWz6c+w43HnsKny5dyqATjuWQ834IwMrlK7jze+fw3vQZABx41vfYb/RIJPHsjRN4/OrrSnFLmfdf4+7iyVdn0r1LJ/5y+XkAnHPtBOYuKMzw9Mmnn9GlQ3v+9Itzee71f3Dl3fezuqKS1q3KOXfEEQwe0B+A0VfcwL8+/oSKykoG7tCP/z7laMrL8jOpU1aaxiq8t9z8nK4uzbNg9VBZGZe/9w+u2PtAhpwxmhVLPuLhK65i6Hnn0KFbV/50/kX022cQC9/8J58uXcqAYQdz+MUXcMXgA9l6wE6Mmngzlw/6GpWrVvGDh/7IXd/7EYtnv1Xq26rXdVMnlroIG2Ta39+iQ9s2nH/DnWsDYXVX3HEfnTq044xvDWXm3Plstklntui2Cf+ct4DRv7qBp669GIDln35Opw7tiAjOvuYWhg7ajcP22fNL52uuyvY6dKMi2fM9eqf+O9134bxmGzXz819XE9nxoCF88NYclrw7j12HH8aUCXcCMGXCnex25OEAvD1lKp8uXQrAnBem0a3X1gD02GkH5rwwjdWffcaaykpmPfUcu3/r8NLcSMbtteN2dO3UscZjEcFDL766NqDt3KcXW3TbBID+vXqwcvVqVq2uAKBTh3YAVFSuYXVFRdop6TOjCIs3lURzL1+LM/D4o5h2170AdNlycz5ZuAiATxYuovMWm30p/36jvs2Mv04G4P0ZM+l/wH507N6d1u3bs8uhh9Ctd6+mK7wB8NI/3mbTTTrRp8fmXzr2yLTX2GnbnrRp/cVTpe9ecT37f/+/6diuHUMH7daURS25xly8qZSK8oxQ0qS6jkfEEcW4bqmVt27Nbkccyp8vuDhV/u2HfJV9R53Mr/cfCsDCv/+Th6+4irMn/5mVy1cw/7XXWVNRUcQSW00emPK3Gpu3s+Yv4DcT7+em805fJ/2m805n5arVnDv2dl54Yxb7fWWHL303q9KuV9zcFauzZB8Kq0jdBbxIyv8QksVcxgB8lbbsTJsiFa84dvnGwbz7t9dYtvhfAHyy6F906bElnyxcRJceW7Js8Qdr8/b8ygC+fdNvufYbR7FiyReT5z4//jaeH38bAMMvu5Cl899v2pvIuYrKSh6dNp17L/3xOukLP1zKD/73Zi4//QS22fLLNfu2bVpz4B4DePxvM3IVCLMRBovXNO4B/CewC4UFmw8GPoiIpyLiqdq+FBHjImJgRAxsaUEQYOCIY5h21x/W7k+f9CD7jDwBgH1GnsD0+x4AoFvvXpz2xzu4+dujWTxr9jrn6Lz5Zmvz7PHvR6xtZlvTmDLjn/Tdekt6bNp1bdonKz7j9N/cyI+OPYw9t++3Nn3F5ytZ/NHHQCGAPvXam/TbaosvnTPLGnk5z5IpSo0wIiqBh4CHJLUFRgBPSrokIq4txjVLrXX79ux08Ne447Sz16Y9fPlVjL7nFvYbdTJL3p3HuGMK688cduF5dNy0GyOuuxKANRUV/HKvIQCM+b/b6bRpdypXr+auM368tlPFGtePf3srU9+czdLlKxjyg4s586hhHD1kMA++8AqH7bPHOnnvmPwM7y76gLF/foSxf34EKDSHAzjjyt+zqqKCyjVrGLxzf447aN8S3E3pZGUarqINn0kC4GEUgmAfCitOjY+I99J8v6UOn8mrljZ8xgo2dvjMq737pP473X3e3GYbNovVWTKBQrP4r8DPImJGMa5jZqWVlbHjxeos+TaFVeq3B86q9nxAFBaf6lKk65pZE2ruz/7SKtYzwoz8P2FmdclIHPS7xmbWcK4RmlnuZSQOOhCaWcP5zRIzy72yjAwkdCA0swZTRrpFHQjNrMHcWWJmuZeROOhAaGYN5xqhmeVeRuKgA6GZNVy5e43NLO8y3zSW1L2uL0bEkrqOm1n2ZSQO1lkjfBkIap6NO4B+NaSbWY5kPhBGRN+mLIiZtTzKyzNCFR4CnAj0jYhLJW0D9IiIqUUvnZk1a1npLEnzgsx1FFalOyHZXwb8rmglMrMWQ0q/NWdpeo33jog9Jb0CEBEfSWp5S8yZWaPLfK9xNasllVPoIEHS5sCaopbKzFqEjMTBVE3ja4A/AVtKugx4FvhFUUtlZi1CMdY1llQu6RVJ9yf7t0iaI+nVZNs9SZekayTNljRd0p7VzjFS0qxkG1nfNeutEUbEHZJeBg5Kko6MiDdT35WZZVaRaoRnA28C1Rd5Ozci7l0v3zeA/sm2NzAW2DsZA30RMJBCS/ZlSZMi4qPaLph2NrEOQHmSv33K75hZxpWVK/WWhqReFNZDvylF9uHArVHwAtBV0lbAUGByRCxJgt9kYFid95GiYBcCE4DuwGbAzZJ+mqKQZpZxG9I0ljRG0kvVtjE1nPJ/gf/gy/0QlyXN36sktU3SegLzquWZn6TVll6rNJ0lI4A9IuLz5MYvB/4G/DzFd80syzZgHGFEjAPG1XZc0uHA4oh4WdKQaocuABYCbZLvnwdcQu1vvdWWXqs0TeO5QLtq+22Bt1J8z8yyrnEHEu4HHCFpLjAROFDS7RGxIGn+rgRuBgYl+ecDvat9vxfwfh3ptao1EEq6VtI1wErgjaTn5mZgBrA8zV2ZWbY1Zq9xRFwQEb0iog9wPPB4RJyUPPeresvtSAoxCGAScHLSezwY+DgiFgAPA4dI6iapG3BIklaruprGLyU/X6YwfKbKk/XekZnlQ3mTrN50RzJ+WcCrwOlJ+oPAocBs4FPgO1CYGUvSpcC0JN8l9c2WVdekCxM2ruxmlnXFmnQhIp4kqXRFxIG15AngjFqOjQfGp71emkkX+gO/BHam2rPCiPA0XGZ5l5FXS9LUa2+mMFCxAvgacCtwWzELZWYtg8qUemvO0gTC9hHxGKCIeCciLgZqrKqaWc5kZPqZNOMIP5dUBsySdCbwHrBFcYtlZi1CM6/ppZUmEP6Qwit2ZwGXUqgN1vsSs5lln5qm17jo0ky6UNUFvZyke9rMDGj2Td606lrF7i/U8VpKRBxRlBKZWYuhbFQI66wR/rrJSmFmLVPWa4QR8VRTFsTMWp7mPiwmrTSdJWZmNct6jdDMrD656TU2M6tV1pvG7jU2s3rloGlc1Wv870AP4PZkfwSFyVrNLOcyv65xVa+xpEsj4oBqh/4i6emil8zMmr+sN42r2VxSv4h4G0BSX2Dz4hbLzFqCPHWWnAM8KentZL8PcFrRSmRmLUfWm8ZVIuKhZHLWHZOkvyeLqJhZzmVlQHWadY07AOcCZ0bEa8A2ybJ7ZpZ3GZmPMO0M1auAfZL9+XhNYzODQmdJ2q0ZS/OMcLuIOE7SCICI+ExN0Gd+/Qczi30Ja0ztO5e6BFYCmR8+U80qSe1JBldL2o7CWsdmlnc56jW+GHgI6C3pDgqr0XuCVjNr9s/+0krTa/yIpJeBwRQWWD47Ij4oesnMrPnLSyCU9FhEHAQ8UEOameVZWcabxpLaUVi0aTNJ3SjUBgG6AFs3QdnMrLnLQY3wNAor2G0NvMwXgfAT4HdFLpeZtQRZD4QRcTVwtaQfRMS1TVgmM2spystLXYJGkaaBv0ZS16odSd0kfb+IZTKzliJHb5aMjoilVTsR8REwunhFMrMWIyOBMM04wjJJioiqAdXlQJviFsvMWoRmHuDSShMIHwbukXQ9hbdLTqcwwNrM8i7rw2eqOY9CD/L3KPQcPwLcVMxCmVkLkZdAGBFrgLHJZmb2haw3jSXdExHHSnqdGlazi4hdi1oyM2v2lIMa4dnJT0/CamY1y3qNMCIWJD/fabrimFmLkvVAKGkZdS/w3qUoJTKzliPrgTAiOgNIugRYCNxGodf4RMDTEZtZZl6xSzN8ZmhE7F1tf6ykF4FfFalMZtZSZKRGmKbLp1LSiZLKJZVJOhGoLHbBzKwFaMRX7CS1kzRV0muS3pD0syS9r6QXJc2SdLekNkl622R/dnK8T7VzXZCk/0PS0PqunSYQngAcCyxKtmOSNDPLu7Ky9Fv9VgIHRsRuwO7AMEmDgSuAqyKiP/ARMCrJPwr4KCL+DbgqyYeknYHjgQHAMOC65NXg2m+jvpJFxNyIGB4Rm0XE5hFxZETMTXNXZpZxjVgjjILlyW7rZAvgQODeJH0CcGTyeXiyT3L8oGSFzeHAxIhYGRFzgNnAoLqunWaB9+0lPSZpRrK/q6Sf1ntXZpZ9GxAIJY2R9FK1bcyXT6dySa8Ci4HJwFvA0oioSLLMB3omn3sC8wCS4x8Dm1ZPr+E7NUpTX70RuABYnVxwOoVqp5nlXXl56i0ixkXEwGrbuPVPFxGVEbE70ItCLW6nGq5aNayvpmpm1JFeqzSBsENETF0vraLGnGaWL0WajzCZA/VJCqtndpVUNcKlF/B+8nk+0LtQDLUCNgGWVE+v4Ts1ShMIP0gWda+aj/BoYEGK75lZ1jVur/HmVbPhS2oPfB14E3gCODrJNhK4L/k8KdknOf54Mm/qJOD4pFe5L9AfWL8yt4404wjPAMYBO0p6D5hDYVC1meVd4066sBUwIenhLQPuiYj7Jc0EJkr6OfAK8Psk/++B2yTNplATPB4gIt6QdA8wk0Lr9YyIqHPIX52BUFIZMDAivi6pI1AWEcsafJtmli2NOKA66X/Yo4b0t6mh1zciPqcwnK+mc10GXJb22nWG82QuwjOTzyscBM1sHWXl6bdmLE3TeLKknwB3AyuqEiNiSdFKZWYtQ1k2XrFLEwhPTX6eUS0tgH6NXxwza1GU/YlZAYiIvk1REDNrgTIy6UK9gVBSO+D7wP4UaoLPANcnDyrNLM9yMFV/lVuBZcC1yf4ICnMT1thbY2Y5kpcaIbBDMhtElSckvVasAplZC9LMe4PTSlOvfSWZCgcASXsDzxWvSGbWYjTuNFwlk6ZGuDdwsqR3k/1tgDerlvn0sp5mOZajpvGwopfCzFqmHA2f8XKeZlazHA2oNjOrWUY6SxwIzazh8tI0NjOrlZvGZpZ7Oeo1NjOrmZvGZpZ7bhqbWe6519jMcs9NYzPLPTeNzSz3XCO0mrz9zjzOufDStfvz3lvAWaNP4dUZM5nz7jwAli1bTufOnbhvwri1+d5fuIjDTjyVM0eNZNQJxzZ5ufNs5cqVnDjqNFatWkVlZSVDv34QZ31vDFOmTuNXV13D6tWrGbDTjlx20U9p1aoVy5Yt59yfXsj7CxZSWVnJqSefxFHDv1nq2ygND5+xmvTbtvfaAFdZWckBw4/j4AP255Tjjlqb5/JrxtKpU8d1vvfLa8by1cFfWrHQmkCbNm2YMO46OnbowOrVFZxw6mj232cw51/4M2654Xf03XZbrr7uBv70lwc45lvDueOeP7Bdv75cf/WVLFnyEcO+dQzfPHQYbVq3LvWtNL1mPr1WWkW5C0k/lLSXpFwH2ikvvULvnlvTc6st16ZFBH99/CkOP/jAtWmPPvUsvbbeiv59+zR9IQ1JdOzQAYCKigoqKiooLy+jTZs29N12WwD2GzyIRx57opAfsWLFp0QEKz77lE026UKr8mz0nm6wjCznWaxw3gu4Glgs6UlJv5B0mKTuRbpes/TAo0+sE/AAXnr1dTbt3o0+vXsB8Olnn3Hj7RM589STS1FES1RWVjL8uBPZ96Ch7Dt4ELvuMoCK1RW8/sZMAB569HEWLloEwInHH8Nbc+by1UMO5YhjTuC/zv0RZRmpGW0wKf3WjBXltxcRP4mIfYEewH8CSygsCzpD0szavidpjKSXJL00bsIdxShak1m1ejWPP/s8ww48YJ30+x99nMO//rW1+9feNIGRxx9Nxw7tm7qIVk15eTn33X0HTz18P9NnzGTWW29z5eU/55e/uYqjTzqFjh07UJ7U+p59/gV22qE/zzzyIH+eeDuXXP4/LF++vMR3UCI5mqF6Y7QHugCbJNv7wOu1ZY6IcUDhAduH86PIZSuqp6dMZcD2/dms+xeV4IqKSiY/+Qx/vPn6tWmvzXyTh594ml//bhyfLF9Omcpo26YNJx19ZCmKnXtdOndm74F78szzUxh18kncOf5GAJ6d8gJz3ylM0v7HSfcz5jsnI4ltt+lNr55b8/bcd9h1lwGlLHppNPOaXlpFCYSSxgEDKKx+9yLwPHBlRHxUjOs1Rw9MfpzD1msWP//Sy/Tbdht6bLH52rQ7x1699vO1N02gQ4f2DoJNbMmSj2jVuhVdOnfm888/5/kXpzL6lJP5cMkSNu3enVWrVnHjLbdy+qjvALBVjy2ZMnUaA/fcgw8+/JA5c9+lV8+eJb6LEvHwmTptA7QFZgHvAfOBpUW6VrPz2eef8/y0l7nkvHPWSX/w0Se+FByt9BZ/8AHnX/gzKtesIdasYdjBX+drB3yVK666hiefeZY1a9Yw4pij2GfQXgB8f/QoLrjoEr55zAgigp+cfSbdu3Ut8V2USDPvBElLEcVpgUoShVrhvsm2C4VnhVMi4qJ6T9DCm8a5075zqUtgDdFhk41q21Y+fXfqv9PyA45rtu3ooj0jjEKEnSFpKfBxsh0ODALqD4Rm1vy5aVw7SWdRqAXuB6ymsA7yFGA8dXSWmFkL486SOvUB7gXOiYgFRbqGmZWaa4S1i4gfFeO8Zta8yDVCM8u9smyEkGzchZmVhucjNLPc8zNCM8s9PyM0s9xzjdDMci8jNcJshHMzK43y8vRbPSSNl7RY0oxqaRdLek/Sq8l2aLVjF0iaLekfkoZWSx+WpM2WdH6a23AgNLOGU1n6rX63AMNqSL8qInZPtgcBJO0MHE9hPoNhwHWSyiWVA78DvgHsDIxI8tbJTWMza7hGbBpHxNOS+qTMPhyYGBErgTmSZlOYxwBgdkS8XSieJiZ5a50QGlwjNLON0bg1wtqcKWl60nTulqT1BOZVyzM/SastvU4OhGbWcBuwZkn1pTiSbUyKK4wFtgN2BxYAv6m6cg15o470OrlpbGYNV54+hKyzFEf67yyq+izpRuD+ZHc+0Lta1l4UlgKhjvRauUZoZg2mQk0v1dbA829VbfdbQFWP8iTgeEltJfUF+gNTgWlAf0l9JbWh0KEyqb7ruEZoZg3XiAOqJd0FDAE2kzSfwgTOQyTtTqF5Oxc4DSAi3pB0D4VOkArgjIioTM5zJvAwUA6Mj4g36r12sabq32ieqr9l8VT9LdNGTtUfs6al/jtV/72a7ehr1wjNrOH8ip2Z5V5GXrFzIDSzhkvx6lxL4EBoZg3nprGZ5Z6bxmZmDoRmlneuEZpZ7jkQmlnuubPEzHIvGxVCB0Iz2xjZiIQOhGbWcH5GaGa550BoZrnnzhIzM9cIzSzv3DQ2s9xzIDQzcyA0s5xr6KJMzY0DoZk1nHuNzSz3XCM0s9xzIDQzcyA0s7xzjdDMci8bcdCB0Mw2gnuNzSz33DQ2M3MgNLO8c43QzHLPgdDMci8jnSWKiFKXIXckjYmIcaUuh6Xj31f2ZSOctzxjSl0A2yD+fWWcA6GZ5Z4DoZnlngNhafh5U8vi31fGubPEzHLPNUIzyz0HwiYgKSTdVm2/laR/Sbq/lOWymkm6StIPq+0/LOmmavu/kfSj0pTOisGBsGmsAHaR1D7ZPxh4r4Tlsbo9D+wLIKkM2AwYUO34vsBzJSiXFYkDYdP5K3BY8nkEcFcJy2J1e44kEFIIgDOAZZK6SWoL7AS8UqrCWeNzIGw6E4HjJbUDdgVeLHF5rBYR8T5QIWkbCgFxCoXf1z7AQGB6RKwqYRGtkfld4yYSEdMl9aFQG3ywtKWxFKpqhfsCVwI9k88fU2g6W4a4Rti0JgG/xs3ilqDqOeFXKDSNX6BQI/TzwQxyIGxa44FLIuL1UhfE6vUccDiwJCIqI2IJ0JVCMJxS0pJZo3MgbEIRMT8iri51OSyV1yn0Fr+wXtrHEfFBaYpkxeI3S8ws91wjNLPccyA0s9xzIDSz3HMgNLPccyA0s9xzIMwRSV0lfb+I5z9F0m/ryXOxpJ9s4HmXb1zJzOrmQJgvXYEaA6Gk8iYui1mz4UCYL5cD20l6VdL/SBoi6QlJdwKvS+ojaUZVZkk/kXRx8nk7SQ9JelnSM5J2rOtCkr4p6UVJr0h6VNKW1Q7vJulxSbMkja72nXMlTZM0XdLPGvfWzWrnSRfy5Xxgl4jYHUDSEGBQkjYnmRSiNuOA0yNilqS9geuAA+vI/ywwOCJC0neB/wB+nBzbFRgMdARekfQAsAvQPymPgEmSDoiIpxt0p2YbwIHQpkbEnLoySOpEYbKBP0iqSm5bz3l7AXdL2gpoA1S/xn0R8RnwmaQnKAS//YFD+GKev04UAqMDoRWdA6GtqPa5gnUfl7RLfpYBS6tqkildC1wZEZOSmufF1Y6t/15nUKgF/jIibtiAa5g1Cj8jzJdlQOc6ji8CtpC0aTIT8+EAEfEJMEfSMQAq2K2ea23CF8sRjFzv2HBJ7SRtCgwBpgEPA6cmtU8k9ZS0RfpbM2s41whzJCI+lPRc0iHyV+CB9Y6vlnQJhdmY5wB/r3b4RGCspJ8CrSnMuP1aHZe7mEJT+j0KM7j0rXZsanLtbYBLkxmh35e0EzAlaX4vB04CFjfwds1S8+wzZpZ7bhqbWe45EJpZ7jkQmlnuORCaWe45EJpZ7jkQmlnuORCaWe45EJpZ7v0/4X/bdAHVkZQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.7483877468762595\n"
     ]
    }
   ],
   "source": [
    "# confusion matrix\n",
    "mat = confusion_matrix(y_test, y_pred)\n",
    "\n",
    "labels = sorted(set(y_test))\n",
    "\n",
    "# plot it\n",
    "sns.heatmap(mat.T, square=True, annot=True, fmt='d', cmap=\"Reds\", xticklabels=labels, yticklabels=labels)\n",
    "plt.xlabel('true label')\n",
    "plt.ylabel('predicted label')\n",
    "\n",
    "# save\n",
    "#plt.savefig('images/')\n",
    "\n",
    "# show\n",
    "plt.show()\n",
    "\n",
    "print('accuracy:', accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is lower than our baseline. It seems adding these features is just confusing the classifier instead of helping.\n",
    "\n",
    "## Predict both genders\n",
    "Now I think this task may be very hard, but I'm going to try it anyways. Given the text of the responder, can we predict both their own gender and the gender they are responding to? Let's get a refresher of what our dataframe looks like in this case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_gender</th>\n",
       "      <th>response_text</th>\n",
       "      <th>response_tokens</th>\n",
       "      <th>response_length</th>\n",
       "      <th>response_avg_slen</th>\n",
       "      <th>source</th>\n",
       "      <th>gender_info</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>W</td>\n",
       "      <td>One day at a time! =]</td>\n",
       "      <td>[One, day, at, a, time, !, =, ]]</td>\n",
       "      <td>8</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>WM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>W</td>\n",
       "      <td>Crazy is synonymous with awesome in this case.</td>\n",
       "      <td>[Crazy, is, synonymous, with, awesome, in, thi...</td>\n",
       "      <td>9</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>WW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>W</td>\n",
       "      <td>You can enter assisted pullups. Select pullups...</td>\n",
       "      <td>[You, can, enter, assisted, pullups, ., Select...</td>\n",
       "      <td>57</td>\n",
       "      <td>11.400000</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>WM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>M</td>\n",
       "      <td>Mirin 3% bodyfat? Yeah, you are.</td>\n",
       "      <td>[Mirin, 3, %, bodyfat, ?, Yeah, ,, you, are, .]</td>\n",
       "      <td>10</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>MW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>M</td>\n",
       "      <td>Hey! I just started a new job, so things are s...</td>\n",
       "      <td>[Hey, !, I, just, started, a, new, job, ,, so,...</td>\n",
       "      <td>26</td>\n",
       "      <td>8.666667</td>\n",
       "      <td>fitocracy</td>\n",
       "      <td>MM</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  op_gender                                      response_text  \\\n",
       "0         W                              One day at a time! =]   \n",
       "1         W     Crazy is synonymous with awesome in this case.   \n",
       "2         W  You can enter assisted pullups. Select pullups...   \n",
       "3         M                   Mirin 3% bodyfat? Yeah, you are.   \n",
       "4         M  Hey! I just started a new job, so things are s...   \n",
       "\n",
       "                                     response_tokens  response_length  \\\n",
       "0                   [One, day, at, a, time, !, =, ]]                8   \n",
       "1  [Crazy, is, synonymous, with, awesome, in, thi...                9   \n",
       "2  [You, can, enter, assisted, pullups, ., Select...               57   \n",
       "3    [Mirin, 3, %, bodyfat, ?, Yeah, ,, you, are, .]               10   \n",
       "4  [Hey, !, I, just, started, a, new, job, ,, so,...               26   \n",
       "\n",
       "   response_avg_slen     source gender_info  \n",
       "0           4.000000  fitocracy          WM  \n",
       "1           9.000000  fitocracy          WW  \n",
       "2          11.400000  fitocracy          WM  \n",
       "3           5.000000  fitocracy          MW  \n",
       "4           8.666667  fitocracy          MM  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "responder_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Welcome! Fitocracy is a great place to track your workouts and keep consistent, with a very supportive community. To help you find your way around:                Getting Started Guides:        On the web - http://bit.ly/I5PoXP        On iPhone - http://bit.ly/N2IXHE                If you need help finding an exercise name, have questions about training, etc--leave me a message on my profile or ask the Welcome to Fitocracy group:        http://ftcy.co/UpSig0        Glad to know ya ;)                            28131\n",
       "Hello and welcome! Fitocracy is a great place to track your workouts and keep consistent, with a very supportive community.        Start by earning points for any exercise...even walking up the stairs. To start:                1. Tap on the \"track\" icon on the bottom center of your screen.        2. Tap on \"start a workout\"        3. Tap on \"Add an exercise\"        If you need help finding an exercise, have questions about training, etc--drop me a line ;)                                                         10009\n",
       "Hi! I've just noticed you're posting statuses, but don't have any followers yet. Like Facebook, or Twitter, only people that are following you are likely to see your posts, so it's better to post a status/ask questions in groups. The best place to start is probably the 'Welcome to Fitocracy Group' https://www.fitocracy.com/group/6460/                                                                                                                                                                                     2764\n",
       "Thanks!                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              2313\n",
       "Welcome! Fitocracy is a great place to track your workouts and keep consistent, with a very supportive community. To help you find your way around:                Getting Started Guides:        On the web - http://bit.ly/I5PoXP        On iPhone - http://www.youtube.com/user/fitocracy/videos                If you need help finding an exercise name, have questions about training, etc--leave me a message on my profile or ask the Welcome to Fitocracy group:        http://ftcy.co/UpSig0        Glad to know ya ;)     1741\n",
       "Name: response_text, dtype: int64"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check for duplicates again\n",
    "responder_df.response_text.value_counts().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, some more seemingly automated messages or things that are too simple, like \"thanks!\". Let's drop duplicates again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "gooooood ... spent like 90% of my morning playing around on fitocracy. ^_^ how about you?                          1\n",
       "True I guess I never seen a girl in knee high socks then to see what they bring out that I could possibly like.    1\n",
       "I love this! Thanks so much for sharing!  :)Keep kicking ass!                                                      1\n",
       ">unless you are an Adonis, you have to be interesting.  Well, that rules me out                                    1\n",
       "> If it makes you feel any better   Nope.                                                                          1\n",
       "Name: response_text, dtype: int64"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "responder_df.drop_duplicates(subset='response_text', keep=False, inplace=True)\n",
    "responder_df.response_text.value_counts().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks good!\n",
    "\n",
    "Recall: the gender_info column first lists the original poster's gender, and then the responder's gender. This is the column we want to predict.\n",
    "\n",
    "What is our baseline here?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MM    0.383260\n",
       "WM    0.240424\n",
       "MW    0.240041\n",
       "WW    0.136274\n",
       "Name: gender_info, dtype: float64"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "responder_df.gender_info.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our highest percentage is male poster/male responder, at 38.3%.\n",
    "\n",
    "### Using Naive Bayes\n",
    "Can we simply use Naive Bayes? Or is this too complicated of a task? Let's see:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "233616\n",
      "58405\n"
     ]
    }
   ],
   "source": [
    "X = list(responder_df.response_text)\n",
    "y = list(responder_df.gender_info)\n",
    "\n",
    "# create train test split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\n",
    "\n",
    "print(len(y_train))\n",
    "print(len(y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use same parameters\n",
    "model = make_pipeline(TfidfVectorizer(max_df=0.5, min_df=5, ngram_range=(1,2), tokenizer=nltk.word_tokenize),\n",
    "                      MultinomialNB())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fit and predict\n",
    "model.fit(X_train, y_train)\n",
    "y_pred = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAEKCAYAAABwhMxUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd4FUX3wPHvuTeUEAgQIFSRLr0IioCCgAIKCj8FBRGxIKKIio3mKygqqCjy2lFUbGBFsPIqIkovIggCUkVQekgglCQ35/fHbkLQ3OQm5OamnM/z3Ifs7OzuDElOZnZmZ0VVMcYY82+eUBfAGGPyKguQxhjjhwVIY4zxwwKkMcb4YQHSGGP8sABpjDF+WIA0xhg/LEAaY4wfFiCNMcaPsFAXwJ8hElkgH/F58ZU7Q12E4Ig9HOoSBI106xPqIgSNp2lHOZPjs/J7+orGndG1QsFakMYY40eebUEaY/K+gt7CsgBpjMm2MMl3veYssQBpjMk2T8GOjxYgjTHZZ11sY4zxw2NdbGOMSZ+1II0xxg+7B2mMMX54rYttjDHpsy62Mcb4YV1sY4zxw1qQxhjjh03zMcYYP8IKdny0AGmMyT7rYhtjjB8eCnYT0gKkMSbbbBTbGGP8sC62Mcb4YS1IY4zxwxbMNcYYP6yLbYwxflgXOx8ZMO1FmvToxpF9+xnf5AIAqjVrwnWvPEeR4sVITkpixh33sWPFqtRjzm51LiOWzuP1a2/k509mU+/ii+gzeULq/kr16/F635tYM/tLBr75MnU7tON4bBwA02+8nV1rfs3dSgJxJxJ4+Nuf2XwgFhHhsS4taV6lHO+u3sL7v2zF6xE61KzM/e2bsDs2nh5v/Y8aUaUAaFY5inGXnAvA15v+5NVlG/GppuYPmahoPFfccmq7THl04RdQvATStB0cOwJA8k9zYNt6qHQ2nq7XOXlFSF70JWxeA6XK4uk+ECIiQZPRNYvQVfNDUKFTTiYkMuDhSSQkJZHkS6brBecy7NoreGDKNNZt3UlYmJemdWowbnB/ioR5iT0az5iX3ubPvQcoViSMx+64gXrVq/o9TygV9Gk+opo3Xz+dnfdi17moLSePxnPj26+mBsi75n7GvMkvsv6bb2l8WRe6PHg3z3bsDoB4PNz97WySTpxk8Rvv8PMns087X4myZRm/5RdGVqtP4vHjDHzzZX794pt/5cuKnHgv9qhvVtCyanl6N6lJgi+ZE4lJbNh3mFeXb+SVXu0oGubl4LETlCtRnN2x8dz+2WLmDLz0tHMcPn6Sq96dx8f9OxNVohijvlnBlQ3Ppk316OwVKiffiy2C5/YnSH73aaRJG0g4ia747vQ8YUXA5wNNhohIPDeOIfmlUVCiJJQsDXv/hKLF8NwwkuRZr8LBPdkvzhm+F1tVOXbiJBHhxUlM8nH9f55m1E3XEHs0nvYtGgNw/5RptGpQl35dO/D0259QIrwYQ/v0YNvuPYx/fQZvjh3u9zzN69XKdtnO9L3Y0yIrBPx7ekvc/nwXTQvULYQtPy3m2KGY09JUleKRTuupeOlIDv916hel47AhrP5kDkf27U/3fOf27sn6r78l8fjx4BU6i46eTGTlrgNc3bgGAEW9HiKLF2Xm2m0MOu8cioZ5AShXoniG5/kzNp4aZUsSVaIYAG2qR/Pt5t1BLXvAzq4Phw9A3CH/eZISneAITrDE/T2Nj3OCI0DCSScwliwT1OJmRkSICHe+H0k+H4k+HyJCh3ObICKICE3q1GDvQednd8uuv7mgcX0AalWtxO79BzlwOM7veULJK4F/8qOgdLFF5KqM9qvqp8G4bno+umcEd82dxdWTHsPj8fBUW6clVaZKZZr/Xw8md+pBjfPOTffYVn2vZt6zL56WduXjD3P5wyPYNG8Bs0aOJSkhIeh1SOvP2HiiwosxZu4qNu4/TKOKZRnVsRk7Yo6yavdBpixaTzGvlwc6NKFJpSgAdsfGc9U731GyaBHuateIVtXKU71MSbYfOsru2Hgqlgpn3pa/SExOztW6+CP1W6IbVp7aPrcD0qg1uucPdP4ncNL9g1W5Bp7LrofIKJK/nH4qYKaIjIKKZ8HfO3Kt7P74fMn0HvEEO/fsp1+3DjSrWzN1X2KSjzk/LmP0TdcAUL9GNb5dtpqWDeqwdvN2/tp/iL0HYyhfJjLD84RCTnaxReQNoAewT1Ubp0kfBtwJJAFfquqDbvoo4BbAB9ylqnPd9G7AFMALvK6qE930msBMIAr4GRigqhn+AgerBfkx8BBOZXsAV6T59AjSNdPV/vZBfDR8FKOrN+Sj4aMYMO0FAPo8N5FZI8aifoJCZKWKVG3SiPVzT3XtZo0ax7j6LZl43sWUiCpLlxHDc6UOafmSld/2HebaZrX4dMAlhBfx8vryTfiSlbgTCczs15H72zfh3i+WoapUiCjOvFsv49MBlzDi4qY8+NVyjp5MpHTxojzcuTn3frmMAR8soErpCLySBzoUHi9Spym66WcAdPWPJE99mOS3noD4OKTj1afy/r2D5DceI/ntp/Bc0BW8af7eFymGp9dgkud9DAkncrcO6fB6Pcya9BDzX53Ar1t28PvOU631R19/n1YN6tKqQV0Abu3Vlbj4Y/zf/Y/x7tc/0KDmWXi93kzPEwoeCfwTgLeAbmkTRKQj0BNoqqqNgEluekOgL9DIPeYlEfGKiBd4EbgMaAj0c/MCPAlMVtW6QAxOcM1QsAZprgauBZoCs4EZqrols4NEZDAwGOAiitGQomdckDYD+/Hh3Q8CsOqjWVz/+vMAnN2qBYNmvgFARPlyNLq8C76kJNbM/hKAVtdcxS+zPic5KSn1XHF79gKQlJDAkjff5ZL77zrj8mVVxVLhVCwVTrPKTuuwS91qvL5iE5VKhnNp3aqICE0rR+ERIeZ4AlEliqV2uxtVLMtZZSLYEXOUxpXK0rF2FTrWrgLAh2u35Y3l82s1crrI7qBM6r+ArlmI5+o7+NdNr0N7IDEBKlSBPTvB48HT61b0t+Ww+ZdcK3ogIiNKcH6jeiz8ZT31qlflxY++ICbuKI/c3z81T8kS4TwxdCDg3CK6ZOgYqkWXy/A8oZKTf1JV9UcRqfGP5NuBiap60s2zz03vCcx007eLyBbgfHffFlXdBiAiM4GeIrIB6AS4I3tMB8YBL2dUpqA0GVR1lqr2BToAW4FnRGShiHTI5LipqtpKVVvlRHAEOPzXHup1uBCAczp1YN/mrQA8VKspY2o2YUzNJqz+eDYz77g3NTgCtOrXmxUzPj7tXJGVKqZ+3axXD/5a91uOlDErKkQUp1KpcLYfcgLH0p37qB1Vik51qrBsp/OzsyPmCIm+ZMqGF+XQsZP4kp2Q8ufho/wRc5RqpSMAOHjMaVnFnkhgxppt9G5SI9fr80/SoBW6YcWphIjIU/vqNUcP/OVslC4HKS3eyCiIiobYg06+bgPQg3vQld/nVrEzdCj2CHHxxwA4cTKBJWs3UrNqJT6at5CFv/zGpLtvweM59asYF3+MhETnD/NH8xbSqkFdSpYI93ueUJIsfLKpHnCRiCwTkQUicp6bXhX4M02+XW6av/RywGFVTfpHeoaCPc3nBBALxAHVgYxHDs7QLe+/Qb2LL6Rk+XJM+HMDn499gndvHcY1U57EGxZG4omTvDf47kzPU+7s6kSdVZXNCxaeln7ze69TqkJ5EGHXL7/y/pB7glWVDI3p2JwHv15Ooi+ZaqUjeLxrK8KLhPHQ3JVcOf1bing9PNGtFSLCyl0HeH7JesLEg8cjjL2kBWXCnT8+E+avYeP+WADuuKABNcqWCkl9UoUVQWrUJ3nu+6lJcvH/IdHVnDGYuIOp+6RqbeTqLs5INkry/z6A4/FQtTaexq3RfbuRgaOANFODQmT/4VhGvTAdX3Iyyap0a9OSji2b0vjaO6hSIYp+Y54C4JLWLRjapztbd+1h5Atv4vV4qF2tMo/dPiDD84RSVhbMTdtDdE1V1amZHBYGlAUuAM4DPhSRWqQfc5X0G32aQf6MyxyMaT7ufYN+OE3e73CawiszPup02Znmkx/kxDSfPCknp/nkMWc6zScvO9NpPh+VrRjw72mfmL2ZXsvtYn+RMkgjIt/gdLF/cLe34gTLQQCqOsFNn4vTZQYYp6pd3fRRbtpEYD9QSVWTRKRN2nz+BOuu/Dyc4LgQKAbcICL/TfkE6ZrGmFyWC13sz3DuHSIi9YCiwAFgDtBXRIq5o9N1geXACqCuiNQUkaI4Azlz1GkJzgd6u+cdiDM+kqFgdbFvJoDmqzEmf8vJeZgiMgO4GCgvIruAscAbwBsisg5IAAa6wW69iHwI/IYz/Weoqvrc89wJzMWZ5vOGqqbcXxkBzBSRx4DVwLTMyhSUAKmqbwXjvMaYvCUn5z2oaj8/u673k/9x4PF00r8CvkonfRunRroDEqyJ4nMy2q+qVwbjusaY3JUHZs4GVbC62G1whtpnAMvI2T80xpg8Ii9MnQ2mYAXISsClOCPZ1wFf4kwWD91cC2NMjivoq/kEa6K4T1W/UdWBOEPyW4Af3GcqjTEFRC6MYodU0CaKi0gxoDtOK7IG8F8g1xapMMYEny2Ymw0iMh1oDHwNPKKq64JxHWNMaEm+bRsGJlgtyAFAPM5zlHelmSslgKpqpL8DjTH5R8EOj8GbB1nQR/+NMVgX2xhj/Croo9gWII0x2Vaww6MFSGPMGbCJ4sYY40cBj48WII0x2WfTfIwxxo/8+jrXQFmANMZkWwGPjxYgjTHZZ11sY4zxw0axjTHGj4L+yJwFSGNMthXwBmTeDZAv/7ks1EUIiuTl/wt1EYJCLuoe6iIEjXi9oS5CnpWV92LnR3k2QBpj8r6CHR4tQBpjzkBOvvY1L7IAaYzJNlvuzBhj/JACHiEtQBpjss1TwOf5WIA0xmSb3YM0xhg/Cnh8tABpjMk+a0EaY4wfBTw+WoA0xmSfPUljjDF+eGyajzHGpE9smo8xxqTPBmmMMcaPAh4fLUAaY7LPWpDGGONHAY+PFiCNMdnntVFsY4xJX0HvYvsdpBeRqIw+uVlIY0zeJBL4J/NzyRsisk9E1qVJe1pENorIWhGZJSJl0uwbJSJbRGSTiHRNk97NTdsiIiPTpNcUkWUisllEPhCRopmVKaNZTKuAle6///yszLy6xpiCLicDJPAW0O0fad8CjVW1KfA7MMq5rjQE+gKN3GNeEhGviHiBF4HLgIZAPzcvwJPAZFWtC8QAt2RWIL9dbFWtGVCVjDGFVk4umKuqP4pIjX+kpX3L3VKgt/t1T2Cmqp4EtovIFuB8d98WVd0GICIzgZ4isgHoBFzn5pkOjANezqhMmc6DF8f1IvIfd7u6iJyf2XHGmILP65GAPzngZuBr9+uqwJ9p9u1y0/yllwMOq2rSP9IzFMggzUtAMk70HQ8cAT4Bzgvg2JD4e99+RkycwoGYw3hEuKZ7F264+gq+WbCIF6bPZOvOXXz44tM0OadO6jGvvv8xn3z9HR6PhzF33spF57Xwe55Qu+TJd4goVgSPRwjzePjozj6p+974cTWTvl7CooduomxEOEdOnGTEB9/x9+GjJCUnc9NFzbmqVQOWbd3NxC8Xph63ff9hJvW9lEsa1QpFlTiZkMiAMRNJSEokyZdM1zatGNavF7v27ue+Z17h8NF4GtY6myfvvpWiRZwf268XLefFmbNBhPo1zmLSvbelnu/oseN0HzaGS1qfy38GXx+SOqX4e/9BRjw3lQMxsc7PUdeO3HBlFzZs+4NxL03nZGIiXq+HsUNuoGm92hyJP8YDz77K3/sP4vP5uOn/LuPqS9o7+V+eTvyx43g8HoZccyWXX9Q6pHXLyhiNiAwGBqdJmqqqUwM8dgyQBLyXkpRONiX9Rp9mkD9DgQTI1qp6roisBlDVmEBuboaS1+tlxJCbaFSvNkePHefqIffRtmVz6taozn8fGcnYyS+dln/Ljj/5av5Cvpj2PPsOHuKmBx7mm+kv+T1PnRpnhahmp7x1a0/KRoSflvb34SMs2bKLymVKpqa9v2QdtaOjeGlgdw4dPc7lz75Pj+b1aF27KrPuuhaAw8dO0G3Se7SrG7p6FS0SxpuPPkBEeHESk5K4fvQELjq3CdPnzOWGK7rQ/aLWjHv5bT6Z9xP9unVkx197ee2Tr3hvwmhKl4zg4OG408733/dncV6jc0JUm9N5vV5G3NyPRrVrOD9H946lbfNGPP3WBwzt15P2LZuxYOUann7rQ955YhTvfTmPOmdV4ZX/DOdQbByX3T6SKzq0pXixYjw5fDA1qlRi78EYet87lgtbNCayZETI6paVUWw3GAYUEP9xjYFAD6CzqqYEtV1A2h/YasBf7tfppR8AyohImNuKTJvfr0AeNU90b3yqW9gKOC3KPCu6XBSN6tUGoGSJcGqfXY29Bw5S++yzqHXWv1vV8xYv4/KOF1K0aBGqVa5I9aqVWbtxs9/z5FVPfrmI+y5rg6T5YykC8ScTUFWOJSRSOrwYYf94kcj/1m3lonrVCS9aJLeLnEpEiAgvDkCSz0eiz4cILP11I13btgKgZ8e2zFv2MwAffbuAfpd1orQbHMqViUw91/qtOzgQG0e75o1yuRbpi44qQ6PaNQD356haFfYejEFEOHrsBABH4o8RHeUM0IpA/PETzvfs+ElKl4wgzOuhZtVK1KhSCYCK5coSVTqSQ3FHQlKnFDk8SJPO+aUbMAK4UlWPpdk1B+grIsVEpCZQF1gOrADquiPWRXEGcua4gXU+p+5hDgRmZ3b9QFqQ/wVmARVF5HH3Ag9lUqmyqhoTwLmDbteevWzYso1mDer5zbP3wCGap9lfqXw59h44lOXz5BYRGPTG5wjCNa0bcs35jfj+t+1ER0ZQv3L50/L2b9OEoW9/RYcJ04k/mcCz/br8a4mqr9dsYeCFzXKzCuny+ZLpff8j7Nyzj36XdaJ6pWgiI0oQ5vUCUKl8FHsPHgbgj7/2AnDdqCfwJSdz57U9uejcJiQnJ/Pkmx/w5D23snTtbyGriz+79u5nw7Y/aHZObUYP6s+gsU/z1JszSU5OZsZT/wGgf/dLuOPx52h/493EHz/Bsw/cgecff9TW/r6VxKQkqleKDkU1UuXkPEgRmQFcDJQXkV3AWJxR62LAt+61lqrqEFVdLyIfAr/hdL2HqqrPPc+dwFzAC7yhquvdS4wAZorIY8BqYFpmZco0QKrqeyKyCujsJvVS1Q2ZHLZJRPYDi4FFwGJV/T2za+W0+OPHuWvck4y64xZKRpTwn1H/fSsi7fc94PPkkveGXEV0ZAQHjx5j0LTPqVWhLK/OX8Xrt/z7/ujC3/+kfuXyvDmoJzsPxjHojTm0rFGFksWduyT74+L5fe9B2tUL/W0Dr9fDrMmPEBd/jGETX2Drrr//lSfl+5Lk8/HH33uZPv5B9h6M4foxE5kzZTyfL1hC+5ZNqVw+703VjT9+grsmPs+oQf0pWSKc5979hJGDrqNr2/P4euEyHnp+Gm+OH8HC1etoULM60x8byc6/93Hzw0/RqtE5lCzh3FLZd+gwD06eysS7b/1X4MxtOTlPXFX7pZPsN4ip6uPA4+mkfwV8lU76Nk6NdAck0P/dEjjR2AOEZ5IXVY0G/g8nOLYFPhWRvSIyW0Qe9HeciAwWkZUisnLqex8GWLT0JSYlcde4J7micwe6XNQmw7wVK5Tj7/0HUrf3HDhIdLmoLJ8nt0RHut3KkiXo3KgmK7b9xe6YI/zflA+55Ml32Bt3lKuf/4j9R44xa9UGLmlUCxHh7PKlqVY2km37TzXuv/l1C5c0rEURt5WWF0RGlOD8xuewZtNW4uKPkeTzAbDnwKHUbmilclF0Pr8FRcLCqFaxAjWrVOKPv/byy6atvP/VPDoPfoCn3vqQ2T8s5pm3PwpldQD352ji81zRoS1d3FsGn32/kC5tnK+7tTuftb9vA2DWvJ+4tE0r53tWpSLVKlZg2y7ndtnRY8cZ8uiz3NP/aprXr5P+xXKRxysBf/KjQKb5PIwzZygKKA+8KSIZdrEBVPV3VX1LVQfjzFl6DGdS5yMZHDNVVVupaqvB/a8JtA7pnYeHJr1A7erVuKlPz0zzd2p7Pl/NX0hCQiK7/t7LH7v/pmn9ulk+T244lpBI/MmE1K8Xb/6TxtWiWfjQTXw3YgDfjRhAxciSfDKsDxVKlaBymVIs3boLgANHjrH9wGHOijp1v+7LNVu4vFndkNQlrUOxccTFO7eYTpxMYMma36hVrTKtG9dn7mLnuYTZ8xfT6fwWAHRu3YJlv24EICbuCDv+2kO1ihV4evhgvn9tEvOmPs2DN15Dz4vbct8NfdK/aC5RVR56fhq1q1Xhpl6n5kFHR5Vh+TqnDkvX/sbZVSoCULl8FEvWOLcHDsTEsn3335xVKZqExCTufOK/9OzYjm4X5o2ZdiIS8Cc/CuQeZD+ghaqeABCRicDPOAEvXSLSFqfl2AZnRGkbziTP691jg+rndRuY/e0P1Kt5Nr0G3wPA8FuuJyExiceef41DsbEMGT2e+nVqMu3JcdStUZ3LLm5H95vvxOv18vCwwXi9Xlb9+lu65+nQulWwq+DXwaPHuOudbwBISk6me/O6XHROdb/5b+/UitEfzaPnczNR4N5uF6SOfu+OiWNP7FHOq1klN4qeof0xsYz67zR8yckkJyvd2p1Hx/OaU+esKtz3zKv89/1ZNKhZnd6XXATAhS0as+iX9fQYNgaPx8P9A6+hbGTJTK4SGj9v2Mzs+Yupd3Y1et3t3GccPqA34++8mcdfexefL5liRYvw6NCbALj92p6MmvIaVwwbA6pu3UoxZ/4iVq7fxOEjR5n1vTNFa8Ldg2hQ6+yQ1Y0CvliFaDr3307LIPI10E9VD7vbZYB3VbVHBsck4wTCZ4HP/jH6FBDdtSHTOUr5UfLy/2WeKR+S+qH7oxFskoduP+Q0OeeCM4pwsR2bB/x7Wnr+L/kumvptQYrI8zhTe04C60XkW3f7UmChv+NcVXBakG2BISIShhMwlwBLUh4DMsbkb/m16xyojLrYKQtSrMKZ5pPih8xOqqp7gE/dDyJSAucxoUeAmjgDPsaY/M5bsN/aldFiFdOze1IRKY1z/zGlFdkC2AJ8jjOybYwpAHJysYq8KNNBGhGpC0zAWTqoeEq6qmb00O4WnEGZxTjPby9X1eNnVlRjTJ5TiLvYKd7EmdE+GegI3ET6D36nUtUKZ140Y0xeV+hbkEC4qs4TEVHVP4BxIvITTtBMl4jMyeiEqnplFstpjMmLrAXJCRHxAJvdZxx3A5k9ANoGZ022GcAyMmlxGmPyKWtBcg/Oo4Z34dxP7ISzEkZGKuFMB+qHs4Lvl8CMNA+NG2MKACmso9gpVHWF++VRnPuPmXJX1fgG+EZEiuEEyh9E5FFVfT67hTXG5DGFtYstIp+TwYq7md1HdANjd5zgWANn2bRPs1VKY0yeJAW7AZlhC3JSdk8qItOBxjjvj3hEVddlcogxJj8qrC1IVV1wBucdAMQD9YC70jyOJM6pNdLfgcaY/MOm+WSDqhbwhrcxBii8LUhjjMlMoR/FNsYYvwprF/tMR7GNMYVAIe5ip4xiX4Uz8ftdd7sfsCOIZTLG5BOFdj3IlFFsERmvqu3T7PpcRH4MesmMMXlfYe1ip1FBRGqlrALuvqTbVusxxtggDTAc5zHBlNck1ABuC1qJjDH5R2HtYqdQ1W/cRXPru0kbVfVkcItljMkPCvpE8UDei10CeAC4U1XXANVFxO8bDY0xhYhI4J98KJAbCG8CCThrPALsIoN3YhtjChGPBP7JhwK5B1lbVa8VkX4AqnpccmNs31Mwb/562lwe6iIEhR7YHeoiBE/5qqEuQZ5VaKf5pJEgIuG4k8ZFpDbOu7KNMYWdjWIzDmfx27NE5D2gHQEunGuMKeAKewtSVf8nIquAC3CWK7tbVQ8EvWTGmLyvsAdIEZmnqp1x3ivzzzRjTGFWQMcKUmS0WEVxnJd1lReRspx6M2EkUCUXymaMyesKcQvyNpw3GlYBVnEqQMYBLwa5XMaY/KCwBkhVnQJMEZFh9iZCY0y6vN5QlyCoArmBkCwiZVI2RKSsiNwRxDIZY/ILe5KGW1X1cMqGqsYAtwavSMaYfKOAB8hA5kF6RERUNWWiuBcoGtxiGWPyhXwa+AIVSAtyLvChiHQWkU7ADJyJ48aYws7jCfwTABEZLiLrRWSdiMwQkeIiUlNElonIZhH5QESKunmLudtb3P010pxnlJu+SUS6Zrt6AeQZAXwP3A4MBeYBD2b3gsaYAiQHA6SIVAXuAlqpamPAC/QFngQmq2pdIAa4xT3kFiBGVesAk918iEhD97hGQDfgJbfnm/XqZZZBVZNV9WVV7a2qV6vqq6rqy87FjDEFTM7fgwwDwkUkDGce9t9AJ+Bjd/90oJf7dU93G3d/Z3chnZ7ATFU9qarbgS3A+dmpXkYTxT9U1WtE5FfSebuhqjbNzgWNMQWH5OCTNKq6W0QmATuB48D/cOZgH1bVJDfbLiBleaWqwJ/usUkiEguUc9OXpjl12mOyJKNBmrvdf21xXGNM+rIwSCMig4HBaZKmqurUNPvL4rT+agKHgY+Ay9I5VUqDLb2LawbpWZbRRPG/3X//yM6JjTGFQBYCpBsMp2aQ5RJgu6rud04tnwJtgTIiEua2IqsBf7n5dwFnAbvcLnlp4FCa9BRpj8kSv+1jETkiInH+Ptm5mDGmgMnZe5A7gQtEpIR7L7Ez8BswH+jt5hkIzHa/nuNu4+7/3p2OOAfo645y1wTqAsuzU72MWpClAETkUWAP8A5O07U/UCo7FzPGFDA5+Kihqi4TkY+Bn4EkYDVOi/NLYKaIPOamTXMPmQa8IyJbcFqOfd3zrBeRD3GCaxIwNLsDy+LO//afQWSZqrbOLC2n6V+bsnXPIM+Tgrk8VEF+5YJUrBHqIgSNRNc4o5nevgm3Bfx76h31ar6bVR7Ib6tPRPqLiFdEPCLSH7BpPsaYAv+oYSAB8jrgGmCv++njphljCrscfpImrwnklQs7cIbejTHmdPm0ZRioQF65UA94Gaioqo1FpClwparm6Xdjj35yCj8sXUliqTC3AAAZvUlEQVS5MqX5/M0XANiwZRvjnn2JkwmJeL1ext4zhKYN6jFt5qd8/t0CAHw+H1t37mLxrHcIL16M6+8eRUJCIj6fjy4d2nHXTaFtPI9+8jl+WLLCqddbLwEw/JEn2b5zFwBxR+OJLBnBZ9OeJyY2jrvHTmDdxs306taZh++5PfU86zZtYdTEyZw8mUD7C1oxZtjgkL7C82RCIgMenkRCYhJJPh9d25zLsGuv5IHnprFu2x+Eeb00rVODcbddT5EwL8vXbWLoUy9RLbo8AJe0bsHQPs6U3bj4Y/zn5XfYvHM3IsJjd9xAi3Nqh6xuoyc8ww+Ll1GubBk+f9uZ5TLl9enM+2kJHo8QVbYME0bfT8Xy5QBYtnoNE/77CklJSZQpXZp3X5gEwFsffMrHX3yNiFC3Vk0mjLqPYsVCvG5MAQ+QgQzSLAAeAF5V1RZu2jr3WcmgOdNBmhVr1lEiPJyREyanBsibH3iYG3v3pH3rlixYupLXZ37KO889cdpx3y9ezvSPZzP92cdRVY6dOEFEeDiJSUn0HzaS0cMG0bxh/ewX7AwHaZx6FWfkE8+mBsi0Jr70OqUiIhg6sB/Hjp9gw+atbN7+B79v/+O0ANlnyHBGDxtM84b1GTxiHAOuvoL2rVtlu1xnOkjj/F+fJCK8OIlJPq5/6ClG3XwtsUfjad/C+VG7/7lptGpYl35dO7B83SbemPMtr4y+81/nGvn8m7RsUJc+l1xIQmISJxISiIwoke2ynekgzYpffnW+Z48/nRogj8bHUzIiAoC3P/6MrTv+4JH77ybuyFH63T6c1555nCoVozkYc5hyZcuwd/8Brht6L1++8xrFixXjnocfo/0F53PV5V3OqGxnPEgz+e7AB2mGT8l30TSQ39YSqvrPOURJ6ebMQ85r1pjSkSVPSxOEo/HHADgSH090uah/HfflvB/p3qm9k1+EiPBwAJKSfCT5kpB0J+nnnvOaNaZ0qfRnWakq38xfSPfOTvlLhBenZdNGFC16eitj38FDHI0/TotGDRARenbtxHcLl6Z3ylzj/F8XByDJ5yPR50MQOpzbBBFBRGhSpwZ7D8ZkeJ6jx46zcsNmenduB0DRImFnFBxzwnnNm1A68vTvWUpwBDh+/ETqz9UX383n0g7tqFIxGoByZVPXqsbn83Hi5EmSknwcP3GSaLfFGVIFfJAmkPUgD4hIbdxHdUSkN84D5H6JyH8z2q+qdwVcwhw0+s5BDHpwLE+98ibJmsyM5586bf/xEydZuOJn/nP3balpPp+Pq2+7l527/+a6XpfTrOE5uV3sgK1cu55yZctQo1rGj53u3X+QShVO/XJVqlCOvfsPBrt4mfL5kuk94nF27tlPv64daFavZuq+xCQfc35cyuibrk1N++X3bfS6bzzRZUvzwMDe1D2rCn/uPUBUZClGvzidTTt20bB2dUbfdC0lihcLRZUyNHnqm8ye+x2lIiKYPsX5Wdzx5y6SknwMGPYA8ceOcUOfXvTqdikVK5Tn5r696dR7AMWKFqPd+edy4fktQ1wD8m3gC1QgLcihwKtAfRHZjfMiryGZHDMEuBDn8Z6VOA+cp/2kS0QGi8hKEVk59d0PAiha1syY/TUj7xjEDx++wag7BvHQ06e/amf+4uW0aNyAMmn+2nu9Xj57fQo/fPQGazdu5vfteffJyy/nLUhtPWbs372ivPBz7vV6mDXpP8x/dSK/btnB7ztPddsffe19WjWsS6uGdQFoWKs6815+gs+e+Q/9L+/InU++DDh/0H7btpO+XTrw6aSHKFGsGK/NypvLlw4ffBM/fPIePS7txLufzgGc1vP6TZt59anxTHvmCV6e/j7bd+4i9sgR5i1cwncfTOfHz97n+PETzJk7L8Q1oMCPYmdYahHx4KzNdglQAaivqhcG8Hx2ZZwZ8F2BAUARYI6qTlfV6f4OUtWpqtpKVVsNvv5af9my7bP/fU+X9m0A6HZxO9Zu/P20/V/N/ym1e/1PkSVLcn7zxvy0/OccL1dOSEry8e1PS7i8Y+YBsmKF8uxJ02Lcs/9g3uiuuSIjSnB+o3osXL0egBc//JyYuCOMHNgnNU/JEuGpXfIO5zYhyecjJu4oFcuVpWK5sqmtzy4XnMtv23fmfiWyoMelHfl2wUIAKlWowIWtW1EivDhly5SmVbMmbNq6jSUrV1OtciWiypahSFgYl3Zox+p1v4W45BT4LnaGAVJVk4E73a/jVfVIICdV1YOq+oqqdgRuBMoA60VkwBmW94xEl4ti+Zp1ACz9eS1nVz31eu8jR+NZsWYdndudekDo0OFY4o4eBeDEyZMsWbWGWtWr5W6hA7Rk1S/UrF6NSu6obkaiy0URUSKcX9ZvRFWZPff70+odCodijxDn3h8+cTKBJWs3UrNqJT76biELf/mNSfcMwpOmFbI/JpaUAca1m7ejmkyZUhFUKFuayuXKsn33HgCW/rqROtUq536FMrHjz1Ot4+8XLqVmdWdthc4XtmHVmnXufcYTrP1tI7XOrk7l6GjWrN/A8RMnUFWWrPqFWmdXD1XxT/F4A//kQ4Hcg/xWRO4HPgDiUxJV9VBmB4rIuUA/4FLgazLoXue0e8c/zYpf1hETG0eHPjcx7MZ+jL//Th5//jV8Ph/Fihbl0fuGpub/duFS2rVqQQm3VQKw/+AhRk58Dl9yMpqsdLv4Qjq2OS+3qpCuex99ihW//OrUq/dAht3Un97du/Dl9z/SI53Wb6drbyb+2DESE5OYt3Ap0yaNp06N6owdfgejJ07mREICF53f8oxGsHPC/phYRr3wFr7kZJJV6da2JR1bNaXxNbdTpUIU/cY8CZyazvO/pT8zY+4CwrxeihUtwjP33Jo6TWnMLX15YMo0EpN8nFWxPI8PHZjRpYPu3nETWLF6LTGxsXS4qj/Dbh7AgqXL2bFzFyIeqlSK5pH7ndvytWtU56LWreh54xA8HqF3j27Uq1UDgC4XX8RVtwwlzOulQd06XHtleiuB5TJP/mwZBiqQaT7b00lWVa2VwTGP4KwjuQGYCXyTZsHLgNiz2PmLPYudP53xNJ9XRwc+zee2J/JdNA3kSZqameVJx3+AbUAz9/OE+9ddnFPaauTGFAj59N5ioAJ5kqY4cAfOqLQCPwGvqOqJDA67D1iI84KdxBwopzEmL8qno9OBCuQe5NvAESBlTkw/nLUh+/g9wnn/wxSgPrAWWAwsApYEcu/SGJNPFPYWJHCOqjZLsz1fRNZkdICq3g/gvr+2Fc6y6TcDr4nIYVVtmN0CG2PykHw6Oh2oQNrHq0XkgpQNEWmN0xoMRDgQifOuiNI4E8eXZbWQxpg8qoBPFA+kBdkauEFEUmbbVgc2pLwONr0BFxGZivPS7iM4AXEx8KyqZvwgrTEmf7EuNt2ycd7qQDFgM7Ab5y1jh7NxHmNMXlZAp62lCGSaT5YfPlbVbu5byRrh3H+8D2gsIodwBmrGZrmkxpi8p4BPFA+kBZkt7usX14nIYSDW/fQAzgcsQBpTEBTwQZqgBEgRuQun5dgOZx7kImAJ8AbwazCuaYwJgcLexc6mGsDHwHBVzXDtSGNMPmZd7KxT1XuDcV5jTB5jo9jGGOOHdbGNMcYP62IbY4wfNoptjDF+WBfbGGP8sC62Mcb4YS1IY4zxw6b5GGOMH/l0GbNAWYA0xmSfjWIbY4wf1sUODSlTKdRFCI5MXrObX0lkhVAXIXiSfaEuQd5lXWxjjPHDWpDGGOOHTfMxxhg/bJDGGGP8KOBP0hTs9rExJrjEE/gn0FOKeEVktYh84W7XFJFlIrJZRD4QkaJuejF3e4u7v0aac4xy0zeJSNfsVs8CpDEm+0QC/wTubmBDmu0ngcmqWheIAW5x028BYlS1DjDZzYeINAT64rw0sBvwkohk616ABUhjTPblcAtSRKoB3YHX3W0BOuG8wgVgOtDL/bqnu427v7ObvycwU1VPqup2YAvOywKzzAKkMSbbRCQrn8EisjLNZ3A6p3wOeBBIdrfLAYdVNcnd3gVUdb+uCvwJ4O6PdfOnpqdzTJbYII0xJvs8gYcQVZ0KTPW3X0R6APtUdZWIXJySnN6pMtmX0TFZYgHSGJN9OTuK3Q64UkQuB4oDkTgtyjIiEua2EqsBf7n5dwFnAbtEJAwoDRxKk54i7TFZYl1sY0z25eA9SFUdparVVLUGziDL96raH5gP9HazDQRmu1/Pcbdx93+vquqm93VHuWsCdYHl2ametSCNMdmXO48ajgBmishjwGpgmps+DXhHRLbgtBz7AqjqehH5EPgNSAKGqmq2HqgXzauLJxyLzaMFO0N59f/b+FeQF6soVe6MIlzyqrkB/0B7WnbNd7PKrQVpjMk+W6zCGGP88Nqz2MYYkz5bzccYY/ywLrYxxvhhLUhjjPHDWpDGGOOHt2CHkIJdO2NMUIm1II0xxg+7B1nwdLq8JxERJfB4PHi9Xj59/22+/vY7XnjlNbZu38FH77xJk0YNQ13MTI0aN54fflpEuaiyfPHRDACef+U1Ppw1m6iyZQC4987b6XBhOxISExn72ATWbdiIiDDmgXtp3aplKIufofTqlmLa2+/y1HPPs2TeXKLKluG7HxYw5aWpeDyC1+tl9P3DadWieYhKnrGTJ0/S/9Y7SEhMxOfz0bVzR+66bRAjxz3G8p9XU6pkSQAmjh1Dg3PqsXXHDkY/8jjrN/7O8Dtu45YB14W4Bv9gLciCafrUl1ODCEC92rV5/pmnGPvYhBCWKmuuuqIH11/bhxEPP3Ja+o39+3LLDdeflvbRp58B8PmH73Pw0CFuvfMePn73LTx59L3G/ur29569LF66nCqVTr03vc3559G5Q3tEhI2/b+aekWP45tMPc7vIASlatCjTX3meiBIlSExK4rpbhtC+7QUAPHjXULpd0um0/GUiIxlz/3Dm/fBjKIqbuQLegizYtcuC2rVqUqvG2aEuRpac17IFpUtHBpR3y7btXHD+eQCUi4qiVKlSrPttQyZHhY6/uk14ZjIP3HPnafe+IkqUSN0+fvwEku5ygHmDiBBRogQASUlJJCUlZXgfr1xUFE0bNSQsLI+2ZYLzyoU8IygBUkR6iUh0MM6dIwRuuWMYV113Ax98MivUpclx733wMVdc059R48YTGxcHQP16dZm34EeSkpL4c/dfrN+wkb/37g1xSbNm3oIfiY6uQP169f6179vvf6DbVddw29338sTYh0JQusD5fD56XjeQtpd2p23r82jWuBEAk1+ayhV9B/DEM1NISEgIcSkD5PUG/smHgtWCvB5Y7b6F7C13qfVGQbpWls1483VmzXiH1154jvc++IgVq34OdZFyTL8+V/HtnE+YPfMdosuXZ+KzUwC4uucVVIqO5urrb+SJSc/SolkTvPnoh/b48RO8Mu0t7h5yW7r7L+10Md98+iEvPvMUU15+NXcLl0Ver5fZ709nwVefsXb9Bn7fspV77xzCN5/M4JO3pxEbF8fU6e+GupiBCcJbDfOSoJRaVXuralXgUuB/QFPgbRHZLyJf+Tsu7Tsrpr7xVjCKBkDF6AqA0325tNPFrF3/W9CuldvKlyuH1+vF4/HQ56qe/OrWLSwsjNH3D2f2zHd5efIkjhw5So3qZ2Vytrxj565d7Nr9Fz37Xk+n7r3Ys28fV/W/gf0HDp6W77yWLdi5axeHYg6HqKSBiyxVitYtW/DTkmVEly+PiFC0aFGuuqJ76vctzyvgXeyg3thQ1R0iUhwIdz8pX/vLf+qdFUFaD/LY8eMkJydTMiKCY8ePs2jJMu4YPCgYlwqJffsPEF2hPADffb+AurVrAU4LTFFKhIezaOkyvF4vdWrVCmVRs+ScunVYMu+b1O1O3Xvx8btvEVW2DH/s/JPqZ1VDRFi/YSOJiUmULVM6hKX171BMDGFhYUSWKsWJEydZvHwltw68nn0HDhBdvjyqyncLfkz9vuV9+TPwBSooAVJERgNtgArAJmAp8AIwOLsr++aUgwcPMfTeBwDnXlCPy7rSvl0bvv1+PuOffIZDMTHcdte9NDinLtNeej6URc3UvaMeYvmqn4k5fJj23XowbMhglq9cxcbfNwNC1SqVeXTMSAAOxhzilqF34xEPFaMr8NT4cSEte2bSq1ufXlemm3fu9/OZ/cVXhIWFUbxYMSZPfCzPTmDed+AgI8eOx5ecjCYn0+3SznS8qB03DLmTmJjDqCr1z6nLI6MeBGD/gYNcfcPNHI2PxyMeps/4gK8+fJ+SJSNCXBNXHv1/zilBWVFcRDYCR4EvgMXAMlWNzdJJbEVxk1fYiuJ+6a4NAf9AS7UG+S6aBqUFqar1RSQKaAtcDIwUkZLAGmCxqr4ZjOsaY3JZPh18CVTQ30njvo6xJdAeuA2oqaqZD59aC9LkFdaC9Ev/2hR4C7LKOdaCBBCRK3Faj+2ARsB6YAlwH06X2xhTIOS7mJclwRrF/gznVY2jgOWqmk9mvRpjsqSAD9IEK0BOBv4PJ0CuFZHFwCJgiaoeCtI1jTG5rYAHyKDegxSRokArnO52G/dzWFUzXyrH7kGavMLuQfqle7cFfg+yYq18F02D/QR8OBAJlHY/fwG/Bvmaxphck+9iXpYEa5BmKs7gzBFgGc7AzLOqGhOM6xljQqSAd7GD1YKsDhQDNgO7gV1A3n841hiTNRYgs05Vu4nzrFcjnPuP9wGNReQQzkDN2GBc1xiT2yxAZos6oz/rROQwEOt+egDnAxYgjSkA8uoz7zklWPcg7+LURPFE3Ck+wBvYII0xBUcBf9QwWC3IGsDHwHBV/TtI1zDGhJq1ILNOVe8NxnmNMXmMBUhjjPHHAqQxxqTPWpDGGONHwY6PFiCNMWfARrGNMcYP62IbY4w/FiCNMSZ91oI0xhg/LEAaY4wfBXyQJuhvNcwPRGSwqk4NdTmCoaDWzeplckPBDv+BGxzqAgRRQa2b1csEnQVIY4zxwwKkMcb4YQHSUZDv+RTUulm9TNDZII0xxvhhLUhjjPGj0ARIEVEReSfNdpiI7BeRL9ztG908ndPk+T83rXcoypyZjOokjgMiUtbdV9nNf2Ga/PtFpFwoyp4REZksIvek2Z4rIq+n2X5GRO516zM+TXp5EUkUkRdyu8yBCLBeKiK90qRtEpGH0mx/IiJX5V6pC7dCEyCBeJw3K4a725fivJI2rV+Bfmm2+wJrcqFs2eW3Tu5L05YBbdx9bYHV7r+IyDnAAVU9mKslDsxiTpXTA5THeUNmirY47znahvMiuBR9gPW5VMbsCKReI9PkKQcc5dT3EPfrxblRWFO4AiTA10B39+t+wIx/7P8JOF9EiohISaAO8Esuli87MqrTItxfNvffZzk9YObVX7S05W4ErAOOiEhZESkGNABigOPABhFp5ea9FvgwtwubBYHUawmnf8++ACq4PYKawHFV3ZPL5S60CluAnAn0FZHiQFOcFlZaCnwHdAV6AnNyt3jZklGdUlssOK/b/Qw4y91OaYXlOar6F5AkItVxyrmEU63hVsBaIMHNnlL/aoAP+Cv3SxyYAOu1DKdXUDRNnk04wTPPfs8KqkIVIFV1Lc4bF/sBX/nJNhOna92Xf7cw85xM6rQcaCEiEUARVT0KbBOROuTtFiScam2lBIklabbTlvsbnFsL/YAPcrmM2ZFhvVT1JM5tgnOBC3ACpr+6myArVAHSNQeYhJ/gp6rLgcZAeVX9PTcLdgbSrZOqHgO2ADcDP7vJS4HLgWiclkleldL6bYLTFV2K09I6rRWlqgnAKuA+4JPcL2aWBVKvxUB7oJSqxrh52mItyFxXGAPkG8CjqvprBnlGAaNzqTw5IaM6LQLuwWmF4P57N7BU8/Yk2EU4AzCHVNWnqoeAMjjBZMk/8j4DjMijA07/FEi9FgG3cWqAcC1Oa7I6eXsQqsApdAFSVXep6pRM8nytqvNzq0xnKpM6LQJqceqX72egGnm/q/Yrzijv0n+kxarqgbQZVXW9qk7PzcKdgUDqtZg03zNVTQL2AStVNTkXy1ro2ZM0xhjjR6FrQRpjTKAsQBpjjB8WII0xxg8LkMYY44cFSGOM8cMCZCEiImVE5I4gnv/GzFbSEZFxInJ/Fs979MxKZkz2WIAsXMoA6QZIEfHmclmMyfMsQBYuE4HaIvKLiDwtIheLyHwReR/4VURqiMi6lMwicr+IjHO/ri0i34jIKhH5SUTqZ3QhEblCRJaJyGoR+U5EKqbZ3UxEvheRzSJya5pjHhCRFSKyVkQeydmqG5N1YaEugMlVI4HGqtocQEQuxlnlp7GqbheRGhkcOxUYoqqbRaQ18BLQKYP8C4ELVFVFZBDwIM7z0uCsOnQBEAGsFpEvcZ5/r+uWR4A5ItJeVX/MVk2NyQEWIM1yVd2eUQZ3bcy2wEcikpJcLJPzVgM+EJHKQFEg7TVmq+px4LiIzMcJihcCXXAW9QUoiRMwLUCakLEAaeLTfJ3E6bddirv/eoDDKS3PAD0PPKuqc9yW6rg0+/75fKvitBonqOqrWbiGMUFl9yALlyNAqQz27wWiRaScu8J1DwBVjQO2i0gfAHd162aZXKs0p15pMfAf+3qKSHH3lQIXAyuAucDNbmsVEakqItGBV82YnGctyEJEVQ+KyCJ3IOZr4Mt/7E8UkUdxFmndDmxMs7s/8LL7AqkiOAsLZ/S+nnE4XfLdOCvX1Eyzb7l77erAeHel7b9EpAGwxO3GHwWux1nFxpiQsNV8jDHGD+tiG2OMHxYgjTHGDwuQxhjjhwVIY4zxwwKkMcb4YQHSGGP8sABpjDF+WIA0xhg//h9l2UZLkFFPcQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.47203150415204176\n"
     ]
    }
   ],
   "source": [
    "# confusion matrix\n",
    "mat = confusion_matrix(y_test, y_pred)\n",
    "\n",
    "labels = sorted(set(y_test))\n",
    "\n",
    "# plot it\n",
    "sns.heatmap(mat.T, square=True, annot=True, fmt='d', cmap=\"Reds\", xticklabels=labels, yticklabels=labels)\n",
    "plt.xlabel('true label')\n",
    "plt.ylabel('predicted label')\n",
    "\n",
    "# save\n",
    "plt.savefig('images/predict_2genders_cm.png')\n",
    "\n",
    "# show\n",
    "plt.show()\n",
    "\n",
    "print('accuracy:', accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "47.2% - which doesn't sound good, but is an okay improvement from our baseline (about 9 points higher). And actually, before dropping duplicates, accuracy was only 44.8%. It looks like the easiest to identify is male poster/male responder.\n",
    "\n",
    "## Predict poster gender\n",
    "\n",
    "What if we just predict the gender they're responding to, and NOT try to also predict their own gender? Let's try to just predict the op_gender column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "M    0.623301\n",
       "W    0.376699\n",
       "Name: op_gender, dtype: float64"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# what is our baseline?\n",
    "responder_df.op_gender.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So our baseline is around 62.3% here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "233616\n",
      "58405\n"
     ]
    }
   ],
   "source": [
    "X = list(responder_df.response_text)\n",
    "y = list(responder_df.op_gender)\n",
    "\n",
    "# create train test split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\n",
    "\n",
    "print(len(y_train))\n",
    "print(len(y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use same parameters\n",
    "model = make_pipeline(TfidfVectorizer(max_df=0.5, min_df=5, ngram_range=(1,2), tokenizer=nltk.word_tokenize),\n",
    "                      MultinomialNB())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fit and predict\n",
    "model.fit(X_train, y_train)\n",
    "y_pred = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAEKCAYAAABwhMxUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmcVmX9//HXe2ZYY1c2AQMRU9zNBbVMSQ1XcEvNjG+ZfC3NLDVzx902/WlfNSlRtAWXCnEPFUUNl0QFlVQEEwQxlU0WgeHz++OcoVuaM3Nzwz0z9z3vZ4/zmPtcZ7k/p3E+XNe5zrkuRQRmZvbfKho7ADOzpsoJ0swsgxOkmVkGJ0gzswxOkGZmGZwgzcwyOEGamWVwgjQzy+AEaWaWoaqxA8hyijr4FZ8ScsNtZzd2CFaAyuEXakOOX5+/09/E4jq/S1JrYBLQiiQ33RMRF0vqB4wFugBTgBMjYqWkVsDtwBeBj4BjI+Kd9FznAicB1cDpEfFIWj4EuA6oBH4XEVfXFZNrkGbWVHwKDI6IHYGdgCGSBgE/A66NiAHAApLER/pzQURsCVyb7oekgcBxwLbAEOBGSZWSKoEbgIOAgcDx6b6ZnCDNrGAV67HUJxKfpKst0iWAwcA9afkYYFj6eWi6Trr9q5KUlo+NiE8jYhYwA9g9XWZExMyIWElSKx1a3/WZmRWkSsp7yUda03sZ+ACYALwNLIyI1ekuc4Be6edewGyAdPsiYJPc8nWOySrP5ARpZgWrUP6LpBGS/pGzjFj3fBFRHRE7Ab1Janzb1PK1Nfc9a8u6UUB5pibbSWNmTd/61LAiYhQwKs99F0p6AhgEdJJUldYSewNz093mAH2AOZKqgI7AxznlNXKPySqvlWuQZlawCinvpT6SukrqlH5uA+wPTAcmAkenuw0H7k0/j0/XSbc/HskAt+OB4yS1SnvABwDPAy8AAyT1k9SSpCNnfF0xuQZpZgXbyDWsnsCYtLe5ArgrIu6X9DowVtLlwEvALen+twB3SJpBUnM8DiAiXpN0F/A6sBo4NSKqASSdBjxC8pjP6Ih4ra6AnCDNrGAVG/QU5WdFxFRg51rKZ5Lcj1y3fAVwTMa5rgCuqKX8QeDBfGNygjSzglXm2Ttdqpwgzaxg5d6J4QRpZgXbmE3spsgJ0swK5hqkmVmGfB7fKWVOkGZWsKryzo9OkGZWODexzcwyVNT6enP5cII0s4K5F9vMLIOb2GZmGVyDNDPLkO9AuKXKCdLMCuYmtplZBjexzcwy+DEfM7MMrkGamWWodII0M6udm9hmZhncxDYzy+DHfMzMMpR5BdIJ0swK5wFzzcwyuIltZpahvOuPTpBmtgHkJraZWe3KOz06QZrZBvA9SDOzDGXewnaCNLPC+VVDM7MM5Z0enSDNbAP4XWwzswwq8zqkE6SZFay806MTpJltADexzcwyuBfbzCxDeafH8n8Q3syKSMp/qf9c6iNpoqTpkl6T9MO0fKSk9yS9nC4H5xxzrqQZkt6Q9LWc8iFp2QxJP80p7yfpOUlvSbpTUsu6YnKCNLOCaT2WPKwGzoyIbYBBwKmSBqbbro2IndLlQYB023HAtsAQ4EZJlZIqgRuAg4CBwPE55/lZeq4BwALgpLoCcoI0s4JpPf5Xn4iYFxFT0s9LgOlArzoOGQqMjYhPI2IWMAPYPV1mRMTMiFgJjAWGKhl6aDBwT3r8GGBYXTE5QZpZwSqV/7I+JPUFdgaeS4tOkzRV0mhJndOyXsDsnMPmpGVZ5ZsACyNi9TrlmZwgzaxg69PEljRC0j9ylhG1nlNqB/wZOCMiFgM3Af2BnYB5wK9yvn5dUUB5JvdiF6iqVSvOmvQwVa1aUlFVxZR77uX+kVeySd/P892xt/K5Lp15d8rL3HriCKpXrVp73C5HDWXEPXdw5a5f4d0XX2Kb/fdj2NUjqWrZktUrV/KXsy/kjYmTaNGmDSPuvp2u/fuxprqaqfc9xLhzRzbeBZeB8++fzJMz5tClbWvGjzjsM9tGP/s6v3x8Cs+ccTSd27bmvldnccvk1wBo27KKi4bswdbdk4rLU2/P5aoJL1AdwdE7bsnJe20HwJyFn3DmuKdYtHwlA3t04erD96JlZWXDXmQDW583aSJiFDCqzvNJLUiS4x8i4i/pcfNztv8WuD9dnQP0yTm8NzA3/Vxb+YdAJ0lVaS0yd/9auQZZoNWffsq1gw/l8p325vKd9mbbIfvTb4/dOPJnl/DYtTdw0VY7s2zBQvY+6Vtrj2nVrh37nX4KM599YW3ZJx9+xI2HHctlO+zJmOGn8O07/vPfz4RfXs/IbXblip2/RP+9B7HtkAMa9BrLzRE7bMGo4wb/V/m8xUuZPGsePTt8bm1Z707tGPPNAxh38qGc8qXtufihZwGoXrOGyx95npuPHcx9Iw7jwdffYca/FwLwq8enMHy3bXj4e0Pp0Lolf3n57Ya5sEa0kXuxBdwCTI+Ia3LKe+bsdgTwavp5PHCcpFaS+gEDgOeBF4ABaY91S5KOnPEREcBE4Oj0+OHAvXXF5AS5AT5duhSAyhYtqGxRRUTwhcFfYco94wCYPOZP7Djs0LX7H37ZBfzt59exesWKtWWzX57KonnvAzD3telUtW5NVcuWrFq+nDefeAqA6lWrmD3lFTr33qyhLq0s7bp5dzq2bvVf5T+b8CJnDt7lM3/EO/fuSsc2yb47brYp8xcvA2Da3I/YvHN7+nRuT8vKSg4a2JfH35pDRPDcv+Zz4DabAzBs+y147M3Z//Vd5aZiPZY87A2cCAxe55Gen0uaJmkqsB/wI4CIeA24C3gdeBg4NSKq09rhacAjJB09d6X7ApwD/FjSDJJ7krfUFZCb2BtAFRWc9+Ikum65BU/e8Fv+/fZMli1cxJrqagAWznmPTr2Sf/z67LQDnfv0YtoDD3PAWT+o9Xy7HDWU2S+9wuqVKz9T3qZjR7Y/bAiPX3dTcS+oGXr8zdl0a99mbfO5Nn9+5W2+3D/5x2n+kmX06NB27bYe7dsyde6HLFz+Ke1bt6CqIkkF3Tu0Zf6SZcUNvgnYmA+KR8TTGad8sI5jrgCuqKX8wdqOi4iZJL3ceSlKgpQ0vq7tEXF4Mb63ocWaNVyx85do07Ejp/z1D/Tc5gu17BRI4phrr2LM/3wv81w9B27NET+7lOsO/OxTBxWVlZz0p9FMvP5mPpz1zka+guZt+arV3Pz3V/ndcV/N3Oe5d97nL6/M4PcnJs8g135HX0QtG8p9QivwvNiF2pOkm/1PJN30ef2/mPZqjQD4Mq0YSJ0PuTcZyxct4s0nnqbfoN1o26kjFZWVrKmuplPvXiyc+z6t2rdns+0G8uMnHgCgQ4/ufH/8WG48/DjeffElOvXajFP++kdu+9YIPpw56zPnPmHU9Xzw1ts8ft2NjXFpZW32giW8t/ATjrgl+b3MX7yMo0Y/yJ3/cxBd27XhjQ8WcNGDz3LzsYPp1DZpbvdo35b3F/+nZvj+kmV0a9+Gzm1bsWTFKlavWUNVRQXzFy+jW7s2jXJdDam802Px7kH2AM4DtgOuAw4APoyIJyPiyayDImJUROwaEbs29eTYbtNNaNOxIwAtWrdm6/335f3pb/LGxEnscnRSC9xz+PFMvfcBVixezFld+3F+v+05v9/2zHr2hbXJsU3Hjpz2wN2MO3ckb//9uc98x+GXXUibjh24+4xzGvz6moOtunXm6TOO4dFTj+DRU4+ge4e2/Pk7B9O1XRvmLlrK6X9+kqsP35u+m3RYe8x2m23CvxYsYc7CT1hZXc1Dr7/DfgN6I4ndP9+dv01/F4Bx02YyeKvejXVpDUZS3kspKkoNMiKqSW6aPiypFXA88ISkSyPi18X4zobWsWcPho/5DRWVlaiighfv+ivTHniYea//k++OvZXDL7+Q2S+9wjO33F7nefY9bQRdt9yCgy/8CQdf+BMArj9wGJUtW3LwBWczb/obnDcl6ax54v9G1Xs+y3bWuKd4/l/zWbj8U/b79V847cs7cNROW9a6701PT2XR8pVc+vDzAFRViLu/czBVFRWcf+BunDz2MdasCY7YsT8DunYC4Mz9duascU9z3aSX2aZ7F47asfZzl5NyH+5MUdvNk41x4iQxHkKSHPuSdMmPjoj38jn+FHUoTmBWFDfcdnZjh2AFqBx+4QaluJf79M3773Sn2e+UXDotVifNGJLm9UPAJRHxaj2HmFkJqijzBwWL1UlzIrAU2Ao4Pef+g4CIiA5ZB5pZ6SjVe4v5KtY9yDL/d8XMIL83ZEqZHxQ3s4K5BmlmlqHM86MTpJkVzm/SmJllqCjzByGdIM2sYCrz7lgnSDMrmDtpzMwylHl+dII0s8K5BmlmlqHM86MTpJkVrtK92GZmtWu2TWxJXeo6MCI+3vjhmFkpKfP8WGcN8kXqnmx7i6JEZGYlo9kmyIjo15CBmFnpUXO/B5lO5n0C0C8iLpO0OdAjIp4venRm1qSVeydNPi8K3UgyS+E30vUlwA1Fi8jMSoaU/1KK8unF3iMidpH0EkBELJDUtKccNLMG0Wx7sXOsklRJOme6pK7AmqJGZWYloczzY14J8nrgr0B3SVcARwMXFDUqMysJzb4GGRF/kPQi8NW0aFhETC9uWGZWCso8P+b9Jk1boKaZ3aZ44ZhZKamoLO8MWW8vtqSLgDFAF2BT4FZJbmKbGZLyXkpRPjXI44GdI2IFgKSrgSnA5cUMzMxKQJk/B5lPgnwHaA2sSNdbAW8XKyAzKyElWjPMV12DVfya5J7jp8Brkiak6wcATzdMeGbWlJVq0zlfddUg/5H+fJHkMZ8aTxQtGjMrLZXlPWtXXYNVjGnIQMys9HiwCmkAcBUwkOReJAAR4eHOzJq7Mm9i51M/vhW4CVgN7AfcDtxRzKDMrDSoQnkv9Z5L6iNpoqTpkl6T9MO0vIukCZLeSn92Tssl6XpJMyRNlbRLzrmGp/u/JWl4TvkXJU1Lj7le9dxEzSdBtomIxwBFxL8iYiQwOI/jzKzcbdzhfFYDZ0bENsAg4FRJA4GfAo9FxADgsXQd4CBgQLqMIKnI1cyGcDGwB7A7cHFNUk33GZFz3JC6AsonQa6QVAG8Jek0SUcA3fK5WjMrcxXKf6lHRMyLiCnp5yXAdKAXMJTkZRXSn8PSz0OB2yPxLNBJUk/ga8CEiPg4IhYAE4Ah6bYOETE5IoKkNVxzrlrl8xzkGSSvGp4OXEZSexxe5xFm1iyoSL3YkvoCOwPPAd0jYh4kSVRSTQWtFzA757A5aVld5XNqKc+Uz2AVL6QfPwG+Xd/+ZtaMrEcnjaQRJM3bGqMiYlQt+7UD/gycERGL67hNmDVf1vqWZ6rrQfH76jo4Ig6v68RmVv60HhXINBn+V0L8zPmkFiTJ8Q8R8Ze0eL6knmntsSfwQVo+B+iTc3hvYG5avu865U+k5b1r2T9TXTXIX9Z1oJnZxnzMJ+1RvgWYHhHX5GwaT3Jb7+r057055adJGkvSIbMoTaKPAFfmdMwcCJwbER9LWiJpEEnT/VvAr+uKqa4HxZ9c7ys0s2ZlIz8ovjdwIjBN0stp2XkkifEuSScB7wLHpNseBA4GZgDLSG8BponwMqDm9uClEfFx+vl7wG0kwzY+lC6Z8h0P0szsv23EGmREPE3t9wnhPwN25+4fwKkZ5xoNjK6l/B/AdvnG5ARpZgUrVi92U+EEaWaFa67vYrsX28zqVebvYufTi30k0AP4fbp+PMkgumbWzDXb8SBrerElXRYR++Rsuk/SpKJHZmZNX3NtYufoKmmLiJgJIKkf0LW4YZlZKXAnDfwIeELSzHS9L/C/RYvIzEpHc21i14iIh9NBc7dOi/4ZEZ8WNywzKwXlPqJ4PvNitwXOBk6LiFeAzSUdWvTIzKzp27jjQTY5+Y4ovhLYM12fg+fENjPYqONBNkX53IPsHxHHSjoeICKW1zdM+cbwm6Wz69/JmoxYuqixQ7BG0Gwf88mxUlIb0ofGJfUnmSvbzJo792IzEngY6CPpDyQjbnjgXDMr2XuL+cqnF/tvkl4kmURHwA8j4sOiR2ZmTV9zT5CSHouIrwIP1FJmZs1ZRTNtYktqTTJZ16bpyLw1/1R0ADZrgNjMrKlrxjXI/yWZ0XAz4EX+kyAXAzcUOS4zKwXNNUFGxHXAdZJ+EBF1zttgZs1UZWVjR1BU+dxAWCOpU82KpM6Svl/EmMysVPhNGk6OiIU1KxGxADi5eCGZWcko8wSZz3OQFZKUTpCDpEqgZXHDMrOSUKKJL1/5JMhHSKZc/A3J2zSnkDw4bmbNXXN9zCfHOSQ92t8j6cn+G/C7YgZlZiWiuSfIiFgD3JQuZmb/0Vyb2JLuioivS5pGLbMbRsQORY3MzJo8NeMa5A/Tnx4c18xq11xrkBExL/35r4YLx8xKSnNNkJKWUEvTukZEdChKRGZWOpprgoyI9gCSLgXeB+4g6cU+AWjfINGZWdNW5q8a5vOYz9ciYo+c9ZskPQf8vEgxmVmpKPMaZD5dUNWSTpBUKalC0glAdbEDM7MSUOavGuaTIL8BfB2Yny7HpGVm1txVVOS/lKB8HhR/Bxha/FDMrOSUaM0wX/WmdUlbSXpM0qvp+g6SLih+aGbW5LmJzW+Bc4FVABExFTiumEGZWYmorMx/KUH59GK3jYjn15kgfHWR4jGzUlKiNcN85VOD/FBSf9KHxiUdDcwralRmVho2YhNb0mhJH9TczkvLRkp6T9LL6XJwzrZzJc2Q9Iakr+WUD0nLZkj6aU55P0nPSXpL0p2S6h3XNp8EeSpwM7C1pPdIJvI6JY/jzKzcbdxe7NuAIbWUXxsRO6XLgwCSBpLc6ts2PebG9FHESpJJBQ8CBgLHp/sC/Cw91wBgAXBSvZdX10ZJFcCuEbE/0BXYOiK+5PezzQzYqDXIiJgEfJznNw8FxkbEpxExC5gB7J4uMyJiZkSsBMYCQ5XcIxwM3JMePwYYVt+X1Jkg07EgT0s/L42IJXkGb2bNQUVl3oukEZL+kbOMyPNbTpM0NW2Cd07LegGzc/aZk5ZllW8CLIyI1euU1315eQQ3QdJZkvpI6lKz5HGcmZW7CuW9RMSoiNg1ZxmVxzfcBPQHdiLp+/hVWl5blTQKKK9TPr3Y30l/nrrOibfI41gzK2cq7hsyETF/7VdJvwXuT1fnAH1ydu0NzE0/11b+IdBJUlVai8zdP1M+b9L0q28fM2umivyYj6SeNWPTAkcANT3c44E/SroG2AwYADxPUlMcIKkf8B5JR843IiIkTQSOJrkvORy4t77vrzdBSmoNfB/4EknN8SngNxGxIu+rNLPytBHfsZb0J2BfYFNJc4CLgX0l7USSe94hmUCQiHhN0l3A6yTPZZ8aEdXpeU4jmY21EhgdEa+lX3EOMFbS5cBLwC31xpROd11X0HcBS4Dfp0XHA50j4pj8LrtAyxbVe3/Amo5YuqixQ7ACqOvmG1QFrL7lorz/TitPurTknirP5x7kFyJix5z1iZJeKVZAZlZCKkrzFcJ85VM/fknSoJoVSXsAzxQvJDMrGc19uDNgD+Bbkt5N1zcHptdMB+vpX82asTJ/FzufBFnbqz9mZkV/zKex5fOYj18rNLPaVbgGaWZWuzLvpHGCNLPCNfcmtplZJjexzcwyuBfbzCyDm9hmZhncxDYzy+BebDOzDG5im5llcBPbzCyDa5CWj8VLlnDBJVfw5ttvI4krL76Afn0/z4/OOZ/35s6j12Y9+X8/v5KOHTqwaPFizht5Ge/OeY9WLVty5cgL2WrL/mvPVV1dzVEnDKd7t67cfP21jXhV5Wnmu7P58UWXr12fPfd9Tv/ucIZ//UjuuGccf/jzvVRVVvKVvfbg7O+fzKrVq7ng6mt4/c23qK6uZuiQA/jfE48H4Pa7/sLd9z1ERHDM4Qcz/OtHNtZlNQ4/5mP5uOLnv+LLew3i+l9ezcpVq1ixYgW/ueVW9tx9N0Z8ZzijRo9h1K1jOPuHP+A3t9zGNl/Yihuu+QVvz3qHS6/+OWNuvnHtuW7/41j69+vLJ0uXNt4FlbEtNu/DuNtuBpJ/jL5yxPHsv8/ePDvlZR5/6u+MH3MzLVu25KMFCwB4+PFJrFq1ivtu/y3LV6zgkG9+l0P2349ly5dz930Pcddvf02LqhacfOa5fGXP3enbp3djXl7DKtFhzPJVlKuTdIak3SQ1iwT8ySef8MKUlzj6iKEAtGzRgg7t2/PYE5MYdtghAAw77BAenfgkAG/PnMWg3XcDoH+/vrw3dx4ffvQRAO/Pn88TTz+z9lxWXJNffIk+vXrSq0d3xv71Pk7+5nG0bNkSgE06JzOMSrBs+QpWr65mxacraVFVRbvPtWXmO++y47Zb06Z1a6qqKtlt5x14dFIzGyp1PaZ9LUXFSv+9geuADyQ9IelKSYeU63Sxs9+bS5fOnTn34ksZdtw3Of+Sy1m2fDkfffQx3bpuCkC3rpvy8cdJjWTrrQYw4bGJAEx99TXmznuf9+d/AMCVv7iWs3/4AyrK/F/mpuLBR5/gkP33A+Cd2XP4x9RpfP3kH/DN037MtOlvAPC1/fahbZvWfHnYsQw+6gS+c/wxdOrQgQFb9OWFl6exYNFilq9YwZOTn2feB/9uzMtpeFL+Swkqyl9hRJwVEXsBPYDzgI9Jpo99VdLrWcflTiw+avRtxQitKFavXs3r/3yD4485inFjf0+bNm0YNXpM5v4jvv0tFi9ZwtBjT+COsXexzRe2oqqykomTnqJLl85sN3CbBoy++Vq5ahWPPzOZIft9BYDq6jUsXvIJd466np98fwRnXHQ5EcG01/9JRUUFk8aN5dG7b+fWsfcw+7159O/7eU7+5rGc9KNzOPnM89h6yy2oqizNmlLBPKL4BmkDdAA6pstcYFrWzulE4slk4iU0aVeP7t3o0a0bO26/HQBD9h/MqFtvZ5NNuvDBvz+kW9dN+eDfH9KlS9Jka9euHVddchEAEcFXDxlG716b8cAjE3j8yaeY9PTf+XTlp3yydClnnX8Rv7zi0ka7tnL21LMvMHCrLdk0/b1077opB+zzJSSxw8CtqZBYsHAR9094nC/vsSstqqrYpHNndtl+W17955v06dWTow89iKMPPQiAa26+hR5duzbmJTW8Eq0Z5qtY9yBHSXoGuBPYE/g7cExE7BoR3y7GdzamrptuSo8e3Zj5TjK28OTnX6D/Fv0Y/JV9GHffAwCMu+8BvrrvPkDS471y1SoA7v7rvey6y060a9eOM08/lUmP3M/jD97LNVdfwaDddnVyLKIHHp24tnkNsP8+e/HclJcAmPXuHFatXk3nTh3p2b0bz055mYhg2fLlvPL6dLb4fDI3fU1Hztz3P2DCk8985nzNgiryX0pQsWqQmwOtgLdIJu+eAyws0nc1CReeczZnnXchq1avpk+vzbjqkotYs2YNZ5xzHveMG0/Pnt257udXAUknzTkXXkJFZQVbbtGPKy6+oJGjb36Wr1jBMy+8yCVnn7G27MhDhnD+Vb/isBNPpkWLKq4+/2wk8Y0jh3Lelb/gsBNPJgiOPPhrfGHLLQA4/fxLWbh4MVWVVVz049Po2KF9Y11S4yjRzpd81TsvdsEnlgRsC+yVLtuR3IucHBEX13uCEmpim+fFLlUbPC/2pDvznxd7n2NLrj1etHuQkWTeVyUtBBaly6HA7kD9CdLMmr4SbTrnqygJUtLpJLXGvYFVJPNoTwZGU0cnjZmVmDLvpClWDbIvcA/wo4iYV6TvMLPG5hrk+ouIHxfjvGbWtMg1SDOzDBXlnULK++rMrLg8HqSZWQbfgzQzy+B7kGZmGVyDNDPL4BqkmVmGMh/erbzrx2ZWXBtxNB9JoyV9IOnVnLIukiZIeiv92Tktl6TrJc2QNFXSLjnHDE/3f0vS8JzyL0qalh5zvfJ4iNMJ0swKt3FHFL8NGLJO2U+BxyJiAPBYug5wEDAgXUYANyXhqAvJWA97kI77UJNU031G5By37nf9FydIMyvcRqxBRsQkkhG/cg0FaobnHwMMyym/PRLPAp0k9QS+BkyIiI8jYgEwARiSbusQEZPTgXRuzzlXJt+DNLPCFb+TpnvNeA4RMU9St7S8FzA7Z785aVld5XNqKa+TE6SZFa4y/xQiaQRJE7fGqHSalULUlpmjgPI6OUGaWcHWZ7CKz8w5lb/5knqmtceewAdp+RygT85+vUnmvJoD7LtO+RNpee9a9q+T70GaWeGKPyfNeKCmJ3o4cG9O+bfS3uxBwKK0Kf4IcKCkzmnnzIHAI+m2JZIGpb3X38o5VybXIM2scBvxHqSkP5HU/jaVNIekN/pq4C5JJwHvAsekuz8IHAzMAJYB3waIiI8lXQa8kO53aUTUdPx8j6SnvA3wULrUHVOx5qTZYJ6TpqR4TprStKFz0sSMF/P+O9WWXyy5125cgzSzwvlVQzOzDGX+qqETpJkVzqP5mJllcBPbzCyLE6SZWe1cgzQzy+AEaWaWwZ00ZmYZyrsC6QRpZhuivDOkE6SZFc73IM3MMjhBmpllcCeNmVkW1yDNzGrnJraZWQYnSDOzLE6QZma1Wp9Ju0qRE6SZFc692GZmGVyDNDPL4ARpZpbFCdLMrHauQZqZZSjv/OgEaWYbwL3YZmYZ3MQ2M8viBGlmVjvXIM3MMjhBmpllKPNOGkVEY8fQ7EgaERGjGjsOy49/X81Xeaf/pmtEYwdg68W/r2bKCdLMLIMTpJlZBifIxuH7WaXFv69myp00ZmYZXIM0M8vgBNkAJIWkO3LWqyT9W9L9jRmX1U7StZLOyFl/RNLvctZ/JenHjROdNSQnyIaxFNhOUpt0/QDgvUaMx+r2d2AvAEkVwKbAtjnb9wKeaYS4rIE5QTach4BD0s/HA39qxFisbs+QJkiSxPgqsERSZ0mtgG2AlxorOGs4TpANZyxwnKTWwA7Ac40cj2WIiLnAakmbkyTKySS/rz2BXYGpEbGyEUO0BuJ3sRtIREyV1Jek9vhg40ZjeaipRe4FXAP0Sj8vImmCWzPgGmTDGg/8EjevS0HNfcjtSZrYz5LUIH3/sRlxgmxYo4FLI2JaYwdi9XoGOBT4OCKqI+JjoBNJkpz6ZG7TAAACUUlEQVTcqJFZg3GCbEARMScirmvsOCwv00h6r59dp2xRRHzYOCFZQ/ObNGZmGVyDNDPL4ARpZpbBCdLMLIMTpJlZBidIM7MMTpDNiKROkr5fxPP/j6T/q2efkZLOWs/zfrJhkZkVxgmyeekE1JogJVU2cCxmTZ4TZPNyNdBf0suSfiFpX0kTJf0RmCapr6RXa3aWdJakkenn/pIelvSipKckbV3XF0k6TNJzkl6S9Kik7jmbd5T0uKS3JJ2cc8zZkl6QNFXSJRv30s3WnweraF5+CmwXETsBSNoX2D0tm5UOppFlFHBKRLwlaQ/gRmBwHfs/DQyKiJD0XeAnwJnpth2AQcDngJckPQBsBwxI4xEwXtI+ETGpoCs12wicIO35iJhV1w6S2pEM0nC3pJriVvWctzdwp6SeQEsg9zvujYjlwHJJE0mS4peAA/nPOIvtSBKmE6Q1GidIW5rzeTWfve3SOv1ZASysqXnm6dfANRExPq2pjszZtu77rUFSa7wqIm5ej+8wKyrfg2xelgDt69g+H+gmaZN05OxDASJiMTBL0jEASuxYz3d15D/TSgxfZ9tQSa0lbQLsC7wAPAJ8J62tIqmXpG75X5rZxucaZDMSER9JeibtiHkIeGCd7askXUoyevYs4J85m08AbpJ0AdCCZIT0V+r4upEkTfL3SEbE6Zez7fn0uzcHLktH8J4raRtgctqM/wT4JvBBgZdrtsE8mo+ZWQY3sc3MMjhBmpllcII0M8vgBGlmlsEJ0swsgxOkmVkGJ0gzswxOkGZmGf4/pgX1HKESFlcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.6525297491653111\n"
     ]
    }
   ],
   "source": [
    "# confusion matrix\n",
    "mat = confusion_matrix(y_test, y_pred)\n",
    "\n",
    "labels = sorted(set(y_test))\n",
    "\n",
    "# plot it\n",
    "sns.heatmap(mat.T, square=True, annot=True, fmt='d', cmap=\"Reds\", xticklabels=labels, yticklabels=labels)\n",
    "plt.xlabel('true label')\n",
    "plt.ylabel('predicted label')\n",
    "\n",
    "# save\n",
    "plt.savefig('images/predict_poster_cm.png')\n",
    "\n",
    "# show\n",
    "plt.show()\n",
    "\n",
    "print('accuracy:', accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "65.3%... only a couple points higher than the baseline. This seems like a difficult task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}