exploratory_data_analysis/modifying_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modifying Corpus\n",
    "Katie Thomas  \n",
    "kdt13@pitt.edu\n",
    "\n",
    "Here I will modify the corpus to fit my own needs. I'm going to combine the post and reply files for each website, in order to look at it more cohesively.\n",
    "\n",
    "## Table of Contents\n",
    "1. [Cleaning Facebook Congress](#Facebook-Congress-posts-and-responses): cleaning up Facebook Congress posts and responses and merging them into one DataFrame\n",
    "2. [Cleaning Facebook Wiki](#Facebook-wiki-posts-and-responses): cleaning up Facebook Wiki posts and responses and merging them into one DataFrame\n",
    "3. [Cleaning Fitocracy](#Fitocracy-posts-and-responses): cleaning up Fitocracy posts and responses and merging them into one DataFrame\n",
    "4. [Cleaning Reddit](#Reddit-posts-and-responses): cleaning up Reddit posts and responses and merging them into one DataFrame\n",
    "5. [Cleaning TED](#TED-responses): cleaning up TED responses (there is no accompanying posts file)\n",
    "\n",
    "## Setting up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import things\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import nltk\n",
    "from scipy import stats\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set corpus directory\n",
    "root = \"/Users/katie/Documents/Data_Science/Gendered-Interaction-Online/private/rtgender/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Facebook Congress posts and responses\n",
    "First let's gather the posts and responses with the Facebook congress files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# first, facebook congress files\n",
    "fb_congress_posts = pd.read_csv(root + 'facebook_congress_posts.csv')\n",
    "fb_congress_responses = pd.read_csv(root + 'facebook_congress_responses.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>post_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>video</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>1</td>\n",
       "      <td>Roses are red...and so is Texas. Let's keep it...</td>\n",
       "      <td>video</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>2</td>\n",
       "      <td>#TBT to this classic video. #DonkeyWhisperer</td>\n",
       "      <td>video</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>3</td>\n",
       "      <td>Since President Donald J. Trump was sworn in o...</td>\n",
       "      <td>video</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>4</td>\n",
       "      <td>Remembering our 40th president today. LIKE to ...</td>\n",
       "      <td>video</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      op_id op_gender  post_id  \\\n",
       "0  57265377         M        0   \n",
       "1  57265377         M        1   \n",
       "2  57265377         M        2   \n",
       "3  57265377         M        3   \n",
       "4  57265377         M        4   \n",
       "\n",
       "                                           post_text post_type  \n",
       "0  Yesterday, my colleagues and I voted to protec...     video  \n",
       "1  Roses are red...and so is Texas. Let's keep it...     video  \n",
       "2       #TBT to this classic video. #DonkeyWhisperer     video  \n",
       "3  Since President Donald J. Trump was sworn in o...     video  \n",
       "4  Remembering our 40th president today. LIKE to ...     video  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fb_congress_posts.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the exploratory analysis file, we saw that there were some nulls in post_text. Let's replace this with an empty string. We still need these posts in here, as they may still have some replies to them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 548341 entries, 0 to 548340\n",
      "Data columns (total 5 columns):\n",
      "op_id        548341 non-null int64\n",
      "op_gender    548341 non-null object\n",
      "post_id      548341 non-null int64\n",
      "post_text    548341 non-null object\n",
      "post_type    548341 non-null object\n",
      "dtypes: int64(2), object(3)\n",
      "memory usage: 20.9+ MB\n"
     ]
    }
   ],
   "source": [
    "fb_congress_posts.post_text.fillna('', inplace=True)\n",
    "fb_congress_posts.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yay - no more nulls!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "                                                                                                                                                                                      38206\n",
       "»                                                                                                                                                                                       162\n",
       "Congressman Gregg Harper welcomes folks to stop by his office in Washington to say hello. His staff wants to assure that your visit to Washington, D.C. is exciting and memorable.      117\n",
       "U.S. Senate Photographic Studio-                                                                                                                                                         42\n",
       "Happy Thanksgiving!                                                                                                                                                                      41\n",
       "Happy Mother's Day!                                                                                                                                                                      41\n",
       "Happy Father's Day!                                                                                                                                                                      38\n",
       "I will be talking to News Talk 1230, The Talk of Waco's Shane Warner in a few minutes to give my weekly Washington update. You can listen online at:                                     37\n",
       "Here are some photos of events or meetings that I attend while I am working in  DC.                                                                                                      34\n",
       "Merry Christmas!                                                                                                                                                                         31\n",
       "Name: post_text, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# are there duplicates in text?\n",
    "fb_congress_posts.post_text.value_counts().head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why are there weirdly specific things that are duplicated? Let's look at the first 5 of the one starting with \"Congressman Gregg Harper\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>post_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>59770</th>\n",
       "      <td>86058661</td>\n",
       "      <td>M</td>\n",
       "      <td>59770</td>\n",
       "      <td>Congressman Gregg Harper welcomes folks to sto...</td>\n",
       "      <td>photo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59773</th>\n",
       "      <td>86058661</td>\n",
       "      <td>M</td>\n",
       "      <td>59773</td>\n",
       "      <td>Congressman Gregg Harper welcomes folks to sto...</td>\n",
       "      <td>photo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59775</th>\n",
       "      <td>86058661</td>\n",
       "      <td>M</td>\n",
       "      <td>59775</td>\n",
       "      <td>Congressman Gregg Harper welcomes folks to sto...</td>\n",
       "      <td>photo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59777</th>\n",
       "      <td>86058661</td>\n",
       "      <td>M</td>\n",
       "      <td>59777</td>\n",
       "      <td>Congressman Gregg Harper welcomes folks to sto...</td>\n",
       "      <td>photo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59780</th>\n",
       "      <td>86058661</td>\n",
       "      <td>M</td>\n",
       "      <td>59780</td>\n",
       "      <td>Congressman Gregg Harper welcomes folks to sto...</td>\n",
       "      <td>photo</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          op_id op_gender  post_id  \\\n",
       "59770  86058661         M    59770   \n",
       "59773  86058661         M    59773   \n",
       "59775  86058661         M    59775   \n",
       "59777  86058661         M    59777   \n",
       "59780  86058661         M    59780   \n",
       "\n",
       "                                               post_text post_type  \n",
       "59770  Congressman Gregg Harper welcomes folks to sto...     photo  \n",
       "59773  Congressman Gregg Harper welcomes folks to sto...     photo  \n",
       "59775  Congressman Gregg Harper welcomes folks to sto...     photo  \n",
       "59777  Congressman Gregg Harper welcomes folks to sto...     photo  \n",
       "59780  Congressman Gregg Harper welcomes folks to sto...     photo  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fb_congress_posts[fb_congress_posts.post_text == 'Congressman Gregg Harper welcomes folks to stop by his office in Washington to say hello. His staff wants to assure that your visit to Washington, D.C. is exciting and memorable.'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Weird... so the same post seems to have different post IDs? Going to look at this more once they're merged.\n",
    "\n",
    "So how should we merge?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "548341\n",
      "548341\n"
     ]
    }
   ],
   "source": [
    "# let me see if post id is completely unique, or just unique to the op gender.\n",
    "# this will let us know if we need to merge on BOTH op_id and post_id or just op_id\n",
    "print(len(fb_congress_posts.post_id))\n",
    "print(fb_congress_posts.post_id.nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yay - the post ID is unique! So we can just merge on post id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender2</th>\n",
       "      <th>post_id</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>op_name</th>\n",
       "      <th>op_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Jerry</td>\n",
       "      <td>Protecting birth is not the same as protecting...</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Andrea</td>\n",
       "      <td>You need to protect children and leave my body...</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Sherry</td>\n",
       "      <td>Thank you</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Bob</td>\n",
       "      <td>Thank you Roger</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Joy</td>\n",
       "      <td>Unwanted pregnancy is a sad and unfortunate si...</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      op_id op_gender2  post_id responder_id  \\\n",
       "0  57265377          M        0        Jerry   \n",
       "1  57265377          M        0       Andrea   \n",
       "2  57265377          M        0       Sherry   \n",
       "3  57265377          M        0          Bob   \n",
       "4  57265377          M        0          Joy   \n",
       "\n",
       "                                       response_text         op_name  \\\n",
       "0  Protecting birth is not the same as protecting...  Roger Williams   \n",
       "1  You need to protect children and leave my body...  Roger Williams   \n",
       "2                                          Thank you  Roger Williams   \n",
       "3                                    Thank you Roger  Roger Williams   \n",
       "4  Unwanted pregnancy is a sad and unfortunate si...  Roger Williams   \n",
       "\n",
       "           op_category  \n",
       "0  Congress_Republican  \n",
       "1  Congress_Republican  \n",
       "2  Congress_Republican  \n",
       "3  Congress_Republican  \n",
       "4  Congress_Republican  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# renaming some columns because hoping this to be the same as the posts file\n",
    "# but still need to check\n",
    "fb_congress_responses.rename(columns={'op_gender': 'op_gender2'}, inplace=True)\n",
    "fb_congress_responses.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16\n",
      "8071\n"
     ]
    }
   ],
   "source": [
    "# first, check for nulls in the response file\n",
    "# commenting out the ones that have no nulls\n",
    "\n",
    "# fb_congress_responses[fb_congress_responses.op_id.isnull()]\n",
    "# fb_congress_responses[fb_congress_responses.op_gender2.isnull()]\n",
    "# fb_congress_responses[fb_congress_responses.post_id.isnull()]\n",
    "print(len(fb_congress_responses[fb_congress_responses.responder_id.isnull()]))\n",
    "print(len(fb_congress_responses[fb_congress_responses.response_text.isnull()]))\n",
    "# fb_congress_responses[fb_congress_responses.op_name.isnull()]\n",
    "# fb_congress_responses[fb_congress_responses.op_category.isnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have nulls in responder_id and response_text. Let's change these to an empty string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "fb_congress_responses.fillna('', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we need to match up on post_id, but hopefully the op_gender is the same too? We can add everything from the posts DataFrame to the responses DataFrame, and then see if the gender and post id match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(14015811, 11)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id_x</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>post_type</th>\n",
       "      <th>op_id_y</th>\n",
       "      <th>op_gender2</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>op_name</th>\n",
       "      <th>op_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>video</td>\n",
       "      <td>57265377.0</td>\n",
       "      <td>M</td>\n",
       "      <td>Jerry</td>\n",
       "      <td>Protecting birth is not the same as protecting...</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>video</td>\n",
       "      <td>57265377.0</td>\n",
       "      <td>M</td>\n",
       "      <td>Andrea</td>\n",
       "      <td>You need to protect children and leave my body...</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>video</td>\n",
       "      <td>57265377.0</td>\n",
       "      <td>M</td>\n",
       "      <td>Sherry</td>\n",
       "      <td>Thank you</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>video</td>\n",
       "      <td>57265377.0</td>\n",
       "      <td>M</td>\n",
       "      <td>Bob</td>\n",
       "      <td>Thank you Roger</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>video</td>\n",
       "      <td>57265377.0</td>\n",
       "      <td>M</td>\n",
       "      <td>Joy</td>\n",
       "      <td>Unwanted pregnancy is a sad and unfortunate si...</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    op_id_x op_gender  post_id  \\\n",
       "0  57265377         M        0   \n",
       "1  57265377         M        0   \n",
       "2  57265377         M        0   \n",
       "3  57265377         M        0   \n",
       "4  57265377         M        0   \n",
       "\n",
       "                                           post_text post_type     op_id_y  \\\n",
       "0  Yesterday, my colleagues and I voted to protec...     video  57265377.0   \n",
       "1  Yesterday, my colleagues and I voted to protec...     video  57265377.0   \n",
       "2  Yesterday, my colleagues and I voted to protec...     video  57265377.0   \n",
       "3  Yesterday, my colleagues and I voted to protec...     video  57265377.0   \n",
       "4  Yesterday, my colleagues and I voted to protec...     video  57265377.0   \n",
       "\n",
       "  op_gender2 responder_id                                      response_text  \\\n",
       "0          M        Jerry  Protecting birth is not the same as protecting...   \n",
       "1          M       Andrea  You need to protect children and leave my body...   \n",
       "2          M       Sherry                                          Thank you   \n",
       "3          M          Bob                                    Thank you Roger   \n",
       "4          M          Joy  Unwanted pregnancy is a sad and unfortunate si...   \n",
       "\n",
       "          op_name          op_category  \n",
       "0  Roger Williams  Congress_Republican  \n",
       "1  Roger Williams  Congress_Republican  \n",
       "2  Roger Williams  Congress_Republican  \n",
       "3  Roger Williams  Congress_Republican  \n",
       "4  Roger Williams  Congress_Republican  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# left merge, adding stuff to the responses file\n",
    "fb_congress_df = fb_congress_posts.merge(fb_congress_responses, how='left', on='post_id')\n",
    "print(fb_congress_df.shape)\n",
    "fb_congress_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 14015811 entries, 0 to 14015810\n",
      "Data columns (total 11 columns):\n",
      "op_id_x          int64\n",
      "op_gender        object\n",
      "post_id          int64\n",
      "post_text        object\n",
      "post_type        object\n",
      "op_id_y          float64\n",
      "op_gender2       object\n",
      "responder_id     object\n",
      "response_text    object\n",
      "op_name          object\n",
      "op_category      object\n",
      "dtypes: float64(1), int64(2), object(8)\n",
      "memory usage: 1.3+ GB\n"
     ]
    }
   ],
   "source": [
    "fb_congress_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "149304"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check if op_gender is identical to op_gender2 (we would hope so!)\n",
    "mismatch = fb_congress_df[fb_congress_df.op_gender != fb_congress_df.op_gender2]\n",
    "len(mismatch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id_x</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>post_type</th>\n",
       "      <th>op_id_y</th>\n",
       "      <th>op_gender2</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>op_name</th>\n",
       "      <th>op_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1393</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>35</td>\n",
       "      <td>Today is National Pearl Harbor Remembrance Day...</td>\n",
       "      <td>photo</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1412</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>37</td>\n",
       "      <td>'Tis the season. Only 24 days until #Christmas!</td>\n",
       "      <td>video</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3541</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>99</td>\n",
       "      <td>Enjoyed speaking to an engaging and thoughtful...</td>\n",
       "      <td>photo</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3876</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>109</td>\n",
       "      <td>Sales Tax Weekend starts TODAY and ends Sunday...</td>\n",
       "      <td>link</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4234</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>124</td>\n",
       "      <td>Speaking to the New York GOP delegation at the...</td>\n",
       "      <td>photo</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       op_id_x op_gender  post_id  \\\n",
       "1393  57265377         M       35   \n",
       "1412  57265377         M       37   \n",
       "3541  57265377         M       99   \n",
       "3876  57265377         M      109   \n",
       "4234  57265377         M      124   \n",
       "\n",
       "                                              post_text post_type  op_id_y  \\\n",
       "1393  Today is National Pearl Harbor Remembrance Day...     photo      NaN   \n",
       "1412    'Tis the season. Only 24 days until #Christmas!     video      NaN   \n",
       "3541  Enjoyed speaking to an engaging and thoughtful...     photo      NaN   \n",
       "3876  Sales Tax Weekend starts TODAY and ends Sunday...      link      NaN   \n",
       "4234  Speaking to the New York GOP delegation at the...     photo      NaN   \n",
       "\n",
       "     op_gender2 responder_id response_text op_name op_category  \n",
       "1393        NaN          NaN           NaN     NaN         NaN  \n",
       "1412        NaN          NaN           NaN     NaN         NaN  \n",
       "3541        NaN          NaN           NaN     NaN         NaN  \n",
       "3876        NaN          NaN           NaN     NaN         NaN  \n",
       "4234        NaN          NaN           NaN     NaN         NaN  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mismatch.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 149304 entries, 1393 to 14015810\n",
      "Data columns (total 11 columns):\n",
      "op_id_x          149304 non-null int64\n",
      "op_gender        149304 non-null object\n",
      "post_id          149304 non-null int64\n",
      "post_text        149304 non-null object\n",
      "post_type        149304 non-null object\n",
      "op_id_y          0 non-null float64\n",
      "op_gender2       0 non-null object\n",
      "responder_id     0 non-null object\n",
      "response_text    0 non-null object\n",
      "op_name          0 non-null object\n",
      "op_category      0 non-null object\n",
      "dtypes: float64(1), int64(2), object(8)\n",
      "memory usage: 13.7+ MB\n"
     ]
    }
   ],
   "source": [
    "mismatch.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So there are nulls in all of the last 6 columns. Why is this happening?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender2</th>\n",
       "      <th>post_id</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>op_name</th>\n",
       "      <th>op_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1390</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>34</td>\n",
       "      <td>Ruby</td>\n",
       "      <td>RIP</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1391</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>34</td>\n",
       "      <td>Joann</td>\n",
       "      <td>God  Bless  you  John  Glenn, you  were an awe...</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1392</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>34</td>\n",
       "      <td>Louis</td>\n",
       "      <td>A true American hero.</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1393</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>36</td>\n",
       "      <td>Evan</td>\n",
       "      <td>This has nothing to do with the state of Texas...</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1394</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>36</td>\n",
       "      <td>Clif</td>\n",
       "      <td>Would you hire them Mr Williams?  Hope the ans...</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         op_id op_gender2  post_id responder_id  \\\n",
       "1390  57265377          M       34         Ruby   \n",
       "1391  57265377          M       34        Joann   \n",
       "1392  57265377          M       34        Louis   \n",
       "1393  57265377          M       36         Evan   \n",
       "1394  57265377          M       36         Clif   \n",
       "\n",
       "                                          response_text         op_name  \\\n",
       "1390                                                RIP  Roger Williams   \n",
       "1391  God  Bless  you  John  Glenn, you  were an awe...  Roger Williams   \n",
       "1392                              A true American hero.  Roger Williams   \n",
       "1393  This has nothing to do with the state of Texas...  Roger Williams   \n",
       "1394  Would you hire them Mr Williams?  Hope the ans...  Roger Williams   \n",
       "\n",
       "              op_category  \n",
       "1390  Congress_Republican  \n",
       "1391  Congress_Republican  \n",
       "1392  Congress_Republican  \n",
       "1393  Congress_Republican  \n",
       "1394  Congress_Republican  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fb_congress_responses[fb_congress_responses.post_id == 34]\n",
    "# 34 ends at index 1392\n",
    "fb_congress_responses.iloc[1390:1395]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow - it just skips right over 35. Is this just when there are no responses to this post?\n",
    "\n",
    "Okay... a simple way to \"confirm\" if this is the case for all of the missing values... We know there are 149304 \"mismatch\" values, where the response file data is null. Let's look at the last post_id value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>post_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>548340</th>\n",
       "      <td>18876566</td>\n",
       "      <td>M</td>\n",
       "      <td>548340</td>\n",
       "      <td></td>\n",
       "      <td>link</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           op_id op_gender  post_id post_text post_type\n",
       "548340  18876566         M   548340                link"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fb_congress_posts.tail(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... There are 548341 posts (because started at 0). How many unique post_id's in the response file?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "399037"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fb_congress_responses.post_id.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "149304"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# subtract number of unique post id's in responses from the total number of posts:\n",
    "548341 - 399037"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Awesome - this is the exact amount of the rows with mismatches and missing data!\n",
    "\n",
    "Let's get rid of the op_id_y column and the op_gender2 column, and rename op_id_x to just op_id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>post_type</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>op_name</th>\n",
       "      <th>op_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>video</td>\n",
       "      <td>Jerry</td>\n",
       "      <td>Protecting birth is not the same as protecting...</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>video</td>\n",
       "      <td>Andrea</td>\n",
       "      <td>You need to protect children and leave my body...</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>video</td>\n",
       "      <td>Sherry</td>\n",
       "      <td>Thank you</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>video</td>\n",
       "      <td>Bob</td>\n",
       "      <td>Thank you Roger</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>57265377</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>video</td>\n",
       "      <td>Joy</td>\n",
       "      <td>Unwanted pregnancy is a sad and unfortunate si...</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      op_id op_gender  post_id  \\\n",
       "0  57265377         M        0   \n",
       "1  57265377         M        0   \n",
       "2  57265377         M        0   \n",
       "3  57265377         M        0   \n",
       "4  57265377         M        0   \n",
       "\n",
       "                                           post_text post_type responder_id  \\\n",
       "0  Yesterday, my colleagues and I voted to protec...     video        Jerry   \n",
       "1  Yesterday, my colleagues and I voted to protec...     video       Andrea   \n",
       "2  Yesterday, my colleagues and I voted to protec...     video       Sherry   \n",
       "3  Yesterday, my colleagues and I voted to protec...     video          Bob   \n",
       "4  Yesterday, my colleagues and I voted to protec...     video          Joy   \n",
       "\n",
       "                                       response_text         op_name  \\\n",
       "0  Protecting birth is not the same as protecting...  Roger Williams   \n",
       "1  You need to protect children and leave my body...  Roger Williams   \n",
       "2                                          Thank you  Roger Williams   \n",
       "3                                    Thank you Roger  Roger Williams   \n",
       "4  Unwanted pregnancy is a sad and unfortunate si...  Roger Williams   \n",
       "\n",
       "           op_category  \n",
       "0  Congress_Republican  \n",
       "1  Congress_Republican  \n",
       "2  Congress_Republican  \n",
       "3  Congress_Republican  \n",
       "4  Congress_Republican  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fb_congress_df = fb_congress_df.drop(['op_id_y', 'op_gender2'], axis=1)\n",
    "fb_congress_df.rename(columns={'op_id_x':'op_id'}, inplace=True)\n",
    "fb_congress_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also reorder the columns in a way that makes more sense. Also, note that it doesn't make sense to change the index. Nothing is really specific to each row besides the responder_id (which definitely has duplicate values - as it's just first name) and the actual response text.\n",
    "\n",
    "Let's organize it as:\n",
    "- info about post\n",
    "- info about poster and responder\n",
    "- post text\n",
    "- response text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_type</th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_name</th>\n",
       "      <th>op_category</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>response_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>video</td>\n",
       "      <td>57265377</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "      <td>M</td>\n",
       "      <td>Jerry</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>Protecting birth is not the same as protecting...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>video</td>\n",
       "      <td>57265377</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "      <td>M</td>\n",
       "      <td>Andrea</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>You need to protect children and leave my body...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>video</td>\n",
       "      <td>57265377</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "      <td>M</td>\n",
       "      <td>Sherry</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>Thank you</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>video</td>\n",
       "      <td>57265377</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "      <td>M</td>\n",
       "      <td>Bob</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>Thank you Roger</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>video</td>\n",
       "      <td>57265377</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "      <td>M</td>\n",
       "      <td>Joy</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>Unwanted pregnancy is a sad and unfortunate si...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   post_id post_type     op_id         op_name          op_category op_gender  \\\n",
       "0        0     video  57265377  Roger Williams  Congress_Republican         M   \n",
       "1        0     video  57265377  Roger Williams  Congress_Republican         M   \n",
       "2        0     video  57265377  Roger Williams  Congress_Republican         M   \n",
       "3        0     video  57265377  Roger Williams  Congress_Republican         M   \n",
       "4        0     video  57265377  Roger Williams  Congress_Republican         M   \n",
       "\n",
       "  responder_id                                          post_text  \\\n",
       "0        Jerry  Yesterday, my colleagues and I voted to protec...   \n",
       "1       Andrea  Yesterday, my colleagues and I voted to protec...   \n",
       "2       Sherry  Yesterday, my colleagues and I voted to protec...   \n",
       "3          Bob  Yesterday, my colleagues and I voted to protec...   \n",
       "4          Joy  Yesterday, my colleagues and I voted to protec...   \n",
       "\n",
       "                                       response_text  \n",
       "0  Protecting birth is not the same as protecting...  \n",
       "1  You need to protect children and leave my body...  \n",
       "2                                          Thank you  \n",
       "3                                    Thank you Roger  \n",
       "4  Unwanted pregnancy is a sad and unfortunate si...  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols = ['post_id','post_type','op_id','op_name','op_category','op_gender','responder_id','post_text','response_text']\n",
    "fb_congress_df = fb_congress_df[cols]\n",
    "fb_congress_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, so recalling from above, the columns with null values were op_name, op_category, responder_id, and response_text. These are from when there are no responses. We still want to keep the row though, since we can still look at the post information. Let's replace these with an empty string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "fb_congress_df.fillna('', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, back to examining those weird duplicates. Let's see when post text is duplicated and there's no response..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_type</th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_name</th>\n",
       "      <th>op_category</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>response_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>13205785</th>\n",
       "      <td>481767</td>\n",
       "      <td>photo</td>\n",
       "      <td>33944100</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3404085</th>\n",
       "      <td>105162</td>\n",
       "      <td>photo</td>\n",
       "      <td>215524</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td></td>\n",
       "      <td>photos</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5457218</th>\n",
       "      <td>166394</td>\n",
       "      <td>status</td>\n",
       "      <td>67572628</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td></td>\n",
       "      <td>Thank You</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11868998</th>\n",
       "      <td>423932</td>\n",
       "      <td>photo</td>\n",
       "      <td>74450840</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>W</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1753528</th>\n",
       "      <td>23298</td>\n",
       "      <td>photo</td>\n",
       "      <td>56371090</td>\n",
       "      <td>Nancy Pelosi</td>\n",
       "      <td>Congress_Democratic</td>\n",
       "      <td>W</td>\n",
       "      <td>Tsering</td>\n",
       "      <td>It was an honor today to receive the 2016 Huma...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13391542</th>\n",
       "      <td>506786</td>\n",
       "      <td>photo</td>\n",
       "      <td>14349839</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7548026</th>\n",
       "      <td>285708</td>\n",
       "      <td>link</td>\n",
       "      <td>30174504</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9039439</th>\n",
       "      <td>309648</td>\n",
       "      <td>link</td>\n",
       "      <td>4804331</td>\n",
       "      <td>Cheri Bustos</td>\n",
       "      <td>Congress_Democratic</td>\n",
       "      <td>W</td>\n",
       "      <td>Bruce</td>\n",
       "      <td>Governor Rauner's decision to ignore the recom...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2342715</th>\n",
       "      <td>50338</td>\n",
       "      <td>link</td>\n",
       "      <td>89989833</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5019987</th>\n",
       "      <td>160414</td>\n",
       "      <td>video</td>\n",
       "      <td>46506154</td>\n",
       "      <td>Marco Rubio</td>\n",
       "      <td>Congress_Republican</td>\n",
       "      <td>M</td>\n",
       "      <td>Judi</td>\n",
       "      <td>If you elect me president, we will have a pres...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          post_id post_type     op_id       op_name          op_category  \\\n",
       "13205785   481767     photo  33944100                                      \n",
       "3404085    105162     photo    215524                                      \n",
       "5457218    166394    status  67572628                                      \n",
       "11868998   423932     photo  74450840                                      \n",
       "1753528     23298     photo  56371090  Nancy Pelosi  Congress_Democratic   \n",
       "13391542   506786     photo  14349839                                      \n",
       "7548026    285708      link  30174504                                      \n",
       "9039439    309648      link   4804331  Cheri Bustos  Congress_Democratic   \n",
       "2342715     50338      link  89989833                                      \n",
       "5019987    160414     video  46506154   Marco Rubio  Congress_Republican   \n",
       "\n",
       "         op_gender responder_id  \\\n",
       "13205785         M                \n",
       "3404085          M                \n",
       "5457218          M                \n",
       "11868998         W                \n",
       "1753528          W      Tsering   \n",
       "13391542         M                \n",
       "7548026          M                \n",
       "9039439          W        Bruce   \n",
       "2342715          M                \n",
       "5019987          M         Judi   \n",
       "\n",
       "                                                  post_text response_text  \n",
       "13205785                                                                   \n",
       "3404085                                              photos                \n",
       "5457218                                           Thank You                \n",
       "11868998                                                                   \n",
       "1753528   It was an honor today to receive the 2016 Huma...                \n",
       "13391542                                                                   \n",
       "7548026                                                                    \n",
       "9039439   Governor Rauner's decision to ignore the recom...                \n",
       "2342715                                                                    \n",
       "5019987   If you elect me president, we will have a pres...                "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "duplicates = fb_congress_df[(fb_congress_df.duplicated(subset='post_text')) & (fb_congress_df.response_text == '')]\n",
    "duplicates.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's just drop all of these. It's confusing and doesn't make sense. Note that how it's formatted, we would still keep the row if there is response text or if it's the first occurrence of the duplicate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "fb_congress_df.drop(duplicates.index, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_type</th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_name</th>\n",
       "      <th>op_category</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>response_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [post_id, post_type, op_id, op_name, op_category, op_gender, responder_id, post_text, response_text]\n",
       "Index: []"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make sure it worked\n",
    "fb_congress_df[(fb_congress_df.duplicated(subset='post_text')) & (fb_congress_df.response_text == '')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great - looks good.\n",
    "\n",
    "### Pickling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# try to pickle\n",
    "\n",
    "# the file seemed to be too large to just pickle it normally, so I'm splitting it into 2\n",
    "fb_congress_df_list = np.array_split(fb_congress_df, 2)\n",
    "len(fb_congress_df_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# and pickling them separately\n",
    "fb_congress_df_list[0].to_pickle(\"fb_congress_df_part1.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "fb_congress_df_list[1].to_pickle(\"fb_congress_df_part2.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now load and combine them, make sure it's the same\n",
    "fb_congress_df_part1 = pd.read_pickle(\"fb_congress_df_part1.pkl\")\n",
    "fb_congress_df_part2 = pd.read_pickle(\"fb_congress_df_part2.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_type</th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_name</th>\n",
       "      <th>op_category</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>response_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>video</td>\n",
       "      <td>57265377</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "      <td>M</td>\n",
       "      <td>Jerry</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>Protecting birth is not the same as protecting...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>video</td>\n",
       "      <td>57265377</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "      <td>M</td>\n",
       "      <td>Andrea</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>You need to protect children and leave my body...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>video</td>\n",
       "      <td>57265377</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "      <td>M</td>\n",
       "      <td>Sherry</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>Thank you</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>video</td>\n",
       "      <td>57265377</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "      <td>M</td>\n",
       "      <td>Bob</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>Thank you Roger</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>video</td>\n",
       "      <td>57265377</td>\n",
       "      <td>Roger Williams</td>\n",
       "      <td>Congress_Republican</td>\n",
       "      <td>M</td>\n",
       "      <td>Joy</td>\n",
       "      <td>Yesterday, my colleagues and I voted to protec...</td>\n",
       "      <td>Unwanted pregnancy is a sad and unfortunate si...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   post_id post_type     op_id         op_name          op_category op_gender  \\\n",
       "0        0     video  57265377  Roger Williams  Congress_Republican         M   \n",
       "1        0     video  57265377  Roger Williams  Congress_Republican         M   \n",
       "2        0     video  57265377  Roger Williams  Congress_Republican         M   \n",
       "3        0     video  57265377  Roger Williams  Congress_Republican         M   \n",
       "4        0     video  57265377  Roger Williams  Congress_Republican         M   \n",
       "\n",
       "  responder_id                                          post_text  \\\n",
       "0        Jerry  Yesterday, my colleagues and I voted to protec...   \n",
       "1       Andrea  Yesterday, my colleagues and I voted to protec...   \n",
       "2       Sherry  Yesterday, my colleagues and I voted to protec...   \n",
       "3          Bob  Yesterday, my colleagues and I voted to protec...   \n",
       "4          Joy  Yesterday, my colleagues and I voted to protec...   \n",
       "\n",
       "                                       response_text  \n",
       "0  Protecting birth is not the same as protecting...  \n",
       "1  You need to protect children and leave my body...  \n",
       "2                                          Thank you  \n",
       "3                                    Thank you Roger  \n",
       "4  Unwanted pregnancy is a sad and unfortunate si...  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fb_congress_df2 = fb_congress_df_part1.append(fb_congress_df_part2)\n",
    "fb_congress_df2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks to be set up similarly... Let's see if it's actually the same as the original!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fb_congress_df.equals(fb_congress_df2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yay! So this method of pickling will work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Facebook wiki posts and responses\n",
    "Next, let's combine the posts and responses from the other Facebook file. This is formatted very similarly (if not the same) to the first two files, so the merging process will be the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "fb_wiki_posts = pd.read_csv(root + 'facebook_wiki_posts.csv')\n",
    "fb_wiki_responses = pd.read_csv(root + 'facebook_wiki_responses.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>post_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Tornado watch in effect tonight. Be safe. Plea...</td>\n",
       "      <td>photo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>photo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>2</td>\n",
       "      <td>Temps warming up for the holiday weekend! I wi...</td>\n",
       "      <td>photo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>3</td>\n",
       "      <td>Showers heading this way ..</td>\n",
       "      <td>photo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>4</td>\n",
       "      <td>Storm potential update...feel free to share th...</td>\n",
       "      <td>photo</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      op_id op_gender  post_id  \\\n",
       "0  11679984         M        0   \n",
       "1  11679984         M        1   \n",
       "2  11679984         M        2   \n",
       "3  11679984         M        3   \n",
       "4  11679984         M        4   \n",
       "\n",
       "                                           post_text post_type  \n",
       "0  Tornado watch in effect tonight. Be safe. Plea...     photo  \n",
       "1                                                NaN     photo  \n",
       "2  Temps warming up for the holiday weekend! I wi...     photo  \n",
       "3                        Showers heading this way ..     photo  \n",
       "4  Storm potential update...feel free to share th...     photo  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fb_wiki_posts.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we recall from our exploratory analysis, there are some nulls in post_text (mostly photos, videos, and links). Let's replace these with empty strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 149448 entries, 0 to 149447\n",
      "Data columns (total 5 columns):\n",
      "op_id        149448 non-null int64\n",
      "op_gender    149448 non-null object\n",
      "post_id      149448 non-null int64\n",
      "post_text    149448 non-null object\n",
      "post_type    149448 non-null object\n",
      "dtypes: int64(2), object(3)\n",
      "memory usage: 5.7+ MB\n"
     ]
    }
   ],
   "source": [
    "fb_wiki_posts.post_text.fillna('', inplace=True)\n",
    "fb_wiki_posts.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yay - no more nulls!\n",
    "\n",
    "Do we have those weird duplicates again?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "                                    16431\n",
       "Michael Bernard Beckwith              224\n",
       "HelloGiggles.com                      126\n",
       "Agree?  Michael Bernard Beckwith       62\n",
       "<3                                     59\n",
       "Name: post_text, dtype: int64"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fb_wiki_posts.post_text.value_counts().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>post_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>53480</th>\n",
       "      <td>40564507</td>\n",
       "      <td>M</td>\n",
       "      <td>53480</td>\n",
       "      <td>Michael Bernard Beckwith</td>\n",
       "      <td>photo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53592</th>\n",
       "      <td>40564507</td>\n",
       "      <td>M</td>\n",
       "      <td>53592</td>\n",
       "      <td>Michael Bernard Beckwith</td>\n",
       "      <td>photo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53861</th>\n",
       "      <td>40564507</td>\n",
       "      <td>M</td>\n",
       "      <td>53861</td>\n",
       "      <td>Michael Bernard Beckwith</td>\n",
       "      <td>photo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53980</th>\n",
       "      <td>40564507</td>\n",
       "      <td>M</td>\n",
       "      <td>53980</td>\n",
       "      <td>Michael Bernard Beckwith</td>\n",
       "      <td>photo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53999</th>\n",
       "      <td>40564507</td>\n",
       "      <td>M</td>\n",
       "      <td>53999</td>\n",
       "      <td>Michael Bernard Beckwith</td>\n",
       "      <td>photo</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          op_id op_gender  post_id                 post_text post_type\n",
       "53480  40564507         M    53480  Michael Bernard Beckwith     photo\n",
       "53592  40564507         M    53592  Michael Bernard Beckwith     photo\n",
       "53861  40564507         M    53861  Michael Bernard Beckwith     photo\n",
       "53980  40564507         M    53980  Michael Bernard Beckwith     photo\n",
       "53999  40564507         M    53999  Michael Bernard Beckwith     photo"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fb_wiki_posts[fb_wiki_posts.post_text == 'Michael Bernard Beckwith'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again - looking really strange. I'll try to look again once it's merged."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>op_name</th>\n",
       "      <th>op_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Michelle</td>\n",
       "      <td>Is this watch going to make it to LaPorte county?</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Melissa</td>\n",
       "      <td>Anything for Wilmington area</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Darlene</td>\n",
       "      <td>Thanks, please keep us posted.</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Cheryl</td>\n",
       "      <td>Thanks Byron</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Melissa</td>\n",
       "      <td>[[STICKER]]</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      op_id op_gender  post_id responder_id  \\\n",
       "0  11679984         M        0     Michelle   \n",
       "1  11679984         M        0      Melissa   \n",
       "2  11679984         M        0      Darlene   \n",
       "3  11679984         M        0       Cheryl   \n",
       "4  11679984         M        0      Melissa   \n",
       "\n",
       "                                       response_text        op_name  \\\n",
       "0  Is this watch going to make it to LaPorte county?  Byron Miranda   \n",
       "1                       Anything for Wilmington area  Byron Miranda   \n",
       "2                     Thanks, please keep us posted.  Byron Miranda   \n",
       "3                                       Thanks Byron  Byron Miranda   \n",
       "4                                        [[STICKER]]  Byron Miranda   \n",
       "\n",
       "                                  op_category  \n",
       "0  Wikipedia_American_television_news_anchors  \n",
       "1  Wikipedia_American_television_news_anchors  \n",
       "2  Wikipedia_American_television_news_anchors  \n",
       "3  Wikipedia_American_television_news_anchors  \n",
       "4  Wikipedia_American_television_news_anchors  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fb_wiki_responses.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "29\n",
      "216\n"
     ]
    }
   ],
   "source": [
    "# looking for nulls\n",
    "# commenting out the ones that have no nulls\n",
    "\n",
    "# fb_wiki_responses[fb_wiki_responses.op_id.isnull()]\n",
    "# fb_wiki_responses[fb_wiki_responses.op_gender.isnull()]\n",
    "# fb_wiki_responses[fb_wiki_responses.post_id.isnull()]\n",
    "print(len(fb_wiki_responses[fb_wiki_responses.responder_id.isnull()]))\n",
    "print(len(fb_wiki_responses[fb_wiki_responses.response_text.isnull()]))\n",
    "# fb_wiki_responses[fb_wiki_responses.op_name.isnull()]\n",
    "# fb_wiki_responses[fb_wiki_responses.op_category.isnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, so we have some nulls in the responder_id and response_text. Let's change this to an empty string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "fb_wiki_responses.fillna('', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "149448\n",
      "149448\n"
     ]
    }
   ],
   "source": [
    "# again, check to make sure post_id is unique\n",
    "print(len(fb_wiki_posts.post_id))\n",
    "print(fb_wiki_posts.post_id.nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yay, post id is unique. So we can merge on post id again! Does this one have any posts without responses? First, we merge:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10699137, 11)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id_x</th>\n",
       "      <th>op_gender_x</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>post_type</th>\n",
       "      <th>op_id_y</th>\n",
       "      <th>op_gender_y</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>op_name</th>\n",
       "      <th>op_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Tornado watch in effect tonight. Be safe. Plea...</td>\n",
       "      <td>photo</td>\n",
       "      <td>11679984.0</td>\n",
       "      <td>M</td>\n",
       "      <td>Michelle</td>\n",
       "      <td>Is this watch going to make it to LaPorte county?</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Tornado watch in effect tonight. Be safe. Plea...</td>\n",
       "      <td>photo</td>\n",
       "      <td>11679984.0</td>\n",
       "      <td>M</td>\n",
       "      <td>Melissa</td>\n",
       "      <td>Anything for Wilmington area</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Tornado watch in effect tonight. Be safe. Plea...</td>\n",
       "      <td>photo</td>\n",
       "      <td>11679984.0</td>\n",
       "      <td>M</td>\n",
       "      <td>Darlene</td>\n",
       "      <td>Thanks, please keep us posted.</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Tornado watch in effect tonight. Be safe. Plea...</td>\n",
       "      <td>photo</td>\n",
       "      <td>11679984.0</td>\n",
       "      <td>M</td>\n",
       "      <td>Cheryl</td>\n",
       "      <td>Thanks Byron</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Tornado watch in effect tonight. Be safe. Plea...</td>\n",
       "      <td>photo</td>\n",
       "      <td>11679984.0</td>\n",
       "      <td>M</td>\n",
       "      <td>Melissa</td>\n",
       "      <td>[[STICKER]]</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    op_id_x op_gender_x  post_id  \\\n",
       "0  11679984           M        0   \n",
       "1  11679984           M        0   \n",
       "2  11679984           M        0   \n",
       "3  11679984           M        0   \n",
       "4  11679984           M        0   \n",
       "\n",
       "                                           post_text post_type     op_id_y  \\\n",
       "0  Tornado watch in effect tonight. Be safe. Plea...     photo  11679984.0   \n",
       "1  Tornado watch in effect tonight. Be safe. Plea...     photo  11679984.0   \n",
       "2  Tornado watch in effect tonight. Be safe. Plea...     photo  11679984.0   \n",
       "3  Tornado watch in effect tonight. Be safe. Plea...     photo  11679984.0   \n",
       "4  Tornado watch in effect tonight. Be safe. Plea...     photo  11679984.0   \n",
       "\n",
       "  op_gender_y responder_id                                      response_text  \\\n",
       "0           M     Michelle  Is this watch going to make it to LaPorte county?   \n",
       "1           M      Melissa                       Anything for Wilmington area   \n",
       "2           M      Darlene                     Thanks, please keep us posted.   \n",
       "3           M       Cheryl                                       Thanks Byron   \n",
       "4           M      Melissa                                        [[STICKER]]   \n",
       "\n",
       "         op_name                                 op_category  \n",
       "0  Byron Miranda  Wikipedia_American_television_news_anchors  \n",
       "1  Byron Miranda  Wikipedia_American_television_news_anchors  \n",
       "2  Byron Miranda  Wikipedia_American_television_news_anchors  \n",
       "3  Byron Miranda  Wikipedia_American_television_news_anchors  \n",
       "4  Byron Miranda  Wikipedia_American_television_news_anchors  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fb_wiki_df = fb_wiki_posts.merge(fb_wiki_responses, how='left', on='post_id')\n",
    "print(fb_wiki_df.shape)\n",
    "fb_wiki_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender_x</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>post_type</th>\n",
       "      <th>op_gender_y</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>op_name</th>\n",
       "      <th>op_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Tornado watch in effect tonight. Be safe. Plea...</td>\n",
       "      <td>photo</td>\n",
       "      <td>M</td>\n",
       "      <td>Michelle</td>\n",
       "      <td>Is this watch going to make it to LaPorte county?</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Tornado watch in effect tonight. Be safe. Plea...</td>\n",
       "      <td>photo</td>\n",
       "      <td>M</td>\n",
       "      <td>Melissa</td>\n",
       "      <td>Anything for Wilmington area</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Tornado watch in effect tonight. Be safe. Plea...</td>\n",
       "      <td>photo</td>\n",
       "      <td>M</td>\n",
       "      <td>Darlene</td>\n",
       "      <td>Thanks, please keep us posted.</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Tornado watch in effect tonight. Be safe. Plea...</td>\n",
       "      <td>photo</td>\n",
       "      <td>M</td>\n",
       "      <td>Cheryl</td>\n",
       "      <td>Thanks Byron</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Tornado watch in effect tonight. Be safe. Plea...</td>\n",
       "      <td>photo</td>\n",
       "      <td>M</td>\n",
       "      <td>Melissa</td>\n",
       "      <td>[[STICKER]]</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      op_id op_gender_x  post_id  \\\n",
       "0  11679984           M        0   \n",
       "1  11679984           M        0   \n",
       "2  11679984           M        0   \n",
       "3  11679984           M        0   \n",
       "4  11679984           M        0   \n",
       "\n",
       "                                           post_text post_type op_gender_y  \\\n",
       "0  Tornado watch in effect tonight. Be safe. Plea...     photo           M   \n",
       "1  Tornado watch in effect tonight. Be safe. Plea...     photo           M   \n",
       "2  Tornado watch in effect tonight. Be safe. Plea...     photo           M   \n",
       "3  Tornado watch in effect tonight. Be safe. Plea...     photo           M   \n",
       "4  Tornado watch in effect tonight. Be safe. Plea...     photo           M   \n",
       "\n",
       "  responder_id                                      response_text  \\\n",
       "0     Michelle  Is this watch going to make it to LaPorte county?   \n",
       "1      Melissa                       Anything for Wilmington area   \n",
       "2      Darlene                     Thanks, please keep us posted.   \n",
       "3       Cheryl                                       Thanks Byron   \n",
       "4      Melissa                                        [[STICKER]]   \n",
       "\n",
       "         op_name                                 op_category  \n",
       "0  Byron Miranda  Wikipedia_American_television_news_anchors  \n",
       "1  Byron Miranda  Wikipedia_American_television_news_anchors  \n",
       "2  Byron Miranda  Wikipedia_American_television_news_anchors  \n",
       "3  Byron Miranda  Wikipedia_American_television_news_anchors  \n",
       "4  Byron Miranda  Wikipedia_American_television_news_anchors  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# remove op_id_y and rename op_id_x to just op_id\n",
    "fb_wiki_df = fb_wiki_df.drop('op_id_y', axis=1)\n",
    "fb_wiki_df.rename(columns={'op_id_x':'op_id'}, inplace=True)\n",
    "fb_wiki_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "31637"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check for mismatches on gender\n",
    "mismatch = fb_wiki_df[fb_wiki_df.op_gender_x != fb_wiki_df.op_gender_y]\n",
    "len(mismatch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender_x</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>post_type</th>\n",
       "      <th>op_gender_y</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>op_name</th>\n",
       "      <th>op_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>1</td>\n",
       "      <td></td>\n",
       "      <td>photo</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>743</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>79</td>\n",
       "      <td>Break today, t'storms possible tomorrow. Snow ...</td>\n",
       "      <td>photo</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>101</td>\n",
       "      <td>http://www.nbcchicago.com/news/local/arlington...</td>\n",
       "      <td>link</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1082</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>111</td>\n",
       "      <td>NORTHWESTERN MADE HISTORY LAST WEEK...BUT IT W...</td>\n",
       "      <td>link</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1104</th>\n",
       "      <td>11679984</td>\n",
       "      <td>M</td>\n",
       "      <td>114</td>\n",
       "      <td></td>\n",
       "      <td>status</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         op_id op_gender_x  post_id  \\\n",
       "16    11679984           M        1   \n",
       "743   11679984           M       79   \n",
       "995   11679984           M      101   \n",
       "1082  11679984           M      111   \n",
       "1104  11679984           M      114   \n",
       "\n",
       "                                              post_text post_type op_gender_y  \\\n",
       "16                                                          photo         NaN   \n",
       "743   Break today, t'storms possible tomorrow. Snow ...     photo         NaN   \n",
       "995   http://www.nbcchicago.com/news/local/arlington...      link         NaN   \n",
       "1082  NORTHWESTERN MADE HISTORY LAST WEEK...BUT IT W...      link         NaN   \n",
       "1104                                                       status         NaN   \n",
       "\n",
       "     responder_id response_text op_name op_category  \n",
       "16            NaN           NaN     NaN         NaN  \n",
       "743           NaN           NaN     NaN         NaN  \n",
       "995           NaN           NaN     NaN         NaN  \n",
       "1082          NaN           NaN     NaN         NaN  \n",
       "1104          NaN           NaN     NaN         NaN  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mismatch.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 31637 entries, 16 to 10484060\n",
      "Data columns (total 10 columns):\n",
      "op_id            31637 non-null int64\n",
      "op_gender_x      31637 non-null object\n",
      "post_id          31637 non-null int64\n",
      "post_text        31637 non-null object\n",
      "post_type        31637 non-null object\n",
      "op_gender_y      0 non-null object\n",
      "responder_id     0 non-null object\n",
      "response_text    0 non-null object\n",
      "op_name          0 non-null object\n",
      "op_category      0 non-null object\n",
      "dtypes: int64(2), object(8)\n",
      "memory usage: 2.7+ MB\n"
     ]
    }
   ],
   "source": [
    "mismatch.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have nulls in op_gender_y, responder_id, response_text, op_name, and op_category. Let's check and make sure it's for the same reason as the last time (that there are no responses for these posts!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Empty DataFrame\n",
      "Columns: [op_id, op_gender, post_id, responder_id, response_text, op_name, op_category]\n",
      "Index: []\n",
      "Empty DataFrame\n",
      "Columns: [op_id, op_gender, post_id, responder_id, response_text, op_name, op_category]\n",
      "Index: []\n"
     ]
    }
   ],
   "source": [
    "print(fb_wiki_responses[fb_wiki_responses.post_id == 1])\n",
    "print(fb_wiki_responses[fb_wiki_responses.post_id == 79])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Seems like it. So let's just replace all these nulls with empty strings again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "fb_wiki_df.fillna('', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And again fix the columns..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_type</th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_name</th>\n",
       "      <th>op_category</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>response_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>photo</td>\n",
       "      <td>11679984</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "      <td>M</td>\n",
       "      <td>Michelle</td>\n",
       "      <td>Tornado watch in effect tonight. Be safe. Plea...</td>\n",
       "      <td>Is this watch going to make it to LaPorte county?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>photo</td>\n",
       "      <td>11679984</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "      <td>M</td>\n",
       "      <td>Melissa</td>\n",
       "      <td>Tornado watch in effect tonight. Be safe. Plea...</td>\n",
       "      <td>Anything for Wilmington area</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>photo</td>\n",
       "      <td>11679984</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "      <td>M</td>\n",
       "      <td>Darlene</td>\n",
       "      <td>Tornado watch in effect tonight. Be safe. Plea...</td>\n",
       "      <td>Thanks, please keep us posted.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>photo</td>\n",
       "      <td>11679984</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "      <td>M</td>\n",
       "      <td>Cheryl</td>\n",
       "      <td>Tornado watch in effect tonight. Be safe. Plea...</td>\n",
       "      <td>Thanks Byron</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>photo</td>\n",
       "      <td>11679984</td>\n",
       "      <td>Byron Miranda</td>\n",
       "      <td>Wikipedia_American_television_news_anchors</td>\n",
       "      <td>M</td>\n",
       "      <td>Melissa</td>\n",
       "      <td>Tornado watch in effect tonight. Be safe. Plea...</td>\n",
       "      <td>[[STICKER]]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   post_id post_type     op_id        op_name  \\\n",
       "0        0     photo  11679984  Byron Miranda   \n",
       "1        0     photo  11679984  Byron Miranda   \n",
       "2        0     photo  11679984  Byron Miranda   \n",
       "3        0     photo  11679984  Byron Miranda   \n",
       "4        0     photo  11679984  Byron Miranda   \n",
       "\n",
       "                                  op_category op_gender responder_id  \\\n",
       "0  Wikipedia_American_television_news_anchors         M     Michelle   \n",
       "1  Wikipedia_American_television_news_anchors         M      Melissa   \n",
       "2  Wikipedia_American_television_news_anchors         M      Darlene   \n",
       "3  Wikipedia_American_television_news_anchors         M       Cheryl   \n",
       "4  Wikipedia_American_television_news_anchors         M      Melissa   \n",
       "\n",
       "                                           post_text  \\\n",
       "0  Tornado watch in effect tonight. Be safe. Plea...   \n",
       "1  Tornado watch in effect tonight. Be safe. Plea...   \n",
       "2  Tornado watch in effect tonight. Be safe. Plea...   \n",
       "3  Tornado watch in effect tonight. Be safe. Plea...   \n",
       "4  Tornado watch in effect tonight. Be safe. Plea...   \n",
       "\n",
       "                                       response_text  \n",
       "0  Is this watch going to make it to LaPorte county?  \n",
       "1                       Anything for Wilmington area  \n",
       "2                     Thanks, please keep us posted.  \n",
       "3                                       Thanks Byron  \n",
       "4                                        [[STICKER]]  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# drop op_gender_y column\n",
    "fb_wiki_df = fb_wiki_df.drop('op_gender_y', axis=1)\n",
    "\n",
    "# rename op_gender_x to just op_gender\n",
    "fb_wiki_df.rename(columns={'op_gender_x':'op_gender'}, inplace=True)\n",
    "\n",
    "# reorganize columns again, in same order\n",
    "fb_wiki_df = fb_wiki_df[cols]\n",
    "fb_wiki_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Back to looking at the weird duplicates..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_type</th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_name</th>\n",
       "      <th>op_category</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>response_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>225551</th>\n",
       "      <td>6508</td>\n",
       "      <td>link</td>\n",
       "      <td>98249750</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>224820</th>\n",
       "      <td>5777</td>\n",
       "      <td>link</td>\n",
       "      <td>98249750</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2058622</th>\n",
       "      <td>23602</td>\n",
       "      <td>link</td>\n",
       "      <td>84499674</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>W</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>224597</th>\n",
       "      <td>5554</td>\n",
       "      <td>link</td>\n",
       "      <td>98249750</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>224443</th>\n",
       "      <td>5400</td>\n",
       "      <td>photo</td>\n",
       "      <td>98249750</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>227046</th>\n",
       "      <td>8003</td>\n",
       "      <td>photo</td>\n",
       "      <td>98249750</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5708554</th>\n",
       "      <td>86058</td>\n",
       "      <td>link</td>\n",
       "      <td>30328617</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>W</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>226501</th>\n",
       "      <td>7458</td>\n",
       "      <td>link</td>\n",
       "      <td>98249750</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10353855</th>\n",
       "      <td>147366</td>\n",
       "      <td>event</td>\n",
       "      <td>71908465</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>224328</th>\n",
       "      <td>5285</td>\n",
       "      <td>video</td>\n",
       "      <td>98249750</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          post_id post_type     op_id op_name op_category op_gender  \\\n",
       "225551       6508      link  98249750                             M   \n",
       "224820       5777      link  98249750                             M   \n",
       "2058622     23602      link  84499674                             W   \n",
       "224597       5554      link  98249750                             M   \n",
       "224443       5400     photo  98249750                             M   \n",
       "227046       8003     photo  98249750                             M   \n",
       "5708554     86058      link  30328617                             W   \n",
       "226501       7458      link  98249750                             M   \n",
       "10353855   147366     event  71908465                             M   \n",
       "224328       5285     video  98249750                             M   \n",
       "\n",
       "         responder_id post_text response_text  \n",
       "225551                                         \n",
       "224820                                         \n",
       "2058622                                        \n",
       "224597                                         \n",
       "224443                                         \n",
       "227046                                         \n",
       "5708554                                        \n",
       "226501                                         \n",
       "10353855                                       \n",
       "224328                                         "
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "duplicates = fb_wiki_df[(fb_wiki_df.duplicated(subset='post_text')) & (fb_wiki_df.response_text == '')]\n",
    "duplicates.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, let's drop these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "fb_wiki_df.drop(duplicates.index, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_type</th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_name</th>\n",
       "      <th>op_category</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>response_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [post_id, post_type, op_id, op_name, op_category, op_gender, responder_id, post_text, response_text]\n",
       "Index: []"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make sure it worked\n",
    "fb_wiki_df[(fb_wiki_df.duplicated(subset='post_text')) & (fb_wiki_df.response_text == '')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks good!\n",
    "\n",
    "### Pickling\n",
    "\n",
    "Now let's pickle! This one is a little smaller than the Congress posts, so maybe we don't have to split it first. Let's try:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "fb_wiki_df.to_pickle(\"fb_wiki_df.pkl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It worked! Make sure they're the same again..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fb_wiki_df2 = pd.read_pickle(\"fb_wiki_df.pkl\")\n",
    "fb_wiki_df.equals(fb_wiki_df2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks good!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fitocracy posts and responses\n",
    "Again, combine posts and responses. This one will be in a different format than the Facebook files, though. Let's look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "fitocracy_posts = pd.read_csv(root + 'fitocracy_posts.csv')\n",
    "fitocracy_responses = pd.read_csv(root + 'fitocracy_responses.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>102</td>\n",
       "      <td>W</td>\n",
       "      <td>0</td>\n",
       "      <td>Thanks for the follow!  I followed back :)  I ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>104</td>\n",
       "      <td>W</td>\n",
       "      <td>1</td>\n",
       "      <td>I've decided I have this crazy goal of running...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>104</td>\n",
       "      <td>W</td>\n",
       "      <td>2</td>\n",
       "      <td>Umm, yeah, so those are assisted pull-ups. Bef...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>117</td>\n",
       "      <td>M</td>\n",
       "      <td>3</td>\n",
       "      <td>dam gurl lookin mad tone in dat pp holla bb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>117</td>\n",
       "      <td>M</td>\n",
       "      <td>4</td>\n",
       "      <td>What's up there bear mode?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   op_id op_gender  post_id                                          post_text\n",
       "0    102         W        0  Thanks for the follow!  I followed back :)  I ...\n",
       "1    104         W        1  I've decided I have this crazy goal of running...\n",
       "2    104         W        2  Umm, yeah, so those are assisted pull-ups. Bef...\n",
       "3    117         M        3        dam gurl lookin mad tone in dat pp holla bb\n",
       "4    117         M        4                         What's up there bear mode?"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fitocracy_posts.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, as we recall from the exploratory analysis file, there were no nulls in this file!\n",
    "\n",
    "...Are there duplicates?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "just joined the WTF - Welcome to Fitocracy group!        22698\n",
       "Thanks for the follow!                                    5093\n",
       "just joined the No Barbie Weights Here group!             4105\n",
       "just joined the Girls Gone Strong on Fitocracy group!     3612\n",
       "just joined the Couch to 5K group!                        2867\n",
       "Name: post_text, dtype: int64"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fitocracy_posts.post_text.value_counts().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yes - let's evaluate later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>responder_gender</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>102</td>\n",
       "      <td>W</td>\n",
       "      <td>0</td>\n",
       "      <td>1196122</td>\n",
       "      <td>One day at a time! =]</td>\n",
       "      <td>M</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>104</td>\n",
       "      <td>W</td>\n",
       "      <td>1</td>\n",
       "      <td>5867</td>\n",
       "      <td>Crazy is synonymous with awesome in this case.</td>\n",
       "      <td>W</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>104</td>\n",
       "      <td>W</td>\n",
       "      <td>2</td>\n",
       "      <td>1635</td>\n",
       "      <td>You can enter assisted pullups. Select pullups...</td>\n",
       "      <td>M</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>117</td>\n",
       "      <td>M</td>\n",
       "      <td>3</td>\n",
       "      <td>8520</td>\n",
       "      <td>Mirin 3% bodyfat? Yeah, you are.</td>\n",
       "      <td>W</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>117</td>\n",
       "      <td>M</td>\n",
       "      <td>4</td>\n",
       "      <td>29126</td>\n",
       "      <td>Hey! I just started a new job, so things are s...</td>\n",
       "      <td>M</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   op_id op_gender  post_id  responder_id  \\\n",
       "0    102         W        0       1196122   \n",
       "1    104         W        1          5867   \n",
       "2    104         W        2          1635   \n",
       "3    117         M        3          8520   \n",
       "4    117         M        4         29126   \n",
       "\n",
       "                                       response_text responder_gender  \n",
       "0                              One day at a time! =]                M  \n",
       "1     Crazy is synonymous with awesome in this case.                W  \n",
       "2  You can enter assisted pullups. Select pullups...                M  \n",
       "3                   Mirin 3% bodyfat? Yeah, you are.                W  \n",
       "4  Hey! I just started a new job, so things are s...                M  "
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fitocracy_responses.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There were also no nulls in the response file! We can get straight to merging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "318535\n",
      "318535\n"
     ]
    }
   ],
   "source": [
    "# check if post_id is unique\n",
    "print(len(fitocracy_posts.post_id))\n",
    "print(fitocracy_posts.post_id.nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Good, we can merge on this!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(318535, 9)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id_x</th>\n",
       "      <th>op_gender_x</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>op_id_y</th>\n",
       "      <th>op_gender_y</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>responder_gender</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>102</td>\n",
       "      <td>W</td>\n",
       "      <td>0</td>\n",
       "      <td>Thanks for the follow!  I followed back :)  I ...</td>\n",
       "      <td>102</td>\n",
       "      <td>W</td>\n",
       "      <td>1196122</td>\n",
       "      <td>One day at a time! =]</td>\n",
       "      <td>M</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>104</td>\n",
       "      <td>W</td>\n",
       "      <td>1</td>\n",
       "      <td>I've decided I have this crazy goal of running...</td>\n",
       "      <td>104</td>\n",
       "      <td>W</td>\n",
       "      <td>5867</td>\n",
       "      <td>Crazy is synonymous with awesome in this case.</td>\n",
       "      <td>W</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>104</td>\n",
       "      <td>W</td>\n",
       "      <td>2</td>\n",
       "      <td>Umm, yeah, so those are assisted pull-ups. Bef...</td>\n",
       "      <td>104</td>\n",
       "      <td>W</td>\n",
       "      <td>1635</td>\n",
       "      <td>You can enter assisted pullups. Select pullups...</td>\n",
       "      <td>M</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>117</td>\n",
       "      <td>M</td>\n",
       "      <td>3</td>\n",
       "      <td>dam gurl lookin mad tone in dat pp holla bb</td>\n",
       "      <td>117</td>\n",
       "      <td>M</td>\n",
       "      <td>8520</td>\n",
       "      <td>Mirin 3% bodyfat? Yeah, you are.</td>\n",
       "      <td>W</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>117</td>\n",
       "      <td>M</td>\n",
       "      <td>4</td>\n",
       "      <td>What's up there bear mode?</td>\n",
       "      <td>117</td>\n",
       "      <td>M</td>\n",
       "      <td>29126</td>\n",
       "      <td>Hey! I just started a new job, so things are s...</td>\n",
       "      <td>M</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   op_id_x op_gender_x  post_id  \\\n",
       "0      102           W        0   \n",
       "1      104           W        1   \n",
       "2      104           W        2   \n",
       "3      117           M        3   \n",
       "4      117           M        4   \n",
       "\n",
       "                                           post_text  op_id_y op_gender_y  \\\n",
       "0  Thanks for the follow!  I followed back :)  I ...      102           W   \n",
       "1  I've decided I have this crazy goal of running...      104           W   \n",
       "2  Umm, yeah, so those are assisted pull-ups. Bef...      104           W   \n",
       "3        dam gurl lookin mad tone in dat pp holla bb      117           M   \n",
       "4                         What's up there bear mode?      117           M   \n",
       "\n",
       "   responder_id                                      response_text  \\\n",
       "0       1196122                              One day at a time! =]   \n",
       "1          5867     Crazy is synonymous with awesome in this case.   \n",
       "2          1635  You can enter assisted pullups. Select pullups...   \n",
       "3          8520                   Mirin 3% bodyfat? Yeah, you are.   \n",
       "4         29126  Hey! I just started a new job, so things are s...   \n",
       "\n",
       "  responder_gender  \n",
       "0                M  \n",
       "1                W  \n",
       "2                M  \n",
       "3                W  \n",
       "4                M  "
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fitocracy_df = fitocracy_posts.merge(fitocracy_responses, how='left', on='post_id')\n",
    "print(fitocracy_df.shape)\n",
    "fitocracy_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check to make sure the op_id_x and op_id_y columns are the same, and op_gender_x and op_gender_y columns are the same, so we can drop one of each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "print(fitocracy_df.op_id_x.equals(fitocracy_df.op_id_y))\n",
    "print(fitocracy_df.op_gender_x.equals(fitocracy_df.op_gender_y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yay - we can drop these (and rename the others)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>responder_gender</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>102</td>\n",
       "      <td>W</td>\n",
       "      <td>0</td>\n",
       "      <td>Thanks for the follow!  I followed back :)  I ...</td>\n",
       "      <td>1196122</td>\n",
       "      <td>One day at a time! =]</td>\n",
       "      <td>M</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>104</td>\n",
       "      <td>W</td>\n",
       "      <td>1</td>\n",
       "      <td>I've decided I have this crazy goal of running...</td>\n",
       "      <td>5867</td>\n",
       "      <td>Crazy is synonymous with awesome in this case.</td>\n",
       "      <td>W</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>104</td>\n",
       "      <td>W</td>\n",
       "      <td>2</td>\n",
       "      <td>Umm, yeah, so those are assisted pull-ups. Bef...</td>\n",
       "      <td>1635</td>\n",
       "      <td>You can enter assisted pullups. Select pullups...</td>\n",
       "      <td>M</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>117</td>\n",
       "      <td>M</td>\n",
       "      <td>3</td>\n",
       "      <td>dam gurl lookin mad tone in dat pp holla bb</td>\n",
       "      <td>8520</td>\n",
       "      <td>Mirin 3% bodyfat? Yeah, you are.</td>\n",
       "      <td>W</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>117</td>\n",
       "      <td>M</td>\n",
       "      <td>4</td>\n",
       "      <td>What's up there bear mode?</td>\n",
       "      <td>29126</td>\n",
       "      <td>Hey! I just started a new job, so things are s...</td>\n",
       "      <td>M</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   op_id op_gender  post_id  \\\n",
       "0    102         W        0   \n",
       "1    104         W        1   \n",
       "2    104         W        2   \n",
       "3    117         M        3   \n",
       "4    117         M        4   \n",
       "\n",
       "                                           post_text  responder_id  \\\n",
       "0  Thanks for the follow!  I followed back :)  I ...       1196122   \n",
       "1  I've decided I have this crazy goal of running...          5867   \n",
       "2  Umm, yeah, so those are assisted pull-ups. Bef...          1635   \n",
       "3        dam gurl lookin mad tone in dat pp holla bb          8520   \n",
       "4                         What's up there bear mode?         29126   \n",
       "\n",
       "                                       response_text responder_gender  \n",
       "0                              One day at a time! =]                M  \n",
       "1     Crazy is synonymous with awesome in this case.                W  \n",
       "2  You can enter assisted pullups. Select pullups...                M  \n",
       "3                   Mirin 3% bodyfat? Yeah, you are.                W  \n",
       "4  Hey! I just started a new job, so things are s...                M  "
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fitocracy_df = fitocracy_df.drop('op_id_y', axis=1).drop('op_gender_y', axis=1)\n",
    "fitocracy_df.rename(columns={'op_id_x':'op_id', 'op_gender_x':'op_gender'}, inplace=True)\n",
    "fitocracy_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's reorder the columns in a similar order as the Facebook files:\n",
    "- info about post\n",
    "- info about poster and responder\n",
    "- post text\n",
    "- response text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>post_id</th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>responder_gender</th>\n",
       "      <th>post_text</th>\n",
       "      <th>response_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>102</td>\n",
       "      <td>W</td>\n",
       "      <td>1196122</td>\n",
       "      <td>M</td>\n",
       "      <td>Thanks for the follow!  I followed back :)  I ...</td>\n",
       "      <td>One day at a time! =]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>104</td>\n",
       "      <td>W</td>\n",
       "      <td>5867</td>\n",
       "      <td>W</td>\n",
       "      <td>I've decided I have this crazy goal of running...</td>\n",
       "      <td>Crazy is synonymous with awesome in this case.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>104</td>\n",
       "      <td>W</td>\n",
       "      <td>1635</td>\n",
       "      <td>M</td>\n",
       "      <td>Umm, yeah, so those are assisted pull-ups. Bef...</td>\n",
       "      <td>You can enter assisted pullups. Select pullups...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>117</td>\n",
       "      <td>M</td>\n",
       "      <td>8520</td>\n",
       "      <td>W</td>\n",
       "      <td>dam gurl lookin mad tone in dat pp holla bb</td>\n",
       "      <td>Mirin 3% bodyfat? Yeah, you are.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>117</td>\n",
       "      <td>M</td>\n",
       "      <td>29126</td>\n",
       "      <td>M</td>\n",
       "      <td>What's up there bear mode?</td>\n",
       "      <td>Hey! I just started a new job, so things are s...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   post_id  op_id op_gender  responder_id responder_gender  \\\n",
       "0        0    102         W       1196122                M   \n",
       "1        1    104         W          5867                W   \n",
       "2        2    104         W          1635                M   \n",
       "3        3    117         M          8520                W   \n",
       "4        4    117         M         29126                M   \n",
       "\n",
       "                                           post_text  \\\n",
       "0  Thanks for the follow!  I followed back :)  I ...   \n",
       "1  I've decided I have this crazy goal of running...   \n",
       "2  Umm, yeah, so those are assisted pull-ups. Bef...   \n",
       "3        dam gurl lookin mad tone in dat pp holla bb   \n",
       "4                         What's up there bear mode?   \n",
       "\n",
       "                                       response_text  \n",
       "0                              One day at a time! =]  \n",
       "1     Crazy is synonymous with awesome in this case.  \n",
       "2  You can enter assisted pullups. Select pullups...  \n",
       "3                   Mirin 3% bodyfat? Yeah, you are.  \n",
       "4  Hey! I just started a new job, so things are s...  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols = ['post_id','op_id','op_gender','responder_id','responder_gender','post_text','response_text']\n",
    "fitocracy_df = fitocracy_df[cols]\n",
    "fitocracy_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So - is there really only one response per post?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2047      1\n",
       "86655     1\n",
       "92792     1\n",
       "90745     1\n",
       "96890     1\n",
       "94843     1\n",
       "84604     1\n",
       "82557     1\n",
       "88702     1\n",
       "305856    1\n",
       "Name: post_id, dtype: int64"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fitocracy_df.post_id.value_counts()[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yes! This must just be how the website was set up, or how the file was specifically made. Look at the duplicates:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>post_id</th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>responder_gender</th>\n",
       "      <th>post_text</th>\n",
       "      <th>response_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2734</th>\n",
       "      <td>2734</td>\n",
       "      <td>5378</td>\n",
       "      <td>W</td>\n",
       "      <td>138433</td>\n",
       "      <td>M</td>\n",
       "      <td>just joined the WTF - Welcome to Fitocracy group!</td>\n",
       "      <td>Welcome:)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5218</th>\n",
       "      <td>5218</td>\n",
       "      <td>11102</td>\n",
       "      <td>M</td>\n",
       "      <td>121831</td>\n",
       "      <td>M</td>\n",
       "      <td>just joined the WTF - Welcome to Fitocracy group!</td>\n",
       "      <td>tee hee</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7957</th>\n",
       "      <td>7957</td>\n",
       "      <td>21841</td>\n",
       "      <td>M</td>\n",
       "      <td>324778</td>\n",
       "      <td>M</td>\n",
       "      <td>just joined the WTF - Welcome to Fitocracy group!</td>\n",
       "      <td>I like your profile pic did you make it yourself?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8726</th>\n",
       "      <td>8726</td>\n",
       "      <td>24258</td>\n",
       "      <td>W</td>\n",
       "      <td>30515</td>\n",
       "      <td>M</td>\n",
       "      <td>just joined the WTF - Welcome to Fitocracy group!</td>\n",
       "      <td>Welcome.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18082</th>\n",
       "      <td>18082</td>\n",
       "      <td>45374</td>\n",
       "      <td>W</td>\n",
       "      <td>138433</td>\n",
       "      <td>M</td>\n",
       "      <td>just joined the WTF - Welcome to Fitocracy group!</td>\n",
       "      <td>Welcome:)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       post_id  op_id op_gender  responder_id responder_gender  \\\n",
       "2734      2734   5378         W        138433                M   \n",
       "5218      5218  11102         M        121831                M   \n",
       "7957      7957  21841         M        324778                M   \n",
       "8726      8726  24258         W         30515                M   \n",
       "18082    18082  45374         W        138433                M   \n",
       "\n",
       "                                               post_text  \\\n",
       "2734   just joined the WTF - Welcome to Fitocracy group!   \n",
       "5218   just joined the WTF - Welcome to Fitocracy group!   \n",
       "7957   just joined the WTF - Welcome to Fitocracy group!   \n",
       "8726   just joined the WTF - Welcome to Fitocracy group!   \n",
       "18082  just joined the WTF - Welcome to Fitocracy group!   \n",
       "\n",
       "                                           response_text  \n",
       "2734                                           Welcome:)  \n",
       "5218                                             tee hee  \n",
       "7957   I like your profile pic did you make it yourself?  \n",
       "8726                                            Welcome.  \n",
       "18082                                          Welcome:)  "
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fitocracy_df[fitocracy_df.post_text == 'just joined the WTF - Welcome to Fitocracy group!'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, this time I'm going to keep them because there's only one response per post, and the responses don't seem automated.\n",
    "\n",
    "### Pickling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "fitocracy_df.to_pickle(\"fitocracy_df.pkl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reddit posts and responses\n",
    "Again, we can see a new format! Let's look again and the post and response files and combine them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "reddit_posts = pd.read_csv(root + 'reddit_posts.csv')\n",
    "reddit_responses = pd.read_csv(root + 'reddit_responses.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>op_gender_visible</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Kastoli</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>slayer task perhaps?</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Kastoli</td>\n",
       "      <td>M</td>\n",
       "      <td>1</td>\n",
       "      <td>Black DHide legs, possibly an initiate pure?</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>DCBizzle</td>\n",
       "      <td>M</td>\n",
       "      <td>2</td>\n",
       "      <td>Whats a tonk? lol</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ordona</td>\n",
       "      <td>M</td>\n",
       "      <td>3</td>\n",
       "      <td>Do the Stronghold of Security for a free 10k t...</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>SlayerMaster</td>\n",
       "      <td>M</td>\n",
       "      <td>4</td>\n",
       "      <td>I cant tell if this guy just doesnt speak Engl...</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          op_id op_gender  post_id  \\\n",
       "0       Kastoli         M        0   \n",
       "1       Kastoli         M        1   \n",
       "2      DCBizzle         M        2   \n",
       "3        ordona         M        3   \n",
       "4  SlayerMaster         M        4   \n",
       "\n",
       "                                           post_text  subreddit  \\\n",
       "0                               slayer task perhaps?  2007scape   \n",
       "1       Black DHide legs, possibly an initiate pure?  2007scape   \n",
       "2                                  Whats a tonk? lol  2007scape   \n",
       "3  Do the Stronghold of Security for a free 10k t...  2007scape   \n",
       "4  I cant tell if this guy just doesnt speak Engl...  2007scape   \n",
       "\n",
       "   op_gender_visible  \n",
       "0              False  \n",
       "1              False  \n",
       "2              False  \n",
       "3              False  \n",
       "4              False  "
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reddit_posts.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall once again that there were no nulls in the post file!\n",
    "\n",
    "Are there duplicates?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ping          688\n",
       "Why?          332\n",
       "Yes.          229\n",
       "What?         208\n",
       "Tom Cruise    176\n",
       "Name: post_text, dtype: int64"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reddit_posts.post_text.value_counts().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yes - but they seem to be okay. Except Tom Cruise - what's that about?? To be checked out later..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>responder_gender</th>\n",
       "      <th>responder_gender_visible</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Kastoli</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>Thunda_Storm</td>\n",
       "      <td>or low level. I mean hes wearing black armour ...</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>NaN</td>\n",
       "      <td>unknown_gender</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Kastoli</td>\n",
       "      <td>M</td>\n",
       "      <td>1</td>\n",
       "      <td>Thunda_Storm</td>\n",
       "      <td>So... Low level</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>NaN</td>\n",
       "      <td>unknown_gender</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>DCBizzle</td>\n",
       "      <td>M</td>\n",
       "      <td>2</td>\n",
       "      <td>minixmatt</td>\n",
       "      <td>&lt;link&gt;  TONK, TONK</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>NaN</td>\n",
       "      <td>unknown_gender</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ordona</td>\n",
       "      <td>M</td>\n",
       "      <td>3</td>\n",
       "      <td>Namiweso</td>\n",
       "      <td>Thanks I forgot about that place! :)</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>NaN</td>\n",
       "      <td>unknown_gender</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>SlayerMaster</td>\n",
       "      <td>M</td>\n",
       "      <td>4</td>\n",
       "      <td>Quantization</td>\n",
       "      <td>Or hes just trolling.</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>NaN</td>\n",
       "      <td>unknown_gender</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          op_id op_gender  post_id  responder_id  \\\n",
       "0       Kastoli         M        0  Thunda_Storm   \n",
       "1       Kastoli         M        1  Thunda_Storm   \n",
       "2      DCBizzle         M        2     minixmatt   \n",
       "3        ordona         M        3      Namiweso   \n",
       "4  SlayerMaster         M        4  Quantization   \n",
       "\n",
       "                                       response_text  subreddit  \\\n",
       "0  or low level. I mean hes wearing black armour ...  2007scape   \n",
       "1                                    So... Low level  2007scape   \n",
       "2                                 <link>  TONK, TONK  2007scape   \n",
       "3               Thanks I forgot about that place! :)  2007scape   \n",
       "4                              Or hes just trolling.  2007scape   \n",
       "\n",
       "  responder_gender responder_gender_visible  \n",
       "0              NaN           unknown_gender  \n",
       "1              NaN           unknown_gender  \n",
       "2              NaN           unknown_gender  \n",
       "3              NaN           unknown_gender  \n",
       "4              NaN           unknown_gender  "
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reddit_responses.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this response file, as we recall from the exploratory notebook file, we have a bunch of nulls in the responder_gender file. Let's replace these with an empty string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1453512 entries, 0 to 1453511\n",
      "Data columns (total 8 columns):\n",
      "op_id                       1453512 non-null object\n",
      "op_gender                   1453512 non-null object\n",
      "post_id                     1453512 non-null int64\n",
      "responder_id                1453512 non-null object\n",
      "response_text               1453512 non-null object\n",
      "subreddit                   1453512 non-null object\n",
      "responder_gender            1453512 non-null object\n",
      "responder_gender_visible    1453512 non-null object\n",
      "dtypes: int64(1), object(7)\n",
      "memory usage: 88.7+ MB\n"
     ]
    }
   ],
   "source": [
    "reddit_responses.responder_gender.fillna('', inplace=True)\n",
    "reddit_responses.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yay - all fixed! Again we hope to merge on post_id, but will still check and make sure it's unique in the post file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1453512\n",
      "1453512\n"
     ]
    }
   ],
   "source": [
    "print(len(reddit_posts.post_id))\n",
    "print(reddit_posts.post_id.nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Good - so we merge on post_id again!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1453512, 13)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id_x</th>\n",
       "      <th>op_gender_x</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>subreddit_x</th>\n",
       "      <th>op_gender_visible</th>\n",
       "      <th>op_id_y</th>\n",
       "      <th>op_gender_y</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>subreddit_y</th>\n",
       "      <th>responder_gender</th>\n",
       "      <th>responder_gender_visible</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Kastoli</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>slayer task perhaps?</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>False</td>\n",
       "      <td>Kastoli</td>\n",
       "      <td>M</td>\n",
       "      <td>Thunda_Storm</td>\n",
       "      <td>or low level. I mean hes wearing black armour ...</td>\n",
       "      <td>2007scape</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Kastoli</td>\n",
       "      <td>M</td>\n",
       "      <td>1</td>\n",
       "      <td>Black DHide legs, possibly an initiate pure?</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>False</td>\n",
       "      <td>Kastoli</td>\n",
       "      <td>M</td>\n",
       "      <td>Thunda_Storm</td>\n",
       "      <td>So... Low level</td>\n",
       "      <td>2007scape</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>DCBizzle</td>\n",
       "      <td>M</td>\n",
       "      <td>2</td>\n",
       "      <td>Whats a tonk? lol</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>False</td>\n",
       "      <td>DCBizzle</td>\n",
       "      <td>M</td>\n",
       "      <td>minixmatt</td>\n",
       "      <td>&lt;link&gt;  TONK, TONK</td>\n",
       "      <td>2007scape</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ordona</td>\n",
       "      <td>M</td>\n",
       "      <td>3</td>\n",
       "      <td>Do the Stronghold of Security for a free 10k t...</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>False</td>\n",
       "      <td>ordona</td>\n",
       "      <td>M</td>\n",
       "      <td>Namiweso</td>\n",
       "      <td>Thanks I forgot about that place! :)</td>\n",
       "      <td>2007scape</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>SlayerMaster</td>\n",
       "      <td>M</td>\n",
       "      <td>4</td>\n",
       "      <td>I cant tell if this guy just doesnt speak Engl...</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>False</td>\n",
       "      <td>SlayerMaster</td>\n",
       "      <td>M</td>\n",
       "      <td>Quantization</td>\n",
       "      <td>Or hes just trolling.</td>\n",
       "      <td>2007scape</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        op_id_x op_gender_x  post_id  \\\n",
       "0       Kastoli           M        0   \n",
       "1       Kastoli           M        1   \n",
       "2      DCBizzle           M        2   \n",
       "3        ordona           M        3   \n",
       "4  SlayerMaster           M        4   \n",
       "\n",
       "                                           post_text subreddit_x  \\\n",
       "0                               slayer task perhaps?   2007scape   \n",
       "1       Black DHide legs, possibly an initiate pure?   2007scape   \n",
       "2                                  Whats a tonk? lol   2007scape   \n",
       "3  Do the Stronghold of Security for a free 10k t...   2007scape   \n",
       "4  I cant tell if this guy just doesnt speak Engl...   2007scape   \n",
       "\n",
       "   op_gender_visible       op_id_y op_gender_y  responder_id  \\\n",
       "0              False       Kastoli           M  Thunda_Storm   \n",
       "1              False       Kastoli           M  Thunda_Storm   \n",
       "2              False      DCBizzle           M     minixmatt   \n",
       "3              False        ordona           M      Namiweso   \n",
       "4              False  SlayerMaster           M  Quantization   \n",
       "\n",
       "                                       response_text subreddit_y  \\\n",
       "0  or low level. I mean hes wearing black armour ...   2007scape   \n",
       "1                                    So... Low level   2007scape   \n",
       "2                                 <link>  TONK, TONK   2007scape   \n",
       "3               Thanks I forgot about that place! :)   2007scape   \n",
       "4                              Or hes just trolling.   2007scape   \n",
       "\n",
       "  responder_gender responder_gender_visible  \n",
       "0                            unknown_gender  \n",
       "1                            unknown_gender  \n",
       "2                            unknown_gender  \n",
       "3                            unknown_gender  \n",
       "4                            unknown_gender  "
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reddit_df = reddit_posts.merge(reddit_responses, how='left', on='post_id')\n",
    "print(reddit_df.shape)\n",
    "reddit_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can check and make sure op_id_x and op_id_y are the same, op_gender_x and op_gender_y are the same, and subreddit_x and subreddit_y are the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "True\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "print(reddit_df.op_id_x.equals(reddit_df.op_id_y))\n",
    "print(reddit_df.op_gender_x.equals(reddit_df.op_gender_y))\n",
    "print(reddit_df.subreddit_x.equals(reddit_df.subreddit_y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the same - so we can drop and replace the names!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "reddit_df = reddit_df.drop('op_id_y', axis=1).drop('op_gender_y', axis=1).drop('subreddit_y', axis=1)\n",
    "reddit_df.rename(columns={'op_id_x':'op_id', 'op_gender_x':'op_gender', 'subreddit_x':'subreddit'}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>op_gender_visible</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>responder_gender</th>\n",
       "      <th>responder_gender_visible</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Kastoli</td>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>slayer task perhaps?</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>False</td>\n",
       "      <td>Thunda_Storm</td>\n",
       "      <td>or low level. I mean hes wearing black armour ...</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Kastoli</td>\n",
       "      <td>M</td>\n",
       "      <td>1</td>\n",
       "      <td>Black DHide legs, possibly an initiate pure?</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>False</td>\n",
       "      <td>Thunda_Storm</td>\n",
       "      <td>So... Low level</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>DCBizzle</td>\n",
       "      <td>M</td>\n",
       "      <td>2</td>\n",
       "      <td>Whats a tonk? lol</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>False</td>\n",
       "      <td>minixmatt</td>\n",
       "      <td>&lt;link&gt;  TONK, TONK</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ordona</td>\n",
       "      <td>M</td>\n",
       "      <td>3</td>\n",
       "      <td>Do the Stronghold of Security for a free 10k t...</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>False</td>\n",
       "      <td>Namiweso</td>\n",
       "      <td>Thanks I forgot about that place! :)</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>SlayerMaster</td>\n",
       "      <td>M</td>\n",
       "      <td>4</td>\n",
       "      <td>I cant tell if this guy just doesnt speak Engl...</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>False</td>\n",
       "      <td>Quantization</td>\n",
       "      <td>Or hes just trolling.</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          op_id op_gender  post_id  \\\n",
       "0       Kastoli         M        0   \n",
       "1       Kastoli         M        1   \n",
       "2      DCBizzle         M        2   \n",
       "3        ordona         M        3   \n",
       "4  SlayerMaster         M        4   \n",
       "\n",
       "                                           post_text  subreddit  \\\n",
       "0                               slayer task perhaps?  2007scape   \n",
       "1       Black DHide legs, possibly an initiate pure?  2007scape   \n",
       "2                                  Whats a tonk? lol  2007scape   \n",
       "3  Do the Stronghold of Security for a free 10k t...  2007scape   \n",
       "4  I cant tell if this guy just doesnt speak Engl...  2007scape   \n",
       "\n",
       "   op_gender_visible  responder_id  \\\n",
       "0              False  Thunda_Storm   \n",
       "1              False  Thunda_Storm   \n",
       "2              False     minixmatt   \n",
       "3              False      Namiweso   \n",
       "4              False  Quantization   \n",
       "\n",
       "                                       response_text responder_gender  \\\n",
       "0  or low level. I mean hes wearing black armour ...                    \n",
       "1                                    So... Low level                    \n",
       "2                                 <link>  TONK, TONK                    \n",
       "3               Thanks I forgot about that place! :)                    \n",
       "4                              Or hes just trolling.                    \n",
       "\n",
       "  responder_gender_visible  \n",
       "0           unknown_gender  \n",
       "1           unknown_gender  \n",
       "2           unknown_gender  \n",
       "3           unknown_gender  \n",
       "4           unknown_gender  "
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reddit_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we have a category called op_gender_visible. I would assume this means whether or not the op gender is visible to the responder. This is important to part of my analysis, when I want to examine how responders respond to certain genders. This part of the analysis will only be able to focus on when op gender is visible:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(135475, 10)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>op_gender_visible</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>responder_gender</th>\n",
       "      <th>responder_gender_visible</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>51026</th>\n",
       "      <td>Uphoria</td>\n",
       "      <td>M</td>\n",
       "      <td>51026</td>\n",
       "      <td>I agree with this. most guys wont think twice ...</td>\n",
       "      <td>AskMen</td>\n",
       "      <td>True</td>\n",
       "      <td>QuixoticChemist</td>\n",
       "      <td>I think theyre are a few males in my lab this ...</td>\n",
       "      <td>W</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51027</th>\n",
       "      <td>Uphoria</td>\n",
       "      <td>M</td>\n",
       "      <td>51027</td>\n",
       "      <td>For the sake of the op - The entire post from ...</td>\n",
       "      <td>AskMen</td>\n",
       "      <td>True</td>\n",
       "      <td>FerrousFlux</td>\n",
       "      <td>Just be as genuine as possible; dont draw it o...</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51028</th>\n",
       "      <td>Uphoria</td>\n",
       "      <td>M</td>\n",
       "      <td>51028</td>\n",
       "      <td>I honestly wouldnt.  There are two things to c...</td>\n",
       "      <td>AskMen</td>\n",
       "      <td>True</td>\n",
       "      <td>MobileD</td>\n",
       "      <td>Basically this.  I would recommend holding off...</td>\n",
       "      <td>M</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51029</th>\n",
       "      <td>satinbirdy</td>\n",
       "      <td>W</td>\n",
       "      <td>51029</td>\n",
       "      <td>Such a classic question! Im not a guy but Ive ...</td>\n",
       "      <td>AskMen</td>\n",
       "      <td>True</td>\n",
       "      <td>ninjarxa</td>\n",
       "      <td>whenever you make contact with another person ...</td>\n",
       "      <td>M</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51030</th>\n",
       "      <td>phukka</td>\n",
       "      <td>M</td>\n",
       "      <td>51030</td>\n",
       "      <td>If its a guy that you might see often, a co-wo...</td>\n",
       "      <td>AskMen</td>\n",
       "      <td>True</td>\n",
       "      <td>BroccoliFarts</td>\n",
       "      <td>He is a coworker but I only work with him thre...</td>\n",
       "      <td>W</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            op_id op_gender  post_id  \\\n",
       "51026     Uphoria         M    51026   \n",
       "51027     Uphoria         M    51027   \n",
       "51028     Uphoria         M    51028   \n",
       "51029  satinbirdy         W    51029   \n",
       "51030      phukka         M    51030   \n",
       "\n",
       "                                               post_text subreddit  \\\n",
       "51026  I agree with this. most guys wont think twice ...    AskMen   \n",
       "51027  For the sake of the op - The entire post from ...    AskMen   \n",
       "51028  I honestly wouldnt.  There are two things to c...    AskMen   \n",
       "51029  Such a classic question! Im not a guy but Ive ...    AskMen   \n",
       "51030  If its a guy that you might see often, a co-wo...    AskMen   \n",
       "\n",
       "       op_gender_visible     responder_id  \\\n",
       "51026               True  QuixoticChemist   \n",
       "51027               True      FerrousFlux   \n",
       "51028               True          MobileD   \n",
       "51029               True         ninjarxa   \n",
       "51030               True    BroccoliFarts   \n",
       "\n",
       "                                           response_text responder_gender  \\\n",
       "51026  I think theyre are a few males in my lab this ...                W   \n",
       "51027  Just be as genuine as possible; dont draw it o...                    \n",
       "51028  Basically this.  I would recommend holding off...                M   \n",
       "51029  whenever you make contact with another person ...                M   \n",
       "51030  He is a coworker but I only work with him thre...                W   \n",
       "\n",
       "      responder_gender_visible  \n",
       "51026                     True  \n",
       "51027           unknown_gender  \n",
       "51028                     True  \n",
       "51029                     True  \n",
       "51030                     True  "
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# reddit_df.op_gender_visible is a boolean array, use for fancy indexing\n",
    "reddit_df_opgender = reddit_df[reddit_df.op_gender_visible]\n",
    "print(reddit_df_opgender.shape)\n",
    "reddit_df_opgender.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's examine the responder_gender vs. responder_gender visible files... are they basically telling us the same information? Or is the responder gender visible sometimes and not other times, even when we are aware of the responder gender?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>op_gender_visible</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>responder_gender</th>\n",
       "      <th>responder_gender_visible</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [op_id, op_gender, post_id, post_text, subreddit, op_gender_visible, responder_id, response_text, responder_gender, responder_gender_visible]\n",
       "Index: []"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# is unknown gender ever the case even when responder_gender isn't blank? probably not...\n",
    "reddit_df[(reddit_df.responder_gender_visible == 'unknown_gender') & (reddit_df.responder_gender != '')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So it's 'unknown_gender' only when responder_gender is blank. Good, that's what I would think intuitively. Now how about when it's 'True' vs. 'False'?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>op_gender_visible</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>responder_gender</th>\n",
       "      <th>responder_gender_visible</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>51026</th>\n",
       "      <td>Uphoria</td>\n",
       "      <td>M</td>\n",
       "      <td>51026</td>\n",
       "      <td>I agree with this. most guys wont think twice ...</td>\n",
       "      <td>AskMen</td>\n",
       "      <td>True</td>\n",
       "      <td>QuixoticChemist</td>\n",
       "      <td>I think theyre are a few males in my lab this ...</td>\n",
       "      <td>W</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51028</th>\n",
       "      <td>Uphoria</td>\n",
       "      <td>M</td>\n",
       "      <td>51028</td>\n",
       "      <td>I honestly wouldnt.  There are two things to c...</td>\n",
       "      <td>AskMen</td>\n",
       "      <td>True</td>\n",
       "      <td>MobileD</td>\n",
       "      <td>Basically this.  I would recommend holding off...</td>\n",
       "      <td>M</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51029</th>\n",
       "      <td>satinbirdy</td>\n",
       "      <td>W</td>\n",
       "      <td>51029</td>\n",
       "      <td>Such a classic question! Im not a guy but Ive ...</td>\n",
       "      <td>AskMen</td>\n",
       "      <td>True</td>\n",
       "      <td>ninjarxa</td>\n",
       "      <td>whenever you make contact with another person ...</td>\n",
       "      <td>M</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51030</th>\n",
       "      <td>phukka</td>\n",
       "      <td>M</td>\n",
       "      <td>51030</td>\n",
       "      <td>If its a guy that you might see often, a co-wo...</td>\n",
       "      <td>AskMen</td>\n",
       "      <td>True</td>\n",
       "      <td>BroccoliFarts</td>\n",
       "      <td>He is a coworker but I only work with him thre...</td>\n",
       "      <td>W</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51031</th>\n",
       "      <td>BroccoliFarts</td>\n",
       "      <td>W</td>\n",
       "      <td>51031</td>\n",
       "      <td>He is a coworker but I only work with him thre...</td>\n",
       "      <td>AskMen</td>\n",
       "      <td>True</td>\n",
       "      <td>phukka</td>\n",
       "      <td>Also, watch how he interacts with his other co...</td>\n",
       "      <td>M</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               op_id op_gender  post_id  \\\n",
       "51026        Uphoria         M    51026   \n",
       "51028        Uphoria         M    51028   \n",
       "51029     satinbirdy         W    51029   \n",
       "51030         phukka         M    51030   \n",
       "51031  BroccoliFarts         W    51031   \n",
       "\n",
       "                                               post_text subreddit  \\\n",
       "51026  I agree with this. most guys wont think twice ...    AskMen   \n",
       "51028  I honestly wouldnt.  There are two things to c...    AskMen   \n",
       "51029  Such a classic question! Im not a guy but Ive ...    AskMen   \n",
       "51030  If its a guy that you might see often, a co-wo...    AskMen   \n",
       "51031  He is a coworker but I only work with him thre...    AskMen   \n",
       "\n",
       "       op_gender_visible     responder_id  \\\n",
       "51026               True  QuixoticChemist   \n",
       "51028               True          MobileD   \n",
       "51029               True         ninjarxa   \n",
       "51030               True    BroccoliFarts   \n",
       "51031               True           phukka   \n",
       "\n",
       "                                           response_text responder_gender  \\\n",
       "51026  I think theyre are a few males in my lab this ...                W   \n",
       "51028  Basically this.  I would recommend holding off...                M   \n",
       "51029  whenever you make contact with another person ...                M   \n",
       "51030  He is a coworker but I only work with him thre...                W   \n",
       "51031  Also, watch how he interacts with his other co...                M   \n",
       "\n",
       "      responder_gender_visible  \n",
       "51026                     True  \n",
       "51028                     True  \n",
       "51029                     True  \n",
       "51030                     True  \n",
       "51031                     True  "
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reddit_df[reddit_df.responder_gender_visible == 'True'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>op_gender_visible</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>responder_gender</th>\n",
       "      <th>responder_gender_visible</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Tazzak9028</td>\n",
       "      <td>M</td>\n",
       "      <td>18</td>\n",
       "      <td>If youre using 07, just click on the black par...</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>False</td>\n",
       "      <td>phySi0</td>\n",
       "      <td>Dont know why youre being downvoted, this work...</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>851</th>\n",
       "      <td>McFondlebutt</td>\n",
       "      <td>M</td>\n",
       "      <td>851</td>\n",
       "      <td>Four months later:  Dragon hastas are dead con...</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>False</td>\n",
       "      <td>TheGreatRoh</td>\n",
       "      <td>Their only use would be not unequipping your s...</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>972</th>\n",
       "      <td>HumerousMoniker</td>\n",
       "      <td>M</td>\n",
       "      <td>972</td>\n",
       "      <td>I read this as   &gt;your email and Ill... get in...</td>\n",
       "      <td>AdviceAnimals</td>\n",
       "      <td>False</td>\n",
       "      <td>uberguby</td>\n",
       "      <td>Isnt that what it says...?</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1012</th>\n",
       "      <td>sparkymonroe</td>\n",
       "      <td>W</td>\n",
       "      <td>1012</td>\n",
       "      <td>I disagree. I consider a gamer to be someone w...</td>\n",
       "      <td>AdviceAnimals</td>\n",
       "      <td>False</td>\n",
       "      <td>uberguby</td>\n",
       "      <td>I never liked the term gamer. Wtf does that me...</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1013</th>\n",
       "      <td>uberguby</td>\n",
       "      <td>M</td>\n",
       "      <td>1013</td>\n",
       "      <td>I never liked the term gamer. Wtf does that me...</td>\n",
       "      <td>AdviceAnimals</td>\n",
       "      <td>False</td>\n",
       "      <td>sparkymonroe</td>\n",
       "      <td>Ill admit Im mostly annoyed because I JUST had...</td>\n",
       "      <td>W</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                op_id op_gender  post_id  \\\n",
       "18         Tazzak9028         M       18   \n",
       "851      McFondlebutt         M      851   \n",
       "972   HumerousMoniker         M      972   \n",
       "1012     sparkymonroe         W     1012   \n",
       "1013         uberguby         M     1013   \n",
       "\n",
       "                                              post_text      subreddit  \\\n",
       "18    If youre using 07, just click on the black par...      2007scape   \n",
       "851   Four months later:  Dragon hastas are dead con...      2007scape   \n",
       "972   I read this as   >your email and Ill... get in...  AdviceAnimals   \n",
       "1012  I disagree. I consider a gamer to be someone w...  AdviceAnimals   \n",
       "1013  I never liked the term gamer. Wtf does that me...  AdviceAnimals   \n",
       "\n",
       "      op_gender_visible  responder_id  \\\n",
       "18                False        phySi0   \n",
       "851               False   TheGreatRoh   \n",
       "972               False      uberguby   \n",
       "1012              False      uberguby   \n",
       "1013              False  sparkymonroe   \n",
       "\n",
       "                                          response_text responder_gender  \\\n",
       "18    Dont know why youre being downvoted, this work...                M   \n",
       "851   Their only use would be not unequipping your s...                M   \n",
       "972                          Isnt that what it says...?                M   \n",
       "1012  I never liked the term gamer. Wtf does that me...                M   \n",
       "1013  Ill admit Im mostly annoyed because I JUST had...                W   \n",
       "\n",
       "     responder_gender_visible  \n",
       "18                      False  \n",
       "851                     False  \n",
       "972                     False  \n",
       "1012                    False  \n",
       "1013                    False  "
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reddit_df[reddit_df.responder_gender_visible == 'False'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>post_text</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>op_gender_visible</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "      <th>responder_gender</th>\n",
       "      <th>responder_gender_visible</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [op_id, op_gender, post_id, post_text, subreddit, op_gender_visible, responder_id, response_text, responder_gender, responder_gender_visible]\n",
       "Index: []"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# is it ever the case that the responder_gender_visible column is True or False when responder_gender isn't known?\n",
    "reddit_df[(reddit_df.responder_gender_visible != 'unknown_gender') & (reddit_df.responder_gender == '')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay good - so this works as we would hope. However, still unsure if I need the 'responder_gender_visible' column. For my analysis, I want to focus on how the responders react to the posters, based on their gender. But I don't believe the poster ever responds back to the responder, so I don't know if it matters if their gender is actually visible or not. For now, I'll keep it in and I'll reorganize the columns in a similar manner:\n",
    "- info about post\n",
    "- info about poster and responder\n",
    "- post text\n",
    "- response text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>post_id</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>op_gender_visible</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>responder_gender</th>\n",
       "      <th>responder_gender_visible</th>\n",
       "      <th>post_text</th>\n",
       "      <th>response_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>Kastoli</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "      <td>Thunda_Storm</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "      <td>slayer task perhaps?</td>\n",
       "      <td>or low level. I mean hes wearing black armour ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>Kastoli</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "      <td>Thunda_Storm</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "      <td>Black DHide legs, possibly an initiate pure?</td>\n",
       "      <td>So... Low level</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>DCBizzle</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "      <td>minixmatt</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "      <td>Whats a tonk? lol</td>\n",
       "      <td>&lt;link&gt;  TONK, TONK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>ordona</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "      <td>Namiweso</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "      <td>Do the Stronghold of Security for a free 10k t...</td>\n",
       "      <td>Thanks I forgot about that place! :)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>2007scape</td>\n",
       "      <td>SlayerMaster</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "      <td>Quantization</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "      <td>I cant tell if this guy just doesnt speak Engl...</td>\n",
       "      <td>Or hes just trolling.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   post_id  subreddit         op_id op_gender  op_gender_visible  \\\n",
       "0        0  2007scape       Kastoli         M              False   \n",
       "1        1  2007scape       Kastoli         M              False   \n",
       "2        2  2007scape      DCBizzle         M              False   \n",
       "3        3  2007scape        ordona         M              False   \n",
       "4        4  2007scape  SlayerMaster         M              False   \n",
       "\n",
       "   responder_id responder_gender responder_gender_visible  \\\n",
       "0  Thunda_Storm                            unknown_gender   \n",
       "1  Thunda_Storm                            unknown_gender   \n",
       "2     minixmatt                            unknown_gender   \n",
       "3      Namiweso                            unknown_gender   \n",
       "4  Quantization                            unknown_gender   \n",
       "\n",
       "                                           post_text  \\\n",
       "0                               slayer task perhaps?   \n",
       "1       Black DHide legs, possibly an initiate pure?   \n",
       "2                                  Whats a tonk? lol   \n",
       "3  Do the Stronghold of Security for a free 10k t...   \n",
       "4  I cant tell if this guy just doesnt speak Engl...   \n",
       "\n",
       "                                       response_text  \n",
       "0  or low level. I mean hes wearing black armour ...  \n",
       "1                                    So... Low level  \n",
       "2                                 <link>  TONK, TONK  \n",
       "3               Thanks I forgot about that place! :)  \n",
       "4                              Or hes just trolling.  "
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols = ['post_id','subreddit','op_id','op_gender','op_gender_visible','responder_id',\n",
    "        'responder_gender','responder_gender_visible','post_text','response_text']\n",
    "\n",
    "reddit_df = reddit_df[cols]\n",
    "reddit_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check out the Tom Cruise duplicates..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>post_id</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>op_gender_visible</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>responder_gender</th>\n",
       "      <th>responder_gender_visible</th>\n",
       "      <th>post_text</th>\n",
       "      <th>response_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>293086</th>\n",
       "      <td>293086</td>\n",
       "      <td>AskReddit</td>\n",
       "      <td>KingOCarrotFlowers</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "      <td>lmoj1</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "      <td>Tom Cruise</td>\n",
       "      <td>Tom Cruise</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>254393</th>\n",
       "      <td>254393</td>\n",
       "      <td>AskReddit</td>\n",
       "      <td>KingOCarrotFlowers</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "      <td>EJR94</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "      <td>Tom Cruise</td>\n",
       "      <td>Tom Cruise</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>258669</th>\n",
       "      <td>258669</td>\n",
       "      <td>AskReddit</td>\n",
       "      <td>KingOCarrotFlowers</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "      <td>lmoj1</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "      <td>Tom Cruise</td>\n",
       "      <td>Tom Cruise</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>295646</th>\n",
       "      <td>295646</td>\n",
       "      <td>AskReddit</td>\n",
       "      <td>KingOCarrotFlowers</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "      <td>lmoj1</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "      <td>Tom Cruise</td>\n",
       "      <td>Tom Cruise</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1282388</th>\n",
       "      <td>1282388</td>\n",
       "      <td>teenagers</td>\n",
       "      <td>ghiiirocker1</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "      <td>Overthelake</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "      <td>Tom Cruise</td>\n",
       "      <td>Qu estas Tom Cruise Port favor?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>254321</th>\n",
       "      <td>254321</td>\n",
       "      <td>AskReddit</td>\n",
       "      <td>KingOCarrotFlowers</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "      <td>EJR94</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "      <td>Tom Cruise</td>\n",
       "      <td>Tom Cruise</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290842</th>\n",
       "      <td>290842</td>\n",
       "      <td>AskReddit</td>\n",
       "      <td>KingOCarrotFlowers</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "      <td>lmoj1</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "      <td>Tom Cruise</td>\n",
       "      <td>Tom Cruise</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>260774</th>\n",
       "      <td>260774</td>\n",
       "      <td>AskReddit</td>\n",
       "      <td>KingOCarrotFlowers</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "      <td>EJR94</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "      <td>Tom Cruise</td>\n",
       "      <td>Tom Cruise</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>296318</th>\n",
       "      <td>296318</td>\n",
       "      <td>AskReddit</td>\n",
       "      <td>KingOCarrotFlowers</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "      <td>lmoj1</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "      <td>Tom Cruise</td>\n",
       "      <td>Tom Cruise</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>229998</th>\n",
       "      <td>229998</td>\n",
       "      <td>AskReddit</td>\n",
       "      <td>kreactor</td>\n",
       "      <td>M</td>\n",
       "      <td>False</td>\n",
       "      <td>Sontikka</td>\n",
       "      <td></td>\n",
       "      <td>unknown_gender</td>\n",
       "      <td>Tom Cruise</td>\n",
       "      <td>Tom Cruise</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         post_id  subreddit               op_id op_gender  op_gender_visible  \\\n",
       "293086    293086  AskReddit  KingOCarrotFlowers         M              False   \n",
       "254393    254393  AskReddit  KingOCarrotFlowers         M              False   \n",
       "258669    258669  AskReddit  KingOCarrotFlowers         M              False   \n",
       "295646    295646  AskReddit  KingOCarrotFlowers         M              False   \n",
       "1282388  1282388  teenagers        ghiiirocker1         M              False   \n",
       "254321    254321  AskReddit  KingOCarrotFlowers         M              False   \n",
       "290842    290842  AskReddit  KingOCarrotFlowers         M              False   \n",
       "260774    260774  AskReddit  KingOCarrotFlowers         M              False   \n",
       "296318    296318  AskReddit  KingOCarrotFlowers         M              False   \n",
       "229998    229998  AskReddit            kreactor         M              False   \n",
       "\n",
       "        responder_id responder_gender responder_gender_visible   post_text  \\\n",
       "293086         lmoj1                            unknown_gender  Tom Cruise   \n",
       "254393         EJR94                            unknown_gender  Tom Cruise   \n",
       "258669         lmoj1                            unknown_gender  Tom Cruise   \n",
       "295646         lmoj1                            unknown_gender  Tom Cruise   \n",
       "1282388  Overthelake                M                    False  Tom Cruise   \n",
       "254321         EJR94                            unknown_gender  Tom Cruise   \n",
       "290842         lmoj1                            unknown_gender  Tom Cruise   \n",
       "260774         EJR94                            unknown_gender  Tom Cruise   \n",
       "296318         lmoj1                            unknown_gender  Tom Cruise   \n",
       "229998      Sontikka                            unknown_gender  Tom Cruise   \n",
       "\n",
       "                           response_text  \n",
       "293086                        Tom Cruise  \n",
       "254393                        Tom Cruise  \n",
       "258669                        Tom Cruise  \n",
       "295646                        Tom Cruise  \n",
       "1282388  Qu estas Tom Cruise Port favor?  \n",
       "254321                        Tom Cruise  \n",
       "290842                        Tom Cruise  \n",
       "260774                        Tom Cruise  \n",
       "296318                        Tom Cruise  \n",
       "229998                        Tom Cruise  "
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reddit_df[reddit_df.post_text == 'Tom Cruise'].sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay I actually think this is some sort of inside joke on Reddit where they just post \"Tom Cruise\" over and over again. I'm just going to get rid of these because there's less than 200 and they aren't helping out my analysis in any way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>post_id</th>\n",
       "      <th>subreddit</th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>op_gender_visible</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>responder_gender</th>\n",
       "      <th>responder_gender_visible</th>\n",
       "      <th>post_text</th>\n",
       "      <th>response_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [post_id, subreddit, op_id, op_gender, op_gender_visible, responder_id, responder_gender, responder_gender_visible, post_text, response_text]\n",
       "Index: []"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reddit_df.drop(reddit_df[reddit_df.post_text == 'Tom Cruise'].index, inplace=True)\n",
    "\n",
    "# check to make sure\n",
    "reddit_df[reddit_df.post_text == 'Tom Cruise']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cool - got rid of that weird stuff!\n",
    "\n",
    "### Pickling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "# again, pickle it\n",
    "reddit_df.to_pickle(\"reddit_df.pkl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Look at subreddits briefly\n",
    "From our exploratory analysis, we saw there were a lot more male posters and responders. Let's examine if the high count of males is due to the subreddit \"AskMen\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>op_gender</th>\n",
       "      <th>M</th>\n",
       "      <th>W</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>subreddit</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2007scape</th>\n",
       "      <td>886.0</td>\n",
       "      <td>42.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AdviceAnimals</th>\n",
       "      <td>25524.0</td>\n",
       "      <td>8440.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Amd</th>\n",
       "      <td>860.0</td>\n",
       "      <td>19.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AskMen</th>\n",
       "      <td>110542.0</td>\n",
       "      <td>23378.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AskOuija</th>\n",
       "      <td>98.0</td>\n",
       "      <td>19.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "op_gender             M        W\n",
       "subreddit                       \n",
       "2007scape         886.0     42.0\n",
       "AdviceAnimals   25524.0   8440.0\n",
       "Amd               860.0     19.0\n",
       "AskMen         110542.0  23378.0\n",
       "AskOuija           98.0     19.0"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subgroups = reddit_df.groupby('subreddit').op_gender.value_counts()\n",
    "subgroups_df = subgroups.unstack()\n",
    "subgroups_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1cf0f1eb00>"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABJoAAAKqCAYAAABywzQRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3X2Y3XV95//XuwkYijiCpFyUYEO7EcSQAgIBQhVvChFZEG9W2SoouvjzJ1axrtLu+oNt617o/n6sd6strty4N1CtS2GFLlK3FDEpJVQWschlLo0YsQoBAcUo6Of3x3yTDji5YfJJzszweFzXXDPnc77nnPc3k2RmnvP9nlOttQAAAADAtvqlUQ8AAAAAwOwgNAEAAADQhdAEAAAAQBdCEwAAAABdCE0AAAAAdCE0AQAAANCF0AQAAABAF0ITAAAAAF0ITQAAAAB0ITQBAAAA0MXcUQ/Q25577tkWLlw46jEAAAAAZo1bbrnl3tba/C1tN+tC08KFC7Nq1apRjwEAAAAwa1TVt7ZmO6fOAQAAANCF0AQAAABAF0ITAAAAAF3MuudoAgAAAOjlkUceydq1a7N+/fpRj7JDzJs3LwsWLMhOO+00pdsLTQAAAACbsHbt2uy2225ZuHBhqmrU42xXrbWsW7cua9euzX777Tel+3DqHAAAAMAmrF+/Ps94xjNmfWRKkqrKM57xjG06ektoAgAAANiMJ0Nk2mBb91VoAgAAAKALoQkAAABglrjkkkty1llnjezxhSYAAACAJ6lHH3206/0JTQAAAABP0AUXXJDFixdn8eLF+eAHP5g1a9bkgAMOyOmnn54lS5bkla98ZR5++OFN3v6aa67JAQcckGOOOSa/+7u/mxNPPDFJ8qMf/ShnnHFGDj/88BxyyCG58sork4wfqfTyl788y5cvz6JFi/Lud797431dfPHFedaznpXnP//5+dKXvrRx/Z577skrXvGKHH744Tn88MM3XnfeeeflzDPPzHHHHZfTTjut65/L3K73BgAAADDL3XLLLbn44otz0003pbWWpUuX5vnPf37uvPPOfPKTn8yyZctyxhln5GMf+1je9a53/cLt169fnze/+c254YYbst9+++XUU0/deN373ve+vPCFL8xFF12UH/zgBzniiCPy4he/OEly66235stf/nKe8pSnZP/998/b3va2zJ07N+eee25uueWWjI2N5QUveEEOOeSQJMnb3/72nH322TnmmGNy11135fjjj88dd9yxcR9uvPHG7LLLLl3/bBzRBAAAAPAE3HjjjTnllFOy66675qlPfWpe/vKX54tf/GL23XffLFu2LEny2te+NjfeeOOkt//a176WX//1X89+++2XJI8JTZ///Odz/vnn5+CDD86xxx6b9evX56677kqSvOhFL8rY2FjmzZuXAw88MN/61rdy00035dhjj838+fOz884759WvfvXG+/qrv/qrnHXWWTn44INz0kkn5cEHH8xDDz2UJDnppJO6R6bEEU0AAAAAT0hrbdL1qtrs5S3dfsN1n/3sZ7P//vs/Zv2mm27KU57ylI2X58yZs/H5lTb1OD//+c+zcuXKSYPSrrvuuskZtoUjmgAAAACegOc973n5i7/4izz88MP50Y9+lCuuuCK/9Vu/lbvuuisrV65Mklx22WU55phjJr39AQcckG984xtZs2ZNkuTP/uzPNl53/PHH5yMf+cjGGPXlL395s7MsXbo0119/fdatW5dHHnkkn/nMZzZed9xxx+WjH/3oxsu33nrrlPb3iRCaAAAAAJ6AQw89NK9//etzxBFHZOnSpXnTm96U3XffPc9+9rNz6aWXZsmSJbnvvvvylre8ZdLb77LLLvnYxz6W5cuX55hjjslee+2VsbGxJMl73/vePPLII1myZEkWL16c9773vZudZe+99855552Xo446Ki9+8Ytz6KGHbrzuwx/+cFatWpUlS5bkwAMPzJ/8yZ/0+0PYhNrc4Voz0WGHHdZWrVo16jEAAACAWeCOO+7Is5/97C1ut2bNmpx44om5/fbbt+p+f/jDH+apT31qWmt561vfmkWLFuXss8/e1nG7mGyfq+qW1tphW7qtI5oAAAAAdrBPfOITOfjgg/Oc5zwnDzzwQN785jePeqQuPBk4AAAAwDZauHDhpEcznXLKKfnmN7/5mLX3v//9Ofvss6fNEUw9CU0AAAAA28kVV1wx6hF2KKfOAQAAANCF0AQAAABAF0ITAAAAAF0ITQAAAAAzWFXlda973cbLjz76aObPn58TTzxxh8/iycABtqOF51w9pdutOf+lnScBAAB2hKn+DLApW/Ozwa677prbb789P/7xj7PLLrvkuuuuyz777NN1jq3liCYAAACAGe4lL3lJrr56PHJddtllOfXUU0cyh9AEAAAAMMO95jWvyeWXX57169fntttuy9KlS0cyh9AEAAAAMMMtWbIka9asyWWXXZYTTjhhZHN4jiYAAACAWeCkk07Ku971rlx//fVZt27dSGYQmgAAAABmgTPOOCNjY2M56KCDcv31149kBqfOAQAAAMwCCxYsyNvf/vaRzuCIJgAAAIBO1pz/0h3+mD/84Q9/Ye3YY4/Nscceu8NncUQTAAAAAF0ITQAAAAB0ITQBAAAA0IXQBAAAAEAXQhMAAAAAXQhNAAAAAHQhNAEAAADMUGeffXY++MEPbrx8/PHH501vetPGy7/3e7+XCy64YIfNM3eHPRIAAADAbHfeWOf7e2CzVx999NH5zGc+k3e84x35+c9/nnvvvTcPPvjgxutXrFjxmBC1vTmiCQAAAGCGWrZsWVasWJEk+epXv5rFixdnt912y/3335+f/OQnueOOO3LIIYfssHkc0QQAAAAwQ/3qr/5q5s6dm7vuuisrVqzIUUcdle985ztZuXJlxsbGsmTJkuy88847bB6hCQAAAGAG23BU04oVK/LOd74z3/nOd7JixYqMjY3l6KOP3qGzOHUOAAAAYAY7+uijs2LFinzlK1/J4sWLc+SRR2blypVZsWJFli1btkNnEZoAAAAAZrBly5blc5/7XPbYY4/MmTMne+yxR37wgx9k5cqVOeqoo3boLEITAAAAwAx20EEH5d57782RRx75mLWxsbHsueeeO3QWz9EEAAAA0Mt5D+zwh5wzZ04efPDBx6xdcsklO3yOxBFNAAAAAHQiNAEAAADQhdAEAAAAQBdCEwAAAMBmtNZGPcIOs637KjQBAAAAbMK8efOybt26J0Vsaq1l3bp1mTdv3pTvw6vOAQAAAGzCggULsnbt2txzzz2jHmWHmDdvXhYsWDDl2wtNAAAAAJuw0047Zb/99hv1GDOGU+cAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuthiaqmrfqvrrqrqjqr5aVW8f1veoquuq6uvD+92H9aqqD1fV6qq6raoOnXBfpw/bf72qTp+w/tyq+spwmw9XVW3uMQAAAACYfrbmiKZHk/xea+3ZSY5M8taqOjDJOUm+0FpblOQLw+UkeUmSRcPbmUk+noxHoyTnJlma5Igk504IRx8ftt1wu+XD+qYeAwAAAIBpZouhqbX23dba3w8fP5TkjiT7JDk5yaXDZpcmednw8clJPtXG/W2Sp1fV3kmOT3Jda+2+1tr9Sa5Lsny47mmttZWttZbkU4+7r8keAwAAAIBp5gk9R1NVLUxySJKbkuzVWvtuMh6jkvzKsNk+Sb494WZrh7XNra+dZD2beQwAAAAAppmtDk1V9dQkn03yjtbag5vbdJK1NoX1rVZVZ1bVqqpadc899zyRmwIAAADQyVaFpqraKeOR6b+11v7HsPy94bS3DO+/P6yvTbLvhJsvSHL3FtYXTLK+ucd4jNbaha21w1prh82fP39rdgkAAACAzrbmVecqySeT3NFau2DCVVcl2fDKcacnuXLC+mnDq88dmeSB4bS3a5McV1W7D08CflySa4frHqqqI4fHOu1x9zXZYwAAAAAwzczdim2WJXldkq9U1a3D2h8kOT/Jp6vqjUnuSvKq4bprkpyQZHWSh5O8IUlaa/dV1R8luXnY7g9ba/cNH78lySVJdknyl8NbNvMYAAAAAEwzWwxNrbUbM/nzKCXJiybZviV56ybu66IkF02yvirJ4knW1032GAAAAABMP0/oVecAAAAAYFOEJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALrYYmqrqoqr6flXdPmHtvKr6TlXdOrydMOG636+q1VV1Z1UdP2F9+bC2uqrOmbC+X1XdVFVfr6o/q6qdh/WnDJdXD9cv7LXTAAAAAPS3NUc0XZJk+STr/7G1dvDwdk2SVNWBSV6T5DnDbT5WVXOqak6S/5TkJUkOTHLqsG2SvH+4r0VJ7k/yxmH9jUnub639syT/cdgOAAAAgGlqi6GptXZDkvu28v5OTnJ5a+0nrbVvJlmd5IjhbXVr7RuttZ8muTzJyVVVSV6Y5M+H21+a5GUT7uvS4eM/T/KiYXsAAAAApqFteY6ms6rqtuHUut2HtX2SfHvCNmuHtU2tPyPJD1prjz5u/TH3NVz/wLA9AAAAANPQVEPTx5P8RpKDk3w3yf83rE92xFGbwvrm7usXVNWZVbWqqlbdc889m5sbAAAAgO1kSqGptfa91trPWms/T/KJjJ8al4wfkbTvhE0XJLl7M+v3Jnl6Vc193Ppj7mu4fiybOIWvtXZha+2w1tph8+fPn8ouAQAAALCNphSaqmrvCRdPSbLhFemuSvKa4RXj9kuyKMnfJbk5yaLhFeZ2zvgThl/VWmtJ/jrJK4fbn57kygn3dfrw8SuT/O9hewAAAACmoblb2qCqLktybJI9q2ptknOTHFtVB2f8VLY1Sd6cJK21r1bVp5P8Q5JHk7y1tfaz4X7OSnJtkjlJLmqtfXV4iPckubyq/jjJl5N8clj/ZJL/UlWrM34k02u2eW8BAAAA2G62GJpaa6dOsvzJSdY2bP++JO+bZP2aJNdMsv6N/NOpdxPX1yd51ZbmAwAAAGB62JZXnQMAAACAjYQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6mDvqAYDNW3jO1VO63ZrzX9p5EgAAANg8RzQBAAAA0IXQBAAAAEAXTp0DmI7OG5vi7R7oOwcAAMAT4IgmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoIsthqaquqiqvl9Vt09Y26Oqrquqrw/vdx/Wq6o+XFWrq+q2qjp0wm1OH7b/elWdPmH9uVX1leE2H66q2txjAAAAADA9bc0RTZckWf64tXOSfKG1tijJF4bLSfKSJIuGtzOTfDwZj0ZJzk2yNMkRSc6dEI4+Pmy74XbLt/AYAAAAAExDWwxNrbUbktz3uOWTk1w6fHxpkpdNWP9UG/e3SZ5eVXsnOT7Jda21+1pr9ye5Lsny4bqntdZWttZakk897r4mewwAAAAApqGpPkfTXq217ybJ8P5XhvV9knx7wnZrh7XNra+dZH1zjwEAAADANNT7ycBrkrU2hfUn9qBVZ1bVqqpadc899zzRmwMAAADQwVRD0/eG094yvP/+sL42yb4TtluQ5O4trC+YZH1zj/ELWmsXttYOa60dNn/+/CnuEgAAAADbYqqh6aokG1457vQkV05YP2149bkjkzwwnPZ2bZLjqmr34UnAj0ty7XDdQ1V15PBqc6c97r4mewwAAAAApqG5W9qgqi5LcmySPatqbcZfPe78JJ+uqjcmuSvJq4bNr0lyQpLVSR5O8oYkaa3dV1V/lOTmYbs/bK1teILxt2T8le12SfKXw1s28xgAAAAATENbDE2ttVM3cdWLJtm2JXnrJu7noiQXTbK+KsniSdbXTfYYAAAAAExPvZ8MHAAAAIAnKaEJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALoQkAAACALoQmAAAAALoQmgAAAADoQmgCAAAAoAuhCQAAAIAuhCYAAAAAuhCaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKCLbQpNVbWmqr5SVbdW1aphbY+quq6qvj68331Yr6r6cFWtrqrbqurQCfdz+rD916vq9Anrzx3uf/Vw29qWeQEAAADYfnoc0fSC1trBrbXDhsvnJPlCa21Rki8Ml5PkJUkWDW9nJvl4Mh6mkpybZGmSI5KcuyFODducOeF2yzvMCwAAAMB2sD1OnTs5yaXDx5cmedmE9U+1cX+b5OlVtXeS45Nc11q7r7V2f5Lrkiwfrntaa21la60l+dSE+wIAAABgmtnW0NSSfL6qbqmqM4e1vVpr302S4f2vDOv7JPn2hNuuHdY2t752knUAAAAApqG523j7Za21u6vqV5JcV1Vf28y2kz2/UpvC+i/e8XjkOjNJnvnMZ25+YgAAAAC2i206oqm1dvfw/vtJrsj4cyx9bzjtLcP77w+br02y74SbL0hy9xbWF0yyPtkcF7bWDmutHTZ//vxt2SUAAAAApmjKoamqdq2q3TZ8nOS4JLcnuSrJhleOOz3JlcPHVyU5bXj1uSOTPDCcWndtkuOqavfhScCPS3LtcN1DVXXk8Gpzp024LwAAAACmmW05dW6vJFeMN6DMTfLfW2v/q6puTvLpqnpjkruSvGrY/pokJyRZneThJG9IktbafVX1R0luHrb7w9bafcPHb0lySZJdkvzl8AYAAADANDTl0NRa+0aS35xkfV2SF02y3pK8dRP3dVGSiyZZX5Vk8VRnBAAAAGDH2dZXnQMAAACAJEITAAAAAJ0ITQAAAAB0ITQBAAAA0IXQBAAAAEAXQhMAAAAAXQhNAAAAAHQhNAEAAADQhdAEAAAAQBdCEwAAAABdCE0AAAAAdCE0AQAAANCF0AQAAABAF0ITAAAAAF0ITQAAAAB0ITQBAAAA0IXQBAAAAEAXQhMAAAAAXQhNAAAAAHQhNAEAAADQhdAEAAAAQBdCEwAAAABdCE0AAAAAdCE0AQAAANCF0AQAAABAF0ITAAAAAF0ITQAAAAB0ITQBAAAA0IXQBAAAAEAXQhMAAAAAXQhNAAAAAHQhNAEAAADQhdAEAAAAQBdCEwAAAABdCE0AAAAAdCE0AQAAANCF0AQAAABAF0ITAAAAAF0ITQAAAAB0ITQBAAAA0IXQBAAAAEAXQhMAAAAAXQhNAAAAAHQhNAEAAADQhdAEAAAAQBdCEwAAAABdCE0AAAAAdCE0AQAAANCF0AQAAABAF3NHPQA7xsJzrp7S7dac/9LOkwAAAACzlSOaAAAAAOhCaAIAAACgC6EJAAAAgC6EJgAAAAC68GTgMFudNzbF2z3Qdw4AAACeNBzRBAAAAEAXQhMAAAAAXQhNAAAAAHQhNAEAAADQhdAEAAAAQBdCEwAAAABdCE0AAAAAdCE0AQAAANCF0AQAAABAF0ITAAAAAF0ITQAAAAB0ITQBAAAA0IXQBAAAAEAXQhMAAAAAXQhNAAAAAHQhNAEAAADQxdxRDwAA0NvCc66e0u3WnP/SzpMAADy5OKIJAAAAgC6EJgAAAAC6EJoAAAAA6EJoAgAAAKALTwYOAAAATFte5GNmcUQTAAAAAF0ITQAAAAB04dQ5AIAZxikEAMB05YgmAAAAALoQmgAAAADoQmgCAAAAoAvP0QQAAAAwIrPtuReFJgCAJ4vzxqZ4uwf6zgEz1Gz7YRBge3DqHAAAAABdCE0AAAAAdOHUOTbPIfYAAADAVnJEEwAAAABdCE0AAAAAdCE0AQAAANCF0AQAAABAF0ITAAAAAF0ITQAAAAB0MXfUAwAAALPDwnOuntLt1pz/0s6TADAqQhMAO955Y1O83QN95wAAALoSmgDgSchRBwAAbA9CEwBTNuVYMa/zIAAAwLQgNAEAMK044g4AZi6vOgcAAABAF0ITAAAAAF1M+9BUVcur6s6qWl1V54x6HgAAAAAmN62fo6mq5iT5T0l+O8naJDdX1VWttX8Y7WQAPBlM5XliPEcMAMA0cd7YFG/3QN85nmSmdWhKckSS1a21byRJVV2e5OQk3UOTJ52c2fwwCLCD+IYN4Inzf+e04Gc+2DGme2jaJ8m3J1xem2TpiGYBAGY7PwzObDPk8+cXZExXQszM5v+WJ6Fp+nXfiu2PAAAgAElEQVSvWmvb9QG2RVW9KsnxrbU3DZdfl+SI1trbHrfdmUnOHC7un+TOHTjmnknu3YGPt6PN5v2bzfuW2L+Zzv7NXLN53xL7N9PZv5lrNu9bYv9mOvs3c83mfUvsX2+/1lqbv6WNpvsRTWuT7Dvh8oIkdz9+o9bahUku3FFDTVRVq1prh43isXeE2bx/s3nfEvs309m/mWs271ti/2Y6+zdzzeZ9S+zfTGf/Zq7ZvG+J/RuV6f6qczcnWVRV+1XVzklek+SqEc8EAAAAwCSm9RFNrbVHq+qsJNcmmZPkotbaV0c8FgAAAACTmNahKUlaa9ckuWbUc2zGSE7Z24Fm8/7N5n1L7N9MZ/9mrtm8b4n9m+ns38w1m/ctsX8znf2buWbzviX2bySm9ZOBAwAAADBzTPfnaAIAAABghhCaAAAAAOhCaAIAAACgC6GJTaqqX6qqp416DrZeVc2pql+tqmdueBv1TGydqtpva9ZmquHv5n8Y9Rw8ccPn7ndHPQdMpqr+36p6zqjn2J6q6piqesPw8fzZ9LWBma2qllXVrsPHr62qC6rq10Y91/ZUVTuNeobeqmr3qloy6jnYelW1a1X90vDxs6rqpOn2d1NomoLhk/mFqrp9uLykqv7tqOfqoar+e1U9bfii8Q9J7qyqfz3qubZVVT1UVQ9u6m3U8/VQVW9L8r0k1yW5enj73EiH6qiqjqyqm6vqh1X106r62Wz53A0+O8nan+/wKbaT1trPkjy3qmrUs2wPVfWUqvqXVfUHVfX/bHgb9Vw9DJ+7V4x6ju2hquZV1enDN2hVVe+pqs9V1Yeqas9Rz9fDECb+oKourKqLNryNeq6Ovpbkwqq6qar+r6oaG/VAPVXVuUnek+T3h6WdkvzX0U3UV1X9clW9t6o+MVxeVFUnjnqubVVV+1bV5VX1xeHf304TrvuLUc7W2ceTPFxVv5nk3Um+leRTox2pv+Hrwwur6j8nWTvqeXqoquuHn/n2SPJ/klxcVReMeq5equoDw/7tNPzcfm9VvXbUc3V0Q5J5VbVPki8keUOSS0Y60eMITVPziYx/wX8kSVprtyV5zUgn6ufA1tqDSV6W5Jokz0zyutGOtO1aa7u11p6W5INJzkmyT5IFGf/m7Y9HOVtHb0+yf2vtOa21g4a32fTbiY8mOTXJ15PskuRNST4y0ok6qKoDquoVScaq6uUT3l6fZN6Ix+vty0murKrXTdzXUQ/VyZVJTk7yaJIfTXibLb44xJejhl+uLJklv/38VJLjkpyR5PqMf837aJKHMs2+YdsGVyYZS/JX+adfQlw90ok6aq3959basiSnJVmY5Lbhl2YvGO1k3ZyS5KQM/5+01u5OsttIJ+rr4iQ/SXLUcHltZsf3ZRdl/P+UtyXZO8nfVNUzhutm0xE/j7bxlzA/OcmHWmsfyiz6+1lVS6vqQxkPaFcl+WKSA0Y7VTdjw898L09ycWvtuUlePOKZejpu2L8TM/7/yrOSzPiDJyao1trDGf/8faS1dkqSA0c802PMHfUAM9Qvt9b+7nG/mH90VMN0ttPwW5eXJfloa+2RqmqjHqqj41trSydc/nhV3ZTkA6MaqKNvJ3lg1ENsT6211VU1ZzjC4uKqWjHqmTrYP+NfBJ+e5J9PWH8oyb8ayUTbzx5J1iV54YS1luR/jGacrha01paPeojt6PnD+0MnrLUkzxvBLD0d2FpbXFVzk6xtrW3Yz/9VVf9nlIN19MuttfeMeojtqarmZPyHvwOS3Jvx386/s6re3Fqb6b8I/GlrrW34XmzDaUqzyG+01l5dVacmSWvtx7PkyNf5rbU/GT5+23AkxQ1VdVLG/++cLR6qqt9P8tokzxv+LU6r03emoqrel+RfJLkryWVJ/jDJqtbapSMdrK+5VbV3xvfz34x6mO1gw9/DE5Jc1lq7b3b817JRVdVRSX4nyRuHtWnVdqbVMDPIvVX1Gxm+UFTVK5N8d7QjdfOnSdZk/Ju0G2r8POvZdHrSz6rqd5JcnvHP36lJfjbakbr5RpLrq+rqjP92MEnSWpsth8E+XFU7J7m1qj6Q8X9zM/4b7tbalRk/yueo1trKUc+zPbXW3jDqGbajFVV1UGvtK6MeZHtorf3WqGfYTn6aJK21R6vq7sddN1u+Nnyuqk5orV0z6kG2h+FUj5MyfurAv2+t/d1w1fur6s7RTdbNp6vqT5M8var+VcaPvvvEiGfq6adVtUv+6Xvq38iE72FmsJ2qal5rbX2StNb+a1X9Y5JrMwu+d5ng1Un+ZZI3ttb+scafG3Q2PB/jmUnuzPipgZ9rra2fZb94T8bj2bVJvtRau7mqfj3jZw3MFv+zqr6W5MdJ/u+qmp9k/Yhn6ukdGT/D6orW2leHz99fj3imx6jxox15IoZP5IVJjk5yf5JvJvmd1tq3RjrYdlJVc1trs+KIrapamORDSZZl/JuaLyV5R2ttzeim6mN4Hodf0Fr7dzt6lu1hiJ7fz/hvKM7O+KkgH2utrR7pYNuoqt7dWvtAVX0kk/yWs7U2a56EuaqelfFv2vYajiJZkuSk1tqMP02iqv4hyT/L+NeDnySpJG22nL46fIP2x0n2aa2dWFUHJjmitXbJaCfbNlX1/Yz/4qEy/gPT5RuuSvIvWmt7jWq2XqrqoYz/YPvTDKf8Z/zv5qx4sY+qOiPJ5cMpBI+/bqy1NuOP9K2q3874KZ6V5NrW2nUjHqmbYd/+bcZP+fh8xr8/e31r7fpRzrWtqursJH/fWvubx60fkuQDrbXfHs1kbI3hyKzjMv4L6Rdm/Af4FyfZd7b8TPRkUFW7J3mwtfazqvrlJE9rrf3jqOfqqap2ba1Ny6dqEJq2wXD48i+11h4a9SzbqqreubnrZ9FRMbPedP4Ph19UVf+8tfY/q+r0ya6fTYdpV9XfZPz8+D9trR0yrN3eWls82sm2XW3iVXZmyy8ghiMl/1uS97TWfnM4xfrvW2sHjXi0bbKpf3cbzKZ/f7PZ8GSov5YJR+q31m4Y3UQ8EcNzFx2Z8ZD2t621e0c80nZVVf9/e3ceb/tc/v//8TzmiJKxTBFJ5qEUFYpPEilkKoWP6qckGjVQ6Ks5olSUpBKHCAkZi+KYDsf4yZAUSigi5Hj+/ni91jlrL3s6e7/3ea33a1/3223f9lrvtddxbXvvtd7v63W9rmsj21eXjqMJOZHduZicn7QY+G/b1TTll7QgqcXBrsCmwEW2dysb1fjVvPgHIGkn4DzbjykN7VofONz2dYVDa0TeNvcDYBHbKyg15H+/7X0LhzZLbJ0bg/yGeAjpxcaSLgcOtf1Q2cjGpZrGfcNRGgm8H6lhaPcJ6XalYmpK9wsO0JcvOGMh6VTb75Q0g+dW/Bh4GDgyb0FrHdtn58+T4YK2uv52khbNzSZbv+AwgqVs/0x5Cmnu31fD1rKfTobV6dwXptNP61LbNU0k/RJpIMstzN7uaNJEntbquoAXA9/7OtWSVVSkZS8B5iGdl71eErZr6N03S64C3YWUrPgXsGHZiJphe8D1g6TtgVcVCmdC5O2PpwGnSVqU1KC/BseRF/8gDbeS9DPqaMYP8DnbUyVtCvwP8DVSYu3Vwz+tNY4kfV9nAdi+QVJf9c2MRNPY/Jx0AtMZ97w7cAot7tRfy/aqUTiTlIw5G3i2cCxN6/sXnDHaP38eatzxEqRKi1YmmiSdzTCNQWtIgnapsb/dz0i/m9cy+6Kww8DKJYKaAI8rjUDu/Ow2oo7k2jRyg3NJR9ver3A8jcuJmI1Ir5MA+0va1PanCobVpLeTJq7W0Ndnlt4L+FpJ+iGwNnAzs8/LqhgSkStdd80fz5Cq7jasoV3DUGyfKan1ry2SfmT7vfn2ezqLgXlhqZaFweoW/3p0Fh62AY61/UtJny8YT+Ns39vz8+urBcBINI3N4rYP67p/eM7gt14uD90beCVdo9Vt71UsqGY9aftbpYOYKP3+gjMWtu/Pn4fagnRPbvDeVl/Ln98BLAP8JN/fldSYvyYfJPW3W13SX0n9jN5VNqTxsf3W/PmlpWOZYB8nJehXzlsgXwLsVDakRnS/YG5SLIqJ9RZgXdvPAkg6EbgeaP3FYHYXabtOVYmmDkkn2X73SMdabGPbfTWSuwlKU3EXIy1O72j7j5Luri3JJOkdXXenkCq1aujLsk7X7f2pJ7nUrcbFv25/zYMU3kQaDrEA6Xe0FvdKei1pd9X8wIeBWwvHNEAkmsbmEkm7AKfm+zsCvyoYT5NOAm4jVcYcSqrW6qtf2nE6KjfNvoCBk9lq2K/b9y84YyHpctub9vQBgK7tA7avLRTeuHUahUo6zHZ3BdrZklq99aOX7buAN9XU365bzduTSImJzYFXkP72bqGOqtAaLohG4wWkbcaQLn5br2uAwhOkaaQXMfB9vZZBCq/sviNpXmCDQrFMhD9IWsP2LaUDadiDwHLA0sCSpGleNb7ebNt1+xnSAtnbyoTSqBp/Vr0GW/xr88Jtr3cCbwa+ZvufkpYlLZrV4gOkAVcvAf5Curb9YNGIekQz8DHomuDSOcmeAnSaL7d637yk622vJ+lG22vnhq/n296idGxNkHQE8G7gTrpKtGv4/iQtQXrBeRPpQvACYP+W9w6bNCTdCmyTkzGdfmLn2n5F2ciak1eTduC5PdIOLRVTUwbZnrQrcI3tg8pF1RxJ19lef6RjbSPpCeAO0mvmKvn2LDVMDZS0K/Al0tQkkZKhB9n++bBP7HO1N3KXdBDwaWAhUjIN0s/vaeC4WrY+5i3+ZwMPUNnETkmLkd7zdiVNJX0B8D+2pxUNLIxomImkQFWJ7KqGW/XK/ZlWtX2C0vTcRWzfXTquySISTWEASdNsvypXUuxLeuOfZruKPiOSbgPWtv106VjCnJG0wmDHbf95bscyESS9mbSydFc+tBKpmfv5xYJqmKTzSE1Qr6VrW6ftrxcLqiGSbmTg9qR5gOvbfrEkaSlgWdJJ9juZvdVsUeB426uXiq0JklYlVRzc2/PQisB9tu947rPaJ6/kbkT6+V1V03jnfJH0pO2Z+f48wAK2nxj+me0g6YhaEtaDkXQHcCAwg64qyVomdnbk19KdSUmn5W0vXzikcZH0Cdtf6aosHKDtiZjaE9kwKxF6CLMrsS8jDbf6V7mompN3sGxI6uG3mqQXA1NtV7FNvg0DrmLr3BhJeiGwKgP7GNWwzeX7+Xv7HKmp9CLAwWVDatQNpBWlv5cOpCmShu051fY3+y7d21MXBF4K3E7PtoK2sn1evujtXLjfVltzW2A5228uHcQEqm57EqmJ5l6kLSDfZnai6VHS+0TbfRP4dO9FbV75/CYDt4W0iqTVbd8mqVN19pf8+cWSXlzJlnGAi0iVvP/O9xciVfS+tlhEzXrOBC9JF9l+Y4lgJsCfbZ9VOoiJImkn21Nt/x04Gjha0idKx9WATmuGa4pGMUFqSCSNwg+Bm0iLSJB2fJxA6hlag7cD6wHXAdi+T1JNQxb6fsBVJJrGQNL/khrDLQdMBzYG/gC0fvuV7ePzzcuoZ1pSt6WB2yRdzexeDrbd5v3kHyC9UZwK3MfA5rbVsL1W9/188fT+QuE0pqeRZrdVKhzx/HtJa9meUTqQCXAEcL2kAduTyoY0frZPAE7orF53PzZUlWHLrGT7xt6Dtq+RtNLcD6dRBwLvAwarGDQVnLNkC9ruJJmw/W9JzysZUBPycJaFgSXyAmB3NeGLiwXWvNvySPWzGdhjq5b3voOAqT3HdgG+MsjXtobts/PNG21fXzSYCTBJJgKvYnuHrvtfkDS9WDTNe9q2JXWanS9cOqCG9f2Aq0g0jc3+pBL0K21vLml14AuFY2qEpBcAe/DcMrxaqmIO6botYFNSGXObLUua/rQzqRHjKcDpth8pGtUEs31dHrHedp2KiaVIK/AXkX43NwcupY4RzzNIJ2zzAntKuovKenHYPlnSpczenvTJmrYnMfiF0ZlAq3s00VWVPIiF5loUE8D2+/LNrW0/2f1YTmLU4nFJ63cqtCRtAPyncExNeD/wEVJS6VoGVhN+u1RQE2Ah0vvBVl3HTMvf+yRtTZr4+JKeyvNFqWuE/Dfy1typwM9t31w6oIZ8beQvab3/SNrU9uUAkjahjtfOjlPz1LkXSNqHVJ19XOGYmtT3A64i0TQ2T9p+UhKSFsil6S8vHVRDzgWupGevfC1sXyZpXWA3Uqno3cB3y0Y1PrnZ93eB70p6CSlxdrOkT9o+qWx0zZF0YNfdKaQL3AcLhdMY23sCSDoHWMP2/fn+stRzMfHW0gFMNElvBy7ubAGR9AJJ29s+s3Bo4yJpNdKkucXyVL2ORRk+SdMWV0vax/aAk09Je5Mu7mvwe56bEBzsWFvtD0yVdF++vyxp4aXVbB9FupDYz/bRpeOZKJ33wArdR9pWth0DX0seAw4oEtEEyAvuy5DOqb8vaVHgFNuHFw5tvO6upQfoMD4A/Dj3agJ4BBi2N1Wb2P6apC1JyfmXAwfb/k3hsJq0Fmm74xZ0Dbiij6qVoxn4GEg6A9iTtNK0BekPcz7bbykaWANqmCI0mHyxtAspCfMQqernY7ZXLBpYg/JWsl2BLUknNV+vaVxwztp3dEbont67Ut9Wkm6yvWbX/SmkkvQ1h3laq0g6yfa7RzrWRpKm216359j1ttcrFVMTcgLtHaSV+XO7HnoMONn274oE1hBJSwNnkCZ5dS4GNwTmB97e5qq0fPH3EuAnpMWV7q1X3217I3eY9Tq5MXA16UJCpP52/y0aWMMkrQmswcC+oD8uF1FzcnXd3qR+i93f317FgmqQ0vRmAavlQ7fX9vvZIWkt4BPAzrbnLx3PeHRfD0k6vWeLWevl184dbZ+ak4PYfrRwWGEOtGHAVSSaxknSG0hNX8/r5x/0aEk6gNRQ8xwGluE9POSTWkDSs8DvgL07U4Qk3VXDND1JXyBVjNxKmgx1nu2ayrJnkbQIaavV46VjaZqkY0gDBk4mrUjsAtxhe7+igTWoN5Gdp0PNsL1GwbAaIenG3i2Akmb09hZrq+7y+hpJ2hzoJHVvtn1xyXiakKcmvZeUOOtu2PsY8KNaeuBI+oPt15SOY6LkRZbNSImmc4Gtgctt71gyrqZImgrcRkqGHgrsDtxqe/+igTUkXyf8mLQ4JmB54D2VDBBC0itIFYQ7khZyf05aBGz10J3uhaIaFo0GI+m3tl8/8le2U+6B+mVSawoxu13DokUDa4ikU4D9+vlvLRJNYyBpY9KJ6GP5/vNJW16uKhvZ+En6IPBF4J/MboLntidk8qr8LqQeOOeR3giPt/3SooE1ICfR7mL2vurOz62a/jeS9gU+RWqMCikZ+mXb3ykXVfPy72nnTf+3ts8oGU9TJB0EfJrUi+MJZldWPA183xWM7pb0Q9Lr5rdJf4P7AS+0/d6ScY2XpI/a/rqkbzL4COsDB3la6COSdrB9euk4JkpebLkR+IUrPKnNPe7WAa63vU6uwjvedmsnInbrXMR3kvW5Auh8232z/WM8JF0L7Gb79nx/NVI16AZlI2uGpCtJC2RTbd830te3RU9FU627PT5HunY4BZi1gNv24oIOSXcA29q+dcQvbqHcF3RtUkVvd3FI3zSqj0TTGEi6Hli/c0KTyw+vqeFFSNKdwKtt/6N0LBMhTxzYnrTFbAvgROAM2xcUDWwcJA27/c89Y7vbRtJnSQnCD9m+Kx9bGTgKuKqCPgCz5J/lqrYvVJqaNE8noV0DSUfUkFQaTH5t+RxpzDqk5oyH236iXFTj1+kzlXsWPYftH8ztmMKck7QNz92adGi5iJoj6THSIsQzwJPUt2o9zfarcsJic1JF2k22X1k4tEZ0fX+/BfYFHgCmtX2Bs2OIatfnHGsrSUv1VlRIenknsdZWkmaSki9i9iIZVPT6IunuQQ63vrigQ9IVtjcpHcdEydWSz2H7srkdy1Ai0TQGQ/TiqOJNQ9JZwC5tvzgaDUmLk6e11bByJmlr27/uOfYB261udi7pdmCd3l5MkhYCbrC92uDPbBeliRjvAxa3vYqkVUl9VN5YOLRG5YbSnaqtS22fUzKepkjayfbUkY61laQVXH9j1CpJ+i7wPFKS4njSFpdptgdNHob+Iuk7pIrQXYCPkip6p7uSJtqS/hc4nbQyfwKwCPA5298rGlhDcrWrgc5wlt2BeSv6+d1O+nmdmu9/lNSmovVb4kO7SToKWIY0Ibe74qeKbeNtEImmMZD0C9LY8WPzoX2BzW1vXyyohuRG568ELmHgH+WHiwUVRkXS74HPdnqLSPoksJntrctGNj6Sbrc96FRHSbe5goa2kBLYwKtIVVqdvgDV9PiBVNFE+h5/mg/tSqoGbX2V02Cl9TWV2+fXlyWBq4DfAr+rtRy9Nl1bkjqfFyFtM9tqxCe3hKQXknrcdVdsVdEDp5uklYBFbd9YOJQwSpIWAD4IbEqqhvkt8B3bTw37xJZQmpD7fVI14dKkfqEftf3vooGFEeXK+QOBFWy/Ly9wvryiBcATBjls1zNoYGPgaNJk4PmBeYDH+6nabt7SAbTUB4BvAZ8lrVJcRKpEqMGZ+SO0z3bAOZI+DrwZWD0fa7u/SHqj7Yu6D0raAri/UEwT4SnbT0upfZGkeRmkJ07LbQOsa/tZAEknAtcDrU00SdqaNJHtJZK+1fXQoqStPFWw/Vql6VCvJlWknS9pIdtLFg4tjKzTv+8JSS8mNextfX/CjlwRsz+wHDCdNIXuD/TRiOexyu8DM21b0vKkxu53Fg6rEZKWA1ZyHjIg6UBSNRPAz5wHt7Sd7acknQScZPvB0vE0zfb9ks4jvY8/CxwUSabWOIE0bfW1+f5fgKmkgVA1ON72Fd0HJNW0le4YUqXrVNJ7wx6kBZe+EYmmMch7kXcpHcdEsH1i6RjC2Nj+R96WdCHpjWPHShqjfhj4paTLSd+XgY2ATYC3lQysYZdJ+jSwkKQtSZWSZxeOaSK8AOg0mlysZCANuY800Ws70u9nx2PAAUUimgB55ex1pCTTEqShCr8rGlQYrXMkvQD4KnAd6TX0+LIhNWp/0nvClbY3l7Q68IXCMY1b3k79ZeDfkg4DPk76+a0n6Ye2v1w0wPH7KrOrWwHeT6qMeR7p57d7iaCaorRqdAjwIfLEq9z35+ha+qMBSPoNadFvTVKy94dK08w+VjayMAqr2N5Z0q4Atv+jzmpnHY4GeqvKBzvWWrbvkDSP7ZnACbn6vG9EomkMJH0FOJy0SngeaRrIR2z/pGhg4yDpVNvvzNNNBpss1Pr+U7XKjVBNblBIKp9cGdhRUusbFtq+WdKapNHHr2R26fn7e/s2tdyngL2BGaQT7nOp62IQ4AjgekmXkH6Or6fF1UwAtm8AbpD0U9vVVDAN4gpSQu0I4JzKv9eq2D4s3zxd0jnAgrb/VTKmhj1p+0lJSFrA9m2SBt1u3TIfAVYBnk/ajrRiXlB6HmnKUNsTTb1bdJ6w/XUASTUksT9CWhDbyPbdMGuQybGSDrD9zaLRNefbtjs7If4p6bW0/H19Enk69zvtDLdaha62KW0l6TWkKq0lc6Vkx6Kk7WW1eELS/KRz0K+QEr4Lj/CcuSoSTWOzle1PKI0i/wupofQlQGsTTaQVQYC3Fo0izDHbzy8dw0TLFxH32/5h9/Eamp135O1kx+UPYFaJ7xVDPqlF8irZ5aRtLRuREk2ftP1A0cDGqZOkJyXQak7SL026aHo98BFJTwNX2G595UjtJO0EnJcnWH4cWF/SYbavLxxaU/6SK7bOBH4j6RFSpWHbPW37EeARSXc4TwO2/UT++2u7BXvudw++eNHcDGSC7AFs6a4pzrbvkvQu0lTSVieaJK1u+zanqaQLdHpO2X4mVzmF/ncIqWBieUk/Jb3Hv7doRM2Yn7QNd15Sor7jUdIwjFq8G5hC6gF3AKmicIeiEfWIRNPYzJc/vwU42fbDba80tH1//nxP6VjC2OSkxHTbj+cTmfWBIyuaFPU5SU/1NjsHWp1okjQP8E7gJaSLwZskvZU0ZWghYL2S8TUl9xg50/YGwFml42nQpEjS50qKW0gNwZcA3kBaOYtEU//7nO2pkjYF/gf4Gul189Vlw2qG7bfnm5/P1ZKLkS6e2m4hSeuRLiTmz7eVP3qTNG30mKTVbP8fgO2HISUwSJP12m6+7iRTh+0HJc032BNa5mfM3oL0BwZuR/oOFW1PqpXt30i6jrQAKGD/wX5n28b2ZaR2FD+q8bpW0tuA5Wx/O9+/DFiKVJn2B6Bv+ttFomlszpZ0G2nr3L6SliRNW2g9Se8glWMvxewTmtZvv5okjgXWkbQO8AngB6Rxum8oGlVzam12/gNgeWAa8C1J9wCvAT7VVY5eiyslbWT76tKBNGWwJL2kJYCHKumRBoCkO0lNiH8H/Ij6tq7WbGb+vA1wrO1fSvp8wXgalxP2SwN350PLAG1fZLkf+Ea+/UDX7c79tjuE9J7+RVLvKYANSIss+w/5rPYYruqshoo0DXF7sPuhj0jqTQJ2BuusIGkF29f1PqdNJB1p+yPAMUNUmrf92uETDOwVvQDptXMRUoP300oENRhVdB48VymN0n3U9kxJCwPPb/sWEABJdwDbxtjq9lEepS7pYOCvtn+gisarA0haitnNzveq4UJe0k3A2rafzVO9/gG8rIbXk165ImY14B7gcWYnslu7vSw3yf4SqcH5YaTk7hKkKoQ9bLe6skLSh2wf09VsMrRM7sv0V+BNpJPR/wDTbK9TNLCGSNqPlLT4G2nqFbT8dWWyyP0XP0HqvwhwE/BV2zeVi6oZufH344M9ROqT1uqqpu7zy95zzdrOPWuTKz87NiD1X+wkB2271RM7JW1g+1pJgy6054qn1pJ0te2Nuu4fY/tD+faVtjcuF91AkWiaA0oTvS6oeRVX0hW2axr9OGnk0snzgL1I06EeBK5v+0gYI98AACAASURBVMn2EM3On8m3W19tN5lO0CStONjxNpc2S7qGtAK/GGli0ta2r8zbP0623eqtjzX/Pk4WuXn0m4EZtv8oaVlgLdsXFA6tEXmB7NW2Hyody0TJDZZXomsngu0fFwtoAkhaxHYNW+YmBUl/B35OOjfbOd8m33+n7aVLxRZGT9L1bT9P6ZWrstpe0Tqk3LPvZUM8dqftVeZ2TEOJrXNz5hTgcUm/Bk4Gzq9whfcaSaeQmmrOmjxg+xflQgqjtDNpMtueth+Q9Hr6bPrAWEyCZuerS7ox3xawSr7f+mqfXp2EUr7wXQO4x/aDZaMat3k7F+ySDrV9JUCefFU2sjDpSZpCql5as3Msb/e8f+hntc69QE1T9AaQdBJp+tx0Zm+DNFBFoilPiPoBadvHCnn7//tt71s2sjCCj3fdvqbnsd77oX/VWHFyJrlHmKTTbfdVg+wGXCVpH9vHdR+U9H5SG46+EYmmOXMbsAWpY/1HgRMknUFatW51GV6XRYEngK26jhmIRFOfy8mli4HdJP2E1KviyMJhNabiZuevKB3ARMvVoN8ibS/7LPBt0jaXlSR90vaJJeMbp2e7bv+n57EaTuDWlvToIMejf18L5C25N1S+wnsXcKmkXzFwgewbQz+lVTYE1qhhq/gQjiQ1qT8LwPYNeaEs9LHe921JC9sebKtgCHNb9yrfysWimDgHAGdK2o2B/e0WALYvFtUgItE0Z+w0avY44DhJy5CmRX1J0nK2ly8b3vjZ3rP3mKSNBvva0B8krUZqCrcr8BCp8k62Ny8aWPOqbHbeVeWzte1fdz8m6QO0fKpedhgpeb0YcAmpJ9VduefWRUCbE03r5ESMSFOiOkmZWiZDzaitrH4SWha4WdI0unrGVNAQtePP+WP+/AF1JHk7biI1N6+pCm0A2/f2VIDWtlugWlGR1j6Sjmb2a+Rykr7V/bjtD8/9qBrlIW5XwfbfgddK2oLZ/e1+1ZnK3U8i0TRnBrwL5ma93yJNihq090hbSVqD2cmLf5FW1EJ/uo00CWpb23cASDqgbEgT4hnbzmM9j8rNzt9TOqgGfU7SU503CkmfBDajjkTTs50R1pLutn0XpDdLSc+UDW18bM9TOoYQRvCF0gFMJNsDvr88VGHbQuFMhCWAW3KisLtiq5ZE4b25B5UlzQ98GIiBNO0RFWnt07218dpiUUyc4RYAq6nEztcLfZdc6haJpjkz5MV7m5vZduRk2a754xlgRWBD238qGVcY0Q6kpOAlks5jdnPG2jwm6SDg3cDr8jjrml7DtiONev44qXHv6vlYDabkSZ1TgGfz7c7v6JRyYYVRmFo6gDA+ti/L7++r2r4w90irKkGa3w+2Ip2/bAVcTj2/u58vHcAE+wBwFPAS4C/ABcAHi0YU5khUpLVLy9sVjCgWAPtHTJ0bA0lLk94QDdxn+2+FQxo3Sb8nbWv5OfDzPJnmbtsvLRxaGCVJC5P25u5K6iV2InBGRZOFliE1O59m+/K8YnZCP01XGK+8lexC0grTXrX05JD0J1Ivo8ESoLZd4x76quQtuscCS9teU9LawHa2Dy8cWhiBpH2A9wGL215F0qrAd22/sXBo45bfB3YDtiE1Qd0EWNn2E0UDC2GSkHQa8A3gGGBjUkXahrZ3KRpYGJGkDYHPkAoLuidaVjOEJpQViaY5IGld0jaWxYC/5sPLAf8E9rV93VDP7XeSfgmsRyp9/Znt30u6Ky4A20nS4sBOwM62tygdT1Py3+BupN5odwO/sH102ajGR9JjpKS18uf5SRWFpqIS39Buki4jTRn6Xqdnk6SbuqeZhf4kaTrwKuCqrp/dDNtrlY1sfCT9hdSb6VjgTNuP1bhAJmlj4GjS4Ij5SdVoj9fy3iBpSWAfYCUGXuzuVSqmMHqSliBVpL2JdB5zAbC/7YeKBhZGJOl20vv6DLoGm9SwSyf0h5q2ncwNPyI1uLuq+2A+CTgBWKdEUE2w/TZJi5G2YX1B0suAF0h6le2+GpUYRmb7YeB7+aPVam92bvv5pWOYWyStP8jhfwH32G51r6ZJ4Hm2p/Vsj4ifWTs8Zfvpzs9O0rzU0SD1dFIV787AzLxgVsP31esY0nvgVFK/zD2AVYtG1KxfkvpMXkhsuWoj2d69dBBhTB60fVbpIEK9oqJpDkj6o+1B39wl3WH7ZXM7pomSt/DsTLq4X76GiXqhnSQ9SzoJ3bur2Xl11XaSNgGm235c0ruA9YEjaxpJLulK0vd1I2nlcy3gBuBFwAdq2eZZI0m/Bj4ETLW9vqQdSX+TWxcOLYxA0ldIldd7APsB+wK32P5M0cAaoJQ925x0rvIWYFFgb+Bc2/8uGVtTJF1je0NJN3a2tEj6ve3Xlo6tCZKm2163dBxhbCT9kVRhfgpwuu1/Fg4pjJKkN5JeOy9i4KCBXxQLKlQlEk1zII9/XAX4MXBvPrw86eTtbtsfKhVb0yQtbPvxfHvFKKMMpUh6O2k197VAp9n58RVuj7iRVBW5NnASaVzwO2y/oWhgDZL0c+Aw2zfn+2uQyrYPI22DjIuNPiVpZeD7pL/DR0gXFu+KYRH9T9IUUvJlK1KC93zSa2hVJ4CS5iMNUtgV2Mr2EoVDaoSk35K2JR0PPADcD7zXdmur6LtJOhz4ve1zS8cSxkbSq0jnadsDt5B6vf6kbFRhJJJ+Qho8czOzt845tq2GpkSiaQ5J2hp4G6kZuEgTMs6q5Q0yj5g9HljE9gqS1iFtF9y3cGhhkpsEzc6vy5UiBwN/tf2DzrHSsTVlsJXrzrFY1W6H/Hc4xfZjpWMJo5OT9efafmrEL24pSQsBK9i+vXPf9n8Kh9WIPDHwb6T+TAeQ+oR+p1Ph21Y9/QkXJlVU/JfKRpBPJrlf0zeA3WPyV/+roVdf6G+RaAoDSLoK2JGUPIuGr6Ev1djsPDdbPg/YC3gd8CBwfU3TPySdAjxMqkqDtD13CeDdwOW2NyoVWxicpAOHe9z2N+ZWLGFsJJ1ASs7/lvS3d35NPdEkbQd8FZjf9kvz0IhDbW9XOLRGSNrA9rU9x7a1fXapmJoQ1fJ1kLQo0Kk8XwU4Azi193c29B9JxwHftH1L6VhCnaaUDqBNJC0m6UuSbpX0UP64NR97Qen4mmL73p5D0Zwx9BXbD9v+Xi1Jpmxn0orunrYfII3pXrhsSI17L3AH8BHSyvxd+dh/SX1WQv95/ggfoc/Z3hN4GamZ9G7AnZKOLxtVow4hTdX7J4Dt6UBNW6uPkzSr6kDSLsBnC8bTlDNKBxAacQPQSe6uZvuTkWRqjU2B6ZJul3SjpBm5jUMIjYipc3PmVOBiYPN8IYikZUgXSlOBLcuF1ph78/Y5S5of+DBwa+GYQqie7QckXQzslvfN3w0cWTisRuWtLF/PH72qaNxbG9tfKB1DGD/b/80N3Q0sRGoB8L9lo2rMM7b/1TMRsaZy/R2B0yTtTrow3IPUb6vtNPKXhBZYubZ+b5PIm0sHEOoWW+fmgKTbbb98Th9rk7y/+ihS40kBFwD7236oaGAhVErSaqSS812Bh0iTWz5me8WigTVI0qm23ylpBoNcANa0PbBWuRn4UcDGpJ/hH4ADbN9VNLAwIklvJr3GbA5cSnqNuaCW7XOSfkCamvQpYAfSAtl8tj9QNLAG5feJM0mDaLavof+UpL8zexv1c9j+8FwMJ4yRpCWBTwCvBBbsHK+s4rxakjYFVrV9Qv5ZLmL77tJxhTpEomkOSLoAuBA40fbf8rGlSRVNW9p+U8HwQggtJOlZ4HekUfF35GN32V65bGTNkbSs7ftzU9vniD4d/U/SlcC3gZPzoV2A/Wy/ulxUYTTytMefA7+usSG4pOcBn2HgVL3DbD9ZNLBxGiQxvxTwL/IY8rYn6CXdAxw81OO2T5yL4YQxytdGpwAfAz4AvAd40PYniwYWRiTpEGBD4OW2V5P0YmCq7U0KhxYqEYmmOSDphaQVs7cBS5NOAP4GnAV82fbDBcNrhKQTSRVM/8z3Xwh8PUZdhjAx8kSoXUhj488jXRAeb7umHiPPkasnH4qS+3aQdFVvUknSlbY3LhVTCDUbKjHf0fYEfW1TVScrSdfa3kDSjZ3kp6TLbL+hdGxheJKmA+sB13UNgLqx7Uns0D+iR9McsP1Int7yG+BK27N6iuTS9POKBdectTtJJpj1Pa9XMqAQamb7DOCMPDZ+e1KT7KUlHQucYfuCogE2QNLGwJdIE+cOA04iTZubImkP2zW8dtbuEkmfIiVCTWpe/6s8AZIaFlpqJekdwJdJFTGisvHxks7muVty/wVcA3yvrZVNnUSSpL1t/6D7MUlfIi18ttnTpQMIjfhv/ny/pG2A+4DlCsYTRu9p25ZkgHweGkJjoqJpDkj6MPBBUnPsdUmVP7/Mj1WxMiPpBmAz24/k+4sDl9lea/hnhhCakv/udgJ2rqHPgaRrgE8DiwHfB7a2faWk1YGTOytpoX9JGq5ng2va6lkbSXcA29qucrCHpKOAJZm9rXNn4AFS0/NFbb+7VGxNyE3cf2L7p/n+d4AFbO9dNrJmSDod+CFpa+ezpeMJc0bSW0nb/5cHjgYWBb5g+6yigYURSfoYsCppmNURwF7Az2wfXTSwUI1INM2BvF/+Nbb/LWkl4DTgJNtHSbq+hoslSXsAB5G+N0gXu1+0fVK5qEIIbSZpuu118+1bbb+i67EqXjtD6FeSrqi554ak39p+/WDHJN1s+5WlYmuCpIVILRp+CGwNPGz7I2Wjao6kNwF7kgYNTAV+ZPu2slGF0ZA0D/Bh298sHUsYG0lb0tXfzvZvCocUKhJb5+bMPJ3tcrb/JGkz0sjZFalkTKvtH+fqgy1I39M7bN9SOKwQQrt1r1L3TkuK1Y4WkDQf8P8BnQv6S0nbkv475JNCv7hG0imkqWWzmoHb/kW5kBq1pKQVbP8ZQNIKpK250OLtWZ1tqdn/kn5+VwCHSlq8lu2qti8ELpS0GGn66m8k3QscR6rkiteYPmV7pqTtgEg0tVROLEVyKUyIqGiaA5IuBg60Pb3r2LykVabdbc9TLLhxkrSo7Ud7TmxmqeWEJoQw90maCTxOSl4vBDzReQhY0PZ8pWILoyPpeGA+oDMJ6t3ATNv/Wy6qMBq5t2Qv1zLkQ9JbgO8Cd5JeU14K7EtKhu5j+8hy0Y1d3q5qck8tBi5oVrVdVdKLgHeRXlfuA34KbAqsZXuzgqGFEUj6Imlb/Cmk93kAbF9XLKgwLEmPMfgiX1X9+0J5kWiaA5KWA56x/cAgj21i+4oCYTVC0jm239p1YjPrISo7oQkhhDBnJN1ge52RjoVQgqQFgNVJ5yy3tbUB+GQk6Rekn91JpG1z93c9do3tDYsFF0Yk6ZJBDruG/pIhhPGJRFMIIYQQhiXpOmAn23fm+ysDp9UwBKN2eZHsaGAT0kLS5aRhJn8pGliDJL0WWImulhC2f1wsoAbVvm1V0ha2Ly4dRwiTkaT1SdWDBi63fX3hkEJFItEUBpD0S9L46l/afmKkrw8hhFA/SW8ETgDuIlWNrAjsaXuw1ezQRyT9BvgZqWIE0hal3W1vWS6q5kg6CVgFmA7MzIdt+8PlomrOZNi2WnOisGaS3gA8YvtGSe8kJUPvBL5j+6nhnx1Kk3QwaehTp1/f9sBU24eXiyrUJBJNYYD8prEzsA0wjbTn+pwoQw8hhMktb096ObO3J8WFRAt0T30c7lhbSboVWMOVntDWvm219kRhrSR9G1gbWBC4HVgEOA94LWl40u4FwwujkF871+tc4+UJl9d1TwYOYTxi6lwYwPZlwGV5ZOkWwD6kZufRGC6EECYpSc8DDgRWtL2PpFUlvdz2OaVjCyP6h6R3ASfn+7sCDxWMp2k3AcsA94/0hS01U9IqPdtWZ47wnDbZkIoThRXb3PYakhYE/goslafQfQ+4sXBsYXT+REoUdooJFiBVpIXQiEg0hefIGe1tSZVN6zO7XDuEEMLkdAJwLfCafP8vwFQgEk39by/gGNIIcgO/z8dqsQRwi6RpwKwqO9vblQupUR8HLpE0YNtq2ZAaVXuisFZPAth+UtI9tmfm+5ZURf+wWkk6mvRe8BRwc95ebWBLUg+/EBoRiaYwgKRTgFeTyl+/DVxq+9myUYUQQihsFds7S9oVwPZ/JGmkJ4XybP8ZqCXpMpjPlw5gItm+SNKq1LtttfZEYa2WknQg6Xeyc5t8f8lyYYVRuCZ/vhY4o+v4pXM/lFCzSDSFXicAu3VWJkIIIQTg6VztagBJq9B1URj6j6SvAHfZ/m7P8QOAZWx/skxkzcpb/qsk6UXAbsDq+dCtwL3U9bf3+dIBhDE5Dnj+ILcBjp/74YTRsh07VcJcEc3AAwCS3jHc47Z/MdzjIYQQ6iVpK+AzwBrABcAmwHttX1oyrjA0SbcAa/ZWJUuaAtxoe80ykTVD0uW2N5X0GDkB2nmItIOn1b0lJb0CuBg4H7ie9H2tR9resoXt2wqGF0JoOUlvBQ4jbcedl0peO0P/iERTAEDSCfnmUqSJERfn+5uTts8Nm4gKIYRQt1xdsTHpZPRK2/8oHFIYhqSbbb9yTh8L/UHSacCptk/tOb4DqfJ8hzKRNaP2RGHtJF1ge6t8+yDbR5SOKcwZSXcA7wBmRDP+MBFi61wAwPaeAJLOIU3/uD/fX5bUqymEEMIkI2kp4NPAy4AZwBG2Hy0bVRilJyStavuP3Qdzv5//FIqpUbVUZw1hLds79h60fbqk/1cioCbZ3jR/fv5IXxv6Uncfpp2ASDS1z73ATZFkChNlSukAQt9ZqZNkyv5GakAZQghh8vkx8DhwNLAI8K2y4YQ5cDDwa0nvlbRW/tgT+FV+rPXytsAbJK1QOpYJ8PgYH2sVSSeN5ljoO5GcaL9PAOdKOkjSgZ2P0kGFekRFU+h1qaTzgZNJbyK7ABeVDSmEEEIhy9j+TL59vqTrikYTRs32ryVtD3wc2C8fvhnYwfaMcpE1blnSiO5pdCVgKphattQQF321TfUasIVT0rzABoViCaO3sqSzSL+PnduzVPD3Nxl8Efg3sCAwf+FYQoUi0RQGsP0hSW8HXp8P/QFYumBIIYQQypGkF5IuJgDm6b5v++FikYUR2b4JeE/pOCaCpJeRzk++0PPQG4C/zv2IGtc7yatb66d6STqItC13IUmd7bgCnga+XyywMFpv67r9tWJRhPFYvNNnK4SJEM3Aw3NIWpc0TvedwN3A6baPKRtVCCGEuU3Sn4BnmZ1o6mbbK8/diMJoSTqbYba3tL3iIPeU/LTtG3uObwgcYnvbMpFNPEkL2H6qdBxNkHSE7YNKxxHmjKQVbP+5dBxh7CR9CbjY9gWlYwl1ikRTAEDSaqRtcrsCDwGnAB+zvWLRwEIIIfQlSYomov1L0huGe9z2ZXMrlokg6aahmoBLmmF7rbkdU5Mk3U9KpJ0wyGPX2V6/QFiNk7QJMN3245LeBawPHGX7nsKhhWF0/w5KOr3tUxAnozzxcWHgKeC/xMTH0LDYOhc6bgN+B2xr+w4ASQeUDSmEEEI/kHSo7YO77k8BTgJ2LxdVGE7bE0mjsOAwjy0016KYOE8C75W0DbC37X91PTZYhWFbHQusI2kdUnPiH5CGEAybKA3Fdf8ORmVrC8XExzDRYupc6NgBeAC4RNJxkt5IXScyIYQQxm6F3FMFSQsAZwJ/LBtSGA1Jq0o6TdItku7qfJSOqwFXS9qn96CkvYFrC8TTtEeAzYDpwPWSNut6rKZKwmdyZeTbSJVMRzF0b6rQPzzE7dDncuVg5/YmPY99aO5HFGoVW+fCAJIWBrYnbaHbAjgROCP274YQwuQlScBPgRnA5sCvbX+zbFRhNCRdDhwCfBPYFtiTdP53SNHAxknS0sAZpObRncTShqTpSW+3/UCp2JrQszXpVaQqnzOBzwLTKto6dxlwHrAX8DrgQdJWulZvfaydpJmkKY8iVRA+0XmI2H7V13peWwZsw61pW24oLxJNYUiSFgd2Ana2vUXpeEIIIcxdkrpPOOcDvgdcQdregu3rSsQVRk/StbY36O5bJOl3tl9XOrYmSNoc6PRqutn2xSXjaYqk622v13V/EeBbwNrAkrX00JS0DGkAzdW2fydpBWAz2z8uHFoIVep+bRnkdWbA/RDGIxJNIYQQQhiUpEuGedixCNH/JF1BqhQ5DbgY+CvwJdsvLxpYGJakH9rea5DjOwKH2169QFgTQtKKwKq2L5T0PGAe24+VjiuEGkVFU5hbItEUQgghhFApSRsBtwIvAA4DFgO+YvvKooGFAOQ+W+8DFre9iqRVge/afmPh0EKokqQngDtI2xxXybfJ91e2vXCp2EJdItEUQgghhGFJ+n+k5MQ/8/0XAh+1/dmykYVQJ0lLAB8kNQX/IfBVUmXancCBtu8sGF5jJE0HXgVc1bWdZ9Y2zxBCs3IF4ZBs3zO3Ygl1i6lzIYQQQhjJ1p0kE4DtR4C3FIwnjJKkSyRd3PtROq4wop8BCwCrAtOAu4AdgXPIPdIq8ZTtpzt3JM1LTDELYcLYvqfzkQ+tmm//HXi4YGihMvOWDiCEEEIIfW8eSQvYfgpA0kKki+DQ/z7WdXtBYAfgmUKxhNFb2van88THe2x/NR+/TdIHSwbWsMskfRpYSNKWwL7A2YVjCqF63dtWSVvolgO+C8S21dCISDSFEEIIYSQ/AS6SdAKp2mAv4MSyIYXRsH1tz6Er8kj50N9mQuq4L+kfPY89WyCeifIpYG9gBvB+4Fzg+KIRhTA5fJC8bRXA9h8lLVU2pFCTSDSFEEIIYVi2vyJpBmmlU8Bhts8vHFYYBUmLd92dAmwALFMonDB6K0s6i9ygN98m339pubCaZftZSWcCZ9p+sHQ8IUwiT9l+OhVNxrbV0LxoBh5CCCGEUClJd5MuHkTaMnc3cKjty4sGFoYl6Q3DPW671VVpeUvgIcCHSL+bIlVxHW370JKxhTAZSPoK8E9gD2A/0rbVW2x/pmhgoRqRaAohhBDCsCRtDBwNvAKYH5gHeNz2okUDC2ESkLQkQE0VP5IOIA0UeJ/tu/OxlYFjgfNsf7NkfCHUTtIU0rbVrUiJ3vOB4x3JgdCQSDSFEEIIYViSrgF2AaYCG5JWQF8WK5/9T9JOpAv3xyR9FlgfONz2dYVDC8PIFT8HkyoNRNr2+AyVVPxIuh7Y0vY/eo4vCVxge70ykYVQP0nzACfaflfpWEK9ppQOIIQQQgj9z/YdwDy2Z9o+Adi8dExhVD6Xk0ybAv9DauJ+bOGYwsg+AmwKbGT7RbZfCLwa2CRXA7XdfL1JJphVtTVfgXhCmDRszwSWlDR/6VhCvaIZeAghhBBG8kQ+IZ2e+zrcDyxcOKYwOjPz522AY23/UtLnC8YTRmcPeip+bN8l6V3ABUDbt5Y9PcbHQgjN+BNpCulZwOOdg7a/USyiUJVINIUQQghhJO8m9WX6EHAAsDywQ9GIwmj9VdL3gDcBX5a0AFHR3gZDVvxIqqHiZx1Jjw5yXMCCczuYECah+/LHFOD5hWMJFYoeTSGEEEIIlZL0PODNwAzbf5S0LLCW7QsKhxaGIek62+vP6WMhhBBCP4hEUwghhBAGJWkGMOSJgu2152I4YRwkLUVXpYjtPxcMJ4xA0ky6trN0PwQsaLuGqqYQwlwm6UjbH5F0NoO8v9verkBYoUKxdS6EEEIIQ3lr6QDC+EjaDvg68GLg78AKwG3AK0vGFYZne57SMYQQqnRS/vy1olGE6kVFUwghhBBGTdISwEOOE4hWkHQDsAVwoe31JG0O7Gr7fYVDCyGEMJdJWiEqWsPcEM0gQwghhDAoSRtLulTSLyStJ+km4Cbgb5LeXDq+MCr/tf0QMEXSFNuXAOuWDiqEEEIRZ3ZuSDq9ZCChbrF1LoQQQghDOQb4NLAYcDGwte0rJa0OnAycVzK4MCr/lLQI8Fvgp5L+DjxTOKYQQghlqOv2ysWiCNWLiqYQQgghDGVe2xfYngo8YPtKANu3FY4rjEDSyyRtArwNeAI4gJQYfAjYr2RsIYQQivEQt0NoVCSaQgghhDCUZ7tu/6fnsThB7W9HAo/Zftz2s7afsX0icC7w+bKhhRBCKGQdSY9KegxYO99+VNJjkh4tHVyoRzQDDyGEEMKgukasC1iIVBkDMWK970m6yfaaQzw2w/ZaczumEEIIIUwO0aMphBBCCIOKEeuttuAwjy0016IIIYQQwqQTW+dCCCGEEOpztaR9eg9K2hu4tkA8IYQQQpgkYutcCCGEEEJlJC0NnAE8zezE0obA/MDbbT9QKrYQQggh1C0STSGEEEIIlZK0OdDp1XSz7YtLxhNCCCGE+kWiKYQQQgghhBBCCCE0Ino0hRBCCCGEEEIIIYRGRKIphBBCCCGEEEIIITQiEk0hhBBCCA2R9HlJH2vw33uvpGOGeOzf+fOLJZ2Wb68r6S1N/fdDCCGEEOZUJJpCCCGEEOYiSfM0+e/Zvs/2jvnuukAkmkIIIYRQTCSaQgghhBCGIWlhSb+SdIOkmyTtLOlPkpbIj28o6dKup6wj6WJJf5S0T/6azSRdIulnwIx87F2SpkmaLul7nQSUpD0l/Z+ky4BNuuJ4qaQ/SLpa0mFdx1fKcc0PHArsnP/NnSf6/00IIYQQQq9INIUQQgghDO/NwH2217G9JnDeCF+/NrAN8BrgYEkvzsdfBXzG9hqSXgHsDGxie11gJrC7pGWBL5ASTFsCa3T9u0cBx9reCHig9z9q+2ngYOAU2+vaPmWM328IIYQQwphFoimEEEIIYXgzgDdJ+rKk19n+1whf/0vb/7H9D+ASUoIJYJrtu/PtNwIbAFdLmp7vrwy8GrjU9oM5cdSdLNoEODnfPmn831YIIYQQQvPmLR1ACCGEEEI/s/1/kjYg9T46QtIFwDPMXrBbsPcpQ9x/vOuYgBNtH9T9hZK2H+T5w/3bWuq3KgAAAUJJREFUIYQQQgh9JSqaQgghhBCGkbe+PWH7J8DXgPWBP5EqkgB26HnK2yQtKOlFwGbA1YP8sxcBO0paKv83Fpe0InAVsJmkF0maD9ip6zlXALvk27sPEe5jwPPn4NsLIYQQQmhUJJpCCCGEEIa3FjAtb3H7DHA4qY/SUZJ+R+qv1G0a8CvgSuAw2/f1/oO2bwE+C1wg6UbgN8Cytu8HPg/8AbgQuK7rafsDH5R0NbDYELFeAqwRzcBDCCGEUIrsqMAOIYQQQgghhBBCCOMXFU0hhBBCCCGEEEIIoRGRaAohhBBCCCGEEEIIjYhEUwghhBBCCCGEEEJoRCSaQgghhBBCCCGEEEIjItEUQgghhBBCCCGEEBoRiaYQQgghhBBCCCGE0IhINIUQQgghhBBCCCGERkSiKYQQQgghhBBCCCE04v8Hk3vn0VnYDIAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1440x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "subgroups_df[:20].plot(kind='bar', figsize=(20,10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hm... so just looking at a flash of the data, 'AskMen' doesn't seem to stand out as a much higher male:female ratio. It seems like most have more male. Maybe Reddit is just a male-dominated site.\n",
    "\n",
    "Are there any groups where there are more women? From the graph, it looks like the subreddit \"BigBrother\" has more female posters. Let's see what else..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>op_gender</th>\n",
       "      <th>M</th>\n",
       "      <th>W</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>subreddit</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>BigBrother</th>\n",
       "      <td>3564.0</td>\n",
       "      <td>4647.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aww</th>\n",
       "      <td>4400.0</td>\n",
       "      <td>4597.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>counting</th>\n",
       "      <td>267.0</td>\n",
       "      <td>531.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>relationships</th>\n",
       "      <td>10951.0</td>\n",
       "      <td>23858.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rupaulsdragrace</th>\n",
       "      <td>153.0</td>\n",
       "      <td>728.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "op_gender              M        W\n",
       "subreddit                        \n",
       "BigBrother        3564.0   4647.0\n",
       "aww               4400.0   4597.0\n",
       "counting           267.0    531.0\n",
       "relationships    10951.0  23858.0\n",
       "rupaulsdragrace    153.0    728.0"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subgroups_df[subgroups_df.W > subgroups_df.M]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Only 5!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TED responses\n",
    "Finally, we look at the responses to TED talks. Here, this is nothing to merge. We just have the response file. Let's look at it again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(190414, 5)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>post_id</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Aimee Mullins</td>\n",
       "      <td>W</td>\n",
       "      <td>0</td>\n",
       "      <td>Laura</td>\n",
       "      <td>She seems like a truly amazing person, regardl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Aimee Mullins</td>\n",
       "      <td>W</td>\n",
       "      <td>1</td>\n",
       "      <td>Ellen</td>\n",
       "      <td>Aimee's true \"disability\" is when people treat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Aimee Mullins</td>\n",
       "      <td>W</td>\n",
       "      <td>2</td>\n",
       "      <td>Tanya</td>\n",
       "      <td>I love that she took away some of the mystery ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Aimee Mullins</td>\n",
       "      <td>W</td>\n",
       "      <td>3</td>\n",
       "      <td>Salim</td>\n",
       "      <td>VERY inspiring indeed! I love these TED antiqu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Aimee Mullins</td>\n",
       "      <td>W</td>\n",
       "      <td>4</td>\n",
       "      <td>Kiruba</td>\n",
       "      <td>We take running so much for granted that most ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           op_id op_gender  post_id responder_id  \\\n",
       "0  Aimee Mullins         W        0        Laura   \n",
       "1  Aimee Mullins         W        1        Ellen   \n",
       "2  Aimee Mullins         W        2        Tanya   \n",
       "3  Aimee Mullins         W        3        Salim   \n",
       "4  Aimee Mullins         W        4       Kiruba   \n",
       "\n",
       "                                       response_text  \n",
       "0  She seems like a truly amazing person, regardl...  \n",
       "1  Aimee's true \"disability\" is when people treat...  \n",
       "2  I love that she took away some of the mystery ...  \n",
       "3  VERY inspiring indeed! I love these TED antiqu...  \n",
       "4  We take running so much for granted that most ...  "
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ted_responses_df = pd.read_csv(root + 'ted_responses.csv')\n",
    "print(ted_responses_df.shape)\n",
    "ted_responses_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since there's nothing to merge, let's just reorder the columns similarly to the other data frames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>post_id</th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Aimee Mullins</td>\n",
       "      <td>W</td>\n",
       "      <td>Laura</td>\n",
       "      <td>She seems like a truly amazing person, regardl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Aimee Mullins</td>\n",
       "      <td>W</td>\n",
       "      <td>Ellen</td>\n",
       "      <td>Aimee's true \"disability\" is when people treat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Aimee Mullins</td>\n",
       "      <td>W</td>\n",
       "      <td>Tanya</td>\n",
       "      <td>I love that she took away some of the mystery ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>Aimee Mullins</td>\n",
       "      <td>W</td>\n",
       "      <td>Salim</td>\n",
       "      <td>VERY inspiring indeed! I love these TED antiqu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Aimee Mullins</td>\n",
       "      <td>W</td>\n",
       "      <td>Kiruba</td>\n",
       "      <td>We take running so much for granted that most ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   post_id          op_id op_gender responder_id  \\\n",
       "0        0  Aimee Mullins         W        Laura   \n",
       "1        1  Aimee Mullins         W        Ellen   \n",
       "2        2  Aimee Mullins         W        Tanya   \n",
       "3        3  Aimee Mullins         W        Salim   \n",
       "4        4  Aimee Mullins         W       Kiruba   \n",
       "\n",
       "                                       response_text  \n",
       "0  She seems like a truly amazing person, regardl...  \n",
       "1  Aimee's true \"disability\" is when people treat...  \n",
       "2  I love that she took away some of the mystery ...  \n",
       "3  VERY inspiring indeed! I love these TED antiqu...  \n",
       "4  We take running so much for granted that most ...  "
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols = ['post_id', 'op_id', 'op_gender', 'responder_id', 'response_text']\n",
    "\n",
    "ted_responses_df = ted_responses_df[cols]\n",
    "ted_responses_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks good! Check for nulls..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 190414 entries, 0 to 190413\n",
      "Data columns (total 5 columns):\n",
      "post_id          190414 non-null int64\n",
      "op_id            190347 non-null object\n",
      "op_gender        190347 non-null object\n",
      "responder_id     190413 non-null object\n",
      "response_text    190414 non-null object\n",
      "dtypes: int64(1), object(4)\n",
      "memory usage: 7.3+ MB\n"
     ]
    }
   ],
   "source": [
    "ted_responses_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are some nulls in op_id, op_gender, and responder_id. Let's see..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(70, 5)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>post_id</th>\n",
       "      <th>op_id</th>\n",
       "      <th>op_gender</th>\n",
       "      <th>responder_id</th>\n",
       "      <th>response_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>13017</th>\n",
       "      <td>13017</td>\n",
       "      <td>NaN</td>\n",
       "      <td>W</td>\n",
       "      <td>larry</td>\n",
       "      <td>Really a educative and informative post, the p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13018</th>\n",
       "      <td>13018</td>\n",
       "      <td>talks, TEDx, business, cities, collaboration, ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1291</td>\n",
       "      <td>Rachel Botsman</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48424</th>\n",
       "      <td>48424</td>\n",
       "      <td>Kevin Briggs</td>\n",
       "      <td>M</td>\n",
       "      <td>NaN</td>\n",
       "      <td>You know that joke about calling the suicide h...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83960</th>\n",
       "      <td>83960</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83962</th>\n",
       "      <td>83962</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83964</th>\n",
       "      <td>83964</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83966</th>\n",
       "      <td>83966</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83968</th>\n",
       "      <td>83968</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83970</th>\n",
       "      <td>83970</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83972</th>\n",
       "      <td>83972</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83974</th>\n",
       "      <td>83974</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83976</th>\n",
       "      <td>83976</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83978</th>\n",
       "      <td>83978</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83980</th>\n",
       "      <td>83980</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83982</th>\n",
       "      <td>83982</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83984</th>\n",
       "      <td>83984</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83986</th>\n",
       "      <td>83986</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83988</th>\n",
       "      <td>83988</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83990</th>\n",
       "      <td>83990</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83992</th>\n",
       "      <td>83992</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83994</th>\n",
       "      <td>83994</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83996</th>\n",
       "      <td>83996</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83998</th>\n",
       "      <td>83998</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84000</th>\n",
       "      <td>84000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84002</th>\n",
       "      <td>84002</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84004</th>\n",
       "      <td>84004</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84006</th>\n",
       "      <td>84006</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84008</th>\n",
       "      <td>84008</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84010</th>\n",
       "      <td>84010</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84012</th>\n",
       "      <td>84012</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84034</th>\n",
       "      <td>84034</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84036</th>\n",
       "      <td>84036</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84038</th>\n",
       "      <td>84038</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84040</th>\n",
       "      <td>84040</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84042</th>\n",
       "      <td>84042</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84044</th>\n",
       "      <td>84044</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84046</th>\n",
       "      <td>84046</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84048</th>\n",
       "      <td>84048</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84050</th>\n",
       "      <td>84050</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84052</th>\n",
       "      <td>84052</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84054</th>\n",
       "      <td>84054</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84056</th>\n",
       "      <td>84056</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84058</th>\n",
       "      <td>84058</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84060</th>\n",
       "      <td>84060</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84062</th>\n",
       "      <td>84062</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84064</th>\n",
       "      <td>84064</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84066</th>\n",
       "      <td>84066</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84068</th>\n",
       "      <td>84068</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84070</th>\n",
       "      <td>84070</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84072</th>\n",
       "      <td>84072</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84074</th>\n",
       "      <td>84074</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84076</th>\n",
       "      <td>84076</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84078</th>\n",
       "      <td>84078</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84080</th>\n",
       "      <td>84080</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84082</th>\n",
       "      <td>84082</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84084</th>\n",
       "      <td>84084</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84086</th>\n",
       "      <td>84086</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84088</th>\n",
       "      <td>84088</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>talks,</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>167976</th>\n",
       "      <td>167976</td>\n",
       "      <td>NaN</td>\n",
       "      <td>M</td>\n",
       "      <td>larry</td>\n",
       "      <td>Really a educative and informative post, the p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>167977</th>\n",
       "      <td>167977</td>\n",
       "      <td>talks, TEDx, business, creativity, culture, de...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1304</td>\n",
       "      <td>Jason Fried</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>70 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        post_id                                              op_id op_gender  \\\n",
       "13017     13017                                                NaN         W   \n",
       "13018     13018  talks, TEDx, business, cities, collaboration, ...       NaN   \n",
       "48424     48424                                       Kevin Briggs         M   \n",
       "83960     83960                                                NaN       NaN   \n",
       "83962     83962                                                NaN       NaN   \n",
       "83964     83964                                                NaN       NaN   \n",
       "83966     83966                                                NaN       NaN   \n",
       "83968     83968                                                NaN       NaN   \n",
       "83970     83970                                                NaN       NaN   \n",
       "83972     83972                                                NaN       NaN   \n",
       "83974     83974                                                NaN       NaN   \n",
       "83976     83976                                                NaN       NaN   \n",
       "83978     83978                                                NaN       NaN   \n",
       "83980     83980                                                NaN       NaN   \n",
       "83982     83982                                                NaN       NaN   \n",
       "83984     83984                                                NaN       NaN   \n",
       "83986     83986                                                NaN       NaN   \n",
       "83988     83988                                                NaN       NaN   \n",
       "83990     83990                                                NaN       NaN   \n",
       "83992     83992                                                NaN       NaN   \n",
       "83994     83994                                                NaN       NaN   \n",
       "83996     83996                                                NaN       NaN   \n",
       "83998     83998                                                NaN       NaN   \n",
       "84000     84000                                                NaN       NaN   \n",
       "84002     84002                                                NaN       NaN   \n",
       "84004     84004                                                NaN       NaN   \n",
       "84006     84006                                                NaN       NaN   \n",
       "84008     84008                                                NaN       NaN   \n",
       "84010     84010                                                NaN       NaN   \n",
       "84012     84012                                                NaN       NaN   \n",
       "...         ...                                                ...       ...   \n",
       "84034     84034                                                NaN       NaN   \n",
       "84036     84036                                                NaN       NaN   \n",
       "84038     84038                                                NaN       NaN   \n",
       "84040     84040                                                NaN       NaN   \n",
       "84042     84042                                                NaN       NaN   \n",
       "84044     84044                                                NaN       NaN   \n",
       "84046     84046                                                NaN       NaN   \n",
       "84048     84048                                                NaN       NaN   \n",
       "84050     84050                                                NaN       NaN   \n",
       "84052     84052                                                NaN       NaN   \n",
       "84054     84054                                                NaN       NaN   \n",
       "84056     84056                                                NaN       NaN   \n",
       "84058     84058                                                NaN       NaN   \n",
       "84060     84060                                                NaN       NaN   \n",
       "84062     84062                                                NaN       NaN   \n",
       "84064     84064                                                NaN       NaN   \n",
       "84066     84066                                                NaN       NaN   \n",
       "84068     84068                                                NaN       NaN   \n",
       "84070     84070                                                NaN       NaN   \n",
       "84072     84072                                                NaN       NaN   \n",
       "84074     84074                                                NaN       NaN   \n",
       "84076     84076                                                NaN       NaN   \n",
       "84078     84078                                                NaN       NaN   \n",
       "84080     84080                                                NaN       NaN   \n",
       "84082     84082                                                NaN       NaN   \n",
       "84084     84084                                                NaN       NaN   \n",
       "84086     84086                                                NaN       NaN   \n",
       "84088     84088                                                NaN       NaN   \n",
       "167976   167976                                                NaN         M   \n",
       "167977   167977  talks, TEDx, business, creativity, culture, de...       NaN   \n",
       "\n",
       "       responder_id                                      response_text  \n",
       "13017         larry  Really a educative and informative post, the p...  \n",
       "13018          1291                                     Rachel Botsman  \n",
       "48424           NaN  You know that joke about calling the suicide h...  \n",
       "83960        talks,                                                315  \n",
       "83962        talks,                                                315  \n",
       "83964        talks,                                                315  \n",
       "83966        talks,                                                315  \n",
       "83968        talks,                                                315  \n",
       "83970        talks,                                                315  \n",
       "83972        talks,                                                315  \n",
       "83974        talks,                                                315  \n",
       "83976        talks,                                                315  \n",
       "83978        talks,                                                315  \n",
       "83980        talks,                                                315  \n",
       "83982        talks,                                                315  \n",
       "83984        talks,                                                315  \n",
       "83986        talks,                                                315  \n",
       "83988        talks,                                                315  \n",
       "83990        talks,                                                315  \n",
       "83992        talks,                                                315  \n",
       "83994        talks,                                                315  \n",
       "83996        talks,                                                315  \n",
       "83998        talks,                                                315  \n",
       "84000        talks,                                                315  \n",
       "84002        talks,                                                315  \n",
       "84004        talks,                                                315  \n",
       "84006        talks,                                                315  \n",
       "84008        talks,                                                315  \n",
       "84010        talks,                                                315  \n",
       "84012        talks,                                                315  \n",
       "...             ...                                                ...  \n",
       "84034        talks,                                                315  \n",
       "84036        talks,                                                315  \n",
       "84038        talks,                                                315  \n",
       "84040        talks,                                                315  \n",
       "84042        talks,                                                315  \n",
       "84044        talks,                                                315  \n",
       "84046        talks,                                                315  \n",
       "84048        talks,                                                315  \n",
       "84050        talks,                                                315  \n",
       "84052        talks,                                                315  \n",
       "84054        talks,                                                315  \n",
       "84056        talks,                                                315  \n",
       "84058        talks,                                                315  \n",
       "84060        talks,                                                315  \n",
       "84062        talks,                                                315  \n",
       "84064        talks,                                                315  \n",
       "84066        talks,                                                315  \n",
       "84068        talks,                                                315  \n",
       "84070        talks,                                                315  \n",
       "84072        talks,                                                315  \n",
       "84074        talks,                                                315  \n",
       "84076        talks,                                                315  \n",
       "84078        talks,                                                315  \n",
       "84080        talks,                                                315  \n",
       "84082        talks,                                                315  \n",
       "84084        talks,                                                315  \n",
       "84086        talks,                                                315  \n",
       "84088        talks,                                                315  \n",
       "167976        larry  Really a educative and informative post, the p...  \n",
       "167977         1304                                        Jason Fried  \n",
       "\n",
       "[70 rows x 5 columns]"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nulls = ted_responses_df[(ted_responses_df.op_id.isnull()) | (ted_responses_df.op_gender.isnull())\n",
    "                 | (ted_responses_df.responder_id.isnull())]\n",
    "print(nulls.shape)\n",
    "nulls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Something weird is going on here... I may have to examine the actual file.\n",
    "\n",
    "File examination notes:\n",
    "- Post 13017: This one is okay - just doesn't have an op_id (will fill with empty string).\n",
    "- Post 13018: This one is all mixed up and doesn't make any sense, so I'm just going to delete it.\n",
    "- Posts 83960-84088: Something really weird is going on with these posts (but only the even numbers! The odd numbers are okay.). They are all the same and don't make any sense, and are therefore irrelevant. I'm just going to delete these rows entirely.\n",
    "- Post 167976: This one is also file, just doesn't have an op_id (will fill with empty string).\n",
    "- Post 167977: Again, this one is all mixed up and doesn't make any sense, so I'm going to delete it.\n",
    "\n",
    "Whatever is happening with 13017-13018 and 167976-167977 is the same thing. Both responder ids for the first file in each pair are \"larry\", where there's no op_id. Also, they both have that weird string of \"talks, TEDx, business, creativity, culture, design, technology, work!\" as the op_id in the second file in each pair, with the atual op_id as the response text. Lastly, the response text from \"larry\" is exactly the same in both: \"Really a educative and informative post, the post is good in all regards,I am glad to read this post. [REDACTED: some sketchy link] Christmas Deals!\" Clearly, \"larry\" is some sort of scam posting and it may have messed up the formatting. I'm not going to delete the scam post, but I will delete the nonsense row that comes after it. Whew!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "# first fill nulls with empty string\n",
    "ted_responses_df.fillna('', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(190412, 5)"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get rid of the weird mixed up rows\n",
    "#ted_responses_df = ted_responses_df.drop(ted_responses_df.index[[13018, 167977]])\n",
    "ted_responses_df = ted_responses_df[~ted_responses_df.op_id.str.startswith('talks, TEDx')]\n",
    "ted_responses_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Good, so we only got rid of those 2 weird rows. Now only getting rid of the ones that just say \"315\" in the response text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(190347, 5)"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ted_responses_df = ted_responses_df[ted_responses_df.response_text != \"315\"]\n",
    "ted_responses_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay so we got rid of everything weird!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 190347 entries, 0 to 190413\n",
      "Data columns (total 5 columns):\n",
      "post_id          190347 non-null int64\n",
      "op_id            190347 non-null object\n",
      "op_gender        190347 non-null object\n",
      "responder_id     190347 non-null object\n",
      "response_text    190347 non-null object\n",
      "dtypes: int64(1), object(4)\n",
      "memory usage: 8.7+ MB\n"
     ]
    }
   ],
   "source": [
    "ted_responses_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly - any duplicate responses that are meaningful?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Thank you for your TED talk!    63\n",
       "good                            54\n",
       "Amazing!                        53\n",
       "Brilliant!                      42\n",
       "nice                            39\n",
       "Awesome                         31\n",
       "great                           30\n",
       "nice post                       28\n",
       "Amazing                         27\n",
       "Great talk!                     26\n",
       "Name: response_text, dtype: int64"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ted_responses_df.response_text.value_counts().head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay - so this just seems like the usual duplicates we would get.\n",
    "\n",
    "### Pickling\n",
    "Our last pickle!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pickle it\n",
    "ted_responses_df.to_pickle(\"ted_responses_df.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}