curriculum/2_data_exploration_and_analysis/text-analysis/JobsData_TextAnalysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Analysis\n",
    "---\n",
    "\n",
    "## Introduction\n",
    "\n",
    "Text Analysis is used for summarizing or getting useful information out of a large amount of unstructured text stored in documents. This opens up the opportunity of using text data alongside more conventional data sources (e.g, surveys and administrative data). The goal of text analysis is to take a large corpus of complex and unstructured text data and extract important and meaningful messages in a comprehensible meaningful way. \n",
    "\n",
    "\n",
    "Text Analysis can help with the following tasks:\n",
    "\n",
    "* **Informationan retrieval**: Help find relevant information in large databases such as a systematic literature review. \n",
    "\n",
    "* **Clustering and text categorization**: Techniques like topic modeling can summarize a large corpus of text by finding the most important phrases. \n",
    "\n",
    "* **Text Summarization**: Create category-sensitive text summaries of a large corpus of text. \n",
    "\n",
    "* **Machine Translation**: Translate from one language to another. \n",
    "\n",
    "In this tutorial, we are going to analyze job advertisements from 2010-2015 using topic modeling to examine the content of our data and document classification to tag the type of job in the advertisement. First we will go over how to transform our data into a matrix that can be read in by an algorithm. \n",
    " \n",
    "\n",
    "\n",
    "## Glossary of Terms\n",
    "\n",
    "* **Corpus**: A corpus of documents is the set of all documents in the dataset.\n",
    "\n",
    "* **Tokenize**: Tokenization is the process by which text is sepearated into meaningful terms or phrases. In english this is fairly triial as words as separated by whitespace. \n",
    "\n",
    "* **Stemming**: Stemming is a type of text normalization where words that have different forms but their essential meaning at the same are normalized to the original dictionary form of a word. For example \"go,\" \"went,\" and \"goes\" all stem from the lemma \"go.\"\n",
    "\n",
    "* **TFIDF**: TFIDF (Term frequency-inverse document frequency) is an example of feature enginnering where the most important words are extracted by taking account their frequency in documents and the entire corpus of documents as a whole.\n",
    "\n",
    "* **Topic Modeling**: Topic modeling is an unsupervised learning method where groups of co-occuring words are clustered into topics. Typically, the words in a cluster should be related and make sense (e.g, boat, ship, captain). Individual documents will then fall into multiple topics. \n",
    "\n",
    "* **LDA**: LDA (latent Dirichlet allocation) is a type of probabilistic model commonly used for topic modelling. \n",
    "\n",
    "* **Stop Words**: Stop words are words that have little semantic meaning like prepositions, articles and common nouns. They can often be ignored. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n",
      "[nltk_data] Downloading package stopwords to /home/akumar/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%pylab inline \n",
    "import nltk\n",
    "import ujson\n",
    "import re\n",
    "import time\n",
    "import progressbar\n",
    "\n",
    "import pandas as pd\n",
    "from __future__ import print_function\n",
    "from six.moves import zip, range \n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer\n",
    "from sklearn.decomposition import LatentDirichletAllocation\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import precision_recall_curve, roc_auc_score, auc\n",
    "from sklearn import preprocessing\n",
    "from collections import Counter, OrderedDict\n",
    "from nltk.corpus import stopwords\n",
    "from nltk import PorterStemmer\n",
    "\n",
    "nltk.download('stopwords') #download the latest stopwords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load the Data\n",
    "\n",
    "Our Dataset for this tutorial will be a subset jobs-ad data from 2010-2015 compiled by the Commonwealth of Virginia. The full data and how this subset was created can be found in the data folder in this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df_jobs_data = pd.read_csv('./data/jobs_subset.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Explore the Data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>normalizedTitle_onetName</th>\n",
       "      <th>normalizedTitle_onetCode</th>\n",
       "      <th>jobDescription</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Lawyers</td>\n",
       "      <td>23-1011.00</td>\n",
       "      <td>This position is in support of the United Stat...</td>\n",
       "      <td>Paralegal (Buffalo, NY)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Software Quality Assurance Engineers and Testers</td>\n",
       "      <td>15-1199.01</td>\n",
       "      <td>The successful candidate will maintain and dev...</td>\n",
       "      <td>System Integration / Test Engineer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Software Quality Assurance Engineers and Testers</td>\n",
       "      <td>15-1199.01</td>\n",
       "      <td>This position is a Senior System Integration &amp;...</td>\n",
       "      <td>System Integration / Test Engineer Staff</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Software Quality Assurance Engineers and Testers</td>\n",
       "      <td>15-1199.01</td>\n",
       "      <td>In this position, candidates may participate i...</td>\n",
       "      <td>Mult Func Fin Analyst</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Software Quality Assurance Engineers and Testers</td>\n",
       "      <td>15-1199.01</td>\n",
       "      <td>This is a business operations analyst position...</td>\n",
       "      <td>Mult Func Fin Analyst</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           normalizedTitle_onetName normalizedTitle_onetCode  \\\n",
       "0                                           Lawyers               23-1011.00   \n",
       "1  Software Quality Assurance Engineers and Testers               15-1199.01   \n",
       "2  Software Quality Assurance Engineers and Testers               15-1199.01   \n",
       "3  Software Quality Assurance Engineers and Testers               15-1199.01   \n",
       "4  Software Quality Assurance Engineers and Testers               15-1199.01   \n",
       "\n",
       "                                      jobDescription  \\\n",
       "0  This position is in support of the United Stat...   \n",
       "1  The successful candidate will maintain and dev...   \n",
       "2  This position is a Senior System Integration &...   \n",
       "3  In this position, candidates may participate i...   \n",
       "4  This is a business operations analyst position...   \n",
       "\n",
       "                                      title  \n",
       "0                   Paralegal (Buffalo, NY)  \n",
       "1        System Integration / Test Engineer  \n",
       "2  System Integration / Test Engineer Staff  \n",
       "3                     Mult Func Fin Analyst  \n",
       "4                     Mult Func Fin Analyst  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_jobs_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our table has 4 fields. `normalizedTitle_onetName`, `normalizedTitle_onetCode`, `jobDescription`, `title`\n",
    "\n",
    "Onet is an online database that contains hundreds of occupational definitions. https://en.wikipedia.org/wiki/Occupational_Information_Network\n",
    "\n",
    "The normalizedTitle_onetName and the normalizedTitle_onetCode are derived from the Onet Database. We wil use the names in the document tagging portion of the tutorial. The jobDescription is the actual jobDescription and the title is derived from the jobDescription. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many unique job titles are in this dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Lawyers', 'Software Quality Assurance Engineers and Testers',\n",
       "       'Graduate Teaching Assistants', 'Financial Examiners',\n",
       "       'Credit Analysts'], dtype=object)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_jobs_data.normalizedTitle_onetName.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Paralegal (Buffalo, NY)', 'System Integration / Test Engineer',\n",
       "       'System Integration / Test Engineer Staff', ...,\n",
       "       'JAVA ENGINEER(perm)', 'BSA Analyst', 'AML/KYC Analyst'], dtype=object)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_jobs_data.title.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2496,)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_jobs_data.title.unique().shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 5 unique categories of jobs using the ONet classification. There are too many unique job titles in the title field to display. We can see the shape of the array of unique titles is 2496 titles. \n",
    "\n",
    "Each job description has a great deal of information contained in unstructered text. We can use text analysis to find overarching concepts that are in our corpus.  This will allow us to the discover the most important words and phrases in the job descriptions and give us a big-picture of the content in our collection. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Topic Modeling\n",
    "\n",
    " We are going to apply topic modeling, an unsuperivised learning method, to our corpus to find the high-level topics in our corpus as a first-go for exploring our data. As we apply topic modeling we will discuss ways of cleaning and preprocessing our data to get the best results.\n",
    "\n",
    "Topic modeling is a broad subfield of machine learning and natural language processing. We are going to focus on one approach, Latent Dirichlet allocation (LDA). LDA is a fully Bayesian extension of probabilistic latent semantic  indexing, itself a probabilistic extension of latent semantic analysis.\n",
    "\n",
    "In topic modeling we first assume the existence of topics in the corpus and that there is a small number of topics that can explain a corpus. Topics, in this case, are a ranked-list of words from our corpus, with the highest probability words at the top. A single document can be explained by multiple topics. For instance, an article on net neutrality has to do with both technology and politics. The set of topics used by a document is known as the document's allocation, hence, the name latent Dirchlet allocation, each document has an allocation of latent topics allocated by Dirchlet distribution. \n",
    "\n",
    "## Processing text data\n",
    "\n",
    "The first important step in working with text data is cleaning and processing the data. This include but is not limited to *forming a corpus of text, tokenization, removing stop-words, finding words colocated together (N-grams), and stemming and lemmatization*. Each of these steps will be discussed below. The ultimate goal is to transform our text data into a form an algorithm can work with. A sequence of symbols cannot be fed directly into an algorithm. Algorithms expect numerical feature vectors with fixed size rather then a document with a variable document length. We will be transforming our text corpus into a *bag of n-grams* to be further analyzed. In this form our text data is represented as a matrix where each row refers to a specific job description (document) and each column is the occurence of a word (feature).\n",
    "\n",
    "\n",
    "### Bag of n-gram representation example\n",
    "\n",
    "Ultimately, we want to take our collection of documents, corpus, and convert it into a matrix. Fortunately sklearn has a pre-built object, `CountVectorizer`, that can tokenize, eliminate stopwords, identify n-grams and stem our corpus, outputing a matrix in one step. Before we apply the vectorizer on our corpus of data we are going to apply it to a toy example so we can understand the output and how a bag of words is represented. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_bag_of_words(corpus,\n",
    "                       NGRAM_RANGE=(0,1),\n",
    "                       stop_words = None,\n",
    "                        stem = False,\n",
    "                       MIN_DF = 0.05,\n",
    "                       MAX_DF = 0.95,\n",
    "                       USE_IDF=False):\n",
    "    \"\"\"\n",
    "    Turn a corpus of text into a bag-of-words.\n",
    "    \n",
    "    Parameters\n",
    "    -----------\n",
    "    corpus: ls\n",
    "        test of documents in corpus    \n",
    "    NGRAM_RANGE: tupule\n",
    "        range of N-gram default (0,1)\n",
    "    stop_words: ls\n",
    "        list of commonly occuring words that have little semantic\n",
    "        value\n",
    "    stem: bool\n",
    "        use a stemmer to stem words\n",
    "    MIN_DF: float\n",
    "       exclude words that have a frequency less than the threshold\n",
    "    MAX_DF: float\n",
    "        exclude words that have a frequency greater than the threshold\n",
    "    \n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    bag_of_words: scipy sparse matrix\n",
    "        scipy sparse matrix of text\n",
    "    features:\n",
    "        ls of words\n",
    "    \"\"\"\n",
    "    #parameters for vectorizer \n",
    "    ANALYZER = \"word\" #unit of features are single words rather then phrases of words \n",
    "    STRIP_ACCENTS = 'unicode'\n",
    "     \n",
    "    if stem:\n",
    "        tokenize = lambda x: [stemmer.stem(i) for i in x.split()]\n",
    "    else:\n",
    "        tokenize = None\n",
    "    vectorizer = CountVectorizer(analyzer=ANALYZER,\n",
    "                                tokenizer=tokenize, \n",
    "                                ngram_range=NGRAM_RANGE,\n",
    "                                stop_words = stop_words,\n",
    "                                strip_accents=STRIP_ACCENTS,\n",
    "                                min_df = MIN_DF,\n",
    "                                max_df = MAX_DF)\n",
    "    \n",
    "    bag_of_words = vectorizer.fit_transform( corpus ) #transform our corpus is a bag of words \n",
    "    features = vectorizer.get_feature_names()\n",
    "\n",
    "    if USE_IDF:\n",
    "        NORM = None #turn on normalization flag\n",
    "        SMOOTH_IDF = True #prvents division by zero errors\n",
    "        SUBLINEAR_IDF = True #replace TF with 1 + log(TF)\n",
    "        transformer = TfidfTransformer(norm = NORM,smooth_idf = SMOOTH_IDF,sublinear_tf = True)\n",
    "        #get the bag-of-words from the vectorizer and\n",
    "        #then use TFIDF to limit the tokens found throughout the text \n",
    "        tfidf = transformer.fit_transform(bag_of_words)\n",
    "        \n",
    "        return tfidf, features\n",
    "    else:\n",
    "        return bag_of_words, features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "toy_corpus = ['this is document one', 'this is document two', 'text analysis on documents is fun'] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "toy_bag_of_words, toy_features = create_bag_of_words(toy_corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The counter_vectorizer outputs a matrix. In this case a sparse matrix, a matrix that has a lot more 0s then 1s. To save space scipy has special methods for storing sparse matrices in a space-efficient way rather than saving many many 0s. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['this is document one',\n",
       " 'this is document two',\n",
       " 'text analysis on documents is fun']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toy_corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 1, 0, 0, 0, 1, 0, 1, 0],\n",
       "       [0, 1, 0, 0, 0, 0, 0, 1, 1],\n",
       "       [1, 0, 1, 1, 1, 0, 1, 0, 0]], dtype=int64)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np_bag_of_words = toy_bag_of_words.toarray()\n",
    "np_bag_of_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['analysis',\n",
       " 'document',\n",
       " 'documents',\n",
       " 'fun',\n",
       " 'on',\n",
       " 'one',\n",
       " 'text',\n",
       " 'this',\n",
       " 'two']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toy_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our data has been transformed into a 3x9 matrix where each row corresponds to a document and the columns correspond to the features. A 1 indicates the existence of the feature or word in the document, 0 indicates the word is not present.Our toy corpus is now ready to be analyzed. We illustrated bag of n-gram with a toy example because the matrix for a much larger corpus would be much larger and harder to interpet on our corpus of data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### word counts\n",
    "\n",
    "As a initial look into the data we can examine what the top few words are in our corpus. We can sum the columns of the bag_of_words and then convert to a numpy array. From here we can zip the features and word_count into a dictionary\n",
    "and display the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_word_counts(bag_of_words, feature_names):\n",
    "    \"\"\"\n",
    "    Get the ordered word counts from a bag_of_words\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    bag_of_words: obj\n",
    "        scipy sparse matrix from CounterVectorizer\n",
    "    feature_names: ls\n",
    "        list of words\n",
    "        \n",
    "    Returns\n",
    "    -------\n",
    "    word_counts: dict\n",
    "        Dictionary of word counts\n",
    "    \"\"\"\n",
    "    np_bag_of_words = bag_of_words.toarray()\n",
    "    word_count = np.sum(np_bag_of_words,axis=0)\n",
    "    np_word_count = np.asarray(word_count).ravel()\n",
    "    dict_word_counts = dict( zip(feature_names, np_word_count) )\n",
    "    \n",
    "    orddict_word_counts = OrderedDict( \n",
    "                                    sorted(dict_word_counts.items(), key=lambda x: x[1], reverse=True), )\n",
    "    \n",
    "    return orddict_word_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('this', 2),\n",
       "             ('document', 2),\n",
       "             ('on', 1),\n",
       "             ('documents', 1),\n",
       "             ('analysis', 1),\n",
       "             ('fun', 1),\n",
       "             ('one', 1),\n",
       "             ('text', 1),\n",
       "             ('two', 1)])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_word_counts(toy_bag_of_words, toy_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text Corpora\n",
    "\n",
    "First we need to form our corpus, a set of multiple similiar documents. In our case, our corpus is the set of all job descriptions. We can pull out the job descriptions from the data frame by pulling out the underlying numpy array using the `.values` attribute. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ \"This position is in support of the United States Attorney's Office in Buffalo, NY located at 138 Delaware Avenue. The estimated length of this assignment is through May 31, 2016 with the possibility of extensions. Overtime and travel may be authorized as needed. The Employee will perform the following tasks, but not limited to: *Serves as a primary resource for the office on all aspects of docketing. *Recommends revisions in office procedures to improve the docketing function by expediting case processing, changing codes for data elements, etc. *Reviews reports to identify recurring errors and trains new employees in the operation and use of the case tracking system. Advises other personnel on appropriate codes for unprecedented cases. Explains the operation of the system and importance of timely information to new attorneys and support staff. *Responsibility for developing/maintaining automated records for the criminal and civil case workload from initial referral to final disposition. *Regularly inputs new information into the database from LIONS coding sheets provided by District Division Chiefs. *Runs regularly scheduled and preprogrammed management reports with the assistance of System Manager. *Requests new or modified reports and/or internal processing procedures to meet perceived changes in the information needs of the office and/or to improve docketing efficiency/effectiveness. *Discusses changes with supervisor. *Maintains a working knowledge of the Districts and Departments docket and reporting procedures in order to assure proper interpretation and data entry of codes. *Assists Legal Assistants and other employees in obtaining information from the database and in interpreting output. *Provides ongoing instruction as to processing requirements. *Receives numerous telephone calls on case status from client agencies, the courts, employees within the office, city, county and state law office, private attorneys and individuals. *Exercises tact, discretion and learned judgment in determining what information and be provided and to whom. *Must be aware of what cases are sensitive and for which no information can be provided. *Perform others duties as necessary. Basic Qualifications *Proven excellent written and oral communication skills. *Paralegal Certificate. An equivalent level of training may be substituted with the approval of the COTR. *At least ONE (1) year of litigation paralegal experience. *Must have basic legal knowledge, including knowledge of standard legal citation system sufficient to perform basic legal research. *Thorough knowledge of legal research tools such as LEXIS and WestLaw. *Should have hands-on familiarity with a variety of computer applications, including word processing, database, spreadsheet, imaging, and telecommunications. *Ability to consistently deliver highest quality of work under extreme pressure will be very important. *Some walking, standing, bending and carrying of small office items *Ability to obtain a security clearance. *Must be a U.S. Citizen. Desired skills *Experience working in a Government and/or Litigation Support environment in conjunction with basic qualifications. *Proficient in Microsoft Office software programs (i.e., Word, Excel, Outlook, MS Access or other data bases), Word Perfect and Adobe Acrobat Professional, strongly preferred. *Experience with docketing processes and procedures. *Attention to detail. *Trial experience, helpful. *Automated litigation support experience (i.e. Concordance, Summation,etc...), very helpful. *Experience with organization and management of documents. *Current or active clearance. As a leading technology innovation company, Lockheed Martins team of 113,000 people works with partners around the world to bring proven performance to our customers toughest challenges. Lockheed Martin has employees based in all 50 states and more than 570 facilities that span 70 countries. Join us at Lockheed Martin, where were engineering a better tomorrow. *Lockheed Martin is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status.* Job Location(s): Buffalo New York\",\n",
       "       \"The successful candidate will maintain and develop electronic production and development test solutions. Provide hands on hardware/software troubleshooting and upgrade support for existing production test stations in a real time environment. Participate in the design, build, integration and validation of automated and manual test equipment. Interpret and understand test requirements, develop/modify test procedures, test reports and other test related documentation. Must be able to interface with the customer during planned meetings and support required presentations. The successful candidate will also have the ability and willingness to on occasion support our Product Support group in maintaining, troubleshooting and servicing electronic fielded equipment. Participate in the integration, installation and validation of fielded equipment. Interpret and understand repair procedures and other repair related documentation. Must be able to interface with the customer during field service trips and internal status meetings. This requires the ability to travel at a moment's notice for 1-2 weeks at a time. Basic Qualifications Ability to function as self-starter. Strong written and oral communication skills. Understanding of engineering fundamentals. test procedure and script development and execution at the system level. Familiarity with test requirement identification/evaluation. Ability to perform effectively in a team environment, interacting with test engineering, test conductors, quality, and production supervisors. A desire to gain direct hands-on hardware/software test experience with automated test systems integration, collecting test data, test analysis, troubleshooting, and anomaly resolution. Ability and willingness to travel frequently. Ability to obtain a secret security clearance. Bachelors degree from an accredited college in a related discipline, or equivalent experience/combined education. Desired skills Knowledge of LabView programming or Pro Engineering with the ability and desire to become proficient. An understanding of schematic development, hands on assembly, software development, integration and troubleshooting. Ability and desire to gain hands on experience with automated test systems including analog, digital and optical test equipment. Familiarity with the MFC-Archbald products would be a benefit. As a leading technology innovation company, Lockheed Martins vast team works with partners around the world to bring proven performance to our customers toughest challenges. Lockheed Martin has employees based in many states throughout the U.S., and Internationally, with business locations in many nations and territories. Join us at Lockheed Martin, where were engineering a better tomorrow. *Lockheed Martin is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status.* Job Location(s): Archbald Pennsylvania\",\n",
       "       'This position is a Senior System Integration & Test Engineer on the RUSH program. The System I&T Engineer will support both the legacy mission management system as well as the transition to operations of new multi-INT capabilities. The successful candidate will develop test plans, procedures, and automated test scripts to verify the system functional and performance requirements and will support enterprise test activities. Applicants selected will be subject to a government security investigation and must meet eligibility requirements for access to classified information. Basic Qualifications - Ability to effectively communicate and collaborate across team boundaries with subcontractors as well as customers - Experience in developing test plans, test procedures, and test scripts - Experience with testing external interfaces - Experience performing test execution and requirement verification - Active TS/SCI clearance Desired skills - Experience with test automation - Security Authorization and Authentication understanding - Knowledge of Agile methodologies - Geospatial systems experience As a leading technology innovation company, Lockheed Martins team of 113,000 people works with partners around the world to bring proven performance to our customers toughest challenges. Lockheed Martin has employees based in all 50 states and more than 570 facilities that span 70 countries. Join us at Lockheed Martin, where were engineering a better tomorrow. *Lockheed Martin is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status.* Job Location(s): Herndon Pennsylvania, Herndon Virginia',\n",
       "       ...,\n",
       "       'Our suburban client is expanding and looking for a Senior Financial Analyst. The Senior Financial Analyst will be responsible for compiling and analyzing financial information in support of the Regional Finance Director. Analysis of monthly, quarterly and annual reports to ensure information has been recorded accurately. The analyst will assist in the forecasting and budgeting process, assist in month end close and provide detailed variance analysis to senior management. Additional duties will include pricing analysis, monthly forecasting, as well as mentoring of entry level staff members. Ad hoc analytical and financial reporting projects will be assigned by senior management. Robert Half Finance & Accounting is the world\\'s leader in specialized financial staffing. We provide exciting full-time opportunities in the areas of accounting, bookkeeping, finance, audit, taxation and more. We pioneered the professional staffing industry, and we\\'ve been successfully matching professionals with employers since 1948. Our proven proprietary processes, along with our relationships in more than 340 locations worldwide, allow us to provide you unparalleled access to exciting career opportunities. But don\\'t take our word for it. Our company once again was named to FORTUNE magazine\\'s list of \"World\\'s Most Admired Companies.\" (March 1, 2015), and 9 out of 10 of our clients and candidates would recommend our service to a colleague. Apply for this job now or contact our nearest office at 1.800.474.4253 for additional information. All applicants applying for U.S. job openings must be authorized to work in the United States. All applicants applying for Canadian job openings must be authorized to work in Canada. Equal Opportunity Employer M/F/Disability/Vet By clicking \\'Apply Now\\' you are agreeing to Robert Half Terms of Use *Req ID:* 02920-109039 *Functional Role:* Financial Analyst-Manager *Country:* USA *State:* NY *City:* BUFFALO *Postal Code:* 14206 *Compensation:* $75,000.00 to $90,000.00 per year *Requirements:* Suburban service company is looking to add a Senior Financial Analyst to the team. To be considered as a candidate you must have a Bachelors degree in Finance or Accounting, an MBA would be a plus. Must have five plus years of experience in a related capacity, excellent analytical skills, strong Excel experience and have the ability to work in a team environment. For immediate consideration, please contact Marc Genareo at (716)842-6384 or submit a resume in strict confidence for immediate review to marc.genareo@roberthalf.com.',\n",
       "       'Remote Opportunity (Pennsylvania) - Robert Half is working with a Multinational Mass Media and Information Company who is seeking a Transactional Attorney for a remote, temporary position. Ideal candidates will have 6+ years of experience as a Business Transactional lawyer in a law firm or as in-house counsel in Pennsylvania. Candidates should have experience with entity formation and governance (corporations, LLCs, LLPs), M&A; deals, and general commercial contract drafting. Responsibilities can include: Research, write, edit, and update state-specific business organizations and corporate transactional resources, including practice notes, checklists and standard documents explaining how to form corporate entities under state law, including guidance related to formation, governance and mergers or acquisitions for use by attorneys at law firms and in-house counsel. Be available to answer questions and perform research on discrete state law issues that arise in final review of these resources. Candidates must be actively licensed in the state of Pennsylvania. This position will begin as soon as the right candidate is found and requires 20 - 40 hours/week completed on a remote basis. Given the remote nature of this position candidates need to have Windows-based PC and high speed internet. The role is expected to last 3+ months. For immediate consideration, please forward your MS Word resume to alicia.reynolds@rht.com. Please include law school class rank or GPA on your resume. Located in major North American and global markets, Robert Half Legal is the premier provider of legal staffing and consulting professionals on a full-time, project, and temporary basis, serving both law firms and corporate legal departments. We also provide project and e-discovery teams and workspace for a wide range of initiatives, including litigation support, M&A; and document review matters. Our dedicated teams\\' industry experience allows us to quickly match skilled legal professionals with the best available jobs. We offer challenging opportunities, competitive compensation and benefits, and skills-enhancement training. Our parent company, Robert Half, once again was named to FORTUNE magazine\\'s list of \"World\\'s Most Admired Companies.\" (March 1, 2015). Call your local Robert Half Legal office at 1.800.870.8367 to discover more about this position. Apply for this job now or contact our branch office for additional information. All applicants applying for U.S. job openings must be authorized to work in the United States. All applicants applying for Canadian job openings must be authorized to work in Canada. Equal Opportunity Employer M/F/Disability/Vet By clicking \\'Apply Now\\' you are agreeing to Robert Half Terms of Use *Req ID:* 04860-9812252 *Functional Role:* Attorney-licensed II 4-9 yrs exp *Country:* USA *State:* PA *City:* Pittsburgh *Postal Code:* 15201 *Compensation:* DOE *Requirements:* 6+ years of experience as a Business Transactional lawyer in a law firm or as in-house counsel in Pennsylvania. Candidates must be actively licensed in the state of Pennsylvania.',\n",
       "       \"The Credit Analyst will support the commercial lending function by providing timely and accurate evaluations of existing and potential commercial customers' creditworthiness. The Credit analyst will also maintain a culture of strong compliance within assigned realm of responsibilities. The primary responsibilities of the Credit Analyst will include: - Preparing credit write-ups which analyze the creditworthiness of commercial customers in support of approving authorities, culminating to an accurate, well-informed decision. - Creating concise and thorough analysis of the risks and factors of commercial loan relationships by analyzing financial statements including, income statements, balance sheets, tax returns, cash flow statements, etc. - Analyzing and spreading financial statements, preparing debt coverage analyses, and assessing financial conditions and collateral. - Completing at least annual reviews of larger commercial relationships. - Communicating effectively with company stakeholders in order to produce accurate and informative evaluations. - Demonstrating a knowledge and sensitivity to current regulatory environment. - Complying and adhering to policies and procedures. - Performing other duties as assigned. The minimum qualifications for the Credit Analyst Position are: - Bachelor's degree, preferably with an emphasis on finance, accounting, or economics. - Strong communication skills (written and verbal). - Deadline-focused and fast-paced personality. - Exceptional analytical and problem-solving skills. - Two (2)+ years experience is preferred. - Proficiency with Microsoft Office Suite - Knowledge of basic accounting principles is preferred. Banking Credit Analysis, Credit Analysis, Credit - Commercial, Microsoft Excel, Microsoft Word, Microsoft PowerPoint.\"], dtype=object)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus = df_jobs_data['jobDescription'].values #pull all the jobDescriptions and put them in a numpy array \n",
    "corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_topics(tfidf, features, N_TOPICS=3, N_TOP_WORDS=5,):\n",
    "    \"\"\"\n",
    "    Given a matrix of features of text data generate topics\n",
    "    \n",
    "    Parameters\n",
    "    -----------\n",
    "    tfidf: scipy sparse matrix\n",
    "        sparse matrix of text features\n",
    "    N_TOPICS: int\n",
    "        number of topics (default 10)\n",
    "    N_TOP_WORDS: int\n",
    "        number of top words to display in each topic (default 10)\n",
    "        \n",
    "    Returns\n",
    "    -------\n",
    "    ls_keywords: ls\n",
    "        list of keywords for each topics\n",
    "    doctopic: array\n",
    "        numpy array with percentages of topic that fit each category\n",
    "    N_TOPICS: int\n",
    "        number of assumed topics\n",
    "    N_TOP_WORDS: int\n",
    "        Number of top words in a given topic. \n",
    "    \"\"\"\n",
    "    \n",
    "    with progressbar.ProgressBar(max_value=progressbar.UnknownLength) as bar:\n",
    "        i=0\n",
    "        lda = LatentDirichletAllocation( n_topics= N_TOPICS,\n",
    "                                       learning_method='online') #create an object that will create 5 topics\n",
    "        bar.update(i)\n",
    "        i+=1\n",
    "        doctopic = lda.fit_transform( tfidf )\n",
    "        bar.update(i)\n",
    "        i+=1\n",
    "        \n",
    "        ls_keywords = []\n",
    "        for i,topic in enumerate(lda.components_):\n",
    "            word_idx = np.argsort(topic)[::-1][:N_TOP_WORDS]\n",
    "            keywords = ', '.join( features[i] for i in word_idx)\n",
    "            ls_keywords.append(keywords)\n",
    "            print(i, keywords)\n",
    "            bar.update(i)\n",
    "            i+=1\n",
    "            \n",
    "    return ls_keywords, doctopic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "corpus_bag_of_words, corpus_features = create_bag_of_words(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's examine our features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['00',\n",
       " '000',\n",
       " '10',\n",
       " '100',\n",
       " '11',\n",
       " '12',\n",
       " '15',\n",
       " '20',\n",
       " '2015',\n",
       " '25',\n",
       " '30',\n",
       " '40',\n",
       " '50',\n",
       " '500',\n",
       " '80',\n",
       " '800',\n",
       " 'aa',\n",
       " 'abilities',\n",
       " 'ability',\n",
       " 'able',\n",
       " 'about',\n",
       " 'above',\n",
       " 'academic',\n",
       " 'access',\n",
       " 'accordance',\n",
       " 'according',\n",
       " 'accounting',\n",
       " 'accounts',\n",
       " 'accredited',\n",
       " 'accuracy',\n",
       " 'accurate',\n",
       " 'achieve',\n",
       " 'across',\n",
       " 'act',\n",
       " 'action',\n",
       " 'active',\n",
       " 'activities',\n",
       " 'addition',\n",
       " 'additional',\n",
       " 'administration',\n",
       " 'administrative',\n",
       " 'admired',\n",
       " 'advance',\n",
       " 'advanced',\n",
       " 'affirmative',\n",
       " 'again',\n",
       " 'against',\n",
       " 'age',\n",
       " 'agencies',\n",
       " 'agency',\n",
       " 'agile',\n",
       " 'agreeing',\n",
       " 'al',\n",
       " 'all',\n",
       " 'along',\n",
       " 'also',\n",
       " 'an',\n",
       " 'analysis',\n",
       " 'analyst',\n",
       " 'analytical',\n",
       " 'analyze',\n",
       " 'and',\n",
       " 'anos',\n",
       " 'any',\n",
       " 'applicable',\n",
       " 'applicants',\n",
       " 'application',\n",
       " 'applications',\n",
       " 'apply',\n",
       " 'applying',\n",
       " 'appropriate',\n",
       " 'architecture',\n",
       " 'are',\n",
       " 'area',\n",
       " 'areas',\n",
       " 'around',\n",
       " 'as',\n",
       " 'aspects',\n",
       " 'assessment',\n",
       " 'assigned',\n",
       " 'assist',\n",
       " 'assistance',\n",
       " 'assistant',\n",
       " 'assisting',\n",
       " 'assists',\n",
       " 'associate',\n",
       " 'associated',\n",
       " 'assurance',\n",
       " 'at',\n",
       " 'attention',\n",
       " 'attorney',\n",
       " 'audit',\n",
       " 'authorized',\n",
       " 'automated',\n",
       " 'automation',\n",
       " 'available',\n",
       " 'bachelor',\n",
       " 'bachelors',\n",
       " 'background',\n",
       " 'base',\n",
       " 'based',\n",
       " 'basic',\n",
       " 'basis',\n",
       " 'be',\n",
       " 'been',\n",
       " 'being',\n",
       " 'benefits',\n",
       " 'best',\n",
       " 'better',\n",
       " 'between',\n",
       " 'billion',\n",
       " 'board',\n",
       " 'both',\n",
       " 'branch',\n",
       " 'bring',\n",
       " 'broad',\n",
       " 'bs',\n",
       " 'build',\n",
       " 'building',\n",
       " 'business',\n",
       " 'but',\n",
       " 'by',\n",
       " 'ca',\n",
       " 'call',\n",
       " 'can',\n",
       " 'canada',\n",
       " 'canadian',\n",
       " 'candidate',\n",
       " 'candidates',\n",
       " 'capabilities',\n",
       " 'care',\n",
       " 'career',\n",
       " 'case',\n",
       " 'cases',\n",
       " 'category',\n",
       " 'center',\n",
       " 'certification',\n",
       " 'challenges',\n",
       " 'challenging',\n",
       " 'change',\n",
       " 'changes',\n",
       " 'changing',\n",
       " 'characteristic',\n",
       " 'citizenship',\n",
       " 'city',\n",
       " 'civil',\n",
       " 'class',\n",
       " 'clear',\n",
       " 'clearance',\n",
       " 'clicking',\n",
       " 'client',\n",
       " 'clients',\n",
       " 'clinical',\n",
       " 'close',\n",
       " 'closely',\n",
       " 'cloud',\n",
       " 'co',\n",
       " 'code',\n",
       " 'coding',\n",
       " 'collaborate',\n",
       " 'collaboration',\n",
       " 'collaborative',\n",
       " 'collection',\n",
       " 'college',\n",
       " 'color',\n",
       " 'com',\n",
       " 'commercial',\n",
       " 'commitment',\n",
       " 'committed',\n",
       " 'communicate',\n",
       " 'communication',\n",
       " 'communications',\n",
       " 'communities',\n",
       " 'community',\n",
       " 'companies',\n",
       " 'company',\n",
       " 'compensation',\n",
       " 'competitive',\n",
       " 'complete',\n",
       " 'completed',\n",
       " 'completion',\n",
       " 'complex',\n",
       " 'compliance',\n",
       " 'components',\n",
       " 'comprehensive',\n",
       " 'computer',\n",
       " 'computing',\n",
       " 'con',\n",
       " 'concepts',\n",
       " 'conditions',\n",
       " 'conduct',\n",
       " 'conocimientos',\n",
       " 'consideration',\n",
       " 'considered',\n",
       " 'consistent',\n",
       " 'consulting',\n",
       " 'contact',\n",
       " 'content',\n",
       " 'continuous',\n",
       " 'contract',\n",
       " 'contribute',\n",
       " 'control',\n",
       " 'controls',\n",
       " 'coordinate',\n",
       " 'core',\n",
       " 'corporate',\n",
       " 'correo',\n",
       " 'cost',\n",
       " 'counsel',\n",
       " 'countries',\n",
       " 'country',\n",
       " 'coverage',\n",
       " 'create',\n",
       " 'creating',\n",
       " 'creation',\n",
       " 'creative',\n",
       " 'credit',\n",
       " 'creed',\n",
       " 'critical',\n",
       " 'cross',\n",
       " 'culture',\n",
       " 'current',\n",
       " 'currently',\n",
       " 'customer',\n",
       " 'customers',\n",
       " 'cv',\n",
       " 'cycle',\n",
       " 'daily',\n",
       " 'data',\n",
       " 'database',\n",
       " 'date',\n",
       " 'day',\n",
       " 'days',\n",
       " 'de',\n",
       " 'debugging',\n",
       " 'decision',\n",
       " 'decisions',\n",
       " 'dedicated',\n",
       " 'degree',\n",
       " 'del',\n",
       " 'deliver',\n",
       " 'delivering',\n",
       " 'delivery',\n",
       " 'demands',\n",
       " 'demonstrate',\n",
       " 'demonstrated',\n",
       " 'dental',\n",
       " 'department',\n",
       " 'departments',\n",
       " 'description',\n",
       " 'design',\n",
       " 'designing',\n",
       " 'designs',\n",
       " 'desirable',\n",
       " 'desired',\n",
       " 'detail',\n",
       " 'details',\n",
       " 'determine',\n",
       " 'develop',\n",
       " 'developers',\n",
       " 'developing',\n",
       " 'development',\n",
       " 'develops',\n",
       " 'device',\n",
       " 'devices',\n",
       " 'difference',\n",
       " 'different',\n",
       " 'diploma',\n",
       " 'direct',\n",
       " 'direction',\n",
       " 'directly',\n",
       " 'disabilities',\n",
       " 'disability',\n",
       " 'discipline',\n",
       " 'discriminate',\n",
       " 'diverse',\n",
       " 'diversity',\n",
       " 'division',\n",
       " 'do',\n",
       " 'document',\n",
       " 'documentation',\n",
       " 'documents',\n",
       " 'does',\n",
       " 'drive',\n",
       " 'driven',\n",
       " 'driving',\n",
       " 'drug',\n",
       " 'during',\n",
       " 'duties',\n",
       " 'dynamic',\n",
       " 'each',\n",
       " 'edge',\n",
       " 'education',\n",
       " 'eeo',\n",
       " 'effective',\n",
       " 'effectively',\n",
       " 'efficient',\n",
       " 'efforts',\n",
       " 'el',\n",
       " 'electrical',\n",
       " 'electronic',\n",
       " 'eligible',\n",
       " 'email',\n",
       " 'employee',\n",
       " 'employees',\n",
       " 'employer',\n",
       " 'employment',\n",
       " 'empresa',\n",
       " 'en',\n",
       " 'enable',\n",
       " 'end',\n",
       " 'engineer',\n",
       " 'engineering',\n",
       " 'engineers',\n",
       " 'english',\n",
       " 'ensure',\n",
       " 'ensures',\n",
       " 'ensuring',\n",
       " 'enterprise',\n",
       " 'entry',\n",
       " 'enviar',\n",
       " 'environment',\n",
       " 'environments',\n",
       " 'equal',\n",
       " 'equipment',\n",
       " 'equivalent',\n",
       " 'escolaridad',\n",
       " 'essential',\n",
       " 'established',\n",
       " 'etc',\n",
       " 'evaluate',\n",
       " 'evaluation',\n",
       " 'every',\n",
       " 'excel',\n",
       " 'excellence',\n",
       " 'excellent',\n",
       " 'exciting',\n",
       " 'execute',\n",
       " 'execution',\n",
       " 'existing',\n",
       " 'expected',\n",
       " 'experience',\n",
       " 'experienced',\n",
       " 'experiences',\n",
       " 'experiencia',\n",
       " 'expert',\n",
       " 'expertise',\n",
       " 'exposure',\n",
       " 'expression',\n",
       " 'extensive',\n",
       " 'external',\n",
       " 'facilities',\n",
       " 'facility',\n",
       " 'familiarity',\n",
       " 'fast',\n",
       " 'features',\n",
       " 'federal',\n",
       " 'feedback',\n",
       " 'field',\n",
       " 'finance',\n",
       " 'financial',\n",
       " 'find',\n",
       " 'firm',\n",
       " 'firms',\n",
       " 'first',\n",
       " 'flexible',\n",
       " 'focus',\n",
       " 'focused',\n",
       " 'follow',\n",
       " 'following',\n",
       " 'for',\n",
       " 'fortune',\n",
       " 'framework',\n",
       " 'frameworks',\n",
       " 'free',\n",
       " 'from',\n",
       " 'full',\n",
       " 'function',\n",
       " 'functional',\n",
       " 'functions',\n",
       " 'future',\n",
       " 'gender',\n",
       " 'general',\n",
       " 'generation',\n",
       " 'genetic',\n",
       " 'get',\n",
       " 'global',\n",
       " 'goals',\n",
       " 'good',\n",
       " 'government',\n",
       " 'graduate',\n",
       " 'great',\n",
       " 'group',\n",
       " 'groups',\n",
       " 'grow',\n",
       " 'growing',\n",
       " 'growth',\n",
       " 'grumman',\n",
       " 'guidance',\n",
       " 'guidelines',\n",
       " 'half',\n",
       " 'handle',\n",
       " 'handling',\n",
       " 'hands',\n",
       " 'hardware',\n",
       " 'has',\n",
       " 'have',\n",
       " 'health',\n",
       " 'healthcare',\n",
       " 'help',\n",
       " 'here',\n",
       " 'high',\n",
       " 'higher',\n",
       " 'highly',\n",
       " 'hire',\n",
       " 'hiring',\n",
       " 'home',\n",
       " 'hours',\n",
       " 'how',\n",
       " 'http',\n",
       " 'human',\n",
       " 'id',\n",
       " 'ideal',\n",
       " 'identify',\n",
       " 'identity',\n",
       " 'if',\n",
       " 'ii',\n",
       " 'immediate',\n",
       " 'impact',\n",
       " 'implement',\n",
       " 'implementation',\n",
       " 'implementing',\n",
       " 'improve',\n",
       " 'improvement',\n",
       " 'improvements',\n",
       " 'in',\n",
       " 'inc',\n",
       " 'include',\n",
       " 'includes',\n",
       " 'including',\n",
       " 'independent',\n",
       " 'independently',\n",
       " 'individual',\n",
       " 'individuals',\n",
       " 'industry',\n",
       " 'information',\n",
       " 'infrastructure',\n",
       " 'ingles',\n",
       " 'initiatives',\n",
       " 'innovation',\n",
       " 'innovative',\n",
       " 'instructions',\n",
       " 'insurance',\n",
       " 'integrated',\n",
       " 'integration',\n",
       " 'interact',\n",
       " 'interesados',\n",
       " 'interested',\n",
       " 'interface',\n",
       " 'internal',\n",
       " 'international',\n",
       " 'interpersonal',\n",
       " 'into',\n",
       " 'investigation',\n",
       " 'involved',\n",
       " 'is',\n",
       " 'issues',\n",
       " 'it',\n",
       " 'its',\n",
       " 'java',\n",
       " 'javascript',\n",
       " 'job',\n",
       " 'jobs',\n",
       " 'join',\n",
       " 'judgment',\n",
       " 'key',\n",
       " 'know',\n",
       " 'knowledge',\n",
       " 'la',\n",
       " 'lab',\n",
       " 'laboratory',\n",
       " 'language',\n",
       " 'languages',\n",
       " 'large',\n",
       " 'largest',\n",
       " 'las',\n",
       " 'law',\n",
       " 'laws',\n",
       " 'lead',\n",
       " 'leader',\n",
       " 'leaders',\n",
       " 'leadership',\n",
       " 'leading',\n",
       " 'learn',\n",
       " 'learning',\n",
       " 'least',\n",
       " 'legal',\n",
       " 'level',\n",
       " 'levels',\n",
       " 'ley',\n",
       " 'license',\n",
       " 'life',\n",
       " 'like',\n",
       " 'limited',\n",
       " 'line',\n",
       " 'linux',\n",
       " 'list',\n",
       " 'litigation',\n",
       " 'lives',\n",
       " 'local',\n",
       " 'located',\n",
       " 'location',\n",
       " 'locations',\n",
       " 'long',\n",
       " 'looking',\n",
       " 'los',\n",
       " 'made',\n",
       " 'magazine',\n",
       " 'maintain',\n",
       " 'maintaining',\n",
       " 'maintains',\n",
       " 'maintenance',\n",
       " 'major',\n",
       " 'make',\n",
       " 'making',\n",
       " 'manage',\n",
       " 'management',\n",
       " 'manager',\n",
       " 'managers',\n",
       " 'managing',\n",
       " 'manner',\n",
       " 'manual',\n",
       " 'manufacturing',\n",
       " 'many',\n",
       " 'march',\n",
       " 'marital',\n",
       " 'market',\n",
       " 'marketing',\n",
       " 'markets',\n",
       " 'master',\n",
       " 'materials',\n",
       " 'matters',\n",
       " 'may',\n",
       " 'medical',\n",
       " 'meet',\n",
       " 'meetings',\n",
       " 'member',\n",
       " 'members',\n",
       " 'met',\n",
       " 'methodologies',\n",
       " 'methods',\n",
       " 'microsoft',\n",
       " 'military',\n",
       " 'minimum',\n",
       " 'mission',\n",
       " 'mobile',\n",
       " 'monitor',\n",
       " 'monitoring',\n",
       " 'months',\n",
       " 'more',\n",
       " 'most',\n",
       " 'motivated',\n",
       " 'ms',\n",
       " 'multi',\n",
       " 'multiple',\n",
       " 'must',\n",
       " 'named',\n",
       " 'national',\n",
       " 'necessary',\n",
       " 'need',\n",
       " 'needed',\n",
       " 'needs',\n",
       " 'network',\n",
       " 'new',\n",
       " 'next',\n",
       " 'no',\n",
       " 'non',\n",
       " 'north',\n",
       " 'northrop',\n",
       " 'northropgrumman',\n",
       " 'not',\n",
       " 'now',\n",
       " 'number',\n",
       " 'object',\n",
       " 'objectives',\n",
       " 'obtain',\n",
       " 'of',\n",
       " 'off',\n",
       " 'offer',\n",
       " 'offers',\n",
       " 'office',\n",
       " 'ofrecemos',\n",
       " 'on',\n",
       " 'once',\n",
       " 'one',\n",
       " 'online',\n",
       " 'only',\n",
       " 'open',\n",
       " 'openings',\n",
       " 'operate',\n",
       " 'operating',\n",
       " 'operational',\n",
       " 'operations',\n",
       " 'opportunities',\n",
       " 'opportunity',\n",
       " 'or',\n",
       " 'oral',\n",
       " 'order',\n",
       " 'organization',\n",
       " 'organizational',\n",
       " 'organizations',\n",
       " 'orientation',\n",
       " 'oriented',\n",
       " 'origin',\n",
       " 'other',\n",
       " 'others',\n",
       " 'our',\n",
       " 'out',\n",
       " 'outside',\n",
       " 'over',\n",
       " 'overall',\n",
       " 'overview',\n",
       " 'own',\n",
       " 'paced',\n",
       " 'package',\n",
       " 'paid',\n",
       " 'para',\n",
       " 'part',\n",
       " 'participate',\n",
       " 'participates',\n",
       " 'partner',\n",
       " 'partners',\n",
       " 'passion',\n",
       " 'passionate',\n",
       " 'patient',\n",
       " 'patients',\n",
       " 'pay',\n",
       " 'people',\n",
       " 'per',\n",
       " 'perform',\n",
       " 'performance',\n",
       " 'performed',\n",
       " 'performing',\n",
       " 'performs',\n",
       " 'person',\n",
       " 'personal',\n",
       " 'personnel',\n",
       " 'physical',\n",
       " 'place',\n",
       " 'plan',\n",
       " 'planning',\n",
       " 'plans',\n",
       " 'platform',\n",
       " 'platforms',\n",
       " 'please',\n",
       " 'plus',\n",
       " 'policies',\n",
       " 'policy',\n",
       " 'por',\n",
       " 'portfolio',\n",
       " 'position',\n",
       " 'positions',\n",
       " 'positive',\n",
       " 'possess',\n",
       " 'post',\n",
       " 'postal',\n",
       " 'posting',\n",
       " 'potential',\n",
       " 'power',\n",
       " 'practice',\n",
       " 'practices',\n",
       " 'pre',\n",
       " 'preferred',\n",
       " 'premier',\n",
       " 'preparation',\n",
       " 'prepare',\n",
       " 'presentation',\n",
       " 'prestaciones',\n",
       " 'previous',\n",
       " 'primary',\n",
       " 'principles',\n",
       " 'prior',\n",
       " 'priorities',\n",
       " 'prioritize',\n",
       " 'problem',\n",
       " 'problems',\n",
       " 'procedures',\n",
       " 'process',\n",
       " 'processes',\n",
       " 'processing',\n",
       " 'product',\n",
       " 'production',\n",
       " 'products',\n",
       " 'professional',\n",
       " 'professionals',\n",
       " 'proficiency',\n",
       " 'proficient',\n",
       " 'profile',\n",
       " 'program',\n",
       " 'programming',\n",
       " 'programs',\n",
       " 'project',\n",
       " 'projects',\n",
       " 'protected',\n",
       " 'protocols',\n",
       " 'proud',\n",
       " 'proven',\n",
       " 'provide',\n",
       " 'provider',\n",
       " 'provides',\n",
       " 'providing',\n",
       " 'pruebas',\n",
       " 'public',\n",
       " 'python',\n",
       " 'qa',\n",
       " 'qualifications',\n",
       " 'qualified',\n",
       " 'quality',\n",
       " 'que',\n",
       " 'questions',\n",
       " 'quickly',\n",
       " 'race',\n",
       " 'range',\n",
       " 're',\n",
       " 'real',\n",
       " 'reasonable',\n",
       " 'receive',\n",
       " 'recommendations',\n",
       " 'record',\n",
       " 'records',\n",
       " 'reference',\n",
       " 'regard',\n",
       " 'regarding',\n",
       " 'regular',\n",
       " 'regulations',\n",
       " 'regulatory',\n",
       " 'related',\n",
       " 'relationships',\n",
       " 'release',\n",
       " 'relevant',\n",
       " 'religion',\n",
       " 'report',\n",
       " 'reporting',\n",
       " 'reports',\n",
       " 'req',\n",
       " 'requests',\n",
       " 'require',\n",
       " 'required',\n",
       " 'requirements',\n",
       " 'requires',\n",
       " 'requisition',\n",
       " 'requisitos',\n",
       " 'research',\n",
       " 'resolution',\n",
       " 'resolve',\n",
       " 'resources',\n",
       " 'responsibilities',\n",
       " 'responsibility',\n",
       " 'responsible',\n",
       " 'results',\n",
       " 'resume',\n",
       " 'retaining',\n",
       " 'review',\n",
       " 'reviews',\n",
       " 'right',\n",
       " 'risk',\n",
       " 'risks',\n",
       " 'robert',\n",
       " 'role',\n",
       " 'roles',\n",
       " 'safety',\n",
       " 'salary',\n",
       " 'sales',\n",
       " 'scale',\n",
       " 'schedule',\n",
       " 'school',\n",
       " 'science',\n",
       " 'scope',\n",
       " 'scripting',\n",
       " 'scripts',\n",
       " 'search',\n",
       " 'security',\n",
       " 'seeking',\n",
       " 'self',\n",
       " 'senior',\n",
       " 'serve',\n",
       " 'server',\n",
       " 'service',\n",
       " 'services',\n",
       " 'set',\n",
       " 'setting',\n",
       " 'sex',\n",
       " 'sexual',\n",
       " 'share',\n",
       " 'shift',\n",
       " 'should',\n",
       " 'significant',\n",
       " 'similar',\n",
       " 'sistemas',\n",
       " 'site',\n",
       " 'skill',\n",
       " 'skilled',\n",
       " 'skills',\n",
       " 'small',\n",
       " 'so',\n",
       " 'software',\n",
       " 'solicita',\n",
       " 'solid',\n",
       " 'solution',\n",
       " 'solutions',\n",
       " 'solve',\n",
       " 'solving',\n",
       " 'some',\n",
       " 'source',\n",
       " 'special',\n",
       " 'specialist',\n",
       " 'specialty',\n",
       " 'specific',\n",
       " 'specifications',\n",
       " 'sql',\n",
       " 'staff',\n",
       " 'staffing',\n",
       " 'standard',\n",
       " 'standards',\n",
       " 'state',\n",
       " 'statement',\n",
       " 'states',\n",
       " 'status',\n",
       " 'strategic',\n",
       " 'strategies',\n",
       " 'strategy',\n",
       " 'strong',\n",
       " 'subject',\n",
       " 'submit',\n",
       " 'success',\n",
       " 'successful',\n",
       " 'successfully',\n",
       " 'such',\n",
       " 'sueldo',\n",
       " 'suite',\n",
       " 'summary',\n",
       " 'supervision',\n",
       " 'support',\n",
       " 'supporting',\n",
       " 'supports',\n",
       " 'system',\n",
       " 'systems',\n",
       " 'take',\n",
       " 'talent',\n",
       " 'talented',\n",
       " 'task',\n",
       " 'tasks',\n",
       " 'team',\n",
       " 'teams',\n",
       " 'technical',\n",
       " 'techniques',\n",
       " 'technologies',\n",
       " 'technology',\n",
       " 'temporary',\n",
       " 'terms',\n",
       " 'test',\n",
       " 'tester',\n",
       " 'testing',\n",
       " 'tests',\n",
       " 'than',\n",
       " 'that',\n",
       " 'the',\n",
       " 'their',\n",
       " 'them',\n",
       " 'then',\n",
       " 'these',\n",
       " 'they',\n",
       " 'this',\n",
       " 'those',\n",
       " 'three',\n",
       " 'through',\n",
       " 'throughout',\n",
       " 'time',\n",
       " 'timely',\n",
       " 'title',\n",
       " 'to',\n",
       " 'today',\n",
       " 'together',\n",
       " 'tools',\n",
       " 'top',\n",
       " 'trabajo',\n",
       " 'track',\n",
       " 'tracking',\n",
       " 'training',\n",
       " 'travel',\n",
       " 'troubleshooting',\n",
       " 'two',\n",
       " 'type',\n",
       " 'under',\n",
       " 'understand',\n",
       " 'understanding',\n",
       " 'unique',\n",
       " 'unit',\n",
       " 'united',\n",
       " 'university',\n",
       " 'up',\n",
       " 'upon',\n",
       " 'us',\n",
       " 'usa',\n",
       " 'use',\n",
       " 'used',\n",
       " 'user',\n",
       " 'users',\n",
       " 'using',\n",
       " 'validation',\n",
       " 'value',\n",
       " 'variety',\n",
       " 'various',\n",
       " 'verbal',\n",
       " 'verification',\n",
       " 'verify',\n",
       " 'very',\n",
       " 'vet',\n",
       " 'veteran',\n",
       " 'veterans',\n",
       " 'vision',\n",
       " 'visit',\n",
       " 'want',\n",
       " 'was',\n",
       " 'way',\n",
       " 'we',\n",
       " 'web',\n",
       " 'week',\n",
       " 'well',\n",
       " 'were',\n",
       " 'what',\n",
       " 'when',\n",
       " 'where',\n",
       " 'which',\n",
       " 'while',\n",
       " 'who',\n",
       " 'wide',\n",
       " 'will',\n",
       " 'windows',\n",
       " 'with',\n",
       " 'within',\n",
       " 'without',\n",
       " 'word',\n",
       " 'work',\n",
       " 'workforce',\n",
       " 'working',\n",
       " 'workplace',\n",
       " 'works',\n",
       " 'world',\n",
       " 'worldwide',\n",
       " 'would',\n",
       " 'write',\n",
       " 'writing',\n",
       " 'written',\n",
       " 'www',\n",
       " 'year',\n",
       " 'years',\n",
       " 'you',\n",
       " 'your',\n",
       " 'zona']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first aspect of the feature list that should stand out for us is that the first few entries are numbers that have no real semantic meaning. There are also other useless words such as prepositions and articles that also have no semantic meaning. The words *ability* or *abilities* or *accuracy* and *accurate* are also quite similiar and mean the same thing. We should try cleaning our corpus of data of these types of words as they just add noise to our analysis. Nevertheless let's try creating topics and seeing the quality of the results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('and', 72036),\n",
       "             ('to', 36758),\n",
       "             ('the', 34940),\n",
       "             ('of', 31793),\n",
       "             ('in', 23879),\n",
       "             ('with', 16877),\n",
       "             ('for', 15605),\n",
       "             ('or', 13556),\n",
       "             ('experience', 11108),\n",
       "             ('is', 10153),\n",
       "             ('de', 8563),\n",
       "             ('as', 8252),\n",
       "             ('be', 7101),\n",
       "             ('on', 7003),\n",
       "             ('work', 6583),\n",
       "             ('our', 6556),\n",
       "             ('will', 5949),\n",
       "             ('are', 5769),\n",
       "             ('an', 5686),\n",
       "             ('job', 5631),\n",
       "             ('software', 5360),\n",
       "             ('skills', 5337),\n",
       "             ('we', 5198),\n",
       "             ('development', 5079),\n",
       "             ('you', 4884),\n",
       "             ('test', 4744),\n",
       "             ('this', 4584),\n",
       "             ('team', 4450),\n",
       "             ('all', 4442),\n",
       "             ('that', 4267),\n",
       "             ('ability', 4248),\n",
       "             ('other', 4207),\n",
       "             ('required', 4147),\n",
       "             ('business', 4010),\n",
       "             ('at', 3679),\n",
       "             ('quality', 3664),\n",
       "             ('requirements', 3652),\n",
       "             ('position', 3494),\n",
       "             ('by', 3452),\n",
       "             ('testing', 3413),\n",
       "             ('knowledge', 3366),\n",
       "             ('years', 3330),\n",
       "             ('management', 3320),\n",
       "             ('must', 3261),\n",
       "             ('systems', 3103),\n",
       "             ('engineering', 3047),\n",
       "             ('services', 3043),\n",
       "             ('information', 2989),\n",
       "             ('support', 2955),\n",
       "             ('your', 2837),\n",
       "             ('legal', 2816),\n",
       "             ('have', 2802),\n",
       "             ('en', 2790),\n",
       "             ('status', 2757),\n",
       "             ('opportunity', 2734),\n",
       "             ('including', 2626),\n",
       "             ('working', 2559),\n",
       "             ('design', 2531),\n",
       "             ('company', 2528),\n",
       "             ('technical', 2487),\n",
       "             ('data', 2359),\n",
       "             ('more', 2355),\n",
       "             ('provide', 2307),\n",
       "             ('time', 2297),\n",
       "             ('degree', 2276),\n",
       "             ('related', 2264),\n",
       "             ('new', 2242),\n",
       "             ('strong', 2206),\n",
       "             ('from', 2152),\n",
       "             ('environment', 2105),\n",
       "             ('qualifications', 2103),\n",
       "             ('system', 2071),\n",
       "             ('compliance', 2019),\n",
       "             ('equal', 1993),\n",
       "             ('preferred', 1925),\n",
       "             ('location', 1899),\n",
       "             ('training', 1839),\n",
       "             ('employer', 1799),\n",
       "             ('product', 1795),\n",
       "             ('medical', 1786),\n",
       "             ('us', 1772),\n",
       "             ('responsibilities', 1770),\n",
       "             ('employment', 1732),\n",
       "             ('process', 1726),\n",
       "             ('program', 1724),\n",
       "             ('any', 1703),\n",
       "             ('law', 1685),\n",
       "             ('disability', 1679),\n",
       "             ('service', 1667),\n",
       "             ('solutions', 1664),\n",
       "             ('customer', 1661),\n",
       "             ('communication', 1659),\n",
       "             ('procedures', 1652),\n",
       "             ('engineer', 1641),\n",
       "             ('project', 1622),\n",
       "             ('computer', 1608),\n",
       "             ('not', 1601),\n",
       "             ('products', 1598),\n",
       "             ('it', 1589),\n",
       "             ('apply', 1579),\n",
       "             ('analysis', 1576),\n",
       "             ('technology', 1562),\n",
       "             ('national', 1556),\n",
       "             ('la', 1545),\n",
       "             ('high', 1536),\n",
       "             ('office', 1533),\n",
       "             ('may', 1523),\n",
       "             ('level', 1521),\n",
       "             ('include', 1513),\n",
       "             ('applicants', 1495),\n",
       "             ('within', 1492),\n",
       "             ('tools', 1484),\n",
       "             ('state', 1474),\n",
       "             ('based', 1472),\n",
       "             ('develop', 1448),\n",
       "             ('performance', 1417),\n",
       "             ('ensure', 1387),\n",
       "             ('health', 1383),\n",
       "             ('responsible', 1373),\n",
       "             ('world', 1370),\n",
       "             ('laboratory', 1370),\n",
       "             ('minimum', 1362),\n",
       "             ('full', 1361),\n",
       "             ('customers', 1358),\n",
       "             ('through', 1354),\n",
       "             ('protected', 1344),\n",
       "             ('duties', 1344),\n",
       "             ('excellent', 1337),\n",
       "             ('microsoft', 1336),\n",
       "             ('their', 1336),\n",
       "             ('description', 1322),\n",
       "             ('education', 1320),\n",
       "             ('id', 1318),\n",
       "             ('department', 1305),\n",
       "             ('gender', 1303),\n",
       "             ('issues', 1291),\n",
       "             ('industry', 1290),\n",
       "             ('well', 1286),\n",
       "             ('application', 1284),\n",
       "             ('global', 1283),\n",
       "             ('part', 1270),\n",
       "             ('one', 1263),\n",
       "             ('perform', 1260),\n",
       "             ('security', 1258),\n",
       "             ('role', 1234),\n",
       "             ('applications', 1233),\n",
       "             ('able', 1233),\n",
       "             ('processes', 1231),\n",
       "             ('review', 1220),\n",
       "             ('veteran', 1220),\n",
       "             ('teams', 1213),\n",
       "             ('title', 1212),\n",
       "             ('about', 1210),\n",
       "             ('activities', 1204),\n",
       "             ('race', 1185),\n",
       "             ('care', 1176),\n",
       "             ('professional', 1174),\n",
       "             ('research', 1169),\n",
       "             ('origin', 1168),\n",
       "             ('use', 1168),\n",
       "             ('religion', 1166),\n",
       "             ('candidate', 1162),\n",
       "             ('please', 1162),\n",
       "             ('color', 1158),\n",
       "             ('projects', 1149),\n",
       "             ('organization', 1146),\n",
       "             ('if', 1137),\n",
       "             ('science', 1130),\n",
       "             ('written', 1129),\n",
       "             ('understanding', 1120),\n",
       "             ('such', 1105),\n",
       "             ('employees', 1104),\n",
       "             ('opportunities', 1097),\n",
       "             ('maintain', 1088),\n",
       "             ('orientation', 1073),\n",
       "             ('people', 1070),\n",
       "             ('following', 1064),\n",
       "             ('age', 1063),\n",
       "             ('sexual', 1056),\n",
       "             ('standards', 1053),\n",
       "             ('has', 1047),\n",
       "             ('control', 1038),\n",
       "             ('without', 1038),\n",
       "             ('also', 1032),\n",
       "             ('who', 1031),\n",
       "             ('financial', 1028),\n",
       "             ('clients', 1025),\n",
       "             ('risk', 1022),\n",
       "             ('qualified', 1018),\n",
       "             ('benefits', 1010),\n",
       "             ('code', 1008),\n",
       "             ('than', 996),\n",
       "             ('leading', 994),\n",
       "             ('hours', 992),\n",
       "             ('functional', 990),\n",
       "             ('career', 986),\n",
       "             ('complex', 977),\n",
       "             ('identity', 976),\n",
       "             ('web', 976),\n",
       "             ('most', 970),\n",
       "             ('good', 964),\n",
       "             ('its', 964),\n",
       "             ('results', 964),\n",
       "             ('assigned', 961),\n",
       "             ('com', 959),\n",
       "             ('reports', 954),\n",
       "             ('con', 952),\n",
       "             ('documentation', 949),\n",
       "             ('basic', 948),\n",
       "             ('both', 947),\n",
       "             ('etc', 946),\n",
       "             ('life', 945),\n",
       "             ('using', 943),\n",
       "             ('technologies', 940),\n",
       "             ('action', 939),\n",
       "             ('regulatory', 937),\n",
       "             ('integration', 934),\n",
       "             ('providing', 925),\n",
       "             ('credit', 923),\n",
       "             ('join', 918),\n",
       "             ('internal', 915),\n",
       "             ('appropriate', 913),\n",
       "             ('plans', 907),\n",
       "             ('field', 906),\n",
       "             ('areas', 903),\n",
       "             ('candidates', 901),\n",
       "             ('insurance', 898),\n",
       "             ('regard', 898),\n",
       "             ('000', 898),\n",
       "             ('client', 897),\n",
       "             ('can', 890),\n",
       "             ('sales', 881),\n",
       "             ('looking', 879),\n",
       "             ('best', 879),\n",
       "             ('el', 876),\n",
       "             ('general', 872),\n",
       "             ('plus', 872),\n",
       "             ('sex', 868),\n",
       "             ('group', 865),\n",
       "             ('clinical', 864),\n",
       "             ('professionals', 861),\n",
       "             ('operations', 859),\n",
       "             ('experiencia', 857),\n",
       "             ('area', 851),\n",
       "             ('senior', 845),\n",
       "             ('make', 845),\n",
       "             ('2015', 843),\n",
       "             ('across', 842),\n",
       "             ('which', 836),\n",
       "             ('developing', 834),\n",
       "             ('corporate', 834),\n",
       "             ('equivalent', 826),\n",
       "             ('needs', 815),\n",
       "             ('assurance', 811),\n",
       "             ('plan', 802),\n",
       "             ('highly', 797),\n",
       "             ('patient', 789),\n",
       "             ('up', 789),\n",
       "             ('functions', 786),\n",
       "             ('but', 785),\n",
       "             ('healthcare', 785),\n",
       "             ('equipment', 782),\n",
       "             ('companies', 777),\n",
       "             ('lab', 768),\n",
       "             ('assist', 762),\n",
       "             ('problems', 761),\n",
       "             ('compensation', 758),\n",
       "             ('year', 756),\n",
       "             ('java', 756),\n",
       "             ('states', 754),\n",
       "             ('assistant', 750),\n",
       "             ('practices', 745),\n",
       "             ('complete', 739),\n",
       "             ('federal', 738),\n",
       "             ('robert', 737),\n",
       "             ('half', 736),\n",
       "             ('basis', 731),\n",
       "             ('manager', 730),\n",
       "             ('staff', 729),\n",
       "             ('provides', 726),\n",
       "             ('policies', 726),\n",
       "             ('programs', 723),\n",
       "             ('employee', 722),\n",
       "             ('effectively', 722),\n",
       "             ('center', 721),\n",
       "             ('do', 720),\n",
       "             ('documents', 708),\n",
       "             ('no', 704),\n",
       "             ('requisition', 704),\n",
       "             ('reviews', 703),\n",
       "             ('multiple', 703),\n",
       "             ('essential', 702),\n",
       "             ('help', 702),\n",
       "             ('lead', 689),\n",
       "             ('meet', 689),\n",
       "             ('under', 687),\n",
       "             ('programming', 685),\n",
       "             ('various', 681),\n",
       "             ('united', 677),\n",
       "             ('personal', 674),\n",
       "             ('create', 674),\n",
       "             ('diverse', 673),\n",
       "             ('implementation', 667),\n",
       "             ('specific', 666),\n",
       "             ('10', 665),\n",
       "             ('problem', 663),\n",
       "             ('manage', 663),\n",
       "             ('tests', 660),\n",
       "             ('eeo', 660),\n",
       "             ('planning', 660),\n",
       "             ('key', 659),\n",
       "             ('commercial', 659),\n",
       "             ('into', 656),\n",
       "             ('counsel', 654),\n",
       "             ('leadership', 654),\n",
       "             ('resources', 654),\n",
       "             ('contact', 651),\n",
       "             ('individuals', 651),\n",
       "             ('affirmative', 650),\n",
       "             ('analyst', 645),\n",
       "             ('learn', 645),\n",
       "             ('demonstrated', 643),\n",
       "             ('patients', 643),\n",
       "             ('automation', 642),\n",
       "             ('analytical', 641),\n",
       "             ('additional', 641),\n",
       "             ('verbal', 639),\n",
       "             ('consideration', 639),\n",
       "             ('local', 638),\n",
       "             ('online', 637),\n",
       "             ('competitive', 636),\n",
       "             ('individual', 633),\n",
       "             ('marital', 631),\n",
       "             ('offer', 631),\n",
       "             ('build', 630),\n",
       "             ('00', 626),\n",
       "             ('type', 626),\n",
       "             ('partner', 623),\n",
       "             ('range', 623),\n",
       "             ('school', 622),\n",
       "             ('regulations', 622),\n",
       "             ('current', 620),\n",
       "             ('qa', 620),\n",
       "             ('matters', 619),\n",
       "             ('travel', 619),\n",
       "             ('committed', 615),\n",
       "             ('bachelor', 614),\n",
       "             ('communicate', 610),\n",
       "             ('tasks', 607),\n",
       "             ('independently', 606),\n",
       "             ('engineers', 603),\n",
       "             ('relevant', 598),\n",
       "             ('cloud', 597),\n",
       "             ('access', 592),\n",
       "             ('identify', 586),\n",
       "             ('should', 586),\n",
       "             ('pruebas', 585),\n",
       "             ('members', 585),\n",
       "             ('day', 585),\n",
       "             ('para', 584),\n",
       "             ('performs', 581),\n",
       "             ('needed', 581),\n",
       "             ('now', 579),\n",
       "             ('end', 579),\n",
       "             ('participate', 578),\n",
       "             ('automated', 578),\n",
       "             ('proven', 576),\n",
       "             ('per', 575),\n",
       "             ('understand', 573),\n",
       "             ('production', 572),\n",
       "             ('self', 570),\n",
       "             ('processing', 570),\n",
       "             ('network', 570),\n",
       "             ('mobile', 569),\n",
       "             ('document', 568),\n",
       "             ('receive', 565),\n",
       "             ('non', 564),\n",
       "             ('cases', 564),\n",
       "             ('policy', 562),\n",
       "             ('applicable', 561),\n",
       "             ('maintenance', 561),\n",
       "             ('oriented', 561),\n",
       "             ('while', 560),\n",
       "             ('safety', 560),\n",
       "             ('writing', 559),\n",
       "             ('successful', 557),\n",
       "             ('class', 557),\n",
       "             ('solving', 556),\n",
       "             ('necessary', 555),\n",
       "             ('when', 555),\n",
       "             ('improve', 555),\n",
       "             ('applying', 552),\n",
       "             ('partners', 552),\n",
       "             ('external', 549),\n",
       "             ('they', 547),\n",
       "             ('word', 546),\n",
       "             ('schedule', 545),\n",
       "             ('building', 545),\n",
       "             ('others', 544),\n",
       "             ('advanced', 544),\n",
       "             ('certification', 541),\n",
       "             ('expertise', 541),\n",
       "             ('sql', 541),\n",
       "             ('drive', 539),\n",
       "             ('mission', 539),\n",
       "             ('deliver', 537),\n",
       "             ('background', 536),\n",
       "             ('delivery', 535),\n",
       "             ('date', 533),\n",
       "             ('agile', 533),\n",
       "             ('city', 531),\n",
       "             ('user', 527),\n",
       "             ('desired', 525),\n",
       "             ('order', 524),\n",
       "             ('requires', 523),\n",
       "             ('reporting', 523),\n",
       "             ('seeking', 523),\n",
       "             ('excel', 523),\n",
       "             ('trabajo', 523),\n",
       "             ('international', 520),\n",
       "             ('where', 519),\n",
       "             ('performing', 513),\n",
       "             ('some', 513),\n",
       "             ('los', 512),\n",
       "             ('summary', 510),\n",
       "             ('physical', 507),\n",
       "             ('making', 507),\n",
       "             ('positions', 507),\n",
       "             ('ms', 506),\n",
       "             ('que', 505),\n",
       "             ('salary', 504),\n",
       "             ('attorney', 503),\n",
       "             ('every', 502),\n",
       "             ('clearance', 501),\n",
       "             ('unit', 501),\n",
       "             ('government', 500),\n",
       "             ('finance', 499),\n",
       "             ('bachelors', 498),\n",
       "             ('category', 498),\n",
       "             ('strategy', 497),\n",
       "             ('openings', 497),\n",
       "             ('leader', 496),\n",
       "             ('electrical', 495),\n",
       "             ('execution', 493),\n",
       "             ('genetic', 492),\n",
       "             ('primary', 490),\n",
       "             ('citizenship', 489),\n",
       "             ('marketing', 489),\n",
       "             ('las', 488),\n",
       "             ('effective', 486),\n",
       "             ('por', 485),\n",
       "             ('available', 484),\n",
       "             ('need', 484),\n",
       "             ('supporting', 482),\n",
       "             ('these', 482),\n",
       "             ('locations', 482),\n",
       "             ('multi', 481),\n",
       "             ('large', 481),\n",
       "             ('major', 480),\n",
       "             ('here', 479),\n",
       "             ('innovative', 478),\n",
       "             ('personnel', 478),\n",
       "             ('techniques', 478),\n",
       "             ('authorized', 476),\n",
       "             ('fast', 476),\n",
       "             ('conduct', 472),\n",
       "             ('windows', 472),\n",
       "             ('relationships', 470),\n",
       "             ('least', 469),\n",
       "             ('variety', 467),\n",
       "             ('hardware', 466),\n",
       "             ('anos', 466),\n",
       "             ('what', 464),\n",
       "             ('accounting', 463),\n",
       "             ('member', 463),\n",
       "             ('operating', 462),\n",
       "             ('firm', 460),\n",
       "             ('worldwide', 459),\n",
       "             ('works', 458),\n",
       "             ('resume', 454),\n",
       "             ('www', 453),\n",
       "             ('enterprise', 452),\n",
       "             ('decisions', 451),\n",
       "             ('each', 449),\n",
       "             ('country', 449),\n",
       "             ('includes', 449),\n",
       "             ('english', 449),\n",
       "             ('linux', 449),\n",
       "             ('detail', 448),\n",
       "             ('over', 447),\n",
       "             ('been', 446),\n",
       "             ('hands', 445),\n",
       "             ('al', 445),\n",
       "             ('wide', 445),\n",
       "             ('college', 445),\n",
       "             ('them', 443),\n",
       "             ('change', 441),\n",
       "             ('experienced', 439),\n",
       "             ('like', 437),\n",
       "             ('hiring', 436),\n",
       "             ('12', 433),\n",
       "             ('disabilities', 433),\n",
       "             ('initiatives', 432),\n",
       "             ('prior', 432),\n",
       "             ('practice', 430),\n",
       "             ('northrop', 429),\n",
       "             ('community', 429),\n",
       "             ('litigation', 428),\n",
       "             ('number', 427),\n",
       "             ('focus', 427),\n",
       "             ('staffing', 426),\n",
       "             ('familiarity', 426),\n",
       "             ('req', 424),\n",
       "             ('grumman', 424),\n",
       "             ('vision', 422),\n",
       "             ('statement', 421),\n",
       "             ('site', 421),\n",
       "             ('operational', 419),\n",
       "             ('out', 419),\n",
       "             ('del', 418),\n",
       "             ('completion', 418),\n",
       "             ('implement', 418),\n",
       "             ('success', 417),\n",
       "             ('organizational', 417),\n",
       "             ('shift', 416),\n",
       "             ('function', 416),\n",
       "             ('impact', 416),\n",
       "             ('open', 415),\n",
       "             ('experiences', 415),\n",
       "             ('located', 414),\n",
       "             ('creed', 413),\n",
       "             ('timely', 410),\n",
       "             ('standard', 410),\n",
       "             ('manufacturing', 410),\n",
       "             ('oral', 406),\n",
       "             ('15', 406),\n",
       "             ('how', 404),\n",
       "             ('take', 404),\n",
       "             ('communications', 404),\n",
       "             ('market', 404),\n",
       "             ('execute', 404),\n",
       "             ('validation', 403),\n",
       "             ('provider', 403),\n",
       "             ('levels', 403),\n",
       "             ('interpersonal', 402),\n",
       "             ('assistance', 401),\n",
       "             ('fortune', 401),\n",
       "             ('top', 399),\n",
       "             ('record', 399),\n",
       "             ('report', 399),\n",
       "             ('continuous', 398),\n",
       "             ('comprehensive', 398),\n",
       "             ('expression', 398),\n",
       "             ('list', 397),\n",
       "             ('around', 397),\n",
       "             ('countries', 397),\n",
       "             ('prepare', 395),\n",
       "             ('analyze', 395),\n",
       "             ('case', 393),\n",
       "             ('cross', 391),\n",
       "             ('closely', 390),\n",
       "             ('used', 390),\n",
       "             ('proud', 390),\n",
       "             ('right', 389),\n",
       "             ('platform', 389),\n",
       "             ('contract', 387),\n",
       "             ('culture', 386),\n",
       "             ('methodologies', 385),\n",
       "             ('interested', 384),\n",
       "             ('throughout', 384),\n",
       "             ('existing', 384),\n",
       "             ('materials', 383),\n",
       "             ('being', 382),\n",
       "             ('considered', 381),\n",
       "             ('package', 380),\n",
       "             ('abilities', 379),\n",
       "             ('goals', 377),\n",
       "             ('architecture', 376),\n",
       "             ('creating', 375),\n",
       "             ('power', 375),\n",
       "             ('email', 375),\n",
       "             ('follow', 374),\n",
       "             ('place', 373),\n",
       "             ('guidance', 372),\n",
       "             ('specifications', 372),\n",
       "             ('value', 371),\n",
       "             ('was', 370),\n",
       "             ('develops', 370),\n",
       "             ('free', 370),\n",
       "             ('tester', 369),\n",
       "             ('limited', 367),\n",
       "             ('critical', 367),\n",
       "             ('database', 367),\n",
       "             ('commitment', 367),\n",
       "             ('established', 365),\n",
       "             ('manner', 365),\n",
       "             ('during', 364),\n",
       "             ('requests', 362),\n",
       "             ('obtain', 362),\n",
       "             ('capabilities', 362),\n",
       "             ('innovation', 362),\n",
       "             ('details', 361),\n",
       "             ('markets', 360),\n",
       "             ('100', 360),\n",
       "             ('empresa', 359),\n",
       "             ('future', 359),\n",
       "             ('only', 358),\n",
       "             ('would', 357),\n",
       "             ('possess', 357),\n",
       "             ('set', 356),\n",
       "             ('directly', 355),\n",
       "             ('laws', 354),\n",
       "             ('first', 353),\n",
       "             ('supervision', 353),\n",
       "             ('methods', 353),\n",
       "             ('veterans', 353),\n",
       "             ('driven', 352),\n",
       "             ('supports', 352),\n",
       "             ('together', 352),\n",
       "             ('organizations', 352),\n",
       "             ('act', 351),\n",
       "             ('workforce', 351),\n",
       "             ('great', 351),\n",
       "             ('human', 350),\n",
       "             ('managers', 349),\n",
       "             ('posting', 348),\n",
       "             ('growth', 348),\n",
       "             ('overview', 348),\n",
       "             ('managing', 347),\n",
       "             ('direct', 346),\n",
       "             ('strategic', 345),\n",
       "             ('administration', 344),\n",
       "             ('server', 343),\n",
       "             ('active', 343),\n",
       "             ('maintaining', 341),\n",
       "             ('paced', 340),\n",
       "             ('responsibility', 339),\n",
       "             ('line', 338),\n",
       "             ('collection', 338),\n",
       "             ('eligible', 336),\n",
       "             ('so', 335),\n",
       "             ('two', 335),\n",
       "             ('ideal', 334),\n",
       "             ('public', 334),\n",
       "             ('daily', 333),\n",
       "             ('devices', 332),\n",
       "             ('improvement', 331),\n",
       "             ('currently', 330),\n",
       "             ('license', 330),\n",
       "             ('strategies', 328),\n",
       "             ('addition', 328),\n",
       "             ('quickly', 327),\n",
       "             ('controls', 327),\n",
       "             ('concepts', 327),\n",
       "             ('cv', 325),\n",
       "             ('source', 325),\n",
       "             ('challenging', 324),\n",
       "             ('departments', 324),\n",
       "             ('collaborate', 323),\n",
       "             ('learning', 323),\n",
       "             ('inc', 323),\n",
       "             ('infrastructure', 322),\n",
       "             ('drug', 321),\n",
       "             ('co', 321),\n",
       "             ('assessment', 320),\n",
       "             ('regular', 320),\n",
       "             ('potential', 320),\n",
       "             ('electronic', 319),\n",
       "             ('motivated', 319),\n",
       "             ('passion', 319),\n",
       "             ('those', 318),\n",
       "             ('interface', 318),\n",
       "             ('attention', 318),\n",
       "             ('20', 318),\n",
       "             ('principles', 318),\n",
       "             ('special', 317),\n",
       "             ('solid', 317),\n",
       "             ('north', 317),\n",
       "             ('call', 317),\n",
       "             ('languages', 316),\n",
       "             ('verification', 316),\n",
       "             ('30', 316),\n",
       "             ('accordance', 315),\n",
       "             ('ley', 315),\n",
       "             ('changes', 314),\n",
       "             ('agency', 314),\n",
       "             ('components', 314),\n",
       "             ('meetings', 313),\n",
       "             ('bs', 313),\n",
       "             ('expected', 313),\n",
       "             ('university', 313),\n",
       "             ('delivering', 312),\n",
       "             ('records', 312),\n",
       "             ('direction', 312),\n",
       "             ('consulting', 311),\n",
       "             ('search', 310),\n",
       "             ('reasonable', 309),\n",
       "             ('accredited', 309),\n",
       "             ('objectives', 309),\n",
       "             ('between', 308),\n",
       "             ('outside', 308),\n",
       "             ('canada', 307),\n",
       "             ('terms', 305),\n",
       "             ('similar', 305),\n",
       "             ('aspects', 305),\n",
       "             ('temporary', 305),\n",
       "             ('prestaciones', 305),\n",
       "             ('language', 304),\n",
       "             ('were', 304),\n",
       "             ('audit', 303),\n",
       "             ('subject', 303),\n",
       "             ('serve', 303),\n",
       "             ('solution', 303),\n",
       "             ('core', 302),\n",
       "             ('50', 302),\n",
       "             ('usa', 302),\n",
       "             ('users', 301),\n",
       "             ('associated', 300),\n",
       "             ('division', 299),\n",
       "             ('conditions', 298),\n",
       "             ('growing', 298),\n",
       "             ('collaboration', 296),\n",
       "             ('specialist', 295),\n",
       "             ('days', 294),\n",
       "             ('resolution', 294),\n",
       "             ('offers', 293),\n",
       "             ('post', 293),\n",
       "             ('pay', 292),\n",
       "             ('jobs', 292),\n",
       "             ('successfully', 292),\n",
       "             ('driving', 292),\n",
       "             ('developers', 291),\n",
       "             ('overall', 291),\n",
       "             ('person', 291),\n",
       "             ('tracking', 290),\n",
       "             ('track', 290),\n",
       "             ('scripts', 290),\n",
       "             ('groups', 290),\n",
       "             ('scale', 289),\n",
       "             ('accurate', 289),\n",
       "             ('monitoring', 289),\n",
       "             ('coordinate', 289),\n",
       "             ('above', 289),\n",
       "             ('lives', 288),\n",
       "             ('advance', 288),\n",
       "             ('javascript', 287),\n",
       "             ('verify', 287),\n",
       "             ('cost', 287),\n",
       "             ('framework', 287),\n",
       "             ('better', 286),\n",
       "             ('submit', 285),\n",
       "             ('skilled', 285),\n",
       "             ('regarding', 285),\n",
       "             ('instructions', 284),\n",
       "             ('discipline', 283),\n",
       "             ('associate', 281),\n",
       "             ('evaluation', 280),\n",
       "             ('real', 280),\n",
       "             ('handling', 280),\n",
       "             ('content', 280),\n",
       "             ('features', 279),\n",
       "             ('require', 279),\n",
       "             ('sueldo', 278),\n",
       "             ('master', 278),\n",
       "             ('challenges', 278),\n",
       "             ('40', 276),\n",
       "             ('ensuring', 276),\n",
       "             ('many', 275),\n",
       "             ('enable', 275),\n",
       "             ('implementing', 275),\n",
       "             ('sistemas', 274),\n",
       "             ('administrative', 274),\n",
       "             ('ofrecemos', 274),\n",
       "             ('graduate', 274),\n",
       "             ('accounts', 273),\n",
       "             ('completed', 272),\n",
       "             ('exciting', 272),\n",
       "             ('week', 272),\n",
       "             ('proficient', 271),\n",
       "             ('cycle', 271),\n",
       "             ('vet', 270),\n",
       "             ('facility', 270),\n",
       "             ('leaders', 270),\n",
       "             ('platforms', 270),\n",
       "             ('diploma', 269),\n",
       "             ('maintains', 269),\n",
       "             ('assists', 268),\n",
       "             ('diversity', 268),\n",
       "             ('resolve', 267),\n",
       "             ('base', 267),\n",
       "             ('dynamic', 267),\n",
       "             ('grow', 266),\n",
       "             ('escolaridad', 265),\n",
       "             ('release', 265),\n",
       "             ('11', 265),\n",
       "             ('proficiency', 263),\n",
       "             ('specialty', 263),\n",
       "             ('does', 262),\n",
       "             ('guidelines', 261),\n",
       "             ('feedback', 261),\n",
       "             ('dedicated', 261),\n",
       "             ('excellence', 261),\n",
       "             ('ensures', 261),\n",
       "             ('device', 261),\n",
       "             ('ca', 260),\n",
       "             ('environments', 260),\n",
       "             ('facilities', 260),\n",
       "             ('once', 260),\n",
       "             ('today', 259),\n",
       "             ('conocimientos', 259),\n",
       "             ('designs', 259),\n",
       "             ('small', 259),\n",
       "             ('coding', 259),\n",
       "             ('agencies', 258),\n",
       "             ('communities', 257),\n",
       "             ('flexible', 256),\n",
       "             ('focused', 256),\n",
       "             ('off', 256),\n",
       "             ('previous', 256),\n",
       "             ('requisitos', 256),\n",
       "             ('preparation', 256),\n",
       "             ('upon', 255),\n",
       "             ('demonstrate', 255),\n",
       "             ('paid', 255),\n",
       "             ('http', 255),\n",
       "             ('canadian', 254),\n",
       "             ('entry', 254),\n",
       "             ('write', 254),\n",
       "             ('questions', 254),\n",
       "             ('way', 253),\n",
       "             ('risks', 253),\n",
       "             ('determine', 253),\n",
       "             ('significant', 252),\n",
       "             ('evaluate', 250),\n",
       "             ('changing', 250),\n",
       "             ('investigation', 249),\n",
       "             ('largest', 249),\n",
       "             ('800', 249),\n",
       "             ('months', 248),\n",
       "             ('firms', 248),\n",
       "             ('protocols', 247),\n",
       "             ('home', 247),\n",
       "             ('accuracy', 247),\n",
       "             ('against', 247),\n",
       "             ('very', 245),\n",
       "             ('portfolio', 245),\n",
       "             ('ii', 245),\n",
       "             ('operate', 245),\n",
       "             ('military', 244),\n",
       "             ('debugging', 244),\n",
       "             ('25', 244),\n",
       "             ('designing', 244),\n",
       "             ('board', 242),\n",
       "             ('aa', 242),\n",
       "             ('different', 242),\n",
       "             ('find', 242),\n",
       "             ('positive', 240),\n",
       "             ('prioritize', 239),\n",
       "             ('higher', 238),\n",
       "             ('hire', 237),\n",
       "             ('extensive', 237),\n",
       "             ('computing', 237),\n",
       "             ('decision', 237),\n",
       "             ('long', 236),\n",
       "             ('desirable', 236),\n",
       "             ('want', 236),\n",
       "             ('workplace', 236),\n",
       "             ('share', 236),\n",
       "             ('python', 235),\n",
       "             ('manual', 235),\n",
       "             ('coverage', 234),\n",
       "             ('named', 234),\n",
       "             ('clicking', 234),\n",
       "             ('recommendations', 233),\n",
       "             ('broad', 233),\n",
       "             ('civil', 233),\n",
       "             ('own', 232),\n",
       "             ('handle', 232),\n",
       "             ('magazine', 232),\n",
       "             ('creation', 231),\n",
       "             ('efforts', 231),\n",
       "             ('interact', 231),\n",
       "             ('three', 231),\n",
       "             ('500', 230),\n",
       "             ('ingles', 230),\n",
       "             ('visit', 229),\n",
       "             ('unique', 228),\n",
       "             ('admired', 228),\n",
       "             ('talent', 228),\n",
       "             ('bring', 228),\n",
       "             ('participates', 227),\n",
       "             ('collaborative', 227),\n",
       "             ('generation', 226),\n",
       "             ('skill', 226),\n",
       "             ('talented', 225),\n",
       "             ('monitor', 224),\n",
       "             ('along', 224),\n",
       "             ('judgment', 224),\n",
       "             ('reference', 223),\n",
       "             ('performed', 223),\n",
       "             ('solve', 221),\n",
       "             ('object', 221),\n",
       "             ('according', 220),\n",
       "             ('integrated', 220),\n",
       "             ('again', 219),\n",
       "             ('march', 219),\n",
       "             ('academic', 218),\n",
       "             ('next', 217),\n",
       "             ('improvements', 216),\n",
       "             ('frameworks', 216),\n",
       "             ('retaining', 215),\n",
       "             ('efficient', 214),\n",
       "             ('roles', 214),\n",
       "             ('interesados', 213),\n",
       "             ('setting', 213),\n",
       "             ('demands', 213),\n",
       "             ('troubleshooting', 213),\n",
       "             ('edge', 212),\n",
       "             ('task', 212),\n",
       "             ('exposure', 212),\n",
       "             ('priorities', 211),\n",
       "             ('postal', 211),\n",
       "             ('correo', 211),\n",
       "             ('assisting', 211),\n",
       "             ('scripting', 210),\n",
       "             ('branch', 210),\n",
       "             ('presentation', 210),\n",
       "             ('contribute', 209),\n",
       "             ('creative', 208),\n",
       "             ('billion', 208),\n",
       "             ('achieve', 207),\n",
       "             ('pre', 206),\n",
       "             ('enviar', 206),\n",
       "             ('agreeing', 206),\n",
       "             ('made', 205),\n",
       "             ('suite', 205),\n",
       "             ('independent', 205),\n",
       "             ('profile', 204),\n",
       "             ('zona', 201),\n",
       "             ('passionate', 198),\n",
       "             ('re', 198),\n",
       "             ('know', 196),\n",
       "             ('consistent', 196),\n",
       "             ('clear', 196),\n",
       "             ('involved', 196),\n",
       "             ('expert', 195),\n",
       "             ('80', 195),\n",
       "             ('premier', 193),\n",
       "             ('solicita', 191),\n",
       "             ('difference', 191),\n",
       "             ('then', 190),\n",
       "             ('scope', 189),\n",
       "             ('met', 189),\n",
       "             ('immediate', 187),\n",
       "             ('characteristic', 187),\n",
       "             ('get', 187),\n",
       "             ('close', 185),\n",
       "             ('discriminate', 183),\n",
       "             ('dental', 179),\n",
       "             ('northropgrumman', 178)])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_word_counts(corpus_bag_of_words, corpus_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our top words are articles, prepositions and conjunctions that do not tell us anything about our courpus. Let's march on create topics anyway."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "| 2 Elapsed Time: 0:00:16                                                                                                                                              \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 and, to, the, of, in\n",
      "1 de, en, la, con, el\n",
      "2 and, to, the, of, in\n"
     ]
    }
   ],
   "source": [
    "ls_corpus_keywords, corpus_doctopic = create_topics(corpus_bag_of_words, corpus_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at these topics we have no real knowledge of what is in our corpus, with the exception that there are job ads written in Spanish. The problem is the the top words in the topics are conjunctions and prepositions that have no semantic information. We have to try and clean and process our data to get more meaningful infomation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text Cleaning and Normalization\n",
    "\n",
    "To clean and normalize text we will remove all special characters, numbers, and punctuation. Then we will make all the text lowercase to normalize the text;  this is so words like \"the\" and \"The\" will be counted as the same in our analysis. To remove the special characters, numbers and punctuation we will use regular expressions. \n",
    "\n",
    "#### Regular Expressions\n",
    "\n",
    ">\"Some people, when confronted with a problem, think \n",
    ">'I know, I'll use regular expressions.'   Now they have two problems.\"\n",
    "> -- Jaime Zawinski\n",
    "\n",
    "Regular Expressions or regexes match a certain amount text in a document based on a set of rules and syntax. The name \"regular expressions\" actually comes from the mathematical theory it is based on. These rules are useful for pulling out useful information in a large amount of text (e.g., email addresses, html-tags, credit card numbers). Regexes often match text much more quickly then plain text sorting and can often reduce their development time. Some regular expressions can become quite complicated and it may then become a good idea to write code using Python. Any developer should keep in mind there is a trade-off between optimization and understandibility. In Python, a general philosophy is code is meant to be as understandable by *people* as much as possible. You should therefore always tend toward the understandabilty side of things rather than overly optimizing your code. Your future-self, code-reviewers, people who inherit your code, and anyone else who has to make sense of your code in the future will appreciate it. \n",
    "\n",
    "For our purposes we are going to use a regular expression to match all characters that are not letters -- punctutation, quotes, special characters and numbers --replace them with spaces and then take all the remaining characters and make them lowercase. \n",
    "\n",
    "A full tutorial on regular expressions would be outside the scope of this tutorial. There are many good tutorials that can be found on-line. There is also a great interactive tool for developing and checking regular expressions regex101.com. \n",
    "\n",
    "We will be using the `re` library in python for regular expression matching."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#get rid of the punctuations and set all characters to lowercase\n",
    "RE_PREPROCESS = r'\\W+|\\d+' #the regular expressions that matches all non-characters\n",
    "\n",
    "#get rid of punctuation and make everything lowercase\n",
    "#the code belows works by looping through the array of text\n",
    "#for a given piece of text we invoke the `re.sub` command where we pass in the regular expression, a space ' ' to\n",
    "#subsitute all the matching characters with\n",
    "#we then invoke the `lower()` method on the output of the re.sub command\n",
    "#to make all the remaining characters\n",
    "#the cleaned document is then stored in a list\n",
    "#once this list has been filed it is then stored in a numpy array\n",
    "\n",
    "processed_corpus = np.array( [ re.sub(RE_PREPROCESS, ' ', comment).lower() for comment in corpus] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### first decription before cleaning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"This position is in support of the United States Attorney's Office in Buffalo, NY located at 138 Delaware Avenue. The estimated length of this assignment is through May 31, 2016 with the possibility of extensions. Overtime and travel may be authorized as needed. The Employee will perform the following tasks, but not limited to: *Serves as a primary resource for the office on all aspects of docketing. *Recommends revisions in office procedures to improve the docketing function by expediting case processing, changing codes for data elements, etc. *Reviews reports to identify recurring errors and trains new employees in the operation and use of the case tracking system. Advises other personnel on appropriate codes for unprecedented cases. Explains the operation of the system and importance of timely information to new attorneys and support staff. *Responsibility for developing/maintaining automated records for the criminal and civil case workload from initial referral to final disposition. *Regularly inputs new information into the database from LIONS coding sheets provided by District Division Chiefs. *Runs regularly scheduled and preprogrammed management reports with the assistance of System Manager. *Requests new or modified reports and/or internal processing procedures to meet perceived changes in the information needs of the office and/or to improve docketing efficiency/effectiveness. *Discusses changes with supervisor. *Maintains a working knowledge of the Districts and Departments docket and reporting procedures in order to assure proper interpretation and data entry of codes. *Assists Legal Assistants and other employees in obtaining information from the database and in interpreting output. *Provides ongoing instruction as to processing requirements. *Receives numerous telephone calls on case status from client agencies, the courts, employees within the office, city, county and state law office, private attorneys and individuals. *Exercises tact, discretion and learned judgment in determining what information and be provided and to whom. *Must be aware of what cases are sensitive and for which no information can be provided. *Perform others duties as necessary. Basic Qualifications *Proven excellent written and oral communication skills. *Paralegal Certificate. An equivalent level of training may be substituted with the approval of the COTR. *At least ONE (1) year of litigation paralegal experience. *Must have basic legal knowledge, including knowledge of standard legal citation system sufficient to perform basic legal research. *Thorough knowledge of legal research tools such as LEXIS and WestLaw. *Should have hands-on familiarity with a variety of computer applications, including word processing, database, spreadsheet, imaging, and telecommunications. *Ability to consistently deliver highest quality of work under extreme pressure will be very important. *Some walking, standing, bending and carrying of small office items *Ability to obtain a security clearance. *Must be a U.S. Citizen. Desired skills *Experience working in a Government and/or Litigation Support environment in conjunction with basic qualifications. *Proficient in Microsoft Office software programs (i.e., Word, Excel, Outlook, MS Access or other data bases), Word Perfect and Adobe Acrobat Professional, strongly preferred. *Experience with docketing processes and procedures. *Attention to detail. *Trial experience, helpful. *Automated litigation support experience (i.e. Concordance, Summation,etc...), very helpful. *Experience with organization and management of documents. *Current or active clearance. As a leading technology innovation company, Lockheed Martins team of 113,000 people works with partners around the world to bring proven performance to our customers toughest challenges. Lockheed Martin has employees based in all 50 states and more than 570 facilities that span 70 countries. Join us at Lockheed Martin, where were engineering a better tomorrow. *Lockheed Martin is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status.* Job Location(s): Buffalo New York\""
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### first description after cleaning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'this position is in support of the united states attorney s office in buffalo ny located at   delaware avenue the estimated length of this assignment is through may     with the possibility of extensions overtime and travel may be authorized as needed the employee will perform the following tasks but not limited to serves as a primary resource for the office on all aspects of docketing recommends revisions in office procedures to improve the docketing function by expediting case processing changing codes for data elements etc reviews reports to identify recurring errors and trains new employees in the operation and use of the case tracking system advises other personnel on appropriate codes for unprecedented cases explains the operation of the system and importance of timely information to new attorneys and support staff responsibility for developing maintaining automated records for the criminal and civil case workload from initial referral to final disposition regularly inputs new information into the database from lions coding sheets provided by district division chiefs runs regularly scheduled and preprogrammed management reports with the assistance of system manager requests new or modified reports and or internal processing procedures to meet perceived changes in the information needs of the office and or to improve docketing efficiency effectiveness discusses changes with supervisor maintains a working knowledge of the districts and departments docket and reporting procedures in order to assure proper interpretation and data entry of codes assists legal assistants and other employees in obtaining information from the database and in interpreting output provides ongoing instruction as to processing requirements receives numerous telephone calls on case status from client agencies the courts employees within the office city county and state law office private attorneys and individuals exercises tact discretion and learned judgment in determining what information and be provided and to whom must be aware of what cases are sensitive and for which no information can be provided perform others duties as necessary basic qualifications proven excellent written and oral communication skills paralegal certificate an equivalent level of training may be substituted with the approval of the cotr at least one   year of litigation paralegal experience must have basic legal knowledge including knowledge of standard legal citation system sufficient to perform basic legal research thorough knowledge of legal research tools such as lexis and westlaw should have hands on familiarity with a variety of computer applications including word processing database spreadsheet imaging and telecommunications ability to consistently deliver highest quality of work under extreme pressure will be very important some walking standing bending and carrying of small office items ability to obtain a security clearance must be a u s citizen desired skills experience working in a government and or litigation support environment in conjunction with basic qualifications proficient in microsoft office software programs i e word excel outlook ms access or other data bases word perfect and adobe acrobat professional strongly preferred experience with docketing processes and procedures attention to detail trial experience helpful automated litigation support experience i e concordance summation etc very helpful experience with organization and management of documents current or active clearance as a leading technology innovation company lockheed martins team of     people works with partners around the world to bring proven performance to our customers toughest challenges lockheed martin has employees based in all   states and more than   facilities that span   countries join us at lockheed martin where were engineering a better tomorrow lockheed martin is an equal opportunity affirmative action employer all qualified applicants will receive consideration for employment without regard to race color religion sex pregnancy sexual orientation gender identity national origin age protected veteran status or disability status job location s buffalo new york'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "processed_corpus[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All lowercase, all numbers and special chracters have been removed. Out text is now normalized."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization\n",
    "\n",
    "Now that we have cleaned our text we can tokenize it by deciding which terms and phrases are the most meaningful. In this case we want to split our text into individual words. Our words are separted by spaces so we can use the `.split()` command to turn are document into a list of words using a space as the character to split on as an example. Normally the `CountVectorizer` handles this for us.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tokens = processed_corpus[0].split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['this',\n",
       " 'position',\n",
       " 'is',\n",
       " 'in',\n",
       " 'support',\n",
       " 'of',\n",
       " 'the',\n",
       " 'united',\n",
       " 'states',\n",
       " 'attorney',\n",
       " 's',\n",
       " 'office',\n",
       " 'in',\n",
       " 'buffalo',\n",
       " 'ny',\n",
       " 'located',\n",
       " 'at',\n",
       " 'delaware',\n",
       " 'avenue',\n",
       " 'the',\n",
       " 'estimated',\n",
       " 'length',\n",
       " 'of',\n",
       " 'this',\n",
       " 'assignment',\n",
       " 'is',\n",
       " 'through',\n",
       " 'may',\n",
       " 'with',\n",
       " 'the',\n",
       " 'possibility',\n",
       " 'of',\n",
       " 'extensions',\n",
       " 'overtime',\n",
       " 'and',\n",
       " 'travel',\n",
       " 'may',\n",
       " 'be',\n",
       " 'authorized',\n",
       " 'as',\n",
       " 'needed',\n",
       " 'the',\n",
       " 'employee',\n",
       " 'will',\n",
       " 'perform',\n",
       " 'the',\n",
       " 'following',\n",
       " 'tasks',\n",
       " 'but',\n",
       " 'not',\n",
       " 'limited',\n",
       " 'to',\n",
       " 'serves',\n",
       " 'as',\n",
       " 'a',\n",
       " 'primary',\n",
       " 'resource',\n",
       " 'for',\n",
       " 'the',\n",
       " 'office',\n",
       " 'on',\n",
       " 'all',\n",
       " 'aspects',\n",
       " 'of',\n",
       " 'docketing',\n",
       " 'recommends',\n",
       " 'revisions',\n",
       " 'in',\n",
       " 'office',\n",
       " 'procedures',\n",
       " 'to',\n",
       " 'improve',\n",
       " 'the',\n",
       " 'docketing',\n",
       " 'function',\n",
       " 'by',\n",
       " 'expediting',\n",
       " 'case',\n",
       " 'processing',\n",
       " 'changing',\n",
       " 'codes',\n",
       " 'for',\n",
       " 'data',\n",
       " 'elements',\n",
       " 'etc',\n",
       " 'reviews',\n",
       " 'reports',\n",
       " 'to',\n",
       " 'identify',\n",
       " 'recurring',\n",
       " 'errors',\n",
       " 'and',\n",
       " 'trains',\n",
       " 'new',\n",
       " 'employees',\n",
       " 'in',\n",
       " 'the',\n",
       " 'operation',\n",
       " 'and',\n",
       " 'use',\n",
       " 'of',\n",
       " 'the',\n",
       " 'case',\n",
       " 'tracking',\n",
       " 'system',\n",
       " 'advises',\n",
       " 'other',\n",
       " 'personnel',\n",
       " 'on',\n",
       " 'appropriate',\n",
       " 'codes',\n",
       " 'for',\n",
       " 'unprecedented',\n",
       " 'cases',\n",
       " 'explains',\n",
       " 'the',\n",
       " 'operation',\n",
       " 'of',\n",
       " 'the',\n",
       " 'system',\n",
       " 'and',\n",
       " 'importance',\n",
       " 'of',\n",
       " 'timely',\n",
       " 'information',\n",
       " 'to',\n",
       " 'new',\n",
       " 'attorneys',\n",
       " 'and',\n",
       " 'support',\n",
       " 'staff',\n",
       " 'responsibility',\n",
       " 'for',\n",
       " 'developing',\n",
       " 'maintaining',\n",
       " 'automated',\n",
       " 'records',\n",
       " 'for',\n",
       " 'the',\n",
       " 'criminal',\n",
       " 'and',\n",
       " 'civil',\n",
       " 'case',\n",
       " 'workload',\n",
       " 'from',\n",
       " 'initial',\n",
       " 'referral',\n",
       " 'to',\n",
       " 'final',\n",
       " 'disposition',\n",
       " 'regularly',\n",
       " 'inputs',\n",
       " 'new',\n",
       " 'information',\n",
       " 'into',\n",
       " 'the',\n",
       " 'database',\n",
       " 'from',\n",
       " 'lions',\n",
       " 'coding',\n",
       " 'sheets',\n",
       " 'provided',\n",
       " 'by',\n",
       " 'district',\n",
       " 'division',\n",
       " 'chiefs',\n",
       " 'runs',\n",
       " 'regularly',\n",
       " 'scheduled',\n",
       " 'and',\n",
       " 'preprogrammed',\n",
       " 'management',\n",
       " 'reports',\n",
       " 'with',\n",
       " 'the',\n",
       " 'assistance',\n",
       " 'of',\n",
       " 'system',\n",
       " 'manager',\n",
       " 'requests',\n",
       " 'new',\n",
       " 'or',\n",
       " 'modified',\n",
       " 'reports',\n",
       " 'and',\n",
       " 'or',\n",
       " 'internal',\n",
       " 'processing',\n",
       " 'procedures',\n",
       " 'to',\n",
       " 'meet',\n",
       " 'perceived',\n",
       " 'changes',\n",
       " 'in',\n",
       " 'the',\n",
       " 'information',\n",
       " 'needs',\n",
       " 'of',\n",
       " 'the',\n",
       " 'office',\n",
       " 'and',\n",
       " 'or',\n",
       " 'to',\n",
       " 'improve',\n",
       " 'docketing',\n",
       " 'efficiency',\n",
       " 'effectiveness',\n",
       " 'discusses',\n",
       " 'changes',\n",
       " 'with',\n",
       " 'supervisor',\n",
       " 'maintains',\n",
       " 'a',\n",
       " 'working',\n",
       " 'knowledge',\n",
       " 'of',\n",
       " 'the',\n",
       " 'districts',\n",
       " 'and',\n",
       " 'departments',\n",
       " 'docket',\n",
       " 'and',\n",
       " 'reporting',\n",
       " 'procedures',\n",
       " 'in',\n",
       " 'order',\n",
       " 'to',\n",
       " 'assure',\n",
       " 'proper',\n",
       " 'interpretation',\n",
       " 'and',\n",
       " 'data',\n",
       " 'entry',\n",
       " 'of',\n",
       " 'codes',\n",
       " 'assists',\n",
       " 'legal',\n",
       " 'assistants',\n",
       " 'and',\n",
       " 'other',\n",
       " 'employees',\n",
       " 'in',\n",
       " 'obtaining',\n",
       " 'information',\n",
       " 'from',\n",
       " 'the',\n",
       " 'database',\n",
       " 'and',\n",
       " 'in',\n",
       " 'interpreting',\n",
       " 'output',\n",
       " 'provides',\n",
       " 'ongoing',\n",
       " 'instruction',\n",
       " 'as',\n",
       " 'to',\n",
       " 'processing',\n",
       " 'requirements',\n",
       " 'receives',\n",
       " 'numerous',\n",
       " 'telephone',\n",
       " 'calls',\n",
       " 'on',\n",
       " 'case',\n",
       " 'status',\n",
       " 'from',\n",
       " 'client',\n",
       " 'agencies',\n",
       " 'the',\n",
       " 'courts',\n",
       " 'employees',\n",
       " 'within',\n",
       " 'the',\n",
       " 'office',\n",
       " 'city',\n",
       " 'county',\n",
       " 'and',\n",
       " 'state',\n",
       " 'law',\n",
       " 'office',\n",
       " 'private',\n",
       " 'attorneys',\n",
       " 'and',\n",
       " 'individuals',\n",
       " 'exercises',\n",
       " 'tact',\n",
       " 'discretion',\n",
       " 'and',\n",
       " 'learned',\n",
       " 'judgment',\n",
       " 'in',\n",
       " 'determining',\n",
       " 'what',\n",
       " 'information',\n",
       " 'and',\n",
       " 'be',\n",
       " 'provided',\n",
       " 'and',\n",
       " 'to',\n",
       " 'whom',\n",
       " 'must',\n",
       " 'be',\n",
       " 'aware',\n",
       " 'of',\n",
       " 'what',\n",
       " 'cases',\n",
       " 'are',\n",
       " 'sensitive',\n",
       " 'and',\n",
       " 'for',\n",
       " 'which',\n",
       " 'no',\n",
       " 'information',\n",
       " 'can',\n",
       " 'be',\n",
       " 'provided',\n",
       " 'perform',\n",
       " 'others',\n",
       " 'duties',\n",
       " 'as',\n",
       " 'necessary',\n",
       " 'basic',\n",
       " 'qualifications',\n",
       " 'proven',\n",
       " 'excellent',\n",
       " 'written',\n",
       " 'and',\n",
       " 'oral',\n",
       " 'communication',\n",
       " 'skills',\n",
       " 'paralegal',\n",
       " 'certificate',\n",
       " 'an',\n",
       " 'equivalent',\n",
       " 'level',\n",
       " 'of',\n",
       " 'training',\n",
       " 'may',\n",
       " 'be',\n",
       " 'substituted',\n",
       " 'with',\n",
       " 'the',\n",
       " 'approval',\n",
       " 'of',\n",
       " 'the',\n",
       " 'cotr',\n",
       " 'at',\n",
       " 'least',\n",
       " 'one',\n",
       " 'year',\n",
       " 'of',\n",
       " 'litigation',\n",
       " 'paralegal',\n",
       " 'experience',\n",
       " 'must',\n",
       " 'have',\n",
       " 'basic',\n",
       " 'legal',\n",
       " 'knowledge',\n",
       " 'including',\n",
       " 'knowledge',\n",
       " 'of',\n",
       " 'standard',\n",
       " 'legal',\n",
       " 'citation',\n",
       " 'system',\n",
       " 'sufficient',\n",
       " 'to',\n",
       " 'perform',\n",
       " 'basic',\n",
       " 'legal',\n",
       " 'research',\n",
       " 'thorough',\n",
       " 'knowledge',\n",
       " 'of',\n",
       " 'legal',\n",
       " 'research',\n",
       " 'tools',\n",
       " 'such',\n",
       " 'as',\n",
       " 'lexis',\n",
       " 'and',\n",
       " 'westlaw',\n",
       " 'should',\n",
       " 'have',\n",
       " 'hands',\n",
       " 'on',\n",
       " 'familiarity',\n",
       " 'with',\n",
       " 'a',\n",
       " 'variety',\n",
       " 'of',\n",
       " 'computer',\n",
       " 'applications',\n",
       " 'including',\n",
       " 'word',\n",
       " 'processing',\n",
       " 'database',\n",
       " 'spreadsheet',\n",
       " 'imaging',\n",
       " 'and',\n",
       " 'telecommunications',\n",
       " 'ability',\n",
       " 'to',\n",
       " 'consistently',\n",
       " 'deliver',\n",
       " 'highest',\n",
       " 'quality',\n",
       " 'of',\n",
       " 'work',\n",
       " 'under',\n",
       " 'extreme',\n",
       " 'pressure',\n",
       " 'will',\n",
       " 'be',\n",
       " 'very',\n",
       " 'important',\n",
       " 'some',\n",
       " 'walking',\n",
       " 'standing',\n",
       " 'bending',\n",
       " 'and',\n",
       " 'carrying',\n",
       " 'of',\n",
       " 'small',\n",
       " 'office',\n",
       " 'items',\n",
       " 'ability',\n",
       " 'to',\n",
       " 'obtain',\n",
       " 'a',\n",
       " 'security',\n",
       " 'clearance',\n",
       " 'must',\n",
       " 'be',\n",
       " 'a',\n",
       " 'u',\n",
       " 's',\n",
       " 'citizen',\n",
       " 'desired',\n",
       " 'skills',\n",
       " 'experience',\n",
       " 'working',\n",
       " 'in',\n",
       " 'a',\n",
       " 'government',\n",
       " 'and',\n",
       " 'or',\n",
       " 'litigation',\n",
       " 'support',\n",
       " 'environment',\n",
       " 'in',\n",
       " 'conjunction',\n",
       " 'with',\n",
       " 'basic',\n",
       " 'qualifications',\n",
       " 'proficient',\n",
       " 'in',\n",
       " 'microsoft',\n",
       " 'office',\n",
       " 'software',\n",
       " 'programs',\n",
       " 'i',\n",
       " 'e',\n",
       " 'word',\n",
       " 'excel',\n",
       " 'outlook',\n",
       " 'ms',\n",
       " 'access',\n",
       " 'or',\n",
       " 'other',\n",
       " 'data',\n",
       " 'bases',\n",
       " 'word',\n",
       " 'perfect',\n",
       " 'and',\n",
       " 'adobe',\n",
       " 'acrobat',\n",
       " 'professional',\n",
       " 'strongly',\n",
       " 'preferred',\n",
       " 'experience',\n",
       " 'with',\n",
       " 'docketing',\n",
       " 'processes',\n",
       " 'and',\n",
       " 'procedures',\n",
       " 'attention',\n",
       " 'to',\n",
       " 'detail',\n",
       " 'trial',\n",
       " 'experience',\n",
       " 'helpful',\n",
       " 'automated',\n",
       " 'litigation',\n",
       " 'support',\n",
       " 'experience',\n",
       " 'i',\n",
       " 'e',\n",
       " 'concordance',\n",
       " 'summation',\n",
       " 'etc',\n",
       " 'very',\n",
       " 'helpful',\n",
       " 'experience',\n",
       " 'with',\n",
       " 'organization',\n",
       " 'and',\n",
       " 'management',\n",
       " 'of',\n",
       " 'documents',\n",
       " 'current',\n",
       " 'or',\n",
       " 'active',\n",
       " 'clearance',\n",
       " 'as',\n",
       " 'a',\n",
       " 'leading',\n",
       " 'technology',\n",
       " 'innovation',\n",
       " 'company',\n",
       " 'lockheed',\n",
       " 'martins',\n",
       " 'team',\n",
       " 'of',\n",
       " 'people',\n",
       " 'works',\n",
       " 'with',\n",
       " 'partners',\n",
       " 'around',\n",
       " 'the',\n",
       " 'world',\n",
       " 'to',\n",
       " 'bring',\n",
       " 'proven',\n",
       " 'performance',\n",
       " 'to',\n",
       " 'our',\n",
       " 'customers',\n",
       " 'toughest',\n",
       " 'challenges',\n",
       " 'lockheed',\n",
       " 'martin',\n",
       " 'has',\n",
       " 'employees',\n",
       " 'based',\n",
       " 'in',\n",
       " 'all',\n",
       " 'states',\n",
       " 'and',\n",
       " 'more',\n",
       " 'than',\n",
       " 'facilities',\n",
       " 'that',\n",
       " 'span',\n",
       " 'countries',\n",
       " 'join',\n",
       " 'us',\n",
       " 'at',\n",
       " 'lockheed',\n",
       " 'martin',\n",
       " 'where',\n",
       " 'were',\n",
       " 'engineering',\n",
       " 'a',\n",
       " 'better',\n",
       " 'tomorrow',\n",
       " 'lockheed',\n",
       " 'martin',\n",
       " 'is',\n",
       " 'an',\n",
       " 'equal',\n",
       " 'opportunity',\n",
       " 'affirmative',\n",
       " 'action',\n",
       " 'employer',\n",
       " 'all',\n",
       " 'qualified',\n",
       " 'applicants',\n",
       " 'will',\n",
       " 'receive',\n",
       " 'consideration',\n",
       " 'for',\n",
       " 'employment',\n",
       " 'without',\n",
       " 'regard',\n",
       " 'to',\n",
       " 'race',\n",
       " 'color',\n",
       " 'religion',\n",
       " 'sex',\n",
       " 'pregnancy',\n",
       " 'sexual',\n",
       " 'orientation',\n",
       " 'gender',\n",
       " 'identity',\n",
       " 'national',\n",
       " 'origin',\n",
       " 'age',\n",
       " 'protected',\n",
       " 'veteran',\n",
       " 'status',\n",
       " 'or',\n",
       " 'disability',\n",
       " 'status',\n",
       " 'job',\n",
       " 'location',\n",
       " 's',\n",
       " 'buffalo',\n",
       " 'new',\n",
       " 'york']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stopwords\n",
    "\n",
    "Stopwords are words that have very little semantic meaning and are found throughout a text. Having the word *the* or *of* will tell us nothing about our corpus, nor will they be meaningful features.  Examples of stopwords are prepositions, articles and common nouns. To process the corpus even further we can eliminate these stopwords by checking if the are in a list of commonly occuring stopwords.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "eng_stopwords =  stopwords.words('english')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['i',\n",
       " 'yours',\n",
       " 'herself',\n",
       " 'which',\n",
       " 'was',\n",
       " 'does',\n",
       " 'because',\n",
       " 'against',\n",
       " 'from',\n",
       " 'further',\n",
       " 'any',\n",
       " 'nor',\n",
       " 't',\n",
       " 'o',\n",
       " 'hasn',\n",
       " 'weren']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#sample of stopwords\n",
    "eng_stopwords[::10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('experience', 11108),\n",
       "             ('de', 8575),\n",
       "             ('work', 6583),\n",
       "             ('job', 5631),\n",
       "             ('software', 5360),\n",
       "             ('skills', 5337),\n",
       "             ('development', 5079),\n",
       "             ('test', 4744),\n",
       "             ('team', 4450),\n",
       "             ('ability', 4248),\n",
       "             ('required', 4147),\n",
       "             ('business', 4010),\n",
       "             ('quality', 3664),\n",
       "             ('requirements', 3653),\n",
       "             ('position', 3494),\n",
       "             ('testing', 3413),\n",
       "             ('knowledge', 3366),\n",
       "             ('years', 3336),\n",
       "             ('management', 3320),\n",
       "             ('must', 3261),\n",
       "             ('systems', 3103),\n",
       "             ('engineering', 3047),\n",
       "             ('services', 3043),\n",
       "             ('information', 2989),\n",
       "             ('support', 2955),\n",
       "             ('legal', 2816),\n",
       "             ('en', 2791),\n",
       "             ('status', 2757),\n",
       "             ('opportunity', 2734),\n",
       "             ('including', 2627),\n",
       "             ('working', 2559),\n",
       "             ('design', 2531),\n",
       "             ('company', 2528),\n",
       "             ('technical', 2487),\n",
       "             ('data', 2359),\n",
       "             ('provide', 2307),\n",
       "             ('time', 2297),\n",
       "             ('degree', 2276),\n",
       "             ('related', 2264),\n",
       "             ('new', 2242),\n",
       "             ('strong', 2206),\n",
       "             ('environment', 2105),\n",
       "             ('qualifications', 2103),\n",
       "             ('system', 2072),\n",
       "             ('compliance', 2019),\n",
       "             ('equal', 1993),\n",
       "             ('preferred', 1925),\n",
       "             ('location', 1899),\n",
       "             ('training', 1839),\n",
       "             ('employer', 1799),\n",
       "             ('product', 1795),\n",
       "             ('medical', 1786),\n",
       "             ('us', 1772),\n",
       "             ('responsibilities', 1770),\n",
       "             ('employment', 1732),\n",
       "             ('process', 1726),\n",
       "             ('program', 1724),\n",
       "             ('law', 1685),\n",
       "             ('disability', 1679),\n",
       "             ('service', 1667),\n",
       "             ('solutions', 1664),\n",
       "             ('customer', 1661),\n",
       "             ('communication', 1659),\n",
       "             ('procedures', 1652),\n",
       "             ('engineer', 1641),\n",
       "             ('project', 1622),\n",
       "             ('computer', 1608),\n",
       "             ('products', 1598),\n",
       "             ('apply', 1582),\n",
       "             ('analysis', 1576),\n",
       "             ('technology', 1562),\n",
       "             ('national', 1556),\n",
       "             ('la', 1545),\n",
       "             ('office', 1539),\n",
       "             ('high', 1536),\n",
       "             ('may', 1523),\n",
       "             ('level', 1521),\n",
       "             ('include', 1513),\n",
       "             ('applicants', 1495),\n",
       "             ('within', 1492),\n",
       "             ('tools', 1484),\n",
       "             ('state', 1474),\n",
       "             ('based', 1472),\n",
       "             ('develop', 1448),\n",
       "             ('performance', 1417),\n",
       "             ('ensure', 1387),\n",
       "             ('health', 1383),\n",
       "             ('responsible', 1373),\n",
       "             ('laboratory', 1371),\n",
       "             ('world', 1370),\n",
       "             ('minimum', 1362),\n",
       "             ('full', 1361),\n",
       "             ('customers', 1358),\n",
       "             ('protected', 1344),\n",
       "             ('duties', 1344),\n",
       "             ('excellent', 1337),\n",
       "             ('microsoft', 1336),\n",
       "             ('description', 1322),\n",
       "             ('education', 1320),\n",
       "             ('id', 1318),\n",
       "             ('department', 1305),\n",
       "             ('gender', 1303),\n",
       "             ('issues', 1291),\n",
       "             ('industry', 1290),\n",
       "             ('well', 1286),\n",
       "             ('application', 1285),\n",
       "             ('global', 1283),\n",
       "             ('part', 1270),\n",
       "             ('one', 1263),\n",
       "             ('perform', 1260),\n",
       "             ('security', 1258),\n",
       "             ('role', 1234),\n",
       "             ('applications', 1233),\n",
       "             ('able', 1233),\n",
       "             ('processes', 1231),\n",
       "             ('review', 1220),\n",
       "             ('veteran', 1220),\n",
       "             ('teams', 1213),\n",
       "             ('title', 1212),\n",
       "             ('activities', 1204),\n",
       "             ('race', 1185),\n",
       "             ('care', 1176),\n",
       "             ('professional', 1174),\n",
       "             ('research', 1169),\n",
       "             ('use', 1169),\n",
       "             ('origin', 1168),\n",
       "             ('religion', 1166),\n",
       "             ('candidate', 1162),\n",
       "             ('please', 1162),\n",
       "             ('color', 1158),\n",
       "             ('projects', 1149),\n",
       "             ('organization', 1147),\n",
       "             ('science', 1130),\n",
       "             ('written', 1129),\n",
       "             ('understanding', 1120),\n",
       "             ('employees', 1104),\n",
       "             ('opportunities', 1097),\n",
       "             ('maintain', 1088),\n",
       "             ('orientation', 1073),\n",
       "             ('people', 1070),\n",
       "             ('following', 1064),\n",
       "             ('age', 1063),\n",
       "             ('sexual', 1056),\n",
       "             ('standards', 1053),\n",
       "             ('control', 1038),\n",
       "             ('without', 1038),\n",
       "             ('also', 1032),\n",
       "             ('financial', 1028),\n",
       "             ('clients', 1025),\n",
       "             ('risk', 1022),\n",
       "             ('qualified', 1018),\n",
       "             ('benefits', 1010),\n",
       "             ('code', 1008),\n",
       "             ('leading', 994),\n",
       "             ('hours', 994),\n",
       "             ('functional', 990),\n",
       "             ('career', 987),\n",
       "             ('complex', 977),\n",
       "             ('web', 977),\n",
       "             ('identity', 976),\n",
       "             ('com', 972),\n",
       "             ('good', 964),\n",
       "             ('results', 964),\n",
       "             ('assigned', 961),\n",
       "             ('reports', 954),\n",
       "             ('con', 952),\n",
       "             ('basic', 949),\n",
       "             ('documentation', 949),\n",
       "             ('etc', 946),\n",
       "             ('life', 945),\n",
       "             ('using', 943),\n",
       "             ('technologies', 940),\n",
       "             ('action', 939),\n",
       "             ('regulatory', 937),\n",
       "             ('integration', 934),\n",
       "             ('providing', 925),\n",
       "             ('credit', 923),\n",
       "             ('join', 918),\n",
       "             ('internal', 915),\n",
       "             ('appropriate', 913),\n",
       "             ('plans', 907),\n",
       "             ('field', 906),\n",
       "             ('areas', 903),\n",
       "             ('candidates', 901),\n",
       "             ('insurance', 898),\n",
       "             ('regard', 898),\n",
       "             ('client', 897),\n",
       "             ('sales', 881),\n",
       "             ('looking', 879),\n",
       "             ('best', 879),\n",
       "             ('el', 878),\n",
       "             ('general', 872),\n",
       "             ('plus', 872),\n",
       "             ('sex', 868),\n",
       "             ('group', 865),\n",
       "             ('clinical', 864),\n",
       "             ('professionals', 861),\n",
       "             ('operations', 859),\n",
       "             ('experiencia', 857),\n",
       "             ('area', 851),\n",
       "             ('senior', 845),\n",
       "             ('make', 845),\n",
       "             ('across', 842),\n",
       "             ('developing', 834),\n",
       "             ('corporate', 834),\n",
       "             ('equivalent', 826),\n",
       "             ('needs', 815),\n",
       "             ('assurance', 811),\n",
       "             ('plan', 802),\n",
       "             ('highly', 797),\n",
       "             ('patient', 789),\n",
       "             ('functions', 786),\n",
       "             ('healthcare', 785),\n",
       "             ('equipment', 782),\n",
       "             ('companies', 777),\n",
       "             ('lab', 768),\n",
       "             ('assist', 762),\n",
       "             ('problems', 761),\n",
       "             ('compensation', 758),\n",
       "             ('java', 758),\n",
       "             ('year', 756),\n",
       "             ('states', 754),\n",
       "             ('assistant', 750),\n",
       "             ('practices', 745),\n",
       "             ('complete', 739),\n",
       "             ('federal', 738),\n",
       "             ('robert', 737),\n",
       "             ('half', 736),\n",
       "             ('basis', 731),\n",
       "             ('manager', 730),\n",
       "             ('staff', 729),\n",
       "             ('provides', 726),\n",
       "             ('policies', 726),\n",
       "             ('programs', 723),\n",
       "             ('employee', 722),\n",
       "             ('effectively', 722),\n",
       "             ('center', 721),\n",
       "             ('documents', 708),\n",
       "             ('requisition', 704),\n",
       "             ('reviews', 703),\n",
       "             ('multiple', 703),\n",
       "             ('essential', 702),\n",
       "             ('help', 702),\n",
       "             ('lead', 689),\n",
       "             ('meet', 689),\n",
       "             ('programming', 685),\n",
       "             ('various', 681),\n",
       "             ('united', 677),\n",
       "             ('personal', 674),\n",
       "             ('create', 674),\n",
       "             ('diverse', 673),\n",
       "             ('implementation', 667),\n",
       "             ('specific', 666),\n",
       "             ('problem', 663),\n",
       "             ('manage', 663),\n",
       "             ('tests', 660),\n",
       "             ('eeo', 660),\n",
       "             ('planning', 660),\n",
       "             ('key', 659),\n",
       "             ('commercial', 659),\n",
       "             ('counsel', 654),\n",
       "             ('leadership', 654),\n",
       "             ('resources', 654),\n",
       "             ('contact', 651),\n",
       "             ('individuals', 651),\n",
       "             ('affirmative', 650),\n",
       "             ('analyst', 645),\n",
       "             ('learn', 645),\n",
       "             ('demonstrated', 643),\n",
       "             ('patients', 643),\n",
       "             ('automation', 642),\n",
       "             ('analytical', 641),\n",
       "             ('additional', 641),\n",
       "             ('verbal', 639),\n",
       "             ('consideration', 639),\n",
       "             ('local', 638),\n",
       "             ('online', 637),\n",
       "             ('competitive', 636),\n",
       "             ('individual', 633),\n",
       "             ('marital', 631),\n",
       "             ('offer', 631),\n",
       "             ('build', 630),\n",
       "             ('type', 626),\n",
       "             ('partner', 623),\n",
       "             ('range', 623),\n",
       "             ('school', 622),\n",
       "             ('regulations', 622),\n",
       "             ('qa', 621),\n",
       "             ('current', 620),\n",
       "             ('matters', 619),\n",
       "             ('travel', 619),\n",
       "             ('committed', 615),\n",
       "             ('bachelor', 614),\n",
       "             ('communicate', 610),\n",
       "             ('tasks', 607),\n",
       "             ('independently', 606),\n",
       "             ('engineers', 603),\n",
       "             ('relevant', 598),\n",
       "             ('cloud', 597),\n",
       "             ('access', 592),\n",
       "             ('identify', 586),\n",
       "             ('pruebas', 585),\n",
       "             ('members', 585),\n",
       "             ('day', 585),\n",
       "             ('para', 584),\n",
       "             ('performs', 581),\n",
       "             ('needed', 581),\n",
       "             ('end', 579),\n",
       "             ('participate', 578),\n",
       "             ('automated', 578),\n",
       "             ('per', 577),\n",
       "             ('proven', 576),\n",
       "             ('understand', 573),\n",
       "             ('production', 572),\n",
       "             ('self', 570),\n",
       "             ('processing', 570),\n",
       "             ('network', 570),\n",
       "             ('mobile', 569),\n",
       "             ('document', 568),\n",
       "             ('receive', 565),\n",
       "             ('non', 564),\n",
       "             ('cases', 564),\n",
       "             ('policy', 562),\n",
       "             ('applicable', 561),\n",
       "             ('maintenance', 561),\n",
       "             ('oriented', 561),\n",
       "             ('safety', 560),\n",
       "             ('writing', 559),\n",
       "             ('successful', 557),\n",
       "             ('class', 557),\n",
       "             ('solving', 556),\n",
       "             ('necessary', 555),\n",
       "             ('improve', 555),\n",
       "             ('applying', 552),\n",
       "             ('partners', 552),\n",
       "             ('external', 549),\n",
       "             ('schedule', 546),\n",
       "             ('word', 546),\n",
       "             ('building', 545),\n",
       "             ('others', 544),\n",
       "             ('advanced', 544),\n",
       "             ('certification', 541),\n",
       "             ('expertise', 541),\n",
       "             ('sql', 541),\n",
       "             ('mission', 539),\n",
       "             ('drive', 539),\n",
       "             ('deliver', 537),\n",
       "             ('background', 536),\n",
       "             ('delivery', 535),\n",
       "             ('date', 533),\n",
       "             ('agile', 533),\n",
       "             ('city', 531),\n",
       "             ('user', 529),\n",
       "             ('desired', 525),\n",
       "             ('order', 524),\n",
       "             ('requires', 523),\n",
       "             ('reporting', 523),\n",
       "             ('seeking', 523),\n",
       "             ('excel', 523),\n",
       "             ('trabajo', 523),\n",
       "             ('international', 520),\n",
       "             ('performing', 513),\n",
       "             ('los', 512),\n",
       "             ('summary', 510),\n",
       "             ('ms', 508),\n",
       "             ('physical', 507),\n",
       "             ('making', 507),\n",
       "             ('positions', 507),\n",
       "             ('que', 505),\n",
       "             ('salary', 504),\n",
       "             ('attorney', 503),\n",
       "             ('every', 502),\n",
       "             ('clearance', 501),\n",
       "             ('unit', 501),\n",
       "             ('government', 500),\n",
       "             ('finance', 499),\n",
       "             ('category', 498),\n",
       "             ('bachelors', 498),\n",
       "             ('strategy', 497),\n",
       "             ('openings', 497),\n",
       "             ('leader', 496),\n",
       "             ('electrical', 495),\n",
       "             ('execution', 493),\n",
       "             ('genetic', 492),\n",
       "             ('primary', 490),\n",
       "             ('citizenship', 489),\n",
       "             ('marketing', 489),\n",
       "             ('las', 488),\n",
       "             ('effective', 487),\n",
       "             ('por', 485),\n",
       "             ('available', 484),\n",
       "             ('need', 484),\n",
       "             ('locations', 482),\n",
       "             ('supporting', 482),\n",
       "             ('multi', 481),\n",
       "             ('large', 481),\n",
       "             ('major', 480),\n",
       "             ('innovative', 478),\n",
       "             ('personnel', 478),\n",
       "             ('techniques', 478),\n",
       "             ('authorized', 476),\n",
       "             ('fast', 476),\n",
       "             ('least', 472),\n",
       "             ('conduct', 472),\n",
       "             ('windows', 472),\n",
       "             ('www', 471),\n",
       "             ('relationships', 470),\n",
       "             ('anos', 470),\n",
       "             ('variety', 467),\n",
       "             ('hardware', 466),\n",
       "             ('accounting', 463),\n",
       "             ('member', 463),\n",
       "             ('operating', 462),\n",
       "             ('firm', 460),\n",
       "             ('worldwide', 459),\n",
       "             ('works', 458),\n",
       "             ('resume', 454),\n",
       "             ('enterprise', 452),\n",
       "             ('decisions', 451),\n",
       "             ('country', 449),\n",
       "             ('includes', 449),\n",
       "             ('linux', 449),\n",
       "             ('english', 449),\n",
       "             ('detail', 448),\n",
       "             ('al', 447),\n",
       "             ('hands', 445),\n",
       "             ('wide', 445),\n",
       "             ('college', 445),\n",
       "             ('change', 441),\n",
       "             ('experienced', 439),\n",
       "             ('like', 437),\n",
       "             ('hiring', 436),\n",
       "             ('disabilities', 433),\n",
       "             ('initiatives', 432),\n",
       "             ('prior', 432),\n",
       "             ('practice', 430),\n",
       "             ('northrop', 429),\n",
       "             ('community', 429),\n",
       "             ('req', 429),\n",
       "             ('litigation', 428),\n",
       "             ('number', 427),\n",
       "             ('focus', 427),\n",
       "             ('staffing', 426),\n",
       "             ('familiarity', 426),\n",
       "             ('grumman', 424),\n",
       "             ('vision', 422),\n",
       "             ('statement', 421),\n",
       "             ('site', 421),\n",
       "             ('operational', 419),\n",
       "             ('del', 418),\n",
       "             ('completion', 418),\n",
       "             ('implement', 418),\n",
       "             ('success', 417),\n",
       "             ('organizational', 417),\n",
       "             ('shift', 416),\n",
       "             ('function', 416),\n",
       "             ('impact', 416),\n",
       "             ('open', 415),\n",
       "             ('experiences', 415),\n",
       "             ('located', 414),\n",
       "             ('creed', 413),\n",
       "             ('timely', 410),\n",
       "             ('standard', 410),\n",
       "             ('manufacturing', 410),\n",
       "             ('oral', 406),\n",
       "             ('fortune', 405),\n",
       "             ('take', 404),\n",
       "             ('communications', 404),\n",
       "             ('market', 404),\n",
       "             ('execute', 404),\n",
       "             ('provider', 403),\n",
       "             ('validation', 403),\n",
       "             ('levels', 403),\n",
       "             ('interpersonal', 402),\n",
       "             ('assistance', 401),\n",
       "             ('top', 399),\n",
       "             ('record', 399),\n",
       "             ('report', 399),\n",
       "             ('comprehensive', 398),\n",
       "             ('expression', 398),\n",
       "             ('continuous', 398),\n",
       "             ('list', 397),\n",
       "             ('around', 397),\n",
       "             ('countries', 397),\n",
       "             ('prepare', 395),\n",
       "             ('analyze', 395),\n",
       "             ('case', 393),\n",
       "             ('cross', 391),\n",
       "             ('used', 390),\n",
       "             ('proud', 390),\n",
       "             ('closely', 390),\n",
       "             ('right', 389),\n",
       "             ('platform', 389),\n",
       "             ('contract', 387),\n",
       "             ('culture', 386),\n",
       "             ('methodologies', 385),\n",
       "             ('interested', 384),\n",
       "             ('throughout', 384),\n",
       "             ('existing', 384),\n",
       "             ('materials', 383),\n",
       "             ('considered', 381),\n",
       "             ('package', 380),\n",
       "             ('abilities', 379),\n",
       "             ('goals', 377),\n",
       "             ('architecture', 376),\n",
       "             ('power', 375),\n",
       "             ('email', 375),\n",
       "             ('creating', 375),\n",
       "             ('follow', 374),\n",
       "             ('place', 373),\n",
       "             ('guidance', 372),\n",
       "             ('specifications', 372),\n",
       "             ('value', 371),\n",
       "             ('develops', 370),\n",
       "             ('free', 370),\n",
       "             ('tester', 369),\n",
       "             ('limited', 367),\n",
       "             ('database', 367),\n",
       "             ('commitment', 367),\n",
       "             ('critical', 367),\n",
       "             ('established', 365),\n",
       "             ('manner', 365),\n",
       "             ('requests', 362),\n",
       "             ('obtain', 362),\n",
       "             ('capabilities', 362),\n",
       "             ('innovation', 362),\n",
       "             ('details', 361),\n",
       "             ('markets', 360),\n",
       "             ('empresa', 359),\n",
       "             ('future', 359),\n",
       "             ('would', 357),\n",
       "             ('possess', 357),\n",
       "             ('set', 356),\n",
       "             ('directly', 355),\n",
       "             ('laws', 354),\n",
       "             ('supervision', 353),\n",
       "             ('methods', 353),\n",
       "             ('first', 353),\n",
       "             ('veterans', 353),\n",
       "             ('driven', 352),\n",
       "             ('supports', 352),\n",
       "             ('together', 352),\n",
       "             ('organizations', 352),\n",
       "             ('act', 352),\n",
       "             ('workforce', 351),\n",
       "             ('great', 351),\n",
       "             ('human', 350),\n",
       "             ('managers', 349),\n",
       "             ('posting', 348),\n",
       "             ('growth', 348),\n",
       "             ('overview', 348),\n",
       "             ('managing', 347),\n",
       "             ('direct', 346),\n",
       "             ('strategic', 345),\n",
       "             ('administration', 344),\n",
       "             ('active', 343),\n",
       "             ('server', 343),\n",
       "             ('maintaining', 341),\n",
       "             ('paced', 340),\n",
       "             ('responsibility', 339),\n",
       "             ('line', 338),\n",
       "             ('collection', 338),\n",
       "             ('eligible', 336),\n",
       "             ('two', 335),\n",
       "             ('public', 334),\n",
       "             ('ideal', 334),\n",
       "             ('daily', 333),\n",
       "             ('devices', 332),\n",
       "             ('pm', 332),\n",
       "             ('improvement', 331),\n",
       "             ('currently', 330),\n",
       "             ('license', 330),\n",
       "             ('strategies', 328),\n",
       "             ('addition', 328),\n",
       "             ('quickly', 327),\n",
       "             ('controls', 327),\n",
       "             ('concepts', 327),\n",
       "             ('cv', 326),\n",
       "             ('source', 325),\n",
       "             ('challenging', 324),\n",
       "             ('departments', 324),\n",
       "             ('collaborate', 323),\n",
       "             ('learning', 323),\n",
       "             ('inc', 323),\n",
       "             ('infrastructure', 322),\n",
       "             ('drug', 321),\n",
       "             ('co', 321),\n",
       "             ('assessment', 320),\n",
       "             ('call', 320),\n",
       "             ('regular', 320),\n",
       "             ('potential', 320),\n",
       "             ('electronic', 319),\n",
       "             ('motivated', 319),\n",
       "             ('passion', 319),\n",
       "             ('interface', 318),\n",
       "             ('attention', 318),\n",
       "             ('principles', 318),\n",
       "             ('north', 317),\n",
       "             ('special', 317),\n",
       "             ('solid', 317),\n",
       "             ('languages', 316),\n",
       "             ('verification', 316),\n",
       "             ('accordance', 315),\n",
       "             ('ley', 315),\n",
       "             ('meetings', 314),\n",
       "             ('changes', 314),\n",
       "             ('agency', 314),\n",
       "             ('components', 314),\n",
       "             ('bs', 313),\n",
       "             ('expected', 313),\n",
       "             ('university', 313),\n",
       "             ('delivering', 312),\n",
       "             ('records', 312),\n",
       "             ('direction', 312),\n",
       "             ('consulting', 311),\n",
       "             ('search', 310),\n",
       "             ('reasonable', 309),\n",
       "             ('accredited', 309),\n",
       "             ('objectives', 309),\n",
       "             ('outside', 308),\n",
       "             ('canada', 307),\n",
       "             ('terms', 305),\n",
       "             ('similar', 305),\n",
       "             ('aspects', 305),\n",
       "             ('temporary', 305),\n",
       "             ('prestaciones', 305),\n",
       "             ('language', 304),\n",
       "             ('audit', 303),\n",
       "             ('subject', 303),\n",
       "             ('usa', 303),\n",
       "             ('serve', 303),\n",
       "             ('solution', 303),\n",
       "             ('core', 302),\n",
       "             ('users', 301),\n",
       "             ('associated', 300),\n",
       "             ('division', 299),\n",
       "             ('growing', 298),\n",
       "             ('conditions', 298),\n",
       "             ('collaboration', 296),\n",
       "             ('days', 295),\n",
       "             ('specialist', 295),\n",
       "             ('resolution', 294),\n",
       "             ('offers', 293),\n",
       "             ('post', 293),\n",
       "             ('pay', 292),\n",
       "             ('jobs', 292),\n",
       "             ('successfully', 292),\n",
       "             ('driving', 292),\n",
       "             ('developers', 291),\n",
       "             ('overall', 291),\n",
       "             ('person', 291),\n",
       "             ('tracking', 290),\n",
       "             ('scripts', 290),\n",
       "             ('track', 290),\n",
       "             ('groups', 290),\n",
       "             ('scale', 289),\n",
       "             ('accurate', 289),\n",
       "             ('monitoring', 289),\n",
       "             ('coordinate', 289),\n",
       "             ('lives', 288),\n",
       "             ('advance', 288),\n",
       "             ('framework', 287),\n",
       "             ('javascript', 287),\n",
       "             ('verify', 287),\n",
       "             ('cost', 287),\n",
       "             ('better', 286),\n",
       "             ('submit', 285),\n",
       "             ('skilled', 285),\n",
       "             ('regarding', 285),\n",
       "             ('instructions', 284),\n",
       "             ('discipline', 283),\n",
       "             ('associate', 281),\n",
       "             ('evaluation', 280),\n",
       "             ('real', 280),\n",
       "             ('handling', 280),\n",
       "             ('content', 280),\n",
       "             ('features', 279),\n",
       "             ('require', 279),\n",
       "             ('sueldo', 278),\n",
       "             ('master', 278),\n",
       "             ('challenges', 278),\n",
       "             ('ensuring', 276),\n",
       "             ('many', 275),\n",
       "             ('enable', 275),\n",
       "             ('implementing', 275),\n",
       "             ('sistemas', 274),\n",
       "             ('graduate', 274),\n",
       "             ('administrative', 274),\n",
       "             ('ofrecemos', 274),\n",
       "             ('accounts', 273),\n",
       "             ('exciting', 272),\n",
       "             ('week', 272),\n",
       "             ('completed', 272),\n",
       "             ('cycle', 271),\n",
       "             ('proficient', 271),\n",
       "             ('vet', 270),\n",
       "             ('facility', 270),\n",
       "             ('leaders', 270),\n",
       "             ('platforms', 270),\n",
       "             ('diploma', 269),\n",
       "             ('maintains', 269),\n",
       "             ('assists', 268),\n",
       "             ('base', 268),\n",
       "             ('diversity', 268),\n",
       "             ('resolve', 267),\n",
       "             ('dynamic', 267),\n",
       "             ('ca', 266),\n",
       "             ('grow', 266),\n",
       "             ('escolaridad', 265),\n",
       "             ('release', 265),\n",
       "             ('proficiency', 263),\n",
       "             ('specialty', 263),\n",
       "             ('guidelines', 261),\n",
       "             ('feedback', 261),\n",
       "             ('dedicated', 261),\n",
       "             ('excellence', 261),\n",
       "             ('ensures', 261),\n",
       "             ('device', 261),\n",
       "             ('environments', 260),\n",
       "             ('facilities', 260),\n",
       "             ('coding', 259),\n",
       "             ('today', 259),\n",
       "             ('conocimientos', 259),\n",
       "             ('designs', 259),\n",
       "             ('small', 259),\n",
       "             ('agencies', 258),\n",
       "             ('br', 257),\n",
       "             ('communities', 257),\n",
       "             ('focused', 256),\n",
       "             ('flexible', 256),\n",
       "             ('previous', 256),\n",
       "             ('requisitos', 256),\n",
       "             ('preparation', 256),\n",
       "             ('upon', 255),\n",
       "             ('demonstrate', 255),\n",
       "             ('paid', 255),\n",
       "             ('http', 255),\n",
       "             ('canadian', 254),\n",
       "             ('write', 254),\n",
       "             ('entry', 254),\n",
       "             ('questions', 254),\n",
       "             ('way', 253),\n",
       "             ('risks', 253),\n",
       "             ('determine', 253),\n",
       "             ('significant', 252),\n",
       "             ('evaluate', 250),\n",
       "             ('changing', 250),\n",
       "             ('investigation', 249),\n",
       "             ('largest', 249),\n",
       "             ('months', 248),\n",
       "             ('firms', 248),\n",
       "             ('protocols', 247),\n",
       "             ('home', 247),\n",
       "             ('accuracy', 247),\n",
       "             ('html', 246),\n",
       "             ('operate', 245),\n",
       "             ('portfolio', 245),\n",
       "             ('ii', 245),\n",
       "             ('military', 244),\n",
       "             ('debugging', 244),\n",
       "             ('designing', 244),\n",
       "             ('board', 242),\n",
       "             ('aa', 242),\n",
       "             ('different', 242),\n",
       "             ('find', 242),\n",
       "             ('positive', 240),\n",
       "             ('prioritize', 239),\n",
       "             ('higher', 238),\n",
       "             ('hire', 237),\n",
       "             ('decision', 237),\n",
       "             ('extensive', 237),\n",
       "             ('computing', 237),\n",
       "             ('long', 236),\n",
       "             ('desirable', 236),\n",
       "             ('want', 236),\n",
       "             ('workplace', 236),\n",
       "             ('share', 236),\n",
       "             ('python', 235),\n",
       "             ('manual', 235),\n",
       "             ('coverage', 234),\n",
       "             ('named', 234),\n",
       "             ('clicking', 234),\n",
       "             ('recommendations', 233),\n",
       "             ('broad', 233),\n",
       "             ('civil', 233),\n",
       "             ('handle', 232),\n",
       "             ('efforts', 232),\n",
       "             ('magazine', 232),\n",
       "             ('creation', 231),\n",
       "             ('interact', 231),\n",
       "             ('three', 231),\n",
       "             ('ingles', 230),\n",
       "             ('visit', 229),\n",
       "             ('admired', 228),\n",
       "             ('talent', 228),\n",
       "             ('unique', 228),\n",
       "             ('bring', 228),\n",
       "             ('participates', 227),\n",
       "             ('collaborative', 227),\n",
       "             ('generation', 226),\n",
       "             ('skill', 226),\n",
       "             ('talented', 225),\n",
       "             ('monitor', 224),\n",
       "             ('along', 224),\n",
       "             ('judgment', 224),\n",
       "             ('reference', 223),\n",
       "             ('performed', 223),\n",
       "             ('solve', 221),\n",
       "             ('object', 221),\n",
       "             ('according', 220),\n",
       "             ('integrated', 220),\n",
       "             ('march', 219),\n",
       "             ('academic', 218),\n",
       "             ('next', 217),\n",
       "             ('improvements', 216),\n",
       "             ('frameworks', 216),\n",
       "             ('retaining', 215),\n",
       "             ('efficient', 214),\n",
       "             ('roles', 214),\n",
       "             ('interesados', 213),\n",
       "             ('setting', 213),\n",
       "             ('demands', 213),\n",
       "             ('troubleshooting', 213),\n",
       "             ('edge', 212),\n",
       "             ('task', 212),\n",
       "             ('exposure', 212),\n",
       "             ('assisting', 211),\n",
       "             ('priorities', 211),\n",
       "             ('postal', 211),\n",
       "             ('correo', 211),\n",
       "             ('scripting', 210),\n",
       "             ('branch', 210),\n",
       "             ('presentation', 210),\n",
       "             ('contribute', 209),\n",
       "             ('creative', 208),\n",
       "             ('billion', 208),\n",
       "             ('pre', 207),\n",
       "             ('achieve', 207),\n",
       "             ('enviar', 206),\n",
       "             ('agreeing', 206),\n",
       "             ('made', 205),\n",
       "             ('suite', 205),\n",
       "             ('independent', 205),\n",
       "             ('profile', 204),\n",
       "             ('zona', 201),\n",
       "             ('passionate', 198),\n",
       "             ('know', 196),\n",
       "             ('consistent', 196),\n",
       "             ('clear', 196),\n",
       "             ('involved', 196),\n",
       "             ('expert', 195),\n",
       "             ('premier', 193),\n",
       "             ('solicita', 191),\n",
       "             ('difference', 191),\n",
       "             ('scope', 189),\n",
       "             ('met', 189),\n",
       "             ('immediate', 187),\n",
       "             ('characteristic', 187),\n",
       "             ('get', 187),\n",
       "             ('close', 185),\n",
       "             ('discriminate', 183),\n",
       "             ('dental', 179),\n",
       "             ('northropgrumman', 178)])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "processed_bag_of_words, processed_features = create_bag_of_words(processed_corpus,stop_words=eng_stopwords)\n",
    "dict_processed_word_counts = get_word_counts(processed_bag_of_words, processed_features)\n",
    "dict_processed_word_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much better! Now let's see how this affects the topics that are produce. Though the top 20 words are likely to be in all the job ads. Let's add them to the stopwords to remove them as well. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "top_20_words = list(dict_processed_word_counts.keys())[:20]\n",
    "domain_specific_stopwords = eng_stopwords + top_20_words\n",
    "processed_bag_of_words, processed_features = create_bag_of_words(processed_corpus,\n",
    "                                                                 stop_words=domain_specific_stopwords)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('systems', 3103),\n",
       "             ('engineering', 3047),\n",
       "             ('services', 3043),\n",
       "             ('information', 2989),\n",
       "             ('support', 2955),\n",
       "             ('legal', 2816),\n",
       "             ('en', 2791),\n",
       "             ('status', 2757),\n",
       "             ('opportunity', 2734),\n",
       "             ('including', 2627),\n",
       "             ('working', 2559),\n",
       "             ('design', 2531),\n",
       "             ('company', 2528),\n",
       "             ('technical', 2487),\n",
       "             ('data', 2359),\n",
       "             ('provide', 2307),\n",
       "             ('time', 2297),\n",
       "             ('degree', 2276),\n",
       "             ('related', 2264),\n",
       "             ('new', 2242),\n",
       "             ('strong', 2206),\n",
       "             ('environment', 2105),\n",
       "             ('qualifications', 2103),\n",
       "             ('system', 2072),\n",
       "             ('compliance', 2019),\n",
       "             ('equal', 1993),\n",
       "             ('preferred', 1925),\n",
       "             ('location', 1899),\n",
       "             ('training', 1839),\n",
       "             ('employer', 1799),\n",
       "             ('product', 1795),\n",
       "             ('medical', 1786),\n",
       "             ('us', 1772),\n",
       "             ('responsibilities', 1770),\n",
       "             ('employment', 1732),\n",
       "             ('process', 1726),\n",
       "             ('program', 1724),\n",
       "             ('law', 1685),\n",
       "             ('disability', 1679),\n",
       "             ('service', 1667),\n",
       "             ('solutions', 1664),\n",
       "             ('customer', 1661),\n",
       "             ('communication', 1659),\n",
       "             ('procedures', 1652),\n",
       "             ('engineer', 1641),\n",
       "             ('project', 1622),\n",
       "             ('computer', 1608),\n",
       "             ('products', 1598),\n",
       "             ('apply', 1582),\n",
       "             ('analysis', 1576),\n",
       "             ('technology', 1562),\n",
       "             ('national', 1556),\n",
       "             ('la', 1545),\n",
       "             ('office', 1539),\n",
       "             ('high', 1536),\n",
       "             ('may', 1523),\n",
       "             ('level', 1521),\n",
       "             ('include', 1513),\n",
       "             ('applicants', 1495),\n",
       "             ('within', 1492),\n",
       "             ('tools', 1484),\n",
       "             ('state', 1474),\n",
       "             ('based', 1472),\n",
       "             ('develop', 1448),\n",
       "             ('performance', 1417),\n",
       "             ('ensure', 1387),\n",
       "             ('health', 1383),\n",
       "             ('responsible', 1373),\n",
       "             ('laboratory', 1371),\n",
       "             ('world', 1370),\n",
       "             ('minimum', 1362),\n",
       "             ('full', 1361),\n",
       "             ('customers', 1358),\n",
       "             ('protected', 1344),\n",
       "             ('duties', 1344),\n",
       "             ('excellent', 1337),\n",
       "             ('microsoft', 1336),\n",
       "             ('description', 1322),\n",
       "             ('education', 1320),\n",
       "             ('id', 1318),\n",
       "             ('department', 1305),\n",
       "             ('gender', 1303),\n",
       "             ('issues', 1291),\n",
       "             ('industry', 1290),\n",
       "             ('well', 1286),\n",
       "             ('application', 1285),\n",
       "             ('global', 1283),\n",
       "             ('part', 1270),\n",
       "             ('one', 1263),\n",
       "             ('perform', 1260),\n",
       "             ('security', 1258),\n",
       "             ('role', 1234),\n",
       "             ('applications', 1233),\n",
       "             ('able', 1233),\n",
       "             ('processes', 1231),\n",
       "             ('review', 1220),\n",
       "             ('veteran', 1220),\n",
       "             ('teams', 1213),\n",
       "             ('title', 1212),\n",
       "             ('activities', 1204),\n",
       "             ('race', 1185),\n",
       "             ('care', 1176),\n",
       "             ('professional', 1174),\n",
       "             ('research', 1169),\n",
       "             ('use', 1169),\n",
       "             ('origin', 1168),\n",
       "             ('religion', 1166),\n",
       "             ('candidate', 1162),\n",
       "             ('please', 1162),\n",
       "             ('color', 1158),\n",
       "             ('projects', 1149),\n",
       "             ('organization', 1147),\n",
       "             ('science', 1130),\n",
       "             ('written', 1129),\n",
       "             ('understanding', 1120),\n",
       "             ('employees', 1104),\n",
       "             ('opportunities', 1097),\n",
       "             ('maintain', 1088),\n",
       "             ('orientation', 1073),\n",
       "             ('people', 1070),\n",
       "             ('following', 1064),\n",
       "             ('age', 1063),\n",
       "             ('sexual', 1056),\n",
       "             ('standards', 1053),\n",
       "             ('control', 1038),\n",
       "             ('without', 1038),\n",
       "             ('also', 1032),\n",
       "             ('financial', 1028),\n",
       "             ('clients', 1025),\n",
       "             ('risk', 1022),\n",
       "             ('qualified', 1018),\n",
       "             ('benefits', 1010),\n",
       "             ('code', 1008),\n",
       "             ('leading', 994),\n",
       "             ('hours', 994),\n",
       "             ('functional', 990),\n",
       "             ('career', 987),\n",
       "             ('complex', 977),\n",
       "             ('web', 977),\n",
       "             ('identity', 976),\n",
       "             ('com', 972),\n",
       "             ('good', 964),\n",
       "             ('results', 964),\n",
       "             ('assigned', 961),\n",
       "             ('reports', 954),\n",
       "             ('con', 952),\n",
       "             ('basic', 949),\n",
       "             ('documentation', 949),\n",
       "             ('etc', 946),\n",
       "             ('life', 945),\n",
       "             ('using', 943),\n",
       "             ('technologies', 940),\n",
       "             ('action', 939),\n",
       "             ('regulatory', 937),\n",
       "             ('integration', 934),\n",
       "             ('providing', 925),\n",
       "             ('credit', 923),\n",
       "             ('join', 918),\n",
       "             ('internal', 915),\n",
       "             ('appropriate', 913),\n",
       "             ('plans', 907),\n",
       "             ('field', 906),\n",
       "             ('areas', 903),\n",
       "             ('candidates', 901),\n",
       "             ('insurance', 898),\n",
       "             ('regard', 898),\n",
       "             ('client', 897),\n",
       "             ('sales', 881),\n",
       "             ('looking', 879),\n",
       "             ('best', 879),\n",
       "             ('el', 878),\n",
       "             ('general', 872),\n",
       "             ('plus', 872),\n",
       "             ('sex', 868),\n",
       "             ('group', 865),\n",
       "             ('clinical', 864),\n",
       "             ('professionals', 861),\n",
       "             ('operations', 859),\n",
       "             ('experiencia', 857),\n",
       "             ('area', 851),\n",
       "             ('senior', 845),\n",
       "             ('make', 845),\n",
       "             ('across', 842),\n",
       "             ('developing', 834),\n",
       "             ('corporate', 834),\n",
       "             ('equivalent', 826),\n",
       "             ('needs', 815),\n",
       "             ('assurance', 811),\n",
       "             ('plan', 802),\n",
       "             ('highly', 797),\n",
       "             ('patient', 789),\n",
       "             ('functions', 786),\n",
       "             ('healthcare', 785),\n",
       "             ('equipment', 782),\n",
       "             ('companies', 777),\n",
       "             ('lab', 768),\n",
       "             ('assist', 762),\n",
       "             ('problems', 761),\n",
       "             ('compensation', 758),\n",
       "             ('java', 758),\n",
       "             ('year', 756),\n",
       "             ('states', 754),\n",
       "             ('assistant', 750),\n",
       "             ('practices', 745),\n",
       "             ('complete', 739),\n",
       "             ('federal', 738),\n",
       "             ('robert', 737),\n",
       "             ('half', 736),\n",
       "             ('basis', 731),\n",
       "             ('manager', 730),\n",
       "             ('staff', 729),\n",
       "             ('provides', 726),\n",
       "             ('policies', 726),\n",
       "             ('programs', 723),\n",
       "             ('employee', 722),\n",
       "             ('effectively', 722),\n",
       "             ('center', 721),\n",
       "             ('documents', 708),\n",
       "             ('requisition', 704),\n",
       "             ('reviews', 703),\n",
       "             ('multiple', 703),\n",
       "             ('essential', 702),\n",
       "             ('help', 702),\n",
       "             ('lead', 689),\n",
       "             ('meet', 689),\n",
       "             ('programming', 685),\n",
       "             ('various', 681),\n",
       "             ('united', 677),\n",
       "             ('personal', 674),\n",
       "             ('create', 674),\n",
       "             ('diverse', 673),\n",
       "             ('implementation', 667),\n",
       "             ('specific', 666),\n",
       "             ('problem', 663),\n",
       "             ('manage', 663),\n",
       "             ('tests', 660),\n",
       "             ('eeo', 660),\n",
       "             ('planning', 660),\n",
       "             ('key', 659),\n",
       "             ('commercial', 659),\n",
       "             ('counsel', 654),\n",
       "             ('leadership', 654),\n",
       "             ('resources', 654),\n",
       "             ('contact', 651),\n",
       "             ('individuals', 651),\n",
       "             ('affirmative', 650),\n",
       "             ('analyst', 645),\n",
       "             ('learn', 645),\n",
       "             ('demonstrated', 643),\n",
       "             ('patients', 643),\n",
       "             ('automation', 642),\n",
       "             ('analytical', 641),\n",
       "             ('additional', 641),\n",
       "             ('verbal', 639),\n",
       "             ('consideration', 639),\n",
       "             ('local', 638),\n",
       "             ('online', 637),\n",
       "             ('competitive', 636),\n",
       "             ('individual', 633),\n",
       "             ('marital', 631),\n",
       "             ('offer', 631),\n",
       "             ('build', 630),\n",
       "             ('type', 626),\n",
       "             ('partner', 623),\n",
       "             ('range', 623),\n",
       "             ('school', 622),\n",
       "             ('regulations', 622),\n",
       "             ('qa', 621),\n",
       "             ('current', 620),\n",
       "             ('matters', 619),\n",
       "             ('travel', 619),\n",
       "             ('committed', 615),\n",
       "             ('bachelor', 614),\n",
       "             ('communicate', 610),\n",
       "             ('tasks', 607),\n",
       "             ('independently', 606),\n",
       "             ('engineers', 603),\n",
       "             ('relevant', 598),\n",
       "             ('cloud', 597),\n",
       "             ('access', 592),\n",
       "             ('identify', 586),\n",
       "             ('pruebas', 585),\n",
       "             ('members', 585),\n",
       "             ('day', 585),\n",
       "             ('para', 584),\n",
       "             ('performs', 581),\n",
       "             ('needed', 581),\n",
       "             ('end', 579),\n",
       "             ('participate', 578),\n",
       "             ('automated', 578),\n",
       "             ('per', 577),\n",
       "             ('proven', 576),\n",
       "             ('understand', 573),\n",
       "             ('production', 572),\n",
       "             ('self', 570),\n",
       "             ('processing', 570),\n",
       "             ('network', 570),\n",
       "             ('mobile', 569),\n",
       "             ('document', 568),\n",
       "             ('receive', 565),\n",
       "             ('non', 564),\n",
       "             ('cases', 564),\n",
       "             ('policy', 562),\n",
       "             ('applicable', 561),\n",
       "             ('maintenance', 561),\n",
       "             ('oriented', 561),\n",
       "             ('safety', 560),\n",
       "             ('writing', 559),\n",
       "             ('successful', 557),\n",
       "             ('class', 557),\n",
       "             ('solving', 556),\n",
       "             ('necessary', 555),\n",
       "             ('improve', 555),\n",
       "             ('applying', 552),\n",
       "             ('partners', 552),\n",
       "             ('external', 549),\n",
       "             ('schedule', 546),\n",
       "             ('word', 546),\n",
       "             ('building', 545),\n",
       "             ('others', 544),\n",
       "             ('advanced', 544),\n",
       "             ('certification', 541),\n",
       "             ('expertise', 541),\n",
       "             ('sql', 541),\n",
       "             ('mission', 539),\n",
       "             ('drive', 539),\n",
       "             ('deliver', 537),\n",
       "             ('background', 536),\n",
       "             ('delivery', 535),\n",
       "             ('date', 533),\n",
       "             ('agile', 533),\n",
       "             ('city', 531),\n",
       "             ('user', 529),\n",
       "             ('desired', 525),\n",
       "             ('order', 524),\n",
       "             ('requires', 523),\n",
       "             ('reporting', 523),\n",
       "             ('seeking', 523),\n",
       "             ('excel', 523),\n",
       "             ('trabajo', 523),\n",
       "             ('international', 520),\n",
       "             ('performing', 513),\n",
       "             ('los', 512),\n",
       "             ('summary', 510),\n",
       "             ('ms', 508),\n",
       "             ('physical', 507),\n",
       "             ('making', 507),\n",
       "             ('positions', 507),\n",
       "             ('que', 505),\n",
       "             ('salary', 504),\n",
       "             ('attorney', 503),\n",
       "             ('every', 502),\n",
       "             ('clearance', 501),\n",
       "             ('unit', 501),\n",
       "             ('government', 500),\n",
       "             ('finance', 499),\n",
       "             ('category', 498),\n",
       "             ('bachelors', 498),\n",
       "             ('strategy', 497),\n",
       "             ('openings', 497),\n",
       "             ('leader', 496),\n",
       "             ('electrical', 495),\n",
       "             ('execution', 493),\n",
       "             ('genetic', 492),\n",
       "             ('primary', 490),\n",
       "             ('citizenship', 489),\n",
       "             ('marketing', 489),\n",
       "             ('las', 488),\n",
       "             ('effective', 487),\n",
       "             ('por', 485),\n",
       "             ('available', 484),\n",
       "             ('need', 484),\n",
       "             ('locations', 482),\n",
       "             ('supporting', 482),\n",
       "             ('multi', 481),\n",
       "             ('large', 481),\n",
       "             ('major', 480),\n",
       "             ('innovative', 478),\n",
       "             ('personnel', 478),\n",
       "             ('techniques', 478),\n",
       "             ('authorized', 476),\n",
       "             ('fast', 476),\n",
       "             ('least', 472),\n",
       "             ('conduct', 472),\n",
       "             ('windows', 472),\n",
       "             ('www', 471),\n",
       "             ('relationships', 470),\n",
       "             ('anos', 470),\n",
       "             ('variety', 467),\n",
       "             ('hardware', 466),\n",
       "             ('accounting', 463),\n",
       "             ('member', 463),\n",
       "             ('operating', 462),\n",
       "             ('firm', 460),\n",
       "             ('worldwide', 459),\n",
       "             ('works', 458),\n",
       "             ('resume', 454),\n",
       "             ('enterprise', 452),\n",
       "             ('decisions', 451),\n",
       "             ('country', 449),\n",
       "             ('includes', 449),\n",
       "             ('linux', 449),\n",
       "             ('english', 449),\n",
       "             ('detail', 448),\n",
       "             ('al', 447),\n",
       "             ('hands', 445),\n",
       "             ('wide', 445),\n",
       "             ('college', 445),\n",
       "             ('change', 441),\n",
       "             ('experienced', 439),\n",
       "             ('like', 437),\n",
       "             ('hiring', 436),\n",
       "             ('disabilities', 433),\n",
       "             ('initiatives', 432),\n",
       "             ('prior', 432),\n",
       "             ('practice', 430),\n",
       "             ('northrop', 429),\n",
       "             ('community', 429),\n",
       "             ('req', 429),\n",
       "             ('litigation', 428),\n",
       "             ('number', 427),\n",
       "             ('focus', 427),\n",
       "             ('staffing', 426),\n",
       "             ('familiarity', 426),\n",
       "             ('grumman', 424),\n",
       "             ('vision', 422),\n",
       "             ('statement', 421),\n",
       "             ('site', 421),\n",
       "             ('operational', 419),\n",
       "             ('del', 418),\n",
       "             ('completion', 418),\n",
       "             ('implement', 418),\n",
       "             ('success', 417),\n",
       "             ('organizational', 417),\n",
       "             ('shift', 416),\n",
       "             ('function', 416),\n",
       "             ('impact', 416),\n",
       "             ('open', 415),\n",
       "             ('experiences', 415),\n",
       "             ('located', 414),\n",
       "             ('creed', 413),\n",
       "             ('timely', 410),\n",
       "             ('standard', 410),\n",
       "             ('manufacturing', 410),\n",
       "             ('oral', 406),\n",
       "             ('fortune', 405),\n",
       "             ('take', 404),\n",
       "             ('communications', 404),\n",
       "             ('market', 404),\n",
       "             ('execute', 404),\n",
       "             ('provider', 403),\n",
       "             ('validation', 403),\n",
       "             ('levels', 403),\n",
       "             ('interpersonal', 402),\n",
       "             ('assistance', 401),\n",
       "             ('top', 399),\n",
       "             ('record', 399),\n",
       "             ('report', 399),\n",
       "             ('comprehensive', 398),\n",
       "             ('expression', 398),\n",
       "             ('continuous', 398),\n",
       "             ('list', 397),\n",
       "             ('around', 397),\n",
       "             ('countries', 397),\n",
       "             ('prepare', 395),\n",
       "             ('analyze', 395),\n",
       "             ('case', 393),\n",
       "             ('cross', 391),\n",
       "             ('used', 390),\n",
       "             ('proud', 390),\n",
       "             ('closely', 390),\n",
       "             ('right', 389),\n",
       "             ('platform', 389),\n",
       "             ('contract', 387),\n",
       "             ('culture', 386),\n",
       "             ('methodologies', 385),\n",
       "             ('interested', 384),\n",
       "             ('throughout', 384),\n",
       "             ('existing', 384),\n",
       "             ('materials', 383),\n",
       "             ('considered', 381),\n",
       "             ('package', 380),\n",
       "             ('abilities', 379),\n",
       "             ('goals', 377),\n",
       "             ('architecture', 376),\n",
       "             ('power', 375),\n",
       "             ('email', 375),\n",
       "             ('creating', 375),\n",
       "             ('follow', 374),\n",
       "             ('place', 373),\n",
       "             ('guidance', 372),\n",
       "             ('specifications', 372),\n",
       "             ('value', 371),\n",
       "             ('develops', 370),\n",
       "             ('free', 370),\n",
       "             ('tester', 369),\n",
       "             ('limited', 367),\n",
       "             ('database', 367),\n",
       "             ('commitment', 367),\n",
       "             ('critical', 367),\n",
       "             ('established', 365),\n",
       "             ('manner', 365),\n",
       "             ('requests', 362),\n",
       "             ('obtain', 362),\n",
       "             ('capabilities', 362),\n",
       "             ('innovation', 362),\n",
       "             ('details', 361),\n",
       "             ('markets', 360),\n",
       "             ('empresa', 359),\n",
       "             ('future', 359),\n",
       "             ('would', 357),\n",
       "             ('possess', 357),\n",
       "             ('set', 356),\n",
       "             ('directly', 355),\n",
       "             ('laws', 354),\n",
       "             ('supervision', 353),\n",
       "             ('methods', 353),\n",
       "             ('first', 353),\n",
       "             ('veterans', 353),\n",
       "             ('driven', 352),\n",
       "             ('supports', 352),\n",
       "             ('together', 352),\n",
       "             ('organizations', 352),\n",
       "             ('act', 352),\n",
       "             ('workforce', 351),\n",
       "             ('great', 351),\n",
       "             ('human', 350),\n",
       "             ('managers', 349),\n",
       "             ('posting', 348),\n",
       "             ('growth', 348),\n",
       "             ('overview', 348),\n",
       "             ('managing', 347),\n",
       "             ('direct', 346),\n",
       "             ('strategic', 345),\n",
       "             ('administration', 344),\n",
       "             ('active', 343),\n",
       "             ('server', 343),\n",
       "             ('maintaining', 341),\n",
       "             ('paced', 340),\n",
       "             ('responsibility', 339),\n",
       "             ('line', 338),\n",
       "             ('collection', 338),\n",
       "             ('eligible', 336),\n",
       "             ('two', 335),\n",
       "             ('public', 334),\n",
       "             ('ideal', 334),\n",
       "             ('daily', 333),\n",
       "             ('devices', 332),\n",
       "             ('pm', 332),\n",
       "             ('improvement', 331),\n",
       "             ('currently', 330),\n",
       "             ('license', 330),\n",
       "             ('strategies', 328),\n",
       "             ('addition', 328),\n",
       "             ('quickly', 327),\n",
       "             ('controls', 327),\n",
       "             ('concepts', 327),\n",
       "             ('cv', 326),\n",
       "             ('source', 325),\n",
       "             ('challenging', 324),\n",
       "             ('departments', 324),\n",
       "             ('collaborate', 323),\n",
       "             ('learning', 323),\n",
       "             ('inc', 323),\n",
       "             ('infrastructure', 322),\n",
       "             ('drug', 321),\n",
       "             ('co', 321),\n",
       "             ('assessment', 320),\n",
       "             ('regular', 320),\n",
       "             ('potential', 320),\n",
       "             ('call', 320),\n",
       "             ('electronic', 319),\n",
       "             ('motivated', 319),\n",
       "             ('passion', 319),\n",
       "             ('interface', 318),\n",
       "             ('attention', 318),\n",
       "             ('principles', 318),\n",
       "             ('north', 317),\n",
       "             ('special', 317),\n",
       "             ('solid', 317),\n",
       "             ('languages', 316),\n",
       "             ('verification', 316),\n",
       "             ('accordance', 315),\n",
       "             ('ley', 315),\n",
       "             ('meetings', 314),\n",
       "             ('changes', 314),\n",
       "             ('agency', 314),\n",
       "             ('components', 314),\n",
       "             ('bs', 313),\n",
       "             ('expected', 313),\n",
       "             ('university', 313),\n",
       "             ('delivering', 312),\n",
       "             ('records', 312),\n",
       "             ('direction', 312),\n",
       "             ('consulting', 311),\n",
       "             ('search', 310),\n",
       "             ('reasonable', 309),\n",
       "             ('accredited', 309),\n",
       "             ('objectives', 309),\n",
       "             ('outside', 308),\n",
       "             ('canada', 307),\n",
       "             ('terms', 305),\n",
       "             ('similar', 305),\n",
       "             ('aspects', 305),\n",
       "             ('temporary', 305),\n",
       "             ('prestaciones', 305),\n",
       "             ('language', 304),\n",
       "             ('audit', 303),\n",
       "             ('subject', 303),\n",
       "             ('usa', 303),\n",
       "             ('serve', 303),\n",
       "             ('solution', 303),\n",
       "             ('core', 302),\n",
       "             ('users', 301),\n",
       "             ('associated', 300),\n",
       "             ('division', 299),\n",
       "             ('growing', 298),\n",
       "             ('conditions', 298),\n",
       "             ('collaboration', 296),\n",
       "             ('days', 295),\n",
       "             ('specialist', 295),\n",
       "             ('resolution', 294),\n",
       "             ('offers', 293),\n",
       "             ('post', 293),\n",
       "             ('pay', 292),\n",
       "             ('successfully', 292),\n",
       "             ('jobs', 292),\n",
       "             ('driving', 292),\n",
       "             ('developers', 291),\n",
       "             ('overall', 291),\n",
       "             ('person', 291),\n",
       "             ('tracking', 290),\n",
       "             ('scripts', 290),\n",
       "             ('track', 290),\n",
       "             ('groups', 290),\n",
       "             ('scale', 289),\n",
       "             ('accurate', 289),\n",
       "             ('monitoring', 289),\n",
       "             ('coordinate', 289),\n",
       "             ('lives', 288),\n",
       "             ('advance', 288),\n",
       "             ('framework', 287),\n",
       "             ('verify', 287),\n",
       "             ('javascript', 287),\n",
       "             ('cost', 287),\n",
       "             ('better', 286),\n",
       "             ('submit', 285),\n",
       "             ('skilled', 285),\n",
       "             ('regarding', 285),\n",
       "             ('instructions', 284),\n",
       "             ('discipline', 283),\n",
       "             ('associate', 281),\n",
       "             ('evaluation', 280),\n",
       "             ('real', 280),\n",
       "             ('handling', 280),\n",
       "             ('content', 280),\n",
       "             ('features', 279),\n",
       "             ('require', 279),\n",
       "             ('sueldo', 278),\n",
       "             ('master', 278),\n",
       "             ('challenges', 278),\n",
       "             ('ensuring', 276),\n",
       "             ('many', 275),\n",
       "             ('enable', 275),\n",
       "             ('implementing', 275),\n",
       "             ('sistemas', 274),\n",
       "             ('graduate', 274),\n",
       "             ('administrative', 274),\n",
       "             ('ofrecemos', 274),\n",
       "             ('accounts', 273),\n",
       "             ('exciting', 272),\n",
       "             ('week', 272),\n",
       "             ('completed', 272),\n",
       "             ('cycle', 271),\n",
       "             ('proficient', 271),\n",
       "             ('vet', 270),\n",
       "             ('facility', 270),\n",
       "             ('leaders', 270),\n",
       "             ('platforms', 270),\n",
       "             ('diploma', 269),\n",
       "             ('maintains', 269),\n",
       "             ('assists', 268),\n",
       "             ('base', 268),\n",
       "             ('diversity', 268),\n",
       "             ('resolve', 267),\n",
       "             ('dynamic', 267),\n",
       "             ('ca', 266),\n",
       "             ('grow', 266),\n",
       "             ('escolaridad', 265),\n",
       "             ('release', 265),\n",
       "             ('proficiency', 263),\n",
       "             ('specialty', 263),\n",
       "             ('guidelines', 261),\n",
       "             ('feedback', 261),\n",
       "             ('dedicated', 261),\n",
       "             ('excellence', 261),\n",
       "             ('ensures', 261),\n",
       "             ('device', 261),\n",
       "             ('environments', 260),\n",
       "             ('facilities', 260),\n",
       "             ('coding', 259),\n",
       "             ('today', 259),\n",
       "             ('conocimientos', 259),\n",
       "             ('designs', 259),\n",
       "             ('small', 259),\n",
       "             ('agencies', 258),\n",
       "             ('br', 257),\n",
       "             ('communities', 257),\n",
       "             ('focused', 256),\n",
       "             ('flexible', 256),\n",
       "             ('previous', 256),\n",
       "             ('requisitos', 256),\n",
       "             ('preparation', 256),\n",
       "             ('upon', 255),\n",
       "             ('demonstrate', 255),\n",
       "             ('paid', 255),\n",
       "             ('http', 255),\n",
       "             ('canadian', 254),\n",
       "             ('write', 254),\n",
       "             ('entry', 254),\n",
       "             ('questions', 254),\n",
       "             ('way', 253),\n",
       "             ('risks', 253),\n",
       "             ('determine', 253),\n",
       "             ('significant', 252),\n",
       "             ('evaluate', 250),\n",
       "             ('changing', 250),\n",
       "             ('investigation', 249),\n",
       "             ('largest', 249),\n",
       "             ('months', 248),\n",
       "             ('firms', 248),\n",
       "             ('protocols', 247),\n",
       "             ('home', 247),\n",
       "             ('accuracy', 247),\n",
       "             ('html', 246),\n",
       "             ('operate', 245),\n",
       "             ('portfolio', 245),\n",
       "             ('ii', 245),\n",
       "             ('military', 244),\n",
       "             ('debugging', 244),\n",
       "             ('designing', 244),\n",
       "             ('board', 242),\n",
       "             ('aa', 242),\n",
       "             ('different', 242),\n",
       "             ('find', 242),\n",
       "             ('positive', 240),\n",
       "             ('prioritize', 239),\n",
       "             ('higher', 238),\n",
       "             ('hire', 237),\n",
       "             ('decision', 237),\n",
       "             ('extensive', 237),\n",
       "             ('computing', 237),\n",
       "             ('long', 236),\n",
       "             ('desirable', 236),\n",
       "             ('want', 236),\n",
       "             ('workplace', 236),\n",
       "             ('share', 236),\n",
       "             ('python', 235),\n",
       "             ('manual', 235),\n",
       "             ('coverage', 234),\n",
       "             ('named', 234),\n",
       "             ('clicking', 234),\n",
       "             ('recommendations', 233),\n",
       "             ('broad', 233),\n",
       "             ('civil', 233),\n",
       "             ('handle', 232),\n",
       "             ('efforts', 232),\n",
       "             ('magazine', 232),\n",
       "             ('creation', 231),\n",
       "             ('interact', 231),\n",
       "             ('three', 231),\n",
       "             ('ingles', 230),\n",
       "             ('visit', 229),\n",
       "             ('admired', 228),\n",
       "             ('talent', 228),\n",
       "             ('unique', 228),\n",
       "             ('bring', 228),\n",
       "             ('participates', 227),\n",
       "             ('collaborative', 227),\n",
       "             ('generation', 226),\n",
       "             ('skill', 226),\n",
       "             ('talented', 225),\n",
       "             ('monitor', 224),\n",
       "             ('along', 224),\n",
       "             ('judgment', 224),\n",
       "             ('reference', 223),\n",
       "             ('performed', 223),\n",
       "             ('solve', 221),\n",
       "             ('object', 221),\n",
       "             ('according', 220),\n",
       "             ('integrated', 220),\n",
       "             ('march', 219),\n",
       "             ('academic', 218),\n",
       "             ('next', 217),\n",
       "             ('improvements', 216),\n",
       "             ('frameworks', 216),\n",
       "             ('retaining', 215),\n",
       "             ('efficient', 214),\n",
       "             ('roles', 214),\n",
       "             ('interesados', 213),\n",
       "             ('setting', 213),\n",
       "             ('demands', 213),\n",
       "             ('troubleshooting', 213),\n",
       "             ('edge', 212),\n",
       "             ('task', 212),\n",
       "             ('exposure', 212),\n",
       "             ('assisting', 211),\n",
       "             ('priorities', 211),\n",
       "             ('postal', 211),\n",
       "             ('correo', 211),\n",
       "             ('scripting', 210),\n",
       "             ('branch', 210),\n",
       "             ('presentation', 210),\n",
       "             ('contribute', 209),\n",
       "             ('creative', 208),\n",
       "             ('billion', 208),\n",
       "             ('pre', 207),\n",
       "             ('achieve', 207),\n",
       "             ('enviar', 206),\n",
       "             ('agreeing', 206),\n",
       "             ('made', 205),\n",
       "             ('suite', 205),\n",
       "             ('independent', 205),\n",
       "             ('profile', 204),\n",
       "             ('zona', 201),\n",
       "             ('passionate', 198),\n",
       "             ('know', 196),\n",
       "             ('consistent', 196),\n",
       "             ('clear', 196),\n",
       "             ('involved', 196),\n",
       "             ('expert', 195),\n",
       "             ('premier', 193),\n",
       "             ('solicita', 191),\n",
       "             ('difference', 191),\n",
       "             ('scope', 189),\n",
       "             ('met', 189),\n",
       "             ('immediate', 187),\n",
       "             ('characteristic', 187),\n",
       "             ('get', 187),\n",
       "             ('close', 185),\n",
       "             ('discriminate', 183),\n",
       "             ('dental', 179),\n",
       "             ('northropgrumman', 178)])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dict_processed_word_counts = get_word_counts(processed_bag_of_words, processed_features)\n",
    "dict_processed_word_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a bit better. Let's see what topics we produce."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "| 2 Elapsed Time: 0:00:10                                                                                                                                              \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 en, la, con, el, experiencia\n",
      "1 systems, engineering, status, design, information\n",
      "2 legal, compliance, company, law, apply\n"
     ]
    }
   ],
   "source": [
    "processed_keywords, processed_doctopic = create_topics(processed_bag_of_words, \n",
    "                                                       processed_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are starting to get somewhere! There are a lot of jobs that have to do with the law and engineering and medicine. We should increase the number of topics and words for each topics to see if we can understand more from our corpus. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "| 4 Elapsed Time: 0:00:12                                                                                                                                              \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 medical, laboratory, health, care, hours, services, duties, time, training, clinical\n",
      "1 compliance, including, support, related, ensure, risk, strong, company, process, responsibilities\n",
      "2 engineering, systems, design, status, technical, engineer, data, solutions, system, opportunity\n",
      "3 legal, robert, half, apply, law, compensation, companies, professionals, company, applying\n",
      "4 en, la, con, el, experiencia, pruebas, para, trabajo, que, las\n"
     ]
    }
   ],
   "source": [
    "processed_keywords, processed_doctopic = create_topics(processed_bag_of_words, \n",
    "                                                       processed_features, \n",
    "                                                      N_TOPICS = 5,\n",
    "                                                      N_TOP_WORDS= 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding more topics has revealed to larger subtopics. Let's see if adding 10 topics will tell us more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "| 9 Elapsed Time: 0:00:15                                                                                                                                              \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 status, gender, protected, race, origin, religion, disability, color, opportunity, sexual, national, orientation, veteran, equal, identity\n",
      "1 medical, laboratory, health, care, clinical, patient, services, lab, healthcare, patients, procedures, training, assistant, duties, hours\n",
      "2 systems, engineering, security, system, engineer, design, support, technical, integration, clearance, qualifications, electrical, computer, candidate, degree\n",
      "3 half, robert, credit, apply, compensation, applying, openings, authorized, applicants, companies, office, professionals, opportunity, provide, world\n",
      "4 legal, law, counsel, corporate, matters, attorney, litigation, including, issues, regulatory, company, global, firm, provide, practice\n",
      "5 compliance, related, ensure, project, process, support, risk, including, procedures, analysis, processes, responsibilities, information, activities, reports\n",
      "6 en, la, con, el, experiencia, pruebas, para, trabajo, que, las, por, anos, los, al, del\n",
      "7 design, new, data, technical, technology, product, tools, engineer, services, working, strong, technologies, environment, automation, web\n",
      "8 sales, us, company, career, insurance, clients, success, life, opportunities, personal, partner, people, industry, client, program\n",
      "9 time, research, please, apply, may, department, employment, application, hours, location, full, information, applicants, opportunity, description\n"
     ]
    }
   ],
   "source": [
    "processed_keywords, processed_doctopic = create_topics(processed_bag_of_words, \n",
    "                                                       processed_features, \n",
    "                                                      N_TOPICS = 10,\n",
    "                                                      N_TOP_WORDS= 15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like we have a good amount of topics. Some of the top words are quite similiar such as engineering and enginner. We can reduce those words to its stem to further refine our features. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stemming and lemmitzation\n",
    "\n",
    "We can further process our text through *stemming and lemmatization*. Words can take on muliple forms with limited change to their meaning. For example \"systems\", \"systematic\" and \"system\" are all different words but they all have the same meeting. We can replace all these words with system witout losing any meaning. The lemma is the original dictionary form of a word (e.g. lying and lie). There are several well known stemming algorithms -- Porter, Snowball, Lancaster -- that all have strengths and weakneses. For this tutorial we are using the Porter Stemmer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lie\n",
      "lie\n",
      "systemat\n",
      "run\n"
     ]
    }
   ],
   "source": [
    "stemmer = PorterStemmer()\n",
    "print(stemmer.stem('lies'))\n",
    "print(stemmer.stem(\"lying\"))\n",
    "print(stemmer.stem('systematic'))\n",
    "print(stemmer.stem(\"running\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "| 9 Elapsed Time: 0:00:20                                                                                                                                              \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 legal, busi, complianc, experi, manag, law, abil, skill, credit, includ, risk, counsel, financi, regulatori, corpor\n",
      "1 insur, provid, compani, benefit, medic, healthcar, employe, life, legal, case, health, requir, compens, vet, canadian\n",
      "2 appli, half, robert, thi, applic, compani, state, opportun, experi, author, legal, open, candid, provid, compens\n",
      "3 requir, posit, servic, patient, laboratori, assist, provid, medic, care, clinic, hour, experi, perform, thi, health\n",
      "4 engin, statu, product, employ, experi, requir, disabl, nation, protect, thi, veteran, perform, gender, skill, develop\n",
      "5 softwar, system, experi, develop, engin, requir, secur, support, applic, program, year, design, integr, c, oper\n",
      "6 en, la, con, el, experiencia, prueba, para, ano, lo, trabajo, al, que, conocimiento, por, empresa\n",
      "7 busi, manag, client, market, sale, opportun, servic, partner, us, program, develop, career, intern, skill, global\n",
      "8 develop, experi, softwar, technolog, engin, product, design, skill, autom, servic, applic, qualiti, data, technic, tool\n",
      "9 requir, manag, qualiti, process, ensur, document, report, perform, plan, abil, experi, review, project, respons, knowledg\n"
     ]
    }
   ],
   "source": [
    "processed_bag_of_words, processed_features = create_bag_of_words(processed_corpus,\n",
    "                                                                 stop_words=domain_specific_stopwords,\n",
    "                                                                 stem=True)\n",
    "processed_keywords, processed_doctopic = create_topics(processed_bag_of_words, \n",
    "                                                       processed_features, \n",
    "                                                      N_TOPICS = 10,\n",
    "                                                      N_TOP_WORDS= 15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not it appears we have picked up some extra topics that describe the educational requirements for a job ad or the equal opportunity clause of a job ad. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### N-grams\n",
    "\n",
    "Individual words are not always the the correct unit of analysis. Prematurely removing stopwords can lead to losing pharases such as \"kick the bucket\", \"commander in chief\", or \"sleeps with the fishes\". Idenitfying these N-grams requires looking for patterns of words that often appear together in fixed patterns. \n",
    "\n",
    "Now let's transform our corpus into a bag of n-grams which in this case is a bag of bi-grams or bag of 2-grams. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "| 9 Elapsed Time: 0:00:20                                                                                                                                              \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 develop, experi, softwar, engin, product, qualiti, design, skill, requir, system, technolog, project, technic, knowledg, manag\n",
      "1 applic, appli, half, thi, state, pleas, posit, candid, employ, experi, applic appli, open, robert, time, opportun\n",
      "2 robert half, compani, compens, appli, provid, open author, robert, legal, opportun, profession, thi, world, role, client, employ f\n",
      "3 medic, patient, laboratori, provid, servic, requir, health, care, clinic, assist, hour, lab, train, posit, healthcar\n",
      "4 system, engin, experi, softwar, develop, requir, secur, support, program, year, design, applic, integr, candid, clearanc\n",
      "5 legal, busi, manag, law, experi, complianc, compani, includ, provid, skill, abil, corpor, intern, counsel, support\n",
      "6 statu, experi, employ, develop, gender, engin, opportun, race, religion, protect, nation origin, nation, color, origin, orient\n",
      "7 requir, abil, perform, manag, posit, document, process, procedur, duti, report, review, skill, experi, qualiti, thi\n",
      "8 credit, financi, account, analyst, bank, client, risk, experi, financ, analysi, busi, custom, servic, manag, degre\n",
      "9 en, la, con, el, experiencia, prueba, para, ano, lo, trabajo, que, al, conocimiento, por, empresa\n"
     ]
    }
   ],
   "source": [
    "processed_bag_of_words, processed_features = create_bag_of_words(processed_corpus,\n",
    "                                                                 stop_words=domain_specific_stopwords,\n",
    "                                                                 stem=True,\n",
    "                                                                 NGRAM_RANGE=(0,2))\n",
    "processed_keywords, processed_doctopic = create_topics(processed_bag_of_words, \n",
    "                                                       processed_features, \n",
    "                                                      N_TOPICS = 10,\n",
    "                                                      N_TOP_WORDS= 15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice one of the top words in one of the topics is \"northrop grumman\", a bi-gram!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TFIDF (Term Frequency Inverse Document Frequency)\n",
    "\n",
    "A final step in cleaning and processing our text data is TFIDF. TFIDF (Term frequency-inverse document frequency) is an example of feature enginnering where the most important words are extracted by taking account their frequency in documents and the entire corpus of documents as a whole. Words that appear in all documents are deemphazized while more meaningful words are emphaized. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "processed_bag_of_words, processed_features = create_bag_of_words(processed_corpus,\n",
    "                                                                 stop_words=domain_specific_stopwords,\n",
    "                                                                 stem=True,\n",
    "                                                                 NGRAM_RANGE=(0,2),\n",
    "                                                                 USE_IDF = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dict_word_counts = get_word_counts(processed_bag_of_words,\n",
    "                   processed_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('experi', 7311.3416952196339),\n",
       "             ('requir', 6500.8966223741509),\n",
       "             ('develop', 6488.0914391328433),\n",
       "             ('skill', 5593.6453202319126),\n",
       "             ('engin', 5455.9568682587324),\n",
       "             ('manag', 5401.7337516352773),\n",
       "             ('softwar', 5263.6749205370761),\n",
       "             ('system', 5223.212599421202),\n",
       "             ('provid', 5179.3108430063685),\n",
       "             ('thi', 5025.0637681779353),\n",
       "             ('servic', 4995.3657639684407),\n",
       "             ('abil', 4949.3012921033323),\n",
       "             ('includ', 4947.7082701889403),\n",
       "             ('applic', 4895.5820176660418),\n",
       "             ('posit', 4797.8973715640514),\n",
       "             ('product', 4666.3367421137855),\n",
       "             ('busi', 4664.8722441135051),\n",
       "             ('perform', 4626.062694094865),\n",
       "             ('employ', 4591.426951549739),\n",
       "             ('support', 4565.6683880206911),\n",
       "             ('year', 4520.5171459340681),\n",
       "             ('opportun', 4509.7157830600036),\n",
       "             ('respons', 4509.5246722098609),\n",
       "             ('compani', 4425.4738630002421),\n",
       "             ('commun', 4379.0829615999692),\n",
       "             ('knowledg', 4370.7031878118223),\n",
       "             ('process', 4355.9202819446255),\n",
       "             ('design', 4270.5505833333627),\n",
       "             ('qualiti', 4241.5666037146502),\n",
       "             ('program', 4138.8969017915451),\n",
       "             ('custom', 4121.1059405317319),\n",
       "             ('statu', 4092.0500138037914),\n",
       "             ('inform', 4089.2604328033408),\n",
       "             ('time', 3944.7732107279357),\n",
       "             ('locat', 3871.0193424384088),\n",
       "             ('relat', 3860.2822000753149),\n",
       "             ('use', 3856.7460895413315),\n",
       "             ('project', 3843.9162822198787),\n",
       "             ('technolog', 3779.7923422363124),\n",
       "             ('technic', 3757.4047986816608),\n",
       "             ('en', 3671.0290084676963),\n",
       "             ('environ', 3638.7165852475973),\n",
       "             ('appli', 3616.1252165524934),\n",
       "             ('function', 3614.2462565520655),\n",
       "             ('plan', 3605.6149870993258),\n",
       "             ('assist', 3598.5972027779458),\n",
       "             ('oper', 3591.7064633079553),\n",
       "             ('state', 3574.8825875021853),\n",
       "             ('prefer', 3573.4967665734107),\n",
       "             ('legal', 3570.485856379531),\n",
       "             ('qualif', 3553.0544643625021),\n",
       "             ('degre', 3545.6116702177778),\n",
       "             ('strong', 3528.2487686992295),\n",
       "             ('document', 3512.5941567408227),\n",
       "             ('new', 3506.9722674339896),\n",
       "             ('profession', 3475.4968086327303),\n",
       "             ('excel', 3449.3000300211497),\n",
       "             ('data', 3413.2051887236057),\n",
       "             ('candid', 3396.5173252955542),\n",
       "             ('report', 3366.7127153238039),\n",
       "             ('review', 3366.6196742763614),\n",
       "             ('disabl', 3321.5394200422252),\n",
       "             ('client', 3299.6603094234761),\n",
       "             ('comput', 3299.289432000061),\n",
       "             ('level', 3281.4722601582466),\n",
       "             ('intern', 3238.5353140805378),\n",
       "             ('activ', 3233.6145654791558),\n",
       "             ('solut', 3223.0893607788666),\n",
       "             ('train', 3212.5854793760977),\n",
       "             ('need', 3212.4743377060236),\n",
       "             ('law', 3210.0240321723791),\n",
       "             ('ensur', 3190.8070760218075),\n",
       "             ('equal', 3176.540591578555),\n",
       "             ('lead', 3175.3691894919348),\n",
       "             ('offic', 3157.7449738860619),\n",
       "             ('maintain', 3140.7214164464526),\n",
       "             ('understand', 3139.4142333102614),\n",
       "             ('base', 3114.4063438110975),\n",
       "             ('organ', 3097.0669420933823),\n",
       "             ('employe', 3089.5919439433164),\n",
       "             ('area', 3074.9892869754322),\n",
       "             ('procedur', 3073.6975083253005),\n",
       "             ('tool', 3019.8956866042899),\n",
       "             ('nation', 3017.5879876419231),\n",
       "             ('protect', 3010.1142955111227),\n",
       "             ('us', 2988.7948121004897),\n",
       "             ('educ', 2987.1720707592744),\n",
       "             ('ani', 2983.8295819505611),\n",
       "             ('industri', 2974.0802724865093),\n",
       "             ('orient', 2970.7343176553063),\n",
       "             ('complet', 2958.9964287799153),\n",
       "             ('equal opportun', 2931.3512352331136),\n",
       "             ('integr', 2927.7081300394298),\n",
       "             ('high', 2902.1189561198767),\n",
       "             ('gener', 2893.7678245726679),\n",
       "             ('la', 2885.6974825316124),\n",
       "             ('analysi', 2884.0195209324538),\n",
       "             ('follow', 2875.4615638915534),\n",
       "             ('world', 2871.009176043864),\n",
       "             ('thi posit', 2864.1807456263841),\n",
       "             ('implement', 2855.5561581750571),\n",
       "             ('depart', 2854.0137131688407),\n",
       "             ('make', 2847.2814050696961),\n",
       "             ('within', 2844.8140750178372),\n",
       "             ('standard', 2840.6947293892408),\n",
       "             ('c', 2838.8465355415715),\n",
       "             ('complianc', 2829.5220606569187),\n",
       "             ('medic', 2813.9077404410932),\n",
       "             ('veteran', 2803.1831591762916),\n",
       "             ('may', 2802.0161170641704),\n",
       "             ('role', 2773.2397688495721),\n",
       "             ('part', 2764.0489700073372),\n",
       "             ('problem', 2763.6143774924249),\n",
       "             ('execut', 2758.676977635148),\n",
       "             ('issu', 2730.6490633933981),\n",
       "             ('control', 2725.7530459654949),\n",
       "             ('well', 2703.3171939473204),\n",
       "             ('global', 2702.4526432160901),\n",
       "             ('minimum', 2699.3177580917859),\n",
       "             ('duti', 2697.798198783481),\n",
       "             ('practic', 2697.6605206210374),\n",
       "             ('e', 2691.8624244102543),\n",
       "             ('descript', 2687.8026197370905),\n",
       "             ('full', 2686.7515366582916),\n",
       "             ('effect', 2676.2944152197897),\n",
       "             ('unit', 2659.7512017240306),\n",
       "             ('gender', 2658.897808362412),\n",
       "             ('assign', 2631.0327678463641),\n",
       "             ('one', 2625.8150123312525),\n",
       "             ('individu', 2623.1039293754811),\n",
       "             ('health', 2613.3154606187491),\n",
       "             ('secur', 2612.8065490526956),\n",
       "             ('year experi', 2606.6922205557448),\n",
       "             ('success', 2606.0508498303416),\n",
       "             ('improv', 2605.6551442450304),\n",
       "             ('offer', 2602.8639001637107),\n",
       "             ('id', 2582.273276374111),\n",
       "             ('code', 2579.5090302720669),\n",
       "             ('build', 2567.1650320156118),\n",
       "             ('polici', 2551.8648144797712),\n",
       "             ('meet', 2550.4311583394292),\n",
       "             ('specif', 2549.067655006168),\n",
       "             ('titl', 2534.6026403263886),\n",
       "             ('demonstr', 2528.3572118970851),\n",
       "             ('person', 2520.1552571339048),\n",
       "             ('scienc', 2506.5447710716589),\n",
       "             ('microsoft', 2498.4321671573471),\n",
       "             ('origin', 2497.6365530424646),\n",
       "             ('regard', 2491.6421188324825),\n",
       "             ('abl', 2482.4016321833365),\n",
       "             ('benefit', 2479.6517389535879),\n",
       "             ('creat', 2478.7917943435018),\n",
       "             ('particip', 2474.1501746770505),\n",
       "             ('qualifi', 2452.0260453400483),\n",
       "             ('opportun employ', 2449.2445241084106),\n",
       "             ('market', 2443.7907837512416),\n",
       "             ('race', 2442.3920286594803),\n",
       "             ('group', 2436.1995958846701),\n",
       "             ('autom', 2433.4585285327225),\n",
       "             ('collabor', 2433.1131783363389),\n",
       "             ('pleas', 2430.1872650906021),\n",
       "             ('action', 2426.0405674970498),\n",
       "             ('career', 2419.8706401616287),\n",
       "             ('peopl', 2418.7087509864414),\n",
       "             ('religion', 2418.3747080297458),\n",
       "             ('color', 2414.4993817958411),\n",
       "             ('care', 2411.2134708597155),\n",
       "             ('nation origin', 2400.4366943638115),\n",
       "             ('written', 2397.9198364976078),\n",
       "             ('bachelor', 2381.9928177515812),\n",
       "             ('result', 2377.3227807002095),\n",
       "             ('innov', 2363.2215604764156),\n",
       "             ('account', 2359.9817817495323),\n",
       "             ('open', 2357.6304020126236),\n",
       "             ('age', 2355.5684007814521),\n",
       "             ('complex', 2351.8918105982152),\n",
       "             ('ha', 2349.1007396236414),\n",
       "             ('commun skill', 2347.3789561858839),\n",
       "             ('research', 2346.152809788884),\n",
       "             ('commit', 2322.6164871905967),\n",
       "             ('patient', 2307.8707591503189),\n",
       "             ('sexual', 2297.485369194585),\n",
       "             ('risk', 2295.0445495705394),\n",
       "             ('deliv', 2294.3395750072877),\n",
       "             ('member', 2293.9898361954997),\n",
       "             ('also', 2293.0961881242797),\n",
       "             ('without', 2291.4987211913754),\n",
       "             ('addit', 2289.2082827296285),\n",
       "             ('hour', 2286.6036219667853),\n",
       "             ('softwar develop', 2275.1639064629671),\n",
       "             ('help', 2262.9636533138787),\n",
       "             ('detail', 2262.4143179325597),\n",
       "             ('chang', 2261.2642312418729),\n",
       "             ('veteran statu', 2258.7046123463542),\n",
       "             ('learn', 2258.3312640390977),\n",
       "             ('prepar', 2244.2599570007551),\n",
       "             ('race color', 2244.2489354093932),\n",
       "             ('look', 2243.9913979357784),\n",
       "             ('appropri', 2236.7481561765817),\n",
       "             ('sexual orient', 2235.2847724181188),\n",
       "             ('join', 2232.9005460448348),\n",
       "             ('good', 2232.7815884295583),\n",
       "             ('corpor', 2226.5722691227784),\n",
       "             ('field', 2225.6921895900091),\n",
       "             ('direct', 2204.0896803042706),\n",
       "             ('advanc', 2200.6016782740767),\n",
       "             ('com', 2198.0931265669192),\n",
       "             ('ident', 2195.5409791442617),\n",
       "             ('divers', 2188.6594541436225),\n",
       "             ('current', 2175.615461025202),\n",
       "             ('softwar engin', 2169.9148012730807),\n",
       "             ('desir', 2169.0830308990326),\n",
       "             ('identifi', 2165.3748736979546),\n",
       "             ('partner', 2164.0213676151757),\n",
       "             ('basic', 2153.9624147341106),\n",
       "             ('web', 2151.1194129381242),\n",
       "             ('laboratori', 2146.3778535373672),\n",
       "             ('u', 2132.5703654102044),\n",
       "             ('continu', 2127.8342969313439),\n",
       "             ('clinic', 2127.1232323195427),\n",
       "             ('assur', 2120.1610624165419),\n",
       "             ('etc', 2119.2525830183426),\n",
       "             ('full time', 2112.4518009425742),\n",
       "             ('con', 2108.0729829842435),\n",
       "             ('financi', 2100.3626558135352),\n",
       "             ('drive', 2098.7413523413347),\n",
       "             ('life', 2096.7374649586941),\n",
       "             ('gender ident', 2090.7704034719932),\n",
       "             ('schedul', 2082.81836963981),\n",
       "             ('resourc', 2075.1564668998099),\n",
       "             ('best', 2060.3234537487797),\n",
       "             ('contact', 2058.2562520825072),\n",
       "             ('task', 2057.5600902682509),\n",
       "             ('plu', 2055.7716527791927),\n",
       "             ('sex', 2052.777312792351),\n",
       "             ('day', 2052.4693542947998),\n",
       "             ('receiv', 2042.9667357542471),\n",
       "             ('analyt', 2041.9022644684258),\n",
       "             ('senior', 2035.0147233288078),\n",
       "             ('case', 2034.7318077718189),\n",
       "             ('post', 2022.8742477716498),\n",
       "             ('equival', 2015.639096612626),\n",
       "             ('independ', 2015.389860720767),\n",
       "             ('bachelor degre', 2004.8283078501984),\n",
       "             ('without regard', 2003.1893783703433),\n",
       "             ('countri', 2002.9611664675667),\n",
       "             ('conduct', 1995.2861701550642),\n",
       "             ('across', 1990.9375855461876),\n",
       "             ('center', 1988.2410478958329),\n",
       "             ('certif', 1986.103456613946),\n",
       "             ('author', 1985.4716716728326),\n",
       "             ('write', 1984.3224244109815),\n",
       "             ('network', 1983.8713247463002),\n",
       "             ('experiencia', 1981.4435139400937),\n",
       "             ('el', 1974.0339075075317),\n",
       "             ('highli', 1969.9952982032676),\n",
       "             ('regard race', 1964.6699844151747),\n",
       "             ('orient gender', 1958.2838381492786),\n",
       "             ('user', 1949.0425870134211),\n",
       "             ('solv', 1946.236546571899),\n",
       "             ('rang', 1933.6119397093244),\n",
       "             ('regulatori', 1933.0000734181865),\n",
       "             ('regul', 1931.0864206949059),\n",
       "             ('evalu', 1927.1554411688578),\n",
       "             ('strategi', 1923.3352771178836),\n",
       "             ('hire', 1914.9855790626734),\n",
       "             ('leader', 1900.5011779517467),\n",
       "             ('type', 1898.9666442316336),\n",
       "             ('requisit', 1892.4595530945387),\n",
       "             ('associ', 1889.3148602726906),\n",
       "             ('insur', 1888.3133276343244),\n",
       "             ('licens', 1885.2959144332701),\n",
       "             ('analyz', 1879.0041759967751),\n",
       "             ('coordin', 1877.2991609518492),\n",
       "             ('challeng', 1877.1797695102355),\n",
       "             ('close', 1875.8214712931849),\n",
       "             ('initi', 1869.2841716808921),\n",
       "             ('compens', 1868.3953088457488),\n",
       "             ('equip', 1866.1157426622058),\n",
       "             ('analyst', 1863.0867137143489),\n",
       "             ('basi', 1848.5305040294795),\n",
       "             ('assess', 1843.8583568405093),\n",
       "             ('lab', 1841.9193533312102),\n",
       "             ('record', 1830.6167449000066),\n",
       "             ('multipl', 1827.0484076624809),\n",
       "             ('feder', 1825.6847144489443),\n",
       "             ('matter', 1814.4056982618724),\n",
       "             ('access', 1807.1931249172253),\n",
       "             ('consider', 1805.4186916962958),\n",
       "             ('collect', 1804.7859998283093),\n",
       "             ('local', 1804.0738827510852),\n",
       "             ('java', 1803.9592660025642),\n",
       "             ('govern', 1803.876977388697),\n",
       "             ('verbal', 1799.0652630555362),\n",
       "             ('staff', 1791.0017746597559),\n",
       "             ('order', 1790.7231783465797),\n",
       "             ('decis', 1790.6393334333736),\n",
       "             ('skill abil', 1789.0593561476167),\n",
       "             ('set', 1788.1398729738185),\n",
       "             ('competit', 1786.6338069961453),\n",
       "             ('eeo', 1778.5474168656785),\n",
       "             ('variou', 1778.2523364829128),\n",
       "             ('firm', 1776.0895514181875),\n",
       "             ('color religion', 1775.9016417882285),\n",
       "             ('administr', 1767.8722896500033),\n",
       "             ('commerci', 1766.5104397735367),\n",
       "             ('object', 1763.6927124125732),\n",
       "             ('healthcar', 1755.8348185499076),\n",
       "             ('half', 1755.6163101949767),\n",
       "             ('comput scienc', 1749.1823298462218),\n",
       "             ('robert', 1749.0352240018451),\n",
       "             ('track', 1740.9022591708235),\n",
       "             ('robert half', 1738.5114187954771),\n",
       "             ('seek', 1738.0707669936328),\n",
       "             ('request', 1736.6338780096009),\n",
       "             ('key', 1735.9752958078677),\n",
       "             ('qualiti assur', 1735.9627174000145),\n",
       "             ('qualifi applic', 1730.828404509828),\n",
       "             ('establish', 1729.760354163692),\n",
       "             ('serv', 1727.0885679171931),\n",
       "             ('background', 1709.4599012286478),\n",
       "             ('travel', 1706.5279205633583),\n",
       "             ('interest', 1705.3417543966577),\n",
       "             ('essenti', 1700.7890501668608),\n",
       "             ('school', 1690.9396703665061),\n",
       "             ('relationship', 1689.4870110197924),\n",
       "             ('marit', 1688.7796523915802),\n",
       "             ('sale', 1688.4855957744821),\n",
       "             ('take', 1688.3778319191938),\n",
       "             ('affirm', 1688.2295542304514),\n",
       "             ('list', 1687.285736564588),\n",
       "             ('g', 1686.0008691956789),\n",
       "             ('leadership', 1685.7402704596941),\n",
       "             ('unit state', 1672.705535998733),\n",
       "             ('affirm action', 1671.0879819436216),\n",
       "             ('onlin', 1666.7308140567429),\n",
       "             ('relev', 1661.2409338856255),\n",
       "             ('languag', 1658.6214622371483),\n",
       "             ('platform', 1655.7781080584591),\n",
       "             ('counsel', 1654.6570062900503),\n",
       "             ('handl', 1650.2487488310508),\n",
       "             ('consult', 1647.9205231992194),\n",
       "             ('valid', 1645.7961398162706),\n",
       "             ('capabl', 1642.2425583119957),\n",
       "             ('contract', 1639.2816206509438),\n",
       "             ('physic', 1637.1709785666371),\n",
       "             ('special', 1636.1686556755267),\n",
       "             ('extern', 1633.0520329426433),\n",
       "             ('familiar', 1615.6435088047053),\n",
       "             ('model', 1614.2252442390309),\n",
       "             ('monitor', 1613.9388010386704),\n",
       "             ('mission', 1612.1444552936757),\n",
       "             ('attorney', 1604.2734811437122),\n",
       "             ('end', 1604.1471225150565),\n",
       "             ('class', 1602.256177311752),\n",
       "             ('site', 1602.222655665425),\n",
       "             ('supervis', 1600.5182753867975),\n",
       "             ('resum', 1595.9481599597223),\n",
       "             ('ano', 1595.27752297966),\n",
       "             ('avail', 1594.6407276460343),\n",
       "             ('proven', 1592.7569409131613),\n",
       "             ('date', 1592.0275420067342),\n",
       "             ('credit', 1590.1675782289558),\n",
       "             ('self', 1588.7484860515419),\n",
       "             ('marit statu', 1585.9434541368032),\n",
       "             ('valu', 1585.3871728330917),\n",
       "             ('script', 1584.3113685743351),\n",
       "             ('statement', 1582.9640893717042),\n",
       "             ('grow', 1582.3716275540953),\n",
       "             ('per', 1577.2847608194638),\n",
       "             ('present', 1573.7166057956285),\n",
       "             ('word', 1568.6277460611457),\n",
       "             ('non', 1562.7644970276167),\n",
       "             ('place', 1555.7852783633732),\n",
       "             ('interact', 1555.7591414173198),\n",
       "             ('prueba', 1548.6580421982189),\n",
       "             ('expect', 1547.2437740503881),\n",
       "             ('citi', 1545.5588025193022),\n",
       "             ('design develop', 1544.7332054800631),\n",
       "             ('agil', 1544.5302615821231),\n",
       "             ('devic', 1535.5593473976123),\n",
       "             ('involv', 1535.4134758619962),\n",
       "             ('necessari', 1534.9190202954105),\n",
       "             ('sourc', 1527.5676915280792),\n",
       "             ('investig', 1527.4951580674669),\n",
       "             ('profici', 1526.9388128434557),\n",
       "             ('packag', 1524.340900764671),\n",
       "             ('expertis', 1519.3236925284821),\n",
       "             ('accord', 1518.7272555184027),\n",
       "             ('obtain', 1514.8310506822668),\n",
       "             ('sex sexual', 1508.6660213845485),\n",
       "             ('call', 1508.3153957714092),\n",
       "             ('recommend', 1507.4403530664338),\n",
       "             ('consid', 1506.5739995237998),\n",
       "             ('qa', 1506.4159032639118),\n",
       "             ('deliveri', 1505.5980153785924),\n",
       "             ('mainten', 1505.2396505047648),\n",
       "             ('summari', 1503.8117283858508),\n",
       "             ('facil', 1501.1619774951241),\n",
       "             ('salari', 1500.5069297223283),\n",
       "             ('methodolog', 1495.7618716534246),\n",
       "             ('sql', 1490.6820808160844),\n",
       "             ('categori', 1490.0338802246781),\n",
       "             ('requisit id', 1487.8483881855395),\n",
       "             ('util', 1485.1577085900788),\n",
       "             ('mobil', 1485.1184736176863),\n",
       "             ('passion', 1484.3524714641962),\n",
       "             ('agenc', 1478.8663456768929),\n",
       "             ('term', 1476.523893100941),\n",
       "             ('contribut', 1475.1009092581035),\n",
       "             ('safeti', 1474.7647771139448),\n",
       "             ('trabajo', 1472.9255310183048),\n",
       "             ('genet', 1472.2413210034249),\n",
       "             ('effici', 1470.7070176445602),\n",
       "             ('shift', 1467.9005858444882),\n",
       "             ('elig', 1463.7089457869411),\n",
       "             ('major', 1459.2176142179899),\n",
       "             ('interfac', 1459.1350166625239),\n",
       "             ('para', 1455.5604984749586),\n",
       "             ('materi', 1455.1400525646015),\n",
       "             ('databas', 1449.5647556230454),\n",
       "             ('impact', 1448.2271893930335),\n",
       "             ('problem solv', 1444.5835684812005),\n",
       "             ('audit', 1443.3650025526824),\n",
       "             ('hand', 1441.4382295501359),\n",
       "             ('student', 1431.2950460861352),\n",
       "             ('citizenship', 1430.5902907343911),\n",
       "             ('al', 1429.0542932258968),\n",
       "             ('talent', 1427.4051716192703),\n",
       "             ('e g', 1424.4030613332325),\n",
       "             ('f', 1422.6741685562617),\n",
       "             ('primari', 1421.5551280332782),\n",
       "             ('wa', 1421.513560596268),\n",
       "             ('electr', 1421.2714763694801),\n",
       "             ('genet inform', 1420.0931041003439),\n",
       "             ('everi', 1419.4312019455444),\n",
       "             ('employ without', 1418.5616441219206),\n",
       "             ('framework', 1416.9326305267316),\n",
       "             ('techniqu', 1415.0161863134506),\n",
       "             ('goal', 1414.2570235688133),\n",
       "             ('multi', 1413.5256619738593),\n",
       "             ('cloud', 1413.4418852340843),\n",
       "             ('lo', 1412.3923221233918),\n",
       "             ('ms', 1410.7386072926649),\n",
       "             ('differ', 1403.6297120178615),\n",
       "             ('fast', 1401.1574784096906),\n",
       "             ('financ', 1400.5158312827218),\n",
       "             ('larg', 1391.4876544637591),\n",
       "             ('click', 1391.3676533425955),\n",
       "             ('varieti', 1390.7384832438322),\n",
       "             ('motiv', 1390.3065076239011),\n",
       "             ('disabl veteran', 1389.6636139487407),\n",
       "             ('hospit', 1388.3384615139475),\n",
       "             ('clearanc', 1387.7770541043212),\n",
       "             ('cultur', 1384.8049265074092),\n",
       "             ('instruct', 1383.9262403143389),\n",
       "             ('protect veteran', 1383.2303094384984),\n",
       "             ('personnel', 1382.9283480867769),\n",
       "             ('enterpris', 1380.5984814302808),\n",
       "             ('electron', 1378.1164936731827),\n",
       "             ('religion sex', 1376.430177124977),\n",
       "             ('por', 1375.2375012594171),\n",
       "             ('master', 1373.2160778298505),\n",
       "             ('www', 1372.8322723622402),\n",
       "             ('architectur', 1371.9694712497469),\n",
       "             ('enabl', 1371.0306800939138),\n",
       "             ('staf', 1370.7052845163071),\n",
       "             ('number', 1368.5445721320145),\n",
       "             ('manufactur', 1367.2410945560393),\n",
       "             ('bank', 1366.3678857910145),\n",
       "             ('determin', 1366.1203898370791),\n",
       "             ('experienc', 1365.2042739747239),\n",
       "             ('act', 1362.8322189914547),\n",
       "             ('line', 1361.8460083176433),\n",
       "             ('right', 1359.8976545169521),\n",
       "             ('connect', 1352.6346009947069),\n",
       "             ('english', 1350.8139135372996),\n",
       "             ('action employ', 1350.6897364720662),\n",
       "             ('wide', 1350.4975878209459),\n",
       "             ('least', 1349.2958238633523),\n",
       "             ('oral', 1344.6641635486526),\n",
       "             ('disciplin', 1344.610443113583),\n",
       "             ('applic appli', 1342.6277589431938),\n",
       "             ('like', 1340.8511815684881),\n",
       "             ('conocimiento', 1340.5752677192256),\n",
       "             ('worldwid', 1338.437827713512),\n",
       "             ('refer', 1336.5041907241709),\n",
       "             ('repres', 1330.0447342822765),\n",
       "             ('origin age', 1328.2470997204412),\n",
       "             ('ani protect', 1320.257320132594),\n",
       "             ('live', 1320.1625001989355),\n",
       "             ('experi requir', 1319.8866270144972),\n",
       "             ('combin', 1319.0971324082489),\n",
       "             ('share', 1317.8038986481426),\n",
       "             ('window', 1316.75535913862),\n",
       "             ('verifi', 1314.7953867619133),\n",
       "             ('comprehens', 1313.1981255156238),\n",
       "             ('req', 1311.6784151871741),\n",
       "             ('open author', 1311.1196942019878),\n",
       "             ('prior', 1308.6852506591297),\n",
       "             ('defin', 1308.5221485088073),\n",
       "             ('flexibl', 1306.4341389567137),\n",
       "             ('linux', 1306.0577751990484),\n",
       "             ('express', 1304.2142616978701),\n",
       "             ('limit', 1302.7880243695131),\n",
       "             ('focu', 1301.7162400620875),\n",
       "             ('possess', 1295.7738622342504),\n",
       "             ('distribut', 1291.431061075253),\n",
       "             ('hardwar', 1291.3488222536096),\n",
       "             ('exist', 1289.4842392388191),\n",
       "             ('que', 1287.9930485573684),\n",
       "             ('way', 1285.6885289753682),\n",
       "             ('accur', 1280.5998288651822),\n",
       "             ('organiz', 1280.3905755175213),\n",
       "             ('file', 1279.9975782979436),\n",
       "             ('creed', 1278.2685721691228),\n",
       "             ('opportun affirm', 1274.4318611646324),\n",
       "             ('relat field', 1273.3144807239082),\n",
       "             ('agreement', 1271.6141585923758),\n",
       "             ('accommod', 1271.5641559303797),\n",
       "             ('empresa', 1268.9928558222425),\n",
       "             ('concept', 1266.7217886563396),\n",
       "             ('discrimin', 1265.1678143812946),\n",
       "             ('tester', 1264.9318139451798),\n",
       "             ('condit', 1261.6377499583637),\n",
       "             ('req id', 1261.265398616112),\n",
       "             ('email', 1259.203319015214),\n",
       "             ('vision', 1256.5491028278534),\n",
       "             ('reason', 1253.7763873154654),\n",
       "             ('interperson', 1253.5903315815319),\n",
       "             ('compon', 1252.4461969450072),\n",
       "             ('top', 1252.0530258254062),\n",
       "             ('ideal', 1250.6194885086829),\n",
       "             ('fortun', 1247.8834557275302),\n",
       "             ('receiv consider', 1245.0747526386965),\n",
       "             ('method', 1244.5804680383137),\n",
       "             ('featur', 1243.4246997730179),\n",
       "             ('colleg', 1241.9887415383796),\n",
       "             ('around', 1241.4663530102803),\n",
       "             ('critic', 1241.2827964143089),\n",
       "             ('find', 1241.0424359658996),\n",
       "             ('studi', 1239.1874376536077),\n",
       "             ('c c', 1238.915675832556),\n",
       "             ('minimum year', 1237.5351445725275),\n",
       "             ('grumman', 1237.0630274478196),\n",
       "             ('northrop', 1234.3090883807811),\n",
       "             ('northrop grumman', 1234.3090883807811),\n",
       "             ('applic receiv', 1232.7929777581405),\n",
       "             ('enhanc', 1232.7041228763958),\n",
       "             ('proud', 1229.5078953615111),\n",
       "             ('region', 1227.1977726305979),\n",
       "             ('consider employ', 1223.955270451961),\n",
       "             ('server', 1222.5947800063832),\n",
       "             ('select', 1221.9729818773208),\n",
       "             ('negoti', 1220.1538636346963),\n",
       "             ('compet', 1216.4818648634284),\n",
       "             ('litig', 1214.3253731023835),\n",
       "             ('throughout', 1213.5309273155551),\n",
       "             ('releas', 1212.6729450108119),\n",
       "             ('month', 1210.6097668263783),\n",
       "             ('recruit', 1210.4325076777172),\n",
       "             ('cross', 1206.451165998169),\n",
       "             ('engag', 1203.9883109486532),\n",
       "             ('public', 1202.3599491305517),\n",
       "             ('updat', 1201.9597894061969),\n",
       "             ('pace', 1194.1932613657846),\n",
       "             ('free', 1194.0609831743232),\n",
       "             ('ident express', 1190.1909594443164),\n",
       "             ('promot', 1190.1005872639046),\n",
       "             ('polici procedur', 1189.2421221260679),\n",
       "             ('depend', 1188.8720492892649),\n",
       "             ('achiev', 1185.7360747016301),\n",
       "             ('b', 1184.0949032280887),\n",
       "             ('del', 1184.0217396556673),\n",
       "             ('consist', 1181.3511593450357),\n",
       "             ('institut', 1180.680266208871),\n",
       "             ('strateg', 1178.8922361496814),\n",
       "             ('custom servic', 1177.5142628593564),\n",
       "             ('power', 1175.2769320551588),\n",
       "             ('statu ani', 1174.5754779573988),\n",
       "             ('inform system', 1172.7160409074374),\n",
       "             ('futur', 1171.6796452147132),\n",
       "             ('basic qualif', 1169.7734744551663),\n",
       "             ('accredit', 1168.1195469563572),\n",
       "             ('submit', 1167.9122432842059),\n",
       "             ('resolv', 1166.6944370220197),\n",
       "             ('guidanc', 1166.3188720190292),\n",
       "             ('doe', 1164.264356779448),\n",
       "             ('ii', 1157.927519079527),\n",
       "             ('manner', 1153.1884415292345),\n",
       "             ('debug', 1150.0435459233352),\n",
       "             ('dure', 1149.668241246073),\n",
       "             ('fast pace', 1148.9516555554389),\n",
       "             ('onli', 1147.9103072114895),\n",
       "             ('great', 1147.7749751977587),\n",
       "             ('would', 1146.8564759473118),\n",
       "             ('togeth', 1145.6752345710968),\n",
       "             ('univers', 1142.9390871196274),\n",
       "             ('overview', 1142.8377450938287),\n",
       "             ('workforc', 1142.2891498852189),\n",
       "             ('directli', 1138.7987147262957),\n",
       "             ('qualiti engin', 1128.6696425665302),\n",
       "             ('dedic', 1128.0201110385563),\n",
       "             ('transact', 1127.9479037901719),\n",
       "             ('driven', 1126.0037959827544),\n",
       "             ('growth', 1121.8290069337495),\n",
       "             ('aspect', 1119.8782539348508),\n",
       "             ('week', 1119.0505036579953),\n",
       "             ('written commun', 1115.8042349398477),\n",
       "             ('name', 1115.4534182587659),\n",
       "             ('attent', 1113.3039545199726),\n",
       "             ('graduat', 1113.2190293466733),\n",
       "             ('first', 1112.4217912192028),\n",
       "             ('potenti', 1112.0654284157201),\n",
       "             ('manual', 1111.4541397184798),\n",
       "             ('cv', 1111.2993589008006),\n",
       "             ('principl', 1109.5422937508563),\n",
       "             ('accept', 1109.4990590222417),\n",
       "             ('educ experi', 1109.1598709249859),\n",
       "             ('human', 1106.8414207317874),\n",
       "             ('excit', 1106.6098477562914),\n",
       "             ('statu disabl', 1106.5500496641246),\n",
       "             ('approv', 1105.1213873156942),\n",
       "             ('specialist', 1104.3315086293171),\n",
       "             ('intellig', 1104.3013800813421),\n",
       "             ('thi role', 1103.6173910799662),\n",
       "             ('divis', 1101.9579391145785),\n",
       "             ('interpret', 1100.8893730220509),\n",
       "             ('cost', 1100.3829874475707),\n",
       "             ('creativ', 1099.8792885337359),\n",
       "             ('two', 1097.3086106150242),\n",
       "             ('address', 1094.7073249641771),\n",
       "             ('focus', 1094.4392327702185),\n",
       "             ('protocol', 1094.1540501149739),\n",
       "             ('draft', 1091.6762401969222),\n",
       "             ('characterist', 1090.2323658069517),\n",
       "             ('defect', 1090.111893909911),\n",
       "             ('daili', 1088.1345784418095),\n",
       "             ('quickli', 1087.9266940561361),\n",
       "             ('deploy', 1086.5448703347208),\n",
       "             ('employ qualifi', 1085.5823194136453),\n",
       "             ('appli thi', 1085.5366364274016),\n",
       "             ('priorit', 1084.3204428511287),\n",
       "             ('start', 1083.907752576157),\n",
       "             ('prefer qualif', 1083.3344893968983),\n",
       "             ('bring', 1080.3278497029321),\n",
       "             ('effort', 1080.0271431856904),\n",
       "             ('demand', 1075.6141942289237),\n",
       "             ('pm', 1074.6101030503542),\n",
       "             ('resolut', 1070.8225391400599),\n",
       "             ('regular', 1069.1666975834835),\n",
       "             ('solid', 1065.8572977105621),\n",
       "             ('ley', 1064.0583958267714),\n",
       "             ('cycl', 1063.3470360666497),\n",
       "             ('correct', 1062.6254103259157),\n",
       "             ('supervisor', 1062.5848595749571),\n",
       "             ('search', 1062.205556127833),\n",
       "             ('infrastructur', 1061.0929607361636),\n",
       "             ('law firm', 1058.6267658560691),\n",
       "             ('respons includ', 1053.6573815791874),\n",
       "             ('bs', 1052.2957964262928),\n",
       "             ('famili', 1051.7058408069072),\n",
       "             ('posit requir', 1047.4893950238115),\n",
       "             ('skill experi', 1047.4573608063222),\n",
       "             ('develop engin', 1046.6934262907589),\n",
       "             ('pay', 1045.0747093781915),\n",
       "             ('dynam', 1041.6233438110012),\n",
       "             ('similar', 1040.5100625615441),\n",
       "             ('prestacion', 1040.2300454171741),\n",
       "             ('structur', 1040.1176643383831),\n",
       "             ('physician', 1039.9009555914872),\n",
       "             ('age disabl', 1038.1685629061935),\n",
       "             ('suppli', 1034.4062064132288),\n",
       "             ('written verbal', 1033.0000314942738),\n",
       "             ('invest', 1030.4934824877284),\n",
       "             ('subject', 1029.3673807661537),\n",
       "             ('high school', 1026.1081428847185),\n",
       "             ('suit', 1025.3714675316112),\n",
       "             ('properti', 1023.9106857575961),\n",
       "             ('core', 1023.38039021126),\n",
       "             ('troubleshoot', 1022.066379530489),\n",
       "             ('drug', 1018.9386271399043),\n",
       "             ('co', 1016.7116965837479),\n",
       "             ('north', 1015.8102494936318),\n",
       "             ('approach', 1015.1553999987335),\n",
       "             ('scale', 1013.8555008347876),\n",
       "             ('experi year', 1013.3255091485374),\n",
       "             ('degre comput', 1010.9218271709382),\n",
       "             ('intern extern', 1010.2425621390502),\n",
       "             ('usa', 1009.7439619263748),\n",
       "             ('essenti function', 1009.3828071680567),\n",
       "             ('check', 1007.3204123763958),\n",
       "             ('verif', 1005.3525962559564),\n",
       "             ('outsid', 1004.6713294315141),\n",
       "             ('form', 1004.0953852070654),\n",
       "             ('temporari', 1003.7292159484588),\n",
       "             ('sistema', 1001.1020662812118),\n",
       "             ('employ f', 999.36368741078036),\n",
       "             ('inc', 996.49977424850897),\n",
       "             ('http', 996.45006960313401),\n",
       "             ('part time', 991.71594055512162),\n",
       "             ('abov', 987.96795314609255),\n",
       "             ('visit', 987.44008145097803),\n",
       "             ('develop experi', 986.87735325969481),\n",
       "             ('project manag', 986.68614925877216),\n",
       "             ('allow', 985.55693209332264),\n",
       "             ('point', 985.19883876441156),\n",
       "             ('attent detail', 983.57856673749075),\n",
       "             ('overal', 981.08488515937267),\n",
       "             ('javascript', 976.64461698280866),\n",
       "             ('configur', 976.22205315799897),\n",
       "             ('board', 973.64433172433633),\n",
       "             ('sueldo', 971.76206410026418),\n",
       "             ('canada', 970.18284349970986),\n",
       "             ('portfolio', 969.85230555601652),\n",
       "             ('expert', 969.77773804222409),\n",
       "             ('match', 969.04328594201911),\n",
       "             ('proud equal', 967.26149097925202),\n",
       "             ('hous', 966.22612501767742),\n",
       "             ('agre', 964.62400430429352),\n",
       "             ('vet', 962.47468755658201),\n",
       "             ('consum', 961.72551288472368),\n",
       "             ('individu disabl', 961.28256104502691),\n",
       "             ('measur', 960.78867603078311),\n",
       "             ('ofrecemo', 959.5988366037268),\n",
       "             ('requir posit', 957.01232705996654),\n",
       "             ('mani', 953.38622537636206),\n",
       "             ('religion nation', 951.96710031301109),\n",
       "             ('profil', 951.14256479866719),\n",
       "             ('includ limit', 951.01672364284241),\n",
       "             ('p', 950.7493489277025),\n",
       "             ('feder state', 950.4625903614766),\n",
       "             ('better', 948.85947543669033),\n",
       "             ('specialti', 948.35861062398203),\n",
       "             ('influenc', 943.83373501924882),\n",
       "             ('perform duti', 942.14854287287665),\n",
       "             ('question', 939.86405800691227),\n",
       "             ('diploma', 938.22056635636977),\n",
       "             ('high qualiti', 937.65513758111285),\n",
       "             ('optim', 937.47120704921872),\n",
       "             ('brand', 937.14875881132718),\n",
       "             ('escolaridad', 936.82117124407182),\n",
       "             ('health care', 934.88998897006707),\n",
       "             ('note', 933.22768854803098),\n",
       "             ('home', 932.51248509653351),\n",
       "             ('real', 932.11965527689199),\n",
       "             ('electr engin', 932.08885076251499),\n",
       "             ('digit', 931.9186705493704),\n",
       "             ('employ opportun', 931.71058509226316),\n",
       "             ('today', 929.26841815511716),\n",
       "             ('think', 928.07402312670501),\n",
       "             ('experi develop', 927.01647763383448),\n",
       "             ('product develop', 925.34496775089337),\n",
       "             ('onc', 925.16122751196565),\n",
       "             ('want', 924.091017590928),\n",
       "             ('possibl', 922.38572667940696),\n",
       "             ('requisito', 921.98001183453653),\n",
       "             ('facilit', 921.21022524440798),\n",
       "             ('interesado', 921.13007350363898),\n",
       "             ('prioriti', 920.80842032364376),\n",
       "             ('ident nation', 919.45024771051067),\n",
       "             ('content', 918.94406567824706),\n",
       "             ('f disabl', 918.89294636443742),\n",
       "             ('equal employ', 918.85708833803801),\n",
       "             ('superior', 918.44920130756918),\n",
       "             ('signific', 917.78866130021993),\n",
       "             ('skill strong', 917.13677790105942),\n",
       "             ('reason accommod', 916.96673835400316),\n",
       "             ('requir experi', 916.64545228028305),\n",
       "             ('extens', 914.53090303801184),\n",
       "             ('progress', 914.30816027026719),\n",
       "             ('retain', 913.38622586208226),\n",
       "             ('proactiv', 911.64177290688758),\n",
       "             ('duti assign', 911.5807732865203),\n",
       "             ('small', 910.74236273462668),\n",
       "             ('balanc', 910.72024858981626),\n",
       "             ('ca', 910.55001996140209),\n",
       "             ('wide rang', 910.03128787559467),\n",
       "             ('verbal written', 908.47056081686947),\n",
       "             ('feedback', 908.23752219813935),\n",
       "             ('abil independ', 907.56806051414208),\n",
       "             ('headquart', 907.37503160282586),\n",
       "             ('cover', 907.09859007462319),\n",
       "             ('readi', 906.98239805097228),\n",
       "             ('found', 906.76740137491925),\n",
       "             ('canadian', 906.50283114489582),\n",
       "             ('click appli', 906.3594700183852),\n",
       "             ('guidelin', 905.62693786748946),\n",
       "             ('event', 904.56123461274126),\n",
       "             ('design implement', 903.9749235797309),\n",
       "             ('upon', 903.24135085837167),\n",
       "             ('interperson skill', 903.09525145411715),\n",
       "             ('read', 900.73576037781515),\n",
       "             ('except', 900.60099391541291),\n",
       "             ('religion creed', 898.46938977700074),\n",
       "             ('school diploma', 896.58681207633867),\n",
       "             ('k', 895.67407988911907),\n",
       "             ('previou', 895.51672571336837),\n",
       "             ('paid', 894.12567691904508),\n",
       "             ('reward', 894.11125973740627),\n",
       "             ('adher', 893.43581291267435),\n",
       "             ('natur', 892.86913867991188),\n",
       "             ('period', 891.74029413048947),\n",
       "             ('mentor', 891.1977738899152),\n",
       "             ('largest', 889.05753353874161),\n",
       "             ('en el', 885.87308753917546),\n",
       "             ('br', 882.73549425356578),\n",
       "             ('requir bachelor', 882.56310409323271),\n",
       "             ('aa', 882.45727851525453),\n",
       "             ('around world', 880.09477900520824),\n",
       "             ('entri', 880.01949240557235),\n",
       "             ('relat experi', 878.26187855266414),\n",
       "             ('experi prefer', 877.97055544756131),\n",
       "             ('microsoft offic', 877.444079794564),\n",
       "             ('recogn', 876.13399537496775),\n",
       "             ('respect', 875.28416468692501),\n",
       "             ('html', 872.25750806805308),\n",
       "             ('veri', 870.09374376169387),\n",
       "             ('revenu', 869.44053406842306),\n",
       "             ('accuraci', 867.00523553309836),\n",
       "             ('skill requir', 866.54874589929386),\n",
       "             ('magazin', 865.20795039782035),\n",
       "             ('civil', 861.51823688294451),\n",
       "             ('higher', 860.8081835098676),\n",
       "             ('demonstr abil', 860.61392816719297),\n",
       "             ('busi unit', 859.08310426256241),\n",
       "             ('situat', 858.4823552147983),\n",
       "             ('knowledg skill', 857.36087168485471),\n",
       "             ('immedi', 857.10704278183084),\n",
       "             ('annual', 857.0777302113421),\n",
       "             ('u citizenship', 856.95896464591681),\n",
       "             ('judgment', 856.95896464591669),\n",
       "             ('disabl vet', 856.54791985757208),\n",
       "             ('reliabl', 854.83914384225591),\n",
       "             ('keep', 852.37290546887903),\n",
       "             ('leverag', 851.34953454269942),\n",
       "             ('uniqu', 849.80737105089054),\n",
       "             ('web servic', 849.47279513526541),\n",
       "             ('long', 848.96248314395029),\n",
       "             ('deadlin', 848.57643620000158),\n",
       "             ('broad', 847.34957298975507),\n",
       "             ('python', 847.08924201016418),\n",
       "             ('three', 846.02454194038),\n",
       "             ('profit', 845.73509456654813),\n",
       "             ('master degre', 845.39855577084995),\n",
       "             ('award', 845.37881074211259),\n",
       "             ('inform ani', 845.09011518731086),\n",
       "             ('equival experi', 843.54897417024483),\n",
       "             ('billion', 840.49200084377105),\n",
       "             ('academ', 840.41425497575392),\n",
       "             ('join us', 839.94842889219501),\n",
       "             ('phone', 839.46739060680204),\n",
       "             ('author unit', 839.27441898649238),\n",
       "             ('militari', 837.32226930311833),\n",
       "             ('duti respons', 837.04482286822861),\n",
       "             ('million', 836.99034771605852),\n",
       "             ('metric', 836.8922457400497),\n",
       "             ('admir', 836.72160753478033),\n",
       "             ('ingl', 836.06923118440386),\n",
       "             ('stand', 835.43778164652406),\n",
       "             ('creation', 834.38619791184442),\n",
       "             ('space', 834.23664079397872),\n",
       "             ('admir compani', 834.00722374487805),\n",
       "             ('inclus', 833.79827942542954),\n",
       "             ('addit inform', 832.51208530529289),\n",
       "             ('countri usa', 832.51208530529289),\n",
       "             ('coverag', 832.41505817843665),\n",
       "             ('type full', 832.23589944952107),\n",
       "             ('world admir', 831.28834454067669),\n",
       "             ('workplac', 829.94521536619038),\n",
       "             ('equipo', 829.44564964115352),\n",
       "             ('branch', 828.83250599822077),\n",
       "             ('privat', 828.82133619381568),\n",
       "             ('know', 828.32868662681119),\n",
       "             ('verbal commun', 828.23981737978272),\n",
       "             ('march', 826.53782571533554),\n",
       "             ('requir skill', 826.27029849687483),\n",
       "             ('disabl ani', 826.01012559269293),\n",
       "             ('adapt', 825.38695578323393),\n",
       "             ('sector', 824.5527697067987),\n",
       "             ('success candid', 824.18289370019147),\n",
       "             ('along', 823.13433057562918),\n",
       "             ('statu genet', 820.55249803961919),\n",
       "             ('divers workforc', 818.5013101596395),\n",
       "             ('fortun magazin', 818.36073204428124),\n",
       "             ('transform', 813.73622479275843),\n",
       "             ('prestacion ley', 813.32198460148254),\n",
       "             ('minimum qualif', 809.75916881467731),\n",
       "             ('trust', 808.85120374957637),\n",
       "             ('get', 807.12566802913409),\n",
       "             ('next', 805.65544360555828),\n",
       "             ('magazin list', 805.08871403668343),\n",
       "             ('colleagu', 804.77346075461969),\n",
       "             ('correo', 803.79515136309578),\n",
       "             ('excel commun', 800.89352491364741),\n",
       "             ('bar', 800.00109971055724),\n",
       "             ('describ', 799.71550948780339),\n",
       "             ('list world', 799.57661356031235),\n",
       "             ('compani march', 799.57661356031235),\n",
       "             ('onc wa', 799.57661356031235),\n",
       "             ('experi softwar', 798.12635389752336),\n",
       "             ('wa name', 796.50430035833301),\n",
       "             ('make decis', 796.20253522962776),\n",
       "             ('stakehold', 794.6085783323208),\n",
       "             ('postal', 793.73358691198712),\n",
       "             ('edg', 793.7103254457412),\n",
       "             ('exposur', 793.20097795071069),\n",
       "             ('becom', 792.97102257693348),\n",
       "             ('respond', 792.77591411366552),\n",
       "             ('acquisit', 790.58855457421248),\n",
       "             ('american', 790.36798891371507),\n",
       "             ('purpos', 788.82848081546092),\n",
       "             ('solv skill', 787.86729359742253),\n",
       "             ('state applic', 787.24758674507291),\n",
       "             ('develop implement', 786.66533533251811),\n",
       "             ('best practic', 786.06049095951562),\n",
       "             ('input', 782.26776047051646),\n",
       "             ('ideal candid', 781.10258817144836),\n",
       "             ('name fortun', 779.80816114583104),\n",
       "             ('appli u', 778.87812811360118),\n",
       "             ('clear', 778.60675683963234),\n",
       "             ('protect statu', 778.57984085399096),\n",
       "             ('canada equal', 777.33834244223749),\n",
       "             ('legal depart', 777.17273904614535),\n",
       "             ('commit hire', 777.00854182453941),\n",
       "             ('deal', 776.69645749005213),\n",
       "             ('appli canadian', 776.07858859365683),\n",
       "             ('u open', 776.07858859365683),\n",
       "             ('author canada', 776.07858859365683),\n",
       "             ('canadian open', 776.07858859365683),\n",
       "             ('comfort', 775.65520330491643),\n",
       "             ('postal code', 774.70606145078011),\n",
       "             ('function role', 774.70606145078011),\n",
       "             ('term use', 774.70606145078011),\n",
       "             ('oper system', 772.83932991173197),\n",
       "             ('agre robert', 771.90114973290019),\n",
       "             ('appli agre', 771.90114973290019),\n",
       "             ('id function', 771.90114973290019),\n",
       "             ('half term', 771.90114973290019),\n",
       "             ('vet click', 771.90114973290019),\n",
       "             ('use req', 771.90114973290019),\n",
       "             ('citizenship requir', 770.85204286333408),\n",
       "             ('zona', 770.58953264425554),\n",
       "             ('scope', 769.92688683353356),\n",
       "             ('usa state', 769.09131200509159),\n",
       "             ('code compens', 769.09131200509159),\n",
       "             ('move', 768.17965744064145),\n",
       "             ('pre', 767.83763906168838),\n",
       "             ('enviar', 765.89815978317426),\n",
       "             ('made', 763.74128201660483),\n",
       "             ('thi contact', 763.45676073624225),\n",
       "             ('premier', 760.2177524423447),\n",
       "             ('servic provid', 757.99886543576201),\n",
       "             ('eeo aa', 754.43313243667501),\n",
       "             ('life cycl', 752.17487054030562),\n",
       "             ('high level', 751.46949301542668),\n",
       "             ('requir year', 747.87372498933689),\n",
       "             ('protect class', 745.70515123142582),\n",
       "             ('lifecycl', 745.18000226754702),\n",
       "             ('phase', 743.7586387812953),\n",
       "             ('solicita', 737.288063533637),\n",
       "             ('qualif bachelor', 737.288063533637),\n",
       "             ('player', 727.98243770041358),\n",
       "             ('met', 724.29610466359111),\n",
       "             ('oral written', 721.9204252884698),\n",
       "             ('horario', 720.38434087101962),\n",
       "             ('ani characterist', 711.41367606583617),\n",
       "             ('industri experi', 710.81937203848975),\n",
       "             ('color gender', 710.4343530539594),\n",
       "             ('benefit packag', 710.16040700984877),\n",
       "             ('desir skill', 708.49095699341137),\n",
       "             ('make differ', 707.69634910666082),\n",
       "             ('age veteran', 705.35902865362425),\n",
       "             ('pace environ', 701.23910316750016),\n",
       "             ('analyt skill', 701.23910316750016),\n",
       "             ('ident marit', 699.48233726476553),\n",
       "             ('statu nation', 699.48233726476553),\n",
       "             ('retain divers', 699.48233726476553),\n",
       "             ('characterist protect', 699.48233726476519),\n",
       "             ('northropgrumman', 698.28797171882707),\n",
       "             ('northropgrumman com', 698.28797171882707),\n",
       "             ('degre requir', 696.74433331746684),\n",
       "             ('dental', 695.6663309484552),\n",
       "             ('system integr', 693.23322012578581),\n",
       "             ('discov', 691.61752670627016),\n",
       "             ('disabl statu', 690.62486659949047),\n",
       "             ('hire retain', 690.62486659949002),\n",
       "             ('grumman commit', 687.6609313736019),\n",
       "             ('decis without', 687.6609313736019),\n",
       "             ('creed sex', 687.6609313736019),\n",
       "             ('com eeo', 687.6609313736019),\n",
       "             ('employ make', 687.6609313736019),\n",
       "             ('class complet', 681.71569485468251)])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dict_word_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The words counts have been reweighted to emphasize the more meaninful words of the corpus while deemphasizing the the are found throughout the corpus. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "| 9 Elapsed Time: 0:00:20                                                                                                                                              \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 medic, patient, care, laboratori, clinic, health, hour, assist, duti, requir, posit, lab, educ, provid, depart\n",
      "1 grumman, northrop grumman, northrop, eeo, u citizenship, disabl ani, aa, divers workforc, citizenship requir, eeo aa, commit hire, protect class, workforc, age veteran, retain\n",
      "2 applic appli, countri usa, postal, worldwid, career, learn, industri, client, busi, onlin, offer, com, network, great, recruit\n",
      "3 qualiti, engin, system, softwar, develop, assur, product, experi, plan, qualiti assur, design, requir, knowledg, project, autom\n",
      "4 legal, complianc, busi, manag, regulatori, risk, counsel, financi, matter, credit, abil, law, issu, polici, corpor\n",
      "5 global, busi, market, revenu, deliv, live, cultur, across, leader, leadership, peopl, place, world, share, role\n",
      "6 half, robert, robert half, open author, compens, author, appli, open, world admir, staf, admir, admir compani, employ f, f disabl, magazin\n",
      "7 en, la, con, experiencia, el, ano, prueba, trabajo, para, al, por, conocimiento, que, empresa, lo\n",
      "8 statu, gender, religion, nation origin, race, color, origin, protect, sexual, race color, sexual orient, employ, veteran statu, nation, ident\n",
      "9 softwar, engin, develop, softwar engin, web, java, c, softwar develop, platform, design, technolog, cloud, passion, experi, javascript\n"
     ]
    }
   ],
   "source": [
    "processed_keywords, processed_doctopic = create_topics(processed_bag_of_words, \n",
    "                                                       processed_features, \n",
    "                                                      N_TOPICS = 10,\n",
    "                                                      N_TOP_WORDS= 15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#grab the topic_id of the majority topic for each document and store it in a list\n",
    "ls_topic_id = [np.argsort(processed_doctopic[comment_id])[::-1][0] for comment_id in range(len(corpus))]\n",
    "df_jobs_data['topic_id'] = ls_topic_id #add to the dataframe so we can compare with the job titles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that each row is tagged with a topic id let's see how well the topics exaplain the job advertisements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "medic, patient, care, laboratori, clinic, health, hour, assist, duti, requir, posit, lab, educ, provid, depart\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>normalizedTitle_onetName</th>\n",
       "      <th>normalizedTitle_onetCode</th>\n",
       "      <th>jobDescription</th>\n",
       "      <th>title</th>\n",
       "      <th>topic_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Graduate Teaching Assistants</td>\n",
       "      <td>25-1191.00</td>\n",
       "      <td>Lab Assistant Category: Clinical Services/Diag...</td>\n",
       "      <td>Lab Assistant</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Graduate Teaching Assistants</td>\n",
       "      <td>25-1191.00</td>\n",
       "      <td>Lab Assistant Category: Clinical Services/Diag...</td>\n",
       "      <td>Lab Assistant</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Graduate Teaching Assistants</td>\n",
       "      <td>25-1191.00</td>\n",
       "      <td>Lab Assistant Category: Clinical Services/Diag...</td>\n",
       "      <td>Lab Assistant</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Graduate Teaching Assistants</td>\n",
       "      <td>25-1191.00</td>\n",
       "      <td>Lab Assistant Category: Clinical Services/Diag...</td>\n",
       "      <td>Lab Assistant</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Graduate Teaching Assistants</td>\n",
       "      <td>25-1191.00</td>\n",
       "      <td>Lab Assistant Category: Clinical Services/Diag...</td>\n",
       "      <td>Lab Assistant</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Graduate Teaching Assistants</td>\n",
       "      <td>25-1191.00</td>\n",
       "      <td>****This position is an \"alternate\" position. ...</td>\n",
       "      <td>Laboratory Assistant (Alternate position)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>Financial Examiners</td>\n",
       "      <td>13-2061.00</td>\n",
       "      <td>Job Title: COMPLIANCE EXAMINER 1, 2 Agency: GO...</td>\n",
       "      <td>COMPLIANCE EXAMINER 1, 2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>Financial Examiners</td>\n",
       "      <td>13-2061.00</td>\n",
       "      <td>Job Title: Compliance Examiner 1, 2, or 3 Agen...</td>\n",
       "      <td>Compliance Examiner 1, 2, or 3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>Software Quality Assurance Engineers and Testers</td>\n",
       "      <td>15-1199.01</td>\n",
       "      <td>Job Title: ENGINEER 4 Agency: DOTD-Engineering...</td>\n",
       "      <td>ENGINEER 4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>Software Quality Assurance Engineers and Testers</td>\n",
       "      <td>15-1199.01</td>\n",
       "      <td>Job Title: CORRECTIONS CADET TEST Agency: DOC ...</td>\n",
       "      <td>CORRECTIONS CADET TEST</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            normalizedTitle_onetName normalizedTitle_onetCode  \\\n",
       "16                      Graduate Teaching Assistants               25-1191.00   \n",
       "17                      Graduate Teaching Assistants               25-1191.00   \n",
       "18                      Graduate Teaching Assistants               25-1191.00   \n",
       "19                      Graduate Teaching Assistants               25-1191.00   \n",
       "20                      Graduate Teaching Assistants               25-1191.00   \n",
       "21                      Graduate Teaching Assistants               25-1191.00   \n",
       "66                               Financial Examiners               13-2061.00   \n",
       "67                               Financial Examiners               13-2061.00   \n",
       "68  Software Quality Assurance Engineers and Testers               15-1199.01   \n",
       "69  Software Quality Assurance Engineers and Testers               15-1199.01   \n",
       "\n",
       "                                       jobDescription  \\\n",
       "16  Lab Assistant Category: Clinical Services/Diag...   \n",
       "17  Lab Assistant Category: Clinical Services/Diag...   \n",
       "18  Lab Assistant Category: Clinical Services/Diag...   \n",
       "19  Lab Assistant Category: Clinical Services/Diag...   \n",
       "20  Lab Assistant Category: Clinical Services/Diag...   \n",
       "21  ****This position is an \"alternate\" position. ...   \n",
       "66  Job Title: COMPLIANCE EXAMINER 1, 2 Agency: GO...   \n",
       "67  Job Title: Compliance Examiner 1, 2, or 3 Agen...   \n",
       "68  Job Title: ENGINEER 4 Agency: DOTD-Engineering...   \n",
       "69  Job Title: CORRECTIONS CADET TEST Agency: DOC ...   \n",
       "\n",
       "                                        title  topic_id  \n",
       "16                              Lab Assistant         0  \n",
       "17                              Lab Assistant         0  \n",
       "18                              Lab Assistant         0  \n",
       "19                              Lab Assistant         0  \n",
       "20                              Lab Assistant         0  \n",
       "21  Laboratory Assistant (Alternate position)         0  \n",
       "66                   COMPLIANCE EXAMINER 1, 2         0  \n",
       "67             Compliance Examiner 1, 2, or 3         0  \n",
       "68                                 ENGINEER 4         0  \n",
       "69                     CORRECTIONS CADET TEST         0  "
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topic_num = 0\n",
    "print(processed_keywords[topic_num])\n",
    "df_jobs_data[ df_jobs_data.topic_id == topic_num ].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Supervised Learning: Document Classification.\n",
    "\n",
    "Now we turn our attention to supervised learning. Previously, using topic modelling, we were inferring relationships between the data. In supervised learning, we produce a label, *y*, given some data *x*. In order to produce labels we need to first have examples our algorithm can learn from, a training set. Developing a training set can be very expensive, as it can require a large amount of human labor or linguistic expertise. Document classification is the case where our x are documents and our y are what the documents are (e.g, title for a job position). A common example of document classification is spam detection in emails. In sentiment analysis our x is our documents and y is the state of the author. This can range from an author being happy or unhappy with a product or the author being pollitically conservative or liberal. There is also Part-of-speech tagging were our x are individual words and y is the part-of-speech. \n",
    "\n",
    "In this section we are going to train a classifier to classify job titles using our jobs dataset.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_train = pd.read_csv('./data/train_corpus_document_tagging.csv')\n",
    "df_test = pd.read_csv('./data/test_corpus_document_tagging.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>normalizedTitle_onetName</th>\n",
       "      <th>jobDescription</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>96011</td>\n",
       "      <td>Credit Analysts</td>\n",
       "      <td>/Join a financial group thats as committed to ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>355594</td>\n",
       "      <td>Credit Analysts</td>\n",
       "      <td>for fastest consideration on this position, pl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>118598</td>\n",
       "      <td>Credit Analysts</td>\n",
       "      <td>As a Credit Analyst youll be responsible for c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>107794</td>\n",
       "      <td>Credit Analysts</td>\n",
       "      <td>We are a specialty retailer offering the very ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>40010</td>\n",
       "      <td>Credit Analysts</td>\n",
       "      <td>Title: KYC Remediation Team Analyst II Locatio...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0 normalizedTitle_onetName  \\\n",
       "0       96011          Credit Analysts   \n",
       "1      355594          Credit Analysts   \n",
       "2      118598          Credit Analysts   \n",
       "3      107794          Credit Analysts   \n",
       "4       40010          Credit Analysts   \n",
       "\n",
       "                                      jobDescription  \n",
       "0  /Join a financial group thats as committed to ...  \n",
       "1  for fastest consideration on this position, pl...  \n",
       "2  As a Credit Analyst youll be responsible for c...  \n",
       "3  We are a specialty retailer offering the very ...  \n",
       "4  Title: KYC Remediation Team Analyst II Locatio...  "
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Credit Analysts', 'Financial Examiners'], dtype=object)"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train['normalizedTitle_onetName'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Credit Analysts': 51, 'Financial Examiners': 51})"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Counter(df_train['normalizedTitle_onetName'].values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>normalizedTitle_onetName</th>\n",
       "      <th>jobDescription</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>356466</td>\n",
       "      <td>Credit Analysts</td>\n",
       "      <td>Collections Analyst to monitor and follow up o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>314705</td>\n",
       "      <td>Credit Analysts</td>\n",
       "      <td>Credit Analyst **Location:** LAVERGNE, TN **Sa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>84895</td>\n",
       "      <td>Credit Analysts</td>\n",
       "      <td>Deputy Head of Credit/Senior Credit Officer - ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>194749</td>\n",
       "      <td>Credit Analysts</td>\n",
       "      <td>Dresden Partners cuenta con 10 anos de experie...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>351689</td>\n",
       "      <td>Credit Analysts</td>\n",
       "      <td>Degreed accountant with manufacturing experien...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0 normalizedTitle_onetName  \\\n",
       "0      356466          Credit Analysts   \n",
       "1      314705          Credit Analysts   \n",
       "2       84895          Credit Analysts   \n",
       "3      194749          Credit Analysts   \n",
       "4      351689          Credit Analysts   \n",
       "\n",
       "                                      jobDescription  \n",
       "0  Collections Analyst to monitor and follow up o...  \n",
       "1  Credit Analyst **Location:** LAVERGNE, TN **Sa...  \n",
       "2  Deputy Head of Credit/Senior Credit Officer - ...  \n",
       "3  Dresden Partners cuenta con 10 anos de experie...  \n",
       "4  Degreed accountant with manufacturing experien...  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Credit Analysts', 'Financial Examiners'], dtype=object)"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test['normalizedTitle_onetName'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Credit Analysts': 77, 'Financial Examiners': 77})"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Counter(df_test['normalizedTitle_onetName'].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our data is job advertisements for credit analysts and financial examiners."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Process our Data\n",
    "\n",
    "In order to feed out data into a classifier we need to pull out the labels, our y's, and a clean corpus of documents, x, for our training and testing set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_labels = df_train.normalizedTitle_onetName.values\n",
    "train_corpus = np.array( [re.sub(RE_PREPROCESS, ' ', text).lower() for text in df_train.jobDescription.values])\n",
    "test_labels = df_test.normalizedTitle_onetName.values\n",
    "test_corpus = np.array( [re.sub(RE_PREPROCESS, ' ', text).lower() for text in df_test.jobDescription.values])\n",
    "labels = np.append(train_labels, test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just as we had done in the unsupervised learning we have to transform our data. This time we have to transform our testing and training set into two different bag-of-words. The classifer will learn from the training set and we will evaluate the clasffier's performance on the testing set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#parameters for vectorizer \n",
    "ANALYZER = \"word\" #unit of features are single words rather then phrases of words \n",
    "STRIP_ACCENTS = 'unicode'\n",
    "TOKENIZER = None\n",
    "NGRAM_RANGE = (0,2) #Range for pharases of words\n",
    "MIN_DF = 0.01 # Exclude words that have a frequency less than the threshold\n",
    "MAX_DF = 0.8  # Exclude words that have a frequency greater then the threshold \n",
    "\n",
    "vectorizer = CountVectorizer(analyzer=ANALYZER,\n",
    "                            tokenizer=None, # alternatively tokenize_and_stem but it will be slower \n",
    "                            ngram_range=NGRAM_RANGE,\n",
    "                            stop_words = stopwords.words('english'),\n",
    "                            strip_accents=STRIP_ACCENTS,\n",
    "                            min_df = MIN_DF,\n",
    "                            max_df = MAX_DF)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time Elapsed: 0.23s\n"
     ]
    }
   ],
   "source": [
    "NORM = None #turn on normalization flag\n",
    "SMOOTH_IDF = True #prvents division by zero errors\n",
    "SUBLINEAR_IDF = True #replace TF with 1 + log(TF)\n",
    "USE_IDF = True #flag to control whether to use TFIDF\n",
    "\n",
    "transformer = TfidfTransformer(norm = NORM,smooth_idf = SMOOTH_IDF,sublinear_tf = True)\n",
    "\n",
    "#get the bag-of-words from the vectorizer and\n",
    "#then use TFIDF to limit the tokens found throughout the text \n",
    "start_time = time.time()\n",
    "train_bag_of_words = vectorizer.fit_transform( train_corpus ) #using all the data on for generating features!! Bad!\n",
    "test_bag_of_words = vectorizer.transform( test_corpus )\n",
    "if USE_IDF:\n",
    "    train_tfidf = transformer.fit_transform(train_bag_of_words)\n",
    "    test_tfidf = transformer.transform(test_bag_of_words)\n",
    "features = vectorizer.get_feature_names()\n",
    "print('Time Elapsed: {0:.2f}s'.format(\n",
    "        time.time()-start_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We cannot pass the label \"Credit Analyst\" or \"Financial Examiner\" into the classifier. Instead we to encode them as 0s and 1s using the labelencoder as a part of sklearn. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#relabel our labels as a 0 or 1\n",
    "le = preprocessing.LabelEncoder() \n",
    "le.fit(labels)\n",
    "labels_binary = le.transform(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to create arrays of indices so we can access the training and testing sets accoringly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_size = df_train.shape[0]\n",
    "train_set_idx = np.arange(0,train_size)\n",
    "test_set_idx = np.arange(train_size, len(labels))\n",
    "train_labels_binary = labels_binary[train_set_idx]\n",
    "test_labels_binary = labels_binary[test_set_idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The classifier we are using in the example is LogisticRegression. As we saw in the Machine Learning tutorial we first decide on a classifier, then we fit the classfier to create a model. We can then test our model with our testing set by passing in the features of the testing set. The model will output the probablity of each document being classfied as a Credit Analyst or Financial Analyst. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "clf = LogisticRegression(penalty='l1')\n",
    "mdl = clf.fit(train_tfidf, labels_binary[train_set_idx]) #train the classifer to get the model\n",
    "y_score = mdl.predict_proba( test_tfidf ) #score of the document being an ad for Credit or Financial Analyst"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_precision_recall_n(y_true, y_prob, model_name):\n",
    "    \"\"\"\n",
    "    y_true: ls\n",
    "        ls of ground truth labels\n",
    "    y_prob: ls\n",
    "        ls of predic proba from model\n",
    "    model_name: str\n",
    "        str of model name (e.g, LR_123)\n",
    "    \"\"\"\n",
    "    from sklearn.metrics import precision_recall_curve\n",
    "    y_score = y_prob\n",
    "    precision_curve, recall_curve, pr_thresholds = precision_recall_curve(y_true, y_score)\n",
    "    precision_curve = precision_curve[:-1]\n",
    "    recall_curve = recall_curve[:-1]\n",
    "    pct_above_per_thresh = []\n",
    "    number_scored = len(y_score)\n",
    "    for value in pr_thresholds:\n",
    "        num_above_thresh = len(y_score[y_score>=value])\n",
    "        pct_above_thresh = num_above_thresh / float(number_scored)\n",
    "        pct_above_per_thresh.append(pct_above_thresh)\n",
    "    pct_above_per_thresh = np.array(pct_above_per_thresh)\n",
    "    plt.clf()\n",
    "    fig, ax1 = plt.subplots()\n",
    "    ax1.plot(pct_above_per_thresh, precision_curve, 'b')\n",
    "    ax1.set_xlabel('percent of population')\n",
    "    ax1.set_ylabel('precision', color='b')\n",
    "    ax1.set_ylim(0,1.05)\n",
    "    ax2 = ax1.twinx()\n",
    "    ax2.plot(pct_above_per_thresh, recall_curve, 'r')\n",
    "    ax2.set_ylabel('recall', color='r')\n",
    "    ax2.set_ylim(0,1.05)\n",
    "    \n",
    "    name = model_name\n",
    "    plt.title(name)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f02312ef0f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkIAAAGFCAYAAAARopyRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3Xd4VNXWx/Hv0AUVEJSiqKAgKCo38V5FKV4LV6yASIzS\nOxYIxYYKCioXkSIKgoUiJfRiFxRRFARJ9CJSVBSVriiIFCmZ9481eTOEtDmZmTPl93meeZKcmTlZ\nOWLOyt5rrw0iIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIu2B\nDCAhl+fP9T2f+TgG7AbeAxqGPjwRiXdF3A5ARAQYDVwBNAAeAOoAHwCJbgYlIrFPiZCIRIKfgVXA\nCmAC0BooDnRzMygRiX1KhEQkEqX5PlZzNQoRiXlKhEQkElX3fdzuahQiEvOKuR2AiAhQFPt9VBSo\nDYwFjgAvuxmUiIiISGG0J7BVY5mPP4DrQh+eiMQ7TY2JSCQYBVwGXAX0A0oC9wEeN4MSkdinqTER\niQRbgHTf5yuAHcBUoBeWJImIhIRGhEQkEk0HlgIDgPLuhiIisUwjQiISDtcCNXI4vi6P9zwErAQe\nA/qGIigRERGRUGpHzsXQmdtpnOP7vE8u758JHATODnmkIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIi\nIiIiIhLl4qV9fRXfQ0RERAKz3feISfGQCFWpXbv2tg0bNrgdh4iISDRajzVFjclkKB4SoQQgberU\nqdSpU8ftWOJGSkoKo0Zpi6hw0jUPP13z8NM1D6/169fTunVrgESy9gOMKXGzxUadOnVISEhwO4y4\nUa5cOV3vMNM1Dz9d8/DTNZdg06arIiIiEreUCImIiEjcUiIkIiIicUuJkIREcnKy2yHEHV3z8NM1\nDz9dcwm2uFk1lpaWpgI7ERGRAKSnp5OYmAgxvGpMI0IiIiIStyIhEWoEvAlsBTKA2wrwnsZAGnAQ\n2AR0C1l0IiIikl3M3LsjIREqDXwJ3Ov72pvP66sD7wAfA/WAZ4DRQItQBSgiIiLHiZl7dyQ0VHzP\n9yio7sBmoI/v643AZUA/YF5QIxMREZGcxMy9OxJGhAJVH1iU7dgi7IIWDX84IiIiko+IvXdHwohQ\noCoBO7Md24n9LBVzeO7/zZgBO3N9VsLhpJOgQwcoXtztSERE4tCECdCwIdSsGe7v7PjeHWrRmAg5\nkpKSwjfflGPv3qxjRYsmU6yYelKE04EDcMopoFYgIiJhtmgRdO4MTz4Jjz9+wtOpqamkpqYed2yv\n/00zRkVjIrQDqJztWCXgKPBbbm8aNWqU+ghFgPr1ITVViZCISFht3my/eP/zH+jfP8eXJCcnn9Cw\n0q+PUGE5uneHQzTWCK0Ars92rAnwBXAs/OFIIO66C957D37/3e1IRETixMGD0KIFlC0L06ZBUVdK\nciL23h0JI0JlAP/JyhrY0rrdwC/AEKAq0M73/DjgPmA48CpWgNURuDNM8UohtGoFKSkwdy506eJ2\nNCJRbOpUWLbM7SgkGmzYAOvXw4oVcNppwTprzNy7IyER+iewxPe5Fxjh+3wSdpEqA9X8Xr8ZuBEY\nifUv2ArcD8wPfahSWJUqwTXX2PSYEiERhxYsgDZtoG5dW4EgkpciRWD6dKhXL5hnjZl7dyQkQkvJ\ne4quQw7HPsH2PZEolJxs9XrbtkHVqm5Hk8XrtRHkffusqLtaNSgWCf+HiPjbuBHatoXbb4fZs8ET\nD1tGSgRaSozcu/VrXsKuRQvo0QNmzbJpssL6+2/44w977NkDf/5pyUygj7/+gmN+M9V9+sDw4YWP\nTyRo9u2D5s3hrLNg4kQlQSJBoERIwq5cOWja1KbHMhOhjAxLYn77zR6//56V3OT3OHgw9+9VqpQt\n18/+qFABzj035+dOPtlWmY4dC/36QZUqYbksIiYjw7Lw1atPfG7nTnusWmX/WEWk0JQIiSuSk+HO\nO+Giiyzx2b37+NGYTKVKQfnyljyVL2+Pc8+Ff/wj6+vsj1NPzUponDZubNzYptSHDoVRowr1o4oE\nZvBgGD3a/ifJ/g/4/POtI2nt2u7EJhKDlAiJK267De6/30b2Tz8dKlbM+lixoi1sKF/eEiE3lCsH\nvXvDM8/Agw+Gp5bJ67XapL17c37Ur2+1sRLD3noLnnjCkqHHHnM7GpG4EA8TzAlAWlpamhoqSkD2\n7rXRp7Zt4fnnC/4+r9fqlHbvtik+/4+5Hduzx77f0aO5n7d8eVi3Dipnb0kmseH77+Gyy6BRI1sV\nViQa27xJrPFrqJgIpLscTkhoREgkF2XLWqnG009bgfexY3knM/7HcprmK17capMqVLARrwoV4OKL\ns0a/ypa1kaiyZU98HD5so0H33GM9mFQj66Lt263I7eefg3vegwdtqeKUKUqCRMJIiZBIHnr2hJEj\n4eqrs44VKWKJi39Sc/75xyc4mR/9Py9TpnAJzJgxcMcdMGeOfRQXHD5sF3/XLnjooeBmpEWK2LnL\nlg3eOUUkX0qERPJQtiykpcGvv2YlNGXLuvMHe8uW1jrm3nvh3/+2WioJs379bMXWxx9b0ZaIRD0l\nQiL5qF7dHpHgxRdtpV2vXrZlkITY/v3WuwfgnXfghResr4KSIJGYoURIJIpUrmyF223aQFIS3Hqr\n2xHFsLQ0mxP966+sY+3bQ/fubkUkIiGgREgkytx9N8yYYffjRo2swFqC7LffrEK+Th0YONCOlSxp\nDaZUqS4SU5QIiUQZjwfGjbMpsr594bXX3I4oxhw7Zs0MDxywJXrVquX/HhGJWkqERKLQWWfZPmhd\nutgUWZMmbkcUxbxeSE+3Zk5gPXyWLIHFi5UEicQBJUIiUapTJ5si69IF1q7V1lOObNliy/DeeOP4\n4889B9dc405MIhJW6tolEqU8HnjlFWvi+PDDbkcTZY4dsyV4F14IX3wBM2fCDz/YY9s2m3MUkbig\nREgkilWvDv/9r63o/vjjwN579GjOHbBj3tdfQ4MGttndXXfZviWtWmX1SahSxe0IRSSMNDUmEuXu\nuccGNDp1gjVroHRpO37ggO0C8dNPWR/9P9+yxTpiL19uzSJj3qFD8NRTMHQo1KwJy5ZZQiQicU2J\nkEiUK1LEVo5deik0bGhTZj/9ZCvA/V9TtSqccw6cfTZceaV9/cQT0K6dlcjE9KrwpUuha1e7MI89\nZnOJJUu6HZWIRAAlQiIxoFYtmDTJHtWqQbNmWUnPOefAmWfapq/Z1agBN99sK9D69Qt31GHw++/w\nwAMwYYKN/ixcaL2BRER8lAiJxIikJHsE4qab4MEHbYDkyivtERO8Xpsv7NUL/v4bxo+Hzp21q7uI\nnECJkEice+op+OwzS6K++so2lo0ov/1mFeH+W13k59tv4aOPbKfa0aNVAC0iuVIiJBLnihe3fkT1\n6kHbtvDmmxE0cHL0qK3oSkuzAueCKlXKpsG0GZuI5EOJkIhw1lkwZQrceCMMGwYPPeR2RD79+8Mn\nn8AHH9gGqCIiQaZESEQAaNoUHnkEHn0U6taFU0+F776DihVdGliZPduyshEjlASJSMgoERKR/zdo\nEHz6qa0k8zdtmvUeDJt166BDBytcSkkJ4zcWkXgTKZUAIhIBihWDt9+Gd96x/csOHrS6oc6drZA6\nPxkZcORIIYPYuxeaN7cuz6+9FuMNjkTEbRoREpHjnHKKTZNlGjfOkqLmzWH1altV9tdfsHFj1mPD\nBvv47bfW2XrJErjkEgffPCPDOjzu2GHfrEyZoP1cIiI5USIkInk66SSYNw8uuwwSE20h19atWc9X\nrgwXXACXX26jR1Om2NTaypUOVq3/97+22uuNNwJbJSYi4pASIRHJ1znnWG4ydqx9fsEFWY+yZY9/\nbatWlhTddpvtbJG591m+Fi2y7S8GDIBbbgn2jyAikiMlQiJSIPXr2yM/Z51lvYgaNoTrrrOptvXr\n7bl33rEVaSfYvBmSk+GGG2DgwGCGLSKSJxVLi0jQJSTArFlw+LBNrd19N5Qvb7VH/tNqgFVkt2gB\n5crB1Kn/383xjz9g+XKrlx41yl4mIhJsGhESkZC46SZ7ZLr/frjiCmva+MknNqXmzfBysG13Sq7b\nwKyUFXz6+GmsW2cjSDt32vs8HsuNFi+G+fOhRAl3fh4RiU1KhEQkLKpWhXfftU3gGza0BWFXfvUS\nww+9TmumMHvkpdSqZZvDN2pkH+vUgVq1snob3XWXbQdSTL+5RCRI9OtERMLmoousfujJJ6FRseX0\nP5LCppt7MmB4aybVyD3Buf56mDPHZtDat4fJk6Fo0bCGLiIxSomQiIRVgwaweMoOSGgJ9S/nvHnP\nQfH833fLLdbhOjnZVqKNH69eiyJSeEqERCT0MjLg88/h77/t68yVYbNmQfECZEE+rVpZ0XT79laE\nPWqUkiERKRwlQiISevfdBy+9lPV1yZLw4YcOOi5a4+mDB6FHD6szevppJUMi4pwSIREJrUmTLAka\nNSqrUWK5cnDaaY5P2b07HDgAfftaXdGTTyoZEhFnlAiJSOikp1vW0qkT9OwZ1GylTx/b7uOhh8Dr\nhUGDlAyJSOCUCIlI8OzZAzNn2hb0Xi8MHw4XXwwvvhiSLOXBB+20Dz5oZUhPPaVkSEQCo0RIRILj\n6FFb3750aVbXw2rVbN17qVIh+7YPPGANF/v1s9xLNUMiEgglQiISHI88Yi2jP/oIGjcO67fu29eS\noT59bGRoyBAlQyJSMEqERKTwZs+G556DESPCngRl6t3bkp/evS0ZeuYZdaAWkfxp01URKZxvvoEO\nHSApCVJSXA0lJQWefx6GDbNZuQcegLVrXQ1JRCKcEiERcW7vXqsLql7dtomPgPmonj0hLQ3uuAMm\nTrRa7cRES5B+/dXt6EQk0igREhFnMjKsu+HOnTBvnnU3jBAJCTB6NGzbZjvWn3OOjQ5VrQq33QZz\n52Y1uRaR+KZESESc+e9/YeFCmDoVatZ0O5oclSgBzZpZnrZtG4wcaR9btrSk6N57YdUqW20mIvFJ\niZCIBO799+Gxx2DAALj5ZrejKZCKFW2njy++sLKmzp1hwQK4/HK48EJYvtztCEXEDZGSCN0D/Agc\nBFYDDfJ5fVtgDbAf2AZMAJz36xeRgvvxR7jrLrjhhqzNU6PMhRfC0KHw88+W0512Glx7LbzxhtuR\niUSdqL9/R0IilASMBAYD9YBlwLtAtVxefzV24V4GLgTuAP4JvBrqQEXi3sGDVhxdrpxNiRWJhF8h\nzhUtCk2a2P6vN90EzZvDuHGwbJl1AujWzXYJEZEcxcT9OxK6bPTBLsIE39e9gf8APYD+Obz+MmAz\n8KLv65+wi/pASKMUiXder+0btnEjrFhRqE1TI02pUrYzSK9etqs9wEknQfnyMG0azJoFN97obowi\nESgm7t9u/zlXAkgAFmU7vgi4Mpf3LAIqAU0Bj+/zO4C3QhSjSPx6+WWoXBlOP92KbF5/3Y5deqnb\nkQVd0aLwwgu2Q8hXX8Gff8J338F118Ett8CYMVZHNGqUjRStWOF2xCKuipn7t9sjQhWBosDObMd3\nAZVzec8abI5xNvYfohiwEOgZohhF4tPHH8M999hUWEKCHatZE26/3d24QsjjOb4xdrFittS+Tx8r\ntAYbPTr9dJg0CV59Fdq0cSVUEbfFzP3b7UTIiSuAScBA4H2gKjAMGAd0zu1NKSkplCtX7rhjycnJ\nJCcnhyxQkai1dSu0agUNG8L06XG9V0XRotaMsWVLOPlkqFs3a5awbVtYv952vY/ycikRUlNTSU1N\nPe7Y3r17g/ktHN2/Q83tNrAlsMrxllhWmOl54BLg3zm8ZyYWdyu/Y1dhRVpVODE7TQDS0tLSSMj8\nq1ZE7G7+228nNtE5dsxGgbZssRbNZ5zhTnwRzuuF4cPhwQetV9GUKRHVU1IkKNLT00lMTARIBPyX\nDoTj/h0Wbv8NcxhIA5pkO349kFtXDw9wLNuxDL/nRCQ/Xi907WpJTqVKxz+qVrWlUnPmKAnKg8cD\n/fpZT8nFi6FBA/jlF7ejEgmbmLl/R8J49whgCtZ/4HOgK3AWNlQGMAQbPmvn+3oBNrTWHSu8qgKM\nAlYCO8IVtEhUe+klK3B55hm46KITn69ZE+rUCX9cUeiWW+Czz+zjv/6V1aRRJA7ExP07EhKhWUAF\nYAB2Ub4GbgQy/7aqzPE9CaYDZYH7gOHAHuBD4KEwxSsS3ZYvt23a778fHnnE7WhiwiWX2FYdLVpA\no0aWZ3bs6HZUIiEXE/fveJhKUo2QSKYdO2wF2HnnWRfBEiXcjiimHD4MPXvC+PG24G7kSF1iiW55\n1AjFDLdrhEQkXI4csZVgXq91CNQdOuhKlLDO1OPHwyuv2LYdO10p/xSRglIiJBIvHnjAugDOmQNV\nqrgdTUzr2tUaM37/PSQm2rSZiEQmJUIi8WD6dGuGM2oUXHWV29HEhSuvtO4D1apZ3dCrr57YqUBE\n3KdESCTWrVkDnTtbC+R77nE7mrhStaqNDLVrB126QPv2sH+/21GJiD8lQiKx7I8/bEv1Cy6w4hVP\nPKyPiCwlS1rN0JQpNit5+eWwYYPbUYlIJiVCIrEqIwNat7ZkaN48KF3a7YjiWuvWViuUkQGXXQbZ\ndjIQEZcoERKJVYMGwbvvWn1Q9epuRyNY78pVq+C22+Cuu2ym8u+/3Y5KJL4pERKJRW+9BU8+CYMH\nww03uB2N+Dn5ZJg61Zouvvaa1a7/+KPbUYnELyVCIrHm++9tHubWW9U5OkJ5PLZ7/fLl8Pvv1uPy\njTdg926YP982cn3rLbejFIkPSoREYsn+/bbPwxlnwOuvQxH9Lx7JEhNtiX3jxjZdVrGi/eebONH2\nLuvRAw4ccDtKkdim35IiscLrtTXaP/xgwwply7odkRRA+fL2n2vaNMtdf/oJdu2yRX6TJ1th9f/+\n53aUIrErEjZdFZH8jB8PM2bk/ZoDB6wSd+bMnHeUl4jl8VjxtL9u3aBhQ0hOtl3tn33W9jHL7IDg\n9aobgkgwKBESiXRvv21zJNddZ1NeeenUyfYTk5hw4YWwcqWVeqWkwPvv2zTa0qXw6adWWzRxItSo\n4XakItFLiZBIJNu0yQqfb74ZFixQzU8cKlXKdrFv0sQ6U3/yCTRoAH372tTZpZfazikdO2qESMQJ\nJUIikWr/fusKXbGiCp+Fpk1h61abEite3I717Qu9e9sOKgsW2Axq1aruxikSbfSbVSQSeb22hfmm\nTVZJW66c2xFJBChWLCsJAjjlFNvMdeFCWL3aSsOmTNHmriKBUCIkEolGj7aO0BMmQN26bkcjEe7W\nW+Gbb+Cmm6BtW1uKv32721GJRAclQiKR5pNPoF8/6NMHkpLcjkaixGmnWcfq+fNt8eBFF9nXGh0S\nyZsSIZFIsm2brfpq0ACGDnU7GolCzZrZ6FDTptCmjX2t0SGR3CkREokUhw9Dy5ZWCDJzpn0UcaBC\nBWvQOH++Lb+/6CL7WqNDIidSIiQSKXr3tv0W5s7Nv1+QSAFkjg7dcIN1YbjtNvjlF7ejEoksSoRE\nIsHkyTB2rBVJX36529FIDKlQweru58+3PPvCC+2f2bFjbkcmEhmUCIm4LT3dtiLv2NGWzIuEQLNm\nsG6drSpLSYH69bWHmQgoERJx1+7dtt143bowZoxaA0tIlS1r/8w++wwOHoTERHjwQe1wL/FNiZCI\nW44ds502//oL5syxvRREwqB+fZsmGzTIpsnq1rV9zETikRIhEbcMGAAffGC7yp9zjtvRSJwpUQL6\n94evv4bq1a2g+u67YdcutyMTCS8lQiJuWLAAnnnGHtdd53Y0Esdq1rR8fNIkeO89uOACmz5TMbXE\nCyVCIuG2caNVrN5+uxVoiLjM44F27WDDBmtldd99cNllsHy525GJhJ4SIZFw2rfPdpQ/80yYOFHF\n0RJRTj8dXnnFmjAWKwZXXQXt28POnW5HJhI6SoREwsXrtSXyW7ZYU5dTTnE7IpEc/etf8PnnMH48\nvPmmTZeNHg1Hj7odmUjwKRESCZfnnrPVYZMnQ+3abkcjkqeiRa2t1bffwp13Wu+hxERYtsztyESC\nS4mQSDh8+CE8/LA9mjd3OxqRAqtQAcaNsx3tS5WCRo1su44tW9yOTCQ4lAiJhNrPP9uf1NdcA089\n5XY0Io5cdhmsWAGvvgqLF0OtWjBwIOzf73ZkIoWjREgklA4dstVhZcpAaqrNN4hEqSJFoFMn+O47\n6NULhg61hOj11yEjw+3oRJxRIiQSSvffbx3r5s2DihXdjkYkKE49FYYMgfXrbWVZu3a2V/Cnn7od\nmUjglAiJhMorr9g8wrhxkJDgdjQiQVe9OsyalVVA3bAh3HEH/Piju3GJBEKJkEgorFplXem6d7dG\nLCIxrEED6z30+utWR1S7tvUK/eMPtyMTyZ8SIZFg27XL6oISEmDUKLejEQmLIkWgTRtrnN6/P4wd\nCzVqWB2RdreXSKZESCSYjh6FpCQ4fNh6BpUs6XZEImFVpoytJtu0yZbZP/647Wf28stqyCiRSYmQ\nSDA98ogVTMyaZdtoiMSpSpXghRds/7Krr4Zu3eCii+zvA6/X7ehEsigREgmWWbOse/Rzz0Hjxm5H\nIxIRatSAadPgyy/hvPOsmPpf/7IeoyKRQImQSDB8843tI3bnndZgRUSOU68evPMOLF1qG7pedx1c\nfz2kpbkdmcQ7JUIihbV3r22bUb26LZfXjvIiuWrcGJYvt32Ht261jtWtWtmeZiJuUCIkUhgZGdC2\nra0Umz/fKkVFJE8eDzRrBmvWwIQJttP9hRdap4lNm9yOTuKNEiGRwhgyBN54A6ZOhfPPdzsakahS\nrBh06GCjQSNGwPvvwwUX2CzzDz+4HZ3ECyVCIk69956tDR44EG6+2e1oRKJWqVLQs6clP889Z7VE\nF1wAnTurS7WEnhIhESd++AHuuguaNoUBA9yORiQmnHQSpKTY/15Dh8Kbb9qmrl26wObNbkcnscph\nIuStBd5u4H0MvAOOf4jEuAMHrHN0+fI2JVZEf0+IBFPp0tCnjyVEQ4bAwoWWEHXrpikzCT4Hv8G9\nXYB1wJNAS6C579HC99GJe4AfgYPAaqBBPq8vCTwNbAYOAd8DHRx+b5GC83pt/7CNG604unx5tyMS\niVllykC/fjY99vTT9r9crVq2lce6dW5HJz5Rf/928qfsY8Cj4KkMnnrg+Yfv4fs8YEnASGAwUA9Y\nBrwLVMvjPbOAfwMdgVrAncAGB99bJDBjx8KUKbZM/pJL3I5GJC6UKQMPPGDTYyNGWC+iiy6ygVn1\nIXJVTNy/nSRC5YHZQYyhD/AqMAHYCPQGfgF65PL6G4BGwI3AEuBnLAtdEcSYRE702WdWwNCrl9UH\niUhYlS5tRdWbNtnfImvWWB+iG26wnW0k7GLi/u0kEZoDNAnS9y8BJACLsh1fBFyZy3tuxS7cw8AW\n7OIPA0oFKSaRE23fDi1bQv36MGyY29GIxLUSJaBTJ1i/HqZPt8aMjRpBw4a2mFN7mYVFzNy/izl4\nz3fAU+CtD6wBjhz/tGd0AOeqCBQFdmY7vguonMt7amBzkAeBZsDpwFigAjbUJhJchw/bBkkej+0n\nVry42xGJCNaHKDkZkpLgrbesjqhpU5u17tfPdrzR/64hEzP3byeJUDfgL2x4q1EOzweSCDlRBMgA\n7gb2+Y71wUaqegB/5/SmlJQUypUrd9yx5ORkkpOTQxepRK/XXoP//c8+//ZbWLUKPv4YKuf2/7eI\nuKVIEbj1VrjlFqsfGjbMGr7372+z2V26wKmnuh1l5EtNTSU1NfW4Y3v37g3mt3B0/w41tzdFKgHs\nx1afLfQ7/jxwCVZQld1kbNitpt+xOsA3vmPZG7QnAGlpaWkkJCQEKWyJaRMnWmvbOnXsT06Px/68\nbNPG7chEpIDWrrXmjNOnW21Rt25W3le1qtuRRZf09HQSExMBEoF0v6fCcf8Oi0I2QPF67OHYYSCN\nE2uOrgeW5/KeT4GqgP+mTrWwLHNLIWIRsSUoPXpYAcK6dVaN+b//KQkSiTJ168KkSbb0vmtXGDcO\nzj3XtvT45hu3o4sJMXP/dtpQsR1412I9AA6Bdw142zqMYQTQGesjUAdbincWMM73/BAsi8w0HdgN\nTPS9vhFWbPUaLg2rSYz47Tdbj3vxxfDii25HIyJBcOaZ8Oyz8PPPVkO0eLElSf/5D7z9tu2bLI7F\nxP3bSUPFPlhx09tAK9/jPeAl33OBmgWkAAOAL7FCqhuxJXhgRVf+PQn2YxlnOaz6fCo2LNfTwfcW\nMceOWdXl/v0wd65tfiQiMaNsWetF9MMP1grs999ti8DateGFF2DfvvzPISeIifu3g2kt74/AE+CZ\nnO14O9/x6oUPK6hUIyT569/fNjdavBiuucbtaEQkxLxeWLECnn/e/vYpU8ZKA++/H2rUcDu6yJFH\njVDMcDI1VgX4LIfjK7C5P5HoMm+ebWj03/8qCRKJEx4PXHklzJxpdUT33AOvvw7nnw+33QZLlqgf\nUbxwsnx+E9ZW++lsx1thPYZEoseGDdCunTVL7NfP7WhExAXVqtnfQo8/DtOm2SjRtddaLVGvXnD3\n3XDSSW5HGb+8MN8+5DqLlfmc12P7ngbEyYjQAOBJ8L4H3sd9u86/Dzzhe04kOuzbB82b22/BCRPs\nT0QRiVulS1vPoa+/hg8+gOrVbcXZWWdZfdH337sdYdzam8/jT7/PA+ZgRMgzF7yXY02QmvkOrgf+\nCZ4vnQQhEnZeL7Rvb735v/gCTjnF7YhEJEJ4PDYidO21tq/Z2LHWY/W55+D666F7d2veqK7V4eGB\n9qE8v8Pl85408NwNnkTfo7WSIIkqzz5rtUGvvw4XXOB2NCISoc47D4YPt7+ZJk+Gv/6yLhvnngsD\nB8Ivv+R7ColwBRwR8p4Knj+zPs9L5utEItQHH9gqsf79oVmz/F8vInHvpJNs2462ba3H6rhxMGIE\nPPWUjQ517w5Nmth2HxJcXluaX6CXemyleEAKOjW2B7yVwbPLPs89CGwTNpHI9NNPthPjddfBoEFu\nRyMiUeh87ScqAAAgAElEQVTSS+Gll2xgedo0+7xpU6sp6tbNulefcYbbUcaUhfm/BLAcJGAFrA71\nXg18Bp4jvs/zOuVSJ4GEkPoIiTl0CBo0gN27YfVqqFDB7YhEJAZ4vbBypSVEM2dat+pmzawv0fXX\nQ9EoHh6Ihz5CBRwR8k9uIi7REcmf12uNQr75Bj77TEmQiASNxwNXXGGPkSOt9PC112yU6KyzbF1G\nhw5q1BipnGyx0RS8Df2+vg+8/wNvKnjLBy80kSB6+WXbVX7cONDIoIiEyGmnQUqK7de8ahXcdBOM\nHm1F19dcY1NpBw+6HWX08kIxLzzghS+8sMMLf/g9fndyTidlXcMA31pj78XAcGzfsRrYhmsikeXz\nz61v/j33WPNEEZEQ83jgn/+0v722b7dRIq8XWreGKlXs19Hq1epe7cAArH3PLKAsloPMBY4BTzo5\noZNE6Fxgne/z24G3wNMf6AE0dRKESMjs3Gldoy+7zMasRUTCrHRpaNMGPvoIvvsO7rsP3njDEqV6\n9ayT9a+/uh1l1Lgb6OKxQZmjQKoHOgODgMudnNBJInQYKOP7/Dpgke/zP4B8ltaLhNHRo5CUZB/n\nzIESJdyOSETi3Pnn25L7n36Cd96BmjWta3XVqrYMf9YsW9chuaoMrPF9/hc2KgQ2M3WzkxM6SYQ+\nA4bb1hr8y/fNAWoCW5wEIRISDz1khdGzZ9tvGRGRCFG0qBVTz5kD27bBqFE2KpSUBJUqQefO8PHH\ntgJNjrOFrA3eNwH/8X1+GfC3kxM6SYTuw4ajbgd6gCcz+bkReNdJECJBN2OGdTsbPhwaNsz/9SIi\nLqlYEe6918oZN260jV6XLIGrr7aVZo8+avtDCwALgGt9n48CBnnhe2AKMMHJCeNhl0n1EYo3a9fC\n5ZdbI4+pU7WZqohEHa/XBrSnTLHeRHv3Wk1RmzbWE/b008MTR6T3EfLCFcBVwHceeMPJOQo4IuS/\nrYb31LwfIi7as8d2lD//fFsyryRIRKKQx2P9X8ePhx07bAqtalXo29c+3nijrUT7M843tfLA5x4Y\n7jQJgoJPje0B7xlZn+f6+MNpICKFlpFhfy799pttqFqmTP7vERGJcKVK2UavCxZYPdHzz9vmr+3a\n2VYeLVpYkfWBA25HGnpe6O+FDjkc7+iFh5ycs6B7jV1DVpJzjZNvJBJyTz8Nb78Nb71l3ctERGJM\nxYrWg+iee2zn+9mzrSQyKcn+9rv1Vps6+89/oGRJt6MNiW5AUg7H1wEzgKGBnlBbbEhsePddGDgQ\nnnjCxoxFRGJctWrQp489Nm2yWqIZM+C226BsWRspuvNO62j955/w5pswf75VD0Rxb9lKwM4cjv8K\nVHFyQidbbHQEb6scjt8B3ui9tBK9Nm2Cu+6yXvaPPeZ2NCIiYXfeedC/v23tsXatNdNftsxGhs44\nwx7t29sS/VNOcTvaQtkCNMjh+JXANicnLOjUmL9HsC6O2f0KjAcmOwlExJEDB+zPngoVbHlFEScd\nIUREYsdFF8HgwTBoEKSnW21RlSq2kDYGWqq9AozyQnHgQ9+x64Bnse02AuYkEaoG/JTD8Z+Ac5wE\nIeKI1wtdu8L331sDjnLl3I5IRCRieDyQmGiPGDIMOA0YA2RWQR3EaoOGODmhk0RoF3ApsDnb8UuA\n3U6CEHHkxRdtK+fUVLj4YrejERGREPNABvCQF54C6mBJ0HcecLwxiZN5hBnAaPBeA96ivse1dowZ\nTgMRCciyZVYh2Lu3VQOKiEg8qQSUBzZ54JDXWT4DOBsRehybAvsA2/YeLIDJQH+ngYgU2LZtcMcd\ncOWVMDTglZIiIhKlvFABmAX8276kJvAD8IoX9nigb6DndJBBef4GTxJQG7gbaAGcB56O9pxICB0+\nbElQ0aLWQax4cbcjEhGR8BmJ7Xd6NuDfQnIm0NTJCZ2MCGXajO1V9gN4jhTiPCIF16cPfPEFfPKJ\nbdEsIiLxpAlwgwe2eI8//j0OF2w56SNUGrwTsExsHbaKDPCOBu/DToIQKZDXX4cxY2D0aLjiCrej\nERGR8CvD8SNBmSoAjmalnBQXDcFWjV2NVWtn+gBQ1aqExpdfQrdu0KGDfRQRkXi0DGjrf8ALRYEH\ngI+cnNDJ1FhzIAk8K6yRy/9bD2iDJwm+3butaeKFF9qIkHaUFxGJV/2ApV64DCiB9Q+qi/UWusrJ\nCZ0kQhWxXkLZlcEquEWC59gxuPtu2LcPli6Fk05yOyIREXGB1xKfMcCtwI3YyvUywFxgjAe2Ozmv\nk0RoNXAT1jfIX2dghZMgRHI1cCAsXgzvvQfnqHG5iEi88sBhr43+7PbAgGCd10ki9DDwHngvxPb6\n6Anei7ANzxoHKzARFi6Ep5+GIUPg+uvdjkZERNw3BeiE5SJB4SAR8iwH71VYYdImbClbOnAFeL4O\nVmAS5zZuhDZtoHlzeOght6MREZHIUBzo7LWNVtOA/b7jHsDrgT6BnjDARMhbHHgZGAyetvm9WsSR\nv/6y4ugzz4RJk1QcLSIimS7GBl8Aavkd9+CwTjnARMhzBLzNgcFOvplIvrxe6NgRfv7ZGieeeqrb\nEYmISITwWOueoHLSR2gB0CzYgYgAMHw4zJ4NkydD7dpuRyMiIjHOSbH0d8AAX53QarLm53w82VeT\niRTMkiVWD/TQQzY1JiIiEmJOEqEuwB4gEUjwO545P6dESAL3yy+QlATXXANPPeV2NCIiEiecrBo7\nN+tzr29qzZMRhFgkXh06BLffDqVLQ2oqFCvMXsAiIiIF56RGCPB2Bu83wCF7eNeCt0swA5M4cv/9\nsGYNzJsHFSu6HY2IiMQRB396ewcDvYEXgM99B68ARoD3XPA8GqzgJA688gq8+ipMmACJiW5HIyIi\nccbJHEQPoCt4pvsdWwjeNVhypERICmbVKrjvPuje3XaVFxERCTMniVBR4Iscjqc7PJ/Ei507oWdP\n+OMP+/qrr+Af/4BRo9yNS0RE4paTGqFp2KhQdl2B6TkcF4EjR6BVK/joIyhXzh433wxz50LJkm5H\nJyIiccrJCI4X6AzeJliNkAe4HDgbmAzeEfz/UnpPwHt+SIx68EFYvtwSoQYN3I5GREQEcJYI+e/z\ncZ7v42++R13f1473/JAYlJpq01+jRysJEhGRiOKkj9DVQY8C7sF2s68MfAOkAJ8W4H1XAR8DXwP/\nCEFcUlhr1kCnTtC6tRVGi4hILIn6+7fDPkJBlQSMxDZyrQcsA94FquXzvnLA68AHaPQpMu3ZY1tl\n1KoF48drF3kRkdgSE/fvSEiE+gCvAhOAjViPol/IuSDb3zhgKrACm4qTSJKRYaNAv/9ujRJLl3Y7\nIhERCa6YuH+7nQiVwPYrW5Tt+CLgyjze1wE4F3iSCLiIkoPBg+Gdd2DaNKhRw+1oREQkuGLm/u12\n35+KWF+indmO78LmG3NSExgCNAC0x1kkevtteOIJGDQImjZ1OxoREQm+mLl/u50IBaoo1qtoIPB9\nIG9MSUmhXLlyxx1LTk4mOTk5eNEJfP+9TYndcgs8qibjIiLRIjU1ldTU1OOO7d27N1ind3z/DjW3\nh6VKAPuBlsBCv+PPA5cA/872+nLA78Axv2NFsJ/jGHA9sDTbexKAtLS0NBISEoIWuORg/36oX992\nk//iCyhb1u2IRESkENLT00m0fSATyWqdA+G5f4eF2yNCh4E0oAnHX8jrgfk5vH4vWb2KMt0LXAPc\nDmwOfohSIF4vdO0KmzbBypVKgkREYlvM3L/dToQARgBTgNVYp+quwFlYVTnYfGJVoB22zG5dtvf/\nChzK4biE0+jRMH06zJgBdbP/WxcRkRgUE/fvSEiEZgEVgAFAFay50o3YEjywoqu8ehJ4iYA+BHHt\nk0+gb1/o0weSktyORkREwiMm7t9u1wiFg2qEQmnrVkhMhDp1YPFiKBYJubWIiARDHjVCMcPtPkIS\nzQ4fhpYtLfmZOVNJkIiIRB3ducS53r0hPd2mxs44w+1oREREAqZESJyZNAnGjrU9xC6/3O1oRERE\nHNHUmAQuPR26d4eOHaFLF7ejERERcUyJkARm927bUf7ii2HMGO0oLyIiUU1TY1Jwx45BcrJ1kP7k\nEyhVyu2IRERECkWJkBTcgAHw4YewaBGcfbbb0YiIiBSaEiEpmPnz4ZlnYOhQuPZat6MREREJCtUI\nSf42bIB27eD22+GBB9yORkREJGiUCEne9u2z4ugzz4SJE1UcLSIiMUVTY5I7rxc6dIAtW2DVKjjl\nFLcjEhERCSolQpK7YcNg7lyYNw9q13Y7GhERkaDT1Jjk7MMP4ZFH7NG8udvRiIiIhIQSITnRzz/D\nnXfa6rDBg92ORkREJGSUCMnxDh2y1WFlysD06VC0qNsRiYiIhIxqhCSL1wv33gtr18Jnn0HFim5H\nJCIiElJKhCTLK6/AhAm2TD4hwe1oREREQk5TY2JWroT774cePaB9e7ejERERCQslQgK7dlldUEIC\njBrldjQiIiJho0Qo3h09CklJcOQIzJkDJUq4HZGIiEjYqEYo3j38MCxbBkuW2DYaIiIicUSJUDyb\nOROGD4eRI6FRI7ejERERCTtNjcWrtWuhUydIToZevdyORkRExBVKhOLR3r22o3yNGrZkXjvKi4hI\nnNLUWLzJyIC2bW2l2OrV1kFaREQkTikRijfPPANvvAFvvQXnn+92NCIiIq7S1Fg8efddGDAABg6E\nm25yOxoRERHXKRGKFz/8AHffDU2bWjIkIiIiSoTiwoEDVhxdvjxMnQpF9J9dREQEVCMU+7xe6N4d\nvv0WPv/ckiEREREBlAjFvjFjYMoUmDYNLrnE7WhEREQiiuZIYtmnn0Lv3tYw8a673I5GREQk4igR\nilXbt8Mdd0D9+jBsmNvRiIiIRCQlQrHo8GFLgooUgVmzoHhxtyMSERGJSKoRikX9+sGqVfDxx1C5\nstvRiIiIRCwlQrFmyhR44QUYO9amxURERCRXmhqLJV99BV27Qrt2tmReRERE8qREKFb8/rs1TaxT\nB156STvKi4iIFICmxmLBsWO2fcbevbBkCZx0ktsRiYiIRAUlQrHgySfh/ffhvffg3HPdjkZERCRq\nKBGKdm+8AYMHwzPPQJMmbkcjIiISVVQjFM2+/RbatIFmzeDhh92ORkREJOooEYpWf/1lxdGVK8Pk\nySqOFhERcUBTY9HI64VOnWDzZmuceOqpbkckIiISlZQIRaORI23rjNmz4cIL3Y5GREQkamlqLNp8\n9BE8+KA9WrZ0OxoREZGopkQomvzyCyQlQePG8PTTbkcjIiIS9ZQIRYu//7YRoFKlYMYMKKZZTRER\nkcKKpEToHuBH4CCwGmiQx2tbAIuBXcBeYDkQ2010eva0vcTmzoXTT3c7GhEREYiBe3ekJEJJwEhg\nMFAPWAa8C1TL5fUNgfeBpkACsAR40/fe2PPaa/Dyy7aj/D//6XY0IiIiECP37khpPrMSyyTv9Tu2\nDlgA9C/gOdYCM7H/IP4SgLS0tDQSEhIKG2f4ffEFNGxoO8qPH+92NCIiEkfS09NJTEwESATSsz0d\nynt32ETCiFAJLFlZlO34IuDKAp6jCHAKsDuIcbnv11/h9tvh0kth9Gi3oxEREckUM/fuSEiEKgJF\ngZ3Zju8CKhfwHH2B0sCsIMblrqNHITkZDh2COXOgZEm3IxIREckUM/fuWFh6lAwMBG4FfsvtRSkp\nKZQrV+74NyYnk5ycHNronHr0UesZ9MEHUC236VYREZHgSE1NJTU19bhje/fuDdW3K9C9OxwiIRH6\nDTgGVMp2vBKwPZ/3JgGvAi2xoqtcjRo1KnpqhObOhWefheeeg3//2+1oREQkDuQ0OOBXI5RdWO7d\n4RAJU2OHgTROXEJ3Pba0LjfJwETgTqxKPTasWwft20OrVtCnj9vRiIiI5CRm7t2RMCIEMAKYglWf\nfw50Bc4CxvmeHwJUBdr5vr4LmAz0BL4gaz7yAPBneEIOgT//hObN4ZxzbMm8dpQXEZHIFRP37khJ\nhGYBFYABQBXga+BG4Bff85U5vi9BF2w0a4zvkWkS0DHEsYZGRoYtkd+xw5bMn3yy2xGJiIjkJSbu\n3ZGSCAG85HvkpEO2r2OvcGboUFiwABYuhFq13I5GRESkIKL+3h0JNUKyaBE89pg9br3V7WhERETi\nhhIht23ebP2CmjSBJ55wOxoREZG4okTITQcPQosWULYsTJsGRYu6HZGIiEhciaQaofji9UKPHrB+\nPaxYAaed5nZEIiIicUeJkFvGjYPJk+H116GeqxvvioiIxC1NjblhxQro1Qvuuw/atHE7GhERkbil\nRCjcduyAli3hX/+C4cPdjkZERCSuKREKpyNHbOuMjAyYPRtKlHA7IhERkbimGqFwevBBmxb76COo\nUsXtaEREROKeEqFwSU2FUaNg9Gho0MDtaERERARNjYXHmjXQqRO0bm0F0iIiIhIRlAiF2h9/WNPE\nWrVg/HjtKC8iIhJBNDUWShkZNgr0+++2n1jp0m5HJCIiIn6UCIXS4MHw7rvw9ttQo4bb0YiIiEg2\nSoRC5e23bRPVQYOgaVO3oxEREZEcqEYoFL7/3qbEbrkFHn3U7WhEREQkF0qEgm3/fiuOPv10mDIF\niugSi4iIRCpNjQWT1wtdusCmTbByJZQt63ZEIiIikgclQsE0erQ1TpwxA+rWdTsaERERyYfmbYLl\n44+hb1/o0weSktyORkRERApAiVAwbN1qm6k2bAhDh7odjYiIiBSQEqHC+vtvaNkSiheHmTOhmGYb\nRUREooXu2oXVuzekp8Mnn8AZZ7gdjYiIiARAiVBhTJoEL71ke4hdfrnb0YiIiEiANDXmVHo6dO8O\nHTvaknkRERGJOkqEnNi925omXnwxjBmjHeVFRESilKbGAnXsGCQnWwfpTz6BUqXcjkhEREQcUiIU\nqMcfhw8/hEWL4Oyz3Y5GRERECkGJUCDmz4chQ6xX0LXXuh2NiIiIFJJqhApqwwZo1w5uvx0eeMDt\naERERCQIlAgVxL590Lw5nHkmTJyo4mgREZEYoamx/Hi90KGDbaOxahWccorbEYmIiEiQKBHKz7Bh\nMHcuzJsHtWu7HY2IiIgEkabG8vLhh/DII/Zo3tztaERERCTIlAjl5qefICnJVocNHux2NCIiIhIC\nSoRycuiQrQ47+WSYPh2KFnU7IhEREQkB1Qhl5/XCvffCN9/AZ59BxYpuRyQiIiIhokQou1degQkT\nbJl8QoLb0YiIiEgIaWrM38qVcP/90KMHtG/vdjQiIiISYkqEMu3caXVBCQkwapTb0YiIiEgYKBEC\nOHoU7rwTjhyBOXOgRAm3IxIREZEwUI0QwMMPw7JlsGSJbaMhIiIicUGJ0MyZMHw4jBwJjRq5HY2I\niIiEUXxPja1dC506QXIy9OrldjQiIiISZvGbCO3ZAy1aQI0atmReO8qLiIjEnficGsvIgLZtYdcu\nWL0aypRxOyIRERFxQXwmQs88A2++CW+9Beef73Y0IiIi4pL4mxp7910YMAAGDoSbbnI7GhEREXFR\npCRC9wA/AgeB1UCDfF7fGEjzvX4T0K1A3+WHH+Duu6FpU0uGJGRSU1PdDiHu6JqHn655+OmaR5zw\n3L9DKBISoSRgJDAYqAcsA94FquXy+urAO8DHvtc/A4wGWuT5XTIyICkJypeHqVOhSCT86LFLv6zC\nT9c8/HTNw0/XPKKE5/4dYpFQI9QHeBWY4Pu6N/AfoAfQP4fXdwc2+94HsBG4DOgHzMv1uxQpAs89\nZ4lQ+fJBCVxERCSOhef+HWJuD4uUABKARdmOLwKuzOU99XN5/WVA0Ty/W+PGcMklgUcpIiIi/sJ7\n/w4htxOhitgPvzPb8V1A5VzeUymH1+/ERrcqBjU6ERERyUnM3L8jYWosLNavX+92CHFlz549pKen\nux1GXNE1Dz9d8/DTNQ+veLh3up0I/QYcw7JEf5WA7bm8ZwcnZpuVgKO+82W3HdjaunVr7aYaZomJ\niW6HEHd0zcNP1zz8dM3Dbisn3pPDcf8OC7cTocPYMromwEK/49cD83N5zwrglmzHmgBfYP9RstsO\n/BOoUqhIRURE4tN2TkxuwnH/jhutgL+BDkAdbCnen2QtvxsCTPZ7/bnAX8Bw3+s7+t7fPDzhioiI\nCLp/B1UPrCHTISwz9G/INBFYku31jbBM9BDWkKlrGGIUERGR4+n+LSIiIiIiIiIiIiIiIiLhFPWb\nvkWhQK55C2Ax1mhrL7AcWykggQn033mmq7DlqV+GKK5YFug1Lwk8jW0jcAj4HisklYIL9Jq3BdYA\n+4Ft2HYPp4UywBjSCHgTWx6fAdxWgPfo/hmBkrCq847ABVjV+j7y3vRtPzDC9/pOvve7uulblAn0\nmo/E9pJJBM4DnvK9v17II40dgV7zTOWwX1bvAepCFxgn13whluhfA5yNbR1QP7RhxpRAr/nVWJJ/\nH3AOlvR/jYv7VkWZG4BBQDMsEbo1n9fr/hmhVgJjsh1bh+1qm5OhwDfZjr2E/fKSggn0mudkLfB4\n0CKKfU6v+QzgSWAgGhEKVKDX/AbgDyz5FGcCveb9sFE3f/cDPwc5rnhQkEQoJu+fbu81Vlgxs+lb\nFHFyzbMrApwC7A5iXLHM6TXvgPXteBLwhCSy2OXkmt+KTeU8DGzBdtYeBpQKUYyxxsk1X4R1Jm6K\n/RuvBNwBvBWiGONdTN4/3e4sXVih2PQt+3NyPCfXPLu+QGlgVhDjimVOrnlNrJlZA+wvPQmMk2te\nA7veB7GphtOBsUAFbKpH8ubkmq/BaoRmY4lUMWx6smeIYox3MXn/jPYRIYk+ydg0TRIu7i0T44oC\n07HrnH3aQEKnCJZ03o2NDL0L9AHaYUXUEnxXAJOwf+sJ2PRkDWCcizFJlIn2EaGY2fQtiji55pmS\ngFeBlpzYbVRyF+g1PwUrTK8HvOg7VgSbOjiC7QW0NBSBxhAn/863Y6uW9vkd24Bd97OwonXJnZNr\n3ht4H9uyAaz2cD+wDHiUKB2hiGAxef+M9hEh/03f/F1P7sVbK3zP+9OmbwXn5JqDjQRNBO7E/lKW\nggv0mu8F6gKX+j3GYTUrlwKrQhZp7HDy7/xToCpQxu9YLWyUaEuwA4xBTq65hxN/b2f4PSfBpftn\nhNKmb+EX6DW/CxuJ6IH9NZH5ODVM8caCQK95dk+gVWOBCvSal8FWK83yvb4R8C0wPkzxxgInv1sO\nA92xKbGrsJvyijDFG+3KYCPH9bAEMsX3ue6fUUibvoVfINf8I+yvhYxsjwlhiTR2BPrv3N9A1EfI\niUCv+QXYKpr9WFI0DNUHBSrQa96DrCmxrcDrQJXQhxkTribr97H/7+jM3826f4qIiIiIiIiIiIiI\niIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiMSzJ7BNMDOAW90NJV9XY3EWdhuYYJ1HRERE\nXBCspKWO71y3AGcAJYJwzlC6msATmKXY3lj+imM/r4hEqGJuByAiASuK3aS9Yfp+wdjF+zzfxzeD\ncK5ocgTY5XYQIiIiblkKvAC8CPwB/AYMzvaaEsCzwBZsZ+fPgcZ+z7f3vfcmYB12cz0H28zzWWxz\nz0PAd9hu0JkuBN4B9gE7sM0oK2SL7XnfOXYD27HNWTNt5vhNcn/I4+e8GNuc8YDvZxyP7WwNNiXm\nf55juZzjat/zNwL/Aw5i16JuttfdDnzj+5l/BPpke34z8BgwHfvZtwL3+T1/ru/7XOJ3rJzvWKNs\nsWSOCJ0GpGL/jfYDa4A7/d4/KYef8ewczlPQ+B/BNr78E/gJ6IKIiEgUWordzEYANYG7sGSns99r\npgHLgKuA6kBfLAk43/d8e+Bv32uu8J2nNDATu0neht3cGwOtfO+pAvwKPAXUAuoB7wMfZottD/A4\nNmLTBruBX+d7viJ2E2+LTe/4J1H+ygDbgNlY8vVvbFfqiX7Pt/Od6wxynyq62veatcC1WAL0BpaA\nZY5eJwJHgUd916cdlpi08zvPZmAv8KDvNfdhyWPmz3UugSdCVbGE5RLf+zPP+U/f86cCnwHj/H7G\nIjmcp6Dx/wZ0B2oAD/neUwsREZEosxS7sfsbgo0IgCUgx7DExd9i4Gnf5+2xm+nFfs/X8h27Jpfv\nOwh4L9uxs3zvyUywlgIfZ3vNSl98mQpSI9QFG1E6ye9YU+zmfbrv62a+c+Xlat9r7vA7Vh5LFFr6\nvp7GiT/XUI6/xpuBt7O9JtXv2LkEngjl5C1gmN/XH2EJr7/s5ylI/D8Ck7O9ZgfQLY9YRMShIm4H\nIBLjvNj0jr/PsVEdD5Dg+/gtNo2T+WiMjQZkOgx87fd1PSyByp7IZErERmb8z7neF09mvY4Xm+Lx\nt53Ai3vrAF9ho1iZlmO/Xy4I8FwAK/w+/wPY6PseALWxkRd/y8m6nmA/14psr/nc7xxOFMVGcdZg\nozX7gCZAtQDPU5D44cT/LjvISipFJIhULC0SenkVGxfBEpoETqyd+cvv84PZnsv+dU7f8w1sWiW7\nHX6fH8n2nBdnfyAFo6C6oOcu7PfKHJnyP0/xfN7TF0gBemEJ6QFgFFanFaiCxB+s/y4ikg/9jyUS\nWh7g8mzHrsBGgLzAl9hoQyWsFsb/kddqo6/JqkHJSTpWY/NTDuc9kE/M/qvRjvjiy8s64FKsbinT\nVVjCsTGf9+akvt/n5bHRkvW+r9f7zu3vKt/3yYzbk+0cYNd8ne/zX30fq/o9Xy+fmBoCC7AC7K+x\n6asLOP5aHSb/Py4LEr+IiEjMWIoVSw/HbpzJ2LSK/yqgKViC0hwrlv4nNpLT1Pd8e2yKKLsJZBVL\nV8eSosz6mipY88JZvvPVwKZyXiNrRGIpJ/a9WUBWkTPYDXoMUBlLSnJyErYyazZwEVnF0hP8XhNI\njdAarPapLrAQSzoyE4x/YLVHj2F1UpnFxm39zrMZKwJ/wPeae7GE7nq/1yzHphXrYHVBK8m7RmgE\nduzD9F4AAAE/SURBVK3r+97ziu97zPM753jfec7BCs09OZynIPH/CPTMdm2+AgYgIiISZT7Cls+P\nxW6cv5FVBJ2pGLbE/AdsddhWYA6WVIAlQr/ncO6SWIK1FVuKvZHjVx+dD8z1vXc/NiIyPFts2Yt7\n53N8AnMzNnp1mLyXz9fFVqRlLp8fx/EjRM3Ifdl8pqvJWj7/te9n+pzji8QBWmDFxX+T8/LzH7FE\nYwY2vbgNuD/bazJrdfYDadiKsmMcnwgdIyuBKY9dmz+xqcUnsSXz/olQTSzB2s/xy+f9z1PQ+LMn\nQl+iREhERKJQTsmG5OxqgrMdRU6JhIhIjlQjJBJaHkJbSCwiIoWgREgktLyoCDYQulYiIiIiIiIi\nIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIikoP/A30qXkArjqVRAAAAAElF\nTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f02312efac8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    " plot_precision_recall_n(labels_binary[test_set_idx], y_score[:,1], 'LR')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we examine our precision-recall curve we can see that our precision is 1 and recall is 0.8 up to 0.4 percent of the population. Unlike the previous example where we are using a precision at k curve to prioritize our resources. We can still use a precision at k curve to see what parts of the corpus can be tagged by the classifier and which should undergo a manual clerical review. Based on this we can make decisions of what documents should be manually tagged by a person during a clerical rewiew, say, the percent of the population above 40%. \n",
    "\n",
    "Alternatively, we can try to maximize the entire precision-recall space. In this case we need a different metric. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_precision_recall(y_true,y_score):\n",
    "    \"\"\"\n",
    "    Plot a precision recall curve\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    y_true: ls\n",
    "        ground truth labels\n",
    "    y_score: ls\n",
    "        score output from model\n",
    "    \"\"\"\n",
    "    precision_curve, recall_curve, pr_thresholds = precision_recall_curve(y_true,y_score[:,1])\n",
    "    plt.plot(recall_curve, precision_curve)\n",
    "    plt.xlabel('Recall')\n",
    "    plt.ylabel('Precision')\n",
    "    auc_val = auc(recall_curve,precision_curve)\n",
    "    print('AUC-PR: {0:1f}'.format(auc_val))\n",
    "    plt.show()\n",
    "    plt.clf()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC-PR: 0.949873\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh0AAAF5CAYAAAA77Zg2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3XmcXFWd9/FPZ+kIWegkYHYgIaDJgMRuQEQMbmAERQk4\nsQcFwqIQR02CjIw6oOPCw/AAUWYGZsAEFGk2F57wmBAEQUBBSQuBYdEEAoGEhK1DyEJCup8/TvXT\n1ZXe6lbVudXVn/frdV9VdepW1a/vK+n69rnnnAuSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmS\nJEmSJEmSVHGmA4uBF4Fm4NPd7D8auAF4GtgJXF7S6iRJUlH0S7sAYHfgL8CXM49butl/ELAB+B7w\naA/2lyRJ2kUzcHwe+/8OuKxEtUiSpCIqh54OSZLUBxg6JElSFAPSLiCCMcDVwAhgR85zS4E7olck\nSVL5+TgwI6dtIPAacBawrtAP6Cuh47hOnpsO/DBiLZIk9UZjMHT03PXXX8+UKVPSLqPPmDt3LgsW\nLEi7jD7FYx6fxzw+j3lcTz75JJ///OeL9n7lEDoGA/tnPZ4ETANeBdYAFwFjgVOz9pmWuR0KvDPz\neDvwRGcfMmXKFGpra4tXtbpUU1Pj8Y7MYx6fxzw+j3nvVg6h41Dg7sz9FtqmwF4LnE5YDGxCzmsa\ns/avBf4BWE0ILJIkqQyVQ+i4h65n0czuoM1ZN5Ik9TJ+eUuSpCgMHSqJ+vr6tEvoczzm8XnM4/OY\n925VaRcQQS2wfPny5Q4+kiQpD42NjdTV1QHU0TaeMjF7OiRJUhSGDkmSFIWhQ5IkRWHokCRJURg6\nJElSFIYOSZIUhaFDkiRFYeiQJElRGDokSVIUhg5JkhSFoUOSJEVh6JAkSVEYOiRJUhSGDkmSFIWh\nQ5IkRWHokCRJURg6JElSFIYOSZIUhaFDkiRFYeiQJElRGDokSVIUhg5JkhSFoUOSJEVh6JAkSVEY\nOiRJUhSGDkmSFIWhQ5IkRWHokCRJURg6JElSFIYOSZIUhaFDkiRFYeiQJElRGDokSVIUhg5JkhSF\noUOSJEVh6JAkSVEYOiRJUhSGDkmSFEU5hI7pwGLgRaAZ+HQPXnMUsBzYCqwCvlSy6iRJUlGUQ+jY\nHfgL8OXM45Zu9p8I/Aa4F5gG/BD4MTCzVAVKkqTCDUi7AGBpZuups4HVwPzM46eBQ4CvA78samWS\nJKloyqGnI1/vB5bltC0jBI/+8cuRJEk9UQ49HfkaBazPaVtP+Fn27OA5AK64AkaPLnFlUi8zfTp8\n4hNpVyGpr+iNoSORG2+cS79+Ne3ahgypZ+jQ+pQqktK1YQPcdZehQ1LQ0NBAQ0NDu7ampqaifkZv\nDB0vAbl9FqOAt4FXOnvRAw8soLa2tpR1Sb3Kl74EjY1pVyGpXNTX11Nf3/4P8cbGRurq6or2Gb1x\nTMcfgaNz2o4B/gzsjF+OJEnqiXIIHYMJU1+nZR5PytyfkHl8EXBd1v5XAfsAlwJTgNMz2/+OUawk\nSUqmHE6vHArcnbnfAlyWuX8tIUyMpi2AQJgueyxwOWFtjxeBrwC/Kn2pkiQpqXIIHffQdY/L7A7a\nfg8U7ySTJEkquXI4vSJJkvoAQ4ckSYrC0CFJkqIwdEiSpCjKYSCpJKWmpQW2b4fNm9u2LVvaP85t\nb2mBuXNhjz3Srl7qXQwdknqNnTvhzTdh0yZ4441w29H25pth60mA2Lw5vG93+veHwYPhHe8IS8jX\n1cEnP1n6n1mqJIYOSSXV0hICQFMTbNzYdpsbFLoKEa3bli1df9bAgTB0aNgGD26/jRgB48fv2j54\nMOy+e/ft1dVQVQXr14eLR7a0xDl+UiUxdEjqUnNzCASvvQavvx5CQ26AyL7Nbdu4MbxHZ1pDQu42\nYULH7cOGdf6aQYPiHRdJ+TN0SH3Yyy/Dj34UAkVnW1NT56FhyBCoqQljG1pvx42DqVPbt+fus8ce\nbb0R/RzOLvUZhg6pjxo3Dp57Dr75zXDqYfjwcDtiBBx0UNv97K01QNTUhB6HAf4GkZQHf2VIfdQF\nF8A3vuEpCSXXOq6lqirdOtR7GDqkPszA0fe0DuxtHcybO4A338dnnQVXXZX2T6XewtAhSb1ISwts\n29Y2aDd7cG9n93MfdzWwt7q644G7w4fD3nu3H8jb0ADPPBPvZ1fvZ+iQpMh27GgfCPIJDK+/HhYz\n68iAASEcDB/eNvZm5EiYPLnt8fDhbYN5O5oJlE/v18MPh3qknjJ0SFICmzbB6tXJehs2b+74Pauq\n2g/WbQ0PY8e2f5x7v/Xxbrs5vkLlzdAhSXlonbFz8skdP986jTg7FEyc2HlgyL4/dKhTiFXZDB2S\nlIeRI2HxYti6ddfehpoapxFLXfG/hyTlyWuuSMnYkSdJkqIwdEiSpCgMHZIkKQpDhyRJisLQIUmS\nojB0SJKkKAwdkiQpCkOHJEmKwtAhSZKiMHRIkqQoDB2SJCkKQ4ckSYrC0CFJkqIwdEiSpCi8tL0k\nqWzs2AHr1sHatWEbNw7e9760q1KxGDokSSXX3Awvv9wWJl58se1+9uMNG9q/btIkWLUqnZpVfIYO\nSVJBtm0LoeGFF9pv2WFi3Tp4++221/TrB6NGwdixYTv88Lb748aF20WL4Be/SO/nUvEZOiRJiVRV\nwV13wW67tW+vqQnBYdw4mDIFPvaxXQPFqFEwoJtvoGHDSle70mHokCQlMn8+HHQQjB/fto0bB0OG\npF2ZypWhQ5KUyGGHhU3qKafMSpKkKAwdkiQpinIJHXOAZ4GtwMPAkd3s/2XgSWAL8BTwhZJWJ0mS\nClYOoWMWcDnwPWAacB+wBJjQyf7nAD8ELgCmAhcC/wF8suSVSpKkxMohdMwHrgEWAk8D84A1hHDR\nkS8AVwG3AKuBm4CfAN8odaGSJCm5tENHNVALLMtpXwYc0cVr3spp2wYcBvQvanWSJKlo0g4dexKC\nwvqc9g3A6E5ecwdwJiGsVAGHAKcTpv/uWZoyJUlSodIOHUl8jzDm40FgO/ArYBEhgDSnWJckKYJt\n22Dr1rSrUBJpLw72CrATGJXTPgpY18lrtgFnAF/M2u9sYBPwcmcfNHfuXGpqatq11dfXU19fn6hw\nSVLpvfEGfOtbsHo1PPtsuF23Dt71LnjqqbSrqywNDQ00NDS0a2tqairqZ1QV9d2SeRBYTpgG2+oJ\nQg/Gt3r4HvcSBp9+voPnaoHly5cvp7a2tpA6JUkRXX01nH12uFbLxIlh23dfWLEC7rsPXnkl7Qor\nX2NjI3V1dQB1QGOh75d2TwfAZcDPCOtzPEjowRhPmKECcBEwFjg183h/4H3AQ8BwwuyXqbhWhyRV\nlLPOgtNPh/45UwQuvjiEDvU+5RA6bgZGEtbdGAM8BhxL6LmAMKA0e82O/oSg8S5gB3A3YabL85Hq\nlSRFkhs41LuVQ+gAuDKzdWR2zuOnCKdMJEnq1o4dsGYNPPMMrFoVblvvDxoE999vuImlXEKHJEmJ\nNTW1DxPZ959/HnbuDPv17w977w2TJsGQIeE0zVtvwe67p1t/X2HokCT1Om++CbNmtYWL115re27Y\nMNhvvxAsTjqp7f6kSSFwDBwY9mtocGxIbIYOSVKvcuCBMGECvPwyHHwwnHBC+2AxYgRUlcPcTO3C\n0CFJ6lWOOy5s6n0MHZKkPu/tt+G558IYkJUrQ9ucOenWVIkMHZKkPu3AA9sPNm01axaMHJlOTZXK\n0CFJ6pPe/344+WQYMyaMCZk8Odw2NoYBqC0taVdYeQwdkqQ+ad994frrd21fsaI479/cDC++GE7X\nrFzZdupm1aow2+b3v4d99inOZ/UWhg5JkhLasSNchC47ULTePvNMWAMEoF+/MF138uSw3XprWLDM\n0CFJkv6/7dtDgPjrX+Fvf2sfLrLHglRXh4vSTZ4MRx/ddrpm8uTQq1JdHfZ76qkQOvoiQ4ckSR34\n7GdDqFi9OpwqARg8uC1InHRS+2AxfrzLqXfH0CFJUpYpU+Cww2CPPeDEE2H//eGAA8LtmDEuPFYI\nQ4ckSVkOOAAeeijtKipTv7QLkCRJfYOhQ5IkRWHokCRJURg6JElSFIYOSZIUhaFDkiRFkWTK7BDg\nfOCjwDtpH1xagElFqEuSJFWYJKHjauAo4GfAS4Sg0cpr8kmSpA4lCR2fAD4J3F/kWiRJUgVLMqaj\nCXit2IVIkqTKliR0/AvwXWBwkWuRJEkVLMnplfnAfsB6YDWwI+u5FqC28LIkSVKlSRI6buviOQeS\nSpKkDiUJHd8pdhGSJKnyFXJp+zpgSub+E0Bj4eVIkqRKlSR0vBO4ibBWR1OmrQa4B5gFvFyUyiRJ\nqmA7d6ZdQXxJZq9cQViV9O+AEZntQGBY5jlJktSJUaNg2DD4whdgyZK0q4krSeiYAXwZeDKr7Qlg\nDmHhMEmS1Inhw+GRR+Dd74Zjj4VTToFXX027qjiShI5+tJ8m22pHwveTJKlPmTgR7rgDFi2CxYth\nyhS4+WZoqfA5oElCwt3AAmBcVtv4TNtdxShKkqRKV1UFp50GTz4J06fDrFlwwgmwdm3alZVOktDx\nFcL4jdXAM5ntWWBo5jlJktRDo0fDrbeG7cEHYepUuOaayuz1SBI6nidMlz2W0LuxIHO/FlhTvNIk\nSeo7Tjwx9HrMnAlnnQUf/SisWpV2VcWVdAxGM3An8OPMdmfRKpIkqY8aPhwWLgzjPZ59Fg46CC69\ntHKm1/Z0nY6vAlcDW4Gv0fVy5z8utChJkvqyY46Bxx6Db38bzjsPbroJli2Dmpq0KytMT0PHPODn\nhNAxD0OHJEklNWQILFgAn/oUfOxjcM898JnPpF1VYXoaOiZm3d+3BHVIkqQOHHxw2hUUTzHW1RgA\nTAOGF+G9JElShUoSOn4EnJG53x+4l3CxtzXAh4tUlyRJyrJtGzQ2hgXF5s2Dj3wE9t0X/vzntCvr\nuSSh4yRgReb+pwinXt5NmDr7/YR1zCGs9bEVeBg4spv9T8nUsBlYCywkXANGkqSKUlUVbuvroa4O\nzjgDbr8dhg6F556Dv/0t3frykeQqsyOBdZn7xwK3AH8lfPF/LcH7zQIuB84BHgDOBpYAU+l43Y8P\nZT5rLrCYsBrqVcA1wMwEny9JUtkaMSIMKN1tN3jPe+DAA8Mg0y1bYPDgtKvLT5LQsZ5whdmXCBd/\nm5Np3x1IMpN4PiEwLMw8ngd8nBBCvtnB/ocQVkP998zj54D/Bs5L8NmSJJW1qir4Wg//pN+4EVas\ngJUrw5Lq5TbFNknoWATcRAgdLcBvM+2H0f7Ksz1RTVjJ9Ic57cuAIzp5zTLgQsIVbZcC7wQ+C9ye\n52dLktTr3Xgj3HILPPpoWFAs2+zZ6dTUmSRjOr4DnEnoXfgAsC3T3gz8rzzfa0/CYNT1Oe0bgNGd\nvGYFYUzHLcBbhFM9rxIWMJMkqU8YOBD22itcr+XNN8Py6T/9aQgfUJ6rmCbp6QC4tYO2awuoIx+H\nZz7rQuAOYCxwCWFcx5mdvWju3LnU5PQz1dfXU19fX7JCJUkqlYEDYd066NevbbBpIRoaGmhoaGjX\n1tTUVPgbZ+lpmaVaBr2aMAPlJOC2rPYfAe+h4ym4NxHq/vustg8A9wFj2LXXpBZYvnz5cmpra/Mo\nTZKk3qmqCq6+Gs7s9E/xnmlsbKSurg7ChV4bC60r7WXQtwPLgWNoHzqOBn7VyWuq2HXAanPWc5Ik\nqQyVwzLolwE/I6zP8SDwRdqmwQJcRDiFcmrm8a8Jp1fOJgwqHUNYI+QhwuBWSZJUhpKO6Simmwlr\nf1xACBCPEdb/aF2jYzQwIWv/G4A9gH8ELgWagLuAb0SqV5IkJZAkdPwS+CNh8Ga2fwIOJUxfzdeV\nma0jHU346Wp/SZJUhpJMmf0gYcXQXEuAoworR5IkVaokoWMIsKOD9reBYYWVI0mSKlWS0PE/wOc6\naJ8FPFFYOZIkqVIlGdPxr8AvgP0IAzgBPgbUk2w8hyRJ6gOShI7/A3wG+BZwImHtjhXAR4F7i1ea\nJEmqJEmnzP7fzCZJktQjScZ0AAwHziIs3DUi01YHjCtGUZIkqfIk6el4D2EsRxNhddKrgdeAE4C9\nCVeAlSRJaidJT8flhGXI96ftsvYAv8F1OiRJUieS9HQcQrg+Sq61hCXLJUmSdpGkp2Mb4donuQ4A\nXi6sHEmSVKmShI7bCBdnq85q2we4mLB+hyRJ0i6ShI7zgD2BDcBuhLU5VgKbCGt3SJIk7SLJmI6N\nhIu+fZgwTbYfsBz4bRHrkiRJFSbf0FENbAGmAXdnNkmSpG7le3plO/A80L8EtUiSpAqWZEzH9wkr\nkY4sci2SJKmCJRnT8RVgMmFdjueAzVnPtQC1RahLkiRVmCSh49dZ96tynmspoBZJklTB8gkduwOX\nEC5rPwi4k9Dr8UoJ6pIkSUXS0gIvvACPPQYf/CAMHZpOHfmM6fgucBpwO3ADcDRwVQlqkiRJBVqx\nAi6+GGbOhHHjYO+94bjj4Oc/T6+mfHo6ZgJnAg2Zx9cDfyDMZNlZ5LokSVJC1dVwxRUwZAgceiic\neiq8733wuc/B9u3p1ZVP6JgA/D7r8Z+AHcBYYE0xi5IkSck98AAMGgRTp0L/rEUu+iWZs1pE+YSO\nAYSQke1tYGDxypEkSYU65JC0K+hYvrNXFhEWCGshzFx5B3AlYZVSMu0zi1adJEmqGPmEjp/SFjZa\n5Q5HccqsJEm9zFtvhdMxpZZP6DitVEVIkqQ4du6Exx+HBx9s2554AhYuhNNOK+1nJ1kcTJIk9VLn\nnhvW7ejXDw46CI48Ep55Bl56qfSfbeiQJKmP+MEPYNs2OPzwMJV2yJDQfuutcT7f0CFJUh8xb166\nn5/yjF1JktRXGDokSVIUhg5JkhSFoUOSJEVh6JAkSVEYOiRJUhSGDkmSFIWhQ5IkRWHokCRJURg6\nJElSFIYOSZIURbmEjjnAs8BW4GHgyC72vRZo7mB7vLQlSpKkQpRD6JgFXA58D5gG3AcsASZ0sv9X\ngdFZ2wTgNeDmklcqSZISK4fQMR+4BlgIPA3MA9YA53Sy/xvAhqztUGA4sKjklUqSpMTSDh3VQC2w\nLKd9GXBED9/jDOBOQlCRJEllakDKn78n0B9Yn9O+gXDqpDtjgRlAfZHrkiRJRZZ26CjUqcDrwK+7\n23Hu3LnU1NS0a6uvr6e+3rwiSdIjjzRw/PEN7dqampqK+hlph45XgJ3AqJz2UcC6bl5bBZwO/Ax4\nu7sPWrBgAbW1tUlqlCSp4k2bVs/557f/Q7yxsZG6urqifUbaYzq2A8uBY3Lajwb+0M1rjwL2A35S\ngrokSVKRpd3TAXAZobfiYeBB4IvAeOCqzPMXEcZunJrzujMy+z8Rp0xJklSIcggdNwMjgQuAMcBj\nwLG0zUZpXYsj2x7ATMKaHZIkqRcoh9ABcGVm68jsDto2AoNLV44kSSq2tMd0SJKkPsLQIUmSojB0\nSJKkKAwdkiQpCkOHJEmKwtAhSZKiMHRIkqQoDB2SJCkKQ4ckSYrC0CFJkqIwdEiSpCgMHZIkKQpD\nhyRJisLQIUmSojB0SJKkKAwdkiQpCkOHJEmKwtAhSZKiMHRIkqQoDB2SJCkKQ4ckSYrC0CFJkqIw\ndEiSpCgMHZIkKQpDhyRJisLQIUmSojB0SJKkKAwdkiQpCkOHJEmKwtAhSZKiMHRIkqQoDB2SJCkK\nQ4ckSYrC0CFJkqIwdEiSpCgMHZIkKQpDhyRJisLQIUmSojB0SJKkKAwdkiQpinIJHXOAZ4GtwMPA\nkd3sPwj4AbAa2AasBGaXsD5JklSgAWkXAMwCLgfOAR4AzgaWAFOBNZ285mZgL+B0QuB4JzCw5JVK\nkqTEyiF0zAeuARZmHs8DPk4IId/sYP8ZwHRgItCUaXu+xDVKkqQCpX16pRqoBZbltC8DjujkNccT\nTsGcD7wAPA1cAryjRDVKkqQiSLunY0+gP7A+p30DMLqT10wijPnYCnyGcJrlP4GRhNMtkiSpDKUd\nOpLoBzQDJwObMm3zgVsJp2TeSqkuSZLUhbRDxyvATmBUTvsoYF0nr1kHrKUtcAA8BVQB44FVHb1o\n7ty51NTUtGurr6+nvr4+/6olSaowjzzSwPHHN7Rra2pq6mTvZNIOHduB5cAxwG1Z7UcDv+rkNfcD\nJwGDgc2ZtgMIvR8vdPZBCxYsoLa2ttB6JUmqSNOm1XP++e3/EG9sbKSurq5on5H2QFKAy4AzCets\nTCFMnx0PXJV5/iLguqz9bwBeBRZl9p9OGEj6Ezy1IklS2Uq7pwPCmhsjgQuAMcBjwLG0rdExGpiQ\ntf9mQk/IFYRZLK8CNwHfjlSvJElKoBxCB8CVma0jHa00+jThlIwkSeolyuH0iiRJ6gMMHZIkKQpD\nhyRJisLQIUmSojB0SJKkKAwdkiQpCkOHJEmKwtAhSZKiMHRIkqQoDB2SJCkKQ4ckSYrC0CFJkqIw\ndEiSpCgMHZIkKQpDhyRJisLQIUmSojB0SJKkKAwdkiQpCkOHJEmKwtAhSZKiMHRIkqQoDB2SJCkK\nQ4ckSYrC0CFJkqIwdEiSpCgMHZIkKQpDhyRJisLQIUmSojB0SJKkKAwdkiQpCkOHJEmKwtAhSZKi\nMHRIkqQoDB2SJCkKQ4ckSYrC0CFJkqIwdEiSpCgMHZIkKQpDhyRJ6tDmzcV9vwHFfTtJktRbtbTA\no4/C0qVwxx1w333FfX97OiRJEtddB2PGwHvfC9//PgwZAueeW9zPKJfQMQd4FtgKPAwc2cW+HwKa\nO9gOKG2JykdDQ0PaJfQ5HvP4PObxecxL48ADoboaTjkF7roLXn0VFi+GWbOK+znlEDpmAZcD3wOm\nAfcBS4AJ3bxuf2B01rayhDUqT/5iiM9jHp/HPD6PeWncc084rfJv/wYf+QgMGlSazymH0DEfuAZY\nCDwNzAPWAOd087pXgA1ZW3MJa5QkSQVKO3RUA7XAspz2ZcAR3bz2L8Ba4LeEUy6SJKmMpR069gT6\nA+tz2jcQTpl0ZC1wFjAzsz0N3EXX40AkSVLKeuOU2b9mtlYPEsZ/nAfc38H+6/baay/mzJlDdXV1\nuydmzJjBjBkzSlZoX9bU1ERjY2PaZfQpHvP4PObxecxLZ+nSpSxdurRd2/bt2yF0DKwrxmdUFeNN\nClANbAZOAm7Lav8R8B7gwz18n28BJwNTO3n+VuDEhDVKktSX/YLwPV2wtHs6tgPLgWNoHzqOBn6V\nx/u8l3DapTNfAX6Yd3WSJKkovRzl4u+Bt4DZwBTC9Nk3aJsyexFwXdb+c4FPE6bM/l3m+WbgM5Hq\nlSRJvdg5hMXBtgF/pv2g0EXA3VmPzyOM6dgCvArcCzgwQ5IkSZIkSZIkSZIkqfzkc8E4gKMIs2a2\nAquAL5W0usqUzzGfCdxJWPRtI/AHwowl5Sfff+etPgC8TVjFV/nJ95gPAn4ArCaMUVtJGCSvnsv3\nmJ8CrCAsv7CWcEmNEaUssIJMBxYDLxImZHy6B6/p89+fswizX04H3kWY/bKJzi8YN5Hwj/OyzP5n\nZF4/s+SVVo58j/nlwNeBOmA/4PuZ108reaWVI99j3qqG8IthKeCKSvlJcsxvI4TqjwB7A4cA7y9t\nmRUl32P+IUKg/kdgH0LAfgz4ZakLrRAzgH8lzP5sBo7vZn+/P4GHgP/IaXuCztfluBj4n5y2Kwm/\nKNQz+R7zjjwO/EvRKqp8SY/5jcB3gQuxpyNf+R7zGcDrhKCnZPI95l9n1yuMfwV4vsh19QU9CR0F\nf3+mfe2VQiW5YNz7O9n/EMJ1YNS1Qi7S16ofMJQw5VndS3rMZwP7EkJH2qsP9zZJjvnxhNMB5wMv\nEK4LdQnwjhLVWGmSHPNlwCjgE4R/46OAzwK3l6jGvq7g78+0VyQtVJILxo3qYP/1hGOxZwfPqb0k\nxzzXucDuwM1FrKuSJTnm+xMWzjuS8BeM8pPkmE8iHO+thO7qvYD/BEYSTheoa0mO+QrCmI5bCKFl\nAOEU11dLVGNfV/D3Z2/v6VDvU0/o6p8FvJJyLZWqP3AD4Tjndj2rdPoRAt7JhB6PJcB84FTCAFMV\n3+HAtYR/67WEU1yTgKtSrEld6O09Ha8AOwnpK9soOl8r/iV2Tc2jCIOR/BLsXpJj3moWcA3hwkF3\nd7Ov2uR7zIcSBu1OA/4909aP0P28g3Bto3tKUWgFSfLvfB1h9sSmrLanCMd9PGFArzqX5JjPA+4A\nLs08fpww0PE+woVA7bkuroK/P3t7T0f2BeOyHU3nA1v+mHk+2zGE5dd3FrW6ypTkmEPo4VgEfI7w\nF6B6Lt9jvhE4EDg4a7uKMMbgYOBPJau0ciT5d34/MBYYnNV2AKH344ViF1iBkhzzKnb9vd2c9ZyK\ny+9P8r9g3L7Am4RkPIVwrvUt4IQ45VaEfI/5PxD+wj6HkJJbt2GR6q0E+R7zXN/B2Sv5yveYDybM\nmrg5s/90wnWi/itSvZUgye+W7cDZhNMqHyB8Af4xUr293WBCj+g0Qlibm7nv92c38rlgHIRfBssz\n+68CvhihxkqTzzH/HSEFN+dsC6NUWjny/Xee7UJcpyOJfI/5uwij+TcTAsglOJ4jX/ke83NoO63y\nIvBTYEzpy6wIH6Lt93H27+jW381+f0qSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEkqQP+0\nC5CkhFYDLcBDmcfNhOvLPJ1WQZK61tuvMispHdfSdp2GHYSrqF5H3GtetGQ2Sb2EoUNSEi3AEsLV\ngvchXBX0w4SLbUlShwwdkpKoIlzSegOwFrgTuAU4PGuf2cCTwNbM7Tk57zEeuBF4lXC57D8Dh2We\n2w+4DXgJ2AT8CfhoCX4OSRENSLsASb1WVdb9ScAMQnAAOAv4DvBlwjiLWuBqwuXHfwoMAe4F1gCf\nAtYB76XtD6HBwO3APxPCzWnAYsKl49eU5seRJEnl6FrCWI5NwBbC2I7bgRGZ558HZuW85tvAA5n7\nXwQ2AjV5fObjhBDT6lngq1mPm4Hj83g/SZHZ0yEpqbsJp0wGA2cSTqeMIsyKGw8sBK7J2n8A0JS5\nPw1ozHoTeaQoAAABDklEQVScazBwIXAcMDbz2t2ACUX9CSRFZeiQlNQW4JnM/a8BBwELgC9k2s6k\nbTprq52Z2620Pz2T6xLgGOBcYCWwDbgVqC64akmpMXRIKpbvAr8j9EasJQwGbehk30eBM4DhwOsd\nPH8ksIgwmBTCGJCJmfeX1Es5e0VSsdxLOGVyHuHUyD8TxlwcQOgFmQ3My+zbQJiZ8mvgCMJA1BNp\nm/2yMvP44Mx2A133jEjqBQwdkpLobGGuS4GZwG8Jp1dOA1YA9wCn0HY6Zgfh9MkG4DeZff4JeDvz\n/DxCD8gfCL0dSwiBRpIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkKar/BxNP\nlKQ/EZ8YAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f0230d9fcf8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f023c7b7588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_precision_recall(labels_binary[test_set_idx],y_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we look at the area under the curve, 0.96, we see we have a very good classifier. The AUC shows how accurate our scores are under different cut-off thresholds. If you recall from the Machine Learning tutorial, the model outputs a score. We then set a cutoff to bin each score as a 0 or 1. The closer our scores are to the true values the more resilent they are to different cutoffs. For instance, if our scores were perfect our AUC would be 1. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Importances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def display_feature_importances(coef,features, labels, num_features=10):\n",
    "    \"\"\"\n",
    "    output feature importances\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    coef: numpy\n",
    "        feature importances\n",
    "    features: ls \n",
    "        feature names\n",
    "    labels: ls\n",
    "        labels for the classifier\n",
    "    num_features: int\n",
    "        number of features to output (default 10)\n",
    "    \n",
    "    Example\n",
    "    --------\n",
    "    \n",
    "    \n",
    "    \"\"\"\n",
    "    coef = mdl.coef_.ravel()\n",
    "\n",
    "    dict_feature_importances = dict( zip(features, coef) )\n",
    "    orddict_feature_importances = OrderedDict( \n",
    "                                    sorted(dict_feature_importances.items(), key=lambda x: x[1]) )\n",
    "\n",
    "    ls_sorted_features  = list(orddict_feature_importances.keys())\n",
    "\n",
    "    num_features = 10\n",
    "    label0_features = ls_sorted_features[:num_features] \n",
    "    label1_features = ls_sorted_features[-num_features:] \n",
    "\n",
    "    print(labels[0],label0_features)\n",
    "    print(labels[1], label1_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Credit Analysts ['credit', 'trabajo', 'product', 'long term', 'customer', 'customers', 'manager', 'portfolio', 'credito', 'lending']\n",
      "Financial Examiner ['officer', 'credit review', 'plus', 'fraud', 'local', 'review officer', 'como', 'national bank', 'program', 'compliance']\n"
     ]
    }
   ],
   "source": [
    "display_feature_importances(mdl.coef_.ravel(), features, ['Credit Analysts','Financial Examiner'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The feature importances are which words are the most relevant for predicting the type of Job Ad. We would expect words like credit, customer and candidate to be found in ads for a Credit Analyst. While words like review officer, compliance would be found in ads for a Financial Examiner. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross-validation\n",
    "\n",
    "Recall from the machine learning tutorial that we are seeking the find the most general pattern in the data in order to have to most general model that will be successfull at classfying new unseen data. Our previous strategy above was the *Out-of-sample and holdout set*. With this strategy we try to find a general pattern by randomly dividing our data into a test and training set based on some percentage split (e.g., 50-50 or 80-20). We train on the test set and evalute on the test set, where we pretend the test set is unforseen data. A significant drawback with this approach is we may be lucky or unlucky with our random split. A possible solution is to create many random splits into training and testing sets and evaluate each split to estimate the performance of a given model. \n",
    "\n",
    "A more sophisticated holdout training and testing procedure is *cross-validation*. In cross-validation we split our data into k-folds or k-partions, usually 5 or 10 folds. We then iterate k times. In each iteration one of the folds is used as a testing set and the rest of the folds are combined to form the training set. We can then evaluate the performance at each iteration to estimate the performance of a given method. An advantage of using cross-validation is all examples of data are used in the training set at least once. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def create_test_train_bag_of_words(train_corpus, test_corpus):\n",
    "    \"\"\"\n",
    "    Create test and training set bag of words\n",
    "    \n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    train_corpus: ls\n",
    "        ls of raw text for text corpus.\n",
    "    test_corpus: ls\n",
    "        ls of raw text for train corpus. \n",
    "        \n",
    "    Returns\n",
    "    -------\n",
    "    (train_bag_of_words,test_bag_of_words): scipy sparse matrix\n",
    "        bag-of-words representation of train and test corpus\n",
    "    features: ls\n",
    "        ls of words used as features. \n",
    "    \"\"\"\n",
    "    #parameters for vectorizer \n",
    "    ANALYZER = \"word\" #unit of features are single words rather then phrases of words \n",
    "    STRIP_ACCENTS = 'unicode'\n",
    "    TOKENIZER = None\n",
    "    NGRAM_RANGE = (0,2) #Range for pharases of words\n",
    "    MIN_DF = 0.01 # Exclude words that have a frequency less than the threshold\n",
    "    MAX_DF = 0.8  # Exclude words that have a frequency greater then the threshold \n",
    "\n",
    "    vectorizer = CountVectorizer(analyzer=ANALYZER,\n",
    "                                tokenizer=None, # alternatively tokenize_and_stem but it will be slower \n",
    "                                ngram_range=NGRAM_RANGE,\n",
    "                                stop_words = stopwords.words('english'),\n",
    "                                strip_accents=STRIP_ACCENTS,\n",
    "                                min_df = MIN_DF,\n",
    "                                max_df = MAX_DF)\n",
    "    \n",
    "    NORM = None #turn on normalization flag\n",
    "    SMOOTH_IDF = True #prvents division by zero errors\n",
    "    SUBLINEAR_IDF = True #replace TF with 1 + log(TF)\n",
    "    USE_IDF = True #flag to control whether to use TFIDF\n",
    "\n",
    "    transformer = TfidfTransformer(norm = NORM,smooth_idf = SMOOTH_IDF,sublinear_tf = True)\n",
    "\n",
    "    #get the bag-of-words from the vectorizer and\n",
    "    #then use TFIDF to limit the tokens found throughout the text \n",
    "    train_bag_of_words = vectorizer.fit_transform( train_corpus ) \n",
    "    test_bag_of_words = vectorizer.transform( test_corpus )\n",
    "    if USE_IDF:\n",
    "        train_tfidf = transformer.fit_transform(train_bag_of_words)\n",
    "        test_tfidf = transformer.transform(test_bag_of_words)\n",
    "    features = vectorizer.get_feature_names()\n",
    "\n",
    "    \n",
    "    return train_tfidf, test_tfidf, features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f022f492748>"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh0AAAFyCAYAAABRKmj1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3Xt8XFW9//9XmrbQJMXeoHd6o216I23SYpWbiBQFRUGh\nJwqKPdAvlAOWq3hDUL/2B4L0iCh4LMrla7VWEQHRqiiCooWkgQIt1/ZA7zcLaRtomuzfH2tyT0oy\nmeyZJK/n4zGPzKxZe+YzO5B5d++11wJJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJGSEr3QXE\nZGjiJkmS2mZz4tZu3SF0DM3Pz9+0du3adNchSVJntAY4mRQEj+4QOgqBkvvuu49Jkyalu5ZuY+HC\nhSxevDjdZXQr7vP4uc/j5z6P15o1azj33HMBioDS9r5ez3ZX1ElMmjSJwsLCdJfRbfTr18/9HTP3\nefzc5/Fzn3duPdJdgCRJ6h4MHZIkKRaGDkmSFAtDhzpEcXFxukvodtzn8XOfx8993rl1m6tXSkpK\nHHwkSVIblJaWUlRUBCm6esUjHZIkKRaZEDpOAB4ENgLVwMdbsc2JQAlQAbwK/J8Oq06SJKVEJoSO\nHGAVcEnicfQu/ccAvwMeA6YD3wa+B5zVUQVKkqT2y4TJwX6fuLXWRcB64IrE4xeBmcBVwK9TWpkk\nSUqZTDjS0VbvA1Y0altBCB7Z8ZcjSZJaIxOOdLTVYGBro7athM8yqJnnAPj15+7l6dy/dHBpkjJB\njx7Q97DO+a8qHdyeiRs50H9vustos56HD6bgs1enu4w2W7M3tfu6M4aOpDzw/KPkZuU1aDsx+4N8\nIPvkNFUkSWqLQ96Bx3OKuO3MdFeSpJKSdFdwcH/+Mzz6aMO2PXtS+hadMXRsAYY0ahsMHAB2tLTR\ntY99n0nTp3dkXZIywKuvwjnnwE9+Akcfne5qlEp7jn+BuQPyOL/oyHSX0iarl9/OtEV38e/fLqP/\n8HHpLqdlRUVwzTUNmtaUlXHuCSek7C06Y+h4EvhYo7Y5wFNAVUsbTcrNpbBv346sS1IGOKQX8DJM\n7AGF/i/fpTzdI5vDevdiQif7W175djmFL7/M9kN6cXgnq53c3JS+XCac8swlXPpacxhibOL+yMTj\nRcDd9frfAYwCbgEmAfMSt5vjKFaSJCUnE450zAJqTiJFwHcT939KCBNDqAsgEC6XPQ24lTC3x0bg\nUuD+ji9VkiQlKxNCx185+BGXzzfT9jfCPPCSJKmTyITTK5IkqRswdEiSpFgYOiRJUiwMHZIkKRaG\nDkmSFAtDhyRJioWhQ5IkxcLQIUmSYmHokCRJsTB0SJKkWBg6JElSLAwdkiQpFoYOSZIUC0OHJEmK\nhaFDkiTFwtAhSZJiYeiQJEmxMHRIkqRYGDokSVIsDB2SJCkWhg5JkhQLQ4ckSYqFoUOSJMXC0CFJ\nkmJh6JAkSbEwdEiSpFgYOiRJUiwMHZIkKRaGDkmSFAtDhyRJikXPdBcgSd3FgQOwYQOsXx9uGzdC\ncTGMHZvuyqR4GDokKUUah4rGtw0boKqq4TZVVXDddXFXKqWHoUOSWqmy8t1DRXV1Xf+hQ2H06HB7\n3/vq7o8eDUceCUcdBVEU84eQ0sjQIUkJ7QkVxx7bNFQcemjMH0DKcIYOSd1GZSW88UbLoWLjxoah\nYtgwQ4WUSoYOSV3SAw/AI4+0PlQcf3zDUDFypKFCSjVDh6QupV8/yMqCm25qGCpOOCH8HDWq7kjF\nIYekt1apuzF0SOpShg+HHTsgN9dQIWUaQ4ekLmfAgHRXIKk5zkgqSZJiYeiQJEmxMHRIkqRYOKZD\nkiQ1UFVdxba921i7fW1KX9fQIUlSN7Gvch+byzezZc8WNu/Z3OB+/bbt+7ZTHVXDptS+v6FDkqRO\nrDqqZlfFrlaFifL95Q227Z3dm6F5QxmSN4ShfYcye/hshvYdWtv25vo3Oe9H56WsVkOHJEkZaH/V\n/hAYGgeI8s0NwsTWPVuprK5ssG2/Q/uFIJE3lOGHDadoaBFD+w6tbau53//Q/mRlZbVYQ+ne0pR+\nJkOHJEkxiaKIt955q8UAUT9g7KrY1WDb7KxsBucNrg0ORw8+mlPHndogTAzJG8KQvCH06dUnTZ/w\n4AwdkqRO55134PXXYd26sK5Ozc9DDoG77oIeGXht5px75/Bi9r+pOFDRoD2nV07t0YeheUOZfPjk\nBqc8au4PyhlEdo/sNFWfGpkSOhYAVwNDgOeBhcATB+n/WeAqYBzwJvD7xONdB9lGktRJVFbChg0N\nQ0XBenh1M9z2W9i0CaIo9O3RIyzQ17s3vPwy/OAHkJOTzuobmjhwIgBnTTqLviPHNTi9MTRvKH0P\n6ZvmCuOTCaFjLnArcDHwd+Ai4BFgMvBGM/0/ANxFCCYPAiOAO4AfA2d1fLmSpPaqqgrBofGRipqf\nGzaEPhAW8Bs2DG7aD4cdDp8/B8aMCQv3jRkDI0ZAr16wdCl8+tPp+0wt6XdoPwC+duLX4Igj0lxN\nemVC6LiCEBjuSjy+HDiVEEK+3Ez/mcB64PuJx/8L/IhwpESSlAGqq2Hr1pZDxeuvh6MZNQYPrgsR\n73tfw1BRsyLw04Vw2GyY8M20fCSlQLpDR2+gEPh2o/YVwPtb2GYF8HXgI4TTKkcAZwMPdVCNkqRG\noiis5ttSqFi/Poy7qDFwYF2QKCysCxSjR8OoUZl1OkQdJ92hYxCQDWxt1L6NML6jOc8SxnT8khBa\negIPAJd1UI2S1C1VVsJrr8HatfDKK01Dxd69dX3f8566EHHaaQ1DxejR0Lf7DFvQQaQ7dCRjNvBT\nwtGOPwDDgO8QxnVc0NJGCxcupF+/fg3aiouLKS4u7rBCJakz+Pe/Q7B48cXws+b26qtw4EDok5tb\nFyI++MGGoWLMGGj051Wd0NKlS1m6dGmDtt27d6f0PdIdOnYAVcDgRu2Dgc0tbHM5IWzcknj8HLAX\neBz4Ck2PmgCwePFiCgsL21uvJHVKVVXh6ET9YFFzf9u2un6jRkF+Pnz4wzBxYrifnw9DhoQBneq6\nmvuHeGlpKUVFRSl7j3SHjv1ACTCHcIqkxinA/S1sk0UIKvVV13tOkrqt8vLmg8XLL9eNscjJqQsU\nH/xgXbAYP96xFepY6Q4dAN8F7gWeBv4JzKfuMliARYRTKJ9LPP4N4fTKRYRBpUOBxcC/gC1xFS1J\n6VJdHS4pbRws1q4Nl6HWGD48hIkTToD588P9iRPDJaaZOHmWMsv+/bBxY2pfMxNCxzJgIHAdIUCs\nBk6jbo6OIcDIev1/BrwH+C/CKZbdwJ+BL8ZUryTFYt8+eOmlpmMtXnopPAfhUtLx40OgmDevLlhM\nnOjgTbWsujqcVnvjjXD5cnM/t2ypm4AtVTIhdAD8MHFrzufb2F+SOo0oCkcnmjsl8r//W9fviCNC\noDjmGDjvvLpTIqNGQXbnnhlbHeDNNw8eKDZsCEcyavTpE2Z1PfJImDIFPvKR8Pjtt+G//it1dWVK\n6JCkbukb34Abbgj3e/aEo44KYeI//qMuWEycCP37p7dOZY533gmhoaVA8cYb8NZbdf2zs8OMrkce\nGYLEe98bftaEjJEjwzwqzQ0ULk3tIrOGDklKl5tuCufMa4LF2LFhOm91X9XV4bRG/QDROFRsbXSN\n5qBBdeHhpJPq7tf8HDo0BNpMkCFlSFL385nPpLsCxa2iIly6/NprIUQ0d9qjZm4UCPOj1ASIggL4\n2McaBooRIzrXFUeGDkmSUiSKYPv2MLHaa6+FW9+/hBVKCwrg2XrXWPbsGa4wqgkQxx7b9LRH//5d\na34UQ4ckSW2wf38Y5FsTLOoHjNdegz176voefjh8LjEep7gYLj86nEYbMyaMs+hug4ANHZIk1RNF\nYWr4+mGifsDYsCGMvYAwBmf06BAkjj0WPvvZcL/m1rcv8Fvg43DttYQlSrsxQ4ckqds5cCCMoWh8\ntKLm55tv1vXt3x/GjQshYvbsukAxblwYU9Hdjla0h6FDktQlvfVW86dAXn01nB6pSiyokZ0dxlCM\nHQuzZsHcuXWhYswYL1dOJUOHJKlLOf74ECp27qxr69s3hIhx4+CTn6wLFWPHhsDhpcrxMHRIkrqE\noiI47bSwIu5ZZ9WFirFjW578SvEydEiSuoQJE+Dhh9NdhQ7GdQYlSVIsDB2SJCkWhg5JkhQLQ4ck\nSYqFoUOSJMXC0CFJkmJh6JAkSbEwdEiSpOZVVKT05ZwcTJKk7uzNN+GVV8KiNK+80vD+pk0pfStD\nhyRJXVkUwY4dDUNF/WCxY0dd3wEDwvzxRx0FJ54YVsO7/vqUlWLokCSps6uuhs2bWw4Wb71V13fI\nkBAsJk6E00+vCxnjxoXQUV9pqaFDkqRup6oKXn+9+WDx6qt14y+ysmDEiBAkiopg7tyGwSIvL20f\nwdAhSVKm2L8f1q1rPlisWweVlaFfdjaMHl13GmTevHD/qKNgzBg49NC0foyWGDokSYrTvn11Ryca\nB4vXXw+nSgAOOQTGjg1Bov5pkKOOgiOPhF690vs5kmDokCQpDmecAW+80fCKkNzcuiBxzjl19486\nCoYPhx5da2YLQ4ckSR1p6lQ46aQwgPOUU+pCxbhxMHhwGIPRTRg6JEnqSGPHwqOPpruKjNC1jttI\nkqSMZeiQJEmxMHRIkqRYGDokSVIsDB2SJCkWhg5JkhQLQ4ckSYqFoUOSJMXC0CFJkmJh6JAkSbEw\ndEiSpFgYOiRJUiwMHZIkKRaGDkmSFAtDhyRJioWhQ5IkxcLQIUmSYmHokCRJsTB0SJKkWBg6JElS\nLJIJHXnAt4AngVeBdfVuryVZx4LE9hXA08Bx79L/EOD/AuuBt4FXgM8n+d6SJCkGPZPY5n+AE4F7\ngS1AVO+5qNktDm4ucCtwMfB34CLgEWAy8EYL2ywDDgfmEQLHEUCvJN5bkiTFJJnQ8RHgo8ATKarh\nCuDHwF2Jx5cDpxJCyJeb6f9h4ARgDLA70fZ6imqRJEkdJJnTK7uBXSl6/95AIbCiUfsK4P0tbHMG\n4RTMtcAG4EXgO8ChKapJkiR1gGRCx9eAG4DcFLz/ICAb2NqofRswpIVtxhLGfEwGPgEsBD4F/CAF\n9UiSpA6SzOmVK4BxhKCwHqis91xEOHLRkXoA1cBngPJ6NS0nnJJ5p7mNFi5cSL9+/Rq0FRcXU1xc\n3HGVSpLUSSxdupSlS5c2aNu9e3cLvZOTTOh44CDPtXUg6Q6gChjcqH0wsLmFbTYDm6gLHABrgSxg\nBOGKmiYWL15MYWFH5yFJkjqn5v4hXlpaSlFRUcreI5nQcX3K3h32AyXAHBqGmVOA+1vY5gnC6ZRc\nYG+ibQLh6MeGFNYmSZJSqD2TgxUB5yZu7TmE8F3gAsI8G5MIl8+OAO5IPL8IuLte/58BO4GfJPqf\nQBhIuoQWTq1IkqT0S+ZIxxHALwhzddSc7OkH/JUw58b2Nr7eMmAgcB0wFFgNnEbdHB1DgJH1+u8l\nHAm5jXAVy85EPV9t4/tKkqQYJRM6biPMSjoFWJNomwzck3juP5J4zR8mbs1pbqbRFwmnZCRJUieR\nTOj4MOFIw5p6bS8QpjL/YyqKkiRJXU8yYzp60PAy2RqVSb6eJEnqBpIJCY8Ci4Hh9dpGJNr+nIqi\nJElS15NM6LgUOIwwMdhrids6oG/iOUmSpCaSGdPxOuFy2ZMJl6xCGN/heA5JktSiZEIHhIm4/ohB\nQ5IktVJrQ8dlwP8AFcAXOPh0599rb1GSJKnraW3ouBz4f4TQcTmGDkmS1EatDR1j6t0f3QF1SJKk\nLi4V82r0BKYD/VPwWpIkqYtKJnT8N/CfifvZwGNAKWGtlJNSVJckSepikgkdnwKeTdz/GOHUSz5h\ncrBvpaguSZLUxSQTOgYCmxP3TwN+CbwE3AUcnaK6JElSF5NM6NhKWGG2J2Hxt5q5OnKAqhTVJUmS\nuphkJgf7CfALYAvh0tk/JdqPoeHKs5IkSbWSCR3XA88BRwLLgLcT7dXA/5easiRJUleT7DToy5tp\n+2k76pAkSV2c06BLkqRYOA26JEmKhdOgS5KkWKRiGnRJkqR3lUzo+DVwdTPt1xAmCpMkSWoimdBx\nPPBIM+2PACe2rxxJktRVJXPJbB5Q2Uz7AeCw9pUjSZLSIYoi9m/dz74X9rH3+b3sfX4va1euTel7\nJBM6ngf+A7ihUftc4IV2VyRJkjpMFEVUbqusDRb1Q8aBXQcAyOqdRU5+Dr2H9k7peycTOr4B/AoY\nB/w50fYhoBg4O0V1SZKkdoiiiMrt9cLF84lw8cJeDuysFy4m5pA7JZf+c/qTOyWX3Mm5HDruUHr0\n7EF2aTb8IXU1JRM6fgt8AvgK8EnC3B3PAicDj6WuNEmS1Br7t+1vctRi7/NNw0XO5Bz6n5IIF1Pq\nwkVckp0G/eHETZIkxWT/9v1Njlrse34flTvCUMusXolwMSWH/ieHcJEzJYc+R/WJNVy0JNnQ0R/4\nFDAW+A6wCygirDy7MTWlSZLUPdWGi3pHLhqHiz4T+oTTIh9MhIvJiXDRK/3hoiXJhI6jCWM5dhNm\nJ/0fQug4k7Dy7GdTVZwkSV3Z/h37645a1AsZldsT4aJnvXBxUn9ypoTxF5keLlqSTOi4lbCi7NVA\neb323wFLU1CTJEldSm24eKHhoM4m4WJyLsMWDKsdc9FnfOcMFy1JJnTMBOY3074JGNK+ciRJ6ryi\nqoiKVyrYU7aHPc/sqf25f9N+IBEuxocjF8MWDCN3cr1w0bvrhIuWJBM63gbe00z7BGB7+8qRJKlz\nOLDnAHtX720QMPau3kv1vmoAeg/rTd70PIZ8bgi5R+eSOzWXnAk53SJctCSZ0PEAcB1wTr22UcCN\nhPk7JEnqMqIo4p2N74RQ8UxdyKh4pQKicPQiZ1IOeQV5HHH2EeQW5JJXkEfvw1M7sVZXkEzouJpw\nuew2oA9hbo4hwJOEuTskSeqUqvdXs2/tvoanR8r21M7U2bNfT3ILchlw2gDyCvLIm55H7uRcehzS\nfY9etEUyoeNNwqJvJxEuk+0BlAB/SmFdkiR1qMpdlSFY1JwaeSYM8owqIwAOHXsoedPzGPGFEeRN\nzyOvII9DjjyErKysNFfeebU1dPQG9gHTgUcTN0mSMlZUHfH2urebHL145413AOhxaA9yp+XSd2Zf\nhl4wNJweOTqPnoclO5WVWtLWPbofeB3I7oBaJElql6p9Vex9bm+DK0f2PrOXqj1VAPQe0pvcglyO\n+PQRtadH+ozPjNk6u4NkYty3gEXAecDO1JYjSVLrvLOl0eDOsj3se2kfVAM9ICc/DO4cdMag2tMj\nvQc7uDOdkgkdlwJHEebl+F9gb73nIqAwBXVJktTE7sd288ypz7CnbA+V28LEWtl9s8kryKP/h/oz\n4sow/iJ3Si7ZfTwon2mSCR2/qXe/8WiaqB21SJLUotypuez+22569OnBsIuG1R69OHT0oWT1cHBn\nZ9CW0JFDWNztE8AhwB8JRz12dEBdkiQ1MOmeSekuQe3UlpEzNwDnAw8BPwNOAe7ogJokSVIX1JYj\nHWcBF1C3qNt9wD8IV7JUpbguSZLUxbTlSMdI4G/1Hq8EKoFhKa1IkiR1SW0JHT0JIaO+A0Cv1JUj\nSZK6qrZevfITwgRhEeHKlUOBHxJmKSXRflbKqpMkSV1GW0LHPdSFjRr/r1EfL5mVJEnNakvoOL+j\nigAWEFavHQI8DywEnmjFdscSVrldDczosOokSVK7ZcJk83OBW4FvEhaSexx4hDBw9WD6EY6+/AmP\nsEiSlPEyIXRcAfwYuAt4EbgceAO4+F22u4Nw2e6TNJ0ZVZIkZZh0h47ehLVaVjRqXwG8/yDbfR4Y\nTZiwzMAhSVInkMzaK6k0iDC52NZG7dsI4zuaM56wyu1xhLUEJUlSJ5Du0NFW2YQp2L8OvNKWDRcu\nXEi/fv0atBUXF1NcXJy66iRJ6qSWLl3K0qVLG7Tt3r07pe+R7lMTvYG9wKeAB+q1/zdwNHBSo/79\ngF00nHa9B+FzVBHWg/lro20KgZKSkhIKCwtTVrgkSV1daWkpRUVFAEVAaXtfL91jOvYDJcCcRu2n\nENZ1aexNYCpQUO92B2EAagFhanZJkpSBMuH0yneBe4GngX8C84ER1K1gu4iwvsvnCJfGvtBo++3A\n2820S5KkDJIJoWMZMBC4DhhKmOjrNMJlsxAGlB5szo4I5+mQJCnjZULogLB+yw9beO7z77LtDYmb\nJEnKYOke0yFJkroJQ4ckSYqFoUOSJMXC0CFJkmJh6JAkSbEwdEiSpFgYOiRJUiwMHZIkKRaGDkmS\nFAtDhyRJioWhQ5IkxcLQIUmSYmHokCRJsTB0SJKkWBg6JElSLAwdkiQpFoYOSZIUC0OHJEmKhaFD\nkiTFwtAhSZJiYeiQJEmxMHRIkqRYGDokSVIsDB2SJCkWhg5JkhQLQ4ckSYqFoUOSJMXC0CFJkmJh\n6JAkSbEwdEiSpFj0THcB6fLyyy9TXl6e7jLUgr59+zJ+/Ph0lyFJSqFuGTpefvllJkyYkO4y9C5e\neuklg4ckdSHdMnTUHOG47777mDRpUpqrUWNr1qzh3HPP9UiUJHUx3TJ01Jg0aRKFhYXpLkOSpG7B\ngaSSJCkWho40uv322+nbty8VFRW1bddffz133nln7eO3336bMWPG1D5+/PHHmT17NpMnT6aoqIiv\nf/3rzb52r169mDFjBlOmTOHCCy8kiiLWr19Pbm4uM2bMYPLkyVx77bUt1vaNb3yD8ePHM3XqVJ56\n6qlm+zz00EPk5+czYcIElixZ0myfe+65h2nTpjF16lR+9KMf1bYvWrSIUaNGMXTo0BZrkCR1LYaO\nNPrlL3/JCSecwMMPP1zblpWV1WL/TZs2cd5553HnnXfywgsv8NRTTzFw4MBm+w4aNIhVq1axevVq\nXnnlFe6//36ysrI4+uijWbVqFatWreKBBx7g6aefbrLt6tWrWbFiBS+99BI///nPueSSS5r0OXDg\nAFdffTWPPfYYq1at4uabb2bXrl0N+mzfvp1vfetbPPnkk5SVlbF06VK2b98OwJw5c/jnP//Zqv0k\nSeoaDB1psmXLFnbv3s2XvvQlli1b1qptfvCDHzBv3jwKCgoA6NGjB5dddtlBt+nRowezZ8/m1Vdf\nbdB+yCGHMH36dF577bUm2zz44IN85jOfISsri6lTp1JdXc3mzZsb9Fm5ciXTpk1j8ODB5Obmcvrp\np7NixYoGfdatW8eUKVPIy8ujZ8+evPe9760NWEVFRR7lkKRuplsPJG1s3z5Yuzb57fPzISendX1/\n9atfceaZZ3Lsscdy/vnnU1FRQZ8+fQ66zZo1azj//PPbVFNFRQV/+ctfuO6664iiqLZ9165d/Otf\n/+KrX/1qk202b97MlClTah+PGDGCTZs2NQgJmzdvZvjw4Q36bNy4scHrHHXUUTz77LNs3bqV3Nxc\n/vjHP3LYYYe1qX5JUtdh6Khn7VooKkp++5ISaO3FMMuWLeP73/8+WVlZzJkzhwcffJBzzjmn2dMr\nBzvl0pKdO3cyY8YMAE4//XROO+001q9fz+rVq5kxYwYvv/wy8+fPbxAuDiaZGgYMGMAtt9zCRz/6\nUfLy8pgxYwbZ2dltfh1JUtdg6KgnPz8Eh/Zs3xpbtmxh5cqVnHHGGUAYLLpt2zbOOeccBgwYwL//\n/e/avrt27WLQoEFAuMS3rKyMj33sYw1eb//+/RxzzDFkZWVx8cUXM3/+fAYOHMiqVauavPe0adN4\n8skn2bhxIx/4wAe44oorGDFiRIM+w4YNa3DUYuPGjQwbNuygfTZs2MDMmTObvN8nPvEJPvGJTwCw\ncOFCJ/uSJHVphUBUUlIS1SgpKYkat8Xptttui6655prax1VVVdHIkSOjPXv2RGVlZdGsWbOiioqK\nKIqi6MYbb4yuuuqqKIqiaOPGjdGoUaOisrKyKIqi6MCBA9GSJUuafY8hQ4Y0aVu3bl00e/bsBnUs\nWLCgSb9nn302Ov7446Oqqqpo9erV0axZs5r0qaysjPLz86MtW7ZE5eXlUX5+frRr164m/bZt21b7\n3tOmTYveeeedd60z3b8fSVJQ8/c48V3abg4kTYPly5fX/usfwmDPk08+mYceeoiCggLOO+883vve\n9zJjxgxKS0v52te+BoSjC3fffTfz589n0qRJFBQUNBlHUaOl0yH12y+44AIefvhhtm7d2qDPtGnT\nOPnkk5k4cSLFxcV8//vfb/I6PXv25KabbuKEE06gsLCQK6+8kv79+wPhdM6WLVsAuOSSS5gyZQpn\nnnkmd955J7179wbgW9/6FiNHjmTHjh2MHDmSO+64o7W7T5LUSbX9RH3nUwiUlJSU1M4+WlpaSlFR\nEfXblDn8/UhSZqj5ewwUAaXtfT2PdEiSpFgYOiRJUiwMHZIkKRaGDkmSFItMCh0LgHVABfA0cNxB\n+p4F/BHYBrwJ/AOY09EFSpKk5GVK6JgL3Ap8E5gOPA48Aoxsof/xwB+AjxCuTnkUeDCxrSRJykCZ\nEjquAH4M3AW8CFwOvAFc3EL/y4GbgRLgVeCrwMvAx1ron5EydWn7N954g2OPPZY+ffo0qKWx9ixt\n/8gjj1BQUMD06dP56Ec/yp49e1p8H0lS15AJoaM34WjFikbtK4D3t/I1egB9gZ0prKvDZerS9ocd\ndhi33norV155ZYu1tHdp+6uvvprly5dTVlbGhAkT+OlPf3qwXSVJ6gIyIXQMArKBrY3atwFDWvka\nVwI5QOvWiM8Amby0/Xve8x6OOeYYevXq1eLrtndp+8GDB/PWW28RRRHl5eVN1naRJHU9XWHBt2Lg\n68AZwI6WOi1cuJB+/foBsHv37mb77Kvcx9odya9tnz8on5xerVvbPpOXtm+N9i5t/53vfIdTTjml\nNvycddZZSdUhSUqNpUuXsnTp0gZtLX1fJisTQscOoAoY3Kh9MLD5XbadSxgL8inCYNIWLV68uMk0\n6I2t3bGWoh8lv7Z9yfwSCoe2btruzra0fTKaW9q+Z8+eRFHEvHnzeOKJJ5g0aRIXXnghP/7xj7ng\nggs6rBZ/RSvgAAATHElEQVRJ0sEVFxdTXFzcoK2l78tkZULo2E8YEDoHeKBe+ynA/QfZrhhYQgge\nj6SikPxB+ZTMT35t+/xBrVvbPtOXtm+NZJe2P+qoo2rHdUyaNAmAT37ykyxbtszQIUldXCaEDoDv\nAvcS5uf4JzAfGAHULD26CBgGfC7x+NPA3cBlwFPUjf3YB7yVbBE5vXJafaSiPZYvX85ll13GjTfe\nCEB1dTWjR49m7969HH/88Vx44YUsXLiQQw89lPvuu48TTzwRgAULFvD+97+fM844g4KCAqqqqrj7\n7ruZN28eZWVlbaph+PDhfOELX2DRokXcfvvtzfapfzqmsVmzZrF69eraUycPP/wwX/nKV5r02759\nO4cffjjr16/n0Ucf5aabbqJnz57s2rWLTZs2MWzYMP70pz+Rn9+6wCZJ6rwyJXQsAwYC1wFDgdXA\naYTLZiGEivpzdlxIGAR7e+JW46fAvA6utd2WL1/OokWLah/XX9p+7ty5tUvb9+jRg4kTJ9Zealp/\nafu33nqL7Oxs5s6d2+x7tHZp+/z8fK677joGD647u7Vv3z4mTJhAeXk52dnZ3HLLLbz00ksNXqf+\n0vZRFHHNNdc0WNp+yZIlDBkyhEsuuYTnn3+e3r17N1ja/vbbb+cjH/kIAOPHj+cb3/hGW3ejJKmT\ncWl7l07POP5+JCkzuLS9JEnqlAwdkiQpFoYOSZIUC0OHJEmKhaFDkiTFwtAhSZJiYehIo0xd2v7+\n+++noKCAgoIC5syZw6ZNm5rtl+zS9lVVVcyYMaP21r9/f773ve8dfGdJkjo9Q0caZerS9iNHjuSx\nxx7jmWee4dOf/jRf+9rXmvRpz9L22dnZtTWsWrWKAQMG1E4JL0nqugwdaZLJS9vPnDmzdkXeo48+\nmg0bNjTp096l7Ws89dRTDBgwgNGjR7/r55ckdW6ZMg16Zti3D9Ymv7Q9+fmQ07WWtr/77rs5+eST\nm7S3d2n7GsuWLWtxKndJUtdi6Khv7VpozxK+JSXQymm7O8PS9g8++CArV67kb3/7W5vfH5pf2j47\nO7tBn1/96lf89a9/Ter1JUmdi6Gjvvz8EBzas30rdIal7Z955hmuueYa/vKXv9CrV68mzye7tP34\n8eNrn1u5ciWHH344Rx555EH3lyRJnUUhEJWUlEQ1SkpKosZtcbrtttuia665pvZxVVVVNHLkyGjP\nnj1RWVlZNGvWrKiioiKKoii68cYbo6uuuiqKoijauHFjNGrUqKisrCyKoig6cOBAtGTJkmbfY8iQ\nIU3a1q1bF82ePbtBHQsWLGjSb/PmzdHEiROj0tLSFj9DZWVllJ+fH23ZsiUqLy+P8vPzo127djXp\nt23bttr3njZtWvTOO+/UPnfllVdGt9xyS5Nt0v37kSQFNX+PE9+l7eZA0jRYvnx57b/+oeHS9gUF\nBbVL28+YMYPS0tLaq0fqL20/adIkCgoKmoyjqNHape0ffvhhtm7d2qDPzTffzPbt25k3bx4zZszg\nnHPOafI69Ze2Lyws5Morr2ywtP2WLVsAuOSSS5gyZQpnnnlmg6XtAX79619z9tlnt2aXSZK6AJe2\nd+n0jOPvR5Iyg0vbS5KkTsnQIUmSYmHokCRJsTB0SJKkWBg6JElSLAwdkiQpFoaONMrUpe1XrlxZ\nu+z8zJkzW5wG/VOf+hQDBgyguLi4xc+4cuVKpk6dyvjx4/nmN79Z2/7qq68yc+ZMxo8fz8UXX9zi\n9pKkrsPQkUaZurT9tGnTKC0tZdWqVdxzzz1ceumlzb7HpZdeyj333HPQz3jppZfyy1/+khdffJHf\n/e53PPfccwB88Ytf5Nvf/jYvv/wyO3fubLL6rCSp6zF0pEkmL23fp0+f2vBTXl7eYrA58cQTycvL\na/G9N23aRBRFTJo0iR49evCZz3yGhx56CAhHQObMmQPAZz/7WR588MGDfg5JUufngm/17KuqYu2+\nfUlvn5+TQ06jVVRbkulL2z/xxBPMnz+fTZs2sWLFija9Z41NmzYxfPjw2scjRozgz3/+Mzt37mwQ\nZIYPH97idO6SpK7D0FHP2n37KGrHKrMlRUUU9u3bqr6ZvrT9cccdxwsvvMDKlSs5//zzeeGFF9pc\nQzJ1S5K6LkNHPfk5OZSEOeaT3r41OsPS9jWOOeYYevbs2eToRI2DBYuhQ4c2OIKxYcMGhg8fzsCB\nA9m5c2dt+8aNGxk2bFiLryNJ6hoc01FPTnY2hX37Jn1r7amV5cuXc9lll7Fu3TrWrVvHxo0bWbly\nJXv37uX444/n17/+NW+//TYA9913HyeeeCIACxYsYMmSJTzzzDMAVFVVcdddd9G7d2/KyspYtWoV\n8+fPb1UNw4cP5wtf+AKLFi1q8tz69es5cOAAEE7pVFRUtDiuo/4pm8aGDRtGVlYWa9asoaqqiqVL\nl9YGplmzZtWetrnnnntqA5gkqesydKRBpi9t//jjjzN9+nRmzJjBBRdcwL333tvsa51++umcc845\n/Pa3v2XkyJG1Yaj+0vbf+973OPvss8nPz+fUU0+tPZ1z44038uUvf5nx48czYMAATjvttNbsOklS\nJ9YdTrq7tH0n4+9HkjKDS9tLkqROydAhSZJiYeiQJEmxMHRIkqRYGDokSVIsDB1plKmrzNZ45pln\n6NmzZ4vToD/00EPk5+czYcIElixZ0myfu+++m6lTpzJt2jTOP/98qqqqAPjDH/7AjBkzmDp1Ktdf\nf/1B65AkdQ2GjjTK1FVmIUz69aUvfYlTTz212QnADhw4wNVXX81jjz3GqlWruPnmm9m1a1eT17j2\n2mt54oknWL16NeXl5fz+97+nurqaiy66iIcffpjnnnuONWvWUFZW1ppdJknqxAwdaZLJq8wC3Hvv\nvZx88skcccQRzT6/cuVKpk2bxuDBg8nNzeX0009vckQkKyuLgQMHsmfPHg4cOEBFRQVDhgxhx44d\n9OvXr3bq85NOOonf/OY3rdoHkqTOy9BRT9W+KspLy5O+Ve2ravV71V9ltrS0tMEplpasWbOmdhG3\n1qpZZXbKlCnNrjLb3IJvb775JkuWLGHhwoVA80dfNm/e3GQF2eZmR73pppuYMmUKw4YN48gjj6So\nqIjDDz+ct956ixdffJHKykoefPBBNm3a1KbPJUnqfFzwrZ59a/dRUpT8KrNFJUX0Lez8q8xef/31\nfPGLXyQ7O5soig66vsrBlJeXc/XVV/Piiy8ycOBAPv7xj/OnP/2JD33oQ9x9993MmzePrKwsioqK\n2L9/f1LvIUnqPAwd9eTk51BUkvwqszn5XWOV2dLS0trTHTt27OD3v/89P//5z/nABz5Q22fYsGFN\nVpCdOXNmg9d54YUXGD9+PEOGDAHgjDPO4B//+Acf+tCHOO644/j73/8OwOLFi6murm7VvpMkKZMV\nAlFJSUlUo6SkJGrcFqfbbrstuuaaa2ofV1VVRSNHjoz27NkTlZWVRbNmzYoqKiqiKIqiG2+8Mbrq\nqquiKIqijRs3RqNGjYrKysqiKIqiAwcOREuWLGn2PYYMGdKkbd26ddHs2bMb1LFgwYKD1nr++edH\nf/jDH5q0V1ZWRvn5+dGWLVui8vLyKD8/P9q1a1eDPlu2bInGjRsXlZeXR9XV1dHZZ58d/eIXv4ii\nKIq2bdsWRVEU7dy5MyooKIi2bt1au126fz+SpKDm73Hiu7TdHNORBpm+ymxr9OzZk5tuuokTTjiB\nwsJCrrzySvr37w/UrTI7ePBgvvjFLzJ79myOPvpo+vXrx9lnnw3AokWLmDx5Mscffzw33HBDiwNW\nJUldh6vMuoppxvH3I0mZwVVmJUlSp2TokCRJsTB0SJKkWBg6JElSLDJlno4FwNXAEOB5YCHwxEH6\nnwh8F5gMbAJuAu48SP9mrVmzps2FquP5e0nO0qVLKS4uTncZ3Yr7PH7u884tE0LHXOBW4GLg78BF\nwCOEQPFGM/3HAL8jhIxPA8cBPwC2A79uzRv27RtmDT333HPbV7k6VM3vSa3jH+P4uc/j5z7v3DIh\ndFwB/Bi4K/H4cuBUQgj5cjP9LwLWJ7YDeBGYCVxFK0PH+PHjeemllygvL0++ah3UwoULWbx4cdLb\n9+3bl/Hjx6ewIklSuqU7dPQmzKPx7UbtK4D3t7DN+xLPN+7/n0A20KpV1/xC61j9+vVzjg1JUgPp\nHkg6iBAUGk+JuY0wvqM5g5vpv5UQoAaltDpJkpQy6T7SERsHJ8Zr9+7dlJa2e/I6tYH7PH7u8/i5\nz+OV6u/OdE+D3hvYC3wKeKBe+38DRwMnNbPNY8AqwhUuNc4EfgH0oenplaHAU8Dw1JQsSVK3shGY\nBWxu7wul+0jHfqAEmEPD0HEKcH8L2zwJfKxR2xxCsGhuPMdmws4a2q5KJUnqnjaTgsCRKc4B3gE+\nD0wiXD77FjAy8fwi4O56/UcDe4BbEv3nJbY/M55yJUlSZ3YxsA54m3DE4rh6z/0EeLRR/xMIR0je\nBl4F5sdQoyRJkiRJkiRJkiRJktpmAWFMSAXwNA3HhDTnRMKYkArCmJD/06HVdU1t2ednAX8kTPr2\nJvAPwhVHapu2/nde41jgAOFSc7VNW/f5IcD/JSzV8DbwCmGQvFqvrfv8s8CzhOkXNhGW1BjQkQV2\nIScADxIuia0GPt6Kbbr99+dcwtUr84CJhKtfyqm7+qWxMYT/OL+b6P+fie3P6vBKu4627vNbCWvj\nFAHjgG8ltp/e4ZV2HW3d5zX6Ef4w/B5wRqW2SWafP0AI1R8EjiSsC/W+ji2zS2nrPv8AIVD/FzCK\nELBX08p1uMSHgW8AnyCEjjPepb/fn8C/gNsbtb1A0/VcatwIPN+o7YeEPxRqnbbu8+Y8B3wtZRV1\nfcnu858DNwBfxyMdbdXWff5h4N+EoKfktHWfX0U4mlTfpcDrKa6rO2hN6Gj392e6115pr5oF45pb\nAK6tC8bNJKwDo4NLZp831gPoC+xMYV1dWbL7/POEeW1uIP2zD3c2yezzMwinA64FNhBWwP4OcGgH\n1djVJLPPVxDW4/oI4b/xwcDZwEMdVGN31+7vz3TPSNpeHbFgXOPn1FAy+7yxK4EcYFkK6+rKktnn\n4wkT6x1H+BeM2iaZfT6WsL8rCIerDwd+AAwknC7QwSWzz58ljOn4JSG09CSc4rqsg2rs7tr9/dnZ\nj3So8ykmHOqfC+xIcy1dVTbwM8J+bnzoWR2nByHgfYZwxOMR4Argc4QBpkq92cBPCf+tFxJOcY0F\n7khjTTqIzn6kYwdhvZXBjdoH0/I88VtompoHEwYj+SX47pLZ5zXmAj8mLPDXeJZZtayt+7wvYdDu\ndOD7ibYehMPPlYS1jf7aEYV2Icn8d76ZcPVEeb22tYT9PoIwoFctS2afXw78gbAsBoSxYnuBx4Gv\n4JHrVGv392dnP9JRf8G4+k6h5YEtTyaer+9gC8apoWT2OYQjHD8B/oPwL0C1Xlv3+ZvAVKCg3u0O\nwhiDAmBlh1XadSTz3/kTwDAgt17bBMLRjw2pLrALSmafZ9H073Z1veeUWn5/4oJx6dDWff5pwr+w\nLyak5JrbYTHV2xW0dZ83dj1evdJWbd3nuYSrJpYl+p8AvATcGVO9XUEyf1v2AxcRTqscS/gCfDKm\neju7XMIR0emEsLYwcd/vz3fhgnHxa8s+/wshBVc3ut0VS6VdR1v/O6/v6zhPRzLaus8nEkbz7yUE\nkO/geI62aus+v5i60yobgXuAoR1fZpfwAer+Htf/G13zt9nvT0mSJEmSJEmSJEmSJEmSJEmSJEmS\nJEmSJEmSJEmSJEntkJ3uAiQpSeuBCPhX4nE1YX2ZF9NVkKSD6+yrzEpKj59St05DJWEV1buJd82L\nKHGT1EkYOiQlIwIeIawWPIqwKuhJhMW2JKlZhg5JycgiLGm9DdgE/BH4JTC7Xp/PA2uAisTPixu9\nxgjg58BOwnLZTwHHJJ4bBzwAbAHKgZXAyR3wOSTFqGe6C5DUaWXVuz8W+DAhOABcCFwPXEIYZ1EI\n/A9h+fF7gDzgMeAN4GPAZmAGdf8QygUeAr5ECDfnAw8Slo5/o2M+jiRJykQ/JYzlKAf2EcZ2PAQM\nSDz/OjC30TZfBf6euD8feBPo14b3fI4QYmqsAy6r97gaOKMNrycpZh7pkJSsRwmnTHKBCwinUwYT\nroobAdwF/Lhe/57A7sT96UBpvceN5QJfB04HhiW27QOMTOknkBQrQ4ekZO0DXkvc/wIwDVgMnJdo\nu4C6y1lrVCV+VtDw9Exj3wHmAFcCrwBvA8uB3u2uWlLaGDokpcoNwF8IRyM2EQaDLm2h7zPAfwL9\ngX838/xxwE8Ig0khjAEZk3h9SZ2UV69ISpXHCKdMriacGvkSYczFBMJRkM8Dlyf6LiVcmfIb4P2E\ngaifpO7ql1cSjwsSt59x8CMjkjoBQ4ekZLQ0MdctwFnAnwinV84HngX+CnyWutMxlYTTJ9uA3yX6\nXAMcSDx/OeEIyD8IRzseIQQaSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZKk\nWP3/0E/1QoRSP04AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f0230362da0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.cross_validation import StratifiedKFold\n",
    "cv = StratifiedKFold(train_labels_binary, n_folds=5)\n",
    "train_labels_binary = le.transform(train_labels)\n",
    "for i, (train,test) in enumerate(cv):\n",
    "    cv_train = train_corpus[train]\n",
    "    cv_test = train_corpus[test]\n",
    "    bag_of_words_train, bag_of_words_test, feature_names = create_test_train_bag_of_words(cv_train, \n",
    "                                                                                          cv_test)\n",
    "    \n",
    "    probas_ = clf.fit(bag_of_words_train, \n",
    "                      train_labels_binary[train]).predict_proba(bag_of_words_test)\n",
    "    cv_test_labels = train_labels_binary[test]\n",
    "    \n",
    "    precision_curve, recall_curve, pr_thresholds = precision_recall_curve(cv_test_labels,\n",
    "                                                                          probas_[:,1])\n",
    "    auc_val = auc(recall_curve,precision_curve)\n",
    "    plt.plot(recall_curve, precision_curve, label='AUC-PR {0} {1:.2f}'.format(i,auc_val))\n",
    "    \n",
    "plt.ylim(0,1.05)    \n",
    "plt.xlabel('Recall')\n",
    "plt.ylabel('Precision')\n",
    "plt.legend(loc=\"lower left\", fontsize='x-small')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case we did 5-fold cross-validation and plotted precision-recall curves for each iteration. We can then average the AUC-PR of each iteration to estimate the performance of our method. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Examples of tagging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Credit Analysts:0.00022944078823155725\n",
      "---\n",
      "a client in the northwest suburbs is looking for a credit analyst on a temporary basis as the credit analyst you will be responsible for the following extend credit to new customers by performing preliminary credit analysis in determining customer credit status and reviewing existing customers credit line responsible for entering new accounts into computer system and maintaining all a r account information analyze financial data compile reports and review each assigned account and make recommendations as to the size of the customers credit line responsible for research efforts outbound calling to customers write offs suspension of credit reconciliation of charge backs insure that orders are released in a timely manner and customer balances are collected within terms responsible for account credit reviews order releases credit holds ucc filings reconciling customer accounts and collection on equipment leases and contracts while minimizing bad debt loss approve or reject customer deals within approved credit authority work as liaison with all departments works closely with branch accounting and customer contacts to ensure timely collection of outstanding receivables to achieve dso targets heavy phone contact with customers weekly calls to all accounts with past due receivables prepare monthly reports summarizing pas due account status and receivable aging trends work closely with credit manager collection attorney to prepare claims for legal action responsible for handling confidential information appropriately if interested please apply directly and contact brian shute at       or brian shute accountemps com basic credit consumer intermediate credit collection basic credit analysis sap financials none none none \n",
      "===\n",
      "Credit Analysts:0.00025946220931228866\n",
      "---\n",
      "auto req id  br position title underwriter iv business credit line of business corporate institutional banking position city il chicago location s il chicago building location code il  sba chicago bco job type regular job status full time position overview as a n underwriter iv within ourbusiness creditorganization you will be based inchicago illinois job profile works independently with little or no oversight exercises exception and elevated approval authority prepares offerings for all types of transactions at any level of complexity or in management of a portfolio as assigned interacts with internal external customers provides recommendations for credit structuring serves as subject matter expert for certain asset classes serves as a coach work lead and trainer for the group using independent judgment to identify risks and mitigants prepares offerings and or scorecards such as risk ratings by analyzing credit collateral strength and financial worthiness of loan transaction parties if relevant performs ongoing credit risk management for assigned portfolio coaches and or reviews the work of other underwriters and fills in for manager as required contacts internal external customers and other parties as appropriate to gather information and to clarify inconsistent incorrect or missing information may perform relevant pre approval and post approval activities identifies and or confirms all external regulatory requirements ensures internal policies procedures are followed makes recommendations on credit decisions or makes credit decisions directly as appropriate core competencies manages risk working experience assesses and effectively manages all of the risks associated with their business objectives and activities to ensure activities are in alignment with the bank rsquo s and unit rsquo s risk appetite and risk management framework customer focus extensive experience knowledge of the values and practices that align customer needs and satisfaction as primary considerations in all business decisions and ability to leverage that information in creating customized customer solutions job specific competencies accuracy and attention to detail extensive experience understanding of the necessity and value of accuracy and attention to detail ability to process information with high levels of accuracy coaching others working experience knowledge of coaching concepts and methods ability to encourage motivate and guide individuals or teams in learning and improving effectiveness credit analysis and verification extensive experience knowledge of and ability to utilize tools techniques and practices for determining the credit worthiness of an applicant decision making and critical thinking extensive experience understanding of the issues related to the decision making process ability to analyze situations fully and accurately and reach productive decisions knowledge of underwriting extensive experience knowledge of the full spectrum of underwriting activities practices tools and considerations and the ability to utilize these to manage the underwriting process managing multiple priorities extensive experience ability to manage multiple concurrent objectives projects groups or activities making effective judgments as to prioritizing and time allocation office support tools working experience knowledge of and ability to use office support tools available at the desktop for example word processing e mail presentation software and spreadsheets effective communications extensive experience understanding of effective communication concepts tools and techniques ability to effectively transmit receive and accurately interpret ideas information and needs through the application of appropriate communication behaviors operational functions extensive experience knowledge of major functional processes and associated operating requirements ability to apply this knowledge appropriately to diverse situations financial analysis extensive experience knowledge of and ability to read interpret and draw accurate conclusions from financial and numerical material required education and experience roles at this level typically require a university college degree with   years of industry relevant experience specific certifications are often required in lieu of a degree a comparable combination of education and experience may be considered eeo statement pnc provides equal employment opportunity to qualified persons regardless of race color sex religion national origin age sexual orientation gender identity disability veteran status or other categories protected by law total hours per week   hr job code   underwriter iv bus credit\n",
      "===\n",
      "Financial Examiners:0.9998983946607216\n",
      "---\n",
      " general responsibilities moneygram is an international high performance company located in more than   countries and territories we are committed to our employees moneygram promotes an environment that is both challenging and rewarding for employees we understand the importance of career development and provide many opportunities to learn and grow we take pride in upholding a culture strongly rooted in our corporate values and committed to the communities around us the compliance analyst iii is responsible for advanced monitoring analysis investigation of transactional data for identification of patterns and trends of questionable and or suspicious activity as it relates to risks associated with violations money laundering fraud terrorist financing corruption and sanctions lists the incumbent is responsible for gathering the proper documentation investigating potentially suspicious financial activity writing details reports summarizing research results and when appropriate files suspicious activity reports sars with country regulatory bodies refers matters to the regional compliance teams for review and reporting conducting an investigation may involve written and verbal communications with agents customers law enforcement and moneygram employees primary responsibilities responsible for qa reviews of surveillance items flagged as potentially suspicious transactions and items identified as potential matches to adverse media alerts designated sanctions targets and prohibited persons lists and aml compliance testing work to standardize qa process e g sampling for all coverage areas determines whether cases investigations have been properly documented determines if cases investigations have been properly decisioned review alert closure codes by analyst and investigate any potential issues formally track and report results of qa to management identify and coordinate training needs policy procedure revisions standardize alert review process and other needs identified during the qa process with the aml global qa supervisor and the global policies procedures and training heads provide training sessions as needed on training gaps new technology process perform qa for below the line testing of transaction monitoring system scenarios alert scoring engine and any other aml models review and investigate potential issues as a result of inputs and outputs monitoring plans assist with program documentationsupport future phases of efforts related to tiered monitoring such as auto closure segmentation profiling other special projects as needed skills and experience required educational requirements ba bs in business finance law enforcement legal studies or a related field may be substituted with equivalent work experience in financial services law enforcement banking or a related field experience requirements     years professional experience in an anti money laundering group within a audit compliance financial services operations experience or equivalent     years experience in resolution of surveillance items flagged as potentially suspicious transactions and items identified as potential matches to adverse media alerts designated sanctions targets and prohibited persons lists excellent time management skills and ability to execute tasks within tight deadlines excellent analytical oral and written communication skills stong interpersonal skills necessary to work effectively with colleagues across regions and business units be able to critically review closure rationale to determine if the resolution was appropriate and appropriately documented be able to identify solutions for recurring issues identified during the qa process skills desired self motivated team player knowledge of case management systems acams certified ability to research and resolve issues independently while working across teams to acquire information ability to manage multiple priorities and tasks with the skills to adapt to changes in a fast paced environment knowledge of products services operations systems and working knowledge of laws regulations and regulatory agencies related to assigned areas options onlineapply refer email this job to a friendemail refer a linkedin connection share on your newsfeed job id     positions   location us tx frisco posted date       category compliance fraud position type full time job type exempt days hours m f   hours more information about this job general responsibilities moneygram is an international high performance company located in more than   countries and territories we are committed to our employees moneygram promotes an environment that is both challenging and rewarding for employees we understand the importance of career development and provide many opportunities to learn and grow we take pride in upholding a culture strongly rooted in our corporate values and committed to the communities around us the compliance analyst iii is responsible for advanced monitoring analysis investigation of transactional data for identification of patterns and trends of questionable and or suspicious activity as it relates to risks associated with violations money laundering fraud terrorist financing corruption and sanctions lists the incumbent is responsible for gathering the proper documentation investigating potentially suspicious financial activity writing details reports summarizing research results and when appropriate files suspicious activity reports sars with country regulatory bodies refers matters to the regional compliance teams for review and reporting conducting an investigation may involve written and verbal communications with agents customers law enforcement and moneygram employees primary responsibilities responsible for qa reviews of surveillance items flagged as potentially suspicious transactions and items identified as potential matches to adverse media alerts designated sanctions targets and prohibited persons lists and aml compliance testing work to standardize qa process e g sampling for all coverage areas determines whether cases investigations have been properly documented determines if cases investigations have been properly decisioned review alert closure codes by analyst and investigate any potential issues formally track and report results of qa to management identify and coordinate training needs policy procedure revisions standardize alert review process and other needs identified during the qa process with the aml global qa supervisor and the global policies procedures and training heads provide training sessions as needed on training gaps new technology process perform qa for below the line testing of transaction monitoring system scenarios alert scoring engine and any other aml models review and investigate potential issues as a result of inputs and outputs monitoring plans assist with program documentationsupport future phases of efforts related to tiered monitoring such as auto closure segmentation profiling other special projects as needed skills and experience required educational requirements ba bs in business finance law enforcement legal studies or a related field may be substituted with equivalent work experience in financial services law enforcement banking or a related field experience requirements     years professional experience in an anti money laundering group within a audit compliance financial services operations experience or equivalent     years experience in resolution of surveillance items flagged as potentially suspicious transactions and items identified as potential matches to adverse media alerts designated sanctions targets and prohibited persons lists excellent time management skills and ability to execute tasks within tight deadlines excellent analytical oral and written communication skills stong interpersonal skills necessary to work effectively with colleagues across regions and business units be able to critically review closure rationale to determine if the resolution was appropriate and appropriately documented be able to identify solutions for recurring issues identified during the qa process skills desired self motivated team player knowledge of case management systems acams certified ability to research and resolve issues independently while working across teams to acquire information ability to manage multiple priorities and tasks with the skills to adapt to changes in a fast paced environment knowledge of products services operations systems and working knowledge of laws regulations and regulatory agencies related to assigned areas \n",
      "===\n",
      "Financial Examiners:0.9999192286188777\n",
      "---\n",
      " general responsibilities the compliance analyst iii is responsible for advanced monitoring analysis investigation of transactional data for identification of patterns and trends of questionable and or suspicious activity as it relates to risks associated with violations money laundering fraud terrorist financing corruption and sanctions lists the incumbent is responsible for gathering the proper documentation investigating potentially suspicious financial activity writing details reports summarizing research results and when appropriate files suspicious activity reports sars with country regulatory bodies refers matters to the regional compliance teams for review and reporting conducting an investigation may involve written and verbal communications with agents customers law enforcement and moneygram employees primary responsibilities o evaluates fraudulent suspicious and or unusual activity for regulatory reporting and or escalation to management and or other departments o determines whether transactions are legitimate and have been properly documented o serves as subject matter expert in analysis and investigative techniques o gathers pertinent evidence proper documentation through confidential research analysis to support analytical finding through more complex monitoring rules conducting internal external research pulling of transaction records and reviewing reports o conducts inbound and or outbound interviews with consumer agent or employees to gather pertinent evidence and additional details o conducts all regulatory submissions within the timelines and guidelines and country regulatory requirements o create and maintains investigative case files ensure appropriate records files are maintained accurately and are properly reconciled per all record keeping requirements mgi and regulatory o gathers information responsive to internal and or external requests for sar supporting documentation subpoenas or other requests as needed o makes sure databases logs are kept current reconciled and confidential o utilizes resources to stay abreast of industry trends and standards o provides and assist management with information on trends patterns to be used for potential control or rule enhancements helping to increase department efficiency effectiveness of the aml process for moneygrams compliance program o partners with regional compliance teams to maintain awareness of transactional patterns in moderate to high risk countries as well as provide aml fraud related advisory services in relations to financial intelligence unit activities o performs other duties as assigned skills and experience required educational requirements ba bs in business finance law enforcement legal studies or a related field may be substituted with equivalent work experience in financial services law enforcement banking or a related field experience requirements   years of data analytics related to complex compliance financial investigations experience in money service business or banking environments and or familiarity with various financial products acfe or acams certification required within   months of hire exceptional analytical cross referencing and deductive reasoning skills excellent attention to detail in analysis and writing with the proven ability to formulate execute an investigation strong written and verbal communication skills o communicate effectively with a culturally diverse agent and consumer base fluent bilingual communication skills preferred o writing skills may include writing policy procedure and management reports o able to effectively communicate investigative findings and recommendations with senior leadership law enforcement proactive organized able to work independently decisive on time managed sensitive assignments demonstrated ability to work successfully in a fast pace highly structured deadline driven culture flexibility to effectively prioritize work assignments with changing priorities experience working with highly confidential information ability to work as an individual contributor within a team setting capable of sharing knowledge mentoring and training other team members demonstrated ability for computer and internet research skills advanced experience with ms office suite particularly excel ms access preferred experience using analyst notebook or other analytics and visualization tools preferred experience with financial products particularly money transfer monetary instruments and store value cards preferred knowledge of domestic and international higher risk countries jurisdictions and corridors of aml ctf concern preferred options onlineapply refer email this job to a friendemail refer a linkedin connection share on your newsfeed job id     positions   location us tx frisco posted date       category compliance fraud position type full time job type exempt days hours m f   hours more information about this job general responsibilities the compliance analyst iii is responsible for advanced monitoring analysis investigation of transactional data for identification of patterns and trends of questionable and or suspicious activity as it relates to risks associated with violations money laundering fraud terrorist financing corruption and sanctions lists the incumbent is responsible for gathering the proper documentation investigating potentially suspicious financial activity writing details reports summarizing research results and when appropriate files suspicious activity reports sars with country regulatory bodies refers matters to the regional compliance teams for review and reporting conducting an investigation may involve written and verbal communications with agents customers law enforcement and moneygram employees primary responsibilities o evaluates fraudulent suspicious and or unusual activity for regulatory reporting and or escalation to management and or other departments o determines whether transactions are legitimate and have been properly documented o serves as subject matter expert in analysis and investigative techniques o gathers pertinent evidence proper documentation through confidential research analysis to support analytical finding through more complex monitoring rules conducting internal external research pulling of transaction records and reviewing reports o conducts inbound and or outbound interviews with consumer agent or employees to gather pertinent evidence and additional details o conducts all regulatory submissions within the timelines and guidelines and country regulatory requirements o create and maintains investigative case files ensure appropriate records files are maintained accurately and are properly reconciled per all record keeping requirements mgi and regulatory o gathers information responsive to internal and or external requests for sar supporting documentation subpoenas or other requests as needed o makes sure databases logs are kept current reconciled and confidential o utilizes resources to stay abreast of industry trends and standards o provides and assist management with information on trends patterns to be used for potential control or rule enhancements helping to increase department efficiency effectiveness of the aml process for moneygrams compliance program o partners with regional compliance teams to maintain awareness of transactional patterns in moderate to high risk countries as well as provide aml fraud related advisory services in relations to financial intelligence unit activities o performs other duties as assigned skills and experience required educational requirements ba bs in business finance law enforcement legal studies or a related field may be substituted with equivalent work experience in financial services law enforcement banking or a related field experience requirements   years of data analytics related to complex compliance financial investigations experience in money service business or banking environments and or familiarity with various financial products acfe or acams certification required within   months of hire exceptional analytical cross referencing and deductive reasoning skills excellent attention to detail in analysis and writing with the proven ability to formulate execute an investigation strong written and verbal communication skills o communicate effectively with a culturally diverse agent and consumer base fluent bilingual communication skills preferred o writing skills may include writing policy procedure and management reports o able to effectively communicate investigative findings and recommendations with senior leadership law enforcement proactive organized able to work independently decisive on time managed sensitive assignments demonstrated ability to work successfully in a fast pace highly structured deadline driven culture flexibility to effectively prioritize work assignments with changing priorities experience working with highly confidential information ability to work as an individual contributor within a team setting capable of sharing knowledge mentoring and training other team members demonstrated ability for computer and internet research skills advanced experience with ms office suite particularly excel ms access preferred experience using analyst notebook or other analytics and visualization tools preferred experience with financial products particularly money transfer monetary instruments and store value cards preferred knowledge of domestic and international higher risk countries jurisdictions and corridors of aml ctf concern preferred \n",
      "===\n"
     ]
    }
   ],
   "source": [
    "num_comments = 2\n",
    "label0_comment_idx = y_score[:,1].argsort()[:num_comments] #SuicideWatch\n",
    "label1_comment_idx = y_score[:,1].argsort()[-num_comments:] #depression\n",
    "test_set_labels = labels[test_set_idx]\n",
    "#convert back to the indices of the original dataset\n",
    "top_comments_testing_set_idx = np.concatenate([label0_comment_idx, \n",
    "                                               label1_comment_idx])\n",
    "\n",
    "\n",
    "#these are the 5 comments the model is most sure of \n",
    "for i in top_comments_testing_set_idx:\n",
    "    print(\n",
    "        u\"\"\"{}:{}\\n---\\n{}\\n===\"\"\".format(test_set_labels[i],\n",
    "                                          y_score[i,1],\n",
    "                                          test_corpus[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the top-2 example for each label that the model is sure of. In this case we can see our important feature words in the ads and see how the model classified these advertisements. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Further Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A great resource for NLP in python is \n",
    "[Natural Language Processing with Python](https://www.amazon.com/Natural-Language-Processing-Python-Analyzing/dp/0596516495)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Work thorugh the Reddit_TextAnalysis.ipynb notebook."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}