Sumrized.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demo for Text Summarization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## English and Arabic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Steps\n",
    "1. Preprocessing\n",
    "    * Split text to sentences\n",
    "    * Get Sentences Tokens\n",
    "    * Remove stop words\n",
    "2. Loading Word2Vec\n",
    "3. Define a threshold\n",
    "4. Identify Centroid Sentence\n",
    "5. TF-IDF \n",
    "6. Similarity between all phrases and centroid sentence\n",
    "7. Sort and Select the phrases with high similarity\n",
    "8. Rearrange following the original order"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sumrized import Sumrized \n",
    "import gensim.models.keyedvectors as word2vec\n",
    "from helper import Helper\n",
    "import nltk, os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Article Language"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "lang = \"ar\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading Word2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Path to word2vec folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "tools = 'path/to/word2vec/folder'\n",
    "tools = 'tools/' # in my case, the folder './tools/' contain simlinks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### English\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "word2vecEnPath = tools + 'wiki.en/wiki.en.vec'\n",
    "word2vecEn = word2vec.KeyedVectors.load_word2vec_format(word2vecEnPath, \n",
    "                                                      binary=True, \n",
    "                                                      unicode_errors='ignore',\n",
    "                                                      limit=50000\n",
    "                                                       )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Arabic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "word2vecArPath = tools + 'wiki.ar/wiki.ar.vec' \n",
    "word2vecAr = word2vec.KeyedVectors.load_word2vec_format(word2vecArPath, \n",
    "                                                      binary=True, \n",
    "                                                      unicode_errors='ignore',\n",
    "                                                      limit=50000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "word2vec = word2vecEn if lang==\"en\" else word2vecAr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "help = Helper(lang=lang)\n",
    "testingArticles = [\n",
    "    'article1.txt',\n",
    "    'article2.txt',\n",
    "    'article3.txt',\n",
    "    'article4.txt',\n",
    "    'article5.txt'\n",
    "]\n",
    "articlePath = 'articles/'+lang+'/'+testingArticles[3]\n",
    "content = help.getArticleContent(articlePath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define Summary size (percent %)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = help.getArticleSentences(content)\n",
    "summarySize = 30 # [10, 100]\n",
    "limit = ( summarySize * len(sentences) ) / 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate Summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'عندما عاد المنتخب الوطني من مونديال البرازيل 2014 بتأهل تاريخي للدور الثاني، كنت أعتقد يومها وبصدق أن الكرة الجزائرية ستعطي دفعا كبيرا لكل الرياضات الأخرى من أجل التألق، بغية المساهمة في رفع الراية الوطنية في مختلف المنافسات الإقليمية والدولية، خاصة أن جميع الجزائريين ودون استثناء وضعوا ثقتهم كاملة في المسؤولين الذين أوكلت إليهم مهمة تسيير الرياضة الجزائرية. لكن الحقيقة المرة التي لا بد من أن نقف عندها اليوم هي أن الرياضة الجزائرية تراجعت كثيرا جراء الصراعات بين أبناء وطن واحد، ما تسبب في تقهقرها بشكل يدعو إلى الخوف والقلق، وهذا رغم الإمكانات التي تمنحها الدولة الجزائرية لهذا القطاع، والأموال الطائلة التي وجهت إلى العديد من المشاريع التي بقيت إلى يومنا هذا معطلة إلى إشعار آخر، ولن تعرف الفرج ما لم يجد صُنّاع القرار حلولا سريعة لإنقاذ ما يمكن إنقاذه.'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sumrized = Sumrized(lang, word2vec)\n",
    "summary = sumrized.summarize(content, limit)\n",
    "summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}