Summarization using Gensim.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gensim.summarization import summarize\n",
    "from gensim.summarization import keywords\n",
    "\n",
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input text:\n",
      "Thomas A. Anderson is a man living two lives. By day he is an average computer programmer and by night a hacker known as Neo. Neo has always questioned his reality, but the truth is far beyond his imagination. Neo finds himself targeted by the police when he is contacted by Morpheus, a legendary computer hacker branded a terrorist by the government. Morpheus awakens Neo to the real world, a ravaged wasteland where most of humanity have been captured by a race of machines that live off of the humans' body heat and electrochemical energy and who imprison their minds within an artificial reality known as the Matrix. As a rebel against the machines, Neo must return to the Matrix and confront the agents: super-powerful computer programs devoted to snuffing out Neo and the entire human rebellion. \n"
     ]
    }
   ],
   "source": [
    "text = \"Thomas A. Anderson is a man living two lives. By day he is an \" + \\\n",
    "    \"average computer programmer and by night a hacker known as \" + \\\n",
    "    \"Neo. Neo has always questioned his reality, but the truth is \" + \\\n",
    "    \"far beyond his imagination. Neo finds himself targeted by the \" + \\\n",
    "    \"police when he is contacted by Morpheus, a legendary computer \" + \\\n",
    "    \"hacker branded a terrorist by the government. Morpheus awakens \" + \\\n",
    "    \"Neo to the real world, a ravaged wasteland where most of \" + \\\n",
    "    \"humanity have been captured by a race of machines that live \" + \\\n",
    "    \"off of the humans' body heat and electrochemical energy and \" + \\\n",
    "    \"who imprison their minds within an artificial reality known as \" + \\\n",
    "    \"the Matrix. As a rebel against the machines, Neo must return to \" + \\\n",
    "    \"the Matrix and confront the agents: super-powerful computer \" + \\\n",
    "    \"programs devoted to snuffing out Neo and the entire human \" + \\\n",
    "    \"rebellion. \"\n",
    "\n",
    "print('Input text:')\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To summarize this text, we pass the raw string data as input to the function \"summarize\", and it will return a summary.\n",
    "Note: make sure that the string does not contain any newlines where the line breaks in a sentence. A sentence with a newline in it (i.e. a carriage return, \"\\n\") will be treated as two sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary:\n",
      "Morpheus awakens Neo to the real world, a ravaged wasteland where most of humanity have been captured by a race of machines that live off of the humans' body heat and electrochemical energy and who imprison their minds within an artificial reality known as the Matrix.\n"
     ]
    }
   ],
   "source": [
    "print ('Summary:')\n",
    "summary = summarize(text)\n",
    "print(summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the \"split\" option if you want a list of strings instead of a single string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\"Morpheus awakens Neo to the real world, a ravaged wasteland where most of humanity have been captured by a race of machines that live off of the humans' body heat and electrochemical energy and who imprison their minds within an artificial reality known as the Matrix.\"]\n"
     ]
    }
   ],
   "source": [
    "summary = summarize(text, split=True)\n",
    "print(summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can adjust how much text the summarizer outputs via the \"ratio\" parameter or the \"word_count\" parameter. Using the \"ratio\" parameter, you specify what fraction of sentences in the original text should be returned as output. Below we specify that we want 50% of the original text (the default is 20%)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "By day he is an average computer programmer and by night a hacker known as Neo. Neo has always questioned his reality, but the truth is far beyond his imagination.\n",
      "Neo finds himself targeted by the police when he is contacted by Morpheus, a legendary computer hacker branded a terrorist by the government.\n",
      "Morpheus awakens Neo to the real world, a ravaged wasteland where most of humanity have been captured by a race of machines that live off of the humans' body heat and electrochemical energy and who imprison their minds within an artificial reality known as the Matrix.\n"
     ]
    }
   ],
   "source": [
    "summary = summarize(text,  ratio=0.5)\n",
    "print(summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the \"word_count\" parameter, we specify the maximum amount of words we want in the summary. Below we have specified that we want no more than 50 words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "By day he is an average computer programmer and by night a hacker known as Neo. Neo has always questioned his reality, but the truth is far beyond his imagination.\n",
      "Morpheus awakens Neo to the real world, a ravaged wasteland where most of humanity have been captured by a race of machines that live off of the humans' body heat and electrochemical energy and who imprison their minds within an artificial reality known as the Matrix.\n"
     ]
    }
   ],
   "source": [
    "summary = summarize(text,  word_count=80)\n",
    "print(summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned earlier, this module also supports keyword extraction. Keyword extraction works in the same way as summary generation (i.e. sentence extraction), in that the algorithm tries to find words that are important or seem representative of the entire text. They keywords are not always single words; in the case of multi-word keywords, they are typically all nouns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keywords:\n",
      "humanity\n",
      "human\n",
      "neo\n",
      "humans body\n",
      "super\n",
      "hacker\n",
      "reality\n"
     ]
    }
   ],
   "source": [
    "print ('Keywords:')\n",
    "print (keywords(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary:\n",
      "Anderson, a software engineer for a Metacortex, the other life as Neo, a computer hacker \"guilty of virtually every computer crime we have a law for.\" Agent Smith asks him to help them capture Morpheus, a dangerous terrorist, in exchange for amnesty.\n",
      "Morpheus explains that he's been searching for Neo his entire life and asks if Neo feels like \"Alice in Wonderland, falling down the rabbit hole.\" He explains to Neo that they exist in the Matrix, a false reality that has been constructed for humans to hide the truth.\n",
      "Neo is introduced to Morpheus's crew including Trinity; Apoc (Julian Arahanga), a man with long, flowing black hair; Switch; Cypher (bald with a goatee); two brawny brothers, Tank (Marcus Chong) and Dozer (Anthony Ray Parker); and a young, thin man named Mouse (Matt Doran).\n",
      "Cypher cuts up a juicy steak and ruminates that he knows the steak is merely the simulation telling his brain that it is delicious and juicy, but after nine years he has discovered that \"ignorance is bliss.\" He strikes a deal for the machines to reinsert his body into a power plant, reinsert him into the Matrix, and he'll help the Agents.\n",
      "\n",
      "Keywords:\n",
      "neo\n",
      "morpheus\n",
      "trinity\n",
      "cypher\n",
      "agents\n",
      "agent\n",
      "smith\n",
      "tank\n",
      "says\n",
      "saying\n"
     ]
    }
   ],
   "source": [
    "\n",
    "\n",
    "text = requests.get('http://rare-technologies.com/the_matrix_synopsis.txt').text\n",
    "\n",
    "print ('Summary:')\n",
    "print (summarize(text, ratio=0.01))\n",
    "\n",
    "print ('\\nKeywords:')\n",
    "print (keywords(text, ratio=0.01))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary:\n",
      "The answering machine records a woman introducing herself as Maude Lebowski and saying that she is the one who took his rug and has sent a car to pick Dude up at his apartment.\n",
      "As he climbs out of bed to make a White Russian, Maude asks about the apartment and Dude explains that Treehorn's thugs most likely vandalized it looking for Lebowski's money.\n",
      "\n",
      "Keywords:\n",
      "dude\n",
      "dudes\n",
      "walter\n",
      "lebowski\n",
      "brandt\n",
      "maude\n",
      "donny\n",
      "bunny\n"
     ]
    }
   ],
   "source": [
    "text = requests.get('http://rare-technologies.com/the_big_lebowski_synopsis.txt').text\n",
    "\n",
    "print ('Summary:')\n",
    "print (summarize(text, ratio=0.01))\n",
    "\n",
    "print ('\\nKeywords:')\n",
    "print (keywords(text, ratio=0.01))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time around, the summary is not of high quality, as it does not tell us much about the movie. In a way, this might not be the algorithms fault, rather this text simply doesn't contain one or two sentences that capture the essence of the text as in \"The Matrix\" synopsis.\n",
    "The keywords, however, managed to find some of the main characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary:\n",
      "Keywords:\n",
      "require human\n"
     ]
    }
   ],
   "source": [
    "# Test\n",
    "text = \"\"\"\n",
    "The theory and development of computer systems able to perform tasks that normally require human intelligence, such as visual perception, speech recognition, decision- making, and translation between languages\n",
    "\"\"\"\n",
    "#summary = summarize(text)\n",
    "print ('Summary:')\n",
    "#print(summary)\n",
    "print ('Keywords:')\n",
    "print (keywords(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}