Notebooks/Summarize.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# NLP. Text summarization.\n",
    "\n",
    "There are two approaches to automatic summarization nowadays: extraction and abstraction. Abstraction method tries to generate a summary based on the text. This summary could have words which aren't present in the text itself. This method looks very promising, but currently it is considered to be too complex. As a result extraction methods are more commonly used. They work by selecting certain words or sentences from the text and creating summary using them.\n",
    "\n",
    "Usually unsupervised approaches are used, as they don't require training data, so that they can summarize a given text without additional information. And their quality is good enough."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Programs\\Anaconda3\\lib\\site-packages\\gensim\\utils.py:855: UserWarning: detected Windows; aliasing chunkize to chunkize_serial\n",
      "  warnings.warn(\"detected Windows; aliasing chunkize to chunkize_serial\")\n"
     ]
    }
   ],
   "source": [
    "from nltk import FreqDist\n",
    "from nltk.tokenize import word_tokenize, sent_tokenize\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "lemma = WordNetLemmatizer()\n",
    "from nltk.corpus import stopwords\n",
    "stop = stopwords.words('english')\n",
    "\n",
    "from bs4 import BeautifulSoup\n",
    "from urllib.request import urlopen\n",
    "\n",
    "from gensim.models import Phrases\n",
    "from gensim.models.phrases import Phraser\n",
    "\n",
    "import os\n",
    "from collections import Counter\n",
    "import string\n",
    "punctuations = list(string.punctuation)\n",
    "#Add some more punctuation, as the list doesn't cover all cases.\n",
    "punctuations.extend(['”', '–', '``', \"''\"])\n",
    "stop = stop + punctuations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The basic idea behind unsupervised summarization is the following:\n",
    "- split text into sentences;\n",
    "- tokenize sentences into separate words;\n",
    "- assign scores to sentences based on importance;\n",
    "- select several top sentences and display them in original order;\n",
    "\n",
    "The main point, obviously, is assigning scores to sentences. Here are some of the ways to do this:\n",
    "- calculate similarity between each pair of sentences and select sentences which are most similar to most sentences;\n",
    "- calculate word frequences, select most frequent words and select sentences which have most of these words;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook I'll use the following news article:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Snap election to be held in March after Northern Ireland government collapses \n",
      " ____________________________________________________________ \n",
      " An early election will be held in Northern Ireland on 2 March after the collapse of its government, it has been announced.\n",
      "\n",
      "Northern Ireland Secretary James Brokenshire said the devolved Northern Ireland Assembly will sit for the last time on 25 January, before it is dissolved the following day.\n",
      "\n",
      "The break-up of the power-sharing government comes amid a dispute between Sinn Fein and the DUP over a botched renewable energy scheme that could have cost the taxpayer £500m.\n",
      "\n",
      "The \"cash for ash\" scandal prompted the resignation of deputy first minister Martin McGuinness, who called for DUP first minister Arlene Foster to quit.\n",
      "\n",
      "She refused, calling Mr McGuinness' actions \"not principled\" and \"purely political\".\n",
      "\n",
      "On Monday afternoon, Sinn Fein announced it would not replace Mr McGuinness - triggering the snap election.\n",
      "\n",
      "Despite a last-ditch attempt by Theresa May to urge a resolution, Sinn Fein MLA Conor Murphy said his party had decided to \"call time on the arrogance of the DUP\".\n",
      "\n",
      "He said: \"We have had scandal after scandal, allegations of corruption need to be investigated properly and the people responsible need to be held to account.\"\n",
      "\n",
      "Mrs Foster, who presided over the controversial renewable energy scheme as enterprise minister, claimed Sinn Fein \"did not like the election result last May and are therefore looking to have another go\".\n",
      "\n",
      ":: What does the Northern Ireland crisis mean for Brexit?\n",
      "\n",
      "Announcing the dissolution of the Northern Ireland Assembly, Mr Brokenshire urged both parties \"to conduct this election with a view to...re-establishing a partnership government at the earliest opportunity after that poll.\"\n",
      "\n",
      "He said: \"This is essential for the operation of devolved government. And this means that all must remain open to dialogue.\"\n",
      "\n",
      "Sinn Fein and the DUP are expected to remain the two largest parties following the election, meaning they will still have to hammer out a power-sharing arrangement.\n",
      "\n",
      "If they fail to agree terms after three weeks, Mrs May could be forced to suspend devolution and reinstate direct rule from Westminster.\n",
      "\n",
      "Sky News Ireland Correspondent David Blevins said the relationship between Sinn Fein and the DUP had been \"slowly breaking down for a period of months\".\n",
      "\n",
      "He said: \"Some would suggest that the British and Irish governments took their eye off the ball.\n",
      "\n",
      "\"The botched renewable energy scheme is being blamed for the collapse of the devolved government but it was just the tip of the iceberg.\"\n",
      "\n",
      "He added that the collapse of the power-sharing government was the \"greatest challenge to face the Northern Ireland peace process in a decade\".\n",
      "\n",
      "\n",
      " © 2017 Sky UK\n",
      "\n"
     ]
    }
   ],
   "source": [
    "url = urlopen('http://news.sky.com/story/snap-election-to-be-held-in-march-after-northern-ireland-government-collapses-10731488')\n",
    "soup = BeautifulSoup(url.read().decode('utf8'), \"lxml\")\n",
    "text = '\\n\\n'.join(map(lambda p: p.text, soup.find_all('p')))\n",
    "\n",
    "text = text[text.find('An early election'):]\n",
    "title = soup.find('h1').text.strip()\n",
    "print(title, '\\n', '_' * 60, '\\n', text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculating the similarity between sentences\n",
    "\n",
    "This method goes through the following steps:\n",
    "\n",
    "- split text into sentences;\n",
    "- split sentences into words/tokens - there are several ways to do it, which give various results, I'll show them;\n",
    "- calculate similarity between sentences - while there are many ways to do it, I'll use a simple one: comparing tokens in each sentence. Similarity between sentences is calculated as number of words which are present in both sentences divided by average length of sentences (for normalization);\n",
    "- assign scores to sentences based on their similarity with other sentences - for each sentence get a sum of similarity scores with each other sentence;\n",
    "- select several best sentences and show them in order, in which they appeare in the article;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At first I'll simply split sentences into words, using space as a separator. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def intersection(sent1, sent2):\n",
    "    s1 = sent1.split(' ')\n",
    "    s2 = sent2.split(' ')\n",
    "\n",
    "    intersection = [i for i in s1 if i in s2]\n",
    "    #Normalization\n",
    "    return len(intersection) / ((len(s1) + len(s2)) / 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now creating a matrix of similarities between each pair of sentences. This is a 2D-matrix with a length equal to the number of sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1.0,\n",
       "  0.40816326530612246,\n",
       "  0.1568627450980392,\n",
       "  0.08695652173913043,\n",
       "  0.0,\n",
       "  0.10256410256410256,\n",
       "  0.15384615384615385,\n",
       "  0.25,\n",
       "  0.1111111111111111,\n",
       "  0.1875,\n",
       "  0.3018867924528302,\n",
       "  0.12121212121212122,\n",
       "  0.0,\n",
       "  0.16326530612244897,\n",
       "  0.08888888888888889,\n",
       "  0.2127659574468085,\n",
       "  0.10256410256410256,\n",
       "  0.34782608695652173,\n",
       "  0.4,\n",
       "  0.0],\n",
       " [0.24489795918367346,\n",
       "  1.0,\n",
       "  0.10714285714285714,\n",
       "  0.11764705882352941,\n",
       "  0.0,\n",
       "  0.09090909090909091,\n",
       "  0.17543859649122806,\n",
       "  0.03773584905660377,\n",
       "  0.1016949152542373,\n",
       "  0.21621621621621623,\n",
       "  0.20689655172413793,\n",
       "  0.21052631578947367,\n",
       "  0.0,\n",
       "  0.18518518518518517,\n",
       "  0.0,\n",
       "  0.19230769230769232,\n",
       "  0.09090909090909091,\n",
       "  0.3137254901960784,\n",
       "  0.24,\n",
       "  0.0]]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences = sent_tokenize(text)\n",
    "matrix = [[intersection(sentences[i], sentences[j]) for i in range(0,len(sentences))] for j in range(0,len(sentences))]\n",
    "matrix[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now calculating the score for each sentence, which is a sum of simiarity scores with other sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'\"The botched renewable energy scheme is being blamed for the collapse of the devolved government but it was just the tip of the iceberg.\"': 3.7260340696948515,\n",
       " ':: What does the Northern Ireland crisis mean for Brexit?': 3.9061312784591125,\n",
       " 'An early election will be held in Northern Ireland on 2 March after the collapse of its government, it has been announced.': 4.195413155308382,\n",
       " 'And this means that all must remain open to dialogue.\"': 2.0006127492497647,\n",
       " 'Announcing the dissolution of the Northern Ireland Assembly, Mr Brokenshire urged both parties \"to conduct this election with a view to...re-establishing a partnership government at the earliest opportunity after that poll.\"': 3.814137780904025,\n",
       " 'Despite a last-ditch attempt by Theresa May to urge a resolution, Sinn Fein MLA Conor Murphy said his party had decided to \"call time on the arrogance of the DUP\".': 3.894282386960945,\n",
       " 'He added that the collapse of the power-sharing government was the \"greatest challenge to face the Northern Ireland peace process in a decade\".': 4.696759739072093,\n",
       " 'He said: \"Some would suggest that the British and Irish governments took their eye off the ball.': 3.4353927892443257,\n",
       " 'He said: \"This is essential for the operation of devolved government.': 4.306593046828432,\n",
       " 'He said: \"We have had scandal after scandal, allegations of corruption need to be investigated properly and the people responsible need to be held to account.\"': 4.112337244524717,\n",
       " 'If they fail to agree terms after three weeks, Mrs May could be forced to suspend devolution and reinstate direct rule from Westminster.': 2.204353090231064,\n",
       " 'Mrs Foster, who presided over the controversial renewable energy scheme as enterprise minister, claimed Sinn Fein \"did not like the election result last May and are therefore looking to have another go\".': 3.8391866673233834,\n",
       " 'Northern Ireland Secretary James Brokenshire said the devolved Northern Ireland Assembly will sit for the last time on 25 January, before it is dissolved the following day.': 3.531232869189094,\n",
       " 'On Monday afternoon, Sinn Fein announced it would not replace Mr McGuinness - triggering the snap election.': 3.393079931658959,\n",
       " 'She refused, calling Mr McGuinness\\' actions \"not principled\" and \"purely political\".': 1.50805253635191,\n",
       " 'Sinn Fein and the DUP are expected to remain the two largest parties following the election, meaning they will still have to hammer out a power-sharing arrangement.': 4.168767988167754,\n",
       " 'Sky News Ireland Correspondent David Blevins said the relationship between Sinn Fein and the DUP had been \"slowly breaking down for a period of months\".': 4.635577167457064,\n",
       " 'The \"cash for ash\" scandal prompted the resignation of deputy first minister Martin McGuinness, who called for DUP first minister Arlene Foster to quit.': 3.7494398066338452,\n",
       " 'The break-up of the power-sharing government comes amid a dispute between Sinn Fein and the DUP over a botched renewable energy scheme that could have cost the taxpayer £500m.': 4.399355207918554,\n",
       " '© 2017 Sky UK': 1.0689655172413792}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = {sentences[i]: sum(matrix[i]) for i in range(len(matrix))}\n",
    "scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now I'll select five best sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['He added that the collapse of the power-sharing government was the \"greatest challenge to face the Northern Ireland peace process in a decade\".',\n",
       " 'Sky News Ireland Correspondent David Blevins said the relationship between Sinn Fein and the DUP had been \"slowly breaking down for a period of months\".',\n",
       " 'The break-up of the power-sharing government comes amid a dispute between Sinn Fein and the DUP over a botched renewable energy scheme that could have cost the taxpayer £500m.',\n",
       " 'He said: \"This is essential for the operation of devolved government.',\n",
       " 'An early election will be held in Northern Ireland on 2 March after the collapse of its government, it has been announced.']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sents = sorted(scores, key=scores.__getitem__, reverse=True)[:5]\n",
    "sents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Maybe there is a better way to sort sentences based on the order in which they appear in text, but this still works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['An early election will be held in Northern Ireland on 2 March after the collapse of its government, it has been announced.',\n",
       " 'He added that the collapse of the power-sharing government was the \"greatest challenge to face the Northern Ireland peace process in a decade\".',\n",
       " 'He said: \"This is essential for the operation of devolved government.',\n",
       " 'Sky News Ireland Correspondent David Blevins said the relationship between Sinn Fein and the DUP had been \"slowly breaking down for a period of months\".',\n",
       " 'The break-up of the power-sharing government comes amid a dispute between Sinn Fein and the DUP over a botched renewable energy scheme that could have cost the taxpayer £500m.']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tuples = [(i, text.find(i)) for i in sents]\n",
    "sorted_tuples = sorted(tuples, key=lambda x: x[0])\n",
    "#Leave only sentences.\n",
    "best_sents = [i[0] for i in sorted_tuples]\n",
    "best_sents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, I'll put everything together with a nice output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def intersection(sent1, sent2):\n",
    "    s1 = sent1.split(' ')\n",
    "    s2 = sent2.split(' ')\n",
    "    intersection = [i for i in s1 if i in s2]\n",
    "    return len(intersection) / ((len(s1) + len(s2)) / 2)\n",
    "\n",
    "def get_summary(text, limit=3):\n",
    "    sentences = sent_tokenize(text)\n",
    "    matrix = [[intersection(sentences[i], sentences[j]) for i in range(0,len(sentences))] for j in range(0,len(sentences))]\n",
    "    scores = {sentences[i]: sum(matrix[i]) for i in range(len(matrix))}\n",
    "    sents = sorted(scores, key=scores.__getitem__, reverse=True)[:limit]\n",
    "    best_sents = [i[0] for i in sorted([(i, text.find(i)) for i in sents], key=lambda x: x[0])]\n",
    "    return best_sents\n",
    "\n",
    "def summarize(text, limit=3):\n",
    "    summary = get_summary(text, limit)\n",
    "    print(title)\n",
    "    print()\n",
    "    print(' '.join(summary))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Snap election to be held in March after Northern Ireland government collapses\n",
      "\n",
      "An early election will be held in Northern Ireland on 2 March after the collapse of its government, it has been announced. He added that the collapse of the power-sharing government was the \"greatest challenge to face the Northern Ireland peace process in a decade\". He said: \"This is essential for the operation of devolved government. Sky News Ireland Correspondent David Blevins said the relationship between Sinn Fein and the DUP had been \"slowly breaking down for a period of months\". The break-up of the power-sharing government comes amid a dispute between Sinn Fein and the DUP over a botched renewable energy scheme that could have cost the taxpayer £500m.\n"
     ]
    }
   ],
   "source": [
    "summarize(text,5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, this is a summary. The number of sentences in summary is arbitrary and can be changed to get the necessary result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How can this algorithm be improved? I think that splitting sentences while calculating intersections should be changed. Splitting by spaces leaves punctuation attached to the words, which leads to mistakes when evaluating similarity between sentences. So I'll tokenize sentences using nltk and remove stopwords and punctuation. Also taking lemmas of words could help (but didn't help in this case - I tried)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def intersection(sent1, sent2):\n",
    "    s1 = [i for i in word_tokenize(sent1) if i not in punctuations and i not in stop]\n",
    "    s2 = [i for i in word_tokenize(sent2) if i not in punctuations and i not in stop]\n",
    "    intersection = [i for i in s1 if i in s2]\n",
    "    return len(intersection) / ((len(s1) + len(s2)) / 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Snap election to be held in March after Northern Ireland government collapses\n",
      "\n",
      "An early election will be held in Northern Ireland on 2 March after the collapse of its government, it has been announced. Announcing the dissolution of the Northern Ireland Assembly, Mr Brokenshire urged both parties \"to conduct this election with a view to...re-establishing a partnership government at the earliest opportunity after that poll.\" He added that the collapse of the power-sharing government was the \"greatest challenge to face the Northern Ireland peace process in a decade\". Sky News Ireland Correspondent David Blevins said the relationship between Sinn Fein and the DUP had been \"slowly breaking down for a period of months\". The break-up of the power-sharing government comes amid a dispute between Sinn Fein and the DUP over a botched renewable energy scheme that could have cost the taxpayer £500m.\n"
     ]
    }
   ],
   "source": [
    "summarize(text,5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the summary changed. And in one last change I'll increase the complexity of the model even further. Tokenizing sentences is good, but a better idea would be to use n_grams. For this I use gensim's Phrases. Phrases detects collocations in text and can be used for finding n_grams in text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('government', 6),\n",
       " ('Northern Ireland', 6),\n",
       " ('election', 5),\n",
       " ('scandal', 3),\n",
       " ('devolved', 3),\n",
       " ('Sinn Fein DUP', 3),\n",
       " ('Mr', 3),\n",
       " ('McGuinness', 3),\n",
       " ('Sinn Fein', 3),\n",
       " ('May', 3),\n",
       " ('renewable energy scheme', 3),\n",
       " ('He said', 3),\n",
       " ('power-sharing', 3),\n",
       " ('minister', 3),\n",
       " ('said', 3),\n",
       " ('collapse', 3),\n",
       " ('The', 3),\n",
       " ('need', 2),\n",
       " ('held', 2),\n",
       " ('Mrs', 2)]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sents = sent_tokenize(text)\n",
    "#Phrases need input as list of lists of tokens.\n",
    "sentence_stream = [[i for i in word_tokenize(sent) if i not in stop] for sent in sents]\n",
    "bigram = Phrases(sentence_stream, min_count=2, threshold=2, delimiter=b' ')\n",
    "#Create Phraser object.\n",
    "bigram_phraser = Phraser(bigram)\n",
    "bigram_tokens = bigram_phraser[sentence_stream]\n",
    "trigram = Phrases(bigram_tokens, min_count=2, threshold=2, delimiter=b' ')\n",
    "trigram_phraser = Phraser(trigram)\n",
    "trigram_tokens = trigram_phraser[bigram_tokens]\n",
    "all_words = [i for j in trigram_tokens for i in j]\n",
    "\n",
    "Counter(all_words).most_common(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that there are bigrams and trigrams among the most common words. Now I'll use this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def intersection(sent1, sent2):\n",
    "    #As sentences are lists of tokens, there is no need to split them.\n",
    "    intersection = [i for i in sent1 if i in sent2]\n",
    "    return len(intersection) / ((len(sent1) + len(sent2)) / 2)\n",
    "\n",
    "def split_sentences(sents):\n",
    "    sentence_stream = [[i for i in word_tokenize(sent) if i not in stop] for sent in sents]\n",
    "    bigram = Phrases(sentence_stream, min_count=2, threshold=2, delimiter=b'_')\n",
    "    bigram_phraser = Phraser(bigram)\n",
    "    bigram_tokens = bigram_phraser[sentence_stream]\n",
    "    trigram = Phrases(bigram_tokens,min_count=2, threshold=2, delimiter=b'_')\n",
    "    trigram_phraser = Phraser(trigram)\n",
    "    trigram_tokens = trigram_phraser[bigram_tokens]\n",
    "    return [i for i in trigram_tokens]\n",
    "\n",
    "def get_summary(text, limit=3):\n",
    "    sents = sent_tokenize(text)\n",
    "    sentences = split_sentences(sents)\n",
    "    matrix = [[intersection(sentences[i], sentences[j]) for i in range(0,len(sentences))] for j in range(0,len(sentences))]\n",
    "    scores = {sents[i]: sum(matrix[i]) for i in range(len(matrix))}\n",
    "    sents = sorted(scores, key=scores.__getitem__, reverse=True)[:limit]\n",
    "    best_sents = [i[0] for i in sorted([(i, text.find(i)) for i in sents], key=lambda x: x[0])]\n",
    "    return best_sents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Snap election to be held in March after Northern Ireland government collapses\n",
      "\n",
      "\"The botched renewable energy scheme is being blamed for the collapse of the devolved government but it was just the tip of the iceberg.\" An early election will be held in Northern Ireland on 2 March after the collapse of its government, it has been announced. Announcing the dissolution of the Northern Ireland Assembly, Mr Brokenshire urged both parties \"to conduct this election with a view to...re-establishing a partnership government at the earliest opportunity after that poll.\" He added that the collapse of the power-sharing government was the \"greatest challenge to face the Northern Ireland peace process in a decade\". The break-up of the power-sharing government comes amid a dispute between Sinn Fein and the DUP over a botched renewable energy scheme that could have cost the taxpayer £500m.\n"
     ]
    }
   ],
   "source": [
    "summarize(text,5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The summary changed again. Various ways to split sentences may work better on some types of texts and worse on others."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculating words' frequencies\n",
    "\n",
    "This method goes through the following steps:\n",
    "\n",
    "- split text into sentences and sentences into tokens;\n",
    "- assign scores to sentences based on the frequency of words in these sentences. It could be simply a point for each frequent word in sentence or some score based on frequency of certain words. Maybe add additional points if word is also in a title (as word in title should be more important). There are other options, but the main idea is that sentences's score is based on words in this sentence. It is similar to the idea of assigning scores based on similarities between sentences, but works in another way;\n",
    "- select several best sentences and show them in order, in which their appear in the article;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def score_sentences(words, sentences):\n",
    "    #Return scores for sentences.\n",
    "    scores = Counter()\n",
    "    #Words - list of words and their scores, first element is the word, second - its score.\n",
    "    for word in words:\n",
    "        for i in range(0, len(sentences)):\n",
    "            #If word is also in title, then add double score to the sentence.\n",
    "            if word[0] in sentences[i] and word[0] in title:\n",
    "                scores[i] += 2 * word[1]\n",
    "            elif word[0] in sentences[i]:\n",
    "                scores[i] += word[1]\n",
    "    sentence_scores = sorted(scores.items(), key=scores.__getitem__, reverse=True)\n",
    "    return sentence_scores\n",
    "\n",
    "def split_sentences(sents):\n",
    "\n",
    "    sentence_stream = [[i for i in word_tokenize(sent) if i not in stop] for sent in sents]\n",
    "    bigram = Phrases(sentence_stream, min_count=2, threshold=2, delimiter=b'_')\n",
    "    bigram_phraser = Phraser(bigram)\n",
    "    bigram_tokens = bigram_phraser[sentence_stream]\n",
    "    trigram = Phrases(bigram_tokens,min_count=2, threshold=2, delimiter=b'_')\n",
    "    trigram_phraser = Phraser(trigram)\n",
    "    trigram_tokens = trigram_phraser[bigram_tokens]\n",
    "    \n",
    "    all_words = [i for j in trigram_tokens for i in j]\n",
    "    frequent_words = [i for i in Counter(all_words).most_common() if i[1] > 1]\n",
    "    sentences = [i for i in trigram_tokens]\n",
    "    \n",
    "    return frequent_words, sentences\n",
    "\n",
    "def get_summary(text, limit=3):\n",
    "    sents = sent_tokenize(text)\n",
    "    frequent_words, sentences = split_sentences(sents)\n",
    "    sentence_scores = score_sentences(frequent_words, sentences)\n",
    "    \n",
    "    limited_sents = [sents[num] for num, count in sentence_scores[:limit]]\n",
    "    best_sents = [i[0] for i in sorted([(i, text.find(i)) for i in limited_sents], key=lambda x: x[0])]\n",
    "    return best_sents\n",
    "\n",
    "def summarize(text, limit=3):\n",
    "    summary = get_summary(text, limit)\n",
    "    print(title)\n",
    "    print()\n",
    "    print(' '.join(summary))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Snap election to be held in March after Northern Ireland government collapses\n",
      "\n",
      "An early election will be held in Northern Ireland on 2 March after the collapse of its government, it has been announced. Northern Ireland Secretary James Brokenshire said the devolved Northern Ireland Assembly will sit for the last time on 25 January, before it is dissolved the following day. She refused, calling Mr McGuinness' actions \"not principled\" and \"purely political\". The \"cash for ash\" scandal prompted the resignation of deputy first minister Martin McGuinness, who called for DUP first minister Arlene Foster to quit. The break-up of the power-sharing government comes amid a dispute between Sinn Fein and the DUP over a botched renewable energy scheme that could have cost the taxpayer £500m.\n"
     ]
    }
   ],
   "source": [
    "summarize(text, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Conclusions\n",
    "\n",
    "As I have shown, there are many ways to summarize articles with extraction methods. Of course, there are many other ideas which could improve the algorithms. And it is difficult to measure the accuracy of summaries - often there are many \"meaningful\" sentences and choosing one best combination of them isn't possible. So we just try several ways and choose the best implementation for a particular case. And try developing abstraction methods, as extraction methods are limited."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}