sample/nlp-sentence_tokenization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "![Sentence Tokenization](http://www.digitalmeetsculture.net/wp-content/uploads/2015/04/article.jpg)\n",
    "\n",
    "Source: http://www.digitalmeetsculture.net/article/article-about-preforma-published-in-archival-science/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Sentence Tokenization\n",
    "\n",
    "In previous article, word tokenization is introduced. What if we want to tokenize sentence? In general, we can easily split sentence by some punctuation such ., ? and !. However, there are lots of exception if we splitting article by those punctuation only.\n",
    "In this article, you will go through why we need to use sentence tokenization and how can we use it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Why?\n",
    "According to researchers, about 86% of article include the importance sentence in first one or two sentences. Believe that it is one of the reason why textsum model use first 2 sentences for training\n",
    "When I am in school, teacher teaches how we should write an article. The importance sentence will be placed in the first sentence most of the time. It may exists in last sentence sometimes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How?\n",
    "So how can we tokenize sentence? You can use the following simple python script to do that or using library such as nltk and spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Capture from https://en.wikipedia.org/wiki/Lexical_analysis\n",
    "\n",
    "article = 'In computer science, lexical analysis, lexing or tokenization is the process of \\\n",
    "converting a sequence of characters (such as in a computer program or web page) into a \\\n",
    "sequence of tokens (strings with an assigned and thus identified meaning). A program that \\\n",
    "performs lexical analysis may be termed a lexer, tokenizer,[1] or scanner, though scanner \\\n",
    "is also a term for the first stage of a lexer. A lexer is generally combined with a parser, \\\n",
    "which together analyze the syntax of programming languages, web pages, and so forth.'\n",
    "\n",
    "article2 = 'ConcateStringAnd123 ConcateSepcialCharacter_!@# !@#$%^&*()_+ 0123456'\n",
    "\n",
    "article3 = 'It is a great moment from 10 a.m. to 1 p.m. every weekend.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Self build"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original Article: In computer science, lexical analysis, lexing or tokenization is the process of converting a sequence of characters (such as in a computer program or web page) into a sequence of tokens (strings with an assigned and thus identified meaning). A program that performs lexical analysis may be termed a lexer, tokenizer,[1] or scanner, though scanner is also a term for the first stage of a lexer. A lexer is generally combined with a parser, which together analyze the syntax of programming languages, web pages, and so forth.\n",
      "\n",
      "-->Sentence 0: In computer science, lexical analysis, lexing or tokenization is the process of converting a sequence of characters (such as in a computer program or web page) into a sequence of tokens (strings with an assigned and thus identified meaning)\n",
      "-->Sentence 1: .\n",
      "-->Sentence 2:  A program that performs lexical analysis may be termed a lexer, tokenizer,[1] or scanner, though scanner is also a term for the first stage of a lexer\n",
      "-->Sentence 3: .\n",
      "-->Sentence 4:  A lexer is generally combined with a parser, which together analyze the syntax of programming languages, web pages, and so forth\n",
      "-->Sentence 5: .\n",
      "-->Sentence 6: \n",
      "Original Article: ConcateStringAnd123 ConcateSepcialCharacter_!@# !@#$%^&*()_+ 0123456\n",
      "\n",
      "-->Sentence 0: ConcateStringAnd123 ConcateSepcialCharacter_\n",
      "-->Sentence 1: !\n",
      "-->Sentence 2: @# \n",
      "-->Sentence 3: !\n",
      "-->Sentence 4: @#$%^&*()_+ 0123456\n",
      "Original Article: It is a great moment from 10 a.m. to 1 p.m. every weekend.\n",
      "\n",
      "-->Sentence 0: It is a great moment from 10 a\n",
      "-->Sentence 1: .\n",
      "-->Sentence 2: m\n",
      "-->Sentence 3: .\n",
      "-->Sentence 4:  to 1 p\n",
      "-->Sentence 5: .\n",
      "-->Sentence 6: m\n",
      "-->Sentence 7: .\n",
      "-->Sentence 8:  every weekend\n",
      "-->Sentence 9: .\n",
      "-->Sentence 10: \n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "for doc in [article, article2, article3]:\n",
    "    print('Original Article: %s' % (doc))\n",
    "    print()\n",
    "\n",
    "    sentences = re.split('(\\.|!|\\?)', doc)\n",
    "    \n",
    "    for i, s in enumerate(sentences):\n",
    "        print('-->Sentence %d: %s' % (i, s))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that, \"a.m.\" should treat as a \"word\". Of course, we can enhance the above regular expression to do it. But I will go for library rather than build the wheel again"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### spaCy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "spaCy Version: 2.0.11\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "print('spaCy Version: %s' % spacy.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "spacy_nlp = spacy.load('en_core_web_sm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original Article: In computer science, lexical analysis, lexing or tokenization is the process of converting a sequence of characters (such as in a computer program or web page) into a sequence of tokens (strings with an assigned and thus identified meaning). A program that performs lexical analysis may be termed a lexer, tokenizer,[1] or scanner, though scanner is also a term for the first stage of a lexer. A lexer is generally combined with a parser, which together analyze the syntax of programming languages, web pages, and so forth.\n",
      "\n",
      "-->Sentence 0: In computer science, lexical analysis, lexing or tokenization is the process of converting a sequence of characters (such as in a computer program or web page) into a sequence of tokens (strings with an assigned and thus identified meaning).\n",
      "-->Sentence 1: A program that performs lexical analysis may be termed a lexer, tokenizer,[1] or scanner, though scanner is also a term for the first stage of a lexer.\n",
      "-->Sentence 2: A lexer is generally combined with a parser, which together analyze the syntax of programming languages, web pages, and so forth.\n",
      "Original Article: ConcateStringAnd123 ConcateSepcialCharacter_!@# !@#$%^&*()_+ 0123456\n",
      "\n",
      "-->Sentence 0: ConcateStringAnd123 ConcateSepcialCharacter_!@# !\n",
      "-->Sentence 1: @#$%^&*()_+ 0123456\n",
      "Original Article: It is a great moment from 10 a.m. to 1 p.m. every weekend.\n",
      "\n",
      "-->Sentence 0: It is a great moment from 10 a.m. to 1 p.m. every weekend.\n"
     ]
    }
   ],
   "source": [
    "for article in [article, article2, article3]:\n",
    "    print('Original Article: %s' % (article))\n",
    "    print()\n",
    "    doc = spacy_nlp(article)\n",
    "    for i, token in enumerate(doc.sents):\n",
    "        print('-->Sentence %d: %s' % (i, token.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can see that spacy handled \"a.m.\" somehow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NLTK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NTLK Version: 3.2.5\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "from nltk.tokenize import sent_tokenize\n",
    "print('NTLK Version: %s' % nltk.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original Article: It is a great moment from 10 a.m. to 1 p.m. every weekend.\n",
      "\n",
      "-->Sentence 0: It is a great moment from 10 a.m. to 1 p.m. every weekend.\n",
      "Original Article: ConcateStringAnd123 ConcateSepcialCharacter_!@# !@#$%^&*()_+ 0123456\n",
      "\n",
      "-->Sentence 0: ConcateStringAnd123 ConcateSepcialCharacter_!\n",
      "-->Sentence 1: @# !\n",
      "-->Sentence 2: @#$%^&*()_+ 0123456\n",
      "Original Article: It is a great moment from 10 a.m. to 1 p.m. every weekend.\n",
      "\n",
      "-->Sentence 0: It is a great moment from 10 a.m. to 1 p.m. every weekend.\n"
     ]
    }
   ],
   "source": [
    "for article in [article, article2, article3]:\n",
    "    print('Original Article: %s' % (article))\n",
    "    print()\n",
    "\n",
    "    doc = sent_tokenize(article)\n",
    "    for i, token in enumerate(doc):\n",
    "        print('-->Sentence %d: %s' % (i, token))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "So far both NLTK and spacy provides similar behavior so it depends on which library do you use in performing other preprocessing. \n",
    "Recently, I works on text mining related project which is classifying news category. Of course, I can build a ML model to classify it but I go for a simple approach. Only focus on the first sentence for every news and performing simple key word searching to build a baseline model. The result is not bad but it is a very quick way to deliver an initial version."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}