Lab1-assignment-toolkits.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab1-Assignment\n",
    "\n",
    "Copyright: Vrije Universiteit Amsterdam, Faculty of Humanities, CLTL\n",
    "\n",
    "This notebook describes the assignment for Lab 1 of the text mining course. \n",
    "\n",
    "**Points**: each exercise is prefixed with the number of points you can obtain for the exercise.\n",
    "\n",
    "We assume you have worked through the following notebooks:\n",
    "* **Lab1.1-introduction**\n",
    "* **Lab1.2-introduction-to-NLTK**\n",
    "* **Lab1.3-introduction-to-spaCy** \n",
    "\n",
    "In this assignment, you will process an English text (**Lab1-apple-samsung-example.txt**) with both NLTK and spaCy and discuss the similarities and differences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Credits\n",
    "The notebooks in this block have been originally created by [Marten Postma](https://martenpostma.github.io). Adaptations were made by [Filip Ilievski](http://ilievski.nl)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tip: how to read a file from disk\n",
    "Let's open the file **Lab1-apple-samsung-example.txt** from disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/iliamarkov/work/VU-TM-2022-changes/ba-text-mining/Lab1-apple-samsung-example.txt\n",
      "does path exist? -> False\n"
     ]
    }
   ],
   "source": [
    "cur_dir = Path().resolve() # this should provide you with the folder in which this notebook is placed\n",
    "path_to_file = Path.joinpath(cur_dir, 'Lab1-apple-samsung-example.txt')\n",
    "print(path_to_file)\n",
    "print('does path exist? ->', Path.exists(path_to_file))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the output from the code cell above states that **does path exist? -> False**, please check that the file **Lab1-apple-samsung-example.txt** is in the same directory as this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of characters 1139\n"
     ]
    }
   ],
   "source": [
    "with open(path_to_file) as infile:\n",
    "    text = infile.read()\n",
    "\n",
    "print('number of characters', len(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [total points: 4] Exercise 1: NLTK\n",
    "In this exercise, we use NLTK to apply **Part-of-speech (POS) tagging**, **Named Entity Recognition (NER)**, and **Constituency parsing**. The following code snippet already performs sentence splitting and tokenization. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.tokenize import sent_tokenize\n",
    "from nltk import word_tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences_nltk = sent_tokenize(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens_per_sentence = []\n",
    "for sentence_nltk in sentences_nltk:\n",
    "    sent_tokens = word_tokenize(sentence_nltk)\n",
    "    tokens_per_sentence.append(sent_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use lists to keep track of the output of the NLP tasks. We can hence inspect the output for each task using the index of the sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SENTENCE The six phones and tablets affected are the Galaxy S III, running the new Jelly Bean system, the Galaxy Tab 8.9 Wifi tablet, the Galaxy Tab 2 10.1, Galaxy Rugby Pro and Galaxy S III mini.\n",
      "TOKENS ['The', 'six', 'phones', 'and', 'tablets', 'affected', 'are', 'the', 'Galaxy', 'S', 'III', ',', 'running', 'the', 'new', 'Jelly', 'Bean', 'system', ',', 'the', 'Galaxy', 'Tab', '8.9', 'Wifi', 'tablet', ',', 'the', 'Galaxy', 'Tab', '2', '10.1', ',', 'Galaxy', 'Rugby', 'Pro', 'and', 'Galaxy', 'S', 'III', 'mini', '.']\n"
     ]
    }
   ],
   "source": [
    "sent_id = 1\n",
    "print('SENTENCE', sentences_nltk[sent_id])\n",
    "print('TOKENS', tokens_per_sentence[sent_id])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [point: 1] Exercise 1a: Part-of-speech (POS) tagging\n",
    "Use `nltk.pos_tag` to perform part-of-speech tagging on each sentence.\n",
    "\n",
    "Use `print` to **show** the output in the notebook (and hence also in the exported PDF!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "pos_tags_per_sentence = []\n",
    "for tokens in tokens_per_sentence:\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "print(pos_tags_per_sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [point: 1] Exercise 1b: Named Entity Recognition (NER)\n",
    "Use `nltk.chunk.ne_chunk` to perform Named Entity Recognition (NER) on each sentence.\n",
    "\n",
    "Use `print` to **show** the output in the notebook (and hence also in the exported PDF!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_tags_per_sentence = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(ner_tags_per_sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [points: 2] Exercise 1c: Constituency parsing\n",
    "Use the `nltk.RegexpParser` to perform constituency parsing on each sentence.\n",
    "\n",
    "Use `print` to **show** the output in the notebook (and hence also in the exported PDF!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "constituent_parser = nltk.RegexpParser('''\n",
    "NP: {<DT>? <JJ>* <NN>*} # NP\n",
    "P: {<IN>}           # Preposition\n",
    "V: {<V.*>}          # Verb\n",
    "PP: {<P> <NP>}      # PP -> P NP\n",
    "VP: {<V> <NP|PP>*}  # VP -> V (NP|PP)*''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "constituency_output_per_sentence = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(constituency_output_per_sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Augment the RegexpParser so that it also detects Named Entity Phrases (NEP), e.g., that it detects *Galaxy S III* and *Ice Cream Sandwich*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "constituent_parser_v2 = nltk.RegexpParser('''\n",
    "NP: {<DT>? <JJ>* <NN>*} # NP\n",
    "P: {<IN>}           # Preposition\n",
    "V: {<V.*>}          # Verb\n",
    "PP: {<P> <NP>}      # PP -> P NP\n",
    "VP: {<V> <NP|PP>*}  # VP -> V (NP|PP)*\n",
    "NEP: {}             # ???''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "constituency_v2_output_per_sentence = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(constituency_v2_output_per_sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [total points: 1] Exercise 2: spaCy\n",
    "Use Spacy to process the same text as you analyzed with NLTK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('en_core_web_sm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(text) # insert code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "small tip: You can use **sents = list(doc.sents)** to be able to use the index to access a sentence like **sents[2]** for the third sentence.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [total points: 7] Exercise 3: Comparison NLTK and spaCy\n",
    "We will now compare the output of NLTK and spaCy, i.e., in what do they differ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [points: 3] Exercise 3a: Part of speech tagging\n",
    "Compare the output from NLTK and spaCy regarding part of speech tagging.\n",
    "\n",
    "* To compare, you probably would like to compare sentence per sentence. Describe if the sentence splitting is different for NLTK than for spaCy. If not, where do they differ?\n",
    "* After checking the sentence splitting, select a sentence for which you expect interesting results and perhaps differences. Motivate your choice.\n",
    "* Compare the output in `token.tag` from spaCy to the part of speech tagging from NLTK for each token in your selected sentence. Are there any differences? This is not a trick question; it is possible that there are no differences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [points: 2] Exercise 3b: Named Entity Recognition (NER)\n",
    "* Describe differences between the output from NLTK and spaCy for Named Entity Recognition. Which one do you think performs better?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [points: 2] Exercise 3c: Constituency/dependency parsing\n",
    "Choose one sentence from the text and run constituency parsing using NLTK and dependency parsing using spaCy.\n",
    "* describe briefly the difference between constituency parsing and dependency parsing\n",
    "* describe differences between the output from NLTK and spaCy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End of this notebook"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}