t81_558_class_11_01_spacy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 11: Natural Language Processing and Speech Recognition**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 11 Material\n",
    "\n",
    "* **Part 11.1: Getting Started with Spacy in Python** [[Video]](https://www.youtube.com/watch?v=bv_iVVrlfbU) [[Notebook]](t81_558_class_11_01_spacy.ipynb)\n",
    "* Part 11.2: Word2Vec and Text Classification [[Video]](https://www.youtube.com/watch?v=qN9hHlZKIL4) [[Notebook]](t81_558_class_11_02_word2vec.ipynb)\n",
    "* Part 11.3: What are Embedding Layers in Keras [[Video]](https://www.youtube.com/watch?v=Ae3GVw5nTYU) [[Notebook]](t81_558_class_11_03_embedding.ipynb)\n",
    "* Part 11.4: Natural Language Processing with Spacy and Keras [[Video]](https://www.youtube.com/watch?v=Ae3GVw5nTYU) [[Notebook]](t81_558_class_11_04_text_nlp.ipynb)\n",
    "* Part 11.5: Learning English from Scratch with Keras and TensorFlow [[Video]](https://www.youtube.com/watch?v=Ae3GVw5nTYU) [[Notebook]](t81_558_class_11_05_english_scratch.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 11.1: Getting Started with Spacy in Python\n",
    "\n",
    "When neural networks are applied to natural language processing you must decide if you want to operate at the word or character level.  Up to this point we've operated primarily at the character level.  This was the case for the Treasure Island text pirate story generator.  We used word-level NLP for the image caption generator.  In this module, the focus will be primarily upon word-level NLP.  Particularly, we will examine some of the NLP tools that can be used to process words before they are sent to the neural network.  There are two very common NLP libraries for Python:\n",
    "\n",
    "* [NLTK](https://www.nltk.org/)\n",
    "* [Spacy](https://spacy.io/)\n",
    "\n",
    "In this course we will focus on Spacy.  I prefer spacy because of the nice object abstraction of sentences that it provides.  However, both are fine libraries.\n",
    "\n",
    "### Installing Spacy\n",
    "\n",
    "Spacy can be installed with a simple PIP install. This was included in the list of packages to install for this course.  You will need to ensure that you've installed a language with Spacy.  If you do not, you will get the following error:\n",
    "\n",
    "```\n",
    "OSError: [E050] Can't find model 'en_core_web_sm'. It doesn't seem to be a shortcut link, a Python package or a valid path to a data directory.\n",
    "```\n",
    "\n",
    "To install English, use the following command:\n",
    "\n",
    "```\n",
    "python -m spacy download en\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a test.\n",
    "\n",
    "Ok, but what about this?\n",
    "\n",
    "Is USA the same as U.S.A.?\n",
    "\n",
    "What is the best data-set to use?\n",
    "\n",
    "I think I will do this-no wait, I will do that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Apple\n",
      "is\n",
      "looking\n",
      "at\n",
      "buying\n",
      "a\n",
      "U.K.\n",
      "startup\n",
      "for\n",
      "$\n",
      "1\n",
      "billion\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(u\"Apple is looking at buying a U.K. startup for $1 billion\")\n",
    "for token in doc:\n",
    "    print(token.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Apple PROPN\n",
      "is VERB\n",
      "looking VERB\n",
      "at ADP\n",
      "buying VERB\n",
      "a DET\n",
      "U.K. PROPN\n",
      "startup NOUN\n",
      "for ADP\n",
      "$ SYM\n",
      "1 NUM\n",
      "billion NUM\n"
     ]
    }
   ],
   "source": [
    "for word in doc:  \n",
    "    print(word.text,  word.pos_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Apple is like number? False\n",
      "is is like number? False\n",
      "looking is like number? False\n",
      "at is like number? False\n",
      "buying is like number? False\n",
      "a is like number? False\n",
      "U.K. is like number? False\n",
      "startup is like number? False\n",
      "for is like number? False\n",
      "$ is like number? False\n",
      "1 is like number? True\n",
      "billion is like number? True\n"
     ]
    }
   ],
   "source": [
    "for word in doc:\n",
    "    print(f\"{word} is like number? {word.like_num}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentence Diagramming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jheaton/miniconda3/envs/tensorflow-2.0/lib/python3.6/runpy.py:193: UserWarning: [W011] It looks like you're calling displacy.serve from within a Jupyter notebook or a similar environment. This likely means you're already running a local web server, so there's no need to make displaCy start another one. Instead, you should be able to replace displacy.serve with displacy.render to show the visualization.\n",
      "  \"__main__\", mod_spec)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<!DOCTYPE html>\n",
       "<html lang=\"en\">\n",
       "    <head>\n",
       "        <title>displaCy</title>\n",
       "    </head>\n",
       "\n",
       "    <body style=\"font-size: 16px; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; padding: 4rem 2rem; direction: ltr\">\n",
       "<figure style=\"margin-bottom: 6rem\">\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"eb11c059dcb5469a893a2e52428d8617-0\" class=\"displacy\" width=\"1450\" height=\"399.5\" direction=\"ltr\" style=\"max-width: none; height: 399.5px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">I</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">PRON</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">want</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">an</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"575\">iPad,</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"575\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"750\">Laptop,</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"750\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"925\">and</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"925\">CCONJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1100\">a</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1100\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"309.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1275\">dog.</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1275\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-eb11c059dcb5469a893a2e52428d8617-0-0\" stroke-width=\"2px\" d=\"M70,264.5 C70,177.0 215.0,177.0 215.0,264.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-eb11c059dcb5469a893a2e52428d8617-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M70,266.5 L62,254.5 78,254.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-eb11c059dcb5469a893a2e52428d8617-0-1\" stroke-width=\"2px\" d=\"M420,264.5 C420,177.0 565.0,177.0 565.0,264.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-eb11c059dcb5469a893a2e52428d8617-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M420,266.5 L412,254.5 428,254.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-eb11c059dcb5469a893a2e52428d8617-0-2\" stroke-width=\"2px\" d=\"M245,264.5 C245,89.5 570.0,89.5 570.0,264.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-eb11c059dcb5469a893a2e52428d8617-0-2\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">dobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M570.0,266.5 L578.0,254.5 562.0,254.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-eb11c059dcb5469a893a2e52428d8617-0-3\" stroke-width=\"2px\" d=\"M595,264.5 C595,177.0 740.0,177.0 740.0,264.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-eb11c059dcb5469a893a2e52428d8617-0-3\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">conj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M740.0,266.5 L748.0,254.5 732.0,254.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-eb11c059dcb5469a893a2e52428d8617-0-4\" stroke-width=\"2px\" d=\"M770,264.5 C770,177.0 915.0,177.0 915.0,264.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-eb11c059dcb5469a893a2e52428d8617-0-4\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">cc</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M915.0,266.5 L923.0,254.5 907.0,254.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-eb11c059dcb5469a893a2e52428d8617-0-5\" stroke-width=\"2px\" d=\"M1120,264.5 C1120,177.0 1265.0,177.0 1265.0,264.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-eb11c059dcb5469a893a2e52428d8617-0-5\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1120,266.5 L1112,254.5 1128,254.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-eb11c059dcb5469a893a2e52428d8617-0-6\" stroke-width=\"2px\" d=\"M770,264.5 C770,2.0 1275.0,2.0 1275.0,264.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-eb11c059dcb5469a893a2e52428d8617-0-6\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">conj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1275.0,266.5 L1283.0,254.5 1267.0,254.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg>\n",
       "</figure>\n",
       "</body>\n",
       "</html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Using the 'dep' visualizer\n",
      "Serving on http://0.0.0.0:5000 ...\n",
      "\n",
      "Shutting down server on port 5000.\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "from spacy import displacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(u\"I want an iPad, Laptop, and a dog.\")\n",
    "displacy.serve(doc, style=\"dep\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I want an iPad, Laptop, and a dog.\n"
     ]
    }
   ],
   "source": [
    "print(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'the stripe bat be hang on -PRON- foot for good'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "# Initialize spacy 'en' model, keeping only tagger component needed for lemmatization\n",
    "nlp = spacy.load('en', disable=['parser', 'ner'])\n",
    "\n",
    "sentence = \"The striped bats are hanging on their feet for best\"\n",
    "\n",
    "# Parse the sentence using the loaded 'en' model object `nlp`\n",
    "doc = nlp(sentence)\n",
    "\n",
    "# Extract the lemma for each token and join\n",
    "\" \".join([token.lemma_ for token in doc])\n",
    "#> 'the strip bat be hang on -PRON- foot for good'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stop Words\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'go', 'seem', '’ve', 'no', 'quite', 'can', 'throughout', 'rather', 'have', 'put', 'hereby', 'n‘t', 'neither', 'unless', 'a', 'few', 'while', 'this', 'within', 'beforehand', 'there', 'thereafter', 'next', 'off', 'alone', 'please', '‘m', 'nothing', 'whence', 'whereupon', 'because', 'less', 'between', 'always', 'then', 'but', 'herself', 'here', 'amount', 'and', 'as', 'his', 'itself', '’s', 'perhaps', 'mostly', 'she', 'thereby', 'never', 'they', 'still', 'cannot', 'any', 'may', 'meanwhile', 'whenever', 'anyone', 'already', 'once', 'myself', 'of', 'am', 'an', 'enough', 'regarding', 'five', 'serious', 'becomes', 'thus', 'bottom', 'move', 'none', 'something', 'such', 'into', 'anyhow', 'their', 'indeed', 'third', 'everything', 'thereupon', 'noone', 'those', 'when', 'among', 'else', \"'d\", 'nowhere', 'us', 'show', 'whereafter', 'yourselves', 'hereupon', 'down', 'keep', 'further', 'nine', 'though', 'one', 're', 'nor', 'former', 'eleven', 'often', 'with', 'empty', 'him', 'where', 'latterly', 'several', 'fifty', '‘s', 'became', 'each', 'along', 'than', \"'s\", 'been', 'ourselves', 'whoever', 'due', 'sometime', 'back', 'least', 'them', 'others', 'below', 'at', 'ours', 'what', 'again', 'could', 'whereby', 'be', 'also', 'whither', 'were', 'onto', 'make', 'every', 'her', 'is', 'during', 'might', 'ten', 'afterwards', 'becoming', 'through', 'whatever', 'more', 'whereas', 'sixty', 'without', 'another', 'herein', 'beyond', 'only', 'over', 'elsewhere', 'which', 'how', 'whose', \"'ll\", 'before', 'formerly', 'our', 'well', 'you', '‘ll', 'besides', 'whom', 'latter', 'now', 'whether', 'yourself', 'side', '‘re', 'had', 'otherwise', '‘ve', 'beside', 'however', 'full', 'seems', 'front', 'twenty', 'under', 'own', 'together', 'twelve', 'the', 'nevertheless', 'upon', 'do', 'see', 'he', 'same', 'seemed', 'against', 'we', 'by', 'above', 'anything', 'get', 'has', 'my', 'using', '‘d', 'would', 'made', 'mine', 'yours', 'wherever', '’d', 'its', 'everyone', 'anywhere', 'hereafter', 'thence', 'nobody', 'somewhere', 'who', 'are', 'someone', 'hence', 'thru', 'out', 'part', 'himself', 'it', 'me', 'up', 'about', 'forty', \"'m\", \"'re\", '’m', 'therein', 'if', 'many', 'take', 'most', 'give', 'so', 'after', 'except', 'being', 'although', 'was', 'hers', 'towards', 'yet', 'since', 'in', 'that', 'i', 'top', 'some', 'did', 'either', 'both', 'first', 'used', 'very', 'other', 'almost', 'even', 'until', 'various', 'much', 'anyway', 'two', 'whole', 'will', 'on', 'behind', 'six', 'somehow', 'too', 'wherein', \"'ve\", 'really', 'these', 'done', 'not', 'does', 'four', 'therefore', 'seeming', 'just', 'eight', 'toward', 'ca', 'n’t', '’ll', 'per', 'all', 'why', 'should', 'amongst', 'your', 'say', 'for', 'sometimes', \"n't\", 'via', 'call', 'hundred', 'or', 'to', 'three', 'around', 'namely', 'across', 'from', 'doing', 'last', 'moreover', 'name', 'themselves', '’re', 'everywhere', 'must', 'become', 'ever', 'fifteen'}\n"
     ]
    }
   ],
   "source": [
    "from spacy.lang.en.stop_words import STOP_WORDS\n",
    "\n",
    "print(STOP_WORDS)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow-2.0)",
   "language": "python",
   "name": "tensorflow-2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}