spaCy_demo_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# spaCy Demo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Import spaCy's English analyzer and the location where data is stored\n",
    "import pandas as pd\n",
    "from spacy.en import English, LOCAL_DATA_DIR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# What's the English object all about?\n",
    "English?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Set up spaCy NLP analyzer (tokenizer, parser, NER-er, etc.)\n",
    "nlp_analyzer = English(data_dir=LOCAL_DATA_DIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# How does one interact with the analyzer object?\n",
    "nlp_analyzer?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "The analyzer expects as input text -- the output contains all types of analysis\n",
    "(parsing, tagging, and entity recognition can be turned off by setting them to\n",
    "False)\n",
    "\n",
    "Here's some input text that we'll use:\n",
    "\"\"\"\n",
    "text = [\"This is a very simple sentence.\",\n",
    "        \"This sentence, which is moderately more complex, is still quite simple.\",\n",
    "        \"The two preceding sentences are easy to understand, hopefully easy to parse too.\",\n",
    "        \"These sentences will be correctly parsed and tokenized if the gods look favorably on this demo.\",\n",
    "        \"I hope that strange words like vapidity and celerity don't confuse the analyser (nor British spellings).\",\n",
    "        \"One would even hopes that ungrammatical sentences not effects the parsing drammatically.\"]\n",
    "text = ' '.join(text)\n",
    "\n",
    "\"\"\"\n",
    "Let's analyze it and also get a sense for how long it takes for a text of this size\n",
    "to be analyzed\n",
    "\"\"\"\n",
    "%timeit nlp_analyzer(text)\n",
    "analyzed_text = nlp_analyzer(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "analyzed_text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentence Recognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### .sents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Let's take a look at what's in the output\n",
    "\n",
    "The output is automatically divided up into the constituent sentences (.sents\n",
    "attribute) and the sentences and text are composed of constituent tokens\n",
    "\"\"\"\n",
    "for sent in analyzed_text.sents:\n",
    "    print('{}\\n'.format(sent))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "The .sents attribute is a generator and it stores the objects corresponding to each\n",
    "recognized sentence\n",
    "\"\"\"\n",
    "sent = next(analyzed_text.sents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Each sentence is of type spacy.tokens.span.Span, which is basically just a sequence\n",
    "of token objects (more on that later)\n",
    "\n",
    "Here you can see the type of the objects\n",
    "\"\"\"\n",
    "type(sent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### .string or .orth_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "To get the string representation of anything (not just a sentence object), i.e., the\n",
    "original token, the original sentence, the lemma, etc., use the .string or .orth_\n",
    "attributes\n",
    "\"\"\"\n",
    "sent.orth_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# The .string attribute contains whitespace\n",
    "sent.string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Miscellaneous String Attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### .is_alpha, .is_oov, .is_space, .like_email, .is_title, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Various pieces of information can be collected about each object representing a\n",
    "token.\n",
    "\"\"\"\n",
    "lines = []\n",
    "for token in sent:\n",
    "    lines.append(dict(Token=token.orth_, letter=token.is_alpha, ASCII=token.is_ascii,\n",
    "                      digit=token.is_digit, lower=token.is_lower, OOV=token.is_oov,\n",
    "                      punct=token.is_punct, space=token.is_space, stop=token.is_stop,\n",
    "                      titlecase=token.is_title, like_email=token.like_email,\n",
    "                      like_number=token.like_num, like_url=token.like_url,\n",
    "                      shape=token.shape_, prefix=token.prefix_, suffix=token.suffix_,\n",
    "                      lowercased=token.lower_))\n",
    "pd.DataFrame(lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### .doc attribute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "If you want the whole document that the sentence occurred in, use the .doc attribute.\n",
    "\"\"\"\n",
    "sent.doc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lemmatization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### .lemma_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "A lemmatized version of the object can be accessed via the .lemma_ attribute\n",
    "\"\"\"\n",
    "sent.lemma_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parts of Speech and Tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tokens = []\n",
    "for token in sent:\n",
    "    tokens.append(dict(Token=token.orth_, tag=token.tag_, part_of_speech=token.pos_))\n",
    "pd.DataFrame(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parsing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### .root"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(\"sentence = {}\".format(sent.orth_))\n",
    "print(\"root of sentence = {}\".format(sent.root))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### .children, .dep_ attributes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Parse tree-related attributes can be accessed for each token, such as the\n",
    "children/parents of the token, the dependency relationships, etc.\n",
    "\"\"\"\n",
    "token = sent[1]\n",
    "print(\"sentence = {}\".format(sent.orth_))\n",
    "print(\"token: {}\".format(token))\n",
    "print(\"children: {}\".format(list(token.children)))\n",
    "print(\"head: {}\".format(token.head))\n",
    "print(\"dependency relationship: {}\".format(token.dep_))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word Representation Vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### .repvec, .has\\_vector, .similarity()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Representing words as vectors allows for similarity calculations.\n",
    "\"\"\"\n",
    "last_sentence = list(analyzed_text.sents)[-1]\n",
    "last_sentence.string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "token1 = last_sentence[5]\n",
    "token1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Does this token have a vector?\n",
    "token1.has_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "token1.repvec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "token2 = last_sentence[6]\n",
    "token2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# How similar are \"ungrammatical\" and \"sentences\"?\n",
    "token1.similarity(token2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# How similar are two random other words?\n",
    "token3 = last_sentence[8]\n",
    "token3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "token4 = last_sentence[1]\n",
    "token4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "token3.similarity(token4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# The similarity value is not as off as one might think, but it's still less"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Log Probabilities and Brown Cluster IDs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### .prob, .cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for i, token in enumerate(sent):\n",
    "    print(\"original:\", token.orth_)\n",
    "    print(\"log probability:\", token.prob)\n",
    "    print(\"Brown cluster id:\", token.cluster)\n",
    "    print(\"----------------------------------------\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NER"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### .ents, .ent\\_label, .ent\\_type_, .ent\\_iob_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Get a list of the entities directly with .ents\n",
    "analyzed_text.ents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Let's print out all of the tokens in the example text only if they\n",
    "# are entities\n",
    "[print(token.orth_, token.ent_type_)\n",
    " for token in analyzed_text\n",
    " if token.ent_type_ != \"\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Can Handle Messy Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "messy_data = \"lol that is rly funny :) This is gr8 i rate it 8/8!!!\"\n",
    "analyzed_messy_data = nlp_analyzer(messy_data)\n",
    "for token in analyzed_messy_data:\n",
    "    print(token.orth_, token.pos_, token.lemma_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Access to the Vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# The vocabulary that the analyzer uses can be accessed and used (and\n",
    "# also it, along with almost every other component of the system, can\n",
    "# be customized)\n",
    "vocab = nlp_analyzer.vocab\n",
    "vocab.length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# If there's a word that's in the vocabulary, then it can be loaded in and\n",
    "# interacted with\n",
    "vapid = vocab['vapid']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "vapid.similarity(vocab['senseless'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Credit: https://nicschrading.com/project/Intro-to-NLP-with-spaCy/\n",
    "# Let's see if it can figure out this analogy\n",
    "# Man is to King as Woman is to ??\n",
    "from numpy import dot\n",
    "from numpy.linalg import norm\n",
    "\n",
    "# Cosine similarity\n",
    "cosine = lambda v1, v2: dot(v1, v2) / (norm(v1) * norm(v2))\n",
    "\n",
    "king = nlp_analyzer.vocab['king']\n",
    "man = nlp_analyzer.vocab['man']\n",
    "woman = nlp_analyzer.vocab['woman']\n",
    "\n",
    "result = king.repvec - man.repvec + woman.repvec\n",
    "\n",
    "# Gather all known words, take only the lowercased versions\n",
    "all_words = list({w for w in nlp_analyzer.vocab\n",
    "                  if w.has_vector\n",
    "                     and w.orth_.islower()\n",
    "                     and w.lower_ != \"king\"\n",
    "                     and w.lower_ != \"man\"\n",
    "                     and w.lower_ != \"woman\"})\n",
    "\n",
    "# Sort by similarity to the result\n",
    "all_words.sort(key=lambda w: cosine(w.repvec, result))\n",
    "all_words.reverse()\n",
    "print(\"Top 3 closest results for king - man + woman:\\n\")\n",
    "for word in all_words[:3]:   \n",
    "    print(\"\\t{}\".format(word.orth_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Most of the methods/attributes that we've been using can also be used in\n",
    "# \"standalone\" mode and further attributes of the analyzer object can be\n",
    "# specified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nlp_analyzer.like_email(\"mulhodm@gmail.com\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nlp_analyzer.tagger.tag_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sources, Links to Guides"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. [spaCy home page](https://spacy.io/) - [tutorials section](http://spacy.io/docs/#tutorials)\n",
    "## 2. [Nic Schrading's Intro to NLP with spaCy](https://nicschrading.com/project/Intro-to-NLP-with-spaCy/), a fantastic guide (which I stole from a little)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}