Final_Introduction_to_NLP_with_spaCy-Copy1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "32bf606c",
   "metadata": {},
   "source": [
    "# Title    : Introduction to Natural Language Processing with spaCy\n",
    "### Author   : Dr. Nimrita Koul, Associate Professor, Machine Learning, Bangalore, India.\n",
    "\n",
    "#### Github   :  https://github.com/NimritaKoul/IntroductiontoNLPwith_spaCy\n",
    "\n",
    "#### Download This Notebook Here: https://raw.githubusercontent.com/NimritaKoul/IntroductiontoNLPwith_spaCy/main/Final_Introduction_to_NLP_with_spaCy-Copy1.ipynb\n",
    "\n",
    "#### LinkedIn:  https://www.linkedin.com/in/nimritakoul/\n",
    "\n",
    "#### References:\n",
    "1. https://spacy.io/usage/spacy-101\n",
    "2. https://course.spacy.io/en/\n",
    "3. https://github.com/explosion/spacy-course/tree/master/exercises/en\n",
    "3. https://www.machinelearningplus.com/spacy-tutorial-nlp/\n",
    "4. https://blog.dominodatalab.com/natural-language-in-python-using-spacy\n",
    "\n",
    "\n",
    "#### Pre-requisites for this attending this session: \n",
    "\n",
    "1. Intermediate knowledge of Python 3 programming and Jupyter Notebook application.\n",
    "2. Knowledge of libraries - pandas, numpy their installation using pip or conda\n",
    "\n",
    "#### After attending this session, the participants will know the answers to these questions:\n",
    "1. What is Natural Language Processing?\n",
    "2. What is spaCy - Architecture, Features and Functionality?\n",
    "3. How to do NLP tasks in spaCy?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9ed59af",
   "metadata": {},
   "source": [
    "### 1. What is Natural Language Processing?\n",
    "\n",
    "- A Natural Language is a language like English, Hindi, German etc. that is used by \"Humans\" to communicate with one another. \n",
    "\n",
    "\n",
    "- Natural language processing (NLP) is a branch of artificial intelligence that aims at building computer systems that can understand natural language (in text or speech) and respond to in natural language. \n",
    "\n",
    "- To \"understand\" means to recognize the meaning, the intent and the sentiment of the natural language text or speech. \n",
    "\n",
    "- NLP makes use of statistical models, machine learning, deep learning to discover insights from your natural language data. \n",
    "  It helps you answer questions like:\n",
    "\n",
    "        1. What is the central idea of the text?\n",
    "        2. Who is doing what to whom?\n",
    "        3. Is the customer happy or upset about the product or service?\n",
    "        4. How similar are there two pieces of text?\n",
    "\n",
    "- It has applications in automated voice response systems, smart assistants like Alexa, chatbots, machine translation etc like Google Translate etc. \n",
    "\n",
    "####  Understanding natural languages involves tasks like- \n",
    "\n",
    "        a. Identifying nouns, verbs and more,  i.e. parts of speech. \n",
    "        b. Identifying relationships among nouns and verbs. (Who is doing what to whom?)\n",
    "        c. Disambiguating meaning of a word in a context. \n",
    "        d. Identifying real world Entities. (Names of people, countries, companies)\n",
    "        e. Sentiment Analysis. (Is the customer happy or upset about the product?) \n",
    "\n",
    " \n",
    "### 2. [What is spaCy?](https://spacy.io/usage/spacy-101)\n",
    "\n",
    "    - spaCy is a free, open-source, high speed library for advanced Natural Language Processing (NLP) in Python.  \n",
    "    - Designed for building productions level NLP systems for information extraction, text processing and understanding.\n",
    "    - Can handle large volumes of text.\n",
    "    - Supports 66+ languages and provides 76 trained NLP Pipelines/Models for 23 languages.\n",
    "    - Provides pretrained transformers like BERT.\n",
    "    - Extensible with your own components and attributes, supports custom models in frameworks like PyTorch, TensorFlow.\n",
    "    \n",
    "    \n",
    "\n",
    "#### 2a. Important Functionality of spaCy\n",
    "\n",
    "|NAME\t| DESCRIPTION| \n",
    "|:-----|:------------|\n",
    "|Tokenization\t| Segmenting text into words, punctuations marks etc.|\n",
    "|Part-of-speech (POS) Tagging |\tAssigning word types to tokens, like verb or noun.|\n",
    "|Dependency Parsing|\tAssigning syntactic dependency labels, describing the relations between individual tokens, like subject or object.|\n",
    "|Lemmatization|\tAssigning the base forms of words. For example, the lemma of “rats” is “rat”, of \"reading\" is \"read\"|\n",
    "|Sentence Boundary Detection (SBD)|\tFinding and segmenting individual sentences.|\n",
    "|Named Entity Recognition (NER)|\tIdentifying and labelling named “real-world” objects, like persons, companies, locations.|\n",
    "|Entity Linking (EL)|\tDisambiguating textual entities to unique identifiers in a knowledge base.|\n",
    "|Similarity|\tComparing words, text spans and documents and how similar they are to each other.|\n",
    "|Text Classification|\tAssigning categories or labels to a whole document, or parts of a document.|\n",
    "|Rule-based Matching|\tFinding tokens that match a specific lexical pattern or linguistic annotaions.|\n",
    "|Training\t|Updating and improving a statistical model’s predictions.|\n",
    "|Serialization|\tSaving objects to files or byte strings.|\n",
    "\n",
    "\n",
    "#### 2b. [Trained Statistical models in spaCy](https://spacy.io/usage/models)\n",
    "\n",
    "- spaCy provides 76 trained pipelines containing statistical models to predict linguistic annotations of your text in about 23 languages. These linguistic annotations include information like POS tags, dependency, entity names for your text. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebb3447f",
   "metadata": {},
   "source": [
    "## 3. Installing spaCy\n",
    "\n",
    "\n",
    "<b> My preferred way of using Python is its [Anaconda Distribution](https://www.anaconda.com/products/distribution). I prefer to code using Jupyter Notebook. That is what is used for this tutorial.</b>\n",
    "\n",
    "\n",
    "#### 3a. Install spacy using Pip\n",
    "\n",
    "<div class = \"alert alert-block alert-info\">\n",
    "     <b>It is recommended to create a separate virtual environment for installing spaCy:</b><br><br>\n",
    "     <code>python -m venv .env </code>  #create a new virtual env <br>\n",
    "     <code>source .env/bin/activate </code>  #activate new environment <br>\n",
    "</div>\n",
    "\n",
    "\n",
    "#### Install spacy Using pip\n",
    "\n",
    "<code>pip install -U pip setuptools wheel</code> #update pip, setuptools and wheel<br>\n",
    "\n",
    "<code>pip install spacy </code>                  # install spaCy   <br>\n",
    "\n",
    "\n",
    "####  Download and install builtin Trained Pipeline for English Langauge(Small sized pipeline) via spaCy’s download command. \n",
    "It takes care of finding the best-matching package compatible with your spaCy installation.<br>\n",
    "\n",
    "<code>python -m spacy download en_core_web_sm</code><br>\n",
    "\n",
    "\n",
    "#### 3b. Install spacy using conda\n",
    "\n",
    "##### Create a new environment using conda  for spaCy  \n",
    "\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "    <b>  Create a new Python environment with name \"myenv\":</b> <br><br>\n",
    "     <code>conda create --name myenv </code>  #create a new virtual env <br>\n",
    "     <code>conda activate myenv </code>  #activate new environment <br>\n",
    "\n",
    "To relfect new environment in Jupyter Notebook, use following two line of code in this order at Ananconda Prompt:<br>\n",
    "\n",
    "    <code>conda install -c anaconda ipykernel</code><br>\n",
    "    <code>python -m ipykernel install --user --name= myenv</code><br>\n",
    "\n",
    "</div>\n",
    "\n",
    "\n",
    "#### Install spaCy in new environment Using conda : \n",
    "\n",
    "<code>conda install -c conda-forge spacy</code>  #use conda-forge channel<br>\n",
    "\n",
    "Download and install builtin Trained Pipeline for English Langauge (Small sized pipeline)<br>\n",
    "\n",
    "<code>python -m spacy download en_core_web_sm</code>\n",
    "\n",
    "\n",
    "#### 3c. Updating spaCy\n",
    "\n",
    "<code>pip install -U spacy</code><br>\n",
    "\n",
    "##### Once installed, you will need to import spacy in your workspace\n",
    "\n",
    "<code>import spacy</code><br>\n",
    "\n",
    "\n",
    "#### A spaCy trained Pipeline has components which are trained on labelled data to perform following NLP tasks:\n",
    "\n",
    "1. Tokenization<br>\n",
    "2. Parts of speech tagging<br>\n",
    "3. Name Entity Detection<br>\n",
    "4. Dependency parsing<br>\n",
    "5. Matching text with patterns<br>\n",
    "\n",
    "##### Names and sizes of some more trained pipelines\n",
    "\n",
    "**English Langauge Pipelines** <br>\n",
    "\n",
    "**en_core_web_sm**   is a small (12MB) English pipeline optimized for CPU. <br>\n",
    "Its components are: tok2vec, tagger, parser, senter, ner, attribute_ruler, lemmatizer.<br>\n",
    "\n",
    "**en_core_web_md**  - medium sized model (31 MB) <br>\n",
    "**en_core_web_lg**  - large sized model  (382 MB)<br>\n",
    "**en_core_web_trf** - English transformer pipeline (483 MB) (roberta-base). <br>\n",
    "                      Components: transformer, tagger, parser, ner, attribute_ruler, lemmatizer.<br>\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "**German Langauge Pipelines** <br>\n",
    "\n",
    "**de_core_news_sm** - Small German pipeline optimized (13 MB) for CPU. <br>\n",
    "                      Components: tok2vec, tagger, morphologizer, parser, lemmatizer (trainable_lemmatizer), senter,ner.<br>\n",
    "\n",
    "**de_core_news_md**<br>\n",
    "**de_core_news_lg**<br>\n",
    "**de_core_news_trf**<br>\n",
    "\n",
    "Once downloaded, you can load langauage specific model with spacy.load() as shown below:\n",
    "\n",
    "<code>\n",
    "import spacy\n",
    "#Load small english model\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "</code>\n",
    "<br>\n",
    "Here nlp is a Langauge object with a lot of built-in text processing functionality. Also known as a **trained pipeline**.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c40a8bb",
   "metadata": {},
   "source": [
    "### 4. [Pipelines and Linguistic annotations provided by them](https://spacy.io/usage/spacy-101#annotations)\n",
    "\n",
    "\n",
    "#### 4a.  [Pipelines](https://spacy.io/usage/spacy-101#pipelines)\n",
    "\n",
    "- When you run **spacy.load()** with the name of a trained language pipeline, a **Language object is created**. This object is usually named as **nlp**. \n",
    "- When you call the **nlp** object on your text, it does a series of operation on it. The operations depend on the components in the trained pipeline. This series of operations is known as a **processing pipeline**. \n",
    "- Typical components in a trained pipeline are - **Tokenizer, POS tagger, lemmatizer, parser, and Entity Recognizer**. \n",
    "- **Tokenizer** splits your text into tokens and create a **Doc** object. Then, the next component takes this **Doc** as input, performs its operation and passes the processed **Doc** to the next component. \n",
    "\n",
    "<img src = \"https://spacy.io/pipeline-fde48da9b43661abcdf62ab70a546d71.svg\" width = 600>\n",
    "<center><i>Source: https://spacy.io/usage/spacy-101#pipelines </i></center>\n",
    "\n",
    "\n",
    "#### 4b.  Linguistic annotations provided by the trained pipelines\n",
    "\n",
    "- spaCy pipelines predict langauge specific information (linguistic annotations) about your text. All the linguistic annotations are available as Token attributes. \n",
    "- After tokenization, spaCy uses the trained components of pipeline in the nlp object to parse the doc object and predict the tags for the tokens in a Doc object. \n",
    "- spaCy encodes all text strings to **hash values**. This helps reduce memory usage and improves efficiency. So to print or read the string representation of an attribute, add an underscore '_' to its name.  \n",
    "\n",
    "\n",
    "##### 1. Linguistic annotations: Tokenization\n",
    "-  First step in a processing pipeline is the tokenization of text. I.e., splitting larger chunks of text into smaller segments as per language specific rules. E.g., sentences into words and punctuation, paragraphs into sentences. \n",
    "- Tokenization in spaCy can be done even without a trained pipeline. \n",
    "\n",
    "<code>\n",
    "import spacy\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "text = \"Hello, My Name is Nimrita\"\n",
    "doc = nlp(text)\n",
    "for token in doc:\n",
    "    print(token.text)\n",
    "</code><br>\n",
    "\n",
    "##### 2. Linguistic annotations: Part-of-speech tags and dependencies \n",
    "\n",
    "spaCy's POS tagger predicts POS label for each token. To see part of speech tag of a token in a Doc object use **token.pos_**.\n",
    "\n",
    "##### 3. Linguistic annotations: Named Entities \n",
    "\n",
    "- Real world objects like a country, a book title, a person, a company that is assigned a name is known as a **\"Named Entity\"**. \n",
    "- spaCy pipeline's **Named Entity Recognizer** component can predict the Named Entity types for text in a document. \n",
    "- Named entities are available as the **ents** property of a Doc object.\n",
    "\n",
    "##### 4. Linguistic annotations: Word vectors and similarity \n",
    "- Machine learning/deep learning or statistical models can be trained on numerical data. Therefore, the text strings are converted into numerical vectors known as **Word Vectors** or **word embeddings**. These are multi-dimensional representations of the meaning of a word. Algorithms like **word2vec** are used to generate these word vectors. \n",
    "\n",
    "- Small sized trained pipelines for any langauge do not contain word vectors. To use word vectors please download and install medium, large or transformer pipelines. \n",
    "\n",
    "<code>!python -m spacy download en_core_web_lg</code><br>\n",
    "\n",
    "- To see vector for a token use **Token.vector**, you can also see Doc.vector or Span.vector. Doc.vector and Span.vector are the average of their token vectors by default. \n",
    "\n",
    "- spaCy computes Similarity among words by comparing their word vectors. It is a floating point number between 0 and 1. spaCy objects Doc, Span, Token and Lexeme come with a .similarity method."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe1bc53a",
   "metadata": {},
   "source": [
    "## 5. [Architecture of spaCy](https://spacy.io/usage/spacy-101#architecture)\n",
    "\n",
    "spaCy architecture has 3 main data structures: \n",
    "\n",
    "1. Language Class or the nlp Object - the nlp object is created by spacy.load() or spacy.blank(). \n",
    "         It processes the raw text, applies pipeline components to it and returns an annotated Doc object.\n",
    "2. Doc Object - contains and owns the tokens and their annotations.    \n",
    "3. [Vocab](https://spacy.io/usage/spacy-101#vocab) Object \n",
    "      -  The **Vocab** object in spaCy is a shared Vocabulary that stores the data from many documents. It uses a hash function to convert text strings, entity labels, part of speech tags etc. into numerical hash values. For example, the word \"coffee\" has the hash 3197928453018144401. If many documents contain the word \"coffee\", spaCy hashes the word and stores it only once in the **StringStore**. The StringStore (**doc.vocab.strings**) is a lookup table in which you can look up a string to get its hash, or a hash to get its string.\n",
    "     - Each entry in Vocabulary is called a **Lexeme**. Vocabulary may not include word text, and can look up it up in an object called the **StringStore** from its hashvalue.\n",
    "\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<code>\n",
    "import spacy\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\"I love coffee\")\n",
    "print(doc.vocab.strings[\"coffee\"])  # 3197928453018144401\n",
    "print(doc.vocab.strings[3197928453018144401])  # 'coffee'\n",
    "</code>\n",
    "</div>\n",
    "      \n",
    " \n",
    "<img src = \"https://d33wubrfki0l68.cloudfront.net/f35bab3fa9d1e91601695dbd6241a607ba11876c/179f6/architecture-415624fc7d149ec03f2736c4aa8b8f3c.svg\" width = 500/>\n",
    "\n",
    "<center><i>Architecture of spaCy.  Source: https://spacy.io/usage/spacy-101</i></center>\n",
    "\n",
    "\n",
    "### 5a. Other Container objects in spaCy\n",
    "\n",
    "Besides Langauge, Doc and Vocab there are other important objects which contain text or linguistic annotations.<br>\n",
    "\n",
    "1. **Token** individual text entities like words, punctuation, spaces, etc. It contains context specific linguistic attributes. \n",
    "    Some common lexical attributes of Token object are: is_punct, is_ascii, is_digit, is_bracket, is_quote, is_upper, is_lower,      like_url etc.Linguistic annotations in a Token object are token.pos_, token.dep_ etc. <br>\n",
    "2. **Lexeme** is an entry in the vocabulary. It contains only structural information of a word, e.g., spelling, nature of characters in the word etc. i.e. the information that won't change based on context. It is unlike a **Token** which contains context dependent linguistic annotaions.<br>\n",
    "   \n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<code>\n",
    "import spacy\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\"I love coffee\")\n",
    "for word in doc:\n",
    "    lexeme = doc.vocab[word.text]\n",
    "    print(lexeme.text, lexeme.orth, lexeme.shape_, lexeme.prefix_, lexeme.suffix_,\\\n",
    "            lexeme.is_alpha, lexeme.is_digit, lexeme.is_title, lexeme.lang_)\n",
    "</code>\n",
    "\n",
    "Here: <br>\n",
    "Text: The original text of the lexeme. <br>\n",
    "Orth: The hash value of the lexeme. <br>\n",
    "Shape: The abstract word shape of the lexeme. <br>\n",
    "Prefix: By default, the first letter of the word string. <br>\n",
    "Suffix: By default, the last three letters of the word string. <br>\n",
    "is alpha: Does the lexeme consist of alphabetic characters? <br>\n",
    "is digit: Does the lexeme consist of digits? <br>\n",
    "</div>\n",
    "\n",
    "3. **Span** object represents a slice of a **Doc** object.<br>\n",
    "4. **SpanGroup** object is a named collection of spans belonging to a **Doc**.<br>\n",
    "5. **DocBin**  is a collection of **Doc** objects used for training data during training of pipeline components.\n",
    "6. **Example** a collection of training annotations containing two Doc objects: the reference data and the predictions.\n",
    "\n",
    "There are [other classes](https://spacy.io/usage/spacy-101#architecture-other) in spaCy : Corpus Class, KnowledgeBase, Lookups, MorphAnalys, Morphology, Scorer, StringStore, Vectors, Vocab.  \n",
    "\n",
    "\n",
    "### 5b. Processing Pipeline in spaCy\n",
    "\n",
    "- The components of the trained model are applied on the Doc object in a specific order. E.g., Tokenizer is run first of all other components. This series of processing steps forms the processing pipeline. \n",
    "\n",
    "- We can add more built-in components or our own custom components to a Langauge object (nlp) using nlp.add_pipe() method. \n",
    "<img src = \"https://d33wubrfki0l68.cloudfront.net/3ad0582d97663a1272ffc4ccf09f1c5b335b17e9/7f49c/pipeline-fde48da9b43661abcdf62ab70a546d71.svg\" width = 500/>\n",
    "<center><i> Typical Pipeline Components. Source: https://spacy.io/usage/spacy-101 </i></center>\n",
    " \n",
    "There are other built-in spacy components that can be added to a pipeline if your task requires so.\n",
    "\n",
    "### 5c. Matchers\n",
    "Matchers help you find and extract matches between patterns or linguistic attributes and the text in a Doc object.    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b45967cb",
   "metadata": {},
   "source": [
    "### 6. [Serialization - Saving your spaCy projects](https://spacy.io/usage/spacy-101#serialization)\n",
    "\n",
    "You can save your spaCy projects or the customization you do to the spaCy pipeline components by converting them into a byte string. This process is called as serialization. spaCy uses Python's builtin Pickle protocol to serialize your objects and to save them to hard disk or to load them from hard disk. All container classes in spaCy - Language (nlp), Doc, Vocab and StringStore can be serialized."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7705af3f",
   "metadata": {},
   "source": [
    "### 7. [Training (Fine Tuning) the Components of spaCy pipeline](https://spacy.io/usage/spacy-101#training)\n",
    "\n",
    "- Components of spaCy's pipeline - POS tagger, parser, text categorizer etc. are statistical models that have been trained using supervised machine learning (Backpropagation and Gradient descent). The model weights are provided as a part of the pipeline. However, some of these components can be fine tuned with your own data or your own annotations. For this, you can use spacy train() method with configuration settings of your choice. \n",
    "\n",
    "#### Trainable components\n",
    "\n",
    "- **Pipe** class in spaCy allows you to create your own components/models to make predictions of your choice on the Doc objects. These components can be tained using train(). You can use these components as part of the spaCy pipeline. \n",
    "\n",
    "#### Training config and lifecycle\n",
    "\n",
    "- Pipeline contains **Training config file** called as **config.cfg** that include all settings and hyperparameters for training your pipeline. You can call train() method and pass the training configuration by writing it in the config.cfg file. There is no need to provide the training parameters on the command line. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f35d4909",
   "metadata": {},
   "source": [
    "### 8. [Language data](https://spacy.io/usage/spacy-101#language-data)\n",
    "\n",
    "The **lang** module of spaCy contains language specific data that is organized in simple Python files. It includes data like which words are very common, what are the exceptions or special cases of word usage in that language. The root directory contains shared language data that has the rules for basic punctuation , emoji, emoticons, single letter abbreviations that can be shared across languages. Subdirectories/submodules contain language specific rules. \n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<code>\n",
    "from spacy.lang.en import English\n",
    "from spacy.lang.de import German\n",
    "nlp_en = English()  # Includes English data\n",
    "nlp_de = German()  # Includes German data\n",
    "</code>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "231370fc",
   "metadata": {},
   "source": [
    "### Let's get coding\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "789f36a4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.3.1\n"
     ]
    }
   ],
   "source": [
    "# import spacy into your workspace and print the version of spacy you have\n",
    "import spacy\n",
    "print(spacy.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "9c94f8aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello\n",
      ",\n",
      "My\n",
      "Name\n",
      "is\n",
      "Nimrita\n"
     ]
    }
   ],
   "source": [
    "## Load small English pipeline and apply it to your text. Print the tokens of your text\n",
    "#import spacy\n",
    "import spacy\n",
    "\n",
    "#load small English language pipeline\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "#declare your text\n",
    "text = \"Hello, My Name is Nimrita\"\n",
    "\n",
    "#apply the pipeline \"nlp\" to your text\n",
    "doc = nlp(text)  #doc contains the tokens and their annotations\n",
    "\n",
    "#print the text of each token\n",
    "for token in doc:\n",
    "    print(token.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c058208",
   "metadata": {},
   "source": [
    "#### [Lemmatization](https://spacy.io/usage/linguistic-features)\n",
    "\n",
    "Lemmatization is the process of identifying the root word from a group of related words. <br>\n",
    "E.g. 'ate', 'eat', 'eating' all are derived from the word 'eat'. <br>\n",
    "Buy, bought, buying, buyer all are derived from the word 'buy'. <br>\n",
    "This helps in correct interpretation of meaning of the text and reduces the overall amount of text to be processed by NLP models.<br>\n",
    "**Token.lemma_** attribute helps you identigy the lemma of a token in spaCy.<br>\n",
    "\n",
    "\n",
    "#### [Dependency Parsing](https://spacy.io/usage/linguistic-features#dependency-parse)\n",
    "\n",
    "[The Stanford NLP Group](https://nlp.stanford.edu/software/nndep.html) defines the [task of a dependency parser](https://nlp.stanford.edu/software/nndep.html) as follows:<br>\n",
    "\n",
    "A dependency parser analyzes the grammatical structure of a sentence, establishing relationships between \"head\" words and words which modify those heads. In the figure below we see a dependency parse of a short sentence. The arrow from the word \"moving\" to the word \"faster\" indicates that \"faster\" modifies \"moving\", and the label \"advmod\" assigned to the arrow describes the  nature of the dependency as adverb modifier.\n",
    "\n",
    "<img src = \"https://nlp.stanford.edu/static/img/nndep-example.png\" width=\"600\"/>\n",
    "<center><i>Source Credits: https://nlp.stanford.edu/static/img/nndep-example.png</i></center>\n",
    "\n",
    "**token.dep_** attribute can be used to know the dependency relationship of a token to other tokens in spaCy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "7504135a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TokenText: Google    Lemma: Google             POS: PROPN   Dep: compound  Shape: Xxxxx  IsAlphabet: 1  Stopword: 0 \n",
      "TokenText: Inc.      Lemma: Inc.               POS: PROPN   Dep: nsubj     Shape: Xxx.   IsAlphabet: 0  Stopword: 0 \n",
      "TokenText: is        Lemma: be                 POS: AUX     Dep: aux       Shape: xx     IsAlphabet: 1  Stopword: 1 \n",
      "TokenText: buying    Lemma: buy                POS: VERB    Dep: ROOT      Shape: xxxx   IsAlphabet: 1  Stopword: 0 \n",
      "TokenText: a         Lemma: a                  POS: DET     Dep: det       Shape: x      IsAlphabet: 1  Stopword: 1 \n",
      "TokenText: U.K.      Lemma: U.K.               POS: PROPN   Dep: compound  Shape: X.X.   IsAlphabet: 0  Stopword: 0 \n",
      "TokenText: company   Lemma: company            POS: NOUN    Dep: dobj      Shape: xxxx   IsAlphabet: 1  Stopword: 0 \n",
      "TokenText: for       Lemma: for                POS: ADP     Dep: prep      Shape: xxx    IsAlphabet: 1  Stopword: 1 \n",
      "TokenText: $         Lemma: $                  POS: SYM     Dep: quantmod  Shape: $      IsAlphabet: 0  Stopword: 0 \n",
      "TokenText: 110       Lemma: 110                POS: NUM     Dep: compound  Shape: ddd    IsAlphabet: 0  Stopword: 0 \n",
      "TokenText: million   Lemma: million            POS: NUM     Dep: nummod    Shape: xxxx   IsAlphabet: 1  Stopword: 0 \n",
      "TokenText: dollars   Lemma: dollar             POS: NOUN    Dep: pobj      Shape: xxxx   IsAlphabet: 1  Stopword: 0 \n"
     ]
    }
   ],
   "source": [
    "# Print token text, lemma, pos tag, syntactic dependency, shape, and some lexical attributes of tokens \n",
    "import spacy\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\"Google Inc. is buying a U.K. company for $110 million dollars\")\n",
    "for token in doc:\n",
    "    print(f\"TokenText: {token.text:9} Lemma: {token.lemma_:<8} \\\n",
    "          POS: {token.pos_:6}  Dep: {token.dep_:9} Shape: {token.shape_:6} IsAlphabet: {token.is_alpha:<2} Stopword: {token.is_stop:<2}\") \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e420cdd5",
   "metadata": {},
   "source": [
    "#### spacy.explain() to understand the tags you don't follow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "537cc63c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'modifier of quantifier'"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Using spacy.explain() function , you can know the explanation or full-form of any Term that you may not know of.\n",
    "spacy.explain('quantmod')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8adbf6e",
   "metadata": {},
   "source": [
    "#### Print all Named Entities in text\n",
    "Named Entities in text are the words that represent real world entities that have a name. For example, a person, a country, a company. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "49eb472d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Entity Text: Google Inc.             Entity Label: ORG       \n",
      "Entity Text: U.K.                    Entity Label: GPE       \n",
      "Entity Text: $110 million dollars    Entity Label: MONEY     \n"
     ]
    }
   ],
   "source": [
    "# Print all Named Entities in text\n",
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\"Google Inc. is buying a U.K. company for $110 million dollars\")\n",
    "\n",
    "for ent in doc.ents:\n",
    "    print(f\"Entity Text: {ent.text:23} Entity Label: {ent.label_:10}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a08ea2a",
   "metadata": {},
   "source": [
    "### Word Vectors and Similarity \n",
    "Word vectors are numerical representations of the meaning of words. Since statistical models work with numbers so all text needs to be represented in the form of numerical vectors. Common algorithms like Word2Vec or Token2vec generate such word embeddings or vector representations from text. Vectors for Doc are average of the vectors for vectors of its tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "e9757578",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TokenText: dog      HasVector:  1 L2 Norm:  7.443447113037109 OOV: False\n",
      "Token Vector \n",
      ": [-0.72483    0.42538    0.025489  -0.39807    0.037463  -0.29811\n",
      " -0.28279    0.29333    0.57775    1.2205    -0.27903    0.80879\n",
      " -0.71291    0.045808  -0.46751    0.55944    0.42745    0.58238\n",
      "  0.20854   -0.42718   -0.40284   -0.048941   0.1149    -0.6963\n",
      " -0.03338    0.052596  -0.22572   -0.35996    0.47961   -0.38386\n",
      " -0.73837    0.1718     0.52188    0.45584   -0.026621   0.48831\n",
      "  0.67996   -0.73345   -0.27078    0.41739    0.1947     0.27389\n",
      " -0.70931   -0.45317   -0.22574   -0.12617    0.03268    0.142\n",
      "  0.53923   -0.61285   -0.5322     0.19479    0.13889   -0.020284\n",
      "  0.088162   0.85337    0.039407   0.11529   -0.42646    0.74832\n",
      "  0.34421   -0.59462    0.0040537  0.027203  -0.063394   0.26538\n",
      "  0.34757    0.21395   -0.39799   -0.027067  -0.36132    0.31979\n",
      "  0.55813   -0.5652     0.55382    0.03928   -0.26933   -0.14705\n",
      "  0.74032   -0.50566    0.023765   0.62273   -0.79388   -0.25165\n",
      "  0.11992   -0.43056    1.0614     0.58571    0.8856    -0.056054\n",
      "  0.055826   0.30485    0.64639   -0.43831   -0.45706    0.036471\n",
      " -0.3466    -0.56219    0.28105   -0.33758   -0.041398   0.22171\n",
      "  0.05262    0.18113    0.65646   -0.56217    0.038915  -0.30335\n",
      "  0.05051   -0.2354     0.3233     0.31744    0.52453   -0.47154\n",
      "  0.13152   -0.15104    0.14265   -0.20747    0.060413  -0.030342\n",
      " -0.092883   0.80421   -0.12497   -0.56199    0.29128   -0.22488\n",
      "  0.30282   -0.0045144 -0.12305    0.20396   -0.32202   -0.11409\n",
      " -0.37613    0.40457    0.21461    0.25741   -0.36489    0.94135\n",
      "  0.42725    0.022925  -1.8699    -0.76035    0.73771    0.36998\n",
      "  0.50214   -0.30617   -0.26526    0.86573    0.3808     0.14754\n",
      "  0.29932   -0.078863  -0.28992   -0.064636  -0.68914    0.19527\n",
      " -0.56368    0.26251   -0.52171   -1.0703     0.42478   -0.0067289\n",
      " -0.28591   -0.77831    0.049342   0.66675   -0.077419  -0.19226\n",
      "  0.12721   -0.18844    0.13647    0.38804    0.21917   -0.24192\n",
      " -0.13465    0.23119   -0.43197    0.48302    0.3598     1.128\n",
      "  0.019894  -0.10861   -0.13515   -0.34137   -0.36379    0.080616\n",
      "  0.28682   -0.045819  -0.12114   -0.44835   -0.054611  -0.10362\n",
      "  0.010954  -0.60063   -0.46665    0.15115   -0.31815   -0.58903\n",
      "  1.1325     0.04406   -0.92863    0.3399    -0.03463   -0.40474\n",
      "  0.17245   -0.19983   -0.095982  -0.074758   0.57472    0.25455\n",
      " -0.20387    0.055758  -0.65017    0.72629   -0.51083    0.11196\n",
      "  0.44724    0.16157   -0.34571    0.19227   -0.063871   0.0057351\n",
      "  0.48703   -0.53762   -0.73398   -0.11488    0.073723   0.58191\n",
      "  0.33192   -0.13303   -0.3478    -0.022676  -0.32494   -0.26496\n",
      "  0.56275    0.098558  -0.16671   -0.40481    0.55477   -0.58692\n",
      " -0.60433   -0.4227    -0.53712    0.2994     0.11339   -0.3154\n",
      " -0.28685    0.43999    0.013623   0.011139  -0.47734   -0.01492\n",
      "  0.52524    0.53583    0.36626    0.23119   -0.1386     0.35374\n",
      " -0.27448    0.066183   0.6224    -0.24851   -0.36066    0.009084\n",
      " -0.58148    0.24371    0.29944   -0.025314  -0.73222    0.33236\n",
      " -0.40339    0.82624    0.006984   0.26737   -0.27695   -0.09713\n",
      " -0.015736   0.1024    -0.026831  -0.26293    0.31401    0.01051\n",
      " -0.048451  -0.74571    0.75827    0.67771    0.054738  -0.23325\n",
      "  0.17996   -0.206      0.019095  -0.34283   -0.58602    0.0095634\n",
      " -0.085052   0.83312    0.31978    0.050317  -0.23159    0.28165  ]\n",
      "TokenText: cat      HasVector:  1 L2 Norm:  7.443447113037109 OOV: False\n",
      "Token Vector \n",
      ": [-0.72483    0.42538    0.025489  -0.39807    0.037463  -0.29811\n",
      " -0.28279    0.29333    0.57775    1.2205    -0.27903    0.80879\n",
      " -0.71291    0.045808  -0.46751    0.55944    0.42745    0.58238\n",
      "  0.20854   -0.42718   -0.40284   -0.048941   0.1149    -0.6963\n",
      " -0.03338    0.052596  -0.22572   -0.35996    0.47961   -0.38386\n",
      " -0.73837    0.1718     0.52188    0.45584   -0.026621   0.48831\n",
      "  0.67996   -0.73345   -0.27078    0.41739    0.1947     0.27389\n",
      " -0.70931   -0.45317   -0.22574   -0.12617    0.03268    0.142\n",
      "  0.53923   -0.61285   -0.5322     0.19479    0.13889   -0.020284\n",
      "  0.088162   0.85337    0.039407   0.11529   -0.42646    0.74832\n",
      "  0.34421   -0.59462    0.0040537  0.027203  -0.063394   0.26538\n",
      "  0.34757    0.21395   -0.39799   -0.027067  -0.36132    0.31979\n",
      "  0.55813   -0.5652     0.55382    0.03928   -0.26933   -0.14705\n",
      "  0.74032   -0.50566    0.023765   0.62273   -0.79388   -0.25165\n",
      "  0.11992   -0.43056    1.0614     0.58571    0.8856    -0.056054\n",
      "  0.055826   0.30485    0.64639   -0.43831   -0.45706    0.036471\n",
      " -0.3466    -0.56219    0.28105   -0.33758   -0.041398   0.22171\n",
      "  0.05262    0.18113    0.65646   -0.56217    0.038915  -0.30335\n",
      "  0.05051   -0.2354     0.3233     0.31744    0.52453   -0.47154\n",
      "  0.13152   -0.15104    0.14265   -0.20747    0.060413  -0.030342\n",
      " -0.092883   0.80421   -0.12497   -0.56199    0.29128   -0.22488\n",
      "  0.30282   -0.0045144 -0.12305    0.20396   -0.32202   -0.11409\n",
      " -0.37613    0.40457    0.21461    0.25741   -0.36489    0.94135\n",
      "  0.42725    0.022925  -1.8699    -0.76035    0.73771    0.36998\n",
      "  0.50214   -0.30617   -0.26526    0.86573    0.3808     0.14754\n",
      "  0.29932   -0.078863  -0.28992   -0.064636  -0.68914    0.19527\n",
      " -0.56368    0.26251   -0.52171   -1.0703     0.42478   -0.0067289\n",
      " -0.28591   -0.77831    0.049342   0.66675   -0.077419  -0.19226\n",
      "  0.12721   -0.18844    0.13647    0.38804    0.21917   -0.24192\n",
      " -0.13465    0.23119   -0.43197    0.48302    0.3598     1.128\n",
      "  0.019894  -0.10861   -0.13515   -0.34137   -0.36379    0.080616\n",
      "  0.28682   -0.045819  -0.12114   -0.44835   -0.054611  -0.10362\n",
      "  0.010954  -0.60063   -0.46665    0.15115   -0.31815   -0.58903\n",
      "  1.1325     0.04406   -0.92863    0.3399    -0.03463   -0.40474\n",
      "  0.17245   -0.19983   -0.095982  -0.074758   0.57472    0.25455\n",
      " -0.20387    0.055758  -0.65017    0.72629   -0.51083    0.11196\n",
      "  0.44724    0.16157   -0.34571    0.19227   -0.063871   0.0057351\n",
      "  0.48703   -0.53762   -0.73398   -0.11488    0.073723   0.58191\n",
      "  0.33192   -0.13303   -0.3478    -0.022676  -0.32494   -0.26496\n",
      "  0.56275    0.098558  -0.16671   -0.40481    0.55477   -0.58692\n",
      " -0.60433   -0.4227    -0.53712    0.2994     0.11339   -0.3154\n",
      " -0.28685    0.43999    0.013623   0.011139  -0.47734   -0.01492\n",
      "  0.52524    0.53583    0.36626    0.23119   -0.1386     0.35374\n",
      " -0.27448    0.066183   0.6224    -0.24851   -0.36066    0.009084\n",
      " -0.58148    0.24371    0.29944   -0.025314  -0.73222    0.33236\n",
      " -0.40339    0.82624    0.006984   0.26737   -0.27695   -0.09713\n",
      " -0.015736   0.1024    -0.026831  -0.26293    0.31401    0.01051\n",
      " -0.048451  -0.74571    0.75827    0.67771    0.054738  -0.23325\n",
      "  0.17996   -0.206      0.019095  -0.34283   -0.58602    0.0095634\n",
      " -0.085052   0.83312    0.31978    0.050317  -0.23159    0.28165  ]\n",
      "TokenText: banana   HasVector:  1 L2 Norm:   6.89589786529541 OOV: False\n",
      "Token Vector \n",
      ": [-0.6334     0.18981   -0.53544   -0.52658   -0.30001    0.30559\n",
      " -0.49303    0.14636    0.012273   0.96802    0.0040354  0.25234\n",
      " -0.29864   -0.014646  -0.24905   -0.67125   -0.053366   0.59426\n",
      " -0.068034   0.10315    0.66759    0.024617  -0.37548    0.52557\n",
      "  0.054449  -0.36748   -0.28013    0.090898  -0.025687  -0.5947\n",
      " -0.24269    0.28603    0.686      0.29737    0.30422    0.69032\n",
      "  0.042784   0.023701  -0.57165    0.70581   -0.20813   -0.03204\n",
      " -0.12494   -0.42933    0.31271    0.30352    0.09421   -0.15493\n",
      "  0.071356   0.15022   -0.41792    0.066394  -0.034546  -0.45772\n",
      "  0.57177   -0.82755   -0.27885    0.71801   -0.12425    0.18551\n",
      "  0.41342   -0.53997    0.55864   -0.015805  -0.1074    -0.29981\n",
      " -0.17271    0.27066    0.043996   0.60107   -0.353      0.6831\n",
      "  0.20703    0.12068    0.24852   -0.15605    0.25812    0.007004\n",
      " -0.10741   -0.097053   0.085628   0.096307   0.20857   -0.23338\n",
      " -0.077905  -0.030906   1.0494     0.55368   -0.10703    0.052234\n",
      "  0.43407   -0.13926    0.38115    0.021104  -0.40922    0.35972\n",
      " -0.28898    0.30618    0.060807  -0.023517   0.58193   -0.3098\n",
      "  0.21013   -0.15557   -0.56913   -1.1364     0.36598   -0.032666\n",
      "  1.1926     0.12825   -0.090486  -0.47965   -0.61164   -0.16484\n",
      " -0.41134    0.19925    0.059183  -0.20842    0.45223    0.27697\n",
      " -0.20745    0.025404  -0.28874    0.040478  -0.22275   -0.43323\n",
      "  0.76957   -0.054327  -0.35213   -0.30842   -0.48791   -0.35564\n",
      "  0.19813   -0.094767  -0.50918    0.18763   -0.087555   0.37709\n",
      " -0.1322    -0.096913  -1.9102     0.55813    0.27391   -0.077744\n",
      " -0.43933   -0.10367   -0.24408    0.41869    0.11659    0.27454\n",
      "  0.81021   -0.11006    0.43131    0.29095   -0.49548   -0.31958\n",
      " -0.072506   0.020286   0.2179     0.22032   -0.29212    0.75639\n",
      "  0.13598    0.019736  -0.83104    0.22836   -0.28669   -1.0529\n",
      "  0.052771   0.41266    0.50149    0.5323     0.51573   -0.31806\n",
      " -0.4619     0.21739   -0.43584   -0.41382    0.042237  -0.57179\n",
      "  0.067623  -0.27854    0.090044   0.20633    0.024678  -0.57703\n",
      " -0.020183  -0.53147   -0.37548   -0.12795   -0.093662  -0.0061183\n",
      "  0.20221   -0.62296   -0.29746    0.26935    0.59009   -0.50382\n",
      " -0.69757    0.20157   -0.33592   -0.45766    0.14061    0.22982\n",
      "  0.044046   0.26386    0.02942    0.34095    1.1496    -0.15555\n",
      " -0.064071   0.30139    0.024211  -0.63515   -0.73347   -0.10346\n",
      " -0.22637   -0.056392  -0.16735   -0.097331  -0.19206   -0.18866\n",
      "  0.15116   -0.038048   0.70205    0.11586   -0.14813    0.0095166\n",
      " -0.33804   -0.10158   -0.23829   -0.22759    0.092504  -0.29839\n",
      " -0.39721    0.26092    0.34594   -0.47396   -0.25725   -0.19257\n",
      " -0.53071    0.1692    -0.47252   -0.17333   -0.40505    0.046446\n",
      " -0.04473    0.33555   -0.5693     0.31591   -0.21167   -0.31298\n",
      " -0.45923   -0.083091   0.086822   0.01264    0.43779    0.12651\n",
      "  0.30156    0.022061   0.26549   -0.29455   -0.14838    0.033692\n",
      " -0.37346   -0.075343  -0.56498   -0.24207   -0.69351   -0.20277\n",
      " -0.0081185  0.030971   0.53615   -0.16613   -0.84087    0.74661\n",
      "  0.029132   0.46936   -0.49755    0.40954   -0.022558   0.21497\n",
      " -0.049528  -0.039799   0.46165    0.26456    0.32985   -0.04219\n",
      " -0.099599  -0.17312   -0.476     -0.019048  -0.41888   -0.2685\n",
      " -0.65281    0.068773  -0.23881   -1.1784     0.25504    0.61171  ]\n",
      "TokenText: afskfsd  HasVector:  0 L2 Norm:                0.0 OOV: True\n",
      "Token Vector \n",
      ": [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "\n",
      "\n",
      "Similarity of doc1 with doc2: 0.7071206680444951 \n",
      "\n",
      "\n",
      "Similarity of doc1 with doc3: 0.3090511739579275 \n",
      "\n",
      "\n",
      "Similarity of doc1 with doc4: 0.9685581106439412 \n"
     ]
    }
   ],
   "source": [
    "## Print word vectors and similarity between words\n",
    "# Small pipelines dont contain word vectors, so we have downloaded and installed medium size English pipeline for this code\n",
    "# Similarity is obtained using the .similarity() method that can be applied on Doc, Span, View objects.\n",
    "\n",
    "import spacy\n",
    "nlp = spacy.load(\"en_core_web_md\")\n",
    "\n",
    "text1 = \"dog cat banana afskfsd\"\n",
    "doc1 = nlp(text1)\n",
    "for token in doc1:\n",
    "    print(f\"TokenText: {token.text:8} HasVector: {token.has_vector:2} L2 Norm: {token.vector_norm:18} OOV: {token.is_oov}\")\n",
    "    print(f\"Token Vector \\n: {token.vector}\")\n",
    "\n",
    "    \n",
    "text2 = \"i have a dog and a cat\"\n",
    "doc2 = nlp(text2)\n",
    "\n",
    "text3 = \"i love rain\"\n",
    "doc3 = nlp(text3)\n",
    "\n",
    "text4 = \"cat banana\"\n",
    "doc4 = nlp(text4)\n",
    "\n",
    "print(f'\\n\\nSimilarity of doc1 with doc2: {doc1.similarity(doc2)} ')\n",
    "\n",
    "print(f'\\n\\nSimilarity of doc1 with doc3: {doc1.similarity(doc3)} ')\n",
    "\n",
    "print(f'\\n\\nSimilarity of doc1 with doc4: {doc1.similarity(doc4)} ')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12443c9f",
   "metadata": {},
   "source": [
    "**'dog' and 'cat'** are common English words, hence these words and their vectors are included in pipeline's vocabulary. But **\"afskfsd\"** is not common and hence **OOV : out-of-vocabulary**, so its vector representation consists of 300 dimensions of 0, which means that this is not a valid existing word in the English langauge. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67844fc1",
   "metadata": {},
   "source": [
    "#### Text-Cleaning / Preprocessing with spaCy\n",
    "- Text cleaning or preprocessing involves removing the parts of text that do not add value to the meaning to the text. These are the stopwords, punctuation or spaces.  \n",
    "- **Stopwords** are the words like “a”, ” the”, “was”, \"it\" that occur very frequently in English lanaguage text and  do not add value to its meaning. \n",
    "- spaCy tokens have attributes that tell you if a word is noise. E.g., token.is_stop, token.is_punct, token.is_space "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71c1797e",
   "metadata": {},
   "source": [
    "##### Let us remove all words like etc. i.e. (tokens with POS == X) , stopwords and punctuation from input text.\n",
    "spaCy uses POS tag == 'X' for words like etc, i.e. indicating that they are slang/junk words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "412f5341",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original Doc: He 26 books. $ 10.5 plus 10% tax etc etc. \n",
      "Number of tokens before preprocessing:  13\n",
      "\n",
      "Now preprocessing to remove words with POS == X, stopwords, and punctuation...\n",
      "Cleaned Doc: [26, books, $, 10.5, plus, 10, tax]\n",
      "\n",
      "No. of tokens after removing stopwords and punctuation:  7\n",
      "\n",
      "Tokens in cleaned doc: \n",
      "Token Text:26          POS: NUM     \n",
      "Token Text:books       POS: NOUN    \n",
      "Token Text:$           POS: SYM     \n",
      "Token Text:10.5        POS: NUM     \n",
      "Token Text:plus        POS: CCONJ   \n",
      "Token Text:10          POS: NUM     \n",
      "Token Text:tax         POS: NOUN    \n"
     ]
    }
   ],
   "source": [
    "# Identifying all POS tags for tokens in a piece of text.\n",
    "import spacy\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "text= \"\"\"He 26 books. $ 10.5 plus 10% tax etc etc. \"\"\"\n",
    "\n",
    "# Pass the Text to Model\n",
    "doc = nlp(text)\n",
    "\n",
    "\n",
    "print(\"Original Doc: {}\".format(doc))\n",
    "print('Number of tokens before preprocessing: ', len(doc))\n",
    "\n",
    "print(\"\\nNow preprocessing to remove words with POS == X, stopwords, and punctuation...\")\n",
    "\n",
    "cleaned_doc = [token for token in doc if not token.pos_=='X' and not token.is_stop and not token.is_punct]\n",
    "\n",
    "print(\"Cleaned Doc: {}\".format(cleaned_doc))\n",
    "print('\\nNo. of tokens after removing stopwords and punctuation: ', len(cleaned_doc))\n",
    "\n",
    "print(\"\\nTokens in cleaned doc: \")\n",
    "for token in cleaned_doc:\n",
    "    print(f'Token Text:{token.text:10}  POS: {token.pos_:8}')\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94128971",
   "metadata": {},
   "source": [
    "#### Printing the tokens which are like numbers, percentage numbers, email addresses or URLs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "1ae2d4b9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A number: 2020\n",
      "A number: 10\n",
      "A percentage number: 10\n",
      "An email address:  jhucasualties@ada.com\n",
      "A URL:  www.jhu.org\n"
     ]
    }
   ],
   "source": [
    "#Printing the tokens which are like numbers or email addresses\n",
    "\n",
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "text= \"\"\"Covid19 made 2020 a catastropic year, 10% people who got infected died. \n",
    "       All casualties were reported to jhucasualties@ada.com. www.jhu.org has the details.\"\"\"\n",
    "\n",
    "doc=nlp(text)\n",
    "\n",
    "for token in doc:\n",
    "    if token.like_num:\n",
    "        print(\"A number:\", token)\n",
    "        index_of_next_token=token.i + 1\n",
    "        next_token = doc[index_of_next_token]\n",
    "        if next_token.text == '%':\n",
    "            print(\"A percentage number:\", token.text)\n",
    "    if token.like_email:\n",
    "        print(\"An email address: \", token.text) \n",
    "    if token.like_url:\n",
    "        print(\"A URL: \", token.text) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9753e2e",
   "metadata": {},
   "source": [
    "#### [Named Entity Recognition](https://spacy.io/usage/linguistic-features#named-entities-101)\n",
    "\n",
    "spaCy's trained language pipeline models can predict various types of named entities in a document.\n",
    "all the named entities are available in **doc.ents** property. \n",
    "\n",
    "The common Named Entity categories supported by spacy are :<br>\n",
    "\n",
    "PERSON : Names of people<br>\n",
    "GPE : Geo physical entity - like counties, cities, states<br>\n",
    "ORG : Organizations or companies<br>\n",
    "WORK_OF_ART : Titles of books, fimls,songs and other arts<br>\n",
    "PRODUCT : Products such as vehicles, food items ,furniture and so on<br>\n",
    "EVENT : Historical events like wars, disasters ,etc <br>\n",
    "LANGUAGE : All the recognized languages across the globe<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "f33a2d4f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('CARDINAL',\n",
       " 'DATE',\n",
       " 'EVENT',\n",
       " 'FAC',\n",
       " 'GPE',\n",
       " 'LANGUAGE',\n",
       " 'LAW',\n",
       " 'LOC',\n",
       " 'MONEY',\n",
       " 'NORP',\n",
       " 'ORDINAL',\n",
       " 'ORG',\n",
       " 'PERCENT',\n",
       " 'PERSON',\n",
       " 'PRODUCT',\n",
       " 'QUANTITY',\n",
       " 'TIME',\n",
       " 'WORK_OF_ART')"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### See all entity types in  specific langauge model\n",
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "nlp.get_pipe(\"ner\").labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "id": "60698c4c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All entities in this doc (Google, Samsung, Microsoft, John, Walmart, Mandarin, Arabic)\n",
      "\n",
      "All the companies: \n",
      "Google\n",
      "Samsung\n",
      "Microsoft\n",
      "Walmart\n",
      "\n",
      "\n",
      "All the people: \n",
      "John\n",
      "\n",
      "\n",
      "All the languages: \n",
      "Mandarin\n",
      "Arabic\n"
     ]
    }
   ],
   "source": [
    "# Identofy all the entities that are ORG in given text\n",
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "text = \"\"\" This is an important industrial region of the country. \n",
    "           It houses the offices of Google, Samsung, Microsoft etc.\n",
    "           Mr. John works in Walmart, he speaks Mandarin and Arabic.\"\"\"\n",
    "\n",
    "# creating spacy doc\n",
    "doc = nlp(text)\n",
    "\n",
    "# Printing the named entities\n",
    "print('All entities in this doc', doc.ents)\n",
    "\n",
    "\n",
    "print(\"\\nAll the companies: \")\n",
    "for entity in doc.ents:\n",
    "    if entity.label_=='ORG':\n",
    "        print(entity.text)\n",
    "\n",
    "        \n",
    "print(\"\\n\\nAll the people: \")        \n",
    "for entity in doc.ents:\n",
    "    if entity.label_== 'PERSON':\n",
    "        print(entity.text)\n",
    "        \n",
    "print(\"\\n\\nAll the languages: \")        \n",
    "for entity in doc.ents:\n",
    "    if entity.label_== 'LANGUAGE':\n",
    "        print(entity.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52bb47af",
   "metadata": {},
   "source": [
    "#### [Masking certain entity types and tokens representing personal information for privacy]\n",
    "In the code below, we mask the token that have entity type PERSON , ORG, GPE , PRODUCT or EVENT, and tokens that are numbers or email addresses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "56797606",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original Doc:\n",
      " [Hi, my name is Dr. John. I live in Italy. I work at Google. My phone number is 123456. My email is abcd@abcd.com]\n",
      "\n",
      "All entities in this doc (John, Italy, Google, 123456)\n",
      "\n",
      "Identified named entities: \n",
      "Entity: John       Entity Label: 380 \n",
      "Entity: Italy      Entity Label: 384 \n",
      "Entity: Google     Entity Label: 383 \n",
      "Entity: 123456     Entity Label: 391 \n",
      "\n",
      "Masked Personal Identification Information\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Hi , my name is Dr. UNKNOWN . I live in UNKNOWN . I work at UNKNOWN . My phone number is UNKNOWN . My email is UNKNOWN'"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Inspired by: https://www.machinelearningplus.com/spacy-tutorial-nlp/#application2automaticallymaskingentities\n",
    "\n",
    "import spacy\n",
    "\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "\n",
    "text = \"\"\"Hi, my name is Dr. John. I live in Italy. I work at Google. My phone number is 123456. My email is abcd@abcd.com\"\"\"\n",
    "\n",
    "# creating spacy doc\n",
    "doc = nlp(text)\n",
    "\n",
    "print(\"Original Doc:\\n [{}]\".format(doc))\n",
    "\n",
    "print('\\nAll entities in this doc', doc.ents)\n",
    "\n",
    "print(\"\\nIdentified named entities: \")\n",
    "for ent in doc.ents:\n",
    "    print(\"Entity: {:<10} Entity Label: {:<4}\".format(ent.text, ent.label))\n",
    "\n",
    "def find_personaldata(token):\n",
    "    if token.ent_type_ == 'PERSON' or token.ent_type_== 'ORG' or token.ent_type_== 'GPE' or token.like_num or token.like_email:\n",
    "        return 'UNKNOWN'\n",
    "    return token.text\n",
    "\n",
    "def mask_personaldata(doc):\n",
    "    with doc.retokenize() as retokenizer:\n",
    "        for ent in doc.ents:\n",
    "            retokenizer.merge(ent)\n",
    "    tokens = map(find_personaldata, doc)\n",
    "    return ' '.join(tokens)\n",
    "\n",
    "print('\\nMasked Personal Identification Information')\n",
    "mask_personaldata(doc)    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afa72530",
   "metadata": {},
   "source": [
    "####  Visualizers - [displaCy and displaCy ENT](https://spacy.io/usage/visualizers)\n",
    "\n",
    "- You can easily visualize dependencies identified by a langauge models in your text using two spaCy visualizers - [displacy](https://explosion.ai/demos/displacy) and displaCy Named Entity Visualizer [displaCy ENT](https://explosion.ai/demos/displacy-ent). Both are part of the core spaCy library. \n",
    "- displaCy can take a single Doc or a list of Doc objects as its first argument, it can return the markup that can be rendered in jupyter notebook or exported. \n",
    "- displaCy.serve creates a simple web server to see your visualization in a browser. \n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "c7c69173",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"6e091df82c2a466c8d306bdb85cc9c22-0\" class=\"displacy\" width=\"750\" height=\"312.0\" direction=\"ltr\" style=\"max-width: none; height: 312.0px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">I</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">PRON</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">love</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">to</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">PART</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"575\">read</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"575\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-6e091df82c2a466c8d306bdb85cc9c22-0-0\" stroke-width=\"2px\" d=\"M70,177.0 C70,89.5 220.0,89.5 220.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-6e091df82c2a466c8d306bdb85cc9c22-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M70,179.0 L62,167.0 78,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-6e091df82c2a466c8d306bdb85cc9c22-0-1\" stroke-width=\"2px\" d=\"M420,177.0 C420,89.5 570.0,89.5 570.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-6e091df82c2a466c8d306bdb85cc9c22-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">aux</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M420,179.0 L412,167.0 428,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-6e091df82c2a466c8d306bdb85cc9c22-0-2\" stroke-width=\"2px\" d=\"M245,177.0 C245,2.0 575.0,2.0 575.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-6e091df82c2a466c8d306bdb85cc9c22-0-2\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">xcomp</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M575.0,179.0 L583.0,167.0 567.0,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Displaying dependency relations between tokens\n",
    "import spacy\n",
    "from spacy import displacy\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "text = 'I love to read'\n",
    "doc = nlp(text)\n",
    "displacy.render(doc,style='dep',jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7653f4a",
   "metadata": {},
   "source": [
    "#### displacy.serve visualizing Entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "bee4df31",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><!DOCTYPE html>\n",
       "<html lang=\"en\">\n",
       "    <head>\n",
       "        <title>displaCy</title>\n",
       "    </head>\n",
       "\n",
       "    <body style=\"font-size: 16px; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; padding: 4rem 2rem; direction: ltr\">\n",
       "<figure style=\"margin-bottom: 6rem\">\n",
       "<div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Sebastian Thrun\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " started working at \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Google Inc.\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " in \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    2007\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       ". He spoke \n",
       "<mark class=\"entity\" style=\"background: #ff8197; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Mandarin\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">LANGUAGE</span>\n",
       "</mark>\n",
       "</div>\n",
       "</figure>\n",
       "</body>\n",
       "</html></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Using the 'ent' visualizer\n",
      "Serving on http://0.0.0.0:5000 ...\n",
      "\n",
      "Shutting down server on port 5000.\n"
     ]
    }
   ],
   "source": [
    "# Visualizing named entities\n",
    "import spacy\n",
    "from spacy import displacy\n",
    "\n",
    "text = \"\"\"Sebastian Thrun started working at Google Inc. in 2007. He spoke Mandarin\"\"\"\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(text)\n",
    "displacy.serve(doc, style=\"ent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4678e6f7",
   "metadata": {},
   "source": [
    "## [Training the Named Entity Recognizer Component to  Recognize  \"phone\" as Entity](https://course.spacy.io/en/chapter4)\n",
    "[Next we will train the Named Entity Recognizer Component of the pipeline for a new entity called \"iphone\"](https://course.spacy.io/en/chapter4)\n",
    "\n",
    "\n",
    "\n",
    "#### Input File: iphone.json\n",
    "\n",
    "#### Download from here:  https://raw.githubusercontent.com/explosion/spacy-course/master/exercises/en/iphone.json\n",
    "\n",
    "##### Content of the file \"iphone.json\" :\n",
    "<br>\n",
    "[<br>\n",
    "  \"How to preorder the iPhone X\",<br>\n",
    "  \"iPhone X is coming\",<br>\n",
    "  \"Should I pay $1,000 for the iPhone X?\",<br>\n",
    "  \"The iPhone 8 reviews are here\",<br>\n",
    "  \"iPhone 11 vs iPhone 8: What's the difference?\",<br>\n",
    "  \"I need a new phone! Any tips?\"<br>\n",
    "]<br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "id": "0736da0c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[iPhone X]\n",
      "[iPhone X]\n",
      "[iPhone X]\n",
      "[iPhone 8]\n",
      "[iPhone 11, iPhone 8]\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "#Source : https://course.spacy.io/en/chapter4\n",
    "# https://github.com/explosion/spacy-course/tree/master/exercises/en\n",
    "\n",
    "#Import required libraries\n",
    "import json\n",
    "import spacy\n",
    "from spacy.matcher import Matcher\n",
    "from spacy.tokens import Span\n",
    "from spacy.tokens import DocBin\n",
    "\n",
    "# You can access the files at https://raw.githubusercontent.com/explosion/spacy-course/master/exercises/en/iphone.json\n",
    "#open and read the file contents in json format\n",
    "with open(\"iphone.json\", encoding=\"utf8\") as f:\n",
    "    TEXTS = json.loads(f.read())\n",
    "\n",
    "# use blank model, it has only Tokenizer, no other component\n",
    "nlp = spacy.blank(\"en\")\n",
    "\n",
    "# create a Matcher object\n",
    "matcher = Matcher(nlp.vocab)\n",
    "\n",
    "# Create two patters that catch the word sequence \"iphone x\" and \"iphone\" followed by any digit\n",
    "\n",
    "# Two tokens whose lowercase forms match \"iphone\" and \"x\"\n",
    "pattern1 = [{\"LOWER\": \"iphone\"}, {\"LOWER\": \"x\"}]\n",
    "\n",
    "# Token whose lowercase form matches \"iphone\" and a digit\n",
    "pattern2 = [{\"LOWER\": \"iphone\"}, {\"IS_DIGIT\": True}]\n",
    "\n",
    "# Add patterns to the matcher and create docs with matched entities\n",
    "matcher.add(\"GADGET\", [pattern1, pattern2])\n",
    "\n",
    "## initialise list of docs\n",
    "docs = []\n",
    "\n",
    "#apply npl to TEXTS read from json file\n",
    "# apply matcher to identify sequences that satisfy the patterns\n",
    "for doc in nlp.pipe(TEXTS):\n",
    "    matches = matcher(doc)\n",
    "    #create spans of matches in the doc\n",
    "    spans = [Span(doc, start, end, label=match_id) for match_id, start, end in matches]\n",
    "    print(spans)\n",
    "    #store entities\n",
    "    doc.ents = spans\n",
    "    docs.append(doc)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "id": "f013f5a9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "How to preorder the iPhone X\n",
      "iPhone X is coming\n",
      "Should I pay $1,000 for the iPhone X?\n",
      "The iPhone 8 reviews are here\n",
      "iPhone 11 vs iPhone 8: What's the difference?\n",
      "I need a new phone! Any tips?\n"
     ]
    }
   ],
   "source": [
    "for doc in nlp.pipe(TEXTS):\n",
    "    print(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "4836bde3",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Split data i.e. docs list into training docs and development docs (for evaluation)\n",
    "import random\n",
    "random.shuffle(docs)\n",
    "train_docs = docs[:len(docs) // 2]\n",
    "dev_docs = docs[len(docs) // 2:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "id": "5bfd544e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Instantiate the DocBin with the list of docs in train_docs.\n",
    "#Save the DocBin to a disk file called train.spacy.\n",
    "# Create and save a collection of training docs\n",
    "train_docbin = DocBin(docs=train_docs)\n",
    "train_docbin.to_disk(\"C:/spacyfiles/train.spacy\")\n",
    "\n",
    "# Create and save a DocBin collection of evaluation docs\n",
    "dev_docbin = DocBin(docs=dev_docs)\n",
    "dev_docbin.to_disk(\"C:/spacyfiles/dev.spacy\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcab768c",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "After this we need to generate a config.cfg file with spacy init config command. <br>\n",
    "For this, use this line of code at your Anaconda prompt. <br>\n",
    "Issue the command in the same environment where you have installed spacy. <br>\n",
    "    \n",
    "<code>python -m spacy init config ./config.cfg --lang en --pipeline ner </code><br>\n",
    "    \n",
    "init config: the command to run<br>\n",
    "config.cfg: output path for the generated config<br>\n",
    "--lang: language class of the pipeline, e.g. en for English<br>\n",
    "--pipeline: comma-separated names of components to include<br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63b628c6",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "Next we train the model by running following command at Anaconda prompt in the environment in which you have installed spacy\n",
    "<br>\n",
    "\n",
    "<code>python -m spacy train config.cfg --output C:/spacymodeloutput  --paths.train  C:/spacyfiles/train.spacy  --paths.dev C:/spacyfiles/dev.spacy </code><br>\n",
    "\n",
    "      \n",
    "train: the command to run<br>\n",
    "config.cfg: the path to the config file<br>\n",
    "--output: the path to the output directory to save the trained pipeline<br>\n",
    "--paths.train: override with path to the training data, provide the path of train.spacy<br>\n",
    "--paths.dev: override with path to the evaluation data, provide the path of dev.spacy<br>\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f154304",
   "metadata": {},
   "source": [
    "## During training\n",
    "\n",
    "<img src = \"https://raw.githubusercontent.com/NimritaKoul/IntroductiontoNLPwith_spaCy/main/TrainingNERwithSpacy.jpg\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e5a363d",
   "metadata": {},
   "source": [
    "#### Loading a trained pipeline\n",
    "Output after training is a regular loadable spaCy pipeline saved in the folder you specified during training<br>\n",
    "model-last: last trained pipeline<br>\n",
    "model-best: best trained pipeline<br>\n",
    "you can load it with spacy.load()<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "id": "a5adfe09",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(iPhone 11, iPhone 8)\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "nlp = spacy.load(\"C:/spacymodeloutput/model-best\")\n",
    "doc = nlp(\"iPhone 11 is better than iPhone 8\")\n",
    "print(doc.ents)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6dc0188",
   "metadata": {},
   "source": [
    "#### So we had a quick tour of the functionality provided by spaCy. You can explore further and apply spaCy to build interesting projects like your own funny chatbot, or recognizing interesting Entities in text. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1cc9884",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "myenv",
   "language": "python",
   "name": "myenv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}