22Sept_NLPusingPython_DrNimritaKoul.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "422059f1",
   "metadata": {},
   "source": [
    "<img src = \"GHCFirstImage.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f498e058",
   "metadata": {},
   "source": [
    "## Natural Language Processing using Python\n",
    "\n",
    "### Author: Dr. Nimrita Koul, Associate Professor, Machine Learning, Bangalore, India\n",
    "\n",
    "\n",
    "### Let us connect: https://www.linkedin.com/in/nimritakoul/\n",
    "\n",
    "### GitHub URL of this Jupyter Notebook: https://tinyurl.com/NLPNimritaKoul1\n",
    "\n",
    "### Pre-requisites:   \n",
    "\n",
    "    1. Familiarity with Python Programming including libraries - Pandas, NumPy.          \n",
    "    2. Basic understanding about handling data \n",
    "           \n",
    "### Agenda: \n",
    "\n",
    "    1. Introduction to Natural Language Processing (NLP)          \n",
    "    2. Representing Text as Numerical Vectors           \n",
    "    3. NLTK Library - Installation, functionality \n",
    "    4. SpaCy Library - Installation, functionality        \n",
    "    5. Transformer Architecture and Large Langauge Models\n",
    "    6. Hugging Face Transformers Library\n",
    "    \n",
    "\n",
    "### Setup Required:\n",
    "\n",
    "I am using Jupyter Notebook in Anaconda Python with version 3.9.12 for the examples in this session. \n",
    "- Anaconda Python 3.9.12\n",
    "- Jupyter Notebook\n",
    "\n",
    "Libraries used:\n",
    "- numpy\n",
    "- pandas\n",
    "- sklearn\n",
    "- re\n",
    "- nltk\n",
    "- spacy\n",
    "- gensim\n",
    "- transformer-embeddings\n",
    "- transformers\n",
    "- sentence-transformers\n",
    "- torch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c72da9b6",
   "metadata": {},
   "source": [
    "# Introduction to Natural Langauge Processing (NLP)\n",
    "\n",
    "- Natural Langauge Processing is a field of linguistics and machine learning that aims at enabling computers to understand and generate human languages. \n",
    "\n",
    "### Teaching computers to understand human languages is very hard. \n",
    "\n",
    "- There are so many langauges spoken all over the world\n",
    "- Human langauges are ambiguous\n",
    "- Words may have multiple meanings based on context\n",
    "- There may be similar sounding and spelling words with different meanings\n",
    "- Often humans use slang or abbreviations \n",
    "- Tone or emotion may change meaning of a sentence\n",
    "- Human languages have a complex grammar \n",
    "- New words keep adding to human languages every now and then. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46354218",
   "metadata": {},
   "source": [
    "<img src = \"textambiguity2.jpg\" width = 500 height = 400>\n",
    "\n",
    "<center>(Image Source:https://64.media.tumblr.com/fb44df360acecb19785e4fa917a2cda8/tumblr_pg3j3pISoR1rwewyjo1_1280.jpg)<center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01408ada",
   "metadata": {},
   "source": [
    "### Yet we have come a long way in NLP (all the way to BARD and ChatGPT!!)\n",
    "\n",
    "- We have made enormous advances in the area of NLP. Now, we use text generators like BARD, GPT-3.5 and GPT-4, that produce almost human like text, we have text to image generation models, audio translation and generation models, we have models that understand DNA sequences and generate meaningful outputs and so on.\n",
    "\n",
    "<img src = \"chatgptbard.png\">\n",
    "<center> (Image Source: https://www.forbes.com/sites/ariannajohnson/2023/03/21/bard-vs-chatgpt-the-major-difference-between-the-ai-chat-tools-explained/?sh=67087aeb684a)</center>\n",
    "\n",
    "We will come back to these large language models later in the session."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb23ea43",
   "metadata": {},
   "source": [
    "## Some NLP terminology:\n",
    "\n",
    "1. Document: A document is a collection of many words. \n",
    "2. Vocabulary: Vocabulary is the set of unique words in a document.\n",
    "3. Token: A token is a basic unit of discrete data. It often refers to a single word or a punctuation mark.\n",
    "\n",
    "<img src = 'tokenization_basic.png'>\n",
    "<center>(Image Source: https://vaclavkosar.com/ml/Tokenization-in-Machine-Learning-Explained)</center>\n",
    "\n",
    "4. Corpus: A corpus is a collection of documents.\n",
    "5. Context : Context of a word/token is the words/tokens that surround it on left and right in the document.\n",
    "6. Vector Embedding: A real-valued vector based representations of text are called as Embedding. E.g., word2vec or GLoVE are unsupervised methods based on corpus statistics. Frameworks like tensorflow and keras support “Embedding layers”. \n",
    "\n",
    "### Text data is sequential\n",
    "- Text data is sequential, i.e, the meaning of a word may depend on words preceeding it.\n",
    "\n",
    "## Text Preprocessing\n",
    "\n",
    "- Before we can apply machine learning models to text data, we need to preprocess it and convert it into a numerical representation. \n",
    "\n",
    "- Text preprocessing involves a set of steps and operations to clean, transform and structure the text to enable ML models to work better with it.\n",
    "\n",
    "### Important steps in text pre-processing are:\n",
    "    1. Text Cleaning: removing special characters, Unicode characters, inconsistent spaces, correcting spellings etc, handling missing data. \n",
    "\n",
    "    2. Tokenization: Splitting long text into constituent sentences and words or sub-words.\n",
    "\n",
    "    3. Normalization: Removing Punctuation, Lowercasing, Stemming, Lemmatization, Stop word removal, text length normalization, Removing Special Characters or numbers, Expanding Contractions: Like \"can't\" to \"cannot\" or \"I'm\" to \"I am.\", Handling URLS, email addresses, having a consistent text encoding (e.g. UTF-8), having consistent spacing throughout.\n",
    "\n",
    "\n",
    "## Word and Sentence Embeddings: Converting text to numerical representation\n",
    "\n",
    "- For machine learning algorithms to work with text, we must convert words and sentences (text) into numerical representations. \n",
    "\n",
    "- An embedding is a real-valued vector that represents a word or a sentence of text. An embedding can have hundreds of dimensions. \n",
    "\n",
    "- A good embedding must capture semantics of language and have these properties:\n",
    "\n",
    "    1. Similar words and words with similar meanings should be close by in numerical vector space. \n",
    "    \n",
    "    2. We often combine words and sentences to form complex concepts, a good embedding should be able to reflect this combination of words/sentences into corresponding numerical vectors for concepts. \n",
    "\n",
    "#### Semantic Similarity:  Similarity among the vectors representing text can be computed by vector operations like dot product or cosine similarity score. Higher the score, higher is the relationship or similarity between corresponding pieces of text. \n",
    "\n",
    "### Some methods for embedding text into numerical vectors\n",
    "\n",
    "#### 1.  Bag of words Technique (Does not convey information about order of structure of words in the document):\n",
    "\n",
    "BoW technique involves following steps:\n",
    "\n",
    "1. Tokenize the text into sentences and further into words.\n",
    "\n",
    "2. Preprocess the text to remove stopwords, punctuations, rare or unusual words, correct spellings etc.\n",
    "\n",
    "3. Create a vocabulary of all words in text.\n",
    "\n",
    "4. Assign a number to each word in vocabulary, this is called a vocabulary dictionary where each word is the key and the number assigned to it is the value.\n",
    "\n",
    "<img src = 'bow_visual.png'>\n",
    "<center>(Source: https://diveki.github.io/projects/wine/tfidf.html)</center>\n",
    "\n",
    "\n",
    "#### 2. TFI-DF\n",
    "\n",
    "Term Frequency-Inverse Document Frequency (TF-IDF) representation has two components: TF and IDF. \n",
    "\n",
    "Term Frequency (TF) captures the frequency of occurence of a term/word in a particular document. It is given by: \n",
    "\n",
    "$$TF_{i, j} = \\frac{Freq-of-word-i-in-document-j}{Total-number-of-words-in-document-j} $$\n",
    "\n",
    "The inverse document frequency or the IDF: is a measure of rarity of a word. IDF is higher if word is present in fewer documents in the corpus.\n",
    "\n",
    "$$IDF_{i} = \\log(\\frac{Total-number-of-document-in-corpus}{Number-of-documents-containing-word-i})$$\n",
    "\n",
    "TF-IDF(i, j) is the product of TF(i,j) and IDF(i):\n",
    "\n",
    "$$TF-IDF_{(i,j)} = TF_{i,j} * IDF_{i}$$\n",
    "\n",
    "##### In a tabular dataset, each row represents a document, and each column represents the vocabulary. Then we can compute the TF-IDF values for each term in each cell of the dataset. This tfidf representation of each row can then be used to train an ML model.\n",
    "\n",
    "#### Word2Vec: Word to vector\n",
    "\n",
    "Word2Vec model uses a 2-layered neural network to predict the words in neighborhood of a given word. Word2vec takes as its input a large corpus of text and produces a vector space of hundreds of dimensions. Each unique word in corpus is assigned a unique vector. Gensim library provides implementation of Word2Vec model.\n",
    "\n",
    "#### GloVe: Global vectors for word representation (https://nlp.stanford.edu/projects/glove/)\n",
    "\n",
    "For each pair of words in the training corpus, GloVe first creates a co-occurrence matrix. Then it uses unsupervised learning to train from this co-occurence matrix to learn vector representations. The optiman vectors are such as they maximize the sum of similarities between vectors of words that appear together frequently. \n",
    "\n",
    "#### FastText:\n",
    "\n",
    "FastText extends Word2Vec approach by also leanring embeddings for subwords and out-of vocabulary words by using character n-grams. A character n-gram is a sequence of 'n' characters. For example, the character n-grams for the word \"cat\" are \"c\", \"ca\", \"cat\", and \"at\".The vector representation of a word is learned by predicting the surrounding words in a sentence.\n",
    "\n",
    "#### ELMo: Embeddings from Language Models (https://paperswithcode.com/method/elmo)\n",
    "\n",
    "- This technique uses a bi-directional language model (biLM) which is a neural network trained on large corpus of text to learn contextualized embeddings for words. This bi-directional model predicts the next word in a sentence given the previous words. \n",
    "\n",
    "#### Transformer Embeddings : BERT and more.\n",
    "\n",
    "- Transformer embeddings are learned using a transformer neural networks from large corpus of text. The transformer model is trained to predict the next word in a sentence, given the previous words. The hidden states of the transformer model at each word position are then used as the transformer embeddings.\n",
    "\n",
    "\n",
    "## Prominent NLP Tasks\n",
    "\n",
    "Natural language tasks are the specific linguistic or computational operations that are performed on text data.These tasks are building blocks to creat NLP applications for the real world problems:\n",
    "\n",
    "- Text Classification\n",
    "\n",
    "<img src = \"https://developers.google.com/static/machine-learning/guides/text-classification/images/TextClassificationExample.png\">\n",
    "<center> (Source: https://developers.google.com/machine-learning/guides/text-classification\") </center>\n",
    "\n",
    "- Named Entity Recognition (NER)\n",
    "<img src = \"https://production-media.paperswithcode.com/thumbnails/task/task-0000000008-1eee4fae_Kb2RSZl.jpg\">\n",
    "<center> (Source:https://production-media.paperswithcode.com/thumbnails/task/task-0000000008-1eee4fae_Kb2RSZl.jpg) </center>\n",
    "\n",
    "- Part-of-Speech Tagging\n",
    "<img src = \"taggingpos.png\">\n",
    "<center> (Source:/https://www.cs.virginia.edu/~hw5x/Course/TextMining-2019Spring/_site/docs/PDFs/POSTagging-SequenceLabeling.pdf) </center>\n",
    "\n",
    "- Text Generation\n",
    "<img src = \"textgeneration.png\" width = 800 >\n",
    "<center> (Source:ChatGPT) </center>\n",
    "\n",
    "- Machine Translation\n",
    "<img src = \"machinetranslation.png\" width = 800 height = 600>\n",
    "<center> (Source:Google Translate) </center>\n",
    "\n",
    "- Sentiment Analysis\n",
    "<img src = \"https://d33wubrfki0l68.cloudfront.net/9e1b2a906ae6b01cfe2d5d237e1e51f5d41864e3/2a5f9/static/348bb1d70089176ca2f61ea402094382/50bf7/main.png\" width = 500 height = 500>\n",
    "<center> (Source:https://monkeylearn.com/sentiment-analysis/) </center>\n",
    "\n",
    "- Text Summarization\n",
    "\n",
    "<img src = \"textsummarization.png\" width = 800>\n",
    "<center> (Source:Google BARD) </center>\n",
    "\n",
    "- Question Answering\n",
    "\n",
    "<img src = \"questionanswering.png\" width = 800>\n",
    "<center> (Source:Google BARD) </center>\n",
    "\n",
    "\n",
    "- Information Extraction\n",
    "<img src = \"informationextraction.png\" width = 800>\n",
    "<center> (Source:Google BARD)</center>\n",
    "<center> (Text Source:https://www.businessinsider.in/thelife/15-frugal-billionaires-who-live-like-regular-people/slidelist/21446560.cms#slideid=21446568)</center>\n",
    "\n",
    "## Realworld Usecases for NLP\n",
    "\n",
    "- Chatbots and Virtual Assistants: These applications use NLP tasks like text classification, named entity recognition, and sentiment analysis to enable human-like interactions in natural language.\n",
    "\n",
    "\n",
    "- Search Engines: Search engines utilize tokenization, information retrieval, and text classification to deliver relevant search results in response to user queries.\n",
    "\n",
    "\n",
    "- Social Media Monitoring: Businesses, governments and even us often wish to track our brand perception, mentions, sentiment of our followers or general population on social media. \n",
    "\n",
    "\n",
    "- Document Management: NLP tasks like text summarization, named entity recognition, document categorization can be used to manage documentation and extract important insights from it. \n",
    "\n",
    "\n",
    "- Language Translation Services: Machine translation services like Goole Translate use NLP tasks like tokenization, intent classification , alignment to translate text from one langauge to another.\n",
    "\n",
    "\n",
    "- Content Generation: Natural Language Generation involves tasks like langauge modelling, summarization, translation etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c057c07f",
   "metadata": {},
   "source": [
    "## This Jupyter Notebook use Python 3.9.12 on Windows 11 Desktop Machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "fa98c8bc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.9.12\n"
     ]
    }
   ],
   "source": [
    "# Check your python version\n",
    "from platform import python_version\n",
    "print(python_version())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0a5d205",
   "metadata": {},
   "source": [
    "## Code walk through for Text to Numerical Vector Representation\n",
    "\n",
    "### BOW Representation using CountVectorizer in sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0d53d8bf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary dictionary(word and a number):\n",
      " {'love': 6, 'mangoes': 7, 'where': 8, 'do': 0, 'you': 10, 'live': 5, 'it': 4, 'is': 3, 'winter': 9, 'in': 2, 'here': 1}\n",
      "\n",
      "Sorted word list in vocabulary\n",
      " ['do', 'here', 'in', 'is', 'it', 'live', 'love', 'mangoes', 'where', 'winter', 'you']\n",
      "\n",
      " Number of documents and maximum number of words in each document: (3, 11)\n",
      "\n",
      "\n",
      "Document and its representation\n",
      " ['I love mangoes', 'where do you live', 'it is winter in here'] : \n",
      " [[0 0 0 0 0 0 1 1 0 0 0]\n",
      " [1 0 0 0 0 1 0 0 1 0 1]\n",
      " [0 1 1 1 1 0 0 0 0 1 0]]\n",
      "\n",
      "\n",
      " ['winter live love'] : [[0 0 0 0 0 1 1 0 0 1 0]]\n",
      "\n",
      "\n",
      " ['I am happy'] : [[0 0 0 0 0 0 0 0 0 0 0]]\n"
     ]
    }
   ],
   "source": [
    "#BoW representation in sklearn\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "# each item/sentence in the list is a separate document.\n",
    "text = [\"I love mangoes\", \"where do you live\", \"it is winter in here\"] \n",
    "\n",
    "# create an object of CountVectorizer\n",
    "vectorizer = CountVectorizer()\n",
    "\n",
    "# Fit method tokenizes the text and builds a vocabulary dicttionary, \n",
    "# and computes document frequency of each word in vocabulary - number of documents that contain the word\n",
    "vectorizer.fit(text)\n",
    "\n",
    "print(\"Vocabulary dictionary(word and a number):\\n\", vectorizer.vocabulary_)\n",
    "\n",
    "print(\"\\nSorted word list in vocabulary\\n\", sorted(vectorizer.vocabulary_))\n",
    "\n",
    "# Convert each sentence in text to a numeric representation based on vocabulary dictionary and document frequency\n",
    "# first tokenise the text, then look up each word in vocaculary dictionary adn get corresponding feature index\n",
    "# then create a vector that contains doc frequency of the words in the document.\n",
    "vector = vectorizer.transform(text)\n",
    "\n",
    "# Number of documents and number of words in each document\n",
    "print(\"\\n Number of documents and maximum number of words in each document:\", vector.shape)\n",
    "\n",
    "#matrix in which each row represents a document, each column contains a 1 if corresponding word is present \n",
    "# in that document else it contains a 0.\n",
    "\n",
    "print(\"\\n\\nDocument and its BOW representation\\n\", text,\":\",\"\\n\", vector.toarray())\n",
    "\n",
    "# Some new document\n",
    "new_text = [\"winter live love\"]\n",
    "vector = vectorizer.transform(new_text)\n",
    "print(\"\\n\\n\",new_text,\":\", vector.toarray())\n",
    "\n",
    "# Some new document\n",
    "new_text = [\"I am happy\"]\n",
    "vector = vectorizer.transform(new_text)\n",
    "print(\"\\n\\n\",new_text,\":\", vector.toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34b65be6",
   "metadata": {},
   "source": [
    "## Using Gensim library's Word2Vec model for converting text to numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "dfc0f76b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\lib\\site-packages\\paramiko\\transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated\n",
      "  \"class\": algorithms.Blowfish,\n"
     ]
    }
   ],
   "source": [
    "# https://radimrehurek.com/gensim/models/word2vec.html\n",
    "import gensim\n",
    "from gensim.models import Word2Vec\n",
    "import nltk\n",
    "import re\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "X =[\"Camera is fantastic in 2020\",\"This phone is worst\",\"Don't buy it\",\"Too pricey\"]\n",
    "\n",
    "corpus = []\n",
    "# Preprocessing text using re library to remove non-alphbets, converting to lower case and tokenization.\n",
    "for i in range(0,len(X)):\n",
    "    tweet = re.sub(\"[^a-zA-Z]\",\" \",X[i])\n",
    "    tweet = tweet.lower()\n",
    "    tweet = tweet.split()\n",
    "    corpus.append(tweet)      "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2046d9b2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['camera', 'is', 'fantastic', 'in'],\n",
       " ['this', 'phone', 'is', 'worst'],\n",
       " ['don', 't', 'buy', 'it'],\n",
       " ['too', 'pricey']]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "40249956",
   "metadata": {},
   "outputs": [],
   "source": [
    "# COnverting to Word2Vec embeddings\n",
    "w2vmodel = gensim.models.Word2Vec(corpus,  window=1,  min_count=0)\n",
    "#w2vmodel.save((\"myword2vecmodel.model\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "754a6afc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-1.9442177e-03, -5.2675223e-03,  9.4471117e-03, -9.2987325e-03,\n",
       "        4.5039463e-03,  5.4041767e-03, -1.4092636e-03,  9.0070916e-03,\n",
       "        9.8853586e-03, -5.4750443e-03, -6.0210014e-03, -6.7469738e-03,\n",
       "       -7.8948829e-03, -3.0479168e-03, -5.5940272e-03, -8.3446810e-03,\n",
       "        7.8290224e-04,  2.9946566e-03,  6.4147427e-03, -2.6289511e-03,\n",
       "       -4.4534779e-03,  1.2495709e-03,  3.9146186e-04,  8.1169987e-03,\n",
       "        1.8280029e-04,  7.2315861e-03, -8.2645155e-03,  8.4335348e-03,\n",
       "       -1.8889094e-03,  8.7011531e-03, -7.6168370e-03,  1.7963862e-03,\n",
       "        1.0564852e-03,  4.6005251e-05, -5.1032542e-03, -9.2476988e-03,\n",
       "       -7.2642183e-03, -7.9511739e-03,  1.9137263e-03,  4.7846555e-04,\n",
       "       -1.8131376e-03,  7.1201660e-03, -2.4756931e-03, -1.3473105e-03,\n",
       "       -8.9005642e-03, -9.9254129e-03,  8.9493962e-03, -5.7539390e-03,\n",
       "       -6.3729975e-03,  5.1994063e-03,  6.6699935e-03, -6.8316413e-03,\n",
       "        9.5975876e-04, -6.0084746e-03,  1.6473436e-03, -4.2892788e-03,\n",
       "       -3.4407973e-03,  2.1856665e-03,  8.6615775e-03,  6.7281104e-03,\n",
       "       -9.6770572e-03, -5.6221057e-03,  7.8803329e-03,  1.9893574e-03,\n",
       "       -4.2560529e-03,  5.9881213e-04,  9.5209600e-03, -1.1027169e-03,\n",
       "       -9.4246389e-03,  1.6084099e-03,  6.2323548e-03,  6.2823701e-03,\n",
       "        4.0916489e-03, -5.6502391e-03, -3.7069322e-04, -5.5317880e-05,\n",
       "        4.5717955e-03, -8.0415895e-03, -8.0183102e-03,  2.6474951e-04,\n",
       "       -8.6083002e-03,  5.8201551e-03, -4.1781188e-04,  9.9711772e-03,\n",
       "       -5.3439783e-03, -4.8614026e-04,  7.7567720e-03, -4.0679337e-03,\n",
       "       -5.0159004e-03,  1.5900708e-03,  2.6506924e-03, -2.5649595e-03,\n",
       "        6.4475276e-03, -7.6599526e-03,  3.3935595e-03,  4.8996927e-04,\n",
       "        8.7321829e-03,  5.9827138e-03,  6.8153618e-03,  7.8225443e-03],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vector = w2vmodel.wv['camera']  # Get numpy vector of a word\n",
    "vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1a66ff4f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.09310115"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2vmodel.wv.similarity(\"camera\", \"is\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efe86dfb",
   "metadata": {},
   "source": [
    "# NLTK Library\n",
    "NLTK is a Python library for working with human langauge data. NLTK has over 50 builtin corpora and lexical resources like WordNet. It has builtin functionality for NLP tasks like tokenization, stemming, POS tagging, text classification, parsing, and semantic reasoning. \n",
    "\n",
    "#### Installation (https://www.nltk.org/install.html)\n",
    "NLTK requires Python versions 3.7, 3.8, 3.9, 3.10 or 3.11\n",
    "<code>\n",
    "pip install nltk\n",
    "</code>\n",
    "\n",
    "or\n",
    "\n",
    "<code>\n",
    "conda install nltk\n",
    "</code>\n",
    "\n",
    "#### Downloading NLTK datasets (https://www.nltk.org/data.html)\n",
    "After installing NLTK you can download datasets which used with many tasks in NLTK library.\n",
    "\n",
    "##### if you wish to interactively pick the datasets you want to download.\n",
    "<code>nltk.download()</code>\n",
    "\n",
    "or if you wish to download just the popular datasets.\n",
    "\n",
    "<code>nltk.download('popular')</code>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce77372c",
   "metadata": {},
   "source": [
    "## NLTK Builtin Resources: Corpora, Wordnet etc.\n",
    "\n",
    "- NLTK provides many built in text corpora, with and without annotations. \n",
    "\n",
    "- Some of the well known text corpora are : Gutenberg Corpus, Web and Chat Text, Brown Corpus, Reuters Corpus, Inaugural Address Cropus. \n",
    "\n",
    "- Annotated text corpora include: Brown Corpus, CESS Treebank, CMU  Pronouncing Dictionary etc. \n",
    "\n",
    "- NLTK also has corpora in other langauges e.g, Universal Declaration of Human Rights in 300 languages.\n",
    "\n",
    "#### Lexical Resources\n",
    "Wordlist corpora, CMU pronouncing dictionary, Swadesh wordlists (common words in many langauges including German, Spanish, French etc.)\n",
    "\n",
    "#### WordNet: \n",
    "WordNet is a thesaurus containing lists of synonym sets and hypernyms(\"is a\" relationships). It is a semantically oriented dictionary of English containining 155287 words and 117659 synonym sets (synsets) with a rich structure. \n",
    "Wordnet allows you to compute semantic similarity between synonym sets.\n",
    "\n",
    "## Code demo with NLTK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72eff51b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Wordnet is a thesaraus of English langauge built into NLTK.\n",
    "#nltk.download('wordnet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c59174f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#nltk.download('stopwords')  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "bf062f04",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentences: ['Hello, Welcome to the world of Natural Langauge Processing.', 'Happy Learning!', '!']\n"
     ]
    }
   ],
   "source": [
    "# Tokenization of text using NLTK\n",
    "import nltk\n",
    "text= \"Hello, Welcome to the world of Natural Langauge Processing. Happy Learning!!\"\n",
    "\n",
    "#sentence tokenization\n",
    "sentences= nltk.sent_tokenize(text)\n",
    "print('Sentences:', sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "5840d4a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokens: ['Hello', ',', 'Welcome', 'to', 'the', 'world', 'of', 'Natural', 'Langauge', 'Processing', '.', 'Happy', 'Learning', '!', '!']\n"
     ]
    }
   ],
   "source": [
    "#word tokenization\n",
    "tokens = nltk.word_tokenize(text)\n",
    "print('Tokens:', tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "e00b511d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokens with POS Tags: [('Hello', 'NNP'), (',', ','), ('Welcome', 'NNP'), ('to', 'TO'), ('the', 'DT'), ('world', 'NN'), ('of', 'IN'), ('Natural', 'NNP'), ('Langauge', 'NNP'), ('Processing', 'NNP'), ('.', '.'), ('Happy', 'JJ'), ('Learning', 'VBG'), ('!', '.'), ('!', '.')]\n"
     ]
    }
   ],
   "source": [
    "#POS tagging\n",
    "tagged = nltk.pos_tag(tokens)\n",
    "print(\"Tokens with POS Tags:\", tagged)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "7e41bc2b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stemming: \n",
      "hello\n",
      ",\n",
      "welcom\n",
      "to\n",
      "the\n",
      "world\n",
      "of\n",
      "natur\n",
      "langaug\n",
      "process\n",
      ".\n",
      "happi\n",
      "learn\n",
      "!\n",
      "!\n"
     ]
    }
   ],
   "source": [
    "#Stemming\n",
    "print(\"Stemming: \")\n",
    "from nltk.stem import PorterStemmer\n",
    "stemmer = PorterStemmer()          \n",
    "for token in tokens:\n",
    "    root_word = stemmer.stem(token)\n",
    "    print(root_word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "ca18c68d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Lemmatization:\n",
      "Lemma of Hello is: Hello\n",
      "Lemma of , is: ,\n",
      "Lemma of Welcome is: Welcome\n",
      "Lemma of to is: to\n",
      "Lemma of the is: the\n",
      "Lemma of world is: world\n",
      "Lemma of of is: of\n",
      "Lemma of Natural is: Natural\n",
      "Lemma of Langauge is: Langauge\n",
      "Lemma of Processing is: Processing\n",
      "Lemma of . is: .\n",
      "Lemma of Happy is: Happy\n",
      "Lemma of Learning is: Learning\n",
      "Lemma of ! is: !\n",
      "Lemma of ! is: !\n"
     ]
    }
   ],
   "source": [
    "#Lemmatization    \n",
    "print('Lemmatization:')\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "lem = WordNetLemmatizer()\n",
    "\n",
    "for token in tokens:\n",
    "    root_word = lem.lemmatize(token)\n",
    "    print(\"Lemma of {} is: {}\".format(token, root_word))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "def0dfd6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Now removing stopwords\n",
      "Stopwords in this text: ['to', 'the', 'of']\n",
      "Text without stopwords: ['Hello', ',', 'Welcome', 'world', 'Natural', 'Langauge', 'Processing', '.', 'Happy', 'Learning', '!', '!']\n"
     ]
    }
   ],
   "source": [
    "#Removing stopwords\n",
    "print(\"Now removing stopwords\")\n",
    "from nltk.corpus import stopwords\n",
    "stopwords_in_text = []\n",
    "text_without_stopwords = []\n",
    "\n",
    "for token in tokens:\n",
    "    if token not in stopwords.words('english'):\n",
    "        text_without_stopwords.append(token)\n",
    "    else:\n",
    "        stopwords_in_text.append(token)\n",
    "\n",
    "print(\"Stopwords in this text:\" , stopwords_in_text)\n",
    "print(\"Text without stopwords:\" , text_without_stopwords)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b58377a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Named Entity Recognition:\n",
      "Entities in the text: (S\n",
      "  (GPE Hello/NNP)\n",
      "  ,/,\n",
      "  (PERSON Welcome/NNP)\n",
      "  to/TO\n",
      "  the/DT\n",
      "  world/NN\n",
      "  of/IN\n",
      "  (ORGANIZATION Natural/NNP Langauge/NNP)\n",
      "  Processing/NNP\n",
      "  ./.\n",
      "  Happy/JJ\n",
      "  Learning/VBG\n",
      "  !/.\n",
      "  !/.)\n"
     ]
    }
   ],
   "source": [
    "# Named Entity Recognition\n",
    "print(\"Named Entity Recognition:\")\n",
    "# Named Entity Recognition\n",
    "entities = nltk.chunk.ne_chunk(tagged)\n",
    "print(\"Entities in the text:\", entities)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5116eb3b",
   "metadata": {},
   "source": [
    "## End to End mini-project walkthrough: text cleaning and classification on Corona NLP Tweets dataset \n",
    "#### Dataset URL: Corona Virus NLP Tweets data set from https://www.kaggle.com/datasets/datatattle/covid-19-nlp-text-classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "77e42b74",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserName</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>Location</th>\n",
       "      <th>TweetAt</th>\n",
       "      <th>OriginalTweet</th>\n",
       "      <th>Sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3799</td>\n",
       "      <td>48751</td>\n",
       "      <td>London</td>\n",
       "      <td>16-03-2020</td>\n",
       "      <td>@MeNyrbie @Phil_Gahan @Chrisitv https://t.co/i...</td>\n",
       "      <td>Neutral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3800</td>\n",
       "      <td>48752</td>\n",
       "      <td>UK</td>\n",
       "      <td>16-03-2020</td>\n",
       "      <td>advice Talk to your neighbours family to excha...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3801</td>\n",
       "      <td>48753</td>\n",
       "      <td>Vagabonds</td>\n",
       "      <td>16-03-2020</td>\n",
       "      <td>Coronavirus Australia: Woolworths to give elde...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3802</td>\n",
       "      <td>48754</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16-03-2020</td>\n",
       "      <td>My food stock is not the only one which is emp...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3803</td>\n",
       "      <td>48755</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16-03-2020</td>\n",
       "      <td>Me, ready to go at supermarket during the #COV...</td>\n",
       "      <td>Extremely Negative</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserName  ScreenName   Location     TweetAt  \\\n",
       "0      3799       48751     London  16-03-2020   \n",
       "1      3800       48752         UK  16-03-2020   \n",
       "2      3801       48753  Vagabonds  16-03-2020   \n",
       "3      3802       48754        NaN  16-03-2020   \n",
       "4      3803       48755        NaN  16-03-2020   \n",
       "\n",
       "                                       OriginalTweet           Sentiment  \n",
       "0  @MeNyrbie @Phil_Gahan @Chrisitv https://t.co/i...             Neutral  \n",
       "1  advice Talk to your neighbours family to excha...            Positive  \n",
       "2  Coronavirus Australia: Woolworths to give elde...            Positive  \n",
       "3  My food stock is not the only one which is emp...            Positive  \n",
       "4  Me, ready to go at supermarket during the #COV...  Extremely Negative  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv('Corona_NLP_train.csv',  encoding='latin-1') #using encoding = \"latin-1\" as the dataset has some non-ASCII characters\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "3251b21b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(41157, 6)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "3c88af29",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['UserName', 'ScreenName', 'Location', 'TweetAt', 'OriginalTweet',\n",
       "       'Sentiment'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8ed8b881",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['OriginalTweet', 'Sentiment'], dtype='object')"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.drop(['UserName', 'ScreenName', 'Location', 'TweetAt'], axis = 1)\n",
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "c67d5e6b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OriginalTweet    0\n",
       "Sentiment        0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "7b8cabeb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Neutral', 'Positive', 'Extremely Negative', 'Negative',\n",
       "       'Extremely Positive'], dtype=object)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Sentiment'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "96ddd5b8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Positive              11422\n",
       "Negative               9917\n",
       "Neutral                7713\n",
       "Extremely Positive     6624\n",
       "Extremely Negative     5481\n",
       "Name: Sentiment, dtype: int64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Sentiment'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "1053f6ee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>OriginalTweet</th>\n",
       "      <th>Sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>@menyrbie @phil_gahan @chrisitv https://t.co/i...</td>\n",
       "      <td>Neutral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>advice talk to your neighbours family to excha...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>coronavirus australia: woolworths to give elde...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>my food stock is not the only one which is emp...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>me, ready to go at supermarket during the #cov...</td>\n",
       "      <td>Extremely Negative</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       OriginalTweet           Sentiment\n",
       "0  @menyrbie @phil_gahan @chrisitv https://t.co/i...             Neutral\n",
       "1  advice talk to your neighbours family to excha...            Positive\n",
       "2  coronavirus australia: woolworths to give elde...            Positive\n",
       "3  my food stock is not the only one which is emp...            Positive\n",
       "4  me, ready to go at supermarket during the #cov...  Extremely Negative"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#We will do data cleaning and pre-processing on the OriginalTweet Column of this dataframe.\n",
    "#1. Case Conversion to Lower Case\n",
    "df['OriginalTweet'] = df['OriginalTweet'].str.lower()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "b5bc9849",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        @menyrbie @phil_gahan @chrisitv https://t.co/i...\n",
       "1        advice talk neighbours family exchange phone n...\n",
       "2        coronavirus australia: woolworths give elderly...\n",
       "3        food stock one empty... please, panic, enough ...\n",
       "4        me, ready go supermarket #covid19 outbreak. i'...\n",
       "                               ...                        \n",
       "41152    airline pilots offering stock supermarket shel...\n",
       "41153    response complaint provided citing covid-19 re...\n",
       "41154    know itâs getting tough @kameronwilds rationi...\n",
       "41155    wrong smell hand sanitizer starting turn on? #...\n",
       "41156    @tartiicat well new/used rift going $700.00 am...\n",
       "Name: OriginalTweet, Length: 41157, dtype: object"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Remove stop words and punctuation marks\n",
    "#https://stackoverflow.com/questions/29523254/python-remove-stop-words-from-pandas-dataframe\n",
    "import nltk\n",
    "import string\n",
    "from nltk.corpus import stopwords\n",
    "stop_words = stopwords.words('english')\n",
    "stopwordsandpunct = stop_words + list(string.punctuation)\n",
    "\n",
    "df['OriginalTweet'] = df['OriginalTweet'].apply(lambda w:' '.join(w for w in w.split() if w not in stopwordsandpunct))\n",
    "df['OriginalTweet']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "87e0a3f9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                   @menyrbie @phil_gahan @chrisitv   \n",
       "1    advice talk neighbours family exchange phone n...\n",
       "2    coronavirus australia: woolworths give elderly...\n",
       "3    food stock one empty... please, panic, enough ...\n",
       "4    me, ready go supermarket #covid19 outbreak. i'...\n",
       "Name: OriginalTweet, dtype: object"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Remove URLs from all the tweets\n",
    "import re\n",
    "def remove_url(tweet):\n",
    "    tweet = re.sub(r'\\w+:\\/{2}[\\d\\w-]+(\\.[\\d\\w-]+)*(?:(?:\\/[^\\s/]*))*', '', tweet)\n",
    "    return tweet\n",
    "\n",
    "df['OriginalTweet'] = df['OriginalTweet'].apply(remove_url)\n",
    "df['OriginalTweet'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "e2014bb8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                                                     \n",
       "1    advice talk neighbours family exchange phone n...\n",
       "2    coronavirus australia: woolworths give elderly...\n",
       "3    food stock one empty... please, panic, enough ...\n",
       "4    me, ready go supermarket  outbreak. i'm parano...\n",
       "Name: OriginalTweet, dtype: object"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# remove mentions and hashtags\n",
    "def remove_mentions_hashs(tweet):\n",
    "    tweet = re.sub(\"@[A-Za-z0-9_]+\",\"\", tweet) #Remove mentions\n",
    "    tweet = re.sub(\"#[A-Za-z0-9_]+\",\"\", tweet) #Remove hashtags\n",
    "    return tweet\n",
    "\n",
    "df['OriginalTweet'] = df['OriginalTweet'].apply(remove_mentions_hashs)\n",
    "df['OriginalTweet'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "6dcd2aa0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>OriginalTweet</th>\n",
       "      <th>Sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td>Neutral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>advice talk neighbours family exchange phone n...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>coronavirus australia: woolworths give elderly...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>food stock one empty... please, panic, enough ...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>me, ready go supermarket  outbreak. i'm parano...</td>\n",
       "      <td>Extremely Negative</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       OriginalTweet           Sentiment\n",
       "0                                                                Neutral\n",
       "1  advice talk neighbours family exchange phone n...            Positive\n",
       "2  coronavirus australia: woolworths give elderly...            Positive\n",
       "3  food stock one empty... please, panic, enough ...            Positive\n",
       "4  me, ready go supermarket  outbreak. i'm parano...  Extremely Negative"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Removing emojis from tweets\n",
    "# https://stackoverflow.com/a/49146722/330558\n",
    "import re\n",
    "def remove_emojis(tweet):\n",
    "    pat =      re.compile(\"[\"\n",
    "                           u\"\\U0001F600-\\U0001F64F\"  # emoticons\n",
    "                           u\"\\U0001F300-\\U0001F5FF\"  # symbols & pictographs\n",
    "                           u\"\\U0001F680-\\U0001F6FF\"  # transport & map symbols\n",
    "                           u\"\\U0001F1E0-\\U0001F1FF\"  # flags (iOS)\n",
    "                           u\"\\U00002702-\\U000027B0\"\n",
    "                           u\"\\U000024C2-\\U0001F251\"\n",
    "                           \"]+\", flags=re.UNICODE)\n",
    "    return pat.sub(r'', tweet)\n",
    "\n",
    "df['OriginalTweet'] = df['OriginalTweet'].apply(remove_emojis)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "19551b6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    "def remove_empty_strings1(tweet):\n",
    "    tweet = re.sub(r\"^\\s+|\\s+$\", 'NaN', tweet)\n",
    "    return tweet\n",
    "df['OriginalTweet'] = df['OriginalTweet'].apply(remove_empty_strings1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "07df49ee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>OriginalTweet</th>\n",
       "      <th>Sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td>Neutral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>advice talk neighbours family exchange phone n...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>coronavirus australia: woolworths give elderly...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>food stock one empty... please, panic, enough ...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>me, ready go supermarket  outbreak. i'm parano...</td>\n",
       "      <td>Extremely Negative</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       OriginalTweet           Sentiment\n",
       "0                                                                Neutral\n",
       "1  advice talk neighbours family exchange phone n...            Positive\n",
       "2  coronavirus australia: woolworths give elderly...            Positive\n",
       "3  food stock one empty... please, panic, enough ...            Positive\n",
       "4  me, ready go supermarket  outbreak. i'm parano...  Extremely Negative"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import unicodedata\n",
    "def remove_nonascii(text):\n",
    "    text = unicodedata.normalize('NFKD', text).encode('ascii', 'ignore').decode('utf-8', 'ignore')\n",
    "    return text\n",
    "df['OriginalTweet'] = df['OriginalTweet'].apply(remove_nonascii)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "833958f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Removing all rows containing NaNs\n",
    "df = df[df['OriginalTweet'] != 'NaN']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "c59a5dec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>OriginalTweet</th>\n",
       "      <th>Sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>advice talk neighbours family exchange phone n...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>coronavirus australia: woolworths give elderly...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>food stock one empty... please, panic, enough ...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>me, ready go supermarket  outbreak. i'm parano...</td>\n",
       "      <td>Extremely Negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>news regionas first confirmed covid-19 case ca...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       OriginalTweet           Sentiment\n",
       "0  advice talk neighbours family exchange phone n...            Positive\n",
       "1  coronavirus australia: woolworths give elderly...            Positive\n",
       "2  food stock one empty... please, panic, enough ...            Positive\n",
       "3  me, ready go supermarket  outbreak. i'm parano...  Extremely Negative\n",
       "4  news regionas first confirmed covid-19 case ca...            Positive"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Resetting index of dataframe to start from 0\n",
    "df = df.reset_index(drop = True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7bfe77bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Spelling correction\n",
    "#from textblob import TextBlob   \n",
    "#df['SpellCorrectedTweet'] = df['OriginalTweet'].apply(lambda x : str(TextBlob(x).correct()))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "b2953f3d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    [advice, talk, neighbours, family, exchange, p...\n",
       "1    [coronavirus, australia:, woolworths, give, el...\n",
       "2    [food, stock, one, empty..., please,, panic,, ...\n",
       "3    [me,, ready, go, supermarket, outbreak., i'm, ...\n",
       "4    [news, regionas, first, confirmed, covid-19, c...\n",
       "Name: OriginalTweet, dtype: object"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now we will use NLTK to perform tokenization and other text normalization steps on our dataset\n",
    "import nltk\n",
    "from nltk import word_tokenize\n",
    "tokenizer = nltk.tokenize.WhitespaceTokenizer()\n",
    "def tokenize(text):\n",
    "    return tokenizer.tokenize(text)\n",
    "\n",
    "df['OriginalTweet'] = df['OriginalTweet'].apply(tokenize)\n",
    "df['OriginalTweet'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "728792dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>OriginalTweet</th>\n",
       "      <th>Sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[advic, talk, neighbour, famili, exchang, phon...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[coronaviru, australia:, woolworth, give, elde...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[food, stock, one, empty..., please,, panic,, ...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[me,, readi, go, supermarket, outbreak., i'm, ...</td>\n",
       "      <td>Extremely Negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[news, regiona, first, confirm, covid-19, case...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       OriginalTweet           Sentiment\n",
       "0  [advic, talk, neighbour, famili, exchang, phon...            Positive\n",
       "1  [coronaviru, australia:, woolworth, give, elde...            Positive\n",
       "2  [food, stock, one, empty..., please,, panic,, ...            Positive\n",
       "3  [me,, readi, go, supermarket, outbreak., i'm, ...  Extremely Negative\n",
       "4  [news, regiona, first, confirm, covid-19, case...            Positive"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Stemming\n",
    "from nltk.stem import PorterStemmer\n",
    "stemmer = PorterStemmer()\n",
    "\n",
    "def stemming(text):\n",
    "    return [stemmer.stem(w) for w in text]\n",
    "\n",
    "df['OriginalTweet'] = df['OriginalTweet'].apply(stemming)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "64ebea95",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: total: 1.62 s\n",
      "Wall time: 1.67 s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>OriginalTweet</th>\n",
       "      <th>Sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[advic, talk, neighbour, famili, exchang, phon...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[coronaviru, australia:, woolworth, give, elde...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[food, stock, one, empty..., please,, panic,, ...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[me,, readi, go, supermarket, outbreak., i'm, ...</td>\n",
       "      <td>Extremely Negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[news, regiona, first, confirm, covid-19, case...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       OriginalTweet           Sentiment\n",
       "0  [advic, talk, neighbour, famili, exchang, phon...            Positive\n",
       "1  [coronaviru, australia:, woolworth, give, elde...            Positive\n",
       "2  [food, stock, one, empty..., please,, panic,, ...            Positive\n",
       "3  [me,, readi, go, supermarket, outbreak., i'm, ...  Extremely Negative\n",
       "4  [news, regiona, first, confirm, covid-19, case...            Positive"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "# Here we perform lemmatization\n",
    "import nltk\n",
    "tokenizer = nltk.tokenize.WhitespaceTokenizer()\n",
    "lemmatizer = nltk.stem.WordNetLemmatizer()\n",
    "\n",
    "def lemmatize(text):\n",
    "    return [lemmatizer.lemmatize(w) for w in text]\n",
    "\n",
    "df['OriginalTweet'] = df['OriginalTweet'].apply(lemmatize)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "835d4513",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Counting most frequent words in tweets\n",
    "#https://docs.python.org/3/library/itertools.html#itertools.chain\n",
    "import itertools\n",
    "import collections\n",
    "all_tweets = list(df[\"OriginalTweet\"])\n",
    "all_tokens = list(itertools.chain(*all_tweets))\n",
    "token_counts = collections.Counter(all_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "51ba0a5d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('price', 7398),\n",
       " ('store', 6532),\n",
       " ('groceri', 6274),\n",
       " ('supermarket', 6056),\n",
       " ('food', 6033),\n",
       " ('covid-19', 5286),\n",
       " ('peopl', 5002),\n",
       " ('consum', 4654),\n",
       " ('shop', 4068),\n",
       " ('go', 3994)]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_counts.most_common(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "e4f2549a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Token</th>\n",
       "      <th>Count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>price</td>\n",
       "      <td>7398</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>store</td>\n",
       "      <td>6532</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>groceri</td>\n",
       "      <td>6274</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>supermarket</td>\n",
       "      <td>6056</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>food</td>\n",
       "      <td>6033</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Token  Count\n",
       "0        price   7398\n",
       "1        store   6532\n",
       "2      groceri   6274\n",
       "3  supermarket   6056\n",
       "4         food   6033"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_token_counts = pd.DataFrame(token_counts.most_common(20), columns=['Token','Count'])\n",
    "df_token_counts.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "e8550616",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAswAAAIlCAYAAADBmq5QAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA800lEQVR4nO3deZhdVZnv8e9LggQFlCEMEiBIR5RBpkij4oRti6KCCAjXARVFaJza1u6gduNw6Y5eZ1tQnAhOEG1pUJwwMgoKAYLMghAhMoUoGlDm9/6xV5GTSmVVhVTtfZL6fp6nnnP2OsN+T1J1zu+svfZakZlIkiRJGtoaXRcgSZIk9TMDsyRJklRhYJYkSZIqDMySJElShYFZkiRJqjAwS5IkSRUGZkkahyLi7Ih4Sx/UMTUiMiImdl2LJC2PgVmSioiYHxEPRMRGg9rnlVA3dSWfPyPi7yq3fygivrmijxtNEbFZ2d8mPW0fWE7bT9qoSZK6ZmCWpKXdBBwysBEROwJrd1dOuzLzNuAG4Hk9zc8Drh2i7dwVeW57kSWtqgzMkrS0bwBv6Nk+FDip9w4R8cSIOCkiFkbE7yPigxGxRrnt7yLinIj4c0TcFRGnlPaBcHl5RNwTEa95LMVFxBsj4saIWBwRN0XEa3tue3NEXBMRf4qIn0bEVj23vTgiri11/TcQld2cSwnHETEB2AX47KC2ZwHnRsQa5fX/PiLuLP8uTyz3GxhucVhE3Az8IiImRMQnyr/NjcA+I319ktQVA7MkLe1XwHoR8fQSDF8DDB4m8XngicBTgOfTBOw3lds+CvwMWB+YUu5LZg70zu6Umetk5ikrWlhEPAH4HPDSzFwXeDYwr9y2H/B+YH9gMnAe8J1y20bA/wAfBDYCfgc8p7KrRwMzTVi+FpgzqG1N4CLgjeXnhTT/HusA/z3o+Z4PPB14CfBW4OXlOaYDB4zk9UlSlwzMkrSsgV7mF9OExT8M3NAToo/OzMWZOR/4JPD6cpcHga2AJ2fmfZl5/ijX9giwQ0SsnZm3ZeZVpf1twH9l5jWZ+RDwn8DOpZf5ZcDVmfm9zHwQ+Axwe2Uf55R9rA88FzgvM68HNupp+1VmPgC8FvhUZt6YmfcARwMHDxp+8aHMvDcz/wYcBHwmM2/JzD8C/zXC1ydJnTEwS9KyvgH8H5qe05MG3bYR8Djg9z1tvwc2L9f/lWa4w0URcVVEvHkF9vsQTc/toyJiYPvBzLyXJqwfAdwWEWdExNPK7VsBn42IuyPibuCPpY7NgScDtww8Z2Zm7/Zg5UvAAmBPml7l88pNF/a0DQwxeTLL/ltMBDbpaevd15MHbT/62GFenyR1xsAsSYNk5u9pTv57GfD9QTffxZJe5AFbUnqhM/P2zHxrZj6Zptf3uBWY4eJmYOqgtq2Bh3ue/6eZ+WJgM5re7y+X+90CvC0zn9Tzs3ZmXgDcBmwx8IQREb3by3EeTTB+FnDBoLY9WRKYb2XZf4uHgDt62rLn+lK1lPsvuePyX58kdcbALElDOwzYq/R6PiozHwZmA8dGxLplyMN7KOOcI+LAiJhS7v4nmrD4cNm+g2ac7/L8BNg2Il4fEWtGxAY0Qyu+l5kPRcQmEfHKMtb3fuCenuf+InB0RGxf6nhiRBxYbjsD2D4i9i9DJd4JbDrM6z+XZljKrZn5l9J2fml7Ik1vMzTjpP85IraOiHVKvaeUYSFDmQ28MyKmlOEdMwZuGOb1SVJnDMySNITM/F1mzl3Oze8A7gVupAmR3wa+Vm57JvDriLgHOB14V2beVG77EDCrDJs4aIh93knTq/024E7gSuDPwJHlLmsA/0LTq/tHmpPp/qk89lTgY8DJEfGX8tiXltvuAg4EZgKLgGnAL4f5JzgH2Li8vgHzaKbYuyQz/1ravkYzhOVcml75+8q/z/J8GfgpcDlwKUv34C/39UlSl6IZyiZJkiRpKPYwS5IkSRUGZkmSJKnCwCxJkiRVGJglSZKkCgOzJEmSVDFx+Lt0a6ONNsqpU6d2XYYkSZJWY5dccsldmTl5qNv6PjBPnTqVuXOXNxWqJEmStPIi4vfLu80hGZIkSVKFgVmSJEmqMDBLkiRJFX0/hnkoDz74IAsWLOC+++7rupROTJo0iSlTprDmmmt2XYokSdJqb5UMzAsWLGDddddl6tSpRETX5bQqM1m0aBELFixg66237rocSZKk1d4qOSTjvvvuY8MNNxx3YRkgIthwww3Hbe+6JElS21bJwAyMy7A8YDy/dkmSpLatsoG5a7fffjsHH3ww22yzDdtttx0ve9nL+O1vfztqz3/22WdzwQUXjNrzSZIk6bFZJccwDzZ1xhmj+nzzZ+5TvT0zedWrXsWhhx7KySefDMC8efO44447eOpTnzoqNZx99tmss846PPvZzx6V55MkSdJjYw/zY3DWWWex5pprcsQRRzzatvPOO7Pnnnvyvve9jx122IEdd9yRU045BWjC78tf/vJH7/v2t7+dE088EWhWMjzmmGPYdddd2XHHHbn22muZP38+X/ziF/n0pz/NzjvvzHnnndfq65MkSdISq0UPc9uuvPJKdtttt2Xav//97zNv3jwuv/xy7rrrLp75zGfyvOc9b9jn22ijjbj00ks57rjj+MQnPsFXvvIVjjjiCNZZZx3e+973jsVLkCRJ0gjZwzyKzj//fA455BAmTJjAJptswvOf/3wuvvjiYR+3//77A7Dbbrsxf/78Ma5SkiRJK8LA/Bhsv/32XHLJJcu0Z+aQ9584cSKPPPLIo9uDp4Rba621AJgwYQIPPfTQKFYqSZKklWVgfgz22msv7r//fr785S8/2nbxxRez/vrrc8opp/Dwww+zcOFCzj33XHbffXe22morrr76au6//37+/Oc/M2fOnGH3se6667J48eKxfBmSJEkaAccwPwYRwamnnsq73/1uZs6cyaRJk5g6dSqf+cxnuOeee9hpp52ICD7+8Y+z6aabAnDQQQfxjGc8g2nTprHLLrsMu49XvOIVHHDAAZx22ml8/vOf57nPfe5YvyxJkiQNIZY3jKBfTJ8+PefOnbtU2zXXXMPTn/70jirqD/4bSJIkjZ6IuCQzpw91m0MyJEmSpAoDsyRJklThGGZJkiT1tdFY1Xm4lZxrVtke5n4fez2WxvNrlyRJatsqGZgnTZrEokWLxmVwzEwWLVrEpEmTui5FkiRpXFglh2RMmTKFBQsWsHDhwq5L6cSkSZOYMmVK12VIkiSNC6tkYF5zzTXZeuutuy5DkiRJ48AqOSRDkiRJaouBWZIkSaowMEuSJEkVBmZJkiSpwsAsSZIkVRiYJUmSpAoDsyRJklRhYJYkSZIqDMySJElShYFZkiRJqjAwS5IkSRUGZkmSJKnCwCxJkiRVGJglSZKkimEDc0RsGxHzen7+EhHvjogNIuLMiLi+XK7f85ijI+KGiLguIl7S075bRFxRbvtcRMRYvTBJkiRpNAwbmDPzuszcOTN3BnYD/gqcCswA5mTmNGBO2SYitgMOBrYH9gaOi4gJ5emOBw4HppWfvUf11UiSJEmjbEWHZLwI+F1m/h7YF5hV2mcB+5Xr+wInZ+b9mXkTcAOwe0RsBqyXmRdmZgIn9TxGkiRJ6ksrGpgPBr5Trm+SmbcBlMuNS/vmwC09j1lQ2jYv1we3S5IkSX1rxIE5Ih4HvBL47nB3HaItK+1D7evwiJgbEXMXLlw40hIlSZKkUbciPcwvBS7NzDvK9h1lmAXl8s7SvgDYoudxU4BbS/uUIdqXkZknZOb0zJw+efLkFShRkiRJGl0rEpgPYclwDIDTgUPL9UOB03raD46ItSJia5qT+y4qwzYWR8QeZXaMN/Q8RpIkSepLE0dyp4h4PPBi4G09zTOB2RFxGHAzcCBAZl4VEbOBq4GHgKMy8+HymCOBE4G1gR+XH0mSJKlvjSgwZ+ZfgQ0HtS2imTVjqPsfCxw7RPtcYIcVL1OSJEnqhiv9SZIkSRUGZkmSJKnCwCxJkiRVGJglSZKkCgOzJEmSVGFgliRJkioMzJIkSVKFgVmSJEmqMDBLkiRJFQZmSZIkqcLALEmSJFUYmCVJkqQKA7MkSZJUYWCWJEmSKgzMkiRJUoWBWZIkSaowMEuSJEkVBmZJkiSpwsAsSZIkVRiYJUmSpAoDsyRJklQxsesCJEmS1L+mzjhjpR4/f+Y+o1RJd+xhliRJkioMzJIkSVKFgVmSJEmqMDBLkiRJFQZmSZIkqcLALEmSJFUYmCVJkqQKA7MkSZJUYWCWJEmSKgzMkiRJUoWBWZIkSaowMEuSJEkVBmZJkiSpYmLXBUiSJGlZU2ecsdLPMX/mPqNQiexhliRJkioMzJIkSVKFgVmSJEmqMDBLkiRJFQZmSZIkqcLALEmSJFUYmCVJkqQKA7MkSZJUYWCWJEmSKgzMkiRJUoWBWZIkSaoYUWCOiCdFxPci4tqIuCYinhURG0TEmRFxfblcv+f+R0fEDRFxXUS8pKd9t4i4otz2uYiIsXhRkiRJ0miZOML7fRb4SWYeEBGPAx4PvB+Yk5kzI2IGMAP4t4jYDjgY2B54MvDziHhqZj4MHA8cDvwK+BGwN/DjUX1FkiRJK2nqjDNW6vHzZ+4zSpWoHwwbmCNiPeB5wBsBMvMB4IGI2Bd4QbnbLOBs4N+AfYGTM/N+4KaIuAHYPSLmA+tl5oXleU8C9sPALEmSipUNqmBY1egbyZCMpwALga9HxGUR8ZWIeAKwSWbeBlAuNy733xy4pefxC0rb5uX64HZJkiSpb40kME8EdgWOz8xdgHtphl8sz1DjkrPSvuwTRBweEXMjYu7ChQtHUKIkSZI0NkYSmBcACzLz12X7ezQB+o6I2AygXN7Zc/8teh4/Bbi1tE8Zon0ZmXlCZk7PzOmTJ08e6WuRJEmSRt2wgTkzbwduiYhtS9OLgKuB04FDS9uhwGnl+unAwRGxVkRsDUwDLirDNhZHxB5ldow39DxGkiRJ6ksjnSXjHcC3ygwZNwJvognbsyPiMOBm4ECAzLwqImbThOqHgKPKDBkARwInAmvTnOznCX+SJEnqayMKzJk5D5g+xE0vWs79jwWOHaJ9LrDDCtQnSZIkdcqV/iRJkqQKA7MkSZJUYWCWJEmSKgzMkiRJUoWBWZIkSaowMEuSJEkVBmZJkiSpwsAsSZIkVRiYJUmSpAoDsyRJklRhYJYkSZIqDMySJElShYFZkiRJqjAwS5IkSRUGZkmSJKnCwCxJkiRVGJglSZKkCgOzJEmSVGFgliRJkioMzJIkSVKFgVmSJEmqMDBLkiRJFQZmSZIkqcLALEmSJFUYmCVJkqQKA7MkSZJUYWCWJEmSKgzMkiRJUoWBWZIkSaowMEuSJEkVBmZJkiSpwsAsSZIkVRiYJUmSpAoDsyRJklRhYJYkSZIqJnZdgCRJ6g9TZ5yx0s8xf+Y+o1CJ1F/sYZYkSZIqDMySJElShYFZkiRJqjAwS5IkSRUGZkmSJKnCwCxJkiRVGJglSZKkCgOzJEmSVGFgliRJkioMzJIkSVKFgVmSJEmqGFFgjoj5EXFFRMyLiLmlbYOIODMiri+X6/fc/+iIuCEirouIl/S071ae54aI+FxExOi/JEmSJGn0rEgP8wszc+fMnF62ZwBzMnMaMKdsExHbAQcD2wN7A8dFxITymOOBw4Fp5WfvlX8JkiRJ0thZmSEZ+wKzyvVZwH497Sdn5v2ZeRNwA7B7RGwGrJeZF2ZmAif1PEaSJEnqSyMNzAn8LCIuiYjDS9smmXkbQLncuLRvDtzS89gFpW3zcn1wuyRJktS3Jo7wfs/JzFsjYmPgzIi4tnLfocYlZ6V92SdoQvnhAFtuueUIS5QkadU0dcYZK/0c82fuMwqVSBrKiHqYM/PWcnkncCqwO3BHGWZBubyz3H0BsEXPw6cAt5b2KUO0D7W/EzJzemZOnzx58shfjSRJkjTKhg3MEfGEiFh34Drwj8CVwOnAoeVuhwKnleunAwdHxFoRsTXNyX0XlWEbiyNijzI7xht6HiNJkiT1pZEMydgEOLXMADcR+HZm/iQiLgZmR8RhwM3AgQCZeVVEzAauBh4CjsrMh8tzHQmcCKwN/Lj8SJIkSX1r2MCcmTcCOw3Rvgh40XIecyxw7BDtc4EdVrxMSZIkqRuu9CdJkiRVGJglSZKkCgOzJEmSVGFgliRJkioMzJIkSVKFgVmSJEmqMDBLkiRJFQZmSZIkqcLALEmSJFUYmCVJkqQKA7MkSZJUYWCWJEmSKgzMkiRJUoWBWZIkSaqY2HUBkiR1aeqMM1b6OebP3GcUKpHUr+xhliRJkioMzJIkSVKFgVmSJEmqMDBLkiRJFQZmSZIkqcLALEmSJFUYmCVJkqQKA7MkSZJUYWCWJEmSKgzMkiRJUoWBWZIkSaowMEuSJEkVBmZJkiSpwsAsSZIkVRiYJUmSpAoDsyRJklRhYJYkSZIqDMySJElShYFZkiRJqjAwS5IkSRUGZkmSJKnCwCxJkiRVGJglSZKkCgOzJEmSVGFgliRJkioMzJIkSVKFgVmSJEmqMDBLkiRJFQZmSZIkqcLALEmSJFUYmCVJkqQKA7MkSZJUMXGkd4yICcBc4A+Z+fKI2AA4BZgKzAcOysw/lfseDRwGPAy8MzN/Wtp3A04E1gZ+BLwrM3O0XowkadUxdcYZK/0c82fuMwqVSFLdivQwvwu4pmd7BjAnM6cBc8o2EbEdcDCwPbA3cFwJ2wDHA4cD08rP3itVvSRJkjTGRhSYI2IKsA/wlZ7mfYFZ5fosYL+e9pMz8/7MvAm4Adg9IjYD1svMC0uv8kk9j5EkSZL60kh7mD8D/CvwSE/bJpl5G0C53Li0bw7c0nO/BaVt83J9cLskSZLUt4YdwxwRLwfuzMxLIuIFI3jOGKItK+1D7fNwmqEbbLnlliPYpSRpRTh+WJJGbiQ9zM8BXhkR84GTgb0i4pvAHWWYBeXyznL/BcAWPY+fAtxa2qcM0b6MzDwhM6dn5vTJkyevwMuRJEmSRtewgTkzj87MKZk5leZkvl9k5uuA04FDy90OBU4r108HDo6ItSJia5qT+y4qwzYWR8QeERHAG3oeI0mSJPWlEU8rN4SZwOyIOAy4GTgQIDOviojZwNXAQ8BRmflwecyRLJlW7sflR5IkSepbKxSYM/Ns4OxyfRHwouXc71jg2CHa5wI7rGiRkiRJUldc6U+SJEmqMDBLkiRJFQZmSZIkqcLALEmSJFUYmCVJkqQKA7MkSZJUYWCWJEmSKgzMkiRJUoWBWZIkSaowMEuSJEkVBmZJkiSpwsAsSZIkVRiYJUmSpAoDsyRJklRhYJYkSZIqDMySJElSxcSuC5Ck8WTqjDNW+jnmz9xnFCqRJI2UPcySJElShYFZkiRJqjAwS5IkSRWOYZY0bjh+WJL0WNjDLEmSJFUYmCVJkqQKA7MkSZJUYWCWJEmSKgzMkiRJUoWBWZIkSaowMEuSJEkVzsMsacw5/7EkaVVmD7MkSZJUYWCWJEmSKgzMkiRJUoWBWZIkSaowMEuSJEkVBmZJkiSpwsAsSZIkVRiYJUmSpAoDsyRJklRhYJYkSZIqDMySJElShYFZkiRJqjAwS5IkSRUTuy5A0tiaOuOMlX6O+TP3GYVKJElaNdnDLEmSJFUYmCVJkqQKA7MkSZJUYWCWJEmSKoYNzBExKSIuiojLI+KqiPhwad8gIs6MiOvL5fo9jzk6Im6IiOsi4iU97btFxBXlts9FRIzNy5IkSZJGx0h6mO8H9srMnYCdgb0jYg9gBjAnM6cBc8o2EbEdcDCwPbA3cFxETCjPdTxwODCt/Ow9ei9FkiRJGn3DBuZs3FM21yw/CewLzCrts4D9yvV9gZMz8/7MvAm4Adg9IjYD1svMCzMzgZN6HiNJkiT1pRGNYY6ICRExD7gTODMzfw1skpm3AZTLjcvdNwdu6Xn4gtK2ebk+uF2SJEnqWyMKzJn5cGbuDEyh6S3eoXL3ocYlZ6V92SeIODwi5kbE3IULF46kREmSJGlMrNAsGZl5N3A2zdjjO8owC8rlneVuC4Ateh42Bbi1tE8Zon2o/ZyQmdMzc/rkyZNXpERJkiRpVA27NHZETAYezMy7I2Jt4B+AjwGnA4cCM8vlaeUhpwPfjohPAU+mObnvosx8OCIWlxMGfw28Afj8aL8gqV+4JLUkSauHYQMzsBkwq8x0sQYwOzN/GBEXArMj4jDgZuBAgMy8KiJmA1cDDwFHZebD5bmOBE4E1gZ+XH4kSZKkvjVsYM7M3wC7DNG+CHjRch5zLHDsEO1zgdr4Z0mSJKmvuNKfJEmSVGFgliRJkioMzJIkSVKFgVmSJEmqGMksGdIqxyndJEnSaLGHWZIkSaowMEuSJEkVBmZJkiSpwsAsSZIkVRiYJUmSpAoDsyRJklRhYJYkSZIqDMySJElShYFZkiRJqjAwS5IkSRUGZkmSJKliYtcFaPRMnXHGSj/H/Jn7dF7HaNQgSZI0WuxhliRJkioMzJIkSVKFgVmSJEmqcAzzKHHcriRJ0urJHmZJkiSpYpXvYe6XmSEkSZK0erKHWZIkSaowMEuSJEkVBmZJkiSpwsAsSZIkVRiYJUmSpAoDsyRJklRhYJYkSZIqDMySJElShYFZkiRJqjAwS5IkSRUGZkmSJKnCwCxJkiRVGJglSZKkCgOzJEmSVGFgliRJkioMzJIkSVKFgVmSJEmqMDBLkiRJFQZmSZIkqcLALEmSJFUYmCVJkqQKA7MkSZJUYWCWJEmSKgzMkiRJUsWwgTkitoiIsyLimoi4KiLeVdo3iIgzI+L6crl+z2OOjogbIuK6iHhJT/tuEXFFue1zERFj87IkSZKk0TGSHuaHgH/JzKcDewBHRcR2wAxgTmZOA+aUbcptBwPbA3sDx0XEhPJcxwOHA9PKz96j+FokSZKkUTdsYM7M2zLz0nJ9MXANsDmwLzCr3G0WsF+5vi9wcmben5k3ATcAu0fEZsB6mXlhZiZwUs9jJEmSpL60QmOYI2IqsAvwa2CTzLwNmlANbFzutjlwS8/DFpS2zcv1we2SJElS3xpxYI6IdYD/Ad6dmX+p3XWItqy0D7WvwyNibkTMXbhw4UhLlCRJkkbdiAJzRKxJE5a/lZnfL813lGEWlMs7S/sCYIueh08Bbi3tU4ZoX0ZmnpCZ0zNz+uTJk0f6WiRJkqRRN5JZMgL4KnBNZn6q56bTgUPL9UOB03raD46ItSJia5qT+y4qwzYWR8Qe5Tnf0PMYSZIkqS9NHMF9ngO8HrgiIuaVtvcDM4HZEXEYcDNwIEBmXhURs4GraWbYOCozHy6POxI4EVgb+HH5kSRJkvrWsIE5M89n6PHHAC9azmOOBY4don0usMOKFChJkiR1yZX+JEmSpAoDsyRJklRhYJYkSZIqDMySJElShYFZkiRJqjAwS5IkSRUGZkmSJKnCwCxJkiRVGJglSZKkCgOzJEmSVGFgliRJkioMzJIkSVKFgVmSJEmqMDBLkiRJFQZmSZIkqcLALEmSJFUYmCVJkqQKA7MkSZJUYWCWJEmSKgzMkiRJUoWBWZIkSaowMEuSJEkVBmZJkiSpwsAsSZIkVRiYJUmSpAoDsyRJklRhYJYkSZIqDMySJElShYFZkiRJqjAwS5IkSRUGZkmSJKnCwCxJkiRVGJglSZKkCgOzJEmSVGFgliRJkioMzJIkSVKFgVmSJEmqMDBLkiRJFQZmSZIkqcLALEmSJFUYmCVJkqQKA7MkSZJUYWCWJEmSKgzMkiRJUoWBWZIkSaowMEuSJEkVBmZJkiSpYtjAHBFfi4g7I+LKnrYNIuLMiLi+XK7fc9vREXFDRFwXES/pad8tIq4ot30uImL0X44kSZI0ukbSw3wisPegthnAnMycBswp20TEdsDBwPblMcdFxITymOOBw4Fp5Wfwc0qSJEl9Z9jAnJnnAn8c1LwvMKtcnwXs19N+cmben5k3ATcAu0fEZsB6mXlhZiZwUs9jJEmSpL71WMcwb5KZtwGUy41L++bALT33W1DaNi/XB7dLkiRJfW20T/obalxyVtqHfpKIwyNibkTMXbhw4agVJ0mSJK2oxxqY7yjDLCiXd5b2BcAWPfebAtxa2qcM0T6kzDwhM6dn5vTJkyc/xhIlSZKklfdYA/PpwKHl+qHAaT3tB0fEWhGxNc3JfReVYRuLI2KPMjvGG3oeI0mSJPWticPdISK+A7wA2CgiFgDHADOB2RFxGHAzcCBAZl4VEbOBq4GHgKMy8+HyVEfSzLixNvDj8iNJkiT1tWEDc2YespybXrSc+x8LHDtE+1xghxWqTpIkSeqYK/1JkiRJFQZmSZIkqcLALEmSJFUYmCVJkqQKA7MkSZJUYWCWJEmSKgzMkiRJUoWBWZIkSaowMEuSJEkVBmZJkiSpwsAsSZIkVRiYJUmSpAoDsyRJklRhYJYkSZIqDMySJElShYFZkiRJqjAwS5IkSRUGZkmSJKnCwCxJkiRVGJglSZKkCgOzJEmSVGFgliRJkioMzJIkSVKFgVmSJEmqMDBLkiRJFQZmSZIkqcLALEmSJFUYmCVJkqQKA7MkSZJUYWCWJEmSKgzMkiRJUoWBWZIkSaowMEuSJEkVBmZJkiSpwsAsSZIkVRiYJUmSpAoDsyRJklRhYJYkSZIqDMySJElShYFZkiRJqjAwS5IkSRUGZkmSJKnCwCxJkiRVGJglSZKkCgOzJEmSVGFgliRJkioMzJIkSVJF64E5IvaOiOsi4oaImNH2/iVJkqQV0WpgjogJwBeAlwLbAYdExHZt1iBJkiStiLZ7mHcHbsjMGzPzAeBkYN+Wa5AkSZJGrO3AvDlwS8/2gtImSZIk9aXIzPZ2FnEg8JLMfEvZfj2we2a+Y9D9DgcOL5vbAtetxG43Au5aicePln6oox9qgP6oox9qgP6oox9qgP6oox9qgP6oox9qgP6oox9qgP6oox9qgP6oox9qgP6oox9qgJWvY6vMnDzUDRNX4kkfiwXAFj3bU4BbB98pM08AThiNHUbE3MycPhrPtarX0Q819Esd/VBDv9TRDzX0Sx39UEO/1NEPNfRLHf1QQ7/U0Q819Esd/VBDv9TRDzWMdR1tD8m4GJgWEVtHxOOAg4HTW65BkiRJGrFWe5gz86GIeDvwU2AC8LXMvKrNGiRJkqQV0faQDDLzR8CPWtzlqAztGAX9UEc/1AD9UUc/1AD9UUc/1AD9UUc/1AD9UUc/1AD9UUc/1AD9UUc/1AD9UUc/1AD9UUc/1ABjWEerJ/1JkiRJqxqXxpYkSZIqDMySJElShYFZkjoWEVuPpE1StyJirZG0afWzWo1hjoj9a7dn5vfbqgUgIh4P/AuwZWa+NSKmAdtm5g9brOGwzPzqoLaZmTmjrRrKPjfIzD+2uU8tKyI2qN3u/1E3IuLSzNx1UNslmblby3U8EfgQ8NzSdA7wkcz8c8t1bArsDiRwcWbe3ub+e+p4BjCVnhPk2/4c6QcRMZlm3YSHgJsy854W991X71nL+Vtdpm08iIgAXgs8JTM/EhFbAptm5kUt17EVMC0zfx4RawMTM3PxaO+n9VkyxtgryuXGwLOBX5TtFwJnA22/0X0duAR4VtleAHwXaC0wAwdExH2Z+S2AiDgO6OLb8K8jYh7Nv8mPs8VvahHxA5oP3iFl5itbrGUy8FaW/RB+c0slXELzbxHAlsCfyvUnATcD465Xc/CHXURcU65+ITP/e4z3/TRge+CJg77wrwdMGst9L8fXgCuBg8r262n+ZqudEaMpIt4C/AfN+3cAn4+Ij2Tm19qqodTxNeAZwFXAI6U5aelzJCI+T/19650t1LAd8Dma96stgcuAjSPiHOBdLX2R6ov3rPIlbnNg7YjYpdQAzd/q41uq4fzM3DMiFrP070YAmZnrtVFHj+No/jb2Aj4CLAb+B3hmWwVExFtpVobeANiG5ovdF4EXjfa+VqvAnJlvAoiIHwLbZeZtZXsz4AsdlLRNZr4mIg4p9f2tfCNr0/7A6RHxCPBS4I+Z+U8t1wDwVOAfgDfTfACeApyYmb9tYd+fKJf7A5sC3yzbhwDzW9h/r9OA84CfAw+3vG8yc2uAiPgicHqZ5pGIeCnN/08rImId4F+BV9O8wT0A/A74Ymae2FYdAIN7hjLz6RGxIbBHC7vfFng5zYf/K3raF9N8sWrbNpn56p7tD5cvum16H7BLZi4CKP8XF9CE+TbtkZnbtbzPXnM73PeArwGHZuZ1EbE7cFRm/n0JKV8FDhjrAvrlPQt4CfBGmverT/W0/wV4fxsFZOae5XLdNvY3An+fmbtGxGUAmfmnsihdm46iORr161LD9RGx8ZjsKTNXux/gykHbawxua6mOC4C1gUvL9jbARS3te4Oen61oegb+e6Ct4/+fFwJ/AO6mOeT7rJb2e+5I2sa4hnld/tv31HHJEG1zW9z/aSz58HkP8O/ANGAW8J8d/HtsCrySJrRu0sH+W/kbGEEdFwJ79mw/B7iw5RrmAI/r2X4c8PMO/i2+StPx0vn/S6lnPWDdlvd5+aDtS3uuX91yLZ2+Z/Xs89Ud/x50kmeWU8uvaRahG8g4k4HL2q6hXF5WLicCvxmLfa1WPcw9zo6InwLfoTlscTBwVgd1HAP8BNgiIr5F8+Hzxpb2PXAYa0AA+5SfBJ7SUh3NzpteotfRHOK9A3gHzbLoO9MMU2njsNrkiHhKZt5Yatqa5g+8TT+MiJdl6SXp0F0R8UGa3vak+b9Z1OL+p+aSnuRPRcTFmfnRiHgTcDUt9dhA3wwBWBQRc2jC+g5l7OwrM/P/tlgDwBHASWUsMzSHvw9tuYY/0AzhOo3md3Nf4KKIeA9AZn6q9uBRNAu4MCJuB+5nyWHvZ7S0fwAiYjrNsJh1m824G3hzZl7Swu5/FxH/TvMlZn9gXqlpTdo/Qt31e9aAX0bEV4EnZ+ZLy7CVZ+Wgc4XGSmY+EhGXR8SWmXlzG/us+BxwKs0wnWNpjjh8sOUazomI99MMlXkx8E/AD8ZiR6vVSX+9ynjAgRNXzs3MUzuoYQOaN9k9yuWvaHoIbmpp/2vQ/CH/so39DVPLb4FvAF/PzAWDbvu3zPxYCzXsTbMK0I2laSrwtsz86Vjvu6eGxcATaD6AH6SjsWfld/MY4Hk0Hz7n0pzc1coJNBFxAfCvmXl+RLwCeHtmvqTcdl1mbttGHQP7A56dg4YAtFzDOTRDEb6UmbuUtiszc4e2aij7fE+5uk65vAf4M03v3ryWajimdntmfrilOm6gOfpxBUvGMJOZv29j/z11/IZmKMR5ZXtP4Lg2gntEPInmy+t2wOXAzMxcXL5QPT0zfzXWNfTU0vueBc171ofbes/qqePHNF9gPpCZO0XERJrezR1brOEXNOOELwLuHWjPds/HWYMm2/yRZrxwAHMy85rqA8emjsOAfyw1/BT4So5BuF1tA3M/iIhfAi/NzL+U7acD323zQzAiLszMZw1/zzGvI8biF/gx1LEW8LSyeW1m3t9lPV2LiHWyxTPee/b7DOArNGPbr6TpMfttOSnykMz8XIu1zKH5O32gbD8O+FFmtjmm++LMfGZEXNYTmOdl5s5t1VD2+W1gOs3Rn4GjUhfT/M18NzM/3mIt69J8mWz997Ps/xeZuVcX+x5Uxy8z8znDtY0XEbEe8EiHvxed/61GxPOHas/Mc9qqodTReb6IiCcA92Xmw2V7ArBWZv51tPe1Wg7JKL3LH6OZLSPo7gzS/wR+EBEvo/nAOYlmCpY2/SwiXg18v+PA+ouIWGb/bXwgRcRemfmLWHbawW0igmxhmqiIeFpmXhsRQ049lJmXjnUNg+p5Nk1gXQfYMiJ2oultb+WE0Mz8Dc2JGoPbF5Ze+Db1wxCAuyJim7J/IuIA4LYW9jvYhsCuA2Gk9PZ+j6ZX7xJgzANzROxAczRqg7J9F/CGzLxqrPc9yLXlC8QPaI4IAZ1MK3dRRHyJJUMMX0Mz7HDXUk+r7x0DIuKEzDy8xf3tSPMZ2vt7cWhmXtlWDcW95SjUwN/qHjRHYVqTmefE0lOpPZ5mLHHb+iFfzKE5+XPgC9TawM9oZkobVatlYKZ5U39F24cGBsvMM8pYrzNpxp/tl5nXt1zGe2iGADwcEX+juy8P7+25PolmdoSHWtr382nGp75iiNvamibqX2hmPfjkcmpouyfr0zRnfZ8OkJmXR8Tz6g9pzYdpDnm25XflZ8Bp5bLNM9GPohku9LSI+ANwE+1/uYZm2q4HerYfBLbKZoafto7GnAC8JzPPAoiIFwBfZgw+AIexNk1Q/seettamleuxc7kcPFTl2Yzxe0csfw7kAF42Vvtdji+x7O/FCbT/e/EemvfNbcpR5Mm0MFtIr1h2KrXNGaOp1IbRmy/uK21t54tJvUcbMvOe8gVi1K2ugfmOLsNyLDt/5no042bfUXo0x3z+zAHZJ9PPDHGCyi/LuM029j3wQfOWgcM2bcvMt5bLF3ax/6Fk5i2x9CyHrf3blHGZQ94EbNJWHdDemNhh7Af8iObk5DVoxiX+QzSLl8xrsY5vA78qve3QfMn8TjnseXVLNTxhIBQBZObZZf+tyjJNadc6fs9YCPyeJXMOw5I5kcdm6q7l64vfC5oxu8+nmRIygOtY8qWmLe1NpVbRJ/ni3ojYdeBIS0TsBvxtLHa0ugbmudHM8/u/dHMobfD8mW2czbxcEfFKlpwocXa2uNJgTw29PRVr0IyT3LTlMm6KiJ8ApwC/aPMQ0hDDQZbSwWHeW8qwjCxjdt8JtPklcxOaHu4/DWoPmukYWxPNLAQfoJl+sXcxmTZnQ5jO0mOHX0szdviIiGht7HA2M5X8CNiz1HFEZg68n7XV431jNDMzfKNsv46mx71VEfF1hlg4JNtbZGigjiey9Mluba6+eCPwohxiNoaIuKWF/S9VSz/8XtAszPHKgSFC5cjcF4DWTvoD7s/MBwY6PMqJh50MieiDfPFu4LsRcWvZ3oxm2NKoW10D83rAX+noUFpmzmpjPyMRETNpzqb9Vml6V0TsmS0vjc3S09w9RLNgyGEt17AtTY/ZUcBXo1ng5uTMPL+FfQ81HGRAF4d5jwA+S3MobwHNmK+jWtz/D4F1huo9jYizW6wDmr+N9zFoNoSWdT52eEA5GtTll/w30wzL+T5NaD8X6KK3t/eDfxLwKuDW5dx3LHW5+uJngPVpVtQbrLXfyaJffi+OAP63zO6zK825Sm0PTzknWppKraYf8kVmXhzNiqkDPf7XZuaDY7EvZ8kYAxExOzMPiogrGLqHorWeq3Loe+fMfKRsT6CZAqftuUTXpvmj3pPm3+Q84PjMvK/6wLGrZ32awPjazOziZAn1iSjLzXZcwzXATj0zdaxFs8jN03vPxld3yvRVP2975oyhZmDoYgaVftH1LBmlhmfRjKm+D9gnMxe2vP/WplIbpo7O8kXlZH5gbI7arpY9zBExieaXaXuangGg1UNp7yqXL29pf8N5Es24K4AnVu43lmbRLCE6MF3YITSH1g5ss4gyHc9raJYJv5glvTZt7X8tmhMep7L04f+PtFzHU4Hj6X6hjH5wTER8heZs665mQ+iHscN9ofxuvpdl/0a6nuJtGs1JkW37W+m1Ox8gIp7DGI3RHIm2Z8fo2W+ns2RExA9YugPs8TSzY3y1nJvU2hzINCekfi0zv1xqm1DaRn0qtRF4Et3ki9ZP5l8tAzNNELuWZozkR2jG3rU2PjMzbyuXrU5wvxz/CVxaDnMHzSHeozuoY9vM3Kln+6yIuLzNAiLiJpqVqmYD78vMe+uPGBOnURaCoCecdeDLlIUyoJnmrUyhNR4D85topn1ckyVDMlodJtMnY4f7xXdpzvj/Ci2eiDpYmd6wNyDdDvxbB6UcCcyKpVdffGMHdQyY3tF+u54l4xMt7WckWptKbRj/BVwWEWexJF+0skprZh5Tetp/nJmz29jnajkkY+AQZkT8JjOfUaZ2+2lbPRRDvNE+ehMtT7kSEd8Arqd5k72ZZt3129vaf08dJwJfzLIyVET8PU3vQCvz/pZ9rpdlEZmuRAerty2njs4n3+8XEXFFtrhKl+rKzCC7dV1HvylDEeiD97CfZObeHez38kGdLkO2jXENE2iyRGuLGi2njr4ZphMRm9GMYw46yBcRcW5mtjIl6hpt7KQDAwO+745mEvwn0hzea0VmrpuZ6w3xs26bYbkYmM/2lcCngC9ExLsq9x9VEXFFGef098AFETG/9PReyJIza9uyXkScGhF3RsQdEfE/ETGl5RouKIcWu9YvC2X0g19FxHZdFzHeRcQGZTadH0TEP0XEZgNtsfz5gMeynueUITFExOsi4lPRLBbRdh2bRMRXgVMy8y8RsV1EtH3C9KO6CMvFjRHx7xExtfx8kJZnySjTkv61p7e/K/dGzyJYMYZTqdVExJzMvC0zT8/M0zLz9mhWTm3TmRHx3ojYYqzfL1bXHua30Ez9siNwIs1qZv+emV/qsq6ulG/FzwReSHOG798y82n1R43avqsfMG0OW4mIM2nGivZOS/TazHxxizVcTTMW8kaaIRkDRx3aPgnzKSw5nPknykIZfTKMqFXlhLttaP4NOvs/Ge/KF+mBOX4HPPoBlZlPabme3wA7Ac+gec/4KrB/Zg65LPEY1vFjmo6PD2TmTtFMIXZZG0dFhhi3u5Q2xu1GxDcy8/XRrLw5lSXDls4BPpyZg6emHOt6ZgN70CxI9uiwvmxxfYWIeCZwMktmbdkMeE0uu97BWO1/Es0Y7rOAF7Dkb3Y9miEST2+jjlLLwPvGUsbi/WJ1Dcy9J1atWZqz7ROr+kH5tvcEmh7d84DzM/PObqvqRj8cxipfINYHnluazgXubjuolr+RA2j+RjagOSFzvP6NDPmlbjx+eegHEXEQ8JPSm/rvNFN3fTTbXz7+0szcNSL+A/hDZn51oK3lOjobPhXNSdLQTGG3KfDNsn0IMD8zx3y8aulkeCnNHOUvpHyhHbg9M/+4nIeOVT2HDtWeLU8nW4aajvlUasvZ97to5j9+MvAHlvyfLAZOyMwvtFjLUDNwfTEzR73HfXU96a9fTqzqB78BdgN2oPk3uTsiLhyLX6ZVwF0R8TrgO2X7EGBRyzXsB7yFJXOJfoPmBLzPt1zHacDdwKV0M7ds3xgIxtGslDVpmLtr7H0wM2dHxJ7Ai2mWkz+eZlhXmxZHxNE0R6KeV47UrTnMY8bCvRGxIUuGT+1B814+5jLznLLPjw4aJ/qDiDi3jRpoTgD9CfAUll4UbCCktXrkoe1gPJQSlo+kZ8GQiPhSW6E5Mz8LfLZ8mfzMoC+3F7ZRQ4+hZuCaxRjMgLW69jD3xYlV/SQi1qGZDeC9wKaZuVbHJbUuIrYE/ht4Fs0b7QXAO3OIVazGsIbfAM8amKGjjJG8sIMhGf6NFNGsVPVJmt6SO2lW/LsmM7fvtLBxKpactP1fwBWZ+e3oYC7qiNgU+D/AxZl5Xnn/eEFmntRyHbvSfKHeHrgKmAwckJnLW15+LGq4hma+4RvL9tbAj1o+9H58Zh7Z1v4qdUyjmR1iO5aetra14B7NNJhr0gRDaBazeTgz39JWDaWOgYkV9qSZkeuTwPszs7Uvt22eDLq69jBfEBE7ZuYVXRfStYh4O83h/92A39OsGnVep0V156M0M3P8CZqTjGimCmpzqdtg6amyHmbpMZtt8W9kiY/SjEn8eQlqL6TppVA3/hARX6KZNutjZfhQ6yeol7P9P9WzfTPNPMBtuxo4lWaO3cXA/wK/bbmGf6bpxbyxbE8F3tZmAf0Qlouv0yxV/mmaISJvov338GcOCoS/iJanaS0GPsv2oRkGcVpEfKjlGi6LiD0GzcD1y7HY0WrVwxxLVtabSB+cWNUPIuJ9NONkL8nMh7qup0tD9VK13XNVTlw5lOYDEJohGidm5mfaqqHUcTXwd3iiGxExNzOnlw+cXTLzkYi4KDN377q28SgiHg/sTdO7fH0001btmJk/a7mO/YGPARvT/H20Pi1oqWM2zSHngeWHDwHWz8y2F31ai2a+cmjGzI7L4Y5Rpj2MnukoI+K8zHzucI8dxRouBQ7MzN+V7acA3+tgfP0PacYw/wNNp9zfgIvGone3UsM1NGO5B44Ub0mz7sYjjPJn2urWw9wvK+v1jcz8f13X0EfWiIj1B/Uwt/o3kJmfimYRmYEzvd+UmZe1WUPx0g722a/uLkOWzgO+FRF3AuP6y2WXMvOv9Cwak81CUF1Mefhx4BWZ2dqiV8vR2aJPsfzlh7eJZnW7NlfD7Bf3RbNgxvXlCO4faL5Utem9NL8HN9J8jmxF09PdtoNovtx+IjPvLl9u39dyDa1Nc7haBWbPatcwPkkzFOF7NEciDgKObbuIcrZ/q2f8D1GDfytL7EvTM3I0zWwAT2TJ1IMav+7og7AMLR5yHkLryw+vAt5NM6XaO2mGc+1Fc9SwFeXk051ojqL3zpLReo9/P3y5bfOzbLUakiENJ5oFKvaieZOZk5lXd1yS+kTptbuJZq7u/8rMbTouSR2KiM/STKX2v/TMttR2r2qbh5wrNUzIZtEO9YGIOCszX9h1HeONgVnSuFTGyj7QO7Y/Io6kmUnl4Mz8bmfFqXMR8fUhmjMz2zxJuC8Wf4qIm2mmdjsF+EWO4+AQEdOBD9AMg3j0KH2b539ExLE0R8JOYenFUzo9crm6MzBLGpci4lfAfmU2BCLiVcCHgfcA/5yZ+3RZn9QvyuIQrwAOpplr94fAyZl5fqeFdSAirqMZp3sFTS8/0PqqtWcN7HagqSkh92qrhvHIwCxpXOqdqzMiDgfeCrwsMxcOzJzRbYXqUkRMoZn/+Dk0weR84F2ZuaDTwjoWEesDnwVem5kTuq6nbRFxfmbu2XEN/8LSy8gnzUwqczNzXld1re5Wq5P+JGkFLIqIY4AtaE7227aE5c2Ax3VbmvrA12nGsw9M3/a60vbizirqUFkm+zU0M+xczBispLaKOKYsHDKH7sa27wZMp1kuPGjmQb4YeFtEfDczP95iLeOGPcySxqWy3PCRwAPA74D3A5fTLEbwgcz8doflqWMRMS8zdx6ubTyIiJuAecBs4PSBlUrHo4j4Js181FexZEhGq2PbI+KnwKsz856yvQ7wPeBVNGsubNdWLeOJPcySxqXMXAT834HtiLiQ5vD7xzLzus4KU7+4KyJeB3ynbB8CLOqwni7tlJl/6bqIPrHTwIIlHdqS5ov+gAeBrTLzbxExLheUaUPry41KUj/KzFsz87uGZRVvphl2cDvN3LIH0M3iEP1gvYg4NSLujIg7IuJ/yhjv8ehXZXrSLn271HFMGVb2S+A7EfEEmqXUNQYckiFJ0iARMQt496CVQT/R9rRy/SAizqQJaQML+ryO5qS/cTeeu8yLvQ3NnO33s2SGitamlSt17MaSFWPPz8y5be5/PDIwS5I0SERclpm7DNc2HjieuxERATwXWGYKOVdPXf05hlmSpGWtERHrD+phHq+fmY7npulGjohPZ+ZuXdei9o3XP35Jkmo+CVwQEd+jmef2IODYbkvqzJtpVsD8NM2/xQWM3/Hcv4qIZ2bmxV0XonY5JEOSpCGUk7v2ohknOiczx+UJVY7nXiIirga2BebTLEvdyRhmtc/ALEmSlsvx3EtExFZDtTuGefXntHKSJKlmjbIkNjC+x3OXYLwFsFe5/lfMUuPCuPyFlyRJI+Z47qLMezydZljG14E1gW/SLHqk1ZhDMiRJUpXjuRsRMQ/YBbh0YEhKRPzGMcyrP3uYJUlSVQnI4zIkD/JAmV4uAcrqehoHHHcjSZI0MrMj4kvAkyLircDPgS93XJNa4JAMSZKkEYqIFwP/WDZ/lplndlmP2uGQDEmSpJG7Alib5gTIKzquRS1xSIYkSdIIRMRbgIuA/YEDaFb+G3cLuIxHDsmQJEkagYi4Dnh2Zi4q2xsCF2Tmtt1WprFmD7MkSdLILAAW92wvBm7pqBa1yB5mSZKkEYiIk4AdgdNoxjDvSzNE47cAmfmp7qrTWPKkP0mSpJH5XfkZcFq5XLeDWtQie5glSZKkCnuYJUmSRiAizqIZirGUzNyrg3LUIgOzJEnSyLy35/ok4NXAQx3VohY5JEOSJOkxiohzMvP5XdehsWUPsyRJ0ghExAY9m2sA04FNOypHLTIwS5IkjcwlNGOYA3gQmA8c1mVBaocLl0iSJI3MvwE7Z+bWwDeAe4G/dluS2mBgliRJGpkPZuZfImJP4MXAicDx3ZakNhiYJUmSRubhcrkP8MXMPA14XIf1qCUGZkmSpJH5Q0R8CTgI+FFErIVZalxwWjlJkqQRiIjHA3sDV2Tm9RGxGbBjZv6s49I0xgzMkiRJUoWHESRJkqQKA7MkSZJU4cIlktSnImJDYE7Z3JTmDP2FZXv3zHyg577zgemZeVerRUrSOGBglqQ+lZmLgJ0BIuJDwD2Z+Ykua5Kk8cghGZK0ComIF0XEZRFxRUR8rUxr1Xv72hHxk4h4a0Q8odzn4vKYfct93hgR3y/3uz4iPt7Nq5GkVYOBWZJWHZNoVhZ7TWbuSHOU8Mie29cBfgB8OzO/DHwA+EVmPhN4IfD/IuIJ5b47A68BdgReExFbtPIKJGkVZGCWpFXHBOCmzPxt2Z4FPK/n9tOAr2fmSWX7H4EZETEPOJsmcG9ZbpuTmX/OzPuAq4Gtxrh2SVplGZgladVx7zC3/xJ4aURE2Q7g1Zm5c/nZMjOvKbfd3/O4h/GcFklaLgOzJK06JgFTI+LvyvbrgXN6bv8PYBFwXNn+KfCOgQAdEbu0VagkrU4MzJK06rgPeBPw3Yi4AngE+OKg+7wbmFRO5PsosCbwm4i4smxLklaQS2NLkiRJFfYwS5IkSRUGZkmSJKnCwCxJkiRVGJglSZKkCgOzJEmSVGFgliRJkioMzJIkSVKFgVmSJEmq+P9u7n6/xdURAQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams[\"figure.figsize\"] = (12,8)\n",
    "df_token_counts.sort_values(by = 'Count').plot.bar(x='Token', y='Count')\n",
    "plt.title('Most Used Words')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "a36107bd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16.85350262697023\n",
      "11.93713469281725\n",
      "16.761093182734975\n",
      "19.152626811594203\n",
      "19.20689655172414\n"
     ]
    }
   ],
   "source": [
    "# WORD-COUNT\n",
    "df['word_count'] = df['OriginalTweet'].apply(lambda x: len(str(x).split()))\n",
    "\n",
    "print(df[df['Sentiment']==\"Positive\"]['word_count'].mean()) #Positive tweets\n",
    "print(df[df['Sentiment']==\"Neutral\"]['word_count'].mean()) #Neutral tweets\n",
    "\n",
    "print(df[df['Sentiment']==\"Negative\"]['word_count'].mean()) #Negative tweets\n",
    "print(df[df['Sentiment']==\"Extremely Positive\"]['word_count'].mean()) #Extremely Positive tweets\n",
    "print(df[df['Sentiment']==\"Extremely Negative\"]['word_count'].mean()) #Extremely Negative tweets\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "e5a92daf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA+MAAAFTCAYAAABS0d2MAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA/wklEQVR4nO3de5hlVX3n//cnoMg1gjQEuhubaHsBxqC2SKIxREJAY4SZiU5j1DbBtPHBRCfOKOj8gmbSCbl4z2BChNCOAunxBnG8ESKgESENolxaQisE2m7pVnRAk6C0398fe5V9qK6qrqquOnXq1Pv1POc5e699OeucWqv2/u619tqpKiRJkiRJUv/8xFxnQJIkSZKkhcZgXJIkSZKkPjMYlyRJkiSpzwzGJUmSJEnqM4NxSZIkSZL6zGBckiRJkqQ+MxiXJKlPkrwlyQfmOh+SJGnuGYxLkhasJGcn+cSotDvGSVvZ39z1T5JK8vg+f+ZFSf6wn58pSdIgMRiXJC1k1wDPSrIHQJKfAh4BPG1U2uPbupOWZM8ZzutuG8Q8SZK0UBmMS5IWsn+iC76PbfPPAT4L3D4q7WtVtTnJ4UkuT3Jfko1JfmtkR60L+oeSfCDJ/cArkhyZ5OokDyS5Aji4Z/1HtXW/neS7Sf4pyaFjZTLJXa0V/7Yk30nyN0ke1bP8BUluavv5QpKnjNr2jUm+Anx/dECeZOQiw5eTfC/Jf2l5/s9t+bNby/nz2/wvJbmpZ/vfTLKh5evTSR7bs+xJSa5ov9ftSV7c0lcDvw68oX3m3+3i7yRJ0tAxGJckLVhV9QPgOrqAm/b+OeDzo9JGAtZLgE3A4cCvAX+U5MSeXZ4KfAh4NPBB4GLgBrog/H8Cq3rWXQX8JLAUeAzw28C/TZDdXwdOBh4HPAH4HwBJngZcCLyq7eevgMuT7NWz7enArwCPrqqHRv0GI9/zZ6pqv6r6W+Bq4ISe7/914Bd65q9un30a8CbgPwGL6H67S9qyfYEr2m9wSMvDeUmOrqrz2+/zp+0zf3WC7y1J0lAyGJckLXRXsyPw/nm6gPJzo9KuTrIUeDbwxqr696q6CXgf8LKefV1bVR+rqh/RBafPAP6/qnqwqq4BeluAf0gXPD++qrZX1Q1Vdf8E+fyLqrqnqu4D1tAFtwC/BfxVVV3X9rMWeBA4vmfbd7dtJwr2R/8mvcH3H/fM/0JbDt0FgD+uqg0tyP8j4NjWOv4C4K6q+puqeqiqbgQ+THcRQ5KkBc9gXJK00F0DPDvJgcCiqroD+ALwcy3tmLbO4cB9VfVAz7b/Aizumb+nZ/pw4DtV9f1R64/438CngUuTbE7yp0keMUE+e/f9L23/AI8FXt+6qH83yXfpWtsPH2fbybgWeELrNn8s8H5gaZKDgePY0VPgscC7ej73PiB0v8ljgWeOytevAz81xbxIkjSUHMhFkrTQXUvXXXw18I8AVXV/ks0tbXNV3ZnkIeCgJPv3BORHAN/o2Vf1TG8BDkyyb09AfsTIOlX1Q+CtwFuTLAM+QXev+gXj5HNpz/QRwOY2fQ+wpqrWTPAda4JlO69c9a9JbgBeC9xSVT9I8gXg9+jun//WqM/+4Oh9tNbxq6vqpJnIkyRJw8aWcUnSgta6bq+nCzQ/17Po8y3tmrbePXQt5n/cBl97CnAG3b3PY+33X9p+35rkkUmeDfz43ugkv5jkP7RR2++n67a+fYKsnplkSZKD6O7T/tuW/tfAbyd5Zjr7JvmVJPtP4We4F/jpUWlXA69hR5f0q0bNA/wlcHaSo9t3+skkL2rLPk7Xuv6yJI9or2ckefIEnylJ0oJhMC5JUhdgHkIXgI/4XEvrfaTZ6cAyulbpjwLnVNUVE+z3JcAz6bpvn0PX3XvET9EN9nY/sKHl4QMT7Oti4DN0g6l9HfhDgKpaT3ff+F8A3wE2Aq+YYD9jeQuwtnUnf3FLuxrYnx3ff/Q8VfVR4E/outrfD9wCPK8tewD4ZWAl3e/1zbbuyMByFwBHtc/82BTzK0nSvJcqe4lJkjTIktwFvLKq/n6u8yJJkmaGLeOSJEmSJPWZwbgkSZIkSX1mN3VJkiRJkvrMlnFJkiRJkvrMYFySJEmSpD4zGJckSZIkqc8MxiVJkiRJ6jODcUmSJEmS+sxgXJIkSZKkPjMYHwBJ3pTkfRMs//Ukn+lnnqRhk+SqJK+c63xIsy3JJ5Osmut8SDMlyQlJNs11PsBzNs2NQaoDu8tj1MMZjE9DkruS/FuS7yW5N8nfJNlvuvurqj+qqle2fS9LUkn27Fn+war65ZnIe68kFyX5w5ne7y4+c2j+mWiHVifuTbJvT9ork1w1Q/uvJI+fiX2Ns/+3JPnAbO1/nM/cqa5rfprt8r+Lz96p7FbV86pq7Sx81qzWw3E+s+/HqWEy6nxl5PUXk9huaI7VnrPt1mfO+3JgHfAYNZtmol4ajE/fr1bVfsDTgGcA/2OO8yPNtT2B187FBxvQagDMWfmXduFXq2q/ntdrZmKn8+z/rudsC5t1wGPU4KoqX1N8AXcBv9Qz/2fAx9v0C4Fbge8CVwFP7lnvjcA3gAeA24ETW/pbgA+06buBAr7XXj8LvAL4fFv+l8Cfj8rPZcDvtenDgQ8D24A7gd8d5zusBn4I/KB9zt8BvwH8Xc86G4F1PfP3AMe26ScBVwD3te/y4p719gL+vH2Xe1ue9wb2Bf4N+FHP9zscOA5YD9zf1n/7XP+NfU2rTpzVysOjW9orgat61pmozFwFvLJnvrfMX9PqxPdbmfkvwAnAplanvgn8b+BA4OOt7H+nTS8Z7zN60k9p9eCHbf9fBn4RuLlnnb8Hru+Z/zxwWpset87RXfA8C/ga8G1gHXBQWzZWXX88cDXw/4BvAX87139bX30p/4+h+x98P/BPwB+OlP+2/F10/3/vB24Afn68sttb1un+F38XOKZnX4vo/g8f0uZfANzU1vsC8JRxvuNY9fBq4D+35c9uy5/f5n8JuKln+98ENrS6+Wngsbv6bRjjONXSxzyW+hq3bP7SOMveC3yoZ/5PgCsZ/1j9FuBDwAdaWXwl8JPABcCW9jf5Q2CPtr9XAP8IvKOVr68DP9fS7wG2Aqt6Pn/Mc4e27ARgU5v+78CHR32X9wDvnMxvgOdsC+qcbfTffwHXAY9RA3qMmvNKMh9f9FRsYCndP/L/CTyhFYKTgEcAb6D75/hI4ImtoB7etlsGPK5Nv4Ud/9iXtcKyZ8/nvYId/9if0/aTNn9gK7SH05343wD8fvvMn6ar+CeP8z0uAv6wZ/6nW2H/CeAw4F+Ab/Qs+05btm/Lw2/QXWl7Gl3gcHRb953A5cBBwP50FfiP27ITaP9Mej73WuBlbXo/4Pi5/hv7ml6dAD4yUqbo+Uc/iTJzFeME422+gMf3zJ8APER34NyL7sThMcB/BvZp5e7/AB/r2eZhnzEq/z+ug23+Ua1eHdzy+01gc9vv3m3ZY3ZV54DXAV8ElrR8/hVwSVu2jJ3r+iXAm9t+HwU8e67/tr76Uv4vba99gKPaur3l/6WtvO0JvL6Vx0eNVXZHl3XgQmBNz7IzgU+16afRnQw+E9gDWNW+y17jfM/R9fAPgPe06TfRXXT6k55l72rTp9EdC5/cvsP/AL4wyd/mIh5+nBr3WOpr/LI5zrJ9gH+m+3/78+13X9KWncDOx+q30J14ntb+R+0NfIzu/9q+wCHA9cCr2vqvoPs//RutfP0hXZDxv+j+H/4y3cnqfm39dzKJcwe685PvsyOo2LOV46fv6jfAc7YFd86GdeDHvwEeowbyGGU39en7WJLv0rWQXQ38Ed1VmP9bVVdU1Q/prm7tTXcVbDtdxTsqySOq6q6q+to0PvdzdIXt59v8rwHXVtVmuq5Xi6rqD6rqB1X1deCvgZWT2XFb/wHgWOAX6K4MfSPJk9r856rqR3RXqe6qqr+pqoeq6ka6K7u/liTAbwH/taruq6oH2m8zUR5+CDw+ycFV9b2q+uLkfw4NmN8HfifJolHp45aZ3fisHwHnVNWDVfVvVfXtqvpwVf1rK3dr6MrtlFXVv9Nd+X8OsAL4Cl1dfxZwPHBHVX2bXde5VwFvrqpNVfUg3UHp1ybo2vZD4LF0/8T/vao+P538a85Mufwn2YPuItI5rezeBjzsXrqq+kAr3w9V1dvojiVPnGSeLgZO75l/SUuD7n/1X1XVdVW1vbp7+B6kK+OTcTU76thzgD/umf+Fthy6evDHVbWhqh6iOyYcm+SxTP1/w0wdSxeSjyX5bs/rtwCq6l/pTqLfTtfS9ztVtat7ZK+tqo+1c4EDgOcBr6uq71fVVroWwN7j/Z3tb7sd+Fu6YPgP2v/tz9C1KD1+KucOVbWFrhXsRS3pFOBbVXXDrn4DPGdbqOds1oGOx6gBPEbNp3sdBs1pVfX3vQlJDqe7MglAVf0oyT3A4qq6Ksnr6E7Gj07yabpuSpun8qFVVUkupSu419AV2pGBER4LHN4OOCP2oDsYTNbVdFffRrrLfpeuwP4sOwrtY4FnjvqcPem6Ci+iu3J2Q/d/BYC0fIznDLorVF9Ncifw1qr6+BTyrAFRVbck+Thdd6gNPYsmKjPTta0FzQAk2YfuIHgKXesDwP5J9mgHwakaqQub2vR36OrCgzy8LkxU5x4LfDTJj3qWbwcOHecz30DXYnN9ku8Ab6uqC6eRd82BaZb/RW36np5lvdMkeT1dK8bhdCf2B9D12piMfwD2TvJMutaKY4GP9uRrVZLf6Vn/ke1zJuNa4AlJDm37fSHw1iQH03Vlvabnc96V5G29XwtYzBT/N1TVxpk4li4wO52vjKiq65N8na5Fb90k9tVbNh9L16K8ped4/xOj1rm3Z/rf2meOTtuPqZ87rAVeTRe8vpRdH0s8Z1vY52zWATxGMaDHKFvGZ9Zmuj8aAO0q11K6+waoqour6tltnaLrYjtaTeJzLqG7WvVYuq4bH27p99BdgXt0z2v/qnr+OPsZ67NG/rH/fJseuarUewXpHuDqUZ+zX1W9mq7rxr/Rdd8YWfaT1Q2cMuZnVtUdVXU63T/CPwE+1Dvio+adc+iuZi7uSZuozEDX3WqfnvV/ahKfM7osvZ7uSuwzq+oAuqug0P1Dneq+YEddeA4T14WJ6tw9wPNGLX9UVX1jrM+sqm9W1W9V1eF0V2rP6/fIoNptUy3/2+i6MS7pWX/pyESSn6e7/+zFwIFV9Wi6MQVGyvWEx4zWerOOLhh4Cd29sg/05GvNqHztU1WXTOaLtlalG+gGBbqlqn5Ad0/f7wFfq6pv9XzOq0Z9zt5V9YVd/DZjfr9JHks1CUnOpGvF2Ux3MXDEeOWqN/0euouTB/f87Q6oqqOnkZVdnTuM9jHgKUmOoWu5+uA0PtNzNs/ZFmId8Bg1YMcog/GZtQ74lSQnJnkEXXDwIPCFJE9M8twkewH/Tlfhxmqt20bX/fanx/uQqvpSW+99wKer6rtt0fXA/UnemGTvJHskOSbJM8bZ1b1jfM7VdINX7V1dV53P0bU0Pgb4Ulvn43RXml6W5BHt9YwkT26V6q+BdyQ5BCDJ4iQn93zmY5L85MgHJnlpkkVt25HvMp2WTA2AqtpI1xXrd3uSxy0zbflNwH9Ksk8LPs8Ytduxyupo+9PVq+8mOYjugDNZ9wLLkvT+T/wCXXB/HN3gbbfSrpCy42rqrurcXwJr2kkYSRYlObUt26muJ3lRkpED3nfo/olbF+aRqZb/6nptfAR4Syv/TwJe3rPt/nQnQtuAPZP8Pl2rw4ixyu5oF9N1yf11dnT/g+5/9W8neWY6+yb5lST7j7Of8Y4Zr2HHif9Vo+ahqwdnJzkaIMlPJhnpXrmr/w0P+8wpHEu1C0meQHcP60uBlwFvSHJsW7zTsXq06rrKfgZ4W5IDkvxEksclmfLtQZM4dxi9/r/TDaR1Md3/57un+pl4zrbgz9kWYh3wGDV4xyiD8RlUVbfTVej30F3h+lW6xyn8gO6q27kt/Zt0VxTfNMY+/pXuXtd/THdfy3j3RVxCNxjDxT3bbm+feSzdqJzfovvnP94/kgvo7mn4bpKPtX38M92IgJ9r8/fTDSjyj23/tCtWv0x3H8vm9n1GBtKC7grZRuCLSe6nG4n6iW3br7a8f7197uF0B45bk3yPbkTGldXT/Vjz0h/QDXoBTKrMvIPuvql76bpejb7C+xZgbSszLx7nM99Jd7/ft+gGTfvUFPL7f9r7t5Pc2PL8feBG4NZWh6Hr8vQv1d0XNpk69y66wVg+k+SBlq9ntm3HquvPAK5rdeFy4LVVdecUvocGw1TL/2voyszIkwEuoQsKoLsP9JN0gwz9C93BvbeL4E5ld7Squo6u98nhbV8j6evpWkj+gu7iz0a6AYfG8xZ2rodX052MXTPOPFX10fZ9L23HhFvo7rOczG8z+jg1qWOpHubv8vBnLH803bgVH6AbzOjLVXUH3e/4v5PsNc6xeiwvp+s2ehtdGfoQ3eBS0zHuucM41gL/gWne7uQ524I6Z7MOPJzHqAE6Ro2M7ihJkgZAkj8BfqqqVs11XqRBleQI4Kt0deX+uc6P1G9zVQc8Rs0sW8YlSZpDSZ6U5CmtG95xdLdpfHRX20kLVevy+nvApQbiWoj6WQc8Rs0uR1OXJGlu7U/X7e9wumeqvg24bE5zJA2odIOF3UvXJfaUOc6O1HdzUAc8Rs0iu6lLkiRJmlFJLqQb6XtrVR0zatl/A/6M7lnr32ppZ9O1um4HfreqPt3Snw5cRDcuzCfoxlMxgNFQsJu6JEmSpJl2EWO03CZZCpwE3N2TdhTdIFlHt23OSzLyjO33AquB5e1ljwgNDYNxSZIkSTOqqq4B7htj0Tvonund27p9Kt39zw+2p4hsBI5LchhwQFVd21rD3w+cNrs5l/pn4O8ZP/jgg2vZsmVznQ2JG2644VtVtWgu82B90CCZ6zphfdAgsT5IO4xXH5K8EPhGVX05Se+ixXSP/xyxqaX9sE2PTh9TktV0rejsu+++T3/Sk5407e8gzZSJjg8DH4wvW7aM9evXz3U2JJL8y1znwfqgQTLXdcL6oEFifZB2GKs+JNkHeDPdc5t3WjxGWk2QPqaqOh84H2DFihVlndAgmOj4MPDBuCRJkqR573HAkcBIq/gS4Mb2uKxNwNKedZcAm1v6kjHSpaHgPeOSJEmSZlVV3VxVh1TVsqpaRhdoP62qvglcDqxMsleSI+kGaru+qrYADyQ5Pl0E/3J8rJaGiMG4JEmSpBmV5BLgWuCJSTYlOWO8davqVmAdcBvwKeDMqtreFr8aeB/doG5fAz45qxmX+shu6pIkSZJmVFWdvovly0bNrwHWjLHeeuCY0enSMLBlXJqCJEuTfDbJhiS3JnltS39Lkm8kuam9nt+zzdlJNia5PcnJPelPT3JzW/bujBpWVJIkSdLwsmVcmpqHgNdX1Y1J9gduSHJFW/aOqvrz3pWTHAWsBI4GDgf+PskTWter99I9fuOLwCeAU7DrlSRJkrQg2DIuTUFVbamqG9v0A8AGJnjeJXAqcGlVPVhVd9Ld73RcksOAA6rq2qoq4P3AabObe0nSbEpyYZKtSW4Zlf47rXfUrUn+tCfdnlOStIAZjEvTlGQZ8FTgupb0miRfaSdjB7a0xcA9PZttammL2/To9LE+Z3WS9UnWb9u2bSa/giRpZl1E18vpx5L8It2F2adU1dHAn7f03p5TpwDnJdmjbTbSc2p5ez1sn5Kk4WAwLk1Dkv2ADwOvq6r76U6cHgccC2wB3jay6hib1wTpOydWnV9VK6pqxaJFi3Y365KkWVJV1wD3jUp+NXBuVT3Y1tna0u05JUkLnMG4NEVJHkEXiH+wqj4CUFX3VtX2qvoR8NfAcW31TcDSns2XAJtb+pIx0iVJw+UJwM8nuS7J1Ume0dLtOSVJC5zBuDQF7b69C4ANVfX2nvTDelb7j8DI/YKXAyuT7JXkSLruhtdX1RbggSTHt32+HLisL19CktRPewIHAscD/x1Y1/7v23NKkhY4R1OfR6Y7fEuNeQjXND0LeBlwc5KbWtqbgNOTHEt3wnQX8CqAqro1yTrgNrqR2M9sI6lD13XxImBvulHU53Ak9d0ZG8gCJs2Ii3ejHr7EejjANgEfaV3Or0/yI+Bg7Dk1a/LW6delOse6pCG0O2NAGkjMKoNxaQqq6vOMHbl+YoJt1gBrxkhfDxwzc7mTJA2gjwHPBa5K8gTgkcC36HpOXZzk7XSPvhzpObU9yQNJjqcbIPTlwHvmJOeSpFllML4AeDFMkqTZl+QS4ATg4CSbgHOAC4EL2+POfgCsaq3k86TnlCRpthiMS5IkzYCqOn2cRS8dZ317TknSAuYAbpIkSZIk9ZnBuCRJkiRJfWYwLkmSJElSn3nPuCRJkoSPRZPUX7aMS5IkSZLUZ7aMS5IkSdIg251nFWtg2TIuSZIkSVKfGYxLkiRJktRnBuOSJEmSJPWZwbgkSZIkSX1mMC5JkiRJUp8ZjEuSJEmS1GcG45IkSZIk9ZnBuCRJkiRJfWYwLkmSJElSnxmMS5IkSZLUZ7sMxpMsTfLZJBuS3JrktS39oCRXJLmjvR/Ys83ZSTYmuT3JyT3pT09yc1v27iSZna8lSZIkSdLgmkzL+EPA66vqycDxwJlJjgLOAq6squXAlW2etmwlcDRwCnBekj3avt4LrAaWt9cpM/hdJEmSJEmaF3YZjFfVlqq6sU0/AGwAFgOnAmvbamuB09r0qcClVfVgVd0JbASOS3IYcEBVXVtVBby/ZxtJkiRJQyLJhUm2JrmlJ+3Pknw1yVeSfDTJo3uW2bNWC86U7hlPsgx4KnAdcGhVbYEuYAcOaastBu7p2WxTS1vcpkenS5IkSRouF7FzL9grgGOq6inAPwNngz1rtXBNOhhPsh/wYeB1VXX/RKuOkVYTpI/1WauTrE+yftu2bZPNoiRJkqQBUFXXAPeNSvtMVT3UZr8ILGnT9qzVgjSpYDzJI+gC8Q9W1Uda8r2tgtDet7b0TcDSns2XAJtb+pIx0ndSVedX1YqqWrFo0aLJfhdJkiRJ88NvAp9s0/as1YI0mdHUA1wAbKiqt/csuhxY1aZXAZf1pK9MsleSI+m6k1zfurI/kOT4ts+X92yzYCTTf0mSJEnzXZI30w0S/cGRpDFWm1LP2rZfe9dqXplMy/izgJcBz01yU3s9HzgXOCnJHcBJbZ6quhVYB9wGfAo4s6q2t329GngfXdeTr7HjapgkSdK8NtaAVT3L/luSSnJwT5oDVmnBSbIKeAHw663rOcxAz1qwd63mn8mMpv75qkpVPaWqjm2vT1TVt6vqxKpa3t7v69lmTVU9rqqeWFWf7ElfX1XHtGWv6amAkqT55xFJPptkQ5Jbk7wWIMlBSa5Ickd7P3BkA4MPDbmLGGNwqSRL6Rou7u5Jc8AqLThJTgHeCLywqv61Z5E9a7UgTWk0dUmSRnl9VT0ZOB44swUYZwFXVtVy4Mo2b/ChoTfWgFXNO4A38PDutQ5YpaGW5BLgWuCJSTYlOQP4C2B/4IrW2/YvwZ61Wrj2nOsMSJLmrR9W1Y0AVfVAkg10A+ucCpzQ1lkLXEXXEvLj4AO4M8lI8HEXLfgASDISfHjCpXkvyQuBb1TVl0d1+FhMN5r0iJGBqX6IA1ZpCFTV6WMkXzDB+muANWOkrweOmcGsSQPDYFyStNuSLAOeClwHHNq6FlJVW5Ic0lbb7eAjyWq6FnSOOOKIGfwG0sxLsg/wZuCXx1o8RtqUHwWL9UGS5i27qUuSdkuS/egef/m6qrp/olXHSJtS8OHgPJpnHgccCXy59QBZAtyY5KfwUbCStOAZjEuSpi3JI+gC8Q9W1Uda8r3tvlfa+9aWPiOj5UrzRVXdXFWHVNWyqlpGV9afVlXfxAGrJGnBMxiXJO2OC4ANVfX2nrTLgVVtehU7AgmDDw21cQasGpMDVkmSvGdckjRd+wEvA25OclNLexNwLrCuBSJ3Ay+CLvhIMhJ8PMTOwcdFwN50gYfBh+adcQas6l2+bNS8A1ZJ0gJmMC5Jmq7vVdV4zwM/caxEgw9JkqSO3dQlSZIkSeozg3FJkiRJkvrMburTkPE6ZUqSJEmSNAm2jEuSJEmS1GcG45IkSZIk9ZnBuCRJkiRJfWYwLkmSJElSnzmAmyRJ893FuzGy6Etq5vIhSZImzZZxaQqSLE3y2SQbktya5LUt/aAkVyS5o70f2LPN2Uk2Jrk9yck96U9PcnNb9u7EcfolSZKkhcJgXJqah4DXV9WTgeOBM5McBZwFXFlVy4Er2zxt2UrgaOAU4Lwke7R9vRdYDSxvr1P6+UUkSZIkzR2DcWkKqmpLVd3Yph8ANgCLgVOBtW21tcBpbfpU4NKqerCq7gQ2AsclOQw4oKquraoC3t+zjSRJkqQhZzAuTVOSZcBTgeuAQ6tqC3QBO3BIW20xcE/PZpta2uI2PTpdkiRJ0gJgMC5NQ5L9gA8Dr6uq+ydadYy0miB9rM9anWR9kvXbtm2bemYlSZIkDRyDcWmKkjyCLhD/YFV9pCXf27qe0963tvRNwNKezZcAm1v6kjHSd1JV51fViqpasWjRopn7IpIkSZLmjI82k6agjXh+AbChqt7es+hyYBVwbnu/rCf94iRvBw6nG6jt+qranuSBJMfTdXN/OfCePn0NSbsy3UeF+ZgwSZI0SQbj0tQ8C3gZcHOSm1ram+iC8HVJzgDuBl4EUFW3JlkH3EY3EvuZVbW9bfdq4CJgb+CT7SVJkiRpATAYl6agqj7P2Pd7A5w4zjZrgDVjpK8Hjpm53EmSJEmaL7xnXJIkSZKkPjMYlyRJkiSpzwzGJUmSJEnqM4NxSZIkSZL6zGBckiRJkqQ+MxiXJEmSJKnPDMYlSZJmQJILk2xNcktP2p8l+WqSryT5aJJH9yw7O8nGJLcnObkn/elJbm7L3p1kvEdqSpLmMYNxSZKkmXERcMqotCuAY6rqKcA/A2cDJDkKWAkc3bY5L8kebZv3AquB5e01ep/SwBvn4tRBSa5Ickd7P7BnmRentODsOdcZkCRpaFzsOeJCVlXXJFk2Ku0zPbNfBH6tTZ8KXFpVDwJ3JtkIHJfkLuCAqroWIMn7gdOAT85u7qUZdxHwF8D7e9LOAq6sqnOTnNXm3zjq4tThwN8neUJVbWfHxakvAp+guzhlfeiX6V77qJrZfAwpW8YlSZL64zfZEUQsBu7pWbappS1u06PTd5JkdZL1SdZv27ZtFrIrTV9VXQPcNyr5VGBtm15Ld6FpJP3Sqnqwqu4ERi5OHUa7OFVVRRfYn4Y0JAzGJUmSZlmSNwMPAR8cSRpjtZogfefEqvOrakVVrVi0aNHMZFSaXYdW1RaA9n5IS9/ti1PSfGQ3dWlo2D1WkgZRklXAC4ATW+sedEHF0p7VlgCbW/qSMdKlYbbbF6eg6y1C16WdI444YmZyJs0iW8YlSZJmSZJTgDcCL6yqf+1ZdDmwMsleSY6kG6jt+tZa+ECS49tAVS8HLut7xqXZcW/rek5739rSZ+TilL1FNN/YMi5JkjQDklwCnAAcnGQTcA7d6Ol7AVe0QaC/WFW/XVW3JlkH3EbXff3MNlgVwKvpBr/am+4e8wU5WFXeao+vIXQ5sAo4t71f1pN+cZK30w3gNnJxanuSB5IcD1xHd3HqPf3PtjQ7DMYlSZJmQFWdPkbyBROsvwZYM0b6euCYGcya1HfjXJw6F1iX5AzgbuBFAF6c0kJlMC5JkiRpRo1zcQrgxHHW9+KUFhzvGZckSZIkqc8MxiVJkiRJ6jODcUmSJEmS+sxgXJIkSZKkPjMYlyRJkiSpzwzGJUmSJEnqM4NxSZIkSZL6zOeMS9pN2Y1ta8ZyIUmSJM0ntoxLkiRJktRnuwzGk1yYZGuSW3rS3pLkG0luaq/n9yw7O8nGJLcnObkn/elJbm7L3p1kd5rTJEmSJEmatybTMn4RcMoY6e+oqmPb6xMASY4CVgJHt23OS7JHW/+9wGpgeXuNtU9JkiRJkobeLoPxqroGuG+S+zsVuLSqHqyqO4GNwHFJDgMOqKprq6qA9wOnTTPPkiRJkiTNa7tzz/hrknyldWM/sKUtBu7pWWdTS1vcpkenS5IkSZK04Ew3GH8v8DjgWGAL8LaWPtZ94DVB+piSrE6yPsn6bdu2TTOLkiRJkiQNpmkF41V1b1Vtr6ofAX8NHNcWbQKW9qy6BNjc0peMkT7e/s+vqhVVtWLRokXTyaIkqQ8c5FOSJGl6phWMt3vAR/xHYOQk7HJgZZK9khxJN1Db9VW1BXggyfHtBOvlwGW7kW9J0mC4CAf5lCRJmrI9d7VCkkuAE4CDk2wCzgFOSHIsXVfzu4BXAVTVrUnWAbcBDwFnVtX2tqtX05207Q18sr0kSfNYVV2TZNkkV//xIJ/AnUlGBvm8izbIJ0CSkUE+PU5IkqShtctgvKpOHyP5ggnWXwOsGSN9PXDMlHInSZqvXpPk5cB64PVV9R26gTu/2LPOyGCeP8RBPiVJ0gKzO6OpS5I0llkb5NMBPiVJ0rAwGJckzajZHOTTAT4lSdKwMBiXJM0oB/mUJEnatV3eMy5J0ngc5FOSJGl6DMYlSdPmIJ/SDkkuBF4AbK2qY1raQcDfAsvoLk69uA1oSJKzgTOA7cDvVtWnW/rT2XFx6hPAa6tqzHEUJEnzl93UJUmSZsZFwCmj0s4Crqyq5cCVbZ4kRwErgaPbNucl2aNt815gNd2tHMvH2KckaQjYMi5J0kJ28ViD2U/SS2ys7VVV1yRZNir5VLpbOQDWAlcBb2zpl1bVg8CdSTYCxyW5Czigqq4FSPJ+4DS8dUOSho4t45IkSbPn0DZIIe39kJa+GLinZ71NLW1xmx6dLkkaMgbjkiRJ/TdWl4SaIH3nHSSrk6xPsn7btm0zmjlJ0uwzGJckSZo994487q+9b23pm4ClPestATa39CVjpO+kqs6vqhVVtWLRokUznnFJ0uwyGJemKMmFSbYmuaUn7S1JvpHkpvZ6fs+ys5NsTHJ7kpN70p+e5Oa27N3t+cqSpOFyObCqTa8CLutJX5lkryRH0g3Udn3ryv5AkuPbceHlPdtIkoaIwbg0dRcx9si276iqY9vrE+BouZK0kCS5BLgWeGKSTUnOAM4FTkpyB3BSm6eqbgXWAbcBnwLOrKrtbVevBt4HbAS+hoO3SdJQcjR1aYrGGS13PI6WK0kLRFWdPs6iE8dZfw2wZoz09cAxM5g1aaAk+a/AK+nGQ7gZ+A1gH+BvgWXAXcCLq+o7bf2zgTOA7cDvVtWn+59raebZMi7NnNck+Urrxn5gS9vt0XIdoEeSJA2LJIuB3wVWVNUxwB50vQjPAq6squXAlW1+V70MpXnNYFyaGe8FHgccC2wB3tbSd3u0XAfokSRJQ2ZPYO8ke9K1iG+m6024ti1fS9djEHp6GVbVnXS3bxzX3+xKs8NgXJoBVXVvVW2vqh8Bf82Og8Ruj5YrSZI0LKrqG8CfA3fTNWD8v6r6DHBoG8CQ9n5I22S8XobSvGcwLs2AkcfWNP8RGBlp3dFyJUmSmnYr36nAkcDhwL5JXjrRJmOkjdmb0Fv7NN84gJs0RW203BOAg5NsAs4BTkhyLN3B4S7gVdCNlptkZLTch9h5tNyLgL3pBm5z8DZJkjTsfgm4s6q2AST5CPBzwL1JDquqLa2RY2tbf7xehjupqvOB8wFWrFgxZsAuDRKDcWmKxhkt94IJ1ne0XEmSpM7dwPFJ9gH+je5pA+uB7wOr6B7/t4odPQYvBy5O8na6lvTlwPX9zrQ0GwzGJUmSJPVFVV2X5EPAjXS9Br9E15q9H7AuyRl0AfuL2voT9TKU5jWDcUmSJEl9U1Xn0N3m1+tBulbysdYfs5ehNN85gJskSZIkSX1mMC5JkiRJUp/ZTV2SJEmSZlvGekqbFjJbxiVJkiRJ6jODcUmSJEmS+sxgXJIkSZKkPluw94x7y8bk7M7vVDVz+ZAkSZKkYWLLuCRJkiRJfWYwLkmSJElSnxmMS5IkSZLUZwbjkiRJkiT1mcG4JEmSJEl9ZjAuSZIkSVKfLdhHm0mSJGl25a0+S1aSxmPLuCRJkiRJfWYwLkmSJElSnxmMS5IkzbIk/zXJrUluSXJJkkclOSjJFUnuaO8H9qx/dpKNSW5PcvJc5l2SNDsMxiVJkmZRksXA7wIrquoYYA9gJXAWcGVVLQeubPMkOaotPxo4BTgvyR5zkXdJ0uwxGJckSZp9ewJ7J9kT2AfYDJwKrG3L1wKntelTgUur6sGquhPYCBzX3+xKkmabwbgkSdIsqqpvAH8O3A1sAf5fVX0GOLSqtrR1tgCHtE0WA/f07GJTS5MkDRGDcUmSpFnU7gU/FTgSOBzYN8lLJ9pkjLQaY7+rk6xPsn7btm0zk1lJUt8YjEuSJM2uXwLurKptVfVD4CPAzwH3JjkMoL1vbetvApb2bL+Erlv7w1TV+VW1oqpWLFq0aFa/gCRp5hmMS5Ikza67geOT7JMkwInABuByYFVbZxVwWZu+HFiZZK8kRwLLgev7nGdJ0izbc64zIEmSNMyq6rokHwJuBB4CvgScD+wHrEtyBl3A/qK2/q1J1gG3tfXPrKrtc5J5SdKsMRiXJEmaZVV1DnDOqOQH6VrJx1p/DbBmtvMlSZo7dlOXJEmSJKnPDMYlSZIkSeozu6lLkiRJuylvHeuJdJNT5+z05DpJC4At45IkSZIk9dkug/EkFybZmuSWnrSDklyR5I72fmDPsrOTbExye5KTe9KfnuTmtuzd7dEekiRJkiQtOJNpGb8IOGVU2lnAlVW1HLiyzZPkKGAlcHTb5rwke7Rt3guspntW5vIx9ilJkiRpyCV5dJIPJflqkg1JfnY6jX3SfLfLYLyqrgHuG5V8KrC2Ta8FTutJv7SqHqyqO4GNwHFJDgMOqKprq6qA9/dsI0mSJGnheBfwqap6EvAzwAam19gnzWvTvWf80KraAtDeD2npi4F7etbb1NIWt+nR6WNKsjrJ+iTrt23bNs0sSpJmm7cySZKmIskBwHOACwCq6gdV9V2m2NjXzzxLs2WmB3Ab6+SpJkgfU1WdX1UrqmrFokWLZixzkqQZdxHeyiRJmryfBrYBf5PkS0nel2Rfpt7YtxMb9DTfTDcYv7d1Pae9b23pm4ClPestATa39CVjpEuS5jFvZZIkTdGewNOA91bVU4Hv0y7ajmPSjXo26Gm+mW4wfjmwqk2vAi7rSV+ZZK8kR9K1blzfrm49kOT41vXw5T3bSJKGy6zeyiRJmtc2AZuq6ro2/yG64HyqjX3SvDeZR5tdAlwLPDHJpiRnAOcCJyW5AzipzVNVtwLrgNuATwFnVtX2tqtXA++jawn5GvDJGf4ukqTBttu3MtkFUZLmt6r6JnBPkie2pBPpYocpNfb1McvSrNlzVytU1enjLDpxnPXXAGvGSF8PHDOl3EmS5qN7kxxWVVtm+lamqjofOB9gxYoV4449IkkaaL8DfDDJI4GvA79B10i4rjX83Q28CLrGviQjjX0P8fDGPmle22UwLknSFI20bpzLzq0bFyd5O3A4O25l2p7kgSTHA9fR3cr0nv5nW5LUD1V1E7BijEVTauyT5juDcUnStLVbmU4ADk6yCTiHLgifauvGq+lGZt+b7jYmb2WSJElDzWBckjRt3sokSZI0PTP9nHFp6CW5MMnWJLf0pB2U5Iokd7T3A3uWnZ1kY5Lbk5zck/70JDe3Ze9uTxqQJEmStAAYjEtTdxFwyqi0s4Arq2o5cGWbJ8lRwErg6LbNeUn2aNu8F1hNd9/s8jH2KUmSJGlIGYxLU1RV1wD3jUo+FVjbptcCp/WkX1pVD1bVnXSP9juujTB9QFVdW1UFvL9nG0mSJElDzmBcmhmHVtUWgPZ+SEtfDNzTs96mlra4TY9OlyRJkrQAGIxLs2us+8BrgvSdd5CsTrI+yfpt27bNaOYkSZIkzQ2DcWlm3Nu6ntPet7b0TcDSnvWWAJtb+pIx0ndSVedX1YqqWrFo0aIZz/jcym68JEmSpPnLYFyaGZcDq9r0KuCynvSVSfZKciTdQG3Xt67sDyQ5vo2i/vKebSRJkiQNOYNxaYqSXAJcCzwxyaYkZwDnAicluQM4qc1TVbcC64DbgE8BZ1bV9rarVwPvoxvU7WvAJ/v6RSRJfZPk0Uk+lOSrSTYk+dnpPBZTkjQ89pzrDEjzTVWdPs6iE8dZfw2wZoz09cAxM5g1SdLgehfwqar6tSSPBPYB3kT3WMxzk5xF91jMN456LObhwN8neULPxVxJ0hCwZVySJGkWJTkAeA5wAUBV/aCqvssUH4vZzzxLkmafwbgkSdLs+mlgG/A3Sb6U5H1J9mXqj8WUJA0Rg3FJkqTZtSfwNOC9VfVU4Pt0XdLHM6nHX/roS0ma3wzGJUmSZtcmYFNVXdfmP0QXnE/1sZgPM9yPvpSk4WcwLkmSNIuq6pvAPUme2JJOpHvKxpQei9nHLEuS+sDR1CVJkmbf7wAfbCOpfx34DbpGkXXtEZl3Ay+C7rGYSUYei/kQD38spiRpSBiMS5IkzbKquglYMcaiKT0WU5I0POymLkmSJElSnxmMS5IkSZLUZwbjkiRJkiT1mcG4JEmSJEl9ZjAuSZIkSVKfGYxLkiRJktRnPtpMkiRJkiYjmescaIjYMi5JkiRJUp8ZjEuSJEmS1GcG45IkSZIk9ZnBuCRJkiRJfWYwLkmSJKmvkuyR5EtJPt7mD0pyRZI72vuBPeuenWRjktuTnDx3udakJdN/LSAG45IkSZL67bXAhp75s4Arq2o5cGWbJ8lRwErgaOAU4Lwke/Q5r9KsMBiXJEmS1DdJlgC/AryvJ/lUYG2bXguc1pN+aVU9WFV3AhuB4/qUVWlWGYxLkiRJ6qd3Am8AftSTdmhVbQFo74e09MXAPT3rbWpp0rxnMC5JkiSpL5K8ANhaVTdMdpMx0mqcfa9Osj7J+m3btk07j1K/GIxLkiRJ6pdnAS9MchdwKfDcJB8A7k1yGEB739rW3wQs7dl+CbB5rB1X1flVtaKqVixatGi28i/NGINxSZIkSX1RVWdX1ZKqWkY3MNs/VNVLgcuBVW21VcBlbfpyYGWSvZIcCSwHru9ztqVZsedcZ0CSJEnSgncusC7JGcDdwIsAqurWJOuA24CHgDOravvcZVOaOQbjkiRJkvquqq4CrmrT3wZOHGe9NcCavmVM6hO7qUuSJEmS1GcG45IkSZIk9ZnBuCRJkiRJfWYwLkmSJElSnxmMS5Ik9UGSPZJ8KcnH2/xBSa5Ickd7P7Bn3bOTbExye5KT5y7XkqTZYjAuSZLUH68FNvTMnwVcWVXLgSvbPEmOonv+8tHAKcB5Sfboc14lSbPMYFySJGmWJVkC/Arwvp7kU4G1bXotcFpP+qVV9WBV3QlsBI7rU1YlSX3ic8YlSZJm3zuBNwD796QdWlVbAKpqS5JDWvpi4Is9621qaXMib81cfbQkDTVbxiVJkmZRkhcAW6vqhsluMkZajbHf1UnWJ1m/bdu23cqjJKn/DMYlSZJm17OAFya5C7gUeG6SDwD3JjkMoL1vbetvApb2bL8E2Dx6p1V1flWtqKoVixYtms38S5JmwW4F40nuSnJzkpuSrG9pjgwqSZLUVNXZVbWkqpbRDcz2D1X1UuByYFVbbRVwWZu+HFiZZK8kRwLLgev7nG1J0iybiZbxX6yqY6tqRZt3ZFBJkqRdOxc4KckdwEltnqq6FVgH3AZ8CjizqrbPWS4lSbNiNrqpOzKoJMneU9IYquqqqnpBm/52VZ1YVcvb+309662pqsdV1ROr6pNzl2NJ0mzZ3WC8gM8kuSHJ6pb2sJFBgd6RQe/p2XZORwaVJPWFvackSZLGsLvB+LOq6mnA84AzkzxngnUnNTIoODqoJA0xe09JkiSxm8F4VW1u71uBj9KdOO3WyKBtf44OKmkXshsv9Ym9pyRJksYx7WA8yb5J9h+ZBn4ZuAVHBpUkdWa895Q9pyRJ0rDYnZbxQ4HPJ/kyXVD9f6vqUzgyqBYwB6ySdpiN3lP2nJIkScNi2sF4VX29qn6mvY6uqjUt3ZFBtdA5YJUWPHtPSZIkTWzPuc6AtACcCpzQptcCVwFvpGfAKuDOJCMDVl07B3mUZtqhwEeTQHesubiqPpXkn4B1Sc4A7gZeBF3vqSQjvacewt5TkiRpyBmMSzNrZMCqAv6qqs5n1IBVSXoHrPpiz7ZjDljVBr5aDXDEEUfMZt6lGVNVXwd+Zoz0bwMnjrPNGmDNLGdNkiRpIBiMSzPrWVW1uQXcVyT56gTrTmrAqhbQnw+wYsWKMR8HKEmSJGl+2d3njEvqMVuP+5MkSZI0XAzGpRnigFWSJEmSJstu6tLMccAqSZIkSZNiMC7NEAeskiRJkjRZdlOXJEmSJKnPDMYlSZIkSeozg3FJkiRJkvrMYFySJEmSpD4zGJckSZLUF0mWJvlskg1Jbk3y2pZ+UJIrktzR3g/s2ebsJBuT3J7k5LnLvTSzHE1dkjScLs5c50CStLOHgNdX1Y1J9gduSHIF8Argyqo6N8lZwFnAG5McBawEjgYOB/4+yRN8HKyGgS3jkiRJkvqiqrZU1Y1t+gFgA7AYOBVY21ZbC5zWpk8FLq2qB6vqTmAjcFxfMy3NEoNxzZpk+i9JkiQNtyTLgKcC1wGHVtUW6AJ24JC22mLgnp7NNrU0ad6zm7okSZqe3bkV4CU1c/mQNO8k2Q/4MPC6qro/47fGjLVgzH8gSVYDqwGOOOKImcimNKtsGZckSZLUN0keQReIf7CqPtKS701yWFt+GLC1pW8ClvZsvgTYPNZ+q+r8qlpRVSsWLVo0O5mXZpDBuCRJ0ixy9Ghph3RN4BcAG6rq7T2LLgdWtelVwGU96SuT7JXkSGA5cH2/8ivNpnkdjHtPsiRJmgdGRo9+MnA8cGYbIfosutGjlwNXtnlGjR59CnBekj3mJOfSzHsW8DLguUluaq/nA+cCJyW5AzipzVNVtwLrgNuATwFnOpK6hoX3jEuSJM2iNhjVyMBUDyTpHT36hLbaWuAq4I30jB4N3JlkZPToa/ubc2nmVdXnGfs+cIATx9lmDbBm1jIlzZF53TIuSZI0nzh6tCRphMG4JElSH4wePXqiVcdI22n06CSrk6xPsn7btm0zlU1JUp/YTV2SJGmWTTR6dFVtmc7o0VV1PnA+wIoVK3xW3DyWt05/QKM6xz+9NF/ZMi5JkjSLHD1akjQWW8YlSZJm18jo0TcnuamlvYlutOh1Sc4A7gZeBN3o0UlGRo9+CEePlqShZDAuSZI0ixw9WhowPudYA8Ju6pIkSZIk9ZnBuCRJkiRJfWYwLkmSJElSnxmMS5IkSZLUZwbjkiRJkiT1mcG4JEmSJEl9ZjAuSZIkSVKfGYxLkiRJktRnBuOSJEmSJPWZwbgkSZIkSX1mMC5JkiRJUp8ZjEuSJEmS1GcG45IkSZIk9ZnBuCRJkiRJfWYwLkmSJElSn+051xmQJEnS7MpbM9dZkCSNYsu4JEmSJEl9Zsu4JEmSJGkwZDd68lTNXD76wGBc0gK0O90159c/eUmSJA0mu6lLkiRJktRnBuOSJEmSJPWZ3dQ1kBbQrSKSJEmSFiCDcUmSJGme2p3H1tU5tmBIc8lu6pIkSZIk9VnfW8aTnAK8C9gDeF9VndvvPEiDwvog7WB9WGAu3o37kV6yMFrzrBPSDtYHDaO+town2QP4X8DzgKOA05Mc1c88SIPC+jBfZZovTcT6ID2cdULawfqgYdXvlvHjgI1V9XWAJJcCpwK39Tkf0iCwPkg7WB+kh7NOaNbNo/vNrQ+anHk2CnS/g/HFwD0985uAZ/Y5Dxpy86gOWh8WlN1pHV8QXXKtD5q86XZxn1/d260T0g7WB82+OQgi+h2Mj/UNd8p5ktXA6jb7vSS3t+mDgW/NUt52xyDmyzxNzsPytIs6+NgZ/uzdrQ8jBvF3nQl+rx8b2G7uM1knZqI+DGuZmQp/g4l+g1+f1brU92PEPKwPg5Yn8zOxCfOTt0xYnwbhnOnBJLfMcD6ma1D+toOSDxi2vEwcRIxbH/odjG8ClvbMLwE2j16pqs4Hzh+dnmR9Va2YvexNzyDmyzxNzhznabfqw4hB/F1ngt9rwdnt+uBv628AQ/Ub7LJOzLf6MGh5Mj8TG7D8TPkYMUj5H5S8DEo+wLyM6Pejzf4JWJ7kyCSPBFYCl/c5D9KgsD5IO1gfpIezTkg7WB80lPraMl5VDyV5DfBpuscSXFhVt/YzD9KgsD5IO1gfpIezTkg7WB80rPr+nPGq+gTwiWluPm5X3Tk2iPkyT5Mzp3nazfowYhB/15ng91pgZqA++Nv6G8AQ/QZDeM40aHkyPxMbqPxMoz4MUv4HJS+Dkg8wLwCk5mAId0mSJEmSFrJ+3zMuSZIkSdKCN2+C8SSnJLk9ycYkZ81RHi5MsrX3MQlJDkpyRZI72vuBfc7T0iSfTbIhya1JXjvX+UryqCTXJ/lyy9Nb5zpPPXnbI8mXknx8UPI0XYNQJ2bCIJbhmTRMZW6QDUt9mKxhrzdTYR3b2VzXh0E7Xxq0+jKo50nDVJfmsg4MSvkflHI/iOV9kMr6vAjGk+wB/C/gecBRwOlJjpqDrFwEnDIq7SzgyqpaDlzZ5vvpIeD1VfVk4HjgzPbbzGW+HgSeW1U/AxwLnJLk+DnO04jXAht65gchT1M2QHViJgxiGZ5JQ1HmBtmQ1YfJGvZ6MxXWsR4DUh8uYrDOlwatvgzqedJQ1KUBqAMXMRjlf1DK/SCW98Ep61U18C/gZ4FP98yfDZw9R3lZBtzSM387cFibPgy4fY5/q8uAkwYlX8A+wI3AM+c6T3TPpLwSeC7w8UH8+03huwxMnZiF7zZQZXg3v8vQlLlBfg1zfZjCbzA09WaK39s6tvNvMhD1YZDPlwapvgzKedIw1aVBqAODWP4HodwPQnkftLI+L1rGgcXAPT3zm1raIDi0qrYAtPdD5iojSZYBTwWum+t8te4fNwFbgSuqas7zBLwTeAPwo560uc7TdA1ynZi2QSrDM+SdDE+ZG2RDWR8mawjrzVS8E+vYaINaHwbi7zIo9WUAz5PeyfDUpUGsA3N9Xr6MOSz3A1be38kAlfX5EoxnjDSHge+RZD/gw8Drqur+uc5PVW2vqmPprj4dl+SYucxPkhcAW6vqhrnMxwwaujoxaGV4dw1hmRtkQ1cfJmvY6s1UWMfGtWDrw64MUn0ZpPOkIaxL1oEeg1DuB6W8D2JZny/B+CZgac/8EmDzHOVltHuTHAbQ3rf2OwNJHkFXyT5YVR8ZlHwBVNV3gavo7p2Zyzw9C3hhkruAS4HnJvnAHOdpdwxynZiyQS7Du2HYytwgG6r6MFlDWm+mwjo2tkGtD3P6dxnU+jIg50nDVpcGsQ7MyW85aOV+AMr7wJX1+RKM/xOwPMmRSR4JrAQun+M8jbgcWNWmV9Hdj9E3SQJcAGyoqrcPQr6SLEry6Da9N/BLwFfnMk9VdXZVLamqZXTl5x+q6qVzmafdNMh1YkoGsQzPhCEsc4NsaOrDZA1rvZkK69i4BrU+zOV5yUDVl0E7TxrCujSIdaDvv+WglPtBKu8DWdb7dXP67r6A5wP/DHwNePMc5eESYAvwQ7qrbmcAj6EbBOCO9n5Qn/P0bLquN18Bbmqv589lvoCnAF9qeboF+P2WPqe/VU/+TmDHgA0Dkadpfo85rxMz9D0GrgzPwnccijI3yK9hqQ9T+L5DX2+m+HtYxx7+e8xpfRi086VBqy+DfJ40LHVpLuvAoJT/QSn3g1reB6Wsp2VAkiRJkiT1yXzppi5JkiRJ0tAwGJckSZIkqc8MxiVJkiRJ6jODcUmSJEmS+sxgXJIkSZKkPjMYlyRJkiSpzwzGJUmSJEnqM4NxSZIkSZL67P8H4fIpjswnQvcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1224x360 with 5 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# PLOTTING WORD-COUNT\n",
    "import matplotlib.pyplot as plt\n",
    "fig,(ax1,ax2,ax3, ax4, ax5)=plt.subplots(1,5,figsize=(17,5))\n",
    "\n",
    "train_words=df[df['Sentiment']=='Positive']['word_count']\n",
    "ax1.hist(train_words,color='blue')\n",
    "ax1.set_title('Positive tweets')\n",
    "\n",
    "train_words=df[df['Sentiment']=='Neutral']['word_count']\n",
    "ax2.hist(train_words,color='yellow')\n",
    "ax2.set_title('Neutral tweets')\n",
    "\n",
    "\n",
    "train_words=df[df['Sentiment']=='Negative']['word_count']\n",
    "ax3.hist(train_words,color='Orange')\n",
    "ax3.set_title('Negative tweets')\n",
    "\n",
    "train_words=df[df['Sentiment']=='Extremely Positive']['word_count']\n",
    "ax4.hist(train_words,color='green')\n",
    "ax4.set_title('Extremely Positive tweets')\n",
    "\n",
    "train_words=df[df['Sentiment']=='Extremely Negative']['word_count']\n",
    "ax5.hist(train_words,color='red')\n",
    "ax5.set_title('Extremely Negative tweets')\n",
    "\n",
    "\n",
    "fig.suptitle('Words per tweet')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6db53c81",
   "metadata": {},
   "source": [
    "### Fitting an SVM classifier on the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "b893e00c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import svm\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "99b7b39b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train - test split\n",
    "X_train, X_test, y_train, y_test = train_test_split(df[\"OriginalTweet\"].apply(lambda x: ', '.join(x)),df[\"Sentiment\"],test_size=0.1,random_state=42)\n",
    "# Converting text to numbers using TFIDF\n",
    "tfidf_vectorizer = TfidfVectorizer(use_idf=True)\n",
    "X_train_vectors_tfidf = tfidf_vectorizer.fit_transform(X_train) \n",
    "X_test_vectors_tfidf = tfidf_vectorizer.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "db80848a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6023334953816237"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Classfication using Naive Bayer Classifier\n",
    "from sklearn import svm\n",
    "from sklearn.metrics import accuracy_score\n",
    "# Multiclass classification\n",
    "clf = svm.SVC(decision_function_shape='ovo')\n",
    "clf.fit(X_train_vectors_tfidf, y_train)  \n",
    "\n",
    "#Predict y value for test dataset\n",
    "y_predict = clf.predict(X_test_vectors_tfidf)\n",
    "\n",
    "accuracy_score(y_test, y_predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bf8420e",
   "metadata": {},
   "source": [
    "# spaCy Library (Source: https://spacy.io)\n",
    "\n",
    "- spaCy is a fast and open-source Python library for building realworld NLP products.\n",
    "\n",
    "1. Supports NLP tasks in 73+ languages. \n",
    "\n",
    "2. You can use spacy for: linguistically-motivated tokenization, named entity recognition, part-of-speech tagging, dependency parsing, sentence segmentation, text classification, lemmatization, morphological analysis, entity linking etc.\n",
    "\n",
    "3. Provides trained pipelines for NLP tasks in 25 languages\n",
    "\n",
    "4. Provides Pretrained word vectors\n",
    "\n",
    "5. Supports pretrained transformers like BERT\n",
    "\n",
    "6. Allows you to create custom models in PyTorch, TensorFlow and other frameworks\n",
    "\n",
    "7. It has many plugins and is extensible.\n",
    "\n",
    "#### The \"spacy-llm\" package integrates Large Language Models (LLMs) into spaCy pipelines. \n",
    "\n",
    "\n",
    "### Installing spaCy\n",
    "\n",
    "#### Please update your pip, setuptools and wheel packages before installing spacy:\n",
    "\n",
    "<code>\n",
    "pip install -U pip setuptools wheel\n",
    "</code>\n",
    "\n",
    "#### It is advised to create a separate environment for installing spacy:\n",
    "\n",
    "<code>\n",
    "python -m venv .env\n",
    "source .env/bin/activate\n",
    "pip install -U spacy\n",
    "</code>\n",
    "    \n",
    "#### Then download a trained pipeline in a langauge you are working in\n",
    "I am downloading English langauge pipeline of small size\n",
    "<code>\n",
    "python -m spacy download en_core_web_sm  \n",
    "</code>\n",
    "\n",
    "#### Using conda, you can install spaCy like this:\n",
    "\n",
    "<code>\n",
    "conda install -c conda-forge spacy\n",
    "</code>\n",
    "\n",
    "### NLP Functionality in spaCy\n",
    "\n",
    "- Tokenization\n",
    "\n",
    "- Part-of-speech \n",
    "\n",
    "- Dependency Parsing\n",
    "\n",
    "- Lemmatization\n",
    "\n",
    "- Sentence Boundary Detection\n",
    "\n",
    "- Named Entity Recognition\n",
    "\n",
    "- Entity Linking (Linking entities to their unique identifiers in a knowledge base). \n",
    "\n",
    "- Similarity\n",
    "\n",
    "- Text Classification\n",
    "\n",
    "- Rule-based Matching (Finding sequences of tokens based on their texts and linguistic annotations)\n",
    "\n",
    "- Training/ updating builtin statistical models\n",
    "\n",
    "- Serialization for saving your models for sharing and reuse \n",
    "\n",
    "##  Trained spaCy Pipelines and Statistical Models \n",
    "\n",
    "- A trained pipeline in spacy is a sequence of operations that are applied sequentially on tokens of text data.\n",
    "\n",
    "- Each statistical model(for an operation) in a pipeline is trained on labeled data.\n",
    "\n",
    "- SpaCy provids pipelines in different sizes: small, medium and large. Speed, memory usage, accuracy and the data included varies with size of the pipeline. You may choose a pipeline size based on your specific task. \n",
    "\n",
    "- You can install a trained pipeline as an individual Python modules. \n",
    "\n",
    "##### Downloading small sized English langauge pipeline\n",
    "<code>\n",
    "python -m spacy download en_core_web_sm  \n",
    "</code>\n",
    "\n",
    "- Small statistical model packages (e.g., the small package for English langauge: en_core_web_sm) contains:\n",
    "    - Binary weights for the part-of-speech tagger, dependency parser and named entity recognizer to predict those annotations in context.\n",
    "    - Lexical entries in the vocabulary, i.e. words and their context-independent attributes like the shape or spelling.\n",
    "    - Data files like lemmatization rules and lookup tables.\n",
    "    - Word vectors\n",
    "    - Configuration settings\n",
    "\n",
    "### \"nlp\" object \n",
    "\n",
    "- Once you’ve downloaded and installed a trained pipeline, you can load it into our workspace using \"spacy.load()\". \n",
    "\n",
    "- spacy.load() returns a Language object (usually called as 'nlp') containing all components and data needed to process text.\n",
    "\n",
    "- When we call the \"nlp\" object on some text, it is first tokenized and the tokens are put in a processed \"Doc\" object. \n",
    "\n",
    "<code>\n",
    "import spacy\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "</code>\n",
    "\n",
    "\n",
    "- Thereafter a pipeline of multiple steps is applied on the contents of this Doc object. \n",
    "\n",
    "- These steps are known as a processing pipeline which typically include a tagger, a lemmatizer, a parser and an entity recognizer. \n",
    "\n",
    "- Each pipeline component returns the processed Doc, which is then passed on to the next component.\n",
    "\n",
    "<img src = 'https://spacy.io/images/pipeline.svg'>\n",
    "<center>(Image Source: https://spacy.io/usage/spacy-101#features)</center>\n",
    "\n",
    "#### Trained pipelines allow spaCy to make predictions about annotations of tokens in Doc\n",
    "\n",
    "\n",
    "### Other components in a trained pipeline:\n",
    "NAME |\tDESCRIPTION\n",
    "----- |-------------\n",
    "AttributeRuler|\tSet token attributes using matcher rules.\n",
    "DependencyParser|\tPredict syntactic dependencies.\n",
    "EditTreeLemmatizer|\tPredict base forms of words.\n",
    "EntityLinker\t|Disambiguate named entities to nodes in a knowledge base.\n",
    "EntityRecognizer|\tPredict named entities, e.g. persons or products.\n",
    "EntityRuler\t|Add entity spans to the Doc using token-based rules or exact phrase matches.\n",
    "Lemmatizer|\tDetermine the base forms of words using rules and lookups.\n",
    "Morphologizer\t|Predict morphological features and coarse-grained part-of-speech tags.\n",
    "SentenceRecognizer|\tPredict sentence boundaries.\n",
    "Sentencizer\t|Implement rule-based sentence boundary detection that doesn’t require the dependency parse.\n",
    "Tagger\t|Predict part-of-speech tags.\n",
    "TextCategorizer|\tPredict categories or labels over the whole document.\n",
    "Tok2Vec\t|Apply a “token-to-vector” model and set its outputs.\n",
    "Tokenizer|\tSegment raw text and create Doc objects from the words.\n",
    "TrainablePipe|\tClass that all trainable pipeline components inherit from.\n",
    "Transformer|\tUse a transformer model and set its outputs.\n",
    "Other functions\t|Automatically apply something to the Doc, e.g. to merge spans of tokens."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58e309d0",
   "metadata": {},
   "source": [
    "## spaCy code walkthrough\n",
    "https://colab.research.google.com/drive/1ZBeYOGetDYVoOHAVxzQAuEiwI5ZjyXOI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8f7c16c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!python.exe -m pip install --upgrade pip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0617c4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install spacy\n",
    "#!python -m spacy download en_core_web_sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2405ffd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Linguistic Annotations\n",
    "#import spacy\n",
    "import spacy\n",
    "\n",
    "#load the English langauge pipeline (small sized) into an object called \"nlp\"\n",
    "# The nlp object is a pipeline of several text pre-processing operations including tokenization and annotations \n",
    "# that are applied on text.\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "# when we apply nlp object on some text, it returns a \"document\" object contains tokens and annotations\n",
    "doc = nlp(\"Elon Musk bought Twitter for $45 Billion \")\n",
    "\n",
    "#printing tokens, their part of speech tags and dependencies\n",
    "for token in doc:\n",
    "    print(token.text, token.pos_, token.dep_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4e06a44",
   "metadata": {},
   "source": [
    "Elon PROPN compound\n",
    "Musk PROPN nsubj\n",
    "bought VERB ROOT\n",
    "Twitter PROPN dobj\n",
    "for ADP prep\n",
    "$ SYM quantmod\n",
    "45 NUM compound\n",
    "Billion NUM pobj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b291ae1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# More annotations\n",
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\"Elon Musk bought Twitter for $45 Billion \")\n",
    "\n",
    "for token in doc:\n",
    "    print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_,\n",
    "            token.shape_, token.is_alpha, token.is_stop)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7bbaf6e",
   "metadata": {},
   "source": [
    "Elon Elon PROPN NNP compound Xxxx True False\n",
    "Musk Musk PROPN NNP nsubj Xxxx True False\n",
    "bought buy VERB VBD ROOT xxxx True False\n",
    "Twitter Twitter PROPN NNP dobj Xxxxx True False\n",
    "for for ADP IN prep xxx True True\n",
    "$ $ SYM $ quantmod $ False False\n",
    "45 45 NUM CD compound dd False False\n",
    "Billion billion NUM CD pobj Xxxxx True False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18d57ecb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Entities in text\n",
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\"Elon Musk bought Twitter for $45 Billion \")\n",
    "\n",
    "for ent in doc.ents:\n",
    "    print(ent.text, ent.start_char, ent.end_char, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a3e3047",
   "metadata": {},
   "source": [
    "Elon Musk 0 9 PERSON\n",
    "Twitter 17 24 PRODUCT\n",
    "$45 Billion 29 40 MONEY"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87b5951c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m spacy download en_core_web_lg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf39d399",
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_lg\")\n",
    "tokens = nlp(\"Mangoes are delicious afskfsd\")\n",
    "\n",
    "for token in tokens:\n",
    "    print(token.text, token.has_vector, token.vector_norm, token.is_oov)\n",
    "print(\"\\n\\n\\n displaying token vectors\")\n",
    "for token in tokens:\n",
    "  print(token.text, token.vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c0e7892",
   "metadata": {},
   "source": [
    "Mangoes True 21.508425 False\n",
    "are True 89.23195 False\n",
    "delicious True 37.606934 False\n",
    "afskfsd False 0.0 True\n",
    "\n",
    "\n",
    "\n",
    " displaying token vectors\n",
    "Mangoes [ 3.7261e-01 -9.5753e-01 -1.0911e+00 -1.1966e-01  2.4696e+00 -1.9787e+00\n",
    "  6.2725e-01 -1.4327e+00 -2.0090e+00 -3.5362e-01  4.0468e-01 -1.2934e+00\n",
    "  8.0839e-01  8.8090e-01  1.1298e+00 -2.7221e+00  1.2850e+00 -1.0303e+00\n",
    "  4.4173e-01 -1.5131e+00  4.6216e-01  1.4290e+00  1.0037e-01 -7.3036e-01\n",
    " -7.3615e-01 -9.9368e-01 -4.0501e-01 -1.7726e-01 -1.5049e-02 -2.0281e+00\n",
    "  2.4067e+00 -8.0549e-01 -4.9072e-01 -2.3763e-01 -1.5955e-01 -8.8682e-01\n",
    "  3.0616e-01 -3.6994e-01 -6.4752e-01 -8.5087e-01 -1.1671e+00  5.9176e-01\n",
    "  1.1781e+00  2.6323e+00  7.7953e-01  6.0910e-01  1.8162e+00  8.6945e-02\n",
    "  6.0705e-01 -1.4667e+00  3.3571e-01 -5.3755e-01 -4.0295e-02 -1.6203e+00\n",
    "  1.0364e+00 -1.5283e-01 -1.3526e+00 -2.0091e+00  2.8092e+00 -6.8126e-02\n",
    "  4.2257e-01  1.1417e+00  1.0178e+00 -1.6325e-01 -1.0920e+00  1.1825e+00\n",
    " -7.6024e-01 -1.3047e+00  2.5205e+00 -1.7210e+00 -8.6924e-01  7.2525e-01\n",
    " -6.6511e-01 -1.1468e+00  1.3706e+00  9.9028e-01 -1.1308e+00 -1.5282e+00\n",
    "  2.4816e+00  1.2173e+00  9.1371e-01  1.7901e-01 -1.8907e+00 -3.4570e-01\n",
    "  1.1806e+00 -1.3556e+00  2.2075e+00  6.4082e-01 -1.5568e+00  1.2447e+00\n",
    "  2.9432e+00  2.0521e+00 -1.1926e+00 -1.0592e+00 -2.3346e-01  2.7805e-01\n",
    "  1.2185e-01 -1.4608e+00  1.1687e+00 -2.3806e+00  1.0062e+00 -3.1754e+00\n",
    "  1.9033e+00  9.5368e-01 -1.9491e+00  6.7252e-01 -1.6473e+00  6.3709e-01\n",
    "  4.8230e-01  9.3853e-01 -2.5497e-02  1.9835e+00  1.5959e+00  7.6522e-01\n",
    "  3.4903e-02 -2.1805e+00 -4.4780e-01 -1.7176e-01 -1.4068e+00  2.2248e+00\n",
    "  1.4623e+00  3.1100e-01 -5.7280e-01  4.5830e-01 -1.2947e-01  4.9900e-01\n",
    "  1.5321e+00  2.6304e-03  1.6270e+00  4.0491e-01 -9.0396e-01  6.5297e-01\n",
    "  1.9459e+00 -3.3929e+00 -9.5945e-01 -8.6485e-01 -1.4496e+00 -1.2312e+00\n",
    " -4.8594e-01 -1.9576e+00 -8.4457e-01 -1.2373e+00  1.3783e+00 -8.6739e-01\n",
    " -9.4009e-01  5.5541e-01 -5.2121e-01 -8.8086e-01  1.1760e+00 -4.8332e-01\n",
    " -7.8664e-01  7.5649e-01  7.2099e-02 -1.0171e+00 -8.8378e-01 -2.4151e-02\n",
    "  8.6603e-01  2.1302e-01 -1.0151e+00 -2.3085e+00  1.8789e+00 -2.5510e+00\n",
    "  8.2600e-02  1.1799e+00 -2.7354e-02 -1.2339e+00  1.2353e+00  2.0520e+00\n",
    "  2.5664e-01  8.1739e-01 -6.1485e-01 -1.1517e+00  1.0616e-01  1.5321e+00\n",
    " -7.8788e-01 -1.3457e+00  7.2659e-01  6.0924e-02  1.9645e+00 -1.2322e+00\n",
    " -2.5589e+00  2.0009e+00 -6.0232e-01  1.8182e-01 -9.4723e-01 -7.9872e-01\n",
    " -9.8441e-02  4.2603e-01  8.8326e-01 -2.5064e-01  1.3996e+00 -1.8146e+00\n",
    "  4.7783e-01  3.0262e-01  2.0340e+00 -3.8093e-01 -1.1920e+00  1.2820e+00\n",
    "  5.7955e-01 -1.1817e+00  1.4135e+00  1.2238e+00 -7.4446e-01 -4.3840e-01\n",
    " -5.1250e-01 -1.4486e+00 -1.1260e+00  9.3215e-01 -1.9804e-01  1.0660e+00\n",
    "  3.6326e-01  3.5567e-01 -9.2854e-01 -4.3382e-01  2.0830e+00 -7.5695e-01\n",
    " -1.0744e+00 -8.7642e-01 -2.2251e-01 -8.8503e-01  1.4023e+00  3.6381e-02\n",
    " -2.2120e-01  5.5886e-01  1.0445e+00  3.3458e-01  1.5931e-01  1.7838e+00\n",
    " -3.0873e-01  2.2479e+00 -7.0775e-01  1.2952e+00  8.5407e-02  3.6748e-01\n",
    " -7.6384e-01  1.5004e+00 -5.0025e-01  1.0485e+00 -1.9283e+00 -2.2880e+00\n",
    "  1.1627e+00  8.9711e-01 -1.0547e+00 -9.8487e-01  1.1572e+00  3.3264e+00\n",
    " -6.2521e-04 -7.1317e-01 -6.9697e-01 -8.6732e-01 -2.5443e+00 -1.2691e+00\n",
    " -1.2761e+00 -7.4113e-01  4.3587e-01 -9.4631e-01 -4.4047e-01  1.1532e+00\n",
    " -1.9103e-01  1.4302e+00  5.6415e-01 -7.3373e-01 -9.6995e-01 -7.5733e-02\n",
    " -5.5080e-01 -1.0180e+00 -5.0887e-01  5.4205e-01 -6.4663e-01 -6.7202e-01\n",
    "  1.4212e+00  8.4726e-01 -3.1883e-01  1.0729e-01  2.8541e+00 -7.4601e-01\n",
    "  1.6358e+00 -8.7510e-02 -5.6470e-01 -1.1505e+00 -3.2069e-01 -1.8381e+00\n",
    " -1.6539e+00 -2.3310e+00  1.6118e+00 -1.7432e+00 -2.8969e-01 -2.6774e-01\n",
    " -4.0667e-01  1.4516e+00  1.2454e+00 -1.8498e+00 -3.6254e-01 -1.5326e+00\n",
    " -1.0209e+00 -1.4226e+00  1.7257e+00 -2.7325e+00 -6.3350e-01  1.1913e+00]\n",
    "are [-7.1395e+00 -2.4339e+00 -2.5181e+00  5.1603e+00  6.6538e-01  6.2771e+00\n",
    " -8.5893e+00  9.2970e+00 -4.6171e+00 -1.9255e+00  4.6743e+00 -1.0143e+00\n",
    " -5.2753e+00 -3.4705e+00  1.1035e+01  6.0578e+00 -5.4488e+00  3.6641e-01\n",
    " -4.9906e-01 -7.2857e+00  2.3550e+00 -6.4081e+00 -9.8889e+00 -7.2545e+00\n",
    " -7.3035e+00  4.3582e+00  4.5672e+00 -8.6448e+00 -5.6271e-01 -5.1761e+00\n",
    "  5.4107e+00  1.6521e+00 -1.1499e+01 -9.0169e+00 -1.4161e+00  5.1452e+00\n",
    " -1.5432e-01  5.3304e+00  9.0325e+00  7.0028e+00  6.7928e+00  6.5654e+00\n",
    " -1.9077e+00  4.6363e+00 -8.2151e+00  5.7743e+00  9.4244e+00 -9.4891e+00\n",
    " -7.8690e-01  4.0913e+00  1.8901e-01  4.0977e+00  7.9927e+00 -1.8402e+01\n",
    "  1.9437e-01  2.4133e+00 -1.5134e+00  3.6646e+00  1.8009e+00 -2.2974e+00\n",
    "  6.1053e+00  3.2767e+00  2.3698e+00 -3.4089e+00  6.0791e+00 -1.0496e+00\n",
    " -5.4553e+00 -1.0631e+00 -2.0511e+00  2.0346e+00 -5.5171e+00  2.0513e+00\n",
    " -2.3924e+00  5.3559e-01 -3.7088e+00 -1.4202e+00 -5.6887e+00 -2.1082e+00\n",
    " -1.6963e+00  1.4000e+00 -3.7038e+00  3.1733e-02 -3.0140e-01  1.6193e+00\n",
    "  7.8388e+00 -1.1760e+00 -4.5592e+00 -3.0969e+00  8.4116e+00  6.4556e+00\n",
    "  2.2805e+00 -6.4488e+00 -2.9915e+00 -4.2826e+00 -5.1191e+00  1.2892e+00\n",
    "  5.1704e-01 -6.4569e+00 -1.8947e+00  7.6637e-01 -5.2927e-01  2.4601e+00\n",
    " -1.3445e-01 -5.5656e-01 -3.6748e+00  6.9035e+00 -3.9277e+00 -1.8049e-01\n",
    " -1.1688e+00  4.4051e+00  9.3867e+00 -3.9732e+00 -5.8361e-01 -5.9747e+00\n",
    " -4.0834e+00  8.4304e+00 -7.3957e+00 -1.2977e+00 -1.5070e+00 -6.4646e+00\n",
    " -8.4846e+00  3.8867e+00 -3.8673e+00  1.7580e+00 -1.1202e+01 -1.4196e+01\n",
    " -4.7069e+00 -5.7320e+00 -3.1907e+00  9.1816e+00 -1.3515e+00 -3.0696e+00\n",
    "  1.0705e+00  2.7934e+00  1.2047e+00  7.6212e+00 -5.1389e+00  1.8877e+00\n",
    "  3.0568e+00  7.3349e-01  1.2372e+00  7.6613e-01 -2.3151e+00 -4.2148e+00\n",
    "  7.7783e+00 -7.5604e+00 -7.0796e+00 -9.7739e+00 -5.7583e+00  2.4196e+00\n",
    "  6.0054e+00 -3.3835e+00 -2.3892e+00  2.8944e-01 -1.0803e+00  2.1553e-01\n",
    "  4.5528e+00 -3.3486e-01  2.2235e-01 -2.8835e+00 -5.2577e+00 -1.7629e-01\n",
    " -8.8074e-01 -9.0020e+00 -4.2951e+00 -1.4936e+00 -8.3325e+00  6.1630e+00\n",
    "  3.2914e+00 -1.1028e+00 -3.6676e+00 -5.1784e+00  4.0418e-01  9.5084e-01\n",
    " -1.2313e+00  1.5689e+00  8.4684e+00  1.0628e+01 -2.4789e+00 -6.3778e+00\n",
    " -7.6133e-01 -4.9293e+00 -4.7340e-01  7.7729e+00  1.3093e+00  2.2021e+00\n",
    " -3.0884e+00  3.3861e-01  2.1974e+00  7.7405e+00 -5.2260e+00  4.3013e-01\n",
    "  6.0505e+00  5.3930e+00  3.2816e+00  8.2091e+00 -4.2903e+00  9.3290e+00\n",
    "  5.3857e+00 -4.9741e-01  2.2052e+00 -4.6191e+00 -1.2407e-01  9.5478e-01\n",
    "  8.4629e+00  1.0570e+00 -6.3333e+00  2.0358e+00  2.5175e+00  4.7564e+00\n",
    " -9.1763e-01  1.9307e+00  2.1470e+00 -1.0705e+00 -4.6793e+00  4.0667e-02\n",
    "  3.6421e+00 -1.1459e+01 -2.1205e+00  8.4685e+00  3.1494e+00  1.7994e+00\n",
    "  6.2841e+00  1.7717e+00 -2.7784e-01  5.5002e+00 -2.3093e+00  4.3914e+00\n",
    "  4.5903e+00 -5.1033e-01  1.1769e+00  2.3991e+00  5.5478e+00  6.7418e+00\n",
    "  2.5821e+00  1.4383e+00  3.1312e+00 -7.7081e-01  4.8816e+00  1.1286e+00\n",
    " -2.2709e+00  1.8717e+00 -3.4470e+00  3.8647e+00 -6.4492e+00 -1.1923e+00\n",
    " -1.1326e+01  3.1822e+00  7.7341e+00  1.1316e+01 -3.2741e+00 -1.2782e-01\n",
    " -3.6462e+00  3.6184e+00  5.7170e+00 -8.3825e+00  1.5262e+00  4.4945e+00\n",
    " -3.0539e+00  8.8721e+00 -6.2620e-01  6.1333e+00 -3.3232e+00  1.4263e+00\n",
    "  8.9952e+00  1.7785e+00  3.3962e+00  8.1608e+00 -4.7375e+00 -1.9417e-01\n",
    "  8.6104e+00 -1.1521e+01  4.6791e+00 -1.0418e+01  3.4825e+00  2.1576e+00\n",
    " -8.4904e+00  1.7844e+00 -1.0307e+00  1.7072e+00  6.8478e+00 -1.6212e-01\n",
    "  3.9700e-01 -1.3995e+00 -3.8334e+00  1.7668e+00  1.2926e+00  1.7469e+00\n",
    "  1.9176e+00 -5.6437e-01  5.9101e+00 -1.1613e+01  1.1822e-02 -7.3256e+00\n",
    " -3.1746e+00 -7.7470e+00 -5.2323e+00 -4.8147e+00 -5.1779e+00  6.7697e+00]\n",
    "delicious [ 0.7092   -4.8515   -4.0027   -0.035481  2.1093   -3.4179    1.8349\n",
    "  1.4426   -4.2782    1.8838    6.4145   -1.3608    0.74578   1.5484\n",
    "  2.8632   -1.6492    0.68588  -1.6119   -0.040331 -2.1478   -1.0315\n",
    "  4.3489   -0.44904  -1.1641   -1.7063   -0.51209  -3.9661    3.0282\n",
    " -1.3324   -0.58008   1.8697    0.25246   1.5428   -2.1581    1.2633\n",
    " -1.3652    0.22877   1.1674   -0.66271  -0.80733  -0.44111   0.75078\n",
    "  0.64708  -1.6667    2.5484    2.3655    3.4202   -0.88643  -0.99133\n",
    "  2.0942   -0.89087  -2.5291    0.78481  -3.4768    0.37535  -0.39447\n",
    " -1.4205    2.9368    2.94     -1.4571    1.5095    1.1199    2.2446\n",
    " -2.4576    2.4069    1.8778   -6.3169   -3.8751    2.8002   -1.0133\n",
    " -3.5313    2.714    -2.8761    1.0113    0.72243   0.95007  -1.5632\n",
    " -1.0918    1.1665   -1.5499   -4.4036    1.3481   -0.12665  -1.6157\n",
    " -0.61046   2.7583    3.2938   -0.16739  -1.9814    1.6755    2.2041\n",
    "  3.0735    1.4237   -5.1918   -0.84215  -2.3758    5.5003   -3.4504\n",
    " -0.4806   -2.1362   -0.19236  -1.3684    0.93712   1.0347   -0.63331\n",
    "  2.617    -4.6014    0.90786  -0.9261   -0.033286  2.2136   -2.3248\n",
    "  0.39649  -2.4453   -1.1921   -0.42687   0.70292  -3.5321   -1.3444\n",
    "  0.35091  -2.0656    2.3486    1.3469    0.38961   0.51143  -2.3646\n",
    "  0.6641   -5.1361    5.3172    1.0536   -1.8335    1.3553    3.0753\n",
    " -3.0187   -1.965    -0.46668  -1.3267   -2.228     1.2863   -1.6847\n",
    " -0.49327  -0.58868  -0.64351   0.35007   1.3616   -1.9437   -0.38134\n",
    " -0.9606    1.1374   -1.2152    1.7473    0.49004  -0.65754   0.042224\n",
    "  0.086731  3.2393    6.9229   -2.0262   -0.27954  -3.326     1.9961\n",
    " -3.1611    0.10049   0.12654   1.9953   -1.7689   -0.35986   1.2455\n",
    "  1.624     1.3113    1.2619    0.060233 -0.68588   0.058745 -1.6855\n",
    "  0.86664   0.087752  2.0942    3.7874   -1.5249   -0.11565   0.70173\n",
    "  0.52575  -3.1256   -1.0617   -0.39125  -1.848    -1.2818    0.42321\n",
    "  0.76795  -3.9126    2.2813   -3.4596    0.89612   1.8615   -2.5603\n",
    " -3.9133   -2.3822    0.68029  -1.5961    0.025493 -0.71174  -1.0092\n",
    " -0.86978   1.3047   -1.8026   -2.0092    2.2363    3.2866    2.4334\n",
    "  4.1527    2.1361    0.21916   0.28868   1.1919    1.0793    1.2192\n",
    " -0.090891 -1.9837   -0.596     0.13117   2.1703    0.29467   2.5128\n",
    "  1.3831    1.1516   -0.9587    1.7328    2.1304    0.33523  -1.5257\n",
    "  1.0437    1.5667    1.835     1.4899    2.3209    0.027331  4.4471\n",
    "  0.064199 -0.52261   1.1813   -2.4516    0.8218    2.3067   -1.9302\n",
    "  3.8455   -0.8806   -0.47642  -0.77793  -0.91675  -3.412     2.3403\n",
    " -4.1014   -1.7691   -0.81565  -3.4245   -1.7689    5.1181   -3.6028\n",
    "  0.58991   0.75776   3.0458   -0.023577  1.6957   -0.21844   2.0647\n",
    " -1.3647    1.5348   -4.5074   -0.58738   1.2836   -0.83738   0.34191\n",
    "  1.0374   -1.8386   -3.6066    4.5494   -0.017348 -1.0501    0.19709\n",
    "  4.0621   -2.9298   -3.4493   -1.0756   -0.19529  -0.8383   -1.8253\n",
    " -1.7412   -2.3181    2.7422    3.3942   -0.6045    0.12461  -4.2546\n",
    " -0.46246  -1.0097    1.0777    1.9044   -3.3072    2.3965  ]\n",
    "afskfsd [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
    " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
    " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
    " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
    " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
    " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
    " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
    " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
    " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
    " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
    " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
    " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
    " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
    "[ ]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8efd94e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# demonstrating similarity of documents\n",
    "#import spacy\n",
    "#nlp = spacy.load(\"en_core_web_lg\")\n",
    "\n",
    "doc1 = nlp(\"mangoes are sweet\")\n",
    "doc2 = nlp(\"apples are delicious\")\n",
    "\n",
    "# Similarity of two documents\n",
    "print(doc1, \"<->\", doc2, doc1.similarity(doc2))\n",
    "\n",
    "# Similarity of tokens and spans\n",
    "mangoes = doc1[0]\n",
    "apples = doc2[0]\n",
    "\n",
    "print(mangoes, \"<->\", apples, mangoes.similarity(apples))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aed38bfe",
   "metadata": {},
   "source": [
    "mangoes are sweet <-> apples are delicious 0.9256013406837312\n",
    "mangoes <-> apples 0.6791767477989197"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3b5e8dee",
   "metadata": {},
   "source": [
    "# Transformer Models for NLP (The architecture behind ChatGPT and BARD)\n",
    "\n",
    "- A transformer model is a deep neural network architecture that uses attention mechanism. \n",
    "\n",
    "- A transformer learns the context and meaning of its input text by tracking similarity in the tokens. \n",
    "\n",
    "- The transformers have proven very effective in tasks like langauge modelling, machine translation, speech recognition, text to speech conversion. \n",
    "\n",
    "#### Text data is sequential\n",
    "\n",
    "- To keep track of dependencies on past tokens in the input text, deep leaning models like RNNs and LSTMs have backward connections, but these models cannot handle long range depenencies in input sentences and are not parallelizable. Transformer model overcomes these shortcomings with Attention mechanism.\n",
    "\n",
    "<img src = \"transformer.png\">\n",
    "<center>(Source: Attention is all you need: https://arxiv.org/abs/1706.03762)</center>\n",
    "\n",
    "### Encoder and Decoder Block\n",
    "\n",
    "- A transformer has two components - A stack of encoders and a stack of decoders.\n",
    "\n",
    "- A transformer may have multiple encoders and multiple decoders. All the encoders in a stack are structurally similar but have different training weights. \n",
    "\n",
    "- An encoder has two sub-layers - Self-Attention and Feed Forward Neural Network. \n",
    "\n",
    "- Encoder uses Self-attention to compute a score for each word in the input sentence. This score measures how every other word in the input influences each other word. \n",
    "\n",
    "- Output of self-attention is fed to feed-forward neural network. The exact same feed-forward network is independently applied to each position in input text. \n",
    "\n",
    "- The first encoder coverts text to vector embeddings. Then each successive encoder receives the output of the encoder preceeding it. \n",
    "\n",
    "- A decoder also has self-attention and feed forward layers, it additionally has a cross-attention layer. Using cross-attention decoder is able to find the relationship scores between its own input and the input received from the encoder.\n",
    "\n",
    "- Final output is generated by the decoder.\n",
    "\n",
    "### Attention Mechanism (Self- Attention, Cross-Attention, Masked Attention, Multi-Headed Attention)\n",
    "\n",
    "- Attention is the ability to assign different importance (weight) to different words in an input sequence based on context. \n",
    "\n",
    "- Self-attention allows the model to weigh the importance of different parts of the input sequence when making predictions for each element in the sequence. This enables the model to consider long-range dependencies and relationships in the data.\n",
    "\n",
    "##### Toy example demonstrating calculation of attention scores:\n",
    "Say we have following input sentence \"GenAI is a helpful AI\"\n",
    "To compute self-attention score for each word in the sentence we perform following steps:\n",
    "\n",
    "Step 1. Represent the words as numerical vectors (Embeddings): E.g.,\n",
    "   \n",
    "   \"GenAI\":   [0.2, 0.5, -0.1]\n",
    "    \n",
    "   \"is\":      [0.7, 0.3, 0.2]\n",
    "   \n",
    "   \"a\":       [-0.3, 0.6, 0.8]\n",
    "   \n",
    "   \"helpful\": [0.1, 0.2, 0.4]\n",
    "   \n",
    "   \"AI\":      [0.5, -0.2, 0.6]\n",
    "\n",
    "Step 2: Calculate Self Attention Scores: One way to compute these scores is by using the dot product of the embeddings followed by a SoftMax to normalize the scores.\n",
    "\n",
    "Let's calculate the self attention scores for the word \"GenAI\" with all other words in this sentence:\n",
    "\n",
    "Self Attention Score (\"GenAI\", \"GenAI\"): \t [0.2, 0.5, -0.1] dot [0.2, 0.5, -0.1] = 0.34\n",
    "\n",
    "Self Attention Score (\"GenAI\", \"is\"):        [0.2, 0.5, -0.1] dot [0.7, 0.3, 0.2] = 0.29\n",
    "\n",
    "Self Attention Score (\"GenAI\", \"a\"):         [0.2, 0.5, -0.1] dot [-0.3, 0.6, 0.8] = 0.01\n",
    "\n",
    "Self Attention Score (\"GenAI\", \"helpful\"):   [0.2, 0.5, -0.1] dot [0.1, 0.2, 0.4] = 0.23S\n",
    "\n",
    "Self Attention Score (\"GenAI\", \"AI\"):        [0.2, 0.5, -0.1] dot [0.5, -0.2, 0.6] = 0.22\n",
    "\n",
    "Step 3:Normalize Scores with SoftMax: \n",
    "SoftMax([0.34, 0.29, 0.01, 0.23, 0.22]) = [0.299, 0.247, 0.116, 0.185, 0.153]\n",
    "\n",
    "These normalized scores represent the attention weights that show how much attention each word should pay to other words in the sentence. In this example, \"GenAI\" has higher attention on itself and a bit on \"is,\" with lower attention on the other words. \n",
    "\n",
    "### Calculation of cross-attention scores\n",
    "Say we have two sentences, a source sentence in English \"I like cats\" and a target sequence in French \"Je préfère les chats\". \n",
    "\n",
    "Step1: Represent all words in both sequences as numerical embeddings:\n",
    "\n",
    "    \"I\":       [0.2, 0.5, -0.1]\n",
    "    \n",
    "    \"like\":    [0.7, 0.3, 0.2]\n",
    "    \n",
    "    \"cats\":    [-0.3, 0.6, 0.8]\n",
    "    \n",
    "    \"Je\":      [0.4, 0.1, -0.2]\n",
    "    \n",
    "    \"préfère\": [0.3, 0.6, 0.5]\n",
    "    \n",
    "    \"les\":     [0.1, 0.2, 0.4]\n",
    "    \n",
    "    \"chats\":   [0.5, -0.2, 0.6]\n",
    "    \n",
    "\n",
    "Step 2: Calculate Cross Attention Scores for each word in the target sequence based on its relationship with the words in the source sequence, using the dot product of the embeddings followed by a SoftMax to normalize the scores.\n",
    "\n",
    "Here we calculate the cross-attention scores for the word \"chats\" in the target sequence:\n",
    "\n",
    "Cross Attention Score (\"chats\", \"I\"):    [0.5, -0.2, 0.6] dot [0.2, 0.5, -0.1] = 0.14\n",
    "\n",
    "Cross Attention Score (\"chats\", \"like\"): [0.5, -0.2, 0.6] dot [0.7, 0.3, 0.2] = 0.31\n",
    "\n",
    "Cross Attention Score (\"chats\", \"cats\"): [0.5, -0.2, 0.6] dot [-0.3, 0.6, 0.8] = 0.09\n",
    "\n",
    "\n",
    "Step3: Normalize Scores with SoftMax: Apply the SoftMax function to the calculated scores to ensure they sum up to 1 and represent the attention weights. SoftMax([0.14, 0.31, 0.09]) = [0.310, 0.430, 0.260]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20f997a6",
   "metadata": {},
   "source": [
    "# Large Langauge Models (E.g., OpenAI's GPT, Google's BARD)\n",
    "\n",
    "### Langauge Modelling: \n",
    "- Langauge modelling is the task of training machine learning models (deep neural networks) to learn the nuances and linguistic intuition from text input in a natural langauge.\n",
    "\n",
    "- Such models can predict the next word or character in a sequence of words or characters. Some popular state-of-the-art language models include Google BERT, GPT-3/4, Megatron-LM.\n",
    "\n",
    "<img src = \"langaugemodel.png\">\n",
    "\n",
    "- These models when trained on massive volumes of text data are known as Large Language Models(LLMs) or Foundational Models. \n",
    "\n",
    "- These LLMs can generate text very accurately. We can further fine tune these models.Foundational models are capable of following tasks: langauge translation, question answering, text-summarization etc.   \n",
    "\n",
    "- Fine-tuning a pretrained foundational model: Fine tuning means to take a foundational model and train it on your own specific dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9365bc4",
   "metadata": {},
   "source": [
    "## Code walk-through Semantic search on your text using a pre-trained langauge model and transformer embeddings from HuggingFace"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f4ab26c",
   "metadata": {},
   "source": [
    "### transformer-embeddings and sentence-transformer libraries from HuggingFace\n",
    "### https://pypi.org/project/transformer-embeddings/\n",
    "\n",
    "- The library 'transformer-embeddings' allows you to generate embeddings for text input. \n",
    "\n",
    "- 'sentence-transformers' library can be used for semantic-search based on the embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30eeeddc",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install transformer-embeddings\n",
    "!pip install transformers\n",
    "!pip install torch\n",
    "!pip install sentence_transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "38b4f9ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "#https://huggingface.co/blog/getting-started-with-embeddings\n",
    "# For running this code you need to have a Hugging Face account and a valid access token\n",
    "import transformers\n",
    "from transformer_embeddings import TransformerEmbeddings\n",
    "model_id = \"sentence-transformers/all-MiniLM-L6-v2\"\n",
    "hf_token = \"useyourhuggingfaceaccesstokenhere\" #you will find it here:https://huggingface.co/settings/tokens\n",
    "transformer = TransformerEmbeddings(model_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "2a34a0ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "api_url = f\"https://api-inference.huggingface.co/pipeline/feature-extraction/{model_id}\"\n",
    "headers = {\"Authorization\": f\"Bearer {hf_token}\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "6c212341",
   "metadata": {},
   "outputs": [],
   "source": [
    "def query(texts):\n",
    "    response = requests.post(api_url, headers=headers, json={\"inputs\": texts, \"options\":{\"wait_for_model\":True}})\n",
    "    return response.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "f7e2bfda",
   "metadata": {},
   "outputs": [],
   "source": [
    "sometext = [\"Generative AI has uses across a wide range of industries.\", \\\n",
    "            \"Prominent uses are in art, writing, software development, product design, healthcare.\"\\\n",
    "            \"Other uses are in finance, gaming, marketing, and fashion.\",\\\n",
    "            \"Investment in generative AI surged during the early 2020s\",\\\n",
    "            \"Large companies such as Microsoft, Google, and Baidu have developed generative AI models\"]\n",
    "output = query(sometext) #the query() returns embeddings of the text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "40531607",
   "metadata": {},
   "outputs": [],
   "source": [
    "# let us save these embeddings as a dataframe and to a csv file\n",
    "import pandas as pd\n",
    "embeddings = pd.DataFrame(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "c9eb6954",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Converting embeddings to a torch tensor\n",
    "import torch\n",
    "dataset_embeddings = torch.from_numpy(embeddings.to_numpy()).to(torch.float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "17c4c7ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Large companies such as Microsoft, Google, and Baidu have developed generative AI models', 'Generative AI has uses across a wide range of industries.']\n"
     ]
    }
   ],
   "source": [
    "# Doing a semantic search on our data using sentence_transformers\n",
    "question = [\"Which companies have generative AI models?\"]\n",
    "output = query(question)\n",
    "query_embeddings = torch.FloatTensor(output)\n",
    "\n",
    "from sentence_transformers.util import semantic_search\n",
    "hits = semantic_search(query_embeddings, dataset_embeddings, top_k=2)\n",
    "\n",
    "print([sometext[hits[0][i]['corpus_id']] for i in range(len(hits[0]))])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c61783a",
   "metadata": {},
   "source": [
    "## Vector Databases\n",
    "(Source: https://learn.microsoft.com/en-us/semantic-kernel/memories/vector-db)\n",
    "\n",
    "- A vector database stores high-dimensional numerical vector representations of data, i.e. Embeddings. \n",
    "\n",
    "- Vector databases allows for fast and accurate similarity search and retrieval of data based on their vector distance or similarity. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b40da09",
   "metadata": {},
   "source": [
    "## The future of natural langauge processing is promising\n",
    "\n",
    "Some of the interesting areas in NLP in coming future include: \n",
    "\n",
    "- Multimodal NLP: That combines text, speech and images for better performance on NLP tasks. \n",
    "\n",
    "- Zero-shot learning: To perform a task on given data without training on that dataset. \n",
    "\n",
    "- Explainable NLP: Improving transparenty and intepretability of NLP models. \n",
    "\n",
    "- Improving ethics, Reducing bias and \n",
    "\n",
    "- Reducing Hallicunatory Responses"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73f92324",
   "metadata": {},
   "source": [
    "## References:\n",
    "- https://web.stanford.edu/class/cs224n/\n",
    "- https://web.stanford.edu/class/cs224n/slides/cs224n-2023-lecture02-wordvecs2.pdf\n",
    "- https://www.deeplearning.ai/resources/natural-language-processing/\n",
    "- https://github.com/NimritaKoul/NLP_WWC2023/blob/main/Session1_Introduction%20to%20Natural%20Language%20Processing.ipynb\n",
    "- https://www.deeplearning.ai/resources/natural-language-processing/\n",
    "- https://courses.cs.washington.edu/courses/csep517/17sp/\n",
    "- http://jalammar.github.io/illustrated-transformer/\n",
    "- https://colab.research.google.com/github/tensorflow/tensor2tensor/blob/master/tensor2tensor/notebooks/hello_t2t.ipynb\n",
    "- https://medium.com/deeper-learning/glossary-of-deep-learning-word-embedding-f90c3cec34ca\n",
    "- http://jalammar.github.io/illustrated-word2vec/\n",
    "- https://txt.cohere.com/sentence-word-embeddings/\n",
    "- https://txt.cohere.com/what-is-similarity-between-sentences/\n",
    "- https://www.sbert.net/docs/pretrained_models.html#sentence-embedding-models/\n",
    "- https://platform.openai.com/docs/guides/gpt-best-practices\n",
    "- https://www.promptingguide.ai/introduction/basics\n",
    "- https://www.deepset.ai/blog/what-is-a-language-model\n",
    "- https://blog.research.google/2021/12/evaluating-syntactic-abilities-of.html\n",
    "- https://paperswithcode.com/task/language- modelling#:~:text=Language%20Modeling%20is%20the%20task,text%20classification%2C%20and%20question%20answering)\n",
    "- https://huggingface.co/docs/transformers/main/en/training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca0cb726",
   "metadata": {},
   "source": [
    "<img src = \"GHCThankYou.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bbc4076",
   "metadata": {},
   "source": [
    "# Let us connect : https://www.linkedin.com/in/nimritakoul/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}