NaiveBayes_Generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Programming Assignment 2: Naive Bayes\n",
    "## Part 1: Language Modelling and Text Generation\n",
    "\n",
    "#### Name: Abdul Rafay\n",
    "#### Roll Number: 24100173\n",
    "\n",
    "### Instructions\n",
    "*   In this part of the assignment you will be implementing an n-gram model for text-generation.\n",
    "*   Your code must be in the Python programming language.\n",
    "*   You are encouraged to use procedural programming and throughly comment your code.\n",
    "*   For Part 1, in addition to standard libraries i.e. numpy, pandas, regex, matplotlib and scipy, you can use [UrduHack](https://docs.urduhack.com/en/stable/index.html) for tokenization, and [NLKT](https://www.nltk.org/) for training your n-grams. However, no other machine learning toolkits or libraries are allowed.\n",
    "*   **Carefully read the submission guidelines, plagiarism and late days policy.**\n",
    "\n",
    "### Submission Guidelines\n",
    "Submit your code both as notebook file (.ipynb) and python script (.py) as individual files on LMS. Name both files as RollNumber_PA2_PartNum, i.e. this part should be named as `2xxxxxxx_PA4_1`. If you don’t know how to save .ipynb as .py see [this](https://i.stack.imgur.com/L1rQH.png). Failing to submit any one of them might result in the reduction of marks. All cells **MUST** be run to get credit.\n",
    "\n",
    "### Plagiarism Policy\n",
    "The code **MUST** be done independently. Any plagiarism or cheating of work from others or the internet will be immediately referred to the DC. If you are confused about what constitutes plagiarism, it is **YOUR** responsibility to consult with the instructor or the TA in a timely manner. No “after the fact” negotiations will be possible. The only way to guarantee that you do not lose marks is **DO NOT LOOK AT ANYONE ELSE'S CODE NOR DISCUSS IT WITH THEM**.\n",
    "\n",
    "### Late Days Policy\n",
    "\n",
    "The deadline for the assignment is final. However, in order to accommodate all the 11th-\n",
    "hour issues, there is a late submission policy i.e. you can submit your assignment within\n",
    "3 days after the deadline with a 25% deduction each day.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction\n",
    "An n-gram is a contiguous sequence of n words. For example \"Machine\" is a unigram, \"Machine Learning\" is a bigram and \"Machine Learning PA2\" is a trigram. In language modeling, n-gram models are probabilistic models of text that use word dependencies and context to predict the likelihood of occurence of an n-gram, i.e. predicting the nth word in an n-gram based on the previous n-1 words:\n",
    "$$\n",
    "P(ngram) =  P(word|context) = P(x^{n}|x^{n-1},...,x^{1})\n",
    "$$\n",
    "One use of the predictions made by such a model is text generation. In this part you will be training your own n-gram model and using it to generate text after learning from the provided Urdu short stories. \n",
    "<br><br>\n",
    "For additional details of the working of n-gram models, you can also consult [Chapter 3](https://web.stanford.edu/~jurafsky/slp3/3.pdf) of the Speech and Language Processing book as and references.\n",
    "\n",
    "\n",
    "### Dataset\n",
    "You will be using the Urdu short stories by Patras Bukhari given in the folder `Urdu Short Stories` in the PA2 zip file for the purposes of this part of the assignment. This contains 6 stories of varying lengths which will serve as inputs for your n-gram model. \n",
    "You're required to implement an n-gram model that uses the given stories to generate Urdu text that mimics the input stories."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start by importing all required libraries here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-12-02 05:50:53.488381: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2022-12-02 05:50:57.712210: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
      "2022-12-02 05:50:57.712253: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "2022-12-02 05:51:04.839144: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory\n",
      "2022-12-02 05:51:04.839411: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory\n",
      "2022-12-02 05:51:04.839421: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "/home/rafay/anaconda3/envs/urduenv/lib/python3.8/site-packages/tensorflow_addons/utils/ensure_tf_install.py:53: UserWarning: Tensorflow Addons supports using Python ops for all Tensorflow versions above or equal to 2.8.0 and strictly below 2.11.0 (nightly versions are not supported). \n",
      " The versions of TensorFlow you are currently using is 2.11.0 and is not supported. \n",
      "Some things might work, some things might not.\n",
      "If you were to encounter a bug, do not file an issue.\n",
      "If you want to make sure you're using a tested and supported configuration, either change the TensorFlow version or the TensorFlow Addons's version. \n",
      "You can find the compatibility matrix in TensorFlow Addon's readme:\n",
      "https://github.com/tensorflow/addons\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "import urduhack\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 - Loading and Preprocessing the Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read in the short story files given and tokenize the text to be preprocessed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Read file:  hostel mein parhna.txt\n",
      "Read file:  maibal aur main.txt\n",
      "Read file:  dost k naam.txt\n",
      "Read file:  cinema ka ishq.txt\n",
      "Read file:  sawere.txt\n",
      "Read file:  lahore ka jughrafiya.txt\n"
     ]
    }
   ],
   "source": [
    "# code here\n",
    "\n",
    "dataset_path = 'DataP1'\n",
    "\n",
    "# store the names of all files in a list inside the dataset_path folder\n",
    "files = os.listdir(dataset_path)\n",
    "files.remove('Urdu Data.rar')\n",
    "\n",
    "# dataset is multiple urdu text files. read all files and store them in a list\n",
    "dataset = []\n",
    "for file in files:\n",
    "    with open(dataset_path + '/' + file, 'r') as f:\n",
    "        dataset.append(f.read())\n",
    "    print(\"Read file: \", file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# normalize the dataset\n",
    "dataset = [urduhack.normalization.normalize(text) for text in dataset]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokenizing text:  1\n",
      "Tokenizing sentences\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-11-29 23:24:10.758741: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory\n",
      "2022-11-29 23:24:10.758771: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "2022-11-29 23:24:10.758791: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (rafay-pc): /proc/driver/nvidia/version does not exist\n",
      "2022-11-29 23:24:10.759019: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2022-11-29 23:24:18.412021: W tensorflow/tsl/framework/cpu_allocator_impl.cc:82] Allocation of 67108864 exceeds 10% of free system memory.\n",
      "2022-11-29 23:24:18.457071: W tensorflow/tsl/framework/cpu_allocator_impl.cc:82] Allocation of 67108864 exceeds 10% of free system memory.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/6 [====>.........................] - ETA: 21s"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-11-29 23:24:19.142371: W tensorflow/tsl/framework/cpu_allocator_impl.cc:82] Allocation of 67108864 exceeds 10% of free system memory.\n",
      "2022-11-29 23:24:19.178817: W tensorflow/tsl/framework/cpu_allocator_impl.cc:82] Allocation of 67108864 exceeds 10% of free system memory.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2/6 [=========>....................] - ETA: 2s "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-11-29 23:24:19.700403: W tensorflow/tsl/framework/cpu_allocator_impl.cc:82] Allocation of 67108864 exceeds 10% of free system memory.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6/6 [==============================] - 7s 472ms/step\n",
      "3/3 [==============================] - 1s 354ms/step\n",
      "3/3 [==============================] - 2s 549ms/step\n",
      "1/1 [==============================] - 1s 523ms/step\n",
      "6/6 [==============================] - 3s 451ms/step\n",
      "4/4 [==============================] - 2s 394ms/step\n",
      "3/3 [==============================] - 1s 341ms/step\n",
      "3/3 [==============================] - 1s 394ms/step\n",
      "10/10 [==============================] - 5s 537ms/step\n",
      "7/7 [==============================] - 4s 511ms/step\n",
      "2/2 [==============================] - 1s 125ms/step\n",
      "4/4 [==============================] - 2s 398ms/step\n",
      "3/3 [==============================] - 2s 518ms/step\n",
      "2/2 [==============================] - 1s 214ms/step\n",
      "2/2 [==============================] - 1s 268ms/step\n",
      "3/3 [==============================] - 2s 506ms/step\n",
      "11/11 [==============================] - 6s 516ms/step\n",
      "4/4 [==============================] - 2s 410ms/step\n",
      "2/2 [==============================] - 1s 435ms/step\n",
      "2/2 [==============================] - 1s 407ms/step\n",
      "5/5 [==============================] - 2s 460ms/step\n",
      "5/5 [==============================] - 3s 498ms/step\n",
      "2/2 [==============================] - 1s 351ms/step\n",
      "2/2 [==============================] - 1s 119ms/step\n",
      "2/2 [==============================] - 1s 351ms/step\n",
      "7/7 [==============================] - 3s 474ms/step\n",
      "2/2 [==============================] - 1s 381ms/step\n",
      "5/5 [==============================] - 2s 469ms/step\n",
      "3/3 [==============================] - 1s 348ms/step\n",
      "10/10 [==============================] - 5s 495ms/step\n",
      "1/1 [==============================] - 0s 201ms/step\n",
      "2/2 [==============================] - 1s 505ms/step\n",
      "12/12 [==============================] - 6s 537ms/step\n",
      "4/4 [==============================] - 2s 437ms/step\n",
      "2/2 [==============================] - 1s 126ms/step\n",
      "1/1 [==============================] - 0s 418ms/step\n",
      "1/1 [==============================] - 0s 450ms/step\n",
      "6/6 [==============================] - 3s 442ms/step\n",
      "1/1 [==============================] - 0s 373ms/step\n",
      "4/4 [==============================] - 2s 422ms/step\n",
      "4/4 [==============================] - 2s 527ms/step\n",
      "5/5 [==============================] - 2s 477ms/step\n",
      "3/3 [==============================] - 1s 413ms/step\n",
      "8/8 [==============================] - 4s 510ms/step\n",
      "3/3 [==============================] - 1s 305ms/step\n",
      "3/3 [==============================] - 1s 468ms/step\n",
      "2/2 [==============================] - 1s 287ms/step\n",
      "3/3 [==============================] - 1s 313ms/step\n",
      "5/5 [==============================] - 2s 438ms/step\n",
      "2/2 [==============================] - 1s 424ms/step\n",
      "3/3 [==============================] - 1s 445ms/step\n",
      "2/2 [==============================] - 1s 151ms/step\n",
      "2/2 [==============================] - 1s 96ms/step\n",
      "1/1 [==============================] - 0s 500ms/step\n",
      "2/2 [==============================] - 1s 82ms/step\n",
      "6/6 [==============================] - 3s 481ms/step\n",
      "10/10 [==============================] - 5s 546ms/step\n",
      "8/8 [==============================] - 4s 511ms/step\n",
      "1/1 [==============================] - 1s 530ms/step\n",
      "2/2 [==============================] - 1s 533ms/step\n",
      "8/8 [==============================] - 4s 497ms/step\n",
      "4/4 [==============================] - 2s 467ms/step\n",
      "4/4 [==============================] - 2s 388ms/step\n",
      "4/4 [==============================] - 2s 483ms/step\n",
      "3/3 [==============================] - 1s 384ms/step\n",
      "4/4 [==============================] - 2s 473ms/step\n",
      "1/1 [==============================] - 1s 517ms/step\n",
      "1/1 [==============================] - 0s 349ms/step\n",
      "1/1 [==============================] - 0s 204ms/step\n",
      "3/3 [==============================] - 1s 333ms/step\n",
      "3/3 [==============================] - 1s 369ms/step\n",
      "3/3 [==============================] - 1s 367ms/step\n",
      "2/2 [==============================] - 1s 53ms/step\n",
      "4/4 [==============================] - 2s 427ms/step\n",
      "3/3 [==============================] - 2s 515ms/step\n",
      "3/3 [==============================] - 1s 313ms/step\n",
      "2/2 [==============================] - 1s 135ms/step\n",
      "4/4 [==============================] - 2s 417ms/step\n",
      "3/3 [==============================] - 2s 555ms/step\n",
      "3/3 [==============================] - 1s 347ms/step\n",
      "2/2 [==============================] - 1s 493ms/step\n",
      "2/2 [==============================] - 1s 403ms/step\n",
      "2/2 [==============================] - 1s 195ms/step\n",
      "4/4 [==============================] - 2s 426ms/step\n",
      "2/2 [==============================] - 1s 116ms/step\n",
      "7/7 [==============================] - 4s 505ms/step\n",
      "5/5 [==============================] - 3s 501ms/step\n",
      "2/2 [==============================] - 1s 360ms/step\n",
      "6/6 [==============================] - 3s 467ms/step\n",
      "6/6 [==============================] - 3s 516ms/step\n",
      "8/8 [==============================] - 4s 509ms/step\n",
      "3/3 [==============================] - 1s 314ms/step\n",
      "2/2 [==============================] - 1s 274ms/step\n",
      "1/1 [==============================] - 0s 366ms/step\n",
      "1/1 [==============================] - 0s 454ms/step\n",
      "1/1 [==============================] - 0s 302ms/step\n",
      "2/2 [==============================] - 1s 332ms/step\n",
      "3/3 [==============================] - 1s 411ms/step\n",
      "1/1 [==============================] - 0s 440ms/step\n",
      "1/1 [==============================] - 0s 483ms/step\n",
      "1/1 [==============================] - 0s 220ms/step\n",
      "4/4 [==============================] - 2s 489ms/step\n",
      "2/2 [==============================] - 1s 528ms/step\n",
      "1/1 [==============================] - 1s 630ms/step\n",
      "1/1 [==============================] - 0s 451ms/step\n",
      "2/2 [==============================] - 1s 234ms/step\n",
      "4/4 [==============================] - 2s 486ms/step\n",
      "2/2 [==============================] - 1s 52ms/step\n",
      "2/2 [==============================] - 1s 200ms/step\n",
      "1/1 [==============================] - 1s 533ms/step\n",
      "1/1 [==============================] - 0s 462ms/step\n",
      "1/1 [==============================] - 1s 501ms/step\n",
      "3/3 [==============================] - 1s 349ms/step\n",
      "2/2 [==============================] - 1s 521ms/step\n",
      "2/2 [==============================] - 1s 182ms/step\n",
      "3/3 [==============================] - 1s 341ms/step\n",
      "2/2 [==============================] - 1s 214ms/step\n",
      "2/2 [==============================] - 1s 454ms/step\n",
      "2/2 [==============================] - 1s 189ms/step\n",
      "3/3 [==============================] - 1s 359ms/step\n",
      "1/1 [==============================] - 0s 488ms/step\n",
      "1/1 [==============================] - 0s 324ms/step\n",
      "1/1 [==============================] - 0s 340ms/step\n",
      "1/1 [==============================] - 0s 400ms/step\n",
      "1/1 [==============================] - 0s 403ms/step\n",
      "4/4 [==============================] - 2s 407ms/step\n",
      "2/2 [==============================] - 1s 120ms/step\n",
      "3/3 [==============================] - 1s 417ms/step\n",
      "1/1 [==============================] - 0s 260ms/step\n",
      "2/2 [==============================] - 1s 161ms/step\n",
      "1/1 [==============================] - 0s 421ms/step\n",
      "4/4 [==============================] - 2s 516ms/step\n",
      "2/2 [==============================] - 1s 503ms/step\n",
      "7/7 [==============================] - 3s 488ms/step\n",
      "4/4 [==============================] - 2s 454ms/step\n",
      "2/2 [==============================] - 1s 462ms/step\n",
      "5/5 [==============================] - 2s 441ms/step\n",
      "3/3 [==============================] - 1s 425ms/step\n",
      "1/1 [==============================] - 0s 349ms/step\n",
      "3/3 [==============================] - 1s 340ms/step\n",
      "2/2 [==============================] - 1s 398ms/step\n",
      "6/6 [==============================] - 3s 452ms/step\n",
      "3/3 [==============================] - 1s 352ms/step\n",
      "1/1 [==============================] - 0s 208ms/step\n",
      "3/3 [==============================] - 2s 474ms/step\n",
      "3/3 [==============================] - 1s 362ms/step\n",
      "2/2 [==============================] - 1s 181ms/step\n",
      "3/3 [==============================] - 2s 550ms/step\n",
      "5/5 [==============================] - 3s 537ms/step\n",
      "8/8 [==============================] - 4s 490ms/step\n",
      "2/2 [==============================] - 1s 522ms/step\n",
      "3/3 [==============================] - 1s 460ms/step\n",
      "2/2 [==============================] - 1s 183ms/step\n",
      "1/1 [==============================] - 0s 353ms/step\n",
      "5/5 [==============================] - 2s 434ms/step\n",
      "2/2 [==============================] - 1s 259ms/step\n",
      "4/4 [==============================] - 2s 466ms/step\n",
      "3/3 [==============================] - 1s 415ms/step\n",
      "2/2 [==============================] - 1s 535ms/step\n",
      "2/2 [==============================] - 1s 373ms/step\n",
      "6/6 [==============================] - 3s 535ms/step\n",
      "2/2 [==============================] - 1s 334ms/step\n",
      "2/2 [==============================] - 1s 204ms/step\n",
      "4/4 [==============================] - 2s 446ms/step\n",
      "2/2 [==============================] - 1s 549ms/step\n",
      "3/3 [==============================] - 2s 532ms/step\n",
      "11/11 [==============================] - 6s 512ms/step\n",
      "3/3 [==============================] - 1s 423ms/step\n",
      "2/2 [==============================] - 1s 443ms/step\n",
      "2/2 [==============================] - 1s 261ms/step\n",
      "4/4 [==============================] - 2s 448ms/step\n",
      "2/2 [==============================] - 1s 148ms/step\n",
      "7/7 [==============================] - 4s 553ms/step\n",
      "3/3 [==============================] - 1s 316ms/step\n",
      "5/5 [==============================] - 3s 525ms/step\n",
      "2/2 [==============================] - 1s 534ms/step\n",
      "2/2 [==============================] - 1s 540ms/step\n",
      "4/4 [==============================] - 2s 375ms/step\n",
      "4/4 [==============================] - 2s 389ms/step\n",
      "3/3 [==============================] - 2s 542ms/step\n",
      "1/1 [==============================] - 0s 488ms/step\n",
      "7/7 [==============================] - 4s 501ms/step\n",
      "2/2 [==============================] - 1s 192ms/step\n",
      "3/3 [==============================] - 1s 463ms/step\n",
      "7/7 [==============================] - 3s 489ms/step\n",
      "1/1 [==============================] - 0s 369ms/step\n",
      "4/4 [==============================] - 2s 405ms/step\n",
      "8/8 [==============================] - 4s 481ms/step\n",
      "3/3 [==============================] - 1s 309ms/step\n",
      "3/3 [==============================] - 1s 321ms/step\n",
      "2/2 [==============================] - 1s 137ms/step\n",
      "3/3 [==============================] - 1s 387ms/step\n",
      "1/1 [==============================] - 1s 520ms/step\n",
      "3/3 [==============================] - 1s 302ms/step\n",
      "3/3 [==============================] - 1s 415ms/step\n",
      "2/2 [==============================] - 1s 118ms/step\n",
      "3/3 [==============================] - 1s 464ms/step\n",
      "3/3 [==============================] - 1s 399ms/step\n",
      "2/2 [==============================] - 1s 98ms/step\n",
      "3/3 [==============================] - 1s 461ms/step\n",
      "3/3 [==============================] - 1s 460ms/step\n",
      "1/1 [==============================] - 0s 488ms/step\n",
      "2/2 [==============================] - 1s 205ms/step\n",
      "2/2 [==============================] - 1s 475ms/step\n",
      "4/4 [==============================] - 2s 411ms/step\n",
      "5/5 [==============================] - 3s 541ms/step\n",
      "2/2 [==============================] - 1s 266ms/step\n",
      "5/5 [==============================] - 2s 445ms/step\n",
      "3/3 [==============================] - 1s 304ms/step\n",
      "3/3 [==============================] - 1s 426ms/step\n",
      "2/2 [==============================] - 1s 164ms/step\n",
      "2/2 [==============================] - 1s 367ms/step\n",
      "2/2 [==============================] - 1s 419ms/step\n",
      "2/2 [==============================] - 1s 235ms/step\n",
      "3/3 [==============================] - 1s 333ms/step\n",
      "4/4 [==============================] - 2s 398ms/step\n",
      "2/2 [==============================] - 1s 333ms/step\n",
      "3/3 [==============================] - 1s 384ms/step\n",
      "2/2 [==============================] - 1s 286ms/step\n",
      "5/5 [==============================] - 2s 475ms/step\n",
      "3/3 [==============================] - 1s 313ms/step\n",
      "3/3 [==============================] - 1s 327ms/step\n",
      "4/4 [==============================] - 2s 354ms/step\n",
      "4/4 [==============================] - 2s 353ms/step\n",
      "7/7 [==============================] - 3s 441ms/step\n",
      "4/4 [==============================] - 2s 481ms/step\n",
      "2/2 [==============================] - 1s 319ms/step\n",
      "2/2 [==============================] - 1s 446ms/step\n",
      "2/2 [==============================] - 1s 402ms/step\n",
      "2/2 [==============================] - 1s 248ms/step\n",
      "3/3 [==============================] - 1s 339ms/step\n",
      "3/3 [==============================] - 1s 376ms/step\n",
      "4/4 [==============================] - 2s 442ms/step\n",
      "1/1 [==============================] - 0s 431ms/step\n",
      "2/2 [==============================] - 1s 468ms/step\n",
      "3/3 [==============================] - 1s 350ms/step\n",
      "2/2 [==============================] - 1s 247ms/step\n",
      "1/1 [==============================] - 0s 249ms/step\n",
      "1/1 [==============================] - 0s 368ms/step\n",
      "3/3 [==============================] - 1s 305ms/step\n",
      "10/10 [==============================] - 5s 507ms/step\n",
      "5/5 [==============================] - 3s 542ms/step\n",
      "7/7 [==============================] - 4s 495ms/step\n",
      "1/1 [==============================] - 0s 275ms/step\n",
      "3/3 [==============================] - 1s 446ms/step\n",
      "5/5 [==============================] - 2s 441ms/step\n",
      "Tokenizing text:  1\n",
      "Tokenizing sentences\n",
      "1/1 [==============================] - 0s 490ms/step\n",
      "4/4 [==============================] - 2s 397ms/step\n",
      "1/1 [==============================] - 0s 463ms/step\n",
      "2/2 [==============================] - 1s 224ms/step\n",
      "3/3 [==============================] - 1s 315ms/step\n",
      "2/2 [==============================] - 1s 303ms/step\n",
      "2/2 [==============================] - 1s 71ms/step\n",
      "2/2 [==============================] - 1s 139ms/step\n",
      "5/5 [==============================] - 2s 464ms/step\n",
      "2/2 [==============================] - 1s 390ms/step\n",
      "2/2 [==============================] - 1s 191ms/step\n",
      "5/5 [==============================] - 2s 472ms/step\n",
      "7/7 [==============================] - 4s 538ms/step\n",
      "3/3 [==============================] - 1s 449ms/step\n",
      "2/2 [==============================] - 1s 190ms/step\n",
      "5/5 [==============================] - 3s 490ms/step\n",
      "2/2 [==============================] - 1s 546ms/step\n",
      "4/4 [==============================] - 2s 435ms/step\n",
      "6/6 [==============================] - 3s 535ms/step\n",
      "2/2 [==============================] - 1s 297ms/step\n",
      "3/3 [==============================] - 1s 389ms/step\n",
      "6/6 [==============================] - 3s 537ms/step\n",
      "4/4 [==============================] - 2s 483ms/step\n",
      "5/5 [==============================] - 3s 511ms/step\n",
      "7/7 [==============================] - 3s 461ms/step\n",
      "2/2 [==============================] - 1s 510ms/step\n",
      "4/4 [==============================] - 2s 594ms/step\n",
      "2/2 [==============================] - 1s 444ms/step\n",
      "3/3 [==============================] - 1s 334ms/step\n",
      "2/2 [==============================] - 1s 458ms/step\n",
      "1/1 [==============================] - 0s 401ms/step\n",
      "2/2 [==============================] - 1s 302ms/step\n",
      "3/3 [==============================] - 2s 557ms/step\n",
      "4/4 [==============================] - 2s 457ms/step\n",
      "2/2 [==============================] - 1s 212ms/step\n",
      "1/1 [==============================] - 1s 526ms/step\n",
      "1/1 [==============================] - 0s 305ms/step\n",
      "3/3 [==============================] - 1s 435ms/step\n",
      "6/6 [==============================] - 3s 445ms/step\n",
      "1/1 [==============================] - 0s 395ms/step\n",
      "3/3 [==============================] - 1s 456ms/step\n",
      "6/6 [==============================] - 3s 517ms/step\n",
      "1/1 [==============================] - 0s 189ms/step\n",
      "1/1 [==============================] - 0s 179ms/step\n",
      "2/2 [==============================] - 1s 205ms/step\n",
      "3/3 [==============================] - 1s 297ms/step\n",
      "1/1 [==============================] - 1s 531ms/step\n",
      "2/2 [==============================] - 1s 117ms/step\n",
      "1/1 [==============================] - 1s 576ms/step\n",
      "2/2 [==============================] - 1s 154ms/step\n",
      "3/3 [==============================] - 2s 528ms/step\n",
      "2/2 [==============================] - 1s 163ms/step\n",
      "2/2 [==============================] - 1s 471ms/step\n",
      "5/5 [==============================] - 3s 505ms/step\n",
      "5/5 [==============================] - 2s 419ms/step\n",
      "5/5 [==============================] - 2s 458ms/step\n",
      "4/4 [==============================] - 2s 448ms/step\n",
      "2/2 [==============================] - 1s 103ms/step\n",
      "3/3 [==============================] - 1s 408ms/step\n",
      "3/3 [==============================] - 2s 522ms/step\n",
      "2/2 [==============================] - 1s 433ms/step\n",
      "1/1 [==============================] - 1s 520ms/step\n",
      "2/2 [==============================] - 1s 419ms/step\n",
      "3/3 [==============================] - 2s 526ms/step\n",
      "4/4 [==============================] - 2s 446ms/step\n",
      "2/2 [==============================] - 1s 198ms/step\n",
      "3/3 [==============================] - 2s 530ms/step\n",
      "3/3 [==============================] - 2s 494ms/step\n",
      "3/3 [==============================] - 1s 304ms/step\n",
      "3/3 [==============================] - 2s 518ms/step\n",
      "3/3 [==============================] - 2s 497ms/step\n",
      "2/2 [==============================] - 1s 91ms/step\n",
      "2/2 [==============================] - 1s 197ms/step\n",
      "1/1 [==============================] - 0s 420ms/step\n",
      "3/3 [==============================] - 1s 356ms/step\n",
      "4/4 [==============================] - 2s 484ms/step\n",
      "3/3 [==============================] - 1s 369ms/step\n",
      "2/2 [==============================] - 1s 506ms/step\n",
      "6/6 [==============================] - 3s 440ms/step\n",
      "1/1 [==============================] - 0s 404ms/step\n",
      "2/2 [==============================] - 1s 367ms/step\n",
      "2/2 [==============================] - 1s 95ms/step\n",
      "1/1 [==============================] - 0s 306ms/step\n",
      "2/2 [==============================] - 1s 145ms/step\n",
      "3/3 [==============================] - 1s 384ms/step\n",
      "1/1 [==============================] - 0s 288ms/step\n",
      "1/1 [==============================] - 0s 320ms/step\n",
      "1/1 [==============================] - 1s 531ms/step\n",
      "2/2 [==============================] - 1s 538ms/step\n",
      "1/1 [==============================] - 0s 460ms/step\n",
      "1/1 [==============================] - 1s 573ms/step\n",
      "2/2 [==============================] - 1s 504ms/step\n",
      "3/3 [==============================] - 1s 426ms/step\n",
      "1/1 [==============================] - 0s 322ms/step\n",
      "2/2 [==============================] - 1s 504ms/step\n",
      "3/3 [==============================] - 2s 495ms/step\n",
      "2/2 [==============================] - 1s 124ms/step\n",
      "2/2 [==============================] - 1s 119ms/step\n",
      "2/2 [==============================] - 1s 517ms/step\n",
      "Tokenizing text:  1\n",
      "Tokenizing sentences\n",
      "6/6 [==============================] - 3s 482ms/step\n",
      "4/4 [==============================] - 2s 495ms/step\n",
      "1/1 [==============================] - 0s 247ms/step\n",
      "3/3 [==============================] - 2s 497ms/step\n",
      "6/6 [==============================] - 3s 496ms/step\n",
      "6/6 [==============================] - 3s 553ms/step\n",
      "2/2 [==============================] - 1s 520ms/step\n",
      "4/4 [==============================] - 2s 407ms/step\n",
      "4/4 [==============================] - 2s 426ms/step\n",
      "5/5 [==============================] - 2s 465ms/step\n",
      "2/2 [==============================] - 1s 541ms/step\n",
      "1/1 [==============================] - 0s 449ms/step\n",
      "3/3 [==============================] - 2s 502ms/step\n",
      "3/3 [==============================] - 1s 414ms/step\n",
      "5/5 [==============================] - 3s 562ms/step\n",
      "6/6 [==============================] - 3s 441ms/step\n",
      "2/2 [==============================] - 1s 350ms/step\n",
      "6/6 [==============================] - 3s 418ms/step\n",
      "4/4 [==============================] - 2s 412ms/step\n",
      "1/1 [==============================] - 0s 307ms/step\n",
      "2/2 [==============================] - 1s 84ms/step\n",
      "4/4 [==============================] - 2s 359ms/step\n",
      "2/2 [==============================] - 1s 179ms/step\n",
      "2/2 [==============================] - 1s 196ms/step\n",
      "3/3 [==============================] - 1s 408ms/step\n",
      "5/5 [==============================] - 2s 467ms/step\n",
      "8/8 [==============================] - 4s 438ms/step\n",
      "4/4 [==============================] - 2s 384ms/step\n",
      "7/7 [==============================] - 3s 418ms/step\n",
      "6/6 [==============================] - 3s 422ms/step\n",
      "3/3 [==============================] - 1s 461ms/step\n",
      "4/4 [==============================] - 2s 411ms/step\n",
      "5/5 [==============================] - 2s 456ms/step\n",
      "5/5 [==============================] - 2s 397ms/step\n",
      "5/5 [==============================] - 2s 455ms/step\n",
      "3/3 [==============================] - 1s 366ms/step\n",
      "11/11 [==============================] - 5s 455ms/step\n",
      "3/3 [==============================] - 1s 324ms/step\n",
      "7/7 [==============================] - 3s 445ms/step\n",
      "7/7 [==============================] - 4s 507ms/step\n",
      "15/15 [==============================] - 7s 477ms/step\n",
      "1/1 [==============================] - 0s 380ms/step\n",
      "6/6 [==============================] - 3s 411ms/step\n",
      "2/2 [==============================] - 1s 47ms/step\n",
      "1/1 [==============================] - 0s 481ms/step\n",
      "1/1 [==============================] - 0s 266ms/step\n",
      "1/1 [==============================] - 0s 163ms/step\n",
      "2/2 [==============================] - 1s 133ms/step\n",
      "2/2 [==============================] - 1s 486ms/step\n",
      "1/1 [==============================] - 0s 368ms/step\n",
      "2/2 [==============================] - 1s 462ms/step\n",
      "3/3 [==============================] - 1s 296ms/step\n",
      "5/5 [==============================] - 2s 433ms/step\n",
      "3/3 [==============================] - 1s 407ms/step\n",
      "9/9 [==============================] - 4s 448ms/step\n",
      "2/2 [==============================] - 1s 221ms/step\n",
      "2/2 [==============================] - 1s 447ms/step\n",
      "3/3 [==============================] - 1s 451ms/step\n",
      "3/3 [==============================] - 1s 405ms/step\n",
      "2/2 [==============================] - 1s 334ms/step\n",
      "1/1 [==============================] - 0s 480ms/step\n",
      "4/4 [==============================] - 2s 341ms/step\n",
      "5/5 [==============================] - 2s 382ms/step\n",
      "4/4 [==============================] - 2s 360ms/step\n",
      "1/1 [==============================] - 1s 504ms/step\n",
      "1/1 [==============================] - 1s 576ms/step\n",
      "4/4 [==============================] - 2s 422ms/step\n",
      "3/3 [==============================] - 1s 287ms/step\n",
      "2/2 [==============================] - 1s 454ms/step\n",
      "3/3 [==============================] - 1s 483ms/step\n",
      "3/3 [==============================] - 1s 321ms/step\n",
      "5/5 [==============================] - 2s 388ms/step\n",
      "9/9 [==============================] - 4s 455ms/step\n",
      "2/2 [==============================] - 1s 243ms/step\n",
      "1/1 [==============================] - 0s 346ms/step\n",
      "1/1 [==============================] - 0s 292ms/step\n",
      "2/2 [==============================] - 1s 255ms/step\n",
      "3/3 [==============================] - 1s 495ms/step\n",
      "3/3 [==============================] - 1s 306ms/step\n",
      "2/2 [==============================] - 1s 46ms/step\n",
      "5/5 [==============================] - 2s 464ms/step\n",
      "2/2 [==============================] - 1s 244ms/step\n",
      "2/2 [==============================] - 1s 131ms/step\n",
      "4/4 [==============================] - 2s 354ms/step\n",
      "8/8 [==============================] - 4s 462ms/step\n",
      "2/2 [==============================] - 1s 103ms/step\n",
      "2/2 [==============================] - 1s 256ms/step\n",
      "1/1 [==============================] - 0s 399ms/step\n",
      "1/1 [==============================] - 0s 390ms/step\n",
      "3/3 [==============================] - 1s 461ms/step\n",
      "3/3 [==============================] - 1s 316ms/step\n",
      "3/3 [==============================] - 1s 294ms/step\n",
      "1/1 [==============================] - 0s 468ms/step\n",
      "2/2 [==============================] - 1s 433ms/step\n",
      "2/2 [==============================] - 1s 86ms/step\n",
      "2/2 [==============================] - 1s 125ms/step\n",
      "4/4 [==============================] - 2s 422ms/step\n",
      "3/3 [==============================] - 1s 296ms/step\n",
      "1/1 [==============================] - 0s 263ms/step\n",
      "5/5 [==============================] - 2s 413ms/step\n",
      "2/2 [==============================] - 1s 62ms/step\n",
      "1/1 [==============================] - 0s 281ms/step\n",
      "2/2 [==============================] - 1s 309ms/step\n",
      "7/7 [==============================] - 3s 442ms/step\n",
      "2/2 [==============================] - 1s 83ms/step\n",
      "16/16 [==============================] - 8s 489ms/step\n",
      "4/4 [==============================] - 2s 362ms/step\n",
      "4/4 [==============================] - 2s 473ms/step\n",
      "5/5 [==============================] - 2s 460ms/step\n",
      "3/3 [==============================] - 1s 279ms/step\n",
      "5/5 [==============================] - 2s 431ms/step\n",
      "1/1 [==============================] - 0s 393ms/step\n",
      "2/2 [==============================] - 1s 365ms/step\n",
      "3/3 [==============================] - 1s 402ms/step\n",
      "3/3 [==============================] - 1s 382ms/step\n",
      "1/1 [==============================] - 0s 258ms/step\n",
      "5/5 [==============================] - 2s 382ms/step\n",
      "3/3 [==============================] - 1s 330ms/step\n",
      "2/2 [==============================] - 1s 142ms/step\n",
      "7/7 [==============================] - 3s 437ms/step\n",
      "3/3 [==============================] - 1s 334ms/step\n",
      "1/1 [==============================] - 0s 325ms/step\n",
      "2/2 [==============================] - 1s 69ms/step\n",
      "1/1 [==============================] - 0s 393ms/step\n",
      "4/4 [==============================] - 2s 408ms/step\n",
      "8/8 [==============================] - 4s 461ms/step\n",
      "2/2 [==============================] - 1s 227ms/step\n",
      "2/2 [==============================] - 1s 228ms/step\n",
      "3/3 [==============================] - 1s 359ms/step\n",
      "2/2 [==============================] - 1s 285ms/step\n",
      "2/2 [==============================] - 1s 116ms/step\n",
      "Tokenizing text:  1\n",
      "Tokenizing sentences\n",
      "3/3 [==============================] - 1s 486ms/step\n",
      "2/2 [==============================] - 1s 416ms/step\n",
      "4/4 [==============================] - 2s 382ms/step\n",
      "1/1 [==============================] - 0s 326ms/step\n",
      "6/6 [==============================] - 3s 433ms/step\n",
      "1/1 [==============================] - 0s 408ms/step\n",
      "3/3 [==============================] - 1s 434ms/step\n",
      "2/2 [==============================] - 1s 380ms/step\n",
      "2/2 [==============================] - 1s 52ms/step\n",
      "4/4 [==============================] - 2s 473ms/step\n",
      "2/2 [==============================] - 1s 182ms/step\n",
      "4/4 [==============================] - 2s 383ms/step\n",
      "7/7 [==============================] - 3s 492ms/step\n",
      "1/1 [==============================] - 0s 175ms/step\n",
      "1/1 [==============================] - 0s 281ms/step\n",
      "1/1 [==============================] - 0s 460ms/step\n",
      "4/4 [==============================] - 2s 412ms/step\n",
      "2/2 [==============================] - 1s 241ms/step\n",
      "3/3 [==============================] - 1s 439ms/step\n",
      "1/1 [==============================] - 0s 142ms/step\n",
      "2/2 [==============================] - 1s 145ms/step\n",
      "2/2 [==============================] - 1s 332ms/step\n",
      "1/1 [==============================] - 0s 478ms/step\n",
      "3/3 [==============================] - 1s 486ms/step\n",
      "2/2 [==============================] - 1s 312ms/step\n",
      "1/1 [==============================] - 0s 444ms/step\n",
      "1/1 [==============================] - 0s 267ms/step\n",
      "1/1 [==============================] - 0s 359ms/step\n",
      "2/2 [==============================] - 1s 138ms/step\n",
      "1/1 [==============================] - 0s 356ms/step\n",
      "1/1 [==============================] - 0s 317ms/step\n",
      "2/2 [==============================] - 1s 370ms/step\n",
      "2/2 [==============================] - 1s 117ms/step\n",
      "4/4 [==============================] - 2s 480ms/step\n",
      "2/2 [==============================] - 1s 168ms/step\n",
      "2/2 [==============================] - 1s 126ms/step\n",
      "1/1 [==============================] - 0s 428ms/step\n",
      "4/4 [==============================] - 2s 370ms/step\n",
      "2/2 [==============================] - 1s 151ms/step\n",
      "5/5 [==============================] - 2s 401ms/step\n",
      "3/3 [==============================] - 1s 423ms/step\n",
      "2/2 [==============================] - 1s 291ms/step\n",
      "3/3 [==============================] - 1s 357ms/step\n",
      "1/1 [==============================] - 0s 346ms/step\n",
      "1/1 [==============================] - 0s 219ms/step\n",
      "4/4 [==============================] - 2s 409ms/step\n",
      "2/2 [==============================] - 1s 339ms/step\n",
      "4/4 [==============================] - 2s 482ms/step\n",
      "1/1 [==============================] - 0s 384ms/step\n",
      "1/1 [==============================] - 0s 474ms/step\n",
      "1/1 [==============================] - 0s 318ms/step\n",
      "1/1 [==============================] - 0s 341ms/step\n",
      "5/5 [==============================] - 2s 469ms/step\n",
      "1/1 [==============================] - 0s 232ms/step\n",
      "2/2 [==============================] - 1s 123ms/step\n",
      "1/1 [==============================] - 0s 334ms/step\n",
      "2/2 [==============================] - 1s 309ms/step\n",
      "3/3 [==============================] - 1s 410ms/step\n",
      "2/2 [==============================] - 1s 374ms/step\n",
      "1/1 [==============================] - 0s 388ms/step\n",
      "1/1 [==============================] - 0s 341ms/step\n",
      "1/1 [==============================] - 0s 251ms/step\n",
      "3/3 [==============================] - 1s 287ms/step\n",
      "2/2 [==============================] - 1s 120ms/step\n",
      "4/4 [==============================] - 2s 469ms/step\n",
      "1/1 [==============================] - 0s 220ms/step\n",
      "1/1 [==============================] - 0s 309ms/step\n",
      "2/2 [==============================] - 1s 52ms/step\n",
      "1/1 [==============================] - 0s 398ms/step\n",
      "1/1 [==============================] - 0s 497ms/step\n",
      "1/1 [==============================] - 0s 287ms/step\n",
      "2/2 [==============================] - 1s 246ms/step\n",
      "5/5 [==============================] - 2s 435ms/step\n",
      "2/2 [==============================] - 1s 482ms/step\n",
      "4/4 [==============================] - 2s 443ms/step\n",
      "2/2 [==============================] - 1s 295ms/step\n",
      "1/1 [==============================] - 0s 477ms/step\n",
      "1/1 [==============================] - 0s 409ms/step\n",
      "2/2 [==============================] - 1s 375ms/step\n",
      "1/1 [==============================] - 0s 302ms/step\n",
      "2/2 [==============================] - 1s 278ms/step\n",
      "1/1 [==============================] - 0s 249ms/step\n",
      "1/1 [==============================] - 0s 288ms/step\n",
      "1/1 [==============================] - 0s 382ms/step\n",
      "1/1 [==============================] - 0s 440ms/step\n",
      "2/2 [==============================] - 1s 124ms/step\n",
      "2/2 [==============================] - 1s 261ms/step\n",
      "3/3 [==============================] - 1s 293ms/step\n",
      "1/1 [==============================] - 0s 345ms/step\n",
      "2/2 [==============================] - 1s 224ms/step\n",
      "1/1 [==============================] - 0s 325ms/step\n",
      "1/1 [==============================] - 0s 253ms/step\n",
      "3/3 [==============================] - 1s 320ms/step\n",
      "2/2 [==============================] - 1s 197ms/step\n",
      "1/1 [==============================] - 0s 367ms/step\n",
      "1/1 [==============================] - 0s 323ms/step\n",
      "1/1 [==============================] - 0s 405ms/step\n",
      "2/2 [==============================] - 1s 254ms/step\n",
      "2/2 [==============================] - 1s 277ms/step\n",
      "3/3 [==============================] - 2s 522ms/step\n",
      "3/3 [==============================] - 1s 322ms/step\n",
      "2/2 [==============================] - 1s 155ms/step\n",
      "1/1 [==============================] - 0s 255ms/step\n",
      "1/1 [==============================] - 0s 253ms/step\n",
      "2/2 [==============================] - 1s 72ms/step\n",
      "2/2 [==============================] - 1s 184ms/step\n",
      "3/3 [==============================] - 1s 276ms/step\n",
      "3/3 [==============================] - 1s 415ms/step\n",
      "3/3 [==============================] - 1s 453ms/step\n",
      "4/4 [==============================] - 2s 451ms/step\n",
      "1/1 [==============================] - 0s 493ms/step\n",
      "3/3 [==============================] - 1s 344ms/step\n",
      "1/1 [==============================] - 0s 315ms/step\n",
      "1/1 [==============================] - 0s 363ms/step\n",
      "4/4 [==============================] - 2s 379ms/step\n",
      "7/7 [==============================] - 3s 443ms/step\n",
      "1/1 [==============================] - 0s 418ms/step\n",
      "4/4 [==============================] - 2s 423ms/step\n",
      "5/5 [==============================] - 2s 460ms/step\n",
      "3/3 [==============================] - 1s 299ms/step\n",
      "3/3 [==============================] - 1s 306ms/step\n",
      "2/2 [==============================] - 1s 456ms/step\n",
      "2/2 [==============================] - 1s 167ms/step\n",
      "2/2 [==============================] - 1s 389ms/step\n",
      "2/2 [==============================] - 1s 182ms/step\n",
      "2/2 [==============================] - 1s 132ms/step\n",
      "2/2 [==============================] - 1s 254ms/step\n",
      "2/2 [==============================] - 1s 366ms/step\n",
      "4/4 [==============================] - 2s 447ms/step\n",
      "3/3 [==============================] - 1s 310ms/step\n",
      "2/2 [==============================] - 1s 202ms/step\n",
      "4/4 [==============================] - 2s 414ms/step\n",
      "2/2 [==============================] - 1s 171ms/step\n",
      "4/4 [==============================] - 2s 352ms/step\n",
      "1/1 [==============================] - 0s 490ms/step\n",
      "2/2 [==============================] - 1s 245ms/step\n",
      "6/6 [==============================] - 3s 418ms/step\n",
      "1/1 [==============================] - 0s 451ms/step\n",
      "6/6 [==============================] - 3s 428ms/step\n",
      "Tokenizing text:  1\n",
      "Tokenizing sentences\n",
      "2/2 [==============================] - 1s 168ms/step\n",
      "6/6 [==============================] - 3s 463ms/step\n",
      "2/2 [==============================] - 1s 370ms/step\n",
      "3/3 [==============================] - 1s 376ms/step\n",
      "2/2 [==============================] - 1s 134ms/step\n",
      "9/9 [==============================] - 4s 449ms/step\n",
      "2/2 [==============================] - 1s 233ms/step\n",
      "3/3 [==============================] - 1s 343ms/step\n",
      "1/1 [==============================] - 0s 375ms/step\n",
      "1/1 [==============================] - 0s 160ms/step\n",
      "1/1 [==============================] - 0s 219ms/step\n",
      "2/2 [==============================] - 1s 347ms/step\n",
      "1/1 [==============================] - 0s 403ms/step\n",
      "2/2 [==============================] - 1s 230ms/step\n",
      "3/3 [==============================] - 1s 359ms/step\n",
      "2/2 [==============================] - 1s 117ms/step\n",
      "2/2 [==============================] - 1s 214ms/step\n",
      "1/1 [==============================] - 0s 419ms/step\n",
      "5/5 [==============================] - 2s 416ms/step\n",
      "2/2 [==============================] - 1s 149ms/step\n",
      "3/3 [==============================] - 1s 291ms/step\n",
      "8/8 [==============================] - 4s 491ms/step\n",
      "3/3 [==============================] - 1s 376ms/step\n",
      "2/2 [==============================] - 1s 347ms/step\n",
      "1/1 [==============================] - 0s 155ms/step\n",
      "1/1 [==============================] - 0s 307ms/step\n",
      "2/2 [==============================] - 1s 72ms/step\n",
      "1/1 [==============================] - 0s 418ms/step\n",
      "2/2 [==============================] - 1s 298ms/step\n",
      "3/3 [==============================] - 1s 309ms/step\n",
      "1/1 [==============================] - 0s 202ms/step\n",
      "1/1 [==============================] - 0s 292ms/step\n",
      "1/1 [==============================] - 0s 152ms/step\n",
      "1/1 [==============================] - 0s 147ms/step\n",
      "1/1 [==============================] - 0s 179ms/step\n",
      "1/1 [==============================] - 0s 268ms/step\n",
      "1/1 [==============================] - 0s 223ms/step\n",
      "4/4 [==============================] - 2s 499ms/step\n",
      "1/1 [==============================] - 0s 494ms/step\n",
      "2/2 [==============================] - 1s 139ms/step\n",
      "2/2 [==============================] - 1s 500ms/step\n",
      "2/2 [==============================] - 1s 316ms/step\n",
      "2/2 [==============================] - 1s 161ms/step\n",
      "5/5 [==============================] - 2s 401ms/step\n",
      "1/1 [==============================] - 0s 379ms/step\n",
      "7/7 [==============================] - 3s 443ms/step\n",
      "2/2 [==============================] - 1s 66ms/step\n",
      "1/1 [==============================] - 0s 225ms/step\n",
      "1/1 [==============================] - 1s 511ms/step\n",
      "1/1 [==============================] - 0s 339ms/step\n",
      "1/1 [==============================] - 0s 449ms/step\n",
      "6/6 [==============================] - 3s 406ms/step\n",
      "2/2 [==============================] - 1s 342ms/step\n",
      "1/1 [==============================] - 0s 341ms/step\n",
      "1/1 [==============================] - 0s 146ms/step\n",
      "3/3 [==============================] - 1s 314ms/step\n",
      "1/1 [==============================] - 0s 244ms/step\n",
      "1/1 [==============================] - 0s 135ms/step\n",
      "1/1 [==============================] - 0s 295ms/step\n",
      "1/1 [==============================] - 0s 193ms/step\n",
      "2/2 [==============================] - 1s 318ms/step\n",
      "1/1 [==============================] - 0s 173ms/step\n",
      "5/5 [==============================] - 2s 444ms/step\n",
      "1/1 [==============================] - 0s 467ms/step\n",
      "5/5 [==============================] - 2s 443ms/step\n",
      "2/2 [==============================] - 1s 368ms/step\n",
      "1/1 [==============================] - 0s 414ms/step\n",
      "3/3 [==============================] - 1s 465ms/step\n",
      "2/2 [==============================] - 1s 118ms/step\n",
      "2/2 [==============================] - 1s 485ms/step\n",
      "3/3 [==============================] - 1s 288ms/step\n",
      "1/1 [==============================] - 0s 399ms/step\n",
      "2/2 [==============================] - 1s 193ms/step\n",
      "3/3 [==============================] - 1s 342ms/step\n",
      "4/4 [==============================] - 2s 375ms/step\n",
      "1/1 [==============================] - 0s 444ms/step\n",
      "4/4 [==============================] - 2s 397ms/step\n",
      "2/2 [==============================] - 1s 466ms/step\n",
      "1/1 [==============================] - 0s 270ms/step\n",
      "2/2 [==============================] - 1s 483ms/step\n",
      "1/1 [==============================] - 0s 455ms/step\n",
      "1/1 [==============================] - 0s 319ms/step\n",
      "2/2 [==============================] - 1s 465ms/step\n",
      "2/2 [==============================] - 1s 134ms/step\n",
      "2/2 [==============================] - 1s 235ms/step\n",
      "7/7 [==============================] - 3s 447ms/step\n",
      "2/2 [==============================] - 1s 169ms/step\n",
      "1/1 [==============================] - 0s 234ms/step\n",
      "2/2 [==============================] - 1s 78ms/step\n",
      "1/1 [==============================] - 0s 179ms/step\n",
      "1/1 [==============================] - 0s 120ms/step\n",
      "2/2 [==============================] - 1s 134ms/step\n",
      "2/2 [==============================] - 1s 69ms/step\n",
      "1/1 [==============================] - 0s 429ms/step\n",
      "1/1 [==============================] - 0s 469ms/step\n",
      "1/1 [==============================] - 0s 178ms/step\n",
      "1/1 [==============================] - 0s 147ms/step\n",
      "1/1 [==============================] - 0s 373ms/step\n",
      "1/1 [==============================] - 0s 402ms/step\n",
      "3/3 [==============================] - 1s 416ms/step\n",
      "1/1 [==============================] - 0s 495ms/step\n",
      "2/2 [==============================] - 1s 462ms/step\n",
      "2/2 [==============================] - 1s 397ms/step\n",
      "1/1 [==============================] - 0s 471ms/step\n",
      "1/1 [==============================] - 0s 408ms/step\n",
      "2/2 [==============================] - 1s 351ms/step\n",
      "6/6 [==============================] - 3s 477ms/step\n",
      "3/3 [==============================] - 1s 312ms/step\n",
      "3/3 [==============================] - 1s 327ms/step\n",
      "1/1 [==============================] - 0s 265ms/step\n",
      "1/1 [==============================] - 0s 419ms/step\n",
      "1/1 [==============================] - 0s 380ms/step\n",
      "2/2 [==============================] - 1s 84ms/step\n",
      "2/2 [==============================] - 1s 348ms/step\n",
      "3/3 [==============================] - 2s 499ms/step\n",
      "1/1 [==============================] - 0s 218ms/step\n",
      "1/1 [==============================] - 0s 494ms/step\n",
      "2/2 [==============================] - 1s 289ms/step\n",
      "1/1 [==============================] - 0s 142ms/step\n",
      "3/3 [==============================] - 1s 290ms/step\n",
      "2/2 [==============================] - 1s 187ms/step\n",
      "3/3 [==============================] - 1s 300ms/step\n",
      "4/4 [==============================] - 2s 390ms/step\n",
      "3/3 [==============================] - 1s 479ms/step\n",
      "5/5 [==============================] - 2s 437ms/step\n",
      "3/3 [==============================] - 1s 386ms/step\n",
      "2/2 [==============================] - 1s 178ms/step\n",
      "3/3 [==============================] - 1s 303ms/step\n",
      "2/2 [==============================] - 1s 490ms/step\n",
      "2/2 [==============================] - 1s 121ms/step\n",
      "1/1 [==============================] - 0s 221ms/step\n",
      "2/2 [==============================] - 1s 199ms/step\n",
      "10/10 [==============================] - 5s 484ms/step\n",
      "6/6 [==============================] - 3s 421ms/step\n",
      "2/2 [==============================] - 1s 45ms/step\n",
      "3/3 [==============================] - 1s 309ms/step\n",
      "2/2 [==============================] - 1s 352ms/step\n",
      "1/1 [==============================] - 0s 465ms/step\n",
      "1/1 [==============================] - 0s 362ms/step\n",
      "2/2 [==============================] - 1s 197ms/step\n",
      "5/5 [==============================] - 2s 467ms/step\n",
      "2/2 [==============================] - 1s 201ms/step\n",
      "5/5 [==============================] - 2s 406ms/step\n",
      "2/2 [==============================] - 1s 179ms/step\n",
      "3/3 [==============================] - 2s 502ms/step\n",
      "2/2 [==============================] - 1s 106ms/step\n",
      "3/3 [==============================] - 1s 440ms/step\n",
      "1/1 [==============================] - 0s 345ms/step\n",
      "2/2 [==============================] - 1s 300ms/step\n",
      "1/1 [==============================] - 1s 502ms/step\n",
      "2/2 [==============================] - 1s 292ms/step\n",
      "2/2 [==============================] - 1s 485ms/step\n",
      "1/1 [==============================] - 0s 433ms/step\n",
      "2/2 [==============================] - 1s 166ms/step\n",
      "1/1 [==============================] - 0s 209ms/step\n",
      "2/2 [==============================] - 1s 169ms/step\n",
      "2/2 [==============================] - 1s 279ms/step\n",
      "2/2 [==============================] - 1s 341ms/step\n",
      "2/2 [==============================] - 1s 354ms/step\n",
      "1/1 [==============================] - 0s 272ms/step\n",
      "3/3 [==============================] - 1s 315ms/step\n",
      "1/1 [==============================] - 0s 492ms/step\n",
      "1/1 [==============================] - 0s 218ms/step\n",
      "1/1 [==============================] - 0s 281ms/step\n",
      "2/2 [==============================] - 1s 274ms/step\n",
      "2/2 [==============================] - 1s 162ms/step\n",
      "1/1 [==============================] - 0s 340ms/step\n",
      "1/1 [==============================] - 0s 451ms/step\n",
      "4/4 [==============================] - 2s 457ms/step\n",
      "3/3 [==============================] - 1s 303ms/step\n",
      "1/1 [==============================] - 0s 256ms/step\n",
      "4/4 [==============================] - 2s 422ms/step\n",
      "1/1 [==============================] - 0s 238ms/step\n",
      "1/1 [==============================] - 0s 463ms/step\n",
      "1/1 [==============================] - 0s 236ms/step\n",
      "1/1 [==============================] - 0s 199ms/step\n",
      "1/1 [==============================] - 0s 242ms/step\n",
      "1/1 [==============================] - 0s 265ms/step\n",
      "1/1 [==============================] - 0s 149ms/step\n",
      "1/1 [==============================] - 0s 349ms/step\n",
      "5/5 [==============================] - 2s 423ms/step\n",
      "2/2 [==============================] - 1s 348ms/step\n",
      "2/2 [==============================] - 1s 447ms/step\n",
      "2/2 [==============================] - 1s 462ms/step\n",
      "1/1 [==============================] - 0s 496ms/step\n",
      "1/1 [==============================] - 0s 476ms/step\n",
      "2/2 [==============================] - 1s 122ms/step\n",
      "1/1 [==============================] - 0s 186ms/step\n",
      "3/3 [==============================] - 1s 301ms/step\n",
      "2/2 [==============================] - 1s 212ms/step\n",
      "4/4 [==============================] - 2s 358ms/step\n",
      "2/2 [==============================] - 1s 72ms/step\n",
      "3/3 [==============================] - 1s 468ms/step\n",
      "1/1 [==============================] - 0s 327ms/step\n",
      "4/4 [==============================] - 2s 358ms/step\n",
      "2/2 [==============================] - 1s 119ms/step\n",
      "1/1 [==============================] - 0s 348ms/step\n",
      "1/1 [==============================] - 0s 336ms/step\n",
      "7/7 [==============================] - 3s 436ms/step\n",
      "2/2 [==============================] - 1s 319ms/step\n",
      "5/5 [==============================] - 2s 395ms/step\n",
      "8/8 [==============================] - 4s 443ms/step\n",
      "2/2 [==============================] - 1s 159ms/step\n",
      "4/4 [==============================] - 2s 423ms/step\n",
      "12/12 [==============================] - 6s 502ms/step\n",
      "2/2 [==============================] - 1s 420ms/step\n",
      "1/1 [==============================] - 0s 390ms/step\n",
      "1/1 [==============================] - 0s 348ms/step\n",
      "Tokenizing text:  1\n",
      "Tokenizing sentences\n",
      "3/3 [==============================] - 1s 493ms/step\n",
      "2/2 [==============================] - 1s 491ms/step\n",
      "4/4 [==============================] - 2s 387ms/step\n",
      "3/3 [==============================] - 1s 394ms/step\n",
      "2/2 [==============================] - 1s 347ms/step\n",
      "3/3 [==============================] - 1s 441ms/step\n",
      "3/3 [==============================] - 1s 396ms/step\n",
      "2/2 [==============================] - 1s 384ms/step\n",
      "1/1 [==============================] - 0s 384ms/step\n",
      "1/1 [==============================] - 0s 470ms/step\n",
      "4/4 [==============================] - 2s 403ms/step\n",
      "2/2 [==============================] - 1s 120ms/step\n",
      "2/2 [==============================] - 1s 232ms/step\n",
      "2/2 [==============================] - 1s 142ms/step\n",
      "4/4 [==============================] - 2s 412ms/step\n",
      "2/2 [==============================] - 1s 57ms/step\n",
      "1/1 [==============================] - 0s 489ms/step\n",
      "2/2 [==============================] - 1s 126ms/step\n",
      "2/2 [==============================] - 1s 170ms/step\n",
      "2/2 [==============================] - 1s 199ms/step\n",
      "2/2 [==============================] - 1s 430ms/step\n",
      "1/1 [==============================] - 0s 460ms/step\n",
      "2/2 [==============================] - 1s 255ms/step\n",
      "5/5 [==============================] - 2s 410ms/step\n",
      "3/3 [==============================] - 1s 333ms/step\n",
      "3/3 [==============================] - 1s 309ms/step\n",
      "1/1 [==============================] - 0s 445ms/step\n",
      "4/4 [==============================] - 2s 350ms/step\n",
      "2/2 [==============================] - 1s 262ms/step\n",
      "3/3 [==============================] - 1s 459ms/step\n",
      "4/4 [==============================] - 2s 415ms/step\n",
      "8/8 [==============================] - 4s 457ms/step\n",
      "6/6 [==============================] - 3s 452ms/step\n",
      "4/4 [==============================] - 2s 378ms/step\n",
      "3/3 [==============================] - 1s 302ms/step\n",
      "2/2 [==============================] - 1s 126ms/step\n",
      "2/2 [==============================] - 1s 348ms/step\n",
      "2/2 [==============================] - 1s 398ms/step\n",
      "2/2 [==============================] - 1s 134ms/step\n",
      "5/5 [==============================] - 2s 422ms/step\n",
      "1/1 [==============================] - 0s 264ms/step\n",
      "2/2 [==============================] - 1s 135ms/step\n",
      "4/4 [==============================] - 2s 395ms/step\n",
      "2/2 [==============================] - 1s 156ms/step\n",
      "7/7 [==============================] - 3s 479ms/step\n",
      "3/3 [==============================] - 1s 393ms/step\n",
      "2/2 [==============================] - 1s 446ms/step\n",
      "2/2 [==============================] - 1s 215ms/step\n",
      "3/3 [==============================] - 1s 357ms/step\n",
      "2/2 [==============================] - 1s 396ms/step\n",
      "4/4 [==============================] - 2s 472ms/step\n",
      "2/2 [==============================] - 1s 156ms/step\n",
      "6/6 [==============================] - 3s 471ms/step\n",
      "3/3 [==============================] - 1s 403ms/step\n",
      "1/1 [==============================] - 0s 218ms/step\n",
      "1/1 [==============================] - 0s 459ms/step\n",
      "3/3 [==============================] - 1s 329ms/step\n",
      "2/2 [==============================] - 1s 346ms/step\n",
      "2/2 [==============================] - 1s 314ms/step\n",
      "2/2 [==============================] - 1s 187ms/step\n",
      "3/3 [==============================] - 1s 495ms/step\n",
      "5/5 [==============================] - 2s 418ms/step\n",
      "1/1 [==============================] - 1s 511ms/step\n",
      "2/2 [==============================] - 1s 183ms/step\n",
      "2/2 [==============================] - 1s 355ms/step\n",
      "2/2 [==============================] - 1s 313ms/step\n",
      "2/2 [==============================] - 1s 164ms/step\n",
      "1/1 [==============================] - 0s 426ms/step\n",
      "2/2 [==============================] - 1s 415ms/step\n",
      "2/2 [==============================] - 1s 308ms/step\n",
      "2/2 [==============================] - 1s 496ms/step\n",
      "4/4 [==============================] - 2s 394ms/step\n",
      "4/4 [==============================] - 2s 454ms/step\n",
      "2/2 [==============================] - 1s 426ms/step\n",
      "3/3 [==============================] - 1s 361ms/step\n",
      "4/4 [==============================] - 2s 497ms/step\n",
      "4/4 [==============================] - 2s 472ms/step\n",
      "5/5 [==============================] - 2s 481ms/step\n",
      "1/1 [==============================] - 0s 448ms/step\n",
      "3/3 [==============================] - 1s 325ms/step\n",
      "2/2 [==============================] - 1s 127ms/step\n",
      "2/2 [==============================] - 1s 163ms/step\n",
      "2/2 [==============================] - 1s 121ms/step\n",
      "2/2 [==============================] - 1s 443ms/step\n",
      "3/3 [==============================] - 1s 307ms/step\n",
      "4/4 [==============================] - 2s 497ms/step\n",
      "5/5 [==============================] - 2s 448ms/step\n",
      "4/4 [==============================] - 2s 415ms/step\n",
      "5/5 [==============================] - 2s 431ms/step\n",
      "8/8 [==============================] - 4s 482ms/step\n",
      "3/3 [==============================] - 1s 376ms/step\n",
      "1/1 [==============================] - 0s 460ms/step\n",
      "3/3 [==============================] - 1s 332ms/step\n",
      "2/2 [==============================] - 1s 279ms/step\n",
      "3/3 [==============================] - 1s 476ms/step\n",
      "4/4 [==============================] - 2s 365ms/step\n",
      "3/3 [==============================] - 1s 496ms/step\n",
      "4/4 [==============================] - 2s 362ms/step\n",
      "2/2 [==============================] - 1s 480ms/step\n",
      "2/2 [==============================] - 1s 193ms/step\n",
      "1/1 [==============================] - 0s 385ms/step\n",
      "2/2 [==============================] - 1s 479ms/step\n",
      "2/2 [==============================] - 1s 132ms/step\n",
      "6/6 [==============================] - 3s 467ms/step\n",
      "2/2 [==============================] - 1s 52ms/step\n",
      "2/2 [==============================] - 1s 453ms/step\n",
      "2/2 [==============================] - 1s 84ms/step\n",
      "5/5 [==============================] - 2s 376ms/step\n",
      "3/3 [==============================] - 1s 324ms/step\n",
      "1/1 [==============================] - 0s 359ms/step\n",
      "1/1 [==============================] - 0s 378ms/step\n",
      "1/1 [==============================] - 0s 385ms/step\n",
      "6/6 [==============================] - 2s 391ms/step\n",
      "3/3 [==============================] - 1s 460ms/step\n",
      "1/1 [==============================] - 0s 197ms/step\n",
      "1/1 [==============================] - 0s 426ms/step\n",
      "4/4 [==============================] - 2s 497ms/step\n",
      "1/1 [==============================] - 1s 513ms/step\n",
      "2/2 [==============================] - 1s 125ms/step\n",
      "2/2 [==============================] - 1s 350ms/step\n",
      "3/3 [==============================] - 1s 391ms/step\n",
      "3/3 [==============================] - 1s 289ms/step\n",
      "1/1 [==============================] - 0s 452ms/step\n",
      "6/6 [==============================] - 3s 419ms/step\n",
      "2/2 [==============================] - 1s 354ms/step\n",
      "3/3 [==============================] - 1s 420ms/step\n",
      "2/2 [==============================] - 1s 150ms/step\n",
      "1/1 [==============================] - 0s 424ms/step\n",
      "1/1 [==============================] - 0s 465ms/step\n",
      "2/2 [==============================] - 1s 349ms/step\n",
      "3/3 [==============================] - 1s 423ms/step\n",
      "6/6 [==============================] - 3s 461ms/step\n",
      "5/5 [==============================] - 2s 391ms/step\n",
      "4/4 [==============================] - 2s 446ms/step\n",
      "5/5 [==============================] - 2s 423ms/step\n",
      "2/2 [==============================] - 1s 152ms/step\n",
      "1/1 [==============================] - 0s 164ms/step\n",
      "1/1 [==============================] - 0s 397ms/step\n",
      "1/1 [==============================] - 0s 185ms/step\n",
      "1/1 [==============================] - 1s 506ms/step\n",
      "1/1 [==============================] - 0s 449ms/step\n",
      "2/2 [==============================] - 1s 311ms/step\n"
     ]
    }
   ],
   "source": [
    "# tokenize the data\n",
    "tokenized_data = []\n",
    "\n",
    "i = 1\n",
    "for text in dataset:\n",
    "    print(\"Tokenizing text: \", i)\n",
    "    sentences = urduhack.tokenization.sentence_tokenizer(text)\n",
    "    words = []\n",
    "    print(\"Tokenizing sentences\")\n",
    "    for sentence in sentences:\n",
    "        words.extend(urduhack.tokenization.word_tokenizer(sentence))\n",
    "    tokenized_data.append(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Preprocess the tokenized data. Go through the data and use your own discretion to decide on what kind of pre-processing might be required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total words in the dataset:  16028\n"
     ]
    }
   ],
   "source": [
    "# sanity check, total number of words in the dataset\n",
    "total_words = 0\n",
    "for text in tokenized_data:\n",
    "    total_words += len(text)\n",
    "print(\"Total words in the dataset: \", total_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a list of tokens extracted from the dataset\n",
    "tokens = []\n",
    "for text in tokenized_data:\n",
    "    tokens.extend(text)\n",
    "\n",
    "# create a dictionary of tokens and their frequencies\n",
    "token_freq = {}\n",
    "for token in tokens:\n",
    "    if token in token_freq:\n",
    "        token_freq[token] += 1\n",
    "    else:\n",
    "        token_freq[token] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the tokens list to a file\n",
    "with open('tokens.txt', 'w') as f:\n",
    "    for token in tokens:\n",
    "        f.write(token + '\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read the tokens from the file\n",
    "with open('tokens.txt', 'r') as f:\n",
    "    tokens = f.read().splitlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# clean up the tokens\n",
    "tokens = [urduhack.preprocessing.remove_punctuation(token).strip() for token in tokens if token != '']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15994"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a dictionary of tokens and their frequencies\n",
    "token_freq = {}\n",
    "for token in tokens:\n",
    "    if token in token_freq:\n",
    "        token_freq[token] += 1\n",
    "    else:\n",
    "        token_freq[token] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 - Creating Unigrams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start by training a unigram model. For a unigram model, the n-gram probability is approximated by probability of the word in the unigram, as the model assumes independence:\n",
    "\n",
    "$$\n",
    "P(word) = \\frac{n}{N}\n",
    "$$\n",
    "\n",
    "where n = count of the word in the corpus and N = total number of words in the corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate a list of unigrams. Print the first 10 unigrams obtained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total unigrams:  15994\n",
      "First 10 unigrams are:  ['ہم', 'نے', 'کالج', 'میں', 'تعلیم', 'تو', 'ضرور', 'پائی', 'اور', 'رفتہ']\n"
     ]
    }
   ],
   "source": [
    "unigrams = tokens\n",
    "\n",
    "print(\"Total unigrams: \", len(unigrams))\n",
    "print(\"First 10 unigrams are: \", unigrams[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the probabilities for each unique unigram. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unigram</th>\n",
       "      <th>Probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ہم</td>\n",
       "      <td>0.006627</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>نے</td>\n",
       "      <td>0.010066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>کالج</td>\n",
       "      <td>0.001501</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>میں</td>\n",
       "      <td>0.031074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>تعلیم</td>\n",
       "      <td>0.000688</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Unigram  Probability\n",
       "0      ہم     0.006627\n",
       "1      نے     0.010066\n",
       "2    کالج     0.001501\n",
       "3     میں     0.031074\n",
       "4   تعلیم     0.000688"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create a dataframe for the unigrams and their probabilities\n",
    "unigram_df = pd.DataFrame(list(token_freq.items()), columns=['Unigram', 'Probability'])\n",
    "unigram_df['Probability'] = unigram_df['Probability'] / unigram_df['Probability'].sum()\n",
    "\n",
    "unigram_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 - Creating Bigrams\n",
    "Now train a bigram model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate a list of bigrams. Print the first 10 bigrams obtained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total bigrams:  15993\n",
      "First 10 bigrams are:  ['ہم نے', 'نے کالج', 'کالج میں', 'میں تعلیم', 'تعلیم تو', 'تو ضرور', 'ضرور پائی', 'پائی اور', 'اور رفتہ', 'رفتہ رفتہ']\n"
     ]
    }
   ],
   "source": [
    "# code here\n",
    "bigrams = []\n",
    "for i in range(len(tokens) - 1):\n",
    "    bigrams.append(tokens[i] + ' ' + tokens[i + 1])\n",
    "\n",
    "print(\"Total bigrams: \", len(bigrams))\n",
    "print(\"First 10 bigrams are: \", bigrams[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the probabilities for each unique bigram. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# code here\n",
    "# create a dictionary of bigrams and their frequencies\n",
    "bigram_freq = {}\n",
    "for bigram in bigrams:\n",
    "    if bigram in bigram_freq:\n",
    "        bigram_freq[bigram] += 1\n",
    "    else:\n",
    "        bigram_freq[bigram] = 1\n",
    "\n",
    "# create a dataframe for the bigrams and their probabilities\n",
    "bigram_df = pd.DataFrame(list(bigram_freq.items()), columns=['Bigram', 'Probability'])\n",
    "\n",
    "# calculate the probability of each bigram\n",
    "bigram_df['Probability'] = bigram_df.apply(lambda row: row['Probability'] / token_freq[row['Bigram'].split()[0]], axis=1)\n",
    "\n",
    "bigram_df.head()\n",
    "\n",
    "# save bigram_df to a csv file\n",
    "bigram_df.to_csv('bigram_df.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 - Creating Trigrams\n",
    "Lastly train a trigram model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate a list of trigrams. Print the first 10 trigrams obtained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total trigrams:  15992\n",
      "First 10 trigrams are:  ['ہم نے کالج', 'نے کالج میں', 'کالج میں تعلیم', 'میں تعلیم تو', 'تعلیم تو ضرور', 'تو ضرور پائی', 'ضرور پائی اور', 'پائی اور رفتہ', 'اور رفتہ رفتہ', 'رفتہ رفتہ بی']\n"
     ]
    }
   ],
   "source": [
    "# code here\n",
    "trigrams = []\n",
    "for i in range(len(tokens) - 2):\n",
    "    trigrams.append(tokens[i] + ' ' + tokens[i + 1] + ' ' + tokens[i + 2])\n",
    "\n",
    "print(\"Total trigrams: \", len(trigrams))\n",
    "print(\"First 10 trigrams are: \", trigrams[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the probabilities for each unique trigram. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Trigram</th>\n",
       "      <th>Probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ہم نے کالج</td>\n",
       "      <td>0.028571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>نے کالج میں</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>کالج میں تعلیم</td>\n",
       "      <td>0.222222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>میں تعلیم تو</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>تعلیم تو ضرور</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Trigram  Probability\n",
       "0      ہم نے کالج     0.028571\n",
       "1     نے کالج میں     1.000000\n",
       "2  کالج میں تعلیم     0.222222\n",
       "3    میں تعلیم تو     0.500000\n",
       "4   تعلیم تو ضرور     1.000000"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# code here\n",
    "# create a dictionary of trigrams and their frequencies\n",
    "trigram_freq = {}\n",
    "for trigram in trigrams:\n",
    "    if trigram in trigram_freq:\n",
    "        trigram_freq[trigram] += 1\n",
    "    else:\n",
    "        trigram_freq[trigram] = 1\n",
    "\n",
    "# create a dataframe for the trigrams and their probabilities\n",
    "trigram_df = pd.DataFrame(list(trigram_freq.items()), columns=['Trigram', 'Probability'])\n",
    "\n",
    "# calculate the probability of each trigram\n",
    "trigram_df['Probability'] = trigram_df.apply(lambda row: row['Probability'] / bigram_freq[' '.join(row['Trigram'].split()[:2])], axis=1)\n",
    "\n",
    "trigram_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.5 - Generating Text\n",
    "Generate a paragraph with ten sentences each containing 9-15 words (pick the length of the sentence randomly within this range) using you language model. Start with trigrams, use back-off technique (i.e. use n-1 gram) if a token is not available. \n",
    "\n",
    "For each word prediction, get top 5 most probabale words using the n-gram model and then pick the next word randomly from within these. This is being done to avoid excessive repetitive sequences in your generated text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generating text\n",
    "def generate_sentence(unigram_df, bigram_df, trigram_df, n=10, start_word=None):\n",
    "    # generate the first word\n",
    "    if start_word is None:\n",
    "        first_word = unigram_df.sample(n=1)['Unigram'].values[0]\n",
    "    \n",
    "    sentence = first_word\n",
    "\n",
    "    # generate the second word\n",
    "    second_word = bigram_df[bigram_df['Bigram'].str.startswith(first_word)].sample(n=1)['Bigram'].values[0].split()[1]\n",
    "    sentence += ' ' + second_word\n",
    "\n",
    "    # generate the rest of the words\n",
    "    for i in range(n - 2):\n",
    "        # pick the top 5 most probable trigrams that start with the last two words\n",
    "        top_5_trigrams = trigram_df[trigram_df['Trigram'].str.startswith(sentence.split()[-2] + ' ' + sentence.split()[-1])].sort_values(by='Probability', ascending=False).head(5)\n",
    "\n",
    "        # if no tri-gram is found, use the bigram model\n",
    "        if len(top_5_trigrams) == 0:\n",
    "            top_5_bigrams = bigram_df[bigram_df['Bigram'].str.startswith(sentence.split()[-1])].sort_values(by='Probability', ascending=False).head(5)\n",
    "\n",
    "            # if no bi-gram is found, use the unigram model\n",
    "            if len(top_5_bigrams) == 0:\n",
    "                top_5_unigrams = unigram_df[unigram_df['Unigram'].str.startswith(sentence.split()[-1])].sort_values(by='Probability', ascending=False).head(5)\n",
    "                next_word = top_5_unigrams.sample(n=1)['Unigram'].values[0]\n",
    "            else:\n",
    "                next_word = top_5_bigrams.sample(n=1)['Bigram'].values[0].split()[1]\n",
    "        else:\n",
    "            next_word = top_5_trigrams.sample(n=1)['Trigram'].values[0].split()[2]\n",
    "\n",
    "        sentence += ' ' + next_word\n",
    "\n",
    "\n",
    "    return sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'اشیاء کا ہےباقی رہے جنگل پہاڑ دریا تو وہاں بھی ایک نہ- قوم کا لکھا ہو سامنے نہایت بے تکلفی سے اپنی- صناعی اور ہنرمندی سے لکھے کہ کاتب قدرت نے بھی یہی مشورہ دیا- سمجھئے کہ لاہور لا ہو رہی ہے اگر اس- موقلم آپ کیون کر رکھ ہے کالج کی تعلیم حاصل کرنے کا- پکتی ہے چنانچہ ابو ثوق سے کہا کہ دل بھی ا نہ تو الہی- عرصے سے کمیٹی کے زیر غور ہے یہ حالت وجاندار اشیاء کا ہےباقی رہے- موقعے زیادہ ملتے رہتے ہیں دوسری قسم جلالی طلباء کی- مت باہم بحث مباحثے رہتے ہم میں سے چند مشہور ہیں قسم اولی جمالی- '"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "paragraph = \"\"\n",
    "for i in range(1, 10):\n",
    "    length = np.random.randint(9, 15)\n",
    "    paragraph += generate_sentence(unigram_df, bigram_df, trigram_df, n=length)\n",
    "    paragraph += '- '\n",
    "\n",
    "paragraph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15/15 [==============================] - 8s 566ms/step\n",
      "Total tokens:  63\n",
      "اشیاء 1\n",
      "کاہے 1\n",
      "باقی 1\n",
      "رہے 1\n",
      "جنگل 1\n",
      "پہاڑدریا 1\n",
      "تو 1\n",
      "وہاں 1\n",
      "بھی 2\n",
      "ایک 1\n",
      "نہ 1\n",
      "قوم 1\n",
      "کالکھاہوسامنے 1\n",
      "نہایت 1\n",
      "بے 1\n",
      "تکلفی 1\n",
      "سے 4\n",
      "اپنی 1\n",
      "صناعی 1\n",
      "اورہنرمندی 1\n",
      "لکھے 1\n",
      "کہ 3\n",
      "کاتب 1\n",
      "قدرت 1\n",
      "نے 1\n",
      "یہی 1\n",
      "مشورہ 1\n",
      "دیا 1\n",
      "سمجھئے 1\n",
      "لاہورلاہورہی 1\n",
      "ہے 1\n",
      "اگر 1\n",
      "اس 1\n",
      "موقلم 1\n",
      "آپ 1\n",
      "کیون 1\n",
      "کر 1\n",
      "رکھہے 1\n",
      "کالج 1\n",
      "کی 1\n",
      "تعلیم 1\n",
      "حاصل 1\n",
      "کرنے 1\n",
      "کاپکتیہے 1\n",
      "چنانچہ 1\n",
      "ابوثوق 1\n",
      "کہا 1\n",
      "دل 1\n",
      "بھیانہ 1\n",
      "توالہی 1\n",
      "عرصے 1\n",
      "کمیٹی 1\n",
      "کے 1\n",
      "زیرغورہے 1\n",
      "یہ 1\n",
      "حالت 1\n",
      "و 1\n"
     ]
    }
   ],
   "source": [
    "# analyze the generated text\n",
    "tokenized_paragraph = urduhack.tokenization.word_tokenizer(urduhack.preprocessing.remove_punctuation(paragraph))\n",
    "\n",
    "# create a dictionary of tokens and their frequencies\n",
    "token_freq = {}\n",
    "for token in tokenized_paragraph:\n",
    "    if token in token_freq:\n",
    "        token_freq[token] += 1\n",
    "    else:\n",
    "        token_freq[token] = 1\n",
    "\n",
    "print(\"Total tokens: \", len(tokenized_paragraph))\n",
    "\n",
    "for token, freq in token_freq.items():\n",
    "    print(token, freq)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.6 - Discussion and Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Analyze the text generated, and mention 3 distinct observations. Also compare it with the input text and how different it is and why might that be.\n",
    "- Is going upto n=3 enough? What do you think would be a good value of n and why?\n",
    "\n",
    "Answer here:\n",
    "\n",
    "1. Most of the tokens are unique and only a few are repeated. This is because the model is trained on a small dataset and the number of tokens is also small.\n",
    "2. Yes, going upto n=3 is generally enough. However, if the dataset is very large and the number of tokens is also large, then a higher value of n can be used. This is because the model will be able to learn more about the context of the tokens and will be able to generate more meaningful text."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.15 ('urduenv')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.15"
  },
  "vscode": {
   "interpreter": {
    "hash": "53da4203b51a5e8642ed38c1240b496943d6923d110002f5bcbe1c2c4e26d767"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}