2. RNN and Bi-directional RNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7ca825f6-25fd-4560-97dd-a48e5833817b",
   "metadata": {},
   "source": [
    "# Recurrent Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76ab7902-6270-4fc5-8471-8c3386fd82f9",
   "metadata": {},
   "source": [
    "## Basic Overview\n",
    "\n",
    "What is a RNN?\n",
    "\n",
    "Recurrent Neural Network(RNN) are a type of Neural Network where the output from previous step are fed as input to the current step. In traditional neural networks, all the inputs and outputs are independent of each other, but in cases like when it is required to predict the next word of a sentence, the previous words are required and hence there is a need to remember the previous words. Thus RNN came into existence, which solved this issue with the help of a Hidden Layer.\n",
    "\n",
    "Why RNN's?\n",
    "\n",
    "https://www.quora.com/Why-do-we-use-an-RNN-instead-of-a-simple-neural-network\n",
    "\n",
    "## In-Depth Understanding\n",
    "\n",
    "* https://medium.com/mindorks/understanding-the-recurrent-neural-network-44d593f112a2\n",
    "* https://www.youtube.com/watch?v=2E65LDnM2cA&list=PL1F3ABbhcqa3BBWo170U4Ev2wfsF7FN8l\n",
    "* https://www.d2l.ai/chapter_recurrent-neural-networks/rnn.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d79d9870-4d51-48f1-85b8-599a034592b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.imports import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b499b97b-80ff-42b7-b7af-0167ff860f8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "xtrain, xvalid, ytrain, yvalid = prepare_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14fa364a-d8fc-4c18-a1c7-029432313dcf",
   "metadata": {},
   "source": [
    "## Tokenization\n",
    "\n",
    "So, if you have watched the videos and referred to the links, you would know that in an RNN we input a sentence word by word. We represent every word as one hot vectors of dimensions : Numbers of words in Vocab + 1. <br>\n",
    "\n",
    "What keras Tokenizer does is, \n",
    "\n",
    "- it takes all the unique words in the corpus, forms a dictionary with words as keys and their number of occurences as values, it then sorts the dictionary in descending order of counts. \n",
    "- It then assigns the first value 1 , second value 2 and so on.\n",
    "\n",
    "So, let's suppose word 'the' occured the most in the corpus then it will assigned index 1 and vector representing 'the' would be a one-hot vector with value 1 at position 1 and rest zereos.<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2337b8d6-e9d1-48c3-a4b0-72e9ef78b452",
   "metadata": {},
   "outputs": [],
   "source": [
    "# using keras tokenizer here\n",
    "token = text.Tokenizer(num_words=None)\n",
    "max_len = 1500"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e5b24aa2-593c-4d66-b87a-b04115c37779",
   "metadata": {},
   "outputs": [],
   "source": [
    "token.fit_on_texts(list(xtrain) + list(xvalid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8779ca45-5bab-4f1c-9cef-d9dc5823a138",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([\"Congratulations\\nfor the Third of May 1808, reaching FA and it's still April, thanks for your input.....\",\n",
       "       'Jimbo.....the crybaby..... \\n\\nSits in front of his computer everyday acting as the arbiter of the grand Wikipedia. The all knowing know it all who is the expert on everything. The savior of the western world. And of course, when questioned about anything, cries to the main office about ill treatment.....By the way......a lousy editor who uses only one source!  A joke!'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xtrain[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6700a7df-d23b-4cf7-9d75-a0e61154008f",
   "metadata": {},
   "outputs": [],
   "source": [
    "xtrain_seq = token.texts_to_sequences(xtrain)\n",
    "xvalid_seq = token.texts_to_sequences(xvalid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c77d0a0a-29e2-4da1-950e-558bf56a5309",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[2858,\n",
       "  12,\n",
       "  1,\n",
       "  686,\n",
       "  3,\n",
       "  85,\n",
       "  6877,\n",
       "  6878,\n",
       "  2859,\n",
       "  4,\n",
       "  72,\n",
       "  153,\n",
       "  1107,\n",
       "  92,\n",
       "  12,\n",
       "  20,\n",
       "  1376]]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xtrain_seq[:1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7b91fcac-dad4-4223-8e67-0210218878bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "#zero pad the sequences\n",
    "xtrain_pad = pad_sequences(xtrain_seq, maxlen=max_len)\n",
    "xvalid_pad = pad_sequences(xvalid_seq, maxlen=max_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "89337814-6bdc-4b3f-ae49-2da32bd9823b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The code token.word_index simply gives the dictionary of vocab that keras created for us\n",
    "word_index = token.word_index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cc3bcb9-09be-4fa2-91bc-b846db7bbef4",
   "metadata": {},
   "source": [
    "Now you might be wondering What is padding? Why its done</b><br><br>\n",
    "\n",
    "Here is the answer :\n",
    "* https://www.quora.com/Which-effect-does-sequence-padding-have-on-the-training-of-a-neural-network\n",
    "* https://machinelearningmastery.com/data-preparation-variable-length-input-sequences-sequence-prediction/\n",
    "* https://www.coursera.org/lecture/natural-language-processing-tensorflow/padding-2Cyzs\n",
    "\n",
    "Also sometimes people might use special tokens while tokenizing like EOS(end of string) and BOS(Begining of string). Here is the reason why it's done\n",
    "* https://stackoverflow.com/questions/44579161/why-do-we-do-padding-in-nlp-tasks\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bf3365f-5b80-4f61-a665-f6b3d1d965aa",
   "metadata": {},
   "source": [
    "# Building the neural network"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98fb228a-4b7e-4a30-80eb-de2b9753f6e0",
   "metadata": {},
   "source": [
    "To understand the Dimensions of input and output given to RNN in keras her is a beautiful article : https://medium.com/@shivajbd/understanding-input-and-output-shape-in-lstm-keras-c501ee95c65e"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "976812a7-e59a-41a5-9f3d-2ac238dea2f5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " embedding (Embedding)       (None, 1500, 300)         4521300   \n",
      "                                                                 \n",
      " simple_rnn (SimpleRNN)      (None, 100)               40100     \n",
      "                                                                 \n",
      " dense (Dense)               (None, 1)                 101       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 4,561,501\n",
      "Trainable params: 4,561,501\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Wall time: 412 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# A simpleRNN without any pretrained embeddings and one dense layer\n",
    "model = Sequential()\n",
    "model.add(Embedding(len(word_index) + 1,\n",
    "                 300,\n",
    "                 input_length=max_len))\n",
    "model.add(SimpleRNN(100))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "    \n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "862ded67-fa87-47d9-86b0-baaeaa160e88",
   "metadata": {},
   "source": [
    "The first line model.Sequential() tells keras that we will be building our network sequentially . Then we first add the Embedding layer.\n",
    "Embedding layer is also a layer of neurons which takes in as input the nth dimensional one hot vector of every word and converts it into 300 dimensional vector , it gives us word embeddings similar to word2vec. We could have used word2vec but the embeddings layer learns during training to enhance the embeddings.\n",
    "Next we add an 100 LSTM units without any dropout or regularization\n",
    "At last we add a single neuron with sigmoid function which takes output from 100 LSTM cells (Please note we have 100 LSTM cells not layers) to predict the results and then we compile the model using adam optimizer "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8b49dc21-715d-43e0-be63-058df8a512d8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "25/25 [==============================] - 25s 962ms/step - loss: 0.3810 - accuracy: 0.8656\n",
      "Epoch 2/5\n",
      "25/25 [==============================] - 25s 991ms/step - loss: 0.2711 - accuracy: 0.9031\n",
      "Epoch 3/5\n",
      "25/25 [==============================] - 24s 948ms/step - loss: 0.3340 - accuracy: 0.8250\n",
      "Epoch 4/5\n",
      "25/25 [==============================] - 24s 944ms/step - loss: 0.3192 - accuracy: 0.9069\n",
      "Epoch 5/5\n",
      "25/25 [==============================] - 24s 945ms/step - loss: 0.2165 - accuracy: 0.9588\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1d524bdc610>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(xtrain_pad, ytrain, epochs=5, batch_size=64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b2c847c8-b8b3-4b2c-ae56-45313cf16d6b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13/13 [==============================] - 3s 195ms/step\n",
      "Auc: 0.74%\n"
     ]
    }
   ],
   "source": [
    "scores = model.predict(xvalid_pad)\n",
    "print(\"Auc: %.2f%%\" % (roc_auc(scores, yvalid)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a7692fa-96e8-4899-a3bf-ec1cbc128d06",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Bi-directional RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beb7fe6c-9919-4221-a0c5-def9689a01ba",
   "metadata": {},
   "source": [
    "## In Depth Explanation\n",
    "\n",
    "* https://www.coursera.org/learn/nlp-sequence-models/lecture/fyXnn/bidirectional-rnn\n",
    "* https://towardsdatascience.com/understanding-bidirectional-rnn-in-pytorch-5bd25a5dd66\n",
    "* https://d2l.ai/chapter_recurrent-modern/bi-rnn.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "67ef3d9b-8585-44b7-92c0-dbbdf5d9b853",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " embedding_1 (Embedding)     (None, 1500, 300)         4521300   \n",
      "                                                                 \n",
      " bidirectional (Bidirectiona  (None, 200)              80200     \n",
      " l)                                                              \n",
      "                                                                 \n",
      " dense_1 (Dense)             (None, 1)                 201       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 4,601,701\n",
      "Trainable params: 4,601,701\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "\n",
    "model.add(Embedding(len(word_index) + 1,\n",
    "                 300,\n",
    "                 input_length=max_len))\n",
    "model.add(Bidirectional(SimpleRNN(100)))\n",
    "\n",
    "model.add(Dense(1,activation='sigmoid'))\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam',metrics=['accuracy'])\n",
    "    \n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a48f4b58-f9b5-42ba-9099-bb7a801e8b0c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "25/25 [==============================] - 108s 4s/step - loss: 0.3505 - accuracy: 0.8838\n",
      "Epoch 2/5\n",
      "25/25 [==============================] - 111s 4s/step - loss: 0.2230 - accuracy: 0.9031\n",
      "Epoch 3/5\n",
      "25/25 [==============================] - 113s 5s/step - loss: 0.1303 - accuracy: 0.9419\n",
      "Epoch 4/5\n",
      "25/25 [==============================] - 119s 5s/step - loss: 0.0391 - accuracy: 0.9975\n",
      "Epoch 5/5\n",
      "25/25 [==============================] - 120s 5s/step - loss: 0.0149 - accuracy: 0.9994\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1d525498640>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(xtrain_pad, ytrain, epochs=5, batch_size=64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2161c852-2351-44d8-abce-2a6cc0fcde9f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13/13 [==============================] - 11s 862ms/step\n",
      "Auc: 0.62%\n"
     ]
    }
   ],
   "source": [
    "scores = model.predict(xvalid_pad)\n",
    "print(\"Auc: %.2f%%\" % (roc_auc(scores, yvalid)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d364c08-6e4b-422d-a33c-14d7ca5bb982",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}