1. Base Notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# About this Notebook\n",
    "\n",
    "NLP is a very hot topic right now and as belived by many experts '2020 is going to be NLP's Year' ,with its ever changing dynamics it is experiencing a boom , same as computer vision once did. Owing to its popularity Kaggle launched two NLP competitions recently and me being a lover of this Hot topic prepared myself to join in my first Kaggle Competition.<br><br>\n",
    "As I joined the competitions and since I was a complete beginner with Deep Learning Techniques for NLP, all my enthusiasm took a beating when I saw everyone Using all  kinds of BERT , everything just went over my head,I thought to quit but there is a special thing about Kaggle ,it just hooks you. I thought I have to learn someday , why not now , so I braced myself and sat on the learning curve. I wrote a kernel on the Tweet Sentiment Extraction competition that has now got a gold medal , it can be viewed here : https://www.kaggle.com/tanulsingh077/twitter-sentiment-extaction-analysis-eda-and-model <br><br>\n",
    "After 10 days of extensive learning(finishing all the latest NLP approaches) , I am back here to share my leaning , by writing a kernel that starts from the very Basic RNN's to built over , all the way to BERT . I invite you all to come and learn alongside with me and take a step closer towards becoming an NLP expert"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Contents\n",
    "\n",
    "In this Notebook I will start with the very Basics of RNN's and Build all the way to latest deep learning architectures to solve NLP problems. It will cover the Following:\n",
    "* Simple RNN's\n",
    "* Word Embeddings : Definition and How to get them\n",
    "* LSTM's\n",
    "* GRU's\n",
    "* BI-Directional RNN's\n",
    "* Encoder-Decoder Models (Seq2Seq Models)\n",
    "* Attention Models\n",
    "* Transformers - Attention is all you need\n",
    "* BERT\n",
    "\n",
    "I will divide every Topic into four subsections:\n",
    "* Basic Overview\n",
    "* In-Depth Understanding : In this I will attach links of articles and videos to learn about the topic in depth\n",
    "* Code-Implementation\n",
    "* Code Explanation\n",
    "\n",
    "This is a comprehensive kernel and if you follow along till the end , I promise you would learn all the techniques completely\n",
    "\n",
    "Note that the aim of this notebook is not to have a High LB score but to present a beginner guide to understand Deep Learning techniques used for NLP. Also after discussing all of these ideas , I will present a starter solution for this competiton"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<span style=\"color:Red\">This kernel has been a work of more than 10 days If you find my kernel useful and my efforts appreciable, Please Upvote it , it motivates me to write more Quality content**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "from tqdm import tqdm\n",
    "from sklearn.model_selection import train_test_split\n",
    "import tensorflow as tf\n",
    "from keras.models import Sequential\n",
    "from keras.layers.recurrent import LSTM, GRU,SimpleRNN\n",
    "from keras.layers.core import Dense, Activation, Dropout\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "from keras.utils import np_utils\n",
    "from sklearn import preprocessing, decomposition, model_selection, metrics, pipeline\n",
    "from keras.layers import GlobalMaxPooling1D, Conv1D, MaxPooling1D, Flatten, Bidirectional, SpatialDropout1D\n",
    "from keras.preprocessing import sequence, text\n",
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "from plotly import graph_objs as go\n",
    "import plotly.express as px\n",
    "import plotly.figure_factory as ff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Configuring TPU's\n",
    "\n",
    "For this version of Notebook we will be using TPU's as we have to built a BERT Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on TPU  grpc://10.0.0.2:8470\n",
      "REPLICAS:  8\n"
     ]
    }
   ],
   "source": [
    "# Detect hardware, return appropriate distribution strategy\n",
    "try:\n",
    "    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n",
    "    # set: this is always the case on Kaggle.\n",
    "    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n",
    "    print('Running on TPU ', tpu.master())\n",
    "except ValueError:\n",
    "    tpu = None\n",
    "\n",
    "if tpu:\n",
    "    tf.config.experimental_connect_to_cluster(tpu)\n",
    "    tf.tpu.experimental.initialize_tpu_system(tpu)\n",
    "    strategy = tf.distribute.experimental.TPUStrategy(tpu)\n",
    "else:\n",
    "    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n",
    "    strategy = tf.distribute.get_strategy()\n",
    "\n",
    "print(\"REPLICAS: \", strategy.num_replicas_in_sync)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"
   },
   "outputs": [],
   "source": [
    "train = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv')\n",
    "validation = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/validation.csv')\n",
    "test = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/test.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will drop the other columns and approach this problem as a Binary Classification Problem and also we will have our exercise done on a smaller subsection of the dataset(only 12000 data points) to make it easier to train the models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train.drop(['severe_toxic','obscene','threat','insult','identity_hate'],axis=1,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(12001, 3)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train = train.loc[:12000,:]\n",
    "train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will check the maximum number of words that can be present in a comment , this will help us in padding later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1403"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train['comment_text'].apply(lambda x:len(str(x).split())).max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Writing a function for getting auc score for validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def roc_auc(predictions,target):\n",
    "    '''\n",
    "    This methods returns the AUC Score when given the Predictions\n",
    "    and Labels\n",
    "    '''\n",
    "    \n",
    "    fpr, tpr, thresholds = metrics.roc_curve(target, predictions)\n",
    "    roc_auc = metrics.auc(fpr, tpr)\n",
    "    return roc_auc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "xtrain, xvalid, ytrain, yvalid = train_test_split(train.comment_text.values, train.toxic.values, \n",
    "                                                  stratify=train.toxic.values, \n",
    "                                                  random_state=42, \n",
    "                                                  test_size=0.2, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Before We Begin\n",
    "\n",
    "Before we Begin If you are a complete starter with NLP and never worked with text data, I am attaching a few kernels that will serve as a starting point of your journey\n",
    "* https://www.kaggle.com/arthurtok/spooky-nlp-and-topic-modelling-tutorial\n",
    "* https://www.kaggle.com/abhishek/approaching-almost-any-nlp-problem-on-kaggle\n",
    "\n",
    "If you want a more basic dataset to practice with here is another kernel which I wrote:\n",
    "* https://www.kaggle.com/tanulsingh077/what-s-cooking\n",
    "\n",
    "Below are some Resources to get started with basic level Neural Networks, It will help us to easily understand the upcoming parts\n",
    "* https://www.youtube.com/watch?v=aircAruvnKk&list=PL_h2yd2CGtBHEKwEH5iqTZH85wLS-eUzv\n",
    "* https://www.youtube.com/watch?v=IHZwWFHWa-w&list=PL_h2yd2CGtBHEKwEH5iqTZH85wLS-eUzv&index=2\n",
    "* https://www.youtube.com/watch?v=Ilg3gGewQ5U&list=PL_h2yd2CGtBHEKwEH5iqTZH85wLS-eUzv&index=3\n",
    "* https://www.youtube.com/watch?v=tIeHLnjs5U8&list=PL_h2yd2CGtBHEKwEH5iqTZH85wLS-eUzv&index=4\n",
    "\n",
    "For Learning how to visualize test data and what to use view:\n",
    "* https://www.kaggle.com/tanulsingh077/twitter-sentiment-extaction-analysis-eda-and-model\n",
    "* https://www.kaggle.com/jagangupta/stop-the-s-toxic-comments-eda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple RNN\n",
    "\n",
    "## Basic Overview\n",
    "\n",
    "What is a RNN?\n",
    "\n",
    "Recurrent Neural Network(RNN) are a type of Neural Network where the output from previous step are fed as input to the current step. In traditional neural networks, all the inputs and outputs are independent of each other, but in cases like when it is required to predict the next word of a sentence, the previous words are required and hence there is a need to remember the previous words. Thus RNN came into existence, which solved this issue with the help of a Hidden Layer.\n",
    "\n",
    "Why RNN's?\n",
    "\n",
    "https://www.quora.com/Why-do-we-use-an-RNN-instead-of-a-simple-neural-network\n",
    "\n",
    "## In-Depth Understanding\n",
    "\n",
    "* https://medium.com/mindorks/understanding-the-recurrent-neural-network-44d593f112a2\n",
    "* https://www.youtube.com/watch?v=2E65LDnM2cA&list=PL1F3ABbhcqa3BBWo170U4Ev2wfsF7FN8l\n",
    "* https://www.d2l.ai/chapter_recurrent-neural-networks/rnn.html\n",
    "\n",
    "## Code Implementation\n",
    "\n",
    "So first I will implement the and then I will explain the code step by step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# using keras tokenizer here\n",
    "token = text.Tokenizer(num_words=None)\n",
    "max_len = 1500\n",
    "\n",
    "token.fit_on_texts(list(xtrain) + list(xvalid))\n",
    "xtrain_seq = token.texts_to_sequences(xtrain)\n",
    "xvalid_seq = token.texts_to_sequences(xvalid)\n",
    "\n",
    "#zero pad the sequences\n",
    "xtrain_pad = sequence.pad_sequences(xtrain_seq, maxlen=max_len)\n",
    "xvalid_pad = sequence.pad_sequences(xvalid_seq, maxlen=max_len)\n",
    "\n",
    "word_index = token.word_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 1500, 300)         13049100  \n",
      "_________________________________________________________________\n",
      "simple_rnn_1 (SimpleRNN)     (None, 100)               40100     \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 101       \n",
      "=================================================================\n",
      "Total params: 13,089,301\n",
      "Trainable params: 13,089,301\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "CPU times: user 620 ms, sys: 370 ms, total: 990 ms\n",
      "Wall time: 1.18 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "with strategy.scope():\n",
    "    # A simpleRNN without any pretrained embeddings and one dense layer\n",
    "    model = Sequential()\n",
    "    model.add(Embedding(len(word_index) + 1,\n",
    "                     300,\n",
    "                     input_length=max_len))\n",
    "    model.add(SimpleRNN(100))\n",
    "    model.add(Dense(1, activation='sigmoid'))\n",
    "    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "    \n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:1: UserWarning:\n",
      "\n",
      "The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "9600/9600 [==============================] - 39s 4ms/step - loss: 0.3714 - accuracy: 0.8805\n",
      "Epoch 2/5\n",
      "9600/9600 [==============================] - 39s 4ms/step - loss: 0.2858 - accuracy: 0.9055\n",
      "Epoch 3/5\n",
      "9600/9600 [==============================] - 40s 4ms/step - loss: 0.2748 - accuracy: 0.8945\n",
      "Epoch 4/5\n",
      "9600/9600 [==============================] - 40s 4ms/step - loss: 0.2416 - accuracy: 0.9053\n",
      "Epoch 5/5\n",
      "9600/9600 [==============================] - 39s 4ms/step - loss: 0.2109 - accuracy: 0.9079\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.callbacks.History at 0x7fae866d75c0>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(xtrain_pad, ytrain, nb_epoch=5, batch_size=64*strategy.num_replicas_in_sync) #Multiplying by Strategy to run on TPU's"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Auc: 0.69%\n"
     ]
    }
   ],
   "source": [
    "scores = model.predict(xvalid_pad)\n",
    "print(\"Auc: %.2f%%\" % (roc_auc(scores,yvalid)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores_model = []\n",
    "scores_model.append({'Model': 'SimpleRNN','AUC_Score': roc_auc(scores,yvalid)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code Explanantion\n",
    "* Tokenization<br><br>\n",
    " So if you have watched the videos and referred to the links, you would know that in an RNN we input a sentence word by word. We represent every word as one hot vectors of dimensions : Numbers of words in Vocab +1. <br>\n",
    "  What keras Tokenizer does is , it takes all the unique words in the corpus,forms a dictionary with words as keys and their number of occurences as values,it then sorts the dictionary in descending order of counts. It then assigns the first value 1 , second value 2 and so on. So let's suppose word 'the' occured the most in the corpus then it will assigned index 1 and vector representing 'the' would be a one-hot vector with value 1 at position 1 and rest zereos.<br>\n",
    "  Try printing first 2 elements of xtrain_seq you will see every word is represented as a digit now"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[664,\n",
       "  65,\n",
       "  7,\n",
       "  19,\n",
       "  2262,\n",
       "  14102,\n",
       "  5,\n",
       "  2262,\n",
       "  20439,\n",
       "  6071,\n",
       "  4,\n",
       "  71,\n",
       "  32,\n",
       "  20440,\n",
       "  6620,\n",
       "  39,\n",
       "  6,\n",
       "  664,\n",
       "  65,\n",
       "  11,\n",
       "  8,\n",
       "  20441,\n",
       "  1502,\n",
       "  38,\n",
       "  6072]]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xtrain_seq[:1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Now you might be wondering What is padding? Why its done</b><br><br>\n",
    "\n",
    "Here is the answer :\n",
    "* https://www.quora.com/Which-effect-does-sequence-padding-have-on-the-training-of-a-neural-network\n",
    "* https://machinelearningmastery.com/data-preparation-variable-length-input-sequences-sequence-prediction/\n",
    "* https://www.coursera.org/lecture/natural-language-processing-tensorflow/padding-2Cyzs\n",
    "\n",
    "Also sometimes people might use special tokens while tokenizing like EOS(end of string) and BOS(Begining of string). Here is the reason why it's done\n",
    "* https://stackoverflow.com/questions/44579161/why-do-we-do-padding-in-nlp-tasks\n",
    "\n",
    "\n",
    "The code token.word_index simply gives the dictionary of vocab that keras created for us"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Building the Neural Network\n",
    "\n",
    "To understand the Dimensions of input and output given to RNN in keras her is a beautiful article : https://medium.com/@shivajbd/understanding-input-and-output-shape-in-lstm-keras-c501ee95c65e\n",
    "\n",
    "The first line model.Sequential() tells keras that we will be building our network sequentially . Then we first add the Embedding layer.\n",
    "Embedding layer is also a layer of neurons which takes in as input the nth dimensional one hot vector of every word and converts it into 300 dimensional vector , it gives us word embeddings similar to word2vec. We could have used word2vec but the embeddings layer learns during training to enhance the embeddings.\n",
    "Next we add an 100 LSTM units without any dropout or regularization\n",
    "At last we add a single neuron with sigmoid function which takes output from 100 LSTM cells (Please note we have 100 LSTM cells not layers) to predict the results and then we compile the model using adam optimizer \n",
    "\n",
    "* Comments on the model<br><br>\n",
    "We can see our model achieves an accuracy of 1 which is just insane , we are clearly overfitting I know , but this was the simplest model of all ,we can tune a lot of hyperparameters like RNN units, we can do batch normalization , dropouts etc to get better result. The point is we got an AUC score of 0.82 without much efforts and we know have learnt about RNN's .Deep learning is really revolutionary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word Embeddings\n",
    "\n",
    "While building our simple RNN models we talked about using word-embeddings , So what is word-embeddings and how do we get word-embeddings?\n",
    "Here is the answer :\n",
    "* https://www.coursera.org/learn/nlp-sequence-models/lecture/6Oq70/word-representation\n",
    "* https://machinelearningmastery.com/what-are-word-embeddings/\n",
    "<br> <br>\n",
    "The latest approach to getting word Embeddings is using pretained GLoVe or using Fasttext. Without going into too much details, I would explain how to create sentence vectors and how can we use them to create a machine learning model on top of it and since I am a fan of GloVe vectors, word2vec and fasttext. In this Notebook, I'll be using the GloVe vectors. You can download the GloVe vectors from here http://www-nlp.stanford.edu/data/glove.840B.300d.zip or you can search for GloVe in datasets on Kaggle and add the file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2196018it [06:43, 5439.09it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 2196017 word vectors.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# load the GloVe vectors in a dictionary:\n",
    "\n",
    "embeddings_index = {}\n",
    "f = open('/kaggle/input/glove840b300dtxt/glove.840B.300d.txt','r',encoding='utf-8')\n",
    "for line in tqdm(f):\n",
    "    values = line.split(' ')\n",
    "    word = values[0]\n",
    "    coefs = np.asarray([float(val) for val in values[1:]])\n",
    "    embeddings_index[word] = coefs\n",
    "f.close()\n",
    "\n",
    "print('Found %s word vectors.' % len(embeddings_index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LSTM's\n",
    "\n",
    "## Basic Overview\n",
    "\n",
    "Simple RNN's were certainly better than classical ML algorithms and gave state of the art results, but it failed to capture long term dependencies that is present in sentences . So in 1998-99 LSTM's were introduced to counter to these drawbacks.\n",
    "\n",
    "## In Depth Understanding\n",
    "\n",
    "Why LSTM's?\n",
    "* https://www.coursera.org/learn/nlp-sequence-models/lecture/PKMRR/vanishing-gradients-with-rnns\n",
    "* https://www.analyticsvidhya.com/blog/2017/12/fundamentals-of-deep-learning-introduction-to-lstm/\n",
    "\n",
    "What are LSTM's?\n",
    "* https://www.coursera.org/learn/nlp-sequence-models/lecture/KXoay/long-short-term-memory-lstm\n",
    "* https://distill.pub/2019/memorization-in-rnns/\n",
    "* https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21\n",
    "\n",
    "# Code Implementation\n",
    "\n",
    "We have already tokenized and paded our text for input to LSTM's"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 43496/43496 [00:00<00:00, 183357.18it/s]\n"
     ]
    }
   ],
   "source": [
    "# create an embedding matrix for the words we have in the dataset\n",
    "embedding_matrix = np.zeros((len(word_index) + 1, 300))\n",
    "for word, i in tqdm(word_index.items()):\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if embedding_vector is not None:\n",
    "        embedding_matrix[i] = embedding_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_2 (Embedding)      (None, 1500, 300)         13049100  \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 100)               160400    \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 101       \n",
      "=================================================================\n",
      "Total params: 13,209,601\n",
      "Trainable params: 160,501\n",
      "Non-trainable params: 13,049,100\n",
      "_________________________________________________________________\n",
      "CPU times: user 1.33 s, sys: 1.46 s, total: 2.79 s\n",
      "Wall time: 3.09 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "with strategy.scope():\n",
    "    \n",
    "    # A simple LSTM with glove embeddings and one dense layer\n",
    "    model = Sequential()\n",
    "    model.add(Embedding(len(word_index) + 1,\n",
    "                     300,\n",
    "                     weights=[embedding_matrix],\n",
    "                     input_length=max_len,\n",
    "                     trainable=False))\n",
    "\n",
    "    model.add(LSTM(100, dropout=0.3, recurrent_dropout=0.3))\n",
    "    model.add(Dense(1, activation='sigmoid'))\n",
    "    model.compile(loss='binary_crossentropy', optimizer='adam',metrics=['accuracy'])\n",
    "    \n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:1: UserWarning:\n",
      "\n",
      "The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "9600/9600 [==============================] - 117s 12ms/step - loss: 0.3525 - accuracy: 0.8852\n",
      "Epoch 2/5\n",
      "9600/9600 [==============================] - 114s 12ms/step - loss: 0.2397 - accuracy: 0.9192\n",
      "Epoch 3/5\n",
      "9600/9600 [==============================] - 114s 12ms/step - loss: 0.1904 - accuracy: 0.9333\n",
      "Epoch 4/5\n",
      "9600/9600 [==============================] - 114s 12ms/step - loss: 0.1659 - accuracy: 0.9394\n",
      "Epoch 5/5\n",
      "9600/9600 [==============================] - 114s 12ms/step - loss: 0.1553 - accuracy: 0.9470\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.callbacks.History at 0x7fae84dac710>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(xtrain_pad, ytrain, nb_epoch=5, batch_size=64*strategy.num_replicas_in_sync)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Auc: 0.96%\n"
     ]
    }
   ],
   "source": [
    "scores = model.predict(xvalid_pad)\n",
    "print(\"Auc: %.2f%%\" % (roc_auc(scores,yvalid)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores_model.append({'Model': 'LSTM','AUC_Score': roc_auc(scores,yvalid)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code Explanation\n",
    "\n",
    "As a first step we calculate embedding matrix for our vocabulary from the pretrained GLoVe vectors . Then while building the embedding layer we pass Embedding Matrix as weights to the layer instead of training it over Vocabulary and thus we pass trainable = False.\n",
    "Rest of the model is same as before except we have replaced the SimpleRNN By LSTM Units\n",
    "\n",
    "* Comments on the Model\n",
    "\n",
    "We now see that the model is not overfitting and achieves an auc score of 0.96 which is quite commendable , also we close in on the gap between accuracy and auc .\n",
    "We see that in this case we used dropout and prevented overfitting the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GRU's\n",
    "\n",
    "## Basic  Overview\n",
    "\n",
    "Introduced by Cho, et al. in 2014, GRU (Gated Recurrent Unit) aims to solve the vanishing gradient problem which comes with a standard recurrent neural network. GRU's are a variation on the LSTM because both are designed similarly and, in some cases, produce equally excellent results . GRU's were designed to be simpler and faster than LSTM's and in most cases produce equally good results and thus there is no clear winner.\n",
    "\n",
    "## In Depth Explanation\n",
    "\n",
    "* https://towardsdatascience.com/understanding-gru-networks-2ef37df6c9be\n",
    "* https://www.coursera.org/learn/nlp-sequence-models/lecture/agZiL/gated-recurrent-unit-gru\n",
    "* https://www.geeksforgeeks.org/gated-recurrent-unit-networks/\n",
    "\n",
    "## Code Implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_3\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_3 (Embedding)      (None, 1500, 300)         13049100  \n",
      "_________________________________________________________________\n",
      "spatial_dropout1d_1 (Spatial (None, 1500, 300)         0         \n",
      "_________________________________________________________________\n",
      "gru_1 (GRU)                  (None, 300)               540900    \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 1)                 301       \n",
      "=================================================================\n",
      "Total params: 13,590,301\n",
      "Trainable params: 541,201\n",
      "Non-trainable params: 13,049,100\n",
      "_________________________________________________________________\n",
      "CPU times: user 1.3 s, sys: 1.29 s, total: 2.59 s\n",
      "Wall time: 2.79 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "with strategy.scope():\n",
    "    # GRU with glove embeddings and two dense layers\n",
    "     model = Sequential()\n",
    "     model.add(Embedding(len(word_index) + 1,\n",
    "                     300,\n",
    "                     weights=[embedding_matrix],\n",
    "                     input_length=max_len,\n",
    "                     trainable=False))\n",
    "     model.add(SpatialDropout1D(0.3))\n",
    "     model.add(GRU(300))\n",
    "     model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "     model.compile(loss='binary_crossentropy', optimizer='adam',metrics=['accuracy'])   \n",
    "    \n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:1: UserWarning:\n",
      "\n",
      "The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "9600/9600 [==============================] - 191s 20ms/step - loss: 0.3272 - accuracy: 0.8933\n",
      "Epoch 2/5\n",
      "9600/9600 [==============================] - 189s 20ms/step - loss: 0.2015 - accuracy: 0.9334\n",
      "Epoch 3/5\n",
      "9600/9600 [==============================] - 189s 20ms/step - loss: 0.1540 - accuracy: 0.9483\n",
      "Epoch 4/5\n",
      "9600/9600 [==============================] - 189s 20ms/step - loss: 0.1287 - accuracy: 0.9548\n",
      "Epoch 5/5\n",
      "9600/9600 [==============================] - 188s 20ms/step - loss: 0.1238 - accuracy: 0.9551\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.callbacks.History at 0x7fae5b01ed30>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(xtrain_pad, ytrain, nb_epoch=5, batch_size=64*strategy.num_replicas_in_sync)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Auc: 0.97%\n"
     ]
    }
   ],
   "source": [
    "scores = model.predict(xvalid_pad)\n",
    "print(\"Auc: %.2f%%\" % (roc_auc(scores,yvalid)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores_model.append({'Model': 'GRU','AUC_Score': roc_auc(scores,yvalid)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'Model': 'SimpleRNN', 'AUC_Score': 0.6949714081921305},\n",
       " {'Model': 'LSTM', 'AUC_Score': 0.9598235453841757},\n",
       " {'Model': 'GRU', 'AUC_Score': 0.9716554069114769}]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bi-Directional RNN's\n",
    "\n",
    "## In Depth Explanation\n",
    "\n",
    "* https://www.coursera.org/learn/nlp-sequence-models/lecture/fyXnn/bidirectional-rnn\n",
    "* https://towardsdatascience.com/understanding-bidirectional-rnn-in-pytorch-5bd25a5dd66\n",
    "* https://d2l.ai/chapter_recurrent-modern/bi-rnn.html\n",
    "\n",
    "## Code Implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_4\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_4 (Embedding)      (None, 1500, 300)         13049100  \n",
      "_________________________________________________________________\n",
      "bidirectional_1 (Bidirection (None, 600)               1442400   \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 1)                 601       \n",
      "=================================================================\n",
      "Total params: 14,492,101\n",
      "Trainable params: 1,443,001\n",
      "Non-trainable params: 13,049,100\n",
      "_________________________________________________________________\n",
      "CPU times: user 2.39 s, sys: 1.62 s, total: 4 s\n",
      "Wall time: 3.41 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "with strategy.scope():\n",
    "    # A simple bidirectional LSTM with glove embeddings and one dense layer\n",
    "    model = Sequential()\n",
    "    model.add(Embedding(len(word_index) + 1,\n",
    "                     300,\n",
    "                     weights=[embedding_matrix],\n",
    "                     input_length=max_len,\n",
    "                     trainable=False))\n",
    "    model.add(Bidirectional(LSTM(300, dropout=0.3, recurrent_dropout=0.3)))\n",
    "\n",
    "    model.add(Dense(1,activation='sigmoid'))\n",
    "    model.compile(loss='binary_crossentropy', optimizer='adam',metrics=['accuracy'])\n",
    "    \n",
    "    \n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:1: UserWarning:\n",
      "\n",
      "The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "9600/9600 [==============================] - 322s 34ms/step - loss: 0.3171 - accuracy: 0.9009\n",
      "Epoch 2/5\n",
      "9600/9600 [==============================] - 318s 33ms/step - loss: 0.1988 - accuracy: 0.9305\n",
      "Epoch 3/5\n",
      "9600/9600 [==============================] - 318s 33ms/step - loss: 0.1650 - accuracy: 0.9424\n",
      "Epoch 4/5\n",
      "9600/9600 [==============================] - 318s 33ms/step - loss: 0.1577 - accuracy: 0.9414\n",
      "Epoch 5/5\n",
      "9600/9600 [==============================] - 319s 33ms/step - loss: 0.1540 - accuracy: 0.9459\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.callbacks.History at 0x7fae5a4ade48>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(xtrain_pad, ytrain, nb_epoch=5, batch_size=64*strategy.num_replicas_in_sync)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Auc: 0.97%\n"
     ]
    }
   ],
   "source": [
    "scores = model.predict(xvalid_pad)\n",
    "print(\"Auc: %.2f%%\" % (roc_auc(scores,yvalid)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores_model.append({'Model': 'Bi-directional LSTM','AUC_Score': roc_auc(scores,yvalid)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code Explanation\n",
    "\n",
    "Code is same as before,only we have added bidirectional nature to the LSTM cells we used before and is self explanatory. We have achieve similar accuracy and auc score as before and now we have learned all the types of typical RNN architectures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**We are now at the end of part 1 of this notebook and things are about to go wild now as we Enter more complex and State of the art models .If you have followed along from the starting and read all the articles and understood everything , these complex models would be fairly easy to understand.I recommend Finishing Part 1 before continuing as the upcoming techniques can be quite overwhelming**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Seq2Seq Model Architecture\n",
    "\n",
    "## Overview\n",
    "\n",
    "RNN's are of many types  and different architectures are used for different purposes. Here is a nice video explanining different types of model architectures : https://www.coursera.org/learn/nlp-sequence-models/lecture/BO8PS/different-types-of-rnns.\n",
    "Seq2Seq is a many to many RNN architecture where the input is a sequence and the output is also a sequence (where input and output sequences can be or cannot be of different lengths). This architecture is used in a lot of applications like Machine Translation, text summarization, question answering etc\n",
    "\n",
    "## In Depth Understanding\n",
    "\n",
    "I will not write the code implementation for this,but rather I will provide the resources where code has already been implemented and explained in a much better way than I could have ever explained.\n",
    "\n",
    "* https://www.coursera.org/learn/nlp-sequence-models/lecture/HyEui/basic-models ---> A basic idea of different Seq2Seq Models\n",
    "\n",
    "* https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html , https://machinelearningmastery.com/define-encoder-decoder-sequence-sequence-model-neural-machine-translation-keras/ ---> Basic Encoder-Decoder Model and its explanation respectively\n",
    "\n",
    "* https://towardsdatascience.com/how-to-implement-seq2seq-lstm-model-in-keras-shortcutnlp-6f355f3e5639 ---> A More advanced Seq2seq Model and its explanation\n",
    "\n",
    "* https://d2l.ai/chapter_recurrent-modern/machine-translation-and-dataset.html , https://d2l.ai/chapter_recurrent-modern/encoder-decoder.html ---> Implementation of Encoder-Decoder Model from scratch\n",
    "\n",
    "* https://www.youtube.com/watch?v=IfsjMg4fLWQ&list=PLtmWHNX-gukKocXQOkQjuVxglSDYWsSh9&index=8&t=0s ---> Introduction to Seq2seq By fast.ai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_81e3fa40_7db4_11ea_96d7_0242ac130202row0_col1 {\n",
       "            background-color:  #08306b;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_81e3fa40_7db4_11ea_96d7_0242ac130202row1_col1 {\n",
       "            background-color:  #083471;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_81e3fa40_7db4_11ea_96d7_0242ac130202row2_col1 {\n",
       "            background-color:  #083a7a;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_81e3fa40_7db4_11ea_96d7_0242ac130202row3_col1 {\n",
       "            background-color:  #f7fbff;\n",
       "            color:  #000000;\n",
       "        }</style><table id=\"T_81e3fa40_7db4_11ea_96d7_0242ac130202\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >Model</th>        <th class=\"col_heading level0 col1\" >AUC_Score</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_81e3fa40_7db4_11ea_96d7_0242ac130202level0_row0\" class=\"row_heading level0 row0\" >2</th>\n",
       "                        <td id=\"T_81e3fa40_7db4_11ea_96d7_0242ac130202row0_col0\" class=\"data row0 col0\" >GRU</td>\n",
       "                        <td id=\"T_81e3fa40_7db4_11ea_96d7_0242ac130202row0_col1\" class=\"data row0 col1\" >0.971655</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_81e3fa40_7db4_11ea_96d7_0242ac130202level0_row1\" class=\"row_heading level0 row1\" >3</th>\n",
       "                        <td id=\"T_81e3fa40_7db4_11ea_96d7_0242ac130202row1_col0\" class=\"data row1 col0\" >Bi-directional LSTM</td>\n",
       "                        <td id=\"T_81e3fa40_7db4_11ea_96d7_0242ac130202row1_col1\" class=\"data row1 col1\" >0.966693</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_81e3fa40_7db4_11ea_96d7_0242ac130202level0_row2\" class=\"row_heading level0 row2\" >1</th>\n",
       "                        <td id=\"T_81e3fa40_7db4_11ea_96d7_0242ac130202row2_col0\" class=\"data row2 col0\" >LSTM</td>\n",
       "                        <td id=\"T_81e3fa40_7db4_11ea_96d7_0242ac130202row2_col1\" class=\"data row2 col1\" >0.959824</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_81e3fa40_7db4_11ea_96d7_0242ac130202level0_row3\" class=\"row_heading level0 row3\" >0</th>\n",
       "                        <td id=\"T_81e3fa40_7db4_11ea_96d7_0242ac130202row3_col0\" class=\"data row3 col0\" >SimpleRNN</td>\n",
       "                        <td id=\"T_81e3fa40_7db4_11ea_96d7_0242ac130202row3_col1\" class=\"data row3 col1\" >0.694971</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7fae5a249e10>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Visualization of Results obtained from various Deep learning models\n",
    "results = pd.DataFrame(scores_model).sort_values(by='AUC_Score',ascending=False)\n",
    "results.style.background_gradient(cmap='Blues')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "        <script type=\"text/javascript\">\n",
       "        window.PlotlyConfig = {MathJaxConfig: 'local'};\n",
       "        if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}\n",
       "        if (typeof require !== 'undefined') {\n",
       "        require.undef(\"plotly\");\n",
       "        requirejs.config({\n",
       "            paths: {\n",
       "                'plotly': ['https://cdn.plot.ly/plotly-latest.min']\n",
       "            }\n",
       "        });\n",
       "        require(['plotly'], function(Plotly) {\n",
       "            window._Plotly = Plotly;\n",
       "        });\n",
       "        }\n",
       "        </script>\n",
       "        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "        \n",
       "        \n",
       "            <div id=\"6bedf72d-928a-47b5-83a4-e0269d18dbf4\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>\n",
       "            <script type=\"text/javascript\">\n",
       "                require([\"plotly\"], function(Plotly) {\n",
       "                    window.PLOTLYENV=window.PLOTLYENV || {};\n",
       "                    \n",
       "                if (document.getElementById(\"6bedf72d-928a-47b5-83a4-e0269d18dbf4\")) {\n",
       "                    Plotly.newPlot(\n",
       "                        '6bedf72d-928a-47b5-83a4-e0269d18dbf4',\n",
       "                        [{\"text\": [\"GRU\", \"Bi-directional LSTM\", \"LSTM\", \"SimpleRNN\"], \"title\": {\"position\": \"top center\", \"text\": \"Funnel-Chart of Sentiment Distribution\"}, \"type\": \"funnelarea\", \"values\": [0.9716554069114769, 0.9666928741352547, 0.9598235453841757, 0.6949714081921305]}],\n",
       "                        {\"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}},\n",
       "                        {\"responsive\": true}\n",
       "                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('6bedf72d-928a-47b5-83a4-e0269d18dbf4');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })\n",
       "                };\n",
       "                });\n",
       "            </script>\n",
       "        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = go.Figure(go.Funnelarea(\n",
    "    text =results.Model,\n",
    "    values = results.AUC_Score,\n",
    "    title = {\"position\": \"top center\", \"text\": \"Funnel-Chart of Sentiment Distribution\"}\n",
    "    ))\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Attention Models\n",
    "\n",
    "This is the toughest and most tricky part. If you are able to understand the intiuition and working of attention block , understanding transformers and transformer based architectures like BERT will be a piece of cake. This is the part where I spent the most time on and I suggest you do the same . Please read and view the following resources in the order I am providing to ignore getting confused, also at the end of this try to write and draw an attention block in your own way :-\n",
    "\n",
    "* https://www.coursera.org/learn/nlp-sequence-models/lecture/RDXpX/attention-model-intuition --> Only watch this video and not the next one\n",
    "* https://towardsdatascience.com/sequence-2-sequence-model-with-attention-mechanism-9e9ca2a613a\n",
    "* https://towardsdatascience.com/attention-and-its-different-forms-7fc3674d14dc\n",
    "* https://distill.pub/2016/augmented-rnns/ \n",
    "\n",
    "## Code Implementation\n",
    "\n",
    "* https://www.analyticsvidhya.com/blog/2019/11/comprehensive-guide-attention-mechanism-deep-learning/ --> Basic Level\n",
    "* https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html ---> Implementation from Scratch in Pytorch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transformers : Attention is all you need\n",
    "\n",
    "So finally we have reached the end of the learning curve and are about to start learning the technology that changed NLP completely and are the reasons for the state of the art NLP techniques .Transformers were introduced in the paper Attention is all you need by Google. If you have understood the Attention models,this will be very easy , Here is transformers fully explained:\n",
    "\n",
    "* http://jalammar.github.io/illustrated-transformer/\n",
    "\n",
    "## Code Implementation\n",
    "\n",
    "* http://nlp.seas.harvard.edu/2018/04/03/attention.html ---> This presents the code implementation of the architecture presented in the paper by Google"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BERT and Its Implementation on this Competition\n",
    "\n",
    "As Promised I am back with Resiurces , to understand about BERT architecture , please follow the contents in the given order :-\n",
    "\n",
    "* http://jalammar.github.io/illustrated-bert/ ---> In Depth Understanding of BERT\n",
    "\n",
    "After going through the post Above , I guess you must have understood how transformer architecture have been utilized by the current SOTA models . Now these architectures can be used in two ways :<br><br>\n",
    "1) We can use the model for prediction on our problems using the pretrained weights without fine-tuning or training the model for our sepcific tasks\n",
    "* EG: http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/ ---> Using Pre-trained BERT without Tuning\n",
    "\n",
    "2) We can fine-tune or train these transformer models for our task by tweaking the already pre-trained weights and training on a much smaller dataset\n",
    "* EG:* https://www.youtube.com/watch?v=hinZO--TEk4&t=2933s ---> Tuning BERT For your TASK\n",
    "\n",
    "We will be using the first example as a base for our implementation of BERT model using Hugging Face and KERAS , but contrary to first example we will also Fine-Tune our model for our task\n",
    "\n",
    "Acknowledgements : https://www.kaggle.com/xhlulu/jigsaw-tpu-distilbert-with-huggingface-and-keras\n",
    "\n",
    "\n",
    "Steps Involved :\n",
    "* Data Preparation : Tokenization and encoding of data\n",
    "* Configuring TPU's \n",
    "* Building a Function for Model Training and adding an output layer for classification\n",
    "* Train the model and get the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading Dependencies\n",
    "import os\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.layers import Dense, Input\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.callbacks import ModelCheckpoint\n",
    "from kaggle_datasets import KaggleDatasets\n",
    "import transformers\n",
    "\n",
    "from tokenizers import BertWordPieceTokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LOADING THE DATA\n",
    "\n",
    "train1 = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\")\n",
    "valid = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/validation.csv')\n",
    "test = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/test.csv')\n",
    "sub = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/sample_submission.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Encoder FOr DATA for understanding waht encode batch does read documentation of hugging face tokenizer :\n",
    "https://huggingface.co/transformers/main_classes/tokenizer.html here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fast_encode(texts, tokenizer, chunk_size=256, maxlen=512):\n",
    "    \"\"\"\n",
    "    Encoder for encoding the text into sequence of integers for BERT Input\n",
    "    \"\"\"\n",
    "    tokenizer.enable_truncation(max_length=maxlen)\n",
    "    tokenizer.enable_padding(max_length=maxlen)\n",
    "    all_ids = []\n",
    "    \n",
    "    for i in tqdm(range(0, len(texts), chunk_size)):\n",
    "        text_chunk = texts[i:i+chunk_size].tolist()\n",
    "        encs = tokenizer.encode_batch(text_chunk)\n",
    "        all_ids.extend([enc.ids for enc in encs])\n",
    "    \n",
    "    return np.array(all_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "#IMP DATA FOR CONFIG\n",
    "\n",
    "AUTO = tf.data.experimental.AUTOTUNE\n",
    "\n",
    "\n",
    "# Configuration\n",
    "EPOCHS = 3\n",
    "BATCH_SIZE = 16 * strategy.num_replicas_in_sync\n",
    "MAX_LEN = 192"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization\n",
    "\n",
    "For understanding please refer to hugging face documentation again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3d4284c824c1426f81c105d5c7737f1b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=995526.0, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Tokenizer(vocabulary_size=119547, model=BertWordPiece, add_special_tokens=True, unk_token=[UNK], sep_token=[SEP], cls_token=[CLS], clean_text=True, handle_chinese_chars=True, strip_accents=True, lowercase=False, wordpieces_prefix=##)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# First load the real tokenizer\n",
    "tokenizer = transformers.DistilBertTokenizer.from_pretrained('distilbert-base-multilingual-cased')\n",
    "# Save the loaded tokenizer locally\n",
    "tokenizer.save_pretrained('.')\n",
    "# Reload it with the huggingface tokenizers library\n",
    "fast_tokenizer = BertWordPieceTokenizer('vocab.txt', lowercase=False)\n",
    "fast_tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 874/874 [00:35<00:00, 24.35it/s]\n",
      "100%|██████████| 32/32 [00:01<00:00, 20.87it/s]\n",
      "100%|██████████| 250/250 [00:11<00:00, 22.06it/s]\n"
     ]
    }
   ],
   "source": [
    "x_train = fast_encode(train1.comment_text.astype(str), fast_tokenizer, maxlen=MAX_LEN)\n",
    "x_valid = fast_encode(valid.comment_text.astype(str), fast_tokenizer, maxlen=MAX_LEN)\n",
    "x_test = fast_encode(test.content.astype(str), fast_tokenizer, maxlen=MAX_LEN)\n",
    "\n",
    "y_train = train1.toxic.values\n",
    "y_valid = valid.toxic.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset = (\n",
    "    tf.data.Dataset\n",
    "    .from_tensor_slices((x_train, y_train))\n",
    "    .repeat()\n",
    "    .shuffle(2048)\n",
    "    .batch(BATCH_SIZE)\n",
    "    .prefetch(AUTO)\n",
    ")\n",
    "\n",
    "valid_dataset = (\n",
    "    tf.data.Dataset\n",
    "    .from_tensor_slices((x_valid, y_valid))\n",
    "    .batch(BATCH_SIZE)\n",
    "    .cache()\n",
    "    .prefetch(AUTO)\n",
    ")\n",
    "\n",
    "test_dataset = (\n",
    "    tf.data.Dataset\n",
    "    .from_tensor_slices(x_test)\n",
    "    .batch(BATCH_SIZE)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model(transformer, max_len=512):\n",
    "    \"\"\"\n",
    "    function for training the BERT model\n",
    "    \"\"\"\n",
    "    input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n",
    "    sequence_output = transformer(input_word_ids)[0]\n",
    "    cls_token = sequence_output[:, 0, :]\n",
    "    out = Dense(1, activation='sigmoid')(cls_token)\n",
    "    \n",
    "    model = Model(inputs=input_word_ids, outputs=out)\n",
    "    model.compile(Adam(lr=1e-5), loss='binary_crossentropy', metrics=['accuracy'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Starting Training\n",
    "\n",
    "If you want to use any another model just replace the model name in transformers._____ and use accordingly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f0c6752930d541c68ee287ed3ede1403",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=618.0, style=ProgressStyle(description_…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3f6ec704e4a3424a8448f1f5fd611f61",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=910749124.0, style=ProgressStyle(descri…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_word_ids (InputLayer)  [(None, 192)]             0         \n",
      "_________________________________________________________________\n",
      "tf_distil_bert_model (TFDist ((None, 192, 768),)       134734080 \n",
      "_________________________________________________________________\n",
      "tf_op_layer_strided_slice (T [(None, 768)]             0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 1)                 769       \n",
      "=================================================================\n",
      "Total params: 134,734,849\n",
      "Trainable params: 134,734,849\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "CPU times: user 34.4 s, sys: 13.3 s, total: 47.7 s\n",
      "Wall time: 50.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "with strategy.scope():\n",
    "    transformer_layer = (\n",
    "        transformers.TFDistilBertModel\n",
    "        .from_pretrained('distilbert-base-multilingual-cased')\n",
    "    )\n",
    "    model = build_model(transformer_layer, max_len=MAX_LEN)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train for 1746 steps, validate for 63 steps\n",
      "Epoch 1/3\n",
      "1746/1746 [==============================] - 255s 146ms/step - loss: 0.1221 - accuracy: 0.9517 - val_loss: 0.4484 - val_accuracy: 0.8479\n",
      "Epoch 2/3\n",
      "1746/1746 [==============================] - 198s 114ms/step - loss: 0.0908 - accuracy: 0.9634 - val_loss: 0.4769 - val_accuracy: 0.8491\n",
      "Epoch 3/3\n",
      "1746/1746 [==============================] - 198s 113ms/step - loss: 0.0775 - accuracy: 0.9680 - val_loss: 0.5522 - val_accuracy: 0.8500\n"
     ]
    }
   ],
   "source": [
    "n_steps = x_train.shape[0] // BATCH_SIZE\n",
    "train_history = model.fit(\n",
    "    train_dataset,\n",
    "    steps_per_epoch=n_steps,\n",
    "    validation_data=valid_dataset,\n",
    "    epochs=EPOCHS\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train for 62 steps\n",
      "Epoch 1/6\n",
      "62/62 [==============================] - 18s 291ms/step - loss: 0.3244 - accuracy: 0.8613\n",
      "Epoch 2/6\n",
      "62/62 [==============================] - 25s 401ms/step - loss: 0.2354 - accuracy: 0.8955\n",
      "Epoch 3/6\n",
      "62/62 [==============================] - 7s 110ms/step - loss: 0.1718 - accuracy: 0.9252\n",
      "Epoch 4/6\n",
      "62/62 [==============================] - 7s 111ms/step - loss: 0.1210 - accuracy: 0.9492\n",
      "Epoch 5/6\n",
      "62/62 [==============================] - 7s 114ms/step - loss: 0.0798 - accuracy: 0.9686\n",
      "Epoch 6/6\n",
      "62/62 [==============================] - 7s 110ms/step - loss: 0.0765 - accuracy: 0.9696\n"
     ]
    }
   ],
   "source": [
    "n_steps = x_valid.shape[0] // BATCH_SIZE\n",
    "train_history_2 = model.fit(\n",
    "    valid_dataset.repeat(),\n",
    "    steps_per_epoch=n_steps,\n",
    "    epochs=EPOCHS*2\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "499/499 [==============================] - 41s 82ms/step\n"
     ]
    }
   ],
   "source": [
    "sub['toxic'] = model.predict(test_dataset, verbose=1)\n",
    "sub.to_csv('submission.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End Notes\n",
    "\n",
    "This was my effort to share my learnings so that everyone can benifit from it.As this community has been very kind to me and helped me in learning all of this , I want to take this forward. I have shared all the resources I used to learn all the stuff .Join me and make these NLP competitions your first ,without being overwhelmed by the shear number of techniques used . It took me 10 days to learn all of this , you can learn it at your pace and dont give in , at the end of all this you will be a different person and it will all be worth it.\n",
    "\n",
    "\n",
    "### I am attaching more resources if you want NLP end to end:\n",
    "\n",
    "1) Books\n",
    "\n",
    "* https://d2l.ai/\n",
    "* Jason Brownlee's Books\n",
    "\n",
    "2) Courses\n",
    "\n",
    "* https://www.coursera.org/learn/nlp-sequence-models/home/welcome\n",
    "* Fast.ai NLP Course\n",
    "\n",
    "3) Blogs and websites\n",
    "\n",
    "* Machine Learning Mastery\n",
    "* https://distill.pub/\n",
    "* http://jalammar.github.io/\n",
    "\n",
    "**<span style=\"color:Red\">This is subtle effort of contributing towards the community, if it helped you in any way please show a token of love by upvoting**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "01e83e13daeb4a90badfdbc4b843351e": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "0bbdcf07967d4c6b8b18fefca5c84579": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "Downloading: 100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_d88deb40c4c1467ea6973e00abb46ac0",
       "max": 910749124,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_312beb8a95ae4657b6239df1285b2eb2",
       "value": 910749124
      }
     },
     "1d41db7e9cba40a29ab87476dfc10db1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "312beb8a95ae4657b6239df1285b2eb2": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "3483f71ff80b489abb049062276fd6c5": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "3d4284c824c1426f81c105d5c7737f1b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_866729a4b89740398e66dc920f605431",
        "IPY_MODEL_ef39655751fc4d88ad9b04adfd6db98d"
       ],
       "layout": "IPY_MODEL_01e83e13daeb4a90badfdbc4b843351e"
      }
     },
     "3f6ec704e4a3424a8448f1f5fd611f61": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_0bbdcf07967d4c6b8b18fefca5c84579",
        "IPY_MODEL_a2e8c70e5560488dbfdb8b43d9b247bf"
       ],
       "layout": "IPY_MODEL_3483f71ff80b489abb049062276fd6c5"
      }
     },
     "4bb353d9a6564540802e9304ea9bae86": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "58dea1d37b2240ed8cef94afecfe6890": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "Downloading: 100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_a9c97235c96f4b84ad2935c265cb3175",
       "max": 618,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_1d41db7e9cba40a29ab87476dfc10db1",
       "value": 618
      }
     },
     "8188423157f6440ba33dd05c0dc507c5": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "866729a4b89740398e66dc920f605431": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "Downloading: 100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_8a69c2b832ac47fda9d5c13caae0994f",
       "max": 995526,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_8188423157f6440ba33dd05c0dc507c5",
       "value": 995526
      }
     },
     "8a69c2b832ac47fda9d5c13caae0994f": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "a2e8c70e5560488dbfdb8b43d9b247bf": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_eb41a1972a0c40679625a5b7623307bb",
       "placeholder": "​",
       "style": "IPY_MODEL_bdb5721f9d5d4ddf89db484e53d98228",
       "value": " 911M/911M [00:25&lt;00:00, 36.0MB/s]"
      }
     },
     "a9c97235c96f4b84ad2935c265cb3175": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "bdb5721f9d5d4ddf89db484e53d98228": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "c618f266c4464e22859d3fb6cc06b872": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "d3b6ddb9d56a4626bc1f231c108c57b5": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "d88deb40c4c1467ea6973e00abb46ac0": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "e2837b7b031a464ba795dbea012d8b97": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "e5a85c50c954478da3b576dd7efb01a5": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "eb41a1972a0c40679625a5b7623307bb": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "ef39655751fc4d88ad9b04adfd6db98d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_c618f266c4464e22859d3fb6cc06b872",
       "placeholder": "​",
       "style": "IPY_MODEL_4bb353d9a6564540802e9304ea9bae86",
       "value": " 996k/996k [00:00&lt;00:00, 4.84MB/s]"
      }
     },
     "f0c6752930d541c68ee287ed3ede1403": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_58dea1d37b2240ed8cef94afecfe6890",
        "IPY_MODEL_fe2918aa43cf4ca5958408dcb5810b09"
       ],
       "layout": "IPY_MODEL_e2837b7b031a464ba795dbea012d8b97"
      }
     },
     "fe2918aa43cf4ca5958408dcb5810b09": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_e5a85c50c954478da3b576dd7efb01a5",
       "placeholder": "​",
       "style": "IPY_MODEL_d3b6ddb9d56a4626bc1f231c108c57b5",
       "value": " 618/618 [00:00&lt;00:00, 1.11kB/s]"
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}